Computer Vision
Secure-by-Disguise: A Systematic Evaluation of Image Disguising for Confidential Medical Image Modeling
Jason Rojas, Jiajie He, Yash Patel et al. (+3 more)
Cloud-based deep learning enables large-scale medical image analysis but raises significant
privacy concerns when sensitive patient images are outsourced for model development. Image disguising has recently emerged as a promising
privacy-enhancing technology (PET) that transforms images into visually unintelligible representations while preserving information for downstream learning. We established a unified framework to evaluate representative methods, DisguisedNets and NeuraCrypt, across four
datasets involving classification and semantic
segmentation tasks. Our analysis assessed predictive utility, efficiency, and robustness against reconstruction attacks. Results showed that image disguising performance varies significantly between tasks; while methods preserved utility for medical image classification, they caused substantial degradation in dense semantic
segmentation. Specifically, Randomized Multidimensional Transformation (RMT) offered the optimal balance of performance and security, whereas AES-based disguising severely impacted utility. Furthermore, regression-based reconstruction attacks effective on natural images proved considerably less successful on realistic medical images. These findings provide a systematic assessment of PET suitability for confidential medical AI applications.
Read the full abstract ▾
All authors Jason Rojas, Jiajie He, Yash Patel, Yuechun Gu, Zeyun Yu, Keke Chen
Computer Vision
Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing
Dominick Reilly, Qiyu Wu, Hiromi Wakaki et al. (+2 more)
Multimodal Large
Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, requiring models to operate under a privileged modality setting, where auxiliary modalities are available only during training. While these modalities contain valuable information, existing MLLMs largely fail to leverage them effectively, as they treat modalities as interchangeable inputs rather than sources of complementary supervision. We propose Mixture of Probes (MoP), a novel framework that disentangles modality-specific and modality-general signals within the MLLM, allowing the model to preserve modality-dependent structure while learning transferable representations across modalities. At its core, MoP achieves this through a structured probing mechanism that extracts and organizes information from intermediate representations of a shared modality encoder, rather than relying only on final-layer
alignment as done in existing MLLMs. To support this disentanglement, we further introduce MoP Cross-modal Training (MoP-X), a training strategy for MoP centered around a probe disentanglement loss that prevents probe collapse and encourages cross-modal learning. We evaluate MoP across two domains spanning eight tasks and four modalities under a comprehensive evaluation protocol tailored to the privileged modality setting, where each modality is independently treated as the sole input at inference time. MoP consistently outperforms strong MLLM baselines, achieving up to 65% relative improvement, demonstrating that auxiliary modalities, even when unavailable at inference, can provide substantial gains when effectively leveraged during training. Code, model checkpoints, and evaluation protocols will be made available at https://github.com/Sony/MoP.
Read the full abstract ▾
All authors Dominick Reilly, Qiyu Wu, Hiromi Wakaki, Srijan Das, Yuki Mistufuji
Computer Vision
StereoSplat+: Feed-Forward Stereo Gaussian Splatting with Diffusion-Assisted Progressive Inference
Zihua Liu and Masatoshi Okutomi
Recent advances in 3D Gaussian Splatting (3DGS) have enabled high-quality, render-ready scene representations for novel-view synthesis. However, most existing 3DGS pipelines rely on multi-view observations (or non-causal access to future frames) to achieve sufficient coverage, which is often unavailable in on-device robotics and AR settings where sensing is restricted to a single stereo rig. Recovering a high-quality 3DGS scene from one stereo observation, therefore, remains challenging due to occlusions, limited field of view, and missing geometry. We present StereoSplat+, a
diffusion-enhanced feed-forward framework that enables causal reconstruction from a single stereo pair. Our method builds on two key components. First, we propose StereoSplat, an input-invariant feed-forward 3D Gaussian
estimator that takes a variable number of posed stereo pairs as input and predicts high-quality 3D Gaussians. StereoSplat fuses complementary geometry cues via a cost-volume branch and a triplane-based 3D volume branch and leverages continuous pose encoding to generalize across view counts and camera configurations. Second, since multiple posed stereo pairs are typically unavailable at inference time, we introduce a
diffusion-enhanced one-shot progressive inference scheme called StereoSplat+: starting from one stereo pair, we render novel stereo views from the predicted 3DGS, refine them with a one-step
diffusion enhancer, and feed them back as additional inputs to update the 3DGS. Experiments on the KITTI-360
dataset show that StereoSplat+ improves novel-view rendering quality and geometry accuracy, especially in occluded regions and under strong view extrapolation, outperforming recent feed-forward 3DGS baselines.
Read the full abstract ▾
Computer Vision
Wat3R: Underwater 3D Geometry Learning without Annotations
Jiangwei Ren, Xingyu Jiang, Zijie Song et al. (+4 more)
Estimating 3D geometry in
underwater environments presents unique challenges due to light attenuation, scattering, and the absence of large-scale, high-quality 3D annotations. Pioneering methods rely on massive dense annotations that are impractical in
underwater settings. In this paper, we propose Wat3R, a cross-domain semi-supervised learning framework designed to adapt feed-forward 3D reconstruction models from air to
underwater scenes. Uniquely, our method eliminates the need for any annotated
underwater data following a teacher-student architecture, that learns robust geometry representations merely on abundant unlabeled real
underwater video footage. We also design a cross-view consistency loss that leverages
geometric cues from other views to compensate for the information degradation in the current view caused by water attenuation and scattering. Furthermore, considering the lack of comprehensive evaluation
benchmarks, we construct Water3D, a diverse
dataset covering various water bodies and
underwater scenarios, designed for
geometric task evaluation. Experimental results demonstrate that Wat3R outperforms current state-of-the-art methods in
underwater multi-view depth estimation and point cloud reconstruction. The
dataset and code are available at https://github.com/LSXI7/Wat3R .
Read the full abstract ▾
All authors Jiangwei Ren, Xingyu Jiang, Zijie Song, Wei Xu, Hongkai Lin, Dingkang Liang, Xiang Bai
Computer Vision
ZipDepth: Bringing Lightweight Zero-Shot Monocular Depth Anywhere, on Any Device
Fabio Tosi, Luca Bartolomei, Matteo Poggi et al. (+1 more)
Monocular depth estimation has seen remarkable progress through foundation models achieving robust zero-shot generalization, yet their computational demands place them far beyond the reach of embedded and mobile platforms. Lightweight alternatives exist, but have been developed almost exclusively within single-domain, self-supervised paradigms, failing silently under domain shift. We present ZipDepth, a compact
monocular depth network that bridges this gap by combining an efficient reparameterizable encoder-decoder with large-scale knowledge
distillation from a foundation model over a large multi-domain training set. Comprising just 6.1M parameters, ZipDepth runs at real-time rates from server GPUs to power-constrained devices, achieving the best trade-off between zero-shot accuracy and deployment efficiency among lightweight models across five
benchmarks, taking a significant step towards the accuracy of foundation models with 50x more parameters.
Read the full abstract ▾
All authors Fabio Tosi, Luca Bartolomei, Matteo Poggi, Stefano Mattoccia
Computer Vision
LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models
Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang et al. (+4 more)
Recovering high-quality
video from sparse event streams is a challenging task. Regression methods often blur textures, while existing generative models struggle with long-term stability. We propose LongE2V, a novel approach that leverages pre-trained
video diffusion priors to jointly handle event-based
video reconstruction, prediction, and frame interpolation. By fine-tuning a foundational
video model, our approach achieves high data efficiency and superior perceptual quality. We introduce Autoregressive Unrolling and Adaptive Context Switching to mitigate temporal drift in extremely long sequences. We also propose Reencoding
Alignment with Cross Residual Correction to ensure precise bidirectional consistency during frame interpolation. Furthermore, Event Voxel Density Augmentation ensures robustness across varying sensor resolutions. Extensive experiments on real-world
benchmarks demonstrate that LongE2V outperforms state-of-the-art methods across all three tasks, exhibiting exceptional temporal coherence and zero-shot generalization. Project page: https://cdfan0627.github.io/LongE2V-page/
Read the full abstract ▾
All authors Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang, Chin-Yang Lin, Kun-Ru Wu, Yu-Chee Tseng, Yu-Lun Liu
Computer Vision
Geometry and Gradient-based Partitioning for Panoramic Outdoor Reconstruction
Weijian Chen, Weibo Yao, Yuhang Zhang et al. (+7 more)
Scaling 3D Gaussian Splatting (3DGS) to large outdoor scenes is costly in both data acquisition and computation. Adopting panoramic images with equirectangular projection (ERP) can reduce capture effort via their full $360^{\circ}$ field of view, yet the resulting omnipresent visibility invalidates existing partitioning strategies that rely on local camera frustums, causing block-wise optimization to degenerate into global training. Thus, we propose PanoLOG, a two-stage coarse-to-fine framework equipped with a Geometry and Gradient-based Partitioning Strategy tailored for large-scale panoramic 3DGS reconstruction. In the global coarse stage, PanoLOG leverages sky-sphere modeling and panoramic
monocular depth supervision for reliable geometry, while in the refinement stage, G$^2$PS builds adaptive bounding volumes via parallax-driven uncertainty and assigns cameras via gradient-based importance scoring. Furthermore, we construct Pano360, the first
benchmark on large-scale panoramic
dataset for outdoor scene reconstruction. Extensive experiments demonstrate that G$^2$PS achieves state-of-the-art rendering quality while maintaining scalable, block-parallel training. Our models, training code, and
dataset are publicly available.
Read the full abstract ▾
All authors Weijian Chen, Weibo Yao, Yuhang Zhang, Xiaolin Tang, Guo Wang, Weijun Zhang, Xitong Gao, Yihao Chen, Hongde Qin, Lu Qi
Computer Vision
OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators
Hongyu Liu, Chun Wang, Feng Gao et al. (+6 more)
We propose OPSD-V, an on-policy self-
distillation paradigm for post-training few-step autoregressive (AR)
video diffusion models. Existing few-step AR
video generators can produce long
videos with low latency, but still suffer from error accumulation and weakened motion dynamics during long autoregressive rollout. OPSD-V reduces long-horizon degradation while preserving the original few-step inference path. The key idea is to introduce real long-
video data as temporal context during training and use it to provide dense trajectory-level supervision. Specifically, the student follows the exact inference-time rollout, generating each chunk conditioned on its own previously generated KV cache. In parallel, the teacher is evaluated at the same student-visited denoising states, but uses a cleaner AR-consistent temporal cache in which older history can be replaced by real-
video context. This provides dense denoising-level corrective targets under on-policy AR cache dynamics, without changing the sampler, number of denoising steps, or inference-time cache mechanism. We apply OPSD-V to representative few-step AR
video models, including Self-Forcing and LongLive. Experiments show consistent improvements in visual quality, motion dynamics, and VBenchLong scores. A user study with 10 participants comparing 20
video pairs shows that OPSD-V is preferred over the base models in 66.0% of overall-preference judgments (82.5% excluding ties).
Read the full abstract ▾
All authors Hongyu Liu, Chun Wang, Feng Gao, Xuanhua He, Yue Ma, Ziyu Wan, Yong Zhang, Xiaoming Wei, Qifeng Chen
Computer Vision
Enhancing In-context Panoramic Generation via Geometric-aware Pretraining
Haoran Feng, Ruiyang Zhang, Longyi Zhang et al. (+2 more)
In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream task-specific fine-tuning. To address the lack of large-scale, high-quality training data tailored to in-context panoramic tasks, we propose Canvas360
Dataset, a collection of 1M high-quality paired panoramic samples for style transfer, inpainting, outpainting, and editing, enabling effective supervision across diverse in-context generation scenarios. On the modeling side, Canvas360 enhances text-to-panorama generation through parallel depth generation, velocity circular padding, and similarity loss regularization, enabling the model to learn geometry-aware representations, capture object distortion details, and improve
geometric consistency and global coherence. Furthermore, empowered by strong panoramic priors, Canvas360 enables a unified in-context panoramic generation framework that supports diverse downstream tasks via
token-level concatenation, surpassing prior methods in both task coverage and modeling flexibility. Extensive experiments show that Canvas360 improves panoramic image fidelity, achieving particularly strong performance on the panorama-specific FAED metric and competitive or leading results across the reported quantitative evaluations. More information can be found on our project page: https://zry000.github.io/Canvas360/
Read the full abstract ▾
All authors Haoran Feng, Ruiyang Zhang, Longyi Zhang, Dizhe Zhang, Lu Qi
Computer Vision
OpenCoF: Learning to Reason Through Video Generation
Xinyan Chen, Ziyu Guo, Renrui Zhang et al. (+2 more)
Reasoning has become a core capability for large models, especially when reliable decisions require understanding logical consequences. Recent
video generation models offer a
reasoning path distinct from previous Chain-of-Thought (CoT):
reasoning can unfold through temporally connected frames, known as Chain-of-Frame (CoF)
reasoning. However, existing
video generators are primarily trained on general
video corpora, still lacking diverse supervision and dedicated designs for CoF
reasoning. To address this gap, we introduce OpenCoF, a framework comprising the OpenCoF-17K
dataset, a
reasoning video dataset spanning 11 task families, and Wan-CoF, a fine-tuned
video model for studying whether diverse temporal supervision improves CoF behavior. Across four
video reasoning benchmarks, Wan-CoF achieves considerable gains over the Wan2.2-I2V-A14B baseline. Building on this, we empirically explore more advanced designs for CoF capabilities, i.e., equipping the model with visual and textual
reasoning tokens. This mechanism respectively captures low-level visual cues and high-level semantic priors for spatial and temporal
reasoning. Through performance comparisons and
attention analysis, we examine how these
tokens contribute across model depth, denoising steps, space, and time. Our results suggest that stronger
video reasoning requires both broad temporal supervision and explicit mechanisms for organizing intermediate
reasoning state. We open-source the
dataset, model, and code to facilitate future research on
reasoning-oriented
video generation.
Read the full abstract ▾
All authors Xinyan Chen, Ziyu Guo, Renrui Zhang, Dongzhi Jiang, Hongsheng Li
Computer Vision · Cross-listed from cs.AI
AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding
Siddharth Damodharan, Radhika Gupta, Ali Alshami et al. (+2 more)
Recent advances in
Vision-
Language Models, Large
Language Models, and
Multimodal Large
Language Models have improved autonomous driving tasks such as scene understanding, decision making, trajectory prediction, and visual question answering. However, evaluating whether these models can reliably reason about safety-critical incidents remains challenging. To address this gap, we present AUTOPILOT-VQA, an incident-centric visual question answering
benchmark for dashcam
video understanding. The
dataset evaluates different systems through structured questions designed around real-world driving incidents and near-incidents. The
benchmark covers diverse safety-relevant categories, including weather and lighting conditions, traffic environment, road layout, road surface state, signage, involved entities, accident occurrence, impact location, and avoidability-related
reasoning. By requiring models to answer grounded questions about both contextual scene properties and event-level incident details, AUTOPILOT-VQA moves beyond object recognition toward temporally grounded, safety-aware
reasoning. The
dataset is released as part of the AUTOPILOT CVPR 2026 competition and provides a standardized
benchmark for assessing the reliability of autonomous driving systems in different scenarios. Our
benchmark support developments for more interpretable, robust, and safety-conscious
vision-
language systems for real-world autonomous driving.
Read the full abstract ▾
All authors Siddharth Damodharan, Radhika Gupta, Ali Alshami, Ryan Rabinowitz, Jugal Kalita
Computer Vision · Cross-listed from cs.GR
ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation
Kaifeng Zhao, Mathis Petrovich, Haotian Zhang et al. (+3 more)
Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and
humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text
prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body
embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive
transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture
dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D
benchmark and the large-scale, high-fidelity Bones Rigplay
dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary
video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.
Read the full abstract ▾
All authors Kaifeng Zhao, Mathis Petrovich, Haotian Zhang, Tingwu Wang, Siyu Tang, Davis Rempe
Computer Vision
WaspMOT: A Benchmark for Long-Term Multi-Object Tracking of Trichogramma Wasps
Tomasz Stanczyk, Yuan Gao, Hardik Agarwal et al. (+4 more)
Multi-object tracking (MOT) has achieved strong performance on
benchmarks dominated by short
video sequences. However, such
datasets do not adequately evaluate long-term identity preservation, where objects must be tracked consistently over extended durations. We introduce WaspMOT, a
benchmark designed to address this gap through long-duration tracking of Trichogramma wasps in controlled ecological experiments. The
dataset contains 10 sequences of approximately 12,000 frames each (over 8 minutes at 25 FPS), with dense MOTChallenge annotations and oracle detections to isolate association performance. Unlike existing
benchmarks, WaspMOT forms a closed-set tracking scenario where all individuals remain present throughout the sequence, requiring consistent identity assignment across thousands of frames despite abrupt jumps, occlusions, and highly similar appearance. We establish a
benchmark by evaluating five tracking-by-detection methods, including ByteTrack, BoT-SORT, C-BIoU, OC-SORT, and McByte, under a unified protocol. Results show that all methods suffer from significant trajectory fragmentation, highlighting the difficulty of long-term identity preservation even with perfect detections. A simple spatial tracklet stitching baseline consistently improves performance, indicating that substantial gains remain possible. WaspMOT provides a new
benchmark for studying long-term association and reveals limitations of current tracking approaches that are not observable on conventional
datasets. The
benchmark will be made publicly available at the project repository: https://github.com/tstanczyk95/WaspMOT/ .
Read the full abstract ▾
All authors Tomasz Stanczyk, Yuan Gao, Hardik Agarwal, Seongro Yoon, Tiantao Zhang, Vincent Calcagno, Francois Bremond
Computer Vision
Pose-to-Biomechanics: Bridging 3D Human Pose Estimation and Biomechanical Attribute Prediction
Ayda Eghbalian and Kevin Desai
Recent progress in 3D human pose estimation has made markerless recovery of skeletal motion increasingly accurate and scalable. However, most pose
estimators remain optimized for
geometric keypoint accuracy, while many real-world applications in rehabilitation, sports science, ergonomics, and clinical movement analysis require biomechanical quantities that describe how the body moves, loads, and activates. In this work, we propose BioModule, a lightweight plug-in temporal
transformer that attaches downstream of any 3D pose
estimator and predicts biomechanical attributes from standard 17-joint 3D skeletons. BioModule is
estimator-agnostic and requires no modification of the upstream pose model, enabling existing pose
estimators to be extended toward physically interpretable motion analysis. To train and evaluate BioModule, we construct a large-scale aligned
dataset pairing Human3.6M
video and 3D keypoints with the biomechanical label space of Human3.6Mplus. We establish and verify anatomical correspondence between coordinate systems of the two
datasets, enabling frame-accurate cross-modal supervision. Using this aligned supervision, BioModule predicts biomechanical quantities. We further
benchmark BioModule across seven state-of-the-art 3D pose
estimators, providing the first systematic analysis of how upstream pose estimation quality propagates to downstream biomechanical prediction fidelity. The results position BioModule as a compact, modular bridge between
vision-based pose estimation and biomechanically meaningful human motion analysis.
Read the full abstract ▾
Computer Vision
LTM: Large-scale Terrain Model for Wildfire-prone Landscapes
Xiao Fu, Yue Hu, Meida Chen et al. (+2 more)
Accurate 3D terrain maps are essential for emergency response when assessing wildfire hazards. However, wildfire-prone regions often span vast areas where conventional reconstruction methods underperform. Airborne
LiDAR systems provide high-resolution terrain data, but they are expensive and infrequently updated. Image-based methods offer a lower-cost alternative, but struggle due to sparse visual features and limited image overlap. We propose a multi-modal reconstruction framework leveraging outdated Digital Elevation Models (DEMs) as
geometric priors for image-based 3D reconstruction. Our key innovation is physics-based pixel-pixel
alignment between images and DEM data, dramatically reducing computational complexity by eliminating expensive feature matching procedures. To validate our approach, we developed a large-terrain simulator based on a real wildfire-prone area, generating realistic images enabling a comprehensive evaluation. Given posed images and legacy DEMs, our method produces high-fidelity depth maps while maintaining real-time performance. We find significant improvements in reconstruction accuracy and computational efficiency over existing techniques, offering a scalable solution for wildfire response.
Read the full abstract ▾
All authors Xiao Fu, Yue Hu, Meida Chen, Peter Anthony Beerel, Barath Raghavan
Computer Vision
HumanForge: A Human-Centric Deepfake Video Benchmark with Multi-Agent Forgery Rationales
Wenbo Xu, Zhimin Chen, Xiaojie Liang et al. (+3 more)
Rapid advancements in
video diffusion models and temporal editing tools have enabled the generation of highly realistic human-centric
videos, presenting unprecedented challenges to digital content forensics. Existing
benchmarks primarily focus on face-swapping or global text-to-
video synthesis, overlooking the crucial dimensions of
multimodal alignment and complex human-object or human-human interactions. To address these limitations, we introduce HumanForge, a unified, large-scale, and multi-paradigm human-centric
video forgery
benchmark containing over 18,000 synthesized
videos across four distinct scenarios: audio-driven, pose-driven, semantic-driven, and interaction. To construct and annotate this
dataset without labor-intensive manual labeling or blind monolithic prompting, we propose Gen2Anno (Generation-to-Annotation), a cooperative multi-
agent pipeline. Gen2Anno orchestrates six specialized
agents-ranging from driving asset profiling to MoE-based reference analysis and closed-loop verification-to dynamically execute
video synthesis and produce structured annotations containing binary authenticity labels, generative model attribution, and natural-
language contrastive forgery rationales. By systematically contrasting expected states derived from generation provenance with actual visual observations, the framework generates logically grounded forensic
reasoning chains. Extensive
benchmarks using state-of-the-art traditional detectors and
Vision-
Language Models demonstrate the significant challenges of cross-generator generalization, perturbation robustness, and explainable
reasoning on HumanForge. The code and
dataset will be publicly released.
Read the full abstract ▾
All authors Wenbo Xu, Zhimin Chen, Xiaojie Liang, Hengrui Liu, Ziqi Sheng, Wei Lu
Computer Vision
SAM-MT: Real-Time Interactive Multi-Target Video Segmentation
Ruiqi Shen, Chang Liu and Henghui Ding
Modern
Video Object
Segmentation (VOS) involves tracking and segmenting user-specified targets. While recent approaches have achieved remarkable performance in single-target scenarios, extending them to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced frame rates (FPS) with unbounded latency as target count increases. Built upon Segment Anything 2 (SAM2), we propose SAM-MT, which addresses this by transforming the model into an interactive framework for real-time Multi-Target
video segmentation. SAM-MT uses explicit queries to represent different individual targets, in parallel with a shared representation for global context. It employs decoupled masked
attention to keep individual identities distinct from cross-target interference, and sparse
memory for stable temporal
evolution, along with specialized strategies for occlusion handling and overlap prevention. SAM-MT successfully decouples latency from the number of targets, achieving real-time speed on par with single-target baselines (>36 FPS for 10 targets) while maintaining SAM2's robust
video segmentation performance.
Read the full abstract ▾
Computer Vision
Multi-Resolution Feature Stem for Diabetic Retinopathy lesion segmentation
Indranil Dutta and Taehee Jeong
Diabetic Retinopathy (DR) is a leading cause of preventable blindness worldwide, requiring automated lesion
segmentation using deep learning models for early detection and monitoring. However, DR lesions vary dramatically in size from tiny microaneurysms to large hemorrhages and exudates. This variability creates conflicting demands on the model architecture and input resolution, posing a challenge for effective design. This work investigates the impact of input resolution on different lesion types. Through systematic experimentation with multiple architectures (U-Net, UNet++,
Vision Transformers, DeepLabV3+) at $512 \times 512$ and $1024 \times 1024$ resolutions, we identify a critical, counter-intuitive phenomenon where increasing input resolution has opposing effects on different lesion types. We demonstrate that while higher resolution is essential for resolving fine-grained microaneurysms, it can unexpectedly degrade performance on larger hemorrhages. This finding challenges the common assumption that higher resolution is uniformly beneficial. To address this, we propose a novel Multi-Resolution Feature Stem, an input-level pyramid integrated with a UNet++ backbone. This architecture processes multiple scales in parallel, capturing fine-grained details without sacrificing contextual information. This work contributes crucial empirical evidence of this complex, resolution-dependent behavior and a practical, parameter-efficient architecture that successfully resolves this trade-off.
Read the full abstract ▾
Computer Vision
Do Transformations Reveal the Truth? Generative Residual Learning for Generalized AI-Generated Image Detection
Kutub Uddin, Nusrat Tasnim, Awais Khan et al. (+2 more)
The rapid advancement of generative AI has enabled the creation of highly realistic deepfake media, posing significant threats, including misinformation, digital identity theft, fraud, and manipulation of public opinion. AI-generated image (AIGI) detection is reliably challenging due to the diversity of generative methods and the subtle artifacts they leave behind. In this work, we propose GenRes, a novel framework for generative residual learning via a neural tensor network, which models fine-grained relational features between original and transformed samples to enhance generalization. To address scenarios involving multiple generative transformations, we introduce GenRes++, which employs a learnable
attention mechanism to aggregate relational features across multiple transformed samples and enables the model to focus on the most informative cues. Both models leverage PE-Core as a feature extractor, providing generalized and semantically rich
embeddings that improve cross-domain performance and enable the detection of AIGI generated by unseen methods. Comprehensive experiments on multiple
benchmark datasets demonstrate that the proposed GenRes++ approach outperforms existing methods.
Read the full abstract ▾
All authors Kutub Uddin, Nusrat Tasnim, Awais Khan, Mohammad Umar Farooq, Khalid Malik
Computer Vision · Cross-listed from cs.RO
Native Video-Action Pretraining for Generalizable Robot Control
Qihang Zhang, Lin Li, Luyao Zhang et al. (+26 more)
The advent of
video-action models offers a promising path for
robot control. Nevertheless, we argue that repurposing
video generative models designed for digital content creation is inherently inadequate for physical environments. To bridge this gap, we present LingBot-VA 2.0, a
video-action foundation model built from the ground up for embodiment. Four core design principles showcase its
evolution from LingBot-VA. (1) Departing from traditional reconstruction-focused VAEs, we introduce a semantic visual-action tokenizer, which aligns visual representations with both semantics and actions, improving instruction following and action precision in subsequent policy learning. (2) Given the strictly causal nature of temporal dynamics, we adopt a causal pretraining paradigm, training from scratch to circumvent the catastrophic forgetting that frequently occurs when adapting bidirectional architectures. (3) To meet the demands of high-frequency inference, our model employs a sparse MoE backbone, expanding model capacity without compromising efficiency. (4) Real-time closed-loop control is realized through an enhanced asynchronous inference scheme, which predicts future latents in parallel with action execution while re-grounding each rollout on the latest observation via learned forward dynamics. Real-world deployment validates LingBot-VA 2.0 as a robust foundation model, as evidenced by its few-shot generalization across complex manipulation tasks.
Read the full abstract ▾
All authors Qihang Zhang, Lin Li, Luyao Zhang, Shuai Yang, Yiming Luo, Shuaiting Li, Ruilin Wang, Junke Wang, Jiahao Shao, Gangwei Xu, Jiaming Zhou, Yishu Shen, Yudong Jin, Fangyi Xu, Shuailei Ma, Jiaqi Liao, Guanxing Lu, Zifan Shi, Yongkun Wen, Yujie Zhao, Weixuan Tang, Xinyang Wang, Chaojian Li, Jiapeng Zhu, Ka Leong Cheng, Nan Xue, Xing Zhu, Yujun Shen, Yinghao Xu
Computer Vision
When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities
Weiduo Liao, Yunqiao Yang and Ying Wei
Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept. However, in
vision-
language models (VLMs), vanilla SAEs struggle to learn modality-consistent concepts, with concepts often exhibiting fragmented coverage (i.e., disjoint regions) in the visual modality. To address this challenge, we propose a Structured Sparse AutoEncoder ($S^2AE$) that enforces concept consistency from both semantic and spatial perspectives in the visual modality. Specifically, we group image patches based on
Transformer attention similarity and spatial proximity, and introduce a structured sparsity regularization when training the vanilla SAE. The regularization consists of exclusive sparsity for inter-group concept disentanglement and group sparsity for intra-group concept consistency, which drives the latent
neurons by SAEs to specialize in distinct, semantically grounded concepts. Evaluated on the Qwen2.5-VL-7B-Instruct model, the method achieves 6.06% average improvement in semantic
alignment (mIoU) and 60.81 in representational efficiency (lower l0 norm) while maintaining near-perfect reconstruction fidelity with an Explained Variance above 99%. Cross-modal analysis further demonstrates that $S^2AE$ enhances neuronal monosemanticity by this visual structural prior, achieving a 3.08% average gain in semantic consistency and a 2.37% average gain in monosemanticity scores for both modalities of
multimodal features, thereby fostering more coherent and disentangled representations.
Read the full abstract ▾
Computer Vision
Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learning
Yiyang Fang, Pei Fu, Jinjie Li et al. (+7 more)
Multimodal Large
Language Models (MLLMs) often follow a fixed Think-then-Answer paradigm, which is inefficient in heterogeneous multitask settings because simple inputs may not require explicit
reasoning while difficult ones can benefit substantially from it. Learning when to think is also unstable during post-training, where imbalanced rollouts can drive the model toward always-thinking or always-direct behavior. We propose Switch-Reasoner, a GRPO-based framework that learns to adaptively select
reasoning modes for MLLMs. It treats thinking as a virtual tool invocation and allows the model to either answer directly or invoke explicit
reasoning before answering. To stabilize this decision, we introduce a dual-level regulation mechanism that balances the overall use of Thinking Mode and Direct Mode while providing sample-level supervision based on the relative benefit of the two choices. Experiments on 11
multimodal tasks show that Switch-Reasoner reduces unnecessary
reasoning while maintaining strong performance, achieving a better accuracy-efficiency trade-off.
Read the full abstract ▾
All authors Yiyang Fang, Pei Fu, Jinjie Li, Jian Liang, Wenke Huang, Ruijie Luo, Shaojie Zhang, Jian Luan, Yi R. Fung, Mang Ye
Computer Vision
VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval
ZhiXin Sun
Open-vocabulary object detection and
segmentation aim to recognize arbitrary objects beyond predefined categories. Although recent
vision-
language and reference-based approaches have significantly advanced this field, they often rely on text
prompts, limited visual examples, or expensive feature matching procedures, making them difficult to scale to large and continuously expanding object repositories. In this work, we propose VocaDet, a sample-driven open-vocabulary object detection and
segmentation framework that learns object concepts directly from user-provided positive and negative sample collections without model retraining. The key idea is to transform continuous visual representations into discrete visual vocabularies and perform efficient
retrieval-based recognition through a scalable vector database. Specifically, we employ DINOv3 as the visual feature extractor and apply agglomerative clustering with adaptive clustering sensitivity to generate multi-granularity visual
tokens. These visual
tokens, together with position-debiased representations and spatial
topology information, are stored as expandable object
memories in a vector database. During inference, query images are converted into visual
tokens and efficiently matched against the stored object
memories for object localization and
segmentation. Furthermore, a background filtering mechanism is introduced to remove frequently occurring background patterns and reduce redundant
retrieval operations in practical fixed-camera scenarios. Experiments on the UA-DETRAC
dataset demonstrate that VocaDet achieves effective open-vocabulary detection performance without conventional detector training, while supporting continuously expandable recognition capability as additional positive and negative samples are accumulated.
Read the full abstract ▾
Computer Vision
Whareformer: Learning to Track What is Where in Long Egocentric Videos
Jacob Chalk, Saptarshi Sinha, Dima Damen et al. (+2 more)
The recently established 'Out of Sight, Not out of Mind' (OSNOM) task for
egocentric videos focuses on tracking objects that are moved by the camera wearer, online, maintaining knowledge of instance locations throughout the
video even when they leave the field of view or become heavily occluded. In this paper, we propose the first learning-based solution to the OSNOM task: Whareformer, a
transformer-based model with two components: an updatable
memory of established tracks and a track assignment module that associates observations with existing tracks in a feed-forward manner. Whareformer jointly reasons over evolving object appearance (what) and updated 3D location (where), and employs a dedicated New Track
token to reason about novel objects. Thanks to its design choices of using relative distances and evolving track representations, Whareformer is trained on a small set of 56
videos but achieves SOTA performance on 260 long test
videos from three
datasets: EPIC-KITCHENS-100 (unseen
videos), IT3DEgo, and HD-EPIC, with significant absolute improvements over prior work.
Read the full abstract ▾
All authors Jacob Chalk, Saptarshi Sinha, Dima Damen, Yannis Kalantidis, Diane Larlus
Computer Vision
Beyond wheelchairs and blindfolds: Investigating disability stereotypes in T2I models with INCLUDE-BENCH
Sophia Lichtenberg, Albert Gatt and Judith Masthoff
Text-to-image (T2I) models have been shown to exhibit social biases. Prior work has mainly focused on gender, skin tone, and cultural representation within restricted occupational associations, and emerging
benchmarks increasingly incorporate these dimensions. However, disability remains systematically underexplored. Current evaluation practices often fail to align with sociologically grounded definitions of stereotyping, limiting principled assessment of representational harms toward people with disabilities (PWD). To address this, we introduce INCLUDE-BENCH, the first large-scale
benchmark for evaluating disability-related bias in T2I models. INCLUDE-BENCH comprises 119K generated images based on
prompt design across multiple bias dimensions and both static and dynamic contexts. We evaluate 15 open-source and 2 closed-source models. Our key findings reveal that: (1) mobility-impaired and default disability
prompts predominantly yield wheelchair depictions across all models; (2) disability-conditioned generations consistently exhibit less diversity; (3) stereotypical portrayals demonstrate stronger disability-text
alignment; and (4) we introduce the Stereotype Content Model (SCM) Score, demonstrating that T2I models reflect real-world stereotypical associations.
Read the full abstract ▾
Computer Vision
Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
Masatoshi Tateno, Alexandros Stergiou, Risa Shinoda et al. (+2 more)
Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing
video-
language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than
reasoning from the appearance and dynamics of hands and objects themselves. To address this limitation, we propose a new learning paradigm that combines (i) hand-object masked training, which enables robust
reasoning from partial hand or object observations, and (ii) an HOI-dynamics-aware decoder that explicitly learns hand- and object-centric
embeddings through auxiliary predictions of their locations and semantics, enhancing sensitivity to both cues. To systematically evaluate such cue-specific
reasoning, we introduce Cue-Isolated HOI (CI-HOI), a new evaluation that assesses models' ability to predict actions from hand- and object-related cues independently. To enable CI-HOI, we curate the DEHOI testbed, which separates hand- and object-related observations for disentangled HOI evaluation through inpainting. Using DEHOI, we demonstrate both quantitatively and qualitatively that our training strategy exploits hand- and object-centric information more effectively than existing models. Our approach improves over existing models on DEHOI, standard action recognition, object state recognition, and even
robot manipulation action recognition, leading to more robust HOI understanding.
Read the full abstract ▾
All authors Masatoshi Tateno, Alexandros Stergiou, Risa Shinoda, Yoichi Sato, Dima Damen
Computer Vision · Cross-listed from cs.LG
Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures
Hafsa Mateen, Radu Timofte and Dmitry Ignatov
Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While classical AutoML approaches often treat the scheduler as a secondary hyperparameter, we systematically investigate its impact on classification accuracy across a diverse pool of architectures. We evaluated 30 representative architectures from convolutional and
transformer families within the LEMUR neural network
dataset. Through automated source-code injection, we applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10. Our best configuration achieved a top-1 accuracy of 86.45%, with 237 variants exceeding 80%. The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies. The resulting accuracy landscape, contributed to the LEMUR nn-
dataset, provides a practical reference for principled scheduler selection.
Read the full abstract ▾
Computer Vision
CT-CLIP Representations for Multimodal Lung Cancer Survival Prediction
Sofie Allgöwer, Mikael Johansson, Andreas Hallqvist et al. (+4 more)
Accurate prognosis prediction is important for treatment planning in lung cancer, but deep learning-driven survival modelling is often limited by the scarcity of curated imaging cohorts with reliable outcome data. This study evaluates whether representations from a domain-specific foundation model can be used for
multimodal survival prediction in data-constrained clinical settings. We assess the foundation model CT-CLIP as a feature extractor for pretreatment computed
tomography images and clinical variables from 242 diagnosed lung cancer patients. The evaluation includes adaptation strategies based on frozen encoders, full fine-tuning, and low-rank adaptation, together with modality ablations and comparisons with clinical and
multimodal baselines. The results show that a frozen CT-CLIP model combined with a trainable lightweight survival head outperforms the clinical baseline and achieves comparable or improved performance relative to other
multimodal approaches, and separates patients into clinically meaningful high- and low-risk groups.
Read the full abstract ▾
All authors Sofie Allgöwer, Mikael Johansson, Andreas Hallqvist, Jonas Andersson, Åse Johnsson, Ida Häggström, Jennifer Alvén
Computer Vision
Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing
Feng Wang, Canmiao Fu, Zhipeng Huang et al. (+3 more)
Recent unified
multimodal models show a single architecture can jointly perform
vision/
language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon
multimodal dialogue due to visual
token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured
Multimodal Agent that externalizes visual information into an Episodic Visual
Memory and selectively reactivates relevant episodes during
reasoning. The
agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive
Retrieval Engine for cross-turn
memory retrieval, and a
Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level
retrieval supervision in existing
datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained
retrieval annotations, enabling reinforcement learning to optimize abstraction and
retrieval policies. We also construct a long-horizon visual-dialogue
benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B
agent achieves 91.4%
retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured
Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent
multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured
memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon
multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/
Read the full abstract ▾
All authors Feng Wang, Canmiao Fu, Zhipeng Huang, Chen Li, Jing Lyu, Ge Li
Computer Vision
VEGAS: Human-Aligned Video Caption Evaluation via Gaze
Shenghui Chen, Po-han Li, Ximeng Sun et al. (+5 more)
Vision-
language models excel at
video captioning, yet typically generate descriptions that fail to capture individual viewers'
attention. We propose VEGAS (
Video caption Evaluation via GAze Score), a training-free metric that leverages test-time gaze to sample personalized,
attention-aligned text. It is a cross-modal, information-theoretic metric that quantifies how well a candidate caption matches a viewer's focus. To evaluate VEGAS, we curate a
dataset of
egocentric activities and instructional slides paired with synchronized gaze and reference annotations. We then select captions based on VEGAS via rejection sampling without model retraining. Experiments show that VEGAS-selected captions align significantly better with human focus and improve downstream caption-to-
video retrieval, demonstrating the practical utility of incorporating viewer
attention during inference.
Read the full abstract ▾
All authors Shenghui Chen, Po-han Li, Ximeng Sun, Shijia Yang, Emad Barsoum, Zicheng Liu, Sandeep Chinchali, Ufuk Topcu