Computer Vision
T-CLIP: Enabling Thermal Perception for Contrastive Language-Image Pretraining
Tayeba Qazi, Ayush Maheshwari, Prerana Mukherjee et al. (+1 more)
Thermal imaging offers a powerful alternative to visible-spectrum
vision under challenging conditions such as low illumination and adverse weather, yet foundational
vision-
language models like CLIP fail to align thermal images with textual descriptions due to a fundamental thermal perception gap. We identify three major challenges: the lack of captioned thermal
datasets, the inability of standard LLMs to reason about thermal phenomena, and a key representational challenge in thermal imaging where global scene context and object-level heat signatures conflict when learned together in a single
embedding space. To address these, we introduce IR-Cap, the first physics-aware thermal captioning pipeline and
dataset providing complementary global and fine-grained thermal descriptions across three public
benchmarks, and T-CLIP, a decoupled dual-LoRA framework that independently adapts CLIP for scene-level and object-level thermal understanding. T-CLIP achieves consistent improvements over all baselines across three thermal
benchmarks in cross-modal
retrieval, and we provide an exploratory demonstration of its applicability to text-conditioned thermal image generation.
Read the full abstract ▾
All authors Tayeba Qazi, Ayush Maheshwari, Prerana Mukherjee, Brejesh Lall
Computer Vision · Cross-listed from cs.RO
SKIP: Sparse Keyframe Interpolation Paradigm for Efficient Embodied World Models
Ziheng He, Yixiang Chen, Ning Yang et al. (+11 more)
Embodied world models have emerged as a promising paradigm in robotics by predicting how
robot actions affect the surrounding scene. However, the rollout inference remains computationally expensive in pixel space, as long-horizon manipulation
videos typically have to be generated frame by frame. This cost cannot be easily reduced by indiscriminately dropping frames, since downstream policies rely on complete preservation of sparse task-relevant events such as approach, contact, grasp, and release. To address this challenge, we propose Sparse Keyframe Interpolation Paradigm (SKIP), an event-preserving sparse-to-dense framework that avoids dense frame-by-frame generation. SKIP first identifies task-relevant keyframes by leveraging
robot-aware
multimodal features. It then synthesizes only these keyframes with a sparse
video diffusion model. A learned gap predictor and an action-conditioned interpolator subsequently reconstruct the missing intervals according to the
robot actions. On LIBERO, SKIP generates dense rollouts $4.16\times$ faster than a dense baseline while improving visual fidelity and reducing aggregate FVD by $89.0\%$. Importantly, SKIP-generated
videos are effective policy-training data. Even when they fully replace real demonstrations, $\pi_{0.5}$ success drops only $1.3$ pp in LIBERO simulation and $6.7$ pp on the real
robot, whereas fully dense frame-by-frame generation collapses by $48$ to $58$ pp.
Read the full abstract ▾
All authors Ziheng He, Yixiang Chen, Ning Yang, Zhanqian Wu, Qisen Ma, Yuan Xu, Jiabing Yang, Peiyan Li, Xiangnan Wu, Xiaofeng Wang, Zheng Zhu, Jing Liu, Nianfeng Liu, Yan Huang
Computer Vision
TAP-JEPA: Frozen Future-Latent Probing and Two-Stage Score Fusion for EPIC-KITCHENS-100 Action Anticipation
Chaoyang Wang and Lexuan Xu
This report presents TAP-JEPA, our runner-up submission to the EPIC-KITCHENS-100 (EK-100) Action Anticipation Challenge at EgoVis 2026. The task is to anticipate the next verb, noun, and verb-noun action from an
egocentric clip that ends before the target action begins. Instead of fine-tuning a large
video backbone, TAP-JEPA builds a compact anticipation model on frozen V-JEPA 2.1 features: a ViT-G/384 encoder extracts visible pre-action
tokens, the pre-trained latent predictor estimates near-future
tokens from the observed context, and both
token groups are fused by attentive probes with task-specific queries for verbs, nouns, and action pairs. For the final submission, we expand supervised training with the official training split and most of the validation split, reserving a small subset for sanity checks and qualitative inspection, and adopt a two-stage score fusion that first averages eight independently initialized probe replicas within each epoch and then merges candidates from epochs 12-20 with field-dependent weights. On the official open-testing leaderboard, our sunshinesky entry achieves 27.91 percent overall action Mean Top-5 Recall (MT5R), ranking second and only 0.04 percentage points behind the top score.
Read the full abstract ▾
Computer Vision
Collaborative Few-Step Distillation and Low-Bit Quantization for Wan2.2 Dual-Expert Video Diffusion Models
Jinyang Du, Shenghao Jin, Ziqian Xu et al. (+5 more)
Large
video diffusion models achieve strong visual quality but remain expensive to deploy because each sample requires many denoising steps and a large resident parameter footprint. This paper studies a deployment-oriented compression pipeline for Wan2.2-T2V-A14B by combining few-step distribution-matching
distillation with low-bit quantization. The pipeline follows the model's dual-expert denoising route, calibrates the high-noise and low-noise branches separately, protects sensitive entrance layers, and uses HiF4-style low-bit representation to improve dynamic-range coverage. Quantization is calibrated on the distilled few-step student rather than on the original long-step trajectory, reducing activation-distribution mismatch during inference. The proposed co-design keeps the quantized model close to the same-step full-precision model and surpasses the original full-precision baseline at 8 and 20 steps on average. The 20-step setting gives the best quality-efficiency trade-off in the tested configurations.
Read the full abstract ▾
All authors Jinyang Du, Shenghao Jin, Ziqian Xu, Ruihao Gong, Shiqiao Gu, Yang Yong, Jinyang Guo, Xianglong Liu
Computer Vision
An Attribute-Based Measure of Video Complexity
Aditya Sarkar, Yi Li, Zihao Wang et al. (+8 more)
A new framework for the estimation of the complexity posed by
video-question pairs to
video-LLMs,
Video Attribute-Based Complexity (VideoABC), is proposed.
Video complexity is defined as the probability of failure of a
video-LLM for a given
video-question pair. VideoABC is a non-parametric complexity measure, using a reference
video dataset and a pre-defined vocabulary of
video attributes informative of complexity, e.g. the scene complexity or the speed of the
video event relevant to the question. In a training phase, reference
videos are projected into the space of these attributes, which is then quantized. The expected ABC of each quantization
cell is then computed. Given a new
video and its projection into the attribute space, complexity is estimated by the expected ABC of the associated quantization
cell. To enable the use of VideoABC with small reference
video datasets, two quantizers are combined: a k-means quantizer that enables accurate complexity estimates for samples in the distribution of the reference
dataset and a universal
lattice quantizer that guarantees generalization to out-of-distribution samples. A synthetic
video generation procedure, inspired by target-distractor manipulations of psychophysics studies, is proposed to populate the
cells of the
lattice quantizer during training, enabling the computation of their expected ABCs. Experimental results show that VideoABC is effective even with very low-dimensional attribute representations, substantially outperforming approaches like '
video-LLM as judge' with much less complexity. Finally, the explainable nature of the VideoABC score, in terms of well-defined attributes, is shown to provide insights on how the attribute composition of
benchmarks affects their complexity.
Read the full abstract ▾
All authors Aditya Sarkar, Yi Li, Zihao Wang, Hao Yuan, Sai Vidyaranya Nuthalapati, Aashu Singh, Jiacheng Cheng, Ang Li, Shlok Kumar Mishra, David Jacobs, Nuno Vasconcelos
Computer Vision
A Systematic Benchmark of Intraoperative Ultrasound-to-MR Synthesis for Brain Tumour Surgery
Olga Esteban-Sinovas, Santiago Cepeda, Ignacio Arrese et al. (+1 more)
Intraoperative ultrasound (ioUS) is a versatile, cost-effective modality in
brain tumour surgery, but its interpretation is difficult: acquisition planes are non-standard, artefacts are modality-specific, and its appearance differs markedly from the preoperative MRI on which
surgical-planning tools,
segmentation models and the surgeon's experience rely. Synthesising MRI-like images from ioUS could let this MRI-based infrastructure be reused intraoperatively without an extra scan. Most prior work evaluates a single architecture in isolation; to our knowledge, no
benchmark has spanned architectural paradigms, inference regimes and downstream-task endpoints under a common protocol. We address this gap on the public ReMIND data set (76 patients; 153 paired ioUS/T2w and 104 paired ioUS/FLAIR studies; 60/16 patient-level train/held-out split). Six generators (four GAN baselines: Pix2Pix, SwinPix2Pix, CycleGAN, CUT; the
transformer-augmented ResViT; and the few-step
diffusion model SynDiff) were each trained under four inference regimes (2D, 2.5D, 2D + 3D-refinement, full-3D) and two targets (T2w only; T2w + FLAIR multi-task), yielding 48 experiments. Image-fidelity metrics (SSIM, PSNR, MAE, LPIPS) were complemented by an nnU-Net v2 downstream
segmentation evaluation (tumour and resection cavity) and by subgroup analyses by histological grade and reoperation. No architecture dominated every axis, and, critically, perceptual quality tracked downstream utility most closely (LPIPS, r=-0.66, p<0.001), whereas higher SSIM was associated with worse utility (r=-0.64, p<0.001); SynDiff-2.5D best preserved downstream
segmentation (U_Dice=0.55). Perceptual and downstream-task metrics should therefore be reported alongside or in preference to global SSIM, and architecture choice conditioned on
surgical phase, patient history and clinical objective.
Read the full abstract ▾
All authors Olga Esteban-Sinovas, Santiago Cepeda, Ignacio Arrese, Rosario Sarabia
Computer Vision
MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
Yue Jiang, Xue Jiang, Lihua Zhang et al. (+6 more)
Multimodal large
language models (MLLMs) demonstrate remarkable visual understanding, yet their reliability in interactive settings is severely undermined by
hallucination snowballing: a phenomenon where initial errors amplify across conversational turns, leading to a collapse in coherence. This failure reveals a fundamental vulnerability where models progressively neglect visual grounding in favor of over-relying on polluted textual history. Existing
benchmarks are predominantly confined to single-turn VQA, which fail to capture the complex dynamics of error propagation in long-horizon interactions. To address this, we introduce MM-Snowball, the first
benchmark for fine-grained diagnosis of
hallucination snowballing within dialogues. Extensive evaluation shows that our
benchmark poses a significant challenge even to advanced MLLMs and reveals the inefficacy of existing mitigation methods designed for single-turn VQA. To counteract this degradation, we propose Conflict-Aware Visual Rectification (CAVR). This training-free method mitigates snowballing through a synergistic dual-mechanism that refreshes visual grounding at the representation level and rectifies output distributions at the logit level, effectively re-anchoring the model to visual facts. Experiments demonstrate that CAVR achieves state-of-the-art performance, offering a promising path toward more reliable interactive AI. Data and code are available at: https://frenkie-chiang.github.io/MM-Snowball
Read the full abstract ▾
All authors Yue Jiang, Xue Jiang, Lihua Zhang, Zhiqiang Wang, Yuhang Lu, Peng Wang, Bo Han, Feng Zheng, Dingkang Yang
Computer Vision
FlowNar: Scalable Streaming Narration for Long-Form Videos
Zeyun Zhong, Manuel Martin, Chengzhi Wu et al. (+4 more)
Recent Large
Multimodal Models (LMMs), primarily designed for offline settings, are ill-suited for the dynamic requirements of streaming
video. While recent online adaptations improve real-time processing, they still face critical scalability challenges, with resource demands typically growing at least linearly with
video duration. To overcome this bottleneck, we propose FlowNar, a novel framework for scalable streaming
video narration. The core of FlowNar is a dynamic context management strategy for historical visual context removal, combined with our CLAM (Cross Linear Attentive
Memory) module for streaming visual history retention, ensuring bounded visual
memory usage and computational complexity, crucial for efficient streaming. We also introduce a realistic self-conditioned evaluation protocol and complementary evaluation metrics to assess streaming narration models under deployment-like conditions. Experiments on the Ego4D, EgoExo4D, and EpicKitchens100
datasets demonstrate that FlowNar substantially improves narration quality over strong baselines while being highly efficient, supporting processing of 10$\times$ longer
videos and achieving 3$\times$ higher throughput (FPS). The code is available at https://github.com/zeyun-zhong/FlowNar.
Read the full abstract ▾
All authors Zeyun Zhong, Manuel Martin, Chengzhi Wu, David Schneider, Frederik Diederichs, Juergen Gall, Juergen Beyerer
Computer Vision
Pause and Think: A Dataset and Benchmark for Video-Grounded Assistive Action Suggestion
Shivam Singh, Saptarshi Majumder, Pratik Prabhanjan Brahma et al. (+2 more)
Recent
Vision-
Language Models (VLMs) struggle with grounded
reasoning, temporal consistency, and context aware planning in
videos. We introduce pause-and-think-T, a
reasoning-centric training
dataset that encourages models to pause, reason over visual evidence, and produce concise, actionable responses. The
dataset promotes structured
reasoning prior to answer generation, guiding models toward human-like, scene-grounded assistance. We fine-tune a compact 4B-parameter model and evaluate it on our pause-and-think-B
benchmark targeting contextual understanding and goal planning tasks. The model achieves 58.0% accuracy at 59x fewer parameters than Qwen3-VL-235B (58.9%), matching GPT-5.2 on scene understanding and surpassing GPT-4o. Beyond our
benchmark, it also shows strong out-of-distribution performance on EgoThink and TempCompass, with substantial gains in affordance, assistance, attribution recognition, situated
reasoning, and temporal order, without
benchmark-specific training. Our results indicate that targeted
reasoning supervision enables compact models to deliver actionable, visually grounded guidance while generalizing beyond training data, without requiring large-scale model expansion.
Read the full abstract ▾
All authors Shivam Singh, Saptarshi Majumder, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum
Computer Vision
FiSeR: Fine-Grained Source Representations for Cross-Domain AI Image Detection
Shan Zhang, Yongxin He, Mingming Zhang et al. (+2 more)
Real-world synthetic image detectors often generalize poorly under domain shift despite strong in-domain performance. Using unsupervised UMAP projections, we find that natural and synthetic features remain partially separable on unseen
datasets, yet performance still drops, suggesting that the classification head overfits to training-domain artifacts. Therefore, the key is to learn more transferable representations so that the decision criterion is more stable and robust to domain shifts. Based on the structural fact that synthetic images are produced by diverse generators, we propose a hierarchical contrastive learning framework that improves the separability between natural and synthetic images while preserving generator identity information. It jointly optimizes (i) a coarse contrastive objective between natural and synthetic images and (ii) a fine contrastive objective among synthetic images using generator identities. Trained on WildFake, our method achieves an average AUROC gain of +10.22 on cross-domain evaluation over Chameleon, AIGIBench, Community Forensics, and GenImage under the same settings as the strong baseline DIRE. For few-shot adaptation, we freeze the backbone and fit an SVM head on 10 labeled samples per class, improving AUROC by +10.64 on AIGIBench and +17.41 on Chameleon, averaged over 12 widely used detectors. Our code is publicly available at: https://github.com/heyongxin233/FiSeR.
Read the full abstract ▾
All authors Shan Zhang, Yongxin He, Mingming Zhang, Huiwen Tian, Lei Ma
Computer Vision
ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training
Rongsheng Wang, Fenghe Tang, Zihang Jiang et al. (+10 more)
Learning transferable and interpretable representations from medical volumetric scans remains challenging due to complex anatomical structures and weak, heterogeneous supervision provided by radiology reports. In this paper, we propose Anatomy-aware Semantically-Adaptive Pre-training (ASAP), a principled
vision-
language pre-training framework for fine-grained medical volumetric representation learning from large-scale chest CT scans and their corresponding radiology reports. ASAP integrates three key components: (1) an anatomy-aware knowledge injection module that incorporates organ-level structural priors via off-the-shelf
segmentation tool to encourage anatomically coherent representations; (2) a semantically-adaptive selective
alignment mechanism that dynamically associates sentence-level findings with localized volumetric regions; and (3) a semantically-adaptive fusion module for effective interaction between anatomically informed visual features and grounded textual cues under dual-modal masked modeling paradigm. Beyond methodological contributions, we establish a comprehensive
benchmark for medical volumetric
vision-
language pre-training on chest CT, covering 15
datasets and 22 downstream tasks spanning abnormality classification,
segmentation, disease prognosis prediction, report generation, vocabulary classification, cross-modal
retrieval and visual question answering. This
benchmark provides standardized evaluation protocols to systematically assess representation quality under diverse clinical settings and data regimes. Extensive experiments demonstrate that ASAP consistently achieves state-of-the-art performance across tasks and
datasets, with particularly pronounced gains under limited supervision and distribution shift, validating its effectiveness in learning transferable and clinically meaningful volumetric representations.
Read the full abstract ▾
All authors Rongsheng Wang, Fenghe Tang, Zihang Jiang, Yingtai Li, Xu Zhang, Haoran Lai, Wenxin Ma, Wei Wei, Zhiyang He, Xiaodong Tao, Rui Yan, Qingsong Yao, Shaohua Kevin Zhou
Computer Vision
Through the PRISM: Principle-Aware, Interpretable, and Multi-Scale Evaluation of Visual Designs
Mona Gandhi, KJ Joseph, Srinivasan Parthasarathy et al. (+1 more)
Effective visual communication stems from the harmony of multiple design principles, such as readability, contrast,
alignment, overlap, and coherence, which collectively govern clarity and intent of the communicator. While human designers reason holistically over these principles, machine
agents typically condense them into a single heuristic score, offering limited interpretability and diagnostic precision. To address this gap, we introduce PRISM (PRinciple-aware, Interpretable, and Structure-guided Design Modifications), a
benchmark that systematically perturbs professional layouts from the Crello
dataset along measurable design principles. The
benchmark comprises 100K perturbed training samples and 10K perturbed validation designs, each isolating a specific principle violation for controlled analysis of
multimodal reasoning about design quality. We show that models like Qwen-2.5-VL and GPT-4o-mini are largely insensitive to targeted principle degradations, whereas GPT-4o exhibits global awareness without fine-grained disentanglement. Building on these insights, we propose a multi-scale evaluation framework that integrates lightweight scorers for quantitative assessment, instruction-tuned
vision-
language models for localised feedback, and
prompt-based methods for global
reasoning. Our framework provides interpretable explanations of design failures. Using these localised insights, we show targeted refinements that improve layout quality. Together, PRISM and our framework lay the foundation for interpretable design-literate
multimodal reasoning systems.
Read the full abstract ▾
All authors Mona Gandhi, KJ Joseph, Srinivasan Parthasarathy, Sayan Nag
Computer Vision
Response-Aware Multimodal Learning for Post-Treatment Visual Acuity Forecasting
Phuoc-Nguyen Bui, Van-Vi Vo, Duc-Tai Le et al. (+5 more)
Long-term visual acuity (VA) outcomes after anti-VEGF therapy are central to patient counseling, expectation setting, and follow-up planning in diabetic macular edema (DME). However, in clinical practice, physicians must often estimate long-term visual trajectories based only on early post-treatment findings, making reliable prognostication difficult. Although prior OCT-based learning approaches have largely focused on short-term response or single-endpoint prediction, modeling VA trajectories across multiple future time points from early longitudinal observations remains insufficiently explored. In this study, we assembled a real-world cohort of 188 anti-VEGF–treated DME patients with paired baseline and month-1 OCT scans, along with tabular OCT-derived biomarkers and non-imaging clinical variables. Using only these early data, we formulate a multi-horizon VA
forecasting problem aimed at predicting visual outcomes at 3, 6, 12, 18, and 24 months, reflecting clinically meaningful follow-up intervals. We propose ReVA, a response-aware
multimodal framework that integrates structural features from baseline and month-1 OCT with the tabular variables to capture baseline disease status and early treatment response. ReVA uses spatial
attention to preserve localized prognostic imaging features and a dependency-aware tabular encoder to model interactions among clinical variables. These
multimodal representations are fused to predict patient-specific long-term visual acuity trajectories. The proposed framework achieves MAE $=0.1246$, RMSE $=0.1621$, and $R^2=0.6064$ for 24-month VA prediction, with consistent performance across all forecast horizons. Our findings show that incorporating early treatment-response signals enables clinically meaningful long-term visual acuity
forecasting, supporting data-driven decision support for routine anti-VEGF management.
Read the full abstract ▾
All authors Phuoc-Nguyen Bui, Van-Vi Vo, Duc-Tai Le, Junghyun Bum, Van-Nguyen Pham, Ki-Young Kim, Seung-Young Yu, Hyunseung Choo
Computer Vision
Improving Visual Representation Alignment Generation with GRPO
Shentong Mo and Sukmin Yun
Recent
diffusion transformers have demonstrated strong image synthesis capabilities but remain inefficient to train due to weak
alignment between generative and discriminative representations. While representation
alignment frameworks such as REPA improve convergence by aligning noisy denoising features with pretrained visual encoders, their externally supervised
alignment loss is static and lacks adaptivity during training and inference. Existing methods rely on fixed cosine
alignment or contrastive objectives, which cannot dynamically balance representation consistency and generation quality, resulting in limited discriminative benefit and failing to optimize
alignment in a task-adaptive manner. To address this, we propose VRPO, a reinforcement-based optimization strategy that replaces REPA's static
alignment loss with a generative representation policy optimization objective. Instead of enforcing a fixed similarity constraint, VRPO treats representation
alignment as a reward-guided process: the model receives adaptive rewards based on generation fidelity, perceptual quality, and semantic coherence between the
diffusion features and pretrained visual
embeddings. This formulation enables the generator to continuously refine its internal representations toward semantically meaningful directions while improving image quality. Our VRPO-driven training seamlessly integrates into
diffusion transformers, introducing negligible computation cost and preserving full compatibility with SiT and DiT architectures. Extensive experiments on ImageNet-256x256 demonstrate that our VRPO-
Alignment substantially enhances both convergence and fidelity, achieving up to +1.8 FID improvement and 2.3x faster training compared to REPA under identical compute budgets.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Sandboxed Coding Agents are Competitive Omni-modal Task Solvers
Dongping Chen, Xuanao Huang, Zhihan Hu et al. (+3 more)
As
multimodal LLMs increasingly target
video and audio, it is often assumed that such tasks require native omnimodal models. We show that this is not always the case: coding
agents with only text+image access and a sandboxed tool-use interface can match, and in several settings outperform, SOTA native omnimodal models and predefined
multimodal agent scaffolds across multiple audio-
video benchmarks. Our trajectory analysis suggests that their strength comes from writing code and orchestrating tools to extract relevant evidence from transcripts, frames, and other modality signals, thereby converting omnimodal tasks into
retrieval and information-processing problems rather than ingesting entire media streams. We further characterize their limitations through a failure taxonomy and process-level trace analysis, and show that simple skill injection, including human-written and self-distilled skills, substantially improves performance. To explore open-source elicitation, we introduce Code-X, a training recipe with the OmniCoding trajectory
dataset and verifiable reward, and provide baselines on Qwen-3.5-9B and Qwen-3.6-27B. Finally, we argue that the next frontier is many-modality processing, and introduce TerminalBench-O, a process-level
benchmark for real-world omnimodal processing tasks. Code will be available at https://github.com/Dongping-Chen/OmniCoding.
Read the full abstract ▾
All authors Dongping Chen, Xuanao Huang, Zhihan Hu, Qingyuan Shi, Dianqi Li, Tianyi Zhou
Computer Vision · Cross-listed from cs.LG
On the Difficulty of Learning a Meta-network for Training Data Selection
Zilin Du, Junqi Zhao and Boyang Albert Li
Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately. A common strategy is to learn data weights via bi-level optimization, which we refer to as Meta-learning for Training-data Selection (MTS). Interestingly, in practice, MTS often performs below expectation. We identify two obstacles in properly training MTS: a poor gradient signal-to-noise ratio (GSNR), which causes optimization difficulties, and lack of informative features that correlates with data quality. We present a mathematical analysis of MTS, which reveals the dynamics of normalized data weights and the relation between disparate data quality and poor GSNR. The analysis suggests a a simple yet effective solution: increasing the batch size. Further, we propose a set of informative features that capture the positions of training data in their distributions and training dynamics. Experiments across four
benchmarks show consistent improvements, achieving average gains of 5.49% over training without selection and 2.89% over the strongest baseline.
Read the full abstract ▾
Computer Vision
Hee Suk Yoon, Eunseop Yoon, Jaehyun Jang et al. (+6 more)
While on-policy
distillation offers dense supervision for training small
reasoning models, its optimization dynamics in the
multimodal domain remain under-explored. In this work, we challenge the standard monolithic view of
Vision-
Language Model (VLM)
distillation by mathematically decomposing the loss into two distinct components: the
language prior and visual grounding. Our analysis uncovers that gradient vectors for these components are nearly orthogonal, indicating that the objective of aligning with the teacher's
language distribution is geometrically independent from the objective of matching its visual perception. Consequently, standard optimization passively follows a suboptimal compromise trajectory that implicitly balances the two objectives. Hypothesizing that visual grounding constitutes the primary bottleneck for
vision-
language reasoning, we introduce Visual Gradient Steering (VGS), a method that dynamically reorients the update vector to prioritize the visual subspace. Experimental results on multiple
distillation settings and complex
multimodal benchmarks demonstrate that VGS significantly outperforms the standard monolithic formulation of on-policy
distillation, achieving superior grounding with minimal training overhead.
Read the full abstract ▾
All authors Hee Suk Yoon, Eunseop Yoon, Jaehyun Jang, SooHwan Eom, Ji Woo Hong, Mark Hasegawa-Johnson, Qi Dai, Chong Luo, Chang D. Yoo
Computer Vision
DeepLatent: Think with Images via Parallel Latent Visual Reasoning
Dongchen Lu, Zhimo Li, Mao Shu et al. (+1 more)
The emerging paradigm of "thinking with images" embeds visual states into intermediate
reasoning steps, defining a new frontier for
Vision-
Language Models. Existing approaches diverge along two lines. Tool-assisted methods apply explicit visual operations but suffer from high latency and restricted manipulation types. Latent
reasoning methods autoregressively produce implicit visual states, but underperform tool-assisted methods, and their latent
tokens fail to capture effective visual information. In this work, we propose DeepLatent, a parallel framework for latent visual
reasoning. First, we introduce LatentFormer. It uses learnable 2D
tokens to generate context-conditioned latent states in parallel, anchoring every visual update directly in the original image features. Second, we design a continuous-space reinforcement learning
algorithm. It optimizes latent modulation parameters directly in the
embedding space, significantly improving latent representation quality. The framework is trained via knowledge
distillation followed by this continuous-space RL
algorithm. Furthermore, we contribute DeepLatent-180K, a large-scale
dataset tailored for latent visual
reasoning. Extensive evaluations across multiple
benchmarks demonstrate that DeepLatent achieves state-of-the-art performance.
Read the full abstract ▾
All authors Dongchen Lu, Zhimo Li, Mao Shu, Huo Cao
Computer Vision
Improving Visual Grounding in Remote Sensing via Cluster-Guided Refinement and Model Ensemble Voting
Panav Shah, Geet Sethi and Ashutosh Gandhe
Visual grounding aims to locate image regions that correspond to natural
language descriptions and is a key component of interpretable
vision systems. In remote sensing imagery, grounding is particularly challenging due to complex scenes, small objects, and large variations in scale. Relying on a single model is often insufficient to address these diverse challenges. In this work, we propose two grounding pipelines, Sequential Grounding Refinement (SGR) and Cluster-Aware Grounding Refinement (CGR), that combine the complementary strengths of RemoteSAM, a visual grounding model specialized for remote sensing, and SAM3, a powerful general-purpose
segmentation model. Our approach first uses RemoteSAM to obtain an initial estimate of object location, which is then refined using SAM3 to produce more accurate and spatially consistent
segmentations. Additionally, we explore an ensemble strategy based on majority voting across six diverse grounding pipelines, each with distinct capabilities. This multi-model framework improves robustness and significantly enhances localization accuracy. Experimental results demonstrate that the proposed pipelines and ensemble approach outperform individual models, leading to more reliable and precise visual grounding predictions.
Read the full abstract ▾
Computer Vision
CAFOSat: A Strongly Annotated Dataset for Infrastructure-Aware CAFO Mapping Using High-Resolution Imagery
Oishee Bintey Hoque, Nibir Chandra Mandal, Mandy L Wilson et al. (+3 more)
Concentrated Animal Feeding Operations (CAFOs) play an important role in agricultural production but are also associated with environmental, public health, and disease surveillance concerns. Large-scale mapping of CAFOs from remote sensing imagery remains challenging due to heterogeneous infrastructure layouts, noisy location records, inconsistent annotations, and incomplete inventories. We introduce CAFOSat, a strongly annotated, infrastructure-aware
dataset for CAFO mapping across the United States. CAFOSat integrates high-resolution National Agriculture Imagery Program (NAIP) imagery with multi-source CAFO inventories collected across multiple states and transforms weak geolocation records into refined annotations through a human-in-the-loop pipeline combining AI-assisted annotation, GradCAM-based localization, and
geometric clustering. To improve
dataset quality, we curate challenging negative samples using land-cover-guided sampling with spatial exclusion constraints and provide infrastructure-level annotations, including barns, manure ponds, and grazing-related features, through manual verification. The resulting
dataset contains more than 45,000 image patches spanning 20 states and four major CAFO categories. We
benchmark a diverse set of convolutional,
transformer-based, and
vision-
language models, demonstrating the value of refined annotations and curated negative samples for CAFO classification and generalization. In addition, we introduce a synthetic augmentation pipeline that generates infrastructure-aware variations to increase training diversity and improve robustness under distribution shifts. CAFOSat provides a large-scale
benchmark for advancing infrastructure-aware agricultural monitoring and CAFO mapping from high-resolution remote sensing imagery.
Read the full abstract ▾
All authors Oishee Bintey Hoque, Nibir Chandra Mandal, Mandy L Wilson, Samarth Swarup, Madhav Marathe, Abhijin Adiga
Computer Vision
ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMs
Yiling Gao, Hongchen Wei and Zhenzhong Chen
In
Vision-
Language Models (VLMs), high-resolution images produce a large number of visual
tokens, resulting in high computational costs and KV-cache overhead during inference. To address this problem, we propose an Extreme
Token Compression (ETC) framework that minimizes task loss when reducing the number of input
tokens based on the principle of variational information
distillation. Specifically, from an information-theoretic perspective, we show that minimizing task loss requires the compact representation to preserve the instruction-aware sufficient statistic of the task-relevant visual information for prediction. In practice, ETC leverages text-to-image cross-
attention to weight the original visual features to approximate the latent instruction-aware predictive statistic. Moreover, ETC introduces a variational information
distillation, enabling the compact representation to preserve the essential information to recover this predictive statistic. Experiments on LLaVA-1.5-7B and Qwen3-VL-2B show that ETC remains effective even under single-
token compression, substantially reducing KV-cache overhead while retaining strong task performance.
Read the full abstract ▾
Computer Vision
A Trajectory-Driven Spatio-Temporal Refinement Solution for CVPR 2026 8th UG2+ Challenge Track 3: DOST
Hongzhen Li, Miao Yu, Leilei Cao et al. (+3 more)
In this work, we present our solution for the 8th UG2+ Challenge (CVPR 2026) Track 3: Dynamic Object
Segmentation in
Turbulence (DOST). Our method is built upon the strong baseline framework Segment Any Motion (SegAnyMo), which provides powerful mask generation and motion tracking capabilities. To further boost the
segmentation performance under severe atmospheric distortions, we propose two key improvements. First, we employ a data-centric domain adaptation strategy. We significantly expand our training data by incorporating selected sequences from the DAVIS
dataset alongside a subset of the DOST
dataset, and apply simulated atmospheric fluctuation degradations to enhance the model's robustness against complex
geometric distortions. Second, we introduce a spatio-temporal post-processing module. This refinement step effectively removes persistent boundary-connected false foregrounds and short-lived fragmented noise, while strictly preserving genuine small targets and maintaining original individual labels across frames. With these combined strategies, our proposed method ranks the 2st place in the challenge.
Read the full abstract ▾
All authors Hongzhen Li, Miao Yu, Leilei Cao, Youwei Pan, Yingfang Zhu, Fengjie Zhu
Computer Vision · Cross-listed from cs.LG
Generate in Reconstruction Space, Match in Semantic Space: Transport Geometry for One-Step Generation
Hugues Van Assel, Edward De Brouwer, Saeed Saremi et al. (+2 more)
Generative modeling and self-supervised representation learning (SSL) optimize structurally different objectives: generative training rewards distributional fidelity, while SSL rewards semantic coherence. Yet recent work repeatedly finds that SSL features improve generative training, though the mechanism of this synergy remains unclear. Here, we study the benefits of SSL in generative modeling in the framework of one-step generation where the role of representation is explicit: frozen SSL features are used to match generated samples to real data. We use the Sinkhorn divergence in that feature space, providing a tractable surrogate for the Wasserstein distance, the population-level discrepancy approximated by Fréchet-style evaluation metrics (such as FID). We find that this objective becomes highly effective when computed in a semantically structured SSL feature space (a 39$\times$ reduction in ImageNet FID). We trace this behavior primarily to matching estimation: semantic SSL features that suppress nuisance reconstruction details induce a more compact geometry, making distribution matching more tractable. As a consequence, the best training SSL features need not match the features used by the evaluation metric. In particular, we show that using Inception as the feature extractor can improve FID while degrading matching stability and sample quality, revealing a form of metric hacking. Using extensive experiments on ImageNet, we identify which SSL feature families lead to best generation performance and show that matching stability is a quantitative criterion for selecting them. Code is available at https://github.com/Genentech/semantic-transport-generation.
Read the full abstract ▾
All authors Hugues Van Assel, Edward De Brouwer, Saeed Saremi, Gabriele Scalia, Aviv Regev
Computer Vision · Cross-listed from cs.LG
Saliency-Aware Model Merging
Jungin Park, Jiyoung Lee and Kwanghoon Sohn
Model merging aims to consolidate multiple task-specific models fine-tuned on different
datasets into a unified architecture that performs cross-domain proficiency. Current data-free model merging methods often struggle to scale as they rely on simple parameter-level heuristics that ignore inter-layer dependencies and non-uniform distribution of expertise. This work proposes SA-Merging, which is built upon connectivity-based saliency formulations from structural
pruning (e.g., SynFlow) and extends them to the data-free model merging setting. We define a saliency score over task vectors relative to a shared base model, and further introduce merge-aware modulation that incorporates agreement across experts to mitigate task interference. Based on this formulation, an iterative saliency-aware merging procedure progressively removes non-informative updates while preserving end-to-end connectivity. Furthermore, we extend SA-Merging to introduce rank-wise saliency decomposition for LoRAs without compromising their structural integrity. Extensive experiments on
vision and
language tasks demonstrate the effectiveness of our saliency-based approach, further reducing the gap between data-free and test-time adaptation methods.
Read the full abstract ▾
Computer Vision
Structure-Aware Consistency Priors for Shape from Polarization in Complex Media
Kaimin Yu, Puyun Wang, Huayang He et al. (+1 more)
Recovering surface normals from single view polarization images in complex media remains challenging. This paper focuses on ice as a representative complex medium, where intricate light matter interactions lead to a nonlinear mapping between polarization observations and surface normals. To address this, a structure-aware polarization prior based on autocorrelation functions is proposed to capture the local spatial consistency of AoLP. Building on this, a dual-branch network (IceSfP) is designed to integrate raw polarization features with priors via cross modal
attention and multi-scale feature fusion, enabling accurate surface normal estimation under complex media conditions. To evaluate the method, the first real-world ice SfP
dataset is constructed. Experimental results show that the method outperforms existing approaches across all metrics, achieving a MAE of 16.01 deg, which is 2.74 deg lower than the second-best method. The framework provides a generalizable solution for high-precision
geometric perception in complex media.
Read the full abstract ▾
All authors Kaimin Yu, Puyun Wang, Huayang He, Xianyu Wu
Computer Vision
Jungin Park, Jiyoung Lee and Kwanghoon Sohn
This study introduces an intriguing phenomenon in
Video LLMs: rather than merely translating frames into textual
embeddings,
Video LLMs establish a continuous
manifold,
token interface, allowing visual
tokens to operate as standalone entities within the architecture. Exploiting this discovery, we propose V-LynX, a scalable framework that integrates novel modalities into
Video LLMs by repurposing the internalized interface. Departing from conventional paradigms that necessitate heavy modality-specific encoders or paired supervision, V-LynX employs a lightweight auxiliary pathway in parallel with the frozen
vision encoder. Our method integrates new sensory inputs with intrinsic
video priors by aligning both
attention responses and statistical distributions using unpaired unimodal data sets. This ensures
manifold compatibility while preserving the integrity of the
Video LLMs. Extensive
benchmarks demonstrate that V-LynX achieves SOTA and efficiency across audio-visual QA, 3D
reasoning, high-frame-rate, and multi-view
video understanding. The code is available at https://github.com/park-jungin/lynx.
Read the full abstract ▾
Computer Vision
OptiWorld: Optimal Control for Video World Generation under Physical Constraints
Yu Yuan, Jianhao Yuan, Xijun Wang et al. (+4 more)
Video generation models are becoming a scalable form of world models, but they mainly generate plausible motion rather than proactively control or optimize the underlying dynamics. As a result, an object in the generated
video may follow trajectories that are unsafe, not smooth, inefficient, or physically inconsistent. In this work, we propose OptiWorld, a framework that brings classical optimal control into
video generation at inference time. OptiWorld first extracts a compact, task-relevant world state, then plans an optimal trajectory under physical constraints, and finally renders the
video conditioned on this trajectory. We formulate planning as a
geometric problem on a continuous
manifold, which converts 3D geometry and task-dependent physical constraints into a unified planning geometry. By adding this optimal-control layer, OptiWorld generates
videos with preferable dynamics, demonstrating strong potential in multiple tasks including goal-conditioned image-to-
video generation,
video dynamics editing, and counterfactual generation.
Read the full abstract ▾
All authors Yu Yuan, Jianhao Yuan, Xijun Wang, Daiqing Li, Liu He, Lu Ling, Stanley H. Chan
Computer Vision
Pre-Deployment Robustness Stress Testing for CT Segmentation Systems Using Clinically Motivated Multi-Corruption Augmentation
CholMin Kanga, Jonghyun Chung, Amanpreet Kaur et al. (+2 more)
Deep learning-based CT
segmentation systems often achieve high accuracy on clean
benchmark images, but their performance may degrade under heterogeneous clinical imaging conditions such as noise, resolution loss, contrast variation, intensity shift, and artifacts. This instability can limit reliable deployment in real-world medical imaging workflows. We propose Robustness via Augmented Multi-corruption Pipeline (RAMP), a robustness-oriented augmentation framework for CT
segmentation. RAMP combines anatomically constrained spatial perturbations, CT intensity transformations, and
stochastic multi-corruption composition to expose models to clinically plausible image degradation during training. Across two CT
segmentation evaluation settings, RAMP achieved the strongest corrupted-image performance and the smallest clean-to-corrupted robustness gap. In the five-organ noisy evaluation
benchmark, RAMP improved mean corrupted Dice from 0.610 to 0.753 and reduced the robustness gap from 0.264 to 0.064 compared with the nnU-Net baseline. In Abdomen1K, RAMP improved mean corrupted Dice from 0.633 to 0.789 and reduced the robustness gap from 0.290 to 0.070. Although RAMP did not achieve the highest clean-image Dice, it substantially mitigated worst-case
segmentation collapse under severe image degradation. These results suggest that multi-corruption augmentation can serve as a practical pre-deployment strategy for improving the reliability of CT
segmentation systems in heterogeneous clinical environments.
Read the full abstract ▾
All authors CholMin Kanga, Jonghyun Chung, Amanpreet Kaur, Nagesh Gulkotwar, Aarthi Sivasankaran
Computer Vision
3D Segment Anything Model with Visual Mamba for Diagnosing Placenta Accreta Spectrum
Yuliang Zhang, Fang He, Lulu Peng et al. (+5 more)
Placenta Accreta Spectrum (PAS) is a rare but highly dangerous obstetric disease. Early and accurate PAS diagnosis is critical for maternal health. Traditional PAS diagnosis relies on experienced doctors by analyzing the cesarean history and
Magnetic Resonance Imaging (MRI) data. However, district-level hospitals often lack the expertise and resources for accurate PAS diagnosis. To address these challenges, we establish the first MRI-based PAS
dataset, which includes both fine-grained
segmentation and classification annotations. Meanwhile, diagnosing PAS can be significantly enhanced by segmenting lesion areas from MRI images of the uterus. To achieve automatic PAS diagnosis, we propose 3DSAMba, a novel feature learning framework for effective lesion
segmentation. More specifically, we first design a 3D Segment Anything Model (SAM) and incorporate medical domain information into the model through an efficient adapter mechanism. In addition, we introduce a Multi-Level Aggregation Mamba (MLAM) to aggregate feature maps across different levels and a Fusion State Space Model (FSSM) to fuse multi-scale features from both the encoder and decoder. Finally, we apply
segmentation masks to the original MRI images through element-wise multiplication, effectively isolating lesion areas for more accurate PAS diagnosis. Extensive experiments validate that our framework significantly improves the PAS diagnostic performance. To facilitate further research in PAS diagnosis, we have released the
dataset and source code at https://github.com/Drchip61/PASD.
Read the full abstract ▾
All authors Yuliang Zhang, Fang He, Lulu Peng, Tianyu Yan, Pingping Zhang, Ting Song, Lili Du, Dunjin Chen
Computer Vision · Cross-listed from cs.CL
Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs
Xin Gao, Cheng Yang, Chufan Shi et al. (+1 more)
Unified
multimodal models (UMMs) have emerged as a promising paradigm for general-purpose
multimodal intelligence. As they are deployed in real-world applications, effectively updating internal knowledge becomes critical. While knowledge editing has matured for text-only models, it remains unclear whether edits that successfully modify textual outputs also transfer to image generation in UMMs. To study this question, we introduce UniKE, the first
benchmark for cross-modality knowledge editing in UMMs, comprising 2,971 edit subjects spanning attribute and relation edits. Using VQA-based visual verification, we reveal a striking modality gap: text-side efficacy can reach approximately 92%, whereas the best overall VQA accuracy under direct image generation is only 18.5%. We further propose
Reasoning-augmented Parameter Editing, which explicitly activates edited knowledge before generation and improves overall VQA accuracy for all evaluated model-editor pairs, with gains up to 18.6 percentage points. Mechanistic analysis shows that this gap is associated with partial
alignment between edited textual representations and the conditioning pathways for visual generation, where edits sufficient for text outputs may remain too weak or misaligned to steer image synthesis. These findings show that textual knowledge edits do not guarantee reliable cross-modality transfer and motivate modality-aware editing methods. Our code and data are available at https://github.com/gxx27/UniKE.
Read the full abstract ▾
All authors Xin Gao, Cheng Yang, Chufan Shi, Taylor Berg-Kirkpatrick