Computer Vision · Cross-listed from cs.LG
Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models
Jonas Ngnawé, Yann Pequignot, Sabyasachi Sahoo et al. (+3 more)
Large
vision-
language models (LVLMs) inherit massive activations from their text-only bases: spikes where a few fixed hidden channels receive values thousands of times above the typical magnitude. The text spike systematically appears in early layers at a fixed initial position, independently of input content. Visual spikes vary across images, but whether their formation follows a consistent pattern across LVLMs and how they respond to image perturbations remain open questions. We find that some LVLMs do not form visual spikes, while others spike at different rates, typically in deeper layers. We identify the trigger direction from model weights and an interpretable location rule: before the
language model decoder runs, eventual spike
tokens are largely restricted to those sharing least with the rest of the image. Crucially, visual spikes are strikingly brittle. Common corruptions frequently create and relocate spikes, and less often remove them, raising overall incidence. Our trigger-guided spike attack deliberately creates or removes spikes under a small $\ell_\infty$ budget, with 1/255 enough in nine of the ten models that spike. Finally, our preventive intervention removes only the trigger component before spikes erupt, eliminating or substantially reducing spikes on clean and perturbed images while leaving the other image
tokens nearly unchanged. Our study spans 25 adapter-based LVLMs built on 18 released text-only bases from 10 families, ranging from 2B to 72B parameters.
Read the full abstract ▾
All authors Jonas Ngnawé, Yann Pequignot, Sabyasachi Sahoo, Christian Gagné, Frédéric Precioso, Sanmi Koyejo
Computer Vision · Cross-listed from cs.AI
PlanGuard: A Guardrail for Multi-Step Plan Safety in Embodied Agents
Junchi Chen, Changtao Miao, Yuxiao Xiang et al. (+9 more)
Embodied task planners may produce multi-step plans whose subtask dependencies and interactions with the environment create physical risks during execution. Yet existing safeguards overlook such compositional risks, as general-purpose guardrails focus on semantic harm and embodied safety detectors assess subtasks in isolation. To address this gap, we introduce PlanGuard, the first pre-execution detector that evaluates the physical safety of a complete multi-step plan in its current environment. For training and evaluation, we construct a Multi-Step Plan Safety (MSP-Safe)
dataset through paired task construction, plan generation using diverse planners, and safety annotation by three judges. Task-oriented SFT on MSP-Safe establishes fundamental plan-safety assessment capabilities, yet a substantial gap remains between compact models suitable for real-time deployment and stronger but costlier large models. Accordingly, we propose Strong-Teacher Adaptive Compensation for On-Policy
Distillation (STAC-OPD), which provides compact models with adaptive strong-teacher supervision along their on-policy trajectories. It combines
token-level distribution transfer from a fine-tuned strong teacher with probability-routed sequence-level compensation, retaining student-generated targets when the student favors the reference safety decision and using teacher-reconstructed targets otherwise. Across all test subsets, PlanGuard-2B achieves average 87.15% ACC and 87.21% F1, demonstrating effective whole-plan physical-risk detection at compact model scale. Code and
dataset will be publicly released.
Read the full abstract ▾
All authors Junchi Chen, Changtao Miao, Yuxiao Xiang, Zhenchao Jin, Haojie Yuan, Qi Chu, Tao Gong, He Liu, Bo Zhang, Jiansheng Cai, Zhe Li, Nenghai Yu
Computer Vision
Latent Space Is Not Flat: Rethinking Latent Structure for 3D Medical Image Synthesis
Haowen Xue, Hao Chen, Hexuan Hu et al. (+6 more)
Latent generative models make 3D medical image synthesis computationally practical by generating in a compressed space. However, we show that the common flat Euclidean assumption induced by $\ell_2$ objectives is imprecise: latent-space geometry is so strongly anisotropic that equal-magnitude errors can produce drastically different decoded distortions. We further find that this anisotropy has a clear feature: sensitive variation concentrates in a low-rank subspace. The dominant low-rank components capture the overall structure, encoding long-range, spatially coordinated variation while remaining resistant to local noise. Its orthogonal residual, in contrast, mainly captures local and image-specific variation. Motivated by this asymmetry, we introduce Latent Structure Flow (LSF). At each block, LSF decomposes the latent state into structure and residual, models structural changes with global context, and predicts residual variation locally while preserving a direct path for the input structure. LSF changes only the generator, leaving the frozen codec and pointwise training objective unchanged. Across cross-modality synthesis and tumor inpainting tasks, LSF outperforms all compared baselines on both global and tumor-specific metrics, demonstrating the benefit of explicitly modeling latent-space structure for 3D medical image synthesis.
Read the full abstract ▾
All authors Haowen Xue, Hao Chen, Hexuan Hu, Qian Huang, Yi Han, Qing Meng, Zaipeng Xie, Chao Li, Haoli Xu
Computer Vision
Hierarchical Frequency-Domain Compression of Implicit Geometric Representations for Large-Scale Point Clouds
Manlin Yao, Jiabin Liu, Guan Wang et al. (+2 more)
Large-scale point cloud representations of complex geome tries incur prohibitive computational and
memory costs, necessitating compressed implicit representations. To ad dress this, we propose a unified framework comprising im plicit
geometric field representation, hierarchical frequency domain compression, and conditional high-frequency predic tion. Specifically, an unordered point cloud is mapped to an implicit field defined within its physical bounding box. A smooth
Fourier pyramid is then constructed, where com pact low-frequency components capture the global geometry. Inter-scale high-frequency residuals are encoded to preserve the spatial information required for reconstructing fine geo metric details. To restore the high-frequency information lost during compression, we develop a hierarchical 3D neural net work. The reconstructed implicit field is converted back into a point cloud through isosurface extraction. Experiments on a complex-boundary point cloud with more than eight mil lion points demonstrate that the proposed method achieves a higher compression ratio than existing point cloud compres sion methods while maintaining comparable reconstruction quality.
Read the full abstract ▾
All authors Manlin Yao, Jiabin Liu, Guan Wang, Haixu Liu, Hui Li
Computer Vision · Cross-listed from cs.LG
Learning When to Recur: Token-Adaptive Recursion for Imbalanced Ophthalmic Domain Incremental Learning
Nanxi Yu, Kang Li, Ye Du et al. (+3 more)
Domain incremental learning is essential for adapting ophthalmic deep learning models to sequential clinical domains while preserving diagnostic expertise. Existing domain incremental learning methods predominantly address the domain shift induced by style variations. However, they often overlook the severe class imbalance inherent in real-world clinical scenarios, such as clinical referral systems. Institutions in these systems encounter drastic fluctuations in class priors, resulting in label distribution shift, a critical form of domain shift that triggers severe catastrophic forgetting. To address these challenges, we propose ToRe, a rehearsal-free and parameter-efficient framework that leverages frozen ophthalmic foundation models for robust incremental adaptation. ToRe employs a parameter isolation strategy to decouple domain-specific optimization paths, thereby helping mitigate catastrophic forgetting driven by both label distribution shift and style variations. Simultaneously, it introduces
token-adaptive recursion that adaptively allocates additional computational depth across
tokens, allowing simple
tokens to exit the recursion loop early while subjecting complex
tokens, such as those associated with lesions, to deeper recursive processing. This mechanism enhances the feature representations for minority classes, thereby supporting generalization throughout the domain incremental learning process. Extensive evaluations on nine heterogeneous
datasets demonstrate that ToRe consistently outperforms state-of-the-art methods in overall performance across the three
benchmarks, while maintaining near-zero forgetting. Together, these results support the applicability of ToRe to dynamic and imbalanced clinical environments. The code is available at https://github.com/Nancyolo/ToRe
Read the full abstract ▾
All authors Nanxi Yu, Kang Li, Ye Du, Xiaowei Hu, Weihua Yang, Shujun Wang
Computer Vision
Long Qian, Bingke Zhu, Jiaqi Wei et al. (+3 more)
Diffusion vision-
language models generate answers by gradually resolving masked
tokens, making accurate conditional prediction in partially resolved states central to post-training. Masking completed answers yields coherent contexts and targets, but prescribed masks do not reflect the model's reveal decisions. Its trajectories capture these decisions, yet their provisional visible
tokens can conflict with the target response. Outcome-based reinforcement learning follows these trajectories but provides only response-level feedback, which loses contrast when sampled rewards tie. To align coherent
token-level supervision with the model's reveal decisions, we introduce Counterfactual Trace On-Policy
Distillation (CT-OPD), which combines completed teacher responses with trajectory masks from the current student. CT-OPD retokenizes each teacher response in the student's vocabulary and extracts unresolved-position masks at successive stages of the student's reverse process. For each mask, it discards provisional rollout values and reconstructs the partial state from the teacher endpoint, so the supervised positions follow the current trajectory while the visible context and targets remain consistent with the same response. The student is trained on these reconstructed states with its native categorical loss, and trajectories are refreshed as the model evolves. Across dense and sparse
diffusion architectures, CT-OPD consistently enhances
multimodal understanding and
reasoning capabilities, with gains of up to 9.80 points on the nine-
benchmark average. On the unified understanding-and-generation architecture, it also improves both visual understanding and image generation, showing that the same principle transfers across architectures and modalities. Ablations further attribute these gains to coherent reconstruction and current-model trajectory masks.
Read the full abstract ▾
All authors Long Qian, Bingke Zhu, Jiaqi Wei, Yu Li, Yingying Chen, Jinqiao Wang
Computer Vision
OmniMoE-VL: A Sparse Vision-Language Model with Coupled Visual-Depth Routing
Long Qian, Bingke Zhu, Jiaqi Wei et al. (+2 more)
Vision-
language models (VLMs) increasingly use sparse mixture-of-experts (MoE) to scale
language-side computation, yet visual information is typically routed only after passing through a fixed cross-modal interface. This leaves an important decision unresolved: which intermediate visual representations should be exposed to
language computation for a given question? We introduce OmniMoE-VL, a sparse VLM with a coupled visual-depth routed projector. For each image-
prompt pair, the projector selects a sparse set of intermediate visual depths and reuses the resulting global preference to guide both local patch fusion and dynamic visual injection into the
language model. This design enables question-dependent visual access while preserving the native visual-
token sequence, and complements
token-level expert routing in the
vision and
language stacks. Across eight image-based
benchmarks, OmniMoE-VL achieves an average score of 85.9 with 28B total and 9B activated parameters. Controlled comparisons show that the routed visual interface provides the dominant architectural gain, while matched route and component controls, same-image route analysis, and route interventions further support the value of coupling and question-conditioned visual access.
Read the full abstract ▾
All authors Long Qian, Bingke Zhu, Jiaqi Wei, Yingying Chen, Jinqiao Wang
Computer Vision · Cross-listed from cs.RO
An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning
Mino Nakura, Sriram Krishna, Yufei Wang et al. (+3 more)
Visual imitation learning is a promising approach to training
robot manipulation policies capable of completing a wide variety of tasks. However, policies today remain brittle to viewpoint perturbations, making deployment in diverse environments a challenge. We present a controlled empirical study of which design choices allow visuomotor policies to generalize across viewpoints. We find that viewpoint generalization improves when dense visual
tokens are retained and the action head participates in
geometric reasoning. On a suite of simulated tasks that span a wide range of camera poses, we show that these design choices yield a policy that remains performant across viewpoints. As a practical consequence, a policy trained with these design choices also transfers zero-shot from simulation to the real world under random camera configurations.
Read the full abstract ▾
All authors Mino Nakura, Sriram Krishna, Yufei Wang, Shubham Tulsiani, Zackory Erickson, David Held
Computer Vision
From Feed-Forward to Flow: Unifying Reconstruction and Generation Is Easier Than You Think
Haoru Wang, Qianfan Shen, Kai Ye et al. (+2 more)
Reconstruct where the images provide evidence, and generate where they do not: recent success of spatial world models such as Atlas (World Labs Team, 2026) highlights the value of unifying reconstruction and generation in one model. Yet the two have long lived in separate paradigms with distinctive failure modes: feed-forward reconstruction averages ambiguity into blur, while conditional generation invents plausible but scene-inconsistent detail. In this work, we present a unified flow-based formulation for reconstruction and generation, where a shared clean-target predictor performs direct reconstruction at its single-step endpoint and unfolds conditional generation through multi-step flow. A controlled toy study reveals the mechanism: with a single step, the predictor collapses to the conditional mean just like feed-forward methods, favoring consistency over diversity. With multi-step inference, the fidelity of generated details grows with context richness: closer observations reduce ambiguity and yield better-matched details. We further instantiate the formulation in appearance and geometry 3D tasks. JiT-LVSM improves perceptual and distributional quality in novel view synthesis, while JUSt3R retains competitive single-step geometry prediction with additional multi-step inference capabilities that reduces veil and flying-pixel artifacts, producing cleaner surface structure with greater test-time compute. Together, they show that reconstruction and generation can share both a formulation and a backbone, with their behavior governed by denoising configuration—making unification surprisingly simple.
Read the full abstract ▾
All authors Haoru Wang, Qianfan Shen, Kai Ye, Wenzheng Chen, Baoquan Chen
Computer Vision · Cross-listed from cs.AI
Readout is not Recovery: Dissociating Coordinate Emission from Visual-Corruption Repair in Vision-Language Models
Drandreb Earl Juanico
VLM bounding-box localization is both
language generation and spatial commitment. Parseable fields such as bbox_2d make localization easy to score, but dimensions that emit coordinate
tokens need not repair localization after visual evidence is damaged. We study this readout/recovery separation in Qwen3-VL-4B-Instruct on single-object COCO grounding. We compare clean coordinate-
token readout rankings with corruption-derived repair rankings, using object-mask endpoint replacement for recovery and clean-input flooring for depth localization. In Qwen3-VL, coordinate-
token rankings are inert through layer 24, load-bearing from layers 32-35, and peak at layer 34; corruption-derived rankings harm layers 16-24 but become beneficial near layer 35/final. A Kimi-VL-A3B diagnostic shows a matching output-proximal transition despite a different box format. Object-mask recovery separates rank budgets: $k=250$ shows necessity, $k=500$ shows Top-$k$ restoration above random, and $k=d/2$ is largely capacity-driven. Partial-occlusion sweeps reveal that high-overlap coordinate-
token sets can hurt at $k=1000$ and help mainly at half-width, while population corruption-derived sets provide no reliable fixed repair set. Edge-attribution patching shows coordinate-
token paths are high precision but low recall for detection recovery, and RMSNorm quasi-layer controls do not close the endpoint-repair gap. Endpoint coordinate triage is therefore a useful
circuit prior, but occlusion recovery requires a separate
benchmark.
Read the full abstract ▾
Computer Vision
LoCoVSR: Local Context Diffusion Posterior Sampling for Video Super-Resolution
Matan Ben Chorin and Michael Elad
Video super-resolution (VSR) is an ill-posed inverse problem that aims to reconstruct a high-resolution (HR)
video from a noisy, low-resolution (LR) version of it. We present LoCoVSR, a
diffusion-based VSR framework that leverages pixel-space denoising
diffusion probabilistic models. LoCoVSR integrates the
Diffusion Posterior Sampling technique with spatio-temporal context learning, operating in a moving-average form. A localized window of adjacent LR frames is used for recovering each center frame, while applying a shared noise trajectory across all frames. The localized windowing enables processing of long
videos without length limitations, supports parallel inference, and prevents error accumulation that may occur in recursive processing. Unlike prior methods, LoCoVSR offers a simple yet very effective VSR solution, avoiding explicit optical flow estimation, or information loss caused by latent space processing. Trained on the VFHQ face
dataset, LoCoVSR achieves accurate, temporally consistent and high-quality upscaling with competitive results against recent
diffusion-based VSR approaches.
Read the full abstract ▾
Computer Vision
AnesTRACE: Benchmarking Intraoperative Anesthesia from Multimodal Perception to Multi-step Decision-Making
Ziwei Huang, Qi Gao, Zhe Ji et al. (+5 more)
Intraoperative anesthesia requires systems to interpret evolving
multimodal evidence, recommend timely management, and revise decisions as patient states change, yet existing
benchmarks usually isolate perception or single-point
reasoning. We introduce AnesTRACE, an evaluation suite comprising AnesTRACE-Bench and AnesTRACE-Eval. Built from public perioperative
datasets with anesthesiologist annotation, AnesTRACE-Bench evaluates Intraoperative Perception, Single-point Anesthesia Decision-Making, and Multi-step Anesthesia Decision-Making. AnesTRACE-Eval assesses open-ended responses through anesthesiologist-defined criteria for Clinical Correctness, Evidence Grounding, Task Completeness, and Safety, with Temporal Consistency for multi-step decisions; its domain-specific evaluator is trained by supervised fine-tuning and preference
alignment on expert-reviewed judgments. Across more than 30 models, fine-grained visual grounding and intervention selection remain difficult: the leading model reaches only 32.2 mIoU for TEE visual grounding and retains a 17.5% Major/Critical Safety Error Rate in multi-step management. Evaluator
alignment with anesthesiologists improves across both training stages, while the best decision quality is accompanied by a 74.3-second P95 Latency. These results show that aggregate performance alone does not establish safe, timely longitudinal decision-making. We release our code at https://zjudbxai.github.io/AnesTRACE/.
Read the full abstract ▾
All authors Ziwei Huang, Qi Gao, Zhe Ji, Yuanyuan Yao, Fengjiang Zhang, Min Yan, Zhongle Xie, Gang Chen
Computer Vision
Gradient-Guided Decoupled Adaptation for Geospatial Vision-Language Models
Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan et al. (+1 more)
Existing geospatial
vision-
language models (Geo-VLMs) typically optimize diverse geospatial tasks through a unified multi-task adaptation paradigm without explicitly accounting for the heterogeneous optimization characteristics. Our empirical observations reveal heterogeneous gradient characteristics across tasks, including
vision-
language differences, intra-branch gradient relationships, and task interference, which hinder effective multi-task optimization. Motivated by these observations, we propose Gradient-Guided Decoupled Adaptation (G2DA), a gradient-aware optimization framework for multi-task Geo-VLM learning. G2DA first partitions tasks into
vision- and
language-centric groups through gradient-guided cross-modal decoupling. It then constructs modality-specific curricula based on task gradient similarity and employs bidirectional rehearsal to mitigate the recency effects introduced by sequential optimization. We evaluate G2DA on three Geo-VLM
benchmarks using six InternVL3 and Qwen3.5-VL variants, along with GeoChat and GeoLLaVA. Across all 24
benchmark-model combinations, G2DA consistently outperforms representative baselines, improving over the strongest competitor by 3.08, 4.30, and 2.81 percentage points on UrBench-MCQ, XLRS-Bench-Lite, and VRS-Bench-VQA, respectively. These results demonstrate the effectiveness of gradient-guided task organization for Geo-VLM adaptation.
Read the full abstract ▾
All authors Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan, Shenhao Wang
Computer Vision
REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection
Aleksandr Gushchin, Khaled Abud, Georgii Bychkov et al. (+5 more)
AI-generated image detectors are often evaluated on
benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on
dataset-specific cues and fail on unfamiliar generators or processed images. Existing
datasets provide limited support for evaluating these challenges jointly across diverse visual content. We introduce REALIS, a
dataset of 1.43 million real and synthetic images generated by 42 modern text-to-image models, including the latest proprietary systems such as Nano Banana 2. REALIS combines
prompts derived from real images, quality filtering, and stratified sampling to reduce class-specific shortcuts while preserving content diversity. We further introduce REALIS-Expert, a stress-test subset for high-quality synthetic images, where real and generated samples are selected with closely matched semantic and visual characteristics. We also propose a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing. Based on REALIS, our
benchmark evaluates pretrained detectors, fine-tuned models, and zero-shot
vision-
language models under generator and post-processing shifts. On the hardest processed split, the best pretrained conventional detector achieves 0.550 ROC-AUC, compared with 0.752 for the best REALIS-trained detector. REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use.
Read the full abstract ▾
All authors Aleksandr Gushchin, Khaled Abud, Georgii Bychkov, Ekaterina Shumitskaya, Artem Filippov, Sergey Lavrushkin, Dmitriy S. Vatolin, Anastasia Antsiferova
Computer Vision
Region-Local Copula Evidence Fusion for Heterogeneous Remote Sensing Change Detection
Zhiyuan Ji, Junjun Yin and Jian Yang
Superpixel copula models provide stable regional evidence for heterogeneous remote sensing change detection, but a single label per region limits localization within mixed superpixels. This letter develops a region-local copula evidence fusion method that retains the regional decision structure while introducing spatially varying local dependence anomalies. Independently fitted local models characterize departures from unchanged cross-image relationships. Reference ranking and an upper-tail gate transform these anomalies for fusion with continuous regional confidence. We derive the resulting regiondependent local decision threshold and identify a condition under which gating is equivalent to reparameterizing ungated fusion. On Lake and UK, whole-image optimized configurations achieve kappa coefficients of 0.78136 and 0.90817 and improve mixedregion and boundary decisions. Four-fold retrospective spatial validation over ten training subsets confirms complementary local information, with ungated reference fusion increasing mean kappa by 0.00693 and 0.01793. Fixed gating yields a larger UK gain of 0.03353 but only 0.00041 on Lake. These results support regional-local dependence interaction, while showing that calibration and gating have scene-dependent benefits.
Read the full abstract ▾
Computer Vision
ProDyGS: Dynamic Gaussian Splatting from a Single Static Monocular Camera
Ugo Leone Cavalcanti, Fabio Tosi, Matteo Poggi et al. (+4 more)
We present ProDyGS, a novel dynamic 3D Gaussian Splatting framework for high-quality novel view synthesis from
videos captured by a single static camera. While existing methods rely on multi-view setups or significant camera motion for
geometric constraints, our approach addresses the challenging scenario where multi-view supervision is completely absent. We overcome this limitation by generating synthetic multi-view supervision through depth-guided proxy image synthesis. Specifically, we estimate temporally consistent depth maps using foundational monocular depth networks, then construct 3D Gaussian representations that generate proxy images from arbitrary viewpoints. A deformation network learns temporal dynamics by warping canonical Gaussians using this augmented supervision. Experiments on the DyNeRF
dataset demonstrate that our method achieves state-of-the-art performance while requiring only monocular depth estimation as external supervision, outperforming approaches that rely on stronger priors such as scene flow.
Read the full abstract ▾
All authors Ugo Leone Cavalcanti, Fabio Tosi, Matteo Poggi, Andrea Conti, Vladimir Zlokolica, Valerio Cambareri, Stefano Mattoccia
Computer Vision
DPAMixerSR: An Efficient Degradation-Pattern-Aware Model for Image Super-Resolution
Song-Li Wu, Haonan Jiang, Jixuan Fan et al. (+3 more)
While content-adaptive schemes have delivered notable advances in image super-resolution (SR), existing approaches typically focus on texture complexity and ignore intrinsic degradation factors (e.g., blur kernels or noise patterns), leading to suboptimal computation allocation and reconstruction performance. To remedy this, we propose DPAMixerSR, a degradation-pattern-aware framework that enables efficient SR through adaptive sparse computation. We design a lightweight Perceptual Degradation Ranking (PDR) module partitions the image into severely and mildly degraded patches, which are routed to the Adaptive Sparse Processing (ASP) and a lightweight convolutional branch, respectively. ASP performs structure-aligned, multi-scale sparse propagation and bidirectional refinement, while the convolutional branch enhances efficiency in mildly degraded regions. By coupling degradation-driven routing with structure-aligned sparse processing, DPAMixerSR establishes a self-regulating framework that dynamically balances computational efficiency and reconstruction fidelity. Extensive experiments on various SR tasks demonstrate that our DPAMixerSR achieves superior structural restoration and perceptual fidelity with markedly reduced computational overhead, providing a novel and scalable framework for degradation-aware, resource-efficient SR.
Read the full abstract ▾
All authors Song-Li Wu, Haonan Jiang, Jixuan Fan, Yufei Huo, Chubin Zhang, Yansong Tang
Computer Vision
MM-OPD: Towards One More Bottleneck Between Perception and Reasoning
Jintao Tong, Yujing Lou, Zhanming Shen et al. (+6 more)
Recent
multimodal large
language models (MLLMs) advance visual
reasoning by strengthening both perception and
reasoning, implicitly assuming a process that transitions seamlessly from perception to
reasoning. However, we observe a counterintuitive phenomenon that challenges this assumption: holding the model, question, and decoding fixed, we replace images with their caption or code representations (symbolic views), which seems to be redundant given the clear image structures, but the performance surprisingly improves by 10.2% to 23.6% across model scales and
datasets. We term this performance gap as the Symbolic Visual Gap and then take a closer look at it. Through experiments, we find that although the visual evidence can already appear in the
reasoning trace for the image-input model, the symbolic-view-input model shows much higher
attention to the correct evidence than the image-input model. This suggests that despite good capabilities from current works in perception and
reasoning themselves, another bottleneck exists between perception and
reasoning in selecting perceived visual information as appropriate evidence for subsequent
reasoning. To handle this bottleneck, since the symbolic view steers
attention toward correct evidence and is readily obtained at scale, it provides supervision for evidence selection without manually labeled evidence. Building on this, we introduce MM-OPD, a
multimodal on-policy self-
distillation framework for symbolic-to-visual correction that transfers guidance from symbolic-conditioned behavior to the image-conditioned policy through residual
token-level targets, steering the model toward correct visual evidence. Experiments across
benchmarks and model scales show that MM-OPD improves a broad range of
multimodal abilities, with gains in visual perception, chart and document understanding, mathematical
reasoning, and general VQA.
Read the full abstract ▾
All authors Jintao Tong, Yujing Lou, Zhanming Shen, Jiaqi Gu, Lubin Fan, Ruixuan Li, Yue Wu, Jieping Ye, Yixiong Zou
Computer Vision
RCVLA: 4D Radar-Grounded Semantic Reasoning and Trajectory Arbitration for Autonomous Driving
Lianqing Zheng, Xiaokai Bai, Yixuan Luo et al. (+6 more)
4D radar provides
geometric and motion cues that complement visual semantics, but integrating it into
vision-
language-action (VLA) models requires both radar–
language alignment for semantic
reasoning and explicit use of radar measurements for trajectory refinement and selection. To support these capabilities, we construct Cap4DR with 86,016 radar-image-text samples for
alignment pretraining and OmniHD-QA with 520,161 question-answer pairs for instruction tuning across scene description, key-object
reasoning, occupancy understanding, and trajectory planning. Building on these
datasets, we propose RCVLA, a radar-camera VLA framework consisting of a radar-grounded semantic
reasoning stage (RCVLA-Sem) and a trajectory arbitration stage (RCVLA-Phys). RCVLA-Sem performs gated bidirectional interaction between camera and radar
tokens for driving question answering and reference trajectory generation, while auxiliary heads provide object and occupancy queries. RCVLA-Phys refines reference-guided trajectory candidates through truncated
diffusion conditioned on these queries and cluster-level radar measurements, then calibrates candidate scores using radar-derived time-to-collision risk. On OmniHD-QA, RCVLA-Sem improves CIDEr by 9.92 points and reduces key-object velocity error by $21.9\%$ relative to OmniDrive. RCVLA-Phys further reduces average L2 error from $0.348$ to $0.259\,\mathrm{m}$ and average open-loop collision rate from $0.576\%$ to $0.175\%$ relative to RCVLA-Sem. Ablation studies further show that
language-aligned radar
tokens improve semantic
reasoning, while cluster-level radar measurements and risk calibration improve trajectory arbitration. Code will be released.
Read the full abstract ▾
All authors Lianqing Zheng, Xiaokai Bai, Yixuan Luo, Runwei Guan, Minghao Liu, Zhiqiang Wei, Hui-liang Shen, Xichan Zhu, Zhixiong Ma
Computer Vision · Cross-listed from cs.LG
The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models
Dongsheng Liu, Chao Jin, Wenkui Yang et al. (+7 more)
GUI World Models (GUI-WMs) are increasingly used to predict future states for
agent planning and simulation, yet most existing formulations condition only on the current GUI observation and action. We identify state aliasing, where the vis- ible interface omits transition-relevant environment state, so identical observable conditions can correspond to different valid futures. To diagnose this failure mode, we introduce StateAliasBench, a diagnostic
benchmark that explicitly isolates such ambiguities via strict pairing. We further propose lightweight predictive- state recovery that infers structured state from history and augments otherwise frozen GUI-WMs through a deterministic state interface. Family-specific special- ists provide state recovery across heterogeneous state types, and multi-teacher dis- tillation consolidates them into a single unified
estimator. Experiments show that existing GUI-WMs exhibit systematic failures under observation-only condition- ing, while predictive-state augmentation substantially restores state-sensitive pre- diction across evaluated WMs, preserves generative fidelity, and improves down- stream performance of GUI
agents on AndroidWorld. These results suggest that reliable GUI world modeling should account not only for what is visible, but also for the hidden transition state that determines what happens next.
Read the full abstract ▾
All authors Dongsheng Liu, Chao Jin, Wenkui Yang, Hejin Wang, Junwei Yang, Zeren Zhang, Ziwei Chen, Huaibo Huang, Jie Cao, Ran He
Computer Vision · Cross-listed from cs.IR
Concepts Complement Dense Semantics: Learning Compact Sparse Spaces for Text-Image Retrieval
Yoonseo Kim, Jungwoo Choi, Cheonyoung Park et al. (+3 more)
Cross-modal
retrieval has been advanced by
vision-
language pre-trained models that encode images and texts into a shared dense
embedding space. While dense representations effectively capture overall semantic similarity, they often obscure fine-grained visual-textual information needed for precise cross-modal matching. Recent methods introduce a learned sparse branch to complement dense matching with lexical evidence, but they rely on a redundant
language-model
token space and lack explicit grounding for sparse dimensions. We propose GRASP, a compact and grounded sparse learning framework that mines visual-textual concepts from the corpus. A lightweight sparse head is trained to predict concepts relevant to each image or text, yielding interpretable concept-level evidence that complements dense semantic matching. Extensive experiments show that GRASP improves
retrieval accuracy over the state-of-the-art dense-sparse baselines while yielding a more compact and grounded sparse space.
Read the full abstract ▾
All authors Yoonseo Kim, Jungwoo Choi, Cheonyoung Park, Youngwook Kim, Yongho Song, SeongKu Kang
Computer Vision · Cross-listed from cs.AI
Yuhao Du and Shunian Chen
Attention weights depend on
token counts, which change with the representation of a signal. We study refinement
symmetry: splitting a representation while preserving content, position, visible context, and total mass should preserve its contribution. Building on proportional and quadrature
attention, we show that split invariance forces the local mass factor to be linear for any fixed positive
attention kernel, provided that factor is nondecreasing. For changed representations, a physical coupling bounds
attention error by separating feature change from weight reallocation. In Qwen2.5-Omni-7B, duplicating half the visual
tokens threefold changes 255 of 3,586 MVBench answers under standard
attention; measure weighting preserves every answer under matched visibility. Under natural frame resampling, it reduces distributional drift. At twofold merging of a frozen
video encoding, a five-seed evaluation shows an all-partition-correct accuracy gain of 1.04 percentage points over global count weighting (average group mass) and 0.93 points over standard
attention. The advantage over global count also holds on WorldSense but depends on the compression budget. The result is a representation principle with a measured benefit in robustness across partitions.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Timestep Weighting: A Hidden Key to Effective ELBO-Based Flow-Matching RL
Qinwei Ma, Jingzhe Shi, Simin Fan et al. (+2 more)
ELBO-based reinforcement learning offers a sampler-agnostic approach to fine-tuning flow matching models with reward feedback. Timestep weighting in ELBO-based RL has large impact on performance, and it also provides a unified view (as we show in this work) to understand prediction losses heuristically chosen in prior work, yet it remains under-researched and is often chosen to inherit pretrain configs. We investigate impacts and dynamics of timestep weighting in ELBO-based RL. We show that effective weighting depends on both the reward landscape and stage of learning. (1) Through experiments on controlled CIFAR image generation, complemented by robotics, we investigate how weighting impacts reward-driven updates across noise levels. (2) Through gradient analysis, we reveal distinct patterns of cross-noise coordination across tasks and their
evolution during training. These findings motivate the hypothesis that useful weighting depends on the gap between the policy's current behavior and the behavior favored by the reward. (3) Guided by this analysis, we study simple static weighting, budgeted profile selection, and dynamic schedules that improve performance beyond conventional target choices. Our results establish timestep weighting as an important design choice for flow-matching RL and motivate further research into methods that choose and adapt it throughout learning.
Read the full abstract ▾
All authors Qinwei Ma, Jingzhe Shi, Simin Fan, Ling Li, Alex Lamb
Computer Vision
InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning
Hanqian Li, Sirui Huang, Chen Ling et al. (+12 more)
Table images preserve structural information that are often lost in text serialization, and
reasoning over them requires locating relevant rows, columns, and
cells step by step. Current
multimodal large
language models (MLLMs) encode the whole image once before
reasoning, so they cannot pick up row-, column-, and
cell-level evidence as the question unfolds. Encoder-side table structure and generic interleaved visual chain-of-thought still do not bind each
reasoning step to that structure. We propose InterTab, an Interleaved structure-aware framework for CoT
reasoning over Table images, interleaves chain-of-thought with tool calls that crop structure-aligned table regions. First, we build InterTab-22K, includes
reasoning trajectories in which each step is tied to both a structural location and a bounding box. InterTab is trained in two stages: supervised structure-aware
alignment (SSA) on InterTab-22K teaches the model to interleave
reasoning with structure-aligned crops, and active localization optimization (ALO) further optimizes answer correctness, localization IoU, and output format, while penalizing missing or excessive tool calls. Experiments on nine table
benchmarks show that InterTab improves the average accuracy of its backbone from 68.28% to 73.17% and achieves the best average performance among all compared methods. Code and data will be released soon.
Read the full abstract ▾
All authors Hanqian Li, Sirui Huang, Chen Ling, Jungang Li, Yu Huang, Kening Zheng, Yonghua Hei, Xiangrong He, Shiyi Wang, Pengcheng Zhu, Dongnan Liu, Wei Zhou, Linjian Mo, Nai Ding, Xuming Hu
Computer Vision · Cross-listed from cs.CL
From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models
Jialuo He and Huangxun Chen
The ability of
vision-
language models (VLMs) to abstain from unanswerable questions is as important as their ability to answer answerable ones accurately. Recently, several
benchmarks have emerged to evaluate and improve VLM abstention, but they have substantial limitations. First, samples often contain shortcut cues in images or questions that reveal answerability, while an explicit "unanswerable" option further prevents accurate assessment of spontaneous abstention. Second, as training data, they generally provide only binary labels without fine-grained explanations for deeper supervision. To address these limitations, we introduce Visual Answerability Diagnosis with Rationales (VAD-R), a
benchmark constructed through a two-stage pipeline of shortcut filtering and quality verification to prevent answerability leakage. Each example is annotated with step-by-step rationales and causal evidence-gap labels. Evaluation of state-of-the-art open- and closed-source VLMs on VAD-R reveals limited spontaneous abstention, with average recall rates of only 11.4% and 16.3%, respectively. Probing analyses show that hidden-state representations in certain layers can effectively distinguish answerability, yet this distinction fails to manifest in final responses. Motivated by this observation, we introduce Rep2Act, a representation-to-action
alignment method that translates latent answerability awareness into explicit abstention decisions. Rep2Act improves action accuracy on VAD-R from 56.67% to 86.33% for Qwen2.5-VL-3B and from 59.33% to 88.67% for Qwen2.5-VL-7B. On the out-of-distribution TUBench, Rep2Act achieves an average F1 score of 53.3% with only a 3B model, surpassing the closed-source GPT-4 Turbo and GPT-4o by 16.2% and 1.1%, respectively.
Read the full abstract ▾
Computer Vision
DraftAttention2: Fast Video Diffusion with Low-Resolution-Guided Mixed-Precision Attention
Rui Ding, Haopeng Li, Weize Ma et al. (+10 more)
Video generation has broad applications in content creation and entertainment.
Diffusion transformers have advanced the quality of generated
videos, but
attention over spatiotemporal
tokens becomes increasingly expensive as
video resolution and duration increase. We present DraftAttention2, a training-free framework that uses the low-resolution draft
attention map to jointly select
attention blocks and assign their numerical precision. Specifically, spatial 2D average- and max-pooled queries and keys capture complementary regional statistics to estimate block importance, and a shared ranking assigns higher precision to important blocks, lower precision to less important retained blocks, and skips the rest under configurable budgets. Our analysis separates sparsification error from
attention-weighted quantization error, establishing when recovering skipped interactions with low-bit computation tightens the output-error bound. This analysis motivates retaining more interactions at low precision while reserving higher precision for blocks with larger
attention mass. To translate these fine-grained assignments into practical speedups, we further develop fused operand preparation and a single
attention kernel with precision-specific phases, sharing data movement, softmax statistics, and output accumulation across precisions. Experiments demonstrate that our method achieves a superior quality-efficiency trade-off over existing efficient
video generation methods. Notably, its advantage is particularly pronounced for few-step
video diffusion, where jointly combining sparsity with 4- and 8-bit mixed-precision computation substantially improves generation quality while retaining significant acceleration. Code is available at https://github.com/anemoi-project/anemoi
Read the full abstract ▾
All authors Rui Ding, Haopeng Li, Weize Ma, Yufa Zhou, Yitong Li, Xiaoling Zhou, Jiashuo Cao, Liyang Li, Hua Geng, Jiuxiang Gu, Jun Lin, Enze Xie, Xuan Shen
Computer Vision · Cross-listed from cs.RO
World SLAM Model: Joint World Modeling for SLAM and Navigation
Minghui Qin, Yijun Yuan, Weicheng Zheng et al. (+7 more)
We introduce World SLAM Model (WSM), a unified framework that brings the SLAM paradigm directly into downstream navigation. Rather than treating SLAM merely as an upstream module that provides poses, maps or
tokens, WSM adopts its core mechanisms, including incremental state updates with persistent
memory and backend refinement of accumulated errors, to maintain a consistent world state during interaction. Given the current observation and a navigation goal, WSM predicts future visual states and jointly estimates their camera motion and dense geometry, grounding visual prediction in an evolving spatial world state. This spatial state is continuously updated as new observations arrive and provides the basis for action generation and closed-loop navigation. WSM is trained end-to-end with a joint navigation–SLAM objective, enabling downstream navigation to benefit directly from SLAM-style state maintenance and refinement while preserving accurate
geometric estimation. Experiments demonstrate improved navigation performance together with strong SLAM accuracy, highlighting the potential of SLAM as an intrinsic mechanism for long-horizon world modeling and embodied interaction.
Read the full abstract ▾
All authors Minghui Qin, Yijun Yuan, Weicheng Zheng, Kenan Li, Weibang Wang, Chang Sun, Junhao Huang, Anmin Liu, Yicheng Yao, Hang Zhao
Computer Vision
Levy-Driven Correspondence Estimation for Registration
Qianliang Wu, Jiaqi Yang, Wankou Yang et al. (+4 more)
Finding reliable point correspondences is difficult when point clouds have low overlap or undergo non-rigid deformation. Iterative refinement can correct uncertain matches, but costly network evaluations limit the number of updates. We present LevyMatch, a Lévy-driven method that uses random jumps to refine a soft matching
matrix. At each step, a network uses the current matching state and
geometric information to predict a target matching
matrix. A Brownian reference bridge gives an explicit formula for the update toward this target. A Gamma random clock sets the time step for each update. The updated matches provide new
geometric feedback for the next target prediction. We further propose a fixed front-loaded Gamma policy that assigns more expected clock time to early updates and less to later ones, without retraining or extra network evaluations. Reordering the same sampled Gamma increments shows that placing larger increments early gives higher accuracy than placing them late. On 4DMatch and 4DLoMatch, our method improves both non-rigid feature matching recall (NFMR) and inlier ratio (IR) over the compared methods. The front-loaded policy achieves 93.09% NFMR and 92.11% IR on 4DMatch, and 82.79% NFMR and 79.07% IR on 4DLoMatch.
Read the full abstract ▾
All authors Qianliang Wu, Jiaqi Yang, Wankou Yang, Le Hui, Jin Xie, Jian Yang, Yaqing Ding
Computer Vision
GAUGE: Group-Wise View-Inconsistency Rectification for Feed-Forward 4D Tracking
Zhuoqian Feng, Weixing Chen, Ziliang Chen et al. (+2 more)
Feed-forward models regress dense 3D point trajectories directly from monocular
video, yet the residual after global
alignment is substantial and lacks a structural explanation. Measured on dynamic query points across models and
datasets, the error concentrates along the view direction, while the scale correction each motion group requires differs. The predicted displacement direction nevertheless supports reliable grouping, with a median angle far below the 90{\deg} random baseline. The systematic part of the residual is therefore a family of radial degrees of freedom per motion group, along directions 2D observations cannot constrain. We call it group-wise view inconsistency. We present GAUGE (Group-wise Adaptive Unsupervised Gauge Estimation), a training-free and model-agnostic post-hoc module. It recovers motion groups from direction consistency and spatial connectivity, then estimates a per-frame radial scale and group-level translation from 1% to 5% metric anchors, four degrees of freedom per group and frame. On dynamic query points of eight trackers, including D4RT, 4RC and SM4RT, our correction lowers endpoint error by 15.1% to 62.6% over the uncorrected predictions, while spending the same anchors on gradient fine-tuning improves the same models by only -1.1% to 15.2%. Code is publicly available at https://github.com/HCPLab-SYSU/GAUGE.
Read the full abstract ▾
All authors Zhuoqian Feng, Weixing Chen, Ziliang Chen, Yang Liu, Liang Lin
Computer Vision
SPACE: Sparse Predictive Attractor via Counterfactual Eviction for Streaming Video Memory
Hongjin Niu, Weizhan Zhang, Shuo Bao et al. (+4 more)
Fixed-capacity streaming
video memory requires repeated eviction decisions whose effects accumulate over time. Yet existing policies are evaluated primarily in terms of retained information or downstream accuracy, leaving how repeated updates alter the futures supported by
memory largely unexamined. We define a
memory's predictive state as the future representations supported by its retained history and formulate eviction as counterfactual control over transitions in this space. We introduce SPACE (Sparse Predictive Attractor via Counterfactual Eviction), which uses a frozen multi-horizon JEPA to predict the future representations induced by alternative eviction actions. Counterfactual utility identifies future-useful alternatives, while slow predictive-basin geometry determines when to correct avoidable drift and when to adapt to sustained predictive change, without online parameter updates. We further introduce MABS-Bench, which evaluates future-task sufficiency, within-regime predictive stability, transition responsiveness, and perturbation recovery under matched causal streams and
memory budgets. Across multiple
video datasets, SPACE yields consistent improvements in
dataset-native task performance while reducing predictive-state drift.
Read the full abstract ▾
All authors Hongjin Niu, Weizhan Zhang, Shuo Bao, Jiahao Wang, Muyan Jiao, Kairui Wen, Yong-Jin Liu