Computer Vision
Emergent Communication between Heterogeneous Visual Agents through Decentralized Learning
Mikako Ochiai, Masatoshi Nagano and Tadahiro Taniguchi
Symbols are shared, but perception is private. We study emergent communication between heterogeneous visual
agents through decentralized learning, asking what visual information can become shareable when
agents have different visual representations. Instead of optimizing messages through a shared external communicative objective, our
agents exchange only discrete
token sequences and update their own models using local perceptual evidence. This setting focuses on an underexplored aspect of emergent communication, examining whether common symbols can arise without shared perceptual access, and how the similarity between private visual spaces constrains the content and
symmetry of the resulting
language. We instantiate this setting in the Metropolis-Hastings Captioning Game (MHCG), where two
agents collaboratively form shared captions by exchanging proposed
token sequences that a listener accepts or rejects using an MH-style criterion evaluated against its own visual features. We compare three pairings of frozen visual encoders, with
agents starting from randomly initialized text modules. Experiments on MS-COCO show that MHCG produces visually informative shared
token sequences that outperform a no-communication baseline in cross-
agent alignment, visual-feature prediction, and image-text
retrieval; all cross-
agent metrics decline as encoder mismatch increases. Moderate encoder heterogeneity reduces the number of shared sequences while preserving per-sequence visual specificity, whereas stronger encoder heterogeneity yields fewer, coarser, and more asymmetric sequences. Ablations show that listener-side MH acceptance is critical for avoiding degenerate
token formation. These results suggest that shared symbols can arise from local perceptual evaluation alone, with visual representational similarity across encoders shaping both the content and
symmetry of the resulting
language.
Read the full abstract ▾
Computer Vision
Kaixuan He, Song Chen and Yi Kang
Vision Transformers (ViTs) incur significant computational overhead due to the quadratic complexity of self-
attention relative to the
token sequence length. While existing
token reduction methods mitigate this issue, they predominantly rely on fixed heuristic metrics, predefined ratios, or static offline masks, which lack the adaptability to capture input-dependent redundancy during inference. In this paper, we propose DORA (Dynamic Online Reinforcement
Agent), the first reinforcement learning (RL)-driven online inference framework for dynamic
token merging in ViTs. We formulate the merging process as a sequential Markov Decision Process (MDP), where a lightweight RL
agent determines the merging strategy for each
Transformer block based on the current feature state and layer-specific context. To balance computational efficiency and feature fidelity, the
agent is optimized via a dense reward function incorporating a non-linear
distillation-based penalty. We implement an asymmetric Actor-Critic architecture that utilizes a high-capacity Critic for stable offline training while retaining a minimal Actor head for low-computation online inference. Evaluations across multiple ViT scales (Tiny to Large) demonstrate that DORA improves the accuracy-efficiency Pareto front compared to current baselines. Under strict negligible accuracy-drop constraints (<= 0.05%), DORA achieves up to a 12.66%
token merging rate, and delivers up to a 569.7% relative improvement over the most efficient baseline. On ImageNet-1K, under aligned accuracy constraints, DORA achieves up to a 76% relative improvement in computational savings compared to state-of-the-art methods. Furthermore, on out-of-distribution (OOD)
benchmarks such as ImageNet-A and ImageNet-C, DORA attains a relative efficiency advantage of over 430%.
Read the full abstract ▾
Computer Vision
ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes
Shivam Kumar
We introduce ShapeCodeBench, a synthetic
benchmark for perception-to-program reconstruction: given a rendered raster image, a model must emit an executable drawing program that a deterministic evaluator re-renders and compares with the target. The v1 DSL has four primitives on a 512 x 512 black-on-white canvas, but every instance is generated from a seeded RNG, so fresh held-out sets can be created to reduce exact-instance contamination. We release a frozen eval_v1 split with 150 samples across easy, medium, and hard tiers, scored by exact match, pixel accuracy, foreground IoU, parse success, and execution success. We evaluate an empty-program floor, a classical computer-
vision heuristic, Claude Opus 4.7 at high and max effort, and GPT-5.5 at medium and extra_high
reasoning effort. The heuristic is competitive on easy scenes but collapses when overlaps fuse components; the strongest
multimodal configuration preserves much of the foreground structure but still misses exact match because of small parameter errors. Best overall exact match remains low, so ShapeCodeBench is far from saturated. The
benchmark code, frozen
dataset, run artifacts, and paper sources are released to support independent replication and extension.
Read the full abstract ▾
Computer Vision
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
Yaze Zhao, Yicong Liu, Yixiong Zou et al. (+2 more)
Cross-Domain Few-Shot Learning (CDFSL) aims to adapt large-scale pretrained models to specialized target domains with limited samples, yet the few-shot fine-tuning of
vision-
language models like CLIP remains underexplored. By establishing multiple fine-tuning baselines of CLIP for CDFSL, we find adapter-based methods (e.g., LoRA) consistently outperform
prompt-based ones (e.g., MaPLe), contrary to in-domain scenarios. To make those effective in-domain methods competitive again in CDFSL, we analyze this phenomenon and discover LoRA's superiority stems from rectifying the collapsed
attention of visual CLS
token, enhancing modality
alignment and class separation by focusing on text-related visual regions. Further, we find textual EOS
token exhibit much better
attention to visual samples, and CLIP's standard contrastive loss weakly constrains modality
alignment. Based on these insights, we propose Semantic Probe, a plug-and-play
attention rectification framework for both adapter- and
prompt-based methods. Extensive experiments on four CDFSL
benchmarks validate our rationale, achieving state-of-the-art performance and benefiting both fine-tuning paradigms. Codes will be released.
Read the full abstract ▾
All authors Yaze Zhao, Yicong Liu, Yixiong Zou, Yuhua Li, Ruixuan Li
Computer Vision
Weather-Robust Cross-View Geo-Localization via Prototype-Based Semantic Part Discovery
Chi-Nguyen Tran, Dao Sy Duy Minh, Huynh Trung Kiet et al. (+3 more)
Cross-view geo-localization (CVGL), which matches an oblique drone view to a geo-referenced satellite tile, has emerged as a key alternative for autonomous drone
navigation when GNSS signals are jammed, spoofed, or unavailable. Despite strong recent progress, three limitations persist: (1) global-descriptor designs compress the patch grid into a single vector without separating layout from texture across the view gap; (2) altitude-related scale variation is retained in the learned
embedding rather than marginalized; and (3) multi-objective training relies on hand-tuned scalars over losses on incompatible gradient scales. We propose SkyPart, a lightweight swappable head for patch-based
vision transformers (ViTs) that institutes explicit part grouping over the patch grid. SkyPart has four theory-grounded components: (i) learnable prototypes competing for patch
tokens via single-pass cosine assignment; (ii) altitude-conditioned linear modulation applied only during training, making the
retrieval embedding altitude-free at inference; (iii) a graph-
attention readout over active prototypes; and (iv) a Kendall uncertainty-weighted multi-objective loss whose stationary points are Pareto-stationary. At 26.95M parameters and 22.14 GFLOPs, SkyPart is the smallest among top-performing methods and sets a new state of the art on SUES-200, University-1652, and DenseUAV under a single-pass, no-re-ranking, no-TTA protocol. Its advantage over the strongest baseline widens under the ten-condition WeatherPrompt corruption
benchmark.
Read the full abstract ▾
All authors Chi-Nguyen Tran, Dao Sy Duy Minh, Huynh Trung Kiet, Nguyen Lam Phu Quy, Phu-Hoa Pham, Long Tran-Thanh
Computer Vision
Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation
Seonghoon Yu, Dongjun Nam, Byung-Kwan Lee et al. (+1 more)
Recent think-answer approaches in VLMs, such as Qwen3-VL-Thinking, boost
reasoning performance by leveraging intermediate thinking steps before the final answer, but their computational cost becomes substantial, especially for larger VLMs. To distill such capabilities into compact think-answer VLMs, a primary objective is to improve the student's ability to utilize visual evidence throughout its
reasoning trace, as long think-answer traces suffer from visual forgetting issues. To this end, we introduce a novel think-answer
distillation framework that encourages the student to anchor its thinking on visual information by masking the student's salient
reasoning prefixes. To compensate for such masked textual cues, the student is encouraged to rely more on visual evidence as an alternative source of information during
distillation. Our masking strategies include: 1)
token-wise salient
reasoning-prefix masking, which masks high-influence
reasoning prefixes selectively for each next-
token prediction, and 2) self-paced masking budget scheduling, which gradually increases the masking scale according to
distillation difficulty, measured by the discrepancy between teacher–student distributions. In the
distillation phase, the student is guided by our salient
reasoning-prefix mask, which blocks both future
tokens and salient
reasoning cues, in place of the standard causal mask used for auto-regressive
language modeling. Experimental results show that our approach outperforms recent open-source VLMs, VLM
distillation, and self-
distillation methods on
multimodal reasoning benchmarks, while further analyzes confirm enhanced visual utilization along the student thinking process.
Read the full abstract ▾
All authors Seonghoon Yu, Dongjun Nam, Byung-Kwan Lee, Jeany Son
Computer Vision
Unlocking UML Class Diagram Understanding in Vision Language Models
Artem Naboichenko and René Peinl
Although
Vision Language Models (VLMs) have seen tremendous progress across all kinds of use cases, they still fall behind in answering questions regard-ing diagrams compared to photos. Although progress has been made in the area of bar charts, line charts and other diagrams like that there is still few research concerned with other types of diagrams, e.g. in the computer science domain. Our work presents a
benchmark for visual question answering based on UML class diagrams which is both challenging and manageable. We further construct a large-scale training
dataset with 16.000 image-question-answer triples and show that a LoRA-based finetune easily outperforms Qwen 3.5 27B, which is a recent and well-performing VLM in many other
benchmarks.
Read the full abstract ▾
Computer Vision
Single-Shot HDR Recovery via a Video Diffusion Prior
Chinmay Talegaonkar, Jinshi He, Christopher McKenna et al. (+1 more)
Recent generative methods for single-shot high dynamic range (HDR) image reconstruction show promising results, but often struggle with preserving fidelity to the input image. They require separate models to handle highlights and shadows, or sacrifice interpretability by directly predicting the final HDR image. We address these limitations by re-casting single-shot HDR reconstruction as conditional
video generation and fusing the generated frames into an HDR image. We finetune a
video diffusion model to generate an exposure bracket, conditioned on a low dynamic range (LDR) input. We fuse this image bracket using per-pixel weights predicted by a light-weight UNet. This formulation is simple, interpretable, and effective. Rather than directly hallucinating an HDR image, it explicitly reconstructs the intermediate exposure stack and fuses it into the final output. Our method eliminates the need for separate models across exposure regimes and produces HDR reconstructions with high input fidelity. On quantitative
benchmarks, we outperform state-of-the-art generative baselines with comparable model capacity on several reconstruction metrics. Human evaluators further prefer our results in 72% of pairwise comparisons against existing methods. Finally, we show that this input-conditioned sequence generation and fusion framework extends beyond HDR to other image reconstruction tasks, such as all-in-focus image recovery from a single defocus-blurred input.
Read the full abstract ▾
All authors Chinmay Talegaonkar, Jinshi He, Christopher McKenna, Nicholas Antipa
Computer Vision
RNA-FM: Flow-Matching Generative Model for Genome-wide RNA-Seq Prediction
Yaxuan Song, Jianan Fan, Tianyi Wang et al. (+4 more)
Histopathology whole-slide images (WSIs) are routinely acquired in clinical practice and contain rich tissue morphology but lack direct molecular architecture and functional programs defining pathological states, whereas RNA sequencing (RNA-seq) provides genome-wide transcriptional profiles at substantial cost, thereby motivating WSI-based genome-wide transcriptomic prediction. Existing approaches for predicting gene expression from WSIs predominantly rely on deterministic regression with one-to-one mapping, limiting their ability to capture biological heterogeneity and predictive uncertainty. We propose RNA-FM, a flow-matching generative framework for genome-wide bulk RNA-seq prediction from WSIs. RNA-FM formulates transcriptomic prediction as a continuous-time conditional transport problem, learning a velocity field that maps a simple prior to the target gene expression distribution conditioned on morphologies. By integrating pathway-level structure, RNA-FM enables scalable and biologically interpretable genome-wide gene expression imputation. Extensive experiments demonstrate that RNA-FM consistently outperforms state-of-the-art approaches while maintaining biological meaningfulness. Code is available at https://github.com/YXSong000/RNA-FM.
Read the full abstract ▾
All authors Yaxuan Song, Jianan Fan, Tianyi Wang, Qiuyue Hu, Hang Chang, Heng Huang, Weidong Cai
Computer Vision
Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances
Qirui Wang, Jingyi He, Yining Pan et al. (+2 more)
Functional affordance grounding requires more than recognizing an object: an
agent must localize the specific region that supports an interaction, such as the handle to pull or the button to press. This is difficult for training-free
vision-
language pipelines because actionable regions are often small, visually ambiguous, and repeated across multiple same-category instances in a scene. We propose AFFORDMEM, a framework that grounds 3D functional affordances by remembering geometry at two levels. The first is cross-scene affordance
memory: the
agent maintains a category-level
memory bank of RGB images with affordance regions rendered as overlays, and recalls the most informative examples at query time to guide a frozen VLM toward small operable subregions that text-only prompting consistently misses. The second is in-scene spatial
memory: as the
agent processes the scene, it organizes candidate instances and their 3D spatial relations into a structured scene graph, enabling the
language model to resolve references over distant or currently unobserved candidates such as "the second handle from the top." AFFORDMEM requires no model fine-tuning and no target-scene annotation, using a reusable
memory bank built from source scenes. On SceneFun3D, our method improves AP50 over the prior training-free state of the art by 3.23 on Split 0 and 3.7 on Split 1. Ablation studies support complementary benefits: cross-scene affordance
memory improves fine-grained localization, while in-scene spatial
memory provides the larger gain on spatially qualified queries. The project homepage is available at the project page.
Read the full abstract ▾
All authors Qirui Wang, Jingyi He, Yining Pan, Xulei Yang, Shijie Li
Computer Vision
Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs
Chaeyoung Jung, Kyeongha Rho and Joon Son Chung
Omnimodal Large
Language Models (Omni-LLMs) incur substantial computational overhead due to the large number of
multimodal input
tokens they process, making
token reduction essential for real-world deployment. Existing Omni-LLM
pruning methods typically reduce this cost by selecting
tokens that are important for the current query or strongly aligned with cross-modal cues. However, such strategies can discard evidence that falls outside these criteria, even when needed for different questions or for understanding context beyond aligned audio-visual cues. To address this limitation, we reframe Omni-LLM
token reduction as preserving broad audio-visual context while removing cross-modal redundancy. We propose ContextGuard, an inference-time
token pruning framework built on this principle. ContextGuard predicts coarse visual semantics from audio and prunes
video tokens whose coarse semantics are likely recoverable from audio, while retaining additional
video tokens to preserve localized visual details that audio alone cannot specify. For further compression, our method merges temporally similar
video tokens. The framework requires no downstream LLM fine-tuning and uses only an independently trained lightweight predictor. On Qwen2.5-Omni and
Video-SALMONN2+ at 3B and 7B scales across six audio-visual
benchmarks, ContextGuard outperforms prior inference-time
pruning methods while
pruning more
tokens. Notably, on Qwen2.5-Omni 7B, ContextGuard achieves full-
token-level performance on five of six
benchmarks while
pruning 55% of input
tokens.
Read the full abstract ▾
Computer Vision
HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation
Conglang Zhang, Yifan Zhan, Qingjie Wang et al. (+10 more)
Closed-loop driving simulation requires real-time interaction beyond short offline clips, pushing current driving world models toward autoregressive (AR) rollout. Existing AR
distillation approaches typically rely on frame sinks or student-side degradation training. The former transfers poorly to driving due to fast ego-motion and rapid scene changes, while the latter remains bounded by the teacher's single-pass output length and thus provides only a limited supervision horizon. A natural question is: can the teacher itself be extended via AR rollout to provide unbounded-horizon supervision at bounded
memory cost? The key difficulty is that a standard teacher drifts under its own predictions, contaminating the supervision it provides. Our key insight is to make the teacher rollout-capable, ensuring reliable supervision from its own AR rollouts. This is instantiated as HorizonDrive, an anti-drifting training-and-
distillation framework for AR driving simulation. First, scheduled rollout recovery (SRR) trains the base model to reconstruct ground-truth future clips from prediction-corrupted histories, yielding a teacher that remains stable across long AR rollouts. Second, the rollout-capable teacher is extended via AR rollout, providing long-horizon distribution-matching supervision under bounded
memory, while a short-window student aligns to it with teacher rollout DMD (TRD) for efficient real-time deployment. HorizonDrive natively supports minute-scale AR rollout under bounded
memory; on nuScenes, HorizonDrive reduces FID by 52% and FVD by 37%, and lowers ARE and DTW by 21% and 9% relative to the strongest long-horizon streaming baselines, while remaining competitive with single-pass driving
video generators.
Read the full abstract ▾
All authors Conglang Zhang, Yifan Zhan, Qingjie Wang, Zhanpeng Ouyang, Yu Li, Zihao Yang, Xiaoyang Guo, Weiqiang Ren, Qian Zhang, Zhen Dong, Yinqiang Zheng, Wei Yin, Zhengqing Chen
Computer Vision
PointForward: Feedforward Driving Reconstruction through Point-Aligned Representations
Cheng Chi, Xianqi Wang, Hongcheng Luo et al. (+9 more)
High-fidelity reconstruction of driving scenes is crucial for autonomous driving. While recent feedforward 3D Gaussian Splatting (3DGS) methods enable fast reconstruction, their per-pixel Gaussian prediction paradigm often suffers from multi-view inconsistency and layering artifacts. Moreover, existing methods often model dynamic instances via dense flow prediction, which lacks explicit cross-view correspondence and instance-level consistency. In this paper, we propose PointForward, a feedforward driving reconstruction framework through point-aligned representations. Unlike pixel-aligned methods, we initialize sparse 3D queries in world space and aggregate multi-view image information via spatial-temporal fusion onto these queries, enforcing explicit cross-view consistency in a single feedforward pass. To handle scene dynamics, we introduce scene graphs that explicitly organize moving instances during reconstruction. By leveraging 3D bounding boxes, our method enables instance-level motion propagation and temporally consistent dynamic representations. Extensive experiments demonstrate that PointForward achieves state-of-the-art performance on large-scale driving
benchmarks. The code will be available upon the publication of the paper.
Read the full abstract ▾
All authors Cheng Chi, Xianqi Wang, Hongcheng Luo, Mingfei Tu, Gangwei Xu, Zehan Zhang, Bing Wang, Guang Chen, Hangjun Ye, Sida Peng, Xin Yang, Haiyang Sun
Computer Vision
Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration
Mingtao Xian, Yifeng Yang, Qinying Gu et al. (+2 more)
Multimodal Large
Language Models (MLLMs) have shown strong performance in multi-image cross-modal
retrieval, yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance. Through empirical analysis, we identify a phenomenon termed Logit-
Attention Divergence, in which output logits are heavily biased while internal
attention maps remain well-aligned with relevant visual evidence. This observation reveals a fundamental limitation of existing logit-level calibration methods such as PriDe. Based on this insight, we propose a training-free,
attention-guided debiasing framework that leverages intrinsic
attention signals for instance-level correction at inference time, requiring only a minimal calibration set with negligible computational overhead. Experiments on MS-COCO-based
benchmarks show that our method substantially improves permutation invariance and achieves state-of-the-art performance, enhancing accuracy by over 40% compared to baselines. Code is available at https://github.com/brightXian/LAD.
Read the full abstract ▾
All authors Mingtao Xian, Yifeng Yang, Qinying Gu, Xinbing Wang, Nanyang Ye
Computer Vision
A Mixture Autoregressive Image Generative Model on Quadtree Regions for Gaussian Noise Removal via Variational Bayes and Gradient Methods
Shota Saito, Yuta Nakahara, Kohei Horinouchi et al. (+3 more)
This paper addresses the problem of image denoising for grayscale images. We propose a probabilistic image generative model that combines a quadtree region-partitioning model with a mixture autoregressive model, and propose a framework that reduces MAP (maximum a posteriori)-estimation-based denoising to the maximization of a variational lower bound. To maximize this lower bound, we develop an
algorithm that alternately applies variational Bayes and gradient methods. We particularly demonstrate that the gradient-based update rule can be computed analytically without numerical computation or approximation. We carried out some experiments to verify that the proposed
algorithm actually removes image noise and to identify directions for future improvement.
Read the full abstract ▾
All authors Shota Saito, Yuta Nakahara, Kohei Horinouchi, Naoki Ichijo, Manabu Kobayashi, Toshiyasu Matsushima
Computer Vision · Cross-listed from eess.IV
NexOP: Joint Optimization of NEX-Aware k-space Sampling and Image Reconstruction for Low-Field MRI
Tal Oved and Efrat Shimron
Modern low-field
magnetic resonance imaging (MRI) technology offers a compelling alternative to standard high-field MRI, with portable, low-cost systems. However, its clinical utility is limited by a low Signal-to-Noise Ratio (SNR), which hampers diagnostic image quality. A common approach to increase SNR is through repetitive signal acquisitions, known as NEX, but this results in excessively long scan durations. Although recent work has introduced methods to accelerate MRI scans through k-space sampling optimization, the NEX dimension remains unexploited; typically, a single sampling mask is used across all repetitions. Here we introduce NexOP, a deep-learning framework for joint optimization of the sampling and reconstruction in multi-NEX acquisitions, tailored for low-SNR settings. NexOP enables optimizing the sampling density probabilities across the extended k-space-NEX domain, under a fixed sampling-budget constraint, and introduces a new deep-learning architecture for reconstructing a single high-SNR image from multiple low-SNR measurements. Experiments with raw low-field (0.3T)
brain data demonstrate that NexOP consistently outperforms competing methods, both quantitatively and qualitatively, across diverse acceleration factors and tissue contrasts. The results also demonstrate that NexOP yields non-uniform sampling strategies, with progressively decreasing sampling across repetitions, hence exploiting the NEX dimension efficiently. Moreover, we present a theoretical analysis supporting these numerical observations. Overall, this work proposes a sampling-reconstruction optimization framework highly suitable for low-field MRI, which can enable faster, higher-quality imaging with low-cost systems and contribute to advancing affordable and accessible healthcare.
Read the full abstract ▾
Computer Vision
The Midas Touch for Metric Depth
Yu Ma, Zizhan Guo, Zuyi Xiong et al. (+5 more)
Recent advances have markedly improved the cross-scene generalization of relative depth estimation, yet its practical applicability remains limited by the absence of metric scale, local inconsistencies, and low computational efficiency. To address these issues, we present \emph{Midas Touch for Depth} (MTD), a mathematically interpretable approach that converts relative depth into metric depth using only extremely sparse 3D data. To eliminate local scale inconsistencies, it applies a segment-wise recovery strategy via sparse graph optimization, followed by a pixel-wise refinement strategy using a discontinuity-aware geodesic cost. MTD exhibits strong generalization and achieves substantial accuracy improvements over previous depth completion and depth estimation methods. Moreover, its lightweight, plug-and-play design facilitates deployment and integration on diverse downstream 3D tasks. Project page is available at https://mias.group/MTD.
Read the full abstract ▾
All authors Yu Ma, Zizhan Guo, Zuyi Xiong, Haoran Zhang, Yi Feng, Hongbo Zhao, Hanli Wang, Rui Fan
Computer Vision
TB-AVA: Text as a Semantic Bridge for Audio-Visual Parameter Efficient Finetuning
Seongah Kim, Dinh Phu Tran, Hyeontaek Hwang et al. (+3 more)
Audio-visual understanding requires effective
alignment between heterogeneous modalities, yet cross-modal correspondence remains challenging when temporally aligned audio and visual signals lack clear semantic correspondence. We propose to use text as a semantic anchor for audio-visual representation learning. To this end, we introduce a parameter-efficient adaptation framework built on frozen audio and visual encoders, centered on Text-Bridged Audio-Visual Adapter (TB-AVA), which enables text-mediated interaction between audio and visual streams. At the core of TB-AVA, Gated Semantic Modulation (GSM) selectively modulates feature channels based on text-inferred semantic relevance. We evaluate the proposed approach on multiple
benchmarks, including AVE, AVS, and AVVP, where the proposed framework achieves state-of-the-art performance, demonstrating text as an effective semantic anchor for parameter-efficient fine-tuning (PEFT) in audio-visual learning.
Read the full abstract ▾
All authors Seongah Kim, Dinh Phu Tran, Hyeontaek Hwang, Saad Wazir, Duc Do Minh, Daeyoung Kim
Computer Vision
Dynamic Execution Commitment of Vision-Language-Action Models
Feng Chen, Xianghui Wang, Yuxuan Chen et al. (+4 more)
Vision-
Language-Action (VLA) models predominantly adopt action chunking, i.e., predicting and committing to a short horizon of consecutive low-level actions in a single forward pass, to amortize the inference cost of large-scale backbones and reduce per-step latency. However, committing these multi-step predictions to real-world execution requires balancing success rate against inference efficiency, a decision typically governed by fixed execution horizons tuned per task. Such heuristics ignore the state-dependent nature of predictive reliability, leading to brittle performance in dynamic or out-of-distribution settings. In this paper, we introduce A3, an Adaptive Action Acceptance mechanism that reframes dynamic execution commitment as a self-speculative prefix verification problem. A3 first computes a trajectory-wise consensus score of actions via group sampling, then selects a representative draft and prioritizes downstream verification. Specifically, it enforces: (1) consensus-ordered conditional invariance, which validates low-consensus actions by judging whether they remain consistent when re-decoded conditioned on high-consensus actions; and (2) prefix-closed sequential consistency, which guarantees physical rollout integrity by accepting only the longest continuous sequence of verified actions starting from the beginning. Consequently, the execution horizon emerges as the longest verifiable prefix satisfying both internal model logic and sequential execution constraints. Experiments across diverse VLA models and
benchmarks demonstrate that A3 eliminates the need for manual horizon tuning while achieving a superior trade-off between execution robustness and inference throughput.
Read the full abstract ▾
All authors Feng Chen, Xianghui Wang, Yuxuan Chen, Boying Li, Yefei He, Zeyu Zhang, Yicheng Wu
Computer Vision
TCP-SSM: Efficient Vision State Space Models with Token-Conditioned Poles
Sara Shoouri, Morteza Tavakoli Taba and Hun-Seok Kim
State Space Models (SSMs) have emerged as a compelling alternative to
attention models for long-range
vision tasks, offering input-dependent recurrence with linear complexity. However, most efficient SSM variants reduce computation cost by modifying scan routes, resolutions, or traversal patterns, while largely leaving the recurrent dynamics implicit. Consequently, the model's state-dependent
memory behavior is difficult to control, particularly in compact backbones where long scan paths can exceed the effective
memory horizon. We propose
Token-Conditioned Poles SSM (TCP-SSM), a structured selective SSM framework that improves efficiency while making recurrence dynamics explicit and interpretable through stable poles. TCP-SSM builds each scan operator with 1) real poles that model monotone or sign-alternating decay, and 2) complex-conjugate poles that capture damped oscillatory responses. Using bounded radius and angle modulation, TCP-SSM converts shared base poles into
token-dependent poles, allowing each scan step to adapt its
memory behavior to the current visual
token while preserving pole stability. For practical scalability, we integrate grouped pole sharing with a lightweight low-rank input pathway, yielding an efficient scan operator that preserves linear-time scan complexity. Across image classification, semantic
segmentation, and object detection, TCP-SSM reduces SSM computation complexity up to 44% in
Vision Mamba-style models while maintaining or surpassing baseline accuracy.
Read the full abstract ▾
Computer Vision
When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs
Fanpu Cao, Xin Zou, Xuming Hu et al. (+1 more)
Multimodal large
language models (MLLMs) have become a key interface for visual
reasoning and grounded question answering, yet they remain vulnerable to visual
hallucinations, where generated responses contradict image content or mention nonexistent objects. A central challenge is that
hallucination is not always caused by a simple lack of visual
attention: the model may still assign substantial
attention mass to image
tokens while internally drifting toward an incorrect answer. In this paper, we show that the high-frequency structure of visual
attention, measured by layer-wise
Laplacian energy, reveals both the layer where hallucinated preferences emerge and the layer where the ground-truth answer transiently recovers. Building on this finding, we propose LaSCD (
Laplacian-Spectral Contrastive Decoding), a training-free decoding strategy that selects informative layers via
Laplacian energy and remaps next-
token logits in closed form. Experiments on
hallucination and general
multimodal benchmarks show that LaSCD consistently reduces
hallucination while preserving general capabilities, highlighting its potential as a faithful decoding paradigm. The code is available at https://github.com/macovaseas/LaSCD.
Read the full abstract ▾
All authors Fanpu Cao, Xin Zou, Xuming Hu, Hui Xiong
Computer Vision
ScribbleDose: Scribble-Guided Dose Prediction in Radiotherapy
Zhenxi Zhang, Yitao Zhuang, Yao Pu et al. (+7 more)
Anatomical structure masks are widely adopted in radiotherapy dose prediction, as they provide explicit
geometric constraints that facilitate structure-dose coupling. However, conventional manual delineation of these masks requires precise annotation of structure boundaries relevant to radiotherapy, which is time-consuming and labor-intensive. To address these limitations, we propose a scribble-guided dose prediction framework that relies solely on anatomical structures annotated with sparse scribbles. Specifically, we design a Scribble Completion Module (SCM) to generate dense anatomical masks by propagating sparse scribble labels to semantically similar voxels. During the propagation process, a supervoxel-based regularization is introduced to preserve
geometric boundary consistency to ensure anatomical plausibility. Furthermore, we propose a Structure-Guided Dose Generation Module (SGDGM) to strengthen the correspondence between sparse structural cues and dose distribution. Herein, the completed dense masks derived from scribbles serve as structural guidance to condition the dose prediction network. This scribble-mask-dose consistency encourages high-dose concentration within target volumes while effectively sparing surrounding organs-at-risk. Extensive experiments on the open-source GDP-HMM
dataset demonstrate that the proposed method maintains superior dose prediction performance while substantially reducing annotation cost, providing a practical paradigm for dose prediction under sparse structural annotation. The code and reannotated scribbles are made publicly available at https://github.com/iCherishxixixi/ScribbleDose.
Read the full abstract ▾
All authors Zhenxi Zhang, Yitao Zhuang, Yao Pu, Peixin Yu, Zirong Li, Yan Xia, Hui Li, Bin Li, Fuchen Zheng, Ge Ren
Computer Vision · Cross-listed from cs.LG
VNDUQE: Information-Theoretic Novelty Detection using Deep Variational Information Bottleneck
Aryan Gondkar, Hayder Radha and Yiming Deng
Detecting out-of-distribution (OOD) samples is critical for safe deployment of neural networks in safety-critical applications. While maximum softmax probability (MSP) provides a simple baseline, it lacks theoretical grounding and suffers from miscalibration. We propose VNDUQE (VIB-based Novelty Detection and Uncertainty Quantification for Nondestructive Evaluation), which investigates novelty detection through the Deep Variational Information Bottleneck (VIB), which explicitly constrains information flow through learned representations. We train VIB models on MNIST with held-out digit classes and evaluate OOD detection using information-theoretic metrics: KL divergence and prediction
entropy. Our results reveal complementary detection signals: KL divergence achieves perfect detection (100% AUROC on noise) on far-OOD samples (noise, domain shift), while prediction
entropy excels at near-OOD detection (94.7% AUROC on novel digit classes). A parallel detection strategy combining both metrics achieves 95.3% average AUROC and 92% true positive rate at 5% false positive rate, which is a 32 percentage point improvement over baseline MSP (85.0% AUROC, 60.1% TPR). Compression via the information bottleneck principle ($\beta=10^{-3}$) reduces Expected Calibration Error by 38%, demonstrating that information-theoretic constraints produce fundamentally more reliable uncertainty estimates. These findings directly support active learning with expensive computational oracles, where well-calibrated novelty detection enables principled threshold selection for oracle queries.
Read the full abstract ▾
Computer Vision
The DAWN of World-Action Interactive Models
Hongbo Lu, Liang Yao, Chenghao He et al. (+6 more)
A plausible scene
evolution depends on the maneuver being considered, while a good maneuver depends on how the scene may evolve. Existing World Action Models (WAMs) largely miss this reciprocity, treating world prediction and action generation as either isolated parallel branches or rigid predict-then-plan pipelines. We formalize this perspective as World-Action Interactive Models (WAIMs), and instantiate it in autonomous driving with DAWN (Denoising Actions and World iNteractive model), a simple yet strong latent generative baseline. DAWN operates in a compact semantic latent space and couples a World Predictor with a World-Conditioned Action Denoiser: the predicted world hypothesis conditions action denoising, while the denoised action hypothesis is fed back to update the world prediction, so that both are recursively refined during inference. Rather than eliminating test-time world
evolution altogether or rolling out the full future in pixel space, DAWN performs a short explicit latent rollout that is sufficient to support long-horizon trajectory generation in complex interactive scenes. Experiments show that DAWN achieves strong planning performance and favorable safety-related results across multiple autonomous driving
benchmarks. More broadly, our results suggest that interactive world-action generation is a principled path toward truly actionable world models.
Read the full abstract ▾
All authors Hongbo Lu, Liang Yao, Chenghao He, Haoyu Wang, Xiang Gu, Xianfei Li, Wenlong Liao, Tao He, Pai Peng
Computer Vision
GeoR-Bench: Evaluating Geoscience Visual Reasoning
Yushuo Zheng, Zicheng Zhang, Huiyu Duan et al. (+7 more)
Geoscience intelligence is expected to understand, reason about, and predict earth system changes to support human decision-making in critical domains such as disaster response,
climate adaptation and environmental protection. Although current research has shown promising progress on specific geoscience tasks, such as remote sensing interpretation, geographic question-answering, existing
benchmarks remain largely task-specific which failing to capture the open-ended real world geoscience problems. As a result, it remains unclear how far current AI systems are from achieving genuine geoscience intelligence. To address this gap, we present GeoR-Bench, a \underline{Bench}mark for evaluating \underline{Geo}science visual \underline{R}easoning through
reasoning informed visual editing tasks. GeoR-Bench contains 440 curated samples spanning 6 geoscience categories and 24 task types, covering earth observation imagery and structured scientific representations such as maps and diagrams. We evaluate outputs along three dimensions, including
reasoning, consistency, and quality.
Benchmark results of 21 closed- and open-source
multimodal models reveal that geoscience
reasoning remains a critical bottleneck. The highest-performing model achieves 42.7% overall strict accuracy, while the best open-source models only get 10.3%. Notably, the visual consistency and image quality of the outputs frequently surpass their scientific accuracy. Ultimately, these findings indicate that current models generate superficially plausible results but fail to capture underlying earth science processes.
Read the full abstract ▾
All authors Yushuo Zheng, Zicheng Zhang, Huiyu Duan, Chunyi Li, Zijian Chen, Ziheng Jia, Yue Shi, Ke Gu, Xiongkuo Min, Guangtao Zhai
Computer Vision · Cross-listed from cs.CL
Checkup2Action: A Multimodal Clinical Check-up Report Dataset for Patient-Oriented Action Card Generation
Sike Xiang, Shuang Chen, Kevin Qinghong Lin et al. (+4 more)
Routine clinical check-up reports combine laboratory measurements, physiological assessments, imaging findings and visually structured information, but rarely tell patients what to do next. Translating them into follow-up actions requires models to connect evidence across pages, tables and modalities, identify clinically relevant issues and communicate next steps without unsupported diagnostic or treatment claims. Yet this report-to-action capability remains poorly benchmarked. We introduce C2A, a
dataset and
benchmark for generating structured Action Cards from
multimodal check-up reports, together with Checkup2Action, a constrained workflow for the task. C2A contains 2,000 de-identified real-world reports covering physical examinations, laboratory tests, cardiovascular assessments and imaging evidence. Each card specifies one issue, its priority, recommended department, follow-up window, patient-facing explanation and questions for clinicians. Our evaluation measures issue coverage and precision, priority consistency, department and timing accuracy, action complexity, usefulness, readability and safety. Experiments across general-purpose and medical large
language models show that no model performs best on every dimension. Clinical experts judged 84% of evaluated outputs fully reasonable and effective. Removing safety constraints increased problem recall from 0.527 to 0.825, but produced more diagnostic overstatement. C2A therefore supports systematic study of the balance between coverage, actionable guidance and safe patient communication.
Read the full abstract ▾
All authors Sike Xiang, Shuang Chen, Kevin Qinghong Lin, Jialin Yu, Yijia Sun, Philip Torr, Amir Atapour-Abarghouei
Computer Vision
XWOD: A Real-World Benchmark for Object Detection under Extreme Weather Conditions
Chih-Hsin Chen, Yu-Tung Liu, Amar Fadillah et al. (+2 more)
Autonomous driving and intelligent transportation systems remain vulnerable under extreme weather. The U.S. Federal Highway Administration reports that roughly 745,000 crashes and 3,800 fatalities per year are weather-related, and recent regulatory investigations have examined failures of Level-2/3 driving systems under reduced-visibility conditions. However,
datasets commonly used to evaluate weather robustness remain limited in scale, diversity, and realism. In this paper, we introduce XWOD (Extreme Weather Object Detection), a large-scale real-world traffic-object detection
benchmark containing 10,010 images and 42,924 bounding boxes across seven extreme weather conditions: rain, snow, fog, haze/sand/
dust, flooding, tornado, and wildfire. The
dataset covers six traffic-object categories, including car, person, truck, motorcycle, bicycle, and bus. XWOD extends the weather taxonomy from one to seven conditions, and is the first to cover the emerging class of
climate-amplified hazards, such as flooding, tornado, and wildfire. To evaluate the quality of our data, we train standard YOLO-family detectors on XWOD and test them zero-shot on external weather
benchmarks, achieving mAP$_{50}$ scores of 63.00% on RTTS, 59.94% on DAWN, and 61.12% on WEDGE, compared with the corresponding published YOLO-based baselines of 40.37%, 32.75%, and 45.41%, respectively, representing relative improvements of 56%, 83%, and 35%. These cross-
dataset results show that XWOD provides a strong source domain for learning weather-robust traffic perception. We release the
dataset, splits, baseline weights, and reproducible evaluation code under a research-use license.
Read the full abstract ▾
All authors Chih-Hsin Chen, Yu-Tung Liu, Amar Fadillah, Kuan-Ting Lai, Dong Liu
Computer Vision
PointGS: Semantic-Consistent Unsupervised 3D Point Cloud Segmentation with 3D Gaussian Splatting
Yixiao Song, Qingyong Li, Wen Wang et al. (+1 more)
Unsupervised point cloud
segmentation is critical for
embodied artificial intelligence and autonomous driving, as it mitigates the prohibitive cost of dense point-level annotations required by fully supervised methods. While integrating 2D pre-trained models such as the Segment Anything Model (SAM) to supplement semantic information is a natural choice, this approach faces a fundamental mismatch between discrete 3D points and continuous 2D images. This mismatch leads to inevitable projection overlap and complex modality
alignment, resulting in compromised semantic consistency across 2D-3D transfer. To address these limitations, this paper proposes PointGS, a simple yet effective pipeline for unsupervised 3D point cloud
segmentation. PointGS leverages 3D Gaussian Splatting as a unified intermediate representation to bridge the discrete-continuous domain gap. Input sparse point clouds are first reconstructed into dense 3D Gaussian spaces via multi-view observations, filling spatial gaps and encoding occlusion relationships to eliminate projection-induced semantic conflation. Multi-view dense images are rendered from the Gaussian space, with 2D semantic masks extracted via SAM, and semantics are distilled to 3D Gaussian primitives through contrastive learning to ensure consistent semantic assignments across different views. The Gaussian space is aligned with the original point cloud via two-step registration, and point semantics are assigned through nearest-neighbor search on labeled Gaussians. Experiments demonstrate that PointGS outperforms state-of-the-art unsupervised methods, achieving +0.9% mIoU on ScanNet-V2 and +2.8% mIoU on S3DIS.
Read the full abstract ▾
All authors Yixiao Song, Qingyong Li, Wen Wang, Zhicheng Yan
Computer Vision
LiBrA-Net: Lie-Algebraic Bilateral Affine Fields for Real-Time 4K Video Dehazing
Yongcong Wang, Chengchao Shen, Guangwei Gao et al. (+5 more)
Currently, there is a gap in the field of ultra-high-definition (UHD)
video dehazing due to the lack of a
benchmark for evaluation. Furthermore, existing
video dehazing methods cannot run on consumer-grade GPUs when processing continuous UHD sequences of 3–5 frames at a time. In this paper, we address both issues with a new
benchmark and an efficient method. Our key observation is that atmospheric dehazing reduces to a per-pixel affine transform governed by the low-frequency depth field, which can be compactly encoded in bilateral grids whose prediction cost is decoupled from the output resolution. Building on this, we propose LiBrA-Net, which factorizes the spatiotemporal affine field into a spatial–color and a temporal bilateral sub-grid predicted at a fixed low resolution, fuses their coefficients in the $\mathfrak{gl}(3)$ Lie
algebra under group-theoretic regularization, maps the result to invertible $GL(3)$ transforms via a Cayley parameterization, and restores high-frequency detail through a lightweight input-guided branch. We further release UHV-4K, the first paired 4K
video dehazing
benchmark with depth, transmission, and optical-flow annotations on every frame. Across UHV-4K, REVIDE, and HazeWorld, LiBrA-Net sets a new state of the art among compared
video dehazing methods while running native 4K at 25FPS on a single GPU with only 6.12M parameters. Code is available at https://github.com/ycwang31/LiBrA-Net.
Read the full abstract ▾
All authors Yongcong Wang, Chengchao Shen, Guangwei Gao, Wei Wang, Pengwen Dai, Dianjie Lu, Guijuan Zhang, Zhuoran Zheng
Computer Vision
Principled Design of Diffusion-based Optimizers for Inverse Problems
Julio Oscanoa, Irmak Sivgin, Cagan Alkan et al. (+4 more)
Score-based
diffusion models achieve state-of-the-art performance for inverse problems, but their practical deployment is hindered by long inference times and cumbersome hyperparameter tuning. While pretrained
diffusion models can be reused across tasks without retraining, inference-time hyperparameters such as the noise schedule and posterior sampling weights typically require ad-hoc adjustment for each problem setup. We propose principled reparameterizations that induce invariances, allowing the same hyperparameters to be reused across multiple problems without re-tuning. In addition, building on the RED-diff framework, which reformulates posterior sampling as an optimization problem, we further develop the OptDiff pipeline. OptDiff provides a simplified tuning framework that facilitates the integration of convex optimization tools to accelerate inference. Experiments on image reconstruction, deblurring, and super-resolution show substantial speedups and improved image quality.
Read the full abstract ▾
All authors Julio Oscanoa, Irmak Sivgin, Cagan Alkan, Daniel Ennis, John Pauly, Mert Pilanci, Shreyas Vasanawala