Computer Vision
DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding
Dong Zhuo, Wenzhao Zheng, Sicheng Zuo et al. (+4 more)
With the growing adoption of
vision-
language-action models and world models in autonomous driving systems, scalable image tokenization becomes crucial as the interface for the visual modality. However, most existing tokenizers are designed for
monocular and 2D scenes, leading to inefficiency and inter-view inconsistency when applied to high-resolution multi-view driving scenes. To address this, we propose DriveTok, an efficient 3D driving scene tokenizer for unified multi-view reconstruction and understanding. DriveTok first obtains semantically rich visual features from
vision foundation models and then transforms them into the scene
tokens with 3D deformable cross-
attention. For decoding, we employ a multi-view
transformer to reconstruct multi-view features from the scene
tokens and use multiple heads to obtain RGB, depth, and semantic reconstructions. We also add a 3D head directly on the scene
tokens for 3D semantic occupancy prediction for better spatial awareness. With the multiple training objectives, DriveTok learns unified scene
tokens that integrate semantic,
geometric, and textural information for efficient multi-view tokenization. Extensive experiments on the widely used nuScenes
dataset demonstrate that the scene
tokens from DriveTok perform well on image reconstruction, semantic
segmentation, depth prediction, and 3D occupancy prediction tasks.
Read the full abstract ▾
All authors Dong Zhuo, Wenzhao Zheng, Sicheng Zuo, Siming Yan, Lu Hou, Jie Zhou, Jiwen Lu
Computer Vision
Rethinking Vector Field Learning for Generative Segmentation
Chaoyang Wang, Yaobo Liang, Boci Peng et al. (+3 more)
Taming
diffusion models for generative
segmentation has attracted increasing
attention. While existing approaches primarily focus on architectural tweaks or training heuristics, there remains a limited understanding of the intrinsic mismatch between continuous flow matching objectives and discrete perception tasks. In this work, we revisit
diffusion segmentation from the perspective of vector field learning. We identify two key limitations of the commonly used flow matching objective: gradient vanishing and trajectory traversing, which result in slow convergence and poor class separation. To tackle these issues, we propose a principled vector field reshaping strategy that augments the learned velocity field with a detached distance-aware correction term. This correction introduces both attractive and repulsive interactions, enhancing gradient magnitudes near centroids while preserving the original
diffusion training framework. Furthermore, we design a computationally efficient, quasi-random category encoding scheme inspired by Kronecker sequences, which integrates seamlessly with an end-to-end pixel neural field framework for pixel-level semantic
alignment. Extensive experiments consistently demonstrate significant improvements over vanilla flow matching approaches, substantially narrowing the performance gap between generative
segmentation and strong discriminative specialists.
Read the full abstract ▾
All authors Chaoyang Wang, Yaobo Liang, Boci Peng, Fan Duan, Jingdong Wang, Yunhai Tong
Computer Vision
LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs
Keda Tao, Yuhua Zheng, Jia Xu et al. (+13 more)
Recent advancements in omnimodal large
language models (OmniLLMs) have significantly improved the comprehension of audio and
video inputs. However, current evaluations primarily focus on short audio and
video clips ranging from 10 seconds to 5 minutes, failing to reflect the demands of real-world applications, where
videos typically run for tens of minutes. To address this critical gap, we introduce LVOmniBench, a new
benchmark designed specifically for the cross-modal comprehension of long-form audio and
video. This
dataset comprises high-quality
videos sourced from open platforms that feature rich audio-visual dynamics. Through rigorous manual selection and annotation, LVOmniBench comprises 275
videos, ranging in duration from 10 to 90 minutes, and 1,014 question-answer (QA) pairs. LVOmniBench aims to rigorously evaluate the capabilities of OmniLLMs across domains, including long-term
memory, temporal localization, fine-grained understanding, and
multimodal perception. Our extensive evaluation reveals that current OmniLLMs encounter significant challenges when processing extended audio-visual inputs. Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%. We anticipate that this
dataset, along with our empirical findings, will stimulate further research and the development of advanced models capable of resolving complex cross-modal understanding problems within long-form audio-visual contexts.
Read the full abstract ▾
All authors Keda Tao, Yuhua Zheng, Jia Xu, Wenjie Du, Kele Shao, Hesong Wang, Xueyi Chen, Xin Jin, Junhan Zhu, Bohan Yu, Weiqiang Wang, Jian Liu, Can Qin, Yulun Zhang, Ming-Hsuan Yang, Huan Wang
Computer Vision
DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoising
Tianjiao Yu, Xinzhuo Li, Muntasir Wahed et al. (+4 more)
Understanding and generating 3D objects as compositions of meaningful parts is fundamental to human perception and
reasoning. However, most text-to-3D methods overlook the semantic and functional structure of parts. While recent part-aware approaches introduce decomposition, they remain largely geometry-focused, lacking semantic grounding and failing to model how parts align with textual descriptions or their inter-part relations. We propose DreamPartGen, a framework for semantically grounded, part-aware text-to-3D generation. DreamPartGen introduces Duplex Part Latents (DPLs) that jointly model each part's geometry and appearance, and Relational Semantic Latents (RSLs) that capture inter-part dependencies derived from
language. A synchronized co-denoising process enforces mutual
geometric and semantic consistency, enabling coherent, interpretable, and text-aligned 3D synthesis. Across multiple
benchmarks, DreamPartGen delivers state-of-the-art performance in
geometric fidelity and text-shape
alignment.
Read the full abstract ▾
All authors Tianjiao Yu, Xinzhuo Li, Muntasir Wahed, Jerry Xiong, Yifan Shen, Ying Shen, Ismini Lourentzou
Computer Vision
Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders
Shang-Jui Ray Kuo and Paola Cascante-Bonilla
Large
vision–
language models (VLMs) often use a frozen
vision backbone, whose image features are mapped into a large
language model through a lightweight connector. While
transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM)
vision backbones can be a strong alternative. We systematically evaluate SSM
vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization. We further adapt both SSM and ViT-family backbones with detection or
segmentation training and find that dense-task tuning generally improves performance across families; after this adaptation, the SSM backbone remains competitive while operating at a substantially smaller model scale. We further observe that (i) higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and (ii) some visual backbones are unstable in localization. Based on these findings, we propose stabilization strategies that improve robustness for both backbone families and highlight SSM backbones as a strong alternative to
transformer-based
vision encoders in VLMs.
Read the full abstract ▾
Computer Vision
RPiAE: A Representation-Pivoted Autoencoder Enhancing Both Image Generation and Editing
Yue Gong, Hongyu Li, Shanyuan Liu et al. (+8 more)
Diffusion models have become the dominant paradigm for image generation and editing, with latent
diffusion models shifting denoising to a compact latent space for efficiency and scalability. Recent attempts to leverage pretrained visual representation models as tokenizer priors either align
diffusion features to representation features or directly reuse representation encoders as frozen tokenizers. Although such approaches can improve generation metrics, they often suffer from limited reconstruction fidelity due to frozen encoders, which in turn degrades editing quality, as well as overly high-dimensional latents that make
diffusion modeling difficult. To address these limitations, We propose Representation-Pivoted AutoEncoder, a representation-based tokenizer that improves both generation and editing. We introduce Representation-Pivot Regularization, a training strategy that enables a representation-initialized encoder to be fine-tuned for reconstruction while preserving the semantic structure of the pretrained representation space, followed by a variational bridge which compress latent space into a compact one for better
diffusion modeling. We adopt an objective-decoupled stage-wise training strategy that sequentially optimizes generative tractability and reconstruction-fidelity objectives. Together, these components yield a tokenizer that preserves strong semantics, reconstructs faithfully, and produces latents with reduced
diffusion modeling complexity. Experiments demonstrate that RPiAE outperforms other visual tokenizers on text-to-image generation and image editing, while delivering the best reconstruction fidelity among representation-based tokenizers.
Read the full abstract ▾
All authors Yue Gong, Hongyu Li, Shanyuan Liu, Bo Cheng, Yuhang Ma, Liebucha Wu, Xiaoyu Wu, Manyuan Zhang, Dawei Leng, Yuhui Yin, Lijun Zhang
Computer Vision
Tinted Frames: Question Framing Blinds Vision-Language Models
Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher et al. (+2 more)
Vision-
Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual
reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of
attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual
reasoning. Using visual
attention as a probe, we quantify how framing alters both the amount and distribution of
attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower
attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift
attention towards uninformative
tokens. We further demonstrate that this
attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight
prompt-tuning method using learnable
tokens that encourages the robust, visually grounded
attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.
Read the full abstract ▾
All authors Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher, Leonid Sigal, Ritwik Gupta
Computer Vision · Cross-listed from cs.RO
FASTER: Rethinking Real-Time Flow VLAs
Yuxiang Lu, Zhe Liu, Xianzhe Fan et al. (+5 more)
Real-time execution is crucial for deploying
Vision-
Language-Action (VLA) models in the physical world. Existing asynchronous inference methods primarily optimize trajectory smoothness, but neglect the critical latency in reacting to environmental changes. By rethinking the notion of reaction in action chunking policies, this paper presents a systematic analysis of the factors governing reaction time. We show that reaction time follows a uniform distribution determined jointly by the Time to First Action (TTFA) and the execution horizon. Moreover, we reveal that the standard practice of applying a constant schedule in flow-based VLAs can be inefficient and forces the system to complete all sampling steps before any movement can start, forming the bottleneck in reaction latency. To overcome this issue, we propose Fast Action Sampling for ImmediaTE Reaction (FASTER). By introducing a Horizon-Aware Schedule, FASTER adaptively prioritizes near-term actions during flow sampling, compressing the denoising of the immediate reaction by tenfold (e.g., in $\pi_{0.5}$ and X-VLA) into a single step, while preserving the quality of long-horizon trajectory. Coupled with a streaming client-server pipeline, FASTER substantially reduces the effective reaction latency on real
robots, especially when deployed on consumer-grade GPUs. Real-world experiments, including a highly dynamic table tennis task, prove that FASTER unlocks substantially improved real-time responsiveness for generalist policies, enabling rapid generation of accurate and smooth trajectories.
Read the full abstract ▾
All authors Yuxiang Lu, Zhe Liu, Xianzhe Fan, Zhenya Yang, Jinghua Hou, Junyi Li, Kaixin Ding, Hengshuang Zhao
Computer Vision
Reconstruction Matters: Learning Geometry-Aligned BEV Representation through 3D Gaussian Splatting
Yiren Lu, Xin Ye, Burhaneddin Yaman et al. (+4 more)
Bird's-Eye-View (BEV) perception serves as a cornerstone for autonomous driving, offering a unified spatial representation that fuses surrounding-view images to enable
reasoning for various downstream tasks, such as semantic
segmentation, 3D object detection, and motion prediction. However, most existing BEV perception frameworks adopt an end-to-end training paradigm, where image features are directly transformed into the BEV space and optimized solely through downstream task supervision. This formulation treats the entire perception process as a black box, often lacking explicit 3D
geometric understanding and interpretability, leading to suboptimal performance. In this paper, we claim that an explicit 3D representation matters for accurate BEV perception, and we propose Splat2BEV, a Gaussian Splatting-assisted framework for BEV tasks. Splat2BEV aims to learn BEV feature representations that are both semantically rich and geometrically precise. We first pre-train a Gaussian generator that explicitly reconstructs 3D scenes from multi-view inputs, enabling the generation of geometry-aligned feature representations. These representations are then projected into the BEV space to serve as inputs for downstream tasks. Extensive experiments on nuScenes and argoverse
dataset demonstrate that Splat2BEV achieves state-of-the-art performance and validate the effectiveness of incorporating explicit 3D reconstruction into BEV perception.
Read the full abstract ▾
All authors Yiren Lu, Xin Ye, Burhaneddin Yaman, Jingru Luo, Zhexiao Xiong, Liu Ren, Yu Yin
Computer Vision · Cross-listed from cs.SD
Few-shot Acoustic Synthesis with Multimodal Flow Matching
Amandine Brunetto
Generating audio that is acoustically consistent with a scene is essential for immersive virtual environments. Recent neural acoustic field methods enable spatially continuous sound rendering but remain scene-specific, requiring dense audio measurements and costly training for each environment. Few-shot approaches improve scalability across rooms but still rely on multiple recordings and, being deterministic, fail to capture the inherent uncertainty of scene acoustics under sparse context. We introduce flow-matching acoustic generation (FLAC), a probabilistic method for few-shot acoustic synthesis that models the distribution of plausible room impulse responses (RIRs) given minimal scene context. FLAC leverages a
diffusion transformer trained with a flow-matching objective to generate RIRs at arbitrary positions in novel scenes, conditioned on spatial,
geometric, and acoustic cues. FLAC outperforms state-of-the-art eight-shot baselines with one-shot on both the AcousticRooms and Hearing Anything Anywhere
datasets. To complement standard perceptual metrics, we further introduce AGREE, a joint acoustic-geometry
embedding, enabling geometry-consistent evaluation of generated RIRs through
retrieval and distributional metrics. This work is the first to apply generative flow matching to explicit RIR synthesis, establishing a new direction for robust and data-efficient acoustic synthesis.
Read the full abstract ▾
Computer Vision
ARIADNE: A Perception-Reasoning Synergy Framework for Trustworthy Coronary Angiography Analysis
Zhan Jin, Yu Luo, Yizhou Zhang et al. (+5 more)
Conventional pixel-wise loss functions fail to enforce topological constraints in coronary vessel
segmentation, producing fragmented vascular trees despite high pixel-level accuracy. We present ARIADNE, a two-stage framework coupling preference-aligned perception with RL-based diagnostic
reasoning for topologically coherent stenosis detection. The perception module employs DPO to fine-tune the Sa2VA
vision-
language foundation model using Betti number constraints as preference signals, aligning the policy toward geometrically complete vessel structures rather than pixel-wise overlap metrics. The
reasoning module formulates stenosis localization as a Markov Decision Process with an explicit rejection mechanism that autonomously defers ambiguous anatomical candidates such as bifurcations and vessel crossings, shifting from coverage maximization to reliability optimization. On 1,400 clinical angiograms, ARIADNE achieves state-of-the-art centerline Dice of 0.838, reduces false positives by 41% compared to
geometric baselines. External validation on multi-center
benchmarks ARCADE and XCAD confirms generalization across acquisition protocols. This represents the first application of DPO for topological
alignment in medical imaging, demonstrating that preference-based learning over structural constraints mitigates topological violations while maintaining diagnostic sensitivity in interventional cardiology workflows.
Read the full abstract ▾
All authors Zhan Jin, Yu Luo, Yizhou Zhang, Ziyang Cui, Yuqing Wei, Xianchao Liu, Xueying Zeng, Qing Zhang
Computer Vision · Cross-listed from cs.RO
Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigation
Swagat Padhan, Lakshya Jain, Bhavya Minesh Shah et al. (+3 more)
Robots collaborating with humans must convert natural
language goals into actionable, physically grounded decisions. For example, executing a command such as "go two meters to the right of the fridge" requires grounding semantic references, spatial relations, and metric constraints within a 3D scene. While recent
vision language models (VLMs) demonstrate strong semantic grounding capabilities, they are not explicitly designed to reason about metric constraints in physically defined spaces. In this work, we empirically demonstrate that state-of-the-art VLM-based grounding approaches struggle with complex metric-semantic
language queries. To address this limitation, we propose MAPG (Multi-
Agent Probabilistic Grounding), an agentic framework that decomposes
language queries into structured subcomponents and queries a VLM to ground each component. MAPG then probabilistically composes these grounded outputs to produce metrically consistent, actionable decisions in 3D space. We evaluate MAPG on the HM-EQA
benchmark and show consistent performance improvements over strong baselines. Furthermore, we introduce a new
benchmark, MAPG-Bench, specifically designed to evaluate metric-semantic goal grounding, addressing a gap in existing
language grounding evaluations. We also present a real-world
robot demonstration showing that MAPG transfers beyond simulation when a structured scene representation is available.
Read the full abstract ▾
All authors Swagat Padhan, Lakshya Jain, Bhavya Minesh Shah, Omkar Patil, Thao Nguyen, Nakul Gopalan
Computer Vision
Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation
Kwanyoung Lee, SeungJu Cha, Yebin Ahn et al. (+3 more)
Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image
datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary
Prompt Blending (AAPB) - a unified framework that stabilizes the
diffusion process in low-density regions. AAPB leverages auxiliary anchor
prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target
prompt. Unlike prior heuristic
prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target
prompt at each
diffusion step. Grounded in Tweedie's identity, our formulation provides a principled and training-free framework for adaptive
prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit
datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines.
Read the full abstract ▾
All authors Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Hyunwoo Oh, Sungho Koh, Dong-Jin Kim
Computer Vision
ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generation
Kwanyoung Lee, Hyunwoo Oh, SeungJu Cha et al. (+2 more)
Generating rare compositional concepts in text-to-image synthesis remains a challenge for
diffusion models, particularly for attributes that are uncommon in the training data. While recent approaches, such as R2F, address this challenge by utilizing LLM for
prompt scheduling, they suffer from inherent variance due to the randomness of
language models and suboptimal guidance from iterative text
embedding switching. To address these problems, we propose the ADAPT framework, a training-free framework that deterministically plans and semantically aligns
prompt schedules, providing consistent guidance to enhance the composition of rare concepts. By leveraging
attention scores and orthogonal components, ADAPT significantly enhances compositional generation of rare concepts in the RareBench
benchmark without additional training or fine-tuning. Through comprehensive experiments, we demonstrate that ADAPT achieves superior performance in RareBench and accurately reflects the semantic information of rare attributes, providing deterministic and precise control over the generation of rare compositions without compromising visual integrity.
Read the full abstract ▾
All authors Kwanyoung Lee, Hyunwoo Oh, SeungJu Cha, Sungho Koh, Dong-Jin Kim
Computer Vision
GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning
Yiren Lu, Yi Du, Disheng Liu et al. (+3 more)
Effective
embodied exploration requires
agents to accumulate and retain spatial knowledge over time. However, existing scene representations, such as discrete scene graphs or static view-based snapshots, lack post-hoc re-observability. If an initial observation misses a target, the resulting
memory omission is often irrecoverable. To bridge this gap, we propose GSMem, a zero-shot
embodied exploration and
reasoning framework built upon 3D Gaussian Splatting (3DGS). By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial
memory that endows the
agent with Spatial Recollection: the ability to render photorealistic novel views from optimal, previously unoccupied viewpoints. To operationalize this, GSMem employs a
retrieval mechanism that simultaneously leverages parallel object-level scene graphs and semantic-level
language fields. This complementary design robustly localizes target regions, enabling the
agent to “hallucinate” optimal views for high-fidelity
Vision-
Language Model (VLM)
reasoning. Furthermore, we introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with
geometric coverage. Extensive experiments on
embodied question answering and lifelong
navigation demonstrate the robustness and effectiveness of our framework
Read the full abstract ▾
All authors Yiren Lu, Yi Du, Disheng Liu, Yunlai Zhou, Chen Wang, Yu Yin
Computer Vision
Revisiting Autoregressive Models for Generative Image Classification
Ilia Sudakov, Artem Babenko and Dmitry Baranchuk
Class-conditional generative models have emerged as accurate and robust classifiers, with
diffusion models demonstrating clear advantages over other visual generative paradigms, including autoregressive (AR) models. In this work, we revisit visual AR-based generative classifiers and identify an important limitation of prior approaches: their reliance on a fixed
token order, which imposes a restrictive inductive bias for image understanding. We observe that single-order predictions rely more on partial discriminative cues, while averaging over multiple
token orders provides a more comprehensive signal. Based on this insight, we leverage recent any-order AR models to estimate order-marginalized predictions, unlocking the high classification potential of AR models. Our approach consistently outperforms
diffusion-based classifiers across diverse image classification
benchmarks, while being up to 25x more efficient. Compared to state-of-the-art self-supervised discriminative models, our method delivers competitive classification performance - a notable achievement for generative classifiers.
Read the full abstract ▾
Computer Vision
CustomTex: High-fidelity Indoor Scene Texturing via Multi-Reference Customization
Weilin Chen, Jiahao Rao, Wenhao Wang et al. (+3 more)
The creation of high-fidelity, customizable 3D indoor scene textures remains a significant challenge. While text-driven methods offer flexibility, they lack the precision for fine-grained, instance-level control, and often produce textures with insufficient quality, artifacts, and baked-in shading. To overcome these limitations, we introduce CustomTex, a novel framework for instance-level, high-fidelity scene texturing driven by reference images. CustomTex takes an untextured 3D scene and a set of reference images specifying the desired appearance for each object instance, and generates a unified, high-resolution texture map. The core of our method is a dual-
distillation approach that separates semantic control from pixel-level enhancement. We employ semantic-level
distillation, equipped with an instance cross-
attention, to ensure semantic plausibility and “reference-instance”
alignment, and pixel-level
distillation to enforce high visual fidelity. Both are unified within a Variational Score
Distillation (VSD) optimization framework. Experiments demonstrate that CustomTex achieves precise instance-level consistency with reference images and produces textures with superior sharpness, reduced artifacts, and minimal baked-in shading compared to state-of-the-art methods. Our work establishes a more direct and user-friendly path to high-quality, customizable 3D scene appearance editing.
Read the full abstract ▾
All authors Weilin Chen, Jiahao Rao, Wenhao Wang, Xinyang Li, Xuan Cheng, Liujuan Cao
Computer Vision
TAU-R1: Visual Language Model for Traffic Anomaly Understanding
Yuqiang Lin, Kehua Chen, Sam Lockyer et al. (+12 more)
Traffic Anomaly Understanding (TAU) is important for traffic safety in Intelligent Transportation Systems. Recent
vision-
language models (VLMs) have shown strong capabilities in
video understanding. However, progress on TAU remains limited due to the lack of
benchmarks and task-specific methodologies. To address this limitation, we introduce Roundabout-TAU, a
dataset constructed from real-world roundabout
videos collected in collaboration with the City of Carmel, Indiana. The
dataset contains 342 clips and is annotated with more than 2,000 question-answer pairs covering multiple aspects of traffic anomaly understanding. Building on this
benchmark, we propose TAU-R1, a two-layer
vision-
language framework for TAU. The first layer is a lightweight anomaly classifier that performs coarse anomaly categorisation, while the second layer is a larger anomaly reasoner that generates detailed event summaries. To improve task-specific
reasoning, we introduce a two-stage training strategy consisting of decomposed-QA-enhanced supervised fine-tuning followed by TAU-GRPO, a GRPO-based post-training method with TAU-specific reward functions. Experimental results show that TAU-R1 achieves strong performance on both anomaly classification and
reasoning tasks while maintaining deployment efficiency. The
dataset and code are available at: https://github.com/siri-rouser/TAU-R1
Read the full abstract ▾
All authors Yuqiang Lin, Kehua Chen, Sam Lockyer, Arjun Yadav, Mingxuan Sui, Shucheng Zhang, Yan Shi, Bingzhang Wang, Yuang Zhang, Markus Zarbock, Florain Stanek, Adrian Evans, Wenbin Li, Yinhai Wang, Nic Zhang
Computer Vision
SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues
Carlos Hinojosa, Clemens Grange and Bernard Ghanem
Vision-
language models (VLMs) are increasingly deployed in real-world and
embodied settings where safety decisions depend on visual context. However, it remains unclear which visual evidence drives these judgments. We study whether
multimodal safety behavior in VLMs can be steered by simple semantic cues. We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content. To evaluate these effects, we propose SAVeS, a
benchmark for situational safety under semantic cues, together with an evaluation protocol that separates behavioral refusal, grounded safety
reasoning, and false refusals. Experiments across multiple VLMs and an additional state-of-the-art
benchmark show that safety decisions are highly sensitive to semantic cues, indicating reliance on learned visual-linguistic associations rather than grounded visual understanding. We further demonstrate that automated steering pipelines can exploit these mechanisms, highlighting a potential vulnerability in
multimodal safety systems.
Read the full abstract ▾
Computer Vision
Multi-Modal Building Change Detection for Large-Scale Small Changes: Benchmark and Baseline
Ye Wang, Wei Lu, Zhihui You et al. (+6 more)
Change detection in optical remote sensing imagery is susceptible to illumination fluctuations, seasonal changes, and variations in surface land-cover materials. Relying solely on RGB imagery often produces pseudo-changes and leads to semantic ambiguity in features. Incorporating near-infrared (NIR) information provides heterogeneous physical cues that are complementary to visible light, thereby enhancing the discriminability of building materials and tiny structures while improving detection accuracy. However, existing multi-modal
datasets generally lack high-resolution and accurately registered bi-temporal imagery, and current methods often fail to fully exploit the inherent heterogeneity between these modalities. To address these issues, we introduce the Large-scale Small-change Multi-modal
Dataset (LSMD), a bi-temporal RGB-NIR building change detection
benchmark dataset targeting small changes in realistic scenarios, providing a rigorous testing platform for evaluating multi-modal change detection methods in complex environments. Based on LSMD, we further propose the Multi-modal Spectral Complementarity Network (MSCNet) to achieve effective cross-modal feature fusion. MSCNet comprises three key components: the Neighborhood Context Enhancement Module (NCEM) to strengthen local spatial details, the Cross-modal
Alignment and Interaction Module (CAIM) to enable deep interaction between RGB and NIR features, and the Saliency-aware Multisource Refinement Module (SMRM) to progressively refine fused features. Extensive experiments demonstrate that MSCNet effectively leverages multi-modal information and consistently outperforms existing methods under multiple input configurations, validating its efficacy for fine-grained building change detection. The source code will be made publicly available at: https://github.com/AeroVILab-AHU/LSMD
Read the full abstract ▾
All authors Ye Wang, Wei Lu, Zhihui You, Keyan Chen, Tongfei Liu, Kaiyu Li, Hongruixuan Chen, Qingling Shu, Sibao Chen
Computer Vision
DROID-SLAM in the Wild
Moyang Li, Zihan Zhu, Marc Pollefeys et al. (+1 more)
We present a robust, real-time RGB SLAM system that handles dynamic environments by leveraging differentiable Uncertainty-aware Bundle Adjustment. Traditional SLAM methods typically assume static scenes, leading to tracking failures in the presence of motion. Recent dynamic SLAM approaches attempt to address this challenge using predefined dynamic priors or uncertainty-aware mapping, but they remain limited when confronted with unknown dynamic objects or highly cluttered scenes where
geometric mapping becomes unreliable. In contrast, our method estimates per-pixel uncertainty by exploiting multi-view visual feature inconsistency, enabling robust tracking and reconstruction even in real-world environments. The proposed system achieves state-of-the-art camera poses and scene geometry in cluttered dynamic scenarios while running in real time at around 10 FPS. Code and
datasets are available at https://github.com/MoyangLi00/DROID-W.git.
Read the full abstract ▾
All authors Moyang Li, Zihan Zhu, Marc Pollefeys, Daniel Barath
Computer Vision
SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation
Oliver Cory, Ozge Mercanoglu Sincan and Richard Bowden
This paper introduces SignAgent, a novel agentic framework that utilises Large
Language Models (LLMs) for scalable, linguistically-grounded Sign
Language (SL) annotation and
dataset curation. Traditional computational methods for SLs often operate at the gloss level, overlooking crucial linguistic nuances, while manual linguistic annotation remains a significant bottleneck, proving too slow and expensive for the creation of large-scale, phonologically-aware
datasets. SignAgent addresses these challenges through SignAgent Orchestrator, a
reasoning LLM that coordinates a suite of linguistic tools, and SignGraph, a knowledge-grounded LLM that provides lexical and linguistic grounding. We evaluate our framework on two downstream annotation tasks. First, on Pseudo-gloss Annotation, where the
agent performs constrained assignment, using multi-modal evidence to extract and order suitable gloss labels for signed sequences. Second, on ID Glossing, where the
agent detects and refines visual clusters by
reasoning over both visual similarity and phonological overlap to correctly identify and group lexical sign variants. Our results demonstrate that our agentic approach achieves strong performance for large-scale, linguistically-aware data annotation and curation.
Read the full abstract ▾
Computer Vision
Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding
Yikai Zheng, Xin Ding, Yifan Yang et al. (+6 more)
Recent advances in Streaming
Video Understanding has enabled a new interaction paradigm where models respond proactively to user queries. Current proactive VideoLLMs rely on per-frame triggering decision making, which suffers from an efficiency-accuracy dilemma. We propose Em-Garde, a novel framework that decouples semantic understanding from streaming perception. At query time, the Instruction-Guided Proposal Parser transforms user queries into structured, perceptually grounded visual proposals; during streaming, a Lightweight Proposal Matching Module performs efficient
embedding-based matching to trigger responses. Experiments on StreamingBench and OVO-Bench demonstrate consistent improvements over prior models in proactive response accuracy and efficiency, validating an effective solution for proactive
video understanding under strict computational constraints.
Read the full abstract ▾
All authors Yikai Zheng, Xin Ding, Yifan Yang, Shiqi Jiang, Hao Wu, Qianxi Zhang, Weijun Wang, Ting Cao, Yunxin Liu
Computer Vision
SwiftTailor: Efficient 3D Garment Generation with Geometry Image Representation
Phuc Pham, Uy Dieu Tran, Binh-Son Hua et al. (+1 more)
Realistic and efficient 3D garment generation remains a longstanding challenge in computer
vision and digital fashion. Existing methods typically rely on large
vision-
language models to produce serialized representations of 2D sewing patterns, which are then transformed into simulation-ready 3D meshes using garment modeling framework such as GarmentCode. Although these approaches yield high-quality results, they often suffer from slow inference times, ranging from 30 seconds to a minute. In this work, we introduce SwiftTailor, a novel two-stage framework that unifies sewing-pattern
reasoning and geometry-based mesh synthesis through a compact geometry image representation. SwiftTailor comprises two lightweight modules: PatternMaker, an efficient
vision-
language model that predicts sewing patterns from diverse input modalities, and GarmentSewer, an efficient dense prediction
transformer that converts these patterns into a novel Garment Geometry Image, encoding the 3D surface of all garment panels in a unified UV space. The final 3D mesh is reconstructed through an efficient inverse mapping process that incorporates remeshing and dynamic stitching
algorithms to directly assemble the garment, thereby amortizing the cost of physical simulation. Extensive experiments on the
Multimodal GarmentCodeData demonstrate that SwiftTailor achieves state-of-the-art accuracy and visual fidelity while significantly reducing inference time. This work offers a scalable, interpretable, and high-performance solution for next-generation 3D garment generation.
Read the full abstract ▾
All authors Phuc Pham, Uy Dieu Tran, Binh-Son Hua, Phong Nguyen
Computer Vision
Measuring 3D Spatial Geometric Consistency in Dynamic Video Generation
Weijia Dou, Wenzhao Zheng, Weiliang Chen et al. (+3 more)
Recent generative models can produce high-fidelity
videos, yet they often exhibit 3D spatial
geometric inconsistencies. Existing evaluation methods fail to accurately characterize these inconsistencies: fidelity-centric metrics like FVD are insensitive to
geometric distortions, while consistency-focused
benchmarks often penalize valid foreground dynamics. To address this gap, we introduce SGC, a metric for evaluating 3D Spatial
Geometric Consistency in dynamically generated
videos. We quantify
geometric consistency by measuring the divergence among multiple camera poses estimated from distinct local regions. Our approach first separates static from dynamic regions, then partitions the static background into spatially coherent sub-regions. We predict depth for each pixel, estimate a local camera pose for each subregion, and compute the divergence among these poses to quantify
geometric consistency. Experiments on real and generative
videos demonstrate that SGC robustly quantifies
geometric inconsistencies, effectively identifying critical failures missed by existing metrics.
Read the full abstract ▾
All authors Weijia Dou, Wenzhao Zheng, Weiliang Chen, Yu Zheng, Jie Zhou, Jiwen Lu
Computer Vision
TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation
Yan Shu, Bin Ren, Zhitong Xiong et al. (+4 more)
Vision-
language models (VLMs) have shown promise in earth observation (EO), yet they struggle with tasks that require grounding complex spatial
reasoning in precise pixel-level visual representations. To address this problem, we introduce TerraScope, a unified VLM that delivers pixel-grounded geospatial
reasoning with two key capabilities: (1) modality-flexible
reasoning: it handles single-modality inputs (optical or SAR) and adaptively fuses different modalities into the
reasoning process when both are available; (2) multi-temporal
reasoning: it integrates temporal sequences for change analysis across multiple time points. In addition, we curate Terra-CoT, a large-scale
dataset containing 1 million samples with pixel-level masks embedded in
reasoning chains across multiple sources. We also propose TerraScope-Bench, the first
benchmark for pixel-grounded geospatial
reasoning with six sub-tasks that evaluates both answer accuracy and mask quality to ensure authentic pixel-grounded
reasoning. Experiments show that TerraScope significantly outperforms existing VLMs on pixel-grounded geospatial
reasoning while providing interpretable visual evidence.
Read the full abstract ▾
All authors Yan Shu, Bin Ren, Zhitong Xiong, Xiao Xiang Zhu, Begüm Demir, Nicu Sebe, Paolo Rota
Computer Vision
FUMO: Prior-Modulated Diffusion for Single Image Reflection Removal
Telang Xu, Chaoyang Zhang, Guangtao Zhai et al. (+1 more)
Single image reflection removal (SIRR) is challenging in real scenes, where reflection strength varies spatially and reflection patterns are tightly entangled with transmission structures. This paper presents a
diffusion model with prior modulation framework (FUMO) that introduces explicit priors for spatially adaptive conditioning and structurally faithful restoration. Two priors are extracted directly from the mixed image, an intensity prior that estimates spatial reflection severity and a high-frequency prior that captures detail-sensitive responses via multi-scale residual aggregation. We propose a coarse-to-fine training paradigm. In the first stage, these cues are combined to gate the conditional residual injections, focusing the conditioning on regions that are both reflection-dominant and structure-sensitive. In the second stage, a fine-grained refinement network corrects local misalignment and sharpens fine details in the image space. Experiments conducted on both standard
benchmarks and challenging images in the wild demonstrate competitive quantitative results and consistently improved perceptual quality. The code is released at https://github.com/Lucious-Desmon/FUMO.
Read the full abstract ▾
All authors Telang Xu, Chaoyang Zhang, Guangtao Zhai, Xiaohong Liu
Computer Vision
SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
Quentin Guimard, Federico Bartsch, Simone Caldarella et al. (+3 more)
Models that bridge
vision and
language, such as CLIP, are key components of
multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP
embedding space, where bias and task-relevant information are highly entangled. This
entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse
Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text
embeddings into disentangled features, SEM identifies and modulates bias-relevant
neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four
benchmark datasets and two CLIP backbones, SEM achieves substantial
fairness gains in
retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of
vision-
language models.
Read the full abstract ▾
All authors Quentin Guimard, Federico Bartsch, Simone Caldarella, Rahaf Aljundi, Elisa Ricci, Massimiliano Mancini
Computer Vision
Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
Anqi Zhang, Xiaokang Ji, Guangyu Gao et al. (+3 more)
Recent
segmentation methods leveraging Multi-modal Large
Language Models (MLLMs) have shown reliable object-level
segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated
segmentation-related
embeddings and visual features, or incorporate multiple additional
tokens to assist. This paper aims to investigate whether and how we can unlock
segmentation from MLLM itSELF with 1
segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the
attention mask with dual perception pathways, i.e., image-to-image and image-to-
segmentation, enabling rich feature interaction between pixels and the
segmentation token. Comprehensive experiments across multiple
segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free
segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.
Read the full abstract ▾
All authors Anqi Zhang, Xiaokang Ji, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, Yunchao Wei
Computer Vision
GenHOI: Generalized Hand-Object Pose Estimation with Occlusion Awareness
Hui Yang, Wei Sun, Jian Liu et al. (+6 more)
Generalized 3D hand-object pose estimation from a single RGB image remains challenging due to the large variations in object appearances and interaction patterns, especially under heavy occlusion. We propose GenHOI, a framework for generalized hand-object pose estimation with occlusion awareness. GenHOI integrates hierarchical semantic knowledge with hand priors to enhance model generalization under challenging occlusion conditions. Specifically, we introduce a hierarchical semantic
prompt that encodes object states, hand configurations, and interaction patterns via textual descriptions. This enables the model to learn abstract high-level representations of hand-object interactions for generalization to unseen objects and novel interactions while compensating for missing or ambiguous visual cues. To enable robust occlusion
reasoning, we adopt a multi-modal masked modeling strategy over RGB images, predicted point clouds, and textual descriptions. Moreover, we leverage hand priors as stable spatial references to extract implicit interaction constraints. This allows reliable pose inference even under significant variations in object shapes and interaction patterns. Extensive experiments on the challenging DexYCB and HO3Dv2
benchmarks demonstrate that our method achieves state-of-the-art performance in hand-object pose estimation.
Read the full abstract ▾
All authors Hui Yang, Wei Sun, Jian Liu, Jian Xiao, Tao Xie, Hossein Rahmani, Ajmal Saeed Mian, Nicu Sebe, Gim Hee Lee