Computer Vision
Understanding and Optimizing Attention-Based Sparse Matching for Diverse Local Features
Qiang Wang
We revisit the problem of training
attention-based sparse image matching models for various local features. We first identify one critical design choice that has been previously overlooked, which significantly impacts the performance of the LightGlue model. We then investigate the role of detectors and descriptors within the
transformer-based matching framework, finding that detectors, rather than descriptors, are often the primary cause for performance difference. Finally, we propose a novel approach to fine-tune existing image matching models using keypoints from a diverse set of detectors, resulting in a universal, detector-agnostic model. When deployed as a zero-shot matcher for novel detectors, the resulting model achieves or exceeds the accuracy of models specifically trained for those features. Our findings offer valuable insights for the deployment of
transformer-based matching models and the future design of local features.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CL
Prism: Spectral-Aware Block-Sparse Attention
Xinghao Wang, Pengyu Wang, Xiaoran Liu et al. (+4 more)
Block-sparse
attention is promising for accelerating long-context LLM pre-filling, yet identifying relevant blocks efficiently remains a bottleneck. Existing methods typically employ coarse-grained
attention as a proxy for block importance estimation, but often resort to expensive
token-level searching or scoring, resulting in significant selection overhead. In this work, we trace the inaccuracy of standard coarse-grained
attention via mean pooling to a theoretical root cause: the interaction between mean pooling and Rotary Positional
Embeddings (RoPE). We prove that mean pooling acts as a low-pass filter that induces destructive interference in high-frequency dimensions, effectively creating a "blind spot" for local positional information (e.g., slash patterns). To address this, we introduce Prism, a training-free spectral-aware approach that decomposes block selection into high-frequency and low-frequency branches. By applying energy-based temperature calibration, Prism restores the attenuated positional signals directly from pooled representations, enabling block importance estimation using purely block-level operations, thereby improving efficiency. Extensive evaluations confirm that Prism maintains accuracy parity with full
attention while delivering up to $\mathbf{5.1\times}$ speedup.
Read the full abstract ▾
All authors Xinghao Wang, Pengyu Wang, Xiaoran Liu, Fangxu Liu, Jason Chu, Kai Song, Xipeng Qiu
Computer Vision
RealSynCol: a high-fidelity synthetic colon dataset for 3D reconstruction applications
Chiara Lena, Davide Milesi, Alessandro Casella et al. (+10 more)
Deep learning has the potential to improve colonoscopy by enabling 3D reconstruction of the colon, providing a comprehensive view of mucosal surfaces and lesions, and facilitating the identification of unexplored areas. However, the development of robust methods is limited by the scarcity of large-scale ground truth data. We propose RealSynCol, a highly realistic synthetic
dataset designed to replicate the endoscopic environment. Colon geometries extracted from 10 CT scans were imported into a virtual environment that closely mimics intraoperative conditions and rendered with realistic vascular textures. The resulting
dataset comprises 28\,130 frames, paired with ground truth depth maps, optical flow, 3D meshes, and camera trajectories. A
benchmark study was conducted to evaluate the available synthetic colon
datasets for the tasks of depth and pose estimation. Results demonstrate that the high realism and variability of RealSynCol significantly enhance generalization performance on clinical images, proving it to be a powerful tool for developing deep learning
algorithms to support endoscopic diagnosis.
Read the full abstract ▾
All authors Chiara Lena, Davide Milesi, Alessandro Casella, Luca Carlini, Joseph C. Norton, James Martin, Bruno Scaglioni, Keith L. Obstein, Roberto De Sire, Marco Spadaccini, Cesare Hassan, Pietro Valdastri, Elena De Momi
Computer Vision
D$^2$-VR: Degradation-Robust and Distilled Video Restoration with Synergistic Optimization Strategy
Jianfeng Liang, Shaocheng Shen, Botao Xu et al. (+2 more)
The integration of
diffusion priors with temporal
alignment has emerged as a transformative paradigm for
video restoration, delivering fantastic perceptual quality, yet the practical deployment of such frameworks is severely constrained by prohibitive inference latency and temporal instability when confronted with complex real-world degradations. To address these limitations, we propose \textbf{D$^2$-VR}, a single-image
diffusion-based
video-restoration framework with low-step inference. To obtain precise temporal guidance under severe degradation, we first design a Degradation-Robust Flow
Alignment (DRFA) module that leverages confidence-aware
attention to filter unreliable motion cues. We then incorporate an
adversarial distillation paradigm to compress the
diffusion sampling trajectory into a rapid few-step regime. Finally, a synergistic optimization strategy is devised to harmonize perceptual quality with rigorous temporal consistency. Extensive experiments demonstrate that D$^2$-VR achieves state-of-the-art performance while accelerating the sampling process by \textbf{12$\times$}
Read the full abstract ▾
All authors Jianfeng Liang, Shaocheng Shen, Botao Xu, Qiang Hu, Xiaoyun Zhang
Computer Vision · Cross-listed from cs.RO
ST-BiBench: Benchmarking Multi-Stream Multimodal Coordination in Bimanual Embodied Tasks for MLLMs
Xin Wu, Zhixuan Liang, Yue Ma et al. (+3 more)
Multimodal Large
Language Models (MLLMs) have significantly advanced the landscape of
embodied AI, yet transitioning to synchronized bimanual coordination introduces formidable challenges in multi-stream
multimodal integration. We introduce ST-BiBench, a comprehensive multi-tier framework for evaluating spatio-temporal
multimodal coordination. Our approach centers on Strategic Coordination Planning, assessing high-level cross-modal
reasoning over multiple action and perception streams. To investigate the "proximity paradox"-where semantically coherent plans fail to align with spatially grounded visual inputs-we incorporate Foundational Spatial Grounding to verify workspace awareness and arm-selection logic. Furthermore, we probe model frontiers through Fine-Grained Action Control, investigating whether MLLMs can directly synthesize high-dimensional continuous action modalities (16-Dim) from complex
multimodal metadata. Evaluating 30+ state-of-the-art MLLMs, we uncover a persistent and pervasive "coordination paradox"-a significant gap between high-level strategic
reasoning and fine-grained physical execution. Results reveal that while frontier MLLMs excel at logic-driven strategy, they frequently suffer from perception-logic disconnection and multi-stream interference during
multimodal fusion. ST-BiBench provides a platform for identifying critical bottlenecks in multi-stream
multimodal fusion and cross-modal
alignment for complex
embodied tasks.
Read the full abstract ▾
All authors Xin Wu, Zhixuan Liang, Yue Ma, Mengkang Hu, Zhiyuan Qin, Xiu Li
Computer Vision
Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformers
Shuo Zhang, Wenzhuo Wu, Huayu Zhang et al. (+8 more)
Recent advances in
diffusion models have significantly improved image editing. However, challenges persist in handling
geometric transformations, such as translation, rotation, and scaling, particularly in complex scenes. Existing approaches suffer from two main limitations: (1) difficulty in achieving accurate
geometric editing of object translation, rotation, and scaling; (2) inadequate modeling of intricate lighting and shadow effects, leading to unrealistic results. To address these issues, we propose GeoEdit, a framework that leverages in-context generation through a
diffusion transformer module, which integrates
geometric transformations for precise object edits. Moreover, we introduce Effects-Sensitive
Attention, which enhances the modeling of intricate lighting and shadow effects for improved realism. To further support training, we construct RS-Objects, a large-scale
geometric editing
dataset containing over 120,000 high-quality image pairs, enabling the model to learn precise
geometric editing while generating realistic lighting and shadows. Extensive experiments on public
benchmarks demonstrate that GeoEdit consistently outperforms state-of-the-art methods in terms of visual quality,
geometric accuracy, and realism.
Read the full abstract ▾
All authors Shuo Zhang, Wenzhuo Wu, Huayu Zhang, Jiarong Cheng, Xianghao Zang, Chao Ban, Hao Sun, Zhongjiang He, Tianwei Cao, Kongming Liang, Zhanyu Ma
Computer Vision
E-VAds: An E-commerce Short Videos Understanding Benchmark for MLLMs
Xianjie Liu, Yiman Hu, Liang Wu et al. (+4 more)
E-commerce short
videos represent a high-revenue segment of the online
video industry characterized by a goal-driven format and dense multi-modal signals. Current models often struggle with these
videos because existing
benchmarks focus primarily on general-purpose tasks and neglect the
reasoning of commercial intent. In this work, we first propose a multi-modal information density assessment framework to quantify the complexity of this domain. Our evaluation reveals that e-commerce content exhibits substantially higher density across visual, audio, and textual modalities compared to mainstream
datasets, establishing a more challenging frontier for
video understanding. To address this gap, we introduce E-commerce
Video Ads
Benchmark, which is the first
benchmark specifically designed for e-commerce short
video understanding. We curated 3,961 high-quality
videos from Taobao covering a wide range of product categories and used a multi-
agent system to generate 19,785 open-ended Q&A pairs, which consist of five distinct tasks. Finally, we develop E-VAds-R1, an RL-based
reasoning model featuring a multi-grained reward design called MG-GRPO. This strategy provides smooth guidance for early exploration while creating a non-linear incentive for expert-level precision. Experimental results demonstrate that E-VAds-R1 achieves a 109.2% performance gain in commercial intent
reasoning with only a few hundred training samples. Data is available at https://github.com/TaobaoTmall-AlgorithmProducts/E-VAds_
Benchmark.
Read the full abstract ▾
All authors Xianjie Liu, Yiman Hu, Liang Wu, Ping Hu, Yixiong Zou, Jian Xu, Bo Zheng
Computer Vision
What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning
Yujin Zhou, Pengcheng Wen, Jiale Chen et al. (+6 more)
The rapid advancement of Large
Vision Language Models (LVLMs) has demonstrated excellent abilities in various visual tasks. Building upon these developments, the thinking with images paradigm has emerged, enabling models to dynamically edit and re-encode visual information at each
reasoning step, mirroring human visual processing. However, this paradigm introduces significant challenges as diverse errors may occur during
reasoning processes. This necessitates Process Reward Models (PRMs) for distinguishing positive and negative
reasoning steps, yet existing
benchmarks for PRMs are predominantly text-centric and lack comprehensive assessment under this paradigm. To address these gaps, this work introduces the first comprehensive
benchmark specifically designed for evaluating PRMs under the thinking with images paradigm. Our main contributions are: (1) Through extensive analysis of
reasoning trajectories and guided search experiments with PRMs, we define 7 fine-grained error types and demonstrate both the necessity for specialized PRMs and the potential for improvement. (2) We construct a comprehensive
benchmark comprising 1,206 manually annotated thinking with images
reasoning trajectories spanning 4 categories and 16 subcategories for fine-grained evaluation of PRMs. (3) Our experimental analysis reveals that current LVLMs fall short as effective PRMs, exhibiting limited capabilities in visual
reasoning process evaluation with significant performance disparities across error types, positive evaluation bias, and sensitivity to
reasoning step positions. These findings demonstrate the effectiveness of our
benchmark and establish crucial foundations for advancing PRMs in LVLMs.
Read the full abstract ▾
All authors Yujin Zhou, Pengcheng Wen, Jiale Chen, Boqin Yin, Han Zhu, Jiaming Ji, Juntao Dai, Chi-Min Chan, Sirui Han
Computer Vision
UrbanGraphEmbeddings: Learning and Evaluating Spatially Grounded Multimodal Embeddings for Urban Science
Jie Zhang, Xingtong Yu, Yuan Fang et al. (+2 more)
Learning transferable
multimodal embeddings for urban environments is challenging because urban understanding is inherently spatial, yet existing
datasets and
benchmarks lack explicit
alignment between street-view images and urban structure. We introduce UGData, a spatially grounded
dataset that anchors street-view images to structured spatial graphs and provides graph-aligned supervision via spatial
reasoning paths and spatial context captions, exposing distance, directionality, connectivity, and neighborhood context beyond image content. Building on UGData, we propose UGE, a two-stage training strategy that progressively and stably aligns images, text, and spatial structures by combining instruction-guided contrastive learning with graph-based spatial encoding. We finally introduce UGBench, a comprehensive
benchmark to evaluate how spatially grounded
embeddings support diverse urban understanding tasks – including geolocation ranking, image
retrieval, urban perception, and spatial grounding. We develop UGE on multiple state-of-the-art VLM backbones, including Qwen2-VL, Qwen2.5-VL, Phi-3-
Vision, and LLaVA1.6-Mistral, and train fixed-dimensional spatial
embeddings with LoRA tuning. UGE built upon Qwen2.5-VL-7B backbone achieves up to 44% improvement in image
retrieval and 30% in geolocation ranking on training cities, and over 30% and 22% gains respectively on held-out cities, demonstrating the effectiveness of explicit spatial grounding for spatially intensive urban tasks.
Read the full abstract ▾
All authors Jie Zhang, Xingtong Yu, Yuan Fang, Rudi Stouffs, Zdravko Trivic
Computer Vision · Cross-listed from cs.AI
CoTZero: Annotation-Free Human-Like Vision Reasoning via Hierarchical Synthetic CoT
Chengyi Du, Yazhe Niu, Dazhong Shen et al. (+1 more)
Recent advances in
vision-
language models (VLMs) have markedly improved image-text
alignment, yet they still fall short of human-like visual
reasoning. A key limitation is that many VLMs rely on surface correlations rather than building logically coherent structured representations, which often leads to missed higher-level semantic structure and non-causal relational understanding, hindering compositional and verifiable
reasoning. To address these limitations by introducing human models into the
reasoning process, we propose CoTZero, an annotation-free paradigm with two components: (i) a dual-stage data synthesis approach and (ii) a cognition-aligned training method. In the first component, we draw inspiration from neurocognitive accounts of compositional productivity and global-to-local analysis. In the bottom-up stage, CoTZero extracts atomic visual primitives and incrementally composes them into diverse, structured question-
reasoning forms. In the top-down stage, it enforces hierarchical
reasoning by using coarse global structure to guide the interpretation of local details and causal relations. In the cognition-aligned training component, built on the synthesized CoT data, we introduce Cognitively Coherent Verifiable Rewards (CCVR) in Reinforcement Fine-Tuning (RFT) to further strengthen VLMs' hierarchical
reasoning and generalization, providing stepwise feedback on
reasoning coherence and factual correctness. Experiments show that CoTZero achieves an F1 score of 83.33 percent on our multi-level semantic inconsistency
benchmark with lexical-perturbation negatives, across both in-domain and out-of-domain settings. Ablations confirm that each component contributes to more interpretable and human-aligned visual
reasoning.
Read the full abstract ▾
All authors Chengyi Du, Yazhe Niu, Dazhong Shen, Luxin Xu
Computer Vision
Language-Guided Transformer Tokenizer for Human Motion Generation
Sheng Yan, Yong Wang, Xin Du et al. (+2 more)
In this paper, we focus on motion discrete tokenization, which converts raw motion into compact discrete
tokens–a process proven crucial for efficient motion generation. In this paradigm, increasing the number of
tokens is a common approach to improving motion reconstruction quality, but more
tokens make it more difficult for generative models to learn. To maintain high reconstruction quality while reducing generation complexity, we propose leveraging
language to achieve efficient motion tokenization, which we term
Language-Guided Tokenization (LG-Tok). LG-Tok aligns natural
language with motion at the tokenization stage, yielding compact, high-level semantic representations. This approach not only strengthens both tokenization and detokenization but also simplifies the learning of generative models. Furthermore, existing tokenizers predominantly adopt convolutional architectures, whose local receptive fields struggle to support global
language guidance. To this end, we propose a
Transformer-based Tokenizer that leverages
attention mechanisms to enable effective
alignment between
language and motion. Additionally, we design a
language-drop scheme, in which
language conditions are randomly removed during training, enabling the detokenizer to support
language-free guidance during generation. On the HumanML3D and Motion-X generation
benchmarks, LG-Tok achieves Top-1 scores of 0.542 and 0.582, outperforming state-of-the-art methods (MARDM: 0.500 and 0.528), and with FID scores of 0.057 and 0.088, respectively, versus 0.114 and 0.147. LG-Tok-mini uses only half the
tokens while maintaining competitive performance (Top-1: 0.521/0.588, FID: 0.085/0.071), validating the efficiency of our semantic representations. Code and checkpoints are available at https://eanson023.github.io/LG-Tok/
Read the full abstract ▾
All authors Sheng Yan, Yong Wang, Xin Du, Junsong Yuan, Mengyuan Liu
Computer Vision · Cross-listed from cs.CL
Cheng Yang, Chufan Shi, Bo Shui et al. (+7 more)
Unified
multimodal models (UMMs) aim to integrate
multimodal understanding and generation within a unified architecture, yet it remains unclear to what extent textual and visual modalities are aligned. To investigate this question, we use
reasoning-guided image generation as a diagnostic task, where models produce textual
reasoning first and then generate images. We introduce UReason, a
benchmark for evaluating
reasoning-to-generation
alignment in this paradigm, consisting of 2,000 human-curated and human-verified instances spanning five
reasoning-intensive tasks: Code, Arithmetic, Spatial, Attribute, and Text. To enable controlled analysis, we develop an evaluation framework that compares direct generation,
reasoning-guided generation, and decontextualized generation, which conditions only on the refined
prompt extracted from
reasoning. Across eight widely used open-source UMMs, while we find that
reasoning-guided generation yields improvements over direct generation, somewhat surprisingly, decontextualized generation consistently outperforms
reasoning-guided generation by a large margin. Our further analyses suggest that the intended visual semantics in textual
reasoning are not reliably reflected in the generated images, despite their unified design and training. Overall, UReason serves as a practical litmus test for
reasoning-to-generation
alignment and provides a challenging
benchmark for developing next-generation, more tightly aligned UMMs.
Read the full abstract ▾
All authors Cheng Yang, Chufan Shi, Bo Shui, Yaokang Wu, Muzi Tao, Huijuan Wang, Ivan Yee Lee, Yong Liu, Xuezhe Ma, Taylor Berg-Kirkpatrick
Computer Vision
CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichment
Yunzuo Hu, Wen Li and Jing Zhang
Audio-visual learning suffers from modality misalignment caused by off-screen sources and background clutter, and current methods usually amplify irrelevant regions or moments, leading to unstable training and degraded representation quality. To address this challenge, we proposed a novel Caption-aligned and Agreement-guided Enhancement framework (CAE-AV) for audio-visual learning, which used two complementary modules: Cross-modal Agreement-guided Spatio-Temporal Enrichment (CASTE) and Caption-Aligned Saliency-guided Enrichment (CASE) to relieve audio-visual misalignment. CASTE dynamically balances spatial and temporal relations by evaluating frame-level audio-visual agreement, ensuring that key information is captured from both preceding and subsequent frames under misalignment. CASE injects cross-modal semantic guidance into selected spatio-temporal positions, leveraging high-level semantic cues to further alleviate misalignment. In addition, we design lightweight objectives, caption-to-modality InfoNCE, visual-audio consistency, and
entropy regularization to guide
token selection and strengthen cross-modal semantic
alignment. With frozen backbones, CAE-AV achieves state-of-the-art performance on AVE, AVVP, AVS, and AVQA
benchmarks, and qualitative analyses further validate its robustness against audio-visual misalignment.
Read the full abstract ▾
Computer Vision
Tighnari v2: Mitigating Label Noise and Distribution Shift in Multimodal Plant Distribution Prediction via Mixture of Experts and Weakly Supervised Learning
Haixu Liu, Yufei Wang, Tianxiang Xu et al. (+2 more)
Large-scale, cross-species plant distribution prediction plays a crucial role in biodiversity conservation, yet modeling efforts in this area still face significant challenges due to the sparsity and bias of observational data. Presence-Absence (PA) data provide accurate and noise-free labels, but are costly to obtain and limited in quantity; Presence-Only (PO) data, by contrast, offer broad spatial coverage and rich spatiotemporal distribution, but suffer from severe label noise in negative samples. To address these real-world constraints, this paper proposes a
multimodal fusion framework that fully leverages the strengths of both PA and PO data. We introduce an innovative pseudo-label aggregation strategy for PO data based on the geographic coverage of satellite imagery, enabling geographic
alignment between the label space and remote sensing feature space. In terms of model architecture, we adopt Swin
Transformer Base as the backbone for satellite imagery, utilize the TabM network for tabular feature extraction, retain the Temporal Swin
Transformer for time-series modeling, and employ a stackable serial tri-modal cross-
attention mechanism to optimize the fusion of heterogeneous modalities. Furthermore, empirical analysis reveals significant geographic distribution shifts between PA training and test samples, and models trained by directly mixing PO and PA data tend to experience performance degradation due to label noise in PO data. To address this, we draw on the mixture-of-experts paradigm: test samples are partitioned according to their spatial proximity to PA samples, and different models trained on distinct
datasets are used for inference and post-processing within each partition. Experiments on the GeoLifeCLEF 2025
dataset demonstrate that our approach achieves superior predictive performance in scenarios with limited PA coverage and pronounced distribution shifts.
Read the full abstract ▾
All authors Haixu Liu, Yufei Wang, Tianxiang Xu, Chuancheng Shi, Hongsheng Xing
Computer Vision
PISCO: Precise Video Instance Insertion with Sparse Control
Xiangbo Gao, Renjie Li, Xinghao Chen et al. (+4 more)
The landscape of AI
video generation is undergoing a pivotal shift: moving beyond general generation - which relies on exhaustive
prompt-engineering and "cherry-picking" - towards fine-grained, controllable generation and high-fidelity post-processing. In professional AI-assisted filmmaking, it is crucial to perform precise, targeted modifications. A cornerstone of this transition is
video instance insertion, which requires inserting a specific instance into existing footage while maintaining scene integrity. Unlike traditional
video editing, this task demands several requirements: precise spatial-temporal placement, physically consistent scene interaction, and the faithful preservation of original dynamics - all achieved under minimal user effort. In this paper, we propose PISCO, a
video diffusion model for precise
video instance insertion with arbitrary sparse keyframe control. PISCO allows users to specify a single keyframe, start-and-end keyframes, or sparse keyframes at arbitrary timestamps, and automatically propagates object appearance, motion, and interaction. To address the severe distribution shift induced by sparse conditioning in pretrained
video diffusion models, we introduce Variable-Information Guidance for robust conditioning and Distribution-Preserving Temporal Masking to stabilize temporal generation, together with geometry-aware conditioning for realistic scene adaptation. We further construct PISCO-Bench, a
benchmark with verified instance annotations and paired clean background
videos, and evaluate performance using both reference-based and reference-free perceptual metrics. Experiments demonstrate that PISCO consistently outperforms strong inpainting and
video editing baselines under sparse control, and exhibits clear, monotonic performance improvements as additional control signals are provided. Project page: xiangbogaobarry.github.io/PISCO.
Read the full abstract ▾
All authors Xiangbo Gao, Renjie Li, Xinghao Chen, Yuheng Wu, Suofei Feng, Qing Yin, Zhengzhong Tu
Computer Vision · Cross-listed from cs.RO
Informative Object-centric Next Best View for Object-aware 3D Gaussian Splatting in Cluttered Scenes
Seunghoon Jeong, Eunho Lee, Jeongyun Kim et al. (+1 more)
In cluttered scenes with inevitable occlusions and incomplete observations, selecting informative viewpoints is essential for building a reliable representation. In this context, 3D Gaussian Splatting (3DGS) offers a distinct advantage, as it can explicitly guide the selection of subsequent viewpoints and then refine the representation with new observations. However, existing approaches rely solely on
geometric cues, neglect manipulation-relevant semantics, and tend to prioritize exploitation over exploration. To tackle these limitations, we introduce an instance-aware Next Best View (NBV) policy that prioritizes underexplored regions by leveraging object features. Specifically, our object-aware 3DGS distills instancelevel information into one-hot object vectors, which are used to compute confidence-weighted information gain that guides the identification of regions associated with erroneous and uncertain Gaussians. Furthermore, our method can be easily adapted to an object-centric NBV, which focuses view selection on a target object, thereby improving reconstruction robustness to object placement. Experiments demonstrate that our NBV policy reduces depth error by up to 77.14% on the synthetic
dataset and 34.10% on the real-world GraspNet
dataset compared to baselines. Moreover, compared to targeting the entire scene, performing NBV on a specific object yields an additional reduction of 25.60% in depth error for that object. We further validate the effectiveness of our approach through real-world robotic manipulation tasks.
Read the full abstract ▾
All authors Seunghoon Jeong, Eunho Lee, Jeongyun Kim, Ayoung Kim
Computer Vision
Moving Beyond Functional Connectivity: Time-Series Modeling for fMRI-Based Brain Disorder Classification
Guoqi Yu, Xiaowei Hu, Angelica I. Aviles-Rivero et al. (+2 more)
Functional
magnetic resonance imaging (fMRI) enables non-invasive
brain disorder classification by capturing blood-oxygen-level-dependent (BOLD) signals. However, most existing methods rely on functional connectivity (FC) via Pearson correlation, which reduces 4D BOLD signals to static 2D
matrices, discarding temporal dynamics and capturing only linear inter-regional relationships. In this work, we
benchmark state-of-the-art temporal models (e.g., time-series models such as PatchTST, TimesNet, and TimeMixer) on raw BOLD signals across five public
datasets. Results show these models consistently outperform traditional FC-based approaches, highlighting the value of directly modeling temporal information such as cycle-like oscillatory fluctuations and drift-like slow baseline trends. Building on this insight, we propose DeCI, a simple yet effective framework that integrates two key principles: (i) Cycle and Drift Decomposition to disentangle cycle and drift within each ROI (Region of Interest); and (ii) Channel-Independence to model each ROI separately, improving robustness and reducing overfitting. Extensive experiments demonstrate that DeCI achieves superior classification accuracy and generalization compared to both FC-based and temporal baselines. Our findings advocate for a shift toward end-to-end temporal modeling in fMRI analysis to better capture complex
brain dynamics. The code is available at https://github.com/Levi-Ackman/DeCI.
Read the full abstract ▾
All authors Guoqi Yu, Xiaowei Hu, Angelica I. Aviles-Rivero, Anqi Qiu, Shujun Wang
Computer Vision · Cross-listed from eess.IV
A Unified Framework for Multimodal Image Reconstruction and Synthesis using Denoising Diffusion Models
Weijie Gan, Xucheng Wang, Tongyao Wang et al. (+6 more)
Image reconstruction and image synthesis are important for handling incomplete
multimodal imaging data, but existing methods require various task-specific models, complicating training and deployment workflows. We introduce Any2all, a unified framework that addresses this limitation by formulating these disparate tasks as a single virtual inpainting problem. We train a single, unconditional
diffusion model on the complete
multimodal data stack. This model is then adapted at inference time to “inpaint” all target modalities from any combination of inputs of available clean images or noisy measurements. We validated Any2all on a PET/MR/CT
brain dataset. Our results show that Any2all can achieve excellent performance on both
multimodal reconstruction and synthesis tasks, consistently yielding images with competitive distortion-based performance and superior perceptual quality over specialized methods.
Read the full abstract ▾
All authors Weijie Gan, Xucheng Wang, Tongyao Wang, Wenshang Wang, Chunwei Ying, Yuyang Hu, Yasheng Chen, Hongyu An, Ulugbek S. Kamilov
Computer Vision · Cross-listed from cs.AI
Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMs
Siqu Ou, Tianrui Wan, Zhiyuan Zhao et al. (+2 more)
While chain-of-thought (CoT)
reasoning has substantially improved
multimodal large
language models (MLLMs) on complex
reasoning tasks, existing approaches largely rely on long textual
reasoning trajectories and provide limited mechanisms for learning stable visual
attention policies. Our analysis shows that current MLLMs exhibit weak visual focus: early-stage visual misalignment is rarely corrected during subsequent
reasoning, leading to error propagation and failed inferences. We argue that this limitation stems from inadequate credit assignment for visual
attention during training. To address this issue, we propose SAYO, a visual
reasoning model trained with a reinforcement learning (RL) framework that introduces a region-level visual
attention-based reward. This reward explicitly aligns optimization signals with visually grounded
reasoning steps, enabling the model to learn more reliable
attention behaviors. Extensive experiments across multiple
multimodal benchmarks demonstrate that SAYO consistently improves performance on diverse
reasoning and perception tasks. Project is open at https://github.com/Cratileo/Sayo.
Read the full abstract ▾
All authors Siqu Ou, Tianrui Wan, Zhiyuan Zhao, Junyu Gao, Xuelong Li
Computer Vision
When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning
Shoubin Yu, Yue Zhang, Zun Wang et al. (+4 more)
Despite rapid progress in MLLMs, visual spatial
reasoning remains unreliable when correct answers depend on how a scene would appear under unseen or alternative viewpoints. Recent work addresses this by augmenting
reasoning with world models for visual imagination, but questions such as when imagination is actually necessary, how much of it is beneficial, and when it becomes harmful, remain poorly understood. In practice, indiscriminate imagination can increase computation and even degrade performance by introducing misleading evidence. In this work, we present an in-depth analysis of test-time visual imagination as a controllable resource for spatial
reasoning. We first study when static visual evidence is sufficient, when imagination improves
reasoning, and how excessive or unnecessary imagination affects accuracy and efficiency. To support this analysis, we then introduce AVIC, an adaptive test-time framework with world models that explicitly reasons about the sufficiency of current visual evidence before selectively invoking and scaling visual imagination. Finally, to further learn this gating and planning behavior without any annotation of when and how much to imagine, we introduce AVIC-R, which trains the policy via GRPO from QA-correctness rewards and penalties by imagination cost. Across spatial
reasoning benchmarks (SAT, MMSI) and an
embodied navigation benchmark (R2R), our results reveal clear scenarios where imagination is critical, marginal, or detrimental, and show that selective control can match or outperform fixed imagination strategies with substantially fewer world-model calls and
language tokens. Our AVIC-R surpasses strong proprietary baselines including GPT-4o and GPT-4.1 while invoking the world model less often. Overall, our findings highlight the importance of analyzing and controlling test-time imagination for efficient and reliable spatial
reasoning.
Read the full abstract ▾
All authors Shoubin Yu, Yue Zhang, Zun Wang, Jaehong Yoon, Huaxiu Yao, Mingyu Ding, Mohit Bansal
Computer Vision
Generating Adversarial Events: A Motion-Aware Point Cloud Framework
Hongwei Ren, Youxin Jiang, Qifei Gu et al. (+1 more)
Event cameras have been widely adopted in safety-critical domains such as autonomous driving, robotics, and human-computer interaction. A pressing challenge arises from the vulnerability of deep neural networks to
adversarial examples, which poses a significant threat to the reliability of event-based systems. Nevertheless, research into
adversarial attacks on events is scarce. This is primarily due to the non-differentiable nature of mainstream event representations, which hinders the extension of gradient-based attack methods. In this paper, we propose MA-ADV, a novel Motion-Aware
Adversarial framework. To the best of our knowledge, this is the first work to generate
adversarial events by leveraging point cloud representations. MA-ADV accounts for high-frequency noise in events and employs a
diffusion-based approach to smooth perturbations, while fully leveraging the spatial and temporal relationships among events. Finally, MA-ADV identifies the minimal-cost perturbation through a combination of sample-wise Adam optimization, iterative refinement, and binary search. Extensive experimental results validate that MA-ADV ensures a 100% attack success rate with minimal perturbation cost, and also demonstrate enhanced robustness against defenses, underscoring the critical security challenges facing future event-based perception systems.
Read the full abstract ▾
All authors Hongwei Ren, Youxin Jiang, Qifei Gu, Xiangqian Wu
Computer Vision
Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval
Jing Zhang, Zhikai Li, Xuewen Liu et al. (+1 more)
Segment Anything Model 2 (SAM2) shows excellent performance in
video object
segmentation tasks; however, the heavy computational burden hinders its application in real-time
video processing. Although there have been efforts to improve the efficiency of SAM2, most of them focus on retraining a lightweight backbone, with little exploration into post-training acceleration. In this paper, we observe that SAM2 exhibits sparse perception pattern as biological
vision, which provides opportunities for eliminating redundant computation and acceleration: i) In mask decoder, the
attention primarily focuses on the foreground objects, whereas the image encoder in the earlier stage exhibits a broad
attention span, which results in unnecessary computation to background regions. ii) In
memory bank, only a small subset of
tokens in each frame contribute significantly to
memory attention, and the salient regions exhibit temporal consistency, making full-
token computation redundant. With these insights, we propose Efficient-SAM2, which promotes SAM2 to adaptively focus on object regions while eliminating task-irrelevant computations, thereby significantly improving inference efficiency. Specifically, for image encoder, we propose object-aware Sparse Window Routing (SWR), a window-level computation allocation mechanism that leverages the consistency and saliency cues from the previous-frame decoder to route background regions into a lightweight shortcut branch. Moreover, for
memory attention, we propose object-aware Sparse
Memory Retrieval (SMR), which allows only the salient
memory tokens in each frame to participate in computation, with the saliency pattern reused from their first recollection. With negligible additional parameters and minimal training overhead, Efficient-SAM2 delivers 1.68x speedup on SAM2.1-L model with only 1.0% accuracy drop on SA-V test set.
Read the full abstract ▾
All authors Jing Zhang, Zhikai Li, Xuewen Liu, Qingyi Gu
Computer Vision
Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehension
Yik Lung Pang and Changjae Oh
Given a textual description, the task of referring expression comprehension (REC) involves the localisation of the referred object in an image.
Multimodal large
language models (MLLMs) have achieved high accuracy on REC
benchmarks through scaling up the model size and training data. Moreover, the performance of MLLMs can be further improved using techniques such as Chain-of-Thought and tool use, which provides additional visual or textual context to the model. In this paper, we analyse the effect of various techniques for providing additional visual and textual context via tool use to the MLLM and its effect on the REC task. Furthermore, we propose a training-free framework named Chain-of-Caption to improve the REC performance of MLLMs. We perform experiments on RefCOCO/RefCOCOg/RefCOCO+ and Ref-L4
datasets and show that individual textual or visual context can improve the REC performance without any fine-tuning. By combining multiple contexts, our training-free framework shows between 5% to 30% performance gain over the baseline model on accuracy at various Intersection over Union (IoU) thresholds.
Read the full abstract ▾
Computer Vision
Geospatial-Reasoning-Driven Vocabulary-Agnostic Remote Sensing Semantic Segmentation
Chufeng Zhou, Jian Wang, Xinyuan Liu et al. (+1 more)
Open-vocabulary semantic
segmentation has become an important direction in remote sensing, as it enables recognition beyond predefined land-cover categories. However, existing methods mainly depend on passive visual-text matching and often struggle with semantic ambiguity in geographically complex scenes, especially when different classes exhibit similar spectral or structural patterns. To address this issue, we propose a Geospatial
Reasoning Chain-of-Thought (GR-CoT) framework for remote sensing open-vocabulary semantic
segmentation. GR-CoT consists of an offline knowledge
distillation stream and an online instance
reasoning stream. The former constructs category interpretation standards for confusing classes, while the latter performs macro-scenario anchoring, visual feature decoupling, and knowledge-driven decision synthesis to generate an image-adaptive vocabulary for downstream
segmentation. Experiments on the LoveDA and GID5
benchmarks indicate that the proposed framework improves overall
segmentation performance and yields more semantically coherent predictions in complex scenes.
Read the full abstract ▾
All authors Chufeng Zhou, Jian Wang, Xinyuan Liu, Xiaokang Zhang
Computer Vision
Generative Regression for Left Ventricular Ejection Fraction Estimation from Echocardiography Video
Jinrong Lv, Xun Gong, Zhaohuan Li et al. (+1 more)
Estimating Left Ventricular Ejection Fraction (LVEF) from echocardiograms constitutes an ill-posed inverse problem. Inherent noise, artifacts, and limited viewing angles introduce ambiguity, where a single
video sequence may map not to a unique ground truth, but rather to a distribution of plausible physiological values. Prevailing deep learning approaches typically formulate this task as a standard regression problem that minimizes the Mean Squared Error (MSE). However, this paradigm compels the model to learn the conditional expectation, which may yield misleading predictions when the underlying posterior distribution is
multimodal or heavy-tailed – a common phenomenon in pathological scenarios. In this paper, we investigate the paradigm shift from deterministic regression toward generative regression. We propose the
Multimodal Conditional Score-based
Diffusion model for Regression (MCSDR), a probabilistic framework designed to model the continuous posterior distribution of LVEF conditioned on echocardiogram
videos and patient demographic attribute priors. Extensive experiments conducted on the EchoNet-Dynamic, EchoNet-Pediatric, and CAMUS
datasets demonstrate that MCSDR achieves state-of-the-art performance. Notably, qualitative analysis reveals that the generation trajectories of our model exhibit distinct behaviors in cases characterized by high noise or significant physiological variability, thereby offering a novel layer of interpretability for AI-aided diagnosis.
Read the full abstract ▾
All authors Jinrong Lv, Xun Gong, Zhaohuan Li, Weili Jiang
Computer Vision
PEGAsus: 3D Personalization of Geometry and Appearance
Jingyu Hu, Bin Hu, Ka-Hei Hui et al. (+4 more)
We present PEGAsus, a new framework capable of generating Personalized 3D shapes by learning shape concepts at both Geometry and Appearance levels. First, we formulate 3D shape personalization as extracting reusable, category-agnostic
geometric and appearance attributes from reference shapes, and composing these attributes with text to generate novel shapes. Second, we design a progressive optimization strategy to learn shape concepts at both the geometry and appearance levels, decoupling the shape concept learning process. Third, we extend our approach to region-wise concept learning, enabling flexible concept extraction, with context-aware and context-free losses. Extensive experimental results show that PEGAsus is able to effectively extract attributes from a wide range of reference shapes and then flexibly compose these concepts with text to synthesize new shapes. This enables fine-grained control over shape generation and supports the creation of diverse, personalized results, even in challenging cross-category scenarios. Both quantitative and qualitative experiments demonstrate that our approach outperforms existing state-of-the-art solutions.
Read the full abstract ▾
All authors Jingyu Hu, Bin Hu, Ka-Hei Hui, Haipeng Li, Zhengzhe Liu, Daniel Cohen-Or, Chi-Wing Fu
Computer Vision · Cross-listed from cs.RO
Chamelion: Reliable Change Detection for Long-Term LiDAR Mapping in Transient Environments
Seoyeon Jang, Alex Junho Lee, I Made Aswin Nahrendra et al. (+1 more)
Online change detection is crucial for mobile
robots to efficiently navigate through dynamic environments. Detecting changes in transient settings, such as active construction sites or frequently reconfigured indoor spaces, is particularly challenging due to frequent occlusions and spatiotemporal variations. Existing approaches often struggle to detect changes and fail to update the map across different observations. To address these limitations, we propose a dual-head network designed for online change detection and long-term map maintenance. A key difficulty in this task is the collection and
alignment of real-world data, as manually registering structural differences over time is both labor-intensive and often impractical. To overcome this, we develop a data augmentation strategy that synthesizes structural changes by importing elements from different scenes, enabling effective model training without the need for extensive ground-truth annotations. Experiments conducted at real-world construction sites and in indoor office environments demonstrate that our approach generalizes well across diverse scenarios, achieving efficient and accurate map updates.\resubmit{Our source code and additional material are available at: https://chamelion-pages.github.io/.
Read the full abstract ▾
All authors Seoyeon Jang, Alex Junho Lee, I Made Aswin Nahrendra, Hyun Myung
Computer Vision
DAS-SK: An Adaptive Model Integrating Dual Atrous Separable and Selective Kernel CNN for Agriculture Semantic Segmentation
Mei Ling Chee, Thangarajah Akilan, Aparna Ravindra Phalke et al. (+1 more)
Semantic
segmentation in high-resolution agricultural imagery demands models that strike a careful balance between accuracy and computational efficiency to enable deployment in practical systems. In this work, we propose DAS-SK, a novel lightweight architecture that retrofits selective kernel convolution (SK-Conv) into the dual atrous separable convolution (DAS-Conv) module to strengthen multi-scale feature learning. The model further enhances the atrous spatial pyramid pooling (ASPP) module, enabling the capture of fine-grained local structures alongside global contextual information. Built upon a modified DeepLabV3 framework with two complementary backbones - MobileNetV3-Large and EfficientNet-B3, the DAS-SK model mitigates limitations associated with large
dataset requirements, limited spectral generalization, and the high computational cost that typically restricts deployment on
UAVs and other edge devices. Comprehensive experiments across three
benchmarks: LandCover.ai, VDD, and PhenoBench, demonstrate that DAS-SK consistently achieves state-of-the-art performance, while being more efficient than CNN-,
transformer-, and hybrid-based competitors. Notably, DAS-SK requires up to 21x fewer parameters and 19x fewer GFLOPs than top-performing
transformer models. These findings establish DAS-SK as a robust, efficient, and scalable solution for real-time agricultural robotics and high-resolution remote sensing, with strong potential for broader deployment in other
vision domains.
Read the full abstract ▾
All authors Mei Ling Chee, Thangarajah Akilan, Aparna Ravindra Phalke, Kanchan Keisham
Computer Vision · Cross-listed from cs.RO
Self-Supervised Bootstrapping of Action-Predictive Embodied Reasoning
Milan Ganai, Katie Luo, Jonas Frey et al. (+2 more)
Embodied Chain-of-Thought (CoT)
reasoning has significantly enhanced
Vision-
Language-Action (VLA) models, yet current methods rely on rigid templates to specify
reasoning primitives (e.g., objects in the scene, high-level plans, structural affordances). These templates can force policies to process irrelevant information that distracts from critical action-prediction signals. This creates a bottleneck: without successful policies, we cannot verify
reasoning quality; without quality
reasoning, we cannot build robust policies. We introduce R&B-EnCoRe, which enables models to bootstrap
embodied reasoning from internet-scale knowledge through self-supervised refinement. By treating
reasoning as a latent variable within importance-weighted variational inference, models can generate and distill a refined
reasoning training
dataset of embodiment-specific strategies without external rewards, verifiers, or human annotation. We validate R&B-EnCoRe across manipulation (Franka Panda in simulation, WidowX in hardware), legged
navigation (bipedal, wheeled, bicycle, quadruped), and autonomous driving embodiments using various VLA architectures with 1B, 4B, 7B, and 30B parameters. Our approach achieves 28% gains in manipulation success, 101% improvement in
navigation scores, and 21% reduction in collision-rate metric over models that indiscriminately reason about all available primitives. R&B-EnCoRe enables models to distill
reasoning that is predictive of successful control, bypassing manual annotation engineering while grounding internet-scale knowledge in physical execution.
Read the full abstract ▾
All authors Milan Ganai, Katie Luo, Jonas Frey, Clark Barrett, Marco Pavone