Computer Vision · Cross-listed from cs.RO
How to Relieve Distribution Shifts in Semantic Segmentation for Off-Road Environments
Ji-Hoon Hwang, Daeyoung Kim, Hyung-Suk Yoon et al. (+2 more)
Semantic
segmentation is crucial for autonomous
navigation in off-road environments, enabling precise classification of surroundings to identify traversable regions. However, distinctive factors inherent to off-road conditions, such as source-target domain discrepancies and sensor corruption from rough terrain, can result in distribution shifts that alter the data differently from the trained conditions. This often leads to inaccurate semantic label predictions and subsequent failures in
navigation tasks. To address this, we propose ST-Seg, a novel framework that expands the source distribution through style expansion (SE) and texture regularization (TR). Unlike prior methods that implicitly apply generalization within a fixed source distribution, ST-Seg offers an intuitive approach for distribution shift. Specifically, SE broadens domain coverage by generating diverse realistic styles, augmenting the limited style information of the source domain. TR stabilizes local texture representation affected by style-augmented learning through a deep texture
manifold. Experiments across various distribution-shifted target domains demonstrate the effectiveness of ST-Seg, with substantial improvements over existing methods. These results highlight the robustness of ST-Seg, enhancing the real-world applicability of semantic
segmentation for off-road
navigation.
Read the full abstract ▾
All authors Ji-Hoon Hwang, Daeyoung Kim, Hyung-Suk Yoon, Dong-Wook Kim, Seung-Woo Seo
Computer Vision
Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning
Xiang Tan, Run He, Yawen Cui et al. (+6 more)
Class-Incremental Learning (CIL) with pre-trained models (PTMs) aims to sequentially adapt PTMs to new categories without forgetting old knowledge. Built upon PTMs, existing adapter-based methods mainly train models via distinct task-specific adapters, and present a uniform knowledge allocation for each adapter during inference. However, this allocation mechanism ignores the nature of task discrepancy and leads to suboptimal utilization of adapters. Also, under CIL constraint, an allocator is prone to forgetting when tasks evolve. To address these issues, we propose a Non-Forgetting Allocation with Bi-Level Competition (NoFA-BC). NoFA-BC constructs a non-forgetting allocator (NFA) by transforming the allocator training into a recursive least-squares problem and achieves an allocator equivalent to that trained with all data. Based on the NFA, a Bi-Level Competition (BLC) including an intra-task level Winner-Takes-All (WTA) mechanism and inter-task Last-Ones-Fall (LOF) elimination is proposed to provide better allocation of adapter knowledge. WTA extracts the most significant logit within a task to represent the adapter's contribution and LOF suppresses the irrelevant adapters. With BLC, participation ratio of each adapter can be tailored for each input. Moreover, a Stability Enhancement (SE) process is incorporated to further improve the performance of old tasks.
Read the full abstract ▾
All authors Xiang Tan, Run He, Yawen Cui, Mengchen Zhao, Yan Wu, Tianyi Chen, Huiping Zhuang, Xiaonan Luo, Guanbin Li
Computer Vision
Brain-IT-VQA: From Brain Signals to Answers
Roman Beliy, Matias Cosarinsky, Oliver Heinimann et al. (+2 more)
Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the
brain. We present
Brain-IT-VQA, a framework for visual question answering from fMRI. Unlike previous methods, which extract a fixed representation from the fMRI signal, our extraction is conditioned on the question itself, so what is decoded from
brain activity depends on what is being asked. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce NSD-VQA, a new
dataset and
benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA
datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together,
Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying
brain representations. Using this
benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different
brain regions across question types.
Read the full abstract ▾
All authors Roman Beliy, Matias Cosarinsky, Oliver Heinimann, Navve Wasserman, Michal Irani
Computer Vision
BitC-3DGS: High-Capacity 3D Gaussian Splatting Watermarking via Bit Compression
Yuquan Bi, Baosheng Yu, Yingke Lei et al. (+5 more)
High-capacity watermarking is necessary for 3D Gaussian Splatting (3DGS) assets to embed rich information (\eg, ownership, provenance, and authentication codes), enabling reliable identification and provenance tracking in large-scale 3D asset pipelines. Existing bit-to-
token watermarking methods based on a pre-trained text encoder are limited to 77-bit messages due to CLIP's fixed 77-
token context length, as
tokens beyond this limit are unsupported by learned positional
embeddings. To address this limitation, we introduce BitC-3DGS, a bit-compression framework that encodes multiple message bits per
token. It employs a bit-compressed tokenization scheme that encodes multiple bits within the same chunk into a single semantic
token. To enable recovery of the compressed information, it further introduces a dual-branch architecture for joint chunk decompression and bit decoding, along with a hard-message sampling strategy to improve combinatorial coverage during decoder training. Extensive experiments on the Blender and LLFF
datasets demonstrate the effectiveness of BitC-3DGS for high-capacity watermarking, achieving high message recovery accuracy and rendering fidelity. For example, it supports 128-bit message capacity with recovery accuracy comparable to that of 64-bit messages in recent state-of-the-art methods.
Read the full abstract ▾
All authors Yuquan Bi, Baosheng Yu, Yingke Lei, Jianwei Yang, Hongsong Wang, Jie Gui, Yuan Yan Tang, James Tin-Yau Kwok
Computer Vision
Shizhe Zhou, Bohan Jia, Kai Wu et al. (+4 more)
While
multimodal large
language models (MLLMs) have achieved rapid progress in
vision-
language understanding, they remain prone to
multimodal hallucinations, producing responses that are inconsistent with the visual input. Existing
benchmarks predominantly focus on detecting
hallucination outcomes rather than evaluating the underlying causes of these failures. Moreover, many
benchmarks rely on simplistic scenarios and limited evaluation formats that no longer challenge state-of-the-art models. To address these limitations, we introduce ReactBench, a cause-driven
hallucination benchmark featuring multiple tasks and an exam-style evaluation format. By generating
adversarial images and
hallucination-inducing queries, ReactBench introduces four targeted tasks: Relational Erasure, Counterfactual Attribute, Alteration Tracing, and Dense Counting. These tasks systematically expose co-occurrence bias,
language priors, cross-image comparative perception deficiencies, and fine-grained perceptual bottlenecks. Beyond standard accuracy-based evaluation, we leverage Chain-of-Thought
reasoning to identify fine-grained sub-causes of
hallucination within each task. Extensive evaluations reveal that current MLLMs remain notably vulnerable to cause-specific
hallucination triggers, demonstrating the value of ReactBench as a systematic and interpretable testbed for diagnosing and improving
multimodal model robustness. The project page is available at https://reactbench.github.io/.
Read the full abstract ▾
All authors Shizhe Zhou, Bohan Jia, Kai Wu, Yan Shen, Tongyun Li, Yuyang Wu, Shaohui Lin
Computer Vision
Mitigating State Aliasing in Vision-Language-Action Models via Inverse Dynamics Learning
Kyujin Lee, Injae Kim, Jihwan Park et al. (+3 more)
Vision-
Language-Action (VLA) models have emerged as a promising framework that unifies perception,
reasoning, and control for
robot manipulation by adapting pretrained
vision-
language models (VLMs) to action prediction. However, VLM-derived representations are often insensitive to subtle visual distinctions required for low-level control, causing state aliasing between visually similar states that require substantially different actions. Prior VLA studies improve visual understanding by generating visual or
reasoning outputs, such as future frames, 2D grounding points or traces, or intermediate spatial
reasoning steps, but these objectives typically shape the
vision encoder only indirectly through end-to-end prediction and do not explicitly analyze state aliasing in the learned visual feature space. To mitigate state aliasing, we introduce inverse dynamics learning as an auxiliary objective that directly supervises the VLA
vision encoder. By predicting the action between current and future observations, our objective encourages the encoder to capture fine-grained visual distinctions that determine low-level actions. We further use pseudo-reversed supervision to expose the encoder to a broader range of action directions and improve generalization under limited
robot demonstrations. Our method applies to diverse VLA baselines, uses only standard observation-action pairs without additional annotations, and preserves the original inference pipeline at test time. Experiments on CALVIN ABC-D and SimplerEnv show consistent gains across diverse VLA baselines. Frozen-encoder probing and state-feature
alignment analyses further show that our method learns state-discriminative visual representations that reduce state aliasing and better align with
robot state changes.
Read the full abstract ▾
All authors Kyujin Lee, Injae Kim, Jihwan Park, Yejun Ju, Minseok Joo, Hyunwoo J. Kim
Computer Vision
Optimizing Latent Representations for Robust Building Damage Assessment Onboard Earth Observation Satellites
Thomas Goudemant and Benjamin Francesconi
Rapid identification of damaged buildings after natural disasters or on war areas is crucial to support emergency response and prioritize interventions. Earth Observation constellations provide timely, large-scale coverage, but actionable information is often delayed by data downlink constraints, on-ground processing, and human interpretation. Reducing this latency is essential to improve decision-making responsiveness. In this work, we propose an original AI-based system that enables object-level building damage assessment (localization and damage classification) directly onboard satellites from pre-disaster and post-disaster highresolution optical imagery. Available pre-disaster images are encoded on ground into compact latent representations, transmitted to the satellite, and compared on-board with newly acquired post-event observations. Leveraging AI interpretation capabilities and increasing processing capabilities on-board satellites, the proposed design enables processing directly at the data source, reducing the amount of information to be downlinked while preserving task-relevant content and improving overall system responsivity. We explore the design space through a systematic
benchmark of onboard-compatible variants, analyzing the impact of siamese processing, cross-
attention, latent-space compression, and robustness-oriented data augmentation. Experiments on xBD
dataset demonstrate reliable and robust damage assessment under misalignment, with minimal performance degradation under strong compression.
Read the full abstract ▾
Computer Vision
DefSynUS: Real-time Patient-specific Intrahepatic Vessel Identification via Deformation-Aware CT-US Domain Adaptation
Karl-Philippe Beaudet, Yordanka Velikova, Sidaty El Hadramy et al. (+4 more)
Purpose: Laparoscopic ultrasound (LUS) enhances the safety of liver surgery by visualizing intrahepatic vessels in real-time. Still, vessel identification remains difficult due to probe constraints, complex vascular structure, and tissue deformation. This work aims to enable real-time, patient-specific vessel identification that remains robust under deformation through deformable ultrasound augmentation. Methods: Preoperative CT vessel annotations are used to generate synthetic ultrasound data via optimized physics-based rendering, coupled with domain adaptation to intraoperative ultrasound. The rendering is trained end-to-end for vessel identification and patient-specificity, eliminating the need for preoperative ultrasound. A deformation-aware augmentation simulates realistic intraoperative motion and tissue deformation within the rendering pipeline. Results: In abdominal phantom and limited clinical feasibility experiments (single-case clinical evaluation), the framework achieved real-time intrahepatic vessel-branch identification, maintaining performance under new patient poses. Conclusion: The framework enables real-time vessel identification without preoperative ultrasound and supports technical feasibility, but multi-patient validation is still needed for generalizability and clinical feasibility.
Read the full abstract ▾
All authors Karl-Philippe Beaudet, Yordanka Velikova, Sidaty El Hadramy, Nassir Navab, Philippe Cattin, Juan Verde, Stéphane Cotin
Computer Vision
From General Vision to Reliable Traversability Estimation: Adapting Vision Foundation Models for Unstructured Outdoor Environments
Ji-Hoon Hwang, Jisung Bae, Dong-Wook Kim et al. (+2 more)
Vision-based approaches have become the dominant paradigm for traversability estimation in unstructured outdoor environments, typically adapting
vision foundation models (VFMs) via semantic
segmentation supervision. However, this paradigm faces three fundamental challenges that undermine its reliability: the task-agnostic design of VFMs, the ambiguity of traversability annotations, and the discrepancy between semantic labels and physical safety. We propose
Vision-to-Traversability Adaptation (ViTA), a framework that adapts VFMs for reliable traversability estimation, instantiated on SAM2. ViTA injects task-specific knowledge through learnable traversability
prompts while preserving the VFM's cross-domain generalization. To handle annotation ambiguity, we introduce Perspective-Diversified Training, which estimates semantic uncertainty to suppress confident predictions at ambiguous boundaries. To bridge the semantic-traversability discrepancy, we distill
geometric knowledge during training, enabling slope and elevation
reasoning from RGB images alone at inference. The semantic and
geometric outputs are fused into a continuous traversability score that reflects both semantic uncertainty and
geometric risk. Evaluations across diverse domains, including challenging real-world off-road
datasets, demonstrate that ViTA achieves state-of-the-art IoU and Precision with substantial false-positive reduction and strong cross-domain generalization.
Read the full abstract ▾
All authors Ji-Hoon Hwang, Jisung Bae, Dong-Wook Kim, Yeonkyu Lee, Seung-Woo Seo
Computer Vision · Cross-listed from cs.AI
Planning with the Views
Kangrui Wang, Linjie Li, Zhengyuan Yang et al. (+7 more)
Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We call this capability view planning, requiring (1)understanding how a single action transforms the view, and (2)composing many such transformations across multi-turn plans to identify a target view. We probe both abilities in our proposed ViewSuite, a 3D point-cloud environment on real ScanNet scenes. Across 13 frontier VLMs, a critical planning gap emerges: they possess basic view-action knowledge but fail to compose it across multi-turn plans, with the gap widening as viewpoint distance grows. To close this gap, we propose an iterative framework that alternates self-exploration with view graph
distillation. The key insight is that all exploration trajectories, regardless of their outcome, collectively form a view graph that compactly captures how viewpoints connect across a scene. Distilling this graph into diverse supervised tasks reshapes the policy distribution and overcomes the sparse rewards that stall pure RL. This improves Qwen2.5-VL-7B from 2.5% to 47.8% on interactive view planning, surpassing GPT-5.4 Pro (18.5%) and Gemini 3.1 Pro (21.4%). Self-exploration emerges as a promising path toward VLMs that can actively reason and plan in 3D space. Code and Data are at https://viewsuite.github.io.
Read the full abstract ▾
All authors Kangrui Wang, Linjie Li, Zhengyuan Yang, Shiqi Chen, Zihan Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Lijuan Wang, Manling Li
Computer Vision · Cross-listed from cs.RO
VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models
Shengyu Si, Yuanzhuo Lu, Ruimeng Yang et al. (+3 more)
Vision-
Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation, yet they still struggle to generalize to unseen tasks that necessitate transferring relevant experience across objects, scenes, and action patterns. This paper proposes VLA-Pro, a plug-and-play framework designed to enhance cross-task generalization by storing task-relevant procedural
memories at training time and transferring these
memories during inference. Specifically, VLA-Pro stores task-specific LoRA adapters as parameterized procedural
memories during training. At inference time, VLA-Pro retrieves relevant procedural
memories based on the current multi-modal context and dynamically fuses these
memories for generating the current action chunk. Experiments on RoboTwin, RLBench, and real-world manipulation tasks show that VLA-Pro consistently improves cross-task generalization across multiple backbones, achieving up to a 207% relative improvement in simulation and increasing real-world success rate from 5.8% to 65.0%. These results suggest that procedural
memory retrieval and adaptation provide an effective mechanism for transferring manipulation experience to novel tasks while preserving modularity and execution stability.
Read the full abstract ▾
All authors Shengyu Si, Yuanzhuo Lu, Ruimeng Yang, Ziyi Ye, Zuxuan Wu, Yu-Gang Jiang
Computer Vision
TAE: Target-aware enhancer for nighttime UAV tracking
Yanyan Chen, Ruigang Fu, Yu Song et al. (+1 more)
Severe image degradation under low-light nighttime conditions constitutes a core bottleneck preventing all-day applications for
UAV-based single object tracking. Existing image enhancement methods often struggle to distinguish between target and background regions, which can easily lead to amplified background noise or compromise target features. To overcome this limitation, we propose TAE, a target-aware low-light enhancement framework tailored for nighttime object tracking. Guided explicitly by weak supervisory signals from tracking bounding boxes, the framework performs region-aware enhancement to ensure operations focus on the target area. It further adopts an adaptive RGB multi-curve fusion mechanism to achieve refined modeling and adaptive adjustment across different regions. To facilitate research in this domain, we also contribute DarkSOT, a new
benchmark for nighttime
UAV tracking, comprising 268 sequences across 9 target categories. Experimental results on the DarkSOT and UAVDark135 demonstrate that TAE significantly improves tracking performance in low-light nighttime scenarios, exhibiting strong robustness and generalization. The DarkSOT
dataset is available at https://github.com/Fu0511/DarkSOT-
Dataset.
Read the full abstract ▾
All authors Yanyan Chen, Ruigang Fu, Yu Song, Ping Zhong
Computer Vision
Learning Representations from 3D Gaussian Splats
Julia Farganus, Krzysztof Żurawicki, Arkadiusz Gaweł et al. (+2 more)
3D Gaussian Splatting (3DGS) is a recent approach for scene rendering. Although primarily designed for view synthesis, its potential for scene understanding tasks remains underexplored. In this work, we conduct a comparative evaluation of various
geometric deep learning architectures for the classification of 3D scenes represented using Gaussian Splatting. We
benchmark point-based and graph-based models across both traditional point cloud
datasets and dedicated Gaussian Splatting
datasets. Scenes are embedded into latent representations, which are evaluated through end-to-end classification, linear probing, and clustering analysis. Our study provides insight into the suitability of different geometry-aware architectures and input feature configurations for learning effective 3D Gaussian Splat representations. The results highlight consistent differences between architectural families and reveal the impact of Gaussian-specific attributes on the quality of representation.
Read the full abstract ▾
All authors Julia Farganus, Krzysztof Żurawicki, Arkadiusz Gaweł, Weronika Jakubowska, Halina Kwaśnicka
Computer Vision
GiPL: Generative augmented iterative Pseudo-Labeling for Cross-Domain Few-Shot Object Detection
Jiacong Liu, Shu Luo, Yikai Qin et al. (+3 more)
Vision-
language foundation models have shown promising zero-shot generalization for Cross-Domain Few-Shot Object Detection (CD-FSOD). However, they face two critical challenges in fine-tuning: insufficient support set utilization due to sparse single-instance annotations, and severe overfitting under extremely limited target-domain samples. To address these issues, this paper proposes GiPL, an efficient two-branch training framework. In the first branch, we design an iterative pseudo-label self-training paradigm, which performs zero-shot inference on the support set to generate reliable pseudo-annotations, fuses them with ground-truth labels, and iteratively optimizes the model to fully exploit support set data. In the second branch, we introduce generative data augmentation pipeline using large
vision-
language models, which synthesizes domain-aligned, multi-object annotated images to enrich training samples and suppress overfitting. Extensive experiments on three challenging CD-FSOD
datasets (RUOD, CARPK, CarDD) under 1/5/10-shot settings demonstrate that GiPL consistently outperforms state-of-the-art methods with significant performance gains. Code is available at \href{https://github.com/z-yaz/CDiscover}{CDiscover}.
Read the full abstract ▾
All authors Jiacong Liu, Shu Luo, Yikai Qin, Yaze Zhao, Yongwei Jiang, Yixiong Zou
Computer Vision
RadioFormer3D: Weakly Supervised 3D Radio Map Estimation in Low-Altitude Airspace via Generative Modeling
Zheng Fang, Junjie Liu, Kangjun Liu et al. (+3 more)
With the emergence of wireless applications in three-dimensional environments, such as the low-altitude airspace and 3D heterogeneous networks,
radio map estimation is increasingly required to characterize signal propagation across both horizontal and vertical dimensions. However, extending
radio map estimation from 2D to 3D remains challenging due to increased spatial sparsity and limited supervision across continuous altitudes. In this paper, we propose \textbf{RadioFormer3D}, a specialized model for volumetric spectrum reconstruction under weak supervision. Building on the dual-stream, multi-granularity fusion architecture of RadioFormer, RadioFormer3D introduces a
Fourier-based sampling encoder and a volumetric decoder to efficiently process sparse measurements in 3D space. To alleviate the lack of vertical supervision, we propose the \textbf{Joint Spectrum Integrity Loss}, which integrates volume-level pseudo-label supervision, map-level geometry-aware
radio rendering, and pixel-level localized constraints within a unified optimization scheme. This design enables the model to capture complex vertical structural relationships more effectively under sparse supervision. Extensive experiments across several
radio map
datasets show that RadioFormer3D achieves superior overall performance compared to representative existing methods. In particular, it demonstrates improved reconstruction quality at unlabeled altitudes while maintaining a favorable trade-off between accuracy and inference efficiency, positioning it as a highly promising solution for future 3D environment-aware wireless networks.
Read the full abstract ▾
All authors Zheng Fang, Junjie Liu, Kangjun Liu, Jianguo Zhang, Yaowei Wang, Ke Chen
Computer Vision · Cross-listed from cs.SD
Half-Truth Audio Detection and Localisation: A Lightweight Cross-Attentive Architecture and a Cross-Corpus Diagnostic Study
S. Sutharya and Remya K. Sasi
Partially manipulated (half-truth)
speech, where a short synthesised segment is spliced into an otherwise genuine utterance, is a harder and more realistic forensic threat than the fully synthesised deepfakes that dominate the literature. We present CAFNet, a lightweight (576K-parameter, 2.24 MB) cross-attentive architecture that fuses MFCC, LFCC, and Chroma-STFT features to jointly classify audio as real, fully fake, or half-truth, and regress the temporal boundaries of the synthesised region, at approximately 14 ms CPU latency. A component ablation shows cross-
attention fusion is CAFNet's most load-bearing component; a deeply supervised auxiliary classification head from earlier iterations is not, and removing it improves every in-domain metric under 3-seed replication with substantially lower variance. On MLADDC T2+T3 the model reaches 97.55%$\pm$0.69% ternary accuracy and 0.037 s boundary mean absolute error (MAE), to our knowledge, the first reported continuous splice- boundary localisation result on this
benchmark. Zero-shot evaluation on two independent
benchmarks shows transfer is capability- and corpus-dependent rather than uniform: on Half-Truth Audio Detection
dataset (HAD), detection recall reaches 84.9% and ternary classification resolves half-truth correctly on half of true half-truth clips (50.4%), while on PartialSpoof, binary detection stays near chance (AUC 0.5544). We treat this asymmetry, not a single generalization verdict, as the finding. HAD localisation improves in absolute terms but degrades in relative terms, since in-domain localisation improved faster. An architectural change validated purely in-domain thus shifted the cross-corpus transfer profile, evidence that cross-corpus evaluation should accompany, not follow, in-domain architecture decisions.
Read the full abstract ▾
Computer Vision
KGEdit: Ambiguity-Aware Knowledge Graphs for Training-Free Precise Video Generation and Editing
Mingshu Cai, Miao Zhang, Chenghe Yang et al. (+3 more)
In recent years, training-free
video generation has progressed remarkably. However, when handling complex textual instructions, existing methods still suffer from semantic ambiguity, incorrect concept binding, and cross-frame inconsistency. To address these issues, we propose KGEdit, a structured semantic control framework for text-to-
video (T2V)
diffusion models. Specifically, we first construct an ambiguity-aware knowledge graph (AAKG) to disentangle and disambiguate the input
prompt, converting it into four types of structured semantics: identity, relation, attribute, and negative constraints. We then design a structured semantic injection module (SSIM) to inject these semantic signals into key layers of the
diffusion Transformer, enabling fine-grained semantic control. In addition, we introduce a temporal-aware semantic control (TASC) module that dynamically schedules semantic objectives according to the stage-wise characteristics of the denoising process, further improving semantic
alignment and temporal consistency. Experiments show that KGEdit outperforms existing methods in editing precision and temporal stability, while offering higher efficiency and controllability in text-driven interaction scenarios.
Read the full abstract ▾
All authors Mingshu Cai, Miao Zhang, Chenghe Yang, Yixuan Li, Osamu Yoshie, Yuya Ieiri
Computer Vision
ESAM++: Efficient Online 3D Perception on the Edge
Qin Liu, Lavisha Aggarwal, Saptarashmi Bandyopadhyay et al. (+4 more)
Online 3D scene perception in real time is essential for robotics, AR/VR, and autonomous systems, particularly in edge computing scenarios where computational resources are limited and
privacy is crucial. Recent state-of-the-art methods like EmbodiedSAM (ESAM) demonstrate the promise of online 3D perception by leveraging the Segment Anything Model (SAM) for real-time, fine-grained, and generalized 3D instance
segmentation. However, ESAM still relies on a computationally expensive 3D sparse UNet for point cloud feature extraction, which accounts for the majority of the 3D inference time, hindering its practicality on resource-constrained devices. In this paper, we propose ESAM++, a lightweight and scalable alternative for online 3D scene perception tailored to edge devices without GPU acceleration. Our method introduces a 3D Sparse Feature Pyramid Network (SFPN) that efficiently captures multi-scale
geometric features from streaming 3D point clouds while significantly reducing computational overhead and model size. We evaluate our approach on four challenging
segmentation benchmarks, namely ScanNet, ScanNet200, SceneNN, and 3RScan, demonstrating that our model achieves competitive accuracy with up to 3 times faster inference with a 2 times smaller model size compared to ESAM, enabling practical deployment on edge devices.
Read the full abstract ▾
All authors Qin Liu, Lavisha Aggarwal, Saptarashmi Bandyopadhyay, Vikas Bahirwani, Marc Niethammer, Ehsan Adeli, Andrea Colaco
Computer Vision · Cross-listed from cs.CL
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
Runze Xu, Arpit Garg, Hemanth Saratchandran et al. (+1 more)
Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large
language models to new domains, tasks, and users. Yet adaptation performance alone can obscure an important failure mode: LoRA updates may improve performance on the target distribution while degrading prior capabilities learned during pretraining and
alignment. We show that this forgetting becomes especially severe when the adaptation distribution differs substantially from the models original training or
alignment distributions. The challenge is amplified in practical settings, where the original training and
alignment data are typically unavailable. Motivated by this constraint, we study how LoRA based adaptation balances new learning against forgetting in a replay-free setting, and introduce a simple output space regularizer that can be added directly to existing training pipelines. Our method removes the ground-truth
token from both the base and adapted model distributions, renormalizes the remaining probabilities, and applies KL regularization only over the non-target vocabulary. This preserves the base models relative preferences among alternative
tokens without directly opposing the cross-
entropy signal required for adaptation. As the regularizer acts only at the loss level, it requires no replay data, architectural changes, adapter redesign, or inference-time overhead, and can be applied directly to existing LoRA variants. Across all LoRA variants tested and across various backbones, our method improves the frontier between new learning and forgetting when the adaptation distribution differs substantially from the base models original training or
alignment distributions, suggesting a broadly applicable route toward more reliable LLM updating.
Read the full abstract ▾
All authors Runze Xu, Arpit Garg, Hemanth Saratchandran, Simon Lucey
Computer Vision · Cross-listed from cs.CL
On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training
Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang et al. (+1 more)
Post-training has greatly improved
reasoning in frontier
vision-
language models, yet its gains for perception remain comparatively limited, creating a bottleneck for end-to-end visual
reasoning. To investigate this gap, we introduce a controlled diagnostic framework with two synthetic tasks that disentangle perception from
reasoning. Our analysis reveals a consistent perception-
reasoning asymmetry: post-training improves
reasoning more substantially than perception, though the underlying mechanism differs across training paradigms. For supervised fine-tuning (SFT), this asymmetry stems from
token imbalance, with perception occupying a smaller fraction of
tokens in chain-of-thought supervision. Reweighting the loss boosts end-to-end performance by up to 18.2 points. For reinforcement learning (RL), the asymmetry instead arises from reward coupling, as outcome rewards correlate more strongly with
reasoning than perception. Adding a perception-aware reward improves end-to-end accuracy by up to 6.0 points; when ground-truth perception rewards are unavailable, a reliable surrogate provides useful signal, yielding gains of 2.2 points. Beyond the controlled setting, these strategies also improve real-world visual
reasoning, with gains of up to 3.3 points across three
benchmarks. Overall, we diagnose the causes of asymmetric optimization and provide actionable guidance that benefits both synthetic and realistic settings.
Read the full abstract ▾
All authors Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang, Nanyun Peng
Computer Vision
AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling
Yiheng Li, Zhuo Li, Ruibing Hou et al. (+4 more)
Conditional human motion generation remains a fundamental challenge in computer
vision and robotics. Despite significant progress, current methods are often constrained by fixed modality configurations and task-specific architectures, leaving cross-modal interactions and the scaling laws of
multimodal-conditioned synthesis largely underexplored. A key bottleneck is the scarcity of large-scale modality-aligned motion data, limiting generalization across diverse control signals. In this work, we introduce OmniHuMo, a large-scale, high-quality
dataset comprising over 5,000 hours of motion and 3.2 million sequences with precisely aligned
multimodal annotations (e.g., text,
speech, music, and trajectory). Leveraging OmniHuMo, we propose AnyMo, a unified
multimodal framework combining a Residual FSQ-based motion tokenizer with a scalable masked modeling
transformer, enabling high-quality motion synthesis under arbitrary modality combinations. Extensive experiments show that AnyMo achieves high-fidelity synthesis while offering flexible control over both spatial and stylistic attributes.
Read the full abstract ▾
All authors Yiheng Li, Zhuo Li, Ruibing Hou, Yingjie Chen, Hong Chang, Hao Liu, Shiguang Shan
Computer Vision
V2VCrafter: Consistent Street-View Image Generation Across Vehicles
Yihang Tao, Yu Guo, Senkang Hu et al. (+4 more)
Connected and autonomous driving (CAD) systems leverage vehicle-to-vehicle (V2V) communication for multi-
agent collaborative perception, yet remain constrained by scarce annotated real-world V2V
datasets and limited generalization across diverse driving conditions. While image generation offers a feasible solution for data augmentation, existing single-vehicle multi-view generation frameworks face two key challenges in multi-
agent settings: (1) the expanded learning objective degrades generation quality, and (2) dynamic inter-
agent variation hinders consistency modeling for physical attributes (e.g., color, category) of jointly observed objects. To bridge this gap, we propose V2VCrafter, the first framework for generating controllable and realistic multi-view driving images across vehicles. For effective learning, we develop a progressive multi-
agent diffusion model based on a single-
agent backbone, using neighboring
agents' latent states to progressively guide single-to-multi-
agent generation. To address cross-vehicle inconsistency, we further propose a cross-
agent attention module that leverages a collaboration view graph and learnable jointly observed object representations to model dynamic cross-vehicle camera view relationships. Experiments on real-world V2X-Real
dataset show that V2VCrafter generates high-fidelity, controllable, and consistent street views across vehicles, thereby effectively enhancing downstream collaborative 3D object detection tasks.
Read the full abstract ▾
All authors Yihang Tao, Yu Guo, Senkang Hu, Yanan Ma, Zihan Fang, Sam Kwong, Yuguang Fang
Computer Vision
Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset
Qian Chen, Xianyin Zhang, Yanzhi Liu et al. (+3 more)
The emergence of Large
Vision-
Language Models (LVLMs) has substantially expanded model capabilities beyond text-only understanding, enabling unified inference across both visual and textual modalities and supporting a broader range of real-world applications. To comprehensively evaluate the perception, understanding,
reasoning, and cognition capabilities of LVLMs throughout the entire financial business workflow in Chinese contexts, we introduce CFMME, a novel Chinese financial
multimodal evaluation
benchmark. CFMME comprises 6,052 instances spanning from fundamental academic knowledge to complex real-world applications, covering eight primary financial image modalities and four core
multimodal tasks. On CFMME, we conduct a thorough evaluation of representative LVLMs. The results show that the state-of-the-art model attains an overall accuracy of 66.11% on the question answering task and an average score of 77.18 on the detection, recognition, and information extraction tasks, indicating substantial room for improvement in current LVLMs. In addition, we conduct detailed analyses of error causes, cross-modal capabilities, and multi-orientation settings, yielding valuable insights for future research. We hope that CFMME will spur further progress in LVLMs, especially by improving their performance on multiple
multimodal tasks in the financial domain.
Read the full abstract ▾
All authors Qian Chen, Xianyin Zhang, Yanzhi Liu, Lifan Guo, Feng Chen, Chi Zhang
Computer Vision
FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation
Zekang Zhang, Guangyu Gao, Youyun Tang et al. (+7 more)
LLM-conditioned
segmentation has recently advanced rapidly by coupling large
language models with iterative mask generation frameworks. However, we identify a persistent failure mode in current propose-then-select pipelines. Although high-quality mask candidates are often generated, the final prediction may fail to match the given linguistic condition. This failure arises because
language semantics are typically used as static
prompts or post-hoc matching signals, rather than participating in the iterative mask generation process. Through systematic analysis, we show that many errors stem from semantic misalignment rather than poor mask quality. To address this issue, we propose FlowSeg, which introduces dynamic semantic guidance via a bidirectional semantic flow between intermediate decoding states and LLM-derived condition
embeddings throughout the generation process.
Language conditions actively guide mask refinement at each stage, while condition
embeddings are progressively updated by emerging visual evidence. This design yields semantically grounded mask representations and visually aligned
language conditions, enabling more reliable matching. We further incorporate a lightweight boundary-aware refinement to selectively enhance uncertain regions without perturbing confident interiors. Extensive experiments on referring expression
segmentation and
reasoning segmentation tasks demonstrate that FlowSeg consistently improves
language-mask
alignment and achieves state-of-the-art performance. Project page: https://zkzhang98.github.io/FlowSeg_page
Read the full abstract ▾
All authors Zekang Zhang, Guangyu Gao, Youyun Tang, ChengJing Wu, Xiaochao Qu, Chi Harold Liu, Jianbo Jiao, Yunchao Wei, Luoqi Liu, Ting Liu
Computer Vision
FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation
Zehao Wang, Guanglei Yang, Yihan Zeng et al. (+4 more)
Federated fine-tuning of foundation models with Low-Rank Adaptation (LoRA) provides an efficient solution for reducing communication and computation costs while preserving data locality. However, the direct combination of FedAvg and LoRA suffers from three key issues: limited update space, which restricts the model's effective learning capacity; inter-round state mismatch, which disrupts cross-round local optimization continuity; and a client-agnostic starting state, which slows local convergence on clients. Although recent methods mitigate the limited update space issue by merging LoRA updates into the backbone across communication rounds, inter-round state mismatch and the client-agnostic starting state remain insufficiently addressed. To address these issues, we propose FedSmoothLoRA, a
federated LoRA tuning framework that preserves the enlarged update space, improves cross-round local optimization continuity, and provides a client-aware starting state for local training. At each communication round, FedSmoothLoRA constructs the local LoRA initialization using two
matrices: a Round-Matching
matrix that preserves cross-round local state continuity, and a Gradient-Aligned
matrix that provides client-specific optimization guidance from gradient signals estimated on local data. Together, these designs enable smoother and faster convergence. Extensive experiments on image classification and natural
language generation tasks demonstrate that FedSmoothLoRA consistently outperforms existing
federated LoRA tuning methods. Code: https://github.com/wangzehao0704/FedSmoothLoRA
Read the full abstract ▾
All authors Zehao Wang, Guanglei Yang, Yihan Zeng, Hang Xu, Hongzhi Zhang, Wangmeng Zuo, Chun-Mei Feng
Computer Vision
Uni-RCM: Unified Reference-guided Cross-modal Mapping for Multi-Class Anomaly Detection
Yangchen Wu and Huiqiang Xie
Multi-modal industrial anomaly detection typically relies on separate models for each product category, fundamentally limiting practical scalability. When shifting to a unified paradigm that handles diverse classes simultaneously, detection accuracy often degrades due to inter-class interference and feature
manifold confusion. To overcome these challenges, we propose a Unified Reference guided Cross-modal Mapping framework, named Uni-RCM. At its core, we propose a reference guide block to dynamically filter out category-specific noise by introducing a learnable reference feature, which captures the commonalities across different modalities. Besides, an offline residual quantizer is proposed to characterize the normal distribution by multiple cascaded codebooks. Extensive evaluations on the MVTec-3D AD
dataset demonstrate the state-of-the-art performance in the challenging multi-class setting and in terms of image-level detection and pixel-level localization.
Read the full abstract ▾
Computer Vision
Comparative evaluation of photogrammetric reconstruction methods and 3D Gaussian Splatting for road surface roughness analysis
Marouane Elmegdar and Teng Xiao
Image-based 3D reconstruction offers a low-cost alternative to traditional sensor-based techniques for road surface assessment. This study compares four reconstruction pipelines–COLMAP, Meshroom, Metashape, and 3D Gaussian Splatting (3DGS)–to evaluate their ability to estimate road surface roughness from smartphone imagery. All point clouds were processed in CloudCompare using a consistent workflow involving orientation
alignment,
segmentation, normal estimation, and roughness computation at neighborhood radiuses of 0.2, 0.4, and 0.6 model units. The results show that COLMAP provides the highest sensitivity to micro-texture, while Meshroom yields balanced reconstructions with moderate roughness variation. Metashape produces the smoothest geometry due to its internal filtering, and 3DGS captures visible irregularities but exhibits higher noise and lower density. The comparison demonstrates that open-source pipelines are viable for relative roughness evaluation, offering a practical approach for low-cost pavement monitoring.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions
Jeff A. Bilmes, Gantavya Bhatt and Arnav M. Das
Neural scaling laws appraise data through
dataset size, while the Vendi Score uses
quantum entropy to measure
dataset value. We show both that common neural-scaling-law objectives and the Vendi Score are submodular. We further show that the Vendi Score is a special case of a broader class of submodular objectives that we call
matrix spectral functions. This also includes determinantal (DPP) objectives, as well as many others. We also introduce weakly
matrix monotone functions and show how they lead to weakly submodular
matrix spectral functions, yielding a broad family of practical objectives for data appraisal. We develop secular-equation-based updates that avoid repeated eigendecompositions during greedy optimization, reducing marginal-gain evaluation for $m$-dimensional
embeddings by an $O(m)$ factor relative to oracle queries. This yields an average empirical speedup of about 35,000x, making direct optimization of the Vendi Score feasible on ImageNet-1K-scale
datasets. Thus enabled, we compare how well several objectives predict the value of training subsets for held-out test performance under fixed-size, class-balanced, and fixed training-budget regimes, including the Vendi Score, DPPs, facility location, and three new
matrix spectral variants. Across multiple
datasets, facility location performs the best. Direct optimization also reveals that, while the Vendi Score is predictive over moderate score ranges, pushing the objective to higher values can make it a poor downstream performance proxy. We also find that uniformly at random fixed-size subsets, both unconstrained and class-balanced, are remarkably concentrated in both appraisal scores and held-out performance. Finally, we show that size, class balance, and training budget do not alone determine data value: even when controlling for these factors, performance ranges smoothly from good to bad.
Read the full abstract ▾
Computer Vision
Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
Tianpeng Bu, Xin Liu, Qihua Chen et al. (+7 more)
While GUI
agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval contains $1,216$ executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates $800k$ high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our
dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI
benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a $47.4\%$ success rate and a $33.8\%$ All-Pass@4 score, suggesting that improved long-horizon error recovery ability contributes to both robustness and overall performance. Our code is available at https://github.com/AlibabaResearch/RoTS.
Read the full abstract ▾
All authors Tianpeng Bu, Xin Liu, Qihua Chen, Hao Jiang, Shurui Li, Hongtao Duan, Lu Jiang, Lulu Hu, Bin Yang, Minying Zhang
Computer Vision
One Click per Cell Type Suffices: Training-free Group Interaction for Cell Instance Segmentation
Sanghyun Jo, Seo Jin Lee, Seohyung Hong et al. (+4 more)
Cell instance
segmentation models trained on
cell-specific
datasets suffer severe performance drops on out-of-distribution
cell types, while interactive foundation models overcome this through per-instance prompting at a cost that is prohibitively expensive for histopathology images containing hundreds to thousands of densely packed instances. We introduce Group Prompting, a new paradigm that shifts interactive
segmentation from per-instance $O(N)$ to per-type $O(T)$, where a single click per
cell type suffices to segment all instances of that type. Our key observation is that the frozen image encoder of the Segment Anything Model (SAM) already clusters same-type
cells in its feature space before any
prompt is given, and that this clustering holds across staining modalities without any training. Exploiting this property, we propose Chain-of-
Prompts (CoP), a training-free framework that recursively expands a single user click by (1) identifying reliable same-type locations through non-parametric gating of multi-scale encoder features, and (2) selecting the most spatially distant reliable point as the next
prompt to maximize coverage. On eleven
benchmarks, CoP generalizes to both unseen
cell types and unseen imaging modalities without any adaptation: with one click per type it retains over 90% of per-instance performance on three
cell-type-annotated
datasets while surpassing fully-supervised methods, and with one click per image it retains over 95% on eight
datasets spanning both H&E and non-H&E imaging. Project Page: https://shjo-april.github.io/Chain-of-
Prompts/
Read the full abstract ▾
All authors Sanghyun Jo, Seo Jin Lee, Seohyung Hong, Yoorim Gang, Hyeongsub Kim, Hyungseok Seo, Kyungsu Kim