Computer Vision
From Baseline to Follow-Up: Counterfactual Spine DXA Image Synthesis in UK Biobank Using a Causal Hierarchical Variational Autoencoder
Yilin Zhang, Nicholas C. Harvey, Nicholas R. Fuggle et al. (+1 more)
Dual-energy X-ray absorptiometry (DXA) is widely used for large-scale skeletal assessment, yet learning controllable and interpretable factor-specific anatomical variation remains challenging. We propose a metadata-conditioned causal hierarchical variational autoencoder (CHVAE) for causally consistent generation of anteroposterior (AP) spine DXA images from the UK Biobank (UKB). The model is trained on 3,743 raw AP spine scans from the first imaging visit and conditioned on basic participant attributes and lumbar morphometry. Causal consistency is evaluated in a baseline-to-follow-up setting using abduction–action–prediction (AAP): latent variables are abducted from baseline images, age is intervened to the repeat-imaging value, and the resulting counterfactual follow-up morphometry is compared with observed repeat-imaging measurements. Results show strong absolute-level agreement for key vertebral morphometry variables under age intervention, supporting intervention-aligned synthesis of anatomically plausible DXA images.
Read the full abstract ▾
All authors Yilin Zhang, Nicholas C. Harvey, Nicholas R. Fuggle, Rahman Attar
Computer Vision · Cross-listed from cs.LG
The Double Dilemma in Multi-Task Radiology Report Generation: A Gradient Dynamics Analysis and Solution
Erjian Zhang, Yatong Hao, Liejun Wang et al. (+1 more)
While multi-task learning based automatic radiology report generation (RRG) is widely adopted to ensure clinical consistency, most focus on architectural designs yet remain limited to coarse linear scalarization strategies. These strategies cannot effectively balance the hard constraints of discriminative clinical supervision with the smoothness requirements of report generation. To address these problems, we analyze the failure mechanism of linear scalarization from the perspective of gradient dynamics, utilizing the
stochastic differential equation (SDE) framework to characterize it as a "Double Dilemma" of drift term deviation and
diffusion term decay. Based on this, we propose a backbone-agnostic optimizer named Conflict-Averse Magnitude-Enhanced Gradient Descent (CAME-Grad). Through conflict-averse direction rectification and magnitude-enhanced energy injection, the
algorithm not only ensures
geometric validity, but also avoids local optimal solutions. Then, the adaptive gradient fusion mechanism is used to establish a dynamic balance between the theoretical optimal direction and the task-specific inductive bias. Experiments show that as a universal plug-and-play optimizer, CAME-Grad brings substantial and consistent improvements across eight diverse RRG methods, elevating overall clinical efficacy performance by an average of 2.3% on MIMIC-CXR and 1.9% on IU X-Ray. Our code is available at https://github.com/vpsg-research/CAME-Grad.
Read the full abstract ▾
All authors Erjian Zhang, Yatong Hao, Liejun Wang, Zhiqing Guo
Computer Vision
AtomicMotion: Learning Human Motion From Different Human Parts
Runzhen Liu, Chuhua Xian and Fa-Ting Hong
Accurately reconstructing full-body poses from sparse head and hand trajectories is a foundational challenge for immersive AR/VR telepresence. Current methods often struggle with error accumulation and unnatural joint coordination, primarily because they treat the human body as a monolithic entity, thereby failing to capture the fine-grained “atomic intents” embedded in subtle signal variations and overlooking the inherent structural
topology. To bridge this gap, we present AtomicMotion, a framework designed to decouple and re-integrate body dynamics through three core innovations. First, we introduce a logical body partitioning scheme that decomposes the skeleton into five distinct clusters based on functional intent; this ensures that each partition preserves internal joint synergies while isolating local motion primitives. Second, to robustly map sparse inputs to high-dimensional poses, we employ a masked full-body pre-conditioning strategy during training, forcing the model to internalize global skeletal
topology and latent kinematic constraints. Finally, addressing the limitations of vanilla spatial
attention, which often ignores fixed physiological connectivity, we propose Kinematic
Attention. By
embedding the classical kinematic tree structure into the
attention mechanism, we ensure biological plausibility in the synthesized motions. Extensive evaluations on the AMASS
dataset demonstrate that AtomicMotion significantly outperforms existing baselines, yielding higher reconstruction fidelity and superior biomechanical realism.
Read the full abstract ▾
Computer Vision
H-Flow: Self-supervised Human Scene Flow via Physics-inspired Joint Multi-modal Learning
Zhanbo Huang, Xiaoming Liu and Yu Kong
Parametric human models capture global pose but cannot represent the non-rigid surface dynamics of clothing and soft tissue. Generic scene flow estimates dense motion but breaks down on articulated bodies, where pixel-level supervision is also intractable to acquire. We introduce H-Flow, a dense human scene flow that captures both skeletal kinematics and surface deformation. A unified multi-head
transformer estimates flow from
monocular video, jointly predicting pose and depth as companion outputs. The challenge lies in the lack of supervision. In place of unattainable labels, we anchor the network in the physics of human motion, encoding
geometric, structural, and biomechanical priors as cross-modal training objectives. We further introduce DynAct4D, a high-fidelity synthetic
benchmark providing dense flow annotations across diverse subjects, garments, and motions. On standard
benchmarks, H-Flow outperforms scene-flow and parametric baselines, and generalizes zero-shot to in-the-wild
video. Code, models, and the DynAct4D
benchmark will be released upon publication
Read the full abstract ▾
Computer Vision
GLeVE: Graph-Guided Lesion Grounding with Proposal Verification in 3D CT
Shuo Jiang, Yuhao Hong, Chunbo Jiang et al. (+9 more)
Grounding radiology report descriptions to 3D CT volumes is essential for verifiable clinical interpretation, yet remains challenging due to the semantic-spatial gap between free-text narratives and volumetric anatomy. Existing report-assisted and
vision-
language grounding methods typically rely on phrase-level
alignment or dense pixel supervision, resulting in limited lesion-wise correspondence and suboptimal localization accuracy. We propose GLeVE, a graph-guided lesion grounding framework with anatomical prior verification and octree-based autoregressive refinement. GLeVE treats each lesion description as an atomic semantic unit and encodes organ attribution, attributes, and inter-lesion relations through relation-aware graph
reasoning to produce discriminative lesion-wise queries. Anatomy-aware proposal generation with region-level verification enforces one-to-one text-lesion
alignment, while hierarchical octree refinement progressively improves boundary delineation. Experiments on AbdomenAtlas 3.0 demonstrate consistent gains over classical
multimodal foundation models and report-supervised baselines in both
segmentation accuracy and lesion-level localization.
Read the full abstract ▾
All authors Shuo Jiang, Yuhao Hong, Chunbo Jiang, Weihong Chen, Huangwei Chen, Shenghao Zhu, Beining Wu, Mingxuan Liu, Zhu Zhu, Feiwei Qin, Min Tan, Yifei Chen
Computer Vision
Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following
Shijing Wang, Yaping Huang, Chaoqun Cui et al. (+4 more)
Gaze following requires both scene understanding and gaze
reasoning to localize the gaze target of an in-scene person. Recently,
vision foundation models (VFMs) have demonstrated strong performance on this task, enabling simpler architectures while outperforming prior methods. However, we observe a key limitation of VFM-based approaches: while VFMs substantially improve scene understanding, they contribute little to gaze
reasoning. As a result, existing methods often rely on semantically salient objects rather than true gaze cues, leading to degraded performance when targets are not salient. To address this, we propose a novel training mechanism to enhance gaze
reasoning in VFMs for gaze following. Our method includes: (1) a head-conditioned local LoRA, which enables localized adaptation to preserve scene
token learning while improving head
token learning for gaze
reasoning; and (2) an out-of-cone penalty, which injects gaze cues into head
tokens while aligning them with scene
tokens. Experiments on the GazeFollow and VAT
datasets demonstrate that our method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient. Our findings offer valuable insights for advancing future gaze following research. Code will be available at https://github.com/shijing-wang/HCloRA.
Read the full abstract ▾
All authors Shijing Wang, Yaping Huang, Chaoqun Cui, David Wong, Yihua Cheng, Alexandros Neophytou, Hyung Jin Chang
Computer Vision · Cross-listed from cs.RO
Decoupling Ego-Motion from Target Dynamics via Dual-Interval Motion Cues for UAV Detection
Liuyang Wang and Feitian Zhang
Object detection from Unmanned Aerial Vehicles (
UAVs) is challenged by severe ego-motion, camera jitter, and large scale variations. While modern detectors perform well on static images, their direct application to
UAV video often fails, particularly for small objects in dynamic scenes. Existing motion-based methods either rely on computationally expensive optical flow or use single-interval differencing, which is sensitive to jitter and limited in capturing diverse motion patterns. We propose a
vision-only motion-guided detection framework that decouples target motion from camera-induced disturbances. A homography-based Global Motion Compensation (GMC) first aligns adjacent frames. We then introduce a Dual-Interval Motion Extraction strategy that captures both short-term and long-term motion cues. To integrate these cues, a lightweight Motion-Guided
Attention (MGA) module enhances feature representations within a Feature Pyramid Network. Experiments on the VisDrone-VID
dataset demonstrate consistent improvements over a strong YOLOv8 baseline under severe ego-motion. Ablation studies further confirm the effectiveness of the dual-interval design and the proposed motion-guided
attention mechanism.
Read the full abstract ▾
Computer Vision
Rethinking Noise-Robust Training for Frozen Vision Foundation Models: A Cross-Dataset Benchmark with a Case Study of Small-Loss Failure
Zitong Li and Haoyu Wang
Frozen
Vision Foundation Models (VFMs) with lightweight classification heads are increasingly used in medical imaging because they offer efficient and reproducible deployment. Yet noisy-label learning methods for this frozen-feature regime remain poorly understood, and most existing methods still rely on a small-loss assumption inherited from end-to-end training. We present a controlled
benchmark of eight noisy-label methods across five medical
datasets, three backbones, two noise types, and five noise rates (150 conditions, 6,000 training runs), evaluated with balanced accuracy. The
benchmark shows that there is no universal winner: Friedman ranking over the 150 conditions yields $\chi^2 = 333.2$ ($p = 4.77 \times 10^{-68}$), ELR wins the most conditions (49/150), while CUFIT attains the best mean rank (2.51). The practical cost of method choice grows sharply with noise severity, from 4.5pp on clean data to 18.8pp at asymmetric 40% noise. To explain these
benchmark-level patterns, we revisit the small-loss assumption in a representative high-risk regime. Under frozen DINOv2 features, clean and noisy loss distributions overlap by 53–61%, and matched-rate clean-sample detection shows that prediction agreement is markedly more stable than loss ranking under asymmetric noise (3pp vs.\ 13pp precision drop). On ISIC2019 with asymmetric 40% noise, Co-Teaching reaches 68% overall accuracy while collapsing to 35.1% balanced accuracy with zero recall on three minority classes. Together, these results recast noisy-label learning for frozen VFMs as a regime-aware method-selection problem rather than a search for a single dominant
algorithm. We conclude with evidence-based guidance and a low-regret feature-space selector for practical recommendation.
Read the full abstract ▾
Computer Vision
SceneAligner: 3D-Grounded Floorplan Localization in the Wild
Junhyeong Cho, Ruojin Cai and Hadar Averbuch-Elor
Many public buildings provide floorplans with a "you are here" indicator to help visitors orient themselves. Floorplan localization seeks to computationally replicate this capability by determining where visual observations were captured within a floorplan. However, existing methods typically assume controlled small-scale environments and precise vectorized floorplans, limiting their ability to operate in large-scale buildings and rasterized floorplans. In this work, we present an approach for performing floorplan localization in the wild by grounding the task in a reconstructed 3D representation of the scene. Given an unconstrained image collection, our method reconstructs a
gravity-aligned 3D scene and projects it into a 2D density map that serves as a floorplan proxy. Floorplan localization is then formulated as aligning this proxy with the input floorplan via a 2D similarity transform. To bridge the appearance gap between density maps and architectural floorplans, we adapt a 2D foundation model to learn cross-modal correspondences, introducing a fine-tuning scheme that encourages semantically aligned matches while preserving structural consistency. Extensive experiments demonstrate substantial improvements over prior methods, including in extremely sparse settings with as little as a single input image. Our code and data will be publicly available.
Read the full abstract ▾
Computer Vision
Beyond Chamfer Distance: Granular Order-aware Evaluation Metric For Online Mapping
Chouaib Bencheikh Lehocine, Adam Lilja, Junsheng Fu et al. (+1 more)
Online map estimation is a crucial component of autonomous driving systems that reduces the reliance on costly high-definition maps. State-of-the-art (SOTA) methods commonly predict map elements as ordered sequences of points that form polylines and polygons. The evaluation of these methods relies predominantly on mean average precision (mAP) based on thresholded Chamfer distance (CD). This framework lacks sensitivity to point ordering and provides limited granularity in assessing
geometric quality, making it difficult to distinguish which methods truly excel over others. In this work, we address these limitations on two fronts. For the single-instance similarity measure, we introduce sequence optimal sub-pattern assignment (SOSPA), an order-aware metric that enables fine-grained evaluation of individual geometries while satisfying all metric axioms. For the multi-instance evaluation framework, we propose polyline localisation and detection (PLD), a soft metric that jointly captures detection quality and
geometric accuracy, replacing the hard thresholding of mAP with a principled soft assignment. Through evaluations on nuScenes, we demonstrate that PLD effectively ranks SOTA online mapping methods (MapTRv2, StreamMapNet, MapTracker) while providing a decomposed error analysis. This analysis identifies detection capability as the dominant bottleneck in current methods, revealing a performance trend that mAP fails to capture. Code for evaluation using our metrics will be released.
Read the full abstract ▾
All authors Chouaib Bencheikh Lehocine, Adam Lilja, Junsheng Fu, Lars Hammarstrand
Computer Vision
SegGuidedNet: Sub-Region-Aware Attention Supervision for Interpretable Brain Tumor Segmentation
Hasaan Maqsood, Saif Ur Rehman Khan, Sebastian Vollmer et al. (+2 more)
Accurate
segmentation of
brain tumour sub-regions from multi-parametric MRI is critical for treatment planning yet remains challenging due to morphological variability, class imbalance, and overlapping appearances of tumour regions across imaging sequences. We propose SegGuidedNet, a three-dimensional residual encoder–decoder network introducing a novel SegAttentionGate module that explicitly supervises the decoder to produce spatially discriminative
attention maps for each tumour sub-region necrotic core, peritumoral oedema, and enhancing tumour via a lightweight auxiliary loss, adding less than 0.2% parameter overhead. This sub-region supervision maintains decoder discriminability between visually ambiguous classes while providing free-of-cost spatial interpretability at inference without any post-hoc explanation method. Evaluated independently on BraTS2021 and BraTS2023 GLI across 251 held-out subjects each, SegGuidedNet achieves mean Dice of 0.905 (ET= 0.873, TC=0.906, WT=0.935) and 0.897 (ET=0.859, TC=0.902, WT=0.931) respectively, surpassing ensemble-based nnU-Net and HNF-Netv2 as a single model and approaching Swin UNETR a 10-model ensemble within 2–4 Dice points at a fraction of the inference cost. The consistency of results across two
benchmark editions further confirms the generalisability of the proposed approach, offering competitive accuracy with built-in interpretability in a lightweight, clinically practical framework.
Read the full abstract ▾
All authors Hasaan Maqsood, Saif Ur Rehman Khan, Sebastian Vollmer, Andreas Dengel, Muhammad Nabeel Asim
Computer Vision
VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis
Jinho Park, Youbin Kim, Hogun Park et al. (+1 more)
Spatio-temporal
reasoning is a core capability for
Multimodal Large
Language Models (MLLMs) operating in the real world. As such, evaluating it precisely has become an essential challenge. However, existing spatio-temporal
reasoning benchmark datasets primarily rely on static image sets or passively curated
video data, which limits the evaluation of fine-grained
reasoning capabilities. In this paper, we introduce VGenST-Bench, a
video benchmark that employs generative models to actively synthesize highly controlled and diverse evaluation scenarios. To construct VGenST-Bench, we propose a multi-
agent pipeline incorporating a human quality control stage, ensuring the quality of all generated
videos and QA pairs. We establish a comprehensive 3x2x2
video taxonomy, encompassing Spatial Scale, Perspective, and Scene Dynamics to span diverse scenarios. Furthermore, we design a hierarchical task suite that decouples low-level visual perception from high-level spatio-temporal
reasoning. By shifting the paradigm from passive curation to active synthesis, VGenST-Bench enables fine-grained diagnosis of spatio-temporal understanding in MLLMs.
Read the full abstract ▾
All authors Jinho Park, Youbin Kim, Hogun Park, Eunbyung Park
Computer Vision
Cell Phantom Video Generation in Elliptical Fourier Descriptor Domain
Francesco Benedetto, Roberto Basla, Luca Magri et al. (+1 more)
Training Deep Neural Networks for tracking individual
cells in biomedical
videos requires a large amount of annotated data. The annotation of
videos for
cell tracking is very time consuming and often requires domain expertise; this explains the limited availability of public annotated data to address important medical problems like tissue repair or cancer treatment. Generating synthetic
videos along with their Ground Truth annotations is a promising solution that relies, as a foundational first step, on the synthesis of single
cell annotations (or phantoms). Phantoms need to be time consistent, as they have to replicate biological processes that are specific to the
cell types. In this work, we propose a novel framework for generating
videos of
cell phantoms in the Elliptical
Fourier Descriptors (EFDs) domain, a compact and geometrically interpretable representation for 2D closed contours. We represent the
cell phantom
evolution as a multivariate time series of EFD coefficients, introducing a strong prior for
cell morphology and enabling the efficient generation of sequences that evolve coherently in time. Our experimental validation proves that modelling the temporal
evolution in EFD space enables the generation of biologically plausible phantom
videos. Our method can be used in generative pipelines for synthesizing annotated data for
cell tracking, thus strongly mitigating the annotation effort for creating new
datasets. Our code is available for download here: https://github.com/FrancescoBenedetto99/efd-
cell-
video-gen.
Read the full abstract ▾
All authors Francesco Benedetto, Roberto Basla, Luca Magri, Giacomo Boracchi
Computer Vision
GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning
Deshui Miao, Xingsen Huang, Yameng Gu et al. (+3 more)
Spatio-temporal
reasoning in
vision-
language models requires visual representations that preserve physical geometry rather than merely semantic appearance. Recent
multimodal models incorporate
geometric information through structural branches, 3D-aware supervision,
reasoning-stage fusion, or long-horizon
memory. While these approaches demonstrate the importance of geometry for spatial intelligence, they typically treat
geometric cues as a shared signal across all visual
tokens. We note that this overlooks a finer-grained challenge: different visual
tokens require different
geometric evidence depending on their spatial roles. To address this limitation, we introduce GeoWeaver, a pre-
reasoning geometric grounding framework that treats geometry as a representational prerequisite for spatio-temporal
reasoning. GeoWeaver constructs a multi-level geometry bank from a frozen geometry encoder and performs
token-adaptive
geometric evidence allocation, enabling each visual
token to retrieve the most relevant
geometric abstractions. The selected evidence is incorporated into visual
tokens via a residual grounding operation prior to
language modeling, yielding geometry-grounded representations for downstream
reasoning. Extensive evaluations on spatial
reasoning benchmarks demonstrate that GeoWeaver consistently enhances geometry-aware
reasoning while retaining general
multimodal capabilities. This indicates that
geometric information yields the greatest benefit not as a late-fusion auxiliary signal but as a fundamental prerequisite that shapes the representational foundation on which large
language models perform
reasoning. All source code and models will be released at https://github.com/yahooo-m/GeoWeaver .
Read the full abstract ▾
All authors Deshui Miao, Xingsen Huang, Yameng Gu, Xin Li, Haijun Zhang, Ming-Hsuan Yang
Computer Vision
FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning
Haokun Wen, Xuemeng Song, Xinghao Xie et al. (+3 more)
Fashion image
retrieval is a cornerstone of modern e-commerce systems. A unified framework that supports diverse query formats and search intentions is highly desired in practice. However, existing approaches focus on narrow
retrieval tasks and do not fully capture such diversity. Therefore, in this work, we aim to develop a unified framework capable of handling diverse realistic fashion
retrieval scenarios, achieving truly versatile fashion image
retrieval. To establish a data foundation, we first introduce U-FIRE, a comprehensive
benchmark that consolidates fragmented fashion
datasets into a unified collection, supplemented by two manually curated
datasets for testing generalization. Building upon this, we propose FashionLens, a unified framework based on
Multimodal Large
Language Models. To handle divergent matching objectives, we design a Proposal-Guided Spherical Query Calibrator that dynamically shifts query representations into task-aligned metric spaces via adaptive spherical linear interpolation. Additionally, to mitigate the optimization imbalance caused by varying task complexities and data scales, we develop a Gradient-Guided Adaptive Sampling strategy that automatically re-weights tasks based on realtime learning difficulty and the data scale prior. Experiments on U-FIRE show that FashionLens achieves state-of-the-art performance across diverse
retrieval scenarios and generalizes robustly to unseen tasks. The data and code are publicly released at https://github.com/haokunwen/FashionLens.
Read the full abstract ▾
All authors Haokun Wen, Xuemeng Song, Xinghao Xie, Xiaolin Chen, Xiangyu Zhao, Weili Guan
Computer Vision
MOTOR: A Multimodal Dataset for Two-Wheeler Rider Behavior Understanding
Varun A. Paturkar, Shankar Gangisetty and C.V. Jawahar
Two-wheelers account for a disproportionately high share of road fatalities in the Global South. Research on two-wheeler rider behavior, however, lags far behind four-wheelers, where
multimodal datasets have driven major advances in Advanced Driver Assistance Systems (ADAS). To address this gap, we present the MOtorized TwO-wheeler Rider (MOTOR)
dataset, the first large-scale, multi-view,
multimodal resource dedicated to two-wheelers in dense, unstructured traffic. MOTOR comprises 1,629 sequences (25+ hours of
video data) collected from 16 riders and integrates synchronized front, rear, and helmet
videos, rider eye-gaze from wearable trackers, on-road audio, and telemetry (GPS, accelerometer, gyroscope). Rich annotations capture traffic context, rider state, 12 riding maneuvers spanning conventional and unconventional behaviors, and legality labels (Legal, Illegal, Unspecified). We
benchmark rider behavior recognition and maneuver legality classification using state-of-the-art
video action recognition backbones (CNN and
Transformer-based), extended with
multimodal fusion, and find that combining RGB, gaze, and telemetry consistently yields the best performance. MOTOR thus provides a unique foundation for advancing safety-critical understanding of two-wheeler riding. It offers the research community a
benchmark to develop and evaluate models for behavior analysis, legality-aware prediction, and intelligent transportation systems.
Dataset and code is available at https: //varuniiith.github.io/MOTOR-
Dataset/
Read the full abstract ▾
Computer Vision
MKG-CARE: Case-Aware Reasoning with Multimodal Knowledge Graphs for Explainable Medical Image Diagnosis
Yiming Xu, Yixuan Liu, Yuhang Zhang et al. (+3 more)
Medical image diagnosis has achieved significant progress with deep learning, yet existing methods often rely on isolated visual evidence and lack the ability to effectively leverage similar cases and external knowledge. In clinical practice, diagnosis is typically supported by similar historical cases and their associated symptoms. To explicitly model this evidence-based diagnostic process, we propose MKG-CARE, a framework that performs case-aware
reasoning using
multimodal knowledge graphs for explainable medical image diagnosis. Specifically, we construct a case-aware
multimodal knowledge graph as a structured diagnostic
memory, where diseases, images, and symptoms are hierarchically organized. Given an input image, MKG-CARE adaptively retrieves similar cases from this
memory and extracts their corresponding case-centered subgraphs. We further introduce a knowledge propagation and injection mechanism, where an image-centric Graph
Attention Network aggregates heterogeneous semantics within the retrieved case subgraphs, followed by bidirectional cross-modal
attention to align and inject the aggregated case knowledge into visual representations. To mitigate
retrieval noise, we design a confidence-calibrated decision refinement scheme that estimates each retrieved case's reliability from prediction confidence and sample similarity, and reweights its contribution to the final prediction for interpretable case-level evidence attribution. Extensive experiments on multiple medical imaging
datasets demonstrate consistent improvements over strong baselines, while ablation and qualitative analyses validate the effectiveness and interpretability of our method. The code is available at https://github.com/lyxuan1022/MKG-CARE.
Read the full abstract ▾
All authors Yiming Xu, Yixuan Liu, Yuhang Zhang, Ling Zheng, Yihan Wang, Qi Song
Computer Vision
Segment Anything with Motion, Geometry, and Semantic Adaptation for Complex Nonlinear Visual Object Tracking
Deyi Zhu, Yuji Wang, Yong Liu et al. (+4 more)
Traditional visual object tracking (VOT) methods typically rely on task-specific supervised training, limiting their generalization to unseen objects and challenging scenarios with distractors, occlusion, and nonlinear motion. Recent
vision foundation models, exemplified by SAM 2, learn strong
video understanding priors from large-scale pretraining and offer a promising foundation for building more robust and generalizable trackers. However, directly applying SAM 2 to VOT remains suboptimal, as it does not explicitly model target motion dynamics or enforce
geometric and semantic consistency across frames, both of which are essential for reliable tracking. To address this issue, we propose SAMOSA, a new tracking framework that adapts SAM 2 to complex VOT scenarios by explicitly leveraging motion, geometry, and semantic cues. Specifically, we introduce a lightweight nonlinear motion predictor to model target dynamics and guide mask selection as well as
memory filtering. We further exploit semantic cues to detect target shifts and recover from tracking failures, while
geometric cues are incorporated as structural constraints to improve tracking stability. In this way, SAMOSA bridges the gap between the implicit
video understanding prior of SAM 2 and explicit tracking-oriented modeling. Extensive experiments show that SAMOSA consistently outperforms state-of-the-art SAM 2–based approaches on general
benchmarks, demonstrates stronger generalization than supervised VOT methods, and achieves substantial gains on anti-
UAV datasets, which typify complex nonlinear motion scenarios. Our code is available at https://github.com/DurYi/SAMOSA.
Read the full abstract ▾
All authors Deyi Zhu, Yuji Wang, Yong Liu, Yansong Tang, Bingyao Yu, Jiwen Lu, Jie Zhou
Computer Vision
SpaceDG: Benchmarking Spatial Intelligence under Visual Degradation
Xiaolong Zhou, Yifei Liu, Ziyang Gong et al. (+8 more)
Multimodal Large
Language Models (MLLMs) have made rapid progress in spatial intelligence, yet existing spatial
reasoning benchmarks largely assume pristine visual inputs and overlook the degradations that commonly occur in real-world deployment, such as motion blur, low light, adverse weather, lens distortion, and compression artifacts. This raises a fundamental question: how robust is the spatial intelligence of current MLLMs when visual observations are imperfect? To answer this question, we introduce SpaceDG, the first large-scale
dataset for degradation-aware spatial understanding. It is constructed with a physically grounded degradation synthesis engine that embeds degradation formation process into 3D Gaussian Splatting (3DGS) rendering, enabling realistic simulation of nine degradation types. The resulting
dataset contains approximately 1M QA pairs from nearly 1,000 indoor scenes. We further introduce SpaceDG-Bench, an human-verified
benchmark with 1,102 questions spanning 11
reasoning categories and 9 visual degradation types, yielding over 10K VQA instances. Evaluating 25 open- and closed-source MLLMs reveals that visual degradations consistently and substantially impair spatial
reasoning, exposing a critical robustness gap. Finally, we show that finetuning on SpaceDG markedly improves degradation robustness and can even surpass human performance under degraded conditions without any performance drop on clean images, highlighting the promise of degradation-aware training for robust spatial intelligence.
Read the full abstract ▾
All authors Xiaolong Zhou, Yifei Liu, Ziyang Gong, Jiarui Li, Qiyue Zhao, Muyao Niu, Yuanyuan Gao, Le Ma, Xue Yang, Hongjie Zhang, Zhihang Zhong
Computer Vision · Cross-listed from cs.AI
LACO: Adaptive Latent Communication for Collaborative Driving
Tianhao Chen, Yuheng Wu and Dongman Lee
Collaborative driving aims to improve safety and efficiency by enabling connected vehicles to coordinate under partial observability. Recent approaches have evolved from sharing visual features for perception to exchanging
language-based
reasoning through foundation models for behavioral coordination. Though communicating in
language provides intuitive information, it introduces two challenges: high latency caused by autoregressive decoding and information loss caused by compressing rich internal representations into discrete
tokens. To address these challenges, we analyze latent communication in collaborative driving under inherent limitations of multi-
agent settings. Our analysis reveals
agent identity confusion, where direct fusion of latent states entangles decision representations across vehicles. Motivated by this, we propose LACO, a training-free LAtent COmmunication paradigm that seamlessly adapts pretrained driving models to collaborative settings. LACO introduces Iterative Latent Deliberation (ILD) for latent
reasoning, Cross-Horizon Saliency Attribution (CHSA) for communication-efficient information selection, and Structured Semantic Knowledge
Distillation (SSKD) to stabilize ego-centric decision making. Closed-loop experiments in CARLA show that LACO notably reduces communication and inference latency while maintaining strong collaborative driving performance.
Read the full abstract ▾
Computer Vision
Training-Free Fine-Grained Semantic Segmentations in Low Data Regimes: A FungiTastic Baseline
Sebastian Cavada, Francesco Pelosin and Lapo Faggi
Fine-grained semantic
segmentation requires both precise localization and discrimination between visually similar classes. In FungiTastic, this problem is further complicated by a long-tailed distribution and strong variation in image acquisition conditions. We propose a training-free two-stage framework that decouples
segmentation from classification. SAM3 first produces class-agnostic mushroom masks using macro-taxonomic
prompts, and DINOv3 then assigns fine-grained labels through prototype matching in the
embedding space. To improve this stage, we apply a simple transformation of the DINOv3 feature space that improves prototype-based classification. Compared with class-specific prompting, our approach is more scalable and keeps the
segmentation cost low. We report results from one-shot to few-hundred-shot regimes, providing, to the best of our knowledge, the first baseline for fine-grained semantic
segmentation in low-data settings.
Read the full abstract ▾
Computer Vision
Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
David Méndez, Roberto Confalonieri and Natalia Díaz Rodríguez
Vision-
Language Models (VLMs) excel at tasks like zero-shot classification and cross-modal
retrieval by mapping images and text to a shared space, but this requires expensive end-to-end training with massive paired
datasets. Current post-hoc
alignment methods reduce computational costs by connecting pretrained encoders through lightweight mappings, yet still demand substantial paired data. In this work, we investigate the potential of repurposing the classification heads of pretrained
vision models as semantic prototypes. The recycling of these weights, typically discarded after pretraining, unlocks two distinct capabilities: it enables zero-shot
alignment by using weights as semantic anchors, and serves as a robust data augmentation strategy by mixing these prototypes with real image-text pairs. We demonstrate that integrating our approach with several state-of-the-art post-hoc
alignment techniques consistently boosts accuracy in cross-modal
retrieval, zero- and few-shot classification tasks.
Read the full abstract ▾
Computer Vision
DeliCIR: Memory-Guided Test-Time Deliberation via Multi-Agent Collaboration for Composed Image Retrieval
Xingtian Pei, Yukun Song, Changwei Wang et al. (+4 more)
Composed Image
Retrieval (CIR) requires both preserving the visual continuity of the reference image and faithfully executing the semantic variables specified in the modification text, which constitute the core challenge of the task. Existing methods often suffer from Perception Myopia in a single space, or fall into Logic Drift in iterative collaboration due to the perception ceiling of the underlying retriever. To address this issue, we propose a one-stop hierarchical Perception-to-Deliberation Framework (PDF), which, to the best of our knowledge, is the first to introduce Test-Time Scaling (TTS) into CIR. Relying on a hierarchical multi-
agent architecture, PDF first utilizes an Intent Routing Manager to dynamically dispatch multi-view Worker perception signals based on modification intents to construct a high-recall candidate pool. Subsequently, the Deliberation Manager combines a training-free
reasoning experience
distillation mechanism with a tournament-style test-time scaling strategy (T-TTS) to perform experience-guided fine-grained
reasoning and produce the final
retrieval results. Experimental results demonstrate that PDF achieves SOTA performance on three
benchmark datasets: CIRR, CIRCO, and Fashion IQ. This study indicates that experience-driven
memory mechanism and TTS represent a highly promising and scalable path for achieving fine-grained multimedia
retrieval. The code will be made publicly available upon acceptance.
Read the full abstract ▾
All authors Xingtian Pei, Yukun Song, Changwei Wang, Shunpeng Chen, Rongtao Xu, Shengpeng Xu, Shibiao Xu
Computer Vision
MaSC: A Masked Similarity Metric for Evaluating Concept-Driven Generation
Patryk Bartkowiak, Lennart Petersen, Bartosz Kotrys et al. (+3 more)
Evaluating single-concept personalization in text-to-image
diffusion requires measuring both concept preservation, which captures identity fidelity to a reference, and
prompt following, which captures whether the generated scene matches the
prompt. Existing metrics commonly compute these signals using global image or text-image
embeddings, such as CLIP-I, DINO, and CLIP-T. We show that such metrics correlate poorly with human perception because they attend to the image as a whole instead of separating the concept subject from the background. We introduce MaSC, a masked similarity metric that uses externally provided foreground concept masks to decompose evaluation into subject-specific concept preservation and background-based
prompt following. MaSC computes both scores from frozen SigLIP2 SO400M-NaFlex features: concept preservation is measured by masked max-cosine matching between foreground reference patches and generated-image patches, while
prompt following is measured by comparing a background-only pooled image
embedding to a subject-stripped
prompt embedding. On DreamBench++ human ratings, MaSC achieves Krippendorff alpha = 0.471 for concept preservation, outperforming all tested non-LLM baselines and GPT-4V, and approaching GPT-4o. On ORIDa, a real-photo identity-preservation
benchmark across physical environments, MaSC achieves AUC = 0.992, nearly perfectly distinguishing same-subject from cross-subject pairs. Its
prompt-following score also outperforms the CLIP-T baseline shipped with DreamBench++. These results show that spatially decomposed aggregation is a strong design principle for evaluating concept-driven generation.
Read the full abstract ▾
All authors Patryk Bartkowiak, Lennart Petersen, Bartosz Kotrys, Dominik Michels, Soren Pirk, Wojtek Palubicki
Computer Vision
Auditing Metrics for Synthetic Data Utility: A Cross-Domain Study of Appearance and Geometry
Patryk Bartkowiak, Dominik Michels, Soren Pirk et al. (+1 more)
We propose SADGE, a quantitative similarity metric that predicts the performance of synthetic image
datasets for common computer
vision tasks without downstream model training. Estimating whether a synthetic
dataset will lead to a model that performs well on real-world data remains a bottleneck in model development. Existing evaluation metrics (e.g., PSNR, FID, CLIP) primarily measure semantic
alignment between real and synthetic images (Appearance Similarity Score). Less commonly, structural similarity between images is considered to assess the domain gap (
Geometric Similarity Score). However, to the best of our knowledge there exists no studies that evaluate which similarity metric is the best downstream predictor for a given synthetic
dataset. In this paper, we show over a wide variety of different synthetic
datasets and downstream tasks that neither appearance nor geometry alone can reliably predict downstream performance; rather, it is their non-linear interplay that dictates synthetic data utility. Specifically, we measure how commonly used Appearance and
Geometric Similarity metrics computed between synthetic and real images correlate with downstream performance in object detection, semantic
segmentation, and pose estimation. Across five public synthetic-to-real
benchmark families and 15
dataset-level variants (79k image pairs), SADGE achieves the strongest association with downstream transfer performance under both linear and rank-based criteria, reaching Pearson r=0.88 and Spearman rho=0.77. We compute for each combination of geometry-based methods and appearance-based approaches SADGE scores across all
benchmark families. The best configuration is obtained by fusing DINOv3 appearance similarity with MASt3R
geometric consistency through a constrained bilinear interaction, outperforming both the strongest geometry-only baseline and the strongest appearance-only baseline .
Read the full abstract ▾
All authors Patryk Bartkowiak, Dominik Michels, Soren Pirk, Wojtek Palubicki
Computer Vision
Making the Discrete Continuous: Synthetic RAW Augmentations for Fine-Grained Evaluation of Person Detection Performance in Low Light
Valeria Pais, Malena Mendilaharzu, Daniele Faccio et al. (+3 more)
Real-world deployment of AI
vision models is both fueled and limited by the data available for training and testing. Real
datasets are sparse and uneven: long-tailed or unbalanced distributions hinder generalization, and the low number of samples in low density regions makes it hard to run evaluations. Synthetic data can fill these gaps, providing us with a way to sample the input space more continuously and improve data coverage for
benchmarks. Focusing on the autonomous driving safety-critical case of pedestrian detection in the dark, we show how synthetic low-light samples can be used to better characterize the performance of a state-of-the-art object detection model as a function of the scene illumination. We use a synthetic RAW image augmentation technique to generate low-light samples that match the noise model of the camera sensor. Performance metrics on real and synthetic low-light data are similar, indicating that the AI model finds it hard to distinguish between them.
Read the full abstract ▾
All authors Valeria Pais, Malena Mendilaharzu, Daniele Faccio, Luis Oala, Christoph Clausen, Bruno Sanguinetti
Computer Vision
Pre-VLA: Preemptive Runtime Verification for Reliable Vision-Language-Action and World-Model Rollouts
Zhen Sun, Yongjian Guo, Haoran Sun et al. (+6 more)
While large
vision-
language-action (VLA) models and generative world models (WM) have advanced long-horizon
embodied intelligence, their practical deployment remains challenged by uncertainty in learning-based action generation. Low-quality actions may cause physical failures during execution or lead to misleading world-model rollouts with redundant rendering costs. To address this issue, we propose Pre-VLA, a unified runtime verification architecture that performs preemptive action validity assessment before physical execution or world-model imagination. Pre-VLA leverages an efficient
multimodal backbone with modality-aware pooling and a lightweight dual-branch head to predict both safety confidence and critic-derived advantage scores for candidate action chunks. To handle severe class imbalance and unstable boundary decisions, we train Pre-VLA with a multi-task objective combining Focal classification, advantage regression, and soft-threshold calibration. During deployment, a dual-mode preemptive resampling scheduler filters low-quality actions and triggers adaptive resampling under a limited computation budget. Experiments on the LIBERO
benchmark show that Pre-VLA improves the average closed-loop success rate across four suites from 30.79% to 37.62% over RynnVLA-002, reduces task execution steps, achieves 183.9 ms average forward verification time per action chunk, and mitigates error accumulation in world-model rollouts.
Read the full abstract ▾
All authors Zhen Sun, Yongjian Guo, Haoran Sun, Luqiao Wang, Wei Lu, Jiachi Ji, Shengzhe Ji, Junwu Xiong, Zhijun Meng
Computer Vision · Cross-listed from eess.IV
Time-varying rPPG signal separation via block-sparse signal model
Kosuke Kurihara, Yoshihiro Maeda, Daisuke Sugimura et al. (+1 more)
Remote photoplethysmography (rPPG) enables non-contact measurement of cardiac pulse signals by analyzing subtle color changes in facial
videos. Nevertheless, extracting rPPG signals remains challenging because of their extremely weak signal strength and susceptibility to illumination noise. In this paper, we propose an rPPG signal extraction method that exploits the quasi-periodic characteristics of rPPG signals. Our approach models quasi-periodicity of the rPPG signal, which arises from the stable cardiac cycle, as a block-sparse structure in the time-frequency domain. To incorporate a block-sparse model and enable adaptive signal separation under illumination fluctuations, we construct a time-varying signal separation framework. Experiments using a public
dataset demonstrate the effectiveness of our method.
Read the full abstract ▾
All authors Kosuke Kurihara, Yoshihiro Maeda, Daisuke Sugimura, Takayuki Hamamoto
Computer Vision
Moment-Reenacting: Inverse Motion Degradation with Cross-shutter Guidance
Xiang Ji, Guixu Lin, Zhengwei Yin et al. (+2 more)
Motion degradation, manifested as blur in global shutter (GS) images or rolling shutter (RS) distortion in RS counterparts, remains a fundamental challenge in computational imaging, especially under fast motion or low-light conditions. While prior works have treated blur decomposition and RS temporal super-resolution as separate tasks, this separation fails to exploit their intrinsic complementarity. In this paper, we propose a unified framework to invert motion degradation and reenact imaging moment by jointly leveraging the complementary characteristics of GS blur and RS distortion. To this end, we introduce a novel dual-shutter setup that captures synchronized blur-RS image pairs and demonstrate that this combination effectively resolves temporal and spatial ambiguities inherent in both modalities. For allowing flexible performance-cost trade-offs, we further extend this dual-shutter setup to a stereo Blur-RS configuration with a narrow baseline. In addition, we construct a triaxial imaging system to collect a real-world
dataset with aligned GS-RS pairs and ground-truth high-speed frames, enabling robust training and evaluation beyond synthetic data. Our proposed network explicitly disentangles motion into context-aware and temporally-sensitive representations via a dual-stream motion interpretation module, followed by a self-prompted frame reconstruction stage. Extensive experiments validate the superiority and generalizability of our approach, establishing a new paradigm for realistic high-speed
video reconstruction under complex motion degradations. Codes and more resources are available at https://jixiang2016.github.io/dualBR_site/.
Read the full abstract ▾
All authors Xiang Ji, Guixu Lin, Zhengwei Yin, Jiancheng Zhao, Yinqiang Zheng
Computer Vision
FastTab: A Fast Table Recognizer with a Tiny Recursive Module and 1D Transformers
Laziz Hamdi, Amine Tamasna, Pascal Boisson et al. (+1 more)
Table structure recognition (TSR) requires both table-level coherence (row/column counts, headers, spanning
cells) and precise separator localization. We introduce FastTab, a grid-centric TSR model that avoids autoregressive HTML decoding by combining (i) a lightweight Tiny Recursive Module (TRM) for global
reasoning and (ii) axial 1D
Transformer encoders that capture long-range dependencies along rows and columns. The model predicts row/column counts, header rows, and separators to construct a grid, then infers rowspan/colspan using ROI-aligned
cell features. Across four
benchmarks (PubTabNet, FinTabNet, PubTables-1M, and SciTSR), FastTab achieves competitive structure recovery performance while operating at low-latency inference. We further study robustness under pixel-level anonymisation and show an extension to curved separators for camera-captured documents. The source code will be made publicly available at https://github.com/hamdilaziz/FastTab .
Read the full abstract ▾
All authors Laziz Hamdi, Amine Tamasna, Pascal Boisson, Thierry Paquet