Computer Vision
STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs
Jing Li and Duygu Sarikaya
Despite recent advances in
surgical vision-
language models (VLMs), temporal
reasoning remains limited because existing supervision is largely frame-centric. Frame-level scene graphs (SGs) have proven effective in providing structured representations of
surgical environments but do not explicitly model the dynamics of
surgical workflows. To explicitly model how
surgical states evolve across time, we introduce a multi-level structured temporal supervision methodology that augments frame-level
surgical SGs with object-level continuity, event-level interaction continuity, and procedure-level connectivity. We then execute temporal queries over the resulting spatio-temporal scene graphs (STSGs) to generate evidence-grounded question-answer pairs, which together form the STSG-VQA
benchmark. Each question is linked to the temporal interval and STSG evidence used to derive its reference answer, enabling traceable verification. The
benchmark contains 18,458 question-answer pairs across seven temporal categories. Fine-tuning Qwen3-VL-4B and Hulu-Med-4B with STSG-derived supervision improves question-level micro accuracy by 24.39 and 19.56 percentage points over their zero-shot baselines and by 16.50 and 14.25 points over static scene-graph supervision, respectively. These gains span all temporal categories, indicating that STSG-derived supervision helps
surgical VLMs reason over temporally grounded interactions rather than isolated frames. The code and
dataset will be made publicly available upon acceptance.
Read the full abstract ▾
Computer Vision
Layer Selection in VLMs for Zero-Shot OOD Detection via Multi-Resolution Entropy Estimation
Shyam Nandan Rai, Francesco Di Salvo, Sebastian Doerrich et al. (+1 more)
Out-of-distribution (OOD) detection is crucial for safe deployment of medical AI systems, where domain shifts arise across institutions, acquisition protocols, and patient populations. VLMs enable zero-shot OOD detection by
embedding images into a
language-aligned latent space, where cross-modal similarity serves as a non-parametric confidence signal for identifying in-distribution samples. Yet existing methods rely almost exclusively on final-layer
embeddings, implicitly assuming that the deepest representations are universally optimal. We first show that this assumption does not hold in medical imaging: intermediate layers provide complementary OOD signals, and the optimal representational depth depends on the respective image modality. While prior work selects layer combinations via
entropy minimization of normalized histograms, we demonstrate that single-resolution
entropy estimation is highly sensitive to binning choices, leading to performance variations of up to 19.3% AUROC. To address this instability, we propose a multi-resolution
entropy estimation strategy that aggregates histogram statistics across multiple discretization scales, enabling robust and stable intermediate-layer selection. Across two medical OOD
benchmarks, namely MIDOG and OASIS, covering distinct imaging modalities, diverse shift types, and different VLM backbones, our method consistently outperforms state-of-the-art approaches, offering a lightweight and stable solution for zero-shot OOD detection.
Read the full abstract ▾
All authors Shyam Nandan Rai, Francesco Di Salvo, Sebastian Doerrich, Christian Ledig
Computer Vision
Temporal State Transport in Video Generation: Diagnosing and Correcting Spectral Imbalance
Luyao Tang, Bingjun Luo, Dong Yi et al. (+5 more)
Reliable
video generation requires more than high-quality frames to form a coherent story: a model must maintain a persistent state, transporting visual attributes such as identity, scene layout, motion, and fine details across time. Existing training-free methods mainly strengthen cross-frame
attention or analyze local
attention entropy, but these views do not reveal whether temporal interactions stay in a healthy transport regime. In this work, we study
video generation through the perspective of Temporal State Transport. We introduce Spectral Tension, a signed diagnostic that compares local
attention diffuseness with global spectral diversity, and use it to identify two opposite temporal failures: fragmented transport and over-mixing hotspots. Based on this diagnosis, we propose Spectral Transport Homeostasis, a training-free regulator that softly corrects pathological temporal states while largely preserving balanced ones. Experiments on pretrained
video generation models show that the original model often occupies imbalanced temporal regimes, whereas our method selectively applies larger corrections to the worst temporal hotspots and improves temporal consistency and visual quality without finetuning. Code: https://github.com/lytang63/temporal-state-transport
Read the full abstract ▾
All authors Luyao Tang, Bingjun Luo, Dong Yi, Jialin Guo, Haoning Xi, Cheng Chen, Yizhou Yu, Chaoqi Chen
Computer Vision
SignRefine: Adapting Foundational Video Models for Sign Language Generation
Anton Pelykh, Edward Fish, Ozge Mercanoglu Sincan et al. (+1 more)
Sign
language video generation demands precise hand and facial articulation, yet modern
video diffusion models, trained predominantly on spoken-
language video, produce artifacts that render signing unintelligible. We propose SignRefine, a sign
language video generation model that produces comprehensible signing from 2D keypoint conditioning alone, generalizing across appearances and visual conditions. Our approach builds on a pretrained
video diffusion transformer and introduces local adapters with spatial grounding to selectively refine hand and face regions, steering the strong base model's prior toward accurate articulation. To enable this work and support broader sign
language research, we present NVSign, a large-scale
dataset of
video content natively produced in sign
language, offering diverse signer appearances, environments, and natural conversational settings. Trained on this data, our model shows up to 30% improvement in hand pose precision metrics over the strongest baseline and is preferred by sign
language users for visual quality and comprehensibility in more than 80% of comparisons.
Read the full abstract ▾
All authors Anton Pelykh, Edward Fish, Ozge Mercanoglu Sincan, Richard Bowden
Computer Vision · Cross-listed from cs.RO
AURORA: Active Uncertainty-Driven Re-Orientation for In-Hand Reconstruction
Feiyu Zhao, Yuetong Li and Chenxi Xiao
Observing objects grasped by a
robot hand is challenging due to severe visual occlusions. Although in-hand manipulation can expose hidden surfaces, existing approaches often rely on predefined or open-loop reorientation strategies that do not explicitly target under-observed regions. We propose AURORA, an active 3D reconstruction framework that closes the loop between online object-centric reconstruction and in-hand reorientation. At its core, Ray-GPIS estimates direction-wise reconstruction uncertainty along candidate viewing rays and selects next-best-view targets using an uncertainty–novelty objective, which are realized through an axis-conditioned in-hand rotation policy. The resulting RGB-D observations are fused incrementally using CAD-free 6D pose tracking and lightweight
geometric reconstruction. Experiments demonstrate that AURORA improves reconstruction quality and information-acquisition efficiency over non-active rotation strategies, while Ray-GPIS also outperforms active view-planning baselines in reconstruction performance, action-ranking quality, and planning efficiency. Targeted ablations further validate its robustness to hand occlusion and pose errors. The project webpage is available at https://aurorahand.github.io/
Read the full abstract ▾
Computer Vision
GSComplete: Gaussian Splat Completion with 2D Diffusion Priors
Elias Brugger, Philipp Erler, Stefan Ohrhallinger et al. (+1 more)
Gaussian splats provide a fast, high-fidelity representation for 3D objects but are often constructed from incomplete input data in practice, leaving missing regions. Existing completion methods either do not preserve the original splats or require scarcely available 3D training data. We propose GSComplete, which combines 3D generation based on Score
Distillation Sampling with a novel preservation loss that encourages the original splats to be preserved where they should be visible. This effectively completes the Gaussian splat object using only 2D
diffusion priors while fully preserving existing splats and generating new splats only in missing regions, without occluding the input. To evaluate our approach, we introduce a new
dataset of partial Gaussian splat objects and show that GSComplete achieves significantly more accurate preservation of the input than existing methods with comparable plausibility of the completed result. Our code and
dataset will be made available upon acceptance.
Read the full abstract ▾
All authors Elias Brugger, Philipp Erler, Stefan Ohrhallinger, Paul Guerrero
Computer Vision
AirAnchor: Bridging Local and Global Spatial Information for Zero-Shot Aerial Vision-and-Language Navigation
Shanwei Fan, Bin Zhang, Zhiwei Xu et al. (+4 more)
Aerial
Vision-and-
Language Navigation requires drones to follow natural-
language instructions and navigate through complex urban environments. Accurate
navigation relies on both local and global spatial information, which support immediate action grounding and long-horizon path planning, respectively. However, existing zero-shot methods typically operate at a single spatial scale, relying either on local representations constructed online from current observations or on global
memories built offline from historical experience. To address this limitation, we propose AirAnchor, a new paradigm that bridges local and global spatial information through spatial anchors and integrates both into a shared
navigation framework, enabling comprehensive spatial grounding for decision-making. AirAnchor consists of three core components: (1) Query-Driven Spatial Anchor Grounding, which identifies decision-relevant anchors from visual observations and organizes them into local spatial representations; (2) Persistent Object Spatial
Memory, which incrementally maintains an object knowledge base as persistent global spatial
memory and retrieves landmark-related spatial priors; and (3) a Spatially-Informed
Navigation Agent, which explicitly integrates both local and global spatial information into an agentic framework for decision-making. Extensive experiments on AerialVLN demonstrate that AirAnchor substantially outperforms existing zero-shot baselines, validating the effectiveness and efficiency of the proposed paradigm.
Read the full abstract ▾
All authors Shanwei Fan, Bin Zhang, Zhiwei Xu, Yingxuan Teng, Siqi Dai, Lin Cheng, Guoliang Fan
Computer Vision
CAR-MIL: Counterfactual Attention Regularization for Multiple Instance Learning
Imane Chraki, Pierre Marza, Stergios Christodoulidis et al. (+1 more)
Multiple Instance Learning (MIL) is widely used for weakly supervised learning, particularly in digital pathology, where fine-grained annotations are costly. Most MIL methods aggregate instance features via
attention mechanisms. However,
attention weights do not always faithfully reflect instance importance and may focus on spuriously correlated regions. In this work, we propose CAR-MIL, a framework that explicitly guides
attention learning through a counterfactual
attention regularization objective inspired by counterfactual explanations. Built on a standard
attention-based MIL architecture, our approach introduces a lightweight counterfactual
attention branch trained to produce an alternative prediction while remaining close to the factual
attention distribution. This encourages prediction changes to arise from minimal, structured redistributions of
attention, leading to more informative evidence allocation. The resulting factual and counterfactual
attention maps capture complementary evidence: the former highlights regions supporting the prediction, while the latter reveals regions whose reweighting would challenge it. We evaluate our method on synthetic MIL
benchmarks with instance-level ground truth enabling controlled analysis of
attention behavior and on five digital pathology
datasets across four tasks. CAR-MIL maintains competitive classification performance, with the largest gains observed on more challenging tasks, while improving
attention reliability, demonstrating the benefits of integrating counterfactual explainability
reasoning into
attention learning. Code is available at: https://github.com/ImaneCR/CAR-MIL/.
Read the full abstract ▾
All authors Imane Chraki, Pierre Marza, Stergios Christodoulidis, Maria Vakalopoulou
Computer Vision
Towards Embodied Air-Ground Cooperative Object Search: Benchmark, Dataset and Agentic Method
Boao Yu, Zimo Chen, Junreng Rao et al. (+4 more)
Air-Ground Object Search (AGOS) in urban environments is a challenging
embodied task, which requires an Unmanned Aerial Vehicle (
UAV) and an Unmanned Ground Vehicle (UGV) to jointly search for and verify a specified target vehicle from multi-view visual references. To study this underexplored problem, we introduce AGOS-Bench, the first dedicated
benchmark for evaluating whether general-purpose
Vision-
Language Models (VLMs) can integrate aerial discoveries and ground-level verification through
UAV-UGV cooperation. We further provide AGOS-
Dataset as the companion resource of exemplary trajectories constructed by an automatic pipeline. It consists of 7.7k episodes for searching objects of diverse categories and attributes, spanning three difficulty levels. To address the AGOS task, we propose AGOS-
Agent, a training-free and tool-augmented approach. The agentic method relieves VLMs from complex and dynamic coordination via a deliberate search-handoff-verify cooperation protocol, only demanding VLMs for scene understanding and decision-making. Extensive experiments on nine VLMs show that AGOS-
Agent improves overall success rate for eight of the nine evaluated backbones while reducing decision steps for all nine. On the hard split, the SR and SPL of Gemini-3.6-Flash increase from 8.6% to 55.7% and from 7.6% to 44.0%, respectively.
Read the full abstract ▾
All authors Boao Yu, Zimo Chen, Junreng Rao, Yue Hu, Zhengqiu Zhu, Yong Zhao, Rusheng Ju
Computer Vision
From Coordinates to Candidate Regions: Temporal Change Localization via Region Selection in Remote Sensing Multimodal LLMs
Juwan Chung, Sungjune Park, Yeongyun Kim et al. (+1 more)
Remote sensing
multimodal large
language models (RS-MLLMs) have advanced scene understanding and visual question answering over satellite imagery, yet localizing specific objects or changed regions remains challenging. Existing approaches rely on generating bounding box coordinates as
token sequences, which is fragile for the small, densely packed objects common in remote sensing and increasingly error-prone when multiple targets must be localized simultaneously. In this work, we present an RS-specific formulation of the region selection paradigm, previously explored in natural-image MLLMs, and extend it to temporal change localization over multi-image sequences. Our framework employs a text-conditioned region proposal module, encodes each candidate as special
tokens carrying per-frame visual features enriched with spatial and temporal cues, and lets the LLM localize targets by selecting region
tokens in its response. We construct a multi-task training and evaluation suite spanning localization, referring expression, visual grounding, and understanding tasks across single-image and multi-temporal settings. Experiments show that our approach substantially outperforms coordinate-generation baselines on temporal change localization, while improving single-image visual grounding and maintaining competitive understanding performance. Oracle analysis decomposes the contributions of the region proposer and the LLM selector, providing diagnostic insight unique to this framework. Our code will be available at https://github.com/juwan-kr/RS-RegionSelect.
Read the full abstract ▾
All authors Juwan Chung, Sungjune Park, Yeongyun Kim, Yong Man Ro
Computer Vision
GALoc: Gravity Aligned Wireframes for Depth-Free Monocular Floorplan Localization
Jeahn Han, Minji Kim, Jeongbin Sohn et al. (+3 more)
Floorplans are compact, appearance-invariant maps ideal for indoor localization, yet existing methods rely on depth networks that are brittle in cluttered scenes. We propose GALoc, a geometry-first framework that replaces depth prediction with
gravity-aligned wireframes that satisfy verticality and coplanarity by construction. Given
monocular RGB, camera intrinsics, relative poses, and IMU orientation, GALoc constructs a linear constraint
matrix encoding verticality and coplanarity, and finds the camera gauge minimizing its smallest singular value via global search. The rectified wireframes are projected into bird's-eye-view layouts through a closed-form, FOV-consistent transformation and matched against the floorplan via metric-free SE(2) search. We evaluate end-to-end on Structured3D, with calibrated noise on Gibson, and on real-world author-collected sequences. When sufficient wall geometry is visible, GALoc matches or outperforms depth-based baselines – achieving 88% sequential localization success at 0.1m over 100-step sequences on Gibson vs the baseline's 68% – while abstaining in structure-blind scenes.
Read the full abstract ▾
All authors Jeahn Han, Minji Kim, Jeongbin Sohn, Jonghyeok Park, Matthias Wuest, Pyojin Kim
Computer Vision
To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models
Siru Jiang, Yuwei Liang, Jian Liang et al. (+2 more)
Test-time adaptation (TTA) has emerged as a prominent strategy for adapting
vision-
language models to distribution shifts during inference. We conduct a per-sample analysis of model predictions before and after adaptation, and observe two failure modes in existing TTA methods that echo previous work. Adaptations are frequently negligible, yielding no change in the model's predictions, and more severely, they can be detrimental by flipping previously correct predictions to incorrect ones. This naturally raises a question: Can we identify and skip such negligible or harmful adaptations? In this work, we introduce a new problem of selective adaptation, which aims to determine whether a given test sample should undergo adaptation or be skipped. To this end, we propose Cross-Augmentation Similarity (CAS), a simple baseline that performs adaptation only when predictions across augmented views exhibit low similarity. Notably, CAS not only preserves but in some cases improves overall accuracy, even when skipping nearly 85% of the adaptation process. We hope other researchers will explore this new direction and surpass the performance of our baseline. Our code is available at https://github.com/sirujiang/selective-adaptation.
Read the full abstract ▾
All authors Siru Jiang, Yuwei Liang, Jian Liang, Ran He, Tieniu Tan
Computer Vision
ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation
Yiran Wang, Zeyu Zhang, Ling Shao et al. (+1 more)
Text-to-motion (T2M) generation maps natural
language to human joint movements, aiding gaming, VR, and robotics.
Retrieval-Augmented Text-to-Motion (RAG-T2M) improves generation on complex descriptions by conditioning on retrieved motion-text pairs. However, existing RAG-T2M models face two challenges: coarse-grained
retrieval and fusion mechanisms overlook the hierarchical, spatial-temporal
topology of human motion, and a representation gap exists because retrieved evidence resides in a semantic space separate from the generator's latents. To address the first, we present ReMoMask, a structure-aware RAG framework coupling Hierarchical Bidirectional Momentum (HBM) contrastive learning to align global and part-level features with text; Semantic Spatial-Temporal
Attention (SSTA) for
topology-aware fusion; and
Topology Structured Masking (TSM) to force robust part-level grounding via adaptive masking. To address the second, we introduce ReMoMask-2, which rebuilds the
retrieval database directly within the generator's pre-quantization latent space and aligns text queries via a distilled lightweight projector, allowing the generator to directly consume the retrieved motion's semantic content. Extensive experiments on HumanML3D, KIT-ML, and SnapMoGen demonstrate our retriever achieves state-of-the-art accuracy, while ReMoMask-2 attains the lowest FID on KIT-ML and SnapMoGen; notably, its single mask-
transformer stage surpasses ReMoMask's full two-stage pipeline and delivers the fastest inference.
Read the full abstract ▾
All authors Yiran Wang, Zeyu Zhang, Ling Shao, Hao Tang
Computer Vision
Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking
Jue Wang, Xuan Wang, Hao Zhou et al. (+6 more)
Moving-object perception must decide which image regions correspond to real motion and keep every instance identified over time. Methods that read motion from appearance, optical flow, or estimated trajectories lose that evidence under poor illumination, adverse weather, reflections, and occlusion. Radar is a natural remedy because it measures radial velocity directly instead of inferring it from photometric correspondence. However, existing
benchmarks do not jointly provide radar measurements, dense moving-instance masks, and temporally consistent identities for surveillance. We therefore introduce RGBTR-Motion, a synchronized and calibrated fixed-camera
benchmark that pairs RGB, thermal, and radar streams with dense instance masks and temporally consistent identities across diverse surveillance scenes. We also develop SAM-Radar, an RGB, thermal, and radar-based
segmentation and tracking framework built on SAM 3. SAM-Radar's radar-aware detector fuses calibrated RGBT features with radar returns that are grounded at their projected image locations, and motion supervision, implemented as foreground classification of those projected returns, teaches the detector to reject clutter without any text
prompt. The tracker associates accepted radar returns with individual trajectories and uses them as physical evidence that a visually degraded target remains present. This allows it to bridge short periods of low visibility or occlusion and reconnect a reappearing target to its existing identity instead of starting a new track. SAM-Radar attains 0.7027 IoU and 0.8090 F1-50, and raises MOTA, HOTA, and IDF1 by 0.2977, 0.1603, and 0.2857 over the strongest competing values.
Read the full abstract ▾
All authors Jue Wang, Xuan Wang, Hao Zhou, Ruixiang Zhou, Yixuan Zhou, Tianshuo Yuan, Jieming Ma, Jie Zhang, Fei Luo
Computer Vision
CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs
Nhat-Tan Bui, Varshini Elangovan, Arun Reddy Anugu et al. (+7 more)
Representing a 3D scene as multi-view images allows 2D VLMs to reason in 3D by reusing priors from pre-training, sidestepping the scarcity of annotated 3D data. However, it produces thousands of redundant visual
tokens whose cost grows with every view. Existing visual
token pruners fall into two families, each limited in the 3D multi-view setting. Learned importance methods rank
tokens by
attention or encoder features; because redundancy here is fundamentally spatial, they keep near-duplicate
tokens from a few prominent regions and leave most of the scene unrepresented. Voxelization methods improve spatial coverage but cannot enforce an exact
token budget and saturate as multi-view observations overlap in 3D, capping retention well below the target. We show that spatial coverage is associated with 3D
reasoning performance and introduce CoVeR, a deterministic, training-free selector that uses only
token coordinates, with no learned signals. CoVeR selects
tokens that collectively cover every region of the scene, and solves the limitations of both families: it enforces an exact per-scene budget, breaks the voxelization saturation plateau, and avoids the near-duplicate selections of learned importance. Extensive experiments show CoVeR outperforms prior SOTAs on all three 3D
reasoning benchmarks and generalizes as a plug-and-play module tested across four VLMs. Notably, with only $\approx$8% of visual
tokens, it preserves 93.5% of full-
token performance, surpassing SOTA by 3.9 percentage points on average across
benchmarks.
Read the full abstract ▾
All authors Nhat-Tan Bui, Varshini Elangovan, Arun Reddy Anugu, Sreyas Mohan, Wei Ye, Dilin Wang, JQ Huang, Rakesh Ranjan, Aviral Chharia, Fernando De la Torre
Computer Vision
VeriScene: Reconstructing Crime Scenes from Legal Evidence via World-Model Agent
Kevin Chuanpu Fu, Yongsen Zheng, Zee Kin Yeong et al. (+1 more)
World models take
multimodal inputs like text, photos, and diagrams to generate dynamic scenes in accordance with the laws of physics, thus opening a compelling application: fusing
multimodal legal evidence to re-create a crime scene and re-enact how an offence could have been committed. However, feeding the raw, unorganized evidence into a world model fails in forensic use: it silently drops evidence, glosses over contradictory testimony, and produces motion that violates the evidentiary record. This paper presents VeriScene, an
agent that orchestrates the world model: it reconstructs crime scenes from forensic photographs and witness statements of varying reliability, keeping every claim traceable to evidence and every motion physically plausible. VeriScene iteratively fuses the evidence into a cited narrative under an
auditing loop, verifies the hypothesized dynamics via probe rollouts in the world model with corrective constraint injection, and renders the offence as a re-enactment
video from a fused keyframe. On a
benchmark of 25 crime scenarios across 7 physically-driven case types (139 forensic-style photographs and 65 statements with planted unreliability), VeriScene attains 0.9014 evidence coverage and 0.7217 factual consistency (0-1 scale) on the 20 test scenes, outperforming an end-to-end
multimodal-LLM baseline by 20.35% in factual consistency and 34.88% in temporal coherence, while generalizing across four LLM orchestration backends at USD 1.82 per scene.
Read the full abstract ▾
All authors Kevin Chuanpu Fu, Yongsen Zheng, Zee Kin Yeong, Kwok-Yan Lam
Computer Vision
EdMCGS: Event-Driven Markov Chain Gaussian Splatting for Extreme-Low-Frame-Rate Dynamic Scene Reconstruction
Yuzhong Wang, Wenmin Wang and Xinxing Yu
We present EdMCGS (Event-driven Markov chain Gaussian Splatting), an end-to-end method for reconstructing dynamic 3D scenes from extreme-low-frame-rate RGB together with an event stream, which can then be rendered at any intermediate timestamp. Methods relying solely on RGB images generate numerous artifacts due to the lack of evidence from between consecutive frames. To supply this missing evidence, we model the scene motion as an event-driven Markov chain, in which the sparse RGB frames anchor the state at their own timestamps while the events recorded within an interval drive the transition across it. Since the transition reads the events of the current interval, it remains active at inference and produces the in-between motion of the 3D Gaussians directly from the events rather than by interpolation, which sets our method apart from prior work that uses events only as training-time supervision. The state is carried by a compact set of control points, each driven by the events sampled in the neighborhood of its own image projection, and a temporal local isometry term keeps the propagated motion locally rigid. Experiments on synthetic and real-world scenes show that EdMCGS outperforms both RGB-based and event-based baselines, while rendering in real time with far fewer Gaussians than the strongest event-based baseline. We release our source code and a new
dataset at https://github.com/joseclipse/EdMCGS.
Read the full abstract ▾
Computer Vision
Do Input-Level Defenses Transfer to Observation-Level Attacks on VideoLLMs?
Bangshuo Zhu, Wei Song, Yuxin Cao et al. (+4 more)
Video Large
Language Models (VideoLLMs) are increasingly deployed in safety-critical applications such as content moderation and
video analytics. To process long
videos efficiently, VideoLLMs rely on frame sampling,
token compression, and modality fusion, which together form an observation pipeline that reduces the raw
video to a compact internal representation. Recent observation-level attacks exploit this pipeline to prevent the model from perceiving harmful content, yet no defense has been explicitly designed for this threat. We introduce DefTEval, a controlled evaluation framework that systematically assesses whether input-level
adversarial defenses, which operate on the pixel content of already-sampled frames, can mitigate observation-level attacks. Across five VideoLLMs, eleven representative defenses, and five attack types, we find that input-level defenses offer limited and inconsistent protection, with harmful detection rates frequently near zero. Critically, defenses fail even against attacks that embed harmful signals in every sampled frame, indicating that the bottleneck extends beyond sampling omission to the suppression of signals that do enter the model.
Token compression discards localized features, and modality fusion systematically down-weights weakened visual signals. Furthermore, defense effectiveness is dominated by model architecture rather than by the defense method itself, and detection rates vary drastically across content categories, exposing structural weaknesses in temporal
reasoning. These findings demonstrate that securing VideoLLMs requires system-level robustness mechanisms spanning sampling-aware coverage guarantees,
token-level preservation of safety-relevant features, and modality-balanced fusion.
Read the full abstract ▾
All authors Bangshuo Zhu, Wei Song, Yuxin Cao, Yuezhong Wu, Zhiquan Liu, Yuekang Li, Jingling Xue
Computer Vision
Supervised Cross-Modal Feature Alignment for Zero-Wearable Freezing of Gait Detection in Parkinsonism
Aryan Singh and Chandan Biswas
Objective assessment of Freezing of Gait (FoG) in Parkinson's disease (PD) relies predominantly on wearable Inertial Measurement Units (IMUs). While IMUs provide optimal kinematic precision, mandatory sensor attachment restricts continuous clinical deployment. Conversely, unobtrusive
vision-based alternatives suffer substantial classification errors during turning-in-place tasks, where
geometric self-occlusion degrades deterministic skeletal coordinates and obscures the high-frequency precursors required for FoG detection. To resolve these physical observation limits, we propose a supervised cross-modal subspace
distillation framework. During optimisation, pre-trained kinematic data from IMU sensors and contextual clinical metadata act as oracles to guide a deployable visual architecture. By incorporating joint velocity and acceleration derivatives, utilising a confidence-based gating mechanism, the visual model mitigates some of the tracking errors during occlusion events. Empirical evaluations confirm this latent
alignment transfers the predictive fidelity of hardware sensors directly into the visual representation, yielding $85.5\%$ accuracy, and $82.4\%$ balanced accuracy. All the while maintaining a
vision only model at inference.
Read the full abstract ▾
Computer Vision
From Glance to Scrutiny: Progressive Distortion Reasoning for Fine-Grained Image Quality Assessment
Aoting Zhang, Mingze Gao, Dongbao Yang et al. (+5 more)
Multi-modal large
language models (MLLMs) have demonstrated significant potential in image quality assessment (IQA) by bridging visual perception with descriptive evaluations. However, existing approaches mainly focus on holistic quality prediction, often functioning as black boxes that provide limited insight into where distortions occur and how they affect perceived quality, hindering fine-grained analysis of localized and heterogeneous degradations. We propose GS-IQA, a framework that reformulates IQA as a progressive Where–What–How diagnosis, emulating the human perceptual process from an initial glance to closer scrutiny. Since a severity judgment is meaningful only for a correctly localized and recognized region, we realize this progression through a two-stage reinforcement learning paradigm that respects such dependencies: the glance stage uses a perception-gated reward to establish where degradations lie and what they are, activating severity feedback only once both are correct, while the scrutiny stage introduces online reward-conditioned degradation generation to synthesize hard examples targeted at the model's perceptual bottlenecks, sharpening its discrimination of subtle severity variations. To enable systematic evaluation, we construct Diag-Bench, a region-level IQA
benchmark of about 25K curated samples spanning 12 distortion types and five ordinal severity levels. Extensive experiments show that GS-IQA consistently surpasses state-of-the-art methods in distortion localization, recognition, and severity estimation, and that its diagnostic representations transfer effectively to conventional global quality prediction across diverse external
benchmarks. Code and data will be released.
Read the full abstract ▾
All authors Aoting Zhang, Mingze Gao, Dongbao Yang, Longyi Chen, Daoxin Zhang, Yi Wu, Yao Hu, Yu Zhou
Computer Vision
FPicker: Topology-Guided Evolution for Filament Tracing in Low-SNR Microscopy
Tingyin Zhao, Mingtao Huang and Yuan Shen
Automating filament tracing in Cryo-Electron Microscopy (Cryo-EM) is essential for 3D helical reconstruction but challenged by intersecting
topologies and extremely low Signal-to-Noise Ratios ($\text{SNR} = \sigma_s^2/\sigma_n^2$ < 0.1 or -10 dB). Existing paradigms fail: pixel-wise segmenters suffer from severe topological fracturing, box-based detectors face ghost center drift, sequential trackers derail due to error accumulation, and traditional active contours collapse under artificial closed-curve constraints. To resolve these bottlenecks, we present FPicker, the first
topology-guided framework reconciling these incompatibilities. It unifies perception via a center-endpoint representation and an open-curve
evolution module to explicitly model non-cyclic connectivity. On simulated
benchmarks, FPicker outperforms top baselines by over $40\%$ relative gain in mean spatio-angular precision (mSAP) and reduces topological gap rates by over $60\%$ under extreme noise ($-20\text{ dB}$). By learning intrinsic physical geometry rather than local texture, FPicker demonstrates strong potential as a resilient
geometric backbone. Its zero-shot performance on the real-world EMPIAR
dataset exhibits robust topological resistance, achieving a state-of-the-art 82.9% mSAP upon fine-tuning. Our results also suggest modeling physical priors is a highly robust path toward bridging the sim-to-real gap in signal-starved scientific imaging. The code is publicly available at: https://github.com/tomzhaosky/FPicker.
Read the full abstract ▾
Computer Vision
TRIUNE-Net: Harmonizing Scale, Shape, and Efficiency in Pancreatic Tumor Segmentation
Amir Hossein Saleknia, Alireza Kheyrkhah, Sanaz Karimijafarbigloo et al. (+5 more)
Pancreatic tumor
segmentation in 3D CT volumes is challenged by extreme scale variability across both the pancreas and tumor, and highly irregular tumor morphology. While recent advances have pushed
segmentation performance, existing methods do not explicitly address these challenges and come at the cost of excessive computational complexity, limiting their practicality in resource-constrained clinical environments. We propose TRIUNE-Net, a lightweight unified architecture that harmonizes scale, shape, and efficiency through three synergistic innovations. A multi-scale context aggregation module with stage-adaptive dilated convolutions enables the model to reason across the broad range of anatomical scales present in both organs. A serial linear-deformable
attention mechanism combines large effective receptive fields with shapeadaptive deformable convolutions to capture irregular, non-convex tumor morphologies. Finally, an information-preserving downsampling module replaces conventional max pooling entirely, retaining all spatial information while adding negligible parameters, preventing small tumors from being discarded before they can be recognized. On both the MSD Pancreas and NVD Pancreas
datasets, TRIUNE-Net achieves state-of-theart results with only 5.86 M parameters and no external pre-training, outperforming all baselines across all key tumor metrics. Specifically, it surpasses the next-best model by 0.45% in tumor Dice, 6.0 points in F1 score, 6.6 points in sensitivity, and 3.4 points in precision, simultaneously reflecting its ability to suppress both missed tumors and false alarms in clinically realistic conditions. Our code is available at: https://github.com/abdora-ai/TRIUNE-Net
Read the full abstract ▾
All authors Amir Hossein Saleknia, Alireza Kheyrkhah, Sanaz Karimijafarbigloo, Reza Azad, Sina Houshmand, Ulas Bagci, Dorit Merhof, Alaa Sulaiman
Computer Vision
Human-Centric Image Captioning with Subject-Centered Spatial Understanding
Bozhou Li, Jiahang Zhang, Yue Ding et al. (+11 more)
While
multimodal large
language models (MLLMs) achieve remarkable performance on generic image captioning, they frequently suffer from structural
hallucinations in human-centric scenarios. Accurately modeling human subjects is foundational for critical downstream applications, such as accurate avatar/
video/image generation and fine-grained human action understanding. However, these tasks require highly precise subject-centered spatial grounding, such as distinguishing
egocentric left/right laterality and maintaining correct anatomical-object bindings. Although catastrophic for structural integrity, these localized spatial inversions are often overshadowed by overall descriptive metrics in existing
benchmarks. To systematically expose and quantify this bottleneck, we introduce SPACE (Subject-centric Poses, Appearance, and Characteristics Evaluation), a
benchmark designed to evaluate subject-centered spatial understanding. On SPACE, we reveal that despite strong generic perception, current MLLMs consistently fail to ground descriptions in the subject's intrinsic frame of reference. To bridge this gap, we propose a specialized data construction and
alignment pipeline. We first extract structured spatial hints from fine-grained body-part localization to guide a two-stage caption rewriting process, yielding highly spatially-faithful training data. Furthermore, we design a rubric-based reward for Group Relative Policy Optimization (GRPO) that explicitly penalizes structurally critical spatial errors during
alignment. Extensive experiments on SPACE demonstrate our framework significantly improves human-centric caption quality, particularly in subject-centered spatial
reasoning, achieving performance competitive with strong closed-source models. Our
benchmark and code are available at https://github.com/JHang2020/SPACE-Eval.
Read the full abstract ▾
All authors Bozhou Li, Jiahang Zhang, Yue Ding, Yushuo Guan, Bohan Zeng, Yiyan Ji, Xinlong Chen, Yang Shi, Yifan Dai, Yuran Wang, Chengzhuo Tong, Pengfei Wan, Yuanxing Zhang, Wentao Zhang
Computer Vision
MARS-CLIP: Multi-Resolution and Attention Refined Zero-Shot Image Segmentation
Nagito Saito, Shintaro Ito, Koichi Ito et al. (+1 more)
Contrastive
Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in zero-shot transfer but often struggles with dense prediction tasks due to low spatial resolution and the loss of structural information. To address these limitations, we propose MARS-CLIP (Multi-resolution and
Attention Refined
Segmentation for CLIP), a novel framework for zero-shot semantic
segmentation. Our approach introduces two key strategies: (i) a multi-resolution feature extraction module that fuses local fine-grained features with global context to overcome input resolution constraints, and (ii) an
attention refinement mechanism that injects spatial and color biases from intermediate layers into the final self-
attention block to accurately restore object boundaries. A set of experiments on six public
datasets demonstrates that MARS-CLIP significantly outperforms state-of-the-art methods.
Read the full abstract ▾
All authors Nagito Saito, Shintaro Ito, Koichi Ito, Takafumi Aoki
Computer Vision
Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Models
Ke Hao, Yuanzhi Liang, Tingxi Chen et al. (+5 more)
Unified
multimodal models integrate visual understanding and generation within a single network, yet the two capabilities are commonly optimized as separate tasks. We introduce Generative Grounding Feedback(GGF), a self-evolving post-training framework that uses only text
prompts and the model's own visual experience. Given a
prompt, the model first generates a visual “dream.” Flow-level feedback compares text-, image-, and repair-conditioned predictions at the same noisy latent state, transferring image-grounded generation directions to the
prompt condition. Dream replay grounding replays this dream through captioning and re-imagination, training claim-level evidence to remain consistent across the replay while separating unrelated visual experiences. Jointly optimized, these two directions let generation provide visual grounding for understanding and understanding refine subsequent generation without paired image–text supervision. Experiments across unified models with different understanding–generation integration designs show consistent improvements in text-to-image generation together with modest gains in visual understanding.
Read the full abstract ▾
All authors Ke Hao, Yuanzhi Liang, Tingxi Chen, Rui Li, Haibin Huang, Chi Zhang, Yun Gu, Xuelong Li
Computer Vision
SAM3-O2D2: Zero-Shot Object Out-of-Distribution Detection by Object Class Prompting of the SAM3-Image Model
Lucas Görnhardt, Timo Bartels and Tim Fingscheidt
Object detectors have shown remarkable performance in various fields, among these medical imaging, surveillance, and autonomous driving. However, they are prone to overconfidence when encountering unseen objects in real-world deployments, causing potential safety issues. To address this, detecting out-of-distribution (OOD) objects is essential for reliable object detection. Modern approaches leverage the broad semantic knowledge of foundation models such as CLIP for post-hoc few- and zero-shot OOD detection. However, these methods typically perform OOD assessment in feature space, which can be sensitive to object detector localization errors and variations in object appearance. Moreover, the current state-of-the-art (SOTA) zero-shot method performs computationally costly
diffusion in inference. In this work, for our proposed zero-shot object OOD detection method SAM3-O2D2, we employ the SAM3-image foundation model in an efficient manner. Specifically, we
prompt SAM3 only with the object detector's predicted classes and compare the predictions of the object detector and SAM3. An object is in-distribution (ID), if SAM3 also detects an object at the corresponding location. If SAM3 does not detect the prompted object, this indicates a mismatch between the detector's prediction and the image content, suggesting that the object is OOD. Experimental results show that our method significantly surpasses the so-far zero-shot SOTA method. Specifically, we achieve new SOTA AuROC and FPR95 metrics over both ID
datasets Pascal-VOC and BDD100K and both OOD
datasets MS-COCO and OpenImages.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Online Signature Verification Using Augmented Path Signature and T-Mamba
Ruiling Li and Danyu Yang
Handwritten signature verification is vital for personal authentication across commercial and financial applications. Although deep learning methods are widely adopted for online signature verification (OSV), they often struggle with capturing highly discriminative features and modelling long-range dependencies. To address these issues, we propose a novel framework that integrates the augmented path signature (APS) descriptor with the T-Mamba model. The APS descriptor first applies time and basepoint augmentations, then computes sliding-window path signatures. The path signature is a non-parametric feature map from rough path theory that effectively captures
geometric structures and nonlinear inter-channel interactions. Inspired by the efficacy of state space models (SSMs) in sequence modelling, our T-Mamba model employs a hybrid design combining two temporal convolutional network (TCN) blocks with a time-scanning Mamba. This design enables the model to learn both local temporal patterns and global long-range dependencies, substantially improving verification accuracy. Our framework achieves state-of-the-art EERs on three public
benchmark datasets (MCYT-100, SVC-2004 Task 2, DeepSignDB), validating its effectiveness and robustness, especially when the training data is limited. Our code is publicly available at https://github.com/DLRL04/OSV-using-APS-and-T-Mamba.
Read the full abstract ▾
Computer Vision
Tracking-by-detection in Multi-object Tracking: Survey and Experiments
Yujin Yang, Kyujin Shim, Kangwook Ko et al. (+1 more)
Multi-object tracking (MOT) is an essential computer
vision task that simultaneously tracks multiple objects in
video sequences, with various applications in surveillance, autonomous
navigation, and human-computer interaction. The tracking-by-detection (TBD) paradigm, which combines object detection with temporal association, has emerged as a leading approach, driven by innovative
algorithms. Despite recent progress, fair evaluation of TBD-based methods remains a challenge. Many studies introduce modules such as similarity metrics, data association strategies, or motion models, but they are often evaluated under inconsistent protocols, with different baseline trackers, hyperparameters, and
datasets. Such inconsistencies obscure the genuine contribution of each module and hinder objective comparison. This survey systematically reviews TBD-based MOT techniques, including similarity measurements, data association, camera motion compensation, and interpolation strategies. Starting from a minimal baseline tracker, we fairly evaluate the contributions of each method across diverse
datasets and accumulate well-balanced methods. Our findings establish a strong baseline tracker and provide a foundation for the principled design of robust and versatile MOT systems suitable for real-world deployment.
Read the full abstract ▾
All authors Yujin Yang, Kyujin Shim, Kangwook Ko, Changick Kim
Computer Vision · Cross-listed from cs.RO
CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations
Aman Mehta and Riya Baviskar
How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are increasingly used as the perception front end of world models for manipulation, and their physical competence is assessed with perturbation
benchmarks and linear probes, almost always in a clean, fixed-camera scene. We show that these assessments cannot distinguish an encoder that infers physics from one that does not. CALIPER (calibrate, then predict) is a direct test: an object of unknown mass and friction is struck twice at known speeds, a third strike is shown only up to the moment of contact, and a linear readout on frozen features must predict how far the object slides. Swapping in another object's calibration clips checks that the evidence is actually used. Across 2,000 simulated episodes and eight representations, from V-JEPA 2 to a randomly initialised ViT and raw pixels, calibration adds +0.50 R^2 and the swap removes it. Yet in the clean scene every representation lands within 0.02 R^2 of the ceiling set by true simulator state, because a fixed camera exposes the object's displacement directly in pixel coordinates. Resampling camera, lighting, and clutter for every clip spreads the same representations across 0.50 R^2; when the readout chooses a push speed for a goal distance, V-JEPA 2 misses by 4 mm and the random ViT by 20 mm, no better than ignoring the object. Linear probes track none of this: a change in frame aggregation moves a probe more than pretraining does, and erasing the probed mass direction from the same representation costs nothing in one scene and 0.35 R^2 in the other. Whether a
benchmark can rank models is an empirical property, and we give three checks that establish it.
Read the full abstract ▾
Computer Vision
CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
SeongJun Jeong, Minjoon Jung, Woo Suk Choi et al. (+2 more)
Vision-
language models (VLMs) demonstrate strong performance across compositional
reasoning benchmarks, which require
reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional
reasoning with robust performance across compositional elements. It also preserves general
vision-
language capabilities such as cross-modal
retrieval and downstream visual
reasoning, while requiring fewer training samples than prior methods.
Read the full abstract ▾
All authors SeongJun Jeong, Minjoon Jung, Woo Suk Choi, Youwon Jang, Byoung-Tak Zhang