Computer Vision
PathPilot: An Adaptive Question-Conditioned Multi-Magnification Navigation Agent for Whole-Slide Image Analysis
Zhengyang Xu, Han Li, Jingsong Liu et al. (+5 more)
The diagnostic capability of WSI
agents relies on effective slide navigation to identify diagnostically relevant regions and efficiently integrate evidence across the slide. While in clinical practice, pathologists dynamically examine slides across multiple scales, integrating global and cellular evidence in a sequential manner, existing AI navigation approaches mostly operate at a single fixed magnification or rely on predefined magnification traversal. This mismatch prevents existing methods from modeling cross-magnification interactions and adaptive magnification selection inherent to real diagnostic workflows. We propose PathPilot, a clinically consistent multi-magnification WSI navigation
agent that explicitly models cross-magnification localization and evidence-adaptive exploration. Specifically, we introduce a CrossMagLoc that learns magnification-specific diagnostic localization from pathologist viewing behavior and integrates information from adjacent magnifications to identify the diagnostically most relevant region within the current ROI, thereby providing a diagnostically informed starting point for subsequent question-conditioned exploration. We further introduce an EviRegionSearch that starts question-conditioned exploration from this region, progressively accumulating question-relevant evidence and adaptively determining the next navigation action according to the remaining evidence requirement. Their iterative interaction enables PathPilot to sequentially explore across ROIs and magnifications until sufficient evidence is acquired. Extensive QA evaluations across five cohorts demonstrate consistent superiority over competing SOTA methods on inter/external cohorts, respectively, validating question-conditioned evidence acquisition across regions and magnifications that mirrors the sequential, knowledge-driven navigation of pathologists.
Read the full abstract ▾
All authors Zhengyang Xu, Han Li, Jingsong Liu, Ayako Ura, Shihui Zu, Shaohua Kevin Zhou, Nassir Navab, Peter Schüffler
Computer Vision · Cross-listed from cs.CL
Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering
Haonan Zhang, Ting Jin, Wenzhong Tang et al. (+1 more)
Visual contrastive decoding corrects answers using the prediction difference between the original image and a reference condition, yet the same contrast also turns supported abstentions into incorrect answers. We find that abstentions are lost at two stages: contrastive decoding with candidate filtering can exclude the abstention
token before any comparison takes place, and the reference branch's response to the abstention instruction is removed when the reference prediction is subtracted. We therefore propose instruction-preserving decoding, which keeps the effect of the abstention instruction on the original-image prediction intact while leaving the contrast strength freely adjustable. Using a perturbed image and an image-free prediction as references, we obtain IP-VCD and IP-M3ID. On decision utility (DU), which credits both correct answers and supported abstentions, both methods outperform all compared non-IP baselines on every one of nine models on VizWiz, and IP-VCD also leads on eight of nine models on Food-101. Candidate measurements and paired per-question comparisons show that computing the instruction effect and the visual contrast term separately lets contrastive decoding support both correct answers and supported abstentions.
Read the full abstract ▾
All authors Haonan Zhang, Ting Jin, Wenzhong Tang, Shuai Wang
Computer Vision
3D-VITAL: Visibility-aware Identity Training from Artificial Liftings of 3D Reconstruction model for Cross-View Object Re-Identification
William Grolleau, Astrid Sabourin, Guillaume Lapouge et al. (+1 more)
We address the Cross-View Object Re-Identification (ReID) task, aiming to match identities across drastically different viewpoints. We focus on Single-View Aerial-Ground ReID (SV AG-ReID), where models train on one viewpoint and must generalize to another at inference. To bridge this viewpoint gap, related work employs generative models to synthesize novel views from source images as additional training data. We leverage recent advances in large-scale single-view 3D mesh reconstruction to lift source images into textured meshes and to render them from new viewpoints. However, this pipeline produces unreliable data in two distinct ways. First, the reconstruction may fail and produce geometrically incorrect meshes, resulting in severe corruption in rendered novel views. Second, even when the geometry is correct, the rendered novel views contain hallucinated textures in regions not visible in the source image. The reconstructed mesh carries a per-pixel partition between regions visible from the source view and regions that the reconstruction model had to hallucinate, available as a deterministic byproduct of rendering. We introduce this per-pixel visibility map and exploit it in a two-stage design. Cross-View Consistency Filtering (CVCF) discards geometrically incorrect meshes by ensuring multi-view consistency between source and novel views over visible patches. Visibility-Aware
Attention Supervision (VAAS) leverages the visibility map to guide the model's
attention toward reliable regions of the rendered view during training. Combined, these two components enable our method, 3D-VITAL, to better leverage the synthetic signal from single-view 3D mesh reconstruction. 3D-VITAL sets a new state-of-the-art on three SV AG-ReID
benchmarks. Code and data will be released at https://github.com/TurtleSmoke/3D-VITAL.
Read the full abstract ▾
All authors William Grolleau, Astrid Sabourin, Guillaume Lapouge, Catherine Achard
Computer Vision
EvoSpatial: Experience-Driven Skill and Tool Evolution for Spatial Reasoning Agents
Bin Yu, Shijie Lian, Zhaolong Shen et al. (+10 more)
Spatial
reasoning agents can inspect tool outputs and repair errors, but reusing these lessons requires preserving both the computation and the conditions that make it valid. We introduce EvoSpatial, a framework that learns from execution experience by updating a library of linked skills and tools. Skills describe procedures,
geometric assumptions, and verification steps; tools implement the operations those procedures require. When solving a task, the
agent retrieves relevant skills and synthesizes reusable computational functions if existing tools are insufficient. After each task, execution feedback and evidence checks guide skill revision and tool consolidation, without ground-truth answers or
benchmark scores. Across 20
benchmarks, EvoSpatial averages 63.9, 4.0 percentage points above the average reported by SpatialClaw. The evolved library also improves held-out and cross-
benchmark performance when updates are disabled, and trajectories generated with evolved skills improve supervised fine-tuning, even without external skills at evaluation. Keeping checked procedures on record, together with the tools that support them, lets an
agent carry spatial
reasoning experience from one task to the next.
Read the full abstract ▾
All authors Bin Yu, Shijie Lian, Zhaolong Shen, Xiaopeng Lin, Yao Zhang, Haishan Liu, Haotian Li, Yuliang Wei, Kai Hu, Yichao Du, Zhirui Zhang, Bailing Wang, Kai Chen
Computer Vision
Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition
Haichen He, Weibin Liu and Weiwei Xing
Vision-
language models (VLMs) like CLIP have shown great success in image tasks, however, applying CLIP to open-vocabulary
video recognition poses challenges. Due to the limited text information, directly fine-tuning CLIP on
video recognition
datasets may lead to overfitting and hinder its generalizability. A prevalent method is to enrich text information by generating
prompts using Large
Language Models (LLMs). However, limited efforts have been made to explore how to generate higher-quality
prompts that can contribute to classify unseen categories, and how to further enhance
vision-text
alignment based on these
prompts. In this paper, we propose P2CLIP (
Prompt-guided CLIP), which aims to bridge this gap by exploring the impact of
prompts across three aspects: category,
video, and frame. At the category level, we argue that LLM-generated
prompts under simple instructions may suffer from semantic overgeneralization, which can blur the category boundaries and hinder the performance of the model. To address this issue, we design Generic-Discriminative
Prompt using LLMs to obtain discriminative category
embeddings with rich semantic information. At the
video and frame levels, we first use VLM and LLM to generate frame and
video descriptions for enhancing
vision-text
alignment. Then we propose a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of
videos and encourage a better construction of the joint visual-textual feature space. Extensive experiments across various
benchmarks, including zero-shot and few-shot settings, demonstrate the effectiveness of our method.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
Zihan You, Zehua Fan, Wenqi Lyu et al. (+19 more)
Recent
Vision-
Language-Action (VLA) models increasingly use natural-
language instructions to condition autonomous driving behavior. Yet existing closed-loop
benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does
language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a
benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate
Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving
benchmarks, including our released CARLA-
Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak
language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving
benchmarks, can substantially overstate
language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.
Read the full abstract ▾
All authors Zihan You, Zehua Fan, Wenqi Lyu, Aoqi Wang, Ni Wang, Yifei Yang, Linge Zhao, Huan Li, Sining Ang, Yuguang Yang, Jueyu Chen, Xiangyu Li, Qianshan Wei, Jieyuan Pei, Yuxin Wang, Zhiwen Chen, Jiaqi Li, Wenxuan Song, Yanyuan Qiao, Benjin Zhu, Haoang Li, Yan Wang
Computer Vision
REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation
Hang Xu, Guohui Zhang, Xiaoxiao Ma et al. (+7 more)
Memory-based long-
video generation aims to combine user control with long-range visual consistency. We organize this process into three functional components: a planner, an adapter, and a generator. Representative multi-shot systems couple
memory to generator-specific interfaces or use predefined
agent workflows, rather than independently training a
memory policy for cross-shot objectives. We introduce REEL, a general-purpose visual-
memory planner. Through a unified tool interface, REEL learns to retrieve references, create missing visual evidence, and update persistent
memory. Backend-specific adapters translate its evidence plans into supported conditioning inputs, allowing the same policy to guide different black-box
video generators. To train REEL, we construct two high-quality
datasets, REEL-SFT-4k and REEL-RL-1k, covering diverse interaction patterns and
prompt types. Training proceeds in three stages: supervised fine-tuning, single-shot Group Relative Policy Optimization (GRPO), and multi-shot GRPO. Multi-shot GRPO carries policy-generated
memory across consecutive shots, so later outcomes guide earlier reference selection and
memory updates. REEL-Bench tests identity preservation and intended state changes under text-only requests and timed reference-image uploads. On JoyAI-Echo~1.5 and MiniMax-H3, REEL improves cross-shot consistency on ST-Bench by 15.1% and 20.7%, and Core-100 on our proposed REEL-Bench by 2.41 and 8.32 points, respectively, over the no-
memory baseline.
Read the full abstract ▾
All authors Hang Xu, Guohui Zhang, Xiaoxiao Ma, Hu Yu, Jie Huang, Jiayin Xiao, Lin Song, Haoyang Huang, Nan Duan, Feng Zhao
Computer Vision
Adaptive Interaction Network for Human Motion Prediction During Human-Robot Collaboration
Mengyuan Liu, Yangting Lin and Qiongjie Cui
Human motion prediction during human-
robot collaboration is critical for achieving safe and efficient interactions in shared environments. Unlike previous studies that have primarily focused on humans and objects, we focus on the
robot-aware human motion prediction task, which explicitly models the influence of
robots on human motion. This task presents new challenges, such as capturing human-
robot heterogeneity and modeling complex spatial interactions. To address these issues, we develop an adaptive interaction network (AINet) model that consists of two branches: a primary branch that predicts future human motion and an auxiliary branch that estimates
robot trajectories. The two branches are jointly optimized and coupled via a Local-Global Spatial Interaction (LGSI) Module, which effectively captures fine-grained and global contextual dependencies between human and
robot motion sequences. Then, we introduce an Adaptive Weighted Aggregation (AWA) Module to dynamically fuse motion features using context-dependent weights, thereby increasing adaptability across diverse scenarios. Furthermore, we adopt a coarse-to-fine prediction strategy, in which a coarse pseudo-trajectory is first predicted, followed by the refinement of detailed human poses based on that trajectory. Extensive experiments on three challenging
datasets reveal that our proposed method achieves superior performance, validating its effectiveness in human-
robot collaboration scenarios. Our code is available at: https://github.com/LyTingHub/AINet.
Read the full abstract ▾
Computer Vision
Shunyuan Bai, Yuchen Ren, Le Yang et al. (+3 more)
With the advancement in
language modeling, research on
adversarial images has recently shifted from image classifiers to
vision-
language models (VLMs). Despite the new, text-
prompt-driven nature of VLMs, we find that existing VLM adversaries have largely overlooked the dynamics of text
prompts. In a typical use case,
adversarial images can protect visual
privacy by rejecting VLMs from answering questions about the image. However, existing protections are
prompt-agnostic, i.e., indiscriminately rejecting any question, including those for potential legitimate use of the image. To address this limitation, we propose
Prompt-Adapted (ProAda)
adversarial images, which are optimized to balance the rejection of harmful
prompts and the correct response of legitimate
prompts. In particular, to achieve a more accurate
adversarial direction, the text
prompt is also iteratively updated with a simple classifier judging whether the harmfulness of the
prompt remains. Extensive experiments on visual-
privacy-related
datasets validate that ProAda can reject over 90% of harmful
prompts while allowing over 90% of legitimate ones, demonstrating superior overall performance compared to existing
prompt-agnostic VLM adversaries. In addition, our ProAda achieves competitive overall cross-model transferability and improved robustness against countermeasures.
Read the full abstract ▾
All authors Shunyuan Bai, Yuchen Ren, Le Yang, Chenhao Lin, Chao Shen, Zhengyu Zhao
Computer Vision
Detecting Chain-of-Thought Hijacking in Backdoored Multimodal Large Reasoning Models
Zhifang Zhang, Qi Wei, Weitong Chen et al. (+5 more)
Chain-of-thought (CoT)
reasoning enables
multimodal large
reasoning models (MLRMs) to solve complex visual problems, but also exposes them to CoT hijacking: an adversary implants a backdoor through poisoned fine-tuning so that a visual trigger induces a hijacking turn, redirecting an otherwise faithful
reasoning chain to an attacker-chosen answer. However, since the hijack acts within the
reasoning chain, existing
multimodal backdoor defenses designed for MLLMs that answer directly struggle to detect it. Moreover, inspecting the chain itself does not reliably expose the hijack, since the hijacking turn can resemble honest self-correction in form. In this paper, we find that honest
reasoning tends to revise its answer at high-
entropy "aha moments", whereas a hijack can overturn a confident answer through a near-deterministic turn. To quantify this contrast, we formalize this phenomenon as belief inertia, the cumulative next-
token entropy of a
reasoning span minus the answer-belief revision it induces, with markedly lower values for hijacked spans than for honest
reasoning spans. Since hijacking
tokens are also image-dependent, we further improve detection by restricting the measure to image-dependent spans, yielding visual belief inertia (VBI). Building on this, we propose InertiaGuard, a training-free, test-time detector that flags a trace when its VBI falls below a threshold calibrated on a small set of clean traces. Across various attacks and
benchmarks, InertiaGuard achieves higher detection performance than existing baselines, while generalizing to different model architectures and trigger types.
Read the full abstract ▾
All authors Zhifang Zhang, Qi Wei, Weitong Chen, Olaf Maennel, Wei Emma Zhang, zuomin ren, Lei Feng, Miao Xu
Computer Vision · Cross-listed from eess.IV
Uncertainty-Aware Cortical Surface Reconstruction and Spherical Registration for Heterogeneous Brain MRI
Jesper Duemose Nielsen, Karthik Gopinath, Andrew Hoopes et al. (+4 more)
Surface-based cortical analysis is useful in many applications as it is a natural way to model the highly folded structure of the cerebral cortex, allowing quantities such as
curvature and thickness to be easily calculated. Existing deep learning (DL) methods have focused on 1x1x1 mm3 T1-weighted (T1w) scans, thus replicating classical pipelines. Here we present a neural network that is trained exclusively on synthetic scans with random contrast and resolution. Consequently, it can be applied to almost any kind of
magnetic resonance imaging (MRI) scan. In addition, our model provides vertex-wise uncertainty estimates of the predicted surface as well as a spherical registration to fsaverage space. Our model performs on par with other state-of-the-art DL methods on 1x1x1 mm3 T1w scans and outperforms the only other method capable of estimating cortical surfaces from low resolution, non-T1w scans. The predicted uncertainty, moreover, correlates with overall scan quality on low resolution scans and highlights areas of poor surface reconstruction on 1x1x1 mm3 T1w scans, suggesting that it can be used for quality control. Finally, the predicted spherical registrations prove comparable to other DL and classical registration methods. These qualities, when combined with the fast inference time, will provide the ability to analyze large, existing databases of clinical MRI scans, with the potential for finding disease-specific patterns that would otherwise be infeasible in a research setting.
Read the full abstract ▾
All authors Jesper Duemose Nielsen, Karthik Gopinath, Andrew Hoopes, Adrian Dalca, Axel Thielscher, Juan Eugenio Iglesias, Oula Puonti
Computer Vision
JtT: Rethinking Point Trajectory Diffusion for Diverse and Rigid Motion from an Image
Gabrijel Boduljak, Yushi Lan, Christian Rupprecht et al. (+1 more)
We study forecasting rigid-body motion from a single image: generating diverse and physically plausible future trajectories for arbitrary points in a scene, without building rigidity into the architecture or relying on
geometric inputs such as depth. Like
video generators, recent trajectory generators fail at this task, even in simple synthetic Kubric scenes, despite scaling up model size and training on real data. We show how to solve this with the right architecture and parameterization. Our model, JtT, is a simple
diffusion transformer that operates directly in coordinate space and decodes arbitrary and dense query patterns, without the trajectory VAEs or fixed grids assumed in prior work. Through a systematic study of the design space of point trajectory
diffusion, we find that diffusing per-frame velocities is the only parameterization that captures rigidity, that it must be paired with the right timestep schedule, and that adding full-resolution RGB at the query points resolves the boundary failures of prior methods. However, evaluating such 2D models rigorously is itself non-trivial. To our knowledge, we are the first to observe that physical plausibility and diversity do not improve together during training: larger models, as used in prior work, memorize the single ground-truth future per training image, becoming more rigid while covering fewer possible futures. Yet prior work has paid little
attention to this, so we show how model rankings depend on model size, metric, and stopping time, and adopt a stricter, checkpoint-averaged evaluation protocol. Under this protocol, JtT improves over prior methods on both distributional and rigidity metrics and approaches ground-truth rigidity, with only about 50M parameters.
Read the full abstract ▾
All authors Gabrijel Boduljak, Yushi Lan, Christian Rupprecht, Andrea Vedaldi
Computer Vision
Code-Interactive Visual Reasoning for Financial Candlestick Understanding
Chenxi Zhang, Youwei Pang, Xiaoqi Zhao et al. (+4 more)
Financial candlestick understanding requires locating visual elements, aligning them across trading dates and panels, and combining observations through numerical and financial
reasoning. Answer accuracy alone does not reveal whether predictions are supported by chart evidence. We introduce Code-Interactive Financial Candlestick
Reasoning and present Fin-Thinker, a model based on Qwen3.5-9B that generates and executes code to acquire evidence and perform computations. The model uses the returned observations to guide subsequent
reasoning. To train these capabilities, we construct FinKLine-32K with 32,184 replay-verified code-observation trajectories collected through teacher-guided student exploration. These include successful solutions with and without error recovery. Our three-stage training framework combines foundational visual-code fine-tuning, financial process supervision of successful
reasoning and corrective continuations, and execution-feedback reinforcement learning. For evaluation, we introduce FinKLineBench with 1,200 questions across 138 financial task types to assess answer correctness, code execution, and evidence grounding separately. On FinKLineBench, Fin-Thinker improves strict accuracy by 13.1 and 33.2 percentage points over the best-performing direct-answer baseline among those evaluated and its base model with sandbox access, respectively. Training-stage ablations further show that financial process supervision and execution-feedback reinforcement learning yield successive improvements in both answer accuracy and GPT-5.6-judged grounded accuracy.
Read the full abstract ▾
All authors Chenxi Zhang, Youwei Pang, Xiaoqi Zhao, Ziliang Gan, Rongjunchen Zhang, Shanshan Feng, Hao Wang
Computer Vision · Cross-listed from stat.ML
Variational Filtering for Bayesian Last Layers with an Independent Binary Probit Surrogate
Dimitrije Markovic, Conor Heins, Alexander Tschantz et al. (+1 more)
Modern deep classifiers trained by gradient descent need many passes over the data to become accurate, and even then are not always well calibrated; post-hoc temperature scaling corrects calibration after training but cannot influence training-time accuracy or convergence. We propose a variational filtering
algorithm for the last layer of deep classifiers, whose parameters are treated as the latent state of a linear-Gaussian state-space model. Each mini-batch is assimilated by an exact prediction step followed by a closed-form variational update, made conjugate by an independent binary probit (IBProbit) surrogate for the categorical likelihood. The filter's process noise stabilises the updates under feature drift and keeps the predictive distribution from collapsing to a point estimate, allowing the same procedure to be used for linear probing on a frozen backbone and for full-network fine-tuning. We evaluate on DINOv3 backbones against two baselines tuned under an identical protocol: AdamW with cross-
entropy and post-hoc temperature scaling, and Variational
Bayesian Last Layers (VBLL). As a linear probe on frozen features, IBProbit attains the best accuracy and negative log-likelihood on all five downstream tasks at the largest backbone, and on two of the five at the smallest. In out-of-distribution detection with CIFAR probes on the largest backbone, IBProbit reduces the false-positive rate at 95% true-positive rate by a factor of 1.4 to 3.2 relative to the better of the two baselines, on all three
dataset pairs. When the whole network is fine-tuned on ImageNet-1k, IBProbit reaches the final performance of the cross-
entropy baseline in half the wall-clock time.
Read the full abstract ▾
All authors Dimitrije Markovic, Conor Heins, Alexander Tschantz, Christopher L. Buckley
Computer Vision
CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction
Jialei Wang, Zehan Wang, XiaoYang Wu et al. (+6 more)
Reconstructing reference
videos as editable dynamic 3D scenes is an important path toward controllable
video generation for filmmaking: these scenes provide modifiable, renderable visual proxies that enable explicit control over scene layout, subject actions, and camera behavior. To evaluate
agents' ability to reconstruct editable dynamic 3D scenes from dynamic reference
videos, we introduce CineRecon, a
benchmark comprising 50 reference
videos and 360 annotated evaluation clips. The reference
videos span film, real-world footage, and games. Given a complete
video, an
agent uses tools to construct, animate, render, and refine its reconstruction. We evaluate this task under three representation settings: Proxy, Simplified, and Detailed. These settings increase
geometric and appearance requirements while retaining the same requirements for spatial relations, camera behavior, and subject dynamics. This design tests whether
agents can reproduce the events in a
video when detailed appearance is unnecessary, and how performance changes as representation requirements increase. For evaluation, we construct checklists that associate observable properties and events with specific entities and times. A
vision-
language judge compares timestamp-aligned reference and rendered frames across 11 categories, grouped into camera, static, and dynamic consistency dimensions. We evaluate 12 models using their corresponding
agent harnesses and find failures across all three dimensions. Subject motions and interactions remain difficult to reproduce even under the Proxy setting.
Read the full abstract ▾
All authors Jialei Wang, Zehan Wang, XiaoYang Wu, Ziang Zhang, Siyu Chen, Jiaer Huang, Yujia Zhang, Tianyu Pang, Hengshuang Zhao
Computer Vision
Dense Endoluminal Mapping and Localisation from Monocular Bronchoscopy
Salvatore Esposito, Matias Mattamala, Daniel Rebain et al. (+4 more)
Bronchoscopy inserts a camera through the airways to diagnose endoluminal disease. Locating the camera along the airway usually requires additional equipment, such as an electromagnetic tracker and a preoperative CT scan. In this paper, we investigate the use of the monocular camera stream to localise the bronchoscope without requiring additional sensing. We introduce a monocular SLAM system that localises the tip of a bronchoscope and reconstructs the airway surface around it from RGB
video alone. A Depth Anything 3 backbone, fine-tuned on over a million photorealistic airway renderings, returns metric per-pixel depth and is the system's only source of scale. DINOv3 features routed through a custom cross-
attention matcher supply sub-pixel correspondences on mucosa where classical descriptors fail. Both enter a factor graph alongside a dense constraint from projective iterative closest point (ICP) registration against the system's own running reconstruction, so surface evidence stabilises the directions the visual factors leave weakly observable. On six airways rendered from patient CT scans, the system reaches 26.1 mm mean absolute trajectory error and 77.5% mean surface F-score. Its mean surface accuracy exceeds that of five recent monocular SLAM and feed-forward baselines. MASt3R-SLAM, which fits a free scale to the ground truth, has a lower mean trajectory error (22.0 mm). ORB-SLAM3 completes only two of the six airways, whereas our system completes all six and also reconstructs two ex vivo specimens.
Read the full abstract ▾
All authors Salvatore Esposito, Matias Mattamala, Daniel Rebain, Liang Zhao, Kev Dhaliwal, Mohsen Khadem, Subramanian Ramamoorthy
Computer Vision
VLA-GAUGE: Systematically Evaluating Generalization and Robustness in Vision-Language-Action Models
Jun Xiao, Rui Ye, Dong Cao et al. (+3 more)
Vision-
Language-Action models (VLAs) have achieved high success rates on familiar manipulation tasks, yet it remains unclear whether such scores reflect genuine zero-shot generalization and perturbation robustness. Existing evaluation frameworks provide limited capability-level diagnosis and combinatorial perturbation coverage, often without connecting diagnosis to targeted model refinement. We introduce VLA-GAUGE, a diagnostic framework that unifies both evaluations under a shared executable task representation. It combines (1) paired in-distribution and out-of-distribution tasks across six capability suites, (2) single-factor sensitivity analysis with combinatorial perturbation screening and evaluation, and (3) diagnosis-driven refinement through targeted data augmentation and model fine-tuning. We evaluate 12 VLAs spanning different model sizes and action-modeling paradigms under a unified protocol. We find that (1) The mean Zero-Shot Score on VLA-GAUGE is only 22.3%, with SimpleVLA-RL scoring 8.0% and ranking second to last among the evaluated models. (2) Under three representative combinatorial perturbation configurations, the mean R-Score is 9.3. (3) Diagnosis-driven fine-tuning improves $\pi_{0.5}$'s zero-shot generalization and robustness under a selected perturbation combination on held-out scenarios by 72.4 and 31.8 scoring points, respectively; the latter exceeds the 10.9-point gain from LIBERO-Plus single-factor fine-tuning despite fewer successful training trajectories. These results show that high reported success rates on familiar manipulation tasks do not guarantee strong zero-shot generalization to unseen tasks or robustness to combinatorial perturbations, while diagnostic evaluation provides actionable signals for targeted model improvement.
Read the full abstract ▾
All authors Jun Xiao, Rui Ye, Dong Cao, Jinyu Gu, Ying Wang, Zhiliang Zhu
Computer Vision
NoduleFlow: Automatic 3D Lung Nodule Detection and Segmentation
Giulia Romoli, Filippo Ruffini, Francesco Di Feola et al. (+3 more)
Low dose Computed
Tomography (CT) is the standard imaging modality for lung cancer screening. On CT scans, lung cancer typically manifests as pulmonary nodules, whose manual identification is time-consuming and prone to observer variability, image quality limitations, and suboptimal reading conditions. Moreover, subtle temporal changes are often difficult to quantify reliably in CT scans from the same patient across follow-up examinations. Deep learning-based approaches for the automatic detection and segmentation of lung nodules have demonstrated improved accuracy and efficiency. However, most existing approaches provide only coarse region-of-interest localisation, rely on prior lesion annotations or human supervision, or treat detection and segmentation as separate tasks. In this work, we present NoduleFlow, a framework for lung nodule detection, classification, and segmentation directly from series of CT volumes. Our proposal consists of four interconnected modules and operates in an automated manner at inference time, without requiring prior lesion annotations, manual initialisation, or region-of-interest pre-selection. We further investigate whether nodule localization benefits from the propagation of candidate regions across follow-up scans through inter-scan registration fields, as opposed to conventional independent per-scan analysis. The generalisability of the framework is systematically evaluated across four independent
datasets. The results underscore the potential of automated pipelines for lung nodule analysis in screening and diagnostic settings.
Read the full abstract ▾
All authors Giulia Romoli, Filippo Ruffini, Francesco Di Feola, Valerio Guarrasi, Matteo Tortora, Paolo Soda
Computer Vision
Heuristics are Not Enough: Calibrated and Corrected Caching for Video Diffusion Acceleration
Jiyue Jiang, Zhiyu Huo, Yuhan Chen et al. (+3 more)
Feature caching has become a mainstream training-free strategy for accelerating
video diffusion models by reusing intermediate outputs across denoising steps. However, existing adaptive methods rely on heuristic difference measures misaligned with actual generation quality, employ a single global threshold that ignores heterogeneous phase dynamics, and never validate their budget estimates against true approximation error, allowing systematic bias to propagate uncorrected. We propose AlignCache, a closed-loop caching framework that directly aligns scheduling decisions with their true influence on the final generated
video through three tightly coupled components: (1) phase-adaptive budget routing that partitions the denoising trajectory into structurally distinct phases, each governed by an independent threshold and a staleness-aware guard; (2) closed-loop threshold calibration that exploits counterfactual error measurement at the classifier-free guided output level to adjust decision boundaries online via an EMA feedback loop; and (3) momentum-augmented draft synthesis combining direction-magnitude gated residual extrapolation with anchor-guided unconditional correction. By transforming scheduling from a static risk accumulator into a feedback-driven, self-correcting controller, AlignCache consistently achieves superior acceleration-quality trade-offs across state-of-the-art
video generation models. Our code is available at https://github.com/jiangjyjy/AlignCache.
Read the full abstract ▾
All authors Jiyue Jiang, Zhiyu Huo, Yuhan Chen, Yanruisheng Shao, Gong Zhang, Lei Deng
Computer Vision
Deep Learning-based Mechanical Characterization of Synthetically Generated Plant Fibers under Transverse Compression
Nikolaos Kegkeroglou, Bilal Sabillah, Myrto Inglezou et al. (+3 more)
Plant fibers are increasingly used as sustainable reinforcements in composites, but the intrinsic variability of their microstructure causes substantial scatter in transverse mechanical behavior, complicating composite design. Characterizing this behavior relies on finite element (FE) simulations requiring several minutes of high-performance computing per fiber, making high-throughput characterization across realistic fiber populations impractical. In this study, we propose a deep learning framework replacing this FE bottleneck with a fast pipeline. It includes two coupled recursive U-Nets predicting the step-by-step
evolution of fiber deformation and internal stress fields under transverse compression, followed by a regression CNN estimating the transverse Young's modulus from the final stress field. Unlike prior approaches predicting stress fields for fixed configurations or idealized geometries, our framework recursively resolves the full temporal stress
evolution for synthetic fibers derived from real hemp cross-sections, with or without a lumen. Trained on 1,000 synthetic fiber geometries with FE ground truth, it reproduces the stress field sequence with a Structural Similarity Index Measure (SSIM) of $0.9574 \pm 0.0074$ and a Root Mean Squared Error (RMSE) of $0.0345 \pm 0.0079$. It estimates transverse Young's modulus with an $R^2$ of $0.938 \pm 0.024$, a Mean Absolute Error (MAE) of $24.31 \pm 1.23$ MPa, and a Mean Absolute Percentage Error (MAPE) of $5.25 \pm 0.91\%$. Inference requires under one second per fiber versus five minutes per FE simulation, drastically reducing computational cost. This offers a fast, reliable tool for mechanical characterization of synthetic plant fibers, with promising applications in designing natural fiber-reinforced composites.
Read the full abstract ▾
All authors Nikolaos Kegkeroglou, Bilal Sabillah, Myrto Inglezou, Vincent Placet, Céndric Clevy, Antonios Porichis
Computer Vision
KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs
Raffaele Scaringi, Alejandro Sierra-Múnera, Gennaro Vessio et al. (+2 more)
Discovering and disambiguating entities from
multimodal and heterogeneous data is a key challenge in discovery science, where relevant knowledge is often implicit and distributed across modalities. We present KARAMEL, a framework for
multimodal entity linking (MEL) - the task of aligning visual and textual mentions to entities in a knowledge base - in the cultural heritage domain. Unlike prior
vision-
language models that primarily rely on surface-level
multimodal alignment, our approach incorporates sociohistorical and contextual knowledge through a knowledge-aware architecture that fuses a
vision-
language encoder with a heterogeneous graph neural network. This design enables contextual
reasoning over complex relational structures and supports robust entity disambiguation under ambiguous or underspecified evidence. To
benchmark multimodal discovery in this setting, we introduce WikiMuSA (Wikidata-based
Multimodal Semantic data for Art), a new
dataset that links artworks to Wikidata entities via images, textual descriptions, and structured contextual knowledge. Experimental results show that KARAMEL surpasses strong baselines, particularly on implicit or abstract mentions, while ablation studies highlight the critical role of domain-specific, structured context in learning expressive representations. While instantiated in the cultural heritage domain, KARAMEL addresses a general knowledge discovery problem and applies to other scientific settings involving
multimodal observations and heterogeneous background knowledge. Both the
dataset and source code are made publicly available.
Read the full abstract ▾
All authors Raffaele Scaringi, Alejandro Sierra-Múnera, Gennaro Vessio, Giovanna Castellano, Ralf Krestel
Computer Vision
Self-Verifier Capture in Unified Visual Generators: Diagnosing and Repairing Closed-Loop Self-Reward
Binyan Chen and Tong Pan
Image-space reward requires decoding and a separate judge; unified visual generators can instead reuse their understanding pathway for decode-free, self-supplied reward. Optimizing against this verifier closes the self-reward loop; shared trainable weights let one update move generation and scoring even with the reward gradient stopped. We call the resulting failure self-verifier capture: self-reward rises while independently measured quality falls. Matched semantic/denoising by shared/frozen
cells on Scale-RAE isolate two routes: the policy exploits on-
manifold blind spots of a frozen semantic judge, while shared updates additionally rewrite the judge. A frozen judge also becomes stale, and repeated self-refresh yields progressively weaker recoveries. The diagnosis yields a two-part repair: locking the warmed reward read blocks rewriting, while sparse, parameter-disjoint external anchors counter residual exploitation. Unanchored rollouts retain semantic reward across multiple latent noise levels, providing dense latent reward that terminal image-space judging cannot supply, while a drift-triggered controller allocates anchors to high-drift rollouts and propagates their calibration within each group. At matched budgets, the controller improves on fixed-dose anchoring, increasingly so as coverage tightens. Both the unanchored failure and the repair transfer to the higher-performing, architecturally distinct Show-o2. Across both bases, the complete method significantly outperforms a construct-matched every-rollout judge on GenEval and DPG-Bench while externally scoring only 1/16 of rollouts. At 1/32 coverage, it scores higher in all four comparisons and remains jointly non-inferior.
Read the full abstract ▾
Computer Vision
MiMaAlign: Hierarchical Micro-Macro Alignment for Physically Plausible Video Generation
Yansong Liu, Litong Gong, Weijie Li et al. (+2 more)
Text-to-
video (T2V)
diffusion models have made remarkable progress in generating high-fidelity visuals, yet their outputs still often exhibit physically implausible phenomena, including discontinuous dynamics and implausible event interactions. Existing representation
alignment methods mainly transfer generic
video features or short-range motion cues, treating physical plausibility as a single-level representation problem. To address this issue, we propose MiMaAlign, a hierarchical micro-macro
alignment framework that transfers complementary dynamics-oriented and event-oriented knowledge into different abstraction depths of a pretrained
video diffusion model. Specifically, micro-level continuous dynamics
alignment imposes constraints by aligning
diffusion features with a continuous-time motion representation learned by a Neural Ordinary Differential Equation (Neural ODE), while macro-level event-commonsense
alignment provides representations of object interactions and event semantics through an instruction-prompted
vision-
language model (VLM). With training-only auxiliary objectives, we align the complementary supervision from the two teachers with shallower and deeper generator features, respectively, enabling the generator to internalize the corresponding physical priors. Empirical evaluations show that, on CogVideoX-2B, MiMaAlign achieves a relative improvement of up to 21.0% in the Physical Commonsense (PC) score on the VideoPhy
benchmark over the baseline, and also delivers substantial gains on the highly challenging VideoPhy2
benchmark, while maintaining general
video quality on VBench and being supported by blind user comparisons.
Read the full abstract ▾
All authors Yansong Liu, Litong Gong, Weijie Li, Tiezheng Ge, Ming Zeng
Computer Vision · Cross-listed from cs.AI
XAI-Guided Fourier Domain Generalisation for Pure-Simulation Sim-to-Real Deep Reinforcement Learning
Wei Yao, Chuyao Wang, Ziwei Wang et al. (+1 more)
Deep reinforcement learning has shown strong potential for
vision-based autonomous driving, but policies trained in simulation often suffer severe performance degradation when deployed in the real world. Existing sim-to-real methods commonly address this gap through image-transfer-based domain adaptation or policy-agnostic domain randomisation. However, the former requires laborious acquisition of target-domain data and is often limited to the deployment domain, whereas the latter introduces unguided perturbations that may not reflect the underlying sim-to-real discrepancy, leading to suboptimal transfer performance. To overcome these limitations, we propose XFD, an explainable artificial intelligence (XAI)-guided
Fourier domain generalisation framework for pure-simulation sim-to-real reinforcement learning. The proposed framework extracts explainability attribution maps from simulated observations, and converts these attributions into policy-conditioned
Fourier-domain interventions. By perturbing appearance-sensitive spectral components while preserving task-defining spatial structures, XFD constructs robustness-enhancing visual domains entirely within simulation. Subsequently, the proposed robust policy optimisation guides the policy towards attending to task-critical features while suppressing sensitivity to irrelevant visual information. Extensive experiments in simulation and zero-shot real-world deployment demonstrate that XFD consistently improves traveled distance and reward over standard policy and randomly domain-randomised baseline, especially on unseen maps with cluttered backgrounds. Attribution-based analysis further proves that the proposed policy preserves more consistent task-relevant visual evidence across simulation and real-world observations.
Read the full abstract ▾
All authors Wei Yao, Chuyao Wang, Ziwei Wang, Nabil Aouf
Computer Vision
Auditing Observation-Quality Shortcuts in the Evaluation of Person-Centered Motion for Video Anomaly Detection
Chaehyun Kim and Changwon Lim
Video anomaly detection (VAD) identifies deviations from normal patterns, but such deviations do not necessarily correspond to safety-critical events. We focus on human-centered emergencies and violence and investigate whether motion-based anomaly discrimination reflects task-relevant motion information or observation-quality differences arising from person detection, pose estimation, and tracking. We introduce an observation-quality-aware evaluation framework that combines quality diagnostics, quality balancing, motion orthogonalization, and robustness analyses. On human-centered categories of UCF-Crime, quality-only features match or exceed the discrimination of multi-cue motion in the original cohort, revealing a strong observation-quality shortcut. After cardinality matching brings all 11 measured quality variables below a prespecified marginal balance threshold, multi-cue motion consistently outperforms quality-only features across Logistic Regression, Random Forest, and RBF-SVM, yielding ROC-AUC gains of 0.046-0.141. Because nonlinear quality-only prediction remains above chance after matching, we interpret this result as additional discrimination beyond measured marginal quality rather than as evidence of quality-free or causal motion understanding. These findings show that high VAD performance should not be interpreted as evidence of abnormal-motion modeling without auditing the reliability of the person-observation pipeline. Our study provides a systematic protocol for evaluating observation-quality shortcuts and for assessing whether motion-based discrimination survives explicit control of detection, pose-estimation, and tracking reliability.
Read the full abstract ▾
Computer Vision
Teaching Vision-Language Models to Decline: Support Completion Against Mirage
Alfaj Uddin Ahmed, Md. Shaown Miah, Md Arid Hasan et al. (+2 more)
Vision-
language models answer questions about images they were never shown, a failure called mirage. Such an answer can be correct without being justified by the evidence, and the difference matters wherever a model's output is checked by someone who did not see its input. We locate the failure in the training objective: cross-
entropy separates a model that reads the image from one that ignores it by only the conditional mutual information between answer and image, and no training example rewards declining. We formalise support completion, TIDE, which also shows each training question with a blank image, a same-domain wrong image and an out-of-domain image, with targets that decline, and prove that under TIDE every image-blind predictor is suboptimal by at least 1.04 nats per example, that the optimum declines exactly on the counterfactual kinds while keeping its behaviour on valid images, and that for any finite model the error of its answer-or-decline decision, declining on valid input plus answering on invalid input, is bounded below by how well its representation separates valid from invalid evidence. On Qwen2.5-VL-7B under a frozen held-out protocol with two judges, the three-condition predecessor of TIDE reduced the invalid-evidence mirage rate from 0.759 to 0.097 and the blank-image rate from 0.700 to 0.003 at a false-abstention cost of 0.126; label-free grounding raised the rate to 0.859, a likelihood-advantage contrast moved it only to 0.718 because its empty-image term carries no gradient, and grounding applied after support completion re-opened the mirage. The four-condition protocol on three base models, with seeds and component ablations, is in progress and not reported here.
Read the full abstract ▾
All authors Alfaj Uddin Ahmed, Md. Shaown Miah, Md Arid Hasan, Syed Ishtiaque Ahmed, Sayeed Shafayet Chowdhury
Computer Vision
GHOST: Geometry-Constrained Hybrid Diffusion for Open-Vocabulary 3D Scene Graph Generation
Changqun Feng, Wangxiandi Yin, Xin Hu et al. (+4 more)
3D scene graph generation requires understanding both object semantics and metric spatial relations. However, existing open-vocabulary methods often struggle with spatial predicates because
language similarity reflects semantic co-occurrence rather than explicit geometry. We propose GHOST, a Geometry-constrained Hybrid
diffusion framework for Open-vocabulary 3D Scene graph generaTion. GHOST jointly models continuous node geometry and discrete edge predicates, following a geometry-to-constraint-to-predicate denoising process to ensure geometrically consistent relation prediction. It introduces
language-grounded constraint induction, which maps predicate
embeddings to differentiable
geometric constraints that guide both layout refinement and predicate reweighting. A Geometry-
Topology Collaborative Denoiser further couples node and edge
reasoning through noise-adaptive interaction and orientation-aware modeling. Experiments on 3DSSG demonstrate state-of-the-art performance in both closed-set and open-vocabulary settings, including over 17-point gains over RelationField on spatial predicates.
Read the full abstract ▾
All authors Changqun Feng, Wangxiandi Yin, Xin Hu, Ruiting Dai, Feiyang Yang, Jielei Wang, Tao He
Computer Vision
Not Every Cue Is Evidence: Cue Supportability for Forensic Distillation in Face Forgery Detection
Binjia Zhou, Dawei Luo, Shuai Chen et al. (+7 more)
Explainable face forgery detection requires not only correct judgments but also forensic evidence supported by the input image. However, training cues are typically derived from the forgery generation pipeline. Since images available to the model are of limited quality and resolution, it is difficult to observe the fine-grained traces these cues refer to. Training methods that further supply such cues as privileged supervision exacerbate the problem, as the student is taught to articulate evidence it cannot perceive. The student consequently resorts to templated or over-generalized expressions, giving rise to the correct-answer, wrong-evidence (CAWE) failure. We propose a dual supportability audit that filters privileged cues before
distillation. Each candidate cue must be predictable from the remaining privileged context and causally tied to high-attribution visual regions in the current image. Only cues satisfying both conditions receive substantial
distillation weight, preventing unsupported details from leaking into the student. Experiments across eight
benchmarks show that our method maintains or improves detection accuracy while significantly raising explanation faithfulness compared to existing paradigms.
Read the full abstract ▾
All authors Binjia Zhou, Dawei Luo, Shuai Chen, Feng Xu, Haoyuan Li, Jia wen Seow, Jiachi Wang, Jie Lei, Zunlei Feng, Yijun Bei
Computer Vision
Tikhonov-Stabilized Bezier Representation Forecasting for Training-free Diffusion Acceleration
Lei Zhu, Mujie Lin, Ruochong Zheng et al. (+5 more)
Diffusion models, particularly
Diffusion Transformers, achieve strong image and
video generation quality but remain expensive at inference time due to repeated denoiser evaluations. Feature caching offers a deployment-friendly acceleration strategy by reusing or predicting intermediate representations without retraining the generator or modifying the sampler. However, existing cache-then-forecast methods often rely on Taylor-style finite-difference extrapolation, which can become unstable over longer cache intervals, and typically predict local module outputs whose errors may accumulate through subsequent denoiser blocks. We propose BeziCast, a training-free
diffusion acceleration framework that forecasts output-proximal denoising representations using low-order B'ezier trajectories. BeziCast estimates B'ezier control points via Tikhonov-stabilized fitting, providing a smooth, capacity-controlled temporal parameterization that avoids explicit high-order derivative estimation and decouples trajectory capacity from cache-query density. To moderate aggressive extrapolation, we further derive equivalent forecasting weights and introduce a convex-hull-inspired guardrail that detects high-risk predictions and softly pulls them toward a conservative simplex-projected estimate. Extensive evaluations across advanced image and
video diffusion models demonstrate the effectiveness of BeziCast. In particular, BeziCast accelerates FLUX.1 by up to (4.79\times) and HunyuanVideo by up to (4.11\times), while preserving substantially better generation quality than competing acceleration baselines.
Read the full abstract ▾
All authors Lei Zhu, Mujie Lin, Ruochong Zheng, Guangyi Wang, Hao Li, Peng Jin, Chang Liu, Jie Chen
Computer Vision
Pixels or Code? Benchmarking Instruction-Guided Visual Artifact Operation
Shun Guo, Shuizhou Shi, Boao Wang et al. (+4 more)
Visual artifacts such as charts, technical diagrams, webpages, and document pages underpin scientific, commercial, and engineering workflows. Editing these artifacts requires direct modification, dependency-aware updates, preservation of unaffected content, and compliance with domain constraints. We define this capability as Visual Artifact Operation (VAO). Prior
benchmarks test parts of this problem, including cascading updates in charts. We study direct edits, dependent updates, and preservation across visual domains and pixel, code, and
agent paradigms under one image-only protocol. We introduce Visual Artifact Operation Bench (VAO-Bench), comprising 1,573 manually screened tasks across nine artifact categories and three difficulty levels. VAO-Bench uses executable artifacts and deterministic rendering to trace reference transformations and annotate direct edits, dependent updates, and preservation constraints. All systems receive the same source image and instruction; we evaluate pixel-space editors, code-generating
multimodal systems, and tool-using
agents on direct-edit accuracy, dependency-aware update, content preservation, domain validity, and visual quality. Experiments show that executable-code systems better support dependent updates, while pixel-space editors offer faster execution and strong visual quality. Explicitly stating dependencies improves both paradigms, identifying implicit-update inference as a central bottleneck. VAO-Bench provides a reproducible foundation for dependable visual artifact editing across generation paradigms.
Read the full abstract ▾
All authors Shun Guo, Shuizhou Shi, Boao Wang, Wei Xiong, Siyu Sun, Jiangtong Li, Jie Li