Computer Vision · Cross-listed from cs.RO
Think Only When Needed: Prompt-Authority Control for Selective Slow-Path Intervention in Vision-Language-Action Manipulation
Zhiruo Zhou, Zelin Li, Xiwen Chen et al. (+4 more)
Retrieval can efficiently and effectively augment a frozen
vision–
language–action (VLA) policy without retraining, yet retrieved text becomes a control intervention once it enters the executed
prompt. In a matched audit, raw appended text reduces mean success from 92.47% to 3.00%, while meaningful and length-matched meaningless appends both fail on all 500 states. This result identifies
prompt-form collapse: changing the instruction form, rather than adding useful semantics, can dominate execution. We introduce TOWN-VLA (Think Only When Needed), a
prompt-authority interface that separates candidate generation from permission to alter the policy input. A fixed compatibility rule authorizes a canonical compact instruction; otherwise, the interface restores the original Base
prompt exactly. Across 900 audited routes, every route follows this contract: 525 routes recover Base with matching hashes, and all 375 authorized
prompts preserve the task signature. On a matched $4\times7$ LIBERO-Plus evaluation with 10{,}030 episodes per method, success rises from 69.5% to 73.1% ($+362$ episodes; 95% CI 1.89–5.45 points), improving on six perturbation axes and all four suites. On a physical PiPER arm with a frozen \pizerofive{} checkpoint, success rises from 52.7% to 78.7% over 150 trials per method ($p=3.16\times10^{-6}$).
Prompt authority is enforceable for a frozen controller; oracle-free admission calibration is the next deployment target.
Read the full abstract ▾
All authors Zhiruo Zhou, Zelin Li, Xiwen Chen, Jiazhuo Li, Chenwei Wang, Huiming Chen, Xiaojun Zhu
Computer Vision
BenthicDINO: Physics-Informed Self-Distillation for View-Invariant Side-Scan Sonar Representations
Taqi Hamoda, Hayat Rajani and Nuno Gracias
Automated interpretation of side-scan sonar (SSS) imagery is traditionally hindered by view-dependence, as recorded intensities inextricably conflate intrinsic seabed reflectivity with transient acquisition geometries and acoustic artifacts. To address this, we introduce BenthicDINO, a physics-informed self-supervised learning framework that decouples acquisition dynamics from seabed representations. Utilizing a ConvNeXt-v2 backbone, physics-guided augmentations, and an HSIC penalty, BenthicDINO explicitly enforces feature invariance to physical viewing parameters like slant range and incidence angle. Evaluations demonstrate exceptional data efficiency - achieving near-peak performance in benthic habitat mapping with only 10% of available labels - and strong cross-domain generalization, outperforming fully supervised baselines on the AI4Shipwrecks
benchmark using frozen representations.
Read the full abstract ▾
Computer Vision
Bee Detection and Tracking at Hive Entrance using YOLO11 and ByteTrack
Thi Thu Thao Nguyen and Johannes Reschke
This work presents an automatic bee entrance monitoring system based on YOLO11 transfer learning and the ByteTrack tracking
algorithm. The study investigates the influence of data augmentation, backbone freezing, and tracker parameter optimization on the detection and counting of small, fast-moving bees. The detector with progressive backbone unfreezing strategy achieved about 97.0% precision and 98.7% mAP50, while providing more stable convergence than full fine-tuning. Experiments also showed that light augmentation outperformed heavy augmentation. For tracking, ByteTrack parameters were optimized to improve trajectory continuity under low-confidence detections. On an independent 25 FPS side-view
video, the optimized YOLO11-ByteTrack system correctly counted 43 of 47 incoming bees (91.5%) and 7 of 30 outgoing bees (23.3%). Error analysis showed that most counting errors were caused by missed detections due to rapid bee motion and motion blur, while tracking failures became less frequent after parameter optimization. Overall, the results indicate that moderate augmentation, progressive backbone unfreezing, and ByteTrack tuning improve the reliability of automatic bee entrance monitoring under realistic recording conditions.
Read the full abstract ▾
Computer Vision
Learning Spherical Occupancy Profiles for Multi-View 3D Reconstruction and Generation
YiHsuan Tsai
We study spherical occupancy profiles-the ray-wise occupancy probability profiles P(r) = T(r) o(r) distilled from multi-view 3D Gaussian reconstructions-as a unified intermediate representation for both discriminative and generative 3D reconstruction from images. On a 999-object subset of Google Scanned Objects with 48 turntable views each, we train (i) a discriminative per-ray decoder that injects global view-averaged and ray-specific image evidence into a FiLM-conditioned profile head, reaching median soft depth error 0.035 (normalized) on an independent 90-object test split, and (ii) a generative pipeline built on a profile VAE and a latent
diffusion model, which supports unconditional sampling that matches the reconstruction
manifold and image-conditioned multi-solution reconstruction whose per-object solution spread is quantifiable and tunable via classifier-free guidance. We further analyze the morphology of predicted profiles: post-hoc power sharpening and a learned sharpening target both recover ground-truth profile width without degrading depth, exposing a monotonic width-peak frontier in the L1-per-ray loss family and motivating a principled redefinition of morphology gates. Real-photo validation on two DTU scenes confirms the pipeline transfers to non-synthetic input. Our results suggest that ray-wise occupancy profiles offer a compact, learned, and uncertainty-aware interface between multi-view reconstruction and generative priors.
Read the full abstract ▾
Computer Vision
Toward a Foundation Plug-and-Play Prior for Computed Tomography Reconstruction via a Multimodal Diffusion Model
Haley Duba-Sullivan, Patxi Fernandez-Zelaia, Obaidullah Rahman et al. (+1 more)
Computed
tomography (CT) throughput is limited by scan time, which grows with both the number of projections acquired and the detector integration time for each projection. Reconstructing high-quality volumes from sparse-view or low-dose measurements therefore depends on using an informative prior, typically a neural network trained for one specific scan setting and retrained whenever the modality, geometry, or material changes. We investigate whether a single
diffusion model trained across several imaging domains can instead serve as a reusable prior for heterogeneous CT reconstruction problems. We evaluate the proposed method using the same
diffusion visual
transformer model and normalized denoising strength on three
datasets that differ in modality, beam geometry, material, and degradation type, spanning additively manufactured metal parts and concrete microstructure imaged with cone-bean X-ray CT and parallel-beam
neutron CT respectively. The proposed method improves upon analytic reconstructions in all three cases, demonstrating transferability across the evaluated problems and providing a step toward a reusable foundation prior for heterogeneous CT reconstruction.
Read the full abstract ▾
All authors Haley Duba-Sullivan, Patxi Fernandez-Zelaia, Obaidullah Rahman, Amirkoushyar Ziabari
Computer Vision
EchoWM: Open and Enterable Omnimodal World Models
Songchun Zhang, Yaowei Li, Junhao Zhuang et al. (+19 more)
We present EchoWM, an omnimodal world model for enterable generative media that responds to continuous
navigation while jointly generating 720p
video, environmental sound, music and
speech. We organize interaction around camera intent: in first-person scenes, it specifies observer motion, while in third-person scenes, camera–character dynamics are learned from data without view-specific controllers. Discrete commands and continuous poses are mapped to a shared metric-scale relative 6-DoF trajectory, with
dataset-level calibration preserving motion magnitude across heterogeneous data. To jointly learn audio-visual generation and trajectory control, we construct a complementary data engine and adopt progressive training followed by autoregressive post-training for long-horizon generation. Extensive evaluations show that \model achieves strong trajectory following and high visual quality on public world-model
benchmarks, supporting both first- and third-person interaction across varied subjects, and maintaining synchronized environmental sound and
speech over long-horizon generation.
Read the full abstract ▾
All authors Songchun Zhang, Yaowei Li, Junhao Zhuang, Weiyang Jin, Haoyu Wang, Xin Lu, Yilang Sun, Shiyi Zhang, Haoran Li, Xiaoxiao Ma, Yuming Li, Yijun Liu, Yaofeng Su, Yanwen Ma, Haoyu Wu, Zihan Su, Yue Ma, Lvmin Zhang, Haoyang Huang, Zeyue Xue, Anyi Rao, Nan Duan
Computer Vision · Cross-listed from cs.LG
A Comparative Study of Label-free Representation Quality Metrics in Deep Learning
Daniel Richards Arputharaj, Daniel Jönsson and Gabriel Eilertsen
We present a comparative study of label-free metrics for assessing the quality of representations in deep neural networks to understand their reliability under a wide variety of configurations. We group existing label-free metrics into three families based on their construction and analytically establish connections between metrics within the same family. We then characterise the sensitivity of spectral metrics through controlled synthetic experiments. Finally, all label-free metrics are evaluated against downstream task accuracy across a diverse set of 260
vision models on six
datasets spanning generic object classification, fine-grained object classification, scene recognition and geospatial task, stratifying results by architecture class and training objective. We find that intrinsic dimensionality (ID) is the most reliable predictor among the metrics considered. However, the reliability of all metrics, including ID, is moderated by architecture class and training objective. Our results provide a clearer understanding of what label-free representation quality metrics measure, when they are reliable, and how to interpret them in practice.
Read the full abstract ▾
Computer Vision
Neighbor-Aware View Synthesis for Restoring Missing Views in Light-Field Camera Arrays
Sakshi Goel, Ayush Goyal, K S Venkatesh et al. (+1 more)
In light-field (LF) imaging systems, dense spatial sampling from a camera array enables powerful post-capture capabilities such as refocusing and depth estimation. However, real-world LF capture is often affected by hardware malfunctions, where one or more cameras in the array fail, leading to missing sub-aperture images and degraded reconstruction quality. This paper addresses the problem of defective or missing view restoration in light-field camera arrays. We propose a novel generative framework that synthesizes the absent views by exploiting information from a carefully selected subset of neighboring cameras. These selected images, along with a positional encoding map indicating both their locations and the desired target view, are fed into a conditional Generative
Adversarial Network (cGAN) trained to generate the missing viewpoint in a geometrically consistent manner. Extensive experiments on synthetic and real-world LF
datasets demonstrate that our method produces visually plausible and photometrically accurate reconstructions, outperforming baselines for view interpolation both quantitatively and qualitatively. The proposed framework thus offers a robust and efficient solution for fault-tolerant light-field image acquisition.
Read the full abstract ▾
All authors Sakshi Goel, Ayush Goyal, K S Venkatesh, Koteswar Rao Jerripothula
Computer Vision
BenthicFlow: Generating Extensible Underwater Environments via Flow Matching
Joaquín Figueira, Camile Lendering, Manfred Gonzalez-Hernandez et al. (+3 more)
Computer
vision applications for 3D scene understanding in
underwater environments remain challenging due to the lack of high-quality 3D data and the inability of surface-trained models to generalize to
underwater scenes. To address this challenge, an emerging trend is to employ generative models to close the data domain gap. However, existing methods assemble large scenes by stitching independently generated tiles post hoc with separately trained models, while demonstrating heterogeneous landscapes only within individual survey sites. We introduce BenthicFlow, a unified framework based on a single conditional flow-matching model that jointly generates aligned textures and depth maps. A MultiDiffusion-inspired sampling procedure reconciles overlapping windows throughout the generative trajectory, enabling spatially extensible RGBD mosaics without a separate stitching model. The generated mosaics are subsequently lifted into explicit 3D benthic environments using surface-aligned Gaussian surfels. Experiments across geographically distinct survey sites demonstrate that BenthicFlow preserves site-specific appearance while generating coherent, large-scale 3D scenes that closely match the target distributions. Code and trained models are available at https://github.com/jacomof/BenthicFlow.
Read the full abstract ▾
All authors Joaquín Figueira, Camile Lendering, Manfred Gonzalez-Hernandez, Giacomo D'Amicantonio, Erkut Akdag, Egor Bondarev
Computer Vision · Cross-listed from cs.CL
CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension
Abhilash Nandy, Rahul Seetharaman, Aman Bansal et al. (+5 more)
Large-scale
vision-
language models (VLMs) have demonstrated remarkable versatility across a wide range of
multimodal tasks. However, understanding humor remains challenging because humorous content often depends on subtle interactions among entities, events, context, and implicit relationships across image and text modalities. These interactions can involve complex chains of
reasoning that are difficult to capture through conventional prompting or linear chain-of-thought
reasoning. In this work, we propose CaRGo-T (Causal
Reasoning Graph-of-Thought), a
reasoning framework that represents the causal and contextual relationships underlying
multimodal humor as a lightweight graph-based
reasoning structure. The graph is serialized into a code-based representation generated by a VLM, which can subsequently be interpreted by the same or a different VLM to produce the final prediction in zero-shot or in-context learning settings. We evaluate CaRGo-T on humor understanding and humor detection across four
datasets spanning diverse forms of comedic content, including satire, sarcasm, and memes. Experiments with state-of-the-art commercial and open-source VLMs show that CaRGo-T consistently improves performance over existing
reasoning-based baselines, achieving gains of approximately 1-20% on humor understanding and 1-3% on humor detection. Further analysis using mutual information indicates that the
reasoning representations produced by CaRGo-T contain more information relevant to the target output than those generated by baseline
reasoning approaches. Code is available at https://github.com/abhi1nandy2/CaRGo-T.
Read the full abstract ▾
All authors Abhilash Nandy, Rahul Seetharaman, Aman Bansal, Rounak Saha, Manav Nitin Kapadnis, Millon Madhur Das, Pawan Goyal, Niloy Ganguly
Computer Vision
An end-to-end-trained vision-language model for native-language prostate pathology report generation
Christian Grashei, Fabian Gülhan, Maximilian Legnar et al. (+4 more)
Prostate cancer is among the most frequently diagnosed malignancies worldwide, and structured reporting of each biopsy core burdens pathologists. Existing tools frame this as classification, leaving pathologists to assemble coherent reports, while many slide-level
vision-
language models rely on English-centric encoders that transfer poorly to other clinical
languages. We present a slide-level framework generating prostate biopsy reports that is
language-independent by construction: tokenizer and model are trained from scratch, demonstrated here in German. To address paired-data scarcity, an automated pipeline uses a locally deployed large
language model to split composite reports into core-specific image-text pairs, yielding 17,344 pairs from 2,402 historical cases without manual annotation. Evaluated for clinical attributes rather than linguistic similarity, the model achieves 96.2% F1 for malignancy detection and 65.2% for Gleason grading, competitive with an FDA-cleared classifier. Grading is further validated on three external cohorts with latent-space augmentation. Institutions can thus train native-
language reporting models on their own archives.
Read the full abstract ▾
All authors Christian Grashei, Fabian Gülhan, Maximilian Legnar, Fabian Stögbauer, Cleo-Aron Weis, Carolin Mogler, Peter Schüffler
Computer Vision
How Merge-Tolerant Are Vision Transformers for Wheat Phenotyping?
Simon Ravé, Pejman Rasti and David Rousseau
Vision-based wheat phenotyping requires repeated measurements under deployment constraints, from growth-stage recognition to wheat-head counting and organ
segmentation. Plain
Vision Transformers (ViTs) provide a common architecture for these tasks, but quadratic
attention limits high-throughput and edge inference. Training-free
token merging is attractive because it can be inserted into trained models without retraining. We provide a systematic
benchmark of ToMe and Mutual Pair Merging across growth-stage classification, wheat-head detection, and wheat-organ
segmentation, measuring task quality, throughput,
token count, and peak GPU
memory, with additional Raspberry Pi 5 measurements. The
benchmark reveals a clear hierarchy: classification is highly merge-tolerant, while detection and
segmentation are constrained by repeated instances, thin organs, dense boundaries, reconstruction, and runtime overhead. Optimized
attention backends can erase apparent speedups, so deployment value must be profiled on the target runtime rather than inferred from
token count.
Read the full abstract ▾
Computer Vision
MIVIFI: Bridging Perspective and Fisheye Domains for Training Multi-View Fisheye Image Generation Models
Matthias Neuwirth-Trapp, Begüm Altunbas, Jiayi Wang et al. (+4 more)
Achieving 360{\deg} coverage is critical for the visual perception systems of autonomous vehicles. Fisheye cameras offer a cost-effective solution by enabling full surround coverage with as few as two sensors. However, existing multi-view fisheye
datasets are limited, and synthesizing rare corner cases typically requires computationally expensive 3D simulations, hindering the training. While generative models have achieved significant success in standard perspective imagery, their application to wide-angle distortion remains unexplored. In this work, we formally introduce the novel problem of multi-view fisheye image generation conditioned on volumetric semantic representations and present two distinct methods. We first propose SyntheOcc-FE, which adapts the SyntheOcc architecture to fisheye data. While effective, this method is constrained by the scarcity of fisheye
datasets, which limits its generalization. To overcome these limitations, we propose our second method, MIVIFI (multi-view fisheye), which leverages cross-domain learning with Equirectangular Projections. By bridging the gap between
dataset domains using KITTI-360 fisheye images alongside nuScenes multi-view standard images, our approach enables high-fidelity manipulation of scene content. This framework enables the structural modification of semantic occupancy inputs to introduce or eliminate specific actors and facilitates the rendering of diverse meteorological conditions and illumination scenarios absent in the limited fisheye
datasets. Quantitative and qualitative experiments demonstrate that our methods achieve robust photorealistic multi-view fisheye image generation and highlight the specific advantages of our cross-domain strategy for handling data scarcity.
Read the full abstract ▾
All authors Matthias Neuwirth-Trapp, Begüm Altunbas, Jiayi Wang, Yan Xia, Maarten Bieshaar, Xinyu Huang, Daniel Cremers
Computer Vision · Cross-listed from cs.RO
Pointing-VLA: Typed Spatial Grounding Interfaces for Vision-Language-Action Manipulation
Xiwen Chen, Zelin Li, Zhiruo Zhou et al. (+3 more)
Vision-
language-action (VLA) models often expose spatial grounding through autoregressive text coordinates or opaque action
tokens, creating brittle interfaces between
multimodal reasoning and
robot execution. We present Pointing-VLA, a typed hidden-state spatial readout built on
Embodied-R1. Geometry-specific heads predict normalized points, object-functional grounding (OFG) heatmaps, and visual trajectories without serializing geometry as text. For the evaluated Bridge/WidowX and physical pick-place deployments, an explicit execution contract assigns PICK to source-conditioned OFG and PLACE to Pointing, providing direct stage-aligned spatial targets. Pointing-VLA achieves SOTA performance on Bridge/WidowX, averaging 72.9% across the evaluated four-task set without Bridge-specific finetuning under collision-enabled CuRobo execution. Pointing and OFG show complementary strengths across native and cross-
dataset evaluations. The OFG/contact readout transfers to NORA-1.5, preserving or improving success while reducing recorded controller time by more than 20$\times$; typed heads are also 6.68–6.90$\times$ faster than
Embodied-R1 text decoding on a shared external suite. When integrated as spatial guidance for a $\pi_{0.5}$ action policy, Pointing-VLA raises autonomous real-
robot success from 52.7% to 80.7% across three visual contexts. These results establish typed spatial readouts as an efficient, inspectable interface between
embodied reasoning and
robot execution.
Read the full abstract ▾
All authors Xiwen Chen, Zelin Li, Zhiruo Zhou, Huiming Chen, Chenwei Wang, Xiaojun Zhu
Computer Vision
Simulate, record, verify: A language-portable framework for muscle-grounded articulatory QA (extended version)
Seungho Eum, Shantong Sun and Unsang Park
Articulatory data describe what the tongue looks like, but geometry alone does not explain the muscle-level why behind a configuration or how to move it toward a target posture. We present a simulator-based framework that constructs this supervision from controlled biomechanical inputs. Each simulated configuration is stored with its generating muscle state and
geometric properties in a structured fact record. Gold answers are derived before linguistic realization, and every naturalized output is checked against its source record, so the same records support new
languages and question types without re-simulation. We instantiate the framework as 3DTongueQA using the ArtiSynth Badin tongue model. From 295,115 meshes, we construct 891,156 record-checked QA instances per
language, and the checker detects over 96.9% of injected corruptions. Korean and Spanish realizations and a new question type are added from the same records, confirming portability. A SpiralNet++-Qwen3 probe reaches 62.9 Muscle EM, above 44.0 for nearest-neighbor transfer and 7.2 for a zero-shot commercial LLM given the same mesh, while mesh shuffling drops it to near zero. The supervision is learnable, useful beyond
retrieval, and grounded in paired geometry. It reflects simulator-defined states rather than measured physiology. Code, templates, and QA sets are available at https://github.com/esh0504/muscle-grounded-qa.
Read the full abstract ▾
Computer Vision
Bridge Damage Detection from Low-Light UAV Imagery via Degradation-Aware Mixture-of-Experts Enhancement
Hu Wang, Hongxu Pu, Zhiqi Hu et al. (+2 more)
Poor illumination obscures small, low-contrast defects in
UAV bridge imagery, reducing the reliability and operational flexibility of automated inspection. This paper investigates whether degradation-aware image restoration can improve bridge damage detection under low-light conditions and transfer from synthetic degradations to real inspection scenes. We propose DaL- MoE, a detector-agnostic restoration front end trained with an ISP-aware low-light synthesis pipeline and equipped with degradation-aware guidance estimation and complementary experts for noise suppression, color adjustment, and structural-detail recovery. On paired synthetic data, DaL-MoE achieves 23.12 dB PSNR and 0.8482 SSIM, increasing YOLOv11m box mAP50 from 0.3097 to 0.4923 and mask mAP50 from 0.2281 to 0.3529. On real low-light
UAV imagery without paired normal-light references, sim-to-real evaluation shows improved defect visibility and more complete detections than direct inference on raw low-light inputs. Future work will develop low-light-aware bridge damage detectors with stronger cross-scene generalization across bridge sites, imaging conditions, and illumination levels.
Read the full abstract ▾
All authors Hu Wang, Hongxu Pu, Zhiqi Hu, Fangzhou Lin, Wang Wang
Computer Vision
Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models
Fan Xu and Luis A. Leiva
Composed Image
Retrieval (CIR) is an emerging paradigm in content-based image
retrieval that enables users to formulate compositional queries by combining a reference image with an auxiliary modality, usually text-based. This approach supports fine-grained search where the target image shares structural elements with the user-provided image while incorporating the modifications specified by the auxiliary text. Conventional CIR methods rely on
multimodal fusion to combine visual and textual features into a joint query
embedding, which requires training modules that align composed queries with the targets. In this work, we propose PeFuse (for pseudo-fusion), a training-free framework that leverages pretrained
Diffusion Models and
Multimodal Large
Language Models to bridge modalities via generative conversion. We introduce two novel strategies: uni-directional and bi-directional conversion, which convert CIR into four single-modality
retrieval problems. These methods reformulate CIR as either intra-modal or cross-modal single-query
retrieval tasks, bypassing the need for dedicated task-specific training. Extensive experiments on standard
benchmarks demonstrate that converting CIR into text-to-image
retrieval tasks is more effective than alternative conversion strategies, achieving competitive or superior performance compared with state-of-the-art methods, while maintaining high flexibility thanks to replaceable components of the conversion pipeline. These results highlight the effectiveness of the pseudo-fusion paradigm for zero-shot CIR. Our code is publicly available at: https://github.com/StevenXuf/PeFuse4CIR.
Read the full abstract ▾
Computer Vision
Loopy: Seamless Video Loop Generation via Anchored Looping Shift of Positional Embedding
Haotian Dong, Wenjing Wang, Chen Li et al. (+3 more)
Looping
videos are essential for practical applications such as web graphics, game development, and social media. However, existing approaches typically fail to generate high-quality looping
videos due to the neglect of how
video generation models perceive temporal order and how this relates to the looping behavior. In this work, we are the first to reveal that position
embedding at different
attention layers within DiT exhibits varying levels of positional control, with the most pronounced layer acting as an anchor. We formulate this anchored layer as the reference point of the looping
video, offering strong contextual priors for the remaining layers to facilitate the generation of seamless and coherent
video content. Based on this insight, we propose an anchored position
embedding shifting strategy that applies layer-specific shift lengths according to each layer's temporal control effect, effectively transforming DiT's temporal perception from a straight line to a circle. Leveraging this strategy, we develop a general framework, Loopy, for high-quality looping
video generation, supporting both RGB and RGBA
videos, while also enabling advanced AIGC features such as identity control and style transfer. Experiments demonstrate that our approach significantly improves temporal consistency and visual fidelity in generated looping
videos. The released model is available on our website: https://donghaotian123.github.io/Loopy.
Read the full abstract ▾
All authors Haotian Dong, Wenjing Wang, Chen Li, Jing Lyu, Xin Wang, Di Lin
Computer Vision
Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?
Xiwei Liu, Yulong Li, Xinlin Zhuang et al. (+5 more)
Vision-
language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial
reasoning remain poorly understood. It is underexplored whether spatial
reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including
token ablation, layer-wise probing,
attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-
reasoning process in which object-aligned
tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of
attention heads mediates the causal effects of both localization and spatial
reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a
token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.
Read the full abstract ▾
All authors Xiwei Liu, Yulong Li, Xinlin Zhuang, Xuhui Li, Zhixiang Lu, Haolin Yang, Imran Razzak, Yutong Xie
Computer Vision · Cross-listed from cs.AI
From Generation to Simulation: How Far Are World Models from Being True Simulators?
Tong Wang, Huan Deng, Mucheng Yang et al. (+3 more)
With the rapid progress of
diffusion models and large-scale
video generation, generative world models are increasingly expected to replace traditional simulators, including physics engines, game engines, and reinforcement-learning environments. Yet the remaining distance from generation to simulation lacks a systematic assessment. We present a capability-based study using an external yardstick: eight capabilities of a traditional simulator, namely asset construction, physics engine, interaction, controllability, stability, state feedback, diversity, and evaluation metrics. We trace three main technical routes–latent dynamics,
video generation, and joint-
embedding prediction–and map exactly 200 representative works published from 2018 to June 2026 onto these capabilities. Our analysis shows that world models have achieved functional substitution in interaction and controllability for specific scenarios, but remain short of traditional simulators in formal guarantees of physical laws, structured state feedback, and reproducible long-horizon
evolution. State feedback is the most neglected cross-route shortcoming: only 6 of 163 implementation papers expose a runtime interface for querying entity states or physical parameters. We identify six research directions: formalized physics, a unified action interface, first-class state feedback, long-horizon stability, downstream-utility evaluation, and cross-route hybridization. Project page: https://github.com/AtongWang/world-model-simulators
Read the full abstract ▾
All authors Tong Wang, Huan Deng, Mucheng Yang, Yang He, Xiaohui Kuang, Gang Zhao
Computer Vision
Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asia
Burak Satar, Zhixin Ma, Cheng Yu-Tong et al. (+3 more)
Cultural understanding in
video means more than recognizing what is visible; it requires grasping the symbolic and temporal significance of cultural concepts. We decompose this into three abilities: naming what a concept symbolizes, visually recognizing it on
video, and locating its sub-events in time. Existing
video-cultural
benchmarks tend to test what is seen, collapsing these three abilities into a single score that hides the bottleneck. We introduce the Cultural Moment
Benchmark (CMB): 306 expert-curated concepts from seven countries in Southeast Asia across five categories. We evaluate each concept through three stages, one per ability. Given a description, Stage 1 (S1) selects from four candidate concept names, Stage 2 (S2) selects from four candidate
video moments, and Stage 3 (S3) predicts the start and end times of the moment in a
video. To keep each stage focused on a distinct ability, we use three design choices: semantic-similarity distractors (S1, S2), unlabeled
video moments (S2), and free-form localization on a different example
video (S3). Across six
vision-
language models, failure modes vary by ability and modality. i) Even the strongest closed-source models score below 30% when all three stages must be correct; ii) The three abilities do not fully cascade: naming a concept correctly helps half the models recognize it on
video, but recognizing it has little effect on locating the sub-event in time; iii) Audio is complementary, redundant, or distracting depending on the concept, more often distracting in non-Latin-script countries; removing both audio and subtitles hurts Games and Music the most. Our 14-rater human study shows that even Expert raters score below chance on concepts from a neighboring country, indicating that CMB requires country-specific cultural knowledge. CMB acts as a diagnostic harness, attributing failures to a specific ability or modality.
Read the full abstract ▾
All authors Burak Satar, Zhixin Ma, Cheng Yu-Tong, Huy Hoang Tran, Phuong Anh Nguyen, Chong-Wah Ngo
Computer Vision
When the Edit Changes the Patient: Measuring Identity Preservation in Counterfactual Retinal Images
Andrea Posada, Wenke Karbole, Bach Ngoc Doan et al. (+9 more)
Counterfactual medical image generation aims to modify an existing image to reflect a hypothetical scenario in which certain characteristics of the imaged subject are altered, while keeping their identity fixed. Most existing works repurpose established image editing methods, which do not directly supervise identity preservation. Instead, they assume that identity is implicitly preserved by anchoring generation to the source image. This assumption is rarely tested and may fail in domains where biometric cues are subtle, such as retinal optical coherence
tomography (OCT). In this work, we explicitly measure identity preservation for three groups of text-conditioned editing methods - source-anchored, structured-
prompt, and paired-training - using referee classifiers,
embedding alignment scores, and a blind reader study. We find that all methods produce high-quality OCT images with comparable editing success, yet their identity preservation differs markedly. Source-anchored editing frequently alters the depicted subject, while paired-training preserves it best. We argue that future work on medical counterfactual generation must explicitly measure and report identity preservation alongside image realism and editing success.
Read the full abstract ▾
All authors Andrea Posada, Wenke Karbole, Bach Ngoc Doan, Alexander Weers, Solmaz Abdolrahimzadeh, Maria Patsiamanidi, Kahkashan Haider, Vaishali Khare, Daniel Rueckert, Andrew Lotery, Sobha Sivaprasad, Martin J. Menten
Computer Vision
AnaDiffusion: Anatomically Compositional Latent Diffusion for Controllable 3D Brain MRI Generation
Huiwen Han, Lulin Liu, Bangya Liu et al. (+8 more)
3D
brain MRI generation has made significant advances in medical imaging, simulation, and controllable anatomical analysis. However, existing generative models typically synthesize 3D volumes monolithically, often overlooking regional anatomical structures and limiting local controllability. To address these limitations, we introduce AnaDiffusion, an anatomically compositional latent
diffusion framework that factorizes the generation process into distinct, anatomically meaningful regions, followed by part-to-whole assembly and global refinement. Our approach first trains part
diffusion models to capture local structural priors. We then inject an assembled anatomical composite of the parts into the whole-
brain latent representation and continue denoising. This mechanism enables the model to resolve global context while preserving the injected anatomy. As a result, AnaDiffusion produces both explicit part assets and a globally coherent volume, thereby enabling controllable part editing without requiring subject-specific dense
segmentation maps at inference time while maintaining consistent part-to-whole
brain structure. On the subject-disjoint ADNI test split, AnaDiffusion achieves the lowest FID across the whole
brain, left and right hemispheres, cerebellar-brainstem complex, and seam regions. It also achieves the best cerebellar and second-best ventricular and brainstem absolute Cohen's d values among the evaluated methods. In localized editing experiments, paired MS-SSIM demonstrates high target transfer and off-target preservation, supporting controllable part replacement with minimal unintended anatomical alterations.
Read the full abstract ▾
All authors Huiwen Han, Lulin Liu, Bangya Liu, Yuanhao Cai, Nuo Chen, Xiaoqing Wang, Ziqian Xie, Chenyu You, Shuiwang Ji, Degui Zhi, Zhiwen Fan
Computer Vision
Misanthrope: A Privacy-Preserving Keypoint Detector
Francesco Vultaggio, Predrag Djindjic, Markus Gerke et al. (+2 more)
Image matching is a core component of applications such as Simultaneous Localization and Mapping (SLAM), Visual Localization, and Structure from Motion (SfM). However, the local image features central to this task are vulnerable to inversion attacks, which enable adversaries to reconstruct
privacy-sensitive scene content from local features. These attacks pose a particular threat in distributed computing scenarios where the pre-computed features leave edge devices to be processed by remote servers. In this work, we introduce Misanthrope, a novel
privacy-preserving keypoint detector trained through self-
distillation to avoid detecting keypoints on people—a predominant source of
privacy-sensitive content in most localization scenarios—thus mitigating inversion attacks at the source rather than through post-hoc obfuscation. We demonstrate how inverted images from traditional feature detection pipelines can be used to detect and re-identify people in the scene, while Misanthrope is able to mitigate these attacks. Furthermore, Misanthrope maintains image matching performance on par with the state of the art and even surpasses it in challenging settings where people act as distractors, such as phototourism and in-the-wild odometry. On the Image Matching Challenge 2021 Phototourism test set, Misanthrope is the top-performing sparse feature extractor in 7 out of 9 scenes. We make our model and its evaluation script available here: https://github.com/fratopa/misanthrope
Read the full abstract ▾
All authors Francesco Vultaggio, Predrag Djindjic, Markus Gerke, Sebastian Tschiatschek, Phillipp Fanta-Jende
Computer Vision
Coarse Indexing, Fine Evidence: Decoupling Temporal Granularity in Long-Video RAG
Zhe Jin, Zhimin Lin, Bin Zheng et al. (+2 more)
Graph-based
retrieval-augmented generation (RAG) provides a scalable paradigm for long-
video understanding, but existing systems typically inherit a fixed temporal granularity from
video segmentation when constructing their
retrieval index. We argue that this design unnecessarily couples indexing granularity with evidence granularity: coarse representations can often suffice for locating relevant temporal regions, while fine-grained evidence remains important for downstream
reasoning. We propose Density-Aware Graph Construction (DAGC), a training-free approach that decouples a query-independent coarse
retrieval index from the original fine-grained evidence space. DAGC constructs a compact, density-adaptive graph index by merging visually redundant neighboring chunks, while preserving mappings to the original temporal units. Retrieved coarse regions are subsequently expanded back to the original chunk granularity for fine-grained evidence refinement and answer generation. Experiments on MLVU, VideoMME, and LongVideoBench show that DAGC retains only about 40–50% of the original graph nodes and achieves $1.3$–$1.7\times$ end-to-end wall-clock acceleration while preserving approximately 99% of the original QA performance. The gains transfer across different LVLM backbones and
video RAG pipelines, suggesting that long-
video RAG need not maintain the same temporal granularity for indexing and evidence
reasoning.
Read the full abstract ▾
All authors Zhe Jin, Zhimin Lin, Bin Zheng, Junhua Fang, Huihua Yang
Computer Vision
ENCORE: Entropy-Guided Cropping and Attention Regularization for Robust Vision–Language Understanding
Yuanhao Sun, Huawei Ji, Jiaxin Ding et al. (+2 more)
Vision-
Language Models (VLMs) perform well on diverse
vision-
language tasks, but
transformer-based visual encoders split images into fixed-resolution sub-images, compromising object integrity in lightweight VLMs. Existing methods only focus on the visual modality and fail to dynamically preserve the integrity of
prompt-relevant regions, limiting performance. In this work, we observe that the early-layer image-text
entropy of cross-modal
attention strongly correlates with answer grounding quality and task accuracy. Building on this finding, we propose ENCORE, an
entropy-guided framework with two components: At inference, an
Entropy-based Cropping Strategy (ECS) evaluates a small set of candidate crops and selects the one with minimal
entropy, preserving contiguous regions relevant to the
prompt. At training,
Entropy Regularization Training (ERT) augments next-
token prediction with an
entropy term that sharpens
attention on key visual
tokens while down-weighting irrelevant ones. Experiments on ten VQA
benchmarks show that ENCORE, fine-tuning only 0.14% of parameters, achieves an average 1.43% accuracy gain and state-of-the-art performance among recent 2B-parameter VLMs. Our code is released in https://github.com/baokou-fw2/ENCORE.
Read the full abstract ▾
All authors Yuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, Xinbing Wang
Computer Vision
Optimize Surgical Triplet Recognition: A Knowledge-Driven Mixture-of-Experts Solution
Yiyi Zhang, Yuchen Yuan, Ying Zheng et al. (+4 more)
Surgical action triplet recognition constitutes a critical task in context-aware
robot-assisted surgery, facilitating automatic
surgical action perception by identifying instrument, verb, target, and their association. However, existing works struggle to analyze such complex
surgical scenes due to three main issues: (1) component-level optimization conflicts caused by entangled feature spaces, (2) category-level optimization conflicts arising from severe data imbalance, and (3) lack of domain knowledge guidance that limits model interpretability and robustness. To address these challenges, we propose a Mixture-of-Experts-guided Co-Optimization (MoeCo) framework powered by knowledge-driven learning. Within the co-optimization pipeline, to first mitigate component-level conflicts, we introduce a component-tailored adapter that disentangles task-specific features across spatial-temporal regimes, facilitating effective component specialization. Next, we develop a coordinated gradient learning strategy to handle category-level conflicts, which adaptively rebalances positive-negative gradients to enhance the perception of rare categories. Notably, inspired by
surgical domain expertise, we introduce a knowledge-driven mixture-of-experts mechanism that dynamically integrates
multimodal large
language model-guided knowledge via activated experts, thereby enriching the co-optimization pipeline with more expressive and robust representations. Extensive experiments on the public CholecT45 and CholecT50
datasets confirm the effectiveness of the proposed co-optimization pipeline and the superiority of dynamic priors integration via the knowledge-driven mixture-of-experts mechanism.
Read the full abstract ▾
All authors Yiyi Zhang, Yuchen Yuan, Ying Zheng, Jialun Pei, Jinpeng Li, Zheng Li, Pheng-Ann Heng
Computer Vision · Cross-listed from cs.AI
ParallelWorld: Test-Time Scaling for Agentic Embodied Reasoning
Min Chen, Shengjun Zhang, Yuxin Li et al. (+4 more)
Embodied AI has emerged as a key paradigm for autonomous systems operating in complex physical environments. While end-to-end models have established strong local capabilities for perception and control, scaling these capabilities to extended, multi-stage tasks remains difficult. Agentic
embodied systems address this challenge by introducing high-level planners that organize perception,
reasoning, and executable skills over longer interaction horizons. However, existing agentic
embodied frameworks typically generate actions for direct execution without explicitly simulating how alternative action sequences shape future observations and task progress, leading
agents to commit to suboptimal interactions before considering their long-term consequences. To address this limitation, we propose ParallelWorld, an adaptive-horizon test-time scaling framework that grounds agentic
embodied reasoning in the prospective simulation of action outcomes. Specifically, we design an
agent system to simulate and evaluate alternative trajectories in parallel before committing to an action. A verifier assesses intermediate state transitions with respect to the task objective, dynamically
pruning unpromising branches and guiding subsequent expansion. Based on the evaluated long-horizon outcomes, the
agent selects a trajectory, executes only its first action, and replans as new observations become available. Experiments on ESI-Bench and LIBERO-Pro demonstrate improvements in active spatial
reasoning and
language-conditioned manipulation, respectively, highlighting the effectiveness of prospective planning for both understanding and manipulation tasks.
Read the full abstract ▾
All authors Min Chen, Shengjun Zhang, Yuxin Li, Zhang Zhang, Xin Fei, Chong Xia, Yueqi Duan
Computer Vision
Simplified Cross-Modal Calibration for Heterogeneous Event-RGB Stereo Systems
Nico Hessenthaler, Adam T. Müller and Nicolaj C. Stache
Accurate extrinsic calibration between event-based and frame-based cameras remains a practical bottleneck for heterogeneous stereo systems. Existing approaches often require sensor or target motion, precise synchronization, or computationally expensive event-to-image reconstruction. We propose a simple, motion-free cross-modal calibration framework that uses a temporally modulated, blended ChArUco target presented on standard consumer displays. By alternating between the original pattern and a partially blended version, the target reliably triggers events while remaining continuously observable to a frame-based camera, avoiding blank frames and reducing synchronization constraints to a coarse, trigger-based
alignment. We discretize events into frames coarsely aligned with the RGB images, apply lightweight denoising, and perform ChArUco-based intrinsic and stereo extrinsic calibration. Extensive experiments assess robustness to blending opacity, display brightness, external illumination, viewing angle, and handheld acquisition. Compared to the strongest motion-based reference (E2Calib + Kalibr) and a non-motion-based reference (Plasberg et al.), our approach reduces the mean reprojection error by $44\%$ and $6\%$, respectively, while substantially simplifying the calibration procedure. Finally, we demonstrate practical utility in a robotic eye-to-hand calibration case study, showing consistent transformations and stable downstream
geometric measurements even under partial occlusions. Code is publicly available at https://github.com/nhessenthaler/simple-evrgb-cal.
Read the full abstract ▾
Computer Vision
WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
Jun Zhang, Qiao Zhao, Cheng Cui et al. (+6 more)
While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing
benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a
benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four
languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from
language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors – 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on
language priors that conventional accuracy metrics cannot capture.
Read the full abstract ▾
All authors Jun Zhang, Qiao Zhao, Cheng Cui, Jianying Qu, Zhongkai Sun, Jianwen Yang, Changda Zhou, ZhuoXin Liu, Shubin Han