Computer Vision
SimWAM: A Simple World Action Model for End-to-End Autonomous Driving
Zongchuang Zhao, Xin Zhou, Tianyang Xu et al. (+6 more)
In autonomous driving, World-Action Models (WAMs) have improved end-to-end planning by transferring
video dynamics priors to action prediction, but many still couple planning with future-
video generation at inference, incurring substantial computational overhead. We present SimWAM, a simple yet effective WAM that leverages future-
video prediction solely as a training-time supervision signal. It co-trains a pretrained
video expert and a lightweight action expert with joint flow matching. An isolated
attention mask keeps action prediction independent of future frames, allowing trajectory prediction without future-frame generation at inference. This design supports multiple pretrained
video backbones and independent action-expert scaling within a shared
attention interface, while preserving the joint learning objective. Moreover, we apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Experiments show that SimWAM achieves $91.9$ PDMS on NAVSIM with a favorable trade-off between accuracy and latency among world-model-based planners, while transferring zero-shot to nuScenes. It also achieves competitive planning accuracy on WOD-E2E and PhysicalAI-Autonomous-Vehicles. These results position SimWAM as a plain yet solid baseline for efficient autonomous driving. The code and model weights are available at https://github.com/H-EmbodVis/SimWAM/.
Read the full abstract ▾
All authors Zongchuang Zhao, Xin Zhou, Tianyang Xu, Zhengyang Sun, Kaixuan Zhou, Yu Wu, Honglin Li, Dingkang Liang, Xiang Bai
Computer Vision
MirrorWorld: Taming Video Diffusion Models for Mirror Reflection Generation
Youjun Zhao, Alex Warren, Gary K.L. Tam et al. (+1 more)
Recent advances in
video diffusion models (VDMs) have enabled high-fidelity
video synthesis. However, generating mirror reflections remains challenging because the content within a mirror must remain consistent with the surrounding scene. Existing VDMs are not specifically designed to model scene-to-mirror relationships, which can lead to reflections with incorrect content or inconsistent spatial arrangements. We observe that mirror reflection generation involves two complementary challenges: determining what scene content should be reflected and how the reflected content should be spatially arranged within the mirror region. Motivated by this observation, we propose MirrorWorld, a reflection-aware
video inpainting framework that models scene-to-mirror relationships during generation. Specifically, we introduce Semantic Relation
Distillation (SRD), which transfers relational information from a frozen visual foundation model to encourage semantic associations between visible scene content and mirror regions. We further propose
Geometric Transformation
Alignment (GTA), which learns a transformation that guides the spatial arrangement of reflected content. The two components play complementary roles, with SRD modeling what should be reflected and GTA modeling how it should be arranged. To facilitate research on this problem, we construct a
benchmark for
video mirror reflection generation by repurposing four existing
video mirror
datasets into a unified reflection reconstruction task. Experimental results show that MirrorWorld achieves improved reflection reconstruction quality over representative image-based reflection generation methods and strong
video inpainting baselines.
Read the full abstract ▾
All authors Youjun Zhao, Alex Warren, Gary K.L. Tam, Rynson W.H. Lau
Computer Vision
SABRE: Scalable and Automated Benchmarking of VLMs under Stress
Zixuan Lan, Luzhe Sun, Matthew R. Walter et al. (+1 more)
Vision-
language models (VLMs) are improving rapidly, but
benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors – learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and
Language Elicitation (answers suggested by
language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed
benchmark.
Read the full abstract ▾
All authors Zixuan Lan, Luzhe Sun, Matthew R. Walter, Jiawei Zhou
Computer Vision
Conformal Coverage Guarantees for Any Video Temporal Grounder
Aseel Mohamed, Rasul Khanbayov, Erchin Serpedin et al. (+1 more)
Event boundaries in continuous
video are ambiguous: re-annotate the same query-
video pair and independent annotators mark moments that overlap by less than half on a large fraction of samples. The ground truth for
video temporal grounding is therefore a distribution over intervals, yet every grounder returns a single interval with no statement of reliability, so at deployment a wrong interval is indistinguishable from a right one. COVER changes the output object: a post-hoc, model-agnostic wrapper that turns any grounder, a trained localizer or a black-box
video–
language model, into one that emits a temporal region containing the true moment with probability at least $1-\alpha$, by calibrating the
quantile of a temporal nonconformity score on held-out labels and widening the base prediction by that amount. The guarantee is finite-sample and distribution-free under exchangeability, and requires neither retraining nor white-box access. We give two score families, a two-sided boundary-widening score for grounders that emit an interval and a super-level-set score for grounders that emit a relevance signal, and develop theory specific to grounding that bounds how large the certified region becomes, when coverage survives conditioning on event length, and how it degrades when moments from one
video break exchangeability. Across three
benchmarks and five grounders, realized coverage tracks the target, and calibration exposes what point metrics hide.
Read the full abstract ▾
All authors Aseel Mohamed, Rasul Khanbayov, Erchin Serpedin, Hasan Kurban
Computer Vision
I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning
Shibo Gao, Chongxiao Wang, Chenglong Huang et al. (+10 more)
Real-world
video reasoning often involves
multimodal, multi-source inputs, whereas existing
video reasoning tasks typically assume a simplified
video-text setting, limiting identity matching and person-centric
reasoning. To bridge this gap, we introduce the Identity-conditioned Queries (ICQ) task, in which models are required to jointly associate and interpret an input
video and a reference image of a person, and leverage this conditioning to address identity grounding, behavior understanding, and temporal
reasoning, among other challenges. Building on ICQ, we present ISYV (I Seek You in
Videos), a systematic solution comprising three components: (1) ISYV-Bench, a challenging evaluation
benchmark with 1,377 real-world complex
videos and 1,377 question-answer pairs, organized into six difficulty levels spanning capabilities from identity recognition to causal
reasoning; (2) ISYV-75K, a large-scale training set of 75K high-quality samples constructed via automated annotation, multi-stage verification, and manual review; and (3) ISYV-Framework, containing an ICQ-oriented model and training strategy for learning to exploit informative
video shots without additional shot-level annotations. Extensive experiments show that both mainstream closed-source and open-source MLLMs struggle on ISYV-Bench, especially in cross-domain identity matching and long-horizon tracking. ISYV-Model outperforms strong baselines and in some aspects approaches closed-source performance. Overall, ISYV provides a unified task definition, scalable
datasets/
benchmarks, and modeling insights for person-centric
video reasoning.
Read the full abstract ▾
All authors Shibo Gao, Chongxiao Wang, Chenglong Huang, Jie Ma, Haolin Shi, Fei Ding, Jing Li, Qiang Lyu, Yangyang Liu, Yang Liu, Jun Liu, Linlin Huang, Peipei Yang
Computer Vision
UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling
An Lanji, Dawei Liu, Jin Li et al. (+3 more)
Joint-
Embedding Predictive Architectures (JEPAs) have emerged as a principled framework for self-supervised learning of world models in compact latent spaces, yet existing methods are fragmented: some predict masked parts of a single image in latent space (I-JEPA), others learn to predict global photometric transformations (Image World Models), while
video-scale JEPAs predict future temporal states and are post-trained for action-conditioned planning (V-JEPA~2, DINO-World, DINO-WM). These objectives are treated as distinct recipes with separate encoders, predictors, and anti-collapse regularizers, hindering a single model from unifying image-level and
video-level world modeling. We present UniJEPA, a unified JEPA that jointly learns photometric prediction (image-level transformations) and temporal prediction (
video-level next-state dynamics) in one shared latent space. A single end-to-end objective, composed of a next-
embedding prediction loss and a Gaussian regularizer, yields a provably anti-collapse encoder-predictor pair trainable from raw pixels without EMA, stop-gradient, or pre-trained encoders. We show that the same latent space supports controllable abstraction: photometric prediction learns invariant structure while temporal prediction learns equivariant dynamics. After action-conditioned post-training on offline trajectories, UniJEPA enables zero-shot planning by treating goal features as prediction targets. On image,
video, and control
benchmarks, UniJEPA matches or surpasses task-specific JEPAs while requiring a single loss hyperparameter, and plans up to tens of times faster than generative world models at comparable accuracy.
Read the full abstract ▾
All authors An Lanji, Dawei Liu, Jin Li, Haoran Xu, Mei Chen, Yu Tian
Computer Vision
Addressable Memory for Video World Models
Xindi Wu, Sven Elflein, James Lucas et al. (+5 more)
We study visual persistence in interactive
video world models. These models rely on a Key-Value (KV) cache as a growing visual
memory to carry forward previously generated frames. However, we find that models can no longer reliably address stored content once rollouts extend beyond the training horizon, because temporal Rotary Positional
Embeddings (RoPE) offsets then fall outside the range seen during training and the model struggles to retrieve the relevant visual information through
attention. Moreover, naively compressing the cache in the RoPE-rotated space corrupts
memory by averaging together incompatible positional phases. To address this, we propose WorldTrace, a training-free
memory framework for long-horizon visual persistence. WorldTrace keeps compressed
memory addressable by assigning each summary slot a distinct, in-distribution virtual position. Within this addressable cache, we study two
memory compression approaches: WorldTrace-Field compresses history for temporal coherence, while WorldTrace-Landmark stores verbatim scene traces at detected transitions for episodic recall. We further introduce LoopBench, a
benchmark evaluating whether a compressed cache can reconstruct a previously visited scene after a long detour. WorldTrace-Field improves temporal consistency by +15.5%, and WorldTrace-Landmark improves episodic recall by +19.5% on LoopBench, extending visually persistent generation without retraining.
Read the full abstract ▾
All authors Xindi Wu, Sven Elflein, James Lucas, Olga Russakovsky, Laura Leal-Taixé, Despoina Paschalidou, Jonathan Lorraine, Aljoša Ošep
Computer Vision
GeoDistill-Refine: Silhouette-First Geometry Distillation for Annotation-Free Spacecraft Segmentation
Yonglong Zhang, Zongwu Xie and Yang Liu
Foundation
segmentation models can provide supervision for spacecraft imagery without manual training masks, but their predictions vary with textual
prompts and may contain
geometric errors that are amplified during
distillation. This paper presents GeoDistill-Refine, a two-stage framework that transfers offline SAM 3 pseudo-masks to a compact
segmentation network. Six fixed
prompts are fused by an unweighted 50% vote to stabilize the teacher output. The student first learns the foreground silhouette and is then refined with signed-distance-field, skeleton, and area objectives derived from the pseudo-mask. A sample-level gate, computed from
prompt agreement, the valid-
prompt ratio, and pseudo-mask area plausibility, reduces the influence of unreliable pseudo-geometry. On the SpaceSense-Bench HJM lockbox set, GeoDistill-Refine improves Image IoU and Boundary F1 by 0.0456 and 0.1380, respectively, over a plain pseudo-label student. External evaluations on the SPEED+ Lightbox and Sunlamp domains and on TANGO show competitive regional overlap together with gains in boundary quality or foreground precision. The deployed TinyUNet contains 0.263 M parameters and requires approximately 1.1 ms per image on an RTX 4090; SAM 3 pseudo-mask construction and the auxiliary geometry branches are used only during training.
Read the full abstract ▾
Computer Vision
SkySeaLand: A Wide-Format Satellite Transportation Benchmark with an Ultra-Lightweight Detection Baseline
Md. Zahid Hasan Riad and Md Sultanul Islam Ovi
Satellite object detection is challenged by small targets and wide-format scenes that lose detail under standard square-input resizing. We introduce SkySeaLand, a public
dataset of 1,307 high-resolution satellite images and 19,101 verified bounding boxes across airplane, boat, car, and ship classes in terrestrial and maritime scenes. Native COCO and YOLO annotations are provided. The collection is dominated by large source images and wide scene geometry: 84.5 percent exceed 3,836 pixels on the longest side and 73.1 percent are near a 3:1 aspect ratio. We evaluate twelve detectors from the YOLO, RT-DETR, DETR, and Faster R-CNN families using a common split and COCO metrics. The tested YOLO and RT-DETR variants obtain 84.4–88.2 mAP50, with no consistent accuracy gain from larger parameter counts under the reported model-specific recipes. We also report SkyDet, a 1.22 M parameter anchor-free baseline that obtains 60.5 mAP50 and 24.32 mAP50-95 in a 4.90 MB footprint, with 13.74 ms latency (72.8 FPS) on a Tesla T4. SkySeaLand provides a compact
benchmark for mixed land–maritime transportation detection, while SkyDet establishes a documented low-footprint reference rather than a state-of-the-art accuracy claim.
Read the full abstract ▾
Computer Vision · Cross-listed from physics.med-ph
Measurements Automatically Extracted from Zero Echo Time MRI Using Deep Learning Image Segmentation and Geometric Modeling Agree with Expert Manual Readings
Jack Consolini, Eric A. Bogner, Meghan Sahr et al. (+3 more)
Computed
tomography (CT) remains the reference for 3D osseous morphometry in femoroacetabular impingement (FAI) but requires ionizing radiation and manual measurement. Zero echo time (ZTE) MRI visualizes cortical bone and yields FAI angles that agree with CT, but automated angle extraction remains limited. We developed and validated automated FAI angle computation from ZTE MRI and assessed agreement with expert manual measurements in a cross-sectional study (level of evidence, 3). Pelvic ZTE MRI was acquired in 73 participants (mean age 36.8 +/- 18.5 years; 51 women, 22 men), yielding 135 hips. nnU-Net was trained on 100 manually curated hips to segment the femur, pelvis, and three osseous landmarks. Custom
geometric algorithms computed alpha, femoral neck-shaft, Tonnis, coronal and sagittal center-edge, and acetabular version angles from inferred
segmentations. Measurements on 35 test hips were compared with the mean of two radiologists' manual measures using intraclass correlation (ICC) and Bland-Altman analysis. Dice exceeded 0.96 for bone and ranged from 0.65 to 0.83 for landmarks. Median landmark error was 0.38 mm (femoral head), 0.82 mm (lateral acetabulum), and <2.5 mm (medial acetabulum, greater trochanter). Interrater ICC was excellent for acetabular version, coronal center-edge, and Tonnis (>=0.82) but poor for alpha and femoral neck-shaft. Model versus rater-mean agreement was excellent for acetabular version, coronal center-edge, and Tonnis (0.92-0.96), good for mid-acetabular sagittal center-edge (0.74), and fair for alpha (0.45) and femoral neck-shaft (0.55). Model Bland-Altman limits of agreement were narrower than interrater limits for most angles. Fully automated morphometric assessment from ZTE MRI is feasible and performs comparably to expert readers for most coverage and version angles.
Read the full abstract ▾
All authors Jack Consolini, Eric A. Bogner, Meghan Sahr, Matthew F. Koff, Kevin M. Koch, Hollis G. Potter
Computer Vision · Cross-listed from cs.RO
Depth-Wise Probing and Pruning of the Planning Token in a Driving Vision-Language-Action Model
Harisankar Babu, Benjamin Coors, Christopher Lang et al. (+3 more)
Vision-
language-action (VLA) models route driving decisions through a deep
language model, but it is unclear how much of that depth the action itself requires. We study a representative driving VLA whose entire plan is carried by a single planning
token that a generative planner decodes into a trajectory. Borrowing the planner as a trajectory-space logit lens, we decode the planning
token from every one of the 32 decoder layers and measure two signals: the linear decodability of the
navigation command and trajectory compatibility with the frozen native planner. Our diagnostic shows that semantic intent is linearly decodable early: command-probe accuracy reaches 97.7% after the first decoder layer, compared with 16.7% chance. In contrast, compatibility with the frozen native planner improves gradually across depth, with open-loop Avg-L2 reaching its minimum of 2.11\,m only at the final layer. Learned readouts from the first layer recover much of this gap, indicating that planning information is already present early but is not yet represented in the format expected by the deployed planner. Ranking decoder layers by the angular deviation they induce in the planning
token permits removal of 8 of 32 layers within an approximately 5% relative open-loop error increase and yields a measured 1.33$\times$ decoder speedup. At the evaluated sample size, no family-specific degradation is statistically resolved. These findings are limited to the evaluated ORION checkpoint and Bench2Drive setup.
Read the full abstract ▾
All authors Harisankar Babu, Benjamin Coors, Christopher Lang, Hendrik Berkemeyer, Tamim Asfour, Simon Foell
Computer Vision
H2AL: Hyperbolic Hierarchy-aware Aggregative Learning for Registration-based Few-shot Medical Image Segmentation
Jia Wang, Jiaming Cai, Zunying Hu et al. (+4 more)
Registration-based Few-shot medical image
segmentation (RFMIS) aims to generate pseudo-labels for unlabeled images by warping a labeled image through registration. However, existing methods primarily perform pixel-level optimization and inference in Euclidean space, treating anatomical structures as flat and disjoint. This neglect of inherent hierarchies degrades pseudo-label quality and weakens the discrimination of ambiguous regions, limiting the
segmentation performance. To overcome this challenge, we propose a
Hyperbolic Hierarchy-aware Aggregative Learning framework for RFMIS, termed H2AL, that enhances both deformation plausibility and anatomical discrimination for dual-task learning. Specifically, we introduce a
Hyperbolic Hierarchy-aware Infusion (H2I) module, which leverages the hierarchical modeling capability of
hyperbolic space to learn precise hierarchy-aware representations via transformation-guided supervised
hyperbolic contrastive learning, and injects such hierarchical priors into Euclidean space through a gated infusion block while preserving semantic richness. Furthermore, we propose an end-to-end joint optimization
algorithm by gradient aggregation, where the gradients from the registration and
segmentation decoders,
embedding semantic and hierarchical cues, are aggregated to update the shared encoder to promote collaborative learning across tasks. Extensive experiments on two anatomical regions, with five experimental settings, demonstrate the effectiveness and efficiency of our method in both registration and
segmentation. The code is publicly available at https://github.com/JiamingCai469/H2AL.
Read the full abstract ▾
All authors Jia Wang, Jiaming Cai, Zunying Hu, Zhanjie Wu, Jinyuan Liu, Hua Cheng, Yun Peng
Computer Vision · Cross-listed from cs.RO
TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models
Ziheng Liu and Quantao Yang
Vision-
language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically apply a single, uniform update strategy to all model components, ignoring their distinct functional roles. We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. TEMPO freezes the pretrained
vision-
language backbone to preserve general semantic representations, and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert. We update them at different rates–the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction. This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing high-level semantic representations while still allowing the action expert to learn efficiently from online feedback. Experiments on the CALVIN
benchmark and real-world manipulation tasks demonstrate that TEMPO consistently outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline, while reaching and maintaining higher evaluation rewards on two real-world tasks.
Read the full abstract ▾
Computer Vision
Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
Zichuan Wang, Songlin Yang, Bo Peng et al. (+4 more)
Large
Vision-
Language Models (LVLMs) often suffer from object
hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual
attention. However, we find that both real and hallucinated objects receive equally strong visual
attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-
attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object
tokens, whereas those for hallucinated objects cannot. Building on this, we identify two
hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the
hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the
hallucination persists and
attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect
hallucination and targeted remedies: High-
Attention Regions Masking (HARM) for visual uncertainty
hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior
hallucination. Our approach achieves state-of-the-art results on multiple
hallucination benchmarks. Code will be available.
Read the full abstract ▾
All authors Zichuan Wang, Songlin Yang, Bo Peng, Zhenchen Tang, Yang Li, Beibei Dong, Jing Dong
Computer Vision
EliSeg: Verified Target Construction for Report-Grounded Abnormality Segmentation
Chengyi Peng, Haoyu Yang, Meixing Shi et al. (+2 more)
Radiology reports describe clinical observations but do not specify executable
segmentation targets. They may contain present, negated, prior,uncertain, or irrelevant findings, while multiple valid abnormalities may coexist. Existing
segmentation methods largely bypass this ambiguity by receiving a target identity or spatial
prompt before inference, which acts as a hidden target oracle. We study report-grounded abnormality
segmentation, where a model must determine target eligibility, cardinality, and finding-to-mask correspondence directly from an unfiltered report before delineating the corresponding regions. We propose EliSeg, an atcor–verify–revise framework that integrates target construction with mask generation. A grammar-constrained Actor proposes target slots and masks, an independent text-only Verifier reconstructs the eligible finding inventory, and Revision selectively re-executes the shared Actor when their target structures disagree. EliSeg requires no predefined target identity, finding
prompt, point, or bounding box. Experiments on MIMIC-CXR-ILS show that EliSeg consistently outperforms direct
segmentation methods and extract-then-segment cascades across findings, while effectively suppressing masks for ineligible report mentions. Ablation studies confirm the complementary roles of verification and revision, and evaluation on CheXlocalize demonstrates effective transfer of the EliSeg to an external
dataset.Code is available at https://github.com/Maybach-dream/EliSeg.
Read the full abstract ▾
All authors Chengyi Peng, Haoyu Yang, Meixing Shi, Yuxiang Cai, Yankai Jiang
Computer Vision
Foundation Models Adaptation for Multi-View Multi-modal Cardiac MRI Segmentation and Direct Ejection Fraction Estimation
Sina Amirrajab, Cian M Scannell, Volker Vehof et al. (+2 more)
Foundation models have shown strong transferability in cardiac MRI (CMR), but their effectiveness for heterogeneous multi-view and multi-sequence CMR analysis remains unclear. In this work, we explore the effectiveness of fine-tuning and combining different CMR foundation models for the Universal Multi-Sequence, Multi-Center and Multi-View CMR
Segmentation (CMR-Multi) Challenge. CineMA was fine-tuned for cine and late gadolinium enhancement (LGE)
segmentation across short-axis and long-axis views. For direct left-ventricular ejection fraction (LVEF) estimation, we used two recent frozen CMR foundation models to extract
embedding vectors that were then combined using
attention-based multiple-instance learning for LVEF regression. In the challenge validation set, cine
segmentation achieved Dice scores of 0.862, 0.883, and 0.902 for short-axis, two-chamber and four-chamber cine MRI, respectively. LGE
segmentation achieved Dice scores between 0.621 and 0.846 across views. The direct LVEF regression model achieved an MAE of 4.96 percentage points and a Pearson correlation of 0.91. These results indicate that foundation models can be effectively adapted and combined for multi-view CMR analysis, while accurate LGE scar
segmentation remains a challenging task.
Read the full abstract ▾
All authors Sina Amirrajab, Cian M Scannell, Volker Vehof, Michael Bietenbeck, Ali Yilmaz
Computer Vision · Cross-listed from cs.CL
Off-the-Shelf Language-Vision Encoders Predict Human Gaze Preferences in Visual World Experiments
Rahul Murali Shankar, Titus von der Malsburg and Sebastian Padó
A central question for computational psycholinguistics is to what extent neural
language models can serve as cognitively adequate models of human
language processing. However, work has been overwhelmingly focused on the unimodal case of written or spoken
language. In contrast,
multimodal paradigms, like visual world experiments that present participants with both visual and linguistic input simultaneously, have been neglected. In this paper, we present a novel approach that predicts patterns of gaze preferences in visual world studies. To do so, it derives target preferences by applying an attribution method to a simple bi-modal CLIP bi-encoder. We demonstrate the ability of this approach to qualitatively replicate the gaze patterns found in a seminal English visual world study demonstrating human predictive processing. It does so without a generative architecture and without the need for fine-tuning, despite not being trained for the task.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN
Yuehao Huang, Yunzi Wu, Xiaotao Zhang et al. (+7 more)
Recent
vision-
language navigation (VLN) systems increasingly adapt pretrained
vision-
language models (VLMs) into
vision-
language-action (VLA) policies that map
egocentric observations and
language instructions directly to
navigation actions. Although semantically capable, such action-centric training does not explicitly model how the
agent's visual observations should evolve under its predicted motion. Generative world-action models (WAMs) jointly predict future observations and actions, yet existing WAMs for continuous VLN do not condition joint future-view and action generation on geometry-aware representations inferred from the observed history. We present WNM-3D, a generative World
Navigation Model with 3D scene conditioning for continuous VLN. To consolidate past observations into persistent scene context, a frozen feed-forward geometry encoder extracts geometry-aware representations from the
monocular egocentric RGB history, and a trainable 3D Scene-to-
Token Adapter converts them into a fixed-length prefix in the
token space of the world-action
Diffusion Transformer. Through block-causal
attention, this prefix conditions every future
video-action block, providing a shared
geometric context for both future-view and action generation. We train WNM-3D through supervised world-action fine-tuning on A*-generated demonstrations, DAgger-style adaptation on policy-visited states, and Counterfactual DanceGRPO refinement for closed-loop execution. Experiments on GN-Bench show that WNM-3D outperforms strong VLM-based
navigation policies and its 2D-conditioned counterpart in closed-loop
navigation. Stage-wise ablations further show that DAgger-SFT provides the larger success-rate gain, while Counterfactual DanceGRPO subsequently improves both
navigation success and path efficiency.
Read the full abstract ▾
All authors Yuehao Huang, Yunzi Wu, Xiaotao Zhang, Xinhai Li, Jiankun Dong, Jiajun Lv, Chi Zhang, Chenjia Bai, Yong Liu, Xuelong Li
Computer Vision
CANIS: Generation-Assisted 3D Canonicalization via an Image-Semantic Bridge
Kendong Liu, Yuxin Yao and Junhui Hou
Canonicalizing 3D object orientation is fundamental to 3D understanding and analysis. Existing approaches often rely on
geometric cues, although 3D canonicalization ultimately requires a semantically meaningful orientation. To address this gap, we propose CANIS, a category-agnostic, generation-assisted framework that introduces the semantic orientation prior of a frozen image-to-3D generative model into 3D canonicalization, without canonicalization-specific training or category-specific templates. Specifically, CANIS first renders the input object from candidate viewpoints, selects an informative view, and generates a proxy in a canonical orientation. During generation, a sparse structural latent encoded from the input guides the proxy to preserve the geometry of an object. CANIS then uses the selected image as a semantic bridge between the input and the proxy. Image patches identify semantic regions on the proxy, and depth back-projection locates the corresponding regions on the input. The resulting semantic anchors constrain
geometric matching, from which we estimate the rigid transformation that canonicalizes the input. Experiments on synthetic
benchmarks validate CANIS and its key components, while qualitative results on partial observations and OmniObject3D suggest its applicability to incomplete and real-world scans. CANIS also improves downstream 3D classification, part
segmentation, and dense correspondence under arbitrary rotations. Project page: https://kenkenzaii.github.io/Canis.
Read the full abstract ▾
Computer Vision
Flow-Corrected Shape Optimization: Taming Manifold Drift in High-Dimensional 3D Models
Emilien Seiler, Nicolas Talabot, Yingxuan You et al. (+2 more)
Optimizing 3D shapes within the latent spaces of deep generative models is fundamental to computer assisted engineering, yet remains prone to a critical failure mode we term
manifold drift: the tendency of gradient-based optimization to move latent vectors away from the
manifold of valid shapes. This problem is exacerbated in state-of-the-art 3D shape generative models that operate in increasingly high-dimensional latent spaces where valid shapes occupy a vanishingly small fraction of the full space. Existing mitigation strategies, including latent regularization and flow-matching approaches, either sacrifice expressiveness, demand a difficult trade-off between objective guidance and generative fidelity that remains prone to
manifold drift, or are computationally infeasible to scale to modern, large-capacity 3D shape models. We introduce a novel optimizer-corrector framework that alternates between gradient steps for objective minimization and guided flow matching to drive the latent state back to the valid shape
manifold. By decoupling objective minimization from flow-based correction, optimizing freely and correcting strictly, this alternating design avoids inherent trade-offs, preserving
geometric validity without sacrificing expressiveness while remaining computationally feasible on modern 3D shape models. We demonstrate its effectiveness across generative priors of varying complexity, from simple vector latent spaces to large-scale architectures across a variety of downstream optimization tasks, including aerodynamic drag reduction and object compliance optimization.
Read the full abstract ▾
All authors Emilien Seiler, Nicolas Talabot, Yingxuan You, Federico Stella, Pascal Fua
Computer Vision · Cross-listed from cs.LG
An AI4AI Framework for Visual Token Pruning
Zhen Liu, Wenli Huang, Wei Song et al. (+3 more)
Visual-
token pruning can substantially reduce the inference cost of
multimodal large
language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error. As
pruning objectives, budgets, and model architectures diversify, manually navigating the expanding design space becomes increasingly difficult. This paper aims to build an AI4AI framework for visual-
token pruning by addressing a natural question: Can large
language models automatically design effective visual-
token reduction
algorithms? Although LLMs possess broad algorithmic knowledge and strong
reasoning capabilities, translating such general knowledge into effective solutions for a specialized task remains nontrivial. We argue that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-
token pruning. Based on this insight, we propose AutoPrune, a training-free framework for LLM-driven visual-
token pruning policy design. At its core, AutoPrune introduces a
Token Pruning Domain-Specific
Language (TPDSL) comprising 131 reusable atoms for budget control,
token scoring, selection constraints, and
token reassembly. A key property of TPDSL is that it represents each search state as a residual modification of a strong base policy. This residual formulation narrows the search space and directs the LLM's
attention toward the policy components that are most consequential for performance. Experiments on 14
multimodal benchmarks and three MLLM backbones demonstrate the effectiveness, efficiency, and transferability of AutoPrune. Even when removing 94.4% of visual
tokens, AutoPrune preserves more than 99% of full-
token performance while reducing FLOPs by 9.9x and prefill latency by 6.4x.
Read the full abstract ▾
All authors Zhen Liu, Wenli Huang, Wei Song, Yuhan Liu, Zhiqin Yang, Jingwen Fu
Computer Vision
Representation-driven Endoscopic Visual Embedding Alignment for Latent Generation
Francisco Caetano, Tim J.M. Jaspers, Haiko Middeljans et al. (+7 more)
Developing foundation generative models for endoscopy is limited by the gap between natural and clinical images and the computational cost of training large
Diffusion Transformers. Although representation
alignment has improved efficiency in general computer
vision, its role within the highly specialized endoscopic image space remains unclear. We introduce REVEAL (Representation-driven Endoscopic Visual
Embedding Alignment), the largest generative foundation model for endoscopy to date, trained on GastroNet-5M (GN-5M), a multicenter
dataset of 5 million endoscopic frames. Instead of depending on out-of-domain priors, REVEAL employs encoders pretrained directly on the endoscopic distribution to align
diffusion latents with domain-specific visual features, preserving fine textures and intricate anatomical structures. Beyond image generation, REVEAL also serves as a powerful feature extractor; in multiple
benchmarks, it delivers performance that is competitive with, and in several cases exceeds, endoscopic foundation models such as EndoViT and Endo-FM, specifically tuned for classification tasks, while demonstrating strong representation robustness under realistic imaging corruptions. REVEAL produces high-fidelity images and maintains robust structural coherence in latent-space edits such as inpainting and outpainting. This high-capacity backbone lowers the computational threshold for building specialized clinical tools, offering an open, versatile foundation for conditional synthesis,
segmentation, and out-of-distribution detection in future intelligent gastroenterology systems.
Read the full abstract ▾
All authors Francisco Caetano, Tim J.M. Jaspers, Haiko Middeljans, Martijn R. Jong, Rixta A.H. van Eijck van Heslinga, Floor Slooter, Albert J. de Groof, Jacques J. Bergman, Peter H.N. De With, Fons van der Sommen
Computer Vision
InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding
Minchao Jiang, Xiaoxuan Ma, Shunyu Jia et al. (+3 more)
Feed-forward 3D Gaussian Splatting (3DGS) enables efficient and generalizable 3D reconstruction, but current feed-forward 3DGS methods for scene understanding remain largely category-oriented. In contrast, instance-aware 3DGS methods typically rely on per-scene optimization and often decouple reconstruction from instance and semantic learning, limiting reciprocal interactions among them. We present InstanceSplat, a unified feed-forward 3DGS framework for generalizable 3D reconstruction and instance-aware scene understanding from pose-free multi-view images. In a single forward pass, InstanceSplat constructs an instance-aware Gaussian representation that jointly encodes appearance, geometry, instance identity, and
language-aligned semantics. Shared 3D Gaussians ground instance identities across views, producing renderable and cross-view-consistent instance features. To allow reconstruction and scene understanding to benefit from each other, we further design an instance-centric learning strategy that connects reconstruction, instance learning, and semantic learning through shared instance structure. Specifically, instance cues guide reconstruction,
language-aligned semantics strengthen the discrimination of confusing same-category instances, and instance regions aggregate semantic evidence into coherent object-level predictions. Experiments on novel-view synthesis, instance
segmentation, and open-vocabulary semantic understanding under varying input-view settings and on an unseen
dataset demonstrate state-of-the-art performance, practical efficiency, and strong generalization.
Read the full abstract ▾
All authors Minchao Jiang, Xiaoxuan Ma, Shunyu Jia, Haoru Wang, Zhang Liang, Wentao Zhu
Computer Vision
Human-AI Perceptual Alignment by Playing Hues and Cues
Nuria Alabau-Bosque, Jorge Vila-Tomás, Paula Daudén-Oliver et al. (+3 more)
Evaluating the perceptual
alignment between Contrastive
Vision-
Language Models (CVLMs) and humans is typically constrained by traditional
benchmarks that overlook fine-grained semantic and cultural nuances. In this work, we propose a novel evaluation framework that leverages the gamified, discrete color space of the board game Hues and Cues. By mapping the board's 480 color
cells to the CIE xy chromaticity diagram, we calculate empirical perceptual distances across a carefully curated 100-word vocabulary spanning seven semantic categories. To properly contextualize model performance, we establish an empirical lower bound of expected error-the Human Consistency baseline-calculated via Leave-One-Out (LOO) cross-validation on a dense
dataset of color associations collected from 325 human observers through a custom digital interface. We evaluate 162 models across multiple architectural families and pre-training
datasets to assess their semantic color grounding. Our results demonstrate that while CVLMs successfully replicate human cognitive biases, such as idealized
memory colors for concrete physical referents (e.g., food and plants), they systematically diverge from the human baseline in abstract, subjective, and pop-culture domains. We identify two distinct failure modes in severely misaligned concepts: semantic misclassification and a systematic uncertainty collapse into a default blue coordinate. Furthermore, we reveal that highly curated pre-training
datasets are significantly more effective than massive, uncurated corpora in mitigating these severe misalignments. Ultimately, this work highlights that despite their broad categorization capabilities, current CVLMs still fail to capture the nuanced, localized consensus of human color
memory, emphasizing the value of gamified tasks in exposing underlying model biases. The data and code are publicly available to test other metrics.
Read the full abstract ▾
All authors Nuria Alabau-Bosque, Jorge Vila-Tomás, Paula Daudén-Oliver, Pablo Hernández-Cámara, Valero Laparra, Jesús Malo
Computer Vision
Multiple Hypothesis Flow Estimation for Video Frame Interpolation under Matching Ambiguity
Zibo Su, Jing Kong, Ruixing Wang et al. (+2 more)
Many flow-based
video frame interpolation (VFI) methods synthesize an intermediate frame by estimating optical flow fields, warping the two input frames, and blending the warped observations. These latent flow fields are typically learned through image-level reconstruction supervision without direct flow annotations. In ambiguous regions containing repetitive or
stochastic textures, rotating symmetric structures, or fast motion with blur, the matching evidence for a single query may contain multiple comparable and spatially separated peaks. Although the ground-truth intermediate frame provides indirect supervision, it may not uniquely identify the latent correspondence in ambiguous regions.When several locations provide multiple plausible matches, a single-flow
estimator can retain only one displacement and discard the remaining candidates. If the selected match is incorrect or inconsistent with those of neighboring pixels, warping samples content from mismatched locations, producing ghosting, structural distortion, or blur.To address this limitation, we propose a multiple hypothesis flow estimation framework that preserves top-K candidate correspondences and selects one per location through a reliability-guided router. Each hypothesis is initialized from a coarse matching anchor and refined separately through anchor-centered local
attention. Frame synthesis is thus conditioned on one selected flow-appearance hypothesis rather than a soft combination of candidate motions.Experiments on the proposed MA-HD
benchmark and public VFI
benchmarks show that our method achieves the best LPIPS and DISTS among the compared methods.
Read the full abstract ▾
All authors Zibo Su, Jing Kong, Ruixing Wang, Zhanhe Zhang, Kun Wei
Computer Vision
Beyond Fluency: A Clinical Benchmark and Anomaly-Enhanced Baseline for Spine MRI Report Generation
Bruno Palau, Franziska Vogt, Daria Laslo et al. (+4 more)
Radiology reporting is time-consuming and subject to inter-rater variability, making automated report generation an attractive clinical application for
Vision-
Language Models (VLMs). We
benchmark state-of-the-art VLMs on lumbar spine MRI with a focus on diagnostic accuracy and demonstrate that standard lexical and semantic metrics poorly reflect clinical correctness: fluent, well-structured reports can score highly while containing clinically meaningful diagnostic errors. To address this failure mode, we propose an architecture-agnostic framework that augments VLM inputs with spatially localized, disc-level anomaly heatmaps generated by a semi-supervised U-Net++ model. These heatmaps both improve anatomical sensitivity through explicit visual grounding and provide an independent interpretability output for clinical oversight, moving us closer to diagnostically reliable, visually grounded VLMs for lumbar spine MRI interpretation.
Read the full abstract ▾
All authors Bruno Palau, Franziska Vogt, Daria Laslo, Haobo Li, Ender Konukoglu, Maria Monzon, Catherine R. Jutzeler
Computer Vision
Geometry-Aware Camera Localization for Bronchoscopy
Lumin Chen, Qingyao Tian, Jinpeng Li et al. (+5 more)
Camera localization in bronchoscopy remains a challenging problem due to stringent accuracy requirements, real-time constraints, and limited training data. Compared to natural scenes, the confined anatomical structures demand millimeter-level precision, while intraoperative guidance necessitates low-latency inference. However, existing methods often fail to effectively exploit preoperative
geometric priors, limiting their robustness and accuracy. To address these limitations, we propose a unified geometry-aware bronchoscope localization framework (GABL) that effectively fuses preoperative structural priors with paired intraoperative
video to estimate 6-DoF camera poses. Specifically, to address visual ambiguity in complex airways, we propose a graph-guided coarse-to-fine localization scheme that effectively leverages structural priors for precise pose estimation. Furthermore, to mitigate pose jitter and bridge the visual-structural gap, we integrate a
Transformer-based tracking model with a novel RGB-depth matching objective, jointly enforcing spatio-temporal and
geometric consistency. Extensive experiments demonstrate that our method yields remarkable reductions of 8.37% and 31.76% in translation and rotation errors over the prior state-of-the-art, alongside 4 times inference speedup (33.6 FPS) for robust real-time bronchoscope localization. Project website: https://paulili08.github.io/GABL/.
Read the full abstract ▾
All authors Lumin Chen, Qingyao Tian, Jinpeng Li, Haoyu Jiang, Huai Liao, Xinyan Huang, Hongbin Liu, Dong Yi
Computer Vision
International Transfer of Stochastic Cortical Self-Reconstruction
Fabian Bongratz, Zhizheng Zhuo, Chao Zhang et al. (+3 more)
Stochastic cortical self-reconstruction (SCSR) enables personalized mapping of gray matter atrophy, a hallmark of neurodegenerative disorders such as Alzheimer's disease (AD), onto high-resolution cortical surfaces. Unlike conventional normative modeling approaches, which typically operate at a coarse regional level and remain inherently constrained by the
covariates included during training, SCSR estimates an individualized healthy reference directly from the observed cortical thickness at the vertex level. This allows the detection of subtle, subject-specific deviations from healthy cortical shape. In this work, we investigate the generalization and transferability of SCSR, originally trained on UK Biobank (UKB) data, to an independent Chinese population
dataset. Specifically, we evaluate the ability of SCSR-derived Z-scores to discriminate between healthy scans, individuals with mild cognitive impairment (MCI), and patients with AD, while also assessing model robustness across the lifespan. We compare four training strategies: direct application of the UKB-trained model, fine-tuning on Chinese data, training from scratch, and joint training on UKB and Chinese cohorts. As reconstruction backbones, we consider both a multilayer perceptron (MLP) and a Spherical UNet (SUNet). Our results demonstrate that SCSR provides robust detection of cortical atrophy in the Chinese population across all evaluated models. The highest discriminative performance was achieved by the fine-tuned SUNet model (average pairwise AUC = 0.848), followed closely by the UKB-trained SUNet. Moreover, reconstruction errors remained low across the lifespan, even when the training population exhibited a substantially narrower age distribution, indicating strong cross-population transferability.
Read the full abstract ▾
All authors Fabian Bongratz, Zhizheng Zhuo, Chao Zhang, Yaou Liu, Dennis M. Hedderich, Christian Wachinger
Computer Vision
RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs
Qiyanhui Lu, Han Wu, Rongjian Xu et al. (+6 more)
Multimodal large
language models (MLLMs) encode images as long visual
token sequences, making prefilling and KV-cache storage expensive. Existing training-free
pruning methods select
tokens by importance, diversity, or spatial coverage, but treat retained
tokens as interchangeable and do not explicitly track which object-related regions are already covered. We present RoRA, a training-free framework that casts visual
token pruning as role-oriented regional evidence allocation. Given a fixed budget, RoRA partitions
tokens into a protected semantic core, complementary context, and fine-grained detail. It first calibrates text-conditioned
attention with a positional prior and a
prompt-calibrated object prior, then builds
Attention-Anchored Regions (AARs) from high-confidence anchors as lightweight proxies for covered object support. Context is explored mainly outside AARs, while a small AAR-guided budget restores local detail; pairwise similarity is used only for context-stage redundancy filtering. Under matched budgets, RoRA consistently outperforms strong training-free baselines across LLaVA and Qwen-VL families, retaining most of the unpruned accuracy even at aggressive
pruning ratios, e.g., 96.5% of full performance at 88.9%
pruning on LLaVA-1.5, and improving over D2Pruner by about 5% on Qwen3-VL at 75-90%
pruning. At a 66.7%
pruning ratio, RoRA requires only 0.7 ms for
token selection and reduces end-to-end inference time by 24.6%, corresponding to a 1.33x speedup over unpruned inference on an NVIDIA H800.
Read the full abstract ▾
All authors Qiyanhui Lu, Han Wu, Rongjian Xu, Tingzhang Luo, Cheng Fan, Xinghao Chen, Minjing Dong, Jufeng Yang, Jianyuan Guo
Computer Vision · Cross-listed from cs.RO
AutoIntervene: Calibrated Intervention for Action-Chunking Imitation Learning Policies
Jinhe Tang and Weiming Zhi
Action-chunking visuomotor policies learn from demonstrations and improve temporal consistency by predicting short action sequences rather than single-step commands. Yet perception errors and execution drift can move the
robot outside the demonstration distribution, while the policy continues to produce smooth action chunks that are inconsistent with the observed state. We present AutoIntervene, an online framework that selectively transfers control between an action-chunking policy and an operator during deployment. AutoIntervene evaluates proposed chunks against a visual-action support
memory built from successful task executions, combining visual similarity with consistency between proposed and reference actions. Phase-local support governs policy-to-operator transfer within the current task phase, whereas global support governs the return to policy control after operator recovery. We calibrate separate switching thresholds for the two directions from empirical
quantiles of evaluation-level scores on held-out expert demonstrations, avoiding direct manual tuning of score cutoffs. Intervention segments retained from successful rollouts target learner-induced states and provide corrective supervision for subsequent policy updates. Experiments on real-world bimanual manipulation tasks show higher post-adaptation task success and lower operator-control time than manual intervention.
Videos and additional results are available at https://aus.bot/research/autointervene/.
Read the full abstract ▾