Computer Vision
De-rendering, Reasoning, and Repairing Charts with Vision-Language Models
Valentin Bonas, Martin Sinnona, Viviana Siless et al. (+1 more)
Data visualizations are central to scientific communication, journalism, and everyday decision-making, yet they are frequently prone to errors that can distort interpretation or mislead audiences. Rule-based visualization linters can flag violations, but they miss context and do not suggest meaningful design changes. Directly querying general-purpose LLMs about visualization quality is unreliable: lacking training to follow visualization design principles, they often produce inconsistent or incorrect feedback. In this work, we introduce a framework that combines chart de-rendering, automated analysis, and iterative improvement to deliver actionable, interpretable feedback on visualization design. Our system reconstructs the structure of a chart from an image, identifies design flaws using
vision-
language reasoning, and proposes concrete modifications supported by established principles in visualization research. Users can selectively apply these improvements and re-render updated figures, creating a feedback loop that promotes both higher-quality visualizations and the development of visualization literacy. In our evaluation on 1,000 charts from the Chart2Code
benchmark, the system generated 10,452 design recommendations, which clustered into 10 coherent categories (e.g., axis formatting, color accessibility, legend consistency). These results highlight the promise of LLM-driven recommendation systems for delivering structured, principle-based feedback on visualization design, opening the door to more intelligent and accessible authoring tools.
Read the full abstract ▾
All authors Valentin Bonas, Martin Sinnona, Viviana Siless, Emmanuel Iarussi
Computer Vision · Cross-listed from cs.RO
UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models
Manish Kumar Govind, Dominick Reilly, Pu Wang et al. (+1 more)
Latent action representations learned from unlabeled
videos have recently emerged as a promising paradigm for pretraining
vision-
language-action (VLA) models without explicit
robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D
geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a
transformer-based VLA model that incorporates
geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared
embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.The project page is at https://manishgovind.github.io/unilact-vla/
Read the full abstract ▾
All authors Manish Kumar Govind, Dominick Reilly, Pu Wang, Srijan Das
Computer Vision
Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device
Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad et al. (+8 more)
Unified
multimodal models can both understand and generate visual content within a single architecture. Existing models, however, remain data-hungry and too heavy for deployment on edge devices. We present Mobile-O, a compact
vision-
language-
diffusion model that brings unified
multimodal intelligence to a mobile device. Its core module, the Mobile Conditioning Projector (MCP), fuses
vision-
language features with a
diffusion generator using depthwise-separable convolutions and layerwise
alignment. This design enables efficient cross-modal conditioning with minimal computational cost. Trained on only a few million samples and post-trained in a novel quadruplet format (generation
prompt, image, question, answer), Mobile-O jointly enhances both visual understanding and generation capabilities. Despite its efficiency, Mobile-O attains competitive or superior performance compared to other unified models, achieving 74% on GenEval and outperforming Show-O and JanusFlow by 5% and 11%, while running 6x and 11x faster, respectively. For visual understanding, Mobile-O surpasses them by 15.3% and 5.1% averaged across seven
benchmarks. Running in only ~3s per 512x512 image on an iPhone, Mobile-O establishes the first practical framework for real-time unified
multimodal understanding and generation on edge devices. We hope Mobile-O will ease future research in real-time unified
multimodal intelligence running entirely on-device with no cloud dependency. Our code, models,
datasets, and mobile application are publicly available at https://amshaker.github.io/Mobile-O/
Read the full abstract ▾
All authors Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad, Ritesh Thawkar, Omkar Thawakar, Senmao Li, Hisham Cholakkal, Ian Reid, Eric P. Xing, Salman Khan, Fahad Shahbaz Khan
Computer Vision
tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction
Chen Wang, Hao Tan, Wang Yifan et al. (+6 more)
We propose tttLRM, a novel large 3D reconstruction model that leverages a Test-Time Training (TTT) layer to enable long-context, autoregressive 3D reconstruction with linear computational complexity, further scaling the model's capability. Our framework efficiently compresses multiple image observations into the fast weights of the TTT layer, forming an implicit 3D representation in the latent space that can be decoded into various explicit formats, such as Gaussian Splats (GS) for downstream applications. The online learning variant of our model supports progressive 3D reconstruction and refinement from streaming observations. We demonstrate that pretraining on novel view synthesis tasks effectively transfers to explicit 3D modeling, resulting in improved reconstruction quality and faster convergence. Extensive experiments show that our method achieves superior performance in feedforward 3D Gaussian reconstruction compared to state-of-the-art approaches on both objects and scenes.
Read the full abstract ▾
All authors Chen Wang, Hao Tan, Wang Yifan, Zhiqin Chen, Yuheng Liu, Kalyan Sunkavalli, Sai Bi, Lingjie Liu, Yiwei Hu
Computer Vision
Maijunxian Wang, Ruisi Wang, Juyi Lin et al. (+53 more)
Rapid progress in
video models has largely focused on visual quality, leaving their
reasoning capabilities underexplored.
Video reasoning grounds intelligence in spatiotemporally consistent visual environments that go beyond what text can naturally capture, enabling intuitive
reasoning over spatiotemporal structure such as continuity, interaction, and causality. However, systematically studying
video reasoning and its scaling behavior is hindered by the lack of large-scale training data. To address this gap, we introduce the Very Big
Video Reasoning (VBVR)
Dataset, an unprecedentedly large-scale resource spanning 200 curated
reasoning tasks following a principled taxonomy and over one million
video clips, approximately three orders of magnitude larger than existing
datasets. We further present VBVR-Bench, a verifiable evaluation framework that moves beyond model-based judging by incorporating rule-based, human-aligned scorers, enabling reproducible and interpretable diagnosis of
video reasoning capabilities. Leveraging the VBVR suite, we conduct one of the first large-scale scaling studies of
video reasoning and observe early signs of emergent generalization to unseen
reasoning tasks. Together, VBVR lays a foundation for the next stage of research in generalizable
video reasoning. The data,
benchmark toolkit, and models are publicly available at https://
video-reason.com/?v=vbvr .
Read the full abstract ▾
All authors Maijunxian Wang, Ruisi Wang, Juyi Lin, Ran Ji, Thaddäus Wiedemer, Qingying Gao, Dezhi Luo, Yaoyao Qian, Lianyu Huang, Zelong Hong, Jiahui Ge, Qianli Ma, Hang He, Yifan Zhou, Lingzi Guo, Lantao Mei, Jiachen Li, Hanwen Xing, Tianqi Zhao, Fengyuan Yu, Weihang Xiao, Yizheng Jiao, Jianheng Hou, Danyang Zhang, Pengcheng Xu, Boyang Zhong, Zehong Zhao, Gaoyun Fang, John Kitaoka, Yile Xu, Hua Xu, Kenton Blacutt, Tin Nguyen, Siyuan Song, Haoran Sun, Shaoyue Wen, Linyang He, Runming Wang, Yanzhi Wang, Mengyue Yang, Ziqiao Ma, Raphaël Millière, Freda Shi, Nuno Vasconcelos, Daniel Khashabi, Alan Yuille, Yilun Du, Ziming Liu, Bo Li, Dahua Lin, Ziwei Liu, Vikash Kumar, Yijiang Li, Lei Yang, Zhongang Cai, Hokin Deng
Computer Vision
Flow3r: Factored Flow Prediction for Scalable Visual Geometry Learning
Zhongxiao Cong, Qitao Zhao, Minsik Jeon et al. (+1 more)
Current feed-forward 3D/4D reconstruction systems rely on dense geometry and pose supervision – expensive to obtain at scale and particularly scarce for dynamic real-world scenes. We present Flow3r, a framework that augments visual geometry learning with dense 2D correspondences (`flow') as supervision, enabling scalable training from unlabeled
monocular videos. Our key insight is that the flow prediction module should be factored: predicting flow between two images using geometry latents from one and pose latents from the other. This factorization directly guides the learning of both scene geometry and camera motion, and naturally extends to dynamic scenes. In controlled experiments, we show that factored flow prediction outperforms alternative designs and that performance scales consistently with unlabeled data. Integrating factored flow into existing visual geometry architectures and training with ${\sim}800$K unlabeled
videos, Flow3r achieves state-of-the-art results across eight
benchmarks spanning static and dynamic scenes, with its largest gains on in-the-wild dynamic
videos where labeled data is most scarce.
Read the full abstract ▾
All authors Zhongxiao Cong, Qitao Zhao, Minsik Jeon, Shubham Tulsiani
Computer Vision · Cross-listed from cs.RO
Simulation-Ready Cluttered Scene Estimation via Physics-aware Joint Shape and Pose Optimization
Wei-Cheng Huang, Jiaheng Han, Xiaohan Ye et al. (+2 more)
Estimating simulation-ready scenes from real-world observations is crucial for downstream planning and policy learning tasks. Regretfully, existing methods struggle in cluttered environments, often exhibiting prohibitive computational cost, poor robustness, and restricted generality when scaling to multiple interacting objects. We propose a unified optimization-based formulation for real-to-sim scene estimation that jointly recovers the shapes and poses of multiple rigid objects under physical constraints. Our method is built on two key technical innovations. First, we leverage the recently introduced shape-differentiable contact model, whose global differentiability permits joint optimization over object geometry and pose while modeling inter-object contacts. Second, we exploit the structured sparsity of the augmented Lagrangian Hessian to derive an efficient linear system solver whose computational cost scales favorably with scene complexity. Building on this formulation, we develop an end-to-end Simulation-ready Physics-Aware Reconstruction for Cluttered Scenes (SPARCS) pipeline, which integrates learning-based object initialization, physics-constrained joint shape-pose optimization, and differentiable texture refinement. Experiments on cluttered scenes with up to 5 objects and 22 convex hulls demonstrate that our approach robustly reconstructs physically valid, simulation-ready object shapes and poses. Project webpage: https://rory-weicheng.github.io/SPARCS/.
Read the full abstract ▾
All authors Wei-Cheng Huang, Jiaheng Han, Xiaohan Ye, Zherong Pan, Kris Hauser
Computer Vision
Do Large Language Models Understand Data Visualization Rules?
Martin Sinnona, Valentin Bonas, Emmanuel Iarussi et al. (+1 more)
Data visualization rules-derived from decades of research in design and perception-ensure trustworthy chart communication. While prior work has shown that large
language models (LLMs) can generate charts or flag misleading figures, it remains unclear whether they can reason about and enforce visualization rules directly. Constraint-based systems such as Draco encode these rules as logical constraints for precise automated checks, but maintaining symbolic encodings requires expert effort, motivating the use of LLMs as flexible rule validators. In this paper, we present the first systematic evaluation of LLMs against visualization rules using hard-verification ground truth derived from Answer Set Programming (ASP). We translated a subset of Draco's constraints into natural-
language statements and generated a controlled
dataset of 2,000 Vega-Lite specifications annotated with explicit rule violations. LLMs were evaluated on both accuracy in detecting violations and
prompt adherence, which measures whether outputs follow the required structured format. Results show that frontier models achieve high adherence (Gemma 3 4B / 27B: 100%, GPT-oss 20B: 98%) and reliably detect common violations (F1 up to 0.82),yet performance drops for subtler perceptual rules (F1 < 0.15 for some categories) and for outputs generated from technical ASP formulations.Translating constraints into natural
language improved performance by up to 150% for smaller models. These findings demonstrate the potential of LLMs as flexible,
language-driven validators while highlighting their current limitations compared to symbolic solvers.
Read the full abstract ▾
All authors Martin Sinnona, Valentin Bonas, Emmanuel Iarussi, Viviana Siless
Computer Vision · Cross-listed from cs.RO
NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning
Jiahui Fu, Junyu Nan, Lingfeng Sun et al. (+7 more)
Solving complex long-horizon robotic tasks requires joint
reasoning over abstract task structure and low-level physical interaction. While combining
Vision-
Language Models (VLMs) and
video generation models offers a promising path for zero-shot planning, their individual tendencies to hallucinate physics or violate
geometric consistency often compound over time, preventing reliable real-world execution. We introduce NovaPlan, a hierarchical framework that enables robust, zero-shot long-horizon manipulation by systematically proposing, verifying, and repairing visual plans. At the high level, a VLM planner decomposes tasks and filters out dynamically inconsistent futures by verifying multiple candidate
video rollouts. To translate these imagined futures into reliable physical actions, NovaPlan utilizes a hybrid
geometric representation that adaptively switches between object-centric flow and human hand flow. Finally, NovaPlan closes the loop by continuously monitoring execution to verify outcomes and synthesize local, non-prehensile corrective behaviors, such as fingertip poking, when failures occur. Across diverse multi-stage tasks, NovaPlan substantially outperforms prior zero-shot systems, achieving complex assembly and dexterous error recovery entirely without task-specific training or demonstrations. Please visit our project website for additional results: https://nova-plan.github.io/
Read the full abstract ▾
All authors Jiahui Fu, Junyu Nan, Lingfeng Sun, Hongyu Li, Jianing Qian, Benjamin Yang, Yilun Du, Jennifer L. Barry, Kris Kitani, George Konidaris
Computer Vision
Kairan Zhao, Iurie Luca and Peter Triantafillou
Machine unlearning (MU) refers to the post-training capability to remove (the influence of) training examples that are incorrect, biased, or leak sensitive/private information. MU is now widely regarded as critical for building safe and fair AI. In parallel, research into
transformer architectures for computer
vision has been highly successful:
Vision Transformers (VTs) increasingly emerge as strong alternatives to CNNs. Yet, MU research for
vision tasks has largely centered on CNNs, not VTs. While MU
benchmarks have been developed for LLMs,
diffusion models, and CNNs, none currently exist for VTs. This work is the first to attempt this, benchmarking MU
algorithm performance across different VT families (ViT, Swin-T, and DINOv2) and at different capacities. The work employs (i) different
datasets, selected to assess the impacts of
dataset scale and complexity; (ii) different MU
algorithms, selected to represent fundamentally different approaches for MU; and (iii) both single-shot and continual unlearning protocols. Additionally, it focuses on benchmarking MU
algorithms that leverage training data memorization, since leveraging memorization has been recently discovered to significantly improve the performance of previously SOTA
algorithms. En route, the work characterizes how VTs memorize training data relative to CNNs, and assesses the impact of different memorization proxies on performance. The
benchmark uses unified evaluation metrics that capture two complementary notions of forget quality along with accuracy on unseen (test) data and on retained data. Overall, this work offers a benchmarking basis, enabling reproducible, fair, and comprehensive comparisons of existing (and future) MU
algorithms on VTs. Importantly, for the first time, it sheds light on how well existing
algorithms work in VT settings, establishing a promising reference performance baseline.
Read the full abstract ▾
Computer Vision
Transcending the Annotation Bottleneck: AI-Powered Discovery in Biology and Medicine
Soumick Chatterjee
The dependence on expert annotation has long constituted the primary rate-limiting step in the application of artificial intelligence to biomedicine. While supervised learning drove the initial wave of clinical
algorithms, a paradigm shift towards unsupervised and self-supervised learning (SSL) is currently unlocking the latent potential of biobank-scale
datasets. By learning directly from the intrinsic structure of data - whether pixels in a
magnetic resonance image (MRI), voxels in a volumetric scan, or
tokens in a genomic sequence - these methods facilitate the discovery of novel phenotypes, the linkage of morphology to genetics, and the detection of anomalies without human bias. This article synthesises seminal and recent advances in "learning without labels," highlighting how unsupervised frameworks can derive heritable cardiac traits, predict spatial gene expression in histology, and detect pathologies with performance that rivals or exceeds supervised counterparts.
Read the full abstract ▾
Computer Vision
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues
Zanxi Ruan, Songqun Gao, Qiuyu Kong et al. (+2 more)
Edge-based representations are fundamental cues for visual understanding, a principle rooted in early
vision research and still central today. We extend this principle to
vision-
language alignment, showing that isolating and aligning structural cues across modalities can greatly benefit fine-tuning on long, detail-rich captions, with a specific focus on improving cross-modal
retrieval. We introduce StructXLIP, a fine-tuning
alignment paradigm that extracts edge maps (e.g., Canny), treating them as proxies for the visual structure of an image, and filters the corresponding captions to emphasize structural cues, making them "structure-centric". Fine-tuning augments the standard
alignment loss with three structure-centric losses: (i) aligning edge maps with structural text, (ii) matching local edge regions to textual chunks, and (iii) connecting edge maps to color images to prevent representation drift. From a theoretical standpoint, while standard CLIP maximizes the mutual information between visual and textual
embeddings, StructXLIP additionally maximizes the mutual information between
multimodal structural representations. This auxiliary optimization is intrinsically harder, guiding the model toward more robust and semantically stable minima, enhancing
vision-
language alignment. Beyond outperforming current competitors on cross-modal
retrieval in both general and specialized domains, our method serves as a general boosting recipe that can be integrated into future approaches in a plug-and-play manner. Code and pretrained models are publicly available at: https://github.com/intelligolabs/StructXLIP.
Read the full abstract ▾
All authors Zanxi Ruan, Songqun Gao, Qiuyu Kong, Yiming Wang, Marco Cristani
Computer Vision
Do Large Language Models Understand Data Visualization Principles?
Martin Sinnona, Valentin Bonas, Viviana Siless et al. (+1 more)
Data visualization principles, derived from decades of research in design and perception, ensure proper visual communication. While prior work has shown that large
language models (LLMs) can generate charts or flag misleading figures, it remains unclear whether they and their
vision-
language counterparts (VLMs) can reason about and enforce visualization principles directly. Constraint based systems encode these principles as logical rules for precise automated checks, but translating them into formal specifications demands expert knowledge. This motivates leveraging LLMs and VLMs as principle checkers that can reason about visual design directly, bypassing the need for symbolic rule specification. In this paper, we present the first systematic evaluation of both LLMs and VLMs on their ability to reason about visualization principles, using hard verification ground truth derived from Answer Set Programming (ASP). We compiled a set of visualization principles expressed as natural-
language statements and generated a controlled
dataset of approximately 2,000 Vega-Lite specifications annotated with explicit principle violations, complemented by over 300 real-world Vega-Lite charts. We evaluated both checking and fixing tasks, assessing how well models detect principle violations and correct flawed chart specifications. Our work highlights both the promise of large (
vision-)
language models as flexible validators and editors of visualization designs and the persistent gap with symbolic solvers on more nuanced aspects of visual perception. They also reveal an interesting asymmetry: frontier models tend to be more effective at correcting violations than at detecting them reliably.
Read the full abstract ▾
All authors Martin Sinnona, Valentin Bonas, Viviana Siless, Emmanuel Iarussi
Computer Vision
SemanticNVS: Improving Semantic Scene Understanding in Generative Novel View Synthesis
Xinya Chen, Christopher Wewer, Jiahao Xie et al. (+2 more)
We present SemanticNVS, a camera-conditioned multi-view
diffusion model for novel view synthesis (NVS), which improves generation quality and consistency by integrating pre-trained semantic feature extractors. Existing NVS methods perform well for views near the input view, however, they tend to generate semantically implausible and distorted images under long-range camera motion, revealing severe degradation. We speculate that this degradation is due to current models failing to fully understand their conditioning or intermediate generated scene content. Here, we propose to integrate pre-trained semantic feature extractors to incorporate stronger scene semantics as conditioning to achieve high-quality generation even at distant viewpoints. We investigate two different strategies, (1) warped semantic features and (2) an alternating scheme of understanding and generation at each denoising step. Experimental results on multiple
datasets demonstrate the clear qualitative and quantitative (4.69%-15.26% in FID) improvement over state-of-the-art alternatives.
Read the full abstract ▾
All authors Xinya Chen, Christopher Wewer, Jiahao Xie, Xinting Hu, Jan Eric Lenssen
Computer Vision
The Invisible Gorilla Effect in Out-of-distribution Detection
Harry Anthony, Ziyun Liang, Hermione Warr et al. (+1 more)
Deep Neural Networks achieve high performance in
vision tasks by learning features from regions of interest (ROI) within images, but their performance degrades when deployed on out-of-distribution (OOD) data that differs from training data. This challenge has led to OOD detection methods that aim to identify and reject unreliable predictions. Although prior work shows that OOD detection performance varies by artefact type, the underlying causes remain underexplored. To this end, we identify a previously unreported bias in OOD detection: for hard-to-detect artefacts (near-OOD), detection performance typically improves when the artefact shares visual similarity (e.g. colour) with the model's ROI and drops when it does not - a phenomenon we term the Invisible Gorilla Effect. For example, in a skin lesion classifier with red lesion ROI, we show the method Mahalanobis Score achieves a 31.5% higher AUROC when detecting OOD red ink (similar to ROI) compared to black ink (dissimilar) annotations. We annotated artefacts by colour in 11,355 images from three public
datasets (e.g. ISIC) and generated colour-swapped counterfactuals to rule out
dataset bias. We then evaluated 40 OOD methods across 7
benchmarks and found significant performance drops for most methods when artefacts differed from the ROI. Our findings highlight an overlooked failure mode in OOD detection and provide guidance for more robust detectors. Code and annotations are available at: https://github.com/HarryAnthony/Invisible_Gorilla_Effect.
Read the full abstract ▾
All authors Harry Anthony, Ziyun Liang, Hermione Warr, Konstantinos Kamnitsas
Computer Vision
HeatPrompt: Zero-Shot Vision-Language Modeling of Urban Heat Demand from Satellite Images
Kundan Thota, Xuanhao Mu, Thorsten Schlachter et al. (+1 more)
Accurate heat-demand maps play a crucial role in decarbonizing space heating, yet most municipalities lack detailed building-level data needed to calculate them. We introduce HeatPrompt, a zero-shot
vision-
language energy modeling framework that estimates annual heat demand using semantic features extracted from satellite images, basic Geographic Information System (GIS), and building-level features. We feed pretrained Large
Vision Language Models (VLMs) with a domain-specific
prompt to act as an energy planner and extract the visual attributes such as roof age, building density, etc, from the RGB satellite image that correspond to the thermal load. A Multi-Layer Perceptron (MLP) regressor trained on these captions shows an $R^2$ uplift of 93.7% and shrinks the mean absolute error (MAE) by 30% compared to the baseline model. Qualitative analysis shows that high-impact
tokens align with high-demand zones, offering lightweight support for heat planning in data-scarce regions.
Read the full abstract ▾
All authors Kundan Thota, Xuanhao Mu, Thorsten Schlachter, Veit Hagenmeyer
Computer Vision
MeanFuser: Fast One-Step Multi-Modal Trajectory Generation and Adaptive Reconstruction via MeanFlow for End-to-End Autonomous Driving
Junli Wang, Yinan Zheng, Xueyi Liu et al. (+9 more)
Generative models have shown great potential in trajectory planning. Recent studies demonstrate that anchor-guided generative models are effective in modeling the uncertainty of driving behaviors and improving overall performance. However, these methods rely on discrete anchor vocabularies that must sufficiently cover the trajectory distribution during testing to ensure robustness, inducing an inherent trade-off between vocabulary size and model performance. To overcome this limitation, we propose MeanFuser, an end-to-end autonomous driving method that enhances both efficiency and robustness through three key designs. (1) We introduce Gaussian Mixture Noise (GMN) to guide generative sampling, enabling a continuous representation of the trajectory space and eliminating the dependency on discrete anchor vocabularies. (2) We adapt “MeanFlow Identity" to end-to-end planning, which models the mean velocity field between GMN and trajectory distribution instead of the instantaneous velocity field used in vanilla flow matching methods, effectively eliminating numerical errors from ODE solvers and significantly accelerating inference. (3) We design a lightweight Adaptive Reconstruction Module (ARM) that enables the model to implicitly select from all sampled proposals or reconstruct a new trajectory when none is satisfactory via
attention weights.Experiments on the NAVSIM closed-loop
benchmark demonstrate that MeanFuser achieves outstanding performance without the supervision of the PDM Score and exceptional inference efficiency, offering a robust and efficient solution for end-to-end autonomous driving. Our code and model are available at https://github.com/wjl2244/MeanFuser.
Read the full abstract ▾
All authors Junli Wang, Yinan Zheng, Xueyi Liu, Zebin Xing, Pengfei Li, Guang Li, Kun Ma, Guang Chen, Hangjun Ye, Zhongpu Xia, Long Chen, Qichao Zhang
Computer Vision · Cross-listed from cs.RO
CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigation
Apoorva Vashisth, Manav Kulshrestha, Pranav Bakshi et al. (+3 more)
Robot navigation typically assumes an obstacle-free path exists between start and goal. In real environments, however, clutter may block all routes. We introduce Lifelong Interactive
Navigation, where a mobile
robot with manipulation capabilities must move objects to forge paths and complete sequential object-placement tasks. Because environment modifications persist, decisions impact future navigability and task difficulty. We propose CoReLIN, an LLM-driven constraint-based
reasoning framework with active perception. CoReLIN reasons over a structured scene graph to decide which objects to relocate, where to place them, and where to explore next. A standard motion planner executes reliable
navigation and manipulation primitives. To evaluate long-horizon behavior, we introduce 2 new metrics - Long-term Efficiency Score (LES), a unified metric capturing success, execution efficiency, environment optimality, captured by Price of Clutter. In ProcTHOR-10k, CoReLIN outperforms best baseline by 16% under standard metrics and LES, and transfers to real-world hardware.
Read the full abstract ▾
All authors Apoorva Vashisth, Manav Kulshrestha, Pranav Bakshi, Damon Conover, Guillaume Sartoretti, Aniket Bera
Computer Vision
Decoupling Defense Strategies for Robust Image Watermarking
Jiahui Chen, Zehang Deng, Zeyu Zhang et al. (+3 more)
Deep learning-based image watermarking, while robust against conventional distortions, remains vulnerable to advanced
adversarial and regeneration attacks. Conventional countermeasures, which jointly optimize the encoder and decoder via a noise layer, face 2 inevitable challenges: (1) decrease of clean accuracy due to decoder
adversarial training and (2) limited robustness due to simultaneous training of all three advanced attacks. To overcome these issues, we propose AdvMark, a novel two-stage fine-tuning framework that decouples the defense strategies. In stage 1, we address
adversarial vulnerability via a tailored
adversarial training paradigm that primarily fine-tunes the encoder while only conditionally updating the decoder. This approach learns to move the image into a non-attackable region, rather than modifying the decision boundary, thus preserving clean accuracy. In stage 2, we tackle distortion and regeneration attacks via direct image optimization. To preserve the
adversarial robustness gained in stage 1, we formulate a principled, constrained image loss with theoretical guarantees, which balances the deviation from cover and previous encoded images. We also propose a quality-aware early-stop to further guarantee the lower bound of visual quality. Extensive experiments demonstrate AdvMark outperforms with the highest image quality and comprehensive robustness, i.e. up to 29%, 33% and 46% accuracy improvement for distortion, regeneration and
adversarial attacks, respectively.
Read the full abstract ▾
All authors Jiahui Chen, Zehang Deng, Zeyu Zhang, Chaoyang Li, Lianchen Jia, Lifeng Sun
Computer Vision
SEAL-pose: Enhancing 3D Human Pose Estimation via a Learned Loss for Structural Consistency
Yeonsung Kim, Junggeun Do, Seunguk Do et al. (+3 more)
3D human pose estimation (HPE) is characterized by intricate local and global dependencies among joints. Conventional supervised losses are limited in capturing these correlations because they treat each joint independently. Previous studies have attempted to promote structural consistency through manually designed priors or rule-based constraints; however, these approaches typically require manual specification and are often non-differentiable, limiting their use as end-to-end training objectives. We propose SEAL-pose, a data-driven framework in which a learnable loss-net trains a pose-net by evaluating structural plausibility. Rather than relying on hand-crafted priors, our joint-graph-based design enables the loss-net to learn complex structural dependencies directly from data. Extensive experiments on three 3D HPE
benchmarks with eight backbones show that SEAL-pose reduces per-joint errors and improves pose plausibility compared with the corresponding backbones across all settings. Beyond improving each backbone, SEAL-pose also outperforms models with explicit structural constraints, despite not enforcing any such constraints. Finally, we analyze the relationship between the loss-net and structural consistency, and evaluate SEAL-pose in cross-
dataset and in-the-wild settings.
Read the full abstract ▾
All authors Yeonsung Kim, Junggeun Do, Seunguk Do, Sangmin Kim, Jaesik Park, Jay-Yoon Lee
Computer Vision
Closing the gap in multimodal medical representation alignment
Eleonora Grassucci, Giordano Cicchetti and Danilo Comminiello
In
multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic
alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex
multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical
alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances
alignment between radiology images and clinical text, improving cross-modal
retrieval and image captioning.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
EEG-Driven Intention Decoding: Offline Deep Learning Benchmarking on a Robotic Rover
Ghadah Alosaimi, Maha Alsayyari, Yixin Sun et al. (+3 more)
Brain-computer interfaces (BCIs) provide a hands-free control modality for mobile robotics, yet decoding user intent during real-world
navigation remains challenging. This work presents a
brain-
robot control framework for offline decoding of driving commands during robotic rover operation. A 4WD Rover Pro platform was remotely operated by 12 participants who navigated a predefined route using a joystick, executing the commands forward, reverse, left, right, and stop. Electroencephalogram (
EEG) signals were recorded with a 16-channel OpenBCI cap and aligned with motor actions at Delta = 0 ms and future prediction horizons (Delta > 0 ms). After preprocessing, several deep learning models were benchmarked, including convolutional neural networks, recurrent neural networks, and
Transformer architectures. ShallowConvNet achieved the highest performance for both action prediction and intent prediction. By combining real-world robotic control with multi-horizon
EEG intention decoding, this study introduces a reproducible
benchmark and reveals key design insights for predictive deep learning-based BCI systems.
Read the full abstract ▾
All authors Ghadah Alosaimi, Maha Alsayyari, Yixin Sun, Stamos Katsigiannis, Amir Atapour-Abarghouei, Toby P. Breckon
Computer Vision
Token-UNet: A New Case for Transformers Integration in Efficient and Interpretable 3D UNets for Brain Imaging Segmentation
Louis Fabrice Tshimanga, Andrea Zanola, Federico Del Pup et al. (+1 more)
We present
Token-UNet, adopting the TokenLearner and TokenFuser modules to encase
Transformers into UNets. While
Transformers have enabled global interactions among input elements in medical imaging, current computational challenges hinder their deployment on common hardware. Models like (Swin)UNETR adapt the UNet architecture by incorporating (Swin)
Transformer encoders, which process
tokens that each represent small subvolumes ($8^3$ voxels) of the input. The
Transformer attention mechanism scales quadratically with the number of
tokens, which is tied to the cubic scaling of 3D input resolution. This work reconsiders the role of convolution and
attention, introducing
Token-UNets, a family of 3D
segmentation models that can operate in constrained computational environments and time frames. To mitigate computational demands, our approach maintains the convolutional encoder of UNet-like models, and applies TokenLearner to 3D feature maps. This module pools a preset number of
tokens from local and global structures. Our results show this tokenization effectively encodes task-relevant information, yielding naturally interpretable
attention maps. The
memory footprint, computation times at inference, and parameter counts of our heaviest model are reduced to 33%, 10%, and 35% of the SwinUNETR values, with better average performance (86.75% $\pm 0.19\%$ Dice score for SwinUNETR vs our 87.21% $\pm 0.35\%$). This work opens the way to more efficient trainings in contexts with limited computational resources, such as 3D medical imaging. Easing model optimization, fine-tuning, and transfer-learning in limited hardware settings can accelerate and diversify the development of approaches, for the benefit of the research community.
Read the full abstract ▾
All authors Louis Fabrice Tshimanga, Andrea Zanola, Federico Del Pup, Manfredo Atzori
Computer Vision
RADE-Net: Robust Attention Network for Radar-Only Object Detection in Adverse Weather
Christof Leitgeb, Thomas Puchleitner, Max Peter Ronecker et al. (+1 more)
Automotive perception systems are obligated to meet high requirements. While optical sensors such as Camera and
Lidar struggle in adverse weather conditions, Radar provides a more robust perception performance, effectively penetrating fog, rain, and snow. Since full Radar tensors have large data sizes and very few
datasets provide them, most Radar-based approaches work with sparse point clouds or 2D projections, which can result in information loss. Additionally, deep learning methods show potential to extract richer and more dense features from low level Radar data and therefore significantly increase the perception performance. Therefore, we propose a 3D projection method for fast-
Fourier-transformed 4D Range-Azimuth-Doppler-Elevation (RADE) tensors. Our method preserves rich Doppler and Elevation features while reducing the required data size for a single frame by 91.9% compared to a full tensor, thus achieving higher training and inference speed as well as lower model complexity. We introduce RADE-Net, a lightweight model tailored to 3D projections of the RADE tensor. The backbone enables exploitation of low-level and high-level cues of Radar tensors with spatial and channel-
attention. The decoupled detection heads predict object center-points directly in the Range-Azimuth domain and regress rotated 3D bounding boxes from rich feature maps in the cartesian scene. We evaluate the model on scenes with multiple different road users and under various weather conditions on the large-scale K-Radar
dataset and achieve a 16.7% improvement compared to their baseline, as well as 6.5% improvement over current Radar-only models. Additionally, we outperform several
Lidar approaches in scenarios with adverse weather conditions. The code is available under https://github.com/chr-is-tof/RADE-Net.
Read the full abstract ▾
All authors Christof Leitgeb, Thomas Puchleitner, Max Peter Ronecker, Daniel Watzenig
Computer Vision
RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflection
Tianyu Wang, Zhiyuan Ma, Qian Wang et al. (+3 more)
Recent advancements in image generation have achieved impressive results in producing high-quality images. However, existing image generation models still generally struggle with a spatial
reasoning dilemma, lacking the ability to accurately capture fine-grained spatial relationships from the
prompt and correctly generate scenes with structural integrity. To mitigate this dilemma, we propose RL-RIG, a Reinforcement Learning framework for Reflection-based Image Generation. Our architecture comprises four primary components: Diffuser, Checker, Actor, and Inverse Diffuser, following a Generate-Reflect-Edit paradigm to spark the Chain of Thought
reasoning ability in image generation for addressing the dilemma. To equip the model with better intuition over generation trajectories, we further develop Reflection-GRPO to train the VLM Actor for edit
prompts and the Image Editor for better image quality under a given
prompt, respectively. Unlike traditional approaches that solely produce visually stunning yet structurally unreasonable content, our evaluation metrics prioritize spatial accuracy, utilizing Scene Graph IoU and employing a VLM-as-a-Judge strategy to assess the spatial consistency of generated images on LAION-SG
dataset. Experimental results show that RL-RIG outperforms existing state-of-the-art open-source models by up to 11% in terms of controllable and precise spatial
reasoning in image generation.
Read the full abstract ▾
All authors Tianyu Wang, Zhiyuan Ma, Qian Wang, Xinyi Zhang, Xinwei Long, Bowen Zhou
Computer Vision
When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators
Krzysztof Adamkiewicz, Brian Bernhard Moser, Stanislav Frolov et al. (+3 more)
Recent text-to-image (T2I)
diffusion models produce visually stunning images and demonstrate excellent
prompt following. But do they perform well as synthetic
vision data generators? In this work, we revisit the promise of synthetic data as a scalable substitute for real training sets and uncover a surprising performance regression. We generate large-scale synthetic
datasets using state-of-the-art T2I models released between 2022 and 2025, train standard classifiers solely on this synthetic data, and evaluate them on real test data. Despite observable advances in visual fidelity and
prompt adherence, classification accuracy on real test data consistently declines with newer T2I models as training data generators. Our analysis reveals a hidden trend: These models collapse to a narrow, aesthetic-centric distribution that undermines diversity and real data distribution coverage. Overall, our findings challenge a growing assumption in
vision research, namely that progress in generative realism implies progress in data realism. We thus highlight an urgent need to rethink the capabilities of modern T2I models as reliable training data generators.
Read the full abstract ▾
All authors Krzysztof Adamkiewicz, Brian Bernhard Moser, Stanislav Frolov, Tobias Christian Nauen, Federico Raue, Andreas Dengel
Computer Vision
Discover, Segment, and Select: A Progressive Mechanism for Zero-shot Camouflaged Object Segmentation
Yilong Yang, Jianxin Tian, Shengchuan Zhang et al. (+1 more)
Current zero-shot Camouflaged Object
Segmentation methods typically employ a two-stage pipeline (discover-then-segment): using MLLMs to obtain visual
prompts, followed by SAM
segmentation. However, relying solely on MLLMs for camouflaged object discovery often leads to inaccurate localization, false positives, and missed detections. To address these issues, we propose the Discover-Segment-Select (DSS) mechanism, a progressive framework designed to refine
segmentation step by step. The proposed method contains a Feature-coherent Object Discovery (FOD) module that leverages visual features to generate diverse object proposals, a
segmentation module that refines these proposals through SAM
segmentation, and a Semantic-driven Mask Selection (SMS) module that employs MLLMs to evaluate and select the optimal
segmentation mask from multiple candidates. Without requiring any training or supervision, DSS achieves state-of-the-art performance on multiple COS
benchmarks, especially in multiple-instance scenes.
Read the full abstract ▾
All authors Yilong Yang, Jianxin Tian, Shengchuan Zhang, Liujuan Cao
Computer Vision
Learning Positive-Incentive Point Sampling in Neural Implicit Fields for Object Pose Estimation
Yifei Shi, Boyan Wan, Xin Xu et al. (+1 more)
Learning neural implicit fields of 3D shapes is a rapidly emerging field that enables shape representation at arbitrary resolutions. Due to the flexibility, neural implicit fields have succeeded in many research areas, including shape reconstruction, novel view image synthesis, and more recently, object pose estimation. Neural implicit fields enable learning dense correspondences between the camera space and the object's canonical space-including unobserved regions in camera space-significantly boosting object pose estimation performance in challenging scenarios like highly occluded objects and novel shapes. Despite progress, predicting canonical coordinates for unobserved camera-space regions remains challenging due to the lack of direct observational signals. This necessitates heavy reliance on the model's generalization ability, resulting in high uncertainty. Consequently, densely sampling points across the entire camera space may yield inaccurate estimations that hinder the learning process and compromise performance. To alleviate this problem, we propose a method combining an SO(3)-equivariant convolutional implicit network and a positive-incentive point sampling (PIPS) strategy. The SO(3)-equivariant convolutional implicit network estimates point-level attributes with SO(3)-equivariance at arbitrary query locations, demonstrating superior performance compared to most existing baselines. The PIPS strategy dynamically determines sampling locations based on the input, thereby boosting the network's accuracy and training efficiency. Our method outperforms the state-of-the-art on three pose estimation
datasets. Notably, it demonstrates significant improvements in challenging scenarios, such as objects captured with unseen pose, high occlusion, novel geometry, and severe noise.
Read the full abstract ▾
All authors Yifei Shi, Boyan Wan, Xin Xu, Kai Xu
Computer Vision · Cross-listed from cs.LG
Expanding the Role of Diffusion Models for Robust Classifier Training
Pin-Han Huang, Shang-Tse Chen and Hsuan-Tien Lin
Incorporating
diffusion-generated synthetic data into
adversarial training (AT) has been shown to substantially improve the training of robust image classifiers. In this work, we extend the role of
diffusion models beyond merely generating synthetic data, examining whether their internal representations, which encode meaningful features of the data, can provide additional benefits for robust classifier training. Through systematic experiments, we show that
diffusion models offer representations that are both diverse and partially robust, and that explicitly incorporating
diffusion representations as an auxiliary learning signal during AT consistently improves robustness across settings. Furthermore, our representation analysis indicates that incorporating
diffusion models into AT encourages more disentangled features, while
diffusion representations and
diffusion-generated synthetic data play complementary roles in shaping representations. Experiments on CIFAR-10, CIFAR-100, and ImageNet validate these findings, demonstrating the effectiveness of jointly leveraging
diffusion representations and synthetic data within AT.
Read the full abstract ▾
Computer Vision
Augmented Radiance Field: A General Framework for Enhanced Gaussian Splatting
Yixin Yang, Bojian Wu, Yang Zhou et al. (+1 more)
Due to the real-time rendering performance, 3D Gaussian Splatting (3DGS) has emerged as the leading method for radiance field reconstruction. However, its reliance on spherical harmonics for color encoding inherently limits its ability to separate diffuse and specular components, making it challenging to accurately represent complex reflections. To address this, we propose a novel enhanced Gaussian kernel that explicitly models specular effects through view-dependent opacity. Meanwhile, we introduce an error-driven compensation strategy to improve rendering quality in existing 3DGS scenes. Our method begins with 2D Gaussian initialization and then adaptively inserts and optimizes enhanced Gaussian kernels, ultimately producing an augmented radiance field. Experiments demonstrate that our method not only surpasses state-of-the-art NeRF methods in rendering performance but also achieves greater parameter efficiency. Project page at: https://xiaoxinyyx.github.io/augs.
Read the full abstract ▾
All authors Yixin Yang, Bojian Wu, Yang Zhou, Hui Huang