Computer Vision
Perceptual 3D Simulation With Physical World Modeling
Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh et al. (+2 more)
Predicting how a scene will evolve after a desired 3D transformation from images is a central goal in
vision, graphics, and robotics. Yet unlike ideal simulators with full access to 3D geometry and dynamics, real world systems must rely on perceptual inputs and local actions that are inherently partial and incomplete. In this work, we present P3Sim, a physical world modeling system that simulates future scene states under both partial observations and incomplete 3D transformation signals. P3Sim is composed of three interacting components: a learned physical world model, a
geometric conditioning module, and a persistent scene
memory. The world model interprets perception as probabilistic inference over
multimodal scene variables, providing predictions of the distributions of any scene variable conditioned on any combination of others. The
geometric conditioning module provides a partial 3D transform signal for conditioning the world model at inference time. The persistent scene
memory integrates predictions over time, enabling online updates and consistency under uncertainty. By combining learned inference with explicit
geometric structure, P3Sim balances data-driven flexibility with built-in inductive bias. This design yields a flexible perceptual simulator that generalizes across diverse 3D transformation tasks, such as novel view synthesis, object manipulation, and dynamic scene prediction, advancing toward general purpose 3D scene understanding and transformation.
Read the full abstract ▾
All authors Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh, Jared Watrous, Daniel L. K. Yamins
Computer Vision
Radar Guided Camera Verification for Automatic Emergency Braking Rethinking Object Detection in Radar Camera Fusion
Ram Charan Akula, Sivanathan Kandhasamy and Manikandan Ganesan
Radar camera fusion is widely used in Automatic Emergency Braking AEB systems because radar provides reliable range and velocity measurements while cameras provide a proper visual confirmation of the objects . Most of the deployed systems perform this confirmation using computationally intensive object detectors. However, if the radar has already localized a target, the camera may only need to verify the obstacles presence rather than solving a full problem by identifying the object. Our work proposes a radar scoped edge density gate that performs obstacle verification within radar guided image regions of interest. This method requires no training data, model weights, or GPU acceleration and was integrated into a complete radar camera fusion AEB system with brake by wire actuation. Evaluated on a real instrumented vehicle across 72 driving sessions and 131,603 camera frames, the proposed approach reduced the camera search space by up to 98.7 percentage, achieved a mean processing latency of 0.121 ms per ROI, an AUC of 0.898, and a recall of 0.994. Across 33 staged threat scenarios, the complete AEB system recorded zero missed brake events.
Read the full abstract ▾
Computer Vision
Understanding Cross-Rig Generalization in Automotive Perception: a Multi-Rig Benchmark and Rig Variation Metrics
Tim Alexander Bader, Tim Dieter Eberhardt, Maximilian Dillitzer et al. (+1 more)
Camera-based perception systems for autonomous driving are typically developed and evaluated using fixed sensor rigs, while real-world vehicle fleets exhibit substantial variation in camera placement, orientation, field of view, and camera count. This mismatch introduces a cross-rig domain gap in which only the
geometric observation process changes. To study this effect under controlled conditions, we introduce Plentiful CARLA Camera Rigs, a
benchmark that renders identical driving scenes under 14 systematically designed camera rigs. This setup enables direct analysis of cross-rig generalization without confounding changes in scene content or appearance. Using the
benchmark, we analyze cross-rig transfer behavior of representative multi-view perception architectures and observe substantial performance shifts induced by
geometric rig variation. To facilitate structured analysis, we further introduce two calibration-based descriptors derived from rig metadata: Rig Variance, capturing internal rig diversity, and Rig Contrastive Distance, measuring
geometric discrepancy between rigs. Our experiments show that
geometric rig differences strongly correlate with relative cross-rig performance shifts and that Rig Contrastive Distance provides a reliable proxy for ranking transfer difficulty between sensor rigs.
Read the full abstract ▾
All authors Tim Alexander Bader, Tim Dieter Eberhardt, Maximilian Dillitzer, Wilhelm Stork
Computer Vision
Beyond MoCap: Scaling Motion Tokenizers with Synthetic Human Motion for Generative Modeling
Yiwen Yan, Wanning He and Yu-Wing Tai
Human motion generation models are fundamentally constrained by the limited diversity of motion capture
datasets, which predominantly contain common, repetitive actions and fail to cover the long tail of complex human movements, resulting in a restricted motion vocabulary in learned latent representations and poor generalization to rare, compositional, and highly dynamic motions. In this work, we propose a framework for expanding the motion representation space by leveraging large-scale synthetic human motion, introducing a data generation pipeline that produces diverse, physically plausible motion sequences beyond the distribution of existing
datasets and integrating it with a redesigned VQ-VAE tokenizer that adapts to this expanded motion space. Unlike conventional tokenizers trained on narrow data distributions, our approach jointly scales both the training distribution and the discrete codebook, enabling the model to capture a significantly richer set of motion primitives. We demonstrate that training with synthetic motion substantially improves the coverage and compositionality of the learned motion vocabulary, leading to consistent gains across motion generation tasks such as text-to-motion and motion continuation, while remaining fully compatible with existing frameworks including MotionGPT. Our results suggest that the primary bottleneck lies in the limited support of the learned motion representation, rather than model architecture alone. Scaling synthetic motion in tandem with representation learning offers a principled path toward more expressive, controllable, and generalizable human motion synthesis.
Read the full abstract ▾
Computer Vision
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
Haoyu Chen, Kaichen Zhou, Hang Hua et al. (+11 more)
Video generation models aspire to simulate dynamic environments, and several
benchmarks now evaluate
memory consistency across frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, a diagnostic
benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment across four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regarding
memory consistency under the disappear-and-reappear paradigm.
Read the full abstract ▾
All authors Haoyu Chen, Kaichen Zhou, Hang Hua, Kaile Zhang, Jingwen Qian, Wufei Ma, Haonan Chen, Chunjiang Liu, Yizhou Zhao, Xiaoyuan Wang, Weiyue Li, Alan Yuille, Paul Pu Liang, Yilun Du
Computer Vision
Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
Thomas Shih-Chao Liang, Zhuoran Yu and Yong Jae Lee
Large
Language Models (LLMs) possess broad conceptual knowledge acquired through large-scale text pretraining, yet their potential to supervise models in other modalities remains underexplored. In this work, we propose LaViD–
Language-to-Visual Knowledge
Distillation–a simple and effective framework for transferring high-level semantic knowledge from a
language-only teacher to a
vision-only student model. Instead of relying on paired
multimodal data, LaViD elicits conceptual signals from an LLM by prompting it to generate multiple-choice questions (MCQs) that probe semantic distinctions between visual classes. Each class is mapped to a soft label distribution over these MCQs, forming a rich conceptual signature that guides the student through an auxiliary
distillation loss. Notably, despite using a
language-only teacher without access to image data, LaViD consistently outperforms recent methods like MaKD that distill from
vision-
language models across multiple fine-grained
benchmarks. It also achieves competitive or superior performance compared to state-of-the-art visual
distillation methods such as DKD and MLKD, with further gains when combined with logit standardization. On the Waterbirds
dataset, LaViD substantially improves worst-group accuracy, demonstrating enhanced robustness to spurious correlations with
distillation. Code is available at https://github.com/lliangthomas/lavid.
Read the full abstract ▾
Computer Vision
Tessellating The Earth
Daniel Cher, Hamza Iqbal, Eric Xing et al. (+2 more)
Geolocation encoders, which map geographic coordinates to learned representations, are emerging as an effective means of capturing visual and non-visual characteristics from a latitude-longitude pair alone. However, existing approaches project coordinates onto fixed bases (e.g., spherical harmonics), allocating representational capacity uniformly and devoting equal resources to the open ocean and to a developing city. We introduce Tessellating the Earth (TTE), a location encoder built from learnable Spherical Voronoi partitions that concentrates representational capacity where it is needed in a fully differentiable, end-to-end manner. Each Voronoi site carries its own
embedding and migrates during training toward discriminative areas. To bridge the gap between local spatial structure and global semantic understanding, we introduce global semantic
tokens: a set of shared learnable concept
tokens that distill semantic knowledge from the satellite imagery into a compact vocabulary the location encoder can reference at inference, enabling geographically distant sites covering similar environments to share semantics. TTE sets a new state of the art for location encoders across a suite of geospatial classification and regression tasks, and achieves the strongest results when used as a geographic prior for fine-grained species classification on iNaturalist-2018. Code, and weights are available at https://github.com/mvrl/TTE.
Read the full abstract ▾
All authors Daniel Cher, Hamza Iqbal, Eric Xing, Brian Wei, Nathan Jacobs
Computer Vision
Structured-Li-GS: Structured 3D Gaussians Splatting with LiDAR Incorporation and Spatial Constraints
Huaiyuan Weng, Huibin Li and Chul Min Yeum
In this study, we develop a Structured framework for Gaussian Splatting (3DGS) with
LiDAR integration (Structured-Li-GS). It is a lightweight Gaussian Splatting pipeline that leverages
LiDAR-inertial-visual SLAM. Structured-Li-GS achieves high-quality 3D reconstructions with fewer Gaussians by training on accurate, dense, colorized point clouds. Gaussian primitives are anchored using sub-sampled point clouds, and their ellipsoidal parameters are initialized from local surface geometry. Our training strategy integrates a comprehensive set of loss terms, including photometric, flattening, offset, depth, and normal losses, guided by the dense point cloud, enabling accurate reconstruction without Gaussian densification. This approach produces up-to-scale, high-fidelity results with a moderate model size. For experimental validation, we develop a custom hardware-synchronized
LiDAR-camera handheld scanner. Experiments on both
benchmark datasets and our real-world in-house
dataset demonstrate that Structured-Li-GS surpasses state-of-the-art methods while using fewer Gaussians.
Read the full abstract ▾
Computer Vision
TruEye: Fine-Grained Detection of AI-Generated Human Subjects in Images
Jay Barot and Dan Lin
AI generated images are proliferating across the Internet. While some are used for entertainment, others are weaponized for fraud and social engineering attacks on social media users. Existing detectors overfit to generators seen during training, treat detection as opaque binary classification, or rely on costly Large
Language Models (LLMs) to explain their outputs. In this paper, we present TruEye, a novel model for fine grained detection and localization of AI manipulated or AI generated humans and scenes. Unlike conventional detectors that assign a single authenticity label, TruEye is the first to distinguish among five compositional categories of synthetic content, including the most challenging case in which a real human is composited into a real scene where they were never physically present. At its core is a mask conditioned dual stream
transformer that separates human and scene
tokens while preserving patch level spatial correspondence. Specialized
reasoning within each stream and region gated cross
attention enforce semantic coherence between subject and background, while
token level supervision and global compositional classification yield robust, interpretable predictions without invoking an LLM. By restricting intra stream
attention to semantically coherent
tokens, TruEye also runs over $100\times$ faster than LLM based competitors. Experiments on 6
datasets and our newly curated FineSyn
dataset, show that TruEye surpasses state of the art detectors with higher accuracy, faster inference, and stronger generalization to unseen AI generated or manipulated images.
Read the full abstract ▾
Computer Vision
ReWorld: Representation Learning for World Action Models
Tianze Xia, Lijun Zhou, Kaixin Xiong et al. (+8 more)
World Action Models (WAMs) unify future environment prediction with action generation for autonomous driving, yet existing approaches optimize only the final outputs, leaving intermediate representations as incidental byproducts. We present ReWorld, the first representation learning framework specifically designed for autonomous-driving WAMs. ReWorld explicitly optimizes the latent world-to-action pathway through three complementary mechanisms. First, it imposes future-predictive supervision on intermediate
Video DiT states to encode temporal scene dynamics, enabling self-guided sampling and a roughly twofold convergence speedup. Second, it aligns Action DiT states with their attended
video readouts so that the retrieved world information is retained in the representations used for planning. Third, it shapes the action space using geometrically close yet low-scoring hard negatives to separate the expert trajectory from nearby unsafe alternatives. ReWorld constructs supervision entirely from the WAM's own generation targets and attended features, requiring no external encoders or teacher models and introducing only 0.3% additional per-step training cost. Experiments show that ReWorld reduces FVD from 81.3 to 61.9 on nuScenes, improves closed-loop PDMS from 89.1 to 90.4 on NAVSIM without reinforcement learning or test-time scoring, and increases frozen linear-probe accuracy from 68.3% to 80.2% on UCF-101 action recognition. These results indicate that explicitly optimized representations are central to translating world knowledge into planning capability in WAMs.
Read the full abstract ▾
All authors Tianze Xia, Lijun Zhou, Kaixin Xiong, Jingfeng Yao, Zhenxin Zhu, Haiyang Sun, Bing Wang, Guang Chen, Wenyu Liu, Hangjun Ye, Xinggang Wang
Computer Vision
Jaume Guasch-Martí, Enrique Lopez-Cuena, Martín Suárez-Fernández et al. (+3 more)
Large
Vision-
Language Models (LVLMs) specialized in healthcare are emerging as a promising research direction due to their potential impact in clinical and biomedical applications. However, progress is constrained by the scarcity of high-quality medical
multimodal data, concerns about robustness in safety-critical settings, and the narrow and potentially contaminated evaluation
benchmarks that limit reliable assessment. To address these issues, the field requires state-of-the-art solutions to be fully open and reproducible systems in which all components can be inspected, evaluated, and improved. This work introduces Aloe-
Vision-Data, a large-scale, quality-filtered mixture which integrates both medical and general domains across
multimodal and text-only sources, designed for direct use in model fine-tuning. Building on this
dataset, we train the Aloe-
Vision family of medical LVLMs, openly released with full weights, training recipes and data, in two scales (7B and 72B). Through comprehensive benchmarking, we demonstrate that high quality training mixtures produce balanced LVLMs which yield significant gains over the baseline models without compromising general capabilities, achieving competitive performance with respect to state-of-the-art alternatives. To support reliable evaluation, we introduce CareQA-
Vision, a carefully curated
vision benchmark derived from MIR and EIR exams, the residency entrance exams for medical and nursing specialists in Spain, offering novel
vision questions with low likelihood of contamination. Finally, we show that current LVLMs remain vulnerable to
adversarial and misleading inputs, underscoring reliability challenges in clinical contexts.
Read the full abstract ▾
All authors Jaume Guasch-Martí, Enrique Lopez-Cuena, Martín Suárez-Fernández, Jordi Bayarri-Planas, Anna Arias-Duart, Dario Garcia-Gasulla
Computer Vision
DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection
Yujin Tang, Chenming Shang, Ruize Xu et al. (+1 more)
Agent benchmarks for measuring
memory largely study textual cases, in which information is deliberately extracted from the environment, written down, and then later retrieved. In other words, they assess what
agents elected to record, not what they happened to see. We introduce DMV-Bench (code: https://github.com/yyyujintang/DMV-Bench), the first interactive
benchmark for visual
memory in
multimodal agents, to study this often-neglected property. DMV-Bench is built on (1) a controlled home-furnishing e-commerce environment, supported by a catalog of 1,000 product variants, and (2) a text-leakage contract which ensures that the primary discriminative signal of each task is solely in the pixels. In DMV-Bench,
agents undergo chains of autonomous shopping sessions in which every visited product image carries a unique, pre-rendered incidental cue that the
agent is later asked to recall. We show that conventional solutions struggle with this task. Inspired by dual-coding theory, we propose a
memory architecture that uses parallel visual and verbal codes, which we call DualMem. On DMV-Bench, DualMem outperforms a caption-only baseline and three recent
multimodal agent-
memory systems across multi-session chain lengths on multiple models. These gains persist even adjusting for
memory-bank size and encoding-position bias. Further experiments also reveal an asymmetric division of labor between the two codes; a weighted coding scheme is often strongest. We view this as a step towards
memory systems that preserve a richer record of
agents' observations.
Read the full abstract ▾
All authors Yujin Tang, Chenming Shang, Ruize Xu, Nikhil Singh
Computer Vision
SelectAnyTree: A Promptable Instance Segmentation Model for 3D Forest LiDAR Point Clouds
Trung Thanh Nguyen, Daniel Lusk, Kilian Gerberding et al. (+10 more)
Instance
segmentation of trees in forest
LiDAR point clouds is constrained by label scarcity: A single hectare holds millions of points and hundreds of overlapping tree crowns, making manual annotation laborious, while automatic pre-
segmentations offer no interactive refinement. Inspired by the promptable paradigm of foundation
segmentation models, we propose SelectAnyTree, which delineates any individual tree in a 3D forest point cloud from a few clicks and is purpose-built for promptable instance
segmentation of 3D forest
LiDAR scenes. The proposed SelectAnyTree couples three lightweight stages: (1) Sparse voxel scene encoder that embeds the forest once into reusable features, (2) Click-to-query
prompt encoder that turns each click into a single content query from its 3D position, positive/negative polarity, and the backbone feature of its nearest voxel, and (3) State-space query decoder that converts this query into one tree mask with linear-time complexity, with a mask feedback that conditions each refinement round on the previous mask. Each additional tree therefore costs only a lightweight
prompt-encoding and decoding pass, and the full model requires just 19.4 M parameters, far fewer than prior promptable 3D models. Additionally, we exploit forest-aware information by detecting treetops as local maxima of the Canopy Height Model (CHM) computed from the scene geometry, and associating one with the user's click as a free initial click. Across seven diverse forest regions and an independent held-out
dataset, SelectAnyTree segments a target tree to 79.9 Intersection-over-Union (IoU) from a single click, 24.7 points above the strongest promptable baseline, and reaches every accuracy target with the fewest clicks. The source code is available at https://github.com/thanhhff/SelectAnyTree.
Read the full abstract ▾
All authors Trung Thanh Nguyen, Daniel Lusk, Kilian Gerberding, Janusch Vajna-Jehle, Tuan-Anh Vu, Duc Viet Le, Tu Vo, Phi Le Nguyen, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide, Julian Frey, Teja Kattenborn
Computer Vision
Fine-tuning a multimodal large language model for clinician-grade autism behavioral scoring from short home videos
Mohammadmahdi Honarmand, Parnian Azizian, Aaron Kline et al. (+9 more)
Autism spectrum disorder (ASD) affects 1 in 31 US children, yet median age at diagnosis exceeds four years. Artificial intelligence pipelines that provide quantified diagnosis using easy to access observational data (e.g., home
videos) could help with earlier diagnosis, and timely delivery of early treatments. We fine-tuned Gemini 2.5 Pro on 400 clinician-rated home
videos with low-rank adaptation, training only on 30 behavioral features previously validated to produce reliable predictions when passed to various ML models. On 99 held-out children (49 ASD, 50 neurotypical), inter-rater reliability with clinicians (per-feature weighted Cohen's kappa) improved by 40% (p<0.001), with 27 of 28 evaluable features improving. As an emergent zero-shot capability, direct ASD diagnosis F1 improved by 53% (p<0.001), matching or exceeding clinician outcomes. Classifier-assisted pipelines using fine-tuned LLM-derived behavioral features matched clinician-scored inputs across all tested pathways and achieved 77% accuracy (95% CI: 68-85%) and an AUC of 86% (95% CI: 78-92%). Fine-tuned
multimodal LLMs can serve as scalable behavioral feature extractors for use in autism assessment and diagnosis.
Read the full abstract ▾
All authors Mohammadmahdi Honarmand, Parnian Azizian, Aaron Kline, Kae Nurge, Zerin Nasrin Tumpa, Saimourya Surabhi, Kaitlyn Dunlap, Yang Qian, Ali Kargarandehkordi, Sameer Neupane, Peter Washington, Dennis P. Wall
Computer Vision
SemCityLoc: Aerial 6DoF Localization Using Semantic 3D City Models
Jingfeng Mao, Xuyang Chen, Qilin Zhang et al. (+6 more)
Aerial 6DoF localization typically relies on precise GNSS signals or radiometrically rich 3D reconstructions, limiting scalability and on-board deployment. We propose SemCityLoc, a semantic-
geometric alignment system that reframes aerial pose estimation as structured surface registration between foundation-model-derived visual priors and standardized LoD-compliant 3D city models. Instead of matching sparse contours or dense texture, our method aligns semantic surfaces and
monocular depth with lightweight semantic 3D building models, increasing pose discriminability in repetitive and occluded urban environments. To enable accurate evaluation, we introduce SemCityLockeD, the first real-world
benchmark combining centimeter-accurate
UAV poses with standardized LoD1–LoD3 semantic city models and challenging low-altitude imagery. Experiments demonstrate substantial improvements over existing map-based approaches, improving recall by up to 36% and reducing mean positional error from 9.89m to 2.62m in challenging urban canyons. Our results indicate that semantically structured geometry provides sufficient and scalable constraints for high-precision aerial localization without radiometric scene reconstructions. The code and data are available at https://albertchen98.github.io/SemCityLoc.
Read the full abstract ▾
All authors Jingfeng Mao, Xuyang Chen, Qilin Zhang, Oussema Dhaouadi, Guangming Wang, Brian Sheil, Daniel Cremers, Yan Xia, Olaf Wysocki
Computer Vision
From Scene-Centric to Observer-Centric: Modeling Observer-Aware Relations for 3D Scene Graph Generation
Jingjun Sun, Chaowei Wang, Zhirui Liu et al. (+5 more)
3D Scene Graph Generation (3DSGG) represents 3D scenes as structured object–relation–object graphs for spatial understanding. In observer-centric spatial perception, the same scene may be expressed under different local observer frames while its structure remains unchanged. However, existing models typically assume a fixed scene-aligned reference frame and may produce semantically inconsistent predictions when the scene is re-expressed in another observer frame. We attribute this failure to the heterogeneous frame dependency of relational predicates. Directional predicates such as $\textit{left}$, $\textit{front}$, $\textit{right}$, and $\textit{behind}$ are $\textbf{Observer-Dependent Relations}$, whereas most contact, support, and semantic predicates, such as $\textit{standing on}$ and $\textit{attached to}$, are approximately $\textbf{Observer-Independent Relations}$. Conventional models do not distinguish these frame responses, leading to degraded relation prediction under observer-frame reorientation. We introduce $\textbf{Observer-Aware Relations (OAR)}$, which combines observer-aware
geometric encoding and relation specialization, supported by frame-stable object encoding, for unified multi-label predicate prediction. Experiments on 3DSSG show that OAR consistently outperforms baselines across controlled observer-frame reorientations without training-time frame-reorientation augmentation, while remaining competitive on the standard
benchmark. The project page is available at https://oar-predicate.github.io/.
Read the full abstract ▾
All authors Jingjun Sun, Chaowei Wang, Zhirui Liu, Jiaxu Tian, Ming Yang, Yaoxing Wang, Yan Di, Shan Gao
Computer Vision
DanceOPD: On-Policy Generative Field Distillation
Wei Zhou, Xiongwei Zhu, Zelin Xu et al. (+8 more)
Modern image generation demands a single model that unifies diverse capabilities, including text-to-image (T2I), local editing, and global editing. However, these capabilities are rarely naturally aligned and often conflict. For instance, editing tends to degrade T2I performance, while global and local editing interfere with each other. Consequently, effectively composing these capabilities has become a central challenge for image generation model training. To tackle this, we introduce DanceOPD, an on-policy generative field
distillation framework for flow-matching models that routes each sample to one capability field, queries one low-noise student-induced state, and trains with a simple velocity MSE objective. With each capability source defined as a velocity field over the shared flow state space, the student learns from fields queried on its own rollout states to compose expert capabilities. This formulation also absorbs operator-defined fields such as classifier-free guidance. Comprehensive experiments on T2I, editing, realism-field absorption, and CFG absorption show that our approach improves multi-capability composition, strengthening target capabilities while preserving anchor generation quality. We believe this work establishes a practical route for generative field
distillation in flow-matching models.
Read the full abstract ▾
All authors Wei Zhou, Xiongwei Zhu, Zelin Xu, Bo Dong, Lixue Gong, Yongyuan Liang, Meng Chu, Leigang Qu, Lingdong Kong, Wei Liu, Tat-Seng Chua
Computer Vision
Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards
Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar et al. (+5 more)
Most unified large
multimodal models (LMMs) that support both visual understanding and image generation still rely on curated post-training supervision, such as human annotations, preference labels, or external reward models. We ask whether a unified LMM can improve both abilities autonomously using only unlabeled images. We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a Solver that answers and evaluates them, and a Generator that synthesizes images. Training uses only self-derived consistency signals, without human annotations, preference labels, or task-trained external reward/judge models. To stabilize learning, we introduce Solver
Token Entropy (STE), a continuous difficulty signal based on
token-level prediction uncertainty that remains useful even when sample-level consistency becomes unreliable. For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning. This creates a solver-mediated coupling, where better visual understanding enables more reliable generation assessment and stronger internal training signals. The framework preserves the same role decomposition, reward logic, and training schedule across
diffusion-based BLIP3o, rectified-flow BAGEL, and autoregressive VARGPT-v1.1 architectures, requiring only each backbone's native prompting and generation interface. Across eight understanding metrics, our method consistently improves over the corresponding base models. On BAGEL, it achieves a $+3.5\%$ absolute gain on MMMU and improves GenEval image generation performance from $82\%$ to $85\%$. Code and models are publicly released.
Read the full abstract ▾
All authors Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar, Abdelrahman Shaker, Fahad Khan, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer
Computer Vision · Cross-listed from cs.RO
World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays
Manish Kumar Govind, Dominick Reilly, Smit Patel et al. (+2 more)
Going beyond predicting
robot actions, World Action Models (WAMs) can also generate future visual observations. We build on this generative capability to propose Recurrent Generative Replay (REGEN), a continual imitation learning framework that synthesizes pseudo-replay trajectories, enabling a
robot policy to rehearse previously learned tasks without storing their original human demonstrations. During continual adaptation, REGEN recursively queries the WAM to synthesize pseudo-replay trajectories conditioned only on prior task instructions and current-task observations. Experiments in both simulation and real-world manipulation settings show that REGEN reduces catastrophic forgetting by up to $50\%$ relative to sequential fine-tuning, while approaching the performance of privileged experience replay methods that require access to real replay data. Finally, we analyze the factors limiting generated replay, identifying long-horizon visual degradation and action-observation inconsistency as the primary bottlenecks. Our results establish WAMs as a promising foundation for continual
robot learning without stored demonstrations.
Read the full abstract ▾
All authors Manish Kumar Govind, Dominick Reilly, Smit Patel, Hieu Le, Srijan Das
Computer Vision
Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar et al. (+4 more)
Recently, self-evolving large
multimodal models (LMMs) have received
attention for improving visual
reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical
language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on
language priors rather than the image during generation, manifesting as insufficient
attention to visual
tokens. As a result, current self-evolving LMMs struggle on
vision–
language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-
Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a
geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18
benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object
hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE
Read the full abstract ▾
All authors Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Salman Khan, Fahad Khan
Computer Vision
DnA: Denoising Attention for Visual Tasks
Ron Campos, Subhajit Maity, Xin Li et al. (+2 more)
The softmax activation in multihead
attention (MHA) is the de facto standard for
attention-based models in visual perception tasks. However, standard softmax can produce noisy
attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising
Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including
video understanding with
video transformers (1.8%) and
video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.
Read the full abstract ▾
All authors Ron Campos, Subhajit Maity, Xin Li, Srijan Das, Aritra Dutta
Computer Vision
Don't Settle at the Mode! Mitigating Diversity Collapse in Pretrained Flow Models via Feature Self-Guidance
Pradhaan S Bhat, Rishubh Parihar, Abhijnya Bhat et al. (+1 more)
State-of-the-art flow models generate stunning images from text or image
prompts. However, they suffer from diversity collapse when generating multiple samples under the same conditioning. Existing methods address this issue via either latent guidance, which has limited effectiveness, or sample selection, which relies on external reward models that incur significant inference-time overhead. In this work, we introduce an efficient, training-free self-guidance mechanism to mitigate diversity collapse without requiring additional reward models. Specifically, we disperse the internal features of the flow model during batch generation with feature self-guidance. Further, to keep the features close to the
manifold, we introduce a
manifold regularization step that projects these dispersed features back onto the data
manifold, ensuring diverse generation without sacrificing
alignment with the input conditions. Our method integrates seamlessly as a plug-and-play module into pretrained flow models, adding only a marginal inference cost. Experiments demonstrate significant improvements in diversity while preserving fidelity across several conditional flow models, including multi-step and few-step text-to-image, depth-to-image, and reference image generation.
Read the full abstract ▾
All authors Pradhaan S Bhat, Rishubh Parihar, Abhijnya Bhat, R. Venkatesh Babu
Computer Vision
PhysiFormer: Learning to Simulate Mechanics in World Space
Yiming Chen, Yushi Lan and Andrea Vedaldi
We present PhysiFormer, a
diffusion transformer for physically-plausible 3D object motion. Unlike
video world models that operate in view-dependent pixel space, PhysiFormer represents objects as 3D meshes expressed in world coordinates. Given the initial vertex positions and velocities, as well as object material type, rigid or elastic, the model samples future vertex trajectories. While related neural physics approaches build on ad-hoc latent spaces or explicitly enforce
rigidity and causality, PhysiFormer shows that excellent results can be obtained without any such inductive biases, by casting vertex trajectory prediction as a single denoising
diffusion process directly in world coordinates. The probabilistic formulation captures uncertainty in the learned dynamics, enabling diverse plausible futures from initial conditions, making this framework potentially useful for applications with unobserved uncertainty. The model features
attention factorised over time, space, and objects for efficiency, enabling permutation-invariant multi-object
reasoning without needing explicit object encoding. Trained on over 100k simulated trajectories, PhysiFormer generates rigid and elastic mechanics, and generalises to mixed-material settings, unseen real-world geometries, and larger object counts. It substantially outperforms autoregressive baselines in trajectory accuracy,
rigidity preservation, and momentum-based physical consistency. Our results position coordinate-space
diffusion as a promising step toward view-invariant, geometry-aware world modelling for robotics, graphics, and physical design. Visualisations, code, and models are available at https://yimingc9.github.io/physiformer.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Error-Conditioned Neural Solvers
Haina Jiang, Liam Wang, Peng-Chen Chen et al. (+4 more)
Neural surrogate models offer fast approximate mappings from PDE parameters to solutions, but they typically treat solving as a purely statistical task: once trained, they struggle to correct their own constraint violations and extrapolate beyond the training distribution. Recent hybrid methods promote physical correctness by targeting the PDE residual via gradient descent or Gauss–Newton steps, but inherit the compute cost and instability of the underlying classical optimizers. We show, theoretically and empirically, that numerically minimizing the PDE residual can be an unreliable proxy for reconstruction accuracy in ill-conditioned systems, explaining why these methods often do not make accurate predictions despite achieving low residuals. We propose error-conditioned Neural Solvers (ENS), built on a different principle: rather than an optimization target, the PDE residual field is passed as a direct input to the network at each iteration, enabling it to read the spatial structure of its own errors and learn an update policy to iteratively correct its predictions. Across four PDE families, ENS attains the highest prediction accuracy in the large majority of settings, with gains reaching $10\times$ on turbulent Kolmogorov flow, while avoiding the expensive compute cost of hybrid methods. ENS's learned correction policy generalizes under distribution shift, including zero-shot parameter changes and cross-equation transfer, where its relative advantage is largest in the ill-conditioned regimes where residual minimization is least reliable. Project website: https://neuralsolver.github.io/.
Read the full abstract ▾
All authors Haina Jiang, Liam Wang, Peng-Chen Chen, Min Seop Kwak, Seungryong Kim, Brian Bell, Jeong Joon Park
Computer Vision
SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers
Minghao Yin, Jiahao Lu, Wenbo Hu et al. (+3 more)
Video diffusion transformers address their
tokens by position on the pixel-time grid: an address in the tensor, not in the world. The address we would want, the world point a
token depicts, lies on a surface not yet generated, while its camera ray is fixed once the user specifies a trajectory. SCoPE therefore treats the ray as a second positional coordinate, and camera control becomes a property of the coordinate system, not an added module. The ray is added to the pretrained
attention's queries and keys, and the score gains a term that reads the two rays alone. Its canonical form, the reciprocal product of line geometry, measures how nearly two lines of sight meet. Normalize-Gate-Inject makes a single encoding trainable across metric and up-to-scale pose sources. The retrofit keeps RoPE bit-exact, starts from the unchanged pretrained DiT, and adds under 0.1/% new parameters. On Wan2.2 at 5B and 14B under matched data and budget, SCoPE improves every camera-controllability and fidelity metric, leads all closed-loop revisit metrics, and shows widening margins with model size. At 14B, rotation error falls 29/% and FVD 43/% below the strongest baseline.
Read the full abstract ▾
All authors Minghao Yin, Jiahao Lu, Wenbo Hu, Wang Zhao, Shan Ying, Kai Han
Computer Vision
SAM2Matting: Generalized Image and Video Matting
Ruiqi Shen, Guangquan Jie, Chang Liu et al. (+1 more)
Despite impressive advances in image matting,
video matting remains challenging due to the inherent gap between high-level tracking, which requires frame-wise understanding, and low-level matting, which focuses on extremely fine-grained details. Existing methods attempt this with expensive and narrowly-scoped
video matting
datasets, which may limit out-of-domain generalization and compromise tracking robustness. We rethink the paradigm with SAM2Matting, a tracker-to-matting framework that advances VOS trackers to high-fidelity
video matting. Specifically, it decouples the task by enhancing a foundational tracker (e.g., SAM2, SAM3) with a region-proposal bridge and dedicated matting heads, enabling the uncompromised tracker to handle temporal consistency while the matting components resolve fine-grained details. Notably, despite being trained only on images, SAM2Matting establishes new state-of-the-art performance on
video matting, supports diverse
prompt types, maintains strong temporal consistency, and demonstrates robust generalization across both human-centric and in-the-wild scenarios.
Read the full abstract ▾
All authors Ruiqi Shen, Guangquan Jie, Chang Liu, Henghui Ding
Computer Vision
RoPEMover: Depth-Aware Object Relocation via Positional Embeddings
Ipek Oztas, Duygu Ceylan, Aybars Bugra Aksoy et al. (+1 more)
Moving an object in a single image requires geometry-consistent spatial rearrangement, including handling occlusions, revealing previously unseen regions, and maintaining coherent shadows and reflections. Existing approaches are not well suited to this setting and often fail to preserve such scene-level consistency. We address this problem by introducing a geometry-aware object motion method that operates directly on the positional representations of
diffusion transformers. Our key insight is that rotary positional
embeddings (RoPE) define a structured spatial field that can be explicitly manipulated to induce controlled motion. We extend 2D RoPE into a depth-aware formulation that encodes 3D spatial structure, enabling consistent object displacement and scene-aware updates. Our model is trained using synthetic data combined with a small set of real images via parameter-efficient fine-tuning. Despite minimal real supervision, it preserves object identity under large spatial displacements, generates plausible content in newly revealed regions, and consistently updates scene-dependent effects such as shadows and illumination. Experimental results on standard object motion
benchmarks demonstrate state-of-the-art performance across all evaluation metrics.
Read the full abstract ▾
All authors Ipek Oztas, Duygu Ceylan, Aybars Bugra Aksoy, Aysegul Dundar
Computer Vision · Cross-listed from cs.CL
Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
Tianyi Men, Zhuoran Jin, Pengfei Cao et al. (+3 more)
Multimodal web
agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open source MLLMs are cost efficient and
privacy preserving compared with commercial large models, they suffer from weak planning and limited cross website generalization. To address these limitations, we introduce the planning experience exploration and utilization (PEEU) method, which autonomously explores environments to discover experiences and utilizes hindsight experience to synthesize strictly aligned, high level training data. To quantitatively analyze the generalization behaviors driving this performance, we propose the task decomposition hierarchical analysis framework (TDHAF) to systematically study compositional generalization across three task granularities: low, middle and high levels. Our analysis reveals that mastering low level atomic skills does not guarantee high level planning competence, while high level task training yields stronger OOD generalization. Experiments on real world
benchmarks demonstrate PEEU's superior effectiveness: our 7B model achieves 30.6% accuracy, outperforming the much larger Qwen2.5-VL-32B model. These demonstrate constructing hindsight high level tasks and leveraging experiences is crucial for OOD planning abilities of small MLLMs.
Read the full abstract ▾
All authors Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Computer Vision · Cross-listed from cs.LG
Hallucination in World Models is Predictable and Preventable
Nicklas Hansen and Xiaolong Wang
Modern generative world models render increasingly realistic action-controllable futures, yet they frequently hallucinate: rollouts remain visually fluent while drifting from the ground-truth dynamics. We hypothesize that
hallucination concentrates in low-coverage regions of the state-action space, where lightweight data-centric signals can both detect it and guide mitigation. To test this, we introduce MMBench2, a 427-hour, 210-task
dataset for visual world modeling with ground-truth actions, rewards, and live simulators, and train a 350M-parameter world model on it. We identify three distinct
hallucination modes: perceptual, action-marginalized, and scene-diverging – each anchored to a different stage of the pipeline, and develop three signals that accurately predict where the model will fail. To close coverage gaps at training time, we develop a coverage-aware sampling technique; to close them online, our
hallucination predictors serve as curiosity rewards for targeted data collection, yielding a data-efficient finetuning recipe that adapts the pretrained world model to entirely unseen environments with as few as 50 real environment trajectories. Overall, our findings reveal that
hallucination in world models is inherently a data coverage issue, and that the same signals used to detect it can also be used for mitigation. An interactive web version of our paper is available at https://www.nicklashansen.com/mmbench2
Read the full abstract ▾
Computer Vision
Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model
Zizhao Yuan, Zhengtu Liang, Taowen Wang et al. (+7 more)
Recent advances in action-conditioned world models show promising progress in modeling complex interactions and
forecasting future states under diverse action sequences. While these models are often driven by stronger visual representations and model capacity, action conditioning itself remains underexplored. Most existing approaches compress the entire action sequence into a single representation, which works well for low-DoF control but becomes less reliable in high-DoF scenarios. We observe that high-DoF dexterous actions are inherently heterogeneous, spanning multiple orders of magnitude, where large-scale motions coexist with subtle but important signals. When uniformly aggregated, optimization exhibits an imbalance across action components, which hinders the modeling of fine-grained effects and affects action fidelity. We therefore propose DexAC-WM, which treats action conditioning as a structured process rather than global compression. DexAC preserves dimension-level semantics via action tokenization and aligns action signals with visual dynamics through local refinement and global modulation. To address the limited high-level semantic grounding in existing world models, we further introduce a semantic branch that provides rich object-scene priors, which enables world model to capture dynamic visual details while supporting high-DoF action-conditioned
video prediction. Experiments on EgoDex and EgoVerse show that combining the semantic branch with DexAC significantly improves FID, FVD, and PCK, demonstrating gains in visual-temporal realism and action-following consistency. We further verify that DexAC extends to other backbones, showing the scalability of our structured action-conditioning design. These results suggest that scaling world models to high-DoF control requires both structured action modeling and semantic grounding.
Read the full abstract ▾
All authors Zizhao Yuan, Zhengtu Liang, Taowen Wang, Qiwei Liang, Yichi Wang, Yunheng Wang, Yuetong Fang, Lusong Li, Zecui Zeng, Renjing Xu