Computer Vision
Video-Arena: A Scalable Agentic Video Benchmark for Spatiotemporal Capability
Kuan Zhang, Yukun Chen, Zhihao Yang et al. (+7 more)
Understanding and tracking a dynamic world are fundamental to visual intelligence. Existing
video benchmarks provide limited insight into whether models can maintain a coherent understanding of evolving world state. Building such a
benchmark from real-world
videos is challenging because the underlying world state is partially observable, making annotations hard to obtain. To close this gap, we introduce
Video-Arena, a
benchmark that turns competitive games into scalable, verifiable tests of how well models understand evolving world states from
videos. Specifically, inspired by how humans understand dynamic events, we evaluate spatiotemporal world state understanding and tracking through three task categories: perceiving critical events, interpreting state updates, and tracking the resulting states. Our pipeline can generate
videos and aligned reference labels from game records at scale. To keep evaluation affordable and reproducible, we curate a golden set of 180 tasks across 9 games and run evaluations under two protocols: native visual understanding and tool-assisted agentic analysis. The best-performing systems achieve F1 scores of and under the two protocols, respectively. The result highlights that native models struggle to perceive events reliably, while tool-assisted systems improve event recovery but still struggle to interpret state updates and track the resulting states consistently over time. Our project page is in https://videoarena-bench.github.io
Read the full abstract ▾
All authors Kuan Zhang, Yukun Chen, Zhihao Yang, Yue Su, Xiangnan Wu, Zirong Chen, Run Luo, Tao Tan, Yinhe Zheng, Yiming Li
Computer Vision · Cross-listed from cs.SE
OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools
Chenyang Yan, Yingying Zhang, Kanzhi Cheng et al. (+21 more)
We introduce OpenMobile-2, a near-frontier mobile
agent with fully open training environments and recipes. We make three key advances: (1) Diverse environments with simulated commercial apps: We build MobileGym++, featuring 35 realistic, functionally rich commercial-style apps with cross-app workflows, while preserving full controllability for reset and verification. Together with newly configured apps in Android emulators, this yields a diverse playground spanning over 110 apps. (2) Open training data at scale: Building on this foundation, we curate nearly 12K mobile interaction trajectories for supervised fine-tuning and the largest open collection of verifiable RL training data for mobile
agents, comprising over 2K executable tasks with automatic rewards. (3) Hybrid GUI and app-native tool use: We explore an experimental mobile-use setting where apps expose selected functionalities as app-native tools alongside their GUIs, allowing
agents to interleave GUI actions and tool calls within a task. We implement this setting in MobileGym++ with over 300 carefully scoped tools across 50+ apps. Additionally, we introduce MobileGym++ Bench for complex, long-horizon commercial mobile scenarios, supporting both GUI-only and hybrid GUI-tool evaluation on a shared task suite. OpenMobile-2 performs competitively across established
benchmarks, including AndroidWorld (79.9) and MobileWorld (50.4), while showing promising transfer to real-device mobile use, nearly doubling SPA-Bench performance from 31.9 to 59.6. All resources are available at https://os-copilot.github.io/OpenMobile2-Home/.
Read the full abstract ▾
All authors Chenyang Yan, Yingying Zhang, Kanzhi Cheng, Qiushi Sun, Zhengyuan Pan, Zheng Ma, Hang Yan, Yian Wang, Nuo Chen, Jialin Cao, Xingdong Gong, Wenpo Song, Han Chen, Zichen Ding, Fangzhi Xu, Shujian Huang, Xinyu Dai, Yichen Liu, Tiankuo Yao, Bo Wang, Ben Kao, Jianbing Zhang, Lewei Lu, Dahua Lin
Computer Vision · Cross-listed from cs.RO
Axiom-0: Hierarchical Self-Refinement for Vision-Language-Action Policies
Renbin Shu, Jing Yang, Xiangyuan You et al. (+7 more)
Reliable manipulation requires policies that generalize across tasks, layouts, and object interactions.
Vision-
language-action (VLA) models combine pretrained
multimodal understanding with learned control, but their performance remains limited on unfamiliar task compositions. We present Axiom-0, a VLA policy that addresses both the quality of its representations and their use in action generation through hierarchical self-refinement. Intra-VLM Representation Refinement (IVRR) strengthens selected intermediate features through consistency supervision with learned feature projections during training. VLM-Action Interface Refinement (VAIR) recursively updates the resulting conditioning before action generation. Joint optimization of the supervised VLM layers, VAIR, and the action generator connects the two components. On RoboCasa365, Axiom-0 achieves 62.32% overall success in our evaluation, exceeding the highest published leaderboard score (Paimon-0, 58.1%) by 4.22 percentage points and the Xiaomi-Robotics-1 reference by 4.92 points. Under the evaluated profiling setup, mean action-chunk generation latency increases by 1.7% over Xiaomi-Robotics-1. These results demonstrate improved task completion with modest inference overhead.
Read the full abstract ▾
All authors Renbin Shu, Jing Yang, Xiangyuan You, Guolin Qiu, Yufei Xia, Yiming Zhu, Nanjun Dai, Fei Zhang, Feng Wu, Keke Tang
Computer Vision
TrajCFG: Token-Wise Classifier-Free Guidance via Denoising Trajectory Feedback
Jingyu Liu, Hao Jiang, Yikang Ding et al. (+6 more)
Classifier-free guidance (CFG) typically applies a shared scale to all latent
tokens, although their velocity predictions evolve differently during sampling. We introduce Trajectory-Adaptive Classifier-Free Guidance (TrajCFG), a training-free method for flow-matching models that adapts
token-wise guidance using historical changes in guided velocity direction. TrajCFG maps this feedback to bounded scales, assigning stronger guidance to larger directional changes and weaker guidance to more consistent directions. It combines adjacent-step and interval-normalized longer-span feedback, reusing standard CFG predictions without additional model evaluations. Controlled diagnostics and ablations examine the roles of directional feedback,
token-wise allocation, and history-span fusion. Experiments on text-to-
video, image-to-
video, text-to-audio-
video, text-to-image, and image-to-3D generation show improvements over base samplers across multiple backbones. For text-to-
video generation, TrajCFG improves VBench Total scores by 0.30-0.75 points over the strongest evaluated baseline on each of three backbones. These results support historical directional feedback as a broadly applicable signal for fine-grained guidance allocation. The project page is available at https://jyliu-98.github.io/trajcfg-project-page/.
Read the full abstract ▾
All authors Jingyu Liu, Hao Jiang, Yikang Ding, Huafeng Shi, Xu Yang, Xiaoqiang Liu, Guoxin Zhang, Pengfei Wan, Kun Gai
Computer Vision · Cross-listed from eess.IV
Beyond Gate Weights: Decision-Level Interpretation of Radiomics-Image Feature Fusion for RCC Classification
Yuan Liang, Sourav Bhattacharjee and Abraham Campbell
Gate coefficients describe how a fusion layer weights intermediate features, but they do not necessarily reflect the magnitude of branch contributions at the classifier output. We investigate this distinction in a fixed 3D MedVAE–radiomics model for clear
cell versus non-clear
cell renal
cell carcinoma classification from contrast-enhanced CT. Because the model uses a linear classification head, the binary decision logit can be decomposed exactly into a gated image term, a direct radiomics term, and a bias. We define effective radiomics dominance as the direct radiomics term's share of the summed absolute non-bias branch contributions and compare it with structural radiomics weighting. Internally, mean structural radiomics weighting was 0.823, whereas mean effective radiomics dominance was 0.148, with a modest case-wise association (h̊o} = 0.290). The gated image term retained substantially stronger discriminative ranking information than the direct radiomics term. The same structural-to-effective discrepancy was observed in exploratory external evaluation. Branch-targeted permutation, integrated-gradient, and Shapley-style analyses further identified radiomics descriptors influencing the direct radiomics term. Because radiomics also generates the gate, the gated image term remains radiomics-dependent, so additive branch contribution should not be interpreted as total input influence. These findings show that structural gate weighting and decision-level branch contribution represent distinct aspects of gated feature fusion and should be interpreted separately.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification
Yuan Liang, Morgan May, Abraham Campbell et al. (+3 more)
Reliable probability estimates are important for the clinical interpretation of machine-learning classifiers, yet radiomics–image fusion models for renal
cell carcinoma (RCC) are commonly evaluated primarily by discrimination. Using 396 patients from the KiTS23
dataset, we evaluated whether adding radiomics to four image-model families improved probabilistic performance in addition to discrimination. Seventeen individual models and four probability-average ensembles were assessed using discrimination, probability calibration, and proper scoring measures. Across 12 matched image-only versus fusion comparisons, AUC and AP improved in all 12, while Brier score and negative log likelihood (NLL) improved in 11 of 12 and expected calibration error (ECE) in eight of 12. The 3D MedVAE gated-fusion model achieved the strongest overall individual-model profile. Probability averaging across concatenation, cross-
attention, and gated fusion produced favourable point estimates across all five primary metrics relative to image-only models in all four families, although paired-bootstrap support was stronger for Brier score and NLL than for ECE. These findings show that radiomics–image fusion can improve probabilistic prediction quality beyond discrimination, while calibration behaviour varies across image architectures and fusion strategies. We show that our proposed probability-average ensemble models improve discrimination and calibration simultaneously across all architectures in the study.
Read the full abstract ▾
All authors Yuan Liang, Morgan May, Abraham Campbell, Sourav Bhattacharjee, Simon Caton, Pierpaolo Dondio
Computer Vision · Cross-listed from eess.IV
C$^2$LUE: Clinically Aligned Patient-Level Ultrasound Diagnosis via Evidence-Aware Multi-Instance Learning
Qin-Yi Zhang, Linying Cao, De-Xing Huang et al. (+7 more)
Patient-level ultrasound diagnosis demands not only high predictive accuracy but also clinically aligned mitigation of asymmetric diagnostic risks. While Multi-Instance Learning (MIL) effectively models routine multi-view exams, standard aggregators often dilute heterogeneous diagnostic cues and optimize solely for global accuracy. This frequently yields undesirable sensitivity-specificity trade-offs. To bridge this gap, we propose C$^2$LUE, a novel MIL framework coupling evidence aware aggregation with risk-aligned regularization. To preserve evidence heterogeneity and prevent unimodal
attention collapse, C$^2$LUE employs a multi-query cross-
attention mechanism built upon a LoRA-adapted DINOv3 backbone. Multiple latent queries explicitly extract and disentangle diverse morphological patterns from the ultrasound sequence. Furthermore, to improve clinical risk
alignment, we introduce a dynamic trade-off regularization objective that penalizes sensitivity-specificity imbalance during training. Rather than leaving the clinical trade-off entirely to post-hoc threshold adjustment, this objective encourages the learned score distribution toward a more balanced operating region. Extensive evaluations on patient-level, biopsy-proven thyroid and breast ultrasound
datasets demonstrate that C$^2$LUE consistently outperforms prior alternatives. By uniting strong discrimination with a balanced error profile, C$^2$LUE provides a promising solution for real-world ultrasound screening and diagnosis.
Read the full abstract ▾
All authors Qin-Yi Zhang, Linying Cao, De-Xing Huang, Haichuan Lin, Yi Yang, Manzhi Wang, Xiao-Hu Zhou, Chen Chen, Shuangyi Wang, Zeng-Guang Hou
Computer Vision · Cross-listed from eess.IV
Frozen Site-Probe Accuracy Is Not a Standalone Harmonization Score: An Audit of Evaluation Practice for Multi-Site MRI
Muhammad Muqsit Islam, Gloria García Cuenco, José M. Martínez et al. (+1 more)
Image-level MRI harmonization is often evaluated with a frozen site probe, a site classifier trained on raw scans and applied unchanged to harmonized ones: a drop in its accuracy is read as removal of site effects. We audit this practice on ABIDE I (1,112 T1-weighted scans, 17 sites) with ten output sets from seven harmonization methods and controls for chance, head shape, intensity, preprocessing and target
alignment. We find that this verdict depends on how the probe is trained, even among probes that classify raw scans well: across random seeds, its balanced accuracy on the same CycleGAN outputs ranges from 0.13 to 0.59 (chance: 0.06). Low frozen-probe accuracy can also arise without harmonization: a Gaussian blur, which increases the intensity distance to the target site, lowers this accuracy to chance. Moreover, where the frozen probe fails, probes trained on the harmonized scans still recover the site, even from intensity histograms alone: on HACA3 outputs, the frozen probe scores 0.25, a probe trained on those outputs 0.75, and one using only their intensity histograms 0.71. Low frozen-probe accuracy therefore does not show that site information has been removed. We recommend reporting several probes trained to convergence, probes trained on harmonized scans, and target
alignment.
Read the full abstract ▾
All authors Muhammad Muqsit Islam, Gloria García Cuenco, José M. Martínez, Pierrick Coupé
Computer Vision · Cross-listed from eess.IV
Exploring the Latent Structure of AMD Severity Using Retinal Foundation Models
Niccolò Fineschi, Jovana Panic, Atsushi Watanabe et al. (+7 more)
Retinal foundation models have demonstrated strong performance across a wide range of ophthalmic tasks, yet relatively little is known about how disease-related information is represented within their learned
embeddings and whether this distribution influences downstream prediction performance. In this work, we use Age-related Macular Degeneration (AMD) as a case study to investigate how disease severity is represented within the
embedding space of RET-CLIP, a retinal
vision-
language foundation model, using bilateral fundus images from the AREDS
dataset, benchmarked against non-retinal encoders. We first characterize the distribution of AMD severity within the learned representation space and then evaluate how this structure influences classification performance across three clinically relevant task formulations of increasing granularity. Finally, we analyze the most influential latent dimensions combining feature attribution, statistical phenotype association, and qualitative image analyses. Our results show that RET-CLIP represents AMD severity as a continuous spectrum rather than a set of discrete categories, with healthy and advanced disease stages exhibiting clearer separation than intermediate ones. This organization was reflected in downstream performance, which decreased from a balanced accuracy of 0.844 and 0.814 for binary AMD and Late-AMD detection respectively to 0.446 for six-class grading. Furthermore, the most influential latent dimensions were statistically associated to distinct retinal phenotypes, including drusen accumulation, pigmentary alterations, and advanced retinal lesions. These findings suggest that the structure of retinal foundation-model representations influences the difficulty of clinically relevant prediction tasks and demonstrate the potential of representation-level analysis for understanding disease organization and improving interpretability.
Read the full abstract ▾
All authors Niccolò Fineschi, Jovana Panic, Atsushi Watanabe, Hideaki Hayashi, Kazuya Nishimura, Yuki Maeno, Samanta Rosati, Gabriella Balestra, Kohji Nishida, Hajime Nagahara
Computer Vision
Towards Physically Grounded Articulated 3D Assets: A Multi-Agent Pipeline and Benchmark
Xudong Cai, Ruijie Zhu, Haoliang Shang et al. (+5 more)
In this paper, we study physically grounded articulated 3D asset generation—moving the goal from reproducing how objects look to capturing how they move and function. Inspired by the literature on procedural and agentic generation, we present Artisan, a system that extends the latest method Articraft by introducing a multi-
agent pipeline with physical simulation feedback, enabling comprehensive attributes beyond geometry, appearance, and physically plausible articulation, such as collision-free motion, stability under
gravity, and accurate mass and material properties. Furthermore, we construct a unified
benchmark spanning four key aspects: spatial fidelity, physical plausibility, downstream task performance, and overall quality. The
benchmark encompasses 100 reference assets from ArtVIP and LightWheel, 50 stability
prompts, and 50 ABO objects with reference masses, with protocols evaluating various metrics. Through extensive evaluation of more than 10 representative models, Artisan achieves competitive spatial fidelity while attaining the lowest collision rates and highest stability rates among the evaluated methods. Together, our system and
benchmark establish insights for advancing 3D generation towards physically plausible, functionally usable assets.
Read the full abstract ▾
All authors Xudong Cai, Ruijie Zhu, Haoliang Shang, Ruining Li, Marc Pollefeys, Shangzhe Wu, Andrea Vedaldi, Chuanxia Zheng
Computer Vision · Cross-listed from cs.RO
Splat2Act: From Captured Scenes to Personalized VLA Manipulation
Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim et al. (+3 more)
A user who brings a generalist manipulation policy into their home or office wants it to pick up their objects from their shelves, counters, and tables. Yet even a strong generalist like $\pi_{0.5}$ achieves 0% pickup success in our captured deployment scenes without adaptation, and collecting real demonstrations in every user's environment is prohibitively expensive. We present Splat2Act, an automated real-to-sim-to-real pipeline that personalizes a pretrained
vision-
language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work that reconstructs individual tabletops, Splat2Act operates on building-scale indoor environments: it builds a photorealistic digital twin with 3D Gaussian Splatting and collision geometry, automatically discovers tabletop and non-tabletop receptacles, constructs feasible pickup tasks, and generates successful demonstrations with a privileged simulation oracle, with only scene and object capture required from a human. Applied to 101 scanned campus environments with a median navigable area of 104.87 m$^2$ and 7 receptacles, the pipeline yields 852 receptacles and over 68,000 pickup demonstrations. We fine-tune $\pi_{0.5}$ in two stages: Stage 1 adapts to hundreds of public object assets, drawn either from the deployment scene alone or from all 101 scenes; Stage 2 personalizes to the user's own objects on the deployment scene's receptacles. Our strongest personalized policy reaches 62.5% aggregate pickup success in the target scene. Deployed on a real Stretch 3, the same policy achieves 63.9% success on a 12-task subset, closely tracking its 66.7% simulated success on those tasks. These results suggest that automated real-to-sim-to-real training can personalize VLA manipulation policies using only simulation-oracle data.
Read the full abstract ▾
All authors Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim, Gunjan Chhablani, Muhammad Zubair Irshad, Zsolt Kira
Computer Vision
InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators
Xuanlang Dai, Jiazi Bu, Yujie Zhou et al. (+5 more)
Diffusion/flow models have emerged as the de facto paradigms for image generation, giving rise to a diverse family of modern generators with increasingly heterogeneous architectures and representations. Despite their remarkable generation capabilities, the cross-model
alignment of latent representations and the consistency of their underlying denoising dynamics remain underexplored. In this work, we uncover an intriguing cross-model compatibility: independently trained generators exhibit similar latent spatial organization, while directly handing off intermediate states between unaligned models can still preserve coarse visual content. Motivated by these observations, we propose InstantFusion, a lightweight shared latent interface that connects the denoising trajectories of heterogeneous image generators. Specifically, to enable flexible cross-model state translation, model-specific Latent AutoEncoders (LAEs) are introduced to project intermediate states from different generators into a shared latent space and decode them into the latent space of a target generator. This shared space is constrained by two complementary objectives: aligning intermediate states derived from the same image at matched noise levels and synchronizing their corresponding local denoising velocities. Once trained, InstantFusion allows denoising to seamlessly transition across frozen generators without requiring a dedicated mapping for every model pair. The versatility of InstantFusion is demonstrated across acceleration, multi-preference composition, cross-architecture on-policy
distillation, and cross-model composition, achieving competitive performance across these diverse settings.
Read the full abstract ▾
All authors Xuanlang Dai, Jiazi Bu, Yujie Zhou, Yuhong Liu, Beichen Zhang, Penghui Yang, Ziyu Liu, Yuhang Zang
Computer Vision
Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback
Yucheng Yang, Xiaoxiao Sun, Shusheng Yang et al. (+2 more)
Multimodal diffusion transformers (MMDiTs) allow the evolving image to update internal text representations through joint
attention. These representations then guide subsequent image updates, raising a question: does this feedback help correct deviations from the
prompt, or preserve them? We find that image-to-text (I2T) feedback can preserve the objects and attributes already represented in the evolving image, including errors. Transplanting I2T messages can rescue failures or disrupt successes. At matched intervention strengths, suppressing the influence of I2T feedback improves generation, whereas opposite controls degrade it. These findings motivate Semantic Feedback Guidance (SFG), which suppresses the influence of I2T feedback by guiding away from an auxiliary branch that strengthens this feedback and weakens text-to-image conditioning. The in-place variant, iSFG, directly suppresses I2T feedback and strengthens text-to-image conditioning, requiring no extra model evaluations. Temporal interventions motivate running the SFG branch during the first 20% of image sampling steps, requiring about $1.2\times$ the base NFE without classifier-free guidance (CFG). Experiments across three image-generation backbones show gains in
alignment and preference-model scores. On FLUX, adding SFG to CFG raises GenEval from $0.6661$ to $0.7479$; on SD3.5M, iSFG adds $+1.33$ HPSv2.1 over CFG at zero extra NFE. The control also transfers to text-to-
video and image-to-
video generation, improving VBench scores. Code is available at https://github.com/yyc1215lll-creator/SFG.
Read the full abstract ▾
All authors Yucheng Yang, Xiaoxiao Sun, Shusheng Yang, Jiakang Chen, Serena Yeung-Levy
Computer Vision
Hide and Seek: HIDE Concepts to SEEK explanations for black box models
Bhanu Tokas, Rahul Nair, Rockwell Jackson et al. (+1 more)
Interpreting machine learning model predictions is a central challenge, particularly for black-box
vision models. Concept-based attribution methods interpret models using human-understandable concepts. Existing concept-based methods can only interpret classification and regression models. They require access to model weights or large concept-annotated
datasets, and most methods cannot provide image-level attribution. We propose Hide and Seek (HnS), a new method for concept-level attribution that does not require access to model weights or labeled concept
datasets. HnS can interpret a wide range of
vision models. Given a concept and image set, HnS retrieves concept-aligned images using a
vision-
language model, localizes the concept within each image, masks the localized region, and measures attribution from the change in the black-box model's output between the original and masked images. This enables faithful attribution by directly measuring the model's behavior and supporting local attributions. We validated HnS on CelebA and COCO-Stuff. Compared to existing methods, HnS showed better agreement with ground-truth attributions. We demonstrated the applicability of HnS on diverse tasks, including image captioning and satellite image segmentation, where no existing concept attribution method can be used. We provided our anonymous code in the supplementary material.
Read the full abstract ▾
All authors Bhanu Tokas, Rahul Nair, Rockwell Jackson, Hannah Kerner
Computer Vision · Cross-listed from cs.RO
Chronos Policy: Learning Robot Policies on Physical Time
Changhao Wang, Jaden Clark and Shuran Song
Robot policies often convert sensing and control into fixed-rate observation-action sequences, leaving physical time implicit in sequence position. This assumption breaks when observation rates or delays vary, or when inference latency changes. We introduce Chronos Policy, which makes physical time explicit by conditioning on observation capture times and desired action execution times. We study two ways to incorporate time into a
diffusion transformer: adding timestamp
embeddings to
token features, or encoding physical time in positional encoding through time-aware rotary positional encodings (time-aware RoPE). We find that time-aware RoPE is more robust, allowing
attention to depend directly on physical time while keeping
token content unchanged. During training, we temporally randomize which observations are sampled and when the action chunk begins, exposing the policy to a wider range of sensing and execution schedules that may occur during deployments. This also enables learning from asynchronous, multi-rate data without forcing all measurements onto a fixed temporal grid. Across simulation and real-world experiments, Chronos Policy improves performance on dynamic tasks, where observation-to-action timing is critical, and on multisensory tasks, where the relative timing between observations matters. On standard quasi-static tasks, it preserves nominal performance while improving robustness to changes in sensing and inference timing.
Read the full abstract ▾
Computer Vision
Can machines reason non-verbally?
Shaheer U. Saeed, Zion Tse, Gregory Slabaugh et al. (+2 more)
Modern machines can solve equations, follow logic and reason through
language, yet they still struggle with problems that require visual non-verbal
reasoning. This is evident even in advanced systems such as DeepSeek, ChatGPT and Gemini, which remain limited on visual tasks such as completing abstract patterns or delineating ambiguous regions-of-interest. Unlike explicit intermediate steps in verbal
reasoning which can be verified through logic or automated tests, visual
reasoning involves implicit intermediate steps with no clear methods for verification. Non-verbal
reasoning is therefore challenging to model and, as a result, remains a fundamental limitation of current machine intelligence. Here we introduce GenVerOpt, a general framework that unifies generation, verification and optimisation into a
reasoning pipeline for machines. By generating candidate solutions, verifying their correctness, and iteratively refining them, GenVerOpt enables per-sample optimisation rather than relying solely on
dataset-level trends captured during model training. This process mirrors human problem-solving where multiple possibilities are imagined, evaluated, and refined until a correct solution emerges. Across diverse domains, GenVerOpt demonstrates capabilities beyond existing systems: it surpasses humans on abstract IQ puzzles, matches clinicians in prostate cancer segmentation, exceeds experts in liver vessel reconstruction and outperforms state-of-the-art approaches in tumour boundary
alignment. These results show that machine
reasoning can extend beyond
language, establishing visual non-verbal
reasoning as a foundation for more flexible and generalisable intelligence.
Read the full abstract ▾
All authors Shaheer U. Saeed, Zion Tse, Gregory Slabaugh, Daniel C. Alexander, Yipeng Hu
Computer Vision
Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning
Junkai Liu and Le Zhang
Self-supervised learning on 3D medical images commonly relies on random sub-volume sampling during pretraining. However, random cropping implicitly treats all spatial regions as equally worth observing, despite anatomical priors and spatial redundancy that make regions inherently unequal in their value for representation learning. To this end, we first show through controlled crop-quality manipulations that better sub-volumes lead to better downstream transfer, revealing the observation policy as an overlooked bottleneck in 3D medical SSL. Motivated by this finding, we revisit sub-volume sampling as a where-to-look problem. We propose \method, a plug-and-play framework that replaces random cropping with an offline coarse-to-fine probe bank guided by anatomy-awareness, informativeness, and diversity. We instantiate \method as a lightweight \lite variant using
geometric and image-statistical cues, and a diagnostic \feat variant using frozen visual features, to examine whether lightweight volume-intrinsic priors suffice or richer semantic cues reliably provide additional value. Extensive experiments across multiple 3D medical SSL backbones and downstream tasks demonstrate that \method-\lite consistently improves transfer performance over random sampling. Mechanistic analyses further attribute these gains to more anatomically plausible and informative observations, reduced spatial redundancy, more efficient learning dynamics, and more structured representations.
Read the full abstract ▾
Computer Vision
Beyond the Snowflakes: Disentangling Weather Effects from Confounding Meta-Factors in LiDAR Object Detection
Yiming Yang and Jeremy P. Bos
Light Detection and Ranging (
LiDAR) is critical for autonomous perception, yet its performance is known to degrade in adverse weather. While existing literature models snow as atmospheric noise that degrades performance, our granular evaluation on the WADS-3D and CADC
datasets reveals a counter-intuitive phenomenon: detection accuracy consistently improves as snow intensity increases. Through ablation study and meta-information analysis, we reconcile this by confirming that while falling snow acts as noise, its negative impact is outweighed by a dominant, concurrent factor: significantly increased object point count. We hypothesize that this object point count boost acts as a
geometric enhancement, driven by snow accumulation increasing vehicle surface area and a shift from specular to diffuse reflectance. Our findings demonstrate that evaluating perception systems using broad weather-severity bins without accounting for correlated object-level data can lead to misleading conclusions about detector robustness. We conclude that accurate evaluation requires decoupling atmospheric noise from surface accumulation effects and outline future research to validate these physical models. The WADS-3D
dataset is available at: https://digitalcommons.mtu.edu/wads-3d/.
Read the full abstract ▾
Computer Vision
Tracking the Invisible: Self-Supervised Wheel Track Detection with Thermal-RGB Fusion
Yiming Yang and Jeremy P. Bos
Accurate detection of drivable area is essential for autonomous driving and advanced driver assistance systems. While state-of-the-art methods focus on general traversability estimation, they prove inadequate for snow-covered roads where following existing wheel tracks offers crucial advantages: improved traction, predictable vehicle dynamics, and verified path stability. To address this, we propose a low-cost, self-supervised approach detecting safety-critical wheel tracks using synchronized RGB-I (RGB + Long-Wave Infrared) imagery. Our method automatically generates training labels by projecting future wheel trajectories from recorded GNSS data. A neural network is designed to identify wheel tracks with the 4-channel RGB-I images from challenging, white-out road surfaces. Experimental results demonstrate the effectiveness of our approach in visually monotonous and featureless adverse environment, offering a robust and cost-effective alternative for winter autonomy. The newly introduced ThermalTrack
dataset is available at: https://digitalcommons.mtu.edu/thermaltrack/.
Read the full abstract ▾
Computer Vision
Make It Flow: Animating Water in 3D Gaussian Splatting Scenes
Kjell Keune, Florian Hahlbohm, Jannis Möller et al. (+3 more)
Explicit 3D scene representations such as 3D Gaussian Splatting (3DGS) have recently enabled high-quality, real-time novel view synthesis. However, these methods assume static scenes, limiting realism in the presence of dynamic phenomena ubiquitous in real-world environments. While coherent motion (e.g., humans or vehicles) can be captured and reconstructed, high-frequency effects such as flowing water remain intractable under casual capture conditions, where dense multi-view, high-speed acquisition is not feasible. We present a method to animate water in static 3DGS scenes, enabling visually plausible motion without requiring dynamic capture or physical simulation. Our approach estimates 2D optical flow from reference images or model renderings and lifts it into a consistent 3D Eulerian motion field. We propose a cinemagraph-inspired
algorithm that applies this motion field to deform Gaussian primitives, producing temporally seamless looping animations. The resulting representation retains real-time rendering performance, significantly enhances perceptual realism, and preserves free-viewpoint navigation within the scene. We evaluate our method on a newly captured
dataset of in-the-wild waterfall scenes and demonstrate clear improvements over prior work in dynamic Gaussian splatting and 3D cinemagraph generation in visual quality and computational efficiency.
Read the full abstract ▾
All authors Kjell Keune, Florian Hahlbohm, Jannis Möller, Susana Castillo, Martin Eisemann, Marcus Magnor
Computer Vision
LapMerge: Differentiable Exact-Budget Token Merging in Linear Time
Magdalena Trędowicz, Aleksandra Nowak, Tomasz Steifer et al. (+3 more)
Modern
vision and
language models process ever longer
token sequences, and reducing them is the main lever on downstream cost.
Token compression, however, faces a fundamental trade-off: selecting
tokens discards information, while existing merging methods preserve it only through pairwise
token-to-
token matching or $MN$
token-to-output interactions. We introduce LapMerge, a differentiable merging operator that resolves this trade-off for ordered sequences: a learned mass distribution partitions the sequence into exactly $N$ equal-mass segments under a Laplace-kernel density, and prefix/suffix scans aggregate each segment, so both the forward and the backward pass run in $O(M+N)$ time. This combination (an exact output budget, end-to-end differentiability, and linear cost) is precisely what previous reducers trade off against one another. The budget can be changed without changing operator parameters, and the learned mass doubles as a
token-level map of where the budget is allocated. Across
vision and
vision-
language tasks, LapMerge matches or improves on strong
token reducers while being up to two orders of magnitude faster at large sequence lengths; it is particularly effective when
token order reflects spatial or temporal locality, and works both as a drop-in reduction module and as a trainable architectural component.
Read the full abstract ▾
All authors Magdalena Trędowicz, Aleksandra Nowak, Tomasz Steifer, Łukasz Struski, Szymon Płotka, Jacek Tabor
Computer Vision
Revisiting the Task of Building Damage Assessment by Satellite Imaging
Anthony Cioppa, Virgile Blua, Sébastien Piérard et al. (+2 more)
Natural disasters such as earthquakes, tropical storms, and floods cause extensive damage to buildings, and rapid, reliable damage maps are essential to guide emergency response. Automating this task, referred to as building damage assessment (BDA), from satellite imagery, has become an active research topic at the intersection of remote sensing, computer
vision, and machine learning, particularly since supervised deep learning methods were shown to outperform earlier handcrafted approaches. Yet, despite a decade of intense activity and the release of several large-scale
datasets, the field lacks a shared foundation. This chapter revisits the task of BDA from first principles, organized around three pillars: data, evaluation, and methods. We first clarify the candidate task for- mulations and argue for a sensor-agnostic, instance-based definition that tolerates missing pre-event imagery. We then review existing
datasets along the axes of sensing modality, temporal configuration, and geographic coverage. On the evaluation side, we show that prevailing practices, largely based on a single aggregate score, are theoretically fragile, and we lay the foundations of a more informative, ranking-based, multi-score protocol. We further review methods ranging from task-specific deep networks to recent Earth observation foundation models, noticing that generalization to unseen events and regions remains an open difficulty. Finally, we discuss how these elements can inform the organization of a BDA challenge, including cross-validation, challenge, and private data-sharing setups designed to build trust between developers and users.
Read the full abstract ▾
All authors Anthony Cioppa, Virgile Blua, Sébastien Piérard, Pietro Ceccato, Marc Van Droogenbroeck
Computer Vision
Event-based Action Recognition by Pre-Training on Multi-Frame Representations
Emma Norrby, Tomas Wilkinson, Per Nyblom et al. (+1 more)
Action recognition is a crucial task in domains like robotics, patient monitoring, or surveillance. Recently, event-based action recognition has attracted much
attention due to the event camera's ability to capture fast motions and its improved
privacy over conventional
vision cameras. While frame-based event representations work well for short event sequences, their discriminative ability declines sharply with longer and more complex event sequences. In this work, we introduce a multi-frame representation that greatly improves recognition performance over its single-frame counterpart. We adopt a masked auto-encoding framework to pre-train a
Vision Transformer using a sparsity-aware masking
algorithm and
token selection mechanism to improve downstream performance and control computational complexity. We also introduce multiple mask
tokens for the decoder with a position-aware
attention mechanism, leading to further improved performance. Our proposed method shows impressive performance for challenging long-duration and complex event sequences, yielding state-of-the-art performance on THU-EACT-50 CHL and DailyDVS-200.
Read the full abstract ▾
All authors Emma Norrby, Tomas Wilkinson, Per Nyblom, David Gustafsson
Computer Vision
SpatialASM: Internalizing Active Mentalization for Spatial Reasoning
Yuhong Liu, Shenxi Wu, Junyi Chi et al. (+5 more)
When solving complex spatial
reasoning tasks, ordinary humans naturally revisit the scenes from task-relevant viewpoints and mentally reconstruct the spatial layouts from those viewpoints. But can Large
Vision-
Language Models (LVLMs) proactively think in the same way? We term this ability Active Spatial Mentalization (ASM), an intrinsic component of visual intelligence that remains substantially underdeveloped in current LVLMs. To bridge this gap, we introduce SpatialASM, a scalable framework to internalize ASM into LVLMs through training-time spatial simulation. SpatialASM employs an allocentric novel-view generator to ground hypothetical viewpoints in visual evidence and produces two complementary forms of supervision: i) explicit ASM trajectories that teach LVLMs when and how to adaptively invoke view planning and imagination during spatial
reasoning, and ii) automatically-constructed perspective-taking tasks that further strengthens viewpoint-conditioned scene imagination. On seven
benchmarks spanning relatively perception-oriented spatial understanding to more challenging tasks requiring multi-image, cross-view and viewpoint-transformation
reasoning, SpatialASM yields performance gain on LVLMs of four LVLM backbones in spatial
reasoning while retaining their original general
vision capabilities. The experimental results demonstrate that our approach improves average accuracy across the seven spatial
benchmarks. And the synthesis of ASM trajectories provides a scalable pathway to alleviate the scarcity of spatial Chain-of-Thought (CoT) supervision for eliciting active viewpoint exploration and mental scene simulation.
Read the full abstract ▾
All authors Yuhong Liu, Shenxi Wu, Junyi Chi, Beichen Zhang, Xuanlang Dai, Jiaqi Wang, Yuhang Zang, Dahua Lin
Computer Vision · Cross-listed from cs.RO
Dream to Remember: Continual Steering of Generalist Robot Policies with World Models
Bahey Tharwat, Iman Nematollahi, Marvin Alles et al. (+3 more)
Generalist
robot policies are trained with imitation learning and are bounded by the quality and coverage of their demonstrations. Reinforcement learning can improve and extend them, either by fine-tuning the policy weights or by training a small module to steer the input noise of a frozen generative policy. When tasks arrive sequentially, both approaches suffer from catastrophic forgetting. Fine-tuning on a task degrades prior tasks, and steering only shifts forgetting to the shared steering module. We propose Dream to Remember (D2R), which continually steers a frozen generalist policy toward new tasks, while retaining its earlier skills. As the steering module learns a new task, a frozen pretrained world model rehearses prior tasks through imagined rollouts, anchoring the module to its earlier behavior without a replay buffer or new data. When the world model is faithful enough, D2R learns the new task entirely in imagination. Otherwise, it learns the task in the environment while using the world model only to rehearse prior tasks. On CALVIN, D2R improves eight tasks in sequence entirely in imagination, raising average success from 63.7% to 74.8%, while LoRA fine-tuning on the same sequence drops it to 11.7%. With a frozen off-the-shelf generalist policy and world model, $\pi_0$ and A2World, D2R learns unseen LIBERO-90 tasks in the environment while keeping $\pi_0$'s performance on the LIBERO suites it was fine-tuned on, even though the world model is not accurate enough for policy optimization. Finally, on a real Franka
robot, D2R improves three tasks in sequence entirely in imagination while retaining the earlier improvements.
Read the full abstract ▾
All authors Bahey Tharwat, Iman Nematollahi, Marvin Alles, Xingyuan Zhang, Akshay L Chandra, Abhinav Valada
Computer Vision · Cross-listed from cs.LG
Spanning Diffusion and Flow-based Posterior Sampling via Exact Gaussian Conditioning
Teo Deveney, Lea Bogensperger and Hok Shing Wong
Flow-matching and
diffusion models are increasingly used as priors for linear inverse problems, however, most methods are bespoke to one class of trajectory prior. Plug-and-play methods have been proposed, however these implicitly decorrelate the guided trajectory, resulting in broad priors that can struggle when data is sparse. This work proposes $\beta$-Flow: a general plug-and-play method indexed by a correlation parameter $\beta$ that generalises over flow,
diffusion, and decorrelated priors, and a confidence parameter $\kappa$ that controls how strongly the path distribution is enforced. Each step requires only one forward evaluation of a denoising network, followed by a Gaussian conditioning step. This step is an exact Kalman update that is not dependent on network gradients. We show that many existing guided samplers can be interpreted as instances or approximations of our method, each implicitly using a different trajectory prior and operator in place of the Kalman gain. From this viewpoint, we provide a qualitative comparison of methods in terms of their implicit biases and assumptions. We demonstrate consistently state-of-the-art level performance across 7 inverse problems over 4
datasets in terms of PSNR and SSIM. Moreover, we demonstrate the utility of this prior control, by showing that confident high correlations (strict priors) work best when observations are less informative, while low correlation is preferred when observations are more informative, reflecting the complementary roles of the prior and likelihood in
Bayesian inference.
Read the full abstract ▾
Computer Vision
OccludeBench: Can VLMs Find What's Missing?
Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi et al. (+3 more)
How well can
vision-
language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated
benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual
reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended
reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which
prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.
Read the full abstract ▾
All authors Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi, Linrong Cai, Xingyu Fu, Zhuang Liu
Computer Vision · Cross-listed from cs.AI
UkrMeme: Multimodal Explainable Persuasion Detection in Ukrainian Memes
Nazar Andrushko, Vera Schmitt and Veronika Solopova
Memes play a central role in modern online communication, especially in contexts of political conflict where visual and textual cues combine to convey persuasive messages. We study persuasive technique detection in Ukrainian memes using
vision-
language models (VLMs). We introduce UkrMeme, a
dataset of 200 organically produced Ukrainian wartime memes annotated for persuasion techniques, and PTM-UA, a Ukrainian translation of a subset of the Propaganda 950
benchmark that preserves original visual content while translating meme text. We evaluate VLMs in a zero-shot prompting setup across image, text, and
multimodal inputs using hierarchical F1 metrics. Finally, we compare Layer-wise Relevance Propagation (LRP) and model-generated
token attributions and analyze the semantic categories of evidence emphasized by each explanation method.
Read the full abstract ▾
Computer Vision
SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models
Changhui Jing, Haochen Tian, Hongchen Li et al. (+8 more)
Driving world action models (WAMs) learn from paired trajectories and future observations. Post-training with reinforcement learning allows the policy to explore alternative trajectories and improve planning using driving rewards. However, driving logs contain future
videos only for the recorded trajectories, leaving these alternatives without corresponding visual supervision. Thus, we introduce SimVTA, a simulation-grounded post-training framework that combines trajectory optimization with
video supervision. First, we sample candidate trajectories for each observation and use a real-world simulator (e.g., World Engine) to render future
videos for high-reward candidates in real-world scenes reconstructed with 3D Gaussian Splatting (3DGS). This provides trajectory-matched supervision independent of the model's own predictions. Under a limited rendering budget, we prioritize
video supervision for higher-error futures among these rendered candidates. Finally, we optimize trajectory planning with Group Relative Policy Optimization (GRPO) and use a flow-matching loss on the matched
videos to update shared representations. At inference time, the model predicts trajectories directly, without simulator calls. On NAVSIM v1, applying SimVTA to a WAM initialization improves PDMS from 86.2 to 91.3 (+5.1). It achieves 88.2 EPDMS on NAVSIM v2 and an average HD-Score of 38.8 in closed-loop evaluation on HUGSIM. Ablations show that updating shared representations with
video supervision improves planning, while prioritizing higher-error futures among high-reward trajectories further improves performance.
Read the full abstract ▾
All authors Changhui Jing, Haochen Tian, Hongchen Li, Tianyu Li, Caojun Wang, Zihao Zhang, Yanhao Liu, Chengen Xie, Bailin Li, Hao Zhao, Zhenjie Yang
Computer Vision
Detection of extremely rare dead cells in fluorescence microscopy images
Prateek Verma, Benjamin Butter, Jian Zhang et al. (+1 more)
Dead
cell detection in fluorescence microscopy is essential for phenotypic screening and drug efficacy assessment. However, automated identification remains challenging due to signal variability, optical artifacts, and colocalization ambiguities across multiple channels. Markers that define nuclear boundaries do not neatly coincide spatially or temporally with apoptotic markers. Furthermore, marker accumulation kinetics vary across individual
cells and apoptosis stages, making universal thresholds unreliable. We present a systematic comparison of several computational approaches for detecting apoptotic
cells in
nucleus-stained specimens: (1) heuristic methods using intensity thresholds, distributions, correlations, and domain guidance; (2) classification using numerical features (Logistic Regression, Random Forest, XGBoost and LightGBM) and image features (ResNet-18 and EfficientNet-B0); and (3) anomaly detection, also using numerical features (Isolation Forest, one-class SVM and PCA reconstruction) and image features (convolutional autoencoders and Deep SVDD). Using a
dataset of 258 images containing 65,527
cells/
nuclei across multiple scenes with 243 manually annotated ground truth dead
cells, we evaluate each method on metrics such as PR-AUC, Precision, Recall, F1-score, and macro-F1. Our results indicate that classification using image features (deep learning) and numerical features (traditional) achieved the best performance, with ResNet-18 and XGBoost significantly outperforming manually designed heuristics and anomaly detection methods. Our findings demonstrate that such biomedical problems remain highly challenging to solve even with the state-of-the-art computational and learning techniques. We aim to provide an effective decision-support tool to researchers while tackling such problems either fully-automatically or in a human-in-the-loop setting as an aide to domain scientists.
Read the full abstract ▾
All authors Prateek Verma, Benjamin Butter, Jian Zhang, Xintao Wu