Computer Vision · Cross-listed from cs.RO
Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models
Senqiao Yang, Chengyao Wang, Yuxin Chen et al. (+13 more)
Scaling
robot data is crucial for building generalist
Vision-
Language-Action (VLA) models, yet
robot trajectories are harder to scale than web-scale image-text data because
embodied collection is costly and sparsely covers the physical world. This makes representation quality a central bottleneck: under a fixed
robot-data budget, continued pre-training must turn limited trajectories into transferable visual-action knowledge rather than merely fit actions. We propose VLAct, a VLA-oriented VLM backbone trained on broad, heterogeneous, multi-embodiment
robot data before task-specific fine-tuning. VLAct preserves the broad VLM prior and encourages shared action semantics across embodiments through VLM-prior preservation, multi-head continuous action co-supervision, and a partially unified cross-embodiment action layout, while allowing task-specific action heads during fine-tuning. Across simulation, real-world, and unseen-embodiment transfer, VLAct consistently improves downstream performance under fixed fine-tuning protocols. On LIBERO-Plus and RoboTwin 2.0, VLAct surpasses industrial VLA systems including ABot-M0 and LingBot-VLA, achieving success rates of 82.6% and 92.5%. On RoboDojo, VLAct ranks sixth among all policies by success rate and outperforms all explicitly designated world-action model (WAM) entries on both metrics. Most notably, on RoboCasa-GR1, an unseen
humanoid embodiment, VLAct using only 20% of downstream trajectories outperforms the full-data GR00T-N1.6 baseline. These results are obtained using fully open-source data and only a 16-GPU training setup, showing that representation-centric continued pre-training can deliver highly competitive performance under a modest compute budget and is an important independent axis of VLA progress beyond data scaling.
Read the full abstract ▾
All authors Senqiao Yang, Chengyao Wang, Yuxin Chen, Zixuan Wang, Longxiang Tang, Haokun Gui, Jinhui Ye, Changsheng Lu, Xiaoyang Wu, Mingkang Zhu, Pengguang Chen, Shu Liu, Zhuotao Tian, Hengshuang Zhao, Bei Yu, Jiaya Jia
Computer Vision
Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning
Hanyang Wang, Yimo Cai, Weiliang Chen et al. (+14 more)
Physical understanding and
reasoning depend on forming compact and generalizable representations of the world. While modern
vision-
language models can recognize and explain diverse physical events, they often lack explicit representations of the underlying mechanisms-such as object states, physical parameters, and governing dynamics-needed for reliably
reasoning how the world evolves and responds to interventions. In this work, we introduce Code-as-World, a paradigm that represents physical worlds through executable world representations. By expressing physical composition, dynamic
evolution, and visual appearance as executable code, Code-as-World provides a compact, quantitatively grounded, and controllable abstraction of the physical world. To construct such representations from
multimodal observations, such as natural-
language descriptions or real-world
videos, we develop an agentic discovery loop inspired by abductive
reasoning, where an
agent proposes, executes, renders, verifies, and iteratively refines executable world hypotheses. As a concrete application, we use verified executable worlds to provide scalable physical supervision for training
vision-
language models on quantitative physical
reasoning. Experiments show that Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and surpasses leading proprietary models, highlighting the potential of executable world representations as a scalable foundation for physical intelligence.
Read the full abstract ▾
All authors Hanyang Wang, Yimo Cai, Weiliang Chen, Jiawei Chi, Haowen Sun, Qiyu Dai, Yi-Hsin Hung, Xingzhuo Guo, Jinshan Ren, Runmao Yao, Ziwei Liu, Mingsheng Long, Yueqi Duan, Jun Gao, Jiangran Lyu, Fangfu Liu, Jialong Wu
Computer Vision · Cross-listed from cs.CR
Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
Benlei Cui, Shen Pang, Yuke Wang et al. (+7 more)
The safety of large
vision-
language models is increasingly stress-tested by
multimodal jailbreaks, yet existing attacks remain largely static at the meta level: template-based attacks freeze the image-text layout, while iterative attacks adapt only the image-text content with fixed attack strategies and frozen attacker parameters. We propose Meta-Adaptive
Multimodal Jailbreaking (MAMJ), which instead optimizes the attacker itself along two axes: an attack strategy
prompt (ASP) governing attack iteration and attacker model weights determining attack effectiveness. Across groups of
multimodal attack trajectories, an LLM-based critique first refines the ASP, after which group-aggregated attack success rate (ASR) rewards update those weights. On MM-SafetyBench, MAMJ achieves 81.0%, 78.9%, and 82.3% ASR against GPT-4o, Gemini-3-Pro-Preview, and Seed 2.0, respectively, outperforming the strongest sample-level baseline by up to 24.1 percentage points. The learned attacker, comprising the optimized ASP and attacker weights, also transfers without retraining to unseen victims and remains effective under representative defenses. These results reveal a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks and motivate defenses against meta-level adversaries. Code is available at https://github.com/Alibaba-VELLDEPTH/MetaJailbreak-VLM.
Read the full abstract ▾
All authors Benlei Cui, Shen Pang, Yuke Wang, Xuemei Dong, Yuwen Zhai, Jingqun Tang, Haiyang Yu, Hui Xue, Longtao Huang, Haiwen Hong
Computer Vision
Revisiting Local Context for Long-Horizon Streaming 3D Reconstruction
Jiarong Han, Jincheng Xiong, Yuzhou Liu et al. (+6 more)
Streaming 3D reconstruction from extremely long
videos requires estimating camera motion and scene geometry online under bounded
memory and computation. Early streaming models achieve causal, bounded-cost inference using finite context buffers or compact recurrent states, yet their estimates often deteriorate as sequences grow. Recent methods improve long-horizon stability by coupling short-range context with persistent or multi-level long-range
memory. We pursue a different route: we keep the learned temporal state strictly local and formulate predictions whose targets remain independent of sequence length. We present ABot-Recon, a simple streaming model that caches KV features from only the preceding 11 frames. It predicts a point map in the current camera coordinate system together with an adjacent-frame relative pose. These predictions remain equivariant under changes of reference frame, and global poses and geometry are recovered through sequential composition. To reduce accumulated drift, a lightweight temporal refiner improves relative rotations using recent visual and motion context, while a composition-aware pose loss supervises multi-step pose composition. Extensive evaluations on challenging long-sequence
benchmarks demonstrate the superior long-horizon performance of our local-context approach. On Oxford Spires, ABot-Recon achieves an ATE of 4.35 m and an RPE-R of $0.12^\circ$, reducing both errors by approximately 40% relative to the best prior results.
Read the full abstract ▾
All authors Jiarong Han, Jincheng Xiong, Yuzhou Liu, Linzhe Shi, Changjie Wu, Ning Guo, Mu Xu, Hang Zhang, Ming Qian
Computer Vision
FVeinSyn: Synthetic Finger Vein Image Generator
Yifan Wang, Jie Gui, Adams Wai Kin Kong et al. (+6 more)
A major challenge in finger vein recognition is the lack of large-scale public
datasets. Existing
datasets contain few identities and limited samples per finger, restricting the advancement of deep learning-based methods. To address this, we propose FVeinSyn, a large-scale controllable synthetic data generation framework for finger vein. It explicitly decouples synthesis of vascular
topology and imaging appearance to mitigate the limitations caused by insufficient training samples, such as inadequate identity diversity and restricted realism. Specifically: first, a finger vein identity generator models vascular
topology under physiological and
geometric constraints using
stochastic L-systems, producing anatomically valid and identity-distinctive vascular patterns. Then, a cascaded region-aware GAN renders the topological maps into realistic near-infrared images. Finally, an intra-class diversity generator introduces
geometric and optical perturbations to simulate realistic intra-class variations. Using FVeinSyn, we generated 500,000 images (10,000 vein identities, 50 samples per identity) and conducted extensive evaluations. Results show that FVeinSyn holds significant advantages in realism, identity diversity, vascular pattern consistency, and intra-class diversity. Models trained with FVeinSyn outperform real-data-only baselines a cross eight public
datasets, achieving an average accuracy improvement of 27.43%. The code is available at: https://github.com/EvanWang98/Synthetic-Finger-Vein-Generator.
Read the full abstract ▾
All authors Yifan Wang, Jie Gui, Adams Wai Kin Kong, Baosheng Yu, Changsheng Chen, Qi Li, Zhenan Sun, James Tin-Yau Kwok, Alex Kot
Computer Vision · Cross-listed from eess.IV
Destroy Me: Automatic Artifact Generation for Histopathology Images
Zuzanna Krawczyk-Borysiak, Adam Krawczyk, Mateusz Miller et al. (+4 more)
Deep learning's diagnostic utility in pathology is constrained by model vulnerability to real-world data imperfections. While current strategies favor "perfect data" by filtering low-quality regions, which can lead to the loss of valuable diagnostic context, we propose a paradigm shift: engineering models to thrive in imperfect environments using "Destroy Me", a hybrid framework for realistic artifact synthesis and robust data augmentation. Our approach combines Stable
Diffusion, fine-tuned to preserve morphological continuity by realistically integrating artifacts with the underlying tissue architecture, with physics-based procedural modeling to synthesize six common artifact types: tissue folds, precipitates, blur, stitching errors,
dust, and pen markers. Artifact fidelity is assessed using Kernel Inception Distance (KID) and color Wasserstein distance metrics. Validating this strategy on lung adenocarcinoma pattern classification with an nnU-Net, we confirm that models trained on "destroyed" patches consistently outperform baselines on independent real-world
datasets. Specifically, we observed a 10.5% relative improvement in macro F1-score and a 15% relative increase in the Cohen's Kappa ($\kappa$) coefficient. Crucially, our results demonstrate that selective, impact-weighted augmentation is vital for balancing practical robustness with the preservation of subtle diagnostic features.
Read the full abstract ▾
All authors Zuzanna Krawczyk-Borysiak, Adam Krawczyk, Mateusz Miller, Gabriela Kaczmarek, Sławomir Pakuło, Małgorzata Sokół, Żaneta Swiderska-Chadaj
Computer Vision · Cross-listed from cs.LG
A Deeper Analysis of Block-Sparse Featurizers
Alexandru-Iulius Jerpelea and Amith Ananthram
The recently introduced block-sparse featurizer (BSF; Fel et al., 2026) is similar to a sparse autoencoder (SAE), but its atomic unit is a small subspace (a block of directions) rather than a single direction. It is designed for features that live on low-dimensional
manifolds, which are especially frequent in
vision. This work studies the BSF's strengths and weaknesses, finding how it still somewhat suffers from classic SAE failure modes, like feature splitting and composition. We propose several architectural changes to the BSF, including a Tournament Top-K selection rule that significantly reduces feature splitting, and we also extend the block paradigm to the crosscoder.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.SE
Image Augmentation as Test Generation for Deep Learning-Based Image Retrieval Systems
Yehan De Silva, Anirudh Sridhar, Armin Lotfy et al. (+6 more)
Ensuring the reliability of deep learning-based image
retrieval systems is a software engineering challenge. This paper presents a dual contribution: (1) a literature review of augmentation and generation techniques which resulted in the identification of 50 techniques which we organized into a ten-category taxonomy, and (2) a large-scale empirical study that evaluates these techniques as test generators for
embedding-based image
retrieval systems. Augmented images are embedded using Amazon Titan and OpenCLIP, and evaluated across four analytical dimensions: (1)
embedding-space similarity, (2)
embedding uncertainty measured via four
estimators, (3) semantic realism scored by LLaVA, and (4)
retrieval failure rate. Experiments are performed on three
datasets: CIFAR-10, ImageNet-1K, and a
dataset from an industrial partner (March Networks). Across all evaluated
datasets and
embedding models, and under the single severity level tested for each technique, weather simulation and SaSPA are the image augmentation/generation techniques that produce the highest
embedding uncertainty and failure rates while maintaining a favorable balance between performance stability, visual realism, and augmentation effectiveness. The results we discuss are configuration-specific and may shift under milder or stronger perturbation settings. In contrast, GAN-based augmentation techniques are among the lowest in realism, indicating the presence of synthetic artifacts and perceptual inconsistencies that reduce their suitability to produce realistic test inputs. Overall, our findings provide practical guidelines for selecting augmentation techniques that maximize test diversity while preserving realistic image characteristics, thereby enabling the construction of comprehensive and effective test suites for image
retrieval systems while reducing the cost of manual data labeling through the use of metamorphic testing.
Read the full abstract ▾
All authors Yehan De Silva, Anirudh Sridhar, Armin Lotfy, Nafiseh Kahani, Yvan Labiche, Ziyu Wang, Frank Ouyang, Clare Carty, Azalia Shamsaei
Computer Vision
UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City
Tianjie Ju, Zheng Wu, Yueqing Sun et al. (+15 more)
Multimodal large
language models (MLLMs) can interpret a street view, but reliable urban action depends on whether such local evidence remains useful after the
agent starts to move. In this paper, we investigate how far current MLLM
agents can turn local urban perception into reliable action in a real-scale city. We propose UrbanGround, an urban sandbox built from Hong Kong's territory-wide 3D geospatial data. It combines the city's geographic structure with continuous, collision-constrained control through a shared evaluation interface.
Agents use first-person observations and an interactive map to select actions across tasks ranging from local question answering to long-horizon
navigation. Our analysis follows the growth of the spatial problem through three research questions. We first test whether an
agent can gather and interpret local visual evidence to answer spatial questions. Then we ask whether these abilities support
navigation as destinations become farther away and less explicit. Finally, we examine whether the resulting behavior survives changes in route availability and pedestrian motion. MLLM
agents usually show useful atomic abilities in visual recognition and short-range spatial
reasoning, while orientation and pedestrian-aware movement remain unreliable. Their central failure emerges over extended exploration, where local abilities do not compose into sustained goal-directed behavior and errors accumulate without effective correction. We hope UrbanGround will support broader study of how far MLLM
agents can explore reliably in open-ended urban environments.
Read the full abstract ▾
All authors Tianjie Ju, Zheng Wu, Yueqing Sun, Yuhan Cui, Bobo Li, Shengqiong Wu, Pengzhou Cheng, Haodong Zhao, Zongru Wu, Xinbei Ma, Doris Zhang, Kunling Li, Mong-Li Lee, Wynne Hsu, Hao Fei, Qi Gu, Gongshen Liu, Zhuosheng Zhang
Computer Vision
Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
Chanho Park, Daehyeon Choi, Jihyun Lee et al. (+1 more)
Vision-
language models (VLMs) can locate an image region referred to by a text
prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by
retrieval heads in large
language models, we ask whether VLMs contain an analogous mechanism for visual
retrieval. We answer affirmatively by introducing Visual
Retrieval Heads (VRHs), a small subset of
attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions. To find them, we recast existing head-scoring methods under a unified design space over query
tokens, key aggregation, and cross-sample aggregation. We then show that scoring
attention from output prediction
tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across eleven VLMs and five referring-expression
benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. Beyond replicating the causal-sparse-universal triad established for text
retrieval heads, VRHs exhibit several properties not previously reported: they generalize across visual reference tasks, remaining causal on attribute, spatial, counting, and visual-math
benchmarks despite being discovered through bounding-box prediction; they are functionally specific, preserving output format while corrupting localization; and they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in
vision encoder, projector, and instruction tuning.
Read the full abstract ▾
All authors Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung
Computer Vision
Reconstructing Humans and Objects in Interaction using Large Reconstruction Models
Agniv Chatterjee and Georgios Pavlakos
Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer
vision with applications in AR/VR, robotics, and
embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful
geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple
benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
Kechen Liu and Ola Shorinwa
State-of-the-art action-conditioned
video models are typically restricted to a single
robot embodiment, preventing them from leveraging the vast corpus of heterogeneous
video data that contains rich signals for learning generalizable physics. To bridge this gap, we introduce CLAP, a framework for cross-embodiment action-conditioned
video generation capable of being trained on diverse, internet-scale
videos across human and robotic
agents. CLAP is grounded in the insight that universal physical laws govern spatiotemporal dynamics regardless of the actor. However, cross-embodiment learning is non-trivial because action representations vary sharply across
robot platforms and are typically absent in human
videos. CLAP addresses this fundamental challenge through the following core contributions. First, CLAP reconciles disparate action spaces using end-effector poses,
language instructions, and latent actions. Second, to resolve their individual limitations, CLAP introduces a curriculum-based cross-embodiment learning recipe that first learns foundational physical priors across unlabeled
video data using latent actions and subsequently grounds them in end-effector action spaces for zero-shot deployment to real-world tasks. Crucially, CLAP approaches or surpasses state-of-the-art single-embodiment
video models in challenging environments like DROID. These performance advantages compound via few-shot adaptation to establish a novel paradigm for training single-embodiment
video world models. Ultimately, CLAP delivers the most comprehensive suite of action-conditioned
video world models to date - spanning diverse action-conditioning spaces (end-effector,
language, and latent) and
robot morphologies (including cross-embodiment, DROID, Bridge, bimanual YAM
robots, and G1
humanoids). We open-source all code and models. Project Website at https://omni-clap.github.io .
Read the full abstract ▾
Computer Vision
LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
Lukas Kuhn, Lucas Maes, Giuseppe Serra et al. (+4 more)
Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first
video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of
tokens the encoder observes; uniform random
token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest
video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric
benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal
attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same
videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed,
video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.
Read the full abstract ▾
All authors Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner
Computer Vision
Successive Capacity Growth: Task-Complexity-Driven Width and Depth Expansion for Vision Transformer Encoders in JEPA World Models
Frederik Berenz
Joint-
Embedding Predictive Architectures (JEPAs) for world modeling typically employ fixed-size
Vision Transformer encoders that are over-provisioned for simple tasks and under-provisioned for complex ones, with significant redundancy across
attention heads. We propose Successive Capacity Growth (SCG), a method that starts from a minimal encoder (1 head, 2 layers, 283K parameters) and grows incrementally in width (adding
attention heads for low-level semantic capacity) or depth (adding
transformer blocks for higher-order semantic abstraction), driven by a task-agnostic test-and-verify mechanism that exploits function-preserving expansion to safely trial architectural changes and roll back if they do not improve prediction loss. The Sketched Isotropic Gaussian Regularizer (SIGReg) ensures that all learned semantic dimensions remain statistically independent and aligned with the predictive objective, preventing collapse even as the architecture grows. On a 60-dimensional multi-object dynamics task, SCG naturally triggers depth expansion, improving prediction loss by 20.3% over the fixed small baseline with 56 times greater parameter efficiency than scaling to the fixed large model; on a 2D
navigation task, a single width expansion yields even an 23% improvement over the fixed large model. Across all three tested environments of increasing complexity, the adaptive encoder matches or exceeds the fixed small baseline, with zero false-positive expansions and bit-exact function preservation (ratio = 1.0, absolute difference = 0.0). The take-away is that JEPA world model encoders need not be pre-allocated at maximum capacity - they can grow successively as the task demands, achieving significant compute and data efficiency while maintaining representation quality.
Read the full abstract ▾
Computer Vision
KnockGS:interaction-Grounded Calibrationof Physical Gaussian Representations
Chenchen Ge, Hanwen Shen, Bowen Jing et al. (+6 more)
Physics-integrated 3D Gaussian representations now allow reconstructed deformable objects to be simulated and rendered under explicit material models. Existing pipelines, however, assume that material parameters are known or manually specified, limiting their applicability when these parameters must be inferred from observed object dynamics. We propose KnockGS, an interaction-response PhysicalGS framework that estimates the elasticity and density scales of a 3D Gaussian object from its dynamics under a known applied force. Rather than treating physical simulation only as a forward process, we turn the force-induced response into a calibration signal: temporal response features are xtracted from the observed dynamics, the two material scales are estimated from those features, and the estimate is then frozen and written back into the same simulator so that it can be tested on an interaction it was never fitted to.We evaluate the framework on both parameter recovery and response-level fidelity. The estimated scales are compared against hidden ground truth, and the re-simulated object is measured against the target using 3D particle trajectories, response-curve statistics, and rendered-frame quality. Across five held-out material targets, our method recovers the scales substantially more accurately than response
retrieval, global regression, or a fixed default material, and the frozen estimate remains predictive under interactions that differ in direction and in magnitude. Interaction response therefore carries enough information to calibrate material scales in physically grounded 3D Gaussian representations.Our study is a first step toward interactive PhysicalGS systems that calibrate a Gaussian asset whose rendered appearance and simulated response are consistent.
Read the full abstract ▾
All authors Chenchen Ge, Hanwen Shen, Bowen Jing, Jiyuan Cai, Xiaofeng Wang, Hongsen Lei, Weitao Zhou, Dandan Zhang, Haibao Yu
Computer Vision
PAWBench: How Far Are We from Probabilistically Aligned World Modeling?
Yuandong Pu, Le Zhuo, Sayak Paul et al. (+11 more)
Recent
video generation models are increasingly framed as world models. Many physical processes can unfold in more than one valid way. Therefore, a world model should reproduce not only a plausible trajectory, but also the distribution of possible behaviors under the same initial observation and action. We call this distribution-level requirement probabilistic
alignment. However, existing evaluations largely assess individual-
video plausibility and do not test whether repeated generations recover the correct distribution. This raises a central question: how far are current
video generators from probabilistically aligned world modeling? To answer it, we formalize probabilistic
alignment as a distributional criterion for world models and introduce PAWBench, a
benchmark for evaluating
video generators as
stochastic samplers of world dynamics. We further introduce PAWEval, an outcome-level protocol that converts repeated
video rollouts into empirical distributions over possible physical behaviors. Across 50 scenarios and eleven current systems, no model consistently matches the reference probabilities while recovering the range of valid behaviors. Having established this gap, we test whether
language prompts, initial noise sampling, or model training can reshape the model's predictive distribution. We believe our work can serve as a foundation for future efforts to move towards probabilistically aligned world modeling.
Read the full abstract ▾
All authors Yuandong Pu, Le Zhuo, Sayak Paul, Gabriel Jorge Menezes, Avram {\DJ}or{\dj}ević, Shiyang Li, Yifan Zhou, Bin Fu, Wenlong Zhang, Junjun He, Yu Qiao, Yihao Liu, Jinbo Xing, Xi Chen
Computer Vision
R2M-Bench: Evaluating Revisit Memory via Relative Consistency in Interactive Video World Models
Qiwen Gu, Bingjie Gao, Rui Chen et al. (+7 more)
High similarity between first-visit and return frames does not necessarily show that a
video world model remembered the scene; the intervening rollout may simply have changed very little. This ambiguity makes absolute revisit scores sensitive to rendering stability, repetitive content, and failed motion. We introduce R2M-Bench (Relative Revisit
Memory Benchmark), a
benchmark of observable revisit-selective consistency. For every detected return, R2M-Bench compares the revisit pair with two controls from the same rollout: a gap-matched non-revisit pair that measures generic temporal stability and a short-range pair that estimates short-horizon consistency. These comparisons produce MemoryGain (MG), the revisit advantage over the temporal baseline, and the Normalized
Memory Ratio (NMR), which normalizes this advantage by the short-to-baseline dynamic range. R2M-Bench combines 100 reference scenes with three leave-and-return trajectories to form 300 instances and evaluates appearance fidelity, scene and object identity, local geometry, and persistent state. Across seven action-conditioned
video world models, Overall NMR correlates with human consistency judgments at Spearman's $\rho=0.547$ (95% CI $[0.45,0.63]$). Its within-model correlation magnitude with generated motion is $0.072$, compared with $0.207$ for raw revisit similarity, indicating that relative calibration substantially reduces the slow-motion shortcut. DreamX-World-Memo achieves the highest Overall NMR among the evaluated
video models. Together, these results support same-rollout relative calibration as a practical way to distinguish revisit-specific consistency from generic temporal stability.
Read the full abstract ▾
All authors Qiwen Gu, Bingjie Gao, Rui Chen, Geng Li, Jifan Li, Qishuai Wen, Li Niu, Jing Tang, Xiangxiang Chu, Junqiao Zhao
Computer Vision · Cross-listed from cs.GR
Comparative Evaluation of 3D Reconstruction Methods for Immersive Visualization of Laboratory Objects
Brian De La Cruz, Aaron Y. Zhao, Maitrey Gramopadhye et al. (+4 more)
In this study, we examined whether current 3D reconstruction methods can support the creation of realistic
holographic representations of laboratory objects for educational use. In this regard, we compared four approaches: photogrammetry, a neural radiance field (NeRF)-based method, Gaussian splatting, and
LiDAR. These methods were used to generate
holographic models of common laboratory items and their fidelity was evaluated by graduate students. Participants assessed the models for shape, color, texture, and visual defects using a repeated-measures design. Across objects, the NeRF-based method produced the most consistently high-fidelity representations, particularly for transparent, reflective, or low-texture items that were difficult to capture with other approaches. Shape and color were generally reproduced more successfully than texture, suggesting that some visual properties remain more challenging to represent accurately in educational holograms. Beyond identifying the strengths and limitations of each reconstruction method, the study demonstrates a practical workflow for creating immersive learning objects that may support pre-laboratory preparation, spatial
reasoning, and student engagement in AR/MR-based educational environments. These findings offer design-relevant insights for educators and researchers developing immersive digital learning experiences.
Read the full abstract ▾
All authors Brian De La Cruz, Aaron Y. Zhao, Maitrey Gramopadhye, Sawyer J. Lazar, Xianming Tan, Daniel Szafir, David S. Lawrence
Computer Vision
Detection of Christmas tree plantations from high-resolution aerial imagery. A case study in the French Morvan
Francesca Razzano, Emanuele Dalsasso, Adrien Baysse-Lainé et al. (+3 more)
Christmas tree plantations are economically relevant, yet a largely unexplored application domain in Remote Sensing (RS). Their delineation is challenging because of high planting density, short rotation cycles, visual confusion with surrounding vegetation, the availability of dense labels for one reference year only, and severe class imbalance at the landscape scale. Although Deep Learning (DL) methods have shown strong potential for vegetation mapping, existing approaches are typically designed for forests, generic plantation systems, or orchards, and do not explicitly address the structural specificity and hard-negative confusion that characterize Christmas tree plantations. In response to these challenges, this work makes three main contributions: (i) it frames Christmas tree plantation mapping as a distinct rare-target semantic
segmentation problem; (ii) it introduces a Hard Negative Mining (HNM) strategy to improve discrimination against confusing background patterns; and (iii) it evaluates the proposed framework across complementary levels, including supervised testing, temporal transfer, and large-scale validation. On the 2020 test set held out, the best model, DeepLabV3 with a ResNet-34 encoder, achieves an IoU of 0.733 and an F1-score of 0.846. HNM substantially improves precision-recall behavior, increasing the area under the precision-recall curve from 0.204 to 0.913. Temporal inference further shows meaningful transferability, reaching IoU/F1 values of 0.751/0.858 on 2017/2018 and 0.691/0.817 on 2023. Large-scale validation further highlights the intrinsic difficulty of the task, as Christmas tree plantations occupied only a very small fraction of the extent of the common evaluation, corresponding to 1,498.4 ha (1.72%) in 2017/2018 and 1,782.2 ha (2.04%) in 2023 out of 87,309.4 ha in total.
Read the full abstract ▾
All authors Francesca Razzano, Emanuele Dalsasso, Adrien Baysse-Lainé, Silvia Liberata Ullo, Gilda Schirinzi, Jocelyn Chanussot
Computer Vision
TADP: Task-Aware Deformable Prediction for Single-Stage 3D Object Detection
Su Wang, Yaochen Li, Min Yang et al. (+3 more)
Most single-stage 3D object detectors complete different tasks with the same extracted features. Nevertheless, it is impossible to project features into a common space that is adaptive for all the tasks. We present a novel task-aware deformable prediction (TADP) method for single-stage 3D object detection to solve this problem. Firstly, a triple feature refinement aggregation module is designed to extract three-level features adaptively. Additionally, we design the multi-scale feature aggregation block to fuse multi-scale features in a scale-aware manner. Finally, the prediction of each task is deformed with the designed plug-and-play task-aware deformation head. It can percept the emphasis and interaction of each task. We also designed three different deformation modules. The experimental results demonstrate that the proposed deformation head shows good results on other detection methods. The experimental results on the KITTI
dataset demonstrate that the car mAP is 80.91%, surpassing many state-of-the-art methods on the KITTI
benchmark.
Read the full abstract ▾
All authors Su Wang, Yaochen Li, Min Yang, Jiaohao Nie, Chang Liu, Yuehu Liu
Computer Vision
Sidecar: Training-Free Semantic Reuse for Character-Consistent Free-form Visual Storytelling
Sibo Dong and Sarah Adel Bargal
Visual storytelling requires generating images that follow a narrative while preserving consistent character identities across frames. In free-form story generation, a character is fully described only when first introduced and is later referred to by a type-level mention or pronoun. Although this setting better reflects natural storytelling, later
prompts may omit important identity-related semantics, making character consistency more difficult to maintain. We propose Sidecar, a plug-and-play semantic augmentation module that preserves entity-level information from the initial description and injects the missing semantics into later
prompt embeddings. Sidecar requires no additional training and does not modify the architecture of the base
diffusion model. Experiments on FreeStoryBench show that Sidecar consistently improves
prompt-image
alignment and character consistency across multiple SDXL- and FLUX-based baselines, with negligible computational overhead.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.DC
Decoupled I/O-Dominant Pipelines for Large-Scale Whole-Slide Image Embedding Extraction
Mayanka Chandrashekar, Xi Zhang, Ethan Seefried et al. (+3 more)
Whole-slide images (WSIs) are central to computational pathology but are prohibitively large, making patch-based processing the practical unit for foundation model inference. At scale, however, generating and handling massive numbers of patches on quickly introduces significant I/O and orchestration overhead, often dominating end-to-end performance. We present a decoupled, I/O-aware pipeline for large-scale WSI
embedding extraction that decomposes the workflow into three stages: (1) patch generation and staging, (2) embarrassingly parallel
embedding inference, and (3) sharded vector database ingestion. This design isolates data movement from compute, enabling efficient patch delivery, scalable multi-node inference with minimal communication. The resulting system produces a distributed vector database where
embeddings are persistently coupled with rich metadata (e.g., patient, slide, and patch attributes), enabling efficient filtering,
retrieval, and downstream reuse. This representation database is compact and reusable for tasks such as
retrieval, classification, and few-shot learning, particularly benefiting low-resource environments. We show that decoupling I/O, computation, and ingestion enables high-throughput WSI
embedding extraction at scale. By characterizing the scaling envelope, we demonstrate that storage dominates beyond moderate concurrency, reframing WSI
embedding extraction as a data-centric systems problem rather than a purely compute-bound workload.
Read the full abstract ▾
All authors Mayanka Chandrashekar, Xi Zhang, Ethan Seefried, Tirthankar Ghosal, John Gounley, Heidi Hanson
Computer Vision
UniFLM: United Segmentation and Measurement on Fetal Limb Ultrasonic Image
Zeen Zhou, Qiuhua Chen, Xiaojun Cao et al. (+3 more)
Prenatal ultrasound examination is crucial for assessing fetal limb development and detecting congenital anomalies. However, existing artificial intelligence models often overlook fetal lethal skeletal dysplasias due to the lack of high-quality annotated data and a unified framework for multiple long bones. Moreover, generic
segmentation models struggle with the inherent noise and semantic gaps in ultrasound images. To address these challenges, we construct the Fetal Limb Bones (FLB)
dataset, comprising high-quality annotations for the humerus, femur, tibia-fibula, and radius-ulna. Furthermore, we propose UniFLM (United
Segmentation and Measurement on Fetal Limb Ultrasonic Image), a unified framework for automatic cross-plane
segmentation and measurement. UniFLM incorporates a Semantic
Alignment Skip Connection (SASC) module to bridge the semantic gap between encoder and decoder features, and a Positive Sampling (PoSamp) strategy to filter noise and extract essential semantic information. Finally, a Point Regression Mapping (PRM) module is introduced to learn clinician annotation patterns for precise bone length measurement. Extensive experiments conducted on the FLB
dataset and the public FetalP5
benchmark demonstrate that UniFLM achieves competitive performance with consistent generalization across four bone categories and external multi-center data, supported by comprehensive statistical validation including Bland-Altman agreement analysis and bootstrap confidence intervals. The source code is publicly available at https://github.com/chosen1203/UniFLM.
Read the full abstract ▾
All authors Zeen Zhou, Qiuhua Chen, Xiaojun Cao, Changmao Chen, Chao Sun, Bo Du
Computer Vision
DINOcular: Self-Supervised Visuospatial Representations
Farkhat Almukhamedov, Sami Azirar and Hermann Blum
We introduce a self-supervised framework for learning joint visuospatial representations from RGB-D observations. While modern
vision foundation models are trained almost exclusively on RGB images, many
embodied systems have access to explicit depth sensing, which provides
geometric information that
monocular inputs cannot recover. Our method integrates depth-derived
geometric priors with a visual backbone through inter-patch and intra-patch fusion, enabling the model to encode both appearance and spatial structure efficiently. The resulting representation shows promising improvements on 3D awareness while preserving semantic transfer: it outperforms prior methods of comparable scale on multiple 3D geometry
benchmarks, and remains competitive when probed for standard RGB-D semantic
segmentation tasks.
Read the full abstract ▾
Computer Vision
CODE: Cross-Modal Calibration and Dynamic Suppression for Open World Object Detection
Hao Xu, Zhaoning Shi, Hehe Jin et al. (+1 more)
Open World Object Detection (OWOD) built on
multimodal foundation models often suffers from semantic ambiguity caused by unidirectional text-to-
vision matching, while rigid outlier penalties may over-suppress unknown objects near known-class decision boundaries. We propose CODE (Cross-Modal Calibration and Dynamic Suppression), a unified inference-time framework with three complementary components. Cross-Modal Joint Confidence Calibration injects global visual prototypes to calibrate text-driven known-class predictions. Uncertainty-Guided Universal Objectness Enhancement measures classification hesitation from local visual responses to strengthen potential unknown objects. Dynamic Outlier Suppression via Confidence Margin replaces rigid suppression with a margin-aware adjustment that preserves ambiguous out-of-distribution instances. Experiments on the Real-World Detection
benchmark demonstrate that, with the OWL-ViT L/14 backbone, CODE achieves 21.7 U-mAP and 40.8 K-mAP in Task 1, surpassing the previous state of the art by 2.6 and 2.3 points, respectively.
Read the full abstract ▾
All authors Hao Xu, Zhaoning Shi, Hehe Jin, Bo Ma
Computer Vision
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
Junjie Liu, Shengyuan Ye and Xu Chen
Vision-
Language Models (VLMs) demonstrate exceptional visual
reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual
tokens. Existing visual
token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-
vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict
token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the
vision encoder and the Large
Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-
Attention Extractor (DDAE) selectively retains visual
tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual
tokens, yielding a 3.1x speedup in time to first
token (TTFT). Our code is available at https://github.com/jjL357/PACE.
Read the full abstract ▾
Computer Vision
Vision-centric generative AI models: A software-hardware perspective
Eleni Tselepi, Cristian Sestito, Shady Agwa et al. (+1 more)
Vision generative artificial intelligence (AI) has emerged as one of the most rapidly advancing areas of deep learning. The explosion of
multimodal models has made them widely associated with text-to-image applications running on large datacentres. However,
vision generative models are equally needed in applications that operate under strict hardware constraints at the edge, including autonomous vehicles, agricultural sensors, and mobile devices. In this Perspective, we argue that progress in
vision generative AI has been driven by output quality, with hardware evolving reactively to accommodate growing model demands. We quantify the parameter cost and energy efficiency of these models across a range of accelerator platforms, and map four generative model families against seven real-world application domains. Finally, we advocate a software-hardware co-design approach, where deployment constraints are considered from the start of the design process, ensuring that the "right model" runs on the "right hardware" to serve the "right application", making generative AI deployment sustainable and accessible across a much broader range of platforms.
Read the full abstract ▾
All authors Eleni Tselepi, Cristian Sestito, Shady Agwa, Themis Prodromakis
Computer Vision · Cross-listed from cs.IT
Knowledge Distillation Driven Semantic NOMA with GAN Refinement for 6G Robotic Vehicle Networks
Qifei Wang, Zhen Gao, Li Qiao et al. (+4 more)
To achieve sustainable intelligent mobility, 6G-empowered robotic vehicles (RVs) require high-fidelity visual perception under stringent bandwidth and energy constraints. Semantic communication offers a spectral-efficient solution but suffers from severe interference in uplink non-orthogonal multiple access (NOMA) RV networks. To address this, we propose a knowledge
distillation-driven and generative models-enhanced NOMA framework for robust and green RV communications, named KDG-SemNOMA. First, we develop a ConvNeXt-based deep joint source-channel coding (DeepJSCC) architecture with an enhanced
attention feature (AF) module for dynamic channel adaptation. Second, to mitigate interference without inference overhead, an orthogonal transmission teacher model guides the NOMA student model via a two-stage knowledge
distillation strategy. Finally, to address the over-smoothing artifacts of pixel-wise optimization, we introduce a channel-conditional GAN (cGAN). By explicitly taking the Stage-I initial reconstruction and channel states as conditional inputs, this module refines coarse outputs into high-fidelity images with realistic textures. Experiments on FFHQ-256 demonstrate that KDG-SemNOMA significantly outperforms state-of-the-art methods in both pixel-level accuracy and perceptual fidelity.
Read the full abstract ▾
All authors Qifei Wang, Zhen Gao, Li Qiao, Ziwei Wan, De Mi, Dapeng Li, Ying Sun
Computer Vision
Unsupervised Adaptation of 3D CT Foundation Models for 3D CBCT Segmentation
Gauthier Miralles, Loic Le Folgoc, Vincent Jugnon et al. (+1 more)
Accurate 3D
segmentation of cone-beam CT (CBCT) is critical for interventional and radiation therapy applications, yet it remains limited by two compounding challenges: the scarcity of annotated CBCT data and the large domain shift from diagnostic CT. Interventional CBCT exhibits fundamental modality differences from conventional CT, driven by acquisition and physics effects as well as contrast-specific vascular content, thereby limiting effective cross-modality model transfer. We propose a novel unsupervised domain adaptation (UDA) framework based on redundancy-reducing feature
alignment, enabling 3D CBCT
segmentation with no target-domain annotations or inference-time adaptation. Our framework is architecture-agnostic, seamlessly adapting both CNN-based and ViT-based foundation models. We evaluate our method on two challenging CT-CBCT liver
segmentation benchmarks: one for interventional vascular procedures and one for radiation therapy, demonstrating that even large-scale pretrained
segmentation networks require explicit feature-space bridging to generalize across acquisition modalities, and that our approach consistently outperforms existing pretrained foundation model and UDA strategies. To support reproducibility and benchmarking, we release the liver
segmentations for a public CBCT
dataset, along with the code, trained models, and weights.
Read the full abstract ▾
All authors Gauthier Miralles, Loic Le Folgoc, Vincent Jugnon, Pietro Gori
Computer Vision
SSMB: Self-Supervised Local Feature Detection under Motion Blur
Zhenjun Zhao, Fabio Bellavia, Wenting Wang et al. (+6 more)
Keypoint detection under motion blur remains a significant challenge, as blur distorts local image structure and degrades the repeatability of feature localization. Existing approaches either rely on computationally expensive deblur-then-detect pipelines that may introduce restoration artifacts, or learn to regress the image positions of handcrafted keypoints extracted on sharp images, which reflects the assumptions of the handcrafted detector rather than what is truly repeatable under blur. We present SSMB, a deblur-free, self-supervised keypoint detector for motion-blurred images that requires neither handcrafted detectors nor external pseudo-labels. SSMB introduces the Local Discriminability Enhancement (LDE) module, which restores fine-grained local discriminability after global feature mixing. Training is performed in two stages. First,
geometric pretraining on synthetic shapes bootstraps spatially discriminative keypoint detection without any external detector, just from the rendered geometry. Second, blur-aware training on real sharp-blur image pairs learns blur-invariant detection through a multi-component self-supervised objective that enforces cross-domain consistency,
geometric alignment, and spatial coverage. Extensive evaluations on keypoint detection, image matching, relative pose estimation, and visual localization under motion blur demonstrate that SSMB establishes a new state-of-the-art among sparse keypoint detectors, consistently outperforming both supervised and self-supervised baselines across all tasks. Code, models, and
datasets will be publicly available upon paper acceptance.
Read the full abstract ▾
All authors Zhenjun Zhao, Fabio Bellavia, Wenting Wang, Fan Zhu, Jiajun Wu, Suryansh Kumar, Mingqiang Wei, Haoang Li, Javier Civera