Computer Vision · Cross-listed from cs.RO
VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
Jingwen Sun, Wenyao Zhang, Zekun Qi et al. (+6 more)
Pretraining
Vision-
Language-Action (VLA) policies on internet-scale
video is appealing, yet current latent-action objectives often learn the wrong thing: they remain anchored to pixel variation rather than action-relevant state transitions, making them vulnerable to appearance bias, nuisance motion, and information leakage. We introduce VLA-JEPA, a JEPA-style pretraining framework that sidesteps these pitfalls by design. The key idea is leakage-free state prediction: a target encoder produces latent representations from future frames, while the student pathway sees only the current observation – future information is used solely as supervision targets, never as input. By predicting in latent space rather than pixel space, VLA-JEPA learns dynamics abstractions that are robust to camera motion and irrelevant background changes. This yields a simple two-stage recipe – JEPA pretraining followed by action-head fine-tuning – without the multi-stage complexity of prior latent-action pipelines. Experiments on LIBERO, LIBERO-Plus, SimplerEnv and real-world manipulation tasks show that VLA-JEPA achieves consistent gains in generalization and robustness over existing methods.
Read the full abstract ▾
All authors Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, Zhibo Chen
Computer Vision
Causality in Video Diffusers is Separable from Denoising
Xingjian Bai, Guande He, Zhengqi Li et al. (+3 more)
Causality – referring to temporal, uni-directional cause-effect relationships between components – underlies many complex generative processes, including
videos,
language, and
robot trajectories. Current causal
diffusion models entangle temporal
reasoning with iterative denoising, applying causal
attention across all layers, at every denoising step, and over the entire context. In this paper, we show that the causal
reasoning in these models is separable from the multi-step denoising process. Through systematic probing of autoregressive
video diffusers, we uncover two key regularities: (1) early layers produce highly similar features across denoising steps, indicating redundant computation along the
diffusion trajectory; and (2) deeper layers exhibit sparse cross-frame
attention and primarily perform intra-frame rendering. Motivated by these findings, we introduce Separable Causal
Diffusion (SCD), a new architecture that explicitly decouples once-per-frame temporal
reasoning, via a causal
transformer encoder, from multi-step frame-wise rendering, via a lightweight
diffusion decoder. Extensive experiments on both pretraining and post-training tasks across synthetic and real
benchmarks show that SCD significantly improves throughput and per-frame latency while matching or surpassing the generation quality of strong causal
diffusion baselines.
Read the full abstract ▾
All authors Xingjian Bai, Guande He, Zhengqi Li, Eli Shechtman, Xun Huang, Zongze Wu
Computer Vision
4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere
Yihang Luo, Shangchen Zhou, Yushi Lan et al. (+2 more)
We present 4RC, a unified feed-forward framework for 4D reconstruction from
monocular videos. Unlike existing approaches that typically decouple motion from geometry or produce limited 4D attributes such as sparse trajectories or two-view scene flow, 4RC learns a holistic 4D representation that jointly captures dense scene geometry and motion dynamics. At its core, 4RC introduces a novel encode-once, query-anywhere and anytime paradigm: a
transformer backbone encodes the entire
video into a compact spatio-temporal latent space, from which a conditional decoder can efficiently query 3D geometry and motion for any query frame at any target timestamp. To facilitate learning, we represent per-view 4D attributes in a minimally factorized form by decomposing them into base geometry and time-dependent relative motion. Extensive experiments demonstrate that 4RC outperforms prior and concurrent methods across a wide range of 4D reconstruction tasks.
Read the full abstract ▾
All authors Yihang Luo, Shangchen Zhou, Yushi Lan, Xingang Pan, Chen Change Loy
Computer Vision
Can Image Splicing and Copy-Move Forgery Be Detected by the Same Model? Forensim: An Attention-Based State-Space Approach
Soumyaroop Nandi and Prem Natarajan
We introduce Forensim, an
attention-based state-space framework for image forgery detection that jointly localizes both manipulated (target) and source regions. Unlike traditional approaches that rely solely on artifact cues to detect spliced or forged areas, Forensim is designed to capture duplication patterns crucial for understanding context. In scenarios such as protest imagery, detecting only the forged region, for example a duplicated act of violence inserted into a peaceful crowd, can mislead interpretation, highlighting the need for joint source-target localization. Forensim outputs three-class masks (pristine, source, target) and supports detection of both splicing and copy-move forgeries within a unified architecture. We propose a visual state-space model that leverages normalized
attention maps to identify internal similarities, paired with a region-based block
attention module to distinguish manipulated regions. This design enables end-to-end training and precise localization. Forensim achieves state-of-the-art performance on standard
benchmarks. We also release CMFD-Anything, a new
dataset addressing limitations of existing copy-move forgery
datasets.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Vendi Novelty Scores for Out-of-Distribution Detection
Amey P. Pasarkar and Adji Bousso Dieng
Out-of-distribution (OOD) detection is critical for the safe deployment of machine learning systems. Existing post-hoc detectors typically rely on model confidence scores or likelihood estimates in feature space, often under restrictive distributional assumptions. In this work, we introduce a third paradigm and formulate OOD detection from a diversity perspective. We propose the Vendi Novelty Score (VNS), an OOD detector based on the Vendi Scores (VS), a family of similarity-based diversity metrics. VNS quantifies how much a test sample increases the VS of the in-distribution feature set, providing a principled notion of novelty that does not require density modeling. VNS is linear-time, non-parametric, and naturally combines class-conditional (local) and
dataset-level (global) novelty signals. Across multiple image classification
benchmarks and network architectures, VNS achieves state-of-the-art OOD detection performance. Remarkably, VNS retains this performance when computed using only 1% of the training data, enabling deployment in
memory- or access-constrained settings.
Read the full abstract ▾
Computer Vision
Spatio-Temporal Attention for Consistent Video Semantic Segmentation in Automated Driving
Serin Varghese, Kevin Ross, Fabian Hueger et al. (+1 more)
Deep neural networks, especially
transformer-based architectures, have achieved remarkable success in semantic
segmentation for environmental perception. However, existing models process
video frames independently, thus failing to leverage temporal consistency, which could significantly improve both accuracy and stability in dynamic scenes. In this work, we propose a Spatio-Temporal
Attention (STA) mechanism that extends
transformer attention blocks to incorporate multi-frame context, enabling robust temporal feature representations for
video semantic
segmentation. Our approach modifies standard self-
attention to process spatio-temporal feature sequences while maintaining computational efficiency and requiring minimal changes to existing architectures. STA demonstrates broad applicability across diverse
transformer architectures and remains effective across both lightweight and larger-scale models. A comprehensive evaluation on the Cityscapes and BDD100k
datasets shows substantial improvements of 9.20 percentage points in temporal consistency metrics and up to 1.76 percentage points in mean intersection over union compared to single-frame baselines. These results demonstrate STA as an effective architectural enhancement for
video-based semantic
segmentation applications.
Read the full abstract ▾
All authors Serin Varghese, Kevin Ross, Fabian Hueger, Kira Maag
Computer Vision
Conformal Prediction Sets for Instance Segmentation
Kerri Lu, Dan M. Kluger, Stephen Bates et al. (+1 more)
Current instance
segmentation models achieve high performance on average predictions, but lack principled uncertainty quantification: their outputs are not calibrated, and there is no guarantee that a predicted mask is close to the ground truth. To address this limitation, we introduce a conformal prediction
algorithm to generate adaptive confidence sets for instance
segmentation. Given an image and a pixel coordinate query, our
algorithm generates a confidence set of instance predictions for that pixel, with a provable guarantee for the probability that at least one of the predictions has high Intersection-Over-Union (IoU) with the true object instance mask. We apply our
algorithm to instance
segmentation examples in agricultural field delineation,
cell segmentation, and vehicle detection. Empirically, we find that our prediction sets vary in size based on query difficulty and attain the target coverage, outperforming baselines (naive best parameter and morphological dilation-based methods). We provide versions of the
algorithm with asymptotic and finite sample guarantees. Our work is the first to capture structural uncertainty in instance
segmentation by constructing confidence sets of diverse
segmentation predictions.
Read the full abstract ▾
All authors Kerri Lu, Dan M. Kluger, Stephen Bates, Sherrie Wang
Computer Vision
Cross-Dataset Linkage of Brain MRI using Image Similarity Measures
Gaurang Sharma, Harri Polonen, Juha Pajula et al. (+2 more)
Head
magnetic resonance imaging (MRI) data are routinely collected and shared for research under strict regulatory frameworks that require the removal of direct identifiers prior to data release. However, even after skull stripping,
brain parenchyma may retain participant-specific features that enable linkage of scans acquired from the same individual across
datasets, posing a potential
privacy risk when combined with auxiliary information. Current regulatory approaches typically assess such risks using qualitative notions of reasonableness. Although prior work has suggested that
brain MRI can support subject linkage, existing demonstrations have relied on training-based or computationally intensive methods. Here, we show that reliable linkage of skull-stripped T1-weighted
brain MRI is possible using standard preprocessing pipelines followed by direct image similarity computations. Using this simple approach, we achieve near-perfect matching accuracy across
datasets acquired at different time points, with varying scanner types, spatial resolutions, and acquisition protocols, and even in the presence of cognitive decline. These experiments simulate realistic scenarios of cross-database matching in large-scale neuroimaging repositories. Our findings highlight a previously underappreciated re-identification risk in shared
brain MRI data and provide empirical evidence relevant to the development of informed, forward-looking data-sharing policies in neuroimaging research.
Read the full abstract ▾
All authors Gaurang Sharma, Harri Polonen, Juha Pajula, Jutta Suksi, Jussi Tohka
Computer Vision
Fake-HR1: Rethinking Reasoning of Vision Language Model for Synthetic Image Detection
Changjiang Jiang, Xinkuan Sha, Fengchang Yu et al. (+5 more)
Recent studies have demonstrated that incorporating Chain-of-Thought (CoT)
reasoning into the detection process can enhance a model's ability to detect synthetic images. However, excessively lengthy
reasoning incurs substantial resource overhead, including
token consumption and latency, which is particularly redundant when handling obviously generated forgeries. To address this issue, we propose Fake-HR1, a large-scale hybrid-
reasoning model that, to the best of our knowledge, is the first to adaptively determine whether
reasoning is necessary based on the characteristics of the generative detection task. To achieve this, we design a two-stage training framework: we first perform Hybrid Fine-Tuning (HFT) for cold-start initialization, followed by online reinforcement learning with Hybrid-
Reasoning Grouped Policy Optimization (HGRPO) to implicitly learn when to select an appropriate
reasoning mode. Experimental results show that Fake-HR1 adaptively performs
reasoning across different types of queries, surpassing existing LLMs in both
reasoning ability and generative detection performance, while significantly improving response efficiency.
Read the full abstract ▾
All authors Changjiang Jiang, Xinkuan Sha, Fengchang Yu, Jingjing Liu, Jian Liu, Mingqi Fang, Chenfeng Zhang, Wei Lu
Computer Vision
Perception with Guarantees: Certified Pose Estimation via Reachability Analysis
Tobias Ladner, Yasser Shoukry and Matthias Althoff
Agents in cyber-physical systems are increasingly entrusted with safety-critical tasks. Ensuring safety of these
agents often requires localizing the pose for subsequent actions. Pose estimates can, e.g., be obtained from various combinations of
lidar sensors, cameras, and external services such as GPS. Crucially, in safety-critical domains, a rough estimate is insufficient to formally determine safety, i.e., guaranteeing safety even in the worst-case scenario, and external services might additionally not be trustworthy. We address this problem by presenting a certified pose estimation in 3D solely from a camera image and a well-known target geometry. This is realized by formally bounding the pose, which is computed by leveraging recent results from reachability analysis and formal neural network verification. Our experiments demonstrate that our approach efficiently and accurately localizes
agents in both synthetic and real-world experiments.
Read the full abstract ▾
Computer Vision
Faster-GS: Analyzing and Improving Gaussian Splatting Optimization
Florian Hahlbohm, Linus Franke, Martin Eisemann et al. (+1 more)
Recent advances in 3D Gaussian Splatting (3DGS) have focused on accelerating optimization while preserving reconstruction quality. However, many proposed methods entangle implementation-level improvements with fundamental algorithmic modifications or trade performance for fidelity, leading to a fragmented research landscape that complicates fair comparison. In this work, we consolidate and evaluate the most effective and broadly applicable strategies from prior 3DGS research and augment them with several novel optimizations. We further investigate underexplored aspects of the framework, including numerical stability, Gaussian truncation, and gradient approximation. The resulting system, Faster-GS, provides a rigorously optimized
algorithm that we evaluate across a comprehensive suite of
benchmarks. Our experiments demonstrate that Faster-GS achieves up to 5$\times$ faster training while maintaining visual quality, establishing a new cost-effective and resource efficient baseline for 3DGS optimization. Furthermore, we demonstrate that optimizations can be applied to 4D Gaussian reconstruction, leading to efficient non-rigid scene optimization.
Read the full abstract ▾
All authors Florian Hahlbohm, Linus Franke, Martin Eisemann, Marcus Magnor
Computer Vision
Efficient Special Stain Classification
Oskar Thaeter, Christian Grashei, Anette Haas et al. (+3 more)
Stains are essential in histopathology to visualize specific tissue characteristics, with Haematoxylin and Eosin (H&E) serving as the clinical standard. However, pathologists frequently utilize a variety of special stains for the diagnosis of specific morphologies. Maintaining accurate metadata for these slides is critical for quality control in clinical archives and for the integrity of computational pathology
datasets. In this work, we compare two approaches for automated classification of stains using whole slide images, covering the 14 most commonly used special stains in our institute alongside standard and frozen-section H&E. We evaluate a Multi-Instance Learning (MIL) pipeline and a proposed lightweight thumbnail-based approach. On internal test data, MIL achieved the highest performance (macro F1: 0.941 for 16 classes; 0.969 for 14 merged classes), while the thumbnail approach remained competitive (0.897 and 0.953, respectively). On external TCGA data, the thumbnail model generalized best (weighted F1: 0.843 vs. 0.807 for MIL). The thumbnail approach also increased throughput by two orders of magnitude (5.635 vs. 0.018 slides/s for MIL with all patches). We conclude that thumbnail-based classification provides a scalable and robust solution for routine visual quality control in digital pathology workflows.
Read the full abstract ▾
All authors Oskar Thaeter, Christian Grashei, Anette Haas, Elisa Schmoeckel, Han Li, Peter J. Schüffler
Computer Vision · Cross-listed from cs.LG
Online Monitoring Framework for Automotive Time Series Data using JEPA Embeddings
Alexander Fertig, Karthikeyan Chandra Sekaran, Lakshman Balasubramanian et al. (+1 more)
As autonomous vehicles are rolled out, measures must be taken to ensure their safe operation. In order to supervise a system that is already in operation, monitoring frameworks are frequently employed. These run continuously online in the background, supervising the system status and recording anomalies. This work proposes an online monitoring framework to detect anomalies in object state representations. Thereby, a key challenge is creating a framework for anomaly detection without anomaly labels, which are usually unavailable for unknown anomalies. To address this issue, this work applies a self-supervised
embedding method to translate object data into a latent representation space. For this, a JEPA-based self-supervised prediction task is constructed, allowing training without anomaly labels and the creation of rich object
embeddings. The resulting expressive JEPA
embeddings serve as input for established anomaly detection methods, in order to identify anomalies within object state representations. This framework is particularly useful for applications in real-world environments, where new or unknown anomalies may occur during operation for which there are no labels available. Experiments performed on the publicly available, real-world nuScenes
dataset illustrate the framework's capabilities.
Read the full abstract ▾
All authors Alexander Fertig, Karthikeyan Chandra Sekaran, Lakshman Balasubramanian, Michael Botsch
Computer Vision
Coupled Inference in Diffusion Models for Semantic Decomposition
Calvin Yeung, Ali Zakeri, Zhuowen Zou et al. (+1 more)
Many visual scenes can be described as compositions of latent factors. Effective recognition,
reasoning, and editing often require not only forming such compositional representations, but also solving the decomposition problem. One popular choice for constructing these representations is through the binding operation. Resonator networks, which can be understood as coupled Hopfield networks, were proposed as a way to perform decomposition on such bound representations. Recent works have shown notable similarities between Hopfield networks and
diffusion models. Motivated by these observations, we introduce a framework for semantic decomposition using coupled inference in
diffusion models. Our method frames semantic decomposition as an inverse problem and couples the
diffusion processes using a reconstruction-driven guidance term that encourages the composition of factor estimates to match the bound vector. We also introduce a novel iterative sampling scheme that improves the performance of our model. Finally, we show that
attention-based resonator networks are a special case of our framework. Empirically, we demonstrate that our coupled inference framework outperforms resonator networks across a range of synthetic semantic decomposition tasks.
Read the full abstract ▾
All authors Calvin Yeung, Ali Zakeri, Zhuowen Zou, Mohsen Imani
Computer Vision
Learning to Detect Baked Goods with Limited Supervision
Thomas H. Schmitt, Maximilian Bundscherer and Tobias Bocklet
Monitoring leftover products provides valuable insights that can be used to optimize future production. This is especially important for German bakeries because freshly baked goods have a very short shelf life. Automating this process can reduce labor costs, improve accuracy, and streamline operations. We propose automating this process using an object detection model to identify baked goods from images. However, the large diversity of German baked goods makes fully supervised training prohibitively expensive and limits scalability. Although open-vocabulary detectors (e.g., OWLv2, Grounding DINO) offer lexibility, we demonstrate that they are insufficient for our task. While motivated by bakeries, our work addresses the broader challenges of deploying computer
vision in industries, where tasks are specialized and annotated
datasets are scarce. We compile
dataset splits with varying supervision levels, covering 19 classes of baked goods. We propose two training workflows to train an object detection model with limited supervision. First, we combine OWLv2 and Grounding DINO localization with image-level supervision to train the model in a weakly supervised manner. Second, we improve viewpoint robustness by fine-tuning on
video frames annotated using Segment Anything 2 as a pseudo-label propagation model. Using these workflows, we train YOLOv11 for our detection task due to its favorable speed accuracy tradeoff. Relying solely on image-level supervision, the model achieves a mean Average Precision (mAP) of 0.91. Finetuning with pseudo-labels raises model performance by 19.3% under non-ideal deployment conditions. Combining these workflows trains a model that surpasses our fully-supervised baseline model under non-ideal deployment conditions, despite relying only on image-level supervision.
Read the full abstract ▾
Computer Vision
Bladder Vessel Segmentation using a Hybrid Attention-Convolution Framework
Franziska Krauß, Matthias Ege, Zoltan Lovasz et al. (+4 more)
Urinary bladder cancer surveillance requires tracking tumor sites across repeated interventions, yet the deformable and hollow bladder lacks stable landmarks for orientation. While blood vessels visible during endoscopy offer a patient-specific "vascular fingerprint" for
navigation, automated
segmentation is challenged by imperfect endoscopic data, including sparse labels, artifacts like bubbles or variable lighting, continuous deformation, and mucosal folds that mimic vessels. State-of-the-art vessel
segmentation methods often fail to address these domain-specific complexities. We introduce a Hybrid
Attention-Convolution (HAC) architecture that combines
Transformers to capture global vessel
topology prior with a CNN that learns a residual refinement map to precisely recover thin-vessel details. To prioritize structural connectivity, the
Transformer is trained on optimized ground truth data that exclude short and terminal branches. Furthermore, to address data scarcity, we employ a physics-aware pretraining, that is a self-supervised strategy using clinically grounded augmentations on unlabeled data. Evaluated on the BlaVeS
dataset, consisting of endoscopic
video frames, our approach achieves high accuracy (0.94) and superior precision (0.61) and clDice (0.66) compared to state-of-the-art medical
segmentation models. Crucially, our method successfully suppresses false positives from mucosal folds that dynamically appear and vanish as the bladder fills and empties during surgery. Hence, HAC provides the reliable structural stability required for clinical
navigation.
Read the full abstract ▾
All authors Franziska Krauß, Matthias Ege, Zoltan Lovasz, Albrecht Bartz-Schmidt, Igor Tsaur, Oliver Sawodny, Carina Veil
Computer Vision
VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization
Yikun Liu, Yuan Liu, Shangzhe Di et al. (+8 more)
Multimodal Large
Language Models (MLLMs) have recently achieved remarkable success in visual-
language understanding, demonstrating superior high-level semantic
alignment within their
vision encoders. An important question thus arises: Can these encoders serve as versatile
vision backbones, capable of reliably performing classic
vision-centric tasks as well? To address the question, we make the following contributions: (i) we identify that the
vision encoders within MLLMs exhibit deficiencies in their dense feature representations, as evidenced by their suboptimal performance on dense prediction tasks (e.g., semantic
segmentation, depth estimation); (ii) we propose VersaViT, a well-rounded
vision transformer that instantiates a novel multi-task framework for collaborative post-training. This framework facilitates the optimization of the
vision backbone via lightweight task heads with multi-granularity supervision; (iii) extensive experiments across various downstream tasks demonstrate the effectiveness of our method, yielding a versatile
vision backbone suited for both
language-mediated
reasoning and pixel-level understanding.
Read the full abstract ▾
All authors Yikun Liu, Yuan Liu, Shangzhe Di, Haicheng Wang, Zhongyin Zhao, Le Tian, Xiao Zhou, Jie Zhou, Jiangchao Yao, Yanfeng Wang, Weidi Xie
Computer Vision
Unbalanced optimal transport for robust longitudinal lesion evolution with registration-aware and appearance-guided priors
Melika Qahqaie, Dominik Neumann, Tobias Heimann et al. (+2 more)
Evaluating lesion
evolution in longitudinal CT scans of can cer patients is essential for assessing treatment response, yet establishing reliable lesion correspondence across time remains challenging. Standard bipartite matchers, which rely on
geometric proximity, struggle when lesions appear, disappear, merge, or split. We propose a registration-aware matcher based on unbalanced optimal transport (UOT) that accommodates unequal lesion mass and adapts priors to patient-level tumor-load changes. Our transport cost blends (i) size-normalized geometry, (ii) local registration trust from the deformation-field Jacobian, and (iii) optional patch-level appearance consistency. The resulting transport plan is sparsified by relative
pruning, yielding one-to-one matches as well as new, disappearing, merging, and splitting lesions without retraining or heuristic rules. On longitudinal CT data, our approach achieves consistently higher edge-detection precision and recall, improved lesion-state recall, and superior lesion-graph component F1 scores versus distance-only baselines.
Read the full abstract ▾
All authors Melika Qahqaie, Dominik Neumann, Tobias Heimann, Andreas Maier, Veronika A. Zimmer
Computer Vision
GeoFormer: A Lightweight Swin Transformer for Joint Building Height and Footprint Estimation from Sentinel Imagery
Han Jinzhen, JinByeong Lee, JiSung Kim et al. (+3 more)
Building height (BH) and footprint (BF) are fundamental urban morphological parameters required by
climate modelling, disaster-risk assessment, and population mapping, yet globally consistent data remain scarce. In this work, we develop GeoFormer, a lightweight Swin
Transformer-based multi-task learning framework that jointly estimates BH and BF on a 100 m grid using only open-access Sentinel-1 SAR, Sentinel-2 multispectral, and DEM data. A geo-blocked data-splitting strategy enforces strict spatial independence between training and evaluation regions across 54 morphologically diverse cities. We set representative CNN baselines (ResNet, UNet, SENet) as
benchmarks and thoroughly evaluate GeoFormer's prediction accuracy, computational efficiency, and spatial transferability. Results show that GeoFormer achieves a BH RMSE of 3.19 m with only 0.32 M parameters – outperforming the best CNN baseline (UNet) by 7.5% – indicating that windowed local
attention is more effective than convolution for scene-level building-parameter
retrieval. Systematic ablation on context window size, model capacity, and input modality further reveals that a 5x5 (500 m) receptive field is optimal, DEM is indispensable for height estimation, and multispectral reflectance carries the dominant predictive signal. Cross-continent transfer tests confirm BH RMSE below 3.5 m without region-specific fine-tuning. All code, model weights, and the resulting global product are publicly released.
Read the full abstract ▾
All authors Han Jinzhen, JinByeong Lee, JiSung Kim, MinKyung Cho, DaHee Kim, HongSik Yun
Computer Vision
Monocular Normal Estimation via Shading Sequence Estimation
Zongrui Li, Xinhua Ma, Minghui Hu et al. (+6 more)
Monocular normal estimation aims to estimate the normal map from a single RGB image of an object under arbitrary lights. Existing methods rely on deep models to directly predict normal maps. However, they often suffer from 3D misalignment: while the estimated normal maps may appear to have a correct appearance, the reconstructed surfaces often fail to align with the
geometric details. We argue that this misalignment stems from the current paradigm: the model struggles to distinguish and reconstruct varying geometry represented in normal maps, as the differences in underlying geometry are reflected only through relatively subtle color variations. To address this issue, we propose a new paradigm that reformulates normal estimation as shading sequence estimation, where shading sequences are more sensitive to various
geometric information. Building on this paradigm, we present RoSE, a method that leverages image-to-
video generative models to predict shading sequences. The predicted shading sequences are then converted into normal maps by solving a simple ordinary least-squares problem. To enhance robustness and better handle complex objects, RoSE is trained on a synthetic
dataset, MultiShade, with diverse shapes, materials, and light conditions. Experiments demonstrate that RoSE achieves state-of-the-art performance on real-world
benchmark datasets for object-based
monocular normal estimation.
Read the full abstract ▾
All authors Zongrui Li, Xinhua Ma, Minghui Hu, Yunqing Zhao, Yingchen Yu, Qian Zheng, Chang Liu, Xudong Jiang, Song Bai
Computer Vision
A benchmark for video-based laparoscopic skill analysis and assessment
Isabel Funke, Sebastian Bodenstedt, Felix von Bechtolsheim et al. (+7 more)
Laparoscopic surgery is a complex
surgical technique that requires extensive training. Recent advances in deep learning have shown promise in supporting this training by enabling automatic
video-based assessment of
surgical skills. However, the development and evaluation of deep learning models is currently hindered by the limited size of available annotated
datasets. To address this gap, we introduce the Laparoscopic Skill Analysis and Assessment (LASANA)
dataset, comprising 1270 stereo
video recordings of four basic laparoscopic training tasks. Each recording is annotated with a structured skill rating, aggregated from three independent raters, as well as binary labels indicating the presence or absence of task-specific errors. The majority of recordings originate from a laparoscopic training course, thereby reflecting a natural variation in the skill of participants. To facilitate benchmarking of both existing and novel approaches for
video-based skill assessment and error recognition, we provide predefined data splits for each task. Furthermore, we present baseline results from a deep learning model as a reference point for future comparisons.
Read the full abstract ▾
All authors Isabel Funke, Sebastian Bodenstedt, Felix von Bechtolsheim, Florian Oehme, Michael Maruschke, Stefanie Herrlich, Jürgen Weitz, Marius Distler, Sören Torge Mees, Stefanie Speidel
Computer Vision
SARS: A Novel Face and Body Shape and Appearance Aware 3D Reconstruction System extends Morphable Models
Gulraiz Khan, Kenneth Y. Wertheim, Kevin Pimbblet et al. (+1 more)
Morphable Models (3DMMs) are a type of morphable model that takes 2D images as inputs and recreates the structure and physical appearance of 3D objects, especially human faces and bodies. 3DMM combines identity and expression blendshapes with a basic face mesh to create a detailed 3D model. The variability in the 3D Morphable models can be controlled by tuning diverse parameters. They are high-level image descriptors, such as shape, texture, illumination, and camera parameters. Previous research in 3D human reconstruction concentrated solely on global face structure or geometry, ignoring face semantic features such as age, gender, and facial landmarks characterizing facial boundaries, curves, dips, and wrinkles. In order to accommodate changes in these high-level facial characteristics, this work introduces a shape and appearance-aware 3D reconstruction system (named SARS by us), a c modular pipeline that extracts body and face information from a single image to properly rebuild the 3D model of the human full body.
Read the full abstract ▾
All authors Gulraiz Khan, Kenneth Y. Wertheim, Kevin Pimbblet, Waqas Ahmed
Computer Vision
AdaTSQ: Pushing the Pareto Frontier of Diffusion Transformers via Temporal-Sensitivity Quantization
Shaoqiu Zhang, Zizhong Ding, Kaicheng Yang et al. (+6 more)
Diffusion Transformers (DiTs) have emerged as the state-of-the-art backbone for high-fidelity image and
video generation. However, their massive computational cost and
memory footprint hinder deployment on edge devices. While post-training quantization (PTQ) has proven effective for large
language models (LLMs), directly applying existing methods to DiTs yields suboptimal results due to the neglect of the unique temporal dynamics inherent in
diffusion processes. In this paper, we propose AdaTSQ, a novel PTQ framework that pushes the Pareto frontier of efficiency and quality by exploiting the temporal sensitivity of DiTs. First, we propose a Pareto-aware timestep-dynamic bit-width allocation strategy. We model the quantization policy search as a constrained pathfinding problem. We utilize a beam search
algorithm guided by end-to-end reconstruction error to dynamically assign layer-wise bit-widths across different timesteps. Second, we propose a Fisher-guided temporal calibration mechanism. It leverages temporal Fisher information to prioritize calibration data from highly sensitive timesteps, seamlessly integrating with Hessian-based weight optimization. Extensive experiments on four advanced DiTs (e.g., Flux-Dev, Flux-Schnell, Z-Image, and Wan2.1) demonstrate that AdaTSQ significantly outperforms state-of-the-art methods like SVDQuant and ViDiT-Q. Our code will be released at https://github.com/Qiushao-E/AdaTSQ.
Read the full abstract ▾
All authors Shaoqiu Zhang, Zizhong Ding, Kaicheng Yang, Junyi Wu, Xianglong Yan, Xi Li, Bingnan Duan, Jianping Fang, Yulun Zhang
Computer Vision
MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation
Jiaxu Wang, Yicheng Jiang, Tianlun He et al. (+7 more)
World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based
forecasting or
reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. This work proposes a novel
embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model imagines the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce
geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Experiments on three
datasets demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices.
Read the full abstract ▾
All authors Jiaxu Wang, Yicheng Jiang, Tianlun He, Jingkai Sun, Qiang Zhang, Jiahang Cao, Zesen Gan, Mingyuan Sun, Qiming Shao, Xiangyu Yue
Computer Vision
BabyMamba-HAR: Lightweight Selective State Space Models for Efficient Human Activity Recognition on Resource Constrained Devices
Mridankan Mandal
Human activity recognition (HAR) on resource constrained devices requires high accuracy across diverse sensor setups. Selective state space models (SSMs) offer efficient linear time sequence processing, presenting a compelling alternative to
attention mechanisms. However, their TinyML design space remains unexplored. This paper introduces BabyMamba-HAR, comprising two lightweight architectures: (1) CI-BabyMamba-HAR, utilizing a channel independent stem for noise robustness, and (2) Crossover-BiDir-BabyMamba-HAR, utilizing an early fusion stem for channel count independent complexity. Both integrate weight tied bidirectional scanning and gated temporal
attention pooling. Across eight
benchmarks, Crossover-BiDir-BabyMamba-HAR averages an 86.52% F1-score with 27K parameters and 2.21M MACs, matching TinyHAR (86.16%) while requiring 11x fewer MACs on high channel
datasets. On-device deployment on the Raspberry Pi Pico 2 and ESP32 utilized a mixed precision C++ runtime (INT8 projections, float32 states). A fused computation strategy with lifetime aware
memory management reduces peak
memory footprint from O(B*dmodel*L*dstate) to O(B*dmodel*dstate), adapting to support weight-tied bidirectional and channel-streaming execution. Both architectures achieved full 8/8
dataset coverage with >99.2% PyTorch parity, whereas INT8 quantized TFLite baselines showed degraded coverage and parity (TinyHAR: 7/8 and 4/8 coverage at 60.4% and 88.6% parity, TinierHAR: 8/8 and 6/8 at 54.2% and 90.8%, DeepConvLSTM: 1/8 and 0/8 on Pico 2 and ESP32, respectively). Crossover-BiDir-BabyMamba-HAR averages 154.4 ms latency on ESP32 and 481.9 ms on Pico 2. Ablations confirm bidirectional scanning and gated
attention improve F1-scores by up to 8.42% and 8.94%, respectively, establishing practical principles for TinyML SSM deployment.
Read the full abstract ▾
Computer Vision
Free-GVC: Towards Training-Free Extreme Generative Video Compression with Temporal Coherence
Xiaoyue Ling, Chuqin Zhou, Chunyi Li et al. (+4 more)
Building on recent advances in
video generation, generative
video compression has emerged as a new paradigm for achieving visually pleasing reconstructions. However, existing methods exhibit limited exploitation of temporal correlations, causing noticeable flicker and degraded temporal coherence at ultra-low bitrates. In this paper, we propose Free-GVC, a training-free generative
video compression framework that reformulates
video coding as latent trajectory compression guided by a
video diffusion prior. Our method operates at the group-of-pictures (GOP) level, encoding
video segments into a compact latent space and progressively compressing them along the
diffusion trajectory. To ensure perceptually consistent reconstruction across GOPs, we introduce an Adaptive Quality Control module that dynamically constructs an online rate-perception surrogate model to predict the optimal
diffusion step for each GOP. In addition, an Inter-GOP
Alignment module establishes frame overlap and performs latent fusion between adjacent groups, thereby mitigating flicker and enhancing temporal coherence. Experiments show that Free-GVC achieves an average of 93.29% BD-Rate reduction in DISTS over the latest neural codec DCVC-RT, and a user study further confirms its superior perceptual quality and temporal coherence at ultra-low bitrates.
Read the full abstract ▾
All authors Xiaoyue Ling, Chuqin Zhou, Chunyi Li, Yunuo Chen, Yuan Tian, Guo Lu, Wenjun Zhang
Computer Vision
Code2World: A GUI World Model via Renderable Code Generation
Yuhao Zheng, Li'an Zhong, Yi Wang et al. (+6 more)
Autonomous GUI
agents interact with environments by perceiving interfaces and executing actions. As a virtual sandbox, the GUI World model empowers
agents with human-like foresight by enabling action-conditioned prediction. However, existing text- and pixel-based approaches struggle to simultaneously achieve high visual fidelity and fine-grained structural controllability. To this end, we propose Code2World, a
vision-
language coder that simulates the next visual state via renderable code generation. Specifically, to address the data scarcity problem, we construct AndroidCode by translating GUI trajectories into high-fidelity HTML and refining synthesized code through a visual-feedback revision mechanism, yielding a corpus of over 80K high-quality screen-action pairs. To adapt existing VLMs into code prediction, we first perform SFT as a cold start for format layout following, then further apply Render-Aware Reinforcement Learning which uses rendered outcome as the reward signal by enforcing visual semantic fidelity and action consistency. Extensive experiments demonstrate that Code2World-8B achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image. Notably, Code2World significantly enhances downstream
navigation success rates in a flexible manner, boosting Gemini-2.5-Flash by +9.5% on AndroidWorld
navigation. The code is available at https://github.com/AMAP-ML/Code2World.
Read the full abstract ▾
All authors Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin
Computer Vision
Towards Explainable Industrial Anomaly Detection via Knowledge-Guided Latent Reasoning
Peng Chen, Wenqiang Wang, Yunkang Cao et al. (+7 more)
Industrial anomaly detection demands precise
reasoning over fine-grained defect patterns. However, existing
multimodal large
language models (MLLMs), pretrained on general-domain data, often struggle to capture category-specific anomalies, thereby limiting both detection accuracy and interpretability. To address these limitations, we propose Reason-IAD, a knowledge-guided dynamic latent
reasoning framework for explainable industrial anomaly detection. Reason-IAD comprises two core components. First, a
retrieval-augmented knowledge module incorporates category-specific textual descriptions into the model input, enabling context-aware
reasoning over domain-specific defects. Second, an
entropy-driven latent
reasoning mechanism conducts iterative exploration within a compact latent space using optimizable latent think
tokens, guided by an
entropy-based reward that encourages confident and stable predictions. Furthermore, a dynamic visual injection strategy selectively incorporates the most informative image patches into the latent sequence, directing the
reasoning process toward regions critical for anomaly detection. Extensive experimental results demonstrate that Reason-IAD consistently outperforms state-of-the-art methods across multiple tasks. The code will be publicly available at https://github.com/chenpeng052/Reason-IAD.
Read the full abstract ▾
All authors Peng Chen, Wenqiang Wang, Yunkang Cao, Chengliang Liu, Wei Wang, Mingbo Yang, Li Shen, Wenqi Ren, Xiaochun Cao, Chao Huang
Computer Vision
Kelix Technical Report
Boyang Ding, Chenglong Chu, Dunju Zang et al. (+28 more)
Autoregressive large
language models (LLMs) scale well by expressing diverse tasks as sequences of discrete natural-
language tokens and training with next-
token prediction, which unifies comprehension and generation under self-supervision. Extending this paradigm to
multimodal data requires a shared, discrete representation across modalities. However, most
vision-
language models (VLMs) still rely on a hybrid interface: discrete text
tokens paired with continuous
Vision Transformer (ViT) features. Because supervision is largely text-driven, these models are often biased toward understanding and cannot fully leverage large-scale self-supervised learning on non-text data. Recent work has explored discrete visual tokenization to enable fully autoregressive
multimodal modeling, showing promising progress toward unified understanding and generation. Yet existing discrete
vision tokens frequently lose information due to limited code capacity, resulting in noticeably weaker understanding than continuous-feature VLMs. We present Kelix, a fully discrete autoregressive unified model that closes the understanding gap between discrete and continuous visual representations.
Read the full abstract ▾
All authors Boyang Ding, Chenglong Chu, Dunju Zang, Han Li, Jiangxia Cao, Kun Gai, Muhao Wei, Ruiming Tang, Shiyao Wang, Siyang Mao, Xinchen Luo, Yahui Liu, Zhixin Ling, Zhuoran Yang, Ziming Li, Chengru Song, Guorui Zhou, Guowang Zhang, Hao Peng, Hao Wang, Jiaxin Deng, Jin Ouyang, Jinghao Zhang, Lejian Ren, Qianqian Wang, Qigen Hu, Tao Wang, Xingmei Wang, Yiping Yang, Zixing Zhang, Ziqi Wang
Computer Vision
Yijie Lin, Guofeng Ding, Haochen Zhou et al. (+3 more)
Existing
multimodal retrieval benchmarks largely emphasize semantic matching on daily-life images and offer limited diagnostics of professional knowledge and complex
reasoning. To address this gap, we introduce ARK, a
benchmark designed to analyze
multimodal retrieval from two complementary perspectives: (i) knowledge domains (five domains with 17 subtypes), which characterize the content and expertise
retrieval relies on, and (ii)
reasoning skills (six categories), which characterize the type of inference over
multimodal evidence required to identify the correct candidate. Specifically, ARK evaluates
retrieval with both unimodal and
multimodal queries and candidates, covering 16 heterogeneous visual data types. To avoid shortcut matching during evaluation, most queries are paired with targeted hard negatives that require multi-step
reasoning. We evaluate 25 representative text-based and
multimodal retrievers and observe a pronounced gap between knowledge- and
reasoning-intensive
retrieval, with fine-grained visual and spatial
reasoning as persistent bottlenecks. We further show that enhancements such as re-ranking, rewriting, and agentic
retrieval yield consistent gains, but substantial headroom remains.
Read the full abstract ▾
All authors Yijie Lin, Guofeng Ding, Haochen Zhou, Haobin Li, Mouxing Yang, Xi Peng