Computer Vision
Make Geometry Matter for Spatial Reasoning
Shihua Zhang, Qiuhong Shen, Shizun Wang et al. (+2 more)
Empowered by large-scale training,
vision-
language models (VLMs) achieve strong image and
video understanding, yet their ability to perform spatial
reasoning in both static scenes and dynamic
videos remains limited. Recent advances try to handle this limitation by injecting geometry
tokens from pretrained 3D foundation models into VLMs. Nevertheless, we observe that naive
token fusion followed by standard fine-tuning in this line of work often leaves such
geometric cues underutilized for spatial
reasoning, as VLMs tend to rely heavily on 2D visual cues. In this paper, we propose GeoSR, a framework designed to make geometry matter by encouraging VLMs to actively reason with geometry
tokens. GeoSR introduces two key components: (1) Geometry-Unleashing Masking, which strategically masks portions of 2D
vision tokens during training to weaken non-
geometric shortcuts and force the model to consult geometry
tokens for spatial
reasoning; and (2) Geometry-Guided Fusion, a gated routing mechanism that adaptively amplifies geometry
token contributions in regions where
geometric evidence is critical. Together, these designs unleash the potential of geometry
tokens for spatial
reasoning tasks. Extensive experiments on both static and dynamic spatial
reasoning benchmarks demonstrate that GeoSR consistently outperforms prior methods and establishes new state-of-the-art performance by effectively leveraging
geometric information. The project page is available at https://suhzhang.github.io/GeoSR/.
Read the full abstract ▾
All authors Shihua Zhang, Qiuhong Shen, Shizun Wang, Tianbo Pan, Xinchao Wang
Computer Vision
RigidDriveGS: Object-Centric Vehicle Reconstruction from Fixed Multi-Camera Drive-Through Captures
Nitin Kulkarni, Akhil Devarashetti, Charlie Cluss et al. (+4 more)
High-fidelity 3D reconstruction of vehicle exteriors improves buyer confidence in online automotive marketplaces, but fixed multi-camera drive-through capture departs from standard Structure-from-Motion (SfM): static cameras observe a moving vehicle amid clutter, rotating wheels, fisheye distortion, and specular reflections. We present RigidDriveGS, an object-centric pipeline combining motion-gated tracking, bifurcated masks that exclude wheels for SfM while optionally retaining them for rendering, mask-guided dense correspondences on raw distorted images, rig-constrained global SfM, and clean object-centric neural rendering with trimap-aware opacity or density supervision and native-alpha white compositing. We evaluate 25 vehicles from 10 dealership installations and six phone-recorded vehicles, each captured from 14 independent handheld positions. Under a staggered per-camera temporal-block holdout, 3DGUT+MCMC achieves 30.42 dB PSNR, 0.90 SSIM, and 0.20 LPIPS, improving PSNR by 7.78 dB over undistorted 3D-GS; independently recorded reference
LiDAR scans yield full/body Chamfer-L1 of 0.026/0.013 m and F1@5 cm of 0.703/0.885. These results support clutter-free, object-centric novel-view synthesis without controlled studio infrastructure.
Read the full abstract ▾
All authors Nitin Kulkarni, Akhil Devarashetti, Charlie Cluss, Livio Forte, Philip Schneider, Chunming Qiao, Alina Vereshchaka
Computer Vision
Think over Trajectories: Leveraging Video Generation to Reconstruct GPS Trajectories from Cellular Signaling
Ruixing Zhang, Hanzhang Jiang, Leilei Sun et al. (+3 more)
Mobile devices continuously interact with cellular base stations, generating massive volumes of signaling records that provide broad coverage for understanding human mobility. However, such records offer only coarse location cues (e.g., serving-
cell identifiers) and therefore limit their direct use in applications that require high-precision GPS trajectories. This paper studies the Sig2GPS problem: reconstructing GPS trajectories from cellular signaling. Inspired by domain experts often lay the signaling trace on the map and sketch the corresponding GPS route, unlike conventional solutions that rely on complex multi-stage engineering pipelines or regress coordinates, Sig2GPS is reframed as an image-to-
video generation task that directly operates in the map-visual domain: signaling traces are rendered on a map, and a
video generation model is trained to draw a continuous GPS path. To support this paradigm, a paired signaling-to-trajectory
video dataset is constructed to fine-tune an open-source
video model, and a trajectory-aware reinforcement learning-based optimization method is introduced to improve generation fidelity via rewards. Experiments on large-scale real-world
datasets show substantial improvements over strong engineered and learning-based baselines, while additional results on next GPS prediction indicate scalability and cross-city transferability. Overall, these results suggest that map-visual
video generation provides a practical interface for trajectory data mining by enabling direct generation and refinement of continuous paths under map constraints.
Read the full abstract ▾
All authors Ruixing Zhang, Hanzhang Jiang, Leilei Sun, Liangzhe Han, Jibin Wang, Weifeng Lv
Computer Vision
VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward
Zhaochong An, Orest Kupyn, Théo Uscidda et al. (+5 more)
Large-scale
video diffusion models achieve impressive visual quality, yet often fail to preserve
geometric consistency. Prior approaches improve consistency either by augmenting the generator with additional modules or applying geometry-aware
alignment. However, architectural modifications can compromise the generalization of internet-scale pretrained models, while existing
alignment methods are limited to static scenes and rely on RGB-space rewards that require repeated VAE decoding, incurring substantial compute overhead and failing to generalize to highly dynamic real-world scenes. To preserve the pretrained capacity while improving
geometric consistency, we propose VGGRPO (Visual Geometry GRPO), a latent geometry-guided framework for geometry-aware
video post-training. VGGRPO introduces a Latent Geometry Model (LGM) that stitches
video diffusion latents to geometry foundation models, enabling direct decoding of scene geometry from the latent space. By constructing LGM from a geometry model with 4D reconstruction capability, VGGRPO naturally extends to dynamic scenes, overcoming the static-scene limitations of prior methods. Building on this, we perform latent-space Group Relative Policy Optimization with two complementary rewards: a camera motion smoothness reward that penalizes jittery trajectories, and a geometry reprojection consistency reward that enforces cross-view
geometric coherence. Experiments on both static and dynamic
benchmarks show that VGGRPO improves camera stability, geometry consistency, and overall quality while eliminating costly VAE decoding, making latent-space geometry-guided reinforcement an efficient and flexible approach to world-consistent
video generation.
Read the full abstract ▾
All authors Zhaochong An, Orest Kupyn, Théo Uscidda, Andrea Colaco, Karan Ahuja, Serge Belongie, Mar Gonzalez-Franco, Marta Tintore Gazulla
Computer Vision
From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning
Yang Liu, Qianqian Xu, Peisong Wen et al. (+3 more)
Recent studies have made notable progress in
video representation learning by transferring image-pretrained models to
video tasks, typically with complex temporal modules and
video fine-tuning. However, fine-tuning heavy modules may compromise inter-
video semantic separability, i.e., the essential ability to distinguish objects across
videos. While reducing the tunable parameters hinders their intra-
video temporal consistency, which is required for stable representations of the same object within a
video. This dilemma indicates a potential trade-off between the intra-
video temporal consistency and inter-
video semantic separability during image-to-
video transfer. To this end, we propose the Consistency-Separability Trade-off Transfer Learning (Co-Settle) framework, which applies a lightweight projection layer on top of the frozen image-pretrained encoder to adjust representation space with a temporal cycle consistency objective and a semantic separability constraint. We further provide a theoretical support showing that the optimized projection yields a better trade-off between the two properties under appropriate conditions. Experiments on eight image-pretrained models demonstrate consistent improvements across multiple levels of
video tasks with only five epochs of self-supervised training. The code is available at https://github.com/yafeng19/Co-Settle.
Read the full abstract ▾
All authors Yang Liu, Qianqian Xu, Peisong Wen, Siran Dai, Xilin Zhao, Qingming Huang
Computer Vision
The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
Gillian Rosenberg, Skylar Stadhard, Bruce C. Hansen et al. (+1 more)
What information is sufficient to learn the full richness of human scene understanding? The distributional hypothesis holds that the statistical co-occurrence of
language and images captures the conceptual knowledge underlying visual cognition.
Vision-
language models (VLMs) are trained on massive paired text-image corpora but lack
embodied experience, making them an ideal test of the distributional hypothesis. We report two experiments comparing descriptions generated by 18 VLMs to those of over 2000 human observers across 15 high-level scene understanding tasks, spanning general knowledge, affordances, sensory experiences, affective responses, and future prediction. Because many tasks lack ground truth answers, we developed a Human-Calibrated Cosine Distance (HCD) metric that measures VLM output similarity to the distribution of human responses, scaled by within-human variability. In Experiment 1, VLMs approached human-level performance on general knowledge tasks, but showed a robust deficit for affordance tasks that resisted
prompt engineering and did not improve with newer model releases. In Experiment 2, we tested six mechanistic hypotheses for explaining this affordance gap, finding that the deficit was structural rather than stylistic and was not resolved by providing explicit spatial information. Corpus analyses revealed that image captioning
datasets contain sparse
agent-addressed affordance
language, consistent with Gricean accounts of why
embodied knowledge may be systematically underrepresented in
language. Together, these findings suggest that distributional learning from images and text is insufficient for affordance-based scene understanding, implying that some dimensions of human visual cognition may require the kind of
agent-centered, three-dimensional experience that no photograph or caption can encode.
Read the full abstract ▾
All authors Gillian Rosenberg, Skylar Stadhard, Bruce C. Hansen, Michelle R. Greene
Computer Vision
From Synthetic Data to Real Restorations: Diffusion Model for Patient-specific Dental Crown Completion
Dávid Pukanec, Tibor Kubík and Michal Španěl
We present ToothCraft, a
diffusion-based model for the contextual generation of tooth crowns, trained on artificially created incomplete teeth. Building upon recent advancements in conditioned
diffusion models for 3D shapes, we developed a model capable of an automated tooth crown completion conditioned on local anatomical context. To address the lack of training data for this task, we designed an augmentation pipeline that generates incomplete tooth geometries from a publicly available
dataset of complete dental arches (3DS, ODD). By synthesising a diverse set of training examples, our approach enables robust learning across a wide spectrum of tooth defects. Experimental results demonstrate the strong capability of our model to reconstruct complete tooth crowns, achieving an intersection over union (IoU) of 81.8% and a Chamfer Distance (CD) of 0.00034 on synthetically damaged testing restorations. Our experiments demonstrate that the model can be applied directly to real-world cases, effectively filling in incomplete teeth, while generated crowns show minimal intersection with the opposing dentition, thus reducing the risk of occlusal interference. Access to the code, model weights, and
dataset information will be available at: https://github.com/ikarus1211/VISAPP_ToothCraft
Read the full abstract ▾
Computer Vision
MA-Bench: Towards Fine-grained Micro-Action Understanding
Kun Li, Jihao Gu, Fei Wang et al. (+3 more)
With the rapid development of
Multimodal Large
Language Models (MLLMs), their potential in Micro-Action understanding, a vital role in human
emotion analysis, remains unexplored due to the absence of specialized
benchmarks. To tackle this issue, we present MA-Bench, a
benchmark comprising 1,000
videos and a three-tier evaluation architecture that progressively examines micro-action perception, relational comprehension, and interpretive
reasoning. MA-Bench contains 12,000 structured question-answer pairs, enabling systematic assessment of both recognition accuracy and action interpretation. The results of 23 representative MLLMs reveal that there are significant challenges in capturing motion granularity and fine-grained body-part dynamics. To address these challenges, we further construct MA-Bench-Train, a large-scale training corpus with 20.5K
videos annotated with structured micro-action captions for fine-tuning MLLMs. The results of Qwen3-VL-8B fine-tuned on MA-Bench-Train show clear performance improvements across micro-action
reasoning and explanation tasks. Our work aims to establish a foundation
benchmark for advancing MLLMs in understanding subtle micro-action and human-related behaviors. Project Page: https://MA-Bench.github.io
Read the full abstract ▾
All authors Kun Li, Jihao Gu, Fei Wang, Zhiliang Wu, Hehe Fan, Dan Guo
Computer Vision
Scene Grounding In the Wild
Tamir Cohen, Leo Segre, Shay Shomer-Chai et al. (+2 more)
Reconstructing accurate 3D models of large-scale real-world scenes from unstructured, in-the-wild imagery remains a core challenge in computer
vision, especially when the input views have little or no overlap. In such cases, existing reconstruction pipelines often produce multiple disconnected partial reconstructions or erroneously merge non-overlapping regions into overlapping geometry. In this work, we propose a framework that grounds each partial reconstruction to a complete reference model of the scene, enabling globally consistent
alignment even in the absence of visual overlap. We obtain reference models from dense, geospatially accurate pseudo-synthetic renderings derived from Google Earth Studio. These renderings provide full scene coverage but differ substantially in appearance from real-world photographs. Our key insight is that, despite this significant domain gap, both domains share the same underlying scene semantics. We represent the reference model using 3D Gaussian Splatting, augmenting each Gaussian with semantic features, and formulate
alignment as an inverse feature-based optimization scheme that estimates a global 6DoF pose and scale while keeping the reference model fixed. Furthermore, we introduce the WikiEarth
dataset, which registers existing partial 3D reconstructions with pseudo-synthetic reference models. We demonstrate that our approach consistently improves global
alignment when initialized with various classical and learning-based pipelines, while mitigating failure modes of state-of-the-art end-to-end models.
Read the full abstract ▾
All authors Tamir Cohen, Leo Segre, Shay Shomer-Chai, Shai Avidan, Hadar Averbuch-Elor
Computer Vision
GVCC: Zero-Shot Video Compression via Codebook-Driven Stochastic Rectified Flow
Ziyue Zeng, Xun Su, Haoyuan Liu et al. (+3 more)
At ultra-low bitrates, high-fidelity reconstruction requires sampling plausible
videos from the posterior rather than regressing to oversmoothed conditional means. We propose Generative
Video Codebook Codec (GVCC), a zero-shot framework in which a pretrained
video generative model serves directly as the decoder, and the transmitted bitstream specifies its generation trajectory. Modern rectified-flow
video models are typically sampled with deterministic ODE solvers, which leave no per-step
stochastic channel for transmitting compressed information. GVCC addresses this by converting the deterministic flow sampler into an equivalent marginal-preserving
stochastic process, so that information can be transmitted by encoding the per-step
stochastic innovations. Unlike images,
videos introduce longer temporal dependencies and more diverse conditioning modes. We instantiate GVCC in three practical modes: Text-to-
Video (T2V) without a reference frame, autoregressive Image-to-
Video (I2V) with tail latent correction, and First-Last-Frame-to-
Video (FLF2V) with boundary-sharing Group of Pictures (GOP) chaining. On the seven-sequence UVG
dataset, local atom-count sweeps characterize the rate–quality behavior of all three variants. We report full-
dataset perceptual and fidelity metrics together with temporal diagnostics, without inferring matched-rate or global RD improvements from these limited local sweeps.
Read the full abstract ▾
All authors Ziyue Zeng, Xun Su, Haoyuan Liu, Bingyu Lu, Yui Tatsumi, Hiroshi Watanabe
Computer Vision
SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching
Lanmiao Liu, Esam Ghaleb, Asl{\i} Özyürek et al. (+1 more)
While the field of co-
speech gesture generation has seen significant advances, producing holistic, semantically grounded gestures remains a challenge. Existing approaches rely on external semantic
retrieval methods, which limit their generalisation capability due to dependency on predefined linguistic rules. Flow-matching-based methods produce promising results; however, the network is optimised using only semantically congruent samples without exposure to negative examples, leading to learning rhythmic gestures rather than sparse motion, such as iconic and metaphoric gestures. Furthermore, by modelling body parts in isolation, the majority of methods fail to maintain crossmodal consistency. We introduce a Contrastive Flow Matching-based co-
speech gesture generation model that uses mismatched audio-text conditions as negatives, training the velocity field to follow the correct motion trajectory while repelling semantically incongruent trajectories. Our model ensures cross-modal coherence by
embedding text, audio, and holistic motion into a composite latent space via cosine and contrastive objectives. Extensive experiments and a user study demonstrate that our proposed approach outperforms state-of-the-art methods on two
datasets, BEAT2 and SHOW.
Read the full abstract ▾
All authors Lanmiao Liu, Esam Ghaleb, Asl{\i} Özyürek, Zerrin Yumak
Computer Vision
Beyond MACs: Hardware Efficient Architecture Design for Vision Backbones
Moritz Nottebaum, Matteo Dunnhofer and Christian Micheloni
Vision backbone networks play a central role in modern computer
vision. Enhancing their efficiency directly benefits a wide range of downstream applications. To measure efficiency, many publications rely on MACs (Multiply Accumulate operations) as a predictor of execution time. In this paper, we experimentally demonstrate the shortcomings of such a metric, especially in the context of edge devices. By contrasting the MAC count and execution time of common architectural design elements, we identify key factors for efficient execution and provide insights to optimize backbone design. Based on these insights, we present LowFormer, a novel
vision backbone family. LowFormer features a streamlined macro and micro design that includes Lowtention, a lightweight alternative to Multi-Head Self-
Attention. Lowtention not only proves more efficient, but also enables superior results on ImageNet. Additionally, we present an edge GPU version of LowFormer, that can further improve upon its baseline's speed on edge GPU and desktop GPU. We demonstrate LowFormer's wide applicability by evaluating it on smaller image classification
datasets, as well as adapting it to several downstream tasks, such as object detection, semantic
segmentation, image
retrieval, and visual object tracking. LowFormer models consistently achieve remarkable speed-ups across various hardware platforms compared to recent state-of-the-art backbones. Code and models are available at https://github.com/altair199797/LowFormer/blob/main/Beyond_MACs.md.
Read the full abstract ▾
Computer Vision
AutoWeather4D: Autonomous Driving Video Weather Conversion via G-Buffer Dual-Pass Editing
Tianyu Liu, Weitao Xiong, Kunming Luo et al. (+4 more)
Generative
video models have significantly advanced the photorealistic synthesis of adverse weather for autonomous driving; however, they consistently demand massive
datasets to learn rare weather scenarios. While 3D-aware editing methods alleviate these data constraints by augmenting existing
video footage, they are fundamentally bottlenecked by costly per-scene optimization and suffer from inherent
geometric and illumination
entanglement. In this work, we introduce AutoWeather4D, a feed-forward 3D-aware weather editing framework designed to explicitly decouple geometry and illumination. At the core of our approach is a G-buffer Dual-pass Editing mechanism. The Geometry Pass leverages explicit structural foundations to enable surface-anchored physical interactions, while the Light Pass analytically resolves light transport, accumulating the contributions of local illuminants into the global illumination to enable dynamic 3D local relighting. Extensive experiments demonstrate that AutoWeather4D achieves comparable photorealism and structural consistency to generative baselines while enabling fine-grained parametric physical control, serving as a practical data engine for autonomous driving.
Read the full abstract ▾
All authors Tianyu Liu, Weitao Xiong, Kunming Luo, Manyuan Zhang, Peng Li, Yuan Liu, Ping Tan
Computer Vision
OVI-MAP:Open-Vocabulary Instance-Semantic Mapping
Zilong Deng, Federico Tombari, Marc Pollefeys et al. (+2 more)
Incremental open-vocabulary 3D instance-semantic mapping is essential for autonomous
agents operating in complex everyday environments. However, it remains challenging due to the need for robust instance
segmentation, real-time processing, and flexible open-set
reasoning. Existing methods often rely on the closed-set assumption or dense per-pixel
language fusion, which limits scalability and temporal consistency. We introduce OVI-MAP that decouples instance reconstruction from semantic inference. We propose to build a class-agnostic 3D instance map that is incrementally constructed from RGB-D input, while semantic features are extracted only from a small set of automatically selected views using
vision-
language models. This design enables stable instance tracking and zero-shot semantic labeling throughout online exploration. Our system operates in real time and outperforms state-of-the-art open-vocabulary mapping baselines on standard
benchmarks.
Read the full abstract ▾
All authors Zilong Deng, Federico Tombari, Marc Pollefeys, Johanna Wald, Daniel Barath
Computer Vision
Learnable Quantum Efficiency Filters for Urban Hyperspectral Segmentation
Imad Ali Shah, Jiarong Li, Ethan Delaney et al. (+4 more)
Hyperspectral sensing provides rich spectral information for scene understanding in urban driving, but its high dimensionality poses challenges for interpretation and efficient learning. We introduce Learnable
Quantum Efficiency (LQE), a physics-inspired, interpretable dimensionality reduction (DR) method that parameterizes smooth high-order spectral response functions that emulate plausible sensor
quantum efficiency curves. Unlike conventional methods or unconstrained learnable layers, LQE enforces physically motivated constraints, including a single dominant peak, smooth responses, and bounded bandwidth. This formulation yields a compact spectral representation that preserves discriminative information while remaining fully differentiable and end-to-end trainable within semantic
segmentation models (SSMs). We conduct systematic evaluations across three publicly available multi-class hyperspectral urban driving
datasets, comparing LQE against six conventional and seven learnable baseline DR methods across six SSMs. Averaged across all SSMs and configurations, LQE achieves the highest average mIoU, improving over conventional methods by 2.45%, 0.45%, and 1.04%, and over learnable methods by 1.18%, 1.56%, and 0.81% on HyKo, HSI-Drive, and Hyperspectral City, respectively. LQE maintains strong parameter efficiency (12–36 parameters compared to 51–22K for competing learnable approaches) and competitive inference latency. Ablation studies show that low-order configurations are optimal, while the learned spectral filters converge to
dataset-intrinsic wavelength patterns. These results demonstrate that physics-informed spectral learning can improve both performance and interpretability, providing a principled bridge between hyperspectral perception and data-driven multispectral sensor design for automotive
vision systems.
Read the full abstract ▾
All authors Imad Ali Shah, Jiarong Li, Ethan Delaney, Enda Ward, Martin Glavin, Edward Jones, Brian Deegan
Computer Vision
Conditional Diffusion for 3D CT Volume Reconstruction from 2D X-rays
Martin Rath, Morteza Ghahremani, Yitong Li et al. (+3 more)
Computed
tomography (CT) provides rich 3D anatomical detail but is often constrained by high radiation exposure, substantial costs, and limited availability. Standard chest X-rays are cost-effective and widely accessible, but provide only 2D projections with limited pathological information. Reconstructing 3D CT volumes from 2D X-rays could markedly increase diagnostic accessibility, yet existing methods rely predominantly on synthetic X-ray projections, limiting clinical generalization. We propose AXON, a multi-stage
diffusion-based framework that reconstructs 3D CT volumes directly from real X-rays with substantially improved fidelity over existing approaches. AXON follows a coarse-to-fine paradigm: a Brownian Bridge
diffusion model first captures global anatomical structure, and a ControlNet-guided refinement stage then enhances local intensity detail and anatomical realism. To alleviate the depth ambiguity inherent in 2D-to-3D reconstruction, AXON incorporates bi-planar X-ray views, enabling more accurate spatial
reasoning and structural recovery. A dedicated super-resolution module further increases the spatial resolution of the generated volumes. Experiments on public and external
datasets show that AXON consistently surpasses state-of-the-art approaches while generalizing across diverse clinical distributions. At our highest-resolution bi-planar setting, AXON achieves an 11.9% improvement in PSNR and an 11.0% increase in SSIM over the strongest baseline evaluated at that resolution. In the $128^3$ single-planar setting, it maintains a lead of 7.8% in PSNR on LIDC-IDRI, with larger margins on the external clinical
dataset of 8.0% in PSNR and 16.9% in SSIM. Our code is available at https://github.com/ai-med/AXON/.
Read the full abstract ▾
All authors Martin Rath, Morteza Ghahremani, Yitong Li, Ashkan Taghipour, Marcus Makowski, Christian Wachinger
Computer Vision
ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better
Mriganka Nath, Anurag Das, Jiahao Xie et al. (+1 more)
Large
vision-
language models (LVLMs) tend to hallucinate, especially when visual inputs are corrupted at test time. We show that such corruptions act as additional distribution shifts, significantly amplifying
hallucination rates in real-world applications. To address this, we propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample. Specifically, we leverage the image-text
alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs. Extensive experiments on standard
hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates
hallucinations and improves descriptive faithfulness under visual corruptions.
Read the full abstract ▾
All authors Mriganka Nath, Anurag Das, Jiahao Xie, Bernt Schiele
Computer Vision
SparseCam4D: Spatio-Temporally Consistent 4D Reconstruction from Sparse Cameras
Weihong Pan, Xiaoyu Zhang, Zhuang Zhang et al. (+4 more)
High-quality 4D reconstruction enables photorealistic and immersive rendering of the dynamic real world. However, unlike static scenes that can be fully captured with a single camera, high-quality dynamic scenes typically require dense arrays of tens or even hundreds of synchronized cameras. Dependence on such costly lab setups severely limits practical scalability. To this end, we propose a sparse-camera dynamic reconstruction framework that exploits abundant yet inconsistent generative observations. Our key innovation is the Spatio-Temporal Distortion Field, which provides a unified mechanism for modeling inconsistencies in generative observations across both spatial and temporal dimensions. Building on this, we develop a complete pipeline that enables 4D reconstruction from sparse and uncalibrated camera inputs. We evaluate our method on multi-camera dynamic scene
benchmarks, achieving spatio-temporally consistent high-fidelity renderings and significantly outperforming existing approaches. Project page available at https://inspatio.github.io/sparse-cam4d/
Read the full abstract ▾
All authors Weihong Pan, Xiaoyu Zhang, Zhuang Zhang, Zhichao Ye, Nan Wang, Haomin Liu, Guofeng Zhang
Computer Vision
HyVIC: A Metric-Driven Spatio-Spectral Hyperspectral Image Compression Architecture Based on Variational Autoencoders
Martin Hermann Paul Fuchs, Behnood Rasti and Begüm Demir
The rapid growth of hyperspectral data archives in remote sensing (RS) necessitates effective compression methods for storage and transmission. Recent advances in learning-based hyperspectral image (HSI) compression have significantly enhanced both reconstruction fidelity and compression efficiency. However, existing methods typically adapt variational image compression models designed for natural images, without adequately accounting for the distinct spatio-spectral redundancies inherent in HSIs. In particular, they lack explicit architectural designs to balance spatial and spectral feature learning, limiting their ability to effectively leverage the unique characteristics of hyperspectral data in RS. To address this issue, in this paper, we aim to study the effects of spatio-spectral feature learning on the rate-distortion (RD) performance of variational HSI compression as a first time in RS. To this end, we propose to use configurable spatial and spectral feature learning blocks within variational HSI compression. To achieve this, we introduce spatio-spectral variational hyperspectral image compression architecture (HyVIC), a configurable variational autoencoder (VAE) for HSI compression. Extensive experiments on two
benchmark datasets demonstrate that the trade-off between spatial and spectral feature learning is crucial for the reconstruction fidelity. Motivated by this, we also present a metric-driven strategy to systematically select the hyperparameters of the proposed model. In detail, HyVIC achieves high spatial and spectral reconstruction fidelity across a wide range of compression ratios (CRs) and improves the state of the art by up to 4.66dB in terms of BD-PSNR. Our code and pre-trained model weights are publicly available at https://git.tu-berlin.de/rsim/hyvic .
Read the full abstract ▾
Computer Vision
Meta-Learned Adaptive Optimization for Robust Human Mesh Recovery with Uncertainty-Aware Parameter Updates
Shaurjya Mandal, Nutan Sharma and John Galeotti
Human mesh recovery from single images remains challenging due to inherent depth ambiguity and limited generalization across domains. While recent methods combine regression and optimization approaches, they struggle with poor initialization for test-time refinement and inefficient parameter updates during optimization. We propose a novel meta-learning framework that trains models to produce optimization-friendly initializations while incorporating uncertainty-aware adaptive updates during test-time refinement. Our approach introduces three key innovations: (1) a meta-learning strategy that simulates test-time optimization during training to learn better parameter initializations, (2) a selective parameter caching mechanism that identifies and freezes converged joints to reduce computational overhead, and (3) distribution-based adaptive updates that sample parameter changes from learned distributions, enabling robust exploration while quantifying uncertainty. Additionally, we employ
stochastic approximation techniques to handle intractable gradients in complex loss landscapes. Extensive experiments on standard
benchmarks demonstrate that our method achieves state-of-the-art performance, reducing MPJPE by 10.3 on 3DPW and 8.0 on Human3.6M compared to strong baselines. Our approach shows superior domain adaptation capabilities with minimal performance degradation across different environmental conditions, while providing meaningful uncertainty estimates that correlate with actual prediction errors. Combining meta-learning and adaptive optimization enables accurate mesh recovery and robust generalization to challenging scenarios.
Read the full abstract ▾
Computer Vision
Image-based Quantification of Postural Deviations on Patients with Cervical Dystonia: A Machine Learning Approach Using Synthetic Training Data
Roland Stenger, Sebastian Löns, Nele Brügge et al. (+36 more)
Cervical dystonia (CD) is the most common form of dystonia, yet current assessment relies on subjective clinical rating scales, such as the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS), which requires expertise, is subjective and faces low inter-rater reliability some items of the score. To address the lack of established objective tools for monitoring disease severity and treatment response, this study validates an automated image-based head pose and shift estimation system for patients with CD. We developed an assessment tool that combines a pretrained head-pose estimation
algorithm for rotational symptoms with a deep learning model trained exclusively on ~16,000 synthetic avatar images to evaluate rare translational symptoms, specifically lateral shift. This synthetic data approach overcomes the scarcity of clinical training examples. The system's performance was validated in a multicenter study by comparing its predicted scores against the consensus ratings of 20 clinical experts using a
dataset of 100 real patient images and 100 labeled synthetic avatars. The automated system demonstrated strong agreement with expert clinical ratings for rotational symptoms, achieving high correlations for torticollis (r=0.91), laterocollis (r=0.81), and anteroretrocollis (r=0.78). For lateral shift, the tool achieved a moderate correlation (r=0.55) with clinical ratings and demonstrated higher accuracy than human raters in controlled
benchmark tests on avatars. By leveraging synthetic training data to bridge the clinical data gap, this model successfully generalizes to real-world patients, providing a validated, objective tool for CD postural assessment that can enable standardized clinical decision-making and trial evaluation.
Read the full abstract ▾
All authors Roland Stenger, Sebastian Löns, Nele Brügge, Feline Hamami, Alexander Münchau, Theresa Paulus, Anne Weissbach, Tatiana Usnich, Max Borsche, Martje G. Pauly, Lara M. Lange, Markus A. Hobert, Rebecca Herzog, Ana Luísa de Almeida Marcelino, Tina Mainka, Friederike Schumann, Lukas L. Goede, Johanna Reimer, Julienne Haas, Jos Becktepe, Alexander Baumann, Robin Wolke, Chi Wang Ip, Thorsten Odorfer, Daniel Zeller, Lisa Harder-Rauschenberger, John-Ih Lee, Philipp Albrecht, Tristan Kölsche, Joachim K. Krauss, Johanna M. Nagel, Joachim Runge, Johanna Doll-Lee, Simone Zittel, Kai Grimm, Pawel Tacik, André Lee, Tobias Bäumer, Sebastian Fudickar
Computer Vision
CPUBone: Efficient Vision Backbone Design for Devices with Low Parallelization Capabilities
Moritz Nottebaum, Matteo Dunnhofer and Christian Micheloni
Recent research on
vision backbone architectures has predominantly focused on optimizing efficiency for hardware platforms with high parallel processing capabilities. This category increasingly includes embedded systems such as mobile phones and embedded AI accelerator modules. In contrast, CPUs do not have the possibility to parallelize operations in the same manner, wherefore models benefit from a specific design philosophy that balances amount of operations (MACs) and hardware-efficient execution by having high MACs per second (MACpS). In pursuit of this, we investigate two modifications to standard convolutions, aimed at reducing computational cost: grouping convolutions and reducing kernel sizes. While both adaptations substantially decrease the total number of MACs required for inference, sustaining low latency necessitates preserving hardware-efficiency. Our experiments across diverse CPU devices confirm that these adaptations successfully retain high hardware-efficiency on CPUs. Based on these insights, we introduce CPUBone, a new family of
vision backbone models optimized for CPU-based inference. CPUBone achieves state-of-the-art Speed-Accuracy Trade-offs (SATs) across a wide range of CPU devices and effectively transfers its efficiency to downstream tasks such as object detection and semantic
segmentation. Models and code are available at https://github.com/altair199797/CPUBone.
Read the full abstract ▾
Computer Vision
SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Training
Le Ma, Thiago Freitas dos Santos, Nadia Magnenat-Thalmann et al. (+1 more)
In
surgical training for medical students, proficiency development relies on expert-led skill assessment, which is costly, time-limited, difficult to scale, and its expertise remains confined to institutions with available specialists. Automated AI-based assessment offers a viable alternative, but progress is constrained by the lack of
datasets containing realistic trainee errors and the multi-view variability needed to train robust computer
vision approaches. To address this gap, we present
Surgical-Hands (SHands), a large-scale multi-view
video dataset for
surgical hand-gesture and error recognition for medical training. SHands captures linear incision and suturing using five RGB cameras from complementary viewpoints, performed by 52 participants (20 experts and 32 trainees), each completing three standardized trials per procedure. The
videos are annotated at the frame level with 15 gesture primitives and include a validated taxonomy of 8 trainee error types, enabling both gesture recognition and error detection. We further define standardized evaluation protocols for single-view, multi-view, and cross-view generalization, and
benchmark state-of-the-art deep learning models on the
dataset. SHands is publicly released to support the development of robust and scalable AI systems for
surgical training grounded in clinically curated domain knowledge.
Read the full abstract ▾
All authors Le Ma, Thiago Freitas dos Santos, Nadia Magnenat-Thalmann, Katarzyna Wac
Computer Vision · Cross-listed from eess.IV
Adapting Frozen Mono-modal Backbones for Multi-modal Registration via Contrast-Agnostic Instance Optimization
Yi Zhang, Yidong Zhao and Qian Tao
Deformable image registration remains a central challenge in medical image analysis, particularly under multi-modal scenarios where intensity distributions vary significantly across scans. While deep learning methods provide efficient feed-forward predictions, they often fail to generalize robustly under distribution shifts at test time. A straightforward remedy is full network fine-tuning, yet for modern architectures such as
Transformers or deep U-Nets, this adaptation is prohibitively expensive in both
memory and runtime when operating in 3D. Meanwhile, the naive fine-tuning struggles more with potential degradation in performance in the existence of drastic domain shifts. In this work, we propose a registration framework that integrates a frozen pretrained mono-modal registration model with a lightweight adaptation pipeline for multi-modal image registration. Specifically, we employ style transfer based on contrast-agnostic representation generation and refinement modules to bridge modality and domain gaps with instance optimization at test time. This design is orthogonal to the choice of backbone mono-modal model, thus avoids the computational burden of full fine-tuning while retaining the flexibility to adapt to unseen domains. We evaluate our approach on the Learn2Reg 2025 LUMIR validation set and observe consistent improvements over the pretrained state-of-the-art mono-modal backbone. In particular, the method ranks second on the multi-modal subset, third on the out-of-domain subset, and achieves fourth place overall in Dice score. These results demonstrate that combining frozen mono-modal models with modality adaptation and lightweight instance optimization offers an effective and practical pathway toward robust multi-modal registration.
Read the full abstract ▾
Computer Vision
Restore, Assess, Repeat: A Unified Framework for Iterative Image Restoration
I-Hsiang Chen, Isma Hadji, Enrique Sanchez et al. (+5 more)
Image restoration aims to recover high quality images from inputs degraded by various factors, such as adverse weather, blur, or low light. While recent studies have shown remarkable progress across individual or unified restoration tasks, they still suffer from limited generalization and inefficiency when handling unknown or composite degradations. To address these limitations, we propose RAR, a Restore, Assess and Repeat process, that integrates Image Quality Assessment (IQA) and Image Restoration (IR) into a unified framework to iteratively and efficiently achieve high quality image restoration. Specifically, we introduce a restoration process that operates entirely in the latent domain to jointly perform degradation identification, image restoration, and quality verification. The resulting model is fully trainable end to end and allows for an all-in-one assess and restore approach that dynamically adapts the restoration process. Also, the tight integration of IQA and IR into a unified model minimizes the latency and information loss that typically arises from keeping the two modules disjoint, (e.g. during image and/or text decoding). Extensive experiments show that our approach consistent improvements under single, unknown and composite degradations, thereby establishing a new state-of-the-art.
Read the full abstract ▾
All authors I-Hsiang Chen, Isma Hadji, Enrique Sanchez, Adrian Bulat, Sy-Yen Kuo, Radu Timofte, Georgios Tzimiropoulos, Brais Martinez
Computer Vision
Dynamic Token Compression for Efficient Video Understanding through Reinforcement Learning
Shida Wang, YongXiang Hua, Zhou Tao et al. (+2 more)
Multimodal Large
Language Models have demonstrated remarkable capabilities in
video understanding, yet face prohibitive computational costs and performance degradation from ”context rot” due to massive visual
token redundancy. Existing compression strategies typically rely on heuristics or fixed transformations that are often decoupled from the downstream task objectives, limiting their adaptability and effectiveness. To address this, we propose SCORE (Surprise-augmented
token COmpression via REinforcement learning), a unified framework that learns an adaptive
token compression policy. SCORE introduces a lightweight policy network conditioned on a surprise-augmented state representation that incorporates inter-frame residuals to explicitly capture temporal dynamics and motion saliency. We optimize this policy using a group-wise reinforcement learning scheme with a split-advantage
estimator, stabilized by a two-stage curriculum transferring from static pseudo-
videos to real dynamic
videos. Extensive experiments on diverse
video understanding
benchmarks demonstrate that SCORE significantly outperforms state-of-the-art baselines. Notably, SCORE achieves a 16x prefill speedup while preserving 99.5% of original performance at a 10% retention ratio, offering a scalable solution for efficient long-form
video understanding.
Read the full abstract ▾
All authors Shida Wang, YongXiang Hua, Zhou Tao, Haoyu Cao, Linli Xu
Computer Vision
HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models
MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh et al. (+4 more)
Understanding the fine-grained articulation of human hands is critical in high-stakes settings such as
robot-assisted surgery, chip manufacturing, and AR/VR-based human-AI interaction. Despite achieving near-human performance on general
vision-
language benchmarks, current
vision-
language models (VLMs) struggle with fine-grained spatial
reasoning, especially in interpreting complex and articulated hand poses. We introduce HandVQA, a large-scale diagnostic
benchmark designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand
datasets (FreiHAND, InterHand2.6M, FPHA), our
benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions. We evaluate several state-of-the-art VLMs (LLaVA, DeepSeek and Qwen-VL) in both base and fine-tuned settings, using lightweight fine-tuning via LoRA. Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect
geometric interpretations, and poor generalization. HandVQA not only exposes these critical
reasoning gaps but provides a validated path to improvement. We demonstrate that the 3D-grounded spatial knowledge learned from our
benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).
Read the full abstract ▾
All authors MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul Baek
Computer Vision
MPDiT: Multi-Patch Global-to-Local Transformer Architecture For Efficient Flow Matching and Diffusion Model
Quan Dao and Dimitris Metaxas
Transformer architectures, particularly
Diffusion Transformers (DiTs), have become widely used in
diffusion and flow-matching models due to their strong performance compared to convolutional UNets. However, the isotropic design of DiTs processes the same number of patchified
tokens in every block, leading to relatively heavy computation during training process. In this work, we introduce a multi-patch
transformer design in which early blocks operate on larger patches to capture coarse global context, while later blocks use smaller patches to refine local details. This hierarchical design could reduces computational cost by up to 50% in GFLOPs while achieving good generative performance. In addition, we also propose improved designs for time and class
embeddings that accelerate training convergence. Extensive experiments on the ImageNet
dataset demonstrate the effectiveness of our architectural choices. Code is released at: https://github.com/quandao10/MPDiT
Read the full abstract ▾
Computer Vision
From Pen to Pixel: Translating Hand-Drawn Plots into Graphical APIs via a Novel Benchmark and Efficient Adapter
Zhenghao Xu and Mengning Yang
As plots play a critical role in modern data visualization and analysis, Plot2API is launched to help non-experts and beginners create their desired plots by directly recommending graphical APIs from reference plot images by neural networks. However, previous works on Plot2API have primarily focused on the recommendation for standard plot images, while overlooking the hand-drawn plot images that are more accessible to non-experts and beginners. To make matters worse, both Plot2API models trained on standard plot images and powerful multi-modal large
language models struggle to effectively recommend APIs for hand-drawn plot images due to the domain gap and lack of expertise. To facilitate non-experts and beginners, we introduce a hand-drawn plot
dataset named HDpy-13 to improve the performance of graphical API recommendations for hand-drawn plot images. Additionally, to alleviate the considerable strain of parameter growth and computational resource costs arising from multi-domain and multi-
language challenges in Plot2API, we propose Plot-Adapter that allows for the training and storage of separate adapters rather than requiring an entire model for each
language and domain. In particular, Plot-Adapter incorporates a lightweight CNN block to improve the ability to capture local features and implements projection
matrix sharing to reduce the number of fine-tuning parameters further. Experimental results demonstrate both the effectiveness of HDpy-13 and the efficiency of Plot-Adapter.
Read the full abstract ▾
Computer Vision
Only Whats Necessary: Pareto Optimal Data Minimization for Privacy Preserving Video Anomaly Detection
Nazia Aslam, Abhisek Ray, Thomas B. Moeslund et al. (+1 more)
Video anomaly detection (VAD) systems are increasingly deployed in safety critical environments and require a large amount of data for accurate detection. However, such data may contain personally identifiable information (PII), including facial cues and sensitive demographic attributes, creating compliance challenges under the EU General Data Protection Regulation (GDPR). In particular, GDPR requires that personal data be limited to what is strictly necessary for a specified processing purpose. To address this, we introduce Only What's Necessary, a
privacy-by-design framework for VAD that explicitly controls the amount and type of visual information exposed to the detection pipeline. The framework combines breadth based and depth based data minimization mechanisms to suppress PII while preserving cues relevant to anomaly detection. We evaluate a range of minimization configurations by feeding the minimized
videos to both a VAD model and a
privacy inference model. We employ two ranking based methods, along with Pareto analysis, to characterize the resulting trade off between
privacy and utility. From the non-dominated frontier, we identify sweet spot operating points that minimize personal data exposure with limited degradation in detection performance. Extensive experiments on publicly available
datasets demonstrate the effectiveness of the proposed framework.
Read the full abstract ▾
All authors Nazia Aslam, Abhisek Ray, Thomas B. Moeslund, Kamal Nasrollahi