Computer Vision
AnimalLift: Reconstructing Animatable 3D Animals from a Single Image by Learning Canonical Shape, Texture, and Fur Maps
Chunyi Sun, Ruyi Zha, Weijian Deng et al. (+3 more)
Reconstructing a fully animatable 3D animal from a single image remains challenging because animation-ready assets require not only plausible geometry, but also a unified
topology, editable appearance, and fur representations compatible with deformation and simulation. Existing image-to-3D approaches often rely on implicit or loosely structured representations that are difficult to rig or edit, while parametric animal models support animation but cannot capture detailed texture and fur appearance. We present AnimalLift, a framework for reconstructing structured, animation-compatible 3D animal assets with explicit fur from a single image. Our method lifts an input image into a shared canonical space with a consistent
topology and UV parameterization across the
dataset, enabling joint prediction of canonical geometry, texture, and fur in a unified feed-forward architecture. A key component of our representation is a UV-aligned fur map that encodes strand geometry in a surface-aligned canonical domain, allowing explicit fur reconstruction compatible with mesh deformation and fur simulation. To train the model, we introduce a procedural data generation pipeline that provides large-scale supervision with aligned geometry, texture, and fur across diverse animal species and appearances. Experiments on synthetic and real-world
datasets demonstrate strong reconstruction quality and generalization across animal categories. Beyond reconstruction, our structured representation directly supports downstream applications including animation, pose transfer, fur editing, and simulation-compatible rendering.
Read the full abstract ▾
All authors Chunyi Sun, Ruyi Zha, Weijian Deng, Junlin Han, Dylan Campbell, Stephen Gould
Computer Vision
RoMa-$\Omega$: What Feed-Forward 3D Models Know About Image Matching
David Nordström, Xinyue Zhang, Thibaut Loiseau et al. (+2 more)
Learned image matching has experienced significant progress in recent years, culminating in robust and accurate matchers such as RoMa, whose robustness is often attributed to its use of frozen DINO features. In a parallel development, feed-forward reconstruction models, such as VGGT, have been trained on ever-growing
datasets to accurately regress dense 3D point maps and camera poses. The distinction between matchers and feed-forward reconstruction models has become increasingly blurred with the introduction of matching losses in models such as MASt3R and VGGT-$\Omega$. This raises a natural question: what do feed-forward 3D models know about image matching? In this work, we answer this question by analyzing three scenarios: (i) zero-shot matching of patch features, (ii) direct matching of 3D point predictions, and (iii) training a full matcher on top of the learned representations. We find that, despite performing poorly in zero-shot matching, especially in later layers, feed-forward reconstruction models provide strong representations for linear probing and full matching pipelines. We further show that, even without any training, their raw predictions alone enable competitive matching, albeit only under moderate viewpoint changes and modality gaps. Based on these insights, we retrain RoMa v2 by replacing its DINO backbone with VGGT-$\Omega$. Our resulting model, \ours, outperforms state-of-the-art matchers on a wide range of
benchmarks, e.g. +8.1 mAA compared to RoMa v2 on WxBS.
Read the full abstract ▾
All authors David Nordström, Xinyue Zhang, Thibaut Loiseau, Vincent Lepetit, Fredrik Kahl
Computer Vision
Learning Global Camera Poses from Noisy View-Graphs for Structure from Motion
Fadi Khatib, Meirav Galun and Ronen Basri
Camera pose estimation is a key step in 3D reconstruction and view-synthesis pipelines. We present a deep, global Structure-from-Motion framework based on learned view-graph aggregation. Our method employs a permutation-equivariant, edge-conditioned graph neural network that takes noisy pairwise relative poses as input and outputs globally consistent camera extrinsics. The network is trained without ground-truth supervision, relying solely on a relative-pose consistency objective. This is followed by 3D point triangulation and robust bundle adjustment. Our approach is efficient, scalable to more than a thousand images, and robust to graph density. We evaluate our method on MegaDepth, 1DSfM, Strecha, and BlendedMVS. These experiments demonstrate that our method achieves superior rotation and translation accuracy compared to deep track-centric methods while registering more images across many scenes, and competitive results compared to state-of-the-art classical pipelines, while being much faster.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Efficient Fairness Auditing Across Guidance Scales in Text-to-Image Diffusion Models via Causal Abstraction
Nabila Tasfiha Rahman, Rajatsubhra Chakraborty, Depeng Xu et al. (+1 more)
Fairness auditing of text-to-image
diffusion models often requires generating large numbers of images across sampling configurations, making comprehensive evaluation computationally expensive. We propose a causal-abstraction-based audit instrument for efficiently evaluating
fairness under interventions on the classifier-free guidance scale. Given a fixed
prompt and a target feature function, we represent the
diffusion process as a low-level structural causal model and construct a corresponding high-level model over abstract denoising states. We characterize the projected causal structure, establish identifiability of the
fairness-relevant interventional query, and provide sufficient conditions under which the high-level model preserves this query. A probabilistic
transformer implements the high-level model as an amortized predictor of target-feature distributions across guidance scales. Experiments evaluate distributional fidelity,
fairness-query accuracy, and computational efficiency. We present two
auditing demonstrations: one using standard Stable
Diffusion 1.5 and another using StayFair, a
fairness-enhanced Stable
Diffusion model, to examine their behavior across guidance scales.
Read the full abstract ▾
All authors Nabila Tasfiha Rahman, Rajatsubhra Chakraborty, Depeng Xu, Lu Zhang
Computer Vision
Infra-Bench CLS: A Global, Open-Source Benchmark for Critical Infrastructure Classification with Earth Observation Foundation Models
Justin Guthrie, Edward Oughton, Konrad Wessels et al. (+2 more)
Critical infrastructure location data is often incomplete and unevenly distributed globally, especially in developing regions. Earth observation foundation models are proposed as a new step in enabling us to more efficiently understand the natural and built environment, raising questions as to their effectiveness in performing challenging downstream tasks. Yet, foundation models remain largely untested for detecting and classifying the facility-scale critical infrastructure that underpins a range of important societal and economic functions. Subsequently, Infra-Bench CLS is introduced as a
benchmark to test foundation models on 18,756 Sentinel-1 SAR and Sentinel-2 multispectral facility-scale critical infrastructure asset images covering seven continents and 13 infrastructure classes, with results reported for the 10 retained classes. Using linear probing and fine-tuning for two training
dataset levels (1.0x and 0.3x), seven foundation models are evaluated (SatlasPretrain S2, SatlasPretrain S1, CROMA, Prithvi-EO-2.0, AlphaEarth Foundations, OlmoEarth v1.1-Base, and DINOv3 ViT-L/16). When comparing macro F1 scores to a ResNet-18 supervised baseline of 39.2 percent, the best foundation model achieved 57.9 percent, a 48 percent improvement. Top performing classes were airports (F1 85.3 percent), train stations (F1 82.1 percent), and data centers (F1 77.6 percent). By contrast, many of the power sector classes perform poorly (F1 27.5-46.2 percent). These findings suggest foundation models can enable superior critical infrastructure classification, but future work should evaluate performance on higher-resolution imagery, particularly for poorly performing sectors, such as power.
Read the full abstract ▾
All authors Justin Guthrie, Edward Oughton, Konrad Wessels, Matthew Rice, Isaac Corley
Computer Vision
LeCor: Learning to Be Corrected by Meta-Learned Test-Time Training for Interactive 3D Lung-Tumour Segmentation
Yi Luo, Yike Guo, Wenxuan Li et al. (+3 more)
Delineating lung tumours on computed
tomography (CT) takes a considerable share of the time spent on radiotherapy planning, and a contour proposed by a model can be refined interactively by the clinician. Promptable foundation models such as SAM 3 support this workflow by writing each correction into a session
memory that conditions the remaining slices, while the model weights stay fixed. On 690 test cases from five public CT cohorts, fine-tuning SAM 3 on lung tumours raises the Dice obtained from a single point
prompt from 0.298 to 0.757, and seven rounds of corrections raise it further to 0.765, but under
memory conditioning alone the accuracy on slices the annotator has not touched stops improving after six rounds. We therefore treat each correction as a training signal and propose LeCor, which performs test-time training on a small set of case adapters that are reset for every case and meta-learned such that a single gradient step driven by a click improves the slices that were not clicked. On the 133 test cases that span at least eight slices, LeCor raises the Dice reached after seven correction rounds from 0.787 with the fine-tuned model to 0.827, reduces the number of cases that never reach a Dice of 0.80 from 47 to 27, and reaches in three correction rounds the accuracy that the fine-tuned model attains in seven.
Read the full abstract ▾
All authors Yi Luo, Yike Guo, Wenxuan Li, Zongwei Zhou, Rui Zhang, Kai Ding
Computer Vision
Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh et al. (+1 more)
Prompt learning adapts CLIP to downstream recognition by replacing hand-written templates with learned continuous context vectors, which in Context Optimization (CoOp) form a dense
prompt matrix $\mathbf{P}\in\mathbb{R}^{m\times d}$ trained from only a few examples per class. We study whether this
matrix is over-parameterized by factorizing it as $\mathbf{P}=\mathbf{B}\mathbf{A}$, which cuts the trainable
prompt parameters from $md$ to $r(m+d)$, and to $rd$ once the
token-side factor $\mathbf{B}$ is fixed. Across seven few-shot
benchmarks and two CLIP backbones, low-rank
prompts match or improve dense CoOp at far fewer parameters, with the clearest gains on low-shot base-to-new generalization. We then find that the
token-side factor need not be learned at all: fixing $\mathbf{B}$ to a Gaussian, orthogonal, SVD-derived, or even random basis and training only the
embedding-side factor $\mathbf{A}$ stays on par with the fully trainable factorization, and a source-trained $\mathbf{B}$ offers no advantage over a random one. A
prompt-factor asymmetry and a local update-space dimension gap show why fixing $\mathbf{B}$ is far less restrictive than fixing $\mathbf{A}$, and a smoothness-only guarantee certifies that optimizing $\mathbf{A}$ over a fixed $\mathbf{B}$ converges. In the CLIP
prompt setting, the
embedding-side coefficients carry the adaptation while the
token basis can simply be fixed.
Read the full abstract ▾
All authors Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh, Ramtin Pedarsani
Computer Vision
Longitudinal tracking of multiple sclerosis lesions in the spinal cord: A validation study
Pierre-Louis Benveniste, Julian McGinnis, Shannon Kolind et al. (+11 more)
Longitudinal characterization of multiple sclerosis (MS) lesions remains constrained by the lack of frameworks capable of establishing consistent instance-level correspondences across time. Conventional
segmentation approaches produce semantic lesion masks at each visit and therefore fail to capture the complex instance temporal patterns associated with lesion appearance, disappearance, splitting, or merging. This study presents a comparative evaluation of five strategies for automated tracking of spinal cord MS lesions in longitudinal MRI data from a multi-site cohort. The investigated strategies rely either on deformable registration or on a spinal anatomical reference system, and encompass overlap-based matching, coordinate-based Hungarian
algorithm, gradient-boosted classification, and Siamese model classification. Tracking accuracy is quantified using instance-level true positives, false positives, and false negatives, allowing to assess the presence of one-to-many and many-to-one associations. Results show best performance for the registration-based overlap method. This study provides the first systematic analysis of lesion-instance correspondence in the spinal cord and outlines the strengths and limitations of registration-based and registration-free paradigms for longitudinal MS assessment. The code is available at http://github.com/ivadomed/longitudinal-sc-ms-lesion-tracking .
Read the full abstract ▾
All authors Pierre-Louis Benveniste, Julian McGinnis, Shannon Kolind, Larry D. Lynd, Sarah A. Morrow, Jiwon Oh, Alexandre Prat, Alice Schabas, Penelope Smyth, Roger Tam, Anthony Traboulsee, Mark Mühlau, Herve Lombaert, Julien Cohen-Adad
Computer Vision
Vision-language models see more about agriculture than they report, and test-time scaling remains bounded by parametric knowledge
Earl Ranario, Jared Smith, Lars Lundqvist et al. (+3 more)
Vision-
language models (VLMs) show promise for agricultural diagnosis but fail unpredictably in zero-shot settings. We test whether this stems from poor visual features or an inability to verbalize what the model already perceives, using a
benchmark spanning classification and detection drawn from 118 AgML classification and 54 detection
datasets. Linear probing shows agricultural features are linearly separable in frozen VLM
vision towers, with Gemma-4-31B's encoder reaching F1-macro of 0.60–0.78 across tasks and approaching the
vision-only DINOv3 backbone, indicating the bottleneck lies downstream of perception. We then test whether test-time scaling, an iterative in-context skill-
memory loop, can close this gap without updating weights. Gains are modest and inconsistent. Skill-
memory improves classification accuracy by up to 3.1 points on three of four tasks while regressing on species, with gains concentrated on classes seen during optimization and unseen classes regressing. Detection F1 is effectively unchanged (0.412 to 0.421), though an oracle filtering condition reaches 0.432 F1, revealing real headroom unconstrained prompting fails to surface. These results indicate VLMs see more agricultural features than they report, but test-time scaling alone fails to surface that hidden parametric knowledge, motivating agentic approaches grounded in external, retrievable knowledge.
Read the full abstract ▾
All authors Earl Ranario, Jared Smith, Lars Lundqvist, Urmil Jatin Chandarana, Bharath Sridhar, J Mason Earles
Computer Vision
VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models
Zaid Pervaiz Bhat, Nimra Nayyar, Arihant Jain et al. (+6 more)
As
Vision-
Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented
Embodied AI evaluated on subject-centric consumer
video. This overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed cameras for open-loop insights like safety monitoring and operational logging. We introduce VANTAGE-Bench, a
benchmark measuring this "Infrastructure AI Gap." It spans three operational domains (Logistics, Transportation, and Smart Spaces), unifies image and
video evaluation across semantic, spatial, temporal, and spatio-temporal capabilities, and moves beyond multiple-choice to eight task formulations including dense captioning and spatio-temporal grounding. It adds a single-pass trajectory protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure
video, scored against specialist trackers. Annotation spans three regimes over 3,346 media assets: 3,342
video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes. Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric
benchmarks is concentrated, not general. Event verification, referring expressions, and temporal localization fall roughly 9 to 24 points at every model scale, while
video question answering stays within 5.3 points of VideoMME and 2D spatial pointing shows no shortfall against BLINK. The temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA_c on dense
video captioning. On tracking, frontier models come within roughly 5 points of specialist trackers over short horizons but separate as the horizon extends. Open-weight models lead 2D object localization outright, so neither scale nor proprietary access explains the pattern. Data, evaluation harness, and leaderboard: https://vantage-bench.org/
Read the full abstract ▾
All authors Zaid Pervaiz Bhat, Nimra Nayyar, Arihant Jain, Lap Fung Chan, John Suchanek, Yu Wang, Varun Praveen, Tomasz Kornuta, Vidya Nariyambut Murali
Computer Vision
OmniPoint: Universal Monocular Metric Pointcloud from Any Camera
Botao Ye, Marc Pollefeys, Ming-Hsuan Yang et al. (+1 more)
Recovering metric 3D geometry from
monocular images is a fundamental computer
vision task, yet current methods remain heavily fragmented by fixed camera model assumptions and inflexible input schemes. We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors, including pinhole, fisheye, and equirectangular projections, while accommodating varying
geometric priors. To overcome projection
rigidity, OmniPoint abandons conventional planar depth regression. It instead adopts a decoupled ray and distance representation alongside a decoupled training objective, explicitly separating the camera projection model from the scene structure. To address the severe scarcity of training data for alternative cameras, we introduce a bidirectional augmentation strategy that explicitly bridges labeled perspective data and unlabeled omnidirectional domains in 3D space. Furthermore, to seamlessly integrate optional inputs like camera intrinsics or sparse depth without destabilizing the network through feature distribution shifts, we propose a robust information injection mechanism. This mechanism utilizes learnable input state
embeddings to resolve architectural ambiguity and applies vectorized Gaussian smoothing to densify irregular measurements. Extensive experiments demonstrate that OmniPoint achieves state-of-the-art zero-shot performance across multiple
benchmarks, establishing a robust new standard for unified
monocular 3D reconstruction.
Read the full abstract ▾
All authors Botao Ye, Marc Pollefeys, Ming-Hsuan Yang, Abhijit Kundu
Computer Vision
The Living Library: Transforming Archival Collections into Conversational Knowledge Systems – Lessons from the Theodore Roosevelt Presidential Library
Pengce Wang, Lucia Ronchi Darre, Matt Briney et al. (+8 more)
We present the Living Library, an end-to-end framework for transforming fragmented digital archives into governed, conversational, in-person exhibit experiences. Developed and deployed at the Theodore Roosevelt Presidential Library, the framework comprises four layers: digitization and corpus creation, AI-powered processing,
retrieval and
reasoning, and an optional
embodied conversational interface. The first three layers aggregate a 300,000-record collection, apply OCR and structured metadata enrichment for expert curatorial review, and publish records to a hybrid dense/semantic index. Expert review is conducted through the Archivist App, a curator-facing interface that supports correction of AI-generated transcriptions and metadata. The governed corpus powers both a researcher-facing interface and Talk to TR, a continuously operating exhibit that embodies Theodore Roosevelt as a full-scale digital human within a museum environment. To support live, face-to-face interactions, Cross-Era Analogical Grounding reframes contemporary questions through documented historical parallels, allowing Roosevelt to address present-day topics without inventing facts. Dual-path
retrieval and end-to-end streaming keep responses grounded and responsive. Layered watchdogs, visitor-session isolation, automated conversation management, and independently restartable services enable reliable unattended operation for hundreds of visitors. Avatar realism, spatial audio, lighting, staging, and conversational design are developed and evaluated as an integrated experience. Rather than report a controlled
benchmark, we describe lessons from operating Talk to TR as a public exhibit and offer a transferable model for transforming archival collections into believable, in-person conversational experiences.
Read the full abstract ▾
All authors Pengce Wang, Lucia Ronchi Darre, Matt Briney, Michaell Bakalars, Dan Rutkowski, Ursula Hardy, David Wolf, Laura Hoffman, Allen Kim, Shawn Wright, Juan Lavista Ferres
Computer Vision
DensePol: Dense-Angle Polarization Dataset for Learning-Based Polarimetric Vision
Param Sangani, Ahmad Moori, Erik Blasch et al. (+2 more)
Polarimetric
vision is gaining increasing
attention because it provides physical cues about scene shape, material, and reflection that are difficult to recover from RGB alone. Recent work has therefore explored predicting polarization directly from conventional RGB images; however, the fidelity of these methods strongly depends on the polarization supervision used for training. Most existing
datasets rely on Division-of-Focal-Plane (DoFP) cameras with four spatially interleaved analyzer orientations, which provide limited angular redundancy and introduce interpolation and instantaneous-field-of-view errors. We introduce DensePol, a high-redundancy RGB–polarization
dataset based on Division-of-Time (DoT) acquisition, capturing 180 full-resolution analyzer orientations at $1^\circ$ intervals. DensePol contains 2,018 paired RGB–polarization images with the angular measurements and fitting residuals retained. Dense angular sampling substantially improves polarization stability, reducing AoLP deviation from $13.36^\circ$ to $2.21^\circ$. We further introduce a deterministic
diffusion-based RGB-to-polarization framework with cyclic AoLP representation and a local DoLP refiner. Experiments demonstrate improved polarization prediction and downstream surface-normal estimation. The
dataset and code will be publicly available.
Read the full abstract ▾
All authors Param Sangani, Ahmad Moori, Erik Blasch, Guna Seetharaman, Hadi Aliakbarpour
Computer Vision
Video-MOPD: Multi-Teacher On-Policy Distillation for Video Understanding
Zhenxin Qin, Peng Shi, Cong Han et al. (+3 more)
Video understanding demands a convergence of complementary capabilities across perception, temporal understanding, and complex
reasoning, which are difficult to jointly optimize within a single model. We introduce
Video-MOPD-8B, an open-weight model dedicated to
video understanding tasks. To fundamentally enhance its capabilities, we conduct targeted reinforcement learning (RL) optimization across three core domains:
video temporal grounding (VTG), general
video comprehension, and
video STEM
reasoning. We then unify their complementary capabilities via Multi-Teacher On-Policy
Distillation (MOPD), which consolidates expert knowledge by supervising student-generated trajectories with routed teacher feedback. We further introduce Reliability-Aware Informative Sampling (RAIS), which selects examples with consistently reliable teacher supervision and large teacher-student performance gaps. Together, these components enable
Video-MOPD-8B to achieve coordinated and comprehensive performance gains across diverse
video understanding tasks. Extensive experiments on comprehensive
benchmarks covering general
video understanding, temporal grounding,
video reasoning, and
video STEM tasks demonstrate that
Video-MOPD-8B achieves state-of-the-art performance among existing models at a comparable scale. The trained model weights are available at https://huggingface.co/LandH/
Video-MOPD-8B.
Read the full abstract ▾
All authors Zhenxin Qin, Peng Shi, Cong Han, Yinlong Qian, Zequn Jie, Lin Ma
Computer Vision · Cross-listed from cs.RO
No Free Checker: A Survey of Verifiers for Robot Policies
Yang Wan, Xihang Yue, Zhirui Liu et al. (+7 more)
A verifier for
robot policies reads a candidate behavior and returns a score for how well it did, used both to evaluate
vision-
language-action policies and to train them. Verifiers range from success detectors and reward models to runtime monitors, safety filters, and temporal-logic specifications. We survey roughly 150 verifiers and compare them along two properties. Availability is how much a verdict costs, how early in a rollout the verdict arrives, and how often a verdict can be asked for. Availability rises as verdicts get cheaper, earlier, and denser. Credibility is how much a high score tells us about the task. Credibility falls as the judgment becomes gameable and self-serving. We group the verifiers by who supplies the judgment: human verifiers, rule-based and formal verifiers, learned and pretrained verifiers, and model-intrinsic verifiers. Across the four families, we find that credibility falls as availability rises. Regardless of who supplies the judgment, there is no free checker. We then examine what validates a verifier itself, and how much a high score tells us. Three measures appear in the literature: agreement with human labels, the performance of the policy it trains, and behavior under reward hacking. We close with nine metrics that make a verifier claim checkable, and coordinates for the verifiers still to be built.
Read the full abstract ▾
All authors Yang Wan, Xihang Yue, Zhirui Liu, Ziyuan Chu, Shuxun Wang, Yuhan Chen, Xiaonan Jiang, Xukun Zhu, Yubo Dong, Linchao Zhu
Computer Vision
SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators
Yuncong Yang, Zhengtao Han, Furkan Ozyurt et al. (+6 more)
World models are increasingly used as policy-in-the-loop imagination environments, where reliable rollouts require fine-grained controllability with respect to low-level
robot actions. A key obstacle to scaling such models in robotics is that actions are not a universal
language in pixel space: changes in visual environment, camera view,
robot placement, or embodiment alter how the same numerical action manifests visually, leading to conflicting supervision under mixed training and brittle generalization at deployment. We introduce SyncWorld, an action-conditioned world model that serves as a zero-shot simulator across unseen environments without any additional training. SyncWorld leverages a visual calibration episode—paired frames and actions that showcase all the controllable degrees of freedom—to specify the setup-specific Action–Visual Mapping in context. Training with visual calibration contexts teaches the model to interpret actions through visual evidence and to leverage interaction history when explicit calibration is unavailable. Experiments show that SyncWorld can accurately simulate action outcomes in previously unseen settings, and that its capability of simulating rollouts enables test-time policy improvement without training.
Read the full abstract ▾
All authors Yuncong Yang, Zhengtao Han, Furkan Ozyurt, Zeyuan Yang, Han Yang, Junyi Cao, Haoyu Zhen, Yilun Du, Chuang Gan
Computer Vision
Point4D: Long-range 4D Motion Reconstruction
Minsik Jeon, Jay Karhade, Deva Ramanan et al. (+1 more)
We introduce Point4D, a feed-forward model for 4D reconstruction of long-range
video sequences. Point4D is able to reliably infer dense per-point 3D trajectories across multi-hundred-frame
videos, unlike existing 4D methods that are limited to short input windows of at most a few dozen frames. A key innovation that enables this is our flexible 3D query-based motion decoder that decouples trajectory prediction from image-plane visibility. The predicted 3D endpoints are then directly re-queried in the next chunk without re-projection or matching. Furthermore, we show that extracting and reusing a visual descriptor from an arbitrary frame where the point is visible leads to better performance than relying solely on the source patch. Overall, Point4D achieves state-of-the-art performance across diverse long-
video tracking
benchmarks spanning over 200 frames and largely outperforms previous feed-forward 4D method. Project page: https://point-4d.github.io
Read the full abstract ▾
All authors Minsik Jeon, Jay Karhade, Deva Ramanan, Shubham Tulsiani
Computer Vision
Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
Siting Li, Zhengyang Wang, Simon Shaolei Du et al. (+2 more)
Image tokenizers define the “visual
language” of unified
multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual
tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during
multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study
multimodal learnability—how well image and text
tokens are jointly modeled—and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss–performance relationship depends on the predicted
token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss–performance relationship shifts with the image-
token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes—the discriminator, semantic supervision, and vocabulary size—to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual
languages, highlighting their interplay with text in joint
multimodal training.
Read the full abstract ▾
All authors Siting Li, Zhengyang Wang, Simon Shaolei Du, Xi Chen, Yang Liu
Computer Vision
Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Xiaofu Chen, Stella Frank and Yova Kementchedjhieva
Visual encoders construct a representation of the image input for
Vision-
Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether
vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a
dataset of objects with canonical colors, and probe
vision encoders for both color and object identity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted object identity, indicating a conceptual link. Extending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the
vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in
vision encoders and VLMs.
Read the full abstract ▾
Computer Vision
Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout
Zhuoran Zhao, Shengju Qian, Tongtong Liang et al. (+7 more)
Autoregressive (AR)
video diffusion models have shown great potential in real-time
video generation. Recent methods distill pretrained bidirectional
video diffusion models into causal AR students through Distribution Matching
Distillation (DMD), but the generated
videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR
diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner
tokens act as denoising guidance for other noisier
tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR
video diffusion distillation methods with higher visual quality efficiently, without incorporating real
video data or additional post-training stages.
Read the full abstract ▾
All authors Zhuoran Zhao, Shengju Qian, Tongtong Liang, Xianghao Kong, Songchun Zhang, Junchao Huang, Guian Fang, Xin Wang, Pan Hui, Anyi Rao
Computer Vision
GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting
Thodoris Betsas, Anastasios Doulamis and Andreas Georgopoulos
Open vocabulary 3D semantic
segmentation methods typically lift CLIP features into 3D. This embeds points in a joint
vision-
language space known to behave like a bag-of-words on compositional tasks. Furthermore, even annotation free variants often require a large 3D training corpus and a dedicated 3D encoder per domain. Instead we use a
vision-
language model purely as a translator. It produces structured, entity-level descriptions of each posed image. These descriptions are grounded, projected, and aggregated directly in a general-purpose,
language-only
embedding space, with no 3D training corpus or encoder required. On ScanNet++, our pipeline is competitive with strong annotation free baselines trained on ScanNet. On a 5-building cultural heritage
benchmark, raw scores initially favor a CLIP-based variant, but a single systematic vocabulary correction reverses this ranking. An effect confirmed by a second, independent correction on a different class, indicating that
language-space
embeddings track physical content more faithfully. This fidelity extends to genuinely out-of-vocabulary (OOV) objects on ScanNet++ proving that
language-space
embeddings separate presence from absence objects far more sharply than CLIP-based
embeddings do. GoDeep also localize these OOV objects within the scene, all without any 2D-3D annotation. Because every representation remains discrete text, predictions are also explainable at the point level. Finally, exploiting both a heuristic weighting, that favors precise over merely frequent observations and GoDeep's explainability property, we propose an aggregation strategy, as a proof of concept, that favors finer elements localization.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Rethinking Learned Occupancy in Autonomous Active Mapping with Observation-Gated Filtering
Jiahui Zhang, Bonian Han, Gongbo Liang et al. (+1 more)
Autonomous 3D active mapping requires a space
robot to choose where to sense while building the geometry needed for
navigation. Learned occupancy completion extends spatial context beyond the current field of view, but one predicted map often serves two planning roles: it scores expected surface gain and constrains collision-free motion. Unsupported occupancy can therefore distort both where the
robot looks and where it believes it can travel. We study this coupled interface in a controlled closed-loop
benchmark by holding the active-mapping system fixed and varying only its planner-facing occupancy across observation-only, learned, oracle-corrected, and ground-truth conditions. Improving occupancy accuracy does not monotonically improve closed-loop coverage: across 25 starts, planning with ground-truth occupancy reaches 70% of the learned baseline's final coverage 12.7 steps earlier on average, while increasing final coverage by only 0.031. Guided by this diagnosis, we introduce an observation-gated filter that retains completion in insufficiently observed regions and suppresses predictions only after repeated frustum exposure without nearby RGB-D support. The filter improves both targeted failure-prone starts without retraining or ground truth. These results motivate online revision of planner-facing geometry during autonomous intervals between communication windows. The current study assumes
benchmark RGB-D observations and sufficiently accurate pose estimates; planetary sensing conditions and accumulated localization drift remain to be evaluated.
Read the full abstract ▾
All authors Jiahui Zhang, Bonian Han, Gongbo Liang, Yu Zhang
Computer Vision
"World Knowledge" in the Weights: Reading Concept Circuits of Vision Transformers
Yanlin Chen, Tang Li and Xi Peng
Vision transformers (ViTs) have achieved remarkable generalization across visual domains, yet little is known about how they internally represent the structure of the world. To address this gap, we use Cross-Layer Transcoders (CLTs) to read concept
circuits from ViTs: directed graphs whose nodes correspond to sparse, interpretable concepts and edges capture concept interactions across layers. Our method yields two complementary views of model behavior. The global concept
circuit is input-invariant and can be recovered directly from learned cross-layer weights, exposing the reusable "world knowledge" encoded in the model. The instance concept
circuit is input-dependent and identifies the concepts and pathways actually used for a specific prediction, enabling faithful example-level explanations. We demonstrate the utility of concept
circuits in three ways: (1) Automatic spurious correlation discovery: leveraging the statistics of our global concept
circuits to identify shortcut dependencies within the model. (2) Spurious correlation removal: intervening on the instance concept
circuit to steer the model towards correct predictions. Empirical results show that our method outperforms existing counterparts by 11.0% on the Waterbird
dataset. (3) Model comparison: contrasting the global concept
circuits of different foundation models (e.g., CLIP vs. DINO) to reveal how supervision paradigms shape representational structure. Our code is available at https://github.com/deep-real/VisionCLT
Read the full abstract ▾
Computer Vision
Task-driven Processing with Coarse-to-Fine Glimpse-based Active Perception
Oleh Kolner, Thomas Ortner, Stanisław Woźniak et al. (+1 more)
State-of-the-art
vision models process images in their entirety, lacking the ability to selectively zoom in on relevant regions. This limitation is particularly acute in scenarios where processing must be conditioned on a specific task - such as instance detection, which requires localizing a specific object in a high-resolution, cluttered scene. In such settings, critical details are easily lost as images are often resized to match the model dimensions and computational constraints. We introduce Coarse-to-Fine Glimpse-based Active Perception (CF-GAP), a task-driven front-end that enhances high-resolution processing of existing instance detectors. CF-GAP selectively directs a sequence of limited view glimpses across the scene, utilizing task information to iteratively refine focus on the most relevant regions. These localized regions are then processed at high resolution by a downstream instance detector. By avoiding full-image processing and eliminating irrelevant confounding information, CF-GAP improves Average Precision (AP) by up to 20% across various state-of-the-art instance detectors on the HR-InsDet and Robotools
benchmarks, while further enabling lightweight detectors to outperform their larger counterparts.
Read the full abstract ▾
All authors Oleh Kolner, Thomas Ortner, Stanisław Woźniak, Angeliki Pantazi
Computer Vision
PIC: Revisiting INR for Image Coding with Fast Encoding and Sub-Millisecond Decoding
Xiang Liu, Jinxiang Wang, Bin Chen et al. (+5 more)
Implicit neural representation (INR) has achieved remarkable progress in novel view synthesis and image/
video coding in recent years.Compared to conventional end-to-end image codecs, INR-based compressors demonstrate significant advantages in decoding complexity. However, their practical application has been hindered by the inferior encoding speed and underutilized decoding efficiency.In this work, we propose a feedforward INR image coding architecture, Practical INR Image Codec (PIC), that computes all the necessary information for INR network in a single forward pass, achieving an encoding speed of 20 FPS. Additionally, we implement a highly optimized decoder that reaches 2000 FPS decoding speed, significantly surpassing JPEG's performance at comparable rate-distortion (RD) performance. To the best of our knowledge, this work presents the first learning-based image codec that simultaneously outperforms or is comparable with JPEG in both RD performance and decoding speed while maintaining practical encoding speed. Code is available at https://github.com/actcwlf/PIC.
Read the full abstract ▾
All authors Xiang Liu, Jinxiang Wang, Bin Chen, Zimo Liu, Mingyao Hong, Jiawei Li, Yaowei Wang, Shu-tao Xia
Computer Vision
Spheriverse: 3D Scene Understanding from Spherical Observations in the Wild
Fei Teng, Sheng Wu, Mengfei Duan et al. (+7 more)
Spherical observations provide global visual context for 3D scene understanding. However, visual information is encoded in an angular domain, whereas the physical world is represented in Cartesian coordinates. This cross-space representation gap complicates
geometric correspondence and semantic evidence aggregation. To delve into this challenge, we introduce Spheriverse, comprising 64,400 temporally aligned spherical image-
LiDAR pairs organized into 644 sequences. The
dataset spans diverse scenes, illumination, and weather conditions, with fine-grained semantic classes. We further establish
benchmarks for semantic occupancy prediction, semantic mapping, and 3D object detection, evaluating 30+ methods through overall and scene-wise comparisons. For dense prediction, we propose SphereOcc, an occupancy framework that couples spherical geometry modeling with semantic evidence
retrieval. Cartesian-Spherical Representation Remodeling (CSRR) incorporates spherical range-azimuth geometry into Cartesian voxel features through region-wise modulation. Spherical Evidence Re-querying (SER) then conditions queries on voxel content and range-height-azimuth geometry to adaptively retrieve relevant semantic evidence from source spherical image features. SphereOcc achieves 13.91% mIoU and 24.65% GeoIoU, yielding relative improvements of 13.9% and 9.3% over the respective best-performing methods, TPVFormer and SurroundOcc. It also ranks first in both metrics across all five scene categories, with consistent advantages across the evaluated spatial partitions and reduced fields of view. The established
benchmark and source code will be available at https://feit-feiteng.github.io/Spheriverse.
Read the full abstract ▾
All authors Fei Teng, Sheng Wu, Mengfei Duan, Guoqiang Zhao, Junhui Ma, Kai Luo, Siyu Li, Hao Shi, Zhiyong Li, Kailun Yang
Computer Vision
A Joint 2D-3D Statistical Shape Model for Orthopedic Reconstruction
Florence Dell'Aniello Picard, Pranav Poudel, Nairouz Shehata et al. (+2 more)
Three-dimensional femoral reconstruction from radiographs supports
surgical planning, implant sizing, and post-operative follow-up, but remains ill-posed as X-ray projections discard depth information. Existing methods often incorporate a 3D statistical shape model (SSM) as a shape prior to guide reconstructions toward anatomically plausible shapes, relying on iterative 3D-to-2D projection matching. Yet, these approaches are computationally expensive and constrain their SSM to a single dimensionality, leaving the statistical relationship between 2D observations and 3D geometry largely unexploited and unexplored. We instead propose a joint 2D-3D SSM that explicitly captures the co-variation between 2D and 3D
segmentations in a shared latent space. During training, 2D and 3D
segmentations are registered to a common 3D template and its corresponding 2D projections, and the resulting stationary velocity fields are jointly decomposed using principal component analysis (PCA). This joint modeling allows the 2D-to-3D mapping to be learned directly from data rather than computing correspondences at inference time. For unseen subjects, the 3D shape is recovered directly by lifting the 2D latent coordinates to the 3D PCA subspace, thereby eliminating the need for iterative 3D-to-2D projection. Experiments on NMDID demonstrate that the proposed joint 2D-3D SSM outperforms a widely-used 3D-only SSM baseline while achieving inference approximately 4 times faster, at under 3 seconds per subject. The code is available at: https://github.com/florence-dellaniello-picard/joint2d3d-ssm.
Read the full abstract ▾
All authors Florence Dell'Aniello Picard, Pranav Poudel, Nairouz Shehata, Frédéric Lavoie, Herve Lombaert
Computer Vision
DXPR: Depth-Based Vision-LiDAR Cross-Modal Place Recognition Using Vision Foundation Models
Yungsoo Han, Youngseok Jang, Seungwon Roh et al. (+2 more)
We present DXPR, a depth-based cross-modal place recognition (CMPR) framework that uses
vision foundation models (VFMs) to match
monocular camera queries against a
LiDAR map without modality-specific encoders. This enables
robots and autonomous vehicles to robustly localize using only cameras within pre-built
LiDAR maps, even under severe seasonal, weather, and illumination changes. The key idea is to convert both camera images and
LiDAR scans into a unified depth image representation so that a single VFM backbone with an aggregation head can learn modality-invariant global descriptors. To make pairwise metric learning faithful to scene geometry, we introduce a geometry-aware overlap miner: after cross-modal scale
alignment of camera and
LiDAR depth, we forward-warp measurements between views to compute a pixel-level overlap score. This score relabels ambiguous pairs and adaptively modulates the positive margin in a multi-similarity loss to avoid overfitting on weakly overlapping views. Extensive experiments on KITTI odometry and Boreas demonstrate strong performance and robustness across seasons, weather, and day/night. On KITTI, DXPR achieves near-perfect Recall@1 on most sequences and outperforms prior CMPR baselines. On Boreas, DXPR achieves intra-sequence performance on par with a strong single-modal baseline (DINOv2-SALAD), while showing clear improvements in the more challenging inter-sequence setting. Compared with RangeBEV, our method consistently performs better in both intra- and inter-sequence evaluations, demonstrating robustness under diverse seasonal and illumination changes.
Read the full abstract ▾
All authors Yungsoo Han, Youngseok Jang, Seungwon Roh, Jeongyeon Seo, H. Jin Kim
Computer Vision
Concentrate After Imagination: Text-Conditioned Evidence Grounding for Partially Relevant Video Retrieval
Shuaiqi Cheng, Siyu You, Yanbi Wu et al. (+3 more)
Partially Relevant
Video Retrieval (PRVR) retrieves untrimmed
videos when queries describe only short moments. Although recent methods improve local representations, uncertainty modeling, and global context, final ranking often still trusts the strongest local response; a coincidentally similar fragment can therefore produce an unsupported peak. We identify this failure as the query-agnostic concentration bottleneck and propose TRACE, a score-level evidence verification operator for PRVR. Given a query and global
video registers, TRACE activates query-relevant registers, routes their support to frame-level evidence, and smoothly marginalizes alternative query-to-register-to-frame paths before localized temporal selection. Unlike representation-level feature fusion, TRACE uses this evidence only as a query-conditioned residual calibration of the original local score. On ActivityNet Captions, Charades-STA, and TVR, TRACE achieves the best SumR on all three
benchmarks and improves the DreamPRVR backbone by 1.2, 1.1, and 1.5 points, respectively. Ablation, routing-corruption, hard-negative, and cross-backbone transfer analyses support the interpretation that the gains arise from query-conditioned evidence verification rather than a generic score offset.
Read the full abstract ▾
All authors Shuaiqi Cheng, Siyu You, Yanbi Wu, Yuxi Chen, Jiahao Zhang, Xuming Hu
Computer Vision
Prior-free relative 6D pose estimation of multiple object instances
Behdad Khodabandehloo, Andrea Caraffa, Davide Boscaini et al. (+1 more)
Object 6D pose estimation formulations have progressively reduced reliance on object-specific priors, evolving from explicit 3D models to multi-view object captures to single reference images. We take this progression to its extreme by introducing prior-free relative 6D pose estimation, which lifts the assumption of knowing which object is to be posed within the scene. This novel setting aims to estimate the relative poses of multiple instances of an unknown object within the same image, without requiring CAD models, templates, or reference images. We solve this by formulating a novel method (PROSE) that finds coarse correspondences between object instances using
multimodal foundation features, thus requiring no training. We refine these correspondences by imposing cycle consistency across tuples of instances, and leverage the resulting globally consistent correspondences to estimate the relative 6D pose between any pair of instances. To enable systematic evaluation, we design a novel
benchmark (PRENCH) built from three multi-instance BOP
datasets and enriched with task-specific metadata. PROSE consistently outperforms baselines obtained by adapting state-of-the-art single-image methods to the proposed setting, while requiring neither task-specific supervision nor additional learned components. Project website: https://tev-fbk.github.io/PROSE/
Read the full abstract ▾
All authors Behdad Khodabandehloo, Andrea Caraffa, Davide Boscaini, Fabio Poiesi