Computer Vision
Beyond Token Accuracy: Prioritizing What Matters for Visual Reconstruction
Zhicheng Liu, Zhouxiang Zhao, Chenliang Wu et al. (+2 more)
Tokens have become a unified interface for
multimodal foundation models, making visual-
token communication a natural paradigm for efficient image delivery. However, existing methods typically rely on static policies that cannot jointly adapt to image content and channel conditions. Moreover, their
token-level utility objectives do not necessarily translate into improved image reconstruction quality. In this paper, we propose AdapToC, an adaptive, reconstruction-oriented visual-
token communication framework. At the transmitter, an adaptive selector jointly models image content, channel state, and communication budget to perform instance-wise resource allocation. Rather than using a fixed
token rate and protection policy, it dynamically determines how many
tokens should be transmitted and assigns different protection levels according to
token importance and current channel conditions. At the receiver, an adaptive MaskGIT receiver incorporates channel reliability into contextual
token modeling. It distinguishes
tokens with different reliability levels, preserves high-confidence observations, corrects potentially corrupted
tokens, and iteratively reconstructs missing content from the received evidence and global visual context. By co-designing
token quantity, unequal protection, and reliability-aware recovery for image-level reconstruction quality, AdapToC achieves a peak mean PSNR gain of 4.20 dB over the strongest static baseline under matched communication costs and state-of-the-art performance among the evaluated visual-
token communication methods.
Read the full abstract ▾
All authors Zhicheng Liu, Zhouxiang Zhao, Chenliang Wu, Zhaohui Yang, Zhaoyang Zhang
Computer Vision
Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis
Keerthi Kaashyap, Dennis Anthony, Akshay Krishnan et al. (+5 more)
This paper examines the role of Novel View Synthesis (NVS) in
geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view
geometric representations. Yet, existing encoder-based NVS methods yield poor representations. This is not because of a lack of supervisory signal, but rather due to inconspicuous architectural choices: spatially expressive decoders that dilute representational capabilities of the scene encoder, and low-level pixel-space targets that hinder feature learning. We present SNAP, a self-supervised encoder-decoder
transformer that addresses both through a pose-conditioned local decoder and a latent-space reconstruction objective. SNAP is task agnostic, and we show that it is competitive with special-purpose geometry-supervised methods. SNAP also performs competitively against self-supervised representations across five tasks: visual localization, pose estimation, point correspondence, depth estimation, and
robot manipulation. Remarkably, SNAP's patch features exhibit emergent viewpoint invariance that approaches heavily supervised models despite lower compute and data budgets. Under camera shifts where standard 2D representations collapse, SNAP degrades more gracefully, revealing that restricting decoder expressivity actively prevents the suppression of transferable
geometric structure. https://snap-nvs.github.io
Read the full abstract ▾
All authors Keerthi Kaashyap, Dennis Anthony, Akshay Krishnan, Nhi Ngoc Nguyen, Jeremy Collins, James Hays, Shreyas Kousik, Animesh Garg
Computer Vision
MoSE3: Learning World-Space SE(3) at Every Pixel
Jiahuan Cheng, Zhiyi Li, Tian Xia et al. (+3 more)
Dense 3D point tracking has been a prominent paradigm for modeling motion in dynamic scenes, but a point track is just a 3-DoF translation curve per pixel: it captures where pixels go, not the rotation of the underlying part, nor which pixels move together as one body. We propose MoSE3, the first feed-forward model that predicts dense SE(3) motion from monocular RGB
video, producing full 6-DoF rigid transforms at every pixel in world space. Per-pixel SE(3) motion offers a richer view of how a scene moves: rotation, translation, and grouping all at once. Directly predicting SE(3) is challenging: rotations lie on a curved
manifold that is ill-suited to Euclidean regression, and annotations for SE(3) are particularly difficult to acquire. To address these challenges, MoSE3 predicts per-pixel SE(3) through two jointly learned intermediates, 3D point tracks and rigidity
embeddings, and recovers SE(3) by differentiably fitting transforms within each soft rigid cluster, enabling end-to-end prediction and supervision. To close the data gap, we introduce Art-Kubric, a large-scale synthetic
dataset with dense SE(3) and rigidity labels for articulated objects with rich physical interactions. MoSE3 achieves state-of-the-art SE(3) estimation at pixel, part, and object levels on both rigid and articulated
benchmarks, and state-of-the-art average 3D point tracking accuracy across three
datasets, while showing strong generalization to real-world
videos despite being trained solely on synthetic motion data.
Read the full abstract ▾
All authors Jiahuan Cheng, Zhiyi Li, Tian Xia, Ruojin Cai, Yilun Du, Qianqian Wang
Computer Vision
4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes
Ruihong Shen, Žiga Kovačič, Peter Kulits et al. (+6 more)
We introduce 4DCodeBench, a
benchmark for 4D inverse graphics through code generation, in which
agents reconstruct dynamic scenes from
video as executable graphics programs. To accomplish this,
agents must translate visual observations into compact representations of scene structure and dynamics, by implementing abstractions such as physical simulations to reproduce complex behavior. To evaluate this capability, we curate a set of real-world
videos and construct synthetic scenes spanning diverse physical phenomena, including deformation, fluid flow, and fracture. We perform extensive benchmarking of frontier models, finding that strong static reconstruction capabilities do not yet translate into reliable reconstruction of complex dynamics. 4DCodeBench provides a testbed for tracking progress toward
agents that can interpret the dynamics of the world through code. Our
benchmark is available at https://github.com/4DCodeBench/4DCodeBench
Read the full abstract ▾
All authors Ruihong Shen, Žiga Kovačič, Peter Kulits, Xingrui Wang, Zizhang Li, Joshua B. Tenenbaum, Alan Yuille, Jieneng Chen, Jiajun Wu
Computer Vision · Cross-listed from cs.LG
What Should World Models Forget? Stratified Retention for Continual Adaptation
Nishit Anand, Ramani Duraiswami and Dinesh Manocha
Continual learning treats degradation on previously seen data as evidence of failure, a convention inherited from settings with a stationary prediction target, where a correct label remains correct indefinitely. World models do not satisfy this condition. Their prediction target is the environment, which changes, so knowledge that was accurate when acquired may later become false, and discarding it is required behavior rather than a defect. Non-stationary ground truth is well studied in the concept drift literature and in the temporal factuality of
language models, but has not been formulated for world models, which are distinctive in that they also encode knowledge that must never be revised. We argue that continual world models require retention stratified by invariance timescale, separating invariants such as physics and object permanence, which must never be revised, from instance-level facts that should be revised as soon as the environment changes. Standard forgetting metrics cannot distinguish a world model that has correctly revised outdated knowledge from one that has suffered catastrophic forgetting, and consequently rank a frozen model highest, while existing physical-
reasoning benchmarks evaluate only frozen checkpoints. We propose differential retention, which reports invariant regression testing across the adaptation stream jointly with revision latency, without aggregation.
Read the full abstract ▾
Computer Vision
Decoding the Functional Roles of Register and High-Norm Patch Tokens in Vision Transformers
Neel Varma, Andrew Rufail, Dipika Khullar et al. (+1 more)
Self-supervised
Vision Transformers (ViTs), such as DINOv2, learn rich visual representations, but the functions of their internal
tokens remain poorly understood. Recent architectures introduce dedicated register
tokens to reduce high-norm out- lier patch
tokens that emerge in background re- gions, yet the semantic and functional roles of both
token types have not been fully established. In this paper, we analyze these roles by training sparse autoencoders (SAEs) on register-
token and outlier-
token activations in DINOv2. Using an automated interpretability pipeline, UMAP clus- tering, and CLIP-space cross-checks, we find that register-
token features are more strongly associ- ated with high-level semantic concepts. Outlier-
token features, by contrast, are more often associ- ated with lower-level structural, background, and texture-dominant patterns. Causal ablations fur- ther reveal a substantial functional asymmetry: disrupting top-activating register-derived features produces a 48.17% drop in representation cosine similarity, whereas disrupting outlier-derived fea- tures produces only a 0.31% drop. Together, our results provide evidence for
token specialization in self-supervised ViTs.
Read the full abstract ▾
All authors Neel Varma, Andrew Rufail, Dipika Khullar, Vasu Sharma
Computer Vision
FlowHMR: Physically Plausible Motion Capture from Video
Zhanke Wang, Chengfeng Zhao, Qing Shuai et al. (+8 more)
We present FlowHMR, a framework for recovering physically plausible global 3D human motion from monocular
video. Previous learning-based methods typically regress human motion directly from
video and train the network with
geometric supervision. However, recovering human motion from monocular
video is inherently ambiguous in depth, and direct regression tends to collapse toward an averaged solution. Moreover, the recovered motions are not guaranteed to be physically plausible, so physics-based tracking of them often fails. To address these challenges, we formulate
video motion capture as a
video-conditioned motion generation problem and first pretrain a flow matching model for this task. Given an input
video, the pretrained model generates diverse motion candidates, but not all of them are faithful to the
video or physically trackable. We therefore post-train the model using Group Relative Policy Optimization (GRPO) with two rewards. A fidelity reward encourages consistency with the input
video. A tracking reward favors motions that a physics-based controller can track successfully. Together, these rewards shift the model's output preference, so the post-trained model stays faithful to the input
video while producing more physically plausible motion. We further introduce Wild-4K, a large and diverse
dataset of about 4K internet
videos, for evaluating human motion recovery in the wild. Qualitative and quantitative experiments on Wild-4K show that our method outperforms state-of-the-art methods in overall motion fidelity and achieves a physical tracking success rate of 82.47%, compared with 62.82% for the strongest baseline, GVHMR.
Read the full abstract ▾
All authors Zhanke Wang, Chengfeng Zhao, Qing Shuai, Jingzhong Lin, Heng Li, Zeyu Ling, Yuxin Wen, Jing Li, Di Kang, Chunchao Guo, Linchao Bao
Computer Vision
SigLIP2 for aerial fire risk classification
Yunus Serhat B{\i}çakç{\i}
We examine the transfer of a pretrained SigLIP2 image encoder to seven class fire risk classification from aerial imagery. We introduce a reproducible partition of the public FireRisk training mirror and an implementation that records data provenance, preprocessing and model selection. Two initial runs compare a frozen encoder probe with full model adaptation. On the validation partition, full adaptation reaches 63.05% accuracy and 58.94% macro F1, compared with 55.95% and 50.19% for the probe. Both runs use one training seed and select their checkpoint on the same validation partition. These development results support further evaluation of SigLIP2 but do not establish performance on an independent test set or unseen regions. The accompanying code provides a common framework for repeated experiments and comparisons with additional visual encoders.
Read the full abstract ▾
Computer Vision
ProAR: Learning Prospective Reasoning with Autoregressive Video Models
Linghui Shen, Tinghui Zhu, Sheng Zhang et al. (+1 more)
Autoregressive (AR)
video models excel at causal generation, but their reliance on next-chunk prediction confines them to a short-sighted, reactive paradigm. This limitation is particularly consequential for
reasoning-oriented generation, where achieving a target outcome through valid intermediate states matters more than local visual plausibility. To address this challenge, we propose Learning Prospective
Reasoning with Autoregressive
Video Models (ProAR), a novel framework that transforms autoregressive
video generation into a goal-oriented
reasoning process. ProAR introduces two key components: (1) To anchor generation to the long-range outcome, we integrate goal-frame prediction into the autoregressive loop via an asymmetric
attention mask, enabling the predicted goal frame to guide the generation of intermediate states without being disrupted by them. (2) To guide short-range transitions, we introduce future representation self-
alignment to encourage current hidden states to anticipate upcoming temporal dynamics. By leveraging teacher-forcing in AR training, we extract clean future representations in a single forward pass and align current representations with them using a lightweight, training-only predictor. Together, these two mechanisms seamlessly combine explicit, sparse target supervision with implicit, dense step-wise guidance, promoting coherent, goal-directed
reasoning progress with modest computational cost. Experiments show that ProAR's complementary components consistently improve performance across diverse visual
reasoning benchmarks. The framework proves highly training-efficient, surpassing fully trained standard AR baselines using only 25% of the training steps. This paradigm also demonstrates promising applicability to embodied
reasoning tasks.
Read the full abstract ▾
All authors Linghui Shen, Tinghui Zhu, Sheng Zhang, Muhao Chen
Computer Vision
On-Board Anomaly Detection for Efficient Marine Environmental Monitoring
Thomas Goudemant, Clotilde Szywala, Benjamin Francesconi et al. (+4 more)
Marine ecosystems are impacted by various threats such as oil spills, algal blooms, and sediment floods, which disrupt habitats, wildlife, and human activities. Advances in satellite imagery and Artificial Intelligence (AI) have enhanced our capabilities for early detection and mitigation of such hazards. In this paper, we propose a marine event detection pipeline for Earth observation satellites equipped with multi- or hyperspectral sensors. Our approach includes a self-supervised neural network encoder that compresses satellite images into a reduced latent space, enabling efficient onboard processing. A machine learning anomaly detection model identifies deviations from normal sea patterns to detect environmental anomalies. We compare its performance against traditional
algorithms such as Isolation Forest, One-Class Support Vector Machine and Local Outlier Factors. Our lightweight, resource-efficient pipeline is optimized for deployment on satellites with limited computational resources, ranging from embedded CPUs to AI hardware accelerators. By prioritizing the transmission of critical information, our solution enhances system responsiveness and optimizes satellite communication bandwidth. Demonstrated through current integration across multiple missions, including European Space Agency's (ESA) Phisat-2 mission and Microsoft/Thales Alenia Space IMAGIN-e mission, our pipeline aims to improve marine environmental monitoring by providing timely alerts and efficient data reduction.
Read the full abstract ▾
All authors Thomas Goudemant, Clotilde Szywala, Benjamin Francesconi, Michelle Aubrun, Yves Bobichon, Marjorie Bellizzi, Adrien Girard
Computer Vision
LoGo: Local-Global Rewards for Consistent Long-Horizon Video Generation
Ziqi Ma, Shreya Sharma, Mohamed El Banani et al. (+8 more)
Camera-controlled
video models are rapidly advancing toward long generation horizons and complex camera control. A key failure mode is 3D inconsistency: as the camera moves, objects lose permanence and scene structures shift. Existing post-training techniques, which assign a single scalar reward to the entire generation, are poorly suited to correcting these inconsistencies over long horizons. We introduce LoGo, which blends global and spatially localized rewards for camera-controlled
video models. The local reward provides fine-grained credit assignment, which substantially improves 3D consistency, while the global reward preserves camera following and
video quality. Across three base models, LoGo shows a clear advantage on DL3DV and TrajectoryBench, a new
benchmark for long-horizon, complex-camera-control generation that current evaluations lack. LoGo effectively reduces local object shifts, artifacts, and global scene changes, illustrating the importance of credit assignment in post-training
video models. Project website: https://ziqi-ma.github.io/logo-website/
Read the full abstract ▾
All authors Ziqi Ma, Shreya Sharma, Mohamed El Banani, Katja Schwarz, Chongjie Ye, Chao-Yuan Wu, Li Fei-Fei, Ben Mildenhall, Georgia Gkioxari, Justin Johnson, Gowthami Somepalli
Computer Vision
Yiqi Liu, Ruifeng Yuan, Yang Wang et al. (+7 more)
Physical fidelity has received increasing
attention in world models and
video generation, yet how
video representations encode physical information remains less understood. We introduce the World
Embedding Benchmark, comprising 8,000 controlled simulation cases from 80 families spanning fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism. Each case pairs a rendered
video with simulation-derived physical annotations, supporting three complementary tasks: text-
video retrieval, physical-property regression, and multiple-choice
video-description pair classification. We use these tasks to distinguish cross-modal physical
alignment from the recoverability of quantitative physical information. Evaluated pre-trained omnimodal
embedding models show weak
retrieval and near-chance within-family pair classification, while lightweight probes recover useful physical information from frozen
video embeddings. Continual contrastive training with physics-specific
video-text pairs improves
retrieval and pair classification but degrades physical-property regression, revealing a trade-off between
alignment and quantitative information recoverability. Finally, we use the
embeddings to retrieve reference
videos for
retrieval-augmented generation with MiniMax-H3. Retrieved references improve the physical fidelity of generated
videos, with stronger
retrieval models yielding larger gains in our experiments. Together, these findings highlight the need to evaluate physical
alignment and property recoverability jointly, and demonstrate the utility of physical representations for improving
video generation.
Read the full abstract ▾
All authors Yiqi Liu, Ruifeng Yuan, Yang Wang, Long Li, Fengyu Cai, Hou Pong Chan, Jialin Yu, Hao Zhang, Chenghua Lin, Chenghao Xiao
Computer Vision · Cross-listed from cs.AI
Video-Time Priors for EEG-fNIRS Emotion Regression on Familiar Videos
Minghao Kong, Ying Gao, Jiurun Chen et al. (+4 more)
Do viewer-specific physiological signals improve continuous emotion regression beyond the stimulus-driven response shared across viewers? We examine this question by predicting emotion ratings for new viewers watching
videos that also appear in the training set. As a stimulus-only baseline, we define a
video-time prior:the smoothed median of training labels for each
video at each second. We then combine this prior, using a fixed fusion weight, with a graph-based model of electroencephalography (
EEG) and functional near-infrared spectroscopy (fNIRS) signals. On MER-PS 2026, five-fold cross-validation grouped by participant over 24 participants yields a mean absolute error (MAE) of 29.01 for fusion, compared with 29.06 for the prior alone. On an external cohort of four participants excluded from training, the corresponding MAEs are 27.72 and 28.04. In this cohort, most of the MAE reduction relative to a global-constant baseline comes from the prior, that is, from
video identity and playback time alone.Fusion lowers MAE for three of the four held-out participants and for nine of the fifteen
videos. The added value of physiological signals is therefore small and varies across participants and
videos. Because the model uses physiological signals recorded after each prediction time, it supports only offline prediction. Generalization to unseen
videos remains untested.
Read the full abstract ▾
All authors Minghao Kong, Ying Gao, Jiurun Chen, Muyao Chen, Zheng Ge, Jianxin Wu, Rongjie Wang
Computer Vision
DEPICT: Scoring Text-to-Image Alignment by Answer Agreement
Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes et al. (+2 more)
Image-text
alignment is a core problem in computer
vision with applications in caption evaluation,
hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations. Recent work addresses this by fine-tuning evaluators on preference data or by prompting a
vision-
language model, either holistically with the caption or with decomposed verification questions. However, existing approaches fall short: fine-tuned metrics remain bound to one backbone and training distribution; holistic metrics miss fine-grained details; and decomposed metrics rely on a fixed-YES assumption that penalizes faithful images whenever that assumption fails. In contrast, we propose DEPICT, a training-free metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighting questions by how decisively the caption determines them. By replacing fixed references, our agreement rule increases negation accuracy from 19% to 88%. To recover the context lost during decomposition, DEPICT merges this agreement score with a holistic score. We evaluate DEPICT on five
benchmarks and eleven backbones from three model families and find that it surpasses all training-free metrics and exceeds fine-tuned evaluators on two out of three human-correlation
benchmarks.
Read the full abstract ▾
All authors Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins
Computer Vision
ManifoldSplat: Language-Guided Semantic Shape Editing of 3D Gaussian Head Avatars
Antonio Canela and Jordi Sànchez-Riera
High-fidelity 3D head avatars have reached near-photorealistic quality. While recent methods enable text-driven manipulation, they struggle to provide fine-grained localized control, often entangling features or lacking
geometric consistency. Modifying geometry through natural
language currently requires slow per-
prompt optimization or compromises identity and rigging. We present ManifoldSplat, the first end-toend framework for
language-guided semantic shape editing of animatable 3D Gaussian Splatting avatars reconstructed from monocular
videos. By performing edits within the structured FLAME
manifold rather than directly optimizing an unstructured Gaussian cloud, we strictly preserve identity and animation. We introduce DeltaRegion, a per-region disentangled Conditional Variational Autoencoder (CVAE) delivering feedforward shape deltas, alongside a refining stage to recover view-consistent details. ManifoldSplat reconstructs and edits an avatar in ~90 seconds on a consumer GPU, rendering at ~800 FPS. Extensive evaluations demonstrate our approach sets a new state-of-the-art in localized
prompt alignment,
geometric coherence, and identity preservation. Project page and code: https://a-canela.github.io/manifoldsplat/
Read the full abstract ▾
Computer Vision
Rethinking What to Cache in Few-Step Diffusion Transformers: Solver-Aware Target Selection
Shuo Yang, Lihao Fang, Yi Zhang et al. (+5 more)
Diffusion Transformers (DiTs) can generate high-quality images and
videos, but generating each sample requires multiple costly DiT forward passes. Two common ways to accelerate DiT sampling are step
distillation, which reduces the number of sampling steps, and caching, which skips some DiT evaluations by reusing a tensor computed at an earlier step. Most caching methods decide in advance which tensor to reuse. After
distillation, adjacent sampling steps are farther apart. Reusing a tensor across this larger gap introduces more error, so choosing what to cache becomes especially important. We therefore introduce AutoTarget, a method that chooses the cached tensor for a given model, solver, and reuse schedule. AutoTarget uses a small set of runs without cache reuse to measure the error caused by reusing each candidate tensor, then selects the candidate with the lowest error. We also analyze how an error at one reuse step affects the final sample. For Euler sampling, we identify cache targets that produce the same trajectory and show why a stored solver update may not. Experiments on distilled image and
video DiTs show that the best cache target changes with the model, image resolution, and solver. AutoTarget reduces DiT evaluations and retained cache storage. Generation quality remains close to the corresponding uncached run. On the tested PixArt-LCM and FLUX.1-schnell settings, its calibration ranking matches the ranking from held-out cached runs. To help others reproduce the method, we provide its core implementation on GitHub at https://github.com/wali1024-offical/AutoTarget.
Read the full abstract ▾
All authors Shuo Yang, Lihao Fang, Yi Zhang, Haixiang Wang, Xincheng Ye, Shufan Chen, Jipeng Guo, Youqing Wang
Computer Vision
DuoMatching: Joint-Marginal Distribution Matching for Few-Step Video Generation
Jiahao Zhan, Yan Wang, Yongrui Ma et al. (+5 more)
Streaming
video generation has benefited from distribution matching
distillation (DMD), which matches the joint distribution of
video frames to a
video teacher's approximation of the real
video distribution. Although this joint matching mitigates drift during autoregressive rollouts, limitations remain in visual quality and semantic
alignment. To address these limitations, we propose DuoMatching, a distribution matching framework that approximates the real
video distribution through a unified joint-marginal formulation. On top of existing joint matching formulations, the additional marginal matching objective provides dedicated frame-level supervision from an image generator, transferring complementary visual and semantic priors from it. To apply this frame-level supervision in
video generation, we introduce LatentBridge to resolve the latent representation mismatch between the
video student and the image teacher. Latent Variation Sampling further distributes such frame-level supervision across distinct temporal segments, reducing redundancy. Experiments demonstrate that DuoMatching improves visual quality, composition, and semantic
alignment while largely preserving motion dynamics. Human evaluations show overall preference rates above 80% against all evaluated baselines. The project page is available at https://johnzhan2023.github.io/DuoMatching/.
Read the full abstract ▾
All authors Jiahao Zhan, Yan Wang, Yongrui Ma, Qunliang Xing, Ruchang Yao, Runtao Liu, Shijie Zhao, Tianfan Xue
Computer Vision
Feedforward Novel View Synthesis for Heterogeneous Cameras
Meng Wei, Cheng Zhang, Boying Li et al. (+4 more)
Feed-forward novel view synthesis has recently shown promising results from sparse posed images, but most existing methods assume that context and target views share a fixed camera family. This homogeneous-camera assumption breaks in practical multi-sensor systems, where perspective, fisheye, and panoramic cameras may coexist and where the target projection may be unseen during training. We study feed-forward NVS across heterogeneous central cameras and identify a key ambiguity introduced by tokenization: a visual
token aggregates a projection-dependent bundle of pixel rays, while existing camera encodings mainly expose absolute rays or
token-center relations. To address this, we combine
token-center relative Camera Positional Encodings and proposed local raymaps, a
token-level representation that explicitly describes the intra-patch ray distribution summarized by each
token. We further propose projection-aware 2D RoPE, which replaces raw image-grid coordinates with ray-induced angular coordinates so that relative positional
reasoning is aligned across camera projections. Together, these components treat diverse cameras as calibrated samplings of a shared ray space rather than separate visual domains. On ScanNet++ with heterogeneous-camera system, our method improves over camera-conditioned baselines under mixed-camera evaluation and demonstrates zero-shot generalization to panoramic views.
Read the full abstract ▾
All authors Meng Wei, Cheng Zhang, Boying Li, Yihang Chen, Jianmin Zheng, Hamid Rezatofighi, Jianfei Cai
Computer Vision · Cross-listed from cs.RO
XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation
Tingting Du, Ziyao Wang, Guoheng Sun et al. (+1 more)
World action models (WAMs) have advanced
robot control by predicting how observations and actions evolve over time. Despite this progress, RGB and action based future prediction does not explicitly address the spatial understanding needed for
robot manipulation. Existing efforts often add a limited set of spatial prediction tasks through specialized heads or branches, leaving both the range of spatial supervision and the model architecture fragmented. We introduce XGenAct, a world action model that represents RGB observations,
robot actions, metric depth, surface normals, and functional role segmentation as RGB
videos through deterministic codecs. By sampling perception and action tasks during training, XGenAct uses one
video diffusion transformer and one objective to learn temporal prediction across these spaces without modality specific learned heads. On held out RLBench tasks, structured perception training improves average closed loop success over RGB only training, and XGenAct achieves 52% success in the five task external comparison, versus 26% for the strongest evaluated baselines. It also predicts future depth and segmentation more accurately than the evaluated pipelines that generate RGB first and then apply a frozen perception expert.
Read the full abstract ▾
All authors Tingting Du, Ziyao Wang, Guoheng Sun, Ang Li
Computer Vision
ProgressNet: Sketching and Prompting with a Frozen Text-to-Image Model
Arkaprabha Basu, Chaitat Utintu and Yi-Zhe Song
Humans draw progressively: a few strokes, a look at the result, a stroke erased, a
prompt revised. Image generators do not work this way. They typically take a finished sketch and produce the image in a single pass, so every edit starts the picture again, and the models that do keep state across turns are driven by text, cannot take a stroke, and are too slow to draw with. We present ProgressNet, a training-free framework that lets a frozen text-to-image model follow a drawing session as it unfolds: strokes are added and erased, the
prompt is revised, and the image keeps up at about a second per turn. It needs no new parameters because the frozen model already has what a progressive generator needs, a pathway through which the previous turn can be remembered, layers that can carry appearance forward without freezing structure, and an internal signal of how far to trust an unfinished sketch; three inference-time mechanisms (Previous-Concept
Memory, Layer-Selective K/V Injection and Banded Adaptive Control) use each in turn. As a sketch fills in, every existing method degrades, the FID of the FLUX+ControlNet baseline doubling between 10% and 100% completion on FS-COCO, while ProgressNet's barely moves; it maintains strong fidelity and progressive coherence across three sketch domains and is preferred by users over five competitors, most widely on erasure.
Read the full abstract ▾
Computer Vision
Weave Forcing: Compositional Memory Routing for Interactive Long Video Generation
Ziyi Wang, Junchi Yao, Heqian Qiu et al. (+5 more)
Recent advances in autoregressive
video generation have improved temporal consistency over extended durations, yet interactive storytelling requires more than continuous scene extension: a new shot may combine characters and backgrounds from different historical shots. Whole
prompt retrieval can overlook the distinct reference needs of individual components, while directly combining all historical
memories may introduce unrelated visual content. To address these problems, we present Weave Forcing, a training-free framework for compositional
memory reuse in interactive long
video generation. First, we use an LLM for semantic slot routing to decompose user
prompts into character and background descriptions and explicitly select suitable historical references for each component. To isolate the required content, masked
memory weaving uses contrasting
attention maps conditioned on semantic slots to construct refined semantic masks, selectively exposing relevant
tokens from compressed historical KV
memories to guide the generation of the current shot. We further introduce coverage adaptive RoPE to adjust temporal offsets and
memory retention according to no, partial, or full reference coverage, addressing visual artifacts observed when incomplete historical references are positioned close to the current generation. Extensive experiments demonstrate that Weave Forcing improves cross-shot subject and background consistency while maintaining competitive visual quality and text
alignment.
Read the full abstract ▾
All authors Ziyi Wang, Junchi Yao, Heqian Qiu, Wenbo Shi, Chengjiu Wang, Jinyang He, Binkai Hong, Hongliang Li
Computer Vision
Fed-ADApt: Federated Anytime Depth Adaptation for Resource-Aware Medical Image Segmentation
Abhijeet Parida, Zhifan Jiang, Pooneh Roshanitabrizi et al. (+6 more)
Federated learning (FL) enables collaborative training of medical image segmentation models without sharing raw patient data, yet existing approaches assume a homogeneous compute budget across institutions, limiting participation of low-resource sites. We propose Fed-ADApt, a depth-adaptive
federated framework for UNet-based segmentation that jointly addresses low-compute training and inference. Fed-ADApt integrates multi-depth supervision with hierarchical depth-wise aggregation, allowing each site to train according to its local compute budget while contributing to a global model that supports dynamic depth selection at deployment. We evaluated Fed-ADApt on multi-site 2D retinal fundus disc segmentation and 3D
brain tumor segmentation. Across both tasks,
federated collaboration substantially improves robustness under domain shift. Fed-ADApt matched the full-resource FedAvg performance in 3D and achieved competitive 2D performance with a 4.7% average Dice reduction, while reducing average inference cost by 19.5% in 3D and 34.5% in 2D and substantially reducing training cost by 98% at the most constrained sites. Importantly, Fed-ADApt enables low-resource institutions that cannot train full-capacity models to participate in federations while maintaining competitive global performance under a favorable accuracy to efficiency trade-off. By considering training and inference compute budgets, Fed-ADApt provides a practical and equitable solution for
federated medical image segmentation across heterogeneous clinical and edge-enabled imaging environments.
Read the full abstract ▾
All authors Abhijeet Parida, Zhifan Jiang, Pooneh Roshanitabrizi, Austin Tapp, Maria J. Ledesma-Carbayo, Syed Muhammad Anwar, Ziyue Xu, Marius George Linguraru, Holger R. Roth
Computer Vision
UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation
Daikun Liu, Xin Zhan, Teng Wang et al. (+2 more)
We propose UniDynamics, a
diffusion-based framework for future 4D dynamic scenes (RGB, depth, and optical flow) generation from a single event-RGB pair, without requiring long histories or control priors as in existing methods, while explicitly modeling future motion fields. The core idea is to leverage event streams to offer an alternative motion prior for single-RGB extrapolation, and to enforce
geometric and motion constraints throughout generation via
multimodal modeling. Specifically, we design an Event Latent Enhancement (ELE) module to align and enhance event latents into
diffusion-injectable conditioning features, providing robust initial motion priors and reliable texture/structure cues. We further introduce a Perceptual Dynamics Space (PDS) embedded in the multi-scale U-Net, which decouples and adaptively interacts depth and flow while continuously feeding back constraints to appearance features, improving
geometric-motion consistency for physically plausible and spatiotemporally coherent prediction. Experiments on VKitti2 and DSEC demonstrate state-of-the-art performance, producing high-quality, temporally coherent, and 4D-consistent future predictions, especially under challenging high-speed motion blur.
Read the full abstract ▾
All authors Daikun Liu, Xin Zhan, Teng Wang, Xiaoping Wang, Changyin Sun
Computer Vision
A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds
Mozhgan Hadadi, Talukder Z. Jubery, Adarsh Krishnamurthy et al. (+1 more)
Editable 3D models of field-grown crops support high-throughput phenotyping and in silico breeding trials, but building them from scanned point clouds requires organ-level segmentation and fitting. Procedural generators can turn an organ-level parameter set into an analysis-suitable 3D model, but obtaining that set requires hours of manual tuning per plant or segmentation models trained on species-specific labels. We present an automated pipeline that reconstructs procedural maize models from raw 3D point clouds without manual tuning or species-specific training data. A
multimodal vision-
language model (VLM) annotates leaf midlines in rendered orthographic views. Deterministic
geometric algorithms back-project the annotations onto the point cloud, merge them into 3D leaves by cross-view consensus, and grow the midlines to full blades on an orientation-weighted surface graph. Measured organ parameters populate a plant descriptor for a Non-Uniform Rational B-Spline (NURBS)-based procedural model generator. Each leaf surface is then refined against its scan points by differentiable NURBS fitting. The pipeline reached a median whole-plant Chamfer distance of 5.4 mm on 100 genotypically diverse field-grown maize plants from the MaizeField3D
dataset. The reconstructions were closer to the scans than those of an earlier semi-automated pipeline based on manual annotations. The pipeline recovered 1,017 of 1,023 (99.4%) curated reference leaves at an intersection-over-union of at least 0.5 without using those labels as input. These results show that VLM annotations become usable organ-level measurements when downstream
geometric stages can correct them. This makes automated generation of editable 3D plant assets feasible at the scale of modern phenotyping experiments.
Read the full abstract ▾
All authors Mozhgan Hadadi, Talukder Z. Jubery, Adarsh Krishnamurthy, Baskar Ganapathysubramanian
Computer Vision
Preserving Anatomical Continuity: Three-Stage Pipeline for Colon Segmentation in 3D Abdominal CT Scans
Deshan Kalupahana, Sonit Singh, Praveen Ravindran et al. (+1 more)
Accurate colon segmentation from CT images is essential for colorectal disease analysis, yet deep learning based methods often produce disconnected predictions due to complex anatomy. This study introduces a three-stage,
topology-preserving segmentation pipeline to address this issue. The first stage performs initial deep learning-based segmentation, followed by centreline bridging to reconnect disjoint regions and a reconstruction stage to refine continuity. Evaluations on TotalSegmentator and RAOS
datasets using overlap, distance and
topology-based metrics demonstrate improved structural consistency while maintaining segmentation accuracy. The proposed method enhances topological integrity, enabling more reliable colon segmentation for clinical and research applications.
Read the full abstract ▾
All authors Deshan Kalupahana, Sonit Singh, Praveen Ravindran, Arcot Sowmya
Computer Vision · Cross-listed from cs.RO
I2CD: Direct Image-to-Convex Decomposition for Simulation-Ready Collision Geometry
Qian Wang, Liam Merz Hoffmeister, Brian Scassellati et al. (+1 more)
Physics simulators and motion planners require convex collision geometry, yet image-to-3D generative models output dense, frequently non-
manifold visual meshes. Bridging the two today takes a slow, brittle reconstruct-then-decompose pipeline of repair, decimation, and approximate convex decomposition. We present I2CD, which predicts a convex decomposition directly from a single RGB image. Rather than train a new image-to-3D model, I2CD freezes the pretrained Hunyuan3D-2 image-conditioned
diffusion transformer and shape decoder and trains only a lightweight cross-
attention head (38M parameters, under ten GPU-hours) whose learned "convex-slot"
tokens emit the halfplane parameters of $K$ convex polytopes. The output is compact, convex by construction, and loads into physics engines without any post-processing, in ${\sim}0.5$s per image. On $227$ held-out OmniObject3D and Google Scanned Objects instances, I2CD attains the highest volumetric IoU among eight reconstruct-then-decompose pipelines while running $6$-$37\times$ faster end-to-end. In a cross-simulator study in MuJoCo, PyBullet, Genesis, and Isaac Sim, every engine uses I2CD geometry as delivered, whereas raw generated meshes "load" everywhere but are silently replaced by a different collision shape in most cases or need seconds to minutes of per-object preprocessing. On a physical xArm7, I2CD produces planner-ready geometry for a $20$-object cluttered scene in $11$s versus $328$s for the strongest baseline, at comparable pick-and-place execution success ($85$ vs. $90$ of $100$ trials).
Read the full abstract ▾
All authors Qian Wang, Liam Merz Hoffmeister, Brian Scassellati, Daniel Rakita
Computer Vision
Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
Arun Josephraj Arokiaraj, Zekun Wu and Adriano Koshiyama
A targeted
adversarial perturbation can drive a
vision-
language model's (VLM's) teacher-forced training loss for a fixed target caption to near zero, yet the same model, allowed to generate freely, produces the original, correct description with no trace of the target. We call this dissociation the train/inference gap, and give it a precise mechanistic account on Qwen2.5-VL-7B-Instruct using a controlled two-stage PGD attack on 200 held-out COCO images. First, we show that image-level pixel statistics, including a correctly re-implemented, texture-based attackability measure from the CNN robustness literature, have essentially no predictive power over which images are corrupted (best predictor r=-0.050, p=0.484; ridge regression R^2=0.069). Second, using the logit lens, we localise the gap to a single autoregressive step: the rank of the target
token, conditioned on the correct first
token already being generated, is fixed at exactly 3,488 out of 152,064 vocabulary entries for every image and every condition, with zero variance. Third, tracking target-
token rank across all 28 LLM decoder layers reveals that the visual encoder corrupts every image's representation by a comparable margin regardless of eventual outcome, but the
language model decoder then differentially arbitrates: amplifying the corrupted signal for susceptible images and actively suppressing it, past its clean-image baseline, for resistant ones (p<0.001, rank-biserial r=0.579). A linear probe on the merger hidden state separates these two outcomes with AUC=0.858, though we flag a circularity concern in this estimate. Together these results argue that
adversarial robustness in autoregressive VLMs is substantially a property of the
language decoder's prior, not the visual encoder, with direct implications for where faithfulness evaluations and defenses for deployed VLM systems should be targeted.
Read the full abstract ▾
Computer Vision
ChromaGS: Text-Driven Semantic Editing of 4D Gaussian Avatars
Antonio Canela and Jordi Sànchez-Riera
We present ChromaGS, a method for real-time,
language-guided color editing of animatable 3D Gaussian head avatars. Given a trained animatable avatar, users can instantly modify the color of semantic regions through natural
language, with edits applied at render time and no retraining required. Our key insight is to augment each Gaussian primitive with learned soft assignments to semantic regions and decompose colors into region-level base colors and Gaussian-level residuals. This decomposition enables coherent color transfer: modifying a region's base color propagates naturally through all associated Gaussians while preserving fine appearance details encoded in residuals. A two-stage
language pipeline translates text instructions into target colors, supporting both absolute specifications and relative adjustments. Unlike generative editing methods that may introduce unintended modifications, our approach provides deterministic, precisely localized semantic control. Experiments demonstrate faithful appearance preservation and intuitive interaction across diverse subjects. Project page and code are available at: https://a-canela.github.io/chromags/
Read the full abstract ▾
Computer Vision
Depth Hypothesis Guided Iterative Refinement for Event-Image Monocular Depth Estimation
Daikun Liu, Teng Wang and Changyin Sun
Event cameras hold excellent dynamic properties, showing great potential for monocular depth estimation (MDE). However, existing methods mainly improve performance by optimizing contextual features, but still struggle with the ill-posed and nonlinear nature of direct full-depth regression. In this paper, we propose HypoDepth, the first event-image monocular depth iterative refinement framework. By introducing a discrete Depth Hypothesis Volume (DHV), we transform the depth regression problem into a constrained depth search task. Specifically, we construct a 3D cost volume between the DHV features and contextual features and perform a multi-scale correlation search to guide stable residual optimization. This lightweight cost volume enables efficient global-to-local refinement across multi-resolution. Our method outperforms existing approaches on DSEC and MVSEC with state-of-the-art results and strong zero-shot generalization. Meanwhile, our tiny model achieves an excellent balance between accuracy and efficiency, enabling real-time performance on resource-limited devices.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation
Danzel Serrano and Przemyslaw Musialski
Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a
geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a
speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced
alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured
speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real
speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real
speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.
Read the full abstract ▾