Computer Vision
MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes
Luca Bompani, Manuele Rusci, Luca Benini et al. (+2 more)
Modern smart
vision sensors need on-device intelligence to process
video streams, as cloud computing is often impractical due to bandwidth, latency, and
privacy constraints. However, these sensory systems typically rely on ultra-low-power microcontrollers (MCUs) with limited
memory and compute, making conventional
video object detection methods, which require feature storage or multi-frame buffering, unfeasible. To address this challenge, we introduce Multi-Resolution Rescored ByteTrack (MR2-ByteTrack), a
Video Object Detection (VOD) method tailored for MCU-based embedded
vision nodes. MR2-ByteTrack reduces computational cost by alternating between full- and low-resolution inference, while linking detections across frames via ByteTrack and correcting misclassifications through the Rescore
algorithm, which applies probability union rules to aggregate detection confidence scores across frames. We apply our approach to both a CNN-based detector and a
Transformer-based model, demonstrating its generality across architectures with fundamentally different spatial processing. Experiments on ImageNetVID demonstrate that MR2-ByteTrack maintains accuracy, achieving mAP scores of up to 49.0 for the CNN-based models and 48.7 for the
Transformer, while reducing multiply-accumulate operations by as much as 53% for the CNNs and 32% for the
Transformer. When deployed on GAP9, an ultra-low-power RISC-V multicore MCU, our method yields up to 55% energy savings compared to processing only full-resolution images, enabling the first real-time
Transformer-based VOD on an MCU-class embedded
vision node. Code available at https://github.com/Bomps4/Multi_Resolution_Rescored_ByteTrack/tree/IEEE_Access
Read the full abstract ▾
All authors Luca Bompani, Manuele Rusci, Luca Benini, Daniele Palossi, Francesco Conti
Computer Vision
U-SEG: Uncertainty in SEGmentation – A systematic multi-variable exploration
Michael Smith and Frank P. Ferrie
In this study, we explore in depth a few under-studied topics at the intersection of uncertainty estimation and
segmentation. Prior work has shown that the quality of uncertainty estimates can be very sensitive to a range of variables. As one of the main uses of uncertainty estimation is to help identify and deal with prediction errors in practical scenarios, any factors that affect this must be clearly identified. For example, do more challenging domains or different
datasets and architectures result in worse performance when using uncertainty estimates? Can prior frames in a
video sequence in fact provide useful uncertainty estimates comparable to other approaches? Is it possible to combine uncertainty estimation approaches, taking advantage of sample diversity, to get better estimates? Finally, when might it make sense to use an ensemble-based uncertainty estimate over a deterministic network? We address these questions by creating a framework for and executing a large scale study across many variables such as
datasets, backbones, and downstream tasks, for both semantic and panoptic
segmentation. We find that a) the more challenging task of panoptic
segmentation usually results in worse performance while high performance variance between
datasets and backbones indicates that generalization is not guaranteed, b) time series samples can be useful for specific configurations, but in many cases are not worth the cost, c) sample diversity shows the most promise in the downstream task of calibration, but otherwise fails to beat simpler alternatives, d) a deterministic approach is adequate for some downstream tasks, but ensembles allow for significant improvements if the right conditions can be achieved in deployment.
Read the full abstract ▾
Computer Vision · Cross-listed from physics.optics
A Differentiable Ray-Wave Framework for Hybrid Refractive-Diffractive System Modeling and Optimization
Jiazhou Cheng, Margaret Gao, Yixuan Shao et al. (+3 more)
Hybrid optical systems combining refractive and diffractive optical responses have the potential to support new types of optical behavior, but they are difficult to model and optimize due to the disparate spatial scales and physics exhibited by ray and wave phenomena. In this work, we present a differentiable ray-wave framework for modeling hybrid refractive-diffractive optical systems that operates as a plug-and-play module within standard ray tracing pipelines. Our model uniquely applies to both planar and curvilinear diffractive surfaces and accommodates arbitrary scalar
holographic profiles with high spatial frequency responses, with each simulation evaluated at a single wavelength. We analyze ray-wave modeling regimes that optimally account for the spatial frequency properties and spatial
curvature of the diffractive surfaces, and we demonstrate the gradient-based end-to-end optimization of hybrid refractive-diffractive systems featuring planar and conformal diffractive surfaces. We anticipate that these modeling capabilities will enable new classes of hybrid optical systems relevant to computational imaging and display applications.
Read the full abstract ▾
All authors Jiazhou Cheng, Margaret Gao, Yixuan Shao, Chenkai Mao, Tom D. Milster, Jonathan A. Fan
Computer Vision · Cross-listed from cs.GR
3DEditSafe: Defending 3D Editing Pipelines from Unsafe Generation
Nicole Meng, Zheyuan Liu, Meng Jiang et al. (+1 more)
Recent advances in 3D generative editing, particularly pipelines based on 3D Gaussian Splatting (3DGS), have achieved high-fidelity, multi-view-consistent scene manipulation from text
prompts. However, we find that these pipelines also introduce new safety risks when unsafe
prompts produce edits that are propagated and optimized across views. In this work, we study unsafe generation in 3D editing pipelines and show that such behavior can lead to coherent, undesirable Not-Safe-For-Work (NSFW) content in the final 3D representation. To address this, we propose 3DEditSafe, a safety-regularized 3D editing framework that constrains unsafe semantic propagation during optimization. 3DEditSafe combines generation-stage safety guidance with rendered-view 3D safety regularization, safe semantic projection, residue suppression, and mask-aware preservation to steer optimization away from unsafe editing directions. We evaluate our approach on EditSplat scenes using an object-compatible unsafe
prompt benchmark and show that 2D safety guidance alone is not consistently sufficient to prevent unsafe 3D edits. 3DEditSafe reduces unsafe semantic
alignment and view-level attack success rates, while revealing a safety-quality tradeoff in which stronger unsafe suppression can introduce artifacts or reduce unsafe-
prompt fidelity. To our knowledge, this work is the first attempt to study and defend against unsafe generation in text-driven 3D editing pipelines, highlighting the need for safety mechanisms that operate directly on optimized 3D representations.
Read the full abstract ▾
All authors Nicole Meng, Zheyuan Liu, Meng Jiang, Yingjie Lao
Computer Vision
ELDOR: A Dataset and Benchmark for Illegal Gold Mining in the Amazon Rainforest
Kangning Cui, Surendra Bohara, Suraj Prasai et al. (+12 more)
Illegal gold mining in the Amazon rainforest causes deforestation, water contamination, and long-term ecosystem disruption, yet remains difficult to monitor at fine spatial scales. Satellite imagery supports large-scale observation, but often misses small mining-related structures and subtle land-cover transitions, especially under frequent cloud cover. We introduce ELDOR, a large-scale
UAV benchmark for monitoring environmental and landscape disturbance from illegal gold mining in the rainforest. ELDOR contains manually annotated orthomosaic imagery covering over 2,500 hectares, with pixel-level semantic labels for both mining-related activities and surrounding ecological structures. With this unified annotation source, we establish four
benchmark tasks: semantic
segmentation,
segmentation-derived recognition, direct multi-label classification, and class-presence recognition with
vision-
language models. Across these tasks, we compare generic and remote-sensing-specific
segmentation models,
vision foundation model-related
segmentation methods, direct multi-label classification methods, and
vision-
language models under a controlled closed-set protocol. Results show that current methods still struggle with rare small-scale mining structures and fine-grained recovery classes, suggesting the need for context-aware and
multimodal modeling. To support domain analysis and practical use, we further build an interactive explorer for domain experts that provides a unified interface for data exploration and model inference.
Read the full abstract ▾
All authors Kangning Cui, Surendra Bohara, Suraj Prasai, Zishan Shao, Wei Tang, Martin Pillaca, Edwin Flores, Zhen Yang, Gregory Larsen, Evan Dethier, David Lutz, Jean-Michel Morel, Miles Silman, Victor Pauca, Fan Yang
Computer Vision · Cross-listed from physics.optics
Frequency-domain Event-based Imaging for Selective Surveillance
Megan Birch, James Rick, Adrish Kar et al. (+2 more)
Event-based cameras (EBCs) are an attractive sensing modality for surveillance due to their reporting of pixel-level radiance changes with microsecond resolution and high dynamic range, enabling motion extraction while suppressing background. Their asynchronous, sparse output, however, necessitate
algorithms that identify targets in event-space without processing full frames. We introduce Frequency Rate Information for Event Space (FRIES), a neuromorphic processing framework that detects periodicity in events, such as rotor rotation and mechanical vibrations, to discriminate and monitor man-made objects. FRIES first applies a time gate to suppress background and noise, then aggregates events into a pixel-wise activity (e.g., density) map and clusters pixels into regions-of-interest (ROIs). A localized spectral analysis is applied to each ROI to extract dominant frequencies used to distinguish structured object signatures from unstructured background and noise. Discriminated targets are visualized using a Resonant Time Surface (RTS), a frequency-selective method that weights events by their phase coherence with the extracted frequencies, rewarding in-sync content and suppressing out-of-sync clutter. We demonstrate FRIES and RTS in a controlled indoor experiment to recover the rotational frequency of a mechanical chopper and drone rotors against a moving background. We further test these methods on an outdoor data to detect a hovering drone against a realistic treeline. These preliminary results establish frequency-domain event processing as a promising front-end for selective surveillance in neuromorphic pipelines and a complementary surveillance modality, leveraging the high temporal resolution to enable spectral discrimination.
Read the full abstract ▾
All authors Megan Birch, James Rick, Adrish Kar, Jason Zutty, Joseph L. Greene
Computer Vision
PanoWorld: Geometry-Consistent Panoramic Video World Modeling
Le Jiang, Xiangyu Bai, Bishoy Galoaa et al. (+7 more)
We present PanoWorld, a panoramic
video world model that generates geometry-consistent 360$\degree$
video from a single image and a caption. Existing panoramic
video methods optimize primarily for visual realism and do not explicitly constrain the underlying 3D scene state, producing outputs that appear plausible yet exhibit inconsistent depth, broken correspondences, and implausible motion across the spherical surface. We address this gap by framing panoramic
video generation as a geometry- and dynamics-consistent latent state modeling problem rather than pure visual synthesis. Building on a pre-trained perspective
video world model, we introduce two lightweight regularizers: a depth consistency loss against pseudo ground-truth panoramic depth, and a trajectory consistency loss that supervises the 3D world-frame positions of tracked points across time. We further apply spherical-geometry-aware adaptation to the conditioning and positional encoding. We additionally introduce PanoGeo, a unified geometry-aware panoramic
video dataset with consistent depth, trajectory, and
prompt annotations across diverse real and synthetic sources, used for both training and stratified evaluation. Experiments show that PanoWorld improves
geometric consistency over prior panoramic generation methods while maintaining competitive visual realism, establishing that panoramic
video generation must be treated as a
geometric modeling problem to support the holistic spatial understanding requirements of
embodied AI applications. Code is available at https://github.com/ostadabbas/PanoWorld.
Read the full abstract ▾
All authors Le Jiang, Xiangyu Bai, Bishoy Galoaa, Shayda Moezzi, Caleb James Lee, Tooba Imtiaz, Edmund Yeh, Jennifer Dy, Yanzhi Wang, Sarah Ostadabbas
Computer Vision
MorphoHELM: A Comprehensive Benchmark for Evaluating Representations for Microscopy-Based Morphology Assays
Emre Hayir, Lorin Crawford and Alex X. Lu
Microscopy images contain rich information about how
cells respond to perturbations, making them essential to applications like drug screening. To quantify images, researchers often use representation extraction methods, and recent years have seen a proliferation of deep learning methods. While measuring the quality of these representations is essential, evaluation remains fragmented, with each proposed model evaluated on different tasks and
datasets, using custom pipelines and metrics, making it difficult to fairly compare models. Here, we introduce MorphoHELM, a comprehensive open
benchmark for evaluating feature extraction methods for
Cell Painting, the most widely-used morphological profiling assay. MorphoHELM consolidates evaluation standards in the field, extends and corrects them to be more robust, and evaluates on the widest range of methods to date. A defining feature of the
benchmark is that each task is evaluated at different degrees of batch effects (or technical noise), directly quantifying how the ability of methods to detect biological signal degrades as noise increases. Together, these properties enable MorphoHELM to detect trade-offs between methods, and we demonstrate that models that excel at certain kinds of biological signal are weaker at others. We show that no existing model outperforms classic computer
vision analytic strategies across all settings, which remain the strongest general use-case representations. All
datasets, code, and evaluation tools are publicly available at https://github.com/microsoft/MorphoHELM.
Read the full abstract ▾
Computer Vision
ChangeFlow – Latent Rectified Flow for Change Detection in Remote Sensing
Blaž Rolih, Matic Fučka, Filip Wolf et al. (+1 more)
Remote sensing change detection (RSCD) localises changes between two images of the same geographic region. Most state-of-the-art methods are trained with a per-pixel discriminative objective that classifies each spatial location independently. In this scenario, the predicted changed region is not modelled as a coherent whole, so predictions tend to be spatially fragmented. Generative modelling offers a principled solution: by learning a distribution over plausible change masks, it treats the mask as a single object and encourages global consistency. Yet existing generative RSCD methods lag behind strong discriminative baselines, held back by costly pixel-space generation and overly complex conditioning. We introduce ChangeFlow, which reformulates change detection as the generative synthesis of change masks in a compact latent space via rectified flow, guided by a structured yet lightweight bi-temporal conditioning signal. Changeflow yields spatially coherent predictions without sacrificing efficiency: across four binary
benchmarks, SYSU, LEVIR, CLCD, and OSCD, ChangeFlow achieves an average F1 of $80.4\%$, a $1.3$-point gain over the previous best with better efficiency. It also extends to semantic change detection, setting a new state-of-the-art $65.9$ $F_{scd}$ on SECOND. Project page: https://blaz-r.github.io/changeflow_cd
Read the full abstract ▾
All authors Blaž Rolih, Matic Fučka, Filip Wolf, Luka Čehovin Zajc
Computer Vision
Discretizing Group-Convolutional Neural Networks for 3D Geometry in Feature Space
Daniel Franzen, Jean Philip Filling and Michael Wand
Group-convolutional neural networks (GCNNs) are among the most important methods for introducing
symmetry as an inductive bias in deep learning: In each linear layer, GCNNs sample a transformation group $G$ densely and correlate data and filters in different poses (with suitable anti-aliasing for steerable GCNNs) to maintain equivariance with respect to $G$. Unfortunately, applying filters to many data items resulting from this sampling is expensive (even for translations alone, i.e., in ordinary CNNs), and costs grow exponentially with increasing degrees of freedom (such as translations and rotations in 3D), which often hinders practical applications. In this paper, we propose sampling in feature space, i.e., replacing geometrically dense samples with representative samples selected by feature similarity. This decouples
geometric resolution from
memory and processing costs during training and inference, providing a novel way to trade off computational effort and accuracy. Our main empirical finding is that a coarse feature-space sampling already preserves classification accuracy remarkably well, which permits precomputation based on
geometric similarity, accelerating the training of equivariant 3D classifiers substantially.
Read the full abstract ▾
Computer Vision
Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
Arsha Nagrani, Jasper Uijilings, Shyamal Buch et al. (+6 more)
Video reasoning models are a core component of
egocentric and
embodied agents. However, standard
benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate
reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a
benchmark for evaluating complex
egocentric visual
reasoning. We extend recent high-quality
video data sources recorded from
egocentric /
embodied settings with a set of challenging, multi-step
multimodal questions and spatiotemporally-dense human-annotated
reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each
reasoning trace in the
dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.
Read the full abstract ▾
All authors Arsha Nagrani, Jasper Uijilings, Shyamal Buch, Tobias Weyand, Sudheendra Vijayanarasimhan, Bo Hu, Ramin Mehran, David A Ross, Cordelia Schmid
Computer Vision
Multimodal Object Detection Under Sparse Forest-Canopy Occlusion
Nitik Jain and Mangal Kothari
Reliable detection of humans beneath forest canopy remains a difficult remote-sensing challenge due to sparse, structured, and viewpoint-dependent occlusion. This paper presents a
multimodal proof-of-concept pipeline that integrates three complementary approaches: (i) experimental evaluation of
LiDAR returns through vegetation to assess the feasibility of active sensing, (ii) visible–thermal image fusion using a multi-scale transform and sparse-representation framework to enhance human saliency, and (iii) synthetic-aperture image formation via Airborne Optical Sectioning (AOS) to suppress canopy clutter. A YOLOv5 detector is fine-tuned on the Teledyne FLIR thermal
dataset and evaluated on thermal and fused imagery. Results show that the tested terrestrial
LiDAR configuration provides limited penetration for object-level detection, while visible–thermal fusion improves target visibility in low-contrast scenes and AOS enhances ground-plane detection in synthetic forest imagery. The fine-tuned YOLOv5 achieves a mean average precision of $\sim$0.83 on the top three FLIR classes. These findings establish an initial baseline for
UAV-deployable search-and-rescue and surveillance systems operating in forested environments, and motivate future work on dedicated forest
datasets and real-time
multimodal integration.
Read the full abstract ▾
Computer Vision
COPRA: Conditional Parameter Adaptation with Reinforcement Learning for Video Anomaly Detection
Darryl Cherian Jacob, Xinyu Liu, Kai Wang et al. (+1 more)
Vision-
language models (VLMs) have shown strong performance in
video anomaly detection (VAD) while providing interpretable predictions. However, existing VLM-based VAD methods suffer from a fundamental mismatch between training and inference in both data distribution and model configuration. First, most approaches rely on static post-training adaptation, limiting generalization under distribution shifts such as unseen environments or anomaly types. Second, they train VLMs on sparse frames from long
videos, but perform inference on densely sampled short segments, creating inconsistencies between training and testing. To address these limitations, we propose COPRA, a conditional parameter adaptation framework for VLM-based VAD. Instead of fixed
prompts or shared parameter updates, COPRA generates input-specific parameter updates to dynamically adapt a frozen VLM for each
video segment during both training and inference. Experiments show strong performance on standard VAD
benchmarks, consistently outperforming static baselines in both in-domain and cross-domain settings. Moreover, COPRA generalizes beyond VAD to unseen tasks such as multiple-choice
Video Question Answering and Dense Captioning. These results highlight COPRA as an effective weight-space generation framework for scalable, adaptive, and context-aware
video understanding. The code will be released at https://github.com/THE-MALT-LAB/COPRA
Read the full abstract ▾
All authors Darryl Cherian Jacob, Xinyu Liu, Kai Wang, Pan He
Computer Vision · Cross-listed from cs.GR
FFAvatar: Few-Shot, Feed-Forward, and Generalizable Avatar Reconstruction
Thuan Hoang Nguyen, Jiahao Luo, Yinyu Nie et al. (+3 more)
Avatar reconstruction has traditionally relied on per-subject optimization that requires hours of computation or on expensive preprocessing that limits scalability. We introduce FFAvatar, a generalizable feed-forward framework that reconstructs high-quality, animatable 3D Gaussian head avatars from few-shot unposed portrait images in seconds. FFAvatar fuses information from multiple source images into a unified canonical Gaussian representation through Multi-View Query-Former, which is animated via FLAME parameters predicted end-to-end directly from pixels, eliminating the overhead of offline FLAME extraction. We further propose a three-stage training curriculum that achieves both broad generalization and high-fidelity reconstruction: (i) scalable pretraining on extensive
monocular video data with over 1M identities to learn strong generalizable priors; (ii) multi-view fine-tuning on a small but high-quality
dataset of 360-degree captures to enhance
geometric fidelity and extreme-view awareness; and (iii) optional personalization that adapts to specific identities for maximum fidelity within 500 optimization steps. Extensive experiments demonstrate that FFAvatar sets a new standard for identity preservation,
geometric consistency, and animation fidelity. On the NeRSemble
benchmark, it outperforms the state-of-the-art LAM by a substantial 5.5 PSNR gain. Furthermore, FFAvatar enables real-time deployment, reconstructing avatars in 2 seconds without personalization and 10 seconds with personalization, while supporting 49 FPS animation on a single NVIDIA A100 GPU.
Read the full abstract ▾
All authors Thuan Hoang Nguyen, Jiahao Luo, Yinyu Nie, Hao Li, Gordon Guocheng Qian, Jian Wang
Computer Vision · Cross-listed from cs.CY
Beyond Performance Disparities: A Three-Level Audit of Representational Harm in CelebA
Sieun Park and Yuanmo He
Large-scale facial
datasets like CelebA are widely used in computer
vision, yet the cultural biases embedded in their labels remain underexplored.
Fairness research has distinguished representational from allocational harms, but audits of computer
vision datasets have mostly examined categorical labels, leaving open how such harms appear in learned features and model
attention. This paper examines CelebA at three levels:
dataset structure, learned feature weights, and spatial
attention, focusing on how gendered double standards of ageing and beauty are encoded in the data and reproduced in model behaviour. First, hierarchical clustering of 202,599 images shows that the 39 attributes organise into latent trait bundles aligned with cultural archetypes: performative femininity (youth, makeup, adornment) and professional masculinity (ageing, facial hair, formal attire). Female faces, though more often rated attractive overall, incur steep penalties when assigned to ageing or masculine-coded clusters. Second, XGBoost with SHAP analysis reveal gender-specific effects, such as adiposity reducing attractiveness only for females. Third, Grad-CAM finds that predictions for female and younger male subgroups concentrate on mid-face cues, whereas predictions for older males drift toward peripheral cues such as hair and clothing. Older males attain the highest accuracy but the lowest average precision, indicating categorical exclusion of groups outside the
dataset's evaluative templates. Cultural double standards thus pass from media representation into
dataset labels, feature weights, and model
attention, producing two representational harms: hyper-scrutiny of women under a narrow evaluative template, and exclusion of older men from the scheme entirely.
Fairness metrics focused on performance disparities mask both, underscoring the need to address representational harm in
fairness research.
Read the full abstract ▾
Computer Vision
One Pass Is Not Enough: Recursive Latent Refinement for Generative Models
Mehdi Esmaeilzadeh, Alexia Jolicoeur-Martineau, Chirag Vashist et al. (+1 more)
Despite remarkable progress, image generation is far from solved. The dominant metric, FID, conflates sample fidelity with mode coverage and is close to being saturated. Yet a model can still exhibit mode collapse while achieving a low FID, since a handful of sharp, near-duplicate images can outscore a model that faithfully covers the full data distribution. We argue that precision and recall are essential complements to FID, and that because FID is already saturated, the more meaningful goal is to improve diversity and coverage. Achieving high recall requires a model that explicitly prioritizes mode coverage, unlike most generative models, which optimize sample fidelity. We introduce RTM, which replaces the single-pass latent mapping in style-based generators with an iterative refinement process, and show that this consistently improves both quality and diversity. Integrated with Implicit Maximum Likelihood Estimation (IMLE), which optimizes mode coverage by design, RTM achieves the highest precision and recall among current state-of-the-art approaches while maintaining competitive FID, with improvements across CIFAR-10, CelebA-HQ at 256x256, and nine few-shot
benchmarks. RTM also improves StyleGAN2 and StyleGAN2-ADA on CIFAR-10 and AFHQ-v1 at 512x512, demonstrating that the benefit is not specific to IMLE. Unlike flow-matching baselines that achieve competitive FID at the expense of coverage, recursive refinement improves both quality and diversity simultaneously.
Read the full abstract ▾
All authors Mehdi Esmaeilzadeh, Alexia Jolicoeur-Martineau, Chirag Vashist, Ke Li
Computer Vision · Cross-listed from cs.GR
Sound Sparks Motion: Audio and Text Tuning for Video Editing
AmirHossein Naghi Razlighi, Aryan Mikaeili, Ali Mahdavi-Amiri et al. (+2 more)
Motion-centric
video editing remains difficult for large generative
video models, which often respond well to appearance changes but struggle to produce specific, localized actions or state transitions in an existing clip. We introduce Sound Sparks Motion, a training-free framework that enables motion editing in an audio-visual
video generation model by tuning its internal
multimodal conditioning signals at test time. Rather than modifying model weights, our method tunes only two lightweight variables: an audio latent derived from the source
video and a residual perturbation in the text-conditioning. We find that this combination can encourage motion edits that the underlying model often struggles to realize under
prompt-only control. Since there is no direct way to evaluate temporal
alignment between text and motion, we guide the tuning process using a
vision-
language model that provides feedback indicating whether the intended motion appears in the generated
video. This simple supervision yields an effective semantic objective for motion editing, while regularization and perceptual-temporal constraints help preserve content and visual quality. Beyond per-
video tuning, we show that the learned latent controls are transferable across
videos, suggesting that they capture reusable motion-edit directions rather than overfitting to a single example. Our results highlight
multimodal conditioning tuning, particularly through the audio pathway, as a promising direction for motion-aware
video editing, and suggest that test-time tuning can serve as a lightweight probing mechanism that helps reveal latent motion controls embedded in the model's
multimodal conditioning. Code and data are available via our project page: https://amirhossein-razlighi.github.io/Sound_Sparks_Motion/
Read the full abstract ▾
All authors AmirHossein Naghi Razlighi, Aryan Mikaeili, Ali Mahdavi-Amiri, Daniel Cohen-Or, Yiorgos Chrysanthou
Computer Vision
Tianyu Yu, Kechen Fang, Zihao Wan et al. (+5 more)
Most
Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector. While effective, recent analysis suggests this architecture suffers from an
alignment challenge: visual features remain distant from the text space in the initial layers of the LLM, forcing the model to waste critical depth~\cite{zhang-etal-2024-investigating,artzy-schwartz-2024-attend} on superficial modality
alignment rather than deep understanding and complex
reasoning. In this work, we propose Deep Pre-
Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large
language model. Comprehensive experiments demonstrate the effectiveness of DPA. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8
multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. Moreover, by offloading
alignment to the perceiver, DPA achieves a 32.9% reduction in
language capability forgetting over 3 text
benchmarks. We further demonstrate that these gains are consistent across different LLM families including Qwen3 and LLaMA 3.2, highlighting the generality of our approach. Beyond performance, DPA also offers a seamless upgrade path for current VLM development, requiring only a modular replacement for the visual encoder with marginal computation overhead.
Read the full abstract ▾
All authors Tianyu Yu, Kechen Fang, Zihao Wan, Kaidong Zhang, Yicheng Zhang, Jun Song, Bo Zheng, Yuan Yao
Computer Vision · Cross-listed from cs.RO
PhysBrain 1.0 Technical Report
Shijie Lian, Bin Yu, Xiaopeng Lin et al. (+10 more)
Vision-
language-action models have advanced rapidly, but
robot trajectories alone provide limited coverage for learning broad physical understanding. PhysBrain 1.0 studies a complementary route: converting large-scale human
egocentric video into structured physical commonsense supervision before
robot adaptation. Our data engine extracts scene elements, spatial dynamics, action execution, and depth-aware relations, then turns them into question-answer supervision for training PhysBrain VLMs. The resulting physical priors are further transferred to VLA policies through a capability-preserving and
language-sensitive adaptation design. Across
multimodal QA
benchmarks and
embodied control
benchmarks, including ERQA, PhysBench, SimplerEnv-WidowX, LIBERO, and RoboCasa, PhysBrain 1.0 achieves SOTA results and shows especially strong out-of-domain performance on SimplerEnv. These results suggest that scaling physical commonsense from human interaction
video can provide an effective bridge from
multimodal understanding to
robot action.
Read the full abstract ▾
All authors Shijie Lian, Bin Yu, Xiaopeng Lin, Changti Wu, Hang Yuan, Xiaolin Hu, Zhaolong Shen, Yuzhuo Miao, Haishan Liu, Yuxuan Tian, Yukun Shi, Cong Huang, Kai Chen
Computer Vision
ReactiveGWM: Flexible Control and NPC Reactivity in Game World Models
Zeqing Wang, Danze Chen, Zhaohu Xing et al. (+4 more)
Existing game world models typically adopt role-specific interactions, where player and NPC roles are bound to fixed characters. This limits their flexibility in multi-character games, where different characters may receive external control while NPCs must react to interactions triggered by players. This setting raises two key challenges: how to flexibly assign control roles to individual characters, and how to support direct player control and reactive NPC behavior within a unified model. These challenges are particularly pronounced in shared-view 2D games, where multiple, potentially visually identical characters share the same viewpoint, making camera cues insufficient to distinguish their roles. To address these challenges, we introduce ReactiveGWM, a reactive game world model that flexibly assigns control modes at initialization and jointly simulates externally controlled players and reactive NPCs. Specifically, ReactiveGWM introduces Spatial Role Binding, which grounds learned character handles to their corresponding regions in the initial frame using instance masks. Building on these handles, Unified Agency Conditioning unifies heterogeneous control signals across characters by encoding player actions and conditional NPC rules into character-specific control-
token groups. Each group is then bound to its corresponding character handle, enabling the model to apply each control signal to its designated character. Meanwhile, causal self-
attention restricts temporal context to the current and preceding latent frames when generating player actions and NPC responses. Experiments on two multi-character 2D games demonstrate that ReactiveGWM supports flexible character control across different player/NPC role assignments while jointly generating accurate player-controlled behaviors and reactive NPC responses, enabling more configurable and richer multi-character interactions.
Read the full abstract ▾
All authors Zeqing Wang, Danze Chen, Zhaohu Xing, Zizhao Tong, Yinhan Zhang, Xingyi Yang, Yeying Jin
Computer Vision · Cross-listed from eess.IV
From Full and Partial Intraoral Scans to Crown Proposal: A Classification-Guided Restoration Assistance Pipeline
Rabin Kunwar, Dikshya Parajuli, Rujal Acharya et al. (+12 more)
Single-unit crown restoration is among the most common procedures in clinical dentistry, with CAD/CAM workflows now designing crowns directly from intraoral scans. Partial scans are often preferred over full-arch scans for single-unit cases due to fewer stitching errors, yet most
segmentation networks trained on full arches fail on partial scans, while end-to-end generative crown methods often produce over-smoothed surfaces that lose occlusal detail. We propose an end-to-end pipeline that takes a raw intraoral scan and target FDI tooth number as input and outputs an initial, patient-specific crown proposal for clinician refinement. The pipeline has three phases: (I) data preparation and pose standardization; (II)
segmentation routed by scan type; and (III) crown proposal generation via context-aware
retrieval and Blender-based fitting. We address partial-scan
segmentation through a classify-then-align strategy: a DGCNN classifier categorizes the scan into one of five anatomical types, then coarse-to-fine RANSAC+ICP registration standardizes the jaw coordinate frame, followed by graph-cut optimization to refine tooth-gingival boundaries. Trained on 1,958 partial scans, the pipeline achieves macro-average DSC 0.9249, Recall 0.8919, and Precision 0.9615 across 17 semantic classes; a fine-tuned full-arch model reaches DSC 0.9347. The prepared tooth and its mesial and distal neighbors achieve DSC 0.9468-0.9569 with sub-millimeter Centroid Errors (0.2666-0.2774 mm). These centroids anchor a
retrieval module using DGCNN
embeddings and cosine similarity over neighboring and opposing teeth, followed by spline-guided
alignment and Blender Python API refinement. The pipeline produces a preliminary crown shell in 2.5-3.5 minutes, offering a practical alternative to end-to-end generative approaches.
Read the full abstract ▾
All authors Rabin Kunwar, Dikshya Parajuli, Rujal Acharya, Romik Gosai, Prince Panta, Kundan Siwakoti, Shuvangi Adhikari, Saugat Kafley, Louis Digiorgio, Amit Regmi, Akio Tanaka, Masahiko Inada, Yuriko Komagamine, Kennta Kashiwazaki, Manabu Kanazawa
Computer Vision
EntityBench: Towards Entity-Consistent Long-Range Multi-Shot Video Generation
Ruozhen He, Meng Wei, Ziyan Yang et al. (+1 more)
Multi-shot
video generation extends single-shot generation to coherent visual narratives, yet maintaining consistent characters, objects, and locations across shots remains a challenge over long sequences. Existing evaluations typically use independently generated
prompt sets with limited entity coverage and simple consistency metrics, making standardized comparison difficult. We introduce EntityBench, a
benchmark of 140 episodes (2,491 shots) derived from real narrative media, with explicit per-shot entity schedules tracking characters, objects, and locations simultaneously across easy / medium / hard tiers of up to 50 shots, 13 cross-shot characters, 8 cross-shot locations, 22 cross-shot objects, and recurrence gaps spanning up to 48 shots. It is paired with a three-pillar evaluation suite that disentangles intra-shot quality,
prompt-following
alignment, and cross-shot consistency, with a fidelity gate that admits only accurate entity appearances into cross-shot scoring. As a baseline, we propose EntityMem, a
memory-augmented generation system that stores verified per-entity visual references in a persistent
memory bank before generation begins. Experiments show that cross-shot entity consistency degrades sharply with recurrence distance in existing methods, and that explicit per-entity
memory yields the highest character fidelity (Cohen's d = +2.33) and presence among methods evaluated. Code and data are available at https://github.com/Catherine-R-He/EntityBench/.
Read the full abstract ▾
All authors Ruozhen He, Meng Wei, Ziyan Yang, Vicente Ordonez
Computer Vision
ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both
Ziyu Guo, Rain Liu, Xinyan Chen et al. (+1 more)
Visual
reasoning, often interleaved with intermediate visual states, has emerged as a promising direction in the field. A straightforward approach is to directly generate images via unified models during
reasoning, but this is computationally expensive and architecturally non-trivial. Recent alternatives include agentic
reasoning through code or tool calls, and latent
reasoning with learnable hidden
embeddings. However, agentic methods incur context-switching latency from external execution, while latent methods lack task generalization and are difficult to train with autoregressive parallelization. To combine their strengths while mitigating their limitations, we propose ATLAS, a framework in which a single discrete 'word', termed as a functional
token, serves both as an agentic operation and a latent visual
reasoning unit. Each functional
token is associated with an internalized visual operation, yet requires no visual supervision and remains a standard
token in the tokenizer vocabulary, which can be generated via next-
token prediction. This design avoids verbose intermediate visual content generation, while preserving compatibility with the vanilla scalable SFT and RL training, without architectural or methodological modifications. To further address the sparsity of functional
tokens during RL, we introduce Latent-Anchored GRPO (LA-GRPO), which stabilizes the training by anchoring functional
tokens with a statically weighted auxiliary objective, providing stronger gradient updates. Extensive experiments and analyses demonstrate that ATLAS achieves superior performance on challenging
benchmarks while maintaining clear interpretability. We hope ATLAS offers a new paradigm inspiring future visual
reasoning research.
Read the full abstract ▾
All authors Ziyu Guo, Rain Liu, Xinyan Chen, Pheng-Ann Heng
Computer Vision
RefDecoder: Enhancing Visual Generation with Conditional Video Decoding
Xiang Fan, Yuheng Wang, Bohan Fang et al. (+2 more)
Video generation powers a vast array of downstream applications. However, while the de facto standard, i.e., latent
diffusion models, typically employ heavily conditioned denoising networks, their decoders often remain unconditional. We observe that this architectural asymmetry leads to significant loss of detail and inconsistency relative to the input image. To address this, we argue that the decoder requires equal conditioning to preserve structural integrity. We introduce RefDecoder, a reference-conditioned
video VAE decoder by injecting high-fidelity reference image signal directly into the decoding process via reference
attention. Specifically, a lightweight image encoder maps the reference frame into the detail-rich high-dimensional
tokens, which are co-processed with the denoised
video latent
tokens at each decoder up-sampling stage. We demonstrate consistent improvements across several distinct decoder backbones (e.g., Wan 2.1 and VideoVAE+), achieving up to +2.1dB PSNR over the unconditional baselines on the Inter4K, WebVid, and Large Motion reconstruction
benchmarks. Notably, RefDecoder can be directly swapped into existing
video generation systems without additional fine-tuning, and we report across-the-board improvements in subject consistency, background consistency, and overall quality scores on the VBench I2V
benchmark. Beyond I2V, RefDecoder generalizes well to a wide range of visual generation tasks such as style transfer and
video editing refinement.
Read the full abstract ▾
All authors Xiang Fan, Yuheng Wang, Bohan Fang, Zhongzheng Ren, Ranjay Krishna
Computer Vision
VGGT-$\Omega$
Jianyuan Wang, Minghao Chen, Shangzhan Zhang et al. (+7 more)
Recent feed-forward reconstruction models, such as VGGT, have proven competitive with traditional optimization-based reconstructors while also providing geometry-aware features useful for other tasks. Here, we show that the quality of these models scales predictably with model and data size. We do so by introducing VGGT-$\Omega$, which substantially improves reconstruction accuracy, efficiency, and capabilities for both static and dynamic scenes. To enable training this model at an unprecedented scale, we introduce architectural changes that improve training efficiency, a high-quality data annotation pipeline that supports dynamic scenes, and a self-supervised learning protocol. We simplify VGGT's architecture by using a single dense prediction head with multi-task supervision and removing the expensive high-resolution convolutional layers. We also use registers to aggregate scene information into a compact representation and introduce register
attention, which restricts inter-frame information exchange to these registers, in part replacing global
attention. In this way, during training, VGGT-$\Omega$ uses only about 30% of the GPU
memory of its predecessor, allowing us to train with 15x more supervised data than prior work and to leverage vast amounts of unlabeled
video data. VGGT-$\Omega$ achieves strong results for reconstruction of static and dynamic scenes across multiple
benchmarks, for example, improving over the previous best camera estimation accuracy on Sintel by 77%. We also show that the learned registers can improve
vision-
language-action models and support
alignment with
language, suggesting that reconstruction can be a powerful and scalable proxy task for spatial understanding. Project Page: http://vggt-omega.github.io/
Read the full abstract ▾
All authors Jianyuan Wang, Minghao Chen, Shangzhan Zhang, Nikita Karaev, Johannes Schönberger, Patrick Labatut, Piotr Bojanowski, David Novotny, Andrea Vedaldi, Christian Rupprecht
Computer Vision
Aligning Latent Geometry for Spherical Flow Matching in Image Generation
Tuna Han Salih Meral, Kaan Oktay, Hidir Yesiltepe et al. (+2 more)
Latent flow matching for image generation usually transports Gaussian noise to variational autoencoder latents along linear paths. Both endpoints, however, concentrate in thin spherical shells, and a Euclidean chord leaves those shells even when preprocessing aligns their radii. By decomposing each latent
token into radial and angular components, we show through component-swap probes that decoded perceptual and semantic content is carried predominantly by direction, with radius contributing much less. We therefore project data latents onto a fixed
token radius, use the radial projection of Gaussian noise as the spherical prior, finetune the decoder with the encoder frozen, and replace linear interpolation with spherical linear interpolation. The resulting geodesic paths stay on the sphere at every timestep, and their velocity targets are purely angular by construction. Under matched training, the method consistently improves class-conditional ImageNet-256 FID across different image tokenizers, leaves the
diffusion architecture unchanged, and requires no auxiliary encoder or representation-
alignment objective.
Read the full abstract ▾
All authors Tuna Han Salih Meral, Kaan Oktay, Hidir Yesiltepe, Adil Kaan Akan, Pinar Yanardag
Computer Vision
RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO
Yanzuo Lu, Ronglai Zuo and Jiankang Deng
Causal autoregressive
video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content. Distilling such generators from high-fidelity bidirectional teachers yields competitive few-step models, yet a persistent gap between the history distributions encountered during training and those arising at inference constrains generation quality over long horizons. We introduce the Real-time Autoregressive
Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states. This formulation aligns training
attention with inference-time extrapolation and allows downstream chunk losses to supervise the history representations on which future predictions depend. We further propose Consistency-model Group Relative Policy Optimization (CM-GRPO), which reformulates a consistency sampling step as a conditional Gaussian transition and applies online Reinforcement Learning (RL) directly to this kernel, avoiding the Euler-Maruyama auxiliary process adopted in prior flow-model RL formulations. Experiments demonstrate that RAVEN surpasses recent causal
video distillation baselines across quality, semantic, and dynamic degree evaluations, and that CM-GRPO provides further gains when combined with RAVEN.
Read the full abstract ▾
Computer Vision
Articraft: An Agentic System for Scalable Articulated 3D Asset Generation
Matt Zhou, Ruining Li, Xiaoyang Lyu et al. (+6 more)
A bottleneck in learning to understand articulated 3D objects is the lack of large and diverse
datasets. In this paper, we propose to leverage large
language models (LLMs) to close this gap and generate articulated assets at scale. We reduce the problem of generating an articulated 3D asset to that of writing a program that builds it. We then introduce a new agentic system, Articraft, that writes such programs automatically. We design a programmatic interface and harness to help the LLM do so effectively. The LLM writes code against a domain-specific SDK for defining parts, composing geometry, specifying joints, and writing tests to validate the resulting assets. The harness exposes a restricted workspace and interface to the LLM, validates the resulting assets, and returns structured feedback. In this way, the LLM is not distracted by details such as authoring a URDF file or managing a complex software environment. We show that this produces higher-quality assets than both state-of-the-art articulated-asset generators and general-purpose coding
agents. Using Articraft, we build Articraft-10K, a curated
dataset of over 10K articulated assets spanning 245 categories, and show its utility both for training models of articulated assets and in downstream applications such as robotics simulation and virtual reality.
Read the full abstract ▾
All authors Matt Zhou, Ruining Li, Xiaoyang Lyu, Zhaomou Song, Zhening Huang, Chuanxia Zheng, Christian Rupprecht, Andrea Vedaldi, Shangzhe Wu
Computer Vision
VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction
Kaixin Zhu, Yiwen Tang, Yifan Yang et al. (+9 more)
High-quality 3D scene reconstruction has recently advanced toward generalizable feed-forward architectures, enabling the generation of complex environments in a single forward pass. However, despite their strong performance in static scene perception, these models remain limited in responding to dynamic human instructions, which restricts their use in interactive applications. Existing editing methods typically rely on a 2D-lifting strategy, where individual views are edited independently and then lifted back into 3D space. This indirect pipeline often leads to blurry textures and inconsistent geometry, as 2D editors lack the spatial awareness required to preserve structure across viewpoints. To address these limitations, we propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. VGGT-Edit introduces depth-synchronized text injection to align semantic guidance with the backbone's spatial poses, ensuring stable instruction grounding. This semantic signal is then processed by a residual transformation head, which directly predicts 3D
geometric displacements to deform the scene while preserving background stability. To ensure high-fidelity results, we supervise the framework with a multi-term objective function that enforces
geometric accuracy and cross-view consistency. We also construct the DeltaScene
Dataset, a large-scale
dataset generated through an automated pipeline with 3D agreement filtering to ensure ground-truth quality. Experiments show that VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed. The project page is https://chriszkxxx.github.io/VGGT-Edit/.
Read the full abstract ▾
All authors Kaixin Zhu, Yiwen Tang, Yifan Yang, Renrui Zhang, Bohan Zeng, Ziyu Guo, Ruichuan An, Zhou Liu, Qizhi Chen, Delin Qu, Jaehong Yoon, Wentao Zhang
Computer Vision
Quantitative Video World Model Evaluation for Geometric-Consistency
Jiaxin Wu, Yihao Pi, Yinling Zhang et al. (+2 more)
Generative
video models are increasingly studied as implicit world models, yet evaluating whether they produce physically plausible 3D structure and motion remains challenging. Most existing
video evaluation pipelines rely heavily on human judgment or learned graders, which can be subjective and weakly diagnostic for
geometric failures. We introduce PDI-Bench (Perspective Distortion Index), a quantitative framework for
auditing geometric coherence in generated
videos. Given a generated clip, we obtain object-centric observations via
segmentation and point tracking (e.g., SAM 2, MegaSaM, and CoTracker3), lift them to 3D world-space coordinates via
monocular reconstruction, and compute a set of projective-geometry residuals capturing three failure dimensions: scale-depth
alignment, 3D motion consistency, and 3D structural
rigidity. To support systematic evaluation, we build PDI-
Dataset, covering diverse scenarios designed to stress these
geometric constraints. Across state-of-the-art
video generators, PDI reveals consistent geometry-specific failure modes that are not captured by common perceptual metrics, and provides a diagnostic signal for progress toward physically grounded
video generation and physical world model. Our code and
dataset can be found at https://pdi-bench.github.io/.
Read the full abstract ▾
All authors Jiaxin Wu, Yihao Pi, Yinling Zhang, Yuheng Li, Xueyan Zou