Computer Vision
VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis
Kangjie Chen, Xiangyu Li, Dongbin Zhang et al. (+12 more)
We present VGGT-Diff, a geometry-routed multi-view
diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while
diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-{\Omega} into a pretrained
video diffusion model. Each visual
token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Read the full abstract ▾
All authors Kangjie Chen, Xiangyu Li, Dongbin Zhang, Chaoda Zheng, Shijia Chen, Jinhao Deng, Hongbin Lin, Choo Sin Wai, Minqi Wang, Minghao Yang, Dake Zhong, Guorui Song, Yu Zhang, Xianming Liu, Boyang Wang
Computer Vision
PARSEE-VAD: Efficient Training-Free Online Video Anomaly Detection via Proposition-Aware Reasoning and Streaming Evidence Escalation
Ji Wang, Shuangqing Zhang, Guo-Sen Xie et al. (+1 more)
Training-free online
video anomaly detection (VAD) with frozen
multimodal language models faces two coupled challenges: extracting reliable current-window semantics under causal and computational constraints, and maintaining temporal continuity without repeatedly transmitting high-dimensional history. Encoding history through text can compress visual evidence and introduce semantic bias, whereas retaining visual history expands
multimodal context. We introduce PARSEE-VAD, a two-module framework that separates semantic evidence acquisition from score-state
evolution. Proposition-Aware
Reasoning (PAR) extracts structured propositional evidence from the current causal window and conditionally activates more specific queries when coarse evidence warrants further refinement. By sharing a reusable causal visual prefix across queries, PAR reduces redundant computation through selective execution. Streaming Evidence Escalation (SEE) maps the acquired proposition evidence into a compact score-domain event state through current evidence escalation, then propagates only the resulting bounded state across decisions to support temporal continuity. Experiments on four
benchmarks demonstrate strong training-free online performance while selective routing reduces specialist computation and score-state propagation remains sparse. These results support a current-first principle for streaming
multimodal inference: resolve present semantics first, then use compact historical state only to repair residual continuity gaps.
Read the full abstract ▾
All authors Ji Wang, Shuangqing Zhang, Guo-Sen Xie, Fang Zhao
Computer Vision
AevaScenes: An FMCW LiDAR Dataset and Benchmark for Long-Range Perception
Gautham Narayan Narasimhan, Heethesh Vhavle, Kumar Bhargav Viswanatha et al. (+2 more)
FMCW
LiDAR measures per-point radial Doppler velocity alongside range, providing a motion cue unavailable in conventional time-of-flight sensors. Exploiting this signal at long range remains understudied. We present an FMCW
LiDAR dataset of 575 sequences (57.5K frames) with over 8 million annotated 3D boxes across 16 detection classes and per-point labels across 24 semantic classes, captured by six commercial FMCW
LiDAR sensors and six paired 4K cameras across eight Bay Area cities, including 237 nighttime sequences, with annotations extending to 400m. We define a
benchmark with three tasks: 3D object detection, scene flow estimation, and semantic segmentation. Detection and scene flow are evaluated across three range bins to 400m, with a public evaluation server. We explore the impact of Doppler measurements on flagship recognition tasks, and find significant improvements up to 2X in detection AP of far-away vehicles and pedestrians, particularly in low-latency single-frame settings. We similarly find scene flow accuracy is significantly improved with Doppler measurements across all ranges. Our
dataset and
benchmark have been publicly released at https://scenes.aeva.com.
Read the full abstract ▾
All authors Gautham Narayan Narasimhan, Heethesh Vhavle, Kumar Bhargav Viswanatha, James Reuther, Deva Ramanan
Computer Vision
Scope-WM: Scoped Computation for Efficient Visual World Models
Chunzheng Li, Zesheng Jia, Hongda Zhang et al. (+4 more)
Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak
memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided
token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected
tokens. It updates the remaining
tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly
token-to-
token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector
distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU
memory usage and planning time to $18.1\%$ and $14.3\%$ of those of dense DINO-WM, respectively, corresponding to a $6.97\times$ planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at https://github.com/ChunZheng2022/Scope-WM.
Read the full abstract ▾
All authors Chunzheng Li, Zesheng Jia, Hongda Zhang, Jiaying Tang, Yuntian Wang, Siao Liu, Jin Wang
Computer Vision
RepFlow: Reciprocal Supervision Improves Generation and Representation in Flow Models
Weili Zeng, Feng Tian, Shengqi Liu et al. (+1 more)
Generative models learn visual structure through denoising, yet their internal states are entangled with both noise level and network depth, making it difficult to obtain a stable visual representation from the generator itself. We introduce RepFlow, which learns such a representation from the generator's evolving computation and uses it to guide generation. Specifically, a separate timestep-free encoder is trained, through a timestep-conditioned predictor, to recover generator states across depths and noise levels from masked clean images. By excluding the reference noise and masked-out content from the encoder's input, we encourage the encoder to distill visual information that is recoverable from the visible image context and predictive of generator states. The representation learned from the generator's evolving states is then fed back to guide and improve the generator, whose updated states provide supervision for further representation learning, forming a reciprocal learning process. This reciprocal process improves multi-step generation and representation quality, as measured by frozen linear probing on ImageNet with latent-space SiT and pixel-space JiT, without an externally pretrained representation teacher. Across the two unconditional settings, FID decreases by $19.2$–$40.8\%$ relative to native training, while linear-probe accuracy improves by 6.80–10.04 percentage points over the best searched raw generator features. The learned representation also serves as a distributional metric for one-step JiT post-training, extending its role from instance-level
alignment to distribution-level supervision.
Read the full abstract ▾
All authors Weili Zeng, Feng Tian, Shengqi Liu, Yichao Yan
Computer Vision
Background Gradients Shape Memorization in Flow Matching
Xuanhua Yin, Boyu Wei, Shuyi Zhang et al. (+3 more)
Repetition is closely associated with memorization in generative models, but how other training images affect the retention and copying of targets remains unclear. We study this question in class-conditioned flow matching, where images outside the target set form the background. At fixed target repetition and same-class background row count, replacing repeated same-class images with distinct images reduces the target extraction rate from 80.7% to 18.0%. To explain this effect, we develop a paired-trajectory framework that isolates target-induced parameter displacement and the background gradient response to it. This response has an exact path-integrated
curvature representation, connecting background loss geometry to target learning. Reciprocal response transfer between repeated and distinct backgrounds changes target retention and copying in both directions, establishing the response's causal role. After target removal, the response correction parallel to the target-induced displacement preserves approximately 90% of the copying effects of full response transfer. Directly scaling the displacement also changes copying without further training. The post-removal copying effects of reciprocal transfer are reproduced across
datasets and architectures. Together, these results identify the background gradient response as a mechanism through which same-class training data shape the retention of target learning and the reproduction of target images.
Read the full abstract ▾
All authors Xuanhua Yin, Boyu Wei, Shuyi Zhang, Shunqi Mao, Chuanzhi Xu, Weidong Cai
Computer Vision · Cross-listed from cs.AI
WorldAgent: Verification-Guided Agentic Physical World Construction
Caoliwen Wang, Mengdi Wang, Yige Chen et al. (+13 more)
Constructing complex physical worlds from
language requires coordinating extensive 3D environments, detailed structures and objects at different spatial scales, and interacting physical processes under both stated goals and implicit physical constraints. We present WorldAgent, an agentic framework for verification-guided physical world construction from a single natural-
language prompt, without iterative user debugging. A world construction layer expands the
prompt into a structured world specification and uses physical knowledge to build scenes and run numerical simulations. After every step, a verification layer inspects scene geometry and simulation states alongside rendered views. Failed checks guide automatic revisions to the specification and re-execution of the affected steps. Accepted worlds pass the required checks and remain editable for further inspection and resimulation. We introduce AgenticSimBench, on which WorldAgent achieves the best scores among the evaluated
agent-based methods on five of seven metrics. In a 26-participant user study, it receives the highest mean ratings across all four criteria.
Read the full abstract ▾
All authors Caoliwen Wang, Mengdi Wang, Yige Chen, Zejia Wu, Bowen Huang, Siyuan Chen, Guanxiong Chen, Lifu Wei, Heng Zhang, Qinghai Zhang, Yin Yang, Guandao Yang, Shiying Xiong, Peng Wang, Chenfanfu Jiang, Peter Yichen Chen
Computer Vision
Structured Residual Connectivity Matters for Diffusion Transformers
Yuhe Liu, Xinyin Ma, Gongfan Fang et al. (+2 more)
Diffusion Transformers (DiTs) have established themselves as a scalable backbone for high-fidelity image synthesis. However, unlike U-Net based
diffusion models that rely on rigid, hand-crafted skip connections, DiTs predominantly use a uniform residual stream that integrates all preceding layers as a monolithic state. In this work, we rethink residual connections in
diffusion transformers and propose to transform them from passive summation into an active
retrieval mechanism optimized for image denoising. First, we conduct a systematic analysis of DiT's internal representation, revealing a latent preference for early-layer feature reuse and symmetric layer guidance. Motivated by this, we introduce a structured connectivity design that explicitly integrates local residual connections with long-range pathways. Instead of static skip connections or dense all-layer routing, our method enables each
transformer block to selectively “attend” to critical earlier representations, dynamically retrieving spatial and semantic cues through direct, differentiable cross-depth paths. Experiments show that our adaptive connectivity leads to faster convergence, with up to $1.73\times$ fewer training iterations, and significant gains in FID and visual quality with less than $0.1\%$ additional parameters, further improving a strong REPA-XL/2 model from $5.9$ to $4.34$ FID without guidance and reaching $1.39$ FID with classifier-free guidance. Our findings suggest that adaptive cross-layer connectivity is a critical yet underexplored factor in
diffusion transformers, and that incorporating structured information pathways provides a simple and effective direction for improving scalable generative models.
Read the full abstract ▾
All authors Yuhe Liu, Xinyin Ma, Gongfan Fang, Songhua Liu, Xinchao Wang
Computer Vision
ReAL: Accelerating Flow Matching through Segment Advancement with Shared Lookahead
Xuanhua Yin, Chuanzhi Xu, Haoxian Zhou et al. (+2 more)
Flow-matching models generate high-quality images and
videos, but repeated neural network evaluations make sampling expensive. Skipping evaluations reduces this cost by extending an available velocity estimate over a longer span. However, local velocity agreement alone does not determine a suitable span, and checking each candidate endpoint adds costly model calls. We introduce ReAL, a training-free sampler that selects how far to advance using one shared lookahead. Our key insight is that the discrepancy between uncorrected and lookahead-corrected endpoint proposals can be computed directly from the observed velocity mismatch and the candidate span beyond the lookahead. This relation provides a span-dependent selection criterion without additional endpoint evaluations. The same lookahead selects the longest passing candidate span, corrects the accepted update, and supplies its velocity as the next starting estimate. After initialization, each regular iteration requires only one fresh evaluation. ReAL uses the pretrained velocity output and original noise schedule, with no additional training or access to internal features. Experiments cover four image-generation backbones,
video generation, and image editing. ReAL achieves 4.91x measured speedup on FLUX.1-dev while retaining 97.0% of dense mean ImageReward. On HunyuanVideo, it achieves a 5.49x speedup while maintaining a VBench score close to that of dense sampling.
Read the full abstract ▾
All authors Xuanhua Yin, Chuanzhi Xu, Haoxian Zhou, Shunqi Mao, Weidong Cai
Computer Vision
FocusDrive: Reasoning with Visual Focus for Autonomous Driving
Zhiyuan Liu, Zehong Ke, Yuanxin Tian et al. (+7 more)
Driving decisions depend on both where to focus and how to act on what is seen. Effective driving
reasoning must establish which objects matter, where they are, and how they inform the intended action. Text-based rationales can describe a driving response while leaving its correspondence to specific visual evidence implicit. Visual focus provides a concrete starting point for this connection by identifying what matters in the scene and where it is. We propose FocusDrive, a structured
multimodal reasoning framework that organizes end-to-end planning around explicit visual focus. It pairs descriptions of decision-relevant objects with image-patch references, bringing explicit visual focus into the
reasoning that generates driving plans and trajectories. We first assess this focus representation through driver gaze prediction, then investigate its role in planning
reasoning using driving-focus annotations within existing NAVSIM training scenes. Experiments on W3DA and NAVSIM demonstrate competitive gaze prediction and end-to-end planning performance, with FocusDrive improving over text-based chain-of-thought. These results support visual focus as an effective link between scene understanding and driving action.
Read the full abstract ▾
All authors Zhiyuan Liu, Zehong Ke, Yuanxin Tian, Hao Cheng, Jinhao Li, Yining Xing, Yanbo Jiang, Zhenhua Xu, Wenhao Yu, Jianqiang Wang
Computer Vision · Cross-listed from cs.LG
When an Evaluation Rule Writes Training Labels: Measuring Human-Reference Forgiveness in NAVSIM
Jiaxuan Guo, Jingxin Yang, Jiaqi Ye et al. (+4 more)
When the human reference scores zero on a metric, the released GTRS-Dense label generator for NAVSIM marks every candidate trajectory in the scene as passing it. NAVSIM's authors introduced this human-reference forgiveness to avoid penalizing contextually justified maneuvers when scoring one trajectory, and warned that it could overlook important failures. In label generation it sets a whole column of 16,384 candidate targets to passing. To measure the consequences for supervision, we re-run the generator with the overwrite disabled and compare the pre-overwrite targets with the released labels on all 103,288 navtrain scenes. The rule erases a candidate distinction that the training loss reads on 11,237 of them (10.8793%). Firing usually changes most of a column: lane keeping carries 9,982 of the 13,042 forgiven loss columns, and its median forgiven column had 14,391 of 16,384 candidates failing before the overwrite. On held-out navtest scenes forgiven on lane keeping, the released lane-keeping head's median AUC against the pre-overwrite outcome is 0.7095; on unforgiven scenes matched on failing-candidate count it is 0.9807. For the Hydra-MDP checkpoint released with GTRS, whose configuration takes the same label file, the two values are 0.6627 and 0.9761. Continuing the released GTRS-Dense checkpoint for 300 optimizer steps with three paired seeds, we observe the forgiven-scene AUC 0.1086-0.1251 higher with pre-overwrite than with published targets, and a narrower gap between matched groups, still above zero. Scoring with forgiveness disabled, we observe lane keeping higher by 2.478-3.524 points on navtest scenes forgiven on any of five loss metrics, with lower adjacent-frame plan consistency. Both changes are larger there than on the rest. EPDMS, scored the same way, does not separate the two target sets.
Read the full abstract ▾
All authors Jiaxuan Guo, Jingxin Yang, Jiaqi Ye, Youran Sun, Shuo Xin, Kejia Zhang, Haizhao Yang
Computer Vision
Can Protein-Derived Knowledge Improve Pathology Foundation Models?
Di Zhang, Zhangpeng Gong, Jiashuai Liu et al. (+10 more)
Molecularly guided pathology foundation models (PFMs) exploit transcriptomic or proteomic information to enrich whole-slide image (WSI) representations, yet effectively leveraging large standalone molecular corpora remains challenging. First, existing molecular foundation models encode
protein sequences or single-
cell states, not the patient-level bulk expression profiles paired with WSIs. Second, because cross-modal supervision is restricted to paired WSI-omics samples, knowledge from standalone molecular corpora reaches the pathology encoder only indirectly, creating a paired-support bottleneck. To address these challenges, we propose a three-stage framework that decouples proteomic knowledge acquisition from cross-modal transfer, yielding ProSlide, a slide-level hierarchical pathology foundation model. First, to close the modality gap, we pretrain a Proteomic Foundation Encoder (PFE) on 12,695 sample-level bulk
protein profiles using virtual profile generation and expression-space multi-view pretraining. Second, we pretrain ProSlide, a patch-region-slide encoder, to predict
protein expression from paired WSI-
protein samples. Third, to relax the paired-support bottleneck, we introduce Prot2Path, a cross-modal relational
distillation objective. For each paired sample, it aligns the similarity distributions of the WSI and its
protein profile over a shared, frozen bank of PFE-encoded paired and standalone profiles. We evaluate ProSlide on 12 downstream tasks across breast, lung, and renal cancers. Despite being pretrained with only 2,229 WSIs and 12,695 sample-level
protein profiles, ProSlide achieves the highest mean accuracy and AUC within each cancer group.
Read the full abstract ▾
All authors Di Zhang, Zhangpeng Gong, Jiashuai Liu, Zhi Zeng, Jiusong Ge, Chunze Yang, Xitong Ling, Kai Yi, Kai He, Weimiao Yu, Mireia Crispin-Ortuzar, Chen Li, Zeyu Gao
Computer Vision · Cross-listed from cs.RO
DeltaWAM: Change-Centric Visual Foresight via Delta Tokens for an Efficient World-Action Model
Tianyun Jiang, Wenrui Bao, Bingxin Xu et al. (+2 more)
World-Action Models (WAMs) offer visual foresight for robotic manipulation, but pixel-space models repeatedly reconstruct entire future scenes, incurring high computational cost and spatio-temporal redundancy. In physical manipulation, consecutive frames often share most of their visual context; the changes between them are what an action policy needs to anticipate. We introduce DeltaWAM, a change-centric WAM that makes a compact delta
token the unit of future prediction. Each
token is a single vector encoding changes between consecutive dense DINO feature maps. DeltaWAM builds on DeltaWorld, a latent world model pretrained on large-scale
videos, to autoregressively predict one delta
token per future frame. A flow-matching action expert then conditions on the predicted transitions and current DINO features, which serve as spatial anchors, to generate action chunks. Trained for 256 GPU hours on two H100 GPUs, DeltaWAM has 0.725B parameters and achieves 92.8% average success on LIBERO. It also shows robust generalization under procedural perturbations on LIBERO-Pro. Inference takes 142.1 ms per action chunk with 3.86 GB peak
memory.
Read the full abstract ▾
All authors Tianyun Jiang, Wenrui Bao, Bingxin Xu, Yu Tian, Yuzhang Shang
Computer Vision
ABO-Med: Accelerated Bilevel Optimization for Few-Shot Medical Image Classification
Ruoxuan Shi, Sheng Yang, Zhengxing Su et al. (+2 more)
In recent years, bilevel optimization has been widely used in a variety of machine learning tasks. However, prior bilevel optimization
algorithms generally require the computation of second-order information, which limits their practical scalability. Only recently has a first-order paradigm for bilevel optimization been established, attaining near-optimal theoretical guarantees for solving bilevel optimization problems. In this paper, we propose ABO-Med, a scalable instantiation of this paradigm for few-shot learning, by incorporating it into the model-agnostic meta-learning (MAML) framework and tailoring it to medical image classification. We also introduce Medical Adaptive RandomAugment (MedRAug), a modality-aware augmentation strategy designed for medical images. Theoretically, ABO-Med establishes the optimality of MAML-type meta-learning approaches. Empirically, ABO-Med outperforms prior baselines on several public medical
datasets, with gains of 1.99% to 18.76%, while MedRAug further improves the average accuracy by 2.20% to 6.34%. Additional cross-domain experiments, augmentation ablation studies, backbone ablation studies, and training efficiency analysis further validate the effectiveness and efficiency of the proposed method.
Read the full abstract ▾
All authors Ruoxuan Shi, Sheng Yang, Zhengxing Su, Xiaoyang Hou, Yating Liu
Computer Vision · Cross-listed from cs.RO
Dynamic Manipulation with World-Action Models via Counterfactual Planning
Sunwoo Park, Wonbin Lee, Seonghyun Jin et al. (+3 more)
World-Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills. We attribute this failure to target-response collapse: as execution advances, the policy becomes increasingly biased toward the learned continuation of its ongoing behavior and less responsive to target relocation. To bridge the gap between what the model has learned and what it can generate from the current context, we formulate dynamic manipulation as counterfactual planning by decoupling the context used for plan generation from the physical state used for execution. Our framework, Dynamic Predictive Planning (DPP), first uses the WAM's predictive rollout to estimate when an interaction is expected to occur, and combines this timing estimate with observed target motion to predict the target's future interaction position. DPP then constructs a counterfactual observation that places this predicted target position in a familiar
robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar
robot-target configuration. The resulting plan is connected to the
robot's actual state during execution. DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data. Experiments in simulation and on a real
robot demonstrate consistent improvements across diverse target motions, with simulation performance surpassing all evaluated baselines, including methods additionally trained on dynamic data. Project page: https://methoder00.github.io/DPP/
Read the full abstract ▾
All authors Sunwoo Park, Wonbin Lee, Seonghyun Jin, Youngmin Kim, Jangho Park, Jong Chul Ye
Computer Vision · Cross-listed from cs.LG
Perturb-and-Solve: Efficient Learned-Operator Conditioning for Latent Diffusion Inverse Problems
Abduragim Shtanchaev, Arip Asadulaev, Luiza Labazanova et al. (+3 more)
Latent
diffusion models serve as powerful priors for solving inverse problems in image restoration, such as deblurring, inpainting, and super-resolution. Current methods have a trade-off between generality and efficiency. Solvers that are restricted to a fixed set of degradation operators are fast and efficient. Methods that support arbitrary degradation operators are slow and require gradients through the
diffusion network. To break this bottleneck, we introduce PASEO (Perturb-And-Solve for Efficient Operator conditioning), a method that uses a small (1M parameters) learned network to degrade
diffusion model predictions in latent space. PASEO supports learned degradation operators without back-propagating through the
diffusion network. We efficiently sample reconstructions from an approximate posterior by combining the
diffusion model's prediction with the observed image. We do this by adding noise and solving linear equations based on a local linear approximation of the learned network, without building or inverting large covariance
matrices. Across super-resolution, deblurring, and inpainting on FFHQ and COCO, PASEO achieves strong perceptual quality while running up to 9x faster and using up to 34% less peak
memory than the tested baselines, with the same or fewer model evaluations.
Read the full abstract ▾
All authors Abduragim Shtanchaev, Arip Asadulaev, Luiza Labazanova, Aidar Alimbayev, Karim Salta, Eric Moulines
Computer Vision
FloodDiffusion 2: Efficient and Path Controllable Streaming Motion Generation
Yiyi Cai, Yuhan Wu, Kunhang Li et al. (+6 more)
We present FloodDiffusion 2 (FD2), an efficient and controllable framework that builds upon FloodDiffusion (FD1), a state-of-the-art streaming motion generation model. While FD1 produces plausible motion, it suffers from low efficiency and limited controllability, as its
attention design requires repeated computation over the entire history, and it lacks precise trajectory control for real-world applications. To address these limitations and improve generation quality, FD2 introduces three advances. First, Partial
Attention makes finalized history representations independent of the active window, enabling KV-cached inference and shared-history packing for efficient training. Second, we establish a necessary-and-sufficient Bregman criterion for regression losses to preserve
diffusion's conditional-mean velocity field. This criterion guides an FK-induced quadratic loss that incorporates motion geometry without online FK evaluation. Third, FD2 introduces precise path conditioning to control the character's root trajectory while preserving natural body motion. Experiments show that FD2 reduces training computation by 4.6$\times$ and accelerates denoising by 11.29$\times$, reaching 2.303 ms per update on long sequences. Alongside these efficiency gains, FD2 improves motion quality over FD1 and achieves state-of-the-art FID scores among streaming methods, with 0.048 on SEED and 0.053 on HumanML3D.
Read the full abstract ▾
All authors Yiyi Cai, Yuhan Wu, Kunhang Li, Tu Fangyuan, Xiangyue Zhang, Qiaoge Li, Zhixiang Wang, Kaipeng Zhang, Haiyang Liu
Computer Vision
FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs
David Restrepo, Chenwei Wu, Luis Filipe Nakayama et al. (+4 more)
Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging. Performance reported on a single
dataset does not capture how models behave under
dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in medical imaging analysis, where robustness, calibration, and
fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-
Dataset Understanding under Shift), a cross-
dataset benchmark for evaluating retinal fundus models that considers
vision-only encoder models (VM),
vision-
language dual-encoder models (VLM), and
multimodal large
language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public
datasets spanning diverse geographies, acquisition conditions, and label protocols. The
benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 MLLM configurations adapted through supervised fine-tuning with low-rank adaptation (LoRA). Results show that no model family consistently dominates across tasks and
datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external
datasets, particularly in calibration. These findings demonstrate that retinal model evaluation is inherently multidimensional. FOCUS provides a practical framework and public
benchmark to assess generalization, reliability, and robustness beyond single-
dataset leaderboards
Read the full abstract ▾
All authors David Restrepo, Chenwei Wu, Luis Filipe Nakayama, Miguel L. Martins, Stergios Christodoulidis, Maria Vakalopoulou, Enzo Ferrante
Computer Vision
DroneWAM: Efficient World Action Model for Drone Visual Navigation
Liang Yao, Fan Liu, Hongbo Lu et al. (+5 more)
World-action models give visual navigation
agents a way to anticipate how candidate actions will change future observations and to act from the predicted consequences. For drones, this capability must operate under tight accuracy and efficiency constraints. We present DroneWAM, an efficient world-action model for drone visual navigation. DroneWAM adopts a JEPA-based architecture to model future states directly in representation space, avoiding the cost of explicit future image generation. A pretrained Resampler further compresses dense encoder features into fewer latent
tokens, reducing the computation repeated at each imagined step. We also introduce adaptive rollout, where a preference-trained Gate adaptively allocates prediction depth according to the current scene. To support learning under richer aerial motion, we construct DroneNav-6D, a simulated visual navigation
dataset with synchronized RGB observations, 6-DoF flight trajectories, control commands, and randomized wind disturbances. On DroneNav-6D, DroneWAM achieves the best trajectory accuracy among the compared methods. Adaptive rollout further reduces the average prediction depth from 8 to 4.58 while improving trajectory accuracy, demonstrating that predictive computation can be allocated more effectively across scenes. \href{https://github.com/1e12Leon/DroneWAM}{Codes and data} will be released.
Read the full abstract ▾
All authors Liang Yao, Fan Liu, Hongbo Lu, Wei Xu, Jianyu Jiang, Yijun Shen, Chuanyi Zhang, Pai Peng
Computer Vision
Train Together or Merge Later? Unifying VLA Experts via a Shared Action Interface
Zhizhen Zhang, Yuxia Fu, Zijian Wang et al. (+2 more)
Co-training offers a straightforward way to build a multi-task
vision-
language-action (VLA) policy, but can fall short of the performance achieved by training each task independently. The challenge is to retain these task-specific gains in a multi-task policy without joint post-training. Combining independently trained experts through model merging is a natural approach, yet strong individual experts do not necessarily yield a strong merged policy. We identify one source of this incompatibility: task-specific changes to the action interface, comprising action normalization and the action encoder and decoder. We propose PolicyWeave, combining merge-compatible post-training with context-guided sparse merging. During post-training, all experts retain the common base policy's action interface, while task adaptation is restricted to LoRA updates in the hidden layers of the action model. This makes the experts more compatible with existing model merging methods. However, merging all experts can still introduce interference from unrelated tasks at deployment. PolicyWeave scores each expert's LoRA updates using the initial visual-
language context, determines the expert set through leave-one-layer-out ranking stability, and forms a sparse weighted merge of the selected updates that remains fixed for current task. We evaluate PolicyWeave with GR00T N1.5 on 18 RoboCasa365 tasks, using only 10% of the target-task demonstrations for supervised fine-tuning (SFT). Preserving the shared action interface raises the average success rate across four static merging methods from 17.0% to 52.8%. PolicyWeave achieves 64.7% success with these SFT experts and 74.1% after task-specific reinforcement learning (RL), compared with 60.7% for joint RL. Further evaluations on LIBERO-10 and an AgileX Piper arm support the deployment of independently learned skills in long-horizon and real-world manipulation.
Read the full abstract ▾
All authors Zhizhen Zhang, Yuxia Fu, Zijian Wang, Helen Huang, Yadan Luo
Computer Vision
Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Models
Tuo Liang, Disheng Liu, Nengbo Wang et al. (+2 more)
Grounding is a core capability of spatial
vision-
language models, yet most existing work focuses only on where a referred object is. Many 3D tasks also require knowing how it is oriented. Although existing 3D VLMs may predict oriented boxes, box pose does not explicitly capture object-centric orientation or
symmetry-induced ambiguities. We introduce orientation grounding, a referring grounding task that predicts an object's 6D orientation and axial
symmetry from a
language or box query in single-view or multi-view scenes. To support this task, we construct ReferOri, with 331K multi-view and 387K single-view orientation-grounding queries obtained through scalable reconstruction, consistency checking, and human verification. We further present OG-VLM, which adapts a 3D VLM with structured box/orientation outputs, sign and
symmetry tokens, and geometry-aware auxiliary losses. Across single-view and multi-view
benchmarks, OG-VLM substantially outperforms orientation-aware VLM baselines and surpasses object-level orientation foundation models on scene-level referring
benchmarks, showing that explicit orientation grounding is a distinct and learnable capability beyond localization. Downstream results validate its benefit for orientation-related spatial
reasoning.
Read the full abstract ▾
All authors Tuo Liang, Disheng Liu, Nengbo Wang, Vipin Chaudhary, Yu Yin
Computer Vision
Octree-based Video Representation
Rungui Zhou, Chuanzhi Zhou, Yuk-Kit Hou et al. (+1 more)
Video models commonly use uniform grids even though visual complexity varies substantially across space and time. We introduce OctVideo, which approximates a
video clip with an octree. This hierarchy recursively partitions a spatio-temporal volume into eight subvolumes, so that smooth regions remain coarse while detailed regions receive finer
cells. Each leaf stores local RGB values and spatio-temporal gradients, supplemented by a lightweight learned residual. For reconstruction, a Conv1D VAE maps the serialized
cells to a regular latent grid and selectively refines details during decoding. Our VAE achieves 36.12 dB PSNR with 38.2M parameters and 189.4 GFLOPs per clip on Kinetics-400 (K400). It also generalizes zero-shot to the high-resolution Densely Annotated
VIdeo Segmentation (DAVIS) 2016
dataset with reconstruction quality comparable to the best evaluated models. On both
datasets, it requires the fewest model FLOPs and achieves the fastest encoding and decoding among the evaluated models. OctVideo also supports
video understanding, achieving competitive recognition performance with few input
tokens when trained from scratch. By exploiting the redundancy already present in
video signals and efficiently processing sparse structures, OctVideo provides an efficient representation for
video.
Read the full abstract ▾
All authors Rungui Zhou, Chuanzhi Zhou, Yuk-Kit Hou, Peng-Shuai Wang
Computer Vision
Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigation
Zhihao Chen, Yiyuan Ge, Ziyang Wang et al. (+2 more)
Recent large-scale
Vision-and-
Language Navigation (VLN) models deliver strong accuracy but remain costly to deploy due to their large parameter counts and computational requirements. We tackle efficient VLN in two steps. First, we build a high-performing teacher that makes navigation evidence selection explicit and compressible. The teacher introduces a small set of learnable query slots to extract global and local action-sufficient navigable evidence from panoramic observations via a Navigable Query Generator, then progressively grounds these evidence
tokens to the instruction with an Instruction-Query Aligner for policy prediction. Second, using this explicit query bottleneck as a
distillation interface, we train a compact student by transferring both where to attend and what to do. We distill the teacher's global and local navigable queries with a navigation-aware
token-adaptive objective, then further match action distributions during fine-tuning. Experiments on standard VLN
benchmarks demonstrate that our student nearly matches the teacher's navigation performance while reducing the number of parameters by 93.65% compared to the teacher.
Read the full abstract ▾
All authors Zhihao Chen, Yiyuan Ge, Ziyang Wang, Pu Cao, Lu Yang
Computer Vision · Cross-listed from cs.CL
OneSign: Unifying Sign Language Understanding Tasks with One Model
Shiwei Gan, Yafeng Yin, Xiao Liu et al. (+3 more)
SLU encompasses a diverse set of tasks, including ISLR, CSLR, and SLT. Although these tasks share basic semantic and linguistic foundations, they are typically addressed with task-specific architectures and training pipelines, which hinders knowledge sharing and requires costly pretraining and finetuning for each task. In this paper, we focus on two aspects of SLU tasks: (1) training and inference pipelines are highly fragmented: most methods rely on pretraining on large-scale SL
datasets followed by task- or
dataset-specific finetuning, which leads to multiple specialized models rather than a single checkpoint. (2) current LLM-based methods may overlook the inherent modality discrepancy between sign and text
tokens, simply concatenating them and processing both modalities with the same decoder layers. In this paper, we present OneSign, a unified framework that addresses multiple SLU tasks within a single model and a single checkpoint. OneSign reformulates ISLR, CSLR, and SLT under a single training paradigm. To accommodate the heterogeneous characteristics of sign and text representations, we introduce a Modality-Adaptive Mixture-of-Experts (MA-MoE) architecture, consisting of a shared expert and modality-specific experts for sign and text
tokens. A modality router dynamically activates the corresponding experts, and their outputs are aggregated to form the final
token representations. By enabling modality-dependent expert specialization while preserving a shared expert path, MA-MoE can effectively model the modality differences between continuous sign representations and discrete text
tokens. Extensive experiments on multiple
benchmarks demonstrate that OneSign achieves competitive or state-of-the-art performance on several
benchmarks, highlighting its effectiveness as a unified SLU model.
Datasets are available at : https://github.com/gswycf/OneSign.
Read the full abstract ▾
All authors Shiwei Gan, Yafeng Yin, Xiao Liu, Desibieer Tuerdaken, Lei Xie, Sanglu Lu
Computer Vision
QSCP: Beyond Class-Name Prompts for Query-Guided Semantic Change Parsing
Yuan Qian and Jie Ma
Traditional change detection (CD) identifies changes between bi-temporal remote sensing images, while semantic change detection (SCD) assigns predefined land-cover classes. However, mapping all changes may not meet a user's specific needs. Referring change detection (RCD) enables selective
retrieval through category
prompts. However, existing category-prompted RCD uses the queried category to specify the destination of a change and returns only a binary mask of the corresponding regions. Users may instead request a particular transition and paired semantic maps to understand what changed into what. Such requests require explicit source and target
reasoning beyond target-class localization. To address these needs, we propose query-guided semantic change parsing (QSCP), which supports category names, synonyms, and intent-bearing sentences and returns a query-specific mask with paired temporal semantic maps. QSCP parses requests into intents and semantic slots, composes bidirectional visual evidence, and predicts both temporal states with a query-conditioned decoder. On SECOND, QSCP outperforms RCDNet on synonym, sentence, and transition queries and improves end-to-end semantic prediction over evaluated semantic baselines. WHU-CDC experiments further assess cross-
dataset transfer and consistency across equivalent expressions without target-domain training. Code is available at https://github.com/qianyuancs/QSCP
Read the full abstract ▾
Computer Vision
dKFD: Phase-Structured Evidence Allocation for Fixed-Budget Localized Event Understanding
Aditya Bagri, Ashutosh Kumar, Chaitanya Lakhchaura et al. (+3 more)
Sparse
video understanding often requires selecting a small set of visual evidence under a fixed frame budget. Most sparse selectors allocate this budget globally, allowing all frames to compete with one another. For temporally localized events, this can be a poor inductive bias: useful evidence is often distributed across pre-event context, the event itself, and post-event consequences. We study fixed-budget evidence allocation for localized event
videos and show that globally competitive Top-$K$ selectors can preserve recognition and grounding while producing unstable event evidence. On DoTA
Video Anomaly Recognition, Global Top-$K$ obtains competitive recognition and temporal grounding, but low selector-event
alignment at $K=12$ (Frame AUC $51.5 \pm 10.8$). We propose dKFD, a phase-structured differentiable selector that reserves evidence capacity across pre-event, event, and post-event phases after full-sequence temporal encoding. Under matched-budget multi-seed evaluation, dKFD improves Frame AUC by $+30.97$ over a matched Global Top-$K$ selector at $K=12$ ($p<0.01$), while yielding modest but statistically significant recognition gains and comparable temporal grounding. Mechanism ablations show that phase supervision is load-bearing: removing it reduces Frame AUC to $41.1 \pm 12.0$ even when phase-partitioned budgets are retained. Downstream diagnostics on VRU-Accident show consistent gains over learned Global Top-$K$ across VLM families, while dense captioning reveals a boundary condition where uniform sampling remains competitive. These results support phase-structured allocation as a controlled fixed-budget approach for event-centric sparse evidence selection, not as a universal
video summarization strategy.
Read the full abstract ▾
All authors Aditya Bagri, Ashutosh Kumar, Chaitanya Lakhchaura, Avinash Anand, Zhengkui Wang, Rajiv Ratn Shah
Computer Vision
SemReward-VL: Semantic Reward-Guided Video-Language Adaptation for Developmental Behavior Assessment
De Jiang, Shuo Zhang, Kehong Yuan et al. (+1 more)
Developmental screening
videos show how children perform specific behaviors, but clinical records usually contain outcomes rather than descriptions of what happened. We present SemReward-VL, which learns to describe item-specific behavior from these outcomes. A
vision-
language model generates a description, and a frozen
language model scores its agreement with the clinical outcome, relevance to the item, abstention on unrelated
video-item pairs, and clarity. Group relative policy optimization (GRPO) updates LoRA adapters using this semantic reward. On 13,379
videos covering 41 items, the method improves aggregate accuracy and the number of items with recall above 0.5. Errors remain in temporal direction, duration, and age-specific interpretations of behavior.
Read the full abstract ▾
All authors De Jiang, Shuo Zhang, Kehong Yuan, Hongen Liao
Computer Vision
Parameter-Efficient 3D Segmentation of Liver and Liver tumors: Depthwise factorization Scales Better Than Dense Convolution with Spatial Dimensionality
Adham M. Alkhadrawi and Mohammed A.B. Mahmoud
Three-dimensional dense convolutional networks are the strongest performers on volumetric medical image segmentation, but their parameter counts scale poorly: moving a dense k x k convolution to k x k x k multiplies its weights by k. We observe that depthwise separable factorization does not share this penalty. Because the cubic kernel term applies only to the depthwise stage while the pointwise projection, which dominates the parameter count, is unchanged, the same architecture grows by 5 % from 2D to 3D where a dense convolutional U-Net grows by 200 %. We exploit this asymmetry to build a 3D U-Net with 536,990 parameters, 24x fewer than an identical dense 3D U-Net. On MSD Task03 Liver (the Medical Segmentation Decathlon liver task, derived from LiTS), evaluated per case under five-fold cross-validation over all 131 public volumes, the model reaches a tumor Dice of 0.577 (95% CI [0.518, 0.633]) and a liver Dice of 0.947 (95% CI [0.941, 0.952]). Its liver Dice exceeds previously reported performance. Its tumor Dice exceeds their low-resolution configuration (0.4701) by 0.107 and their 2D configuration (0.5394), at approximately one twenty-fourth of the parameters and roughly half the in-plane resolution. Trained under identical conditions on a common held-out split, it exceeds a dense 3D U-Net on liver by +0.031 Dice (paired p = 0.006) and on tumor by +0.041 (95 % CI [+0.005, +0.081], paired p = 0.056), suggesting the factorization also acts as a regularizer in the small-data regime characteristic of medical imaging. We further show, on both LiTS and a 2D endoscopy
benchmark, that a large fraction of the network's learnable spatial filters can be replaced by fixed shifts at no cost in accuracy, but that replacing all of them is measurably worse, the placement of spatial capacity matters more than its total amount.
Read the full abstract ▾
Computer Vision
NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation
Avinash Anand, Aman Kumar, Chaitanya Lakhchaura et al. (+4 more)
Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes,
vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited
multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and
protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM), which uses textual guidance to reweight and align RGB-depth frequency components; (2) an Ingredient-Aware Mask-based Prediction Head (IA-MPH), whose gating and channel masks are conditioned on food identity; and (3) modality-specific Internal Semantic Modeling (ISM) blocks. On the Nutrition5k
dataset, NutriVision achieves a mean PMAE of $\mathbf{13.60\pm0.10\%}$, outperforming our IGSMNet implementation by $0.89$ percentage points and OmniFood8k by $2.90$ percentage points (both $p<0.001$). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by $1.50\pm0.17$ percentage points ($p<0.001$). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit
geometric and semantic cues without requiring specialized depth-sensing hardware
Read the full abstract ▾
All authors Avinash Anand, Aman Kumar, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol, Timothy Liu, Zhengkui Wang