Computer Vision
PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation
Kexu Cheng, Zicheng Liu, Mingju Gao et al. (+2 more)
Developing physically aware
video generation models remains a significant challenge due to the difficulty in capturing diverse physical phenomena, such as thermal dynamics, mechanics, and optics. In this work, we introduce PhysRAG, a novel pipeline that enhances physical awareness in
video generation through
Retrieval-Augmented Generation (RAG). To address the issue of limited high-quality data, we design a two-stage data filtering pipeline based on the WISA-80K
dataset, resulting in a curated set of 7K high-quality
videos for training. Furthermore, we construct a physical
video database and develop a mechanism to inject physical knowledge into a
video diffusion model using learnable queries. Our method achieves state-of-the-art performance in both visual quality and physical rule compliance, surpassing existing models in
benchmarks such as PhyGenBench and VBench. We conduct extensive ablation studies to validate the effectiveness of our key components, including the data filtering pipeline, RAG mechanism, and method for physical information extraction. To facilitate future research, our code, data, and models are prepared for release at https://github.com/sediment1024/PhysRAG.
Read the full abstract ▾
All authors Kexu Cheng, Zicheng Liu, Mingju Gao, Chunhe Song, Hao Tang
Computer Vision · Cross-listed from cs.GR
Neural Texture Compression using Hypernetworks
Belcour Laurent
Recent work on neural texture compression has demonstrated that it is possible to learn small, per-material texture representations (composed of latent textures and a small Multi-Layer Perceptron decoder) that can be decoded in real-time during shading to reproduce the input to a physically based shading model. However, existing methods require performing gradient-descent optimization per material for a given MLP and latent configuration. In this work, we train a single hypernetwork that outputs both the latent features and the MLP's weights and biases. Though the solution space is high-dimensional, this approach produces results comparable in quality to the current reference neural texture compressors. We further extend this approach to infer multiple decoders at once or even produce decoders that learn super-resolution.
Read the full abstract ▾
Computer Vision
Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generation
Zekai Zhang, Jiahao Li, Jie Zhang et al. (+18 more)
While text-to-image (T2I) models have achieved remarkable progress, they struggle with real-world requests that are often underspecified, implicit, or dependent on up-to-date knowledge. We identify this challenge as the Context Gap: the mismatch between the user context and the sufficient generation context for T2I models. To bridge this gap, we propose Qwen-Image-
Agent, a unified agentic framework that integrates plan, reason, search,
memory and feedback in a context-centric manner. Qwen-Image-
Agent treats user input as partial context and progressively constructs the generation context through Context-Aware Planning and Context Grounding. Specifically, Context-Aware Planning identifies missing context and plans how it should be acquired and used, while Context Grounding gathers this context from reason, search,
memory, and feedback. To evaluate agentic image generation, we further introduce Image
Agent Bench (IA-Bench), a
benchmark covering four core image
agent capabilities: Plan, Reason, Search, and
Memory. Experiments on IA-Bench, Mindbench and WISE-Verified show that Qwen-Image-
Agent outperforms strong baselines and achieves state-of-the-art performance.
Read the full abstract ▾
All authors Zekai Zhang, Jiahao Li, Jie Zhang, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Shengming Yin, Tianhe Wu, Xiaoyue Chen, Xiao Xu, Yan Shu, Yanran Zhang, Yixian Xu, Yuxiang Chen, Zhendong Wang, Zihao Liu, Zikai Zhou, Huishuai Zhang, Dongyan Zhao, Chenfei Wu
Computer Vision
Confidence-Aware Tool Orchestration for Robust Video Understanding
Yangfan He, Yujin Choi and Jaehong Yoon
Video reasoning language models implicitly assume that every input frame is equally reliable. This leads to what we term the Blind Trust Problem: under realistic perturbations such as motion blur, glare, or occlusion, frontier
video reasoning models can suffer 15-30%p accuracy drops on real-world
embodied benchmarks, while remaining unaware that their visual evidence has been degraded. To address this challenge, we propose Robust-TO, an agentic
video understanding framework that explicitly integrates per-frame trustworthiness into every stage of
reasoning. Robust-TO organizes heterogeneous visual perception tools under a unified evidence interface. Each tool receives a sub-query derived from the original question and a set of trustworthy frames selected by the reliability-relevance score. It returns evidence in a shared format: a concrete prediction (e.g., a bounding box, motion trajectory, recognized text, or action label), temporal grounding, and a calibrated reliability score. During
reasoning, these calibrated scores guide evidence weighting in a three-tier synthesis process (high/medium/low) and define a confidence-cost GRPO reward that jointly optimizes correctness, evidence reliability, and efficiency. On two
video reasoning benchmarks spanning eight tasks, Robust-TO achieves 56.4% average accuracy on clean inputs, surpassing the strongest open-source baseline by 10.6%p and outperforming Gemini-2.5-Pro (46.2%). Under five realistic corruption types, Robust-TO maintains 54.3% average accuracy, 5.8%p above the strongest open-source baseline, while exhibiting the smallest clean-to-corrupted accuracy drop among all compared methods.
Read the full abstract ▾
Computer Vision
Tractography-Driven Synthetic Data Generation for Fiber Bundle Segmentation in Tracer Histology
Kyriaki-Margarita Bintsi, Sparsh Makharia, Yaël Balbastre et al. (+4 more)
Diffusion MRI (dMRI) tractography enables non-invasive reconstruction of white-matter pathways, but its accuracy is fundamentally limited by indirect, low-resolution measurements of axonal organization. Tracer injection studies in non-human primates provide a gold standard for validating dMRI tractography. This, however, requires time-consuming manual annotation of fiber bundles in histology sections. We propose a synthetic-data augmented framework for automated fiber bundle
segmentation in macaque tracer histology. Our approach uses ex vivo dMRI tractography as a generative prior to synthesize 2D image patches for training. This provides us with sufficiently realistic foreground texture, which we compose with backgrounds from blockface photos and diversify via domain randomization. A 2D U-Net is trained on mixed real and synthetic patches. Experiments on held-out
brains demonstrate improved generalization across
brains and fiber bundle densities compared to training with real data only. Training with synthetic data only leads to poor performance, underscoring the need for real supervision. Overall, our approach achieves performance comparable to the state-of-the-art while requiring 3x less manually annotated data.
Read the full abstract ▾
All authors Kyriaki-Margarita Bintsi, Sparsh Makharia, Yaël Balbastre, Joselyn Romero Avila, Julia F. Lehman, Suzanne N. Haber, Anastasia Yendiki
Computer Vision
Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI
Minh Duc Do, Tillmann Rheude, Noel Kronenberg et al. (+2 more)
Brain MRI poses a fundamental challenge for machine learning: models must learn from high-dimensional 3D data spanning multiple co-registered modalities, despite the limited sample sizes typical of neuroimaging studies relative to the diversity in anatomy, pathology, and acquisition conditions. While
multimodal imaging provides complementary information critical for clinical interpretation, effectively integrating these signals remains difficult. We propose
Multimodal Intra- and Cross-Context
Vision Transformer (MICViT), a 3D
vision transformer that explicitly models both modality-specific representations and cross-modal interactions across local and global contexts. Concretely, MICViT combines four
attention mechanisms: modality-specific local and global
attention for intra-modal feature learning, and cross-modal local and global
attention to capture interactions between modalities. We evaluate MICViT on
brain age prediction across three heterogeneous
datasets (UK Biobank, n=41,404; SOOP, n=1,062; Cam-CAN, n=613) using multiple MRI modalities (e.g. T1, FLAIR, DWI, SWI). MICViT consistently outperforms state-of-the-art CNN and
transformer baselines in 3D settings. Notably, it benefits more strongly from
multimodal inputs, yielding larger performance gains as additional modalities are incorporated. These results demonstrate that explicitly modeling intra- and cross-modal interactions is key to unlocking the full potential of
multimodal brain MRI, highlighting a promising direction for representation learning in neuroimaging.
Read the full abstract ▾
All authors Minh Duc Do, Tillmann Rheude, Noel Kronenberg, Roland Eils, Benjamin Wild
Computer Vision
Bridging Vision and Language Concepts through Optimal Transport Semantic Flow
Chenyang Zhang, Anqi Dong, Guangming Zhu et al. (+4 more)
Concept Bottleneck Models (CBMs) promise transparent
reasoning by predicting through human-interpretable concepts, yet their effectiveness fundamentally depends on how well visual and textual representations are aligned or matched. Existing
vision-
language CBMs often rely on pre-aligned encoders or global cosine similarity, which obscures fine-grained concept localization and fails to reflect true semantic geometry. In this work, we rethink concept
alignment as a dynamic cross-modal transport process instead of static projection and propose the Optimal Transport Flow Concept Bottleneck Model (OTF-CBM). It first learns a data-driven semantic cost via Inverse Optimal Transport to measure cross-modal distances, and then performs unbalanced optimal-transport-based flow matching to model semantic transitions between visual patches and textual concepts. With velocity-based concept activation, OTF-CBM captures interpretable
geometric relations without ODE integration. Experiments further show that OTF-CBM achieves superior classification accuracy and concept faithfulness, offering a new
geometric and dynamical perspective for interpretable cross-modal
reasoning.
Read the full abstract ▾
All authors Chenyang Zhang, Anqi Dong, Guangming Zhu, Nuoye Xiong, Siyuan Wang, Lin Mei, Liang Zhang
Computer Vision
RIS-Assisted Proactive Handover for Reliable mmWave Wireless Networks
Alaa Adnan, Mohammad Al-Quraan, Ahmed Zoha et al. (+5 more)
Millimeter-wave (mmWave) networks are highly susceptible to line-of-sight (LoS) blockages.
Vision-aided wireless communications (VAWC) enable proactive handovers (PHO) to mitigate such blockages; however, PHO becomes challenging when no nearby base station (BS) is available. In such cases, reconfigurable intelligent surfaces (RIS) can be used to restore connectivity. To ensure timely PHO, the RIS configuration time must be taken into account, as the large number of RIS elements can limit responsiveness in time-sensitive scenarios. This work proposes a novel RIS-assisted PHO approach that optimizes the number of allocated RIS elements to balance signal processing complexity and link quality under handover timing constraints, making the RIS-assisted link more energy-efficient. An optimization problem based on particle swarm optimization (PSO) is formulated to determine the optimal end-to-end RIS link setup that runs offline to bypass latency constraints. Results show that reducing the number of RIS elements by 12% leads to a 10% decrease in dissipated energy without compromising the signal-to-noise ratio (SNR). Moreover, the RIS-assisted link achieves a 15–30 dB improvement in blocked regions while maintaining accurate PHO timing.
Read the full abstract ▾
All authors Alaa Adnan, Mohammad Al-Quraan, Ahmed Zoha, M. Majid Butt, Sami Muhaidat, Muhammad Ali Imran, Marco Di Renzo, Lina Mohjazi
Computer Vision
SpatialFlow-GRPO: Where Spatial Credit Drives Image Editing
Yankai Yang, Yancheng Long, Wei Chen et al. (+7 more)
Recent online reinforcement learning has substantially improved image editing quality. However, existing Flow-GRPO-style methods usually rely on a single whole-image reward, which makes fine-grained editing optimization difficult. We observe that a key obstacle in image editing is this spatial uniformity assumption: a whole-image reward cannot distinguish how different spatial regions contribute to image quality. To address this issue, we propose SpatialFlow-GRPO, a training framework that introduces spatially fine-grained reward feedback. The framework converts region-aware rewards into semantic-region-level optimization signals and aligns region advantages with the corresponding latent positions during policy updates. We also train a region-aware reward model, SFReward, construct SFReward-14K with region-annotated editing samples, and introduce MultiEditBench to evaluate multi-region editing ability. On OmniGen2 and FLUX.2-klein-4B, SpatialFlow-GRPO outperforms Flow-GRPO on GEdit-Bench, ImgEdit-Bench, and MultiEditBench. The results show that SpatialFlow-GRPO converts local feedback into spatially aligned update signals and improves editing quality.
Read the full abstract ▾
All authors Yankai Yang, Yancheng Long, Wei Chen, Xingyu Lu, Hongyang Wei, Bin Wen, Fan Yang, Tingting Gao, Han Li, Shuo Yang
Computer Vision
Rolling Shutter Relative Pose Estimation Made Practical
Daniel Barath
Rolling shutter (RS) cameras equip virtually all consumer devices, yet RS-aware relative pose estimation has remained impractical: the state-of-the-art solver requires a minimum of 20 point correspondences, making RANSAC-based robust estimation prohibitively expensive due to the exponential dependence of the iteration count on the sample size. We make RS relative pose estimation practical by introducing affine correspondences (ACs) into the RS two-view geometry. We derive novel RS-corrected affine constraints that account for the coupling between point perturbations and the row-dependent essential
matrix, providing two equations per correspondence beyond the standard epipolar constraint. Building on these constraints, we develop a linearized algebraic solver that estimates pose and RS motion from only 7 ACs. The solver exploits the physical smallness of RS parameters to linearize the constraints, eliminates the 12 RS unknowns via null-space projection, and solves the remaining degree-20 system via action
matrices in 1.2\,ms. On the TUM RS
benchmark, our method achieves the best pose and RS parameter accuracy among all tested methods and, uniquely among RS solvers, provides accurate translational velocity estimates – which are poorly conditioned from point correspondences alone due to a $\vec{v}$-$\vec{t}$ coupling. On the global-shutter EuRoC MAV
dataset, the solver achieves comparable accuracy to the standard 5-point
algorithm, demonstrating that it generalizes well to the GS setting. Code is at https://github.com/danini/rolling_shutter_made_practical.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
Appearance-Preserving Refinement of Generated 3D Assets for Monochromatic Fabrication
Chentao Shen, Chen Jia, Mingjie Huang et al. (+3 more)
Recent advances in 3D mesh generation have enabled the creation of visually realistic assets. However, much of their visual fidelity is encoded in textures rather than geometry. When such assets are fabricated using monochromatic materials, texture information is largely lost, causing visually important details to disappear even when the original geometry is faithfully preserved. A key challenge is that the
geometric perturbations required to recover texture-dependent appearance cues often introduce sharp local features and high-frequency surface structures, which may increase stress concentration and fabrication risk. In this paper, we present GenMF, an appearance-oriented geometry refinement framework for monochromatic fabrication. GenMF transforms texture-dependent visual cues into geometry-induced shading effects and formulates geometry refinement as a balance between appearance preservation and fabrication-oriented robustness. To discourage structurally and narrow the gap between simulation and physical manufacturing, we further introduce a differentiable stress-aware regularization based on a learned thermal-stress predictor. Experimental results demonstrate that GenMF significantly improves appearance preservation under monochromatic rendering while reducing stress concentration under a consistent thermo-mechanical simulation setting. Physical 3D printing examples further show that the refined geometries preserve more recognizable visual details while remaining suitable for fabrication. These results suggest that appearance-aware geometry refinement provides an effective bridge between generated 3D assets and fabrication-ready monochromatic objects.
Read the full abstract ▾
All authors Chentao Shen, Chen Jia, Mingjie Huang, Zhuang Zhang, Haisen Zhao, Xiangru Huang
Computer Vision
Liquid Fusion of Heterogeneous Representations Towards General Salient Object Detection
Ke Chen, Ling Zhou, Guangqi Jiang et al. (+3 more)
General Salient Object Detection (SOD) aims to identify and segment visually interesting objects from uni-modality or multi-modality scenes, recently advanced by cutting-edge State Space Models (SSMs). However, a critical limitation of current approaches is their neglect of the inherent spectral biases exhibited by different neural network paradigms. By digging to the
dataset-level spectral analysis of Convolutional Neural Networks (CNNs) and SSMs, their semantic representations are inherently complementary based on their complementary frequency preferences. Inspired by this, we harmonize heterogeneous representations from SSMs and CNNs to bridge their spectral biases for general salient object detection. To this end, inspired by the dynamic information propagation of
Liquid Neural Networks (LNNs), we introduce a
liquid fusion to dynamically integrates features from two backbones, including VMamba and ConvNeXt, referred to
Liquid Fusion Network (LFNet). Concretely, by treating the continuous VMamba features and ConvNeXt features as evolving states and exogenous stimulus, respectively, LFNet employs a dynamic gating mechanism for content-aware feature aggregation. Crucially, this state-stimulus paradigm enables to scale to multi-modal cues, resulting in flexibility in general SOD. Besides, a Saliency-Guided Upsampling (SGU) operator to propagate the features to the shallow layer, which leverages a spectral-spatial co-design to suppress upsampling artifacts while preserving semantics. Extensive experiments across five diverse tasks (RGB, RGB-D, RGB-T, VSOD, and VDT) demonstrate that LFNet achieves state-of-the-art performance, offering a superior trade-off between detection accuracy and model efficiency. Code has been released at https://github.com/cke520/LFNet.
Read the full abstract ▾
All authors Ke Chen, Ling Zhou, Guangqi Jiang, Gengshen Wu, Yi Liu, Shoukun Xu
Computer Vision · Cross-listed from cs.RO
Ordinal Neural Collapse as a Representation Prior for Visual Navigation
E-In Son, Jung-Taak Kim and Seung-Woo Seo
Learning robust
navigation policies directly from visual observations remains a fundamental challenge in
vision-based robotic
navigation. In end-to-end imitation learning approaches, the visual encoder and action decoder are jointly optimized using a single action loss, which provides only an indirect supervisory signal to the encoder. This indirect supervision frequently results in the encoder learning ambiguous, action-agnostic representations. The problem is further complicated by substantial variations in scene structure and appearance across diverse environments, as well as the prevalence of visual distractors inherent to real-world
navigation settings. Such action-agnostic features cause the
navigation policy to produce inconsistent actions at ambiguous decision points, leading to
navigation failure. To overcome these limitations, we propose ORION (Ordinal Neural Collapse for Visual
Navigation), a method that explicitly organizes the encoder's representation space according to the ordinal structure of
navigation actions. In the context of goal-directed
navigation, ego-centric control categories from Far Left to Far Right exhibit a natural ordinal relationship in which neighboring classes share similar visual contexts, while semantically opposing classes differ substantially in appearance. We encourage class representations to be arranged sequentially along a single discriminative axis, while suppressing off-axis variance within each class. The pretrained encoder is then integrated into a
diffusion-based
navigation framework, and the full pipeline is fine-tuned end-to-end. Extensive experiments in both simulation and real-world settings show that ORION consistently outperforms end-to-end and neural collapse baselines in
navigation success rate and goal progress, with notable gains in visually challenging scenarios such as complex multi-way intersections.
Read the full abstract ▾
Computer Vision
Identifying the Unknown: Prompt-Free Open Vocabulary Anomaly Recognition for Robot-Object Interaction
Philipp Allgeuer, Jan-Gerrit Habekost and Stefan Wermter
Robots operating in real-world environments must in general be able to recognize previously unseen objects. As robotic systems move toward open-world autonomy, there is a growing, yet largely unmet, need for open vocabulary object detectors that are
prompt-free and efficient enough for continuous deployment. We present AnomNOVIC, a two-stage known-workspace framework that combines a masked autoencoder (MAE) trained for anomaly detection, with NOVIC, a powerful real-time
prompt-free open vocabulary image classifier. The MAE produces generic object-agnostic bounding boxes, allowing NOVIC to classify salient image regions without requiring a predefined candidate class list. We evaluate AnomNOVIC against strong open vocabulary baselines in a tabletop
robot-object environment featuring the NICOL
humanoid robot, reaching 47.1% AP / 57.5% AP50 for
prompt-free recognition, and 59.0% AP / 72.5% AP50 if class candidates are provided. Across additional
datasets, including an in-the-wild test set with 48 unique objects, AnomNOVIC reaches up to 82.6%
prompt-free detection and classification accuracy. These results significantly surpass all tested open vocabulary baselines, including YOLO-World-v2, OWLv2, and YOLOE.
Read the full abstract ▾
Computer Vision
Learning Adversarial Augmentation Policies for Robust Garlic Seedling Detection
Soeun Lee, Chanho Kim, Yeji Kang et al. (+2 more)
Accurate seedling detection during early growth stages is essential for timely replanting and effective crop management in precision agriculture. However, existing studies are mostly evaluated under relatively stable imaging conditions, such as
UAV imagery or greenhouse environments, leaving robust detection under severe and spatially heterogeneous illumination in ground-based outdoor monitoring insufficiently explored. In addition, many illumination-robust detection methods rely on additional enhancement or feature-extraction modules, which increase inference-time overhead and are not tailored to seedling detection and downstream missing seedling localization. To address these gaps, we construct a new garlic seedling
dataset captured using a ground-based monitoring platform under real outdoor field conditions with highly variable illumination. We further propose an illumination-robust seedling detection framework based on
adversarial augmentation policy learning. The proposed method jointly optimizes a
stochastic augmentation policy
agent and an object detector, enabling the detector to learn robust representations under challenging visual conditions. A structural penalty is introduced to prevent unrealistic distortions while encouraging challenging augmentations during training. Extensive experiments show that the proposed approach achieves an AP$_{50}$ of 91.6%, improving the baseline by 0.9 percentage points and outperforming the previous best-performing method by 0.2 percentage points. For downstream missing seedling localization, it achieves 75.0% precision and a 67.0% F1-score, improving the baseline by 4.8 and 2.0 percentage points, respectively. These results demonstrate the effectiveness of the proposed framework for practical ground-based agricultural monitoring under complex outdoor lighting conditions without additional inference-time computational overhead.
Read the full abstract ▾
All authors Soeun Lee, Chanho Kim, Yeji Kang, YoungKi Hong, Byeongkeun Kang
Computer Vision
Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction
Xilai Li, Xiaosong Li, Haishu Tan et al. (+3 more)
Multi-modality image fusion (MMIF) enhances scene representation by exploiting complementary cues from different modalities. Adverse weather, however, causes significant image degradation, disrupting feature representation and requiring simultaneous feature restoration and cross-modal complementarity. Existing methods often struggle with effective representation learning under such conditions, limiting their practical performance. To address these challenges, we propose a mask-guided MMIF method that integrates feature restoration and interaction. We first introduce "Pseudo Ground Truth" to simplify training, promoting faster and more effective feature learning. Then, we design a mask generation mechanism based on the mapping relationship between the fused result and the source images, quantifying the relative contribution of each modality during the fusion process. By incorporating the proposed mask-guided cross-modal cross-
attention mechanism, the network is encouraged to selectively attend to informative features during modality interaction, mitigating the risk of overfitting to the static distribution of the "Pseudo Ground Truth". Additionally, we propose a mask-guided learning strategy and a task-coupled degradation-aware learning strategy to balance feature restoration and interaction. Extensive experiments on synthetic and real-world
datasets demonstrate that our method surpasses state-of-the-art approaches in visual quality, quantitative metrics, and downstream tasks. The source code is available at https://github.com/ixilai/AMG-Fuse.
Read the full abstract ▾
All authors Xilai Li, Xiaosong Li, Haishu Tan, Tao Ye, Huafeng Li, Hongbin Wang
Computer Vision · Cross-listed from cs.RO
Improving Vision-Language-Action Model Fine-Tuning with Structured Stage and Keyframe Supervision
Yuan Xu, Yixiang Chen, Kai Wang et al. (+5 more)
Vision-
Language-Action (VLA) models have shown strong potential for generalizable robotic manipulation. During fine-tuning, however, action supervision applies equally across all timesteps, without structured supervision on which manipulation stage the
robot is in or what the next gripper-event target should be. This causes failures to concentrate around challenging gripper-event transitions. To address this, we propose StaKe, a plug-in auxiliary supervision framework that automatically derives two complementary signals from demonstration gripper states without manual annotation: a stage classifier that identifies the current manipulation stage, and a keyframe predictor that estimates the target joint action at the next gripper transition. Both are modeled as lightweight auxiliary heads that enrich the learned representations during training, while leaving the base VLA policy architecture and inference loop unchanged. Experiments on bimanual simulation and single-arm Franka real-
robot tasks show that StaKe consistently improves success rates (relative gains of 14% and 56%, respectively), with larger improvements on longer-horizon tasks that involve more gripper-event transitions. Ablation studies validate each design choice, and qualitative analysis confirms that the learned representations faithfully track manipulation stages. These results indicate that structured supervision is an effective and general strategy for enhancing VLA fine-tuning in long-horizon manipulation. Project website: https://hi-yuanxu.github.io/StaKe-Web/
Read the full abstract ▾
All authors Yuan Xu, Yixiang Chen, Kai Wang, Jiabing Yang, Peiyan Li, Qisen Ma, Yan Huang, Liang Wang
Computer Vision
NaviCache: Test-Time Self-Calibration Caching for Video Generation
Zheqi Lv, Zhibo Zhu, Jinke Wang et al. (+6 more)
Video Diffusion Models (VDMs) is constrained by immense computational costs. While offline calibration-based acceleration suffers from calibration data dependency, prohibitive calibration duration, and susceptibility to distribution shifts, offline calibration-free methods eliminate these hurdles. However, since they rely on instantaneous zero-order approximations where the mapping between input and output differences varies in real-time, they are susceptible to observational noise and ignore the intrinsic momentum within the
diffusion trajectory. In this paper, we propose NaviCache, a plug-and-play test-time self-calibration method re-conceptualizing feature
evolution as an Inertial
Navigation System (INS) problem. NaviCache bridges the fundamental domain gap and the non-stationary nature of
diffusion by modeling the relative coupling between input and output variations. We introduce a dual-state estimation architecture that adaptively tracks the feature change ratio and its latent drift, initialized via a specialized Initial
Alignment phase. By integrating a time-dependent noise schedule with an uncertainty-aware Measurement Update mechanism, NaviCache provides a theoretically grounded mechanism for error-bounded computation skipping. Extensive experiments on the HunyuanVideo, Wan, and Open-Sora series demonstrate that NaviCache exhibits more accurate error judgment for computation skipping and achieves outstanding comprehensive performance.
Read the full abstract ▾
All authors Zheqi Lv, Zhibo Zhu, Jinke Wang, Qi Tian, Shengyu Zhang, Zhengyu Chen, Chengxi Zang, Zhou Zhao, Fei Wu
Computer Vision
ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP
Sicheng Zhang, Muzammal Naseer, Binzhu Xie et al. (+5 more)
CLIP and its variants are widely adopted visual backbones in
multimodal systems, but their pretraining remains dominated by descriptive image-text
alignment. As downstream applications increasingly demand visually grounded commonsense inference and compositional
reasoning, it remains unclear whether CLIP-style encoders can support such
reasoning without architectural changes. To address this, we present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale
reasoning supervision into CLIP-style models through our two-stage strategy, which progressively integrates
reasoning signals while preserving descriptive
alignment, followed by category-structured
reasoning supervision. To support this framework, we construct two complementary
datasets and a
benchmark: ReasonLite-42M, with open-form, visually verifiable
reasoning captions; ReasonPro-16M, with category-specific
reasoning supervision; and RCLIP-Bench for diagnostic evaluation of visually grounded
reasoning. We train a family of ReasonCLIP that improves visually grounded commonsense and compositional
reasoning while also enhancing zero-shot
retrieval performance. As a drop-in visual encoder for
multimodal large
language models such as LLaVA-NeXT, ReasonCLIP delivers consistent gains without additional inference cost, demonstrating that structured
reasoning supervision enhances the expressive capacity of CLIP-style visual representations. All
datasets, models, and training code are available at https://github.com/RISys-Lab/ReasonCLIP.
Read the full abstract ▾
All authors Sicheng Zhang, Muzammal Naseer, Binzhu Xie, Naufal Suryanto, Shi Qiu, Jamal Bentahar, Naveed Akhtar, Mubarak Shah
Computer Vision
Event-based Gaze Control System for Accurate Real-time Spin Estimation in Professional Ball Games
Yunpu Hu, Fabian Schilling, Valentina Cavinato et al. (+6 more)
Spin plays a crucial role in many ball sports due to its effect on the trajectory of the ball.
Vision-based estimation of the ball's
spin during a game with conventional cameras is challenging due to the ball's small size, high speed, and fast rotation. To address these challenges, we propose an event-based active
vision system that can track unmodified balls and measure their
spin in real time. The system consists of an event camera for its high temporal resolution and minimal motion blur, high-speed pan/tilt galvanometer mirrors to keep the ball in the field of view, and a low-latency focus-tunable telephoto lens to increase the spatial resolution on the ball and keep it in focus. To track the ball, we use a hybrid approach that combines 2D event-based detection for centering and 3D positions from a ball localization system for re-initialization. For high-accuracy
spin estimation, we propose an offline method that performs contrast maximization on the sphere (s-CMax). This method achieves state-of-the-art accuracy on static balls across multiple sports (table tennis, baseball, tennis, and golf), with mean magnitude and axis errors of 1.2% and 1.5 degrees, respectively. We then develop a low-latency online method for table tennis as a case study in real-time applications. This method uses an uncertainty-aware convolutional neural network trained on pseudo-ground-truth
spin labels from the offline approach, combined with a GPU-accelerated batch implementation of contrast maximization for refinement. We demonstrate reliable tracking and
spin estimation with a three-view setup during professional table tennis matches, with high accuracy (8.8% magnitude and 6.4 degrees axis mismatch w.r.t. the offline method), 3 ms latency, and 750 Hz throughput.
Read the full abstract ▾
All authors Yunpu Hu, Fabian Schilling, Valentina Cavinato, Asude Aydin, Agis Politis, Ricardo Tapiador Morales, Kirk Y.W. Scheper, Peter Dürr, Naoya Takahashi
Computer Vision
LearniBridge: Learnable Calibration of Feature Caching for Diffusion Models Acceleration
Xuyue Huang, Zhe Chen, Wang Shen et al. (+1 more)
Diffusion Transformers (DiTs) have driven substantial progress in image and
video generation but suffer from prohibitive computational costs. Feature caching accelerates inference by reusing intermediate representations. Existing methods rely on historical features for implementation simplicity, yet suffer from severe error accumulation at high acceleration ratios. To address this limitation, we investigate the nature of the requisite feature correction. We demonstrate that the optimal calibration update is characterized by a shared low-rank subspace across diverse
prompts. Guided by this structural insight, we propose LearniBridge, a learnable calibration mechanism for feature caching that bridges multiple timesteps through lightweight LoRA updates. This mechanism enables effective calibration requiring only 3-5 training samples. Extensive experiments on image and
video generation show that LearniBridge achieves up to $5.87\times$, $5.75\times$, and $4.10\times$ acceleration on FLUX, HunyuanVideo, and WAN2.1, respectively. On WAN2.1, it improves VBench by 1.28% over the previous SOTA at $4.10\times$ acceleration. Our code is available at https://github.com/Iiiiiiirene/LearniBridge.
Read the full abstract ▾
All authors Xuyue Huang, Zhe Chen, Wang Shen, Xiao-Ping Zhang
Computer Vision · Cross-listed from cs.AI
ResilPhase: Plug-and-Play Phase Mapping and Noise-Resilient Macro-Trajectory Extrapolation for Diffusion Acceleration
Qicheng Zhao, Yu Li, Qi Sun et al. (+1 more)
The adoption of powerful
diffusion models is hindered by their significant inference latency. Recent “cache-then-forecast” schemes alleviate this issue by accelerating DiTs using derivative-based
polynomials, but they suffer from severe quality degradation at high acceleration ratios. Our analysis reveals its root cause: the discrete extrapolation performed on representations that are misaligned with the continuous
diffusion trajectory and are numerically unstable. Thus, accelerated DiTs suffer from accumulated spatial errors, noisy derivative amplification, and high-order instability. We therefore reformulate accelerated inference as stable macro-trajectory extrapolation in ordinary differential equation (ODE) space. Instead of predicting intermediate features, we align
forecasting with the model's Global Drift (GD), i.e., the end-to-end state
evolution, thereby eliminating feature inconsistency and
memory overhead. However, even this smooth macro-trajectory remains vulnerable to the derivative fallacy: its higher-order temporal derivatives are intrinsically noisy. Thus, we introduce a derivative-free barycentric Lagrange extrapolator to effectively bypass derivative instability and approximation error. We further propose a bounded Phase Mapping that regularizes the extrapolation domain, suppressing oscillatory error growth. These elements collectively constitute ResilPhase, a noise-resilient acceleration framework. Experiments on FLUX.1-dev and HunyuanVideo demonstrate state-of-the-art fidelity under aggressive acceleration ratios.
Read the full abstract ▾
All authors Qicheng Zhao, Yu Li, Qi Sun, Zheyu Yan
Computer Vision
Anatomy-Guided Residual Motion Diffusion for Controllable 4D Cardiac MRI Synthesis
Yiheng Cao, Gustavo Andrade-Miranda, Jiatian Zhang et al. (+2 more)
Developing robust artificial intelligence models for 4D (3D + time) medical imaging is constrained by limited annotated data, inter-device domain shifts, and
privacy restrictions. To address this, we propose a 4D controllable generative framework for anatomically consistent data augmentation. A semi-supervised variational autoencoder learns a compact latent representation of anatomical volumes while jointly predicting aligned
segmentation masks in a unified framework. Anatomical structure is then disentangled from temporal dynamics through a cascaded latent
diffusion model (LDM). A static LDM generates subject-specific anatomy conditioned on clinical priors (diagnosis and volumes measures) and a subsequent motion LDM estimates residual latent motions, ensuring strict temporal coherence across the 4D sequence. The proposed approach was evaluated on cine cardiac MRI as a representative 4D imaging application. Experiments across multiple
datasets demonstrate high controllability of static anatomy (Pearson r > 0.8) and strong temporal coherence (FVD = 288.08). In cross-vendor generalization experiments, augmenting training sets with synthetic 4D sequences significantly improves downstream
segmentation performance. Using nnU-Net, the proposed augmentation strategy improves the average Dice score by 1.4% and reduces the Hausdorff Distance by 3.0mm compared to training on real data alone, for the left ventricle, Dice improves by 2.8% with a 5.4mm reduction in boundary error. Overall, this framework provides a scalable and controllable solution for 4D medical image synthesis, supporting the development of more robust models with limited annotations and cross-vendor variability. Code available on https://github.com/cyiheng/4DCardiacMRISynthesis.
Read the full abstract ▾
All authors Yiheng Cao, Gustavo Andrade-Miranda, Jiatian Zhang, Lingxiao Zhao, Xin Gao
Computer Vision
Calibrated Harmonic Overlaid Implicit Neural Representations for Multi-Dimensional Data
Honghang Chen, Xiujun Zhang, Xiaoli Sun et al. (+1 more)
Implicit neural representation (INR) has emerged as a powerful prior for multi-dimensional data (e.g., multispectral images and
videos). However, most INR methods employing periodic activation functions (e.g., Sine) predominantly rely on function composition. This mechanism introduces optimization instability as network depth increases, thereby limiting their performance. Meanwhile, these methods fail to incorporate proper physical priors to effectively alleviate spectrum bias. To address these issues, inspired by the commonalities between deep periodic networks and generalized
Fourier series, we propose a novel Calibrated Harmonic Overlaid Implicit Neural Representation (CHOIR). Specifically, we utilize Coordinated Harmonic Superposition (CHS) to replace the conventional function composition used in most INRs, thereby ensuring optimization stability when scaling network depth. Furthermore, we introduce a Perceptual Spectrum Calibration (PSC) to mitigate spectrum bias. This calibration embeds the ubiquitous power-law spectrum prior of natural images and adjusts the globally fixed spectrum towards a physically plausible log-uniform distribution. Extensive experiments on various multidimensional data recovery problems demonstrate that our method achieves superior performance over state-of-the-art approaches. Code is available at https://github.com/chorl0229/CHOIR.
Read the full abstract ▾
All authors Honghang Chen, Xiujun Zhang, Xiaoli Sun, Mingqing Xiao
Computer Vision
ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory
Le Tu Ngoc Minh, Jinyeong Lim and Dongsu Han
Streaming
video understanding (SVU) must answer queries that arrive asynchronously while visual
tokens stream continuously under strict GPU-
memory and query-time latency budgets. A key challenge is delayed query: decisive cues may appear briefly, yet many subsequent updates occur before the query arrives, increasing the risk that those cues are evicted or diluted under bounded
memory. We propose ProtoKV, a constant-footprint SVU
memory that represents far history as a fixed-capacity summary state rather than retaining
token instances. ProtoKV keeps an exact near-window KV cache and aggregates older content into a semantic-spatial prototype bank with residual statistics. At query time, each prototype is exposed through a bounded pseudo-
token interface that is drop-in compatible with standard
attention. Under matched budgets and comparable query-time cost, ProtoKV improves accuracy by up to 12.5 points over
token-retention baselines on SVU
benchmarks in the long-delay regime, with gains that grow as query delay increases.
Read the full abstract ▾
Computer Vision
Capacity-Controlled Multi-View Stylization of 3D Gaussian Splatting
Zhihao Wen, Yixin Yang, Bojian Wu et al. (+4 more)
While 3D Gaussian Splatting (3DGS) provides an efficient and explicit representation for novel view synthesis, enforcing stylistic coherence across viewpoints remains challenging. Existing 3D stylization methods typically apply 2D feature-matching losses independently per rendered view, which leads to unstable style allocation, many-to-one feature reuse, and limited cross-view consistency. We propose a capacity-controlled framework for multi-view stylization of 3DGS, grounded in optimal transport. Specifically, we reformulate local style matching as a semi-balanced optimal transport problem. By introducing explicit column-capacity constraints with tunable strength, our formulation mitigates many-to-one matching and enables controllable allocation of style features. This transport-based objective provides a principled mechanism for balancing feature coverage and stylistic diversity while maintaining stable correspondences across viewpoints. To further enhance cross-view coherence, we incorporate a novel cross-view matching guidance to constrain correspondences between scene content and style patterns. In addition, we introduce several
geometric regularizations to enhance the vanilla 3DGS, thereby enabling optimized Gaussian primitives to represent finer-grained textures during stylization. Extensive experiments demonstrate that our approach significantly improves multi-view stylistic consistency and produces stable, expressive 3D stylizations while preserving the core semantic structure of the scene.
Read the full abstract ▾
All authors Zhihao Wen, Yixin Yang, Bojian Wu, Yang Zhou, Dani Lischinski, Daniel Cohen-Or, Hui Huang
Computer Vision
Depth-Semantic Alignment and Affinity-Guided Fusion for Structured Radar Point Cloud Generation
Amjad Hussain, Wenjie Liu, Yuchen Tan et al. (+4 more)
Point clouds are an important carrier of three-dimensional spatial information, and their quality directly affects the performance of downstream perception tasks such as object detection and tracking. However, millimeter-wave radar point clouds are typically sparse, noisy, and structurally incomplete. To address these limitations, this paper proposes a
multimodal point cloud generation method based on
vision-radar fusion. The proposed method leverages image semantic information to impose structural constraints and achieve spatial
alignment for radar point clouds, while incorporating a sparse completion strategy to enhance point density and recover missing structures. The generated point clouds are further evaluated in object detection and tracking tasks. Experimental results demonstrate that the proposed method effectively improves point cloud quality and enhances the detection accuracy and robustness of perception models in complex environments, providing a practical solution for multisensor point cloud generation and intelligent perception systems.
Read the full abstract ▾
All authors Amjad Hussain, Wenjie Liu, Yuchen Tan, Fuyuan Ai, Zecheng Li, Chunyi Song, Xin Qiu
Computer Vision · Cross-listed from cs.RO
PressMimic: Pressure-Guided Motion Capture and Control for Humanoid Robot Imitation
Yi Lu, Shenghao Ren, Tianyu Xiong et al. (+6 more)
Humanoid motion imitation requires not only accurate perception of human kinematics but also faithful reproduction of physical interactions with the environment. However, existing pipelines rely primarily on
vision-based motion capture and kinematic imitation, largely ignoring contact dynamics, leading to artifacts such as foot sliding, floor penetration, and unstable behaviors. In this work, we revisit
humanoid motion imitation from the perspective of physical grounding and leverage pressure as a unified modality across perception and control. We present PressMimic, a framework that integrates pressure into the full pipeline from motion capture to
humanoid control. In the perception stage, we introduce FRAPPE++, a
multimodal model that fuses RGB and pressure to jointly estimate 3D pose and global motion, where pressure provides explicit contact and support constraints to resolve ambiguity in
vision-based estimation. In the control stage, we propose a pressure-supervised policy (PSP) that incorporates pressure-derived signals into reinforcement learning, enabling physically consistent contact patterns during execution. We further construct MotionPRO, a large-scale
dataset with synchronized RGB, pressure, and motion capture data. Experiments show that pressure improves motion estimation accuracy, trajectory consistency, and execution stability. These results demonstrate that pressure serves as an effective physical grounding signal, bridging perception and control for physically consistent
humanoid motion imitation.
Read the full abstract ▾
All authors Yi Lu, Shenghao Ren, Tianyu Xiong, Zhaoxiang Li, Jiaqi Li, He Zhang, Tao Yu, Qiu Shen, Xun Cao
Computer Vision
LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing
Xinyu Wang, Chongbo Zhao, Fangneng Zhan et al. (+1 more)
Streaming
video editing has made rapid progress, yet practical deployment is still limited by two core issues: maintaining stable backgrounds and non-edited regions over time, and achieving the low latency required for real-time interactive scenarios. Meanwhile, recent streaming
video generation methods are mostly developed for synthesis and cannot be directly applied to editing due to the strict preservation requirement and region-specific control. In this work, we present a novel streaming
video editing framework that performs causal, frame-by-frame editing with strong content preservation and real-time responsiveness. Our key design is a three-stage
distillation pipeline that progressively transfers editing capability from a powerful bidirectional foundation model to an efficient unidirectional streaming editor, enabling stable long-horizon edits without sacrificing visual fidelity. To further support real-time deployment, we introduce an AR-oriented mask cache that reuses region-related computation across frames, substantially reducing redundant processing and accelerating inference. Finally, we establish a dedicated
benchmark for streaming
video editing. Extensive evaluations demonstrate that our method achieves state-of-the-art visual quality among streaming baselines while drastically boosting inference speed to 12.66 FPS, making it suitable for interactive and augmented reality applications.
Read the full abstract ▾
All authors Xinyu Wang, Chongbo Zhao, Fangneng Zhan, Yue Ma
Computer Vision
Do Image Editing Models Understand Lighting?
Tim Küchler, Johann-Friedrich Feiden, Matthias Nießner et al. (+1 more)
While recent advancements in generative image editing models have achieved stunning visual fidelity, it remains an open question whether these systems possess an intrinsic knowledge of real-world lighting. Existing
benchmarks typically evaluate high-level plausibility of perceptual light transport on curated internet imagery, using VLMs or human judgement, or they rely on synthetically generated
datasets. In this work, we introduce the 3D-anchored Light Probe (3DLP)
benchmark, for which we have captured a new high-fidelity HDR
dataset of real-world lighting changes. The
dataset consists of 1K image pairs of diverse indoor scenery in which light probes are physically turned on and off. To allow for a granular performance analysis, we annotated specific image regions such as cast shadows or metallic surfaces. With this data, we evaluate a range of state-of-the-art image editing models by measuring how well their light probe edits align with reality. The evaluation uses two new scores to compensate for AI-generated photographic effects, such as adjusted white balance. Our results show that the overall performance of models differs considerably, with differences slightly less pronounced for specular highlights. The best image editing models are remarkably consistent with real-world physics, however, they still leave room for improvement. We observe that image regions that receive less light from the light probe are more prone to errors for all models. Furthermore, building on their success in evaluating macroscopic lighting plausibility, we test VLMs on our task but find that they are unsuitable for pixel-level light transport analysis. We will make the
benchmark, together with the real-world
dataset, publicly available to encourage future research on this topic.
Read the full abstract ▾
All authors Tim Küchler, Johann-Friedrich Feiden, Matthias Nießner, Carsten Rother