Computer Vision
ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation
Hao Zhang, Lue Fan, Weikang Bian et al. (+4 more)
We present ReinDriveGen, a framework that enables full controllability over dynamic driving scenes, allowing users to freely edit actor trajectories to simulate safety-critical corner cases such as front-vehicle collisions, drifting cars, vehicles spinning out of control, pedestrians jaywalking, and cyclists cutting across lanes. Our approach constructs a dynamic 3D point cloud scene from multi-frame
LiDAR data, introduces a vehicle completion module to reconstruct full 360{\deg} geometry from partial observations, and renders the edited scene into 2D condition images that guide a
video diffusion model to synthesize realistic driving
videos. Since such edited scenarios inevitably fall outside the training distribution, we further propose an RL-based post-training strategy with a pairwise preference model and a pairwise reward mechanism, enabling robust quality improvement under out-of-distribution conditions without ground-truth supervision. Extensive experiments demonstrate that ReinDriveGen outperforms existing approaches on edited driving scenarios and achieves state-of-the-art results on novel ego viewpoint synthesis.
Read the full abstract ▾
All authors Hao Zhang, Lue Fan, Weikang Bian, Zehuan Wu, Lewei Lu, Zhaoxiang Zhang, Hongsheng Li
Computer Vision
Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
Reyhaneh Ahani Manghotay and Jie Liang
Leveraging the rich semantic features of
vision-
language models (VLMs) like CLIP for
monocular depth estimation tasks is a promising direction, yet often requires extensive fine-tuning or lacks
geometric precision. We present a parameter-efficient framework, named MoA-DepthCLIP, that adapts pretrained CLIP representations for
monocular depth estimation with minimal supervision. Our method integrates a lightweight Mixture-of-Adapters (MoA) module into the pretrained
Vision Transformer (ViT-B/32) backbone combined with selective fine-tuning of the final layers. This design enables spatially-aware adaptation, guided by a global semantic context vector and a hybrid prediction architecture that synergizes depth bin classification with direct regression. To enhance structural accuracy, we employ a composite loss function that enforces
geometric constraints. On the NYU Depth V2
benchmark, MoA-DepthCLIP achieves competitive results, significantly outperforming the DepthCLIP baseline by improving the $\delta_1$ accuracy from 0.390 to 0.745 and reducing the RMSE from 1.176 to 0.520. These results are achieved while requiring substantially few trainable parameters, demonstrating that lightweight,
prompt-guided MoA is a highly effective strategy for transferring VLM knowledge to fine-grained
monocular depth estimation tasks.
Read the full abstract ▾
Computer Vision
ProTPS: Prototype-Guided Text Prompt Selection for Continual Learning
Jie Mei, Li-Leng Peng, Keith Fuller et al. (+1 more)
For continual learning, text-
prompt-based methods leverage text encoders and learnable
prompts to encode semantic features for sequentially arrived classes over time. A common challenge encountered by existing works is how to learn unique text
prompts, which implicitly carry semantic information of new classes, so that the semantic features of newly arrived classes do not overlap with those of trained classes, thereby mitigating the catastrophic forgetting problem. To address this challenge, we propose a novel approach Prototype-guided Text
Prompt Selection (ProTPS)” to intentionally increase the training flexibility thus encouraging the learning of unique text
prompts. Specifically, our ProTPS learns class-specific
vision prototypes and text
prompts.
Vision prototypes guide the selection and learning of text
prompts for each class. We first evaluate our ProTPS in both class incremental (CI) setting and cross-
datasets continual (CDC) learning setting. Because our ProTPS achieves performance close to the upper bounds, we further collect a real-world
dataset with 112 marine species collected over a span of six years, named Marine112, to bring new challenges to the community. Marine112 is authentically suited for the class and domain incremental (CDI) learning setting and is under natural long-tail distribution. The results under three settings show that our ProTPS performs favorably against the recent state-of-the-art methods. The implementation code and Marine112
dataset will be released upon the acceptance of our paper.
Read the full abstract ▾
All authors Jie Mei, Li-Leng Peng, Keith Fuller, Jenq-Neng Hwang
Computer Vision · Cross-listed from cs.SD
TRACE: Training-Free Partial Audio Deepfake Detection via Embedding Trajectory Analysis of Speech Foundation Models
Awais Khan, Muhammad Umar Farooq, Kutub Uddin et al. (+1 more)
Partial audio deepfakes, where synthesized segments are spliced into genuine recordings, are particularly deceptive because most of the audio remains authentic. Existing detectors are supervised: they require frame-level annotations, overfit to specific synthesis pipelines, and must be retrained as new generative models emerge. We argue that this supervision is unnecessary. We hypothesize that
speech foundation models implicitly encode a forensic signal: genuine
speech forms smooth, slowly varying
embedding trajectories, while splice boundaries introduce abrupt disruptions in frame-level transitions. Building on this, we propose TRACE (Training-free Representation-based Audio Countermeasure via
Embedding dynamics), a training-free framework that detects partial audio deepfakes by analyzing the first-order dynamics of frozen
speech foundation model representations without any training, labeled data, or architectural modification. We evaluate TRACE on four
benchmarks that span two
languages using six
speech foundation models. In PartialSpoof, TRACE achieves 8.08% EER, competitive with fine-tuned supervised baselines. In LlamaPartialSpoof, the most challenging
benchmark featuring LLM-driven commercial synthesis, TRACE surpasses a supervised baseline outright (24.12% vs. 24.49% EER) without any target-domain data. These results show that temporal dynamics in
speech foundation models provide an effective, generalize signal for training-free audio forensics.
Read the full abstract ▾
All authors Awais Khan, Muhammad Umar Farooq, Kutub Uddin, Khalid Malik
Computer Vision
ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data
Yaoqin Ye, Yiteng Xu, Qin Sun et al. (+3 more)
Human behaviors in real-world environments are inherently interactive, with an individual's motion shaped by surrounding
agents and the scene. Such capabilities are essential for applications in virtual avatars, interactive animation, and human-
robot collaboration. We target real-time human interaction-to-reaction generation, which generates the ego's future motion from dynamic multi-source cues, including others' actions, scene geometry, and optional high-level semantic inputs. This task is fundamentally challenging due to (i) limited and fragmented interaction data distributed across heterogeneous single-person, human-human, and human-scene domains, and (ii) the need to produce low-latency yet high-fidelity motion responses during continuous online interaction. To address these challenges, we propose ReMoGen (Reaction Motion Generation), a modular learning framework for real-time interaction-to-reaction generation. ReMoGen leverages a universal motion prior learned from large-scale single-person motion
datasets and adapts it to target interaction domains through independently trained Meta-Interaction modules, enabling robust generalization under data-scarce and heterogeneous supervision. To support responsive online interaction, ReMoGen performs segment-level generation together with a lightweight Frame-wise Segment Refinement module that incorporates newly observed cues at the frame level, improving both responsiveness and temporal coherence without expensive full-sequence inference. Extensive experiments across human-human, human-scene, and mixed-modality interaction settings show that ReMoGen produces high-quality, coherent, and responsive reactions, while generalizing effectively across diverse interaction scenarios.
Read the full abstract ▾
All authors Yaoqin Ye, Yiteng Xu, Qin Sun, Xinge Zhu, Yujing Sun, Yuexin Ma
Computer Vision
ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Prediction
Yuheng Zhang, Mengfei Duan, Kunyu Peng et al. (+5 more)
3D semantic occupancy prediction is central to autonomous driving, yet current methods are vulnerable to long-tailed class bias and out-of-distribution (OOD) inputs, often overconfidently assigning anomalies to rare classes. We present ProOOD, a lightweight, plug-and-play method that couples prototype-guided refinement with training-free OOD scoring. ProOOD comprises (i) prototype-guided semantic imputation that fills occluded regions with class-consistent features, (ii) prototype-guided tail mining that strengthens rare-class representations to curb OOD absorption, and (iii) EchoOOD, which fuses local logit coherence with local and global prototype matching to produce reliable voxel-level OOD scores. Extensive experiments on five
datasets demonstrate that ProOOD achieves state-of-the-art performance on both in-distribution 3D occupancy prediction and OOD detection. On SemanticKITTI, it surpasses baselines by +3.57% mIoU overall and +24.80% tail-class mIoU; on VAA-KITTI, it improves AuPRCr by +19.34 points, with consistent gains across
benchmarks. These improvements yield more calibrated occupancy estimates and more reliable OOD detection in safety-critical urban driving. The source code is publicly available at https://github.com/7uHeng/ProOOD.
Read the full abstract ▾
All authors Yuheng Zhang, Mengfei Duan, Kunyu Peng, Yuhang Wang, Di Wen, Danda Pani Paudel, Luc Van Gool, Kailun Yang
Computer Vision
PHASOR: Anatomy- and Phase-Consistent Volumetric Diffusion for CT Virtual Contrast Enhancement
Zilong Li, Dongyang Li, Chenglong Ma et al. (+6 more)
Contrast-enhanced computed
tomography (CECT) is pivotal for highlighting tissue perfusion and vascularity, yet its clinical ubiquity is impeded by the invasive nature of contrast
agents and radiation risks. While virtual contrast enhancement (VCE) offers an alternative to synthesizing CECT from non-contrast CT (NCCT), existing methods struggle with anatomical heterogeneity and spatial misalignment, leading to inconsistent enhancement patterns and incorrect details. This paper introduces PHASOR, a volumetric
diffusion framework for high-fidelity CT VCE. By treating CT volumes as coherent sequences, we leverage a
video diffusion model to enhance structural coherence and volumetric accuracy. To ensure anatomy-phase consistent synthesis, we introduce two complementary modules. First, anatomy-routed mixture-of-experts (AR-MoE) anchors distinct enhancement patterns to anatomical semantics, with organ-specific
memory to capture salient details. Second, intensity-phase aware representation
alignment (IP-REPA) highlights intricate contrast signals while mitigating the impact of imperfect spatial
alignment. Extensive experiments across three
datasets demonstrate that PHASOR significantly outperforms state-of-the-art methods in both synthesis quality and enhancement accuracy.
Read the full abstract ▾
All authors Zilong Li, Dongyang Li, Chenglong Ma, Zhan Feng, Dakai Jin, Junping Zhang, Hao Luo, Fan Wang, Hongming Shan
Computer Vision
CROWD: A global dataset of continuous urban dashcam driving
Md Shadab Alam, Olena Bazilinska and Pavlo Bazilinskyy
We introduce CROWD (City Road Observations With Dashcams), a manually curated
dataset of ordinary, variable duration, temporally contiguous, unedited, front facing urban dashcam segments screened and segmented from publicly available YouTube
videos. CROWD is designed to support cross-domain robustness and interaction analysis by prioritising routine driving and explicitly excluding crashes, crash aftermath, and other edited or incident-focused content. The release contains 107,407 segment records spanning 36,204.26 hours (82,484 unique uploads), covering 10,538 named inhabited places in 238 countries and territories across all six inhabited continents (Africa, Asia, Europe, North America, South America and Oceania), with segment level manual labels for time of day (day or night) and vehicle type. To lower the barrier for reproducible baseline analysis, we provide per-segment CSV files of automatic detections for all 80 MS-COCO classes produced with YOLOv11x, together with segment-local multi-object tracks generated with BoT-SORT; e.g. person, bicycle, motorcycle, car, bus, truck, traffic light, stop sign, etc. CROWD is distributed as
video identifiers with segment boundaries and derived detection and tracking pseudo-labels, enabling reproducible research without redistributing the underlying
videos. To support leakage-free evaluation, we also provide canonical splits grouped by source channel or by locality, and weights that balance the contribution of localities and countries.
Read the full abstract ▾
Computer Vision
ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration
Fengyuan Yang, Luying Huang, Jiazhi Guan et al. (+8 more)
Recent advances in
Video Foundation Models (VFMs) have revolutionized human-centric
video synthesis, yet fine-grained and independent editing of subjects and scenes remains a critical challenge. Recent attempts to incorporate richer environment control through rigid 3D
geometric compositions often encounter a stark trade-off between precise control and generative flexibility. Furthermore, the heavy 3D pre-processing still limits practical scalability. In this paper, we propose ONE-SHOT, a parameter-efficient framework for compositional human-environment
video generation. Our key insight is to factorize the generative process into disentangled signals. Specifically, we introduce a canonical-space injection mechanism that decouples human dynamics from environmental cues via cross-
attention. We also propose Dynamic-Grounded-RoPE, a novel positional
embedding strategy that establishes spatial correspondences between disparate spatial domains without any heuristic 3D
alignments. To support long-horizon synthesis, we introduce a Hybrid Context Integration mechanism to maintain subject and scene consistency across minute-level generations. Experiments demonstrate that our method significantly outperforms state-of-the-art methods, offering superior structural control and creative diversity for
video synthesis. Our project has been available on: https://martayang.github.io/ONE-SHOT/.
Read the full abstract ▾
All authors Fengyuan Yang, Luying Huang, Jiazhi Guan, Quanwei Yang, Dongwei Pan, Jianglin Fu, Haocheng Feng, Wei He, Kaisiyuan Wang, Hang Zhou, Angela Yao
Computer Vision
Foundation Model-guided Iteratively Prompting and Pseudo-Labeling for Partially Labeled Medical Image Segmentation
Qiaochu Zhao, Wei Wei, David Horowitz et al. (+2 more)
Automated medical image
segmentation has achieved remarkable progress with fully labeled data. However, site-specific clinical priorities and the high cost of manual annotation often yield scans with only a subset of organs labeled, leading to the partially labeled problem that degrades performance. To address this issue, we propose IPnP, an Iteratively Prompting and Pseudo-labeling framework, for partially labeled medical image
segmentation. IPnP iteratively generates and refines pseudo-labels for unlabeled organs through collaboration between a trainable
segmentation network (specialist) and a frozen foundation model (generalist), progressively recovering full-organ supervision. On the public
dataset AMOS with the simulated partial-label setting, IPnP consistently improves
segmentation performance over prior methods and approaches the performance of the fully labeled reference. We further evaluate on a private, partially labeled
dataset of 210 head-and-neck cancer patients and demonstrate our effectiveness in real-world clinical settings.
Read the full abstract ▾
All authors Qiaochu Zhao, Wei Wei, David Horowitz, Richard Bakst, Yading Yuan
Computer Vision
Sub-metre Lunar DEM Generation and Validation from Chandrayaan-2 OHRC Multi-View Imagery Using an Open-Source Pipeline
Aaranay Aadi, Jai Singla, Nitant Dube et al. (+1 more)
High-resolution digital elevation models (DEMs) of the lunar surface are essential for surface mobility planning, landing site characterization, and planetary science. The Orbiter High Resolution Camera (OHRC) on board Chandrayaan-2 has the best ground sampling capabilities of any lunar orbital imaging currently in use by acquiring panchromatic imagery at a resolution of roughly 20-30 cm per pixel. This work presents, for the first time, the generation of sub-metre DEMs from OHRC multi-view imagery using an exclusively open-source pipeline. Candidate stereo pairs are identified from non-paired OHRC archives through
geometric analysis of image metadata, employing baseline-to-height (B/H) ratio computation and convergence angle estimation. Dense stereo correspondence and ray triangulation are then applied to generate point clouds, which are gridded into DEMs at effective spatial resolutions between approximately 24 and 54 cm across five geographically distributed lunar sites. Absolute elevation consistency is established through Iterative Closest Point (ICP)
alignment against Lunar Reconnaissance Orbiter Narrow Angle Camera (NAC) Digital Terrain Models, followed by constant-bias offset correction. Validation against NAC reference terrain yields a vertical RMSE of 5.85 m (at native OHRC resolution), and a horizontal accuracy of within a pixel (approximately 30 cm) assessed by planimetric feature matching.
Read the full abstract ▾
All authors Aaranay Aadi, Jai Singla, Nitant Dube, Oleg Alexandrov
Computer Vision
Diff3R: Feed-forward 3D Gaussian Splatting with Uncertainty-aware Differentiable Optimization
Yueh-Cheng Liu, Jozef Hladký, Matthias Nießner et al. (+1 more)
Recent advances in 3D Gaussian Splatting (3DGS) present two main directions: feed-forward models offer fast inference in sparse-view settings, while per-scene optimization yields high-quality renderings but is computationally expensive. To combine the benefits of both, we introduce Diff3R, a novel framework that explicitly bridges feed-forward prediction and test-time optimization. By incorporating a differentiable 3DGS optimization layer directly into the training loop, our network learns to predict an optimal initialization for test-time optimization rather than a conventional zero-shot result. To overcome the computational cost of backpropagating through the optimization steps, we propose computing gradients via the Implicit Function
Theorem and a scalable,
matrix-free PCG solver tailored for 3DGS optimization. Additionally, we incorporate a data-driven uncertainty model into the optimization process by adaptively controlling how much the parameters are allowed to change during optimization. This approach effectively mitigates overfitting in under-constrained regions and increases robustness against input outliers. Since our proposed optimization layer is model-agnostic, we show that it can be seamlessly integrated into existing feed-forward 3DGS architectures for both pose-given and pose-free methods, providing improvements for test-time optimization.
Read the full abstract ▾
All authors Yueh-Cheng Liu, Jozef Hladký, Matthias Nießner, Angela Dai
Computer Vision
Neerja Thakkar, Shiry Ginosar, Jacob Walker et al. (+3 more)
Visual intelligence requires anticipating the future behavior of
agents, yet
vision systems lack a general representation for motion and behavior. We propose dense point trajectories as visual
tokens for behavior, a structured mid-level representation that disentangles motion from appearance and generalizes across diverse non-rigid
agents, such as animals in-the-wild. Building on this abstraction, we design a
diffusion transformer that models unordered sets of trajectories and explicitly reasons about occlusion, enabling coherent forecasts of complex motion patterns. To evaluate at scale, we curate 300 hours of unconstrained animal
video with robust shot detection and camera-motion compensation. Experiments show that
forecasting trajectory
tokens achieves category-agnostic, data-efficient prediction, outperforms state-of-the-art baselines, and generalizes to rare species and morphologies, providing a foundation for predictive visual intelligence in the wild.
Read the full abstract ▾
All authors Neerja Thakkar, Shiry Ginosar, Jacob Walker, Jitendra Malik, Joao Carreira, Carl Doersch
Computer Vision · Cross-listed from cs.CR
AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration
Ruhao Liu, Weiqi Huang, Qi Li et al. (+1 more)
Membership Inference Attacks (MIAs) serve as a fundamental
auditing tool for evaluating training data leakage in machine learning models. However, existing methodologies predominantly rely on static, handcrafted heuristics that lack adaptability, often leading to suboptimal performance when transferred across different large models. In this work, we propose AutoMIA, an agentic framework that reformulates membership inference as an automated process of self-exploration and strategy
evolution. Given high-level scenario specifications, AutoMIA self-explores the attack space by generating executable logits-level strategies and progressively refining them through closed-loop evaluation feedback. By decoupling abstract strategy
reasoning from low-level execution, our framework enables a systematic, model-agnostic traversal of the attack search space. Extensive experiments demonstrate that AutoMIA consistently matches or outperforms state-of-the-art baselines while eliminating the need for manual feature engineering.
Read the full abstract ▾
All authors Ruhao Liu, Weiqi Huang, Qi Li, Xinchao Wang
Computer Vision
PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
Jingning Xu, Haochen Luo and Chen Liu
Vision-
language models (VLMs) are vulnerable to
adversarial image perturbations. Existing works based on
adversarial training against task-specific
adversarial examples are computationally expensive and often fail to generalize to unseen attack types. To address these limitations, we introduce Paraphrase-Decomposition-Aggregation (PDA), a training-free defense framework that leverages text augmentation to enhance VLM robustness under diverse
adversarial image attacks. PDA performs
prompt paraphrasing, question decomposition, and consistency aggregation entirely at test time, thus requiring no modification on the underlying models. To balance robustness and efficiency, we instantiate PDA as invariants that reduce the inference cost while retaining most of its robustness gains. Experiments on multiple VLM architectures and
benchmarks for visual question answering, classification, and captioning show that PDA achieves consistent robustness gains against various
adversarial perturbations while maintaining competitive clean accuracy, establishing a generic, strong and practical defense framework for VLMs during inference.
Read the full abstract ▾
Computer Vision
Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding
Yiheng Wang, Lichen Zhu, Yueqian Lin et al. (+4 more)
Multimodal Large
Language Models (MLLMs) have shown strong performance on
video question answering, but their application to long-form
videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework motivated by an information-theoretic perspective. We consider maximizing the conditional mutual information between the selected frames and the answer given the query as an idealized objective, and use a modular singleton-gain surrogate to motivate a tractable frame-wise selection criterion. Guided by this motivation, we train a query-conditioned evidence scoring network using annotated answer-supporting segments. A contrastive objective encourages greater aggregate score mass on frames from evidence regions than on background frames, providing a practical evidence-based criterion beyond semantic similarity. Experiments on long-form
video understanding
benchmarks show that our method consistently outperforms prior sampling strategies under comparable frame budgets. With Qwen2.5-VL-7B under the same 32-frame budget, our method improves over uniform sampling by 10.1 points on LVBench and 4.4 points on the long-
video subset of VideoMME.
Read the full abstract ▾
All authors Yiheng Wang, Lichen Zhu, Yueqian Lin, Yudong Liu, Jingyang Zhang, Hai "Helen" Li, Yiran Chen
Computer Vision
EgoSim: Egocentric World Simulator for Embodied Interaction Generation
Jinkun Hao, Mingda Jia, Ruiyan Wang et al. (+7 more)
We introduce EgoSim, a closed-loop
egocentric world simulator that generates spatially consistent interaction
videos and persistently updates the underlying 3D scene state for continuous simulation. Existing
egocentric simulators either lack explicit 3D grounding, causing structural drift under viewpoint changes, or treat the scene as static, failing to update world states across multi-stage interactions. EgoSim addresses both limitations by modeling 3D scenes as updatable world states. We generate embodiment interactions via a Geometry-action-aware Observation Simulation model, with spatial consistency from an Interaction-aware State Updating module. To overcome the critical data bottleneck posed by the difficulty in acquiring densely aligned scene-interaction training pairs, we design a scalable pipeline that extracts static point clouds, camera trajectories, and embodiment actions from in-the-wild large-scale
monocular egocentric videos. We further introduce EgoCap, a capture system that enables low-cost real-world data collection with uncalibrated smartphones. Extensive experiments demonstrate that EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization to complex scenes and in-the-wild dexterous interactions, while supporting cross-embodiment transfer to robotic manipulation. Codes and
datasets will be open soon. The project page is at egosimulator.github.io.
Read the full abstract ▾
All authors Jinkun Hao, Mingda Jia, Ruiyan Wang, Hongrui Zhu, Jiafei Cao, Xihui Liu, Ran Yi, Lizhuang Ma, Jiangmiao Pang, Xudong Xu
Computer Vision
Large Vision Model-Guided Masked Low-Rank Approximation for Ground-Roll Attenuation
Jiacheng Liao, Feng Qian, Ziyin Fan et al. (+1 more)
Ground roll is a common type of coherent noise in seismic records, and its attenuation remains challenging due to its substantial overlap with useful reflections in localized regions. Existing attenuation methods can be broadly classified into global and local categories according to whether ground-roll-contaminated regions are explicitly identified. Global methods, however, typically impose uniform attenuation on both contaminated and uncontaminated regions, which may result in signal leakage or distortion of reflections. By contrast, local methods restrict attenuation to contaminated regions and are therefore less prone to unnecessary modification of clean areas. However, their performance is often limited by manually designed or simplistic model-based mask estimation strategies. To address these limitations, we propose a large
vision model-guided masked low-rank approximation (LVM-LRA) framework for ground-roll attenuation. Within this framework, a promptable LVM is first employed to identify ground-roll-dominant regions in seismic records through
multimodal prompting and to generate accurate, fine-grained masks. The estimated masks are then incorporated into an LRA model for ground-roll attenuation. A global low-rank constraint is imposed on the reflection component to preserve event continuity, whereas a mask-guided local low-rank constraint is imposed on the ground-roll component so that its separation is confined to the masked regions. An iterative optimization
algorithm based on the alternating direction method of multipliers (ADMM) is further developed to solve the resulting model efficiently. Experiments on synthetic and field
datasets demonstrate that the proposed method achieves more effective ground-roll attenuation and better suppresses signal leakage than the baseline methods.
Read the full abstract ▾
All authors Jiacheng Liao, Feng Qian, Ziyin Fan, Yongjian Guo
Computer Vision
Maximizing T2-Only Prostate Cancer Localization from Expected Diffusion Weighted Imaging
Weixi Yi, Yipei Wang, Wen Yan et al. (+10 more)
Multiparametric MRI is increasingly recommended as a first-line noninvasive approach to detect and localize prostate cancer, requiring at minimum
diffusion-weighted (DWI) and T2-weighted (T2w) MR sequences. Early machine learning attempts using only T2w images have shown promising diagnostic performance in segmenting radiologist-annotated lesions. Such uni-modal T2-only approaches deliver substantial clinical benefits by reducing costs and expertise required to acquire other sequences. This work investigates an arguably more challenging application using only T2w at inference, but to localize individual cancers based on independent histopathology labels. We formulate DWI images as a latent modality (readily available during training) to classify cancer presence at local Barzell zones, given only T2w images as input. In the resulting expectation-maximization
algorithm, a latent modality generator (implemented using a flow matching-based generative model) approximates the latent DWI image posterior distribution in the E-steps, while in M-steps a cancer localizer is simultaneously optimized with the generative model to maximize the expected likelihood of cancer presence. The proposed approach provides a novel theoretical framework for learning from a privileged DWI modality, yielding superior cancer localization performance compared to approaches that lack training DWI images or existing frameworks for privileged learning and incomplete modalities. The proposed T2-only methods perform competitively or better than baseline methods using multiple input sequences (e.g., improving the patient-level F1 score by 14.4% and zone-level QWK by 5.3% over the T2w+DWI baseline). We present quantitative evaluations using internal and external
datasets from 4,133 prostate cancer patients with histopathology-verified labels.
Read the full abstract ▾
All authors Weixi Yi, Yipei Wang, Wen Yan, Hanyuan Zhang, Natasha Thorley, Alexander Ng, Shonit Punwani, Fernando Bianco, Mark Emberton, Veeru Kasivisvanathan, Dean C. Barratt, Shaheer U. Saeed, Yipeng Hu
Computer Vision
ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration
Bei Yan, Yuecong Min, Jie Zhang et al. (+2 more)
Large
Vision-
Language Models (LVLMs) frequently suffer from severe
hallucination issues. Existing mitigation strategies predominantly rely on isolated, single-step states to enhance visual focus or suppress strong linguistic priors. However, these static approaches neglect dynamic context changes across the generation process and struggles to correct inherited information loss. To address this limitation, we propose Adaptive Context inTegration (ACT), a training-free inference intervention method that mitigates
hallucination through the adaptive integration of contextual information. Specifically, we first propose visual context exploration, which leverages spatio-temporal profiling to adaptively amplify
attention heads responsible for visual exploration. To further facilitate
vision-
language alignment, we propose semantic context aggregation that marginalizes potential semantic queries to effectively aggregate visual evidence, thereby resolving the information loss caused by the discrete nature of
token prediction. Extensive experiments across diverse LVLMs demonstrate that ACT significantly reduces
hallucinations and achieves competitive results on both discriminative and generative
benchmarks, acting as a robust and highly adaptable solution without compromising fundamental generation capabilities.
Read the full abstract ▾
All authors Bei Yan, Yuecong Min, Jie Zhang, Shiguang Shan, Xilin Chen
Computer Vision
DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Driving
Yiyao Zhu, Ying Xue, Haiming Zhang et al. (+8 more)
Vision-based autonomous driving has gained much
attention due to its low costs and excellent performance. Compared with dense BEV (Bird's Eye View) or sparse query models, Gaussian-centric method is a comprehensive yet sparse representation by describing scene with 3D semantic Gaussians. In this paper, we introduce DLWM, a novel paradigm with Dual Latent World Models specifically designed to enable holistic gaussian-centric pre-training in autonomous driving using two stages. In the first stage, DLWM predicts 3D Gaussians from queries by self-supervised reconstructing multi-view semantic and depth images. Equipped with fine-grained contextual features, in the second stage, two latent world models are trained separately for temporal feature learning, including Gaussian-flow-guided latent prediction for downstream occupancy perception and
forecasting tasks, and ego-planning-guided latent prediction for motion planning. Extensive experiments in SurroundOcc and nuScenes
benchmarks demonstrate that DLWM shows significant performance gains across Gaussian-centric 3D occupancy perception, 4D occupancy
forecasting and motion planning tasks.
Read the full abstract ▾
All authors Yiyao Zhu, Ying Xue, Haiming Zhang, Guangfeng Jiang, Wending Zhou, Xu Yan, Jiantao Gao, Yingjie Cai, Bingbing Liu, Zhen Li, Shaojie Shen
Computer Vision
Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization
Hao Fang, Wenbo Yu, Bin Chen et al. (+4 more)
Federated Learning (FL) has emerged as a compelling paradigm for
privacy-preserving distributed machine learning, allowing multiple clients to collaboratively train a global model by transmitting locally computed gradients to a central server without exposing their private data. Nonetheless, recent studies find that the gradients exchanged in the FL system are also vulnerable to
privacy leakage, e.g., an attacker can invert shared gradients to reconstruct sensitive data by leveraging pre-trained generative
adversarial networks (GAN) as prior knowledge. However, existing attacks simply perform gradient inversion in the latent space of the GAN model, which limits their expression ability and generalizability. To tackle these challenges, we propose Gradient Inversion over Feature Domains (GIFD), which disassembles the GAN model and searches the hierarchical features of the intermediate layers. Instead of optimizing only over the initial latent code, we progressively change the optimized layer, from the initial latent space to intermediate layers closer to the output images. In addition, we design a regularizer to avoid unreal image generation by adding a small ${l_1}$ ball constraint to the searching range. We also extend GIFD to the out-of-distribution (OOD) setting, which weakens the assumption that the training sets of GANs and FL tasks obey the same data distribution. Furthermore, we consider the challenging OOD scenario of label inconsistency and propose a label mapping technique as an effective solution. Extensive experiments demonstrate that our method can achieve pixel-level reconstruction and outperform competitive baselines across a variety of FL scenarios.
Read the full abstract ▾
All authors Hao Fang, Wenbo Yu, Bin Chen, Xuan Wang, Shu-Tao Xia, Qing Liao, Ke Xu
Computer Vision
YieldSAT: A Multimodal Benchmark Dataset for High-Resolution Crop Yield Prediction
Miro Miranda, Deepak Pathak, Patrick Helber et al. (+10 more)
Crop yield prediction requires substantial data to train scalable models. However, creating yield prediction
datasets is constrained by high acquisition costs, heterogeneous data quality, and data
privacy regulations. Consequently, existing
datasets are scarce, low in quality, or limited to regional levels or single crop types, hindering the development of scalable data-driven solutions. In this work, we release YieldSAT, a large, high-quality, and
multimodal dataset for high-resolution crop yield prediction. YieldSAT spans various
climate zones across multiple countries, including Argentina, Brazil, Uruguay, and Germany, and includes major crop types, including corn, rapeseed, soybeans, and wheat, across 2,173 expert-curated fields. In total, over 12.2 million yield samples are available, each with a spatial resolution of 10 m. Each field is paired with multispectral satellite imagery, resulting in 113,555 labeled satellite images, complemented by auxiliary environmental data. We demonstrate the potential of large-scale and high-resolution crop yield prediction as a pixel regression task by comparing various deep learning models and data fusion architectures. Furthermore, we highlight open challenges arising from severe distribution shifts in the ground truth data under real-world conditions. To mitigate this, we explore a domain-informed Deep Ensemble approach that exhibits significant performance gains. The
dataset is available at https://yieldsat.github.io/.
Read the full abstract ▾
All authors Miro Miranda, Deepak Pathak, Patrick Helber, Benjamin Bischke, Hiba Najjar, Francisco Mena, Cristhian Sanchez, Akshay Pai, Diego Arenas, Matias Valdenegro-Toro, Marcela Charfuelan, Marlon Nuske, Andreas Dengel
Computer Vision
EmoScene: A Dual-space Dataset for Controllable Affective Image Generation
Li He, Longtai Zhang, Wenqiang Zhang et al. (+2 more)
Text-to-image
diffusion models achieve high visual fidelity, yet fine-grained affective control remains difficult because textual
emotion cues often fail to specify the visual perceptual factors underlying affective expression. Existing visual-affect
datasets are likewise often limited to discrete labels, specific domains, or limited supervision of perceptual attributes. We introduce EmoScene, a large-scale dual-space
dataset for controllable affective image generation, containing 1.2M images across more than 300 scene categories. Its affective space jointly represents discrete
emotions and continuous valence–arousal–dominance (VAD), its perceptual space records measurable appearance attributes, and contextual descriptions ground both in scene semantics. EmoScene combines multi-model annotation with human-in-the-loop quality control. A random audit of 30,519 images yields 91.22% agreement on discrete
emotion labels, while an independent multi-rater evaluation yields Fleiss' $\kappa=0.85$. After controlling for source and scene composition, affective dimensions and perceptual attributes exhibit stable associations across data sources, reflecting statistical tendencies rather than deterministic visual rules. To demonstrate the
dataset's utility, we further develop AffectCtrl, which learns residuals in the conditioning space of frozen
diffusion models to support categorical
emotion generation and continuous control over VAD, brightness, and saturation. AffectCtrl achieves 85.75% categorical
emotion accuracy, outperforms EmotiCrafter in valence and arousal control under a shared evaluation protocol, and obtains Pearson correlations of 0.673–0.765 across all five continuous axes. These results demonstrate that EmoScene provides a scalable data foundation for analyzing and controlling affective expression in visual generation.
Read the full abstract ▾
All authors Li He, Longtai Zhang, Wenqiang Zhang, Yan Wang, Lizhe Qi
Computer Vision
Autoregressive Appearance Prediction for 3D Gaussian Avatars
Michael Steiner, Zhang Chen, Alexander Richard et al. (+3 more)
A photorealistic and immersive human avatar experience demands capturing fine, person-specific details such as cloth and hair dynamics, subtle facial expressions, and characteristic motion patterns. Achieving this requires large, high-quality
datasets, which often introduce ambiguities and spurious correlations when very similar poses correspond to different appearances. Models that fit these details during training can overfit and produce unstable, abrupt appearance changes for novel poses. We propose a 3D Gaussian Splatting avatar model with a spatial MLP backbone that is conditioned on both pose and an appearance latent. The latent is learned during training by an encoder, yielding a compact representation that improves reconstruction quality and helps disambiguate pose-driven renderings. At driving time, our predictor autoregressively infers the latent, producing temporally smooth appearance
evolution and improved stability. Overall, our method delivers a robust and practical path to high-fidelity, stable avatar driving.
Read the full abstract ▾
All authors Michael Steiner, Zhang Chen, Alexander Richard, Vasu Agrawal, Markus Steinberger, Michael Zollhöfer
Computer Vision
Learning Quantised Structure-Preserving Motion Representations for Dance Fingerprinting
Arina Kharlamova, Bowei He, Chen Ma et al. (+1 more)
We present DANCEMATCH, an end-to-end framework for motion-based dance
retrieval, the task of identifying semantically similar choreographies directly from raw
video, defined as DANCE FINGERPRINTING. While existing motion analysis and
retrieval methods can compare pose sequences, they rely on continuous
embeddings that are difficult to index, interpret, or scale. In contrast, DANCEMATCH constructs compact, discrete motion signatures that capture the spatio-temporal structure of dance while enabling efficient large-scale
retrieval. Our system integrates Skeleton Motion Quantisation (SMQ) with Spatio-Temporal
Transformers (STT) to encode human poses, extracted via Apple CoMotion, into a structured motion vocabulary. We further design DANCE
RETRIEVAL ENGINE (DRE), which performs sub-linear
retrieval using a histogram-based index followed by re-ranking for refined matching. To facilitate reproducible research, we release DANCETYPESBENCHMARK, a pose-aligned
dataset annotated with quantised motion
tokens. Experiments demonstrate robust
retrieval across diverse dance styles and strong generalisation to unseen choreographies, establishing a foundation for scalable motion fingerprinting and quantitative choreographic analysis.
Read the full abstract ▾
All authors Arina Kharlamova, Bowei He, Chen Ma, Xue Liu
Computer Vision
Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis
Dylan B. Lewis, Jens Gregor and Hector Santos-Villalobos
Modern
vision pipelines increasingly rely on pretrained image encoders whose representations are reused across tasks and models, yet these representations are often overcomplete and model-specific. We propose a simple, training-free method to improve the efficiency of image representations via a post-hoc canonical correlation analysis (CCA) operator. By leveraging the shared structure between representations produced by two pre-trained image encoders, our method finds linear projections that serve as a principled form of representation selection and dimensionality reduction, retaining shared semantic content while discarding redundant dimensions. Unlike standard dimensionality reduction techniques such as PCA, which operate on a single
embedding space, our approach leverages cross-model agreement to guide representation
distillation and refinement. The technique allows representations to be reduced by more than 75% in dimensionality with improved downstream performance, or enhanced at fixed dimensionality via post-hoc representation transfer from larger or fine-tuned models. Empirical results on ImageNet-1k, CIFAR-100, MNIST, and additional
benchmarks show consistent improvements over both baseline and PCA-projected representations, with accuracy gains of up to 12.6%.
Read the full abstract ▾
Computer Vision
Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment
Zhuchenyang Liu, Yao Zhang and Yu Xiao
2D assembly diagrams are often abstract and hard to follow, creating a need for intelligent assistants that can monitor progress, detect errors, and provide step-by-step guidance. In mixed reality settings, such systems must recognize completed and ongoing steps from the camera feed and align them with the diagram instructions.
Vision Language Models (VLMs) show promise for this task, but face a depiction gap because assembly diagrams and
video frames share few visual features. To systematically assess this gap, we construct IKEA-Bench, a
benchmark of 1,623 questions across 6 task types on 29 IKEA furniture products, and evaluate 19 VLMs (2B-38B) under three
alignment strategies. Our key findings: (1) assembly instruction understanding is recoverable via text, but text simultaneously degrades diagram-to-
video alignment; (2) architecture family predicts
alignment accuracy more strongly than parameter count; (3)
video understanding remains a hard bottleneck unaffected by strategy. A three-level mechanistic analysis further reveals that diagrams and
video occupy disjoint ViT subspaces, and that adding text shifts models from visual to text-driven
reasoning. These results identify visual encoding as the primary target for improving cross-depiction robustness. Project page: https://ryenhails.github.io/IKEA-Bench/
Read the full abstract ▾
Computer Vision
ProCap: Projection-Aware Captioning for Spatial Augmented Reality
Zimo Cao, Yuchen Deng, Haibin Ling et al. (+1 more)
Spatial augmented reality (SAR) directly projects digital content onto physical scenes using projectors, creating immersive experience without head-mounted displays. However, for SAR to support intelligent interaction, such as
reasoning about the scene or answering user queries, it must semantically distinguish between the physical scene and the projected content. Standard
Vision Language Models (VLMs) struggle with this virtual-physical ambiguity, often confusing the two contexts. To address this issue, we introduce ProCap, a novel framework that explicitly decouples projected content from physical scenes. ProCap employs a two-stage pipeline: first it visually isolates virtual and physical layers via automated
segmentation; then it uses region-aware
retrieval to avoid ambiguous semantic context due to projection distortion. To support this, we present RGBP (RGB + Projections), the first large-scale SAR semantic
benchmark dataset, featuring 65 diverse physical scenes and over 180,000 projections with dense, decoupled annotations. Finally, we establish a dual-captioning evaluation protocol using task-specific
tokens to assess physical scene and projection descriptions independently. Our experiments show that ProCap provides a robust semantic foundation for future SAR research. The source code, pre-trained models and the RGBP
dataset are available on the project page: https://ZimoCao.github.io/ProCap/.
Read the full abstract ▾
All authors Zimo Cao, Yuchen Deng, Haibin Ling, Bingyao Huang
Computer Vision
JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
Issa Sugiura, Koki Maeda, Shuhei Kurita et al. (+3 more)
Reliable evaluation is essential for the development of
vision-
language models (VLMs). However, Japanese VQA
benchmarks have undergone far less iterative refinement than their English counterparts. As a result, many existing
benchmarks contain issues such as ambiguous questions, incorrect answers, and instances that can be solved without visual grounding, undermining evaluation reliability and leading to misleading conclusions in model comparisons. To address these limitations, we introduce JAMMEval, a refined collection of Japanese
benchmarks for reliable VLM evaluation. It is constructed by systematically refining seven existing Japanese
benchmark datasets through two rounds of human annotation, improving both data quality and evaluation reliability. In our experiments, we evaluate open-weight and proprietary VLMs on JAMMEval and analyze the capabilities of recent models on Japanese VQA. We further demonstrate the effectiveness of our refinement by showing that the resulting
benchmarks yield evaluation scores that better reflect model capability, exhibit lower run-to-run variance, and improve the ability to distinguish between models of different capability levels. We release our
dataset and code to advance reliable evaluation of VLMs.
Read the full abstract ▾
All authors Issa Sugiura, Koki Maeda, Shuhei Kurita, Yusuke Oda, Daisuke Kawahara, Naoaki Okazaki