Computer Vision
Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation
Changwang Mei, Peisong Wang, Zekun Li et al. (+7 more)
Visual Autoregressive (VAR) models deliver high-quality image generation but suffer from significant inference latency at high resolutions. Recent acceleration approaches most rely on heuristic measures with layer features to prune
tokens. Such heuristics are sensitive to complex contextual semantics, leading to inaccurate identification of redundant computation and poor adaptability across
prompts. We rethink redundancy in VAR from the perspective of its impact on pixel-space generation and introduce Latent Discrepancy. This unified metric quantifies a
token's contribution by measuring the change in model states during generation. Our analysis shows that redundancy is more accurately identified when guided by image latent or pixel-space signals. We further observed that in classifier-free guidance (CFG), the convergence trend of the discrepancy between conditional and unconditional branches exhibits high dynamics with different
prompts. Based on these findings, we propose LD-
Pruning (Latent Discrepancy
Pruning), a training-free framework that removes redundancy via latent discrepancy by integrating decoding-free region selection and adaptive unconditional-branch skipping. Extensive experiments show that LD-
Pruning substantially reduces inference latency while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
Read the full abstract ▾
All authors Changwang Mei, Peisong Wang, Zekun Li, Changsheng Li, Shuang Qiu, Qinghao Hu, Gang Li, Yifan Zhang, Zhihui Wei, Jian Cheng
Computer Vision
Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion
Jiayi Wu, Haoming Cai, Cornelia Fermuller et al. (+2 more)
Video Diffusion Models (VDMs) synthesize high-quality
videos, but precise camera and motion control remains challenging when large viewpoint shifts, changing occlusions, or complex dynamics reveal unseen regions.
Diffusion priors alone can yield structural inconsistencies, while warped guidance can contain incomplete geometry and misleading view-dependent appearance cues. Real2SAM2Real complements VDM priors with a coarse 3D foreground cache of topologically complete meshes for intuitive editing and animation. Generative 3D lifting estimates these object and human meshes, including unobserved surfaces, from a single reference image. Normal maps rendered under specified camera trajectories and foreground motions guide
video synthesis in the VDM, while the reference image provides appearance conditioning. Soft spatial-aligned injection and minimally invasive fine-tuning incorporate this guidance while largely preserving pretrained synthesis priors. Decoupled conditioning enables independent appearance and geometry editing. We train on around 2,000
monocular clips using instance-masked normal sequences with instance-level spatial perturbations to reduce sensitivity to differences between
video-estimated training normals and normals rendered from coarse 3D caches at inference. Training requires neither explicit 3D assets nor paired multi-trajectory recordings. Real2SAM2Real supports decoupled camera and multi-instance motion control, including articulated humans. Across these challenging settings, experiments show strong spatiotemporal consistency and better overall control accuracy and
video fidelity than evaluated state-of-the-art methods. Preliminary demonstrations suggest potential robotics applications.
Read the full abstract ▾
All authors Jiayi Wu, Haoming Cai, Cornelia Fermuller, Christopher Metzler, Yiannis Aloimonos
Computer Vision
Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models
Zijie Zhou, Dandan Zhu, Hangxiangpan Wang et al. (+3 more)
Large
Vision-
Language Models (LVLMs) have demonstrated impressive performance on
multimodal tasks through scaled architectures and extensive training. Recent studies introduce Mixture of Experts (MoE) into LVLMs for improved computational efficiency. However, existing MoE approaches treat visual and linguistic modalities with symmetric architectures, overlooking the inherent asymmetry in how these two modalities are processed. This asymmetry causes two critical issues. First, text and
vision form hierarchical rather than parallel relationships, as text queries typically describe partial aspects of complete visual scenes. Euclidean expert space struggles to encode such containment structures. Second,
language experts in deeper layers progressively shift from evidence-based processing to parametric
memory dependence, losing grounding in the provided visual and linguistic information. To address these issues, we propose AsyMoE, a novel architecture that explicitly models this asymmetry through three specialized expert groups. Intra-modality experts handle modality-specific processing.
Hyperbolic inter-modality experts capture hierarchical cross-modal relationships through negative
curvature geometry. Evidence-priority
language experts suppress parametric
memory activation and maintain contextual grounding throughout network depth. Extensive experiments demonstrate that AsyMoE achieves consistent improvements over baseline methods, with average gains of 1.5% over MoE variants and up to 3.8% on
hallucination-sensitive tasks. AsyMoE activates 25.45% fewer parameters compared to dense models.
Read the full abstract ▾
All authors Zijie Zhou, Dandan Zhu, Hangxiangpan Wang, Heng Zhang, Huishen Jiao, Yi Zhao
Computer Vision
StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement
Junwon Seo, Sushant Veer, Ran Tian et al. (+6 more)
Video world models (WMs) have shown promise for policy evaluation and improvement by imagining realistic future observations conditioned on ego-
robot actions. While WMs can model distributions over futures, policy evaluation and improvement typically rely on nominal imaginations, which can miss high-impact outcomes of
robot actions unless prohibitively many samples are drawn. To enable robust policy evaluation and improvement over WM imaginations, we propose StressDream, which steers imaginations toward high-impact yet plausible outcomes specified at inference time by optimizing the initial noise of
diffusion-based WMs. However, optimizing high-dimensional noise is challenging: the optimization must reason about nuanced, scene-dependent target events in generated
videos while avoiding out-of-distribution (OOD) noise that yields implausible imaginations. We address this with two complementary objectives: a semantic objective with a
Vision-
Language Model that provides informative gradients by
reasoning about the generated
video, and a plausibility objective that prevents the optimized noise from drifting OOD. With state-of-the-art
video world models for autonomous driving and robotic manipulation, we show that StressDream effectively steers imaginations toward high-impact yet plausible outcomes specified by text at inference time, such as task failures, enabling robust policy evaluation and improvement by identifying actions whose plausible futures include undesirable outcomes.
Video results are available at https://stressdream.github.io/.
Read the full abstract ▾
All authors Junwon Seo, Sushant Veer, Ran Tian, Wenhao Ding, Apoorva Sharma, Karen Leung, Edward Schmerling, Marco Pavone, Andrea Bajcsy
Computer Vision
The Harsh Truth: Segment-Level Analysis of Harsh Driving Events in Milan Using Large-Scale Telematics, Street Networks, and Google Street View
Andrea La Grotteria, Paolo Santi, Titus Venverloo et al. (+2 more)
Police-reported crash statistics remain the standard input for urban road-safety assessment, but their incompleteness and reporting lag limit their usefulness for timely, fine-grained intervention design. Harsh acceleration and braking events are widely used as surrogate safety indicators, but have so far been studied only in comparatively small urban samples. This study analyses harsh events across the urban road network of Milan, combining high-resolution telematics from more than 4.2 million vehicles equipped with On-Board Units, segment-level traffic metrics from TomTom, street-network and infrastructure attributes from OpenStreetMap, and visual streetscape features extracted from Google Street View via semantic
segmentation using a OneFormer model. We employ an analytical framework combining non-parametric Mann–Whitney U tests of segment-feature distributions between high- and low-harshness groups with supervised machine-learning regressors. We find that, once exposure is controlled for, wider carriageways, crossings and transit stops, and more open visual fields (higher sky- and road-pixel proportions) are associated with higher harsh-event intensity, while denser built frontage is associated with lower intensity. Finally, the cycling-infrastructure case study identifies a gradient in harsh-event intensity across facility types: markings-only cycle lanes are associated with a 19.5% higher harshness score, and mixed-traffic configurations with an 11.5% higher score, relative to physically separated cycle paths, conditional on the included controls. These results support context-specific rather than uniform urban-safety interventions and illustrate how large-scale telematics combined with open geospatial and visual data can inform
Vision Zero decision-making at the metropolitan scale.
Read the full abstract ▾
All authors Andrea La Grotteria, Paolo Santi, Titus Venverloo, Umberto Fugiglando, Carlo Ratti
Computer Vision
LastAct: Trajectory-Guided Latest-Activity Localization for Real-Time Smart-Home Activity Recognition
Zishuai Liu, Ruili Fang, Jin Lu et al. (+1 more)
Human Activity Recognition (HAR) from ambient sensors enables smart-home applications such as health monitoring and assisted living. In realistic deployments, however, sensor events arrive as a continuous stream and activity boundaries are unknown. Sliding-window inference therefore produces many windows that straddle transitions and contain mixed activities, creating boundary contamination that violates the pre-segmented instance assumption used by most
benchmarks and models. Moreover, many pipelines under-use spatial context by treating sensor IDs as independent
tokens. We present LastAct, a trajectory-centric framework for streaming smart-home HAR that targets the most recent activity under mixed windows while explicitly modeling spatial structure. LastAct projects sensor events onto the home floorplan to form a layout-aligned trajectory image sequence that preserves spatial continuity. A lightweight gate identifies contaminated windows, and a boundary localizer estimates the last transition to enable boundary-guided masking that emphasizes post-boundary evidence and suppresses stale context. For efficiency, we reuse a precomputed layout-aligned template cache to avoid repeated rendering. Empirically, across four public smart-home
datasets under near-realistic mixed-activity protocols, LastAct achieves competitive or superior performance on pure windows and yields substantial Macro-F1 gains on cross/mixed windows, demonstrating improved robustness under near-realistic sliding-window regimes.
Read the full abstract ▾
All authors Zishuai Liu, Ruili Fang, Jin Lu, Fei Dou
Computer Vision
APE: Agentic Prompt Enhancer for Image Generation and Editing
Zijian Huang, Jay Zhangjie Wu, Zian Wang et al. (+5 more)
Natural
language has become a powerful interface for image generation and editing, yet text-guided visual systems remain highly sensitive to
prompt formulation. Semantically similar requests can produce different outputs depending on wording, specificity, and how explicitly visual constraints are stated, motivating
prompt enhancement as a trainable component rather than a peripheral user choice. Existing strong enhancers often rely on large, proprietary LLMs such as ChatGPT or Gemini, adding cost, latency, and deployment dependence to the visual generation pipeline. We propose Agentic
Prompt Enhancer (APE), a lightweight framework that post-trains small
language models (SLMs) as
prompt-enhancement
agents. APE supports both single-
agent rewriting and role-specialized multi-
agent enhancement. Its single-
agent instantiation, SAPE, rewrites the
prompt in one pass, while its multi-
agent instantiation, MAPE, decomposes enhancement into a router–rewriter–composer process for handling compositional constraints over objects, attributes, spatial relations, and edits. With task-aware rewards and post-training protocols, APE improves visual
alignment and
prompt following without modifying the downstream visual model. Experiments on challenging image generation and editing
benchmarks demonstrate that post-trained small
prompt enhancers reliably outperform their base counterparts, narrowing the gap to closed-source
prompt enhancers; in addition, MAPE proves particularly strong on complex compositional tasks within these
benchmarks.
Read the full abstract ▾
All authors Zijian Huang, Jay Zhangjie Wu, Zian Wang, Tianshi Cao, Jiasi Chen, Sanja Fidler, Huan Ling, Xuanchi Ren
Computer Vision · Cross-listed from cs.RO
Safe2Drive: Evaluating Safe Driving Behaviors of E2E Autonomous Driving Models
Nishad Sahu, Kalpana Panda, Congyuan Yu et al. (+3 more)
Recent end-to-end (E2E) autonomous driving policies achieve high driving scores in closed-loop simulations. Yet it remains unclear whether these policies handle common safety-critical scenarios. We present Safe2Drive (S2D), a set of Bench2Drive-aligned scenario extensions focused on three frequent families of road hazards: work zones, pedestrian jaywalking, and occluded vulnerable road users (VRUs). Safe2Drive adds 100 common but challenging scenarios and introduces SafeDriving Score (SDS), a safety-centric metric that augments prior evaluators with pre-crash braking, work zone-object contact, lane centering, and smoothness checks. Evaluating two state-of-the-art policies (LEAD and SimLingo) on S2D, we find that their driving scores drop sharply relative to their reported Bench2Drive baselines (LEAD: from 94.70 DS on Bench2Drive to 39.95 DS on S2D; SimLingo: from 85.07 DS on Bench2Drive to 41.00 DS on S2D) and that SDS on S2D is low (11.85 for LEAD and 15.27 for Sim-Lingo). These results are consistent with brittle safe-driving behaviors such as poor work-zone understanding, red-light violations, and late or absent braking for pedestrians. This study highlights a lack of safe behavioral
reasoning in E2E models even when tested on CARLA towns that are part of the training set. We plan to release the code and
videos for all 100 S2D scenarios.
Read the full abstract ▾
All authors Nishad Sahu, Kalpana Panda, Congyuan Yu, Changzhong Qian, Shounak Sural, Ragunathan Rajkumar
Computer Vision · Cross-listed from cs.GR
PaintBench: Deterministic Evaluation of Precise Visual Editing
Kai Xu, Ellis Brown, Shrikar Madhu et al. (+3 more)
While current
multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable
benchmark targeting 20 fundamental precise visual editing operations across four categories:
geometric transformation, structural manipulation, color change, and symbolic
reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (
geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained
benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores ($R^2 = 0.91$, $p < 0.001$). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise
multimodal visual editing.
Read the full abstract ▾
All authors Kai Xu, Ellis Brown, Shrikar Madhu, Rob Fergus, He He, Saining Xie
Computer Vision
MyoSem: Aligning Electromyography with Natural-Language Action Semantics
Chiyue Wang, Yang Gao and Zhanpeng Jin
Electromyography (EMG) records muscle activation for hand gesture recognition, prosthetic control, and wearable interaction. Existing EMG methods usually formulate hand action understanding as prediction over fixed gesture labels, leaving EMG signals and action-semantic text unaligned in a common feature space and limiting bidirectional querying between signals and hand-action descriptions. We present MyoSem, an EMG–action semantic
alignment framework that maps hand EMG and multi-view natural-
language action descriptions into a shared feature space for
retrieval. MyoSem builds semantic prototypes from multi-view descriptions, encodes EMG with local activation-aware representations, and uses semantic queries to aggregate local EMG evidence for bidirectional EMG–text
retrieval. We evaluate MyoSem under seven protocols on NinaPro DB2, EMG2Pose, and NinaPro DB2$\rightarrow$DB3, covering random splits, user shifts, held-out action classes, and able-bodied-to-amputee transfer. Across this evaluation, MyoSem achieves the strongest overall bidirectional
retrieval performance among the evaluated methods. Ablations and representation analyses show how semantic prototypes, activation-aware encoding, and semantic queries shape the learned space. Overall, MyoSem moves EMG hand action understanding from fixed-label recognition toward
language-queryable semantic
retrieval.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.HC
UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment
Zheng Wang, Shuo Wang and Junhong Wang
In recent years,
emotion recognition based on physiological signals such as electroencephalogram (
EEG) has gained considerable
attention, as internal physiological data offer greater objectivity and reliability compared to external behavioral data like facial expressions. However, due to distribution shifts caused by individual and contextual differences, along with variations in sample quality across modalities, constructing a cross-domain
multimodal emotion recognition model with high generalization and robustness remains a key challenge. In this study, we propose a Unified Framework with Adaptive
Multimodal Alignment (UF-AMA) to address cross-subject and cross-session
emotion recognition using
multimodal physiological signals. First, we construct a cross-modal feature fusion network comprising
Transformer encoders and multi-head cross-
attention modules, enabling the deep integration of
EEG signals and eye-tracking data. Subsequently, we introduce a confidence-aware screening mechanism that dynamically assesses the predictive reliability of each modality branch on target domain samples, partitions samples into different quality subsets, and accordingly applies global consistency
alignment and cross-modal
distillation. Finally, we propose a multi-level domain adaptation framework that jointly optimizes the marginal and conditional distributions of both local modality-specific and global fusion features, thereby reducing cross-domain distribution shifts at multiple granularities. Extensive experiments on the SEED and SEED-IV
datasets demonstrate that UF-AMA achieves state-of-the-art (SOTA) performance in both cross-subject and cross-session tasks. The source code is available at: https://github.com/BetterCoderLab/UF-AMA.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Modeling Robotics Dataset Construction as an Artifact-Based Build Process
Leon Pohl, Lukas Beer, George Sebastian et al. (+1 more)
Robotic systems generate large volumes of
multimodal sensor data, but converting ROS bag recordings into machine learning
datasets is often handled by ad hoc sequential scripts, creating engineering overhead and slow iteration cycles. We model
dataset construction as an artifact-based build process over a dependency graph and implement this approach in Bagzel, an open-source Bazel extension for reproducible, incremental
dataset generation (including nuScenes-format export). We compare Bagzel and Bagzel-xattr (server-side digest management) against a sequential rosbag2nuscenes baseline. Bagzel reduces runtime in all evaluated execution modes, with the largest gains in iterative workflows (up to 386.26x in warm builds and 7.21x in incremental builds on a 20.4 GB
dataset). Across
dataset sizes from 5.1 to 20.4 GB, Bagzel variants show markedly better scaling behavior than the baseline, especially in warm and incremental modes. Bagzel-xattr provides additional gains, with a mean runtime reduction of 5.9% compared to Bagzel in the input granularity study. Overall, modeling robotics
dataset construction as an artifact-based build process substantially reduces
dataset update latency while maintaining a deterministic build design that supports reproducibility.
Read the full abstract ▾
All authors Leon Pohl, Lukas Beer, George Sebastian, Mirko Maehlisch
Computer Vision
Digital-to-Physical Transfer of Adversarial Patches for Aerial Vehicle Detection
Jung Heum Woo and Eun-Kyu Lee
Deep neural network (DNN)-based object detectors are widely used for analyzing aerial and satellite imagery in applications such as environmental monitoring and urban analytics. Despite their strong performance, these models are known to be vulnerable to
adversarial examples, and physical
adversarial attacks using printable patterns pose realistic security threats. In this paper, we evaluate physical
adversarial patch attacks against an aerial vehicle detector by bridging digital optimization and real-world deployment.
Adversarial patches are optimized in the digital domain using a loss function that minimizes the maximum objectness score while incorporating non-printability score (NPS) and total variation (TV) constraints to ensure both printability and spatial smoothness. The optimized patches are printed and deployed in three configurations: ON, OFF, and OFF-Side. Experiments using a YOLOv3 detector show that while the OFF patch achieves the highest effectiveness in the digital domain (85.51% Average Objectness Reduction Rate (AORR)), the ON patch demonstrates superior robustness in physical environments (0.197-0.343 Objectness Score Ratio (OSR)) due to its consistent visibility. Furthermore, our results indicate that weather-based augmentation does not necessarily improve patch optimization in this domain. These findings provide critical insights into the practical vulnerabilities of aerial object detection systems.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Training-Free Continuous Bitrate Control for Scalable Image Coding for Humans and Machines
Yui Tatsumi and Hiroshi Watanabe
Continuous variable-rate compression is highly demanded in real-world applications, but remains underexplored in scalable image coding for humans and machines. In this paper, we propose a training-free variable-rate scalable image coding framework. By adaptively adjusting quantization step sizes based on predicted scale values, the proposed method enables independent and continuous bitrate control for the machine and enhancement layers while preserving important latent information in each layer. Experimental results demonstrate the effectiveness of the proposed method and highlight the importance of bitrate allocation between the two layers.
Read the full abstract ▾
Computer Vision
DiffCrossGait: Trajectory-Level Alignment for 2D-3D Cross-Modal Gait Recognition via Latent Diffusion
Zhiyang Lu and Ming Cheng
Cross-modal 2D-3D gait recognition is impeded by inherent domain discrepancies between 2D silhouette and 3D
LiDAR range-view representations. While prior methods align only final
embeddings, we propose DiffCrossGait, which reformulates cross-modal matching as trajectory-level
alignment in an identity-relevant latent
diffusion space, rather than assuming full equivalence between 2D and 3D observations. By driving both modalities with shared Gaussian noise within a latent space, we enable continuous
alignment throughout the generative
evolution. We introduce a Tri-Phase
Alignment Strategy that exploits varying noise intensities to enforce identity anchoring, dynamics consistency, and cross-modal structural recoverability, thereby constraining both modalities to share denoising dynamics and bottleneck structure, which promotes modality-invariant gait features. Crucially, our framework decouples generative
alignment from the discriminative backbone; the
diffusion mechanism serves exclusively as a training objective, ensuring high inference efficiency by eliminating the computational overhead of iterative denoising. Extensive experiments on the SUSTech1K and FreeGait
benchmarks demonstrate that DiffCrossGait achieves state-of-the-art performance.
Read the full abstract ▾
Computer Vision
StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning
Xixiang He, Baiqi Wu, Xingming Li et al. (+4 more)
Multimodal large
language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual
reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR
benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We introduce StemBind, a shared-stem diagnostic
benchmark that probes the same visual stem with three aligned questions: Perception (what is in the image), Rule (what pattern governs it), and Full (which option completes it), so a final-answer error can be attributed to a specific sub-step on the same evidence. StemBind contains 2,298 curated knowledge-light stems across nine auditable visual operations, totaling 19,533 P/R/F tasks, with each full item annotated by Sternberg's four
reasoning stages (S1 Encode, S2 Infer, S3 Map, S4 Apply). Evaluating 24 frontier MLLM configurations yields four findings. (i) The R-F chasm: rule accuracy exceeds full-item accuracy on 22 of 24 models, so most failures happen after the rule is identified. (ii) A persistent binding gap: even when P and R are both correct on the same stem, models still answer F incorrectly 51.2% of the time. (iii) The bottleneck is S3: process diagnostics and Stage-wise Stimulus Augmentation localize the dominant failure to rule-to-instance mapping. (iv) Scaling and thinking do not help: neither larger models nor explicit thinking mode reliably closes the gap, and thinking even lowers rule and full-item accuracy. StemBind reframes AVR evaluation from final-answer ranking to locating where abstract visual
reasoning breaks down, identifying rule-to-instance binding as a concrete next target for
vision-grounded
reasoning.
Read the full abstract ▾
All authors Xixiang He, Baiqi Wu, Xingming Li, Ao Cheng, Qiyao Sun, Xuanyu Ji, Qingyong Hu
Computer Vision · Cross-listed from eess.IV
Multi-Contrast MRI Motion Correction via Parameter-Informed Disentanglement and Adaptive Experts
Honglin Xiong, Yuxian Tang, Feng Li et al. (+4 more)
Motion artifacts in
magnetic resonance imaging (MRI) degrade diagnostic reliability. Existing deep learning methods are typically contrast-specific and fail to generalize across diverse modalities and artifact severities. We propose a unified framework combining parameter-informed contrast disentanglement with severity-aware adaptive correction. ScanCLIP, pretrained on over 30,000 MRI text-image pairs, derives contrast
embeddings from acquisition parameters to disentangle contrast style from anatomical content, yielding contrast-free features. A
Vision Transformer then estimates motion severity and routes features through a Mixture-of-Experts network, enabling targeted artifact correction. A dual-pathway decoder reconstructs both the clean image and residual artifact map, enforcing image-space consistency. On IXI and HCP
benchmarks, our method improves PSNR by 0.75 dB and SSIM by up to 0.0279 over state-of-the-art approaches, with larger gains at higher artifact severities. It further demonstrates robust zero-shot generalization on real-world clinical data acquired with unseen scanning parameters, where existing methods either fail to remove artifacts or introduce additional distortions.
Read the full abstract ▾
All authors Honglin Xiong, Yuxian Tang, Feng Li, Yulin Wang, Lei Xiang, Dinggang Shen, Qian Wang
Computer Vision
Representation Forcing for Bottleneck-Free Unified Multimodal Models
Yuqing Wang, Zhijie Lin, Ceyuan Yang et al. (+10 more)
Unified
multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate
tokens before pixels; these
tokens then stay in context to guide pixel
diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.
Read the full abstract ▾
All authors Yuqing Wang, Zhijie Lin, Ceyuan Yang, Yang Zhao, Fei Xiao, Hao He, Qi Zhao, Zihan Ding, Fuyun Wang, Shuai Wang, Youliang Zhang, Haoqi Fan, Xihui Liu
Computer Vision
Lumos-Nexus: Efficient Frequency Bridging with Homogeneous Latent Space for Video Unified Models
Jiazheng Xing, Hangjie Yuan, Lingling Cai et al. (+9 more)
Connector-based
video unified models have demonstrated strong capability in instruction-grounded
video synthesis, but integrating a large high-fidelity generator into the unified training loop is computationally prohibitive, limiting achievable visual quality. We therefore propose Lumos-Nexus, a training-efficient unified
video generation framework that facilitates the development of strong
reasoning-driven generation capabilities while significantly enhancing visual fidelity. Lumos-Nexus adopts a two-stage design: 1) During training, only a lightweight generator is aligned with the understanding block to learn to take in
reasoning-driven semantic control. 2) During inference, we introduce Unified Progressive Frequency Bridging (UPFB) to progressively hand off generation to a high-capacity pretrained generator in the shared latent space, enabling coarse-to-fine refinement and producing high-fidelity
videos without compromising
reasoning quality. To fill the gap in
reasoning-driven
video generation
benchmarks, we introduce VR-Bench, which assesses a model's capability to translate inferred intent into coherent and semantically aligned
video content. Extensive experiments demonstrate that Lumos-Nexus achieves substantial gains in visual realism and temporal coherence on VBench, while exhibiting strong
reasoning-based generative performance on VR-Bench. Code and models are available at https://jiazheng-xing.github.io/nexus-lumos-home/.
Read the full abstract ▾
All authors Jiazheng Xing, Hangjie Yuan, Lingling Cai, Xinyu Liu, Yujie Wei, Fei Du, Tao Feng, Hai Ci, Jiasheng Tang, Weihua Chen, Fan Wang, Yong Liu
Computer Vision
Linear Scaling Video VLMs for Long Video Understanding
Cristobal Eyzaguirre, Jiajun Wu and Juan Carlos Niebles
Video vision-
language models (VLMs) are increasingly used in long-horizon and streaming settings, yet most
video encoders still rely on spatiotemporal self-
attention, causing compute and latency to grow quadratically with the number of frames. Existing efficiency methods improve scalability but often lose accuracy relative to full self-
attention, for example through aggressive frame/
token dropping or coarse
attention approximations. We introduce StateKV, an inference-time method that adapts pretrained long-
video VLMs to linear-time
video prefill by carrying cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used for decoding. Across three long-
video benchmarks and seven models spanning three families and multiple scales, StateKV remains close to full self-
attention and consistently outperforms dominant sliding-window / recency-based streaming approximations, without fine-tuning or architectural changes. StateKV also reduces
video-prefill cost measured FLOPs, enabling stronger accuracy at a fixed compute budget by running larger models. These results suggest a practical step toward scalable long-
video understanding.
Read the full abstract ▾
Computer Vision
SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models
Olaf Dünkel, Basavaraj Sunagad, Haoran Wang et al. (+3 more)
Measuring structured object understanding in
vision foundation models remains challenging due to inconsistent evaluation protocols and limited part-level supervision. Semantic correspondence (SC) evaluates this capability by testing whether object parts can be matched across instances and categories under large variations in appearance, viewpoint, and geometry. To enable a systematic SC evaluation, we introduce SOCO, a new
benchmark for Semantic Object Correspondence that introduces a taxonomy of correspondence types and provides consistent, functionally meaningful keypoint annotations across 100 categories and over 1M correspondence pairs. In addition, SOCO includes keypoint
language descriptions, enabling the evaluation of large
vision-
language models (LVLMs) and their fine-grained part-level understanding. Comprehensive experiments reveal that (i)
vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories and only partially capture object-part position, (ii) LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching, exposing a gap between
language-grounded localization and fine-grained visual correspondence, and (iii) correspondence performance predicts performance on dense downstream tasks, including
segmentation, tracking, 3D pose estimation, and 3D detection, more strongly than ImageNet classification. Together, these findings position SOCO as a
benchmark for structured, part-level representation quality in
vision and
multimodal foundation models.
Read the full abstract ▾
All authors Olaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, Adam Kortylewski
Computer Vision
KLIP: localized distribution shift detection via KL-divergence with diffusion priors in Inverse Problems
Alireza Kheirandish, Jihoon Hong and Sara Fridovich-Keil
Diffusion models have shown promising performance as data-driven priors for computational imaging, as well as some capacity to detect out-of-distribution (OOD) images. However, existing approaches to OOD detection often require some knowledge of the shifted distribution, fail to detect subtle or localized distribution shifts, and operate on full images, rather than the indirect measurements available in inverse problems. We propose an OOD detection metric based on the Kullback-Leibler divergence between the
diffusion prior and the posterior distribution, that (i) does not require any calibration data or knowledge of the shifted distribution, and (ii) can detect whole images as OOD as well as localize OOD patches within an image. Experimentally, we show that this metric can detect subtle yet semantically meaningful distribution shifts, such as the shift from healthy liver CT scans to those with tumors, and generalizes across different types of
diffusion models,
datasets, and inverse problems. Our code can be found at https://github.com/voilalab/KLIP.
Read the full abstract ▾
Computer Vision
Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction
Mungyeom Kim, Minkyeong Jeon, Honggyu An et al. (+10 more)
Dynamic scene reconstruction from
monocular video remains a fundamental challenge in computer
vision. Existing feed-forward methods predict 3D Gaussians pixel-wise for each frame, suffering from duplicated Gaussians and view-dependent biases that hinder effective learning of scene motion. We present C4G, a feed-forward 4D reconstruction framework built upon a compact set of timestamp-conditioned learnable Gaussian query
tokens. Each
token aggregates corresponding features across the full temporal context and decodes a 3D Gaussian whose position is modulated by the target timestamp, enabling globally coherent motion modeling without per-scene optimization. To capture fine-grained details, we further introduce a
video diffusion model-based rendering enhancement module. Since our framework effectively aggregates features into Gaussians, we extend this capability to feature lifting, producing a 4D feature field that supports point tracking and dynamic scene understanding. C4G achieves strong novel-view synthesis performance using significantly fewer Gaussians and without requiring camera poses, while exhibiting stronger motion modeling and robustness to large temporal gaps.
Read the full abstract ▾
All authors Mungyeom Kim, Minkyeong Jeon, Honggyu An, Jaewoo Jung, Hyuna Ko, Jisang Han, Hyeonseo Yu, Donghwan Shin, Sunghwan Hong, Takuya Narihira, Kazumi Fukuda, Yuki Mitsufuji, Seungryong Kim
Computer Vision
CoFiDA-M: Concept-Aware Feature Modulation for Cross-Domain Adaptation with Image-Only Inference
Nurjahan Sultana, Moi Hoon Yap, Xinqi Fan et al. (+1 more)
Models for AI-based skin cancer screening suffer a severe performance drop when shifting from expert dermoscopic (source) images to consumer-grade clinical (target) images, hindering real-world deployment. Existing domain adaptation methods often ignore crucial semantic invariants, such as clinical concepts. While new foundation models like MONET can provide this semantic information as dense, probabilistic scores, this metadata is unavailable at test time, creating a deployment paradox for practical image-only screening tools. We address this gap by proposing CoFiDA-M, a privileged information framework that learns from concepts at training time but deploys as an image-only model. Our method trains a teacher network that uses MONET concept probabilities to guide a FiLM modulator, transforming visual features into a semantically “edited" feature space. A lightweight, image-only student is then trained to reproduce this edited representation, not just the teacher's final predictions. This
distillation “bakes" the clinical
reasoning into the student's weights. On a challenging multi-
dataset benchmark, our image-only student significantly outperforms state-of-the-art approaches, especially in melanoma recall. Our work provides a practical and generalizable framework for leveraging noisy, probabilistic metadata as privileged information, demonstrating strong cross-
dataset robustness and potential for real-world deployment beyond dermatology. Implementation code is available at: https://github.com/mmu-dermatology-research/CoFiDA.git
Read the full abstract ▾
All authors Nurjahan Sultana, Moi Hoon Yap, Xinqi Fan, Wenqi Lu
Computer Vision
TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generation
Ruotong Liao, Guowen Huang, Qing Cheng et al. (+6 more)
Text-to-
video (T2V) generation faces challenging questions when generating
videos with long horizons containing multiple events. Inspired by the intrinsics of the
diffusion process, we probe
video diffusion transformers (DiTs) and uncover intrinsic turning points in the DiT denoising trajectory where conditioning text affects generation from global layout to fine-grained details. Building on this finding, we present TunerDiT, a simple yet effective progressive steering method that requires no additional training for multi-event generation. TunerDiT comprises two steering handles: (1) Event-Partitioned Masking that enforces event boundaries while allowing cross-event transition bands; (2) Cross-Event
Prompt Fusion that injects neighboring event semantics for late-stage refinement. We contribute a self-curated
prompt suite for benchmarking multi-event generation, i.e., Meve. TunerDiT achieves state-of-the-art performance across 8 metrics and offers a tunable trade-off between
video consistency and event separation, compared with other training-free methods. The improvement in text
alignment increases with the event count, indicating a scaling possibility with increasing event count.
Read the full abstract ▾
All authors Ruotong Liao, Guowen Huang, Qing Cheng, Guangyao Zhai, Lei Zhang, Xun Xiao, Thomas Seidl, Daniel Cremers, Volker Tresp
Computer Vision
Recognizing Co-Speech Gestures in-the-Wild
Sindhu B Hegde, K R Prajwal and Andrew Zisserman
While humans naturally gesture during
speech, only a sparse subset of these co-
speech gestures are visually depictive and semantically linked to specific spoken words. In this paper, we introduce a large-scale
dataset – Gesture Recognition in the Wild (GRW), comprising co-
speech gestures corresponding to a diverse vocabulary of 155 words. GRW contains 140k manually annotated
video clips where the word is spoken, with 17k instances of semantic co-
speech gestures including their frame-level temporal boundaries. The
video clips are collected 'in the wild' from public-facing discourse, including lectures, talk shows, and interviews, covering a diverse range of speakers and visual conditions. We also introduce
video models to: (a) classify gestures as semantic or not; (b) recognize the word corresponding to a co-
speech gesture; and (c) temporally localize the gesture. These models are trained and evaluated on the GRW
dataset and compared against a range of strong baselines, establishing
benchmark results for all three tasks. The
dataset, annotations, and trained models are publicly available on the project website: https://www.
robots.ox.ac.uk/~vgg/research/grw.
Read the full abstract ▾
Computer Vision
SurGe: Improved Surface Geometry in Point Maps
Karim Knaebel, Gonzalo Martin Garcia, Christian Schmidt et al. (+4 more)
Recent feedforward 3D reconstruction methods predict point maps and estimate global 3D geometry remarkably well. However, their predictions still exhibit inaccurate local surface geometry, which is clearly visible qualitatively but only weakly reflected in common metrics. To make these errors more explicit in evaluation, we introduce a point map normal metric that evaluates the local surface orientation induced by neighboring 3D predictions. To reduce these errors, we propose two complementary components: a point gradient matching loss that supervises depth-normalized 3D finite differences, and a Neighborhood
Attention Decoder (NAD) that progressively upsamples features and uses Neighborhood
Attention for local feature mixing. Across eight zero-shot
monocular geometry
benchmarks, our model, SurGe, achieves the best average rank for global point map AbsRel and consistently improves local point map and point map normal evaluations.
Read the full abstract ▾
All authors Karim Knaebel, Gonzalo Martin Garcia, Christian Schmidt, Ilya Fradlin, Lucas Nunes, Daan de Geus, Bastian Leibe
Computer Vision
Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement
Aziz Al-Najjar, Marzieh Amini, James R. Green et al. (+1 more)
Most learning-based camera-
LiDAR calibration methods treat each camera-
LiDAR pair independently, ignoring the rigid
geometric coupling in multi-camera platforms. As a result, per-camera estimates may be individually accurate yet inconsistent at the system level. We present a two-stage framework for joint multi-camera
LiDAR extrinsic calibration that combines learned pairwise matching with
geometric refinement. First, CMRNext is applied independently to each camera to produce initial extrinsic estimates and dense 2D-3D correspondences. These predictions are then jointly refined through a multi-frame bundle adjustment with reprojection, per-camera prior, and relative-pose prior terms. This approach converts pairwise predictions into a globally consistent multi-camera calibration. Experiments on KITTI (in-domain for CMRNext) and Walkley (out-of-domain)
datasets show improved per-camera accuracy and inter-camera consistency. On KITTI, the method achieves 0.89 cm translation error and 0.038 rotation error. On Walkley, it reduces translation error from 108.6 cm to 3.1 cm, highlighting the benefit of explicit multi-camera coupling when single-camera predictions are less reliable.
Read the full abstract ▾
All authors Aziz Al-Najjar, Marzieh Amini, James R. Green, Felix Kwamena
Computer Vision
nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving
Zhiyu Huang, Johnson Liu, Rui Song et al. (+12 more)
Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations, infer
agent interactions, and make safe decisions. However, existing AD
datasets and
benchmarks mainly target perception, prediction, or planning, and provide limited supervision for
reasoning over realistic long-tail driving scenes. We introduce nuReasoning, a large-scale real-world
dataset and
benchmark for
reasoning-centric AD. Following the lineage of nuScenes and nuPlan, nuReasoning advances real-world AD
datasets and
benchmarks toward
reasoning in long-tail driving scenarios. The
dataset contains 20,000 clips, each 20 seconds long, collected across multiple cities, with synchronized multi-camera images,
LiDAR data, HD maps, object annotations, and human-verified
reasoning annotations spanning Spatial
Reasoning, Decision
Reasoning, and Counterfactual
Reasoning. Unlike prior
datasets that focus primarily on visual question answering, nuReasoning supports both
reasoning evaluation and planning evaluation, enabling a direct study of how
reasoning supervision affects driving performance. Experiments show that fine-tuning VLMs on nuReasoning substantially improves driving-specific question answering, while incorporating
reasoning supervision into VLA training improves planning performance even when textual
reasoning outputs are disabled at inference time. These results establish nuReasoning as a foundation for evaluating and improving robust, interpretable,
reasoning-driven AD systems in realistic long-tail settings.
Read the full abstract ▾
All authors Zhiyu Huang, Johnson Liu, Rui Song, Xuewei Qi, Yiluan Guo, Zewei Zhou, Hanyin Zhang, Mingxuan Gao, Valeria Xu, Jiali Chen, Yun Zhang, Tianhui Cai, Ruining Yang, Yishan Shen, Jiaqi Ma
Computer Vision
Rosario Forte, Giuseppe Lando and Antonino Furnari
Continuous episodic
memory is a core capability for autonomous
agents operating in dynamic, real-world environments, yet current streaming
video benchmarks provide limited tools for diagnosing what models remember and for how long. We introduce Egostream, a diagnostic
benchmark for streaming episodic
memory evaluation in
egocentric vision. \egostream organizes 2,250 curated questions along seven cognitive dimensions: detail, spatial, temporal, event, social, causal, and prospective
memory. We introduce the Answer Validity Window (AVW), which specifies the temporal span an answer remains valid as the observed scene evolves. This allows us to expand the questions into 8,528 recall-conditioned evaluations, enabling controlled testing from instant to ultra-long-term recall while separating genuine model forgetting from natural world-state changes. We rigorously establish baseline performance through a unified streaming MLLM framework that compares several state-of-the-art
memory-management mechanisms, covering sliding windows,
attention sinks, KV-cache
pruning, merging, and offloading. Experiments within a unified Qwen3-VL backbone reveal that comparable aggregate accuracies mask starkly different
memory profiles. For instance,
token pruning preserves fine-grained details and temporal structure significantly better than
token merging, while quantized offloading rescues ultra-long-term recall. Ultimately, all mechanisms operate well below real-time (>1s per frame), and top performing methods ceil at about 45% accuracy, exposing critical gaps in current architectures. Egostream provides the diagnostic testbed needed to close these gaps. Project website, news and updates at: https://saroo25.github.io/Egostream/
Read the full abstract ▾