Computer Vision
Causal-EVC: Breaking Emotional Spurious Causality via Spatiotemporal Grounding and Counterfactual Intervention
Cheng Ye, Weidong Chen, Peipei Song et al. (+1 more)
Emotional
Video Captioning aims to generate factually accurate and emotionally empathetic descriptions. While recent methods have recognized the importance of visual causes to guide emotion perception and caption generation, they fundamentally rely on simple
attention matching, which inevitably suffers from {causal redundancy and spurious correlations} in co-occurrence bias (e.g., misclassifying “sadness” as “joy” on a sunny beach), leading to severe shortcut learning from confusing backgrounds. Furthermore, existing evaluations fail to verify whether models have genuinely mastered causal
reasoning or merely exploited background confounders. To address these limitations, we first construct {EVC-CauseGround}, a comprehensive
benchmark with dense spatio-temporal causal annotations. Crucially, it introduces a carefully selected {Causal-Faithfulness Subset} to explicitly quantify genuine emotion-cause attribution. Second, we propose {Causal-EVC}, an emotion-grounding captioning framework, which introduces a Motion-guided Causal Spatiotemporal Localization module to precisely decouple causal triggers from background confounders. Besides, we introduce an Interpretable Sparse Emotion Routing module. By synthesizing counterfactual representations and formulating a novel counterfactual contrastive objective, we enforce the model to anchor its emotion predictions strictly on authentic causal triggers instead of confusing background. Extensive experiments show that Causal-EVC not only achieves the best performance on semantic metrics but also exhibits significant advantages in the causal-faithfulness subset, which demonstrates that our model could mine emotional cues from genuine visual causes and mitigate co-occurrence bias for interpretable
multimodal emotion understanding.
Read the full abstract ▾
All authors Cheng Ye, Weidong Chen, Peipei Song, Zhendong Mao
Computer Vision
DSPO: Diversity-aware Subjective Policy Optimization for Robust Emotional Reasoning
Cheng Ye, Weidong Chen, Bingyan Xu et al. (+1 more)
Reinforcement Learning has significantly advanced the complex
reasoning capabilities of MLLMs. However, prevailing RL
algorithms suffer a severe failure in emotion
reasoning tasks. These methods heavily rely on deterministic hard-label supervision and point-wise isolated evaluation, creating a fundamental gap with the inherently subjective and continuously distributed nature of human emotions. Furthermore, unlike explicit physical objects, emotional states are deeply implicit within visual cues. This abstract nature exacerbates visual
hallucinations in MLLMs, leading to plausible yet ungrounded emotional evidence. To address these limitations, we propose Diversity-Aware Subjective Policy Optimization (DSPO), a reinforcement learning framework that jointly promotes subjective affective coverage and visual grounding. First, we construct a context-grounded emotional distribution prior in the VAD space by combining the lexical prior of the annotated emotion with image-specific contextual information. Based on this prior, we introduce a Distribution-Aligned Emotional Diversity Reward (DEDR), which measures the leave-one-out marginal contribution of each candidate emotion within a rollout. DEDR rewards candidates whose inclusion brings the predicted affective set closer to the context-grounded prior, thereby preserving plausible subjective interpretations without encouraging unconstrained dispersion. We further develop Counterfactual Visual Intervention Gating (CVIG), which masks the visual region highlighted in the
reasoning process and uses the resulting candidate-wise probability changes to reduce the weights of interpretations unsupported by visual evidence. Extensive experiments demonstrate that DSPO achieves state-of-the-art performance across multiple public
benchmarks, especially on the cross-domain performance, i.e., improving +10.8% on average cross-domain accuracy than EMO-R3.
Read the full abstract ▾
All authors Cheng Ye, Weidong Chen, Bingyan Xu, Zhendong Mao
Computer Vision · Cross-listed from cs.AI
Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradients
Aram Davtyan, Pablo Acuaviva, Sebastian Stapf et al. (+1 more)
Can a population of neural networks develop a useful division of labor without a shared gate or gradients between
agents? We study a setting where each network has its own weights, trains independently on the same heterogeneous data, and can ask another
agent for help through a forward pass. Unlike mixtures of experts, where a jointly trained gate assigns inputs to experts, specialization here must emerge without central control. We test this in a small scale proxy for predictive visual pretraining. Initially identical
agents are finetuned on an unlabeled mixture of six visual domains using masked prediction of frozen DINOv3 features. We measure specialization by asking whether the best
agent for an input aligns with its latent domain, and utilization by asking whether responsibility is distributed across
agents. We progressively remove central control, ending with DISCO (DIStributed COllaboration) where each
agent locally selects a helper, reads its internal state through a gradient free channel, and rewards its router only for the improvement that help provides. Specialization emerges and is useful. Randomly routed populations underperform a single generalist, while semantically routed populations outperform it, showing that specialization rather than population size drives the gain. Specialization persists without a central router, and gradient free communication lets nonexperts exploit emergent expertise. In DISCO, a random
agent helped by the expert matches the solo generalist, while experts surpass it, including on data outside the specialization mixture. Local routers select the emergent expert for 98% of inputs. These effects persist across population size, model capacity, data imbalance, and finetuning seeds, providing measurable evidence for the dynamics needed by decentralized predictive pretraining.
Read the full abstract ▾
All authors Aram Davtyan, Pablo Acuaviva, Sebastian Stapf, Paolo Favaro
Computer Vision
Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning
Chiyuan He, Zihuan Qiu, Fanman Meng et al. (+5 more)
Benefiting from transferable visual-textual
alignment, CLIP has been widely adopted for class-incremental learning (CIL). However, existing learners either repeatedly update components shared across tasks, leading to knowledge overwriting, or overly isolate new-task updates, hindering the reuse of CLIP's transferable knowledge and limiting plasticity. Moreover, the text-based or bimodal classifier designs still fail to effectively integrate complementary information from the visual and textual modalities. To address these challenges, we introduce DuLBE, which couples dual-mode low-rank learning with a bridge-prototype ensemble classifier for exemplar-free CIL. DuLBE allocates two visual low-rank update modes according to the gradient demand and uses gradient routing to coordinate them: a compact and rewritable shared mode is selected from historically occupied visual directions to reuse transferable knowledge, while residual modes provide low-interference channels for task-specific variations. Building on the resulting stable inter-modal structure, we further construct geodesic bridges between visual prototypes and text
embeddings on the unit hypersphere, and ensemble reliable bridge prototypes to compensate for the modality-gap limitations of textual decision boundaries. Extensive experiments under multiple settings show that DuLBE achieves state-of-the-art CIL performance while retaining the high parameter efficiency of low-rank tuning.
Read the full abstract ▾
All authors Chiyuan He, Zihuan Qiu, Fanman Meng, Chao Wang, Liangjiang Chen, Linfeng Xu, Qingbo Wu, Hongliang Li
Computer Vision
FastVR: Efficient Streaming Video Restoration with One-Step Diffusion
Xiaoxu Chen, Qin Yang, Haoran Bai et al. (+2 more)
Diffusion-based
video restoration recovers realistic details, but its practical deployment is limited by two efficiency bottlenecks: costly VAE encoding and decoding, and the quadratic cost of full self-
attention in
diffusion transformers (DiTs). This paper presents FastVR, a streaming
video restoration framework built on a one-step
diffusion model, which delivers strong restoration quality and temporal consistency while processing 1080p
video at 11 FPS on a single H20 GPU. To improve inference efficiency, FastVR combines a lightweight VAE with chunk-wise causal
attention, which substantially reduces the computational cost. During training, it further adopts velocity consistency regularization and continuous trajectory learning, which improve restoration quality. Extensive experiments show that FastVR is more efficient than the evaluated
diffusion baselines while achieving state-of-the-art performance on synthetic and real-world
benchmarks. We hope that this work supports further progress in the community.
Read the full abstract ▾
All authors Xiaoxu Chen, Qin Yang, Haoran Bai, Sibin Deng, Ying Chen
Computer Vision
NesTok: Nested Self-Aligned 1D Tokenizer for Autoregressive Image Generation
Jiawei Zhang, Shuhao Liu, Rong Huang et al. (+4 more)
One-dimensional (1D) variable-length visual tokenizers enable adaptive compression by varying the number of
tokens, allowing downstream autoregressive (AR) models to flexibly trade off generation quality against computational cost using a single tokenizer. However, existing approaches based on nested dropout often fail to fully exploit the representational capacity of the tokenizer, resulting in suboptimal performance in both image reconstruction and generation. In this work, we introduce NesTok, a nested self-
alignment framework tailored to dynamic visual tokenizers. NesTok introduces cross-length training, which jointly optimizes reconstruction across
token lengths while using the full-length sequence to guide shorter counterparts, enabling shorter
token sequences to approach the reconstruction quality of full-length sequences. On ImageNet, NesTok improves substantially over standard training and achieves an rFID score of 0.98. On downstream image generation, it achieves the state-of-the-art gFID score of 1.46 on ImageNet 256$\times$256 among existing variable-length autoregressive image generation methods. Code will be available at https://github.com/Jiawei804/NesTok.
Read the full abstract ▾
All authors Jiawei Zhang, Shuhao Liu, Rong Huang, Yuancheng Li, Zhihui Li, Xiaojun Chang, Changlin Li
Computer Vision
Drag as Evidence: Motion-Grounded Latent Recomposition for Drag-Based Editing
Xinyu Pu, Hongsong Wang, Jie Gui et al. (+1 more)
Modern image editors excel at semantic manipulation and visual synthesis, yet remain limited in precise spatial control, motivating the development of drag-based editing. However, existing drag-based methods often struggle to balance drag accuracy with natural, plausible, and intent-aligned generation. We propose MoRe-Drag, a motion-grounded drag-based editing method. Our key insight is to treat pixel-space warping as coarse motion evidence, and to inject this evidence into the generative sampling trajectory. Specifically, MoRe-Drag performs region-aware latent recomposition over refinement, inpainting, and anchor regions, coupled with stage-adaptive conditioning that progressively shifts from motion-grounded structure formation to semantic refinement. We further support an instruction-free interface by adapting the MLLM-based text encoder for drag-aware instruction inference. Experiments on DragBench-SR and DragBench-DR show that MoRe-Drag substantially improves drag precision over strong base editors and achieves superior drag accuracy among SOTA drag-based methods, while delivering strong semantic consistency and visually realistic results. Code and
dataset will be publicly released.
Read the full abstract ▾
All authors Xinyu Pu, Hongsong Wang, Jie Gui, Pan Zhou
Computer Vision · Cross-listed from quant-ph
Quantum Fidelity Landscape-Guided Prior Calibration for Single-Circuit QGAN Image Generation
Xue Yang, Rigui Zhou, Dax Enshan Koh et al. (+5 more)
Quantum Generative
Adversarial Networks (QGANs) have emerged as representative generative models in the Noisy Intermediate-Scale
Quantum (NISQ) era and have attracted increasing
attention in
quantum machine learning. However, most existing QGAN methods rely on patch-based decomposition strategies, which weaken the global consistency of generated images and increase
quantum resource overhead. In this work, we investigate a simpler approach: pixel-level, end-to-end image generation using a single-
quantum-
circuit QGAN. By analyzing the structural matching relationship between the
quantum prior and the target data distribution in
Hilbert space, we provide a new theoretical perspective for understanding the training behavior of naive end-to-end QGANs. Specifically, we introduce the
Quantum Fidelity Landscape (QFL), defined as the pairwise-fidelity structure induced by an ensemble of
quantum states and preserved under shared
unitary transformations of the
quantum generation process. We show that, under a fixed Lipschitz readout, this invariant imposes a one-sided bound on decoded sample separation, motivating calibration of the prior-induced QFL before
adversarial training. To validate this theoretical insight, we propose BasicQGAN, a QGAN framework incorporating
quantum prior calibration. Before
adversarial optimization, BasicQGAN aligns the prior-induced QFL with the data-induced QFL. Experimental results on small-scale grayscale image
datasets show that BasicQGAN achieves stable and effective end-to-end pixel-level image generation while requiring fewer
qubits and trainable parameters than representative patch-based
quantum generators. Furthermore, experiments with different initial
quantum-state ensembles show that QFL-calibrated ensembles achieve better generative performance.
Read the full abstract ▾
All authors Xue Yang, Rigui Zhou, Dax Enshan Koh, Siong Thye Goh, Yitao Tang, ShiZheng Jia, Young-Wook Cho, Hongyu Chen
Computer Vision
AESplat: Advancing Pose-Free Feed-Forward 3D Gaussian Splatting via Decoupled Appearance Modeling
Shiwei Ren, Zhiang Liu, Yongchun Fang et al. (+1 more)
Pose-free feed-forward 3D Gaussian Splatting (3DGS) has demonstrated remarkable potential for generalized novel view synthesis. However, existing methods typically predict Gaussian appearance attributes represented by spherical harmonics (SH) in the same manner, overlooking the fundamental distinction between view-independent and view-dependent appearance, which results in suboptimal rendering quality. In this paper, we present AESplat, a novel and general framework for pose-free feed-forward 3DGS that introduces an effective decoupled appearance modeling strategy based on an analysis of SH, enabling higher-quality rendering. Specifically, AESplat directly derives the zeroth-order SH coefficient, which represents the base view-independent appearance component, from the input images without training. The higher-order SH coefficients are subsequently predicted by a shallow multilayer perceptron equipped with two efficient 3D-aware inductive biases to model view-dependent appearance variations. Extensive experiments across multiple
datasets demonstrate that our method significantly outperforms state-of-the-art approaches, achieving a $0.8$ dB improvement in PSNR over the pose-free method NAS3R and a $1.1$ dB improvement over the pose-required method DepthSplat on the RealEstate10K
dataset. Project page: https://aesplat.github.io/.
Read the full abstract ▾
All authors Shiwei Ren, Zhiang Liu, Yongchun Fang, Hongwei Chen
Computer Vision
When Semantics Matter: Reliability-Aware Semantic-Rhythm Control for Co-Speech Gesture Generation
Zhirui Xing, Long Ye, Kaige Li et al. (+2 more)
Co-
speech gesture generation aims to synthesize natural gestures that are both temporally synchronized with
speech and semantically consistent with the spoken content. Although recent methods can generate rhythmically plausible motions, they often rely heavily on acoustic prosody while underutilizing textual semantics, especially when semantic annotations are incomplete, noisy, or unavailable. Consequently, the generated gestures may follow
speech rhythm while failing to express the intended semantics. To address this problem, we propose a reliability-aware semantic-rhythm control framework for co-
speech gesture generation. We first learn a discrete motion prior that represents continuous gestures in a compact and structured motion-code space. We then introduce a dual-branch semantic contribution estimation mechanism consisting of a full
multimodal branch and an audio-only branch. Their distributional discrepancy is formulated as conditional information gain to quantify how much textual semantics changes the predicted motion. Based on this estimate, a controllable semantic-rhythm objective selectively strengthens semantic guidance in content-relevant segments while limiting unnecessary semantic intervention in rhythm-dominant segments. Furthermore, we treat background noise as an acoustic reliability condition and introduce noise-conditioned feature modulation together with beneficial latent perturbation to improve generation robustness under realistic acoustic environments. Experiments on
benchmark datasets demonstrate that the proposed framework achieves a favorable balance among semantic expressiveness, rhythmic synchronization, motion diversity, and robustness, enabling reliable and controllable co-
speech gesture generation.
Read the full abstract ▾
All authors Zhirui Xing, Long Ye, Kaige Li, Ziyi Xu, Ming Meng
Computer Vision
Reprogramming Vision-Language Models via Structured Prompt Reparameterization
Zizhao Li, Chengyi Cai, Mohammed Yaqoob Ansari et al. (+3 more)
Visual reprogramming adapts pretrained models to downstream tasks by modifying their input and output interfaces while keeping the backbone fixed. In
vision-
language models, existing methods mainly rely on intra-class
prompt aggregation and do not explicitly model relationships among classes. However, fine-grained categories often exhibit highly overlapping attribute descriptions and strong inter-class correlation in the text
embedding space, where discriminative cues lie in subtle low-variance components. We propose Reparameterized Inter-Class Visual Reprogramming (RVP), a structured framework that aggregates multiple text
prompts within each class and applies residual correction across classes. We also show that CLIP-based visual reprogramming with input-independent linear output aggregation can be expressed as a linear mapping from frozen image
embeddings to downstream logits, and use this view to design a structured reparameterization that models shared semantic components and class-specific differences. RVP uses only a single visual
prompt and can be reparameterized at inference into a frozen backbone followed by a linear classifier, incurring nearly zero computational overhead. Across 11 few-shot classification
benchmarks and four CLIP backbones, RVP consistently improves over prior visual reprogramming methods with comparable or better inference efficiency.
Read the full abstract ▾
All authors Zizhao Li, Chengyi Cai, Mohammed Yaqoob Ansari, Feng Liu, Joseph West, Kourosh Khoshelham
Computer Vision
ReWorld-Track: A Recursive Event World Model for Language-Guided Multi-Camera Tracking
Haoyang Wu, Shoudong Han, Chaoyue Li et al. (+3 more)
Language-guided multi-camera tracking must preserve a target identity across unobserved gaps, where similar candidates and uncertain returns can make early associations unreliable. A wrong match can corrupt the history used to predict later observations and propagate identity errors across subsequent camera handoffs. We propose ReWorld-Track, a recursive event world model that carries association uncertainty into future predictions. Candidate matches and continued waiting define alternative target states, whose posterior probabilities are used to update a persistent recurrent belief. This representation preserves uncertainty about alternative trajectories through successive observations. This belief predicts the next camera, arrival time, and entry region, while appearance and
language evidence guide association. By training across successive handoffs, the model learns to retain uncertainty that remains useful for later predictions and identity decisions. ReWorld-Track achieves HOTA scores of 65.19 on CityFlowV2 and 45.36 on MTMMC, with improved identity continuity across repeated handoffs. On MTMMC, its structured posterior update gains 0.50 HOTA points over a similarly sized generic updater and 0.94 points over fixed-moment soft association, raising next-camera accuracy from 86.03% to 87.41% and reducing median arrival-time error from 0.78 s to 0.71 s for subsequent target returns.
Read the full abstract ▾
All authors Haoyang Wu, Shoudong Han, Chaoyue Li, Sijia Chen, Zhenyang Xie, Sihan Wang
Computer Vision
You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming
Zizhao Li, Mohammed Yaqoob Ansari, Xinyu Su et al. (+3 more)
Visual reprogramming is a parameter-efficient method for adapting pretrained models, yet its training can remain computationally expensive: even with a frozen backbone, visual
prompts are often optimized through the full model for hundreds of epochs. Before changing what the pretrained model sees, we ask whether we are fully using what it already tells us. We find that modeling the full source response can already yield strong downstream predictions without
prompt optimization. Motivated by this observation, we introduce You Only Reprogram Once (YORO), which constructs a downstream predictor from the frozen response space in a single forward-only traversal. Its
Bayesian Discriminant Mapping (BDM) derives a covariance-aware affine mapping from streaming class statistics, requiring no backpropagation, optimizer updates, or repeated visits to the training set. When further input adaptation helps, YORO-FP optionally refines the visual
prompt for 20 epochs. BDM also extends naturally to CLIP by treating attribute-
prompt similarities as source responses. Across three full-data settings, YORO improves average accuracy over the strongest prior gradient-free mapping by 18.4–24.4%. On 16-shot CLIP, it raises the four-backbone average from 71.4% to 77.2%. YORO-FP provides further gains on selected tasks, while validation often retains the one-pass predictor. These results suggest a different default for visual reprogramming: read out the frozen response first, and optimize the input only when needed.
Read the full abstract ▾
All authors Zizhao Li, Mohammed Yaqoob Ansari, Xinyu Su, Jiayang Ao, Joseph West, Kourosh Khoshelham
Computer Vision
Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptation
Youjia Zhang, Huiling Liu, Soyun Choi et al. (+2 more)
Continual test-time adaptation (CTTA) adapts a source model to an unlabeled test stream whose distribution may change over time. Existing TTA methods often assess prediction reliability using confidence or
entropy, which primarily reflect the model's self-certainty for the current sample. In CTTA, accumulated target observations can provide complementary evidence for correcting the source prediction, but this history may become misaligned as the target distribution changes. The key question is therefore not how much the correction differs from the source prediction, but whether and how strongly it should be applied. This paper proposes Gain-Aware INtervention (GAIN), a backpropagation-free CTTA framework guided by a simple principle: history proposes, gain decides. GAIN maintains compact target statistics to form a correction proposal and a posterior-predictive evaluator that accounts for estimation uncertainty. The resulting source-relative gain estimates the proposal's benefit and determines a sample-specific intervention strength along a continuous path through efficient one-dimensional optimization. Gain-controlled predictions then update the target statistics online, limiting the propagation of unreliable corrections, all without backpropagation, sample storage, or replay. Across five
benchmarks, our method achieves strong predictive performance, with favorable accuracy–calibration–efficiency trade-offs in continual adaptation. On ImageNet-C, for example, GAIN achieves 61.9% accuracy with near-source calibration. It remains stable under diverse and challenging continual shifts while running 15.9x faster than a representative optimization-based CTTA baseline.
Read the full abstract ▾
All authors Youjia Zhang, Huiling Liu, Soyun Choi, Jaehong Yoon, Sungeun Hong
Computer Vision
FocusVTC: Efficient and High-Performance Visual Text Compression with Adaptive Resolution
FangZhi Zhong, Xuerui Qiu, Yuqi Pan et al. (+4 more)
Long-context
reasoning in large
language models incurs substantial computation and
memory costs. Visual text compression (VTC) reduces input length by rendering text as images, but fixed-resolution rendering creates a compression-performance trade-off: low DPI saves
tokens at the expense of legibility, whereas high DPI spends
tokens on irrelevant content. We introduce FocusVTC, which breaks this trade-off through adaptive resolution while preserving general
multimodal capabilities. It combines compressed low-DPI global views with selective region enhancement, integrating enhanced views into ongoing
reasoning. We construct 29.4K high-quality
Reasoning-Evidence Localization (REL) chain-of-thought examples (REL-CoT) that link
reasoning traces to page indices and bounding boxes. Multi-resolution REL supervised fine-tuning (REL-SFT) teaches the model to localize relevant regions, and Group Relative Policy Optimization learns when to enhance resolution and how to use the resulting observations, without a separate continual-pretraining stage. At 72 DPI on RULER v1, FocusVTC scores 87.4 at $2.9\times$ input compression, including tool observations, versus 57.5 for Glyph at $3.0\times$ input compression. It surpasses its text-input backbone on LongBench (56.40 versus 55.86), improves the MRCR macro-average by 13.91 points, and achieves a 51.19 macro-average on VTCBench. The MRCR latency evaluation also shows a $2.79\times$ online end-to-end speedup over Text. General
multimodal capabilities are preserved, with MMMU increasing from 65.12 to 66.73 and MME from 2424.02 to 2457.62.
Read the full abstract ▾
All authors FangZhi Zhong, Xuerui Qiu, Yuqi Pan, Ya Liu, Shaowei Gu, Bo Xu, Guoqi Li
Computer Vision
VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
Yuanwei Hu, Bo Peng, Yuheng Jia et al. (+3 more)
Vision-
language pre-training has reshaped image clustering, giving rise to
language-assisted image clustering (LaIC), which leverages textual semantics to complement visual representations. Despite the rapid proliferation of LaIC methods, it remains unclear how much LaIC has actually advanced image clustering, as existing studies generally suffer from major limitations, including inconsistent experimental settings, inadequate
dataset selection, and limited evaluation dimensions. To address this gap, we introduce VLM4Cluster, a comprehensive
benchmark for image clustering in the era of pre-trained
vision-
language models (VLMs). VLM4Cluster implements 17 representative methods spanning classical, deep, and
language-assisted image clustering, and evaluates them on 20
datasets covering classical, challenging, fine-grained, large-scale, and out-of-distribution settings. Beyond effectiveness, VLM4Cluster systematically investigates image clustering along three complementary dimensions: robustness to
adversarial perturbations, generalization under distribution shifts, and computational efficiency. Our study shows that LaIC substantially advances the clustering performance frontier on many semantically demanding
benchmarks, generally exhibits stronger generalization under distribution shifts, and achieves a more favorable effectiveness-efficiency trade-off. However, its gains become less consistent on large-scale and fine-grained
datasets, while
language assistance does not systematically reduce sensitivity to
adversarial perturbations. VLM4Cluster is released at https://github.com/YuanweiHuu/VLM4Cluster.
Read the full abstract ▾
All authors Yuanwei Hu, Bo Peng, Yuheng Jia, Xinting Hu, Yadan Luo, Wenjie Zhu
Computer Vision
OCA: ODE-Driven Cross-Attention for Image-to-Point-Cloud Registration
Pei An, Jiaqi Yang, Yulong Wang et al. (+2 more)
Cross-
attention is a crucial component in learning-based image-to-point-cloud (I2P) registration. Although existing cross-
attention mechanisms have achieved promising progress,
attention ambiguity remains a fundamental challenge that hinders the learning of discriminative 2D-3D correspondences. To address this problem, we revisit cross-
attention and establish ordinary differential equations (ODEs) to model the ideal I2P feature interaction. Based on this formulation, we develop an ODE-driven cross-
attention (OCA) module that refines feature representations and
attention matrices through ODEs. In practice, OCA can be seamlessly integrated into existing I2P registration frameworks. To validate its effectiveness, we incorporate OCA into five state-of-the-art baselines and evaluate on four public
benchmark datasets. Experimental results demonstrate that OCA improves registration recall by up to 5%, 9%, and 15% under the standard, fine-tuning, and zero-shot settings, respectively.
Read the full abstract ▾
All authors Pei An, Jiaqi Yang, Yulong Wang, Siwen Quan, Liangliang Nan
Computer Vision · Cross-listed from cs.LG
PE-OPSD: Internalizing Prompt Enhancement into Flow-matching Models via On-Policy Self-Distillation
Mingfeng Lin, Chengfei Cai, Lin Xu et al. (+3 more)
Text-to-image users often provide concise and underspecified
prompts, whereas generative models benefit from detailed textual conditions for reliable instruction following. Existing systems bridge this gap with
Prompt Enhancers (PEs) that rewrite raw
prompts at inference time, introducing additional latency and leaving
prompt elaboration external to the generator. We instead view enhanced
prompts as privileged training information and ask whether their benefits can be internalized. We propose
Prompt-Enhanced On-Policy Self-
Distillation (PE-OPSD) for text-to-image flow-matching models. During training, a raw-
prompt student follows its own generation trajectory, while an enhanced-
prompt teacher provides vector-field targets at the states visited by the student. This on-policy supervision distills the behavior induced by enhanced
prompts into the raw-
prompt student without requiring additional text–image pairs. At inference, both the PE and teacher are removed, and the student generates directly from raw
prompts. Across multiple model families, PEs, and
benchmarks, PE-OPSD achieves the strongest aggregate
prompt fidelity among the evaluated baselines, yields positive aggregate visual appeal gains, and retains the base-model inference efficiency.
Read the full abstract ▾
All authors Mingfeng Lin, Chengfei Cai, Lin Xu, Chengqian Ma, Yuxiang Wei, Liang Han
Computer Vision
Beyond Binary Preferences: Graded Preference Optimization for Limb-Motion Captioning
Yanan Wang, Tingsong Li, Kaixun Jiang et al. (+3 more)
Vision-
Language Models (VLMs) can generate rich
video captions, yet often misidentify which person performs an action or which limb is involved, particularly across camera cuts. Improving these details requires evaluation and training that distinguish missing information from incorrect assertions. We introduce FlexBench, a
benchmark spanning 3,105 shots and 18,161 evaluation queries, with human-verified identities and systematic per-person coverage of fine-grained limb actions and states. Its reference-derived checklists support automated assessment of complete captions in their person and shot contexts. Our Graded Physical
Alignment score (GPA) awards credit for correct content and deducts points for incorrect or fabricated actions, making these errors explicit in the aggregate score. Building on this rubric, we propose Graded Margin Direct Preference Optimization (GM-DPO), which assigns stronger preference margins and greater training weight to more severe action errors. Across three VLM backbones, GM-DPO achieves the highest substantive-action and GPA scores among the evaluated preference objectives, improving GPA over DPO by 2.02-3.40 points. On Qwen3-8B, it reduces the weighted
hallucination rate by 21.3% relative to DPO. These gains accompany sustained long-form output, improved shot structure, and competitive performance on three additional
multimodal benchmarks.
Read the full abstract ▾
All authors Yanan Wang, Tingsong Li, Kaixun Jiang, Chongyang Zhong, Chenwei Xoe, Zhaohe Liao
Computer Vision
CrossTimeEdit: A Decade-Spanning Cross-View Dataset and Reward-Guided Editing for Historical Street-View Generation
Hanwen Lu, Jun He, Mingjia Yang et al. (+4 more)
Historical street-view imagery records urban
evolution, but uneven coverage leaves substantial gaps in historical records. Generating plausible past appearances requires restoring changed structures while preserving persistent scene content. We construct VIGOR-his, a decade-spanning cross-view
dataset containing 43,653 location-level quadruplets across 11 cities on three continents. Its automated pipeline performs spatial pairing, consistency screening, change classification, and the generation and validation of satellite-based change descriptions and local editing instructions. Based on VIGOR-his, we propose CrossTimeEdit, a model that reformulates historical street-view generation as editing, using recent street views to constrain viewpoint and unchanged appearance and temporal satellite differences as change evidence. Starting from FLUX.2 [Klein] 4B, we train CrossTimeEdit through supervised fine-tuning (SFT) followed by online reinforcement learning (RL). We design three street-view editing criteria, namely Instruction
Alignment (IA), Background Preservation (BP), and Quality and Physical Plausibility (QP), as both RL reward dimensions and evaluation metrics. We optimize this multi-reward objective using Within Group Relative Policy Optimization for flow-matching models (Flow-GRPO) with Group reward-Decoupled Normalization Policy Optimization (GDPO), which normalizes each reward dimension before aggregation. CrossTimeEdit improves overall performance across the three editing criteria by 17.12% over the pretrained baseline and outperforms cross-view generation models in scene consistency, visual realism, and perceptual quality. The implementation code,
dataset, and model weights are available at https://luhanwen67.github.io/CrossTimeEdit-release/.
Read the full abstract ▾
All authors Hanwen Lu, Jun He, Mingjia Yang, Hao Wei, Jinhao Huang, Yi Lin, Xiang Zhang
Computer Vision
Pixel-wise Exposure for Highly Robust In-Vehicle Remote-PPG
Jieying Wang, Xinqi Cai, Caifeng Shan et al. (+1 more)
Remote photoplethysmography (rPPG) offers a promising non-contact solution for heart rate monitoring, yet its real-world robustness is fundamentally limited by an inherent hardware limitation: existing camera exposure control paradigms, whether fixed or auto-exposure, impose a uniform exposure time across all pixels within a frame. In high-dynamic-range scenes such as automotive cabins with strong directional sunlight, this spatially invariant exposure constraint inevitably leads to localized facial overexposure or underexposure, irreversibly corrupting the subtle pulsatile signals essential for rPPG at the point of capture, a physical degradation that no downstream
algorithm can recover. To overcome this bottleneck, we propose PixExpo (Pixel-wise Exposure), a "temporal-for-spatial" framework that sequentially captures frames under a predefined cyclic exposure schedule and performs non-iterative pixel-wise fusion. At each pixel location, PixExpo selects the observation closest to an rPPG-motivated target intensity. This criterion seeks to reduce local saturation and severe underexposure rather than optimize perceptual appearance. PixExpo requires no sensor modification but assumes programmable frame-level exposure control. We validate the proposed PixExpo framework using our newly introduced MEX-Drive
dataset, comprising 48 participants under real-world driving conditions. Experimental results demonstrate that PixExpo outperforms manufacture-default auto-exposure methods, reducing the mean absolute error (MAE) by 7.21 bpm (from 13.94 to 6.73 bpm) and increasing the success rate by 37.29 percentage points (from 25.95% to 63.24%) across challenging driving scenarios.
Read the full abstract ▾
All authors Jieying Wang, Xinqi Cai, Caifeng Shan, Wenjin Wang
Computer Vision
Scaling Video Generation for Reasoning: At What Cost?
Weihang Guo, Xiaoyu Wu, Yifei Wang et al. (+2 more)
We study whether scaling
video generation enables models to reason about hidden information from the past frames, and at what computational cost. Our controlled
benchmark requires predicting nine prescribed moves of an initially solved 2x2x2 Rubik's Cube from a fixed view of three faces. Correct predictions require inferring how actions change hidden states, and the simulator provides exact ground truth for evaluation. Models learn plausible cube geometry early, while correct sticker configurations require substantially more training. Although validation MSE follows approximate power-law scaling, lower MSE loss does not reliably indicate downstream
reasoning capabilities. Smaller autoregressive models achieve higher state accuracy with limited compute, while larger models reach higher accuracy after more training. At roughly 0.1 PF-days, the 70M-parameter model correctly predicts the visible sticker configuration in 44.6% of post-action frames, compared with 0.3% for the 1B model, which reaches 83.7% at 3.14 PF-days. Symbolic state supervision raises the 20M model's frame accuracy from 31.1% to 67.3% at the same training-data budget, suggesting that learning representations of state changes can complement scaling.
Read the full abstract ▾
All authors Weihang Guo, Xiaoyu Wu, Yifei Wang, Niloofar Mireshghallah, Lydia E. Kavraki
Computer Vision
Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation
Ziying Zhang, Litao Li, Junchao Liao et al. (+4 more)
A
video can exhibit convincing motion and photorealism yet fail immediately when visual text collapses. Unlike generic scene content, visual text is unforgiving in
video generation: minor stroke corruption, temporal instability, or editing errors instantly break legibility and realism. Existing
benchmarks overlook this challenge by treating text as incidental or using static OCR metrics that ignore temporal dynamics. We introduce VidScribe, a unified diagnostic
benchmark spanning four generation regimes: writing from
language (T2V), transferring text identity from a reference (R2V), sustaining text under dynamics (I2V), and localized text editing (V2V). VidScribe contains 803 human-verified samples across a 12-axis conditionally orthogonal factor space covering Intrinsic Text Properties, Physical Imaging Conditions, and Temporal Behavior. For reliable evaluation, we build a track-grounded, gated suite with 11 shared metrics and 2 task-specific probes under strict measurability conditions. Benchmarking 11 commercial and open-source systems shows that
video text capability is non-monolithic, with content recognition decoupled from stroke-level glyph correctness. Performance is highly task-asymmetric: I2V sustains text most reliably, whereas V2V editing is the primary bottleneck. Counter-intuitively, degradation concentrates on a small subset of text-centric structural and temporal factors rather than adverse imaging conditions. Further probes show that visual references improve glyph and typographic fidelity rather than content accuracy, while localized editing fails to isolate target text without corrupting undeclared source text. Beyond evaluation, VidScribe also provides an actionable training signal, where
benchmark-aligned preference optimization measurably improves visual text generation. https://huggingface.co/
datasets/Vicky0720/VidScribe.
Read the full abstract ▾
All authors Ziying Zhang, Litao Li, Junchao Liao, Tianyi Zeng, Siyu Zhu, Long Qin, Zhenghao Zhang
Computer Vision
AffectReveal: Event-Grounded Emotion Recognition Beyond Visual Appearances
Yihao Qian, Runhao Zeng, Sicheng Zhao et al. (+3 more)
Visual emotion recognition commonly assumes that all evidence required for prediction is contained in the observed image or
video. Yet the same visible reaction can convey different emotions depending on events beyond the input: tears, for example, may indicate grief or joy. We formulate Event-Grounded Emotion Recognition (EGER), where emotion recognition requires recovering the affect-determining event. We construct EGER-Bench, comprising 10,052
videos and 10,734 images across 11 emotions, two source domains, and four visual settings. A study with six annotators shows that event context raises human recognition accuracy from 33.96% to 72.08%, confirming that visual evidence alone is often insufficient. Semantic relevance alone does not solve EGER: a plausible event may imply the wrong emotion if its identity, focal-person role, relationship, or outcome is misinterpreted. We therefore propose AffectReveal, a tuning-free framework that first constructs and independently verifies evidence-grounded alternatives over these affect-critical factors. It then cross-checks the recovered event against face-masked in-media facts through bidirectional atomic evidence support, while retaining the original unmasked input for final prediction. Across three downstream models and four input settings, AffectReveal yields average UAR gains of 5.26–10.53 points. For three fine-tunable models, it also enables untuned models to outperform their fine-tuned visual-only counterparts in all 12 accuracy comparisons, without updating downstream parameters.
Read the full abstract ▾
All authors Yihao Qian, Runhao Zeng, Sicheng Zhao, Feng Liang, Hongmin Cai, Mingkui Tan
Computer Vision
ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language Models
Ruochen Zhang, Yao Huang, Yitong Sun et al. (+5 more)
While
Multimodal Large
Language Models (MLLMs) are increasingly deployed in safety-critical domains, their reliability is threatened by
multimodal implicit risks. Unlike explicit threats, these hazards emerge when individually benign text and neutral visual entities logically converge to induce unsafe outputs. Current detection methods fail to address this because they overlook the underlying risk activation mechanisms that govern cross-modal risk activation, leading to single-modality shortcut learning and hallucinated rationalizations. To bridge this gap, we first construct TriggerBench, the first
dataset explicitly modeling risk compositionality (5,600 instances). By formally isolating Key Elements and Trigger Elements to build counterfactual contrastive pairs, TriggerBench eliminates risk residues and forces models to perform genuine logical deduction rather than superficial pattern matching, which provides a rigorous foundation for both large-scale training and fine-grained evaluation. Building on this, we propose a Step-Supervised Structured
Reasoning training framework and employ it to train ThinkingGuard, a specialized guard model. Inspired by Situation Awareness theory, we decouple implicit risk identification into progressive cognitive stages, and utilize a step-reward Monte Carlo Tree Search
algorithm to explore optimal
reasoning trajectories, which are then distilled into the model through Dual-Constraint Preference
Alignment. Extensive experiments across both standard and implicit safety
benchmarks demonstrate that ThinkingGuard achieves strong performance. Project resources are available at https://github.com/FroggyChen/ThinkingGuard.
Read the full abstract ▾
All authors Ruochen Zhang, Yao Huang, Yitong Sun, Jiahe Xie, Jin Yan, Jifan Ma, Yuanfang Guo, Xingxing Wei
Computer Vision
FM-ReID: Selective Competitive Token Routing for Object Re-Identification
Zhiqi Li, Xiaowei Zhou, Zeyuan Sun et al. (+2 more)
Object re-identification (ReID) faces a recurring challenge: different identities can share highly similar global appearances, while the cues that distinguish them are localized, heterogeneous, and visible only under particular viewpoints. This challenge arises in animal ReID through markings, contours, and scars, in person ReID through subtle clothing and accessory cues, and in vehicle ReID through localized appearance details. Although visual foundation models encode such information in dense
tokens, a single holistic descriptor can obscure discriminative local signals. We propose FM-ReID, an end-to-end framework that formulates local representation learning as selective competitive
token routing. Its Competitive Fine-grained Mining module uses multiple mining queries and a residual query to compete for dense DINOv3
tokens. Above-prior selection retains
tokens preferentially allocated to each mining query, while the residual slot receives
tokens excluded from the
retrieval descriptors. The resulting multi-query descriptors are jointly trained with a holistic representation for
retrieval, without fixed spatial partitions or equal-area constraints. FM-ReID achieves strong results on animal, person, and vehicle ReID
benchmarks, supporting competitive
token routing as an effective way to augment holistic foundation-model representations.
Read the full abstract ▾
All authors Zhiqi Li, Xiaowei Zhou, Zeyuan Sun, Feng Gao, Junyu Dong
Computer Vision
How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
Janet Wang, Yunbei Zhang, Xiao Wang et al. (+1 more)
Medical
Vision-
Language Models (VLMs) show significant promise for clinical image understanding, offering accurate diagnosis with interpretable
reasoning. However, a critical performance gap exists between their strong
vision encoders and the full
multimodal model: in dermatology, the MedSigLIP encoder outperforms MedGemma by an average of 10.26 percentage points even when both use zero target-task labels; few-shot linear probing provides further evidence of strong visual representations. This gap motivates an investigation of how visual information is used in end-to-end diagnosis and why plausible-sounding predictions can lack grounding in image evidence. Using dermatology as our primary testbed, we systematically investigate three hypotheses for this phenomenon. We further provide a mechanistic analysis of the model's internal
attention patterns, showing that a simple describe-then-decide prompting strategy increases
vision attention by 30-40% during generation. Task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in our evaluation. To address these challenges, we combine label-free prompting with low-label encoder-assisted reranking while keeping the VLM frozen. We validate the interventions across five VLM backbones in dermatology and provide supporting representation and
attention analyses across additional medical modalities.
Read the full abstract ▾
All authors Janet Wang, Yunbei Zhang, Xiao Wang, Jihun Hamm
Computer Vision
SCCM: Spherically Consistent Coarse Matching for ERP Dense Feature Correspondence
Gyeonggwan Lee, Eunsoo Im, Seunghwan Hong et al. (+1 more)
Dense feature matching between 360$^\circ$ panoramas underpins omnidirectional pose estimation, 3D reconstruction, and SLAM. Such panoramas are stored in the equirectangular projection (ERP), which unrolls the viewing sphere onto a flat chart and thereby introduces three distinct distortions – a longitudinal seam (
topology), latitude-dependent stretch (metric), and non-uniform pixel area (area) – that the coarse stage of perspective-trained dense matchers does not model, so these matchers degrade systematically on ERP. We show that correcting the three distortions at the coarse-stage interfaces where they arise – pairwise distortions in
attention, per-pixel distortion in covisibility gating – improves PCK@1$^\circ$ from 0.230 to 0.275 on Matterport3D under a fixed coarse scaffold, with the refiner architecture unchanged – our central result. Concretely, SCCM (Spherically Consistent Coarse Matching) augments a chart-naive cross-
attention/dual-softmax coarse matcher with two sphere-derived priors: Spherical Positional
Attention (SPA) pairs a yaw-periodic RoPE (
topology) with a tangent-plane bias (metric), and Area-Aware Covisibility (AAC) applies a pre-sigmoid log-area correction (area). The chart-naive scaffold serves as a controlled reference, separating the scaffold-replacement effect from the spherical-prior effect. Instantiated in the RoMa V1 framework with the same frozen encoder, refiner architecture, and loss, SCCM also outperforms the ERP-native EDM (0.163) and an ERP-retrained RoMa V1 (0.198) under a unified ERP dense matching protocol, while perspective-trained matchers largely fail on ERP. It further transfers zero-shot to Stanford2D3D and, when trained on outdoor Holo360D, leads there as well.
Read the full abstract ▾
All authors Gyeonggwan Lee, Eunsoo Im, Seunghwan Hong, Junghun Suh
Computer Vision
Foresight at the Event Boundary: Evaluating Physical Prediction in Video World Models
Estela Monserrat Arriaga Santana, Julian Rosas Scull, Ehécatl Sacamch'en Núñez Rico et al. (+1 more)
Video world models are largely regarded as predictive models of the physical world and are therefore expected to anticipate the consequences of observed events. However, evaluation has mainly focused on reference similarity, physical-law consistency, or judgment plausibility, estimating anticipation only indirectly. We address this directly: when a release or impact has just occurred but its consequence is withheld, can a world model anticipate what should happen next? We introduce an event-anchored evaluation based on 62 controlled real-world free-fall recordings and 124 clips spanning three object types, with fine-grained release and impact annotations and ground-truth trajectories. The protocol separates consequence production, temporal placement, and physical realization. Across six contemporary
video generation and world models, Runway and Veo produce release and subsequent impact events at rates above 93% but often initiate them substantially late, whereas Cosmos-Predict-2.5 and MAGI-1 frequently preserve the pre-event state and produce little or no measurable consequence. Among measurable falls, plausible timing does not necessarily imply physically consistent motion. We further conduct a 15-participant, 20-condition human study in which participants describe the expected consequence from a single event-anchored frame and draw its trajectory. Human predictions favor the recorded future in aggregate while revealing genuine ambiguity among plausible continuations. Overall, physical foresight emerges as a sequence of distinct challenges: initiating a consequence, anchoring it in time, and realizing its motion.
Read the full abstract ▾
All authors Estela Monserrat Arriaga Santana, Julian Rosas Scull, Ehécatl Sacamch'en Núñez Rico, Hugo Jair Escalante
Computer Vision · Cross-listed from cs.RO
Distilling Privileged Control Barrier Functions into RGB-Only Safety Filters for Dynamic Visual Navigation
Seungyeon Yoo, Gawon Lee, Seungwoo Jung et al. (+2 more)
RGB-only end-to-end visual navigation policies remain vulnerable to collisions in real-world dynamic environments, motivating a dedicated safety layer. Existing visual Control Barrier Function (CBF) approaches seek to provide safety from RGB observations, but often rely on real-time rendering or explicit scene reconstruction and are primarily designed for static scenes, limiting their practicality for onboard deployment. We propose a teacher-student visual
distillation framework that transfers the safety behavior of a privileged CBF teacher to an RGB-only student filter for dynamic environments. The student maps a short RGB history,
robot velocity, and a nominal control action directly to a safe action, while the teacher uses ground-truth
robot and obstacle states in a real-to-sim dynamic Gaussian Splatting environment. To reduce the teacher-student information gap, the teacher constructs safety constraints only from obstacles observable within the student's RGB history. It also accounts for obstacle-velocity uncertainty to improve robustness to motion variations, while action augmentation exposes the student to diverse safe and unsafe nominal actions to better capture the safety boundary. At deployment, the student requires only RGB observations and
robot velocity, without explicit 3D reconstruction or online rendering. Experiments show that the proposed method outperforms visual CBF baselines and improves the safety of RGB-based navigation policies under dynamic obstacle motion. Project page: https://syeon-yoo.github.io/distill-cbf-site/.
Read the full abstract ▾
All authors Seungyeon Yoo, Gawon Lee, Seungwoo Jung, Inkyu Jang, H. Jin Kim