Computer Vision
Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation
Bowen Zheng, Weijian Luo, Guang Yang et al. (+2 more)
Most discrete visual tokenizers rely on a default design: every position in the sequence shares the same codebook. Such a constant-codebook design can consume the finite-data
entropy budget very quickly. We observe that the per-position empirical conditional
entropy decays so quickly along the sequence that, after a few positions, the conditional distribution becomes essentially deterministic. On ImageNet with $K=16384$, this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the
dataset. We call this phenomenon the
Entropy Cliff. Under full conditional utilization, larger codebooks exhaust the finite-data
entropy budget in fewer positions. The empirical
entropy decreases more gradually in our
language comparison, where the effective
entropy per position stays below the codebook capacity. To address this, we propose Variable Codebook Size Quantization (VCQ), where the codebook size $K_t$ grows monotonically along the sequence from $K_{\min}=2$ to $K_{\max}$, leaving the loss function, parameter count, and AR training procedure unchanged. With a vanilla autoregressive
Transformer and standard next-
token prediction, VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet $256\times256$, compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively. Scaled up, it reaches gFID 1.71 with 684M autoregressive parameters, without any extra training techniques such as semantic regularization or causal
alignment. The extreme information bottleneck at $K_{\min}=2$ encourages semantic concentration in early encoder features: a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks. These results show that what matters is not only the total capacity of the codebook, but also how that capacity is distributed and organized.
Read the full abstract ▾
All authors Bowen Zheng, Weijian Luo, Guang Yang, Colin Zhang, Tianyang Hu
Computer Vision
Bridging visual saliency and large language models for explainable deep learning in medical imaging
Paul Valery Nguezet, Elie Tagne Fute, Yusuf Brima et al. (+2 more)
The opaque nature of deep learning models remains a significant barrier to their clinical adoption in medical imaging. This paper presents a
multimodal explainability framework that bridges the gap between convolutional neural network (CNN) predictions and clinically actionable insights for
brain tumor classification, leveraging large
language models (LLMs) to deliver human-interpretable diagnostic narratives. The proposed framework operates through three coupled stages. First, nine CNN architectures are extended with a dual-output hybrid formulation that simultaneously optimises a classification head and a
segmentation head, enabling spatially richer feature learning. Second, visual saliency attribution methods, namely Grad-CAM, Grad-CAM++, and ScoreCAM, are applied to generate class-discriminative heatmaps, which are subsequently refined into binary tumor masks via an adaptive percentile thresholding pipeline. Third, the resulting masks are mapped onto the Harvard-Oxford cortical atlas to translate pixel-level evidence into named neuroanatomical structures, and the extracted findings are encoded into a structured JSON file that conditions three LLMs (Grok3, Mistral, and LLaMA) to generate coherent, radiological-style diagnostic reports. Evaluated on a
dataset of 4,834 contrast-enhanced T1-weighted
brain MRI images spanning three tumor classes, InceptionResNetV2 achieved the highest classification performance and Grad-CAM++ yielded the best
segmentation overlap. Among the
language models, Grok3 led in lexical diversity and coherence, while LLaMA achieved the highest readability score. By integrating visual, anatomical, and linguistic modalities into a unified pipeline, the framework produces explanations that are technically grounded and meaningfully interpretable, advancing the transparency and clinical accountability of artificial intelligence assisted
brain tumor diagnosis.
Read the full abstract ▾
All authors Paul Valery Nguezet, Elie Tagne Fute, Yusuf Brima, Benoit Martin Azanguezet, Marcellin Atemkeng
Computer Vision
EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields
Zhaoyang Yang, Yurun Jin, Lizhe Qi et al. (+2 more)
Pretrained
video diffusion models provide powerful spatiotemporal generative priors, making them a natural foundation for robotic world models. While recent world-action models jointly optimize future
videos and actions, they predominantly treat
video generation as an auxiliary representation for policy learning. Consequently, they insufficiently explore the inverse problem: leveraging action signals to guide
video synthesis, thereby often failing to preserve precise
robot spatial geometry and fine-grained
robot-object interaction dynamics in the generated rollouts. To bridge this gap, we present EA-WM, an Event-Aware Generative World Model that effectively closes the loop between kinematic control and visual perception. Rather than injecting joint or end-effector actions as abstract, low-dimensional
tokens, EA-WM projects actions and kinematic states directly into the target camera view as Structured Kinematic-to-Visual Action Fields. To fully exploit this geometrically grounded representation, we introduce event-aware bidirectional fusion blocks that modulate cross-branch
attention, capturing object state changes and interaction dynamics. Evaluated on the comprehensive WorldArena
benchmark, EA-WM achieves state-of-the-art performance, outperforming existing baselines by a significant margin.
Read the full abstract ▾
All authors Zhaoyang Yang, Yurun Jin, Lizhe Qi, Cong Huang, Kai Chen
Computer Vision · Cross-listed from cs.AI
Event-Causal RAG: A Retrieval-Augmented Generation Framework for Long Video Reasoning in Complex Scenarios
Peizheng Yan, Yu Zhao, Liang Xie et al. (+3 more)
Large
vision-
language models perform well on short- and medium-length
video understanding but still struggle to maintain coherent event
memory and recover long-range relationships in ultra-long
videos. End-to-end methods are limited by visual-
token growth and context length, while fixed-segment
retrieval often fragments complete events and weakens state-transition modeling.We propose Event-Causal RAG (EC-RAG), a lightweight
retrieval-augmented framework for ultra-long and streaming
video reasoning. A dual visual-audio sentinel mechanism segments
video streams into semantically complete events, represented as State-Event-State (SES) structures that organize observable pre-event states, central events, and post-event states as event-local causal transitions. These transitions are stored in dual vector-graph
memory and temporally connected through entity-consistent trajectories. During question answering, bidirectional graph
retrieval recovers relevant predecessor and successor events, and answers are generated using both structured
memory and the corresponding
video evidence.We further introduce ECV-1H, an hour-scale long-
video QA
benchmark dedicated to directed event-causal
reasoning, with all source
videos exceeding one hour. It covers over 150 hours of untrimmed
video and contains 1,251 fully human-annotated QA pairs. EC-RAG improves overall accuracy by 4.96%–11.67% across three open-source
video foundation models and achieves consistent gains across public
datasets. On a single RTX 5090 GPU with 32 GB of
memory, EC-RAG can continuously process
videos while maintaining controlled streaming
memory usage.
Read the full abstract ▾
All authors Peizheng Yan, Yu Zhao, Liang Xie, Juntong Qi, Mingming Wang, Erwei Yin
Computer Vision
SuperFace: Preference-Aligned Facial Expression Estimation Beyond Pseudo Supervision
Zejian Kang, Xuanyang Xu, Wentao Yang et al. (+6 more)
Accurate facial estimation is crucial for realistic digital human animation, and ARKit blendshape coefficients offer an interpretable representation by mapping facial motions to semantic animation controls. However, learning high-quality ARKit coefficient prediction remains limited by the absence of reliable ground-truth supervision. Existing methods typically rely on capture software such as Live Link Face to provide pseudo labels, which may contain noisy activations, biased coefficient magnitudes, and missing or inaccurate facial actions. Consequently, models trained with supervised learning tend to reproduce imperfect pseudo labels rather than optimize for perceptual expression fidelity. In this paper, we propose SuperFace, a preference-driven framework that moves ARKit facial expression estimation from pseudo-label imitation toward human-aligned perceptual optimization. Instead of treating software-estimated coefficients as fixed ground truth, SuperFace uses them only as an initialization and further improves coefficient prediction through human preference feedback on rendered facial expressions. By aligning the model with perceptual judgments rather than numerical pseudo labels, SuperFace enables more visually faithful and expressive facial animation. Experiments show that SuperFace improves expression fidelity over Live Link Face supervision, demonstrating the effectiveness of preference-driven optimization for semantic facial action prediction.
Read the full abstract ▾
All authors Zejian Kang, Xuanyang Xu, Wentao Yang, Kai Zheng, Yuanchen Fei, Hongyuan Zou, Hui Shan, Shuo Yang, Xiangru Huang
Computer Vision
Retina-RAG: Retrieval-Augmented Vision-Language Modeling for Joint Retinal Diagnosis and Clinical Report Generation
Abdelrahman Zaian, Sheethal Bhat, Mohamed Abdalkader et al. (+1 more)
Diabetic Retinopathy (DR) is a leading cause of preventable blindness among working-age adults worldwide, yet most automated screening systems are limited to image-level classification and lack clinically structured reporting. We propose Retina-RAG, a low-cost modular framework that jointly performs DR severity grading, macular edema (ME) detection, and report generation. The architecture decouples a high-performance retinal classifier and a parameter-efficient
vision-
language model (Qwen2.5-VL-7B-Instruct) adapted via Low-Rank Adaptation (LoRA), enabling flexible component integration. A
retrieval-augmented generation (RAG) module injects curated ophthalmic knowledge together with structured classifier outputs at inference time to improve diagnostic consistency and reduce
hallucinations. Retina-RAG achieves an F1-score of 0.731 for DR grading and 0.948 for ME detection, substantially outperforming zero-shot Qwen (0.096, 0.732) and MMed-RAG (0.541, 0.641) on a retinal disease detection
dataset with captions. For report generation, Retina-RAG attains ROUGE-L 0.438 and SBERT similarity 0.884, exceeding all baselines. The full framework operates on a single consumer-grade GPU, demonstrating that clinically structured retinal AI can be achieved with modest computational resources.
Read the full abstract ▾
All authors Abdelrahman Zaian, Sheethal Bhat, Mohamed Abdalkader, Andreas Maier
Computer Vision
DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models
Juntong Wang, Jiarui Wang, Huiyu Duan et al. (+3 more)
Existing text-to-image (T2I)
benchmarks largely rely on fixed
prompt sets, leaving them vulnerable to overfitting and
benchmark contamination once publicly released and repeatedly reused. In this work, we propose DynT2I-Eval, a fully automated dynamic evaluation framework for T2I models. It constructs a structured visual semantic space from long-form descriptions, decomposing
prompts into controllable dimensions (e.g., subject, logical constraint, environment, and composition). This enables the continuous generation of fresh
prompts via task-specific spaces and difficulty-aware sampling. DynT2I-Eval evaluates model performance across text
alignment, perceptual quality, and aesthetics. Heterogeneous outputs are unified into
prompt-conditioned pairwise comparisons, allowing a dynamic scheduler, micro-batch aggregation, and weighted
Bayesian updates to maintain a stable online leaderboard despite changing
prompt distributions and model injection. Experiments with independently sampled
prompt streams demonstrate that continually refreshed
prompts provide a robust evaluation protocol, reducing the impact of
prompt-set-specific tuning. Simulations and ablations further confirm that the proposed ranking framework achieves a strong balance among cold-start convergence, late-entry discovery, and long-run ranking fidelity.
Read the full abstract ▾
All authors Juntong Wang, Jiarui Wang, Huiyu Duan, Lewei Li, Guangtao Zhai, Xiongkuo Min
Computer Vision · Cross-listed from cs.LG
Mean Mode Screaming: Mean–Variance Split Residuals for 1000-Layer Diffusion Transformers
Pengqi Lu
Scaling
Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes
token representations and suppresses centered variation. Through mechanistic
auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of
attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than
token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.
Read the full abstract ▾
Computer Vision
Beyond Forgetting in Continual Medical Image Segmentation: A Comprehensive Benchmark Study
Bomin Wang, Hangqi Zhou, Yibo Gao et al. (+1 more)
Continual learning (CL) is essential for deploying medical image
segmentation models in clinical environments where imaging domains, anatomical targets, and diagnostic tasks evolve over time. However, continual
segmentation still faces three main challenges. First, the scenarios for this task remain insufficiently standardized for real-world clinical settings. Second, existing research has been primarily focused on mitigating forgetting, overlooking the other essential properties such as plasticity. Third, a
benchmark work with comprehensive evaluation on existing methods is stll desirable. To address these gaps, we present such
benchmark study of continual medical image
segmentation. We first define three clinically motivated scenarios, namely Domain-CL, Class-CL, and Organ-CL, to respectively capture the cross-center domain shift, the incremental anatomical structure
segmentation, and the cross-organ
segmentation. We then introduce an evaluation framework that measures not only general performance and forgetting, but also plasticity, forward generalizability, parameter efficiency, and replay burden. The results, from extensive experiments with representative CL methods, showed that it was still challenging to develop a model that could satisfy all the requirements simultaneously. Nevertheless, these studies also suggested that the replay-based methods achieve the best overall balance between stability and plasticity, the parameter-isolation methods should be effective at reducing forgetting, though at the cost of increased model size, and the forward generalizability remain a significantly understudied aspect of this research field. Finally, we discuss related learning paradigms and outline future directions for continual medical image
segmentation.
Read the full abstract ▾
All authors Bomin Wang, Hangqi Zhou, Yibo Gao, Xiahai Zhuang
Computer Vision · Cross-listed from cs.CR
Secure Seed-Based Multi-bit Watermarking for Diffusion Models from First Principles
Enoal Gesny and Eva Giboulot
The rapid emergence of generative image models has led to the development of specialized watermarking techniques, particularly in-generation methods such as seed-based
embedding. However, current evaluations in this area remain largely empirical, making them heavily reliant on the specific model architectures used for generation and inversion. This prevents any clear conclusion on the performance of any method, especially regarding security, for which a rigorous definition is lacking. Against this approach, we argue that the effectiveness of a watermarking scheme should be established purely through a thorough theoretical analysis. This is enabled by decoupling the model-dependent part from the actual decision mechanism of the watermarking system. Using this decoupling, we introduce a formal evaluation framework based on security, robustness, and fidelity. This allows precise comparisons between watermarking systems through a characteristic surface representing the trade-off between these three quantities, independent of any generative model. Based on this framework, we propose SSB, a novel watermarking method that generalizes previous seed-based methods by allowing to reach any security-robustness-fidelity regime on its characteristic surface. This work opens the door to the design of modern watermarking systems with theoretical guarantees that do not necessitate any costly empirical evaluations.
Read the full abstract ▾
Computer Vision
Learning Discrete Autoregressive Priors with Wasserstein Gradient Flow
Bowen Zheng, Yihong Luo and Tianyang Hu
Discrete image tokenizers are commonly trained in two stages: first for reconstruction, and then with a prior model fitted to the frozen
token sequences. This decoupling leaves the tokenizer unaware of the model that will later generate its
tokens. As a result, the learned
tokens may preserve image information well but still be difficult for an autoregressive (AR) prior to predict from left to right. We analyze this mismatch using Tripartite Variational Consistency (TVC), which decomposes latent-variable learning into three consistency conditions: conditional-likelihood consistency, prior consistency, and posterior consistency. TVC shows that two-stage training preserves the reconstruction side but leaves prior consistency outside the tokenizer objective: the overall
token distribution is fixed before the AR prior participates in training. Motivated by this view, we add a distribution-level prior-matching signal during tokenizer training, while keeping the reconstruction objective unchanged. We optimize this signal with a Wasserstein-gradient-flow update. For hard categorical
tokens, the update reduces to a
token-level contrast between an auxiliary AR model that tracks the tokenizer's current
token distribution and the target AR prior. It requires only forward passes through the two AR models and does not backpropagate through either of them. The resulting tokenizer, wAR-Tok, reduces AR loss and improves generation FID on CIFAR-10 and ImageNet at comparable reconstruction quality.
Read the full abstract ▾
Computer Vision
AI-Generated Images: What Humans and Machines See When They Look at the Same Image
Silvia Poletti, Justin Ilyes, Marcel Hasenbalg et al. (+2 more)
The misuse of generative AI in online disinformation campaigns highlights the urgent need for transparent and explainable detection systems. In this work, we investigate how detectors for AI-generated images can be more effective in providing human-understandable explanations for their predictions. To this end, we develop a suite of detectors with various architectures and fine-tuning strategies, trained on our large-scale photorealistic fake image
dataset, AIText2Image, and assess their performance on state-of-the-art text-to-image AI generators. We integrate 16 different explainable AI (XAI) methods into our detection framework, and the visual explanations are comprehensively refined and evaluated through a novel approach that prioritizes human understanding of AI-generated images, using both textual and visual responses collected from a survey of 100 participants. This framework offers insights into visual-
language cues in fake image detection and into the clarity of XAI methods from a human perspective, measuring the
alignment of XAI outputs with human preferences.
Read the full abstract ▾
All authors Silvia Poletti, Justin Ilyes, Marcel Hasenbalg, David Fischinger, Martin Boyer
Computer Vision
Autoregressive Visual Generation Needs a Prologue
Bowen Zheng, Weijian Luo, Guang Yang et al. (+2 more)
In this work, we propose Prologue, an approach to bridging the reconstruction-generation gap in autoregressive (AR) image generation. Instead of modifying visual
tokens to satisfy both reconstruction and generation, Prologue generates a small set of prologue
tokens prepended to the visual
token sequence. These prologue
tokens are trained exclusively with the AR cross-
entropy (CE) loss, while visual
tokens remain dedicated to reconstruction. This decoupled design lets us optimize generation through the AR model's true distribution without affecting reconstruction quality, which we further formalize from an ELBO perspective. On ImageNet 256x256, Prologue-Base reduces gFID from 21.01 to 10.75 without classifier-free guidance while keeping reconstruction almost unchanged; Prologue-Large reaches a competitive rFID of 0.99 and gFID of 1.46 using a standard AR model without auxiliary semantic supervision. Interestingly, driven only by AR gradients, prologue
tokens exhibit emergent semantic structure: linear probing on 16 prologue
tokens reaches 35.88% Top-1, far above the 23.71% of the first 16
tokens from a standard tokenizer; resampling with fixed prologue
tokens preserves a similar high-level semantic layout. Our results suggest a new direction: generation quality can be improved by introducing a separate learned generative representation while leaving the original representation intact.
Read the full abstract ▾
All authors Bowen Zheng, Weijian Luo, Guang Yang, Colin Zhang, Tianyang Hu
Computer Vision
Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration
Haisen He, Xiangyu Zou, SongLin Dong et al. (+3 more)
Real-world image degradation is often unknown, spatially non-uniform, and compositional, requiring all-in-one restoration models to adapt a single set of weights to diverse local corruption patterns without test-time degradation labels. Existing methods typically modulate a shared backbone with global
prompts or degradation descriptors, or route features through predefined expert pools. However, compact global conditioning can bottleneck localized degradation evidence, while static expert routing may produce homogeneous updates or rely on unstable sparse assignments. We propose Continuous Expert Assembly (CEA), a
token-wise dynamic parameterization framework for all-in-one image restoration. CEA employs a lightweight Cross-
Attention Hyper-Adapter to probe intermediate spatial features and synthesize instance-conditioned low-rank routing bases and residual directions. Each spatial
token then assembles its own residual update via dense signed dot-product affinities over the generated rank-wise components, avoiding external
prompts, static expert banks, and discrete Top- selection. The resulting assembly rule also admits a linear-
attention perspective, making its dense
token-wise routing behavior transparent. Experiments on AIO-3, AIO-5, and CDD-11 show that CEA improves average restoration quality over strong
prompt-, descriptor-, and expert-based baselines, with the clearest gains on spatially varying and compositional degradations, while maintaining favorable parameter, FLOP, and runtime efficiency.
Read the full abstract ▾
All authors Haisen He, Xiangyu Zou, SongLin Dong, Heng Li, Yihong Gong, Zhiheng Ma
Computer Vision
Pest-Thinker: Learning to Think and Reason like Entomologists via Reinforcement Learning
Xueheng Li, Yu Wang, Tao Hu et al. (+6 more)
Pest-induced crop losses pose a major threat to global food security and sustainable agricultural development. While recent advances in
Multimodal Large
Language Models (MLLMs) have shown strong potential for visual understanding and smart agriculture, their direct application to pest recognition remains limited due to the domain's unique challenges such as high inter-species complexity, intra-species variability, and the scarcity of expert-annotated data. In this work, we introduce Pest-Thinker, a knowledge-driven reinforcement learning (RL) framework that enables MLLMs to reason over fine-grained pest morphology. We first construct two high-definition pest
benchmarks, QFSD and AgriInsect, comprising diverse species and expert-annotated morphological traits. Leveraging these
datasets, we synthesize Chain-of-Thought (CoT)
reasoning trajectories to facilitate structured learning of pest-specific visual cues through Supervised Fine-Tuning (SFT). Subsequently, we employ Group Relative Policy Optimization (GRPO) with a novel feature reward that guides the model to focus on observable morphological evidence, assessed by an LLM-as-a-Judge strategy. Extensive experiments demonstrate that Pest-Thinker substantially improves both in-domain and out-of-domain morphological understanding, marking a step toward expert-level visual
reasoning for intelligent agricultural pest analysis. The
datasets and source code are available upon acceptance.
Read the full abstract ▾
All authors Xueheng Li, Yu Wang, Tao Hu, Ji Huang, Ke Cao, Qize Yang, Rui Li, Jie Zhang, Chengjun Xie
Computer Vision
Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking
Shiao Wang, Xiao Wang, Duoqing Yang et al. (+5 more)
Despite significant progress, RGB-based trackers remain vulnerable to challenging imaging conditions, such as low illumination and fast motion. Event cameras offer a promising alternative by asynchronously capturing pixel-wise brightness changes, providing high dynamic range and high temporal resolution. However, existing event-based trackers often neglect the intrinsic spatial sparsity and temporal density of event data, while relying on a single fixed temporal-window sampling strategy that is suboptimal under varying motion dynamics. In this paper, we propose an event sparsity-aware tracking framework that explicitly models event-density variations across multiple temporal scales. Specifically, the proposed framework progressively injects sparse, medium-density, and dense event search regions into a three-stage
Vision Transformer backbone, enabling hierarchical multi-density feature learning. Furthermore, we introduce a sparsity-aware Mixture-of-Experts module to encourage expert specialization under different sparsity patterns, and design a dynamic pondering strategy to adaptively adjust the inference depth according to tracking difficulty. Extensive experiments on FE240hz, COESOT, and EventVOT demonstrate that the proposed approach achieves a favorable trade-off between tracking accuracy and computational efficiency. The source code will be released on https://github.com/Event-AHU/OpenEvTracking.
Read the full abstract ▾
All authors Shiao Wang, Xiao Wang, Duoqing Yang, Wenhao Zhang, Bo Jiang, Lin Zhu, Yonghong Tian, Bin Luo
Computer Vision · Cross-listed from cs.CL
Uncovering Entity Identity Confusion in Multimodal Knowledge Editing
Shu Wu, Xiaotian Ye, Xinyu Mou et al. (+3 more)
Multimodal knowledge editing (MKE) aims to correct the internal knowledge of large
vision-
language models after deployment, yet the behavioral patterns of post-edit models remain underexplored. In this paper, we identify a systemic failure mode in edited models, termed Entity Identity Confusion (EIC): edited models exhibit an absurd behavior where text-only queries about the original entity's identity unexpectedly return information about the new entity. To rigorously investigate EIC, we construct EC-Bench, a diagnostic
benchmark that directly probes how image-entity bindings shift before and after editing. Our analysis reveals that EIC stems from existing methods failing to distinguish between Image-Entity (I-E) binding and Entity-Entity (E-E) relational knowledge in the model, causing models to overfit E-E associations as a shortcut: the image is still perceived as the original entity, with the new entity's name serving only as a spurious identity label. We further explore potential mitigation strategies, showing that constraining edits to the model's I-E processing stage encourages edits to act more faithfully on I-E binding, thereby substantially reducing EIC. Based on these findings, we discuss principled desiderata for faithful MKE and provide methodological guidance for future research.
Read the full abstract ▾
All authors Shu Wu, Xiaotian Ye, Xinyu Mou, Dongsheng Liu, Xiaohan Wang, Mengqi Zhang
Computer Vision
Metonymy in vision models undermines attention-based interpretability
Ananthu Aniraj, Cassio F. Dantas, Dino Ienco et al. (+2 more)
Part-based
reasoning is a classical strategy to make a computer
vision model directly focus on the object parts that are relevant to the downstream task. In the context of deep learning, this also serves to improve by-design interpretability, often by using part-centric
attention mechanisms on top of a latent image representation provided by a standard, black-box model. This approach is based on a locality assumption: that the latent representation of an object part encodes primarily information about the corresponding image region. In this work, we test this basic assumption, measuring intra-object leakage in
vision models using part-based attribute annotations. Through a comprehensive experimental evaluation, we show that modern pretrained
vision transformers violate the locality assumption and exhibit a strong intra-object leakage, in which each part encodes information from the whole object, a visual metonymy that compromises the faithfulness of
attention-based interpretable-by-design methods for part-based
reasoning, ultimately rendering them uninterpretable. In addition, we establish an upper bound using a two-stage approach that prevents leakage by design. We then show that this inherently disentangled feature extraction improves attribute-driven part discovery on a variety of tasks, confirming the practical impact of intra-object leakage. Our results uncover a neglected issue affecting the interpretability of part-based representations, such as those in CBMs relying on part-centric concepts, highlighting that two-stage approaches offer a promising way to mitigate it.
Read the full abstract ▾
All authors Ananthu Aniraj, Cassio F. Dantas, Dino Ienco, Massimiliano Mancini, Diego Marcos
Computer Vision
Hao Lin, Kunyang Lv, Xu Jiang et al. (+5 more)
Training VideoLLMs for complex
reasoning remains challenging due to sparse sequence level rewards and the lack of fine grained credit assignment over long, temporally grounded
reasoning trajectories. While reinforcement learning with verifiable rewards (RLVR) provides reliable supervision, it fails to capture
token level contributions, leading to inefficient learning. Conversely, existing self
distillation methods offer dense supervision but lack structure and diagnostic specificity, and often interact unstably with reinforcement learning. In this work, we propose VISD, a structured self
distillation framework that introduces diagnostically meaningful privileged information for
video reasoning. VISD employs a
video aware judge model to decompose
reasoning quality into multiple dimensions, including answer correctness, logical consistency, and spatio-temporal grounding, and uses this structured feedback to guide a teacher policy for
token level supervision. To stably integrate dense supervision with RL, we introduce a direction magnitude decoupling mechanism, where rollout level advantages computed from rewards determine update direction, while structured privileged signals modulate
token level update magnitudes. This design enables semantically aligned and fine grained credit assignment, improving both
reasoning faithfulness and training efficiency. Additionally, VISD incorporates curriculum scheduling and EMA based teacher stabilization to support robust optimization over long
video sequences. Experiments on diverse
benchmarks show that VISD consistently outperforms strong baselines, improving answer accuracy and spatio temporal grounding quality. Notably, VISD reaches these gains with nearly 2x faster convergence in optimization steps, highlighting the effectiveness of structured self supervision in improving both performance and sample efficiency for VideoLLMs.
Read the full abstract ▾
All authors Hao Lin, Kunyang Lv, Xu Jiang, Jingqi Tian, Zhongjing Du, Jiayu Ding, Qiaoman Zhang, Hongbo Jin
Computer Vision
Boosting Self-Supervised Tracking with Contextual Prompts and Noise Learning
Yaozong Zheng, Qihua Liang, Bineng Zhong et al. (+4 more)
Learning robust contextual knowledge from unlabeled
videos is essential for advancing self-supervised tracking. However, conventional self-supervised trackers lack effective context modeling, while existing context association methods based on non-semantic queries struggle to adapt to unlabeled tracking scenarios, making it difficult to learn reliable contextual cues. In this work, we propose a novel self-supervised tracking framework, named \tracker, which introduces a dual-modal context association mechanism that jointly leverages fine-grained semantic
prompts and contextual noise to drive the model toward learning robust tracking representations. Adherent to the easy-to-hard learning principle, our contextual association mechanism operates based on two stages. During early training, instance patch
tokens (
prompts) are assigned to both forward and backward tracking branches to facilitate the acquisition of tracking knowledge. As training progresses, contextual noise is gradually injected into the model to perturb feature, encouraging the tracker to learn robust tracking representations in a more complex feature space. Thus, this novel contextual association mechanism enables our self-supervised model to learn high-quality tracking representations from unlabeled
videos, while being applied exclusively during training to preserve efficient inference. Extensive experiments demonstrate the superiority of our method.
Read the full abstract ▾
All authors Yaozong Zheng, Qihua Liang, Bineng Zhong, Shuimu Zeng, Yuanliang Xue, Ning Li, Shuxiang Song
Computer Vision
OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention
Kunyi Li, Michael Niemeyer, Sen Wang et al. (+3 more)
Understanding open-vocabulary 3D scenes with Gaussian-based representations remains challenging due to fragmented and spatially inconsistent semantic predictions across multi-view observations. In this paper, we present OpenGaFF, a novel framework for open-vocabulary 3D scene understanding built upon 3D Gaussian Splatting. At the core of our method is a Gaussian Feature Field that models semantics as a continuous function of Gaussian geometry and appearance. By explicitly conditioning semantic predictions on
geometric structure, this formulation strengthens the coupling between geometry and semantics, leading to improved spatial coherence across similar structures in 3D space. To further enforce object-level semantic consistency, we introduce a structured codebook that serves as a set of shared semantic primitives. Furthermore, a codebook-guided
attention mechanism is proposed to retrieve
language features via similarity matching between query
embeddings and learned codebook entries, enabling robust open-vocabulary
reasoning while reducing intra-object feature variance. Extensive experiments on standard 2D and 3D open-vocabulary
benchmarks demonstrate that our method consistently outperforms prior approaches, achieving improved
segmentation quality, stronger 3D semantic consistency and a semantically interpretable codebook that provides insight into the learned representation.
Read the full abstract ▾
All authors Kunyi Li, Michael Niemeyer, Sen Wang, Stefano Gasperini, Nassir Navab, Federico Tombari
Computer Vision
LARGO: Low-Rank Hypernetwork for Handling Missing Modalities
Niels Vyncke, Pooya Ashtari and Aleksandra Pižurica
Addressing missing modalities is an important challenge in
multimodal image analysis and often relies on complex architectures that do not transfer easily to different
datasets without architectural modifications or hyperparameter tuning. While most existing methods tackle this problem in feature space by engineering representations that are robust to missing inputs, we instead operate in weight space. We propose LARGO, a hypernetwork that compresses the $2^N-1$ dedicated missing-modality models into a single network by modelling the convolutional weights using the Canonical Polyadic (CP) tensor decomposition. Extensive experimental validation on BraTS 2018 (4 modalities, 15 scenarios) and ISLES 2022 (3 modalities, 7 scenarios) shows that our method ranks first in 47 out of 52 configurations, achieving average Dice improvements of +0.68$\%$ and +2.53$\%$ over state-of-the-art baselines (mmFormer, M$^{3}$AE, ShaSpec, SimMLM). A proof-of-concept experiment on avMNIST suggests that LARGO may extend beyond medical imaging to heterogeneous non-medical modalities.
Read the full abstract ▾
Computer Vision
AMIEOD: Adaptive Multi-Experts Image Enhancement for Object Detection in Low-Illumination Scenes
Xiaochen Huang, Honggang Chen, Weicheng Zhang et al. (+4 more)
In multimedia application scenarios, images captured under low-illumination conditions often lead to lower accuracy in visual perception tasks compared to those taken in well-lit environments. To tackle this challenge, we propose AMIEOD, an image enhancement-enabled object detection framework for low-illumination scenes, where the two tasks are jointly optimized in a detection performance-oriented manner. Specifically, to fully exploit the information in poorly lit images, a Multi-Experts Image Enhancement Module (MEIEM) is proposed, which leverages diverse enhancement strategies. On this basis, aiming to better align the MEIEM with the detection task, we propose a Detection-Guided Regression Loss (DGRL) that utilizes the detection result to decide the regression target. Moreover, to dynamically select the most suitable enhancement strategy from MEIEM during inference, we construct an Expert Selection Module (ESM) guided by the proposed Detection-Guided Cross-
Entropy (DGCE) loss, which formulates the optimization of ESM as a classification task. The improved method is well-matched with current detection
algorithms to improve their performance in dim scenes. Extensive experiments on multiple
datasets demonstrate that the proposed method significantly improves object detection accuracy in low-illumination conditions. Our code has been released at https://github.com/scujayfantasy/AMIEOD
Read the full abstract ▾
All authors Xiaochen Huang, Honggang Chen, Weicheng Zhang, Xiaobo Dai, Yongyi Li, Linbo Qing, Xiaohai He
Computer Vision
Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval
Jun Li, Peifeng Lai, Xuhang Lou et al. (+5 more)
Partially relevant
video retrieval aims to retrieve untrimmed
videos using text queries that describe only partial content. However, the inherent asymmetry between brief queries and rich
video content inevitably introduces uncertainty into the
retrieval process. In this setting, vague queries often induce semantic ambiguity across
videos, a challenge that is further exacerbated by the sparse temporal supervision within
videos, which fails to provide sufficient matching evidence. To address this, we propose Holmes, a hierarchical evidential learning framework that aggregates multi-granular cross-modal evidence to quantify and model uncertainty explicitly. At the inter-
video level, similarity scores are interpreted as evidential support and modeled via a Dirichlet distribution. Based on the proposed three-fold principle, we perform fine-grained query identification, which then guides query-adaptive calibrated learning. At the intra-
video level, to accumulate denser evidence, we formulate a soft query-clip
alignment via flexible optimal transport with an adaptive dustbin, which alleviates sparse temporal supervision while suppressing spurious local responses. Extensive experiments demonstrate that Holmes outperforms state-of-the-art methods. Code is released at https://github.com/lijun2005/ICML26-Holmes.
Read the full abstract ▾
All authors Jun Li, Peifeng Lai, Xuhang Lou, Jinpeng Wang, Yuting Wang, Ke Chen, Yaowei Wang, Shu-Tao Xia
Computer Vision
MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation
Shichao Kan, Xuyang Zhang, Haojie Zhang et al. (+7 more)
Evaluating image captions without references remains challenging because global
embedding similarity often misses fine-grained mismatches such as hallucinated objects, missing attributes, or incorrect relations. We propose MSD-Score, a reference-free metric that models image patch and text
token embeddings as von Mises-Fisher mixtures on the unit hypersphere. Instead of treating each modality as a single point, MSD-Score formulates image-text matching as a multi-scale distributional scoring problem. Semantic discrepancies are quantified via a weighted bi-directional KL divergence and combined with global similarity in a multi-scale framework for both single- and multi-candidate evaluations. Extensive experiments show that MSD-Score achieves state-of-the-art correlation with human judgments among reference-free metrics. Beyond accuracy, its probabilistic formulation yields transparent and decomposable diagnostics of local grounding errors, providing a deterministic complementary signal to holistic similarity metrics and judge-based evaluators.
Read the full abstract ▾
All authors Shichao Kan, Xuyang Zhang, Haojie Zhang, Zhe Zhu, Yigang Cen, Yixiong Liang, Lianlei Shan, Linna Zhang, Zhe Qu, Jiazhi Xia
Computer Vision
Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models
Zhikai Li, Yue Zhao, Edward Zhongwei Zhang et al. (+4 more)
Reinforcement learning from human feedback (RLHF) effectively promotes preference
alignment of text-to-image (T2I)
diffusion models. To improve computational efficiency, direct preference optimization (DPO), which avoids explicit reward modeling, has been widely studied. However, its reliance on binary feedback limits it to coarse-grained modeling on chosen-rejected pairs, resulting in suboptimal optimization. In this paper, we propose ArenaPO, which leverages Arena scores as offline rewards to provide refined feedback, thus achieving efficient and fine-grained optimization without a reward model. This enables ArenaPO to benefit from both the rich rewards of traditional RLHF and the efficiency of DPO. Specifically, we first construct a model Arena in which each model's capability is represented as a Gaussian distribution, and infer these capabilities by traversing the annotated pairwise preferences. Each output image is treated as a sample from the corresponding capability distribution. Then, for a image pair, conditioned on the two capability distributions and the observed pairwise preference, the absolute quality gap is estimated using latent-variable inference based on truncated normal distribution, which serves as fine-grained feedback during training. It does not require a reward model and can be computed offline, thus introducing no additional training overhead. We conduct ArenaPO training on Pick-a-Pic v2 and HPD v3
datasets, showing that ArenaPO consistently outperforms existing baselines.
Read the full abstract ▾
All authors Zhikai Li, Yue Zhao, Edward Zhongwei Zhang, Xuewen Liu, Jing Zhang, Qingyi Gu, Zhen Dong
Computer Vision
PersonaGesture: Single-Reference Co-Speech Gesture Personalization for Unseen Speakers
Xiangyue Zhang, Yiyi Cai, Kunhang Li et al. (+6 more)
We propose PersonaGesture, a
diffusion-based pipeline for single-reference co-
speech gesture personalization of unseen speakers. Given target
speech and one motion clip from a new speaker, the model must synthesize gestures that follow the new utterance while retaining speaker-specific pose choices, without per-speaker optimization. This setting is useful for avatars and virtual
agents, but it is hard because the reference mixes stable speaker habits with utterance-specific trajectories. PersonaGesture consists of two key components, Adaptive Style Infusion (ASI) and Implicit Distribution Rectification (IDR), to separate temporal identity evidence from residual statistic correction. A Style Perceiver first encodes the variable-length reference into compact speaker-
memory tokens. ASI injects these
tokens into denoising through zero-initialized residual cross-
attention, enabling style evidence to affect motion formation without replacing the pretrained
speech-to-motion prior. Building on this, IDR applies a length-aware diagonal affine map in latent space to correct residual channel-wise moments estimated from the same reference. Across BEAT2 and ZeroEGGS, we evaluate quantitative metrics, reference-identity controls, same-audio diagnostics, qualitative comparisons, and human preference. Experiments show that separating denoising-time speaker
memory from conservative post-generation moment correction improves unseen-speaker personalization over collapsed style codes, full-reference
attention, and one-clip finetuning. Project: https://xiangyue-zhang.github.io/PersonaGesture.
Read the full abstract ▾
All authors Xiangyue Zhang, Yiyi Cai, Kunhang Li, Kaixing Yang, You Zhou, Zhengqing Li, Xuangeng Chu, Jiaxu Zhang, Haiyang Liu
Computer Vision · Cross-listed from cs.LG
Towards Self-Explainable Document Visual Question Answering with Chain-of-Explanation Predictions
Kjetil Indrehus, Adrian Duric, Changkyu Choi et al. (+1 more)
Document Visual Question Answering (DocVQA) requires
vision-
language models to reason not only about what information in a document is relevant to a question, but also where the answer is grounded on the page. Existing DocVQA models entangle question-relevant evidence and answer localization and operate largely as black boxes, offering limited means to verify how predictions depend on visual evidence. We propose CoExVQA, a self-explainable DocVQA framework with a grounded
reasoning process through a chain-of-explanation design. CoExVQA first identifies question-relevant evidence, then explicitly localizes the answer region, and finally decodes the answer exclusively from the grounded region. Prediction via CoExVQA's chain-of-explanation enables direct inspection and verification of the
reasoning process across modalities. Empirical results show that restricting decoding to grounded evidence achieves SotA explainable DocVQA performance on PFL-DocVQA, improving ANLS by 12% over the current explainable baselines while providing transparent and verifiable predictions.
Read the full abstract ▾
All authors Kjetil Indrehus, Adrian Duric, Changkyu Choi, Ali Ramezani-Kebrya
Computer Vision
RealCam: Real-Time Novel-View Video Generation with Interactive Camera Control
Youcan Xu, Jiaxin Shi, Zhen Wang et al. (+5 more)
Camera-controlled
video-to-
video (V2V) generation enables dynamic viewpoint synthesis from
monocular footage, holding immense potential for interactive filmmaking and live broadcasting. However, existing implicit synthesis methods fundamentally rely on non-causal, full-sequence processing and rigid prefix-style temporal concatenation. This architectural paradigm mandates bidirectional
attention, resulting in prohibitive computational latency, quadratic complexity scaling, and inherent incompatibility with real-time streaming or variable-length inputs. To overcome these limitations, we introduce RealCam, a novel autoregressive framework for interactive, real-time camera-controlled V2V generation. We first design a high-fidelity teacher model grounded in a Cross-frame In-context Learning paradigm. By interleaving source and target frames into synchronized contextual pairs, our design inherently enables length-agnostic generalization and naturally facilitates causal adaptation, breaking the rigid prefix bottleneck. We then distill this teacher into a few-step causal student via Self-Forcing with Distribution Matching
Distillation, enabling efficient, on-the-fly streaming synthesis. Furthermore, to mitigate severe loop inconsistency in closed-loop trajectories, we propose Loop-Closed Data Augmentation (LoopAug), a novel paradigm that synthesizes globally consistent loop sequences from existing multiview
datasets. Extensive experiments demonstrate that RealCam achieves state-of-the-art visual fidelity and temporal consistency while enabling truly interactive camera control with orders-of-magnitude faster inference than existing paradigms. Our project page is at https://xyc-fly.github.io/RealCam/.
Read the full abstract ▾
All authors Youcan Xu, Jiaxin Shi, Zhen Wang, Wensong Song, Feifei Shao, Chen Liang, Jun Xiao, Long Chen
Computer Vision
Fusion in Your Way: Aligning Image Fusion with Heterogeneous Demands via Direct Preference Optimization
Weijian Su, Songqian Zhang, Yuqi Han et al. (+3 more)
As a key technique in multi-modal processing, infrared and visible image fusion (IVIF) plays a crucial role in integrating complementary spectral information for visual enhancement and downstream
vision tasks. Despite remarkable progress, existing methods struggle to flexibly accommodate heterogeneous demands. Achieving adaptive fusion that aligns with various preferences from both human and machine
vision remains an open and challenging problem. To address this challenge, we propose DPOFusion, a direct preference optimization (DPO) framework integrating the property-aligned latent
diffusion model (PALDM) and the preference-controllable latent
diffusion model (PCLDM), enabling task-guided, preference-adaptive IVIF for both human and machine
vision. The PALDM leverages a latent fusion prior and a joint conditional loss to generate diverse candidate fusion results with various properties. PCLDM is subsequently fine-tuned via instance direct preference optimization (IDPO), enabling direct control of the final fusion results with heterogeneous preference signals. Experimental results demonstrate that our framework not only attains precise preference
alignment among humans,
vision-
language models, and task-driven networks, but also sets a new
benchmark for adaptive fusion quality and task-oriented transferability.
Read the full abstract ▾
All authors Weijian Su, Songqian Zhang, Yuqi Han, Jian Zhuang, Yongdong Huang, Qiang Zhang