Computer Vision
Composition-Incremental Learning for Compositional Generalization
Zhen Li, Yuwei Wu, Chenchen Jing et al. (+3 more)
Compositional generalization has achieved substantial progress in computer
vision on pre-collected training data. Nonetheless, real-world data continually emerges, with possible compositions being nearly infinite, long-tailed, and not entirely visible. Thus, an ideal model is supposed to gradually improve the capability of compositional generalization in an incremental manner. In this paper, we explore Composition-Incremental Learning for Compositional Generalization (CompIL) in the context of the compositional zero-shot learning (CZSL) task, where models need to continually learn new compositions, intending to improve their compositional generalization capability progressively. To quantitatively evaluate CompIL, we develop a
benchmark construction pipeline leveraging existing
datasets, yielding MIT-States-CompIL and C-GQA-CompIL. Furthermore, we propose a pseudo-replay framework utilizing a visual synthesizer to synthesize visual representations of learned compositions and a linguistic primitive
distillation mechanism to maintain aligned primitive representations across the learning process. Extensive experiments demonstrate the effectiveness of the proposed framework.
Read the full abstract ▾
All authors Zhen Li, Yuwei Wu, Chenchen Jing, Che Sun, Chuanhao Li, Yunde Jia
Computer Vision · Cross-listed from cs.RO
SMF-VO: Direct Ego-Motion Estimation via Sparse Motion Fields
Sangheon Yang, Yeongin Yoon, Hong Mo Jung et al. (+1 more)
Traditional Visual Odometry (VO) and Visual Inertial Odometry (VIO) methods rely on a 'pose-centric' paradigm, which computes absolute camera poses from the local map thus requires large-scale landmark maintenance and continuous map optimization. This approach is computationally expensive, limiting their real-time performance on resource-constrained devices. To overcome these limitations, we introduce Sparse Motion Field Visual Odometry (SMF-VO), a lightweight, 'motion-centric' framework. Our approach directly estimates instantaneous linear and angular velocity from sparse optical flow, bypassing the need for explicit pose estimation or expensive landmark tracking. We also employed a generalized 3D ray-based motion field formulation that works accurately with various camera models, including wide-field-of-view lenses. SMF-VO demonstrates superior efficiency and competitive accuracy on
benchmark datasets, achieving over 100 FPS on a Raspberry Pi 5 using only a CPU. Our work establishes a scalable and efficient alternative to conventional methods, making it highly suitable for mobile robotics and wearable devices.
Read the full abstract ▾
All authors Sangheon Yang, Yeongin Yoon, Hong Mo Jung, Jongwoo Lim
Computer Vision
Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference
Chengze Jiang, Minjing Dong, Xinli Shi et al. (+1 more)
Vision-
language pre-training models (VLPs) demonstrate strong
multimodal understanding and zero-shot generalization, yet remain vulnerable to
adversarial examples, raising concerns about their reliability. Recent work, Test-Time Counterattack (TTC), improves robustness by generating perturbations that maximize the
embedding deviation of
adversarial inputs using PGD, pushing them away from their
adversarial representations. However, due to the fundamental difference in optimization objectives between
adversarial attacks and counterattacks, generating counterattacks solely based on gradients with respect to the
adversarial input confines the search to a narrow space. As a result, the counterattacks could overfit limited
adversarial patterns and lack the diversity to fully neutralize a broad range of perturbations. In this work, we argue that enhancing the diversity and coverage of counterattacks is crucial to improving
adversarial robustness in test-time defense. Accordingly, we propose Directional Orthogonal Counterattack (DOC), which augments counterattack optimization by incorporating orthogonal gradient directions and momentum-based updates. This design expands the exploration of the counterattack space and increases the diversity of perturbations, which facilitates the discovery of more generalizable counterattacks and ultimately improves the ability to neutralize
adversarial perturbations. Meanwhile, we present a directional sensitivity score based on averaged cosine similarity to boost DOC by improving example discrimination and adaptively modulating the counterattack strength. Extensive experiments on 16
datasets demonstrate that DOC improves
adversarial robustness under various attacks while maintaining competitive clean accuracy. Code is available at https://github.com/bookman233/DOC.
Read the full abstract ▾
All authors Chengze Jiang, Minjing Dong, Xinli Shi, Jie Gui
Computer Vision
VietMEAgent: Culturally-Aware Few-Shot Multimodal Explanation for Vietnamese Visual Question Answering
Hai-Dang Nguyen, Minh-Anh Dang, Minh-Tan Le et al. (+1 more)
Contemporary Visual Question Answering (VQA) systems remain constrained when confronted with culturally specific content, largely because cultural knowledge is under-represented in training corpora and the
reasoning process is not rendered interpretable to end users. This paper introduces VietMEAgent, a
multimodal explainable framework engineered for Vietnamese cultural understanding. The method integrates a cultural object detection backbone with a structured program generation layer, yielding a pipeline in which answer prediction and explanation are tightly coupled. A curated knowledge base of Vietnamese cultural entities serves as an explicit source of background information, while a dual-modality explanation module combines
attention-based visual evidence with structured, human-readable textual rationales. We further construct a Vietnamese Cultural VQA
dataset sourced from public repositories and use it to demonstrate the practicality of programming-based methodologies for cultural AI. The resulting system provides transparent explanations that disclose both the computational rationale and the underlying cultural context, supporting education and cultural preservation with an emphasis on interpretability and cultural sensitivity.
Read the full abstract ▾
All authors Hai-Dang Nguyen, Minh-Anh Dang, Minh-Tan Le, Minh-Tuan Le
Computer Vision
PAN: A World Model for General, Actionable, and Long-Horizon World Simulation
PAN Team Institute of Foundation Models: Zihan Liu, Yi Gu, Mingkai Deng et al. (+11 more)
A world model is a cognitive simulator of the real-world environment allowing biological
agents to reason about how the world evolves, whether spontaneously or in response to their actions, and accordingly to plan and strategize. In building Artificial Intelligence (AI) systems, world models represent the next frontier beyond large
language models (LLMs) to enable physical and
embodied intelligence in AI
agents, allowing them to perform decision-making through simulative
reasoning and reinforcement-learning through simulative trials. Recent advancements in world modeling have yielded impressive progress in
video generation, 3-D scene
evolution, robotic dynamics, and game simulation, but limitations persist in general, open-domain, action-driven prediction, long-horizon consistency, and abstract
reasoning and planning. Moreover, fundamental architectural questions, whether it be state representation, information flow, or training objectives, remain unresolved. In this paper, we introduce PAN, a world model built on the Generative Latent Prediction (GLP) architecture. GLP combines stateful latent representations of world states; an encoder–decoder closed-loop information flow; an LLM/
diffusion-based mixed
reasoning backbone; and a non-degenerate generative reconstruction objective whose fidelity is “dampable” to balance fine-grained detail against semantic saliency. Compared to several existing systems, PAN demonstrates advantages beyond standard
video generation in action-conditioned world simulation, long-horizon
forecasting, and simulative
reasoning and planning, capabilities we argue should serve as the primary criteria for evaluating world models.
Read the full abstract ▾
All authors PAN Team Institute of Foundation Models: Zihan Liu, Yi Gu, Mingkai Deng, Guangyi Liu, Zeyu Feng, Qiyue Gao, Yiyan Hu, Benhao Huang, Yichi Yang, Kun Zhou, Jiannan Xiang, Zhiting Hu, Zhengzhong Liu, Eric P. Xing
Computer Vision
4KDehazeFlow: Ultra-High-Definition Image Dehazing via Flow Matching
Xingchi Chen, Pu Wang, Xuerui Li et al. (+7 more)
Ultra-High-Definition (UHD) image dehazing faces challenges such as limited scene adaptability in prior-based methods and high computational complexity with color distortion in deep learning approaches. To address these issues, we propose 4KDehazeFlow, a novel method based on Flow Matching and the Haze-Aware vector field. This method models the dehazing process as a progressive optimization of continuous vector field flow, providing efficient data-driven adaptive nonlinear color transformation for high-quality dehazing. Specifically, our method has the following advantages: 1) 4KDehazeFlow is a general method compatible with various deep learning networks, without relying on any specific network architecture. 2) We propose a learnable 3D lookup table (LUT) that encodes haze transformation parameters into a compact 3D mapping
matrix, enabling efficient inference through precomputed mappings. 3) We utilize a fourth-order Runge-Kutta (RK4) ordinary differential equation (ODE) solver to stably solve the dehazing flow field through an accurate step-by-step iterative method, effectively suppressing artifacts. Extensive experiments show that 4KDehazeFlow exceeds seven state-of-the-art methods. It delivers a 2dB PSNR increase and better performance in dense haze and color fidelity.
Read the full abstract ▾
All authors Xingchi Chen, Pu Wang, Xuerui Li, Chaopeng Li, Juxiang Zhou, Jianhou Gan, Dianjie Lu, Guijuan Zhang, Wenqi Ren, Zhuoran Zheng
Computer Vision
USF-Net: A Unified Spatiotemporal Fusion Network for Ground-Based Remote Sensing Cloud Image Sequence Extrapolation
Penghui Niu, Taotao Cai, Suqi Zhang et al. (+4 more)
Ground-based remote sensing cloud image sequence extrapolation is a key research area in the development of photovoltaic power systems. However, existing approaches exhibit several limitations:(1)they primarily rely on static kernels to augment feature information, lacking adaptive mechanisms to extract features at varying resolutions dynamically;(2)temporal guidance is insufficient, leading to suboptimal modeling of long-range spatiotemporal dependencies; and(3)the quadratic computational cost of
attention mechanisms is often overlooked, limiting efficiency in practical deployment. To address these challenges, we propose USF-Net, a Unified Spatiotemporal Fusion Network that integrates adaptive large-kernel convolutions and a low-complexity
attention mechanism, combining temporal flow information within an encoder-decoder framework. Specifically, the encoder employs three basic layers to extract features. Followed by the USTM, which comprises:(1)a SiB equipped with a SSM that dynamically captures multi-scale contextual information, and(2)a TiB featuring a TAM that effectively models long-range temporal dependencies while maintaining computational efficiency. In addition, a DSM with a TGM is introduced to enable unified modeling of temporally guided spatiotemporal dependencies. On the decoder side, a DUM is employed to address the common "ghosting effect." It utilizes the initial temporal state as an
attention operator to preserve critical motion signatures. As a key contribution, we also introduce and release the ASI-CIS
dataset. Extensive experiments on ASI-CIS demonstrate that USF-Net significantly outperforms state-of-the-art methods, establishing a superior balance between prediction accuracy and computational efficiency for ground-based cloud extrapolation. The
dataset and source code will be available at https://github.com/she1110/ASI-CIS.
Read the full abstract ▾
All authors Penghui Niu, Taotao Cai, Suqi Zhang, Junhua Gua, Ping Zhanga, Qiqi Liu, Jianxin Li
Computer Vision
Dense Cross-Scale Image Alignment With Fully Spatial Correlation and Just Noticeable Difference Guidance
Jinkun You, Jiaxue Li, Jie Zhang et al. (+1 more)
Existing unsupervised image
alignment methods exhibit limited accuracy and high computational complexity. To address these challenges, we propose a dense cross-scale image
alignment model. It takes into account the correlations between cross-scale features to decrease the
alignment difficulty. Our model supports flexible trade-offs between accuracy and efficiency by adjusting the number of scales utilized. Additionally, we introduce a fully spatial correlation module to further improve accuracy while maintaining low computational costs. We incorporate the just noticeable difference to encourage our model to focus on image regions more sensitive to distortions, eliminating noticeable
alignment errors. Extensive quantitative and qualitative experiments demonstrate that our method surpasses state-of-the-art approaches.
Read the full abstract ▾
All authors Jinkun You, Jiaxue Li, Jie Zhang, Yicong Zhou
Computer Vision · Cross-listed from eess.SP
RadHARSimulator V2: Video to Doppler Generator
Weicheng Gao
Radar-based human activity recognition (HAR) still lacks a comprehensive simulation method. Existing software is developed based on models or motion-captured data, resulting in limited flexibility. To address this issue, a simulator that directly generates Doppler spectra from recorded
video footage (RadHARSimulator V2) is presented in this paper. Both computer
vision and radar modules are included in the simulator. In computer
vision module, the real-time model for object detection with global nearest neighbor is first used to detect and track human targets in the
video. Then, the high-resolution network is used to estimate two-dimensional poses of the detected human targets. Next, the three-dimensional poses of the detected human targets are obtained by nearest matching method. Finally, smooth temporal three-dimensional pose estimation is achieved through Kalman filtering. In radar module, pose interpolation and smoothing are first achieved through the Savitzky-Golay method. Second, the delay model and the mirror method are used to simulate echoes in both free-space and through-the-wall scenarios. Then, range-time map is generated using pulse compression, moving target indication, and DnCNN. Next, Doppler-time map (DTM) is generated using short-time
Fourier transform and DnCNN again. Finally, the ridge features on the DTM are extracted using the maximum local energy method. In addition, a hybrid parallel-serial neural network architecture is proposed for radar-based HAR. Numerical experiments are conducted and analyzed to demonstrate the effectiveness of the designed simulator and the proposed network model. The open-source code of this work can be found in: https://github.com/JoeyBGOfficial/RadHARSimulatorV2-
Video-to-Doppler-Generator.
Read the full abstract ▾
Computer Vision
Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
Liu Yu, Zhonghao Chen, Ping Kuang et al. (+4 more)
Object
hallucination remains a critical challenge in Large
Vision-
Language Models (LVLMs), where models generate content inconsistent with visual inputs. Existing
language-decoder based mitigation approaches often regulate visual or textual
attention independently, overlooking their interaction as two key causal factors. To address this, we propose Owl (Bi-mOdal
attention reWeighting for Layer-wise
hallucination mitigation), a causally-grounded framework that models
hallucination process via a structural causal graph, treating decomposed visual and textual attentions as mediators. We introduce VTACR (Visual-to-Textual
Attention Contribution Ratio), a novel metric that quantifies the modality contribution imbalance during decoding. Our analysis reveals that
hallucinations frequently occur in low-VTACR scenarios, where textual priors dominate and visual grounding is weakened. To mitigate this, we design a fine-grained
attention intervention mechanism that dynamically adjusts
token- and layer-wise
attention guided by VTACR signals. Finally, we propose a dual-path contrastive decoding strategy: one path emphasizes visually grounded predictions, while the other amplifies hallucinated ones – letting visual truth shine and
hallucination collapse. Experimental results on the POPE and CHAIR
benchmarks show that Owl achieves significant
hallucination reduction, setting a new SOTA in faithfulness while preserving
vision-
language understanding capability. Our code is available at https://github.com/CikZ2023/OWL
Read the full abstract ▾
All authors Liu Yu, Zhonghao Chen, Ping Kuang, Zhikun Feng, Fan Zhou, Lan Wang, Gillian Dobbie
Computer Vision · Cross-listed from cs.RO
UniMM-V2X: MoE-Enhanced Multi-Level Fusion for End-to-End Cooperative Autonomous Driving
Ziyi Song, Chen Xia, Chenbing Wang et al. (+3 more)
Autonomous driving holds transformative potential but remains fundamentally constrained by the limited perception and isolated decision-making with standalone intelligence. While recent multi-
agent approaches introduce cooperation, they often focus merely on perception-level tasks, overlooking the
alignment with downstream planning and control, or fall short in leveraging the full capacity of the recent emerging end-to-end autonomous driving. In this paper, we present UniMM-V2X, a novel end-to-end multi-
agent framework that enables hierarchical cooperation across perception, prediction, and planning. At the core of our framework is a multi-level fusion strategy that unifies perception and prediction cooperation, allowing
agents to share queries and reason cooperatively for consistent and safe decision-making. To adapt to diverse downstream tasks and further enhance the quality of multi-level fusion, we incorporate a Mixture-of-Experts (MoE) architecture to dynamically enhance the BEV representations. We further extend MoE into the decoder to better capture diverse motion patterns. Extensive experiments on the DAIR-V2X
dataset demonstrate our approach achieves state-of-the-art (SOTA) performance with a 39.7% improvement in perception accuracy, a 7.2% reduction in prediction error, and a 33.2% improvement in planning performance compared with UniV2X, showcasing the strength of our MoE-enhanced multi-level cooperative paradigm.
Read the full abstract ▾
All authors Ziyi Song, Chen Xia, Chenbing Wang, Haibao Yu, Sheng Zhou, Zhisheng Niu
Computer Vision
T-Rex-Omni: Integrating Negative Visual Prompt in Generic Object Detection
Jiazhou Zhou, Qing Jiang, Kanghao Chen et al. (+4 more)
Object detection methods have evolved from closed-set to open-set paradigms over the years. Current open-set object detectors, however, remain constrained by their exclusive reliance on positive indicators based on given
prompts like text descriptions or visual exemplars. This positive-only paradigm experiences consistent vulnerability to visually similar but semantically different distractors. We propose T-Rex-Omni, a novel framework that addresses this limitation by incorporating negative visual
prompts to negate hard negative distractors. Specifically, we first introduce a unified visual
prompt encoder that jointly processes positive and negative visual
prompts. Next, a training-free Negating Negative Computing (NNC) module is proposed to dynamically suppress negative responses during the probability computing stage. To further boost performance through fine-tuning, our Negating Negative Hinge (NNH) loss enforces discriminative margins between positive and negative
embeddings. T-Rex-Omni supports flexible deployment in both positive-only and joint positive-negative inference modes, accommodating either user-specified or automatically generated negative examples. Extensive experiments demonstrate remarkable zero-shot detection performance, significantly narrowing the performance gap between visual-prompted and text-prompted methods while showing particular strength in long-tailed scenarios (51.2 AP_r on LVIS-minival). This work establishes negative
prompts as a crucial new dimension for advancing open-set visual recognition systems.
Read the full abstract ▾
All authors Jiazhou Zhou, Qing Jiang, Kanghao Chen, Lutao Jiang, Yuanhuiyi Lyu, Ying-Cong Chen, Lei Zhang
Computer Vision · Cross-listed from cs.LG
Fast $k$-means clustering in Riemannian manifolds via Fréchet maps: Applications to large-dimensional SPD matrices
Ji Shi, Nicolas Charon, Andreas Mang et al. (+2 more)
We introduce a novel, efficient framework for clustering data on high-dimensional, non-Euclidean
manifolds that overcomes the computational challenges associated with standard intrinsic methods. The key innovation is the use of the $p$-Fréchet map $F^p : \mathcal{M} \to \mathbb{R}^\ell$ – defined on a generic metric space $\mathcal{M}$ – which embeds the
manifold data into a lower-dimensional Euclidean space $\mathbb{R}^\ell$ using a set of reference points $\{r_i\}_{i=1}^\ell$, $r_i \in \mathcal{M}$. Once embedded, we can efficiently and accurately apply standard Euclidean clustering techniques such as k-means. We rigorously analyze the mathematical properties of $F^p$ in the Euclidean space and the challenging
manifold of $n \times n$ symmetric positive definite
matrices $\mathit{SPD}(n)$. Extensive numerical experiments using synthetic and real $\mathit{SPD}(n)$ data demonstrate significant performance gains: our method reduces runtime by up to two orders of magnitude compared to intrinsic
manifold-based approaches, all while maintaining high clustering accuracy, including scenarios where existing alternative methods struggle or fail.
Read the full abstract ▾
All authors Ji Shi, Nicolas Charon, Andreas Mang, Demetrio Labate, Robert Azencott
Computer Vision
An ICTM-RMSAV Framework for Bias-Field Aware Image Segmentation under Poisson and Multiplicative Noise
Xinyu Wang, Wenjun Yao, Fanghui Song et al. (+1 more)
Image
segmentation is a core task in image processing, yet many methods degrade when images are heavily corrupted by noise and exhibit intensity inhomogeneity. Within the iterative-convolution thresholding method (ICTM) framework, we propose a variational
segmentation model that integrates denoising terms. Specifically, the denoising component consists of an I-divergence term and an adaptive total-variation (TV) regularizer, making the model well suited to images contaminated by Gamma–distributed multiplicative noise and Poisson noise. A spatially adaptive weight derived from a gray-level indicator guides
diffusion differently across regions of varying intensity. To further address intensity inhomogeneity, we estimate a smoothly varying bias field, which improves
segmentation accuracy. Regions are represented by characteristic functions, with contour length encoded accordingly. For efficient optimization, we couple ICTM with a relaxed modified scalar auxiliary variable (RMSAV) scheme. Extensive experiments on synthetic and real-world images with intensity inhomogeneity and diverse noise types show that the proposed model achieves superior accuracy and robustness compared with competing approaches.
Read the full abstract ▾
All authors Xinyu Wang, Wenjun Yao, Fanghui Song, Zhichang Guo
Computer Vision
WDT-MD: Wavelet Diffusion Transformers for Microaneurysm Detection in Fundus Images
Yifei Sun, Yuzhi He, Junhao Jia et al. (+4 more)
Microaneurysms (MAs), the earliest pathognomonic signs of Diabetic Retinopathy (DR), present as sub-60 $\mu m$ lesions in fundus images with highly variable photometric and morphological characteristics, rendering manual screening not only labor-intensive but inherently error-prone. While
diffusion-based anomaly detection has emerged as a promising approach for automated MA screening, its clinical application is hindered by three fundamental limitations. First, these models often fall prey to "identity mapping", where they inadvertently replicate the input image. Second, they struggle to distinguish MAs from other anomalies, leading to high false positives. Third, their suboptimal reconstruction of normal features hampers overall performance. To address these challenges, we propose a Wavelet
Diffusion Transformer framework for MA Detection (WDT-MD), which features three key innovations: a noise-encoded image conditioning mechanism to avoid "identity mapping" by perturbing image conditions during training; pseudo-normal pattern synthesis via inpainting to introduce pixel-level supervision, enabling discrimination between MAs and other anomalies; and a wavelet
diffusion Transformer architecture that combines the global modeling capability of
diffusion Transformers with multi-scale wavelet analysis to enhance reconstruction of normal retinal features. Comprehensive experiments on the IDRiD and e-ophtha MA
datasets demonstrate that WDT-MD outperforms state-of-the-art methods in both pixel-level and image-level MA detection. This advancement holds significant promise for improving early DR screening.
Read the full abstract ▾
All authors Yifei Sun, Yuzhi He, Junhao Jia, Jinhong Wang, Ruiquan Ge, Changmiao Wang, Hongxia Xu
Computer Vision · Cross-listed from cs.GR
A Finite Difference Approximation of Second Order Regularization of Neural-SDFs
Haotian Yin, Aleksander Plocharski, Michal Jan Wlodarczyk et al. (+1 more)
We introduce a finite-difference framework for
curvature regularization in neural signed distance field (SDF) learning. Existing approaches enforce
curvature priors using full Hessian information obtained via second-order automatic differentiation, which is accurate but computationally expensive. Others reduced this overhead by avoiding explicit Hessian assembly, but still required higher-order differentiation. In contrast, our method replaces these operations with lightweight finite-difference stencils that approximate second derivatives using the well known Taylor expansion with a truncation error of O(h^2), and can serve as drop-in replacements for Gaussian
curvature and rank-deficiency losses. Experiments demonstrate that our finite-difference variants achieve reconstruction fidelity comparable to their automatic-differentiation counterparts, while reducing GPU
memory usage and training time by up to a factor of two. Additional tests on sparse, incomplete, and non-CAD data confirm that the proposed formulation is robust and general, offering an efficient and scalable alternative for
curvature-aware SDF learning.
Read the full abstract ▾
All authors Haotian Yin, Aleksander Plocharski, Michal Jan Wlodarczyk, Przemyslaw Musialski
Computer Vision · Cross-listed from cs.MM
Spatio-Temporal Data Enhanced Vision-Language Model for Traffic Scene Understanding
Jingtian Ma, Jingyuan Wang, Wayne Xin Zhao et al. (+2 more)
Nowadays,
navigation and ride-sharing apps have collected numerous images with spatio-temporal data. A core technology for analyzing such images, associated with spatiotemporal information, is Traffic Scene Understanding (TSU), which aims to provide a comprehensive description of the traffic scene. Unlike traditional spatio-temporal data analysis tasks, the dependence on both spatio-temporal and visual-textual data introduces distinct challenges to TSU task. However, recent research often treats TSU as a common image understanding task, ignoring the spatio-temporal information and overlooking the interrelations between different aspects of the traffic scene. To address these issues, we propose a novel SpatioTemporal Enhanced Model based on CILP (ST-CLIP) for TSU. Our model uses the classic
vision-
language model, CLIP, as the backbone, and designs a Spatio-temporal Context Aware Multiaspect
Prompt (SCAMP) learning method to incorporate spatiotemporal information into TSU. The
prompt learning method consists of two components: A dynamic spatio-temporal context representation module that extracts representation vectors of spatio-temporal data for each traffic scene image, and a bi-level ST-aware multi-aspect
prompt learning module that integrates the ST-context representation vectors into word
embeddings of
prompts for the CLIP model. The second module also extracts low-level visual features and image-wise high-level semantic features to exploit interactive relations among different aspects of traffic scenes. To the best of our knowledge, this is the first attempt to integrate spatio-temporal information into visionlanguage models to facilitate TSU task. Experiments on two realworld
datasets demonstrate superior performance in the complex scene understanding scenarios with a few-shot learning strategy.
Read the full abstract ▾
All authors Jingtian Ma, Jingyuan Wang, Wayne Xin Zhao, Guoping Liu, Xiang Wen
Computer Vision
Efficient and Effective In-context Demonstration Selection with Coreset
Zihua Wang, Jiarui Wang, Haiyang Xu et al. (+6 more)
In-context learning (ICL) has emerged as a powerful paradigm for Large Visual
Language Models (LVLMs), enabling them to leverage a few examples directly from input contexts. However, the effectiveness of this approach is heavily reliant on the selection of demonstrations, a process that is NP-hard. Traditional strategies, including random, similarity-based sampling and infoscore-based sampling, often lead to inefficiencies or suboptimal performance, struggling to balance both efficiency and effectiveness in demonstration selection. In this paper, we propose a novel demonstration selection framework named Coreset-based Dual
Retrieval (CoDR). We show that samples within a diverse subset achieve a higher expected mutual information. To implement this, we introduce a cluster-
pruning method to construct a diverse coreset that aligns more effectively with the query while maintaining diversity. Additionally, we develop a dual
retrieval mechanism that enhances the selection process by achieving global demonstration selection while preserving efficiency. Experimental results demonstrate that our method significantly improves the ICL performance compared to the existing strategies, providing a robust solution for effective and efficient demonstration selection.
Read the full abstract ▾
All authors Zihua Wang, Jiarui Wang, Haiyang Xu, Ming Yan, Fei Huang, Xu Yang, Xiu-Shen Wei, Siya Mi, Yu Zhang
Computer Vision · Cross-listed from cs.HC
Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent Disambiguation
Sicheng Yang, Yukai Huang, Weitong Cai et al. (+6 more)
The performance of
egocentric AI
agents is fundamentally limited by
multimodal intent ambiguity. This challenge arises from a combination of underspecified
language, imperfect visual data, and deictic gestures, which frequently leads to task failure. Existing monolithic
Vision-
Language Models (VLMs) struggle to resolve these
multimodal ambiguous inputs, often failing silently or hallucinating responses. To address these ambiguities, we introduce the Plug-and-Play Clarifier, a zero-shot and modular framework that decomposes the problem into discrete, solvable sub-tasks. Specifically, our framework consists of three synergistic modules: (1) a text clarifier that uses dialogue-driven
reasoning to interactively disambiguate linguistic intent, (2) a
vision clarifier that delivers real-time guidance feedback, instructing users to adjust their positioning for improved capture quality, and (3) a cross-modal clarifier with grounding mechanism that robustly interprets 3D pointing gestures and identifies the specific objects users are pointing to. Extensive experiments demonstrate that our framework improves the intent clarification performance of small
language models (4–8B) by approximately 30%, making them competitive with significantly larger counterparts. We also observe consistent gains when applying our framework to these larger models. Furthermore, our
vision clarifier increases corrective guidance accuracy by over 20%, and our cross-modal clarifier improves semantic answer accuracy for referential grounding by 5%. Overall, our method provides a plug-and-play framework that effectively resolves
multimodal ambiguity and significantly enhances user experience in
egocentric interaction.
Read the full abstract ▾
All authors Sicheng Yang, Yukai Huang, Weitong Cai, Shitong Sun, You He, Jiankang Deng, Hang Zhang, Jifei Song, Zhensong Zhang
Computer Vision
AuthSig: Safeguarding Scanned Signatures Against Unauthorized Reuse in Paperless Workflows
RuiQiang Zhang, Zehua Ma, Guanjie Wang et al. (+3 more)
With the deepening trend of paperless workflows, signatures as a means of identity authentication are gradually shifting from traditional ink-on-paper to electronic formats.Despite the availability of dynamic pressure-sensitive and PKI-based digital signatures, static scanned signatures remain prevalent in practice due to their convenience. However, these static images, having almost lost their authentication attributes, cannot be reliably verified and are vulnerable to malicious copying and reuse. To address these issues, we propose AuthSig, a novel static electronic signature framework based on generative models and watermark, which binds authentication information to the signature image. Leveraging the human visual system's insensitivity to subtle style variations, AuthSig finely modulates style
embeddings during generation to implicitly encode watermark bits-enforcing a One Signature, One Use policy.To overcome the scarcity of handwritten signature data and the limitations of traditional augmentation methods, we introduce a keypoint-driven data augmentation strategy that effectively enhances style diversity to support robust watermark
embedding. Experimental results show that AuthSig achieves over 98% extraction accuracy under both digital-domain distortions and signature-specific degradations, and remains effective even in print-scan scenarios.
Read the full abstract ▾
All authors RuiQiang Zhang, Zehua Ma, Guanjie Wang, Chang Liu, Hengyi Wang, Weiming Zhang
Computer Vision · Cross-listed from cond-mat.mtrl-sci
MicroEvoEval: A Systematic Evaluation Framework for Image-Based Microstructure Evolution Prediction
Qinyi Zhang, Duanyu Feng, Ronghui Han et al. (+2 more)
Simulating microstructure
evolution (MicroEvo) is vital for materials design but demands high numerical accuracy, efficiency, and physical fidelity. Although recent studies on deep learning (DL) offer a promising alternative to traditional solvers, the field lacks standardized
benchmarks. Existing studies are flawed due to a lack of comparing specialized MicroEvo DL models with state-of-the-art spatio-temporal architectures, an overemphasis on numerical accuracy over physical fidelity, and a failure to analyze error propagation over time. To address these gaps, we introduce MicroEvoEval, the first comprehensive
benchmark for image-based microstructure
evolution prediction. We evaluate 14 models, encompassing both domain-specific and general-purpose architectures, across four representative MicroEvo tasks with
datasets specifically structured for both short- and long-term assessment. Our multi-faceted evaluation framework goes beyond numerical accuracy and computational cost, incorporating a curated set of structure-preserving metrics to assess physical fidelity. Our extensive evaluations yield several key insights. Notably, we find that modern architectures (e.g., VMamba), not only achieve superior long-term stability and physical fidelity but also operate with an order-of-magnitude greater computational efficiency. The results highlight the necessity of holistic evaluation and identify these modern architectures as a highly promising direction for developing efficient and reliable surrogate models in data-driven materials science.
Read the full abstract ▾
All authors Qinyi Zhang, Duanyu Feng, Ronghui Han, Yangshuai Wang, Hao Wang
Computer Vision
FGM-HD: Boosting Generation Diversity of Fractal Generative Models through Hausdorff Dimension Induction
Haowei Zhang, Yuanpei Zhao, Ji-Zhe Zhou et al. (+1 more)
Improving the diversity of generated results while maintaining high visual quality remains a significant challenge in image generation tasks. Fractal Generative Models (FGMs) are efficient in generating high-quality images, but their inherent self-similarity limits the diversity of output images. To address this issue, we propose a novel approach based on the Hausdorff Dimension (HD), a widely recognized concept in fractal geometry used to quantify structural complexity, which aids in enhancing the diversity of generated outputs. To incorporate HD into FGM, we propose a learnable HD estimation method that predicts HD directly from image
embeddings, addressing computational cost concerns. However, simply introducing HD into a hybrid loss is insufficient to enhance diversity in FGMs due to: 1) degradation of image quality, and 2) limited improvement in generation diversity. To this end, during training, we adopt an HD-based loss with a monotonic momentum-driven scheduling strategy to progressively optimize the hyperparameters, obtaining optimal diversity without sacrificing visual quality. Moreover, during inference, we employ HD-guided rejection sampling to select geometrically richer outputs. Extensive experiments on the ImageNet
dataset demonstrate that our FGM-HD framework yields a 39% improvement in output diversity compared to vanilla FGMs, while preserving comparable image quality. To our knowledge, this is the very first work introducing HD into FGM. Our method effectively enhances the diversity of generated outputs while offering a principled theoretical contribution to FGM development.
Read the full abstract ▾
All authors Haowei Zhang, Yuanpei Zhao, Ji-Zhe Zhou, Mao Li
Computer Vision
Neural B-frame Video Compression with Bi-directional Reference Harmonization
Yuxi Liu, Dengchao Jin, Shuai Huo et al. (+5 more)
Neural
video compression (NVC) has made significant progress in recent years, while neural B-frame
video compression (NBVC) remains underexplored compared to P-frame compression. NBVC can adopt bi-directional reference frames for better compression performance. However, NBVC's hierarchical coding may complicate continuous temporal prediction, especially at some hierarchical levels with a large frame span, which could cause the contribution of the two reference frames to be unbalanced. To optimize reference information utilization, we propose a novel NBVC method, termed Bi-directional Reference Harmonization
Video Compression (BRHVC), with the proposed Bi-directional Motion Converge (BMC) and Bi-directional Contextual Fusion (BCF). BMC converges multiple optical flows in motion compression, leading to more accurate motion compensation on a larger scale. Then BCF explicitly models the weights of reference contexts under the guidance of motion compensation accuracy. With more efficient motions and contexts, BRHVC can effectively harmonize bi-directional references. Experimental results indicate that our BRHVC outperforms previous state-of-the-art NVC methods, even surpassing the traditional coding, VTM-RA (under random access configuration), on the HEVC
datasets. The source code is released at https://github.com/kwai/NVC.
Read the full abstract ▾
All authors Yuxi Liu, Dengchao Jin, Shuai Huo, Jiawen Gu, Chao Zhou, Huihui Bai, Ming Lu, Zhan Ma
Computer Vision
Boosting Adversarial Transferability via Ensemble Non-Attention
Yipeng Zou, Qin Liu, Jie Wu et al. (+4 more)
Ensemble attacks integrate the outputs of surrogate models with diverse architectures, which can be combined with various gradient-based attacks to improve
adversarial transferability. However, previous work shows unsatisfactory attack performance when transferring across heterogeneous model architectures. The main reason is that the gradient update directions of heterogeneous surrogate models differ widely, making it hard to reduce the gradient variance of ensemble models while making the best of individual model. To tackle this challenge, we design a novel ensemble attack, NAMEA, which for the first time integrates the gradients from the non-
attention areas of ensemble models into the iterative gradient optimization process. Our design is inspired by the observation that the
attention areas of heterogeneous models vary sharply, thus the non-
attention areas of ViTs are likely to be the focus of CNNs and vice versa. Therefore, we merge the gradients respectively from the
attention and non-
attention areas of ensemble models so as to fuse the transfer information of CNNs and ViTs. Specifically, we pioneer a new way of decoupling the gradients of non-
attention areas from those of
attention areas, while merging gradients by meta-learning. Empirical evaluations on ImageNet
dataset indicate that NAMEA outperforms AdaEA and SMER, the state-of-the-art ensemble attacks by an average of 15.0% and 9.6%, respectively. This work is the first attempt to explore the power of ensemble non-
attention in boosting cross-architecture transferability, providing new insights into launching ensemble attacks.
Read the full abstract ▾
All authors Yipeng Zou, Qin Liu, Jie Wu, Yu Peng, Guo Chen, Hui Zhou, Guanghui Ye
Computer Vision · Cross-listed from cs.RO
Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigation
Ningnan Wang, Weihuang Chen, Liming Chen et al. (+4 more)
Embodied visual
navigation remains a challenging task, as
agents must explore unknown environments with limited knowledge. Existing zero-shot studies have shown that incorporating
memory mechanisms to support goal-directed behavior can improve long-horizon planning performance. However, they overlook visual frontier boundaries, which fundamentally dictate future trajectories and observations, and fall short of inferring the relationship between partial visual observations and
navigation goals. In this paper, we propose Semantic Cognition Over Potential-based Exploration (SCOPE), a zero-shot framework that explicitly leverages frontier information to drive potential-based exploration, enabling more informed and goal-relevant decisions. SCOPE estimates exploration potential with a
Vision-
Language Model and organizes it into a spatio-temporal potential graph, capturing boundary dynamics to support long-horizon planning. In addition, SCOPE incorporates a self-reconsideration mechanism that revisits and refines prior decisions, enhancing reliability and reducing overconfident errors. Experimental results on two diverse
embodied navigation tasks show that SCOPE outperforms state-of-the-art baselines by 4.6% in accuracy. Further analysis demonstrates that its core components lead to improved calibration, stronger generalization, and higher decision quality.
Read the full abstract ▾
All authors Ningnan Wang, Weihuang Chen, Liming Chen, Haoxuan Ji, Zhongyu Guo, Xuchong Zhang, Hongbin Sun
Computer Vision
From Structure to Detail: Hierarchical Distillation for Efficient Diffusion Model
Hanbo Cheng, Peng Wang, Kaixiang Lei et al. (+4 more)
The inference latency of
diffusion models remains a critical barrier to their real-time application. While trajectory-based and distribution-based step
distillation methods offer solutions, they present a fundamental trade-off. Trajectory-based methods preserve global structure but act as a "lossy compressor", sacrificing high-frequency details. Conversely, distribution-based methods can achieve higher fidelity but often suffer from mode collapse and unstable training. This paper recasts them from independent paradigms into synergistic components within our novel Hierarchical
Distillation (HD) framework. We leverage trajectory
distillation not as a final generator, but to establish a structural “sketch", providing a near-optimal initialization for the subsequent distribution-based refinement stage. This strategy yields an ideal initial distribution that enhances the ceiling of overall performance. To further improve quality, we introduce and refine the
adversarial training process. We find standard discriminator structures are ineffective at refining an already high-quality generator. To overcome this, we introduce the Adaptive Weighted Discriminator (AWD), tailored for the HD pipeline. By dynamically allocating
token weights, AWD focuses on local imperfections, enabling efficient detail refinement. Our approach demonstrates state-of-the-art performance across diverse tasks. On ImageNet $256\times256$, our single-step model achieves an FID of 2.26, rivaling its 250-step teacher. It also achieves promising results on the high-resolution text-to-image MJHQ
benchmark, proving its generalizability. Our method establishes a robust new paradigm for high-fidelity, single-step
diffusion models.
Read the full abstract ▾
All authors Hanbo Cheng, Peng Wang, Kaixiang Lei, Qi Li, Zhen Zou, Pengfei Hu, Jun Du
Computer Vision · Cross-listed from eess.IV
ROI-based Deep Image Compression with Implicit Bit Allocation
Kai Hu, Han Wang, Renhe Liu et al. (+3 more)
Region of Interest (ROI)-based image compression has rapidly developed due to its ability to maintain high fidelity in important regions while reducing data redundancy. However, existing compression methods primarily apply masks to suppress background information before quantization. This explicit bit allocation strategy, which uses hard gating, significantly impacts the statistical distribution of the
entropy model, thereby limiting the coding performance of the compression model. In response, this work proposes an efficient ROI-based deep image compression model with implicit bit allocation. To better utilize ROI masks for implicit bit allocation, this paper proposes a novel Mask-Guided Feature Enhancement (MGFE) module, comprising a Region-Adaptive
Attention (RAA) block and a Frequency-Spatial Collaborative
Attention (FSCA) block. This module allows for flexible bit allocation across different regions while enhancing global and local features through frequencyspatial domain collaboration. Additionally, we use dual decoders to separately reconstruct foreground and background images, enabling the coding network to optimally balance foreground enhancement and background quality preservation in a datadriven manner. To the best of our knowledge, this is the first work to utilize implicit bit allocation for high-quality regionadaptive coding. Experiments on the COCO2017
dataset show that our implicit-based image compression method significantly outperforms explicit bit allocation approaches in rate-distortion performance, achieving optimal results while maintaining satisfactory visual quality in the reconstructed background regions.
Read the full abstract ▾
All authors Kai Hu, Han Wang, Renhe Liu, Zhilin Li, Shenghui Song, Yu Liu
Computer Vision · Cross-listed from cs.HC
"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models
Kapil Garg, Xinru Tang, Jimin Heo et al. (+4 more)
Vision-
Language Models (VLMs) are increasingly used by blind and low-
vision (BLV) people to identify and understand products in their everyday lives, such as food, personal care items, and household goods. Despite their prevalence, we lack an empirical understanding of how common image quality issues–such as blur, misframing, and rotation–affect the accuracy of VLM-generated captions and whether the resulting captions meet BLV people's information needs. Based on a survey of 86 BLV participants, we develop an annotated
dataset of 1,859 product images from BLV people to systematically evaluate how image quality issues affect VLM-generated captions. While the best VLM achieves 98% accuracy on images with no quality issues, accuracy drops to 75% overall when quality issues are present, worsening considerably as issues compound. We discuss the need for model evaluations that center on disabled people's experiences throughout the process and offer concrete recommendations for HCI and ML researchers to make VLMs more reliable for BLV people.
Read the full abstract ▾
All authors Kapil Garg, Xinru Tang, Jimin Heo, Dwayne R. Morgan, Darren Gergle, Erik B. Sudderth, Anne Marie Piper
Computer Vision
Machines Serve Human: A Novel Variable Human-machine Collaborative Compression Framework
Zifu Zhang, Shengxi Li, Xiancheng Sun et al. (+3 more)
Human-machine collaborative compression has been receiving increasing research efforts for reducing image/
video data, serving as the basis for both human perception and machine intelligence. Existing collaborative methods are dominantly built upon the de facto human-
vision compression pipeline, witnessing deficiency on complexity and bit-rates when aggregating the machine-
vision compression. Indeed, machine
vision solely focuses on the core regions within the image/
video, requiring much less information compared with the compressed information for human
vision. In this paper, we thus set out the first successful attempt by a novel collaborative compression method based on the machine-
vision-oriented compression, instead of human-
vision pipeline. In other words, machine
vision serves as the basis for human
vision within collaborative compression. A plug-and-play variable bit-rate strategy is also developed for machine
vision tasks. Then, we propose to progressively aggregate the semantics from the machine-
vision compression, whilst seamlessly tailing the
diffusion prior to restore high-fidelity details for human
vision, thus named as
diffusion-prior based feature compression for human and machine
visions (Diff-FCHM). Experimental results verify the consistently superior performances of our Diff-FCHM, on both machine-
vision and human-
vision compression with remarkable margins. Our code will be released upon acceptance.
Read the full abstract ▾
All authors Zifu Zhang, Shengxi Li, Xiancheng Sun, Mai Xu, Zhengyuan Liu, Jingyuan Xia
Computer Vision
SPEED-Q: Staged Processing with Enhanced Distillation towards Efficient Low-bit On-device VLM Quantization
Tianyu Guo, Shanwei Zhao, Shiai Zhu et al. (+1 more)
Deploying
Vision-
Language Models (VLMs) on edge devices (e.g., smartphones and
robots) is crucial for enabling low-latency and
privacy-preserving intelligent applications. Given the resource constraints of these devices, quantization offers a promising solution by improving
memory efficiency and reducing bandwidth requirements, thereby facilitating the deployment of VLMs. However, existing research has rarely explored aggressive quantization on VLMs, particularly for the models ranging from 1B to 2B parameters, which are more suitable for resource-constrained edge devices. In this paper, we propose SPEED-Q, a novel Staged Processing with Enhanced
Distillation framework for VLM low-bit weight-only quantization that systematically addresses the following two critical obstacles: (1) significant discrepancies in quantization sensitivity between
vision (ViT) and
language (LLM) components in VLMs; (2) training instability arising from the reduced numerical precision inherent in low-bit quantization. In SPEED-Q, a staged sensitivity adaptive mechanism is introduced to effectively harmonize performance across different modalities. We further propose a
distillation-enhanced quantization strategy to stabilize the training process and reduce data dependence. Together, SPEED-Q enables accurate, stable, and data-efficient quantization of complex VLMs. SPEED-Q is the first framework tailored for quantizing entire small-scale billion-parameter VLMs to low bits. Extensive experiments across multiple
benchmarks demonstrate that SPEED-Q achieves up to 6x higher accuracy than existing quantization methods under 2-bit settings and consistently outperforms prior on-device VLMs under both 2-bit and 4-bit settings. Our code and models are available at https://github.com/antgroup/SPEED-Q.
Read the full abstract ▾
All authors Tianyu Guo, Shanwei Zhao, Shiai Zhu, Chenguang Ma