Computer Vision
Prompt-Free Universal Region Proposal Network
Qihong Tang, Changhan Liu, Shaofeng Zhang et al. (+3 more)
Identifying potential objects is critical for object recognition and analysis across various computer
vision applications. Existing methods typically localize potential objects by relying on exemplar images, predefined categories, or textual descriptions. However, their reliance on image and text
prompts often limits flexibility, restricting adaptability in real-world scenarios. In this paper, we introduce a novel
Prompt-Free Universal Region Proposal Network (PF-RPN), which identifies potential objects without relying on external
prompts. First, the Sparse Image-Aware Adapter (SIA) module performs initial localization of potential objects using a learnable query
embedding dynamically updated with visual features. Next, the Cascade Self-
Prompt (CSP) module identifies the remaining potential objects by leveraging the self-prompted learnable
embedding, autonomously aggregating informative visual features in a cascading manner. Finally, the Centerness-Guided Query Selection (CG-QS) module facilitates the selection of high-quality query
embeddings using a centerness scoring network. Our method can be optimized with limited data (e.g., 5% of MS COCO data) and applied directly to various object detection application domains for identifying potential objects without fine-tuning, such as
underwater object detection, industrial defect detection, and remote sensing image object detection. Experimental results across 19
datasets validate the effectiveness of our method. Code is available at https://github.com/tangqh03/PF-RPN.
Read the full abstract ▾
All authors Qihong Tang, Changhan Liu, Shaofeng Zhang, Wenbin Li, Qi Fan, Yang Gao
Computer Vision · Cross-listed from eess.IV
Deep Learning-Based Airway Segmentation in Systemic Lupus Erythematosus Patients with Interstitial Lung Disease (SLE-ILD): A Comparative High-Resolution CT Analysis
Sirong Piao, Ying Ming, Ruijie Zhao et al. (+11 more)
To characterize lobar and segmental airway volume differences between systemic lupus erythematosus (SLE) patients with interstitial lung disease (ILD) and those without ILD (non-ILD) using a deep learning-based approach on non-contrast chest high-resolution CT (HRCT). Methods: A retrospective analysis was conducted on 106 SLE patients (27 SLE-ILD, 79 SLE-non-ILD) who underwent HRCT. A customized deep learning framework based on the U-Net architecture was developed to automatically segment airway structures at the lobar and segmental levels via HRCT. Volumetric measurements of lung lobes and segments derived from the
segmentations were statistically compared between the two groups using two-sample t-tests (significance threshold: p < 0.05). Results: At lobar level, significant airway volume enlargement in SLE-ILD patients was observed in the right upper lobe (p=0.009) and left upper lobe (p=0.039) compared to SLE-non-ILD. At the segmental level, significant differences were found in segments including R1 (p=0.016), R3 (p<0.001), and L3 (p=0.038), with the most marked changes in the upper lung zones, while lower zones showed non-significant trends. Conclusion: Our study demonstrates that an automated deep learning-based approach can effectively quantify airway volumes on HRCT scans and reveal significant, region-specific airway dilation in patients with SLE-ILD compared to those without ILD. The pattern of involvement, predominantly affecting the upper lobes and specific segments, highlights a distinct topographic phenotype of SLE-ILD and implicates airway structural alterations as a potential biomarker for disease presence. This AI-powered quantitative imaging biomarker holds promise for enhancing the early detection and monitoring of ILD in the SLE population, ultimately contributing to more personalized patient management.
Read the full abstract ▾
All authors Sirong Piao, Ying Ming, Ruijie Zhao, Jiaru Wang, Ran Xiao, Rui Zhao, Zicheng Liao, Qiqi Xu, Shaoze Luo, Bing Li, Lin Li, Zhuangfei Ma, Fuling Zheng, Wei Song
Computer Vision
ProGVC: Progressive-based Generative Video Compression via Auto-Regressive Context Modeling
Daowen Li, Ruixiao Dong, Kai Li et al. (+3 more)
Perceptual
video compression leverages generative priors to reconstruct realistic textures and motions at low bitrates. However, existing perceptual codecs often lack native support for variable bitrate and progressive delivery, and their generative modules are weakly coupled with
entropy coding, limiting bitrate reduction. Inspired by the next-scale prediction in the Visual Auto-Regressive (VAR) models, we propose ProGVC, a Progressive-based Generative
Video Compression framework that unifies progressive transmission, efficient
entropy coding, and detail synthesis within a single codec. ProGVC encodes
videos into hierarchical multi-scale residual
token maps, enabling flexible rate adaptation by transmitting a coarse-to-fine subset of scales in a progressive manner. A
Transformer-based multi-scale autoregressive context model estimates
token probabilities, utilized both for efficient
entropy coding of the transmitted
tokens and for predicting truncated fine-scale
tokens at the decoder to restore perceptual details. Extensive experiments demonstrate that as a new coding paradigm, ProGVC delivers promising perceptual compression performance at low bitrates while offering practical scalability at the same time.
Read the full abstract ▾
All authors Daowen Li, Ruixiao Dong, Kai Li, Ying Chen, Ding Ding, Li Li
Computer Vision
Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models
Linghao Zhang, Jungang Li, Yonghua Hei et al. (+12 more)
Multimodal large
language models (MLLMs) are typically trained in multiple stages, with
video-based supervised fine-tuning (
Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained
evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how
Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern:
Video-SFT reliably improves
video performance, but often yields limited gains or even degradation on static image
benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves
video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-
video trade-off. Our results indicate that
Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-
video training.
Read the full abstract ▾
All authors Linghao Zhang, Jungang Li, Yonghua Hei, Sicheng Tao, Song Dai, Yibo Yan, Zihao Dongfang, Weiting Liu, Chenxi Qin, Hanqian Li, Xin Zou, Jiahao Zhang, Shuhang Xun, Haiyun Jiang, Xuming Hu
Computer Vision
Learning Coordinate-based Convolutional Kernels for Continuous SE(3) Equivariant and Efficient Point Cloud Analysis
Jaein Kim, Hee Bin Yoo, Dong-Sig Han et al. (+1 more)
A
symmetry on rigid motion is one of the salient factors in efficient learning of 3D point cloud problems. Group convolution has been a representative method to extract equivariant features, but its realizations have struggled to retain both rigorous
symmetry and scalability simultaneously. We advocate utilizing the intertwiner framework to resolve this trade-off, but previous works on it, which did not achieve complete SE(3)
symmetry or scalability to large-scale problems, necessitate a more advanced kernel architecture. We present Equivariant Coordinate-based Kernel Convolution, or ECKConv. It acquires SE(3) equivariance from the kernel domain defined in a double coset space, and its explicit kernel design using coordinate-based networks enhances its learning capability and
memory efficiency. The experiments on diverse point cloud tasks, e.g., classification, pose registration, part
segmentation, and large-scale semantic
segmentation, validate the rigid equivariance,
memory scalability, and outstanding performance of ECKConv compared to state-of-the-art equivariant methods.
Read the full abstract ▾
All authors Jaein Kim, Hee Bin Yoo, Dong-Sig Han, Byoung-Tak Zhang
Computer Vision
Rel-Zero: Harnessing Patch-Pair Invariance for Robust Zero-Watermarking Against AI Editing
Pengzhen Chen, Yanwei Liu, Xiaoyan Gu et al. (+3 more)
Recent advancements in
diffusion-based image editing pose a significant threat to the authenticity of digital visual content. Traditional
embedding-based watermarking methods often introduce perceptible perturbations to maintain robustness, inevitably compromising visual fidelity. Meanwhile, existing zero-watermarking approaches, typically relying on global image features, struggle to withstand sophisticated manipulations. In this work, we uncover a key observation: while individual image patches undergo substantial alterations during AI-based editing, the relational distance between patch pairs remains relatively invariant. Leveraging this property, we propose Relational Zero-Watermarking (Rel-Zero), a novel framework that requires no modification to the original image but derives a unique zero-watermark from these editing-invariant patch relations. By grounding the watermark in intrinsic structural consistency rather than absolute appearance, Rel-Zero provides a non-invasive yet resilient mechanism for content authentication. Extensive experiments demonstrate that Rel-Zero achieves substantially improved robustness across diverse editing models and manipulations compared to prior zero-watermarking approaches.
Read the full abstract ▾
All authors Pengzhen Chen, Yanwei Liu, Xiaoyan Gu, Xiaojun Chen, Wu Liu, Weiping Wang
Computer Vision
AdapTS: Lightweight Teacher-Student Approach for Multi-Class and Continual Visual Anomaly Detection
Manuel Barusco, Davide Dalle Pezze, Francesco Borsatti et al. (+1 more)
Visual Anomaly Detection (VAD) is crucial for industrial inspection, yet most existing methods are limited to single-category scenarios, failing to address the multi-class and continual learning demands of real-world environments. While Teacher-Student (TS) architectures are efficient, they remain unexplored for the Continual Setting. To bridge this gap, we propose AdapTS, a unified TS framework designed for multi-class and continual settings, optimized for edge deployment. AdapTS eliminates the need for two different architectures by utilizing a single shared frozen backbone and injecting lightweight trainable adapters into the student pathway. Training is enhanced via a
segmentation-guided objective and synthetic Perlin noise, while a prototype-based task identification mechanism dynamically selects adapters at inference with 99% accuracy. Experiments on MVTec AD and VisA demonstrate that AdapTS matches the performance of existing TS methods across multi-class and continual learning scenarios, while drastically reducing
memory overhead. Our lightest variant, AdapTS-S, requires only 8 MB of additional
memory, 13x less than STFPM (95 MB), 48x less than RD4AD (360 MB), and 149x less than DeSTSeg (1120 MB), making it a highly scalable solution for edge deployment in complex industrial environments.
Read the full abstract ▾
All authors Manuel Barusco, Davide Dalle Pezze, Francesco Borsatti, Gian Antonio Susto
Computer Vision
MM-OVSeg:Multimodal Optical-SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing
Yimin Wei, Aoran Xiao, Hongruixuan Chen et al. (+2 more)
Open-vocabulary
segmentation enables pixel-level recognition from an open set of textual categories, allowing generalization beyond fixed classes. Despite great potential in remote sensing, progress in this area remains largely limited to clear-sky optical data and struggles under cloudy or haze-contaminated conditions. We present MM-OVSeg, a
multimodal Optical-SAR fusion framework for resilient open-vocabulary
segmentation under adverse weather conditions. MM-OVSeg leverages the complementary strengths of the two modalities–optical imagery provides rich spectral semantics, while synthetic aperture radar (SAR) offers cloud-penetrating structural cues. To address the cross-modal domain gap and the limited dense prediction capability of current
vision-
language models, we propose two key designs: a cross-modal unification process for multi-sensor representation
alignment, and a dual-encoder fusion module that integrates hierarchical features from multiple
vision foundation models for text-aligned
multimodal segmentation. Extensive experiments demonstrate that MM-OVSeg achieves superior robustness and generalization across diverse cloud conditions. The source
dataset and code are available at https://github.com/Jimmyxichen/MM-OVSeg.
Read the full abstract ▾
All authors Yimin Wei, Aoran Xiao, Hongruixuan Chen, Junshi Xia, Naoto Yokoya
Computer Vision
PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation
Jianjian Yin, Tao Chen, Yi Chen et al. (+4 more)
Recent advances in
vision-
language models (VLMs) have garnered substantial
attention in open-vocabulary semantic and part
segmentation (OSPS). However, existing methods extract image-text
alignment cues from cost volumes through a serial structure of spatial and class aggregations, leading to knowledge interference between class-level semantics and spatial context. Therefore, this paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the above challenge, enabling the model to capture richer
vision-
language alignment information from cost volumes. Specifically, we design an expert-driven perceptual learning (EPL) module that efficiently integrates semantic and contextual streams. It incorporates a multi-expert parser to extract complementary features from multiple perspectives. In addition, a coefficient mapper is designed to adaptively learn pixel-specific weights for each feature, enabling the integration of complementary knowledge into a unified and robust feature
embedding. Furthermore, we propose a feature orthogonalization decoupling (FOD) strategy to mitigate redundancy between the semantic and contextual streams, which allows the EPL module to learn diverse knowledge from orthogonalized features. Extensive experiments on eight
benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
Read the full abstract ▾
All authors Jianjian Yin, Tao Chen, Yi Chen, Gensheng Pei, Xiangbo Shu, Yazhou Yao, Fumin Shen
Computer Vision
UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images
Guibiao Liao, Qian Ren, Kaimin Liao et al. (+4 more)
Semantic-aware 3D reconstruction from sparse, unposed images remains challenging for feed-forward 3D Gaussian Splatting (3DGS). Existing methods often predict an over-complete set of Gaussian primitives under sparse-view supervision, leading to unstable geometry and inferior depth quality. Meanwhile, they rely solely on 2D segmenter features for semantic lifting, which provides weak 3D-level and limited generalizable supervision, resulting in incomplete 3D semantics in novel scenes. To address these issues, we propose UniSem, a unified framework that jointly improves depth accuracy and semantic generalization via two key components. First, Error-aware Gaussian Dropout (EGD) performs error-guided capacity control by suppressing redundancy-prone Gaussians using rendering error cues, producing meaningful, geometrically stable Gaussian representations for improved depth estimation. Second, we introduce a Mix-training Curriculum (MTC) that progressively blends 2D segmenter-lifted semantics with the model's own emergent 3D semantic priors, implemented with object-level prototype
alignment to enhance semantic coherence and completeness. Extensive experiments on ScanNet and Replica show that UniSem achieves superior performance in depth prediction and open-vocabulary 3D
segmentation across varying numbers of input views. Notably, with 16-view inputs, UniSem reduces depth Rel by 15.2% and improves open-vocabulary
segmentation mAcc by 3.7% over strong baselines.
Read the full abstract ▾
All authors Guibiao Liao, Qian Ren, Kaimin Liao, Hua Wang, Zhi Chen, Luchao Wang, Yaohua Tang
Computer Vision
Early Intervention for VFM-based Multimodal Medical Image Classification
Qijie Wei, Hailan Lin and Xirong Li
Current methods for
multimodal medical image classification (M3IC) face two major challenges. First, the prevailing "fusion after unimodal image
embedding" paradigm cannot fully exploit the complementary and correlated information in the
multimodal data. Second, the scarcity of labeled
multimodal medical images, coupled with their substantial domain shift from natural images, impedes the use of cutting-edge
Vision Foundation Models (VFMs) for medical image
embedding. To jointly address the challenges, we propose a novel Early Intervention (EI) framework. Treating one modality as target and the rest as reference, EI harnesses high-level semantic
tokens from the reference as intervention
tokens to steer the target modality's
embedding process at an early stage. For parameter-efficient VFM adaptation, we introduce Mixture of varied-rank LoRAs (MoR), which employs a small set of LoRA experts with distinct ranks and a bypass-allowed router. Extensive experiments on three public
datasets, covering retinal disease recognition, skin lesion recognition, and knee anomaly classification, verify the effectiveness of the proposed method against a number of competitive baselines.
Read the full abstract ▾
Computer Vision
Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation
Jiawei Zhou, Chi Zhang, Xiang Feng et al. (+6 more)
We present Omni-I2C, a comprehensive
benchmark designed to evaluate the capability of Large
Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code. We argue that this task represents a non-trivial challenge for the current generation of LMMs: it demands an unprecedented synergy between high-fidelity visual perception – to parse intricate spatial hierarchies and symbolic details – and precise generative expression – to synthesize syntactically sound and logically consistent code. Unlike traditional descriptive tasks, Omni-I2C requires a holistic understanding where any minor perceptual
hallucination or coding error leads to a complete failure in visual reconstruction. Omni-I2C features 1080 meticulously curated samples, defined by its breadth across subjects, image modalities, and programming
languages. By incorporating authentic user-sourced cases, the
benchmark spans a vast spectrum of digital content – from scientific visualizations to complex symbolic notations – each paired with executable reference code. To complement this diversity, our evaluation framework provides necessary depth; by decoupling performance into perceptual fidelity and symbolic precision, it transcends surface-level accuracy to expose the granular structural failures and
reasoning bottlenecks of current LMMs. Our evaluation reveals a substantial performance gap among leading LMMs; even state-of-the-art models struggle to preserve structural integrity in complex scenarios, underscoring that
multimodal code generation remains a formidable challenge. Data and code are available at https://github.com/MiliLab/Omni-I2C.
Read the full abstract ▾
All authors Jiawei Zhou, Chi Zhang, Xiang Feng, Qiming Zhang, Haibo Qiu, Lihuo He, Dengpan Ye, Xinbo Gao, Jing Zhang
Computer Vision
UAV-CB: A Complex-Background RGB-T Dataset and Local Frequency Bridge Network for UAV Detection
Shenghui Huang, Menghao Hu, Longkun Zou et al. (+6 more)
Detecting Unmanned Aerial Vehicles (
UAVs) in low-altitude environments is essential for perception and defense systems but remains highly challenging due to complex backgrounds, camouflage, and
multimodal interference. In real-world scenarios,
UAVs are frequently visually blended with surrounding structures such as buildings, vegetation, and power lines, resulting in low contrast, weak boundaries, and strong confusion with cluttered background textures. Existing
UAV detection
datasets, though diverse, are not specifically designed to capture these camouflage and complex-background challenges, which limits progress toward robust real-world perception. To fill this gap, we construct
UAV-CB, a new RGB-T
UAV detection
dataset deliberately curated to emphasize complex low-altitude backgrounds and camouflage characteristics. Furthermore, we propose the Local Frequency Bridge Network (LFBNet), which models features in localized frequency space to bridge both the frequency-spatial fusion gap and the cross-modality discrepancy gap in RGB-T fusion. Extensive experiments on
UAV-CB and public
benchmarks demonstrate that LFBNet achieves state-of-the-art detection performance and strong robustness under camouflaged and cluttered conditions, offering a frequency-aware perspective on
multimodal UAV perception in real-world applications.
Read the full abstract ▾
All authors Shenghui Huang, Menghao Hu, Longkun Zou, Hongyu Chi, Zekai Li, Feng Gao, Fan Yang, Qingyao Wu, Ke Chen
Computer Vision
UniSAFE: A Comprehensive Benchmark for Safety Evaluation of Unified Multimodal Models
Segyu Lee, Boryeong Cho, Hojung Jung et al. (+8 more)
Unified
Multimodal Models (UMMs) offer powerful cross-modality capabilities but introduce new safety risks not observed in single-task models. Despite their emergence, existing safety
benchmarks remain fragmented across tasks and modalities, limiting the comprehensive evaluation of complex system-level vulnerabilities. To address this gap, we introduce UniSAFE, the first comprehensive
benchmark for system-level safety evaluation of UMMs across 7 I/O modality combinations, spanning conventional tasks and novel
multimodal-context image generation settings. UniSAFE is built with a shared-target design that projects common risk scenarios across task-specific I/O configurations, enabling controlled cross-task comparisons of safety failures. Comprising 6,802 curated instances, we use UniSAFE to evaluate 15 state-of-the-art UMMs, both proprietary and open-source. Our results reveal critical vulnerabilities across current UMMs, including elevated safety violations in multi-image composition and multi-turn settings, with image-output tasks consistently more vulnerable than text-output tasks. These findings highlight the need for stronger system-level safety
alignment for UMMs. Our code and data are publicly available at https://github.com/segyulee/UniSAFE
Read the full abstract ▾
All authors Segyu Lee, Boryeong Cho, Hojung Jung, Seokhyun An, Juhyeong Kim, Jaehyun Kwak, Yongjin Yang, Sangwon Jang, Youngrok Park, Wonjun Chang, Se-Young Yun
Computer Vision
Revisiting Cross-Attention Mechanisms: Leveraging Beneficial Noise for Domain-Adaptive Learning
Zelin Zang, Yehui Yang, Fei Wang et al. (+2 more)
Unsupervised Domain Adaptation (UDA) seeks to transfer knowledge from a labeled source domain to an unlabeled target domain but often suffers from severe domain and scale gaps that degrade performance. Existing cross-
attention-based
transformers can align features across domains, yet they struggle to preserve content semantics under large appearance and scale variations. To explicitly address these challenges, we introduce the concept of beneficial noise, which regularizes cross-
attention by injecting controlled perturbations, encouraging the model to ignore style distractions and focus on content. We propose the Domain-Adaptive Cross-Scale Matching (DACSM) framework, which consists of a Domain-Adaptive
Transformer (DAT) for disentangling domain-shared content from domain-specific style, and a Cross-Scale Matching (CSM) module that adaptively aligns features across multiple resolutions. DAT incorporates beneficial noise into cross-
attention, enabling progressive domain translation with enhanced robustness, yielding content-consistent and style-invariant representations. Meanwhile, CSM ensures semantic consistency under scale changes. Extensive experiments on VisDA-2017, Office-Home, and DomainNet demonstrate that DACSM achieves state-of-the-art performance, with up to +2.3% improvement over CDTrans on VisDA-2017. Notably, DACSM achieves a +5.9% gain on the challenging "truck" class of VisDA, evidencing the strength of beneficial noise in handling scale discrepancies. These results highlight the effectiveness of combining domain translation, beneficial-noise-enhanced
attention, and scale-aware
alignment for robust cross-domain representation learning.
Read the full abstract ▾
All authors Zelin Zang, Yehui Yang, Fei Wang, Liangyu Li, Baigui Sun
Computer Vision
VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection
Chupeng Liu, Jiyong Rao, Shangquan Sun et al. (+2 more)
Monocular 3D object detection typically relies on pseudo-labeling techniques to reduce dependency on real-world annotations. Recent advances demonstrate that deterministic linguistic cues can serve as effective auxiliary weak supervision signals, providing complementary semantic context. However, hand-crafted textual descriptions struggle to capture the inherent visual diversity of individuals across scenes, limiting the model's ability to learn scene-aware representations. To address this challenge, we propose Visual-referred Probabilistic
Prompt Learning (VirPro), an adaptive multi-modal pretraining paradigm that can be seamlessly integrated into diverse weakly supervised
monocular 3D detection frameworks. Specifically, we generate a diverse set of learnable, instance-conditioned
prompts across scenes and store them in an Adaptive
Prompt Bank (APB). Subsequently, we introduce Multi-Gaussian
Prompt Modeling (MGPM), which incorporates scene-based visual features into the corresponding textual
embeddings, allowing the text
prompts to express visual uncertainties. Then, from the fused
vision-
language embeddings, we decode a
prompt-targeted Gaussian, from which we derive a unified object-level
prompt embedding for each instance. RoI-level contrastive matching is employed to enforce modality
alignment, bringing
embeddings of co-occurring objects within the same scene closer in the latent space, thus enhancing semantic coherence. Extensive experiments on the KITTI
benchmark demonstrate that integrating our pretraining paradigm consistently yields substantial performance gains, achieving up to a 4.8% average precision improvement than the baseline. Code is available at https://github.com/AustinLCP/VirPro.
Read the full abstract ▾
All authors Chupeng Liu, Jiyong Rao, Shangquan Sun, Runkai Zhao, Weidong Cai
Computer Vision
AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization
Dailan He, Guanlin Feng, Xingtong Ge et al. (+5 more)
Streaming autoregressive (AR)
video generators combined with few-step
distillation achieve low-latency, high-quality synthesis, yet remain difficult to align via reinforcement learning from human feedback (RLHF). Existing SDE-based GRPO methods face challenges in this setting: few-step ODEs and consistency model samplers deviate from standard flow-matching ODEs, and their short, low-stochasticity trajectories are highly sensitive to initialization noise, rendering intermediate SDE exploration ineffective. We propose AR-CoPO (AutoRegressive Contrastive Policy Optimization), a framework that adapts the Neighbor GRPO contrastive perspective to streaming AR generation. AR-CoPO introduces chunk-level
alignment via a forking mechanism that constructs neighborhood candidates at a randomly selected chunk, assigns sequence-level rewards, and performs localized GRPO updates. We further propose a semi-on-policy training strategy that complements on-policy exploration with exploitation over a replay buffer of reference rollouts, improving generation quality across domains. Experiments on Self-Forcing demonstrate that AR-CoPO improves both out-of-domain generalization and in-domain human preference
alignment over the baseline, providing evidence of genuine
alignment rather than reward hacking.
Read the full abstract ▾
All authors Dailan He, Guanlin Feng, Xingtong Ge, Yi Zhang, Bingqi Ma, Guanglu Song, Yu Liu, Hongsheng Li
Computer Vision
FACE-net: Factual Calibration and Emotion Augmentation for Retrieval-enhanced Emotional Video Captioning
Weidong Chen, Cheng Ye, Zhendong Mao et al. (+5 more)
Emotional
Video Captioning (EVC) is an emerging task, which aims to describe factual content with the intrinsic
emotions expressed in
videos. Existing works perceive global emotional cues and then combine with
video content to generate descriptions. However, insufficient factual and emotional cues mining and coordination during generation make their methods difficult to deal with the factual-emotional bias, which refers to the factual and emotional requirements being different in different samples on generation. To this end, we propose a
retrieval-enhanced framework with FActual Calibration and
Emotion augmentation (FACE-net), which through a unified architecture collaboratively mines factual-emotional semantics and provides adaptive and accurate guidance for generation, breaking through the compromising tendency of factual-emotional descriptions in all sample learning. Technically, we firstly introduces an external repository and retrieves the most relevant sentences with the
video content to augment the semantic information. Subsequently, our factual calibration via uncertainty estimation module splits the retrieved information into subject-predicate-object triplets, and self-refines and cross-refines different components through
video content to effectively mine the factual semantics; while our progressive visual
emotion augmentation module leverages the calibrated factual semantics as experts, interacts with the
video content and
emotion dictionary to generate visual queries and candidate
emotions, and then aggregates them to adaptively augment
emotions to each factual semantics. Moreover, to alleviate the factual-emotional bias, we design a dynamic bias adjustment routing module to predict and adjust the degree of bias of a sample.
Read the full abstract ▾
All authors Weidong Chen, Cheng Ye, Zhendong Mao, Peipei Song, Xinyan Liu, Lei Zhang, Xiaojun Chang, Yongdong Zhang
Computer Vision
AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
Siqi Pei, Liang Tang, Tiaonan Duan et al. (+9 more)
GUI grounding is a critical capability for
vision-
language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural
language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural
language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding
dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public
benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI
agent deployment.
Read the full abstract ▾
All authors Siqi Pei, Liang Tang, Tiaonan Duan, Long Chen, Shuxian Li, Kaer Huang, Yanzhe Jing, Yiqiang Yan, Bo Zhang, Chenghao Jiang, Borui Zhang, Jiwen Lu
Computer Vision
ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation
Xiangyu Kong, Xiaoyu Jin, Yihan Pan et al. (+6 more)
In natural face-to-face interaction, participants seamlessly alternate between speaking and listening, producing facial behaviors (FBs) that are finely informed by long-range context and naturally exhibit contextual appropriateness and emotional rationality. Interactive Head Generation (IHG) aims to synthesize lifelike avatar head
video emulating such capabilities. Existing IHG methods typically condition on dual-track signals (i.e., human user's behaviors and pre-defined audio for avatar) within a short temporal window, jointly driving generation of avatar's audio-aligned lip articulation and non-verbal FBs. However, two main challenges persist in these methods: (i) the reliance on short-clip behavioral cues without long-range contextual modeling leads them to produce facial behaviors lacking contextual appropriateness; and (ii) the entangled, role-agnostic fusion of dual-track signals empirically introduces cross-signal interference, potentially compromising lip-region synchronization during speaking. To this end, we propose ECHO, a novel IHG framework comprising two key components: a Long-range Contextual Understanding (LCU) component that facilitates contextual understanding of both behavior-grounded dynamics and linguistic-driven affective semantics to promote contextual appropriateness and emotional rationality of synthesized avatar FBs; and a block-wise Spatial-aware Decoupled Cross-
attention Modulation (SDCM) module, that preserves self-audio-driven lip articulation while adaptively integrating user contextual behavioral cues for non-lip facial regions, complemented by our designed two-stage training paradigm, to jointly enhance lip synchronization and visual fidelity. Extensive experiments demonstrate the effectiveness of proposed components and ECHO's superior IHG performance.
Read the full abstract ▾
All authors Xiangyu Kong, Xiaoyu Jin, Yihan Pan, Haoqin Sun, Hengde Zhu, Xiaoming Xu, Xiaoming Wei, Lu Liu, Siyang Song
Computer Vision
Xi Ye, Wenjia Yang, Yangyang Xu et al. (+4 more)
Image-conditioned
video diffusion models achieve impressive visual realism but often suffer from weakened motion fidelity, e.g., reduced motion dynamics or degraded long-term temporal coherence, especially after fine-tuning. We study motion
alignment in
video diffusion models post-training. To address this, we introduce pixel-motion rewards based on pixel flux dynamics, capturing both instantaneous and long-term motion consistency. We further propose \underline{S}mooth \underline{H}ybr\underline{i}d \underline{F}ine-\underline{t}uning (SHIFT), a scalable reward-driven framework that unifies supervised fine-tuning and advantage-weighted fine-tuning. Benefiting from novel
adversarial advantages, SHIFT improves convergence speed and mitigates reward hacking. Experiments show that our approach efficiently resolves dynamic-degree collapse in modern
video diffusion models supervised fine-tuning. Project page: https://xiye20.github.io/projects/SHIFT/.
Read the full abstract ▾
All authors Xi Ye, Wenjia Yang, Yangyang Xu, Xiaoyang Liu, Duo Su, Mengfei Xia, Jun Zhu
Computer Vision · Cross-listed from eess.IV
Structured SIR: Efficient and Expressive Importance-Weighted Inference for High-Dimensional Image Registration
Ivor J. A. Simpson and Neill D. F. Campbell
Image registration is an ill-posed dense
vision task, where multiple solutions achieve similar loss values, motivating probabilistic inference. Variational inference has previously been employed to capture these distributions, however restrictive assumptions about the posterior form can lead to poor characterisation, overconfidence and low-quality samples. More flexible posteriors are typically bottlenecked by the complexity of high-dimensional covariance
matrices required for dense 3D image registration. In this work, we present a
memory and computationally efficient inference method, Structured SIR, that enables expressive, multi-modal, characterisation of uncertainty with high quality samples. We propose the use of a Sampled Importance Resampling (SIR)
algorithm with a novel
memory-efficient high-dimensional covariance parameterisation as the sum of a low-rank covariance and a sparse, spatially structured Cholesky precision factor. This structure enables capturing complex spatial correlations while remaining computationally tractable. We evaluate the efficacy of this approach in 3D dense image registration of
brain MRI data, which is a very high-dimensional problem. We demonstrate that our proposed method produces uncertainty estimates that are significantly better calibrated than those produced by variational methods, achieving equivalent or better accuracy. Crucially, we show that the model yields highly structured multi-modal posterior distributions, enable effective and efficient uncertainty quantification.
Read the full abstract ▾
Computer Vision
Towards Motion-aware Referring Image Segmentation
Chaeyun Kim, Seunghoon Yi, Yejin Kim et al. (+2 more)
Referring Image
Segmentation (RIS) requires identifying objects from images based on textual descriptions. We observe that existing methods significantly underperform on motion-related queries compared to appearance-based ones. To address this, we first introduce an efficient data augmentation scheme that extracts motion-centric phrases from original captions, exposing models to more motion expressions without additional annotations. Second, since the same object can be described differently depending on the context, we propose
Multimodal Radial Contrastive Learning (MRaCL), performed on fused image-text
embeddings rather than unimodal representations. For comprehensive evaluation, we introduce a new test split focusing on motion-centric queries, and introduce a new
benchmark called M-Bench, where objects are distinguished primarily by actions. Extensive experiments show our method substantially improves performance on motion-centric queries across multiple RIS models, maintaining competitive results on appearance-based descriptions. Codes are available at https://github.com/snuviplab/MRaCL
Read the full abstract ▾
All authors Chaeyun Kim, Seunghoon Yi, Yejin Kim, Yohan Jo, Joonseok Lee
Computer Vision
Mutually Causal Semantic Distillation Network for Zero-Shot Learning
Shiming Chen, Shuhuang Chen, Guo-Sen Xie et al. (+1 more)
Zero-shot learning (ZSL) aims to recognize the unseen classes in the open-world guided by the side-information (e.g., attributes). Its key task is how to infer the latent semantic knowledge between visual and attribute features on seen classes, and thus conducting a desirable semantic knowledge transfer from seen classes to unseen ones. Prior works simply utilize unidirectional
attention within a weakly-supervised manner to learn the spurious and limited latent semantic representations, which fail to effectively discover the intrinsic semantic knowledge (e.g., attribute semantic) between visual and attribute features. To solve the above challenges, we propose a mutually causal semantic
distillation network (termed MSDN++) to distill the intrinsic and sufficient semantic representations for ZSL. MSDN++ consists of an attribute$\rightarrow$visual causal
attention sub-net that learns attribute-based visual features, and a visual$\rightarrow$attribute causal
attention sub-net that learns visual-based attribute features. The causal attentions encourages the two sub-nets to learn causal
vision-attribute associations for representing reliable features with causal visual/attribute learning. With the guidance of semantic
distillation loss, the two mutual
attention sub-nets learn collaboratively and teach each other throughout the training process. Extensive experiments on three widely-used
benchmark datasets (e.g., CUB, SUN, AWA2, and FLO) show that our MSDN++ yields significant improvements over the strong baselines, leading to new state-of-the-art performances.
Read the full abstract ▾
All authors Shiming Chen, Shuhuang Chen, Guo-Sen Xie, Xinge You
Computer Vision
Joint Degradation-Aware Arbitrary-Scale Super-Resolution for Variable-Rate Extreme Image Compression
Xinning Chai, Zhengxue Cheng, Xin Li et al. (+2 more)
Recent
diffusion-based extreme image compression methods have demonstrated remarkable performance at ultra-low bitrates. However, most approaches require training separate
diffusion models for each target bitrate, resulting in substantial computational overhead and hindering practical deployment. Meanwhile, recent studies have shown that joint super-resolution can serve as an effective approach for enhancing low-bitrate reconstruction. However, when moving toward ultra-low bitrate regimes, these methods struggle due to severe information loss, and their reliance on fixed super-resolution scales prevents flexible adaptation across diverse bitrates. To address these limitations, we propose ASSR-EIC, a novel image compression framework that leverages arbitrary-scale super-resolution (ASSR) to support variable-rate extreme image compression (EIC). An arbitrary-scale downsampling module is introduced at the encoder side to provide controllable rate reduction, while a
diffusion-based, joint degradation-aware ASSR decoder enables rate-adaptive reconstruction within a single model. We exploit the compression- and rescaling-aware
diffusion prior to guide the reconstruction, yielding high fidelity and high realism restoration across diverse compression and rescaling settings. Specifically, we design a global compression-rescaling adaptor that offers holistic guidance for rate adaptation, and a local compression-rescaling modulator that dynamically balances generative and fidelity-oriented behaviors to achieve fine-grained, bitrate-adaptive detail restoration. To further enhance reconstruction quality, we introduce a dual semantic-enhanced design. Extensive experiments demonstrate that ASSR-EIC delivers state-of-the-art performance in extreme image compression while simultaneously supporting flexible bitrate control and adaptive rate-dependent reconstruction.
Read the full abstract ▾
All authors Xinning Chai, Zhengxue Cheng, Xin Li, Rong Xie, Li Song
Computer Vision
Motion-Adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion
Rui Hong and Shuxue Quan
We present a motion-adaptive temporal
attention mechanism for parameter-efficient
video generation built upon frozen Stable
Diffusion models. Rather than treating all
video content uniformly, our method dynamically adjusts temporal
attention receptive fields based on estimated motion content: high-motion sequences attend locally across frames to preserve rapidly changing details, while low-motion sequences attend globally to enforce scene consistency. We inject lightweight temporal
attention modules into all UNet
transformer blocks via a cascaded strategy – global
attention in down-sampling and middle blocks for semantic stabilization, motion-adaptive
attention in up-sampling blocks for fine-grained refinement. Combined with temporally correlated noise initialization and motion-aware gating, the system adds only 25.8M trainable parameters (2.9% of the base UNet) while achieving competitive results on WebVid validation when trained on 100K
videos. We demonstrate that the standard denoising objective alone provides sufficient implicit temporal regularization, outperforming approaches that add explicit temporal consistency losses. Our ablation studies reveal a clear trade-off between noise correlation and motion amplitude, providing a practical inference-time control for diverse generation behaviors.
Read the full abstract ▾
Computer Vision
Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation
Rui Hong and Jana Kosecka
Estimating 3D hand pose from
monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign
language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative
embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint
token Transformer guided by gesture
embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.
Read the full abstract ▾
Computer Vision
FMMC: Harnessing the Power of Foundation Models for Accurate Material Classification
Qingran Lin, Fengwei Yang and Chaolun Zhu
Material classification has emerged as a critical task in computer
vision and graphics, supporting the assignment of accurate material properties to a wide range of digital and real-world applications. While traditionally framed as an image classification task, this domain faces significant challenges due to the scarcity of annotated data, limiting the accuracy and generalizability of trained models. Recent advances in
vision-
language foundation models (VLMs) offer promising avenues to address these issues, yet existing solutions leveraging these models still exhibit unsatisfying results in material recognition tasks. In this work, we propose a novel framework that effectively harnesses foundation models to overcome data limitations and enhance classification accuracy. Our method integrates two key innovations: (a) a robust image generation and auto-labeling pipeline that creates a diverse and high-quality training
dataset with material-centric images, and automatically assigns labels by fusing object semantics and material attributes in text
prompts; (b) a prior incorporation strategy to distill information from VLMs, combined with a joint fine-tuning method that optimizes a pre-trained
vision foundation model alongside VLM-derived priors, preserving broad generalizability while adapting to material-specific features. Extensive experiments demonstrate significant improvements on multiple
datasets. We show that our synthetic
dataset effectively captures the characteristics of real world materials, and the integration of priors from
vision-
language models significantly enhances the final performance. The source code and
dataset will be released.
Read the full abstract ▾
Computer Vision
Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis
Rui Hong and Jana Kosecka
Generating natural, correct, and visually smooth 3D avatar sign
language motion conditioned on the text inputs continues to be very challenging. In this work, we train a generative model of 3D body motion and explore the role of phonological attribute conditioning for sign
language motion generation, using ASL-LEX 2.0 annotations such as hand shape, hand location and movement. We first establish a strong
diffusion baseline using an Human Motion MDM-style
diffusion model with SMPL-X representation, which outperforms SignAvatar, a state-of-the-art CVAE method, on gloss discriminability metrics. We then systematically study the role of text conditioning using different text encoders (CLIP vs. T5), conditioning modes (gloss-only vs. gloss+phonological attributes), and attribute notation format (symbolic vs. natural
language). Our analysis reveals that translating symbolic ASL-LEX notations to natural
language is a necessary condition for effective CLIP-based attribute conditioning, while T5 is largely unaffected by this translation. Furthermore, our best-performing variant (CLIP with mapped attributes) outperforms SignAvatar across all metrics. These findings highlight input representation as a critical factor for text-encoder-based attribute conditioning, and motivate structured conditioning approaches where gloss and phonological attributes are encoded through independent pathways.
Read the full abstract ▾
Computer Vision
VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm
Hongbo Lu, Liang Yao, Chenghao He et al. (+4 more)
A fundamental bottleneck in Novel View Synthesis (NVS) for autonomous driving is the inherent supervision gap on novel trajectories: models are tasked with synthesizing unseen views during inference, yet lack ground truth images for these shifted poses during training. In this paper, we propose VisionNVS, a camera-only framework that fundamentally reformulates view synthesis from an ill-posed extrapolation problem into a self-supervised inpainting task. By introducing a “Virtual-Shift” strategy, we use
monocular depth proxies to simulate occlusion patterns and map them onto the original view. This paradigm shift allows the use of raw, recorded images as pixel-perfect supervision, effectively eliminating the domain gap inherent in previous approaches. Furthermore, we address spatial consistency through a Pseudo-3D Seam Synthesis strategy, which integrates visual data from adjacent cameras during training to explicitly model real-world photometric discrepancies and calibration errors. Experiments demonstrate that VisionNVS achieves superior
geometric fidelity and visual quality compared to
LiDAR-dependent baselines, offering a robust solution for scalable driving simulation.
Read the full abstract ▾
All authors Hongbo Lu, Liang Yao, Chenghao He, Fan Liu, Wenlong Liao, Tao He, Pai Peng