Computer Vision
Stylized Text-to-Motion Generation via Hypernetwork-Driven Low-Rank Adaptation
Junhyuk Jeon, Seokhyeon Hong and Junyong Noh
Text-driven motion
diffusion models are capable of generating realistic human motions, but text alone often struggles to express fine-level nuances of motion, commonly referred to as style. Recent approaches have tackled this challenge by attaching a style injection mechanism to a pretrained text-driven
diffusion model. Existing stylization methods, however, either require style-specific fine-tuning of existing models or rely on heavy ControlNet-based architectures, limiting efficiency and generalization to unseen styles. We propose a lightweight style conditioning framework that dynamically modulates a pretrained
diffusion model through hypernetwork-generated LoRA parameters. A style reference motion is encoded into a global style
embedding, which is mapped by a hypernetwork to low-rank updates applied at each denoising step of the
diffusion model. By structuring the style latent space with a supervised contrastive loss, our framework reliably captures diverse stylistic attributes, improves generalization to unseen styles, and supports optimization-based guidance without requiring predefined style categories. Experiments on the HumanML3D and 100STYLE
datasets show state-of-the-art stylization results, while achieving improved stylization for unseen styles.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Yunheng Wang, Yuetong Fang, Taowen Wang et al. (+9 more)
Vision-and-
Language Navigation (VLN) is a cornerstone of
embodied intelligence. However, current
agents often suffer from significant performance degradation when transitioning from simulation to real-world deployment, primarily due to perceptual instability (e.g., lighting variations and motion blur) and under-specified instructions. While existing methods attempt to bridge this gap by scaling up model size and training data, we argue that the bottleneck lies in the lack of robust spatial grounding and cross-domain priors. In this paper, we propose StereoNav, a robust
Vision-
Language-Action framework designed to enhance real-world
navigation consistency. To address the inherent gap between synthetic training and physical execution, we introduce Target-Location Priors as a persistent bridge. These priors provide stable visual guidance that remains invariant across domains, effectively grounding the
agent even when instructions are vague. Furthermore, to mitigate visual disturbances like motion blur and illumination shifts, StereoNav leverages stereo
vision to construct a unified representation of semantics and geometry, enabling precise action prediction through enhanced depth awareness. Extensive experiments on R2R-CE and RxR-CE demonstrate that StereoNav achieves state-of-the-art
egocentric RGB performance, with SR and SPL scores of 81.1% and 68.3%, and 67.5% and 52.0%, respectively, while using significantly fewer parameters and less training data than prior scaling-based approaches. More importantly, real-world robotic deployments confirm that StereoNav substantially improves
navigation reliability in complex, unstructured environments. Project page: https://yunheng-wang.github.io/stereonav-public.github.io.
Read the full abstract ▾
All authors Yunheng Wang, Yuetong Fang, Taowen Wang, Lusong Li, Kun Liu, Junzhe Xu, Zizhao Yuan, Yixiao Feng, Jiaxi Zhang, Wei Lu, Zecui Zeng, Renjing Xu
Computer Vision
KamonBench: A Grammar-Based Dataset for Evaluating Compositional Factor Recovery in Vision-Language Models
Richard Sproat and Stefano Peluchetti
Kamon (family crests) are an important part of Japanese culture and a natural test case for compositional visual recognition: each crest combines a small number of symbolic choices, but the space of possible descriptions is sparse. We introduce KamonBench, a grammar-based image-to-structure
benchmark with 20,000 synthetic composite crests and auxiliary component examples. Each composite crest is paired with a formal kamon description
language (kamon yogo) description, a segmented Japanese analysis, an English translation, and a non-linguistic program code. Because each synthetic crest is generated from known factors, namely container, modifier, and motif, KamonBench supports evaluation beyond caption-level accuracy: direct program-code factor metrics, controlled factor recombination splits, counterfactual motif-sensitivity groups under fixed container-modifier contexts, and linear probes of factor accessibility. We include baseline results for a ViT encoder with a
Transformer decoder, two VGG n-gram decoders with and without learned positional masks, and fine-tuned Qwen3-VL-4B and Qwen3-VL-8B models. KamonBench therefore provides a controlled testbed for sparse compositional visual recognition and factor recovery in
vision-
language models.
Read the full abstract ▾
Computer Vision
Test-time Sparsity for Extreme Fast Action Diffusion
Kangye Ji, Yuan Meng, Jianbo Zhou et al. (+3 more)
Action
diffusion excels at high-fidelity action generation but incurs heavy computational costs owing to its iterative denoising nature. Despite current technologies showing promise in accelerating
diffusion transformers by reusing the cached features, they struggle to adapt to policy dynamics arising from diverse perceptions and multi-round rollout iterations in open environments. We propose test-time sparsity to tackle this challenge, which aims to accelerate action
diffusion by dynamically predicting prunable residual computations for each model forward at test time. However, two bottlenecks remain in this paradigm: 1) repetitive conditional encoding and
pruning offset most potential speed gains, and 2) the features cached from previous denoising timesteps cannot constrain large
pruning errors under aggressive sparsity. To address the first bottleneck, we design a highly parallelized inference pipeline that minimizes the non-decoder delay to milliseconds. Specifically, we first design a lightweight pruner that shares the encoder with the
diffusion transformer. Then, we decouple the encoding and
pruning from the autoregressive denoising loop by processing all denoising timesteps in parallel, and overlap the pruner with the decoder forward inference through asynchronism. To overcome the second bottleneck, we introduce an omnidirectional reusing strategy, which achieves 95% sparsity by selectively reusing features cached from the current forward, previous denoising timesteps, and earlier rollout iterations. To learn the rollout-level reusing strategies, we sample a few action trajectories to supervise the sparsified
diffusion step by step. Extensive experiments demonstrate that our method reduces FLOPs by 92% and accelerates action generation by 5x, achieving lossless performance with an inference frequency of 47.5 Hz. Our code is available at https://github.com/ky-ji/Test-time-Sparsity.
Read the full abstract ▾
All authors Kangye Ji, Yuan Meng, Jianbo Zhou, Ye Li, Chen Tang, Zhi Wang
Computer Vision
Color Constancy in Hyperspectral Imaging via Reduced Spectral Spaces
G. Dofri Vidarsson, Liying Lu and Sabine Süsstrunk
Illuminant estimation aims to infer scene illumination from image measurements despite intrinsic ambiguities between surface reflectance and lighting. Most existing methods operate on trichromatic RGB images and are therefore fundamentally limited by the restricted spectral information available. Hyperspectral imaging provides a much richer representation of scene radiance and has the potential to alleviate these ambiguities. However, its high dimensionality poses computational and statistical challenges. In this work, we systematically study the effect of spectral dimensionality and representation choice on illuminant estimation performance using hyperspectral data. We adopt the practical and effective Color-by-Correlation (CbC) framework as the estimation backbone and analyze its behavior under different spectral dimensionality reduction strategies. Our results offer practical insights into how hyperspectral information can be efficiently exploited for illuminant estimation and identify conditions under which compact spectral representations outperform conventional RGB-based approaches. The code is available at https://github.com/IVRL/Reduced-Spectral-Color-Constancy.
Read the full abstract ▾
Computer Vision
Img2CADSeq: Image-to-CAD Generation via Sequence-Based Diffusion
Shiyu Tan, Zixuan Zhao, Hao Gao et al. (+3 more)
Boundary Representation (BRep) is the standard format for Computer-Aided Design (CAD), yet reconstructing high-quality BReps from single-view images remains challenging due to the complexity of topological constraints and operation sequences. We present Img2CADSeq, a multi-stage pipeline that overcomes these limitations by encoding CAD sequences into a three-level hierarchical codebook. Guided by an importance prioritization, this strategy values profiles over details, compressing long sequences into a stable discrete latent space. To bridge the modality gap, we leverage a coarse-to-fine point cloud intermediate, aligning 2D visual features with 3D CAD sequences via contrastive learning to condition a VQ-
Diffusion model. Supported by newly introduced CAD-220K and PrintCAD
datasets, our approach ensures robust industrial domain adaptation. Extensive experiments demonstrate that Img2CADSeq significantly outperforms state-of-the-art methods, producing standard STEP files that can be directly used in commercial CAD software.
Read the full abstract ▾
All authors Shiyu Tan, Zixuan Zhao, Hao Gao, Zhiheng Chen, Xiaolong Yin, Enya Shen
Computer Vision · Cross-listed from cs.CL
Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generation
Weiqing Luo, Zongye Hu, Xiao Wang et al. (+3 more)
Visual evidence selection is a critical component of
multimodal retrieval-augmented generation (RAG), yet existing methods typically rely on semantic relevance or surface-level similarity, which are often misaligned with the actual utility of visual evidence for downstream
reasoning. We reformulate
multimodal evidence selection from an information-theoretic perspective by defining evidence utility as the information gain induced on a model's output distribution. To overcome the intractability of answer-space optimization, we introduce a latent notion of evidence helpfulness and theoretically show that, under mild assumptions, ranking evidence by information gain on this latent variable is equivalent to answer-space utility. We further propose a training-free, surrogate-accelerated framework that efficiently estimates evidence utility using lightweight
multimodal models. Experiments on MRAG-Bench and Visual-RAG across multiple model families demonstrate that our method consistently outperforms state-of-the-art RAG baselines while achieving substantial reductions in computational cost.
Read the full abstract ▾
All authors Weiqing Luo, Zongye Hu, Xiao Wang, Zhiyuan Yu, Haofeng Zhang, Ziyi Huang
Computer Vision
X-Restormer++: 1st Place Solution for the UG2+ CVPR 2026 All-Weather Restoration Challenge
Youwei Pan, Leilei Cao, Yingfang Zhu et al. (+1 more)
In this work, we present our winning solution for the 8th UG2+ Challenge (CVPR 2026) Track 1: Image Restoration under All-weather Conditions. Our method is built upon the X-Restormer baseline, which captures both channel-wise global dependencies and spatially-local structural information through its dual-
attention design (Multi-DConv Head Transposed
Attention and Overlapping Cross-
Attention), augmented with the spatially-adaptive input scaling mechanism from Restormer-Plus. We adopt a two-stage training strategy with dual-model ensemble inference. In the first stage, Model B is trained from scratch on a large-scale diverse
dataset randomly sampled from the FoundIR training set (approximately 800 GB out of 4.84 TB), covering five degradation types: blur, haze, rain, snow, and composite conditions such as co-occurring rain and haze. In the second stage, Model A is fine-tuned on the WeatherStream
dataset (rain and snow splits) using Model B's final checkpoint as pretrained initialization, enabling efficient domain adaptation with a substantially smaller
dataset. To better preserve structural details during training, we propose a novel Gradient-Guided Edge-Aware (GGEA) Loss, which applies Sobel operators to the ground-truth image to construct a spatially adaptive weight map that assigns higher supervision to edge and high-frequency regions. This is incorporated alongside L1 and Multi-Scale SSIM losses in a unified training objective. At inference time, predictions from the two models are fused via a weighted average, out = 0.4 x outA + 0.6 x outB, where the higher weight assigned to Model B reflects its stronger generalization ability from large-scale pretraining. With these strategies, our proposed method successfully ranks 1st in the challenge.
Read the full abstract ▾
All authors Youwei Pan, Leilei Cao, Yingfang Zhu, Fengjie Zhu
Computer Vision
ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding
Xiao Liu, Nayu Liu, Junnan Zhu et al. (+6 more)
Video understanding requires active evidence seeking, motivating tool-augmented
video agents for temporal
reasoning, cross-modal understanding, and complex question answering. Existing
video agents have improved
video reasoning with
retrieval,
memory, frame inspection, and verifier tools, but they still face two limitations: (1) a coarse tool space that lacks fine-grained operations for compositional
reasoning; and (2) a flat action space that forces high-level
video intents into primitive executable tool calls. In this paper, we address these challenges with two complementary designs. First, we construct a MetaAug-
Video Tool Library (MVTL), an extensible tool library with 134 registered tools, including 26 base tools for general
multimodal signal processing and 108 meta tools for filtering, aggregation, reranking, formatting, and other intermediate-result operations. MVTL supports dual-level access to both structured
video information and raw modal evidence, enabling diverse
video reasoning scenarios. Second, we propose ReTool-
Video, a recursive tool-using method that grounds high-level
video intents into executable tool chains. In ReTool-
Video, matched actions are executed directly, while unmatched intents are delegated to a resolver for parameter repair, tool substitution, or decomposition. This allows abstract actions such as temporal merging, cross-modal verification, or repeated-event aggregation to be progressively translated into concrete
multimodal operations at runtime. Experiments on MVBench, MLVU, and
Video-MME w/o sub. show that ReTool-
Video consistently outperforms strong baselines. Further analysis demonstrates that recursive grounding and fine-grained meta tools improve the stability and effectiveness of complex
video understanding.
Read the full abstract ▾
All authors Xiao Liu, Nayu Liu, Junnan Zhu, Ruirui Chen, Guohui Xiang, Changjian Wang, Kaiwen Wei, Rongzhen Li, Jiang Zhong
Computer Vision
Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation
Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait et al. (+4 more)
Text-to-image (T2I) generation has advanced rapidly, making reliable evaluation critical as performance differences between models narrow. Existing evaluation practices typically apply uniform annotation mechanisms, such as Likert-scale or binary question answering (BQA), across heterogeneous evaluation skills, despite fundamental differences in their nature. In this work, we revisit T2I evaluation through the lens of skill-aligned annotation, where annotation strategies reflect the underlying characteristics of each evaluation skill. We systematically compare skill-aligned annotation against uniform baselines and show that it produces more consistent evaluation signals, with higher inter-annotator agreement and improved stability across models. Finally, we present an automated pipeline that instantiates the proposed evaluation protocol, enabling scalable and fine-grained evaluation with spatially grounded feedback. Our work highlights that improving the foundations of image evaluation can increase reliability and efficiency without simply scaling annotation effort. We hope this motivates further research on refining evaluation protocols as a central component of reliable model assessment.
Read the full abstract ▾
All authors Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait, Ansar Khangeldin, Karen Sanchez, Tong Zhang, Bernard Ghanem
Computer Vision
STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition
Hongli Liu, Yu Wang and Shengjie Zhao
Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with
vision-
language models, existing approaches still suffer from semantic-temporal misalignment, where static textual
prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (
STAR), a unified framework, consisting of a semantic-
alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic
alignment module introduces a Temporal Semantic
Attention (TSA) mechanism, which performs frame-level cross-modal
alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from large
language models (LLMs) provide long-range semantic guidance. Extensive experiments on five FSAR
benchmarks demonstrate the consistent superiority of
STAR over state-of-the-art methods. For instance,
STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small
datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/
STAR-main.
Read the full abstract ▾
Computer Vision
FIKA-Bench: From Fine-grained Recognition to Fine-Grained Knowledge Acquisition
Geng Li and Yuxin Peng
Fine-grained recognition in everyday life is often not a closed-book classification problem: when encountering unfamiliar objects, humans actively search, compare visual details, and verify evidence before deciding. Existing
benchmarks primarily evaluate visually recognition, leaving this active external knowledge acquisition ability underexplored. We study fine-grained knowledge acquisition, where a system must seek, verify, and use external evidence to answer open-ended fine-grained recognition questions. We introduce FIKA-Bench, a leakage-aware and evidence-grounded collection of 311 public-source and real-life instances. To ensure high quality, every example is filtered against frontier closed-book models to remove memorized cases and audited to eliminate image-answer leakage, retaining only samples supported by verified evidence. Our evaluation of latest Large
Multimodal Models (LMMs) and
agents reveals that the task remains a formidable challenge: the best system reaches only 25.1% accuracy, with no model exceeding 30%. Crucially, we find that merely equipping models with tools is insufficient to bridge this gap;
agent failures are predominantly driven by wrong entity
retrieval and poor visual judgement. These results show that reliable knowledge acquisition needs better
agent designs that focus on fine-grained recognition.
Read the full abstract ▾
Computer Vision
DiffST: Spatiotemporal-Aware Diffusion for Real-World Space-Time Video Super-Resolution
Zheng Chen, Ruofan Yang, Jin Han et al. (+5 more)
Diffusion-based models have shown strong performance in
video super-resolution (VSR) and
video frame interpolation (VFI). However, their role in the coupled space-time
video super-resolution (STVSR) setting remains limited. Existing
diffusion-based STVSR approaches suffer from two issues: (1) low inference efficiency and (2) insufficient utilization of spatiotemporal information. These limitations impede deployment. To address these issues, we introduce DiffST, an efficient spatiotemporal-aware
video diffusion framework for real-world STVSR. To improve efficiency, we adapt a pre-trained
diffusion model for one-step sampling and process the entire
video directly rather than operating on individual frames. Furthermore, to enhance spatiotemporal information utilization, we introduce cross-frame context aggregation (CFCA) and
video representation guidance (VRG). The CFCA module aggregates information across multiple keyframes to produce intermediate frames. The VRG module extracts
video-level global features to guide the
diffusion process. Extensive experiments show that DiffST obtains leading results on real-world STVSR tasks. It also maintains high inference efficiency, running about 17$\times$ faster than previous
diffusion-based STVSR methods. Code is available at: https://github.com/zhengchen1999/DiffST.
Read the full abstract ▾
All authors Zheng Chen, Ruofan Yang, Jin Han, Dehua Song, Zichen Zou, Chunming He, Yong Guo, Yulun Zhang
Computer Vision
Does Engram Do Memory Retrieval in Autoregressive Image Generation?
Jinghao Wang, Qiyuan He, Chunbin Gu et al. (+1 more)
The Engram module – a hash-keyed, O(1) associative
memory injected into
Transformer layers – was recently shown to improve large
language model pretraining, with the appealing interpretation that it provides a content-addressed shortcut to recurring local
token patterns. We ask whether this interpretation transfers to autoregressive (AR) image generation, or whether the observed gains, if any, come from a different mechanism. We adapt the Engram module to
vision with 2D spatial $n$-gram hashing, gated fusion, and KV-cache-compatible incremental inference, and inject it into a class-conditional AR generator trained on ImageNet 256x256. Across a sweep of backbone-to-
memory budget ratios $\rho{\in}[0.17, 0.90]$, every Engram-augmented variant trails the pure AR baseline in FID, indicating that the module saves backbone FLOPs but does not, by itself, improve sample quality. We then probe how the module is used. A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic, yet a tiny constant gate (g=0.10) matches or beats the learned gate – inconsistent with a heavily content-addressed recall mechanism. A donor-probe experiment shows that swapping the hash inputs for matched,
adversarial, or random same-class exemplars produces statistically indistinguishable next-
token distributions, while collapsing or randomising the table degrades them by two to three orders of magnitude. Finally, training a model from scratch with the entire
memory table frozen to $\mathcal{N}(0, 1)$ noise costs only $\Delta\text{FID}{=}0.10$ and actually raises Inception Score. Together, these findings indicate that the Engram in AR image generation behaves not as a content-addressed retriever but as a gated architectural side-pathway: a hash-keyed residual stream whose benefit is dominated by the pathway itself, with the learned table contributing only a small distributional refinement.
Read the full abstract ▾
All authors Jinghao Wang, Qiyuan He, Chunbin Gu, Pheng-Ann Heng
Computer Vision
CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models
Sangin Lee and Yukyung Choi
In large
vision-
language models, visual
tokens typically constitute the majority of input
tokens, leading to substantial computational overhead. To address this, recent studies have explored
pruning redundant or less informative visual
tokens for image understanding tasks. However, these methods struggle with pixel grounding tasks, where
token importance is highly contingent on the input text. Through an in-depth analysis of CLIP, we observe that visual
tokens within referent regions often exhibit low similarity to their textual representation. Motivated by this insight, we introduce LiteLVLM, a training-free, text-guided
token pruning strategy for efficient pixel grounding inference. By reversing the ranking of CLIP's visual-text similarity, LiteLVLM effectively retains visual
tokens covering the referent regions, while recovering context
tokens to enable clear foreground-background separation. Extensive experiments demonstrate that LiteLVLM significantly outperforms existing methods by over 5% across diverse
token budgets. Without any training or fine-tuning, LiteLVLM maintains 90% of the original performance with a 22% speedup and a 2.3X
memory reduction. Our code is available at https://github.com/sejong-rcv/LiteLVLM.
Read the full abstract ▾
Computer Vision
PanoWorld: Towards Spatial Supersensing in 360$^\circ$ Panorama World
Changpeng Wang, Xin Lin, Junhan Liu et al. (+5 more)
Multimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For
navigation, robotic search, and 3D scene understanding, 360-degree panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial
reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware,
language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-
Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic
benchmark for evaluating ERP-native spatial
reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen
benchmarks. These results demonstrate that robust panoramic
reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released.
Read the full abstract ▾
All authors Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen
Computer Vision · Cross-listed from cs.CR
LoREnc: Low-Rank Encryption for Securing Foundation Models and LoRA Adapters
Beomjin Ahn, Jungmin Kwon, Chanyong Jung et al. (+1 more)
Foundation models and low-rank adapters enable efficient on-device generative AI but raise risks such as intellectual property leakage and model recovery attacks. Existing defenses are often impractical because they require retraining or access to the original
dataset. We propose LoREnc, a training-free framework that secures both FMs and adapters via spectral truncation and compensation. LoREnc suppresses dominant low-rank components of FM weights, compensates for the missing information in authorized adapters, and further applies orthogonal reparameterization to obscure structural fingerprints of the protected adapter. Unauthorized users produce structurally collapsed outputs, while authorized users recover exact performance. Experiments demonstrate that LoREnc provides strong protection against model recovery with under 1% computational overhead.
Read the full abstract ▾
All authors Beomjin Ahn, Jungmin Kwon, Chanyong Jung, Jaewook Chung
Computer Vision
A$_3$B$_2$: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning
Yiyun Zhou, Zhonghua Jiang, Wenkang Han et al. (+4 more)
Efficient transfer learning methods for large-scale
vision-
language models ($e.g.$, CLIP) enable strong few-shot transfer, yet existing adaptation methods follow a fixed fine-tuning paradigm that implicitly assumes a uniform importance of the image and text branches, which has not been systematically studied in image classification. Through extensive analysis, we reveal a Branch Bias issue in
vision-
language image classification: adapting the image encoder does not always improve performance under out-of-distribution settings. Motivated by this observation, we propose A$_3$B$_2$, an Adaptive Asymmetric Adapter that alleviates Branch Bias in few-shot learning. A$_3$B$_2$ introduces Uncertainty-Aware Adapter Dampening (UAAD), which automatically suppresses image-branch adaptation when prediction uncertainty is high, enabling soft and data-driven control without manual intervention. Architecturally, A$_3$B$_2$ adopts a lightweight asymmetric design inspired by mixture-of-experts with Load Balancing Regularization. Extensive experiments on three few-shot image classification tasks across 11
datasets demonstrate that A$_3$B$_2$ consistently outperforms 11 competitive
prompt- and adapter-based baselines.
Read the full abstract ▾
All authors Yiyun Zhou, Zhonghua Jiang, Wenkang Han, Kunxi Li, Mingjing Xu, Chang Yao, Jingyuan Chen
Computer Vision
Unifying Physically-Informed Weather Priors in A Single Model for Image Restoration Across Multiple Adverse Weather Conditions
Jiaqi Xu, Xiaowei Hu, Lei Zhu et al. (+1 more)
Image restoration under multiple adverse weather conditions aims to develop a single model to recover the underlying scene with high visibility. Weather-related artifacts vary with the particle's distance to the camera according to the established scene visibility analysis, where close and faraway regions are more affected by falling drops and fog effects, respectively. Existing methods fail to consider this weather-specific physical visual process; thus, the restoration performance is limited. In this work, we analyze the common visual factors in adverse weather conditions and present a unified imaging model that considers the individually visible particles and fog-like aggregate scattering effects. Further, we design a novel weather-prior-based network, which leverages the weather-related prior information to help recover the scene by enhancing the features using the estimated occlusion and transmission. Experimental results in multiple adverse scenarios show the superiority of our method against state-of-the-art methods.
Read the full abstract ▾
All authors Jiaqi Xu, Xiaowei Hu, Lei Zhu, Pheng-Ann Heng
Computer Vision
Jiaxin Liu, Ding Zhong, Yue Wang et al. (+6 more)
Vision-
language models (VLMs) have demonstrated remarkable capabilities in bridging visual perception and natural
language understanding, enabling a wide range of
multimodal reasoning tasks. However, they often produce object
hallucinations, describing content absent from the input image, which limits their reliability and interpretability. To address this limitation, we propose Dual-Pathway
Circuit Analysis, a framework that identifies and characterizes
hallucination-related
circuits in VLMs for mechanistic understanding and causal probing. We first apply activation patching across five architecturally diverse VLMs to identify a visual grounding pathway that supports correct predictions and a
hallucination pathway that drives erroneous outputs. We then introduce Conditional Pathway Analysis (CPA) to characterize pathway-level interactions, revealing that grounding components remain strongly redundant in both correct and hallucinating samples but undergo a consistent polarity flip, shifting from supporting the ground truth on correct samples to aligning with the hallucinated answer on erroneous ones. We further perform targeted suppression of
hallucination-pathway components, showing that scaling these components reduces object
hallucination by up to 76% with minimal accuracy cost, and validate that the same
circuit selectively transfers to relational but not attribute
hallucination. Evaluations on POPE-
adversarial and AMBER show that the identified
circuits are consistent across architectures, support causal intervention, and transfer selectively across
hallucination types. Code is available at https://github.com/jiaxin26/DualPath-VLM.
Read the full abstract ▾
All authors Jiaxin Liu, Ding Zhong, Yue Wang, Zhidong Yang, Zhaolu Kang, Guangyuan Dong, Qishi Zhan, Pengcheng Fang, Aofan Liu
Computer Vision
Pareto-Guided Optimal Transport for Multi-Reward Alignment
Ying Ba, Tianyu Zhang, Mohan Zhou et al. (+5 more)
Text-to-image generation models have achieved remarkable progress in preference optimization, yet achieving robust
alignment across diverse reward models remains a significant challenge. Existing multi-reward fusion approaches rely on weighted summation, which is costly to tune and insufficient for balancing conflicting objectives. More critically, optimization with reward models is highly susceptible to reward hacking, where reward scores increase while the perceived quality of generated images deteriorates. We demonstrate that optimizing against a unified global target under heterogeneous reward upper bounds can induce reward hacking, a risk further exacerbated by the inherent instability of weak reward models. To mitigate this, we propose a Pareto Frontier-Guided Optimal Transport (PG-OT) framework. Our method constructs a
prompt-specific Pareto frontier and maps dominated samples toward it via distribution-aware optimal transport. Furthermore, we develop both online and offline optimization strategies tailored to diverse reward signal characteristics. To provide a more rigorous assessment, we introduce the Joint Domination Rate (JDR) and Joint Collapse Rate (JCR) as principled metrics to quantify multi-reward synergy and reward hacking. Experimental results show that our approach outperforms strong baselines with an 11% gain in JDR and achieves a near 80% win rate in human evaluations.
Read the full abstract ▾
All authors Ying Ba, Tianyu Zhang, Mohan Zhou, Yalong Bai, Wenyi Mo, Guiwei Zhang, Bing Su, Ji-Rong Wen
Computer Vision
EvObj: Learning Evolving Object-centric Representations for 3D Instance Segmentation without Scene Supervision
Jiahao Chen, Zihui Zhang, Yafei Yang et al. (+4 more)
We introduce EvObj for unsupervised 3D instance
segmentation that bridges the
geometric domain gap between synthetic pretraining data and real-world point clouds. Current methods suffer from structural discrepancies when transferring object priors from synthetic
datasets (e.g., ShapeNet) to real scans (e.g., ScanNet), particularly due to morphological variations and occlusion artifacts. To address this, EvObj integrates two innovative modules: (1) An object discerning module that dynamically refines object candidates, enabling continuous adaptation of object priors to target domains; and (2) An object completion module that reconstructs partial geometries after discovering objects. We conduct extensive experiments on both real-world and synthetic
datasets, demonstrating superior 3D object
segmentation performance over all baselines while achieving state-of-the-art results.
Read the full abstract ▾
All authors Jiahao Chen, Zihui Zhang, Yafei Yang, Jinxi Li, Shenxing Wei, Zhixuan Sun, Bo Yang
Computer Vision
GenCape: Structure-Inductive Generative Modeling for Category-Agnostic Pose Estimation
Jiyong Rao, Yu Wang and Shengjie Zhao
Category-agnostic pose estimation (CAPE) aims to localize keypoints on query images from arbitrary categories, using only a few annotated support examples for guidance. Recent approaches either treat keypoints as isolated entities or rely on manually defined skeleton priors, which are costly to annotate and inherently inflexible across diverse categories. Such oversimplification limits the model's capacity to capture instance-wise structural cues critical for accurate pixel-level localization. To overcome these limitations, we propose GenCape, a Generative-based framework for CAPE that infers keypoint relationships solely from image-based support inputs, without additional textual descriptions or predefined skeletons. Our framework consists of two principal components: an iterative Structure-aware Variational Autoencoder (i-SVAE) and a Compositional Graph Transfer (CGT) module. The former infers soft, instance-specific adjacency
matrices from support features through variational inference, embedded layer-wise into the Graph
Transformer Decoder for progressive structural priors refinement. The latter adaptively aggregates multiple latent graphs into a query-aware structure via
Bayesian fusion and
attention-based reweighting, enhancing resilience to visual uncertainty and support-induced bias. This structure-aware design facilitates effective message propagation among keypoints and promotes semantic
alignment across object categories with diverse keypoint
topologies. Experimental results on the MP-100
dataset show that our method achieves substantial gains over graph-support baselines under both 1- and 5-shot settings, while maintaining competitive performance against text-support counterparts.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Understanding Generalization through Decision Pattern Shift
Huiqi Deng, Yibo Li, Quanshi Zhang et al. (+3 more)
Understanding why deep neural networks (DNNs) fail to generalize to unseen samples remains a long-standing challenge. Existing studies mainly examine changes in externally observable factors such as data, representations, or outputs, yet offer limited insight into how a model's internal decision mechanism evolves from training to test. To address this gap, we introduce Decision Pattern Shift (DPS), a new perspective that defines generalization through the stability of internal decision patterns and quantifies failure as their deviation from those learned during training. Specifically, we represent each sample's decision pattern as a GradCAM-based channel-contribution vector, which captures how feature channels collectively support a prediction, and we propose the DPS metric to measure its discrepancy from the class-average pattern. Empirical analyses across multiple
datasets and architectures show that, (i) decision patterns form a highly structured, class-consistent space with strong intra-class cohesion and low inter-class confusion, enabling direct analysis of a model's decision logic; (ii) the DPS magnitude correlates linearly with the generalization gap (nearly all Pearson r > 0.8), revealing generalization as a systematic drift in the model's internal decision mechanism; (iii) the DPS spectrum organizes diverse generalization degradation scenarios (covering ideal generalization, in-distribution degradation, domain shift, out-of-distribution, and shortcut learning) into a continuous trajectory, providing a unified explanation of their failure modes. These findings open up new possibilities for early generalization-risk detection, failure-mode diagnosis, and channel-level defect localization.
Read the full abstract ▾
All authors Huiqi Deng, Yibo Li, Quanshi Zhang, Peng Zhang, Hongbin Pei, Xia Hu
Computer Vision · Cross-listed from stat.ML
On Hallucinations in Inverse Problems: Fundamental Limits and Provable Assessment Methods
David Iagaru, Nina M. Gottschling, Anders C. Hansen et al. (+1 more)
Artificial intelligence (AI) has transformed imaging inverse problems, from medical diagnostics to Earth observation. Yet deep neural networks can produce
hallucinations, realistic-looking but incorrect details, undermining their reliability, especially when ground truth data is unavailable. We develop a theoretical framework showing that such
hallucinations are not merely artifacts of particular models, but can arise from the ill-posed nature of the inverse problem itself. We derive necessary and sufficient conditions for
hallucinations, together with computable bounds on their magnitude that depend only on the forward model. Building on this theory, we introduce
algorithms to: (1) estimate the minimum
hallucination magnitude achievable by any reconstruction model for a given input; (2) assess the faithfulness of reconstructed details by a given reconstruction model. Experiments across three imaging tasks demonstrate that our approach applies broadly, including to modern generative models, and provides a principled way to quantify and evaluate AI
hallucinations.
Read the full abstract ▾
All authors David Iagaru, Nina M. Gottschling, Anders C. Hansen, Josselin Garnier
Computer Vision
Multi-Modal Guided Multi-Source Domain Adaptation for Object Detection
Sangin Lee, Seokjun Kwon, Jeongmin Shin et al. (+2 more)
General object detection (OD) struggles to detect objects in the target domain that differ from the training distribution. To address this, recent studies demonstrate that training from multiple source domains and explicitly processing them separately for multi-source domain adaptation (MSDA) outperforms blending them for unsupervised domain adaptation (UDA). However, existing MSDA methods learn domain-agnostic features from domain-specific RGB images while preserving domain-specific information from the domain-agnostic feature map. To address this, we propose MS-DePro: Multi-Source Detector with Depth and
Prompt, composed of (1) depth-guided localization and (2) multi-modal guided
prompt learning. We leverage domain-agnostic input modalities, namely depth maps and text, to encode domain-agnostic characteristics. Specifically, we utilize depth maps to generate domain-agnostic region proposals for localization and integrate multi-modal features to align learnable text
embeddings for classification. MS-DePro achieves state-of-the-art performance on MSDA
benchmarks, and comprehensive ablations demonstrate the effectiveness of our contributions. Our code is available on https://github.com/sejong-rcv/Multi-Modal-Guided-Multi-Source-Domain-Adaptation-for-Object-Detection.
Read the full abstract ▾
All authors Sangin Lee, Seokjun Kwon, Jeongmin Shin, Namil Kim, Yukyung Choi
Computer Vision · Cross-listed from cs.GR
Rigel3D: Rig-aware Latents for Animation-Ready 3D Asset Generation
Nikitas Chatzis, Marios Loizou and Evangelos Kalogerakis
Recent 3D generative models can synthesize high-quality assets, but their outputs are typically static: they lack the skeletal rigs, joint hierarchies, and skinning weights required for animation. This limits their use in games, film, simulation, virtual
agents, and
embodied AI, where assets must not only look plausible but also move plausibly. We introduce Rigel3D, a generative method for animation-ready 3D assets represented as rigged meshes. Unlike post-hoc auto-rigging methods that attach rigs to completed shapes, our method jointly models geometry and rig structure through coupled surface and skeleton structured latent representations. A rig-aware autoencoder decodes these representations into mesh geometry, skeleton
topology, joint coordinates, and skinning weights, while a two-stage latent generative model synthesizes both surface and skeleton representations for image-conditioned generation. To support downstream animation workflows, we further introduce an open-vocabulary joint labeling module that embeds generated joints into a shared
vision-
language space, enabling correspondence to arbitrary retargeting templates. Experiments on large-scale rigged asset
datasets demonstrate that our method generates diverse, high-quality animation-ready assets and outperforms existing rigging baselines across multiple metrics.
Read the full abstract ▾
Computer Vision
Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation
Jingxuan He, Xiyu Wang, Yunke Wang et al. (+2 more)
Instruction-based image editing (IIE) models have recently demonstrated strong capability in modifying specific image regions according to natural
language instructions, which implicitly requires identifying where an edit should be applied. This indicates that such models inherently perform
language-conditioned visual semantic grounding. In this work, we investigate whether this implicit grounding can be leveraged for zero-shot referring image
segmentation (RIS), a task that requires pixel-level localization of objects described by natural
language expressions. Through systematic analysis, we reveal that strong foreground-background separability is already present in the internal representations of these models at the earliest denoising timestep, well before any visible image transformation occurs. Building on this insight, we propose a training-free framework that repurposes pretrained image editing models for RIS by exploiting their intermediate representations. Our approach decomposes localization into two complementary components:
attention-based spatial priors that estimate where to focus, and feature-based semantic discrimination that determines what to segment. By leveraging feature-space separability, the framework produces accurate
segmentation masks using only a single denoising step, without requiring full image synthesis. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate that our method achieves superior performance over existing zero-shot baselines. Project page: \url{https://jessie459.github.io/EarlySemanticGrounding/}.
Read the full abstract ▾
All authors Jingxuan He, Xiyu Wang, Yunke Wang, Mengyu Zheng, Chang Xu
Computer Vision · Cross-listed from cs.RO
Zixing Lei, Changxing Liu, Yichen Xiong et al. (+5 more)
Vision-
language-action (VLA) models are effective
robot action executors, but they remain limited on long-horizon tasks due to the dual burden of extended closed-loop planning and diverse physical operations. We therefore propose VLAs-as-Tools, a strategy that distributes this burden across a high-level
vision language model (VLM)
agent for temporal
reasoning and a family of specialized VLA tools for diverse local physical operations. The VLM handles scene analysis, global planning, and recovery, while each VLA tool executes a bounded subtask. To tightly couple
agent planning with VLA tool execution in long-horizon tasks, we introduce a VLA tool-family interface that exposes explicit tool selection and in-execution progress feedback, enabling efficient event-triggered
agent replanning without continuous
agent polling. To obtain diverse specialized VLA tools that faithfully follow
agent invocations, we further propose Tool-Aligned Post-Training (TAPT), which constructs invocation-aligned training units for instruction following and adopts tool-family residual adapters for efficient tool specialization. Across five simulated
benchmarks, VLAs-as-Tools improves the success rate of pi05 by 8.9 percentage points on average and enhances invocation fidelity by 15.0 points in Non-biased Rate. On three real-world Franka tasks, it improves success rate from 16.7% to 58.3% under matched data and compute. Code will be released.
Read the full abstract ▾
All authors Zixing Lei, Changxing Liu, Yichen Xiong, Minhao Xiong, Yuanzhuo Ding, Zhipeng Zhang, Weixin Li, Siheng Chen
Computer Vision
Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation
Jiayu Chen, Junbei Tang, Wenbiao Zhao et al. (+6 more)
Autoregressive
video generation enables streaming and open-ended long
video synthesis, but still suffers from long-term degradation caused by accumulated errors. Existing KVCache strategies usually apply unified historical-frame retention, implicitly assuming homogeneous historical dependencies across
attention heads. We revisit historical-frame
attention and reveal three distinct head types: Anchor Heads require broad long-range context, Wave Heads exhibit periodic temporal dependencies, and Veil Heads focus on initial and adjacent frames. Based on this finding, we propose Pyramid Forcing, a head-aware pyramidal KVCache framework that identifies head types offline, assigns behavior-specific cache policies, and supports heterogeneous cache lengths via efficient ragged-cache
attention. Experiments on Self Forcing and Causal Forcing show that Pyramid Forcing consistently improves long-horizon generation quality on VBench-Long, increasing the 60-second Self Forcing score from 77.87 to 81.21 while enhancing motion dynamics, visual fidelity, and semantic consistency. Project: https://if-lab-pku.github.io/Pyramid-Forcing/.
Read the full abstract ▾
All authors Jiayu Chen, Junbei Tang, Wenbiao Zhao, Maoliang Li, Jiayi Luo, Zihao Zheng, Jiawei Yang, Guojie Luo, Xiang Chen