Computer Vision
Omni-Weather: A Unified Multimodal Model for Weather Radar Understanding and Generation
Zhiwang Zhou, Yuandong Pu, Xuming He et al. (+10 more)
Weather modeling requires both accurate prediction and mechanistic interpretation, yet existing methods treat these goals in isolation, separating generation from understanding. To address this gap, we present Omni-Weather, the first
multimodal foundation model that unifies weather generation and understanding within a single architecture. Omni-Weather integrates a radar encoder for weather generation tasks, followed by unified processing using a shared self-
attention mechanism. Moreover, we construct a Chain-of-Thought
dataset for causal
reasoning in weather generation, enabling interpretable outputs and improved perceptual quality. Extensive experiments show Omni-Weather achieves state-of-the-art performance in both weather generation and understanding. Our findings further indicate that generative and understanding tasks in the weather domain can mutually enhance each other. Omni-Weather also demonstrates the feasibility and value of unifying weather generation and understanding.
Read the full abstract ▾
All authors Zhiwang Zhou, Yuandong Pu, Xuming He, Yidi Liu, Yixin Chen, Junchao Gong, Xiang Zhuang, Wanghan Xu, Qinglong Cao, Shixiang Tang, Yihao Liu, Wenlong Zhang, Lei Bai
Computer Vision
TrackTeller: Temporal Multimodal 3D Grounding for Behavior-Dependent Object References
Jiahong Yu, Ziqi Wang, Hailiang Zhao et al. (+3 more)
Understanding natural-
language references to objects in dynamic 3D driving scenes is essential for interactive autonomous systems. In practice, many referring expressions describe targets through recent motion or short-term interactions, which cannot be resolved from static appearance or geometry alone. We study temporal
language-based 3D grounding, where the objective is to identify the referred object in the current frame by leveraging multi-frame observations. We propose TrackTeller, a temporal
multimodal grounding framework that integrates
LiDAR-image fusion,
language-conditioned decoding, and temporal
reasoning in a unified architecture. TrackTeller constructs a shared UniScene representation aligned with textual semantics, generates
language-aware 3D proposals, and refines grounding decisions using motion history and short-term dynamics. Experiments on the NuPrompt
benchmark demonstrate that TrackTeller consistently improves
language-grounded tracking performance, outperforming strong baselines with a 70% relative improvement in Average Multi-Object Tracking Accuracy and a 3.15-3.4 times reduction in False Alarm Frequency.
Read the full abstract ▾
All authors Jiahong Yu, Ziqi Wang, Hailiang Zhao, Wei Zhai, Xueqiang Yan, Shuiguang Deng
Computer Vision
Training-Free Disentangled Text-Guided Image Editing via Sparse Latent Constraints
Mutiara Shabrina, Nova Kurnia Putri, Jefri Satria Ferdiansyah et al. (+2 more)
Text-driven image manipulation often suffers from attribute
entanglement, where modifying a target attribute (e.g., adding bangs) unintentionally alters other semantic properties such as identity or appearance. The Predict, Prevent, and Evaluate (PPE) framework addresses this issue by leveraging pre-trained
vision-
language models for disentangled editing. In this work, we analyze the PPE framework, focusing on its architectural components, including BERT-based attribute prediction and StyleGAN2-based image generation on the CelebA-HQ
dataset. Through empirical analysis, we identify a limitation in the original regularization strategy, where latent updates remain dense and prone to semantic leakage. To mitigate this issue, we introduce a sparsity-based constraint using L1 regularization on latent space manipulation. Experimental results demonstrate that the proposed approach enforces more focused and controlled edits, effectively reducing unintended changes in non-target attributes while preserving facial identity.
Read the full abstract ▾
All authors Mutiara Shabrina, Nova Kurnia Putri, Jefri Satria Ferdiansyah, Sabita Khansa Dewi, Novanto Yudistira
Computer Vision
SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration
Zhiyuan Liu, Daocheng Fu, Pinlong Cai et al. (+5 more)
High-fidelity and controllable 3D simulation is essential for addressing the long-tail data scarcity in Autonomous Driving (AD), yet existing methods struggle to simultaneously achieve photorealistic rendering and interactive traffic editing. Current approaches often falter in large-angle novel view synthesis and suffer from
geometric or lighting artifacts during asset manipulation. To address these challenges, we propose SymDrive, a unified
diffusion-based framework capable of joint high-quality rendering and scene editing. We introduce a Symmetric Auto-regressive Online Restoration paradigm, which constructs paired symmetric views to recover fine-grained details via a ground-truth-guided dual-view formulation and utilizes an auto-regressive strategy for consistent lateral view generation. Furthermore, we leverage this restoration capability to enable a training-free harmonization mechanism, treating vehicle insertion as context-aware inpainting to ensure seamless lighting and shadow consistency. Extensive experiments demonstrate that SymDrive achieves state-of-the-art performance in both novel-view enhancement and realistic 3D vehicle insertion.
Read the full abstract ▾
All authors Zhiyuan Liu, Daocheng Fu, Pinlong Cai, Lening Wang, Ying Liu, Yilong Ren, Botian Shi, Jianqiang Wang
Computer Vision
CausalFSFG: Rethinking Few-Shot Fine-Grained Visual Categorization from Causal Perspective
Zhiwen Yang, Jinglin Xu and Yuxin Pen
Few-shot fine-grained visual categorization (FS-FGVC) focuses on identifying various subcategories within a common superclass given just one or few support examples. Most existing methods aim to boost classification accuracy by enriching the extracted features with discriminative part-level details. However, they often overlook the fact that the set of support samples acts as a confounding variable, which hampers the FS-FGVC performance by introducing biased data distribution and misguiding the extraction of discriminative features. To address this issue, we propose a new causal FS-FGVC (CausalFSFG) approach inspired by causal inference for addressing biased data distributions through causal intervention. Specifically, based on the structural causal model (SCM), we argue that FS-FGVC infers the subcategories (i.e., effect) from the inputs (i.e., cause), whereas both the few-shot condition disturbance and the inherent fine-grained nature (i.e., large intra-class variance and small inter-class variance) lead to unobservable variables that bring spurious correlations, compromising the final classification performance. To further eliminate the spurious correlations, our CausalFSFG approach incorporates two key components: (1) Interventional multi-scale encoder (IMSE) conducts sample-level interventions, (2) Interventional masked feature reconstruction (IMFR) conducts feature-level interventions, which together reveal real causalities from inputs to subcategories. Extensive experiments and thorough analyses on the widely-used public
datasets, including CUB-200-2011, Stanford Dogs, and Stanford Cars, demonstrate that our CausalFSFG achieves new state-of-the-art performance. The code is available at https://github.com/PKU-ICST-MIPL/CausalFSFG_TMM.
Read the full abstract ▾
Computer Vision
TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistant
Rongpei Hong, Jian Lang, Ting Zhong et al. (+2 more)
Multimodal Large
Language Model (MLLM) Personalization is a critical research problem that facilitates personalized dialogues with MLLMs targeting specific entities (known as personalized concepts). However, existing methods and
benchmarks focus on the simple, context-agnostic visual identification and textual replacement of the personalized concept (e.g., "A yellow puppy" -> "Your puppy Mochi"), overlooking the ability to support long-context conversations. An ideal personalized MLLM assistant is capable of engaging in long-context dialogues with humans and continually improving its experience quality by learning from past dialogue histories. To bridge this gap, we propose LCMP, the first Long-Context MLLM Personalization evaluation
benchmark. LCMP assesses the capability of MLLMs in perceiving variations of personalized concepts and generating contextually appropriate personalized responses that reflect these variations. As a strong baseline for LCMP, we introduce a novel training-free and state-aware framework TAME. TAME endows MLLMs with double
memories to manage the temporal and persistent variations of each personalized concept in a differentiated manner. In addition, TAME incorporates a new training-free Retrieve-then-Align Augmented Generation (RA2G) paradigm. RA2G introduces an
alignment step to extract the contextually fitted information from the multi-
memory retrieved knowledge to the current questions, enabling better interactions for complex real-world user queries. Experiments on LCMP demonstrate that TAME achieves the best performance, showcasing remarkable and evolving interaction experiences in long-context scenarios.
Read the full abstract ▾
All authors Rongpei Hong, Jian Lang, Ting Zhong, Yong Wang, Fan Zhou
Computer Vision · Cross-listed from cs.LG
An Empirical Study of Machine Learning Robustness and Scalability for Imbalanced Tabular Clinical Data in Emergency and Critical Care
Yusuf Brima and Marcellin Atemkeng
Every year, millions of patients pass through emergency departments and intensive care units, where clinicians must make high-stakes decisions under time pressure and uncertainty. Machine learning could support prediction of deterioration, triage, and rare critical outcomes, but clinical data are often severely imbalanced, biasing models toward majority classes and reducing predictive performance. Developing robust and efficient models for imbalanced clinical tabular data therefore remains an important challenge. We evaluated six model families on imbalanced tabular data from the MIMIC-IV-ED and eICU databases: Decision Tree, Random Forest, XGBoost, TabNet, TabICL, and TabPFN v2.6. Trainable models were optimized using
Bayesian hyperparameter tuning, while foundation models were evaluated in their pretrained inference regime without task-specific reweighting. Models were assessed using Macro F1-score, robustness to increasing imbalance, and computational scalability across seven clinical prediction tasks. Results differed across
datasets. On MIMIC-IV-ED, TabPFN v2.6 and TabICL achieved the strongest average Macro F1 ranks, with XGBoost remaining competitive. On eICU, XGBoost consistently performed best, followed by other tree-based methods, while foundation models achieved intermediate performance. Across both
datasets, TabNet showed the largest degradation under increasing imbalance and the highest computational cost. Training-time analysis showed that tree-based methods scaled most favorably with
dataset size, while foundation models offered low per-task adaptation cost. These findings suggest that no single model family dominates across all clinical settings. However, tabular foundation models are narrowing the performance gap with strong classical baselines while offering a distinct efficiency-performance trade-off that may benefit resource-constrained clinical environments.
Read the full abstract ▾
Computer Vision
Resolving compositional and conformational heterogeneity in cryo-EM with deformable 3D Gaussian representations
Bintao He, Yiran Cheng, Hongjia Li et al. (+4 more)
Understanding
protein flexibility and its dynamic interactions with other molecules is essential for studying
protein function. Although cryogenic electron microscopy(cryo-EM) provides an opportunity to observe macromolecular dynamics directly, computational analysis of
datasets mixing continuous and discrete structural states remains a formidable challenge. Here we introduce GaussianEM, a Gaussian-based pseudo-atomic framework that simultaneously resolves compositional and conformational heterogeneity from cryo-EM images. GaussianEM employs a dual-encoder-single-decoder architecture to decompose images into learnable Gaussian components, with variability encoded through modulated parameters. This explicit parameterization yields a continuous, intuitive representation of conformational dynamics that inherently preserves local structural integrity. By modeling displacements in Gaussian space, we capture atomic-scale conformational landscapes, bridging density maps and all-atom models. In comprehensive experiments, GaussianEM successfully reconstructs complex compositional and conformational variability,and resolves previously unobserved details in public
datasets. Quantitative evaluations further confirm its ability to capture broader conformational diversity without sacrificing structural fidelity.
Read the full abstract ▾
All authors Bintao He, Yiran Cheng, Hongjia Li, Xiang Gao, Xin Gao, Fa Zhang, Renmin Han
Computer Vision
From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvement
Jian Lang, Rongpei Hong, Ting Zhong et al. (+3 more)
The proliferation of harmful memes on online media poses significant risks to public health and stability. Existing detection methods heavily rely on large-scale labeled data for training, which necessitates substantial manual annotation efforts and limits their adaptability to the continually evolving nature of harmful content. To address these challenges, we present ALARM, the first lAbeL-free hARmful Meme detection framework powered by Large
Multimodal Model (LMM)
agent self-improvement. The core innovation of ALARM lies in exploiting the expressive information from "shallow" memes to iteratively enhance its ability to tackle more complex and subtle ones. ALARM consists of a novel Confidence-based Explicit Meme Identification mechanism that isolates the explicit memes from the original
dataset and assigns them pseudo-labels. Besides, a new Pairwise Learning Guided
Agent Self-Improvement paradigm is introduced, where the explicit memes are reorganized into contrastive pairs (positive vs. negative) to refine a learner LMM
agent. This
agent autonomously derives high-level detection cues from these pairs, which in turn empower the
agent itself to handle complex and challenging memes effectively. Experiments on three diverse
datasets demonstrate the superior performance and strong adaptability of ALARM to newly evolved memes. Notably, our method even outperforms label-driven methods. These results highlight the potential of label-free frameworks as a scalable and promising solution for adapting to novel forms and topics of harmful memes in dynamic online environments.
Read the full abstract ▾
All authors Jian Lang, Rongpei Hong, Ting Zhong, Leiting Chen, Qiang Gao, Fan Zhou
Computer Vision · Cross-listed from stat.ML
Residual Prior Diffusion: A Probabilistic Framework Integrating Coarse Latent Priors with Diffusion Models
Takuro Kutsuna
Diffusion models have become a central tool in deep generative modeling, but standard formulations rely on a single network and a single
diffusion schedule to transform a simple prior, typically a standard normal distribution, into the target data distribution. As a result, the model must simultaneously represent the global structure of the distribution and its fine-scale local variations, which becomes difficult when these scales are strongly mismatched. This issue arises both in natural images, where coarse
manifold-level structure and fine textures coexist, and in low-dimensional distributions with highly concentrated local structure. To address this issue, we propose Residual Prior
Diffusion (RPD), a two-stage framework in which a coarse prior model first captures the large-scale structure of the data distribution, and a
diffusion model is then trained to represent the residual between the prior and the target data distribution. We formulate RPD as an explicit probabilistic model with a tractable evidence lower bound, whose optimization reduces to the familiar objectives of noise prediction or velocity prediction. We further introduce auxiliary variables that leverage information from the prior model and theoretically analyze how they reduce the difficulty of the prediction problem in RPD. Experiments on synthetic
datasets with fine-grained local structure show that standard
diffusion models fail to capture local details, whereas RPD accurately captures fine-scale detail while preserving the large-scale structure of the distribution. On natural image generation tasks, RPD achieved generation quality that matched or exceeded that of representative
diffusion-based baselines and it maintained strong performance even with a small number of inference steps.
Read the full abstract ▾
Computer Vision
UltraLBM-UNet: Ultralight Bidirectional Mamba-based Model for Skin Lesion Segmentation
Linxuan Fan, Juntao Jiang, Weixuan Liu et al. (+4 more)
Skin lesion
segmentation is a crucial step in dermatology for guiding clinical decision-making. However, existing methods for accurate, robust, and resource-efficient lesion analysis have limitations, including low performance and high computational complexity. To address these limitations, we propose UltraLBM-UNet, a lightweight U-Net variant that integrates a bidirectional Mamba-based global modeling mechanism with multi-branch local feature perception. The proposed architecture integrates efficient local feature injection with bidirectional state-space modeling, enabling richer contextual interaction across spatial dimensions while maintaining computational compactness suitable for point-of-care deployment. Extensive experiments on the ISIC 2017, ISIC 2018, and PH2
datasets demonstrate that our model consistently achieves state-of-the-art
segmentation accuracy, outperforming existing lightweight and Mamba counterparts with only 0.034M parameters and 0.060 GFLOPs. In addition, we introduce a hybrid knowledge
distillation strategy to train an ultra-compact student model, where the distilled variant UltraLBM-UNet-T, with only 0.011M parameters and 0.019 GFLOPs, achieves competitive
segmentation performance. These results highlight the suitability of UltraLBM-UNet for point-of-care deployment, where accurate and robust lesion analyses are essential. The source code is publicly available at https://github.com/LinLinLin-X/UltraLBM-UNet.
Read the full abstract ▾
All authors Linxuan Fan, Juntao Jiang, Weixuan Liu, Zhucun Xue, Jiajun Lv, Jiangning Zhang, Yong Liu
Computer Vision
LLM-Free Image Captioning Evaluation in Reference-Flexible Settings
Shinnosuke Hirano, Yuiga Wada, Kazuki Matsuda et al. (+2 more)
We focus on the automatic evaluation of image captions in both reference-based and reference-free settings. Existing metrics based on large
language models (LLMs) favor their own generations; therefore, the neutrality is in question. Most LLM-free metrics do not suffer from such an issue, whereas they do not always demonstrate high performance. To address these issues, we propose Pearl, an LLM-free supervised metric for image captioning, which is applicable to both reference-based and reference-free settings. We introduce a novel mechanism that learns the representations of image–caption and caption–caption similarities. Furthermore, we construct a human-annotated
dataset for image captioning metrics, that comprises approximately 333k human judgments collected from 2,360 annotators across over 75k images. Pearl outperformed other existing LLM-free metrics on the Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, and FOIL
datasets in both reference-based and reference-free settings. Our project page is available at https://pearl.kinsta.page/.
Read the full abstract ▾
All authors Shinnosuke Hirano, Yuiga Wada, Kazuki Matsuda, Seitaro Otsuki, Komei Sugiura
Computer Vision
Haoyi Zhou, Shuo Li, Tianyu Chen et al. (+3 more)
While large
vision-
language models (VLMs) demonstrate strong long-context understanding, their prevalent small branches fail on linguistics-photography
alignment for a limited window size. We discover that knowledge
distillation improves students' capability as a complement to Rotary Position
Embeddings (RoPE) on window sizes (anchored from large models). Building on this insight, we propose LAid, which directly aims at the transfer of long-range
attention mechanisms through two complementary components: (1) a progressive distance-weighted
attention matching that dynamically emphasizes longer position differences during training, and (2) a learnable RoPE response gain modulation that selectively amplifies position sensitivity where needed. Extensive experiments across multiple model families demonstrate that LAid-distilled models achieve up to 3.2 times longer effective context windows compared to baseline small models, while maintaining or improving performance on standard VL
benchmarks. Spectral analysis also suggests that LAid successfully preserves crucial low-frequency
attention components that conventional methods fail to transfer. Our work not only provides practical techniques for building more efficient long-context VLMs but also offers theoretical insights into how positional understanding emerges and transfers during
distillation.
Read the full abstract ▾
All authors Haoyi Zhou, Shuo Li, Tianyu Chen, Qi Song, Chonghan Gao, Jianxin Li
Computer Vision
Exploration of Reproducible Generated Image Detection
Yihang Duan
While the technology for detecting AI-Generated Content (AIGC) images has advanced rapidly, the field still faces two core issues: poor reproducibility and insufficient gen eralizability, which hinder the practical application of such technologies. This study addresses these challenges by re viewing 7 key papers on AIGC detection, constructing a lightweight test
dataset, and reproducing a representative detection method. Through this process, we identify the root causes of the reproducibility dilemma in the field: firstly, papers often omit implicit details such as prepro cessing steps and parameter settings; secondly, most detec tion methods overfit to exclusive features of specific gener ators rather than learning universal intrinsic features of AIGC images. Experimental results show that basic perfor mance can be reproduced when strictly following the core procedures described in the original papers. However, de tection performance drops sharply when preprocessing dis rupts key features or when testing across different genera tors. This research provides empirical evidence for improv ing the reproducibility of AIGC detection technologies and offers reference directions for researchers to disclose ex perimental details more comprehensively and verify the generalizability of their proposed methods.
Read the full abstract ▾
Computer Vision
Toward Intelligent Scene Augmentation for Context-Aware Object Placement and Sponsor-Logo Integration
Unnati Saraswat, Tarun Rao, Namah Gupta et al. (+4 more)
Intelligent image editing increasingly relies on advances in computer
vision,
multimodal reasoning, and generative modeling. While
vision-
language models (VLMs) and
diffusion models enable guided visual manipulation, existing work rarely ensures that inserted objects are contextually appropriate. We introduce two new tasks for advertising and digital media: (1) context-aware object insertion, which requires predicting suitable object categories, generating them, and placing them plausibly within the scene; and (2) sponsor-product logo augmentation, which involves detecting products and inserting correct brand logos, even when items are unbranded or incorrectly branded. To support these tasks, we build two new
datasets with category annotations, placement regions, and sponsor-product labels.
Read the full abstract ▾
All authors Unnati Saraswat, Tarun Rao, Namah Gupta, Shweta Swami, Shikhar Sharma, Prateek Narang, Dhruv Kumar
Computer Vision
EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal
Sanghyun Jo, Donghwan Lee, Eunji Jung et al. (+2 more)
Object removal must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity, rather than merely filling a hole plausibly. Recent
dataset-free approaches manipulate the
diffusion model's internal self-
attention to prevent it from referencing the masked region, yet they fail in two critical ways: (i) they treat the masked region as the sole foreground, misinterpreting non-target objects as background and regenerating them, and (ii) they apply uniform
attention constraints without distinguishing diverse background subtypes, leading to textural blurring and structural misalignment. Both failures stem from the absence of explicit background-aware
reasoning. We propose EraseLoRA, a
dataset-free framework that replaces
attention surgery with background-aware
reasoning and test-time adaptation. The first stage, Background-aware Foreground Exclusion (BFE), leverages a
multimodal large-
language model to separate target foreground, non-target foregrounds, and clean background from a single image-mask pair. The second stage, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as complementary pieces, enforcing their consistent integration through reconstruction and
alignment objectives without explicit
attention intervention. As a model-agnostic plug-in applicable to diverse
diffusion backbones, EraseLoRA reconstructs backgrounds at least 23% more faithful to the original scene than previous
dataset-free methods while nearly halving unwanted foreground re-generation, and surpasses all
dataset-driven approaches in both aspects despite requiring no training data. Code is available at https://shjo-april.github.io/EraseLoRA.
Read the full abstract ▾
All authors Sanghyun Jo, Donghwan Lee, Eunji Jung, Seong Je Oh, Kyungsu Kim
Computer Vision
Dongchen Han, Tianyu Li, Ziyi Wang et al. (+1 more)
The self-
attention mechanism has been a key factor in the advancement of
vision Transformers. However, its quadratic complexity imposes a heavy computational burden in high-resolution scenarios, restricting the practical application. Previous methods attempt to mitigate this issue by introducing handcrafted patterns such as locality or sparsity, which inevitably compromise model capacity. In this paper, we present a novel
attention paradigm termed Circulant
Attention by exploiting the inherent efficient pattern of self-
attention. Specifically, we first identify that the self-
attention matrix in
vision Transformers often approximates the Block Circulant
matrix with Circulant Blocks (BCCB), a kind of structured
matrix whose multiplication with other
matrices can be performed in $\mathcal{O}(N\log N)$ time. Leveraging this interesting pattern, we explicitly model the
attention map as its nearest BCCB
matrix and propose an efficient computation
algorithm for fast calculation. The resulting approach closely mirrors vanilla self-
attention, differing only in its use of BCCB
matrices. Since our design is inspired by the inherent efficient paradigm, it not only delivers $\mathcal{O}(N\log N)$ computation complexity, but also largely maintains the capacity of standard self-
attention. Extensive experiments on diverse visual tasks demonstrate the effectiveness of our approach, establishing circulant
attention as a promising alternative to self-
attention for
vision Transformer architectures. Code is available at https://github.com/LeapLabTHU/Circulant-
Attention.
Read the full abstract ▾
All authors Dongchen Han, Tianyu Li, Ziyi Wang, Gao Huang
Computer Vision
Hierarchy-Aware Fine-Tuning of Vision-Language Models
Jiayu Li, Rajesh Gangireddy, Samet Akcay et al. (+2 more)
Vision-
Language Models (VLMs) learn powerful
multimodal representations through large-scale image-text pretraining, but adapting them to hierarchical classification is underexplored. Standard approaches treat labels as flat categories and require full fine-tuning, which is expensive and produces inconsistent predictions across taxonomy levels. We propose an efficient hierarchy-aware fine-tuning framework that updates a few parameters while enforcing structural consistency. We combine two objectives: Tree-Path KL Divergence (TP-KL) aligns predictions along the ground-truth label path for vertical coherence, while Hierarchy-Sibling Smoothed Cross-
Entropy (HiSCE) encourages consistent predictions among sibling classes. Both losses work in the VLM's shared
embedding space and integrate with lightweight LoRA adaptation. Experiments across multiple
benchmarks show consistent improvements in Full-Path Accuracy and Tree-based Inconsistency Error with minimal parameter overhead. Our approach provides an efficient strategy for adapting VLMs to structured taxonomies.
Read the full abstract ▾
All authors Jiayu Li, Rajesh Gangireddy, Samet Akcay, Wei Cheng, Juhua Hu
Computer Vision · Cross-listed from cs.LG
Global-Graph Guided and Local-Graph Weighted Contrastive Learning for Unified Clustering on Incomplete and Noise Multi-View Data
Hongqing He, Jie Xu, Wenyuan Yang et al. (+3 more)
Recently, contrastive learning (CL) plays an important role in exploring complementary information for multi-view clustering (MVC) and has attracted increasing
attention. Nevertheless, real-world multi-view data suffer from data incompleteness or noise, resulting in rare-paired samples or mis-paired samples which significantly challenges the effectiveness of CL-based MVC. That is, rare-paired issue prevents MVC from extracting sufficient multi-view complementary information, and mis-paired issue causes contrastive learning to optimize the model in the wrong direction. To address these issues, we propose a unified CL-based MVC framework for enhancing clustering effectiveness on incomplete and noise multi-view data. First, to overcome the rare-paired issue, we design a global-graph guided contrastive learning, where all view samples construct a global-view affinity graph to form new sample pairs for fully exploring complementary information. Second, to mitigate the mis-paired issue, we propose a local-graph weighted contrastive learning, which leverages local neighbors to generate pair-wise weights to adaptively strength or weaken the pair-wise contrastive learning. Our method is imputation-free and can be integrated into a unified global-local graph-guided contrastive learning framework. Extensive experiments on both incomplete and noise settings of multi-view data demonstrate that our method achieves superior performance compared with state-of-the-art approaches.
Read the full abstract ▾
All authors Hongqing He, Jie Xu, Wenyuan Yang, Yonghua Zhu, Guoqiu Wen, Xiaofeng Zhu
Computer Vision
DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPO
Henglin Liu, Huijuan Huang, Jing Wang et al. (+3 more)
Reinforcement learning (RL), particularly GRPO, improves image generation quality significantly by comparing the relative performance of images generated within the same group. However, in the later stages of training, the model tends to produce homogenized outputs, lacking creativity and visual diversity, which restricts its application scenarios. This issue can be analyzed from both reward modeling and generation dynamics perspectives. First, traditional GRPO relies on single-sample quality as the reward signal, driving the model to converge toward a few high-reward generation modes while neglecting distribution-level diversity. Second, conventional GRPO regularization neglects the dominant role of early-stage denoising in preserving diversity, causing a misaligned regularization budget that limits the achievable quality–diversity trade-off. Motivated by these insights, we revisit the diversity degradation problem from both reward modeling and generation dynamics. At the reward level, we propose a distributional creativity bonus based on semantic grouping. Specifically, we construct a distribution-level representation via spectral clustering over samples generated from the same caption, and adaptively allocate exploratory rewards according to group sizes to encourage the discovery of novel visual modes. At the generation level, we introduce a structure-aware regularization, which enforces stronger early-stage constraints to preserve diversity without compromising reward optimization efficiency. Experiments demonstrate that our method achieves a 13%–18% improvement in semantic diversity under matched quality scores, establishing a new Pareto frontier between image quality and diversity for GRPO-based image generation.
Read the full abstract ▾
All authors Henglin Liu, Huijuan Huang, Jing Wang, Chang Liu, Xiu Li, Xiangyang Ji
Computer Vision
MuS-Polar3D: A Benchmark Dataset for Computational Polarimetric 3D Imaging under Multi-Scattering Conditions
Puyun Wang, Kaimin Yu, Huayang He et al. (+1 more)
Polarization-based
underwater 3D imaging exploits polarization cues to suppress background scattering, exhibiting distinct advantages in turbid water. Although data-driven polarization-based
underwater 3D reconstruction methods show great potential, existing public
datasets lack sufficient diversity in scattering and observation conditions, hindering fair comparisons among different approaches, including single-view and multi-view polarization imaging methods. To address this limitation, we construct MuS-Polar3D, a
benchmark dataset comprising polarization images of 42 objects captured under seven quantitatively controlled scattering conditions and five viewpoints, together with high-precision 3D models (+/- 0.05 mm accuracy), normal maps, and foreground masks. The
dataset supports multiple
vision tasks, including normal estimation, object
segmentation, descattering, and 3D reconstruction. Inspired by computational imaging, we further decouple
underwater 3D reconstruction under scattering into a two-stage pipeline, namely descattering followed by 3D reconstruction, from an imaging-chain perspective. Extensive evaluations using multiple baseline methods under complex scattering conditions demonstrate the effectiveness of the proposed
benchmark, achieving a best mean angular error of 15.49 degrees. To the best of our knowledge, MuS-Polar3D is the first publicly available
benchmark dataset for quantitative turbidity
underwater polarization-based 3D imaging, enabling accurate reconstruction and fair
algorithm evaluation under controllable scattering conditions. The
dataset and code are publicly available at https://github.com/WangPuyun/MuS-Polar3D.
Read the full abstract ▾
All authors Puyun Wang, Kaimin Yu, Huayang He, Xianyu Wu
Computer Vision
Fixed-Threshold Evaluation of a Hybrid CNN-ViT for AI-Generated Image Detection Across Photos and Art
Md Ashik Khan and Arafat Alam Jion
AI image generators create both photorealistic images and stylized art, necessitating robust detectors that maintain performance under common post-processing transformations (JPEG compression, blur, downscaling). Existing methods optimize single metrics without addressing deployment-critical factors such as operating point selection and fixed-threshold robustness. This work addresses misleading robustness estimates by introducing a fixed-threshold evaluation protocol that holds decision thresholds, selected once on clean validation data, fixed across all post-processing transformations. Traditional methods retune thresholds per condition, artificially inflating robustness estimates and masking deployment failures. We report deployment-relevant performance at three operating points (Low-FPR, ROC-optimal, Best-F1) under systematic degradation testing using a lightweight CNN-ViT hybrid with gated fusion and optional frequency enhancement. Our evaluation exposes a statistically validated forensic-semantic spectrum: frequency-aided CNNs excel on pristine photos but collapse under compression (93.33% to 61.49%), whereas ViTs degrade minimally (92.86% to 88.36%) through robust semantic pattern recognition. Multi-seed experiments demonstrate that all architectures achieve 15% higher AUROC on artistic content (0.901-0.907) versus photorealistic images (0.747-0.759), confirming that semantic patterns provide fundamentally more reliable detection cues than forensic artifacts. Our hybrid approach achieves balanced cross-domain performance: 91.4% accuracy on tiny-genimage photos, 89.7% on AiArtData art/graphics, and 98.3% (competitive) on CIFAKE. Fixed-threshold evaluation eliminates retuning inflation, reveals genuine robustness gaps, and yields actionable deployment guidance: prefer CNNs for clean photo verification, ViTs for compressed content, and hybrids for art/graphics screening.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Missing Pattern Tree based Decision Grouping and Ensemble for Enhancing Pair Utilization in Deep Incomplete Multi-View Clustering
Jie Xu, Wenyuan Yang, Yazhou Ren et al. (+3 more)
Real-world multi-view data often exhibit highly inconsistent missing patterns, posing significant challenges for incomplete multi-view clustering (IMVC). Although existing IMVC methods have made progress from both imputation-based and imputation-free routes, they largely overlook the issue of pair underutilization. Specifically, inconsistent missing patterns prevent incomplete but available multi-view pairs from being fully exploited, thereby limiting the model performance. To address this limitation, we propose a novel missing-pattern tree based IMVC framework. Specifically, to fully leverage available multi-view pairs, we first introduce a missing-pattern tree model to group data into multiple decision sets according to their missing patterns, and then perform multi-view clustering within each set. Furthermore, a multi-view decision ensemble module is proposed to aggregate clustering results across all decision sets. This module infers uncertainty-based weights to suppress unreliable clustering decisions and produce robust outputs. Finally, we develop an ensemble-to-individual knowledge
distillation module module, which transfers ensemble knowledge to view-specific clustering models. This design enables mutual enhancement between ensemble and individual modules by optimizing cross-view consistency and inter-cluster discrimination losses. Extensive theoretical analysis supports our key designs, and empirical experiments on multiple
benchmark datasets demonstrate that our method effectively mitigates the pair underutilization issue and achieve superior IMVC performance.
Read the full abstract ▾
All authors Jie Xu, Wenyuan Yang, Yazhou Ren, Lifang He, Philip S. Yu, Xiaofeng Zhu
Computer Vision
Fixed-Budget Parameter-Efficient Training with Frozen Encoders Improves Multimodal Chest X-Ray Classification
Md Ashik Khan and Md Nahid Siddique
Multimodal chest X-Ray analysis often fine-tunes large
vision-
language models, which is computationally costly. We study parameter-efficient training (PET) strategies, including frozen encoders, BitFit, LoRA, and adapters for multi-label classification on the Indiana University Chest X-Ray
dataset (3,851 image-report pairs; 579 test samples). To mitigate data leakage, we redact pathology terms from reports used as text inputs while retaining clinical context. Under a fixed parameter budget (2.37M parameters, 2.51% of total), all PET variants achieve AUROC between 0.892 and 0.908, outperforming full fine-tuning (0.770 AUROC), which uses 94.3M trainable parameters, a 40x reduction. External validation on CheXpert (224,316 images, 58x larger) confirms scalability: all PET methods achieve >0.69 AUROC with <9% trainable parameters, with Adapter achieving best performance (0.7214 AUROC). Budget-matched comparisons reveal that
vision-only models (0.653 AUROC, 1.06M parameters) outperform budget-matched
multimodal models (0.641 AUROC, 1.06M parameters), indicating improvements arise primarily from parameter allocation rather than cross-modal synergy. While PET methods show degraded calibration (ECE: 0.29-0.34) compared to simpler models (ECE: 0.049), this represents a tractable limitation addressable through post-hoc calibration methods. These findings demonstrate that frozen encoder strategies provide superior discrimination at substantially reduced computational cost, though calibration correction is essential for clinical deployment.
Read the full abstract ▾
Computer Vision
SVBench: Evaluation of Video Generation Models on Social Reasoning
Wenshuo Peng, Gongxuan Wang, Tianmeng Yang et al. (+4 more)
Recent text-to-
video generation models have made remarkable progress in visual realism, motion fidelity, and text-
video alignment, yet they still struggle to produce socially coherent behavior. Unlike humans, who readily infer intentions, beliefs,
emotions, and social norms from brief visual cues, current models often generate literal scenes without capturing the underlying causal and psychological dynamics. To systematically assess this limitation, we introduce the first
benchmark for social
reasoning in
video generation. Grounded in developmental and social psychology, the
benchmark covers thirty classic social cognition paradigms spanning seven core dimensions: mental-state inference, goal-directed action, joint
attention, social coordination, prosocial behavior, social norms, and multi-
agent strategy. To operationalize these paradigms, we build a fully training-free
agent-based pipeline that distills the
reasoning structure of each paradigm, synthesizes diverse
video-ready scenarios, enforces conceptual neutrality and difficulty control through cue-based critique, and evaluates generated
videos with a high-capacity VLM judge along five interpretable dimensions of social
reasoning. Using this framework, we conduct the first large-scale evaluation of seven state-of-the-art
video generation systems. Results show a clear gap between surface-level plausibility and deeper social
reasoning, suggesting that current models remain limited in their ability to generate socially grounded behavior. https://github.com/Gloria2tt/SVBench-Evaluation
Read the full abstract ▾
All authors Wenshuo Peng, Gongxuan Wang, Tianmeng Yang, Chuanhao Li, Xiaojie Xu, Hui He, Kaipeng Zhang
Computer Vision
Generative Multi-Focus Image Fusion
Xinzhe Xie, Buyu Guo, Bolin Li et al. (+4 more)
Multi-focus image fusion aims to generate an all-in-focus image from a sequence of partially focused input images. Existing fusion
algorithms generally assume that, for every spatial location in the scene, there is at least one input image in which that location is in focus. Furthermore, current fusion models often suffer from edge artifacts caused by uncertain focus estimation or hard-selection operations in complex real-world scenarios. To address these limitations, we propose a generative multi-focus image fusion framework, termed GMFF, which operates in two sequential stages. In the first stage, deterministic fusion is implemented using StackMFF V4, the latest version of the StackMFF series, and integrates the available focal plane information to produce an initial fused image. The second stage, generative restoration, is realized through IFControlNet, which leverages the generative capabilities of latent
diffusion models to reconstruct content from missing focal planes, restore fine details, and eliminate edge artifacts. Each stage is independently developed and functions seamlessly in a cascaded manner. Extensive experiments demonstrate that GMFF achieves state-of-the-art fusion performance and exhibits significant potential for practical applications, particularly in scenarios involving complex multi-focal content. The implementation is publicly available at https://github.com/Xinzhe99/StackMFF-Series.
Read the full abstract ▾
All authors Xinzhe Xie, Buyu Guo, Bolin Li, Shuangyan He, Yanzhen Gu, Qingyan Jiang, Peiliang Li
Computer Vision
GPF-Net: Gated Progressive Fusion Learning for Polyp Re-Identification
Suncheng Xiang, Xiaoyang Wang, Junjie Jiang et al. (+2 more)
Colonoscopic Polyp Re-Identification (ReID) aims to match the same polyp across a large gallery of images captured from different viewpoints and with different cameras, playing a critical role in computer-aided diagnosis for the prevention and treatment of colorectal cancer. However, the coarse granularity of high-level features often limits performance on small polyps, where fine-grained details are essential for accurate matching. To address this challenge, we propose a novel
multimodal feature fusion architecture, termed the Gated Progressive Fusion Network, which selectively integrates features from multiple levels through fully connected gating mechanisms. Building on this framework, we introduce a gated progressive fusion strategy that enables layer-wise refinement of semantic information, facilitating multi-level feature interactions to enhance both generalization ability and robustness. Extensive experiments on standard
benchmarks demonstrate the advantages of the
multimodal setting over state-of-the-art unimodal ReID models, particularly when combined with the proposed fusion strategy tailored for general-purpose scenarios.
Read the full abstract ▾
All authors Suncheng Xiang, Xiaoyang Wang, Junjie Jiang, Hejia Wang, Dahong Qian
Computer Vision
IMA++: ISIC Archive Multi-Annotator Dermoscopic Skin Lesion Segmentation Dataset
Kumar Abhishek, Jeremy Kawahara and Ghassan Hamarneh
Multi-annotator medical image
segmentation is an important research problem, but requires annotated
datasets that are expensive to collect. Dermoscopic skin lesion imaging allows human experts and AI systems to observe morphological structures otherwise not discernable from regular clinical photographs. However, currently there are no large-scale publicly available multi-annotator skin lesion
segmentation (SLS)
datasets with annotator-labels for dermoscopic skin lesion imaging. We introduce ISIC MultiAnnot++, a large public multi-annotator skin lesion
segmentation dataset for images from the ISIC Archive. The final
dataset contains 17,684
segmentation masks spanning 14,967 dermoscopic images, where 2,394 dermoscopic images have 2-5
segmentations per image, making it the largest publicly available SLS
dataset. Further, metadata about the
segmentation, including the annotators' skill level and
segmentation tool, is included, enabling research on topics such as annotator-specific preference modeling for
segmentation and annotator metadata analysis. We provide an analysis on the characteristics of this
dataset, curated data partitions, and consensus
segmentation masks.
Read the full abstract ▾
Computer Vision
CCAD: Compressed Global Feature Conditioned Anomaly Detection
Xiao Jin, Liang Diao, Qixin Xiao et al. (+4 more)
Anomaly detection holds considerable industrial significance, especially in scenarios with limited anomalous data. Currently, reconstruction-based and unsupervised representation-based approaches are the primary focus. However, unsupervised representation-based methods struggle to extract robust features under domain shift, whereas reconstruction-based methods often suffer from low training efficiency and performance degradation due to insufficient constraints. To address these challenges, we propose a novel method named Compressed Global Feature Conditioned Anomaly Detection (CCAD). CCAD synergizes the strengths of both paradigms by adapting global features as a new modality condition for the reconstruction model. Furthermore, we design an adaptive compression mechanism to enhance both generalization and training efficiency. Extensive experiments demonstrate that CCAD consistently outperforms state-of-the-art methods in terms of AUC while achieving faster convergence. In addition, we contribute a reorganized and re-annotated version of the DAGM 2007
dataset with new annotations to further validate our method's effectiveness. The code for reproducing main results is available at https://github.com/chloeqxq/CCAD.
Read the full abstract ▾
All authors Xiao Jin, Liang Diao, Qixin Xiao, Yifan Hu, Ziqi Zhang, Yuchen Liu, Haisong Gu
Computer Vision
Intelligent recognition of GPR road hidden defect images based on feature fusion and attention mechanism
Haotian Lv, Yuhui Zhang, Jiangbo Dai et al. (+3 more)
Ground Penetrating Radar (GPR) has emerged as a pivotal tool for non-destructive evaluation of subsurface road defects. However, conventional GPR image interpretation remains heavily reliant on subjective expertise, introducing inefficiencies and inaccuracies. This study introduces a comprehensive framework to address these limitations: (1) A DCGAN-based data augmentation strategy synthesizes high-fidelity GPR images to mitigate data scarcity while preserving defect morphology under complex backgrounds; (2) A novel Multi-modal Chain and Global
Attention Network (MCGA-Net) is proposed, integrating Multi-modal Chain Feature Fusion (MCFF) for hierarchical multi-scale defect representation and Global
Attention Mechanism (GAM) for context-aware feature enhancement; (3) MS COCO transfer learning fine-tunes the backbone network, accelerating convergence and improving generalization. Ablation and comparison experiments validate the framework's efficacy. MCGA-Net achieves Precision (92.8%), Recall (92.5%), and mAP@50 (95.9%). In the detection of Gaussian noise, weak signals and small targets, MCGA-Net maintains robustness and outperforms other models. This work establishes a new paradigm for automated GPR-based defect detection, balancing computational efficiency with high accuracy in complex subsurface environments.
Read the full abstract ▾
All authors Haotian Lv, Yuhui Zhang, Jiangbo Dai, Hanli Wu, Jiaji Wang, Dawei Wang