Computer Vision
Provably Contractive and High-Quality Denoisers for Convergent Restoration
Shubhi Shukla and Pravin Nair
Image restoration, the recovery of clean images from degraded measurements, has applications in various domains like surveillance, defense, and medical imaging. While modern convolutional and
attention-based networks achieve strong restoration quality, they typically lack explicit stability guarantees under small input perturbations. In this work, we develop provably contractive denoiser networks with global Lipschitz constant smaller than one. Our design composes proximal layers obtained from unfolding techniques, with Lipschitz-controlled convolutional refinements. By contractivity, the proposed denoiser guarantees that input perturbations of strength $\|\delta\|\leq \varepsilon$ induce at most $\varepsilon$ change at the output, whereas strong unconstrained baselines such as DnCNN and Restormer can exhibit larger deviations under the same perturbations. On image denoising, our model substantially improves over prior provably 1-Lipschitz / contractive baselines and narrows the gap to strong unconstrained denoisers. In addition, when used as a prior in Plug-and-Play restoration, the proposed denoiser yields provable convergence and strong empirical performance. Overall, our results show that strict Lipschitz control need not preclude practically competitive restoration quality, and suggest that stable, verifiable restoration models can be achieved without a severe loss in performance. Codes and pretrained models are available at https://github.com/SHUBHI1553/Contractive-Denoisers
Read the full abstract ▾
Computer Vision
Gaussian Shannon: High-Precision Diffusion Model Watermarking Based on Communication
Yi Zhang, Hongbo Huang and Liang-Jie Zhang
Diffusion models generate high-quality images but pose serious risks like copyright violation and disinformation. Watermarking is a key defense for tracing and authenticating AI-generated content. However, existing methods rely on threshold-based detection, which only supports fuzzy matching and cannot recover structured watermark data bit-exactly, making them unsuitable for offline verification or applications requiring lossless metadata (e.g., licensing instructions). To address this problem, in this paper, we propose Gaussian Shannon, a watermarking framework that treats the
diffusion process as a noisy communication channel and enables both robust tracing and exact bit recovery. Our method embeds watermarks in the initial Gaussian noise without fine-tuning or quality loss. We identify two types of channel interference, namely local bit flips and global
stochastic distortions, and design a cascaded defense combining error-correcting codes and majority voting. This ensures reliable end-to-end transmission of semantic payloads. Experiments across three Stable
Diffusion variants and seven perturbation types show that Gaussian Shannon achieves state-of-the-art bit-level accuracy while maintaining a high true positive rate, enabling trustworthy rights attribution in real-world deployment. The source code have been made available at: https://github.com/Rambo-Yi/Gaussian-Shannon
Read the full abstract ▾
Computer Vision
IPV-Bench: Benchmarking Image Protection Methods under Diverse Image-to-Video Generation Scenarios
Xiaofeng Li, Leyi Sheng, Yifan Zhao et al. (+4 more)
Image-to-
video (I2V) generation models can be misused to animate a single image into a convincing fake
video, motivating perturbation-based image protection methods that aim to disrupt such generation. Yet these methods remain difficult to compare: they are reported under inconsistent metrics and generation settings, are often validated only on the single generator they were optimized against, and are evaluated on narrow, single-domain image sets that do not reflect real misuse. To address these challenges, we introduce IPV-Bench (Image Protection against
Video generation), the first systematic
benchmark for image protection in I2V generation scenarios. IPV-Bench couples a unified protocol that jointly scores protection effectiveness, visual fidelity, and robustness to preprocessing attacks together with IPV-500, a
prompt-paired
dataset spanning five misuse-relevant domains. Based on this
benchmark, we evaluate five representative protection methods across four I2V models covering distinct architectures and both open-source and commercial systems. Extensive experiments show a consistently sobering picture: image protection and
video disruption trade off against each other, most methods fail to disrupt generation beyond noise, protection rarely transfers across generators, and the few effective cases are broken by simple preprocessing. We further find that image content governs the perceptual cost of protection but not its benefit: no image domain offers an easier target. Overall, IPV-Bench provides a rigorous, reproducible, and extensible foundation for developing protection methods that work in practice.
Read the full abstract ▾
All authors Xiaofeng Li, Leyi Sheng, Yifan Zhao, Zhen Sun, Zongmin Zhang, Jiaheng Wei, Xinlei He
Computer Vision
Efficient Few-Shot Learning for Edge AI via Knowledge Distillation on MobileViT
Shuhei Tsuyuki, Reda Bensaid, Jérémy Morlier et al. (+4 more)
Efficient and adaptable deep learning models are an important area of deep learning research, driven by the need for highly efficient models on edge devices. Few-shot learning enables the use of deep learning models in low-data regimes, a capability that is highly sought after in real-world applications where collecting large annotated
datasets is costly or impractical. This challenge is particularly relevant in edge scenarios, where connectivity may be limited, low-latency responses are required, or energy consumption constraints are critical. We propose and evaluate a pre-training method for the MobileViT backbone designed for edge computing. Specifically, we employ knowledge
distillation, which transfers the generalization ability of a large-scale teacher model to a lightweight student model. This method achieves accuracy improvements of 14% and 6.7% for one-shot and five-shot classification, respectively, on the MiniImageNet
benchmark, compared to the ResNet12 baseline, while reducing by 69% the number of parameters and by 88% the computational complexity of the model, in FLOPs. Furthermore, we deployed the proposed models on a Jetson Orin Nano platform and measured power consumption directly at the power supply, showing that the dynamic energy consumption is reduced by 37% with a latency of 2.6 ms. These results demonstrate that the proposed method is a promising and practical solution for deploying few-shot learning models on edge AI hardware.
Read the full abstract ▾
All authors Shuhei Tsuyuki, Reda Bensaid, Jérémy Morlier, Mathieu Léonardon, Naoya Onizawa, Vincent Gripon, Takahiro Hanyu
Computer Vision · Cross-listed from cs.LG
PruneFuse: Efficient Data Selection via Weight Pruning and Network Fusion
Humaira Kousar, Hasnain Irshad Bhatti and Jaekyun Moon
Efficient data selection is crucial for enhancing the training efficiency of deep neural networks and minimizing annotation requirements. Traditional methods often face high computational costs, limiting their scalability and practical use. We introduce PruneFuse, a novel strategy that leverages pruned networks for data selection and later fuses them with the original network to optimize training. PruneFuse operates in two stages: First, it applies structured
pruning to create a smaller pruned network that, due to its structural coherence with the original network, is well-suited for the data selection task. This small network is then trained and selects the most informative samples from the
dataset. Second, the trained pruned network is seamlessly fused with the original network. This integration leverages the insights gained during the training of the pruned network to facilitate the learning process of the fused network while leaving room for the network to discover more robust solutions. Extensive experimentation on various
datasets demonstrates that PruneFuse significantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process.
Read the full abstract ▾
Computer Vision
InstaVSR: Taming Diffusion for Efficient and Temporally Consistent Video Super-Resolution
Jintong Hu, Bin Chen, Zhenyu Hu et al. (+3 more)
Video super-resolution (VSR) seeks to reconstruct high-resolution frames from low-resolution inputs. While
diffusion-based methods have substantially improved perceptual quality, extending them to
video remains challenging for two reasons: strong generative priors can introduce temporal instability, and multi-frame
diffusion pipelines are often too expensive for practical deployment. To address both challenges simultaneously, we propose InstaVSR, a lightweight
diffusion framework for efficient
video super-resolution. InstaVSR combines three ingredients: (1) a pruned one-step
diffusion backbone that removes several costly components from conventional
diffusion-based VSR pipelines, (2) recurrent training with flow-guided temporal regularization to improve frame-to-frame stability, and (3) dual-space
adversarial learning in latent and pixel spaces to preserve perceptual quality after backbone simplification. On an NVIDIA RTX 4090, InstaVSR processes a 30-frame
video at 2K$\times$2K resolution in under one minute with only 7 GB of
memory usage, substantially reducing the computational cost compared to existing
diffusion-based methods while maintaining favorable perceptual quality with significantly smoother temporal transitions.
Read the full abstract ▾
All authors Jintong Hu, Bin Chen, Zhenyu Hu, Jiayue Liu, Guo Wang, Lu Qi
Computer Vision
TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life
Mridul Khurana, Amin Karimi Monsefi, Justin Lee et al. (+9 more)
Accurately generating images across the Tree of Life is difficult: there are over 10M distinct species on Earth, many of which differ only by subtle visual traits. Despite the remarkable progress in text-to-image synthesis, existing models often fail to capture the fine-grained visual cues that define species identity, even when their outputs appear photo-realistic. To this end, we propose TaxaAdapter, a simple and lightweight approach that incorporates
Vision Taxonomy Models (VTMs) such as BioCLIP to guide fine-grained species generation. Our method injects VTM
embeddings into a frozen text-to-image
diffusion model, improving species-level fidelity while preserving flexible text control over attributes such as pose, style, and background. Extensive experiments demonstrate that TaxaAdapter consistently improves morphology fidelity and species-identity accuracy over strong baselines, with a cleaner architecture and training recipe. To better evaluate these improvements, we also introduce a
multimodal Large
Language Model-based metric that summarizes trait-level descriptions from generated and real images, providing a more interpretable measure of morphological consistency. Beyond this, we observe that TaxaAdapter exhibits strong generalization capabilities, enabling species synthesis in challenging regimes such as few-shot species with only a handful of training images and even species unseen during training. Overall, our results highlight that VTMs are a key ingredient for scalable, fine-grained species generation.
Read the full abstract ▾
All authors Mridul Khurana, Amin Karimi Monsefi, Justin Lee, Medha Sawhney, David Carlyn, Julia Chae, Jianyang Gu, Rajiv Ramnath, Sara Beery, Wei-Lun Chao, Anuj Karpatne, Cheng Zhang
Computer Vision
Finding Distributed Object-Centric Properties in Self-Supervised Transformers
Samyak Rawlekar, Amitabh Swain, Yujun Cai et al. (+3 more)
Self-supervised
Vision Transformers (ViTs) like DINO show an emergent ability to discover objects, typically observed in [CLS]
token attention maps of the final layer. However, these maps often contain spurious activations resulting in poor localization of objects. This is because the [CLS]
token, trained on an image-level objective, summarizes the entire image instead of focusing on objects. This aggregation dilutes the object-centric information existing in the local, patch-level interactions. We analyze this by computing inter-patch similarity using patch-level
attention components (query, key, and value) across all layers. We find that: (1) Object-centric properties are encoded in the similarity maps derived from all three components ($q, k, v$), unlike prior work that uses only key features or the [CLS]
token. (2) This object-centric information is distributed across the network, not just confined to the final layer. Based on these insights, we introduce Object-DINO, a training-free method that extracts this distributed object-centric information. Object-DINO clusters
attention heads across all layers based on the similarities of their patches and automatically identifies the object-centric cluster corresponding to all objects. We demonstrate Object-DINO's effectiveness on two applications: enhancing unsupervised object discovery (+3.6 to +12.4 CorLoc gains) and mitigating object
hallucination in
Multimodal Large
Language Models by providing visual grounding. Our results demonstrate that using this distributed object-centric information improves downstream tasks without additional training.
Read the full abstract ▾
All authors Samyak Rawlekar, Amitabh Swain, Yujun Cai, Yiwei Wang, Ming-Hsuan Yang, Narendra Ahuja
Computer Vision
Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR
Jinda Lu, Junkang Wu, Jinghan Li et al. (+6 more)
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) for
multimodal large
language models (MLLMs) have mainly focused on improving final answer correctness and strengthening visual grounding. However, a critical bottleneck remains: although models can attend to relevant visual regions, they often fail to effectively incorporate visual evidence into subsequent
reasoning, leading to
reasoning chains that are weakly grounded in visual facts. To address this issue, we propose Trajectory-Guided Reinforcement Learning (TGRL), which guides the policy model to integrate visual evidence into fine-grained
reasoning processes using expert
reasoning trajectories from stronger models. We further introduce
token-level reweighting and trajectory filtering to ensure stable and effective policy optimization. Extensive experiments on multiple
multimodal reasoning benchmarks demonstrate that TGRL consistently improves
reasoning performance and effectively bridges the gap between visual perception and logical
reasoning.
Read the full abstract ▾
All authors Jinda Lu, Junkang Wu, Jinghan Li, Kexin Huang, Shuo Yang, Mingzhu Chen, Jiancan Wu, Kuien Liu, Xiang Wang
Computer Vision
SkinGPT-X: A Self-Evolving Collaborative Multi-Agent System for Transparent and Trustworthy Dermatological Diagnosis
Zhangtianyi Chen, Yuhao Shen, Florensia Widjaja et al. (+6 more)
While recent advancements in Large
Language Models have significantly advanced dermatological diagnosis, monolithic LLMs frequently struggle with fine-grained, large-scale multi-class diagnostic tasks and rare skin disease diagnosis owing to training data sparsity, while also lacking the interpretability and traceability essential for clinical
reasoning. Although multi-
agent systems can offer more transparent and explainable diagnostics, existing frameworks are primarily concentrated on Visual Question Answering and conversational tasks, and their heavy reliance on static knowledge bases restricts adaptability in complex real-world clinical settings. Here, we present SkinGPT-X, a
multimodal collaborative multi-
agent system for dermatological diagnosis integrated with a self-evolving dermatological
memory mechanism. By simulating the diagnostic workflow of dermatologists and enabling continuous
memory evolution, SkinGPT-X delivers transparent and trustworthy diagnostics for the management of complex and rare dermatological cases. To validate the robustness of SkinGPT-X, we design a three-tier comparative experiment. First, we
benchmark SkinGPT-X against four state-of-the-art LLMs across four public
datasets, demonstrating its state-of-the-art performance with a +9.6% accuracy improvement on DDI31 and +13% weighted F1 gain on Dermnet over the state-of-the-art model. Second, we construct a large-scale multi-class
dataset covering 498 distinct dermatological categories to evaluate its fine-grained classification capabilities. Finally, we curate the rare skin disease
dataset, the first
benchmark to address the scarcity of clinical rare skin diseases which contains 564 clinical samples with eight rare dermatological diseases. On this
dataset, SkinGPT-X achieves a +9.8% accuracy improvement, a +7.1% weighted F1 improvement, a +10% Cohen's Kappa improvement.
Read the full abstract ▾
All authors Zhangtianyi Chen, Yuhao Shen, Florensia Widjaja, Yan Xu, Liyuan Sun, Zijian Wang, Hongyi Chen, Wufei Dai, Juexiao Zhou
Computer Vision · Cross-listed from eess.IV
FINDER: Zero-Shot Field-Integrated Network for Distortion-free EPI Reconstruction in Diffusion MRI
Namgyu Han, Seong Dae Yun, Chaeeun Lim et al. (+7 more)
Echo-planar imaging (EPI) remains the cornerstone of
diffusion MRI, but it is prone to severe
geometric distortions due to its rapid sampling scheme that renders the sequence highly sensitive to $B_{0}$ field inhomogeneities. While deep learning has helped improve MRI reconstruction, integrating robust
geometric distortion correction into a self-supervised framework remains an unmet need. To address this, we present FINDER (Field-Integrated Network for Distortion-free EPI Reconstruction), a novel zero-shot, scan-specific framework that reformulates reconstruction as a joint optimization of the underlying image and the $B_{0}$ field map. Specifically, we employ a physics-guided unrolled network that integrates dual-domain denoisers and virtual coil extensions to enforce robust data consistency. This is coupled with an Implicit Neural Representation (INR) conditioned on spatial coordinates and latent image features to model the off-resonance field as a continuous, differentiable function. Employing an alternating minimization strategy, FINDER synergistically updates the reconstruction network and the field map, effectively disentangling susceptibility-induced
geometric distortions from anatomical structures. Experimental results demonstrate that FINDER achieves superior
geometric fidelity and image quality compared to state-of-the-art baselines, offering a robust solution for high-quality
diffusion imaging.
Read the full abstract ▾
All authors Namgyu Han, Seong Dae Yun, Chaeeun Lim, Sunghyun Seok, Sunju Kim, Yoonhwan Kim, Yohan Jun, Tae Hyung Kim, Berkin Bilgic, Jaejin Cho
Computer Vision
SDDF: Specificity-Driven Dynamic Focusing for Open-Vocabulary Camouflaged Object Detection
Jiaming Liang, Yifeng Zhan, Chunlin Liu et al. (+6 more)
Open-vocabulary object detection (OVOD) aims to detect known and unknown objects in the open world by leveraging text
prompts. Benefiting from the emergence of large-scale
vision–
language pre-trained models, OVOD has demonstrated strong zero-shot generalization capabilities. However, when dealing with camouflaged objects, the detector often fails to distinguish and localize objects because the visual features of the objects and the background are highly similar. To bridge this gap, we construct a
benchmark named OVCOD-D by augmenting carefully selected camouflaged object images with fine-grained textual descriptions. Due to the limited scale of available camouflaged object
datasets, we adopt detectors pre-trained on large-scale object detection
datasets as our baseline methods, as they possess stronger zero-shot generalization ability. In the specificity-aware sub-descriptions generated by
multimodal large models, there still exist confusing and overly decorative modifiers. To mitigate such interference, we design a sub-description principal component contrastive fusion strategy that reduces noisy textual components. Furthermore, to address the challenge that the visual features of camouflaged objects are highly similar to those of their surrounding environment, we propose a specificity-guided regional weak
alignment and dynamic focusing method, which aims to strengthen the detector's ability to discriminate camouflaged objects from background. Under the open-set evaluation setting, the proposed method achieves an AP of 56.4 on the OVCOD-D
benchmark.
Read the full abstract ▾
All authors Jiaming Liang, Yifeng Zhan, Chunlin Liu, Weihua Zheng, Bingye Peng, Qiwei Liang, Boyang Cai, Xiaochun Mai, Qiang Nie
Computer Vision · Cross-listed from cs.LG
Accurate Precipitation Forecast by Efficiently Learning from Massive Atmospheric Variables and Unbalanced Distribution
Shuangliang Li, Siwei Li, Li Li et al. (+3 more)
Short-term (0-24 hours) precipitation
forecasting is highly valuable to socioeconomic activities and public safety. However, the highly complex
evolution patterns of precipitation events, the extreme imbalance between precipitation and non-precipitation samples, and the inability of existing models to efficiently and effectively utilize large volumes of multi-source atmospheric observation data hinder improvements in precipitation
forecasting accuracy and computational efficiency. To address the above challenges, this study developed a novel
forecasting model capable of effectively and efficiently utilizing massive atmospheric observations by automatically extracting and iteratively predicting the latent features strongly associated with precipitation
evolution. Furthermore, this study introduces a 'WMCE' loss function, designed to accurately discriminate extremely scarce precipitation events while precisely predicting their intensity values. Extensive experiments on two
datasets demonstrate that our proposed model substantially and consistently outperforms all prevalent baselines in both accuracy and efficiency. Moreover, the proposed
forecasting model substantially lowers the computational cost required to obtain valuable predictions compared to existing approaches, thereby positioning it as a milestone for efficient and practical precipitation
forecasting.
Read the full abstract ▾
All authors Shuangliang Li, Siwei Li, Li Li, Weijie Zou, Jie Yang, Maolin Zhang
Computer Vision · Cross-listed from cs.LG
AcTTA: Rethinking Test-Time Adaptation via Dynamic Activation
Hyeongyu Kim, Geonhui Han and Dosik Hwang
Test-time adaptation (TTA) aims to mitigate performance degradation under distribution shifts by updating model parameters during inference. Existing approaches have primarily framed adaptation around affine modulation, focusing on recalibrating normalization layers. This perspective, while effective, overlooks another influential component in representation dynamics: the activation function. We revisit this overlooked space and propose AcTTA, an activation-aware framework that reinterprets conventional activation functions from a learnable perspective and updates them adaptively at test time. AcTTA reformulates conventional activation functions (e.g., ReLU, GELU) into parameterized forms that shift their response threshold and modulate gradient sensitivity, enabling the network to adjust activation behavior under domain shifts. This functional reparameterization enables continuous adjustment of activation behavior without modifying network weights or requiring source data. Despite its simplicity, AcTTA achieves robust and stable adaptation across diverse corruptions. Across CIFAR10-C, CIFAR100-C, and ImageNet-C, AcTTA consistently surpasses normalization-based TTA methods. Our findings highlight activation adaptation as a compact and effective route toward domain-shift-robust test-time learning, broadening the prevailing affine-centric view of adaptation.
Read the full abstract ▾
Computer Vision
CD-Buffer: Complementary Dual-Buffer Framework for Test-Time Adaptation in Adverse Weather Object Detection
Youngjun Song, Hyeongyu Kim and Dosik Hwang
Test-Time Adaptation (TTA) enables real-time adaptation to domain shifts without off-line retraining. Recent TTA methods have predominantly explored additive approaches that introduce lightweight modules for feature refinement. Recently, a subtractive approach that removes domain-sensitive channels has emerged as an alternative direction. We observe that these paradigms exhibit complementary effectiveness patterns: subtractive methods excel under severe shifts by removing corrupted features, while additive methods are effective under moderate shifts requiring refinement. However, each paradigm operates effectively only within limited shift severity ranges, failing to generalize across diverse corruption levels. This leads to the following question: can we adaptively balance both strategies based on measured feature-level domain shift? We propose CD-Buffer, a novel complementary dual-buffer framework where subtractive and additive mechanisms operate in opposite yet coordinated directions driven by a unified discrepancy metric. Our key innovation lies in the discrepancy-driven coupling: Our framework couples removal and refinement through a unified discrepancy metric, automatically balancing both strategies based on feature-level shift severity. This establishes automatic channel-wise balancing that adapts differentiated treatment to heterogeneous shift magnitudes without manual tuning. Extensive experiments on KITTI, Cityscapes, and ACDC
datasets demonstrate state-of-the-art performance, consistently achieving superior results across diverse weather conditions and severity levels.
Read the full abstract ▾
Computer Vision
Learnable Instance Attention Filtering for Adaptive Detector Distillation
Chen Liu, Qizhen Lan, Zhicheng Ding et al. (+2 more)
As deep
vision models grow increasingly complex to achieve higher performance, deployment efficiency has become a critical concern. Knowledge
distillation (KD) mitigates this issue by transferring knowledge from large teacher models to compact student models. While many feature-based KD methods rely on spatial filtering to guide
distillation, they typically treat all object instances uniformly, ignoring instance-level variability. Moreover, existing
attention filtering mechanisms are typically heuristic or teacher-driven, rather than learned with the student. To address these limitations, we propose Learnable Instance
Attention Filtering for Adaptive Detector
Distillation (LIAF-KD), a novel framework that introduces learnable instance selectors to dynamically evaluate and reweight instance importance during
distillation. Notably, the student contributes to this process based on its evolving learning state. Experiments on the KITTI and COCO
datasets demonstrate consistent improvements, with a 2% gain on a GFL ResNet-50 student without added complexity, outperforming state-of-the-art methods.
Read the full abstract ▾
All authors Chen Liu, Qizhen Lan, Zhicheng Ding, Xinyu Chu, Qing Tian
Computer Vision · Cross-listed from eess.SY
Experimental study on surveillance video-based indoor occupancy measurement with occupant-centric control
Irfan Qaisar, Kailai Sun, Qingshan Jia et al. (+1 more)
Accurate occupancy information is essential for closed-loop occupant-centric control (OCC) in smart buildings. However, existing
vision-based occupancy measurement methods often struggle to provide stable and accurate measurements in real indoor environments, and their implications for downstream HVAC control remain insufficiently studied. To achieve Net Zero emissions by 2050, this paper presents an experimental study of large
language models (LLMs)-enhanced
vision-based indoor occupancy measurement and its impact on OCC-enabled HVAC operation. Detection-only, tracking-based, and LLM-based refinement pipelines are compared under identical conditions using real surveillance data collected from a research laboratory in China, with frame-level manual ground-truth annotations. Results show that tracking-based methods improve temporal stability over detection-only measurement, while LLM-based refinement further improves occupancy measurement performance and reduces false unoccupied prediction. The best-performing pipeline, YOLOv8+DeepSeek, achieves an accuracy of 0.8824 and an F1-score of 0.9320. This pipeline is then integrated into an HVAC supervisory model predictive control framework in OpenStudio-EnergyPlus. Experimental results demonstrate that the proposed framework can support more efficient OCC operation, achieving a substantial HVAC energy-saving potential of 17.94%. These findings provide an effective methodology and practical foundation for future research in AI-enhanced smart building operations.
Read the full abstract ▾
All authors Irfan Qaisar, Kailai Sun, Qingshan Jia, Qianchuan Zhao
Computer Vision
When Identities Collapse: A Stress-Test Benchmark for Multi-Subject Personalization
Zhihan Chen, Yuhuan Zhao, Yijie Zhu et al. (+1 more)
Subject-driven text-to-image
diffusion models have achieved remarkable success in preserving single identities, yet their ability to compose multiple interacting subjects remains largely unexplored and highly challenging. Existing evaluation protocols typically rely on global CLIP metrics, which are insensitive to local identity collapse and fail to capture the severity of multi-subject
entanglement. In this paper, we identify a pervasive "Illusion of Scalability" in current models: while they excel at synthesizing 2-4 subjects in simple layouts, they suffer from catastrophic identity collapse when scaled to 6-10 subjects or tasked with complex physical interactions. To systematically expose this failure mode, we construct a rigorous stress-test
benchmark comprising 75
prompts distributed across varying subject counts and interaction difficulties (Neutral, Occlusion, Interaction). Furthermore, we demonstrate that standard CLIP-based metrics are fundamentally flawed for this task, as they often assign high scores to semantically correct but identity-collapsed images (e.g., generating generic clones). To address this, we introduce the Subject Collapse Rate (SCR), a novel evaluation metric grounded in DINOv2's structural priors, which strictly penalizes local
attention leakage and homogenization. Our extensive evaluation of state-of-the-art models (MOSAIC, XVerse, PSR) reveals a precipitous drop in identity fidelity as scene complexity grows, with SCR approaching 100% at 10 subjects. We trace this collapse to the semantic shortcuts inherent in global
attention routing, underscoring the urgent need for explicit physical disentanglement in future generative architectures.
Read the full abstract ▾
All authors Zhihan Chen, Yuhuan Zhao, Yijie Zhu, Xinyu Yao
Computer Vision
MUST: Modality-Specific Representation-Aware Transformer for Diffusion-Enhanced Survival Prediction with Missing Modality
Kyungwon Kim and Dosik Hwang
Accurate survival prediction from
multimodal medical data is essential for precision oncology, yet clinical deployment faces a persistent challenge: modalities are frequently incomplete due to cost constraints, technical limitations, or retrospective data availability. While recent methods attempt to address missing modalities through feature
alignment or joint distribution learning, they fundamentally lack explicit modeling of the unique contributions of each modality as opposed to the information derivable from other modalities. We propose MUST (Modality-Specific representation-aware
Transformer), a novel framework that explicitly decomposes each modality's representation into modality-specific and cross-modal contextualized components through algebraic constraints in a learned low-rank shared subspace. This decomposition enables precise identification of what information is lost when a modality is absent. For the truly modality-specific information that cannot be inferred from available modalities, we employ conditional latent
diffusion models to generate high-quality representations conditioned on recovered shared information and learned structural priors. Extensive experiments on five TCGA cancer
datasets demonstrate that MUST achieves state-of-the-art performance with complete data while maintaining robust predictions in both missing pathology and missing genomics conditions, with clinically acceptable inference latency.
Read the full abstract ▾
Computer Vision
PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery
Elkhan Ismayilzada, Yufei Zhang and Zijun Cui
Significant advancements made in reconstructing hands from images have delivered accurate single-frame estimates, yet they often lack physics consistency and provide no notion of how confidently the motion satisfies physics. In this paper, we propose a novel physics-aware conditional
diffusion framework that refines noisy pose sequences into physically plausible hand motion while estimating the physics variance in motion estimates. Building on a MeshCNN-
Transformer backbone, we formulate Euler-Lagrange dynamics for articulated hands. Unlike prior works that enforce zero residuals, we treat the resulting dynamic residuals as virtual observables to more effectively integrate physics. Through a last-layer Laplace approximation, our method produces per-joint, per-time variances that measure physics consistency and offers interpretable variance maps indicating where physical consistency weakens. Experiments on two well-known hand
datasets show consistent gains over strong image-based initializations and competitive
video-based methods. Qualitative results confirm that our variance estimations are aligned with the physical plausibility of the motion in image-based estimates.
Read the full abstract ▾
Computer Vision
R-PGA: Robust Physical Adversarial Camouflage Generation via Relightable 3D Gaussian Splatting
Tianrui Lou, Siyuan Liang, Jiawei Liang et al. (+2 more)
Physical
adversarial camouflage poses a severe security threat to autonomous driving systems by mapping
adversarial textures onto 3D objects. Nevertheless, current methods remain brittle in complex dynamic scenarios, failing to generalize across diverse
geometric (e.g., viewing configurations) and radiometric (e.g., dynamic illumination, atmospheric scattering) variations. We attribute this deficiency to two fundamental limitations in simulation and optimization. First, the reliance on coarse, oversimplified simulations (e.g., via CARLA) induces a significant domain gap, confining optimization to a biased feature space. Second, standard strategies targeting average performance result in a rugged loss landscape, leaving the camouflage vulnerable to configuration shifts.To bridge these gaps, we propose the Relightable Physical 3D Gaussian Splatting (3DGS) based Attack framework (R-PGA). Technically, to address the simulation fidelity issue, we leverage 3DGS to ensure photo-realistic reconstruction and augment it with physically disentangled attributes to decouple intrinsic material from lighting. Furthermore, we design a hybrid rendering pipeline that leverages precise Relightable 3DGS for foreground rendering, while employing a pre-trained image translation model to synthesize plausible relighted backgrounds that align with the relighted foreground.To address the optimization robustness issue, we propose the Hard Physical Configuration Mining (HPCM) module, designed to actively mine worst-case physical configurations and suppress their corresponding loss peaks. This strategy not only diminishes the overall loss magnitude but also effectively flattens the rugged loss landscape, ensuring consistent
adversarial effectiveness and robustness across varying physical configurations.
Read the full abstract ▾
All authors Tianrui Lou, Siyuan Liang, Jiawei Liang, Yuze Gao, Xiaochun Cao
Computer Vision
MuDD: A Multimodal Deception Detection Dataset and GSR-Guided Progressive Distillation for Non-Contact Deception Detection
Peiyuan Jiang, Yao Liu, Yanglei Gan et al. (+4 more)
Non-contact automatic deception detection remains challenging because visual and auditory deception cues often lack stable cross-subject patterns. In contrast, galvanic skin response (GSR) provides more reliable physiological cues and has been widely used in contact-based deception detection. In this work, we leverage stable deception-related knowledge in GSR to guide representation learning in non-contact modalities through cross-modal knowledge
distillation. A key obstacle, however, is the lack of a suitable
dataset for this setting. To address this, we introduce MuDD, a large-scale
Multimodal Deception Detection
dataset containing recordings from 130 participants over 690 minutes. In addition to
video, audio, and GSR, MuDD also provides Photoplethysmography, heart rate, and personality traits, supporting broader scientific studies of deception. Based on this
dataset, we propose GSR-guided Progressive
Distillation (GPD), a cross-modal
distillation framework for mitigating the negative transfer caused by the large modality mismatch between GSR and non-contact signals. The core innovation of GPD is the integration of progressive feature-level and digit-level
distillation with dynamic routing, which allows the model to adaptively determine how teacher knowledge should be transferred during training, leading to more stable cross-modal knowledge transfer. Extensive experiments and visualizations show that GPD outperforms existing methods and achieves state-of-the-art performance on both deception detection and concealed-digit identification.
Read the full abstract ▾
All authors Peiyuan Jiang, Yao Liu, Yanglei Gan, Jiaye Yang, Lu Liu, Daibing Yao, Qiao Liu
Computer Vision
Pioneering Perceptual Video Fluency Assessment: A Novel Task with Benchmark Dataset and Baseline
Qizhi Xie, Kun Yuan, Yunpeng Qu et al. (+3 more)
Accurately estimating humans' subjective feedback on
video fluency, e.g., motion consistency and frame continuity, is crucial for various applications like streaming and gaming. Yet, it has long been overlooked, as prior arts have focused on solving it in the
video quality assessment (VQA) task, merely as a sub-dimension of overall quality. In this work, we conduct pilot experiments and reveal that current VQA predictions largely underrepresent fluency, thereby limiting their applicability. To this end, we pioneer
Video Fluency Assessment (VFA) as a standalone perceptual task focused on the temporal dimension. To advance VFA research, 1) we construct a fluency-oriented
dataset, FluVid, comprising 4,606 in-the-wild
videos with balanced fluency distribution, featuring the first-ever scoring criteria and human study for VFA. 2) We develop a large-scale
benchmark of 23 methods, the most comprehensive one thus far on FluVid, gathering insights for VFA-tailored model designs. 3) We propose a baseline model called FluNet, which deploys temporal permuted self-
attention (T-PSA) to enrich input fluency information and enhance long-range inter-frame interactions. Our work not only achieves state-of-the-art performance but, more importantly, offers the community a roadmap to explore solutions for VFA.
Read the full abstract ▾
All authors Qizhi Xie, Kun Yuan, Yunpeng Qu, Ming Sun, Chao Zhou, Jihong Zhu
Computer Vision
Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification
Zizhao Chen, Ping Wei, Ziyang Ren et al. (+2 more)
As
multimodal misinformation becomes more sophisticated, its detection and grounding are crucial. However, current
multimodal verification methods, relying on passive holistic fusion, struggle with sophisticated misinformation. Due to 'feature dilution,' global
alignments tend to average out subtle local semantic inconsistencies, effectively masking the very conflicts they are designed to find. We introduce MaLSF (Mask-aware Local Semantic Fusion), a novel framework that shifts the paradigm to active, bidirectional verification, mimicking human cognitive cross-referencing. MaLSF utilizes mask-label pairs as semantic anchors to bridge pixels and words. Its core mechanism features two innovations: 1) a Bidirectional Cross-modal Verification (BCV) module that acts as an interrogator, using parallel query streams (Text-as-Query and Image-as-Query) to explicitly pinpoint conflicts; and 2) a Hierarchical Semantic Aggregation (HSA) module that intelligently aggregates these multi-granularity conflict signals for task-specific
reasoning. In addition, to extract fine-grained mask-label pairs, we introduce a set of diverse mask-label pair extraction parsers. MaLSF achieves state-of-the-art performance on both the DGM4 and
multimodal fake news detection tasks. Extensive ablation studies and visualization results further verify its effectiveness and interpretability.
Read the full abstract ▾
All authors Zizhao Chen, Ping Wei, Ziyang Ren, Huan Li, Xiangru Yin
Computer Vision
Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays
Kang Liu, Zhuoqi Ma, Siyu Liang et al. (+5 more)
Despite recent advances in medical
vision-
language pretraining, existing models still struggle to capture the diagnostic workflow: radiographs are typically treated as context-agnostic images, while radiologists' gaze – a crucial cue for visual
reasoning – remains largely underexplored by existing methods. These limitations hinder the modeling of disease-specific patterns and weaken cross-modal
alignment. To bridge this gap, we introduce CoGaze, a Context- and Gaze-guided
vision-
language pretraining framework for chest X-rays. We first propose a context-infused
vision encoder that models how radiologists integrate clinical context – including patient history, symptoms, and diagnostic intent – to guide diagnostic
reasoning. We then present a multi-level supervision paradigm that (1) enforces intra- and inter-modal semantic
alignment through hybrid-positive contrastive learning, (2) injects diagnostic priors via disease-aware cross-modal representation learning, and (3) leverages radiologists' gaze as probabilistic priors to guide
attention toward diagnostically salient regions. Extensive experiments demonstrate that CoGaze consistently outperforms state-of-the-art methods across diverse tasks, achieving up to +2.0% CheXbertF1 and +1.2% BLEU2 for free-text and structured report generation, +23.2% AUROC for zero-shot classification, and +12.2% Precision@1 for image-text
retrieval. Code is available at https://github.com/mk-runner/CoGaze.
Read the full abstract ▾
All authors Kang Liu, Zhuoqi Ma, Siyu Liang, Yunan Li, Xiyue Gao, Chao Liang, Kun Xie, Qiguang Miao
Computer Vision
Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation
Daiqiang Li, Zihao Pan, Zeyu Zhang et al. (+8 more)
In recent years, GUI
agents have demonstrated strong potential in
navigation tasks. However, preserving complete historical screenshots introduces substantial computational overhead. This paper investigates how
token pruning, a plug-and-play inference acceleration technique, can be effectively applied to GUI
agent navigation scenarios. Firstly, we address the question of where
pruning should occur. We identify a system-level redundancy overlooked by existing methods: as the same screenshot is repeatedly fed into the model across different steps, its ViT encoding is redundantly recomputed each time. We show that its ViT-encoded
embeddings can be fully cached and reused across steps, substantially reducing FLOPs while preserving model performance. This finding suggests that inference acceleration efforts should focus on the subsequent Large
Language Model (LLM). Building on this, we further address the question of how to prune within the LLM, and distill two key insights: (i) from a semantic perspective, the
token budget should be balanced between foreground and background regions; (ii) from a spatial perspective, the spatial uniformity of retained
tokens should be maintained to preserve the model's global spatial perception. These findings provide practical guidance for the design of inference acceleration and
token pruning for GUI
agent navigation.
Read the full abstract ▾
All authors Daiqiang Li, Zihao Pan, Zeyu Zhang, Xuyang Liu, Shijia Xu, Ronghao Chen, Huacan Wang, Honggang Chen, Linfeng Zhang, Zhangquan Chen, Haiyun Jiang
Computer Vision
Face2Parts: Exploring Coarse-to-Fine Inter-Regional Facial Dependencies for Generalized Deepfake Detection
Kutub Uddin, Nusrat Tasnim and Byung Tae Oh
Multimedia data, particularly images and
videos, is integral to various applications, including surveillance, visual interaction, biometrics, evidence gathering, and advertising. However, amateur or skilled counterfeiters can simulate them to create deepfakes, often for slanderous motives. To address this challenge, several forensic methods have been developed to ensure the authenticity of the content. The effectiveness of these methods depends on their focus, with challenges arising from the diverse nature of manipulations. In this article, we analyze existing forensic methods and observe that each method has unique strengths in detecting deepfake traces by focusing on specific facial regions, such as the frame, face, lips, eyes, or nose. Considering these insights, we propose a novel hybrid approach called Face2Parts based on hierarchical feature representation ($HFR$) that takes advantage of coarse-to-fine information to improve deepfake detection. The proposed method involves extracting features from the frame, face, and key facial regions (i.e., lips, eyes, and nose) separately to explore the coarse-to-fine relationships. This approach enables us to capture inter-dependencies among facial regions using a channel-
attention mechanism and deep triplet learning. We evaluated the proposed method on
benchmark deepfake
datasets in both intra-, inter-
dataset, and inter-manipulation settings. The proposed method achieves an average AUC of 98.42% on FF++, 79.80% on CDF1, 85.34% on CDF2, 89.41% on DFD, 84.07% on DFDC, 95.62% on DTIM, 80.76% on PDD, and 100% on WLDR, respectively. The results demonstrate that our approach generalizes effectively and achieves promising performance to outperform the existing methods.
Read the full abstract ▾
Computer Vision
Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMs
Jiazheng Xing, Chao Xu, Hangjie Yuan et al. (+4 more)
Multimodal Large
Language Models (MLLMs) have propelled the field of few-shot action recognition (FSAR). However, preliminary explorations in this area primarily focus on generating captions to form a suboptimal feature->caption->feature pipeline and adopt metric learning solely within the visual space. In this paper, we propose FSAR-LLaVA, the first end-to-end method to leverage MLLMs (such as
Video-LLaVA) as a
multimodal knowledge base for directly enhancing FSAR. First, at the feature level, we leverage the MLLM's
multimodal decoder to extract spatiotemporally and semantically enriched representations, which are then decoupled and enhanced by our
Multimodal Feature-Enhanced Module into distinct visual and textual features that fully exploit their semantic knowledge for FSAR. Next, we leverage the versatility of MLLMs to craft input
prompts that flexibly adapt to diverse scenarios, and use their aligned outputs to drive our designed Composite Task-Oriented Prototype Construction, effectively bridging the distribution gap between meta-train and meta-test sets. Finally, to enable
multimodal features to guide metric learning jointly, we introduce a training-free
Multimodal Prototype Matching Metric that adaptively selects the most decisive cues and efficiently leverages the decoupled feature representations produced by MLLMs. Extensive experiments demonstrate superior performance across various tasks with minimal trainable parameters.
Read the full abstract ▾
All authors Jiazheng Xing, Chao Xu, Hangjie Yuan, Mengmeng Wang, Jun Dan, Hangwei Qian, Yong Liu
Computer Vision
Learning to Trim: End-to-End Causal Graph Pruning with Dynamic Anatomical Feature Banks for Medical VQA
Zibo Xu, Qiang Li, Weizhi Nie et al. (+1 more)
Medical Visual Question Answering (MedVQA) models often exhibit limited generalization due to reliance on
dataset-specific correlations, such as recurring anatomical patterns or question-type regularities, rather than genuine diagnostic evidence. Existing causal approaches are typically implemented as static adjustments or post-hoc corrections. To address this issue, we propose a Learnable Causal Trimming (LCT) framework that integrates causal
pruning into end-to-end optimization. We introduce a Dynamic Anatomical Feature Bank (DAFB), updated via a momentum mechanism, to capture global prototypes of frequent anatomical and linguistic patterns, serving as an approximation of
dataset-level regularities. We further design a differentiable trimming module that estimates the dependency between instance-level representations and the global feature bank. Features highly correlated with global prototypes are softly suppressed, while instance-specific evidence is emphasized. This learnable mechanism encourages the model to prioritize causal signals over spurious correlations adaptively. Experiments on VQA-RAD, SLAKE, SLAKE-CP and PathVQA demonstrate that LCT consistently improves robustness and generalization over existing debiasing strategies.
Read the full abstract ▾
All authors Zibo Xu, Qiang Li, Weizhi Nie, Yuting Su
Computer Vision
Unlabeled Cross-Center Automatic Analysis for TAAD: An Integrated Framework from Segmentation to Clinical Features
Mengdi Liu, Qiang Li, Weizhi Nie et al. (+2 more)
Type A Aortic Dissection (TAAD) is a life-threatening cardiovascular emergency that demands rapid and precise preoperative evaluation. While key anatomical and pathological features are decisive for
surgical planning, current research focuses predominantly on improving
segmentation accuracy, leaving the reliable, quantitative extraction of clinically actionable features largely under-explored. Furthermore, constructing comprehensive TAAD
datasets requires labor-intensive, expert level pixel-wise annotations, which is impractical for most clinical institutions. Due to significant domain shift, models trained on a single center
dataset also suffer from severe performance degradation during cross-institutional deployment. This study addresses a clinically critical challenge: the accurate extraction of key TAAD clinical features during cross-institutional deployment in the total absence of target-domain annotations. To this end, we propose an unsupervised domain adaptation (UDA)-driven framework for the automated extraction of TAAD clinical features. The framework leverages limited source-domain labels while effectively adapting to unlabeled data from target domains. Tailored for real-world emergency workflows, our framework aims to achieve stable cross-institutional multi-class
segmentation, reliable and quantifiable clinical feature extraction, and practical deployability independent of high-cost annotations. Extensive experiments demonstrate that our method significantly improves cross-domain
segmentation performance compared to existing state-of-the-art approaches. More importantly, a reader study involving multiple cardiovascular surgeons confirms that the automatically extracted clinical features provide meaningful assistance for preoperative assessment, highlighting the practical utility of the proposed end-to-end
segmentation-to-feature pipeline.
Read the full abstract ▾
All authors Mengdi Liu, Qiang Li, Weizhi Nie, Shaopeng Zhang, Yuting Su