Computer Vision
SDHSI-Net: Learning Better Representations for Hyperspectral Images via Self-Distillation
Prachet Dev Singh, Shyamsundar Paramasivam, Sneha Barman et al. (+4 more)
Hyperspectral image (HSI) classification presents unique challenges due to its high spectral dimensionality and limited labeled data. Traditional deep learning models often suffer from overfitting and high computational costs. Self-
distillation (SD), a variant of knowledge
distillation where a network learns from its own predictions, has recently emerged as a promising strategy to enhance model performance without requiring external teacher networks. In this work, we explore the application of SD to HSI by treating earlier outputs as soft targets, thereby enforcing consistency between intermediate and final predictions. This process improves intra-class compactness and inter-class separability in the learned feature space. Our approach is validated on two
benchmark HSI
datasets and demonstrates significant improvements in classification accuracy and robustness, highlighting the effectiveness of SD for spectral-spatial learning. Codes are available at https://github.com/Prachet-Dev-Singh/SDHSI.
Read the full abstract ▾
All authors Prachet Dev Singh, Shyamsundar Paramasivam, Sneha Barman, Mainak Singha, Ankit Jha, Girish Mishra, Biplab Banerjee
Computer Vision
Forecast the Principal, Stabilize the Residual: Subspace-Aware Feature Caching for Efficient Diffusion Transformers
Guantao Chen, Shikang Zheng, Yuqi Lin et al. (+1 more)
Diffusion Transformer (DiT) models have achieved unprecedented quality in image and
video generation, yet their iterative sampling process remains computationally prohibitive. To accelerate inference, feature caching methods have emerged by reusing intermediate representations across timesteps. However, existing caching approaches treat all feature components uniformly. We reveal that DiT feature spaces contain distinct principal and residual subspaces with divergent temporal behavior: the principal subspace evolves smoothly and predictably, while the residual subspace exhibits volatile, low-energy oscillations that resist accurate prediction. Building on this insight, we propose SVD-Cache, a subspace-aware caching framework that decomposes
diffusion features via Singular Value Decomposition (SVD), applies exponential moving average (EMA) prediction to the dominant low-rank components, and directly reuses the residual subspace. Extensive experiments demonstrate that SVD-Cache achieves near-lossless across diverse models and methods, including 5.55$\times$ speedup on FLUX and HunyuanVideo, and compatibility with model acceleration techniques including
distillation, quantization and sparse
attention. Our code is in supplementary material and will be released on Github.
Read the full abstract ▾
All authors Guantao Chen, Shikang Zheng, Yuqi Lin, Linfeng Zhang
Computer Vision · Cross-listed from cs.LG
OceanSAR-2: A Universal Feature Extractor for SAR Ocean Observation
Alexandre Tuel, Thomas Kerdreux, Quentin Febvre et al. (+6 more)
We present OceanSAR-2, the second generation of our foundation model for SAR-based ocean observation. Building on our earlier release, which pioneered self-supervised learning on Sentinel-1 Wave Mode data, OceanSAR-2 relies on improved SSL training and dynamic data curation strategies, which enhances performance while reducing training cost. OceanSAR-2 demonstrates strong transfer performance across downstream tasks, including geophysical pattern classification, ocean surface wind vector and significant wave height estimation, and iceberg detection. We release standardized
benchmark datasets, providing a foundation for systematic evaluation and advancement of SAR models for ocean applications.
Read the full abstract ▾
All authors Alexandre Tuel, Thomas Kerdreux, Quentin Febvre, Alexis Mouche, Antoine Grouazel, Jean-Renaud Miadana, Antoine Audras, Chen Wang, Bertrand Chapron
Computer Vision
Learning Dynamic Collaborative Network for Semi-supervised 3D Vessel Segmentation
Jiao Xu, Xin Chen and Lihe Zhang
In this paper, we present a new dynamic collaborative network for semi-supervised 3D vessel
segmentation, termed DiCo. Conventional mean teacher (MT) methods typically employ a static approach, where the roles of the teacher and student models are fixed. However, due to the complexity of 3D vessel data, the teacher model may not always outperform the student model, leading to cognitive biases that can limit performance. To address this issue, we propose a dynamic collaborative network that allows the two models to dynamically switch their teacher-student roles. Additionally, we introduce a multi-view integration module to capture various perspectives of the inputs, mirroring the way doctors conduct medical analysis. We also incorporate
adversarial supervision to constrain the shape of the segmented vessels in unlabeled data. In this process, the 3D volume is projected into 2D views to mitigate the impact of label inconsistencies. Experiments demonstrate that our DiCo method sets new state-of-the-art performance on three 3D vessel
segmentation benchmarks. The code repository address is https://github.com/xujiaommcome/DiCo
Read the full abstract ▾
Computer Vision
HiVid-Narrator: Hierarchical Video Narrative Generation with Scene-Primed ASR-anchored Compression
Haoxuan Li, Mengyan Li and Junjun Zheng
Generating structured narrations for real-world e-commerce
videos requires models to perceive fine-grained visual details and organize them into coherent, high-level stories–capabilities that existing approaches struggle to unify. We introduce the E-commerce Hierarchical
Video Captioning (E-HVC)
dataset with dual-granularity, temporally grounded annotations: a Temporal Chain-of-Thought that anchors event-level observations and Chapter Summary that compose them into concise, story-centric summaries. Rather than directly prompting chapters, we adopt a staged construction that first gathers reliable linguistic and visual evidence via curated ASR and frame-level descriptions, then refines coarse annotations into precise chapter boundaries and titles conditioned on the Temporal Chain-of-Thought, yielding fact-grounded, time-aligned narratives. We also observe that e-commerce
videos are fast-paced and information-dense, with visual
tokens dominating the input sequence. To enable efficient training while reducing input
tokens, we propose the Scene-Primed ASR-anchored Compressor (SPA-Compressor), which compresses
multimodal tokens into hierarchical scene and event representations guided by ASR semantic cues. Built upon these designs, our HiVid-Narrator framework achieves superior narrative quality with fewer input
tokens compared to existing methods.
Read the full abstract ▾
Computer Vision
Seeing Right but Saying Wrong: Inter- and Intra-Layer Refinement in MLLMs without Training
Shezheng Song, Shasha Li and Jie Yu
Multimodal Large
Language Models (MLLMs) have demonstrated strong capabilities across a variety of
vision-
language tasks. However, their internal
reasoning often exhibits a critical inconsistency: although deeper layers may attend to the correct visual regions, final predictions are frequently misled by noisy
attention from earlier layers. This results in a disconnect between what the model internally understands and what it ultimately expresses, a phenomenon we describe as seeing it right but saying it wrong. To address this issue, we propose DualPD, a dual-perspective decoding refinement strategy that enhances the visual understanding without any additional training. DualPD consists of two components. (1) The layer-wise
attention-guided contrastive logits module captures how the belief in the correct answer evolves by comparing output logits between layers that exhibit the largest
attention shift. (2) The head-wise information filtering module suppresses low-contribution
attention heads that focus on irrelevant regions, thereby improving
attention quality within each layer. Experiments conducted on both the LLaVA and Qwen-VL model families across multiple
multimodal benchmarks demonstrate that DualPD consistently improves accuracy without training, confirming its effectiveness and generalizability. The code will be released upon publication.
Read the full abstract ▾
Computer Vision
PulseMind: A Multi-Modal Medical Model for Real-World Clinical Diagnosis
Jiao Xu, Junwei Liu, Jiangwei Lao et al. (+9 more)
Recent advances in medical multi-modal models focus on specialized image analysis like dermatology, pathology, or radiology. However, they do not fully capture the complexity of real-world clinical diagnostics, which involve heterogeneous inputs and require ongoing contextual understanding during patient-physician interactions. To bridge this gap, we introduce PulseMind, a new family of multi-modal diagnostic models that integrates a systematically curated
dataset, a comprehensive evaluation
benchmark, and a tailored training framework. Specifically, we first construct a diagnostic
dataset, MediScope, which comprises 98,000 real-world multi-turn consultations and 601,500 medical images, spanning over 10 major clinical departments and more than 200 sub-specialties. Then, to better reflect the requirements of real-world clinical diagnosis, we develop the PulseMind
Benchmark, a multi-turn diagnostic consultation
benchmark with a four-dimensional evaluation protocol comprising proactiveness, accuracy, usefulness, and
language quality. Finally, we design a training framework tailored for multi-modal clinical diagnostics, centered around a core component named Comparison-based Reinforcement Policy Optimization (CRPO). Compared to absolute score rewards, CRPO uses relative preference signals from multi-dimensional com-parisons to provide stable and human-aligned training guidance. Extensive experiments demonstrate that PulseMind achieves competitive performance on both the diagnostic consultation
benchmark and public medical
benchmarks.
Read the full abstract ▾
All authors Jiao Xu, Junwei Liu, Jiangwei Lao, Qi Zhu, Yunpeng Zhao, Congyun Jin, Shinan Liu, Zhihong Lu, Lihe Zhang, Xin Chen, Jian Wang, Ping Wang
Computer Vision
Reconstruction Guided Few-shot Network For Remote Sensing Image Classification
Mohit Jaiswal, Naman Jain, Shivani Pathak et al. (+4 more)
Few-shot remote sensing image classification is challenging due to limited labeled samples and high variability in land-cover types. We propose a reconstruction-guided few-shot network (RGFS-Net) that enhances generalization to unseen classes while preserving consistency for seen categories. Our method incorporates a masked image reconstruction task, where parts of the input are occluded and reconstructed to encourage semantically rich feature learning. This auxiliary task strengthens spatial understanding and improves class discrimination under low-data settings. We evaluated the efficacy of EuroSAT and PatternNet
datasets under 1-shot and 5-shot protocols, our approach consistently outperforms existing baselines. The proposed method is simple, effective, and compatible with standard backbones, offering a robust solution for few-shot remote sensing classification. Codes are available at https://github.com/stark0908/RGFS.
Read the full abstract ▾
All authors Mohit Jaiswal, Naman Jain, Shivani Pathak, Mainak Singha, Nikunja Bihari Kar, Ankit Jha, Biplab Banerjee
Computer Vision
OSCAR: Open-Set CAD Retrieval from a Language Prompt and a Single Image
Tessa Pulli, Jean-Baptiste Weibel, Peter Hönig et al. (+3 more)
6D object pose estimation plays a crucial role in scene understanding for applications such as robotics and augmented reality. To support the needs of ever-changing object sets in such context, modern zero-shot object pose
estimators were developed to not require object-specific training but only rely on CAD models. Such models are hard to obtain once deployed, and a continuously changing and growing set of objects makes it harder to reliably identify the instance model of interest. To address this challenge, we introduce an Open-Set CAD
Retrieval from a
Language Prompt and a Single Image (OSCAR), a novel training-free method that retrieves a matching object model from an unlabeled 3D object database. During onboarding, OSCAR generates multi-view renderings of database models and annotates them with descriptive captions using an image captioning model. At inference, GroundedSAM detects the queried object in the input image, and multi-modal
embeddings are computed for both the Region-of-Interest and the database captions. OSCAR employs a two-stage
retrieval: text-based filtering using CLIP identifies candidate models, followed by image-based refinement using DINOv2 to select the most visually similar object. In our experiments we demonstrate that OSCAR outperforms all state-of-the-art methods on the cross-domain 3D model
retrieval benchmark MI3DOR. Furthermore, we demonstrate OSCAR's direct applicability in automating object model sourcing for 6D object pose estimation. We propose using the most similar object model for pose estimation if the exact instance is not available and show that OSCAR achieves an average precision of 90.48% during object
retrieval on the YCB-V object
dataset. Moreover, we demonstrate that the most similar object model can be utilized for pose estimation using Megapose achieving better results than a reconstruction-based approach.
Read the full abstract ▾
All authors Tessa Pulli, Jean-Baptiste Weibel, Peter Hönig, Matthias Hirschmanner, Markus Vincze, Andreas Holzinger
Computer Vision
Revisiting the Ordering of Channel and Spatial Attention: A Comprehensive Study on Sequential and Parallel Designs
Zhongming Liu and Bingbing Jiang
Attention mechanisms have become a core component of deep learning models, with Channel
Attention and Spatial
Attention being the two most representative architectures. Current research on their fusion strategies primarily bifurcates into sequential and parallel paradigms, yet the selection process remains largely empirical, lacking systematic analysis and unified principles. We systematically compare channel-spatial
attention combinations under a unified framework, building an evaluation suite of 18
topologies across four classes: sequential, parallel, multi-scale, and residual. Across two
vision and nine medical
datasets, we uncover a "data scale-method-performance" coupling law: (1) in few-shot tasks, the "Channel-Multi-scale Spatial" cascaded structure achieves optimal performance; (2) in medium-scale tasks, parallel learnable fusion architectures demonstrate superior results; (3) in large-scale tasks, parallel structures with dynamic gating yield the best performance. Additionally, experiments indicate that the "Spatial-Channel" order is more stable and effective for fine-grained classification, while residual connections mitigate vanishing gradient problems across varying data scales. We thus propose scenario-based guidelines for building future
attention modules. Code is open-sourced at https://github.com/DWlzm.
Read the full abstract ▾
Computer Vision
Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understanding
Jianghao Yin, Qingbin Li, Kun Sun et al. (+10 more)
While
Multimodal Large
Language Models (MLLMs) excel at single-image understanding, they exhibit significantly degraded performance in multi-image
reasoning scenarios. Multi-image
reasoning presents fundamental challenges including complex inter-relationships between images and scattered critical information across image sets. Inspired by human cognitive processes, we propose a Cognition-Inspired Meta-Action Framework (CINEMA), which decomposes multi-image
reasoning into five structured meta-actions: Global, Focus, Hint, Think, and Answer, explicitly modeling the sequential cognitive steps humans naturally employ. For cold-start training, we introduce a
Retrieval-Based Tree Sampling strategy that generates high-quality meta-action trajectories to bootstrap the model with
reasoning patterns. During reinforcement learning, we adopt a two-stage paradigm: an exploration phase with Diversity-Preserving Strategy to avoid
entropy collapse, followed by an annealed exploitation phase with DAPO to gradually strengthen exploitation. To train our model, we construct a
dataset of 56k cold-start and 58k reinforcement learning instances spanning multi-image, multi-frame, and single-image tasks. We conduct extensive evaluations on multi-image
reasoning benchmarks,
video understanding
benchmarks, and single-image
benchmarks, achieving competitive state-of-the-art performance on several key
benchmarks. Our model surpasses GPT-4o on the MUIR and MVMath
benchmarks and notably outperforms specialized
video reasoning models on
video understanding
benchmarks, demonstrating the effectiveness and generalizability of our human cognition-inspired
reasoning framework.
Read the full abstract ▾
All authors Jianghao Yin, Qingbin Li, Kun Sun, Cheng Ding, Jie Wang, Qin Chen, Jie Zhou, Nan Wang, Changqing Li, Pei Wu, Jian Xu, Zheming Yang, Liang He
Computer Vision
Inference-Time Scaling for Visual AutoRegressive modeling by Searching Representative Samples
Weidong Tang, Xinyan Wan, Siyu Li et al. (+1 more)
While inference-time scaling has significantly enhanced generative quality in large
language and
diffusion models, its application to vector-quantized (VQ) visual autoregressive modeling (VAR) remains unexplored. We introduce VAR-Scaling, the first general framework for inference-time scaling in VAR, addressing the critical challenge of discrete latent spaces that prohibit continuous path search. We find that VAR scales exhibit two distinct pattern types: general patterns and specific patterns, where later-stage specific patterns conditionally optimize early-stage general patterns. To overcome the discrete latent space barrier in VQ models, we map sampling spaces to quasi-continuous feature spaces via kernel density estimation (KDE), where high-density samples approximate stable, high-quality solutions. This transformation enables effective
navigation of sampling distributions. We propose a density-adaptive hybrid sampling strategy: Top-k sampling focuses on high-density regions to preserve quality near distribution modes, while Random-k sampling explores low-density areas to maintain diversity and prevent premature convergence. Consequently, VAR-Scaling optimizes sample fidelity at critical scales to enhance output quality. Experiments in class-conditional and text-to-image evaluations demonstrate significant improvements in inference process. The code is available at https://github.com/WD7ang/VAR-Scaling.
Read the full abstract ▾
All authors Weidong Tang, Xinyan Wan, Siyu Li, Xiumei Wang
Computer Vision
A Visual Semantic Adaptive Watermark grounded by Prefix-Tuning for Large Vision-Language Model
Qi Zheng, Shuliang Liu, Yu Huang et al. (+8 more)
Watermarking has emerged as a pivotal solution for content traceability and intellectual property protection in Large
Vision-
Language Models (LVLMs). However,
vision-agnostic watermarks introduce visually irrelevant
tokens and disrupt visual grounding by enforcing indiscriminate pseudo-random biases, while some semantic-aware methods incur prohibitive inference latency due to rejection sampling. In this paper, we propose the VIsual Semantic Adaptive Watermark (VISA-Mark), a novel framework that embeds detectable signals while strictly preserving visual fidelity. Our approach employs a lightweight, efficiently trained prefix-tuner to extract dynamic Visual-Evidence Weights, which quantify the evidentiary support for candidate
tokens based on the visual input. These weights guide an adaptive vocabulary partitioning and logits perturbation mechanism, concentrating watermark strength specifically on visually-supported
tokens. By actively aligning the watermark with visual evidence, VISA-Mark effectively maintains visual fidelity. Empirical results confirm that VISA-Mark outperforms conventional methods with a 7.8% improvement in visual consistency (Chair-I) and superior semantic fidelity. The framework maintains highly competitive detection accuracy (96.88% AUC) and robust attack resilience (99.3%) without sacrificing inference efficiency, effectively establishing a new standard for reliability-preserving
multimodal watermarking.
Read the full abstract ▾
All authors Qi Zheng, Shuliang Liu, Yu Huang, Sihang Jia, Jungang Li, Lyuhao Chen, Junhao Chen, Hanqian Li, Aiwei Liu, Yibo Yan, Xuming Hu
Computer Vision
VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understanding
Jiapeng Shi, Junke Wang, Zuyao You et al. (+2 more)
This paper presents VideoLoom, a unified
Video Large
Language Model (
Video LLM) for joint spatial-temporal understanding. To facilitate the development of fine-grained spatial and temporal localization capabilities, we curate LoomData-8.7k, a human-centric
video dataset with temporally grounded and spatially localized captions. With this, VideoLoom achieves state-of-the-art or highly competitive performance across a variety of spatial and temporal
benchmarks (e.g., 63.1 J&F on ReVOS for referring
video object
segmentation, and 48.3 R1@0.7 on Charades-STA for temporal grounding). In addition, we introduce LoomBench, a novel
benchmark consisting of temporal, spatial, and compositional
video-question pairs, enabling a comprehensive evaluation of
Video LLMs from diverse aspects. Collectively, these contributions offer a universal and effective suite for joint spatial-temporal
video understanding, setting a new standard in
multimodal intelligence.
Read the full abstract ▾
All authors Jiapeng Shi, Junke Wang, Zuyao You, Bo He, Zuxuan Wu
Computer Vision
Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Models
Yuanyang Yin, Yufan Deng, Shenghai Yuan et al. (+3 more)
The task of Image-to-
Video (I2V) generation aims to synthesize a
video from a reference image and a text
prompt. This requires
diffusion models to reconcile high-frequency visual constraints and low-frequency textual guidance during the denoising process. However, while existing I2V models prioritize visual consistency, how to effectively couple this dual guidance to ensure strong adherence to the text
prompt remains underexplored. In this work, we observe that in
Diffusion Transformer (DiT)-based I2V models, certain intermediate layers exhibit weak semantic responses (termed Semantic-Weak Layers), as indicated by a measurable drop in text-visual similarity. We attribute this to a phenomenon called Condition Isolation, where
attention to visual features becomes partially detached from text guidance and overly relies on learned visual priors. To address this, we propose Focal Guidance (FG), which enhances the controllability from Semantic-Weak Layers. FG comprises two mechanisms: (1) Fine-grained Semantic Guidance (FSG) leverages CLIP to identify key regions in the reference frame and uses them as anchors to guide Semantic-Weak Layers. (2)
Attention Cache transfers
attention maps from semantically responsive layers to Semantic-Weak Layers, injecting explicit semantic signals and alleviating their over-reliance on the model's learned visual priors, thereby enhancing adherence to textual instructions. To further validate our approach and address the lack of evaluation in this direction, we introduce a
benchmark for assessing instruction following in I2V models. On this
benchmark, Focal Guidance proves its effectiveness and generalizability, raising the total score on Wan2.1-I2V to 0.7250 (+3.97%) and boosting the MMDiT-based HunyuanVideo-I2V to 0.5571 (+7.44%).
Read the full abstract ▾
All authors Yuanyang Yin, Yufan Deng, Shenghai Yuan, Kaipeng Zhang, Xiao Yang, Feng Zhao
Computer Vision
GenDet: Painting Colored Bounding Boxes on Images via Diffusion Model for Object Detection
Chen Min, Chengyang Li, Fanjie Kong et al. (+3 more)
This paper presents GenDet, a novel framework that redefines object detection as an image generation task. In contrast to traditional approaches, GenDet adopts a pioneering approach by leveraging generative modeling: it conditions on the input image and directly generates bounding boxes with semantic annotations in the original image space. GenDet establishes a conditional generation architecture built upon the large-scale pre-trained Stable
Diffusion model, formulating the detection task as semantic constraints within the latent space. It enables precise control over bounding box positions and category attributes, while preserving the flexibility of the generative model. This novel methodology effectively bridges the gap between generative models and discriminative tasks, providing a fresh perspective for constructing unified visual understanding systems. Systematic experiments demonstrate that GenDet achieves competitive accuracy compared to discriminative detectors, while retaining the flexibility characteristic of generative methods.
Read the full abstract ▾
All authors Chen Min, Chengyang Li, Fanjie Kong, Qi Zhu, Dawei Zhao, Liang Xiao
Computer Vision
PALUM: Part-based Attention Learning for Unified Motion Retargeting
Siqi Liu, Maoyu Wang, Bo Dai et al. (+1 more)
Retargeting motion between characters with different skeleton structures is a fundamental challenge in computer animation. When source and target characters have vastly different bone arrangements, maintaining the original motion's semantics and quality becomes increasingly difficult. We present PALUM, a novel approach that learns common motion representations across diverse skeleton
topologies by partitioning joints into semantic body parts and applying
attention mechanisms to capture spatio-temporal relationships. Our method transfers motion to target skeletons by leveraging these skeleton-agnostic representations alongside target-specific structural information. To ensure robust learning and preserve motion fidelity, we introduce a cycle consistency mechanism that maintains semantic coherence throughout the retargeting process. Extensive experiments demonstrate superior performance in handling diverse skeletal structures while maintaining motion realism and semantic fidelity, even when generalizing to previously unseen skeleton-motion combinations. We will make our implementation publicly available to support future research.
Read the full abstract ▾
All authors Siqi Liu, Maoyu Wang, Bo Dai, Cewu Lu
Computer Vision
From Landslide Conditioning Factors to Satellite Embeddings: Evaluating the Utilisation of Google AlphaEarth for Landslide Susceptibility Mapping using Deep Learning
Yusen Cheng, Qinfeng Zhu and Lei Fan
Data-driven landslide susceptibility mapping (LSM) typically relies on landslide conditioning factors (LCFs), whose availability, heterogeneity, and preprocessing-related uncertainties can constrain mapping reliability. Recently, Google AlphaEarth (AE)
embeddings, derived from multi-source geospatial observations, have emerged as a unified representation of Earth surface conditions. This study evaluated the potential of AE
embeddings as alternative predictors for LSM. Two AE representations, including retained principal components and the full set of 64
embedding bands, were systematically compared with conventional LCFs across three study areas (Nantou County, Taiwan; Hong Kong; and part of Emilia-Romagna, Italy) using three deep learning models (CNN1D, CNN2D, and
Vision Transformer). Performance was assessed using multiple evaluation metrics, ROC-AUC analysis, error statistics, and spatial pattern assessment. Results showed that AE-based models consistently outperformed LCFs across all regions and models, yielding higher F1-scores, AUC values, and more stable error distributions. Such improvement was most pronounced when using the full 64-band AE representation, with F1-score improvements of approximately 4% to 15% and AUC increased ranging from 0.04 to 0.11, depending on the study area and model. AE-based susceptibility maps also exhibited clearer spatial correspondence with observed landslide occurrences and enhanced sensitivity to localised landslide-prone conditions. Performance improvements were more evident in Nantou and Emilia than in Hong Kong. SHAP analysis further revealed regional variability in the contribution of individual AE bands, while several bands showed consistently high importance across all three study areas. These findings highlight the strong potential of AE
embeddings as a standardised and information-rich alternative to conventional LCFs for LSM.
Read the full abstract ▾
Computer Vision
Universal Adversarial Purification with DDIM Metric Loss for Stable Diffusion
Li Zheng, Liangbin Xie, Jiantao Zhou et al. (+1 more)
Stable
Diffusion (SD) often produces degraded outputs when the training
dataset contains
adversarial noise.
Adversarial purification offers a promising solution by removing
adversarial noise from contaminated data. However, existing purification methods are primarily designed for classification tasks and fail to address SD-specific
adversarial strategies, such as attacks targeting the VAE encoder, UNet denoiser, or both. To address the gap in SD security, we propose Universal
Diffusion Adversarial Purification (UDAP), a novel framework tailored for defending
adversarial attacks targeting SD models. UDAP leverages the distinct reconstruction behaviors of clean and
adversarial images during Denoising
Diffusion Implicit Models (DDIM) inversion to optimize the purification process. By minimizing the DDIM metric loss, UDAP can effectively remove
adversarial noise. Additionally, we introduce a dynamic epoch adjustment strategy that adapts optimization iterations based on reconstruction errors, significantly improving efficiency without sacrificing purification quality. Experiments demonstrate UDAP's robustness against diverse
adversarial methods, including PID (VAE-targeted), Anti-DreamBooth (UNet-targeted), MIST (hybrid), and robustness-enhanced variants like Anti-
Diffusion (Anti-DF) and MetaCloak. UDAP also generalizes well across SD versions and text
prompts, showcasing its practical applicability in real-world scenarios.
Read the full abstract ▾
All authors Li Zheng, Liangbin Xie, Jiantao Zhou, He YiMin
Computer Vision · Cross-listed from cs.RO
HERE: Hierarchical Active Exploration of Radiance Field with Epistemic Uncertainty Minimization
Taekbeom Lee, Dabin Kim, Youngseok Jang et al. (+1 more)
We present HERE, an active 3D scene reconstruction framework based on neural radiance fields, enabling high-fidelity implicit mapping. Our approach centers around an active learning strategy for camera trajectory generation, driven by accurate identification of unseen regions, which supports efficient data acquisition and precise scene reconstruction. The key to our approach is epistemic uncertainty quantification based on evidential deep learning, which directly captures data insufficiency and exhibits a strong correlation with reconstruction errors. This allows our framework to more reliably identify unexplored or poorly reconstructed regions compared to existing methods, leading to more informed and targeted exploration. Additionally, we design a hierarchical exploration strategy that leverages learned epistemic uncertainty, where local planning extracts target viewpoints from high-uncertainty voxels based on visibility for trajectory generation, and global planning uses uncertainty to guide large-scale coverage for efficient and comprehensive reconstruction. The effectiveness of the proposed method in active 3D reconstruction is demonstrated by achieving higher reconstruction completeness compared to previous approaches on photorealistic simulated scenes across varying scales, while a hardware demonstration further validates its real-world applicability. Project page: https://taekbum.github.io/here/
Read the full abstract ▾
All authors Taekbeom Lee, Dabin Kim, Youngseok Jang, H. Jin Kim
Computer Vision
Language-Grounded Multi-Domain Image Translation via Semantic Difference Guidance
Jongwon Ryu, Joonhyung Park, Jaeho Han et al. (+4 more)
Multi-domain image-to-image translation re quires grounding semantic differences ex pressed in natural
language prompts into corresponding visual transformations, while preserving unrelated structural and seman tic content. Existing methods struggle to maintain structural integrity and provide fine grained, attribute-specific control, especially when multiple domains are involved. We propose LACE (
Language-grounded Attribute Controllable Translation), built on two compo nents: (1) a GLIP-Adapter that fuses global semantics with local structural features to pre serve consistency, and (2) a Multi-Domain Control Guidance mechanism that explicitly grounds the semantic delta between source and target
prompts into per-attribute translation vec tors, aligning linguistic semantics with domain level visual changes. Together, these modules enable compositional multi-domain control with independent strength modulation for each attribute. Experiments on CelebA(Dialog) and BDD100K demonstrate that LACE achieves high visual fidelity, structural preservation, and interpretable domain-specific control, surpass ing prior baselines. This positions LACE as a cross-modal content generation framework bridging
language semantics and controllable visual translation.
Read the full abstract ▾
All authors Jongwon Ryu, Joonhyung Park, Jaeho Han, Yeong-Seok Kim, Hye-rin Kim, Sunjae Yoon, Junyeong Kim
Computer Vision
VENUS: Visual Editing with Noise Inversion Using Scene Graphs
Thanh-Nhan Vo, Trong-Thuan Nguyen, Tam V. Nguyen et al. (+1 more)
State-of-the-art text-based image editing models often struggle to balance background preservation with semantic consistency, frequently resulting either in the synthesis of entirely new images or in outputs that fail to realize the intended edits. In contrast, scene graph-based image editing addresses this limitation by providing a structured representation of semantic entities and their relations, thereby offering improved controllability. However, existing scene graph editing methods typically depend on model fine-tuning, which incurs high computational cost and limits scalability. To this end, we introduce VENUS (Visual Editing with Noise inversion Using Scene graphs), a training-free framework for scene graph-guided image editing. Specifically, VENUS employs a split
prompt conditioning strategy that disentangles the target object of the edit from its background context, while simultaneously leveraging noise inversion to preserve fidelity in unedited regions. Moreover, our proposed approach integrates scene graphs extracted from
multimodal large
language models with
diffusion backbones, without requiring any additional training. Empirically, VENUS substantially improves both background preservation and semantic
alignment on PIE-Bench, increasing PSNR from 22.45 to 24.80, SSIM from 0.79 to 0.84, and reducing LPIPS from 0.100 to 0.070 relative to the state-of-the-art scene graph editing model (SGEdit). In addition, VENUS enhances semantic consistency as measured by CLIP similarity (24.97 vs. 24.19). On EditVal, VENUS achieves the highest fidelity with a 0.87 DINO score and, crucially, reduces per-image runtime from 6-10 minutes to only 20-30 seconds. Beyond scene graph-based editing, VENUS also surpasses strong text-based editing baselines such as LEDIT++ and P2P+DirInv, thereby demonstrating consistent improvements across both paradigms.
Read the full abstract ▾
All authors Thanh-Nhan Vo, Trong-Thuan Nguyen, Tam V. Nguyen, Minh-Triet Tran
Computer Vision
SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesis
Jeongjun Choi, Yeonsoo Park and H. Jin Kim
We present SceneNAT, a masked non-autoregressive
Transformer for 3D indoor scene synthesis from natural
language instructions. It generates complete scenes in a few parallel decoding passes, improving both quality and efficiency over prior methods. SceneNAT is trained via masked modeling over fully discretized representations of both semantic and spatial attributes. By applying a masking strategy at both the attribute level and the instance level, the model can better capture intra-object and inter-object structure. To boost relational
reasoning, SceneNAT employs a relational
reasoning module (RRM) that captures implicit spatial constraints. By formulating relation modeling as a set prediction task, it extracts structure-aware features to guide the layout generation without explicit sequential parsing. Extensive experiments on 3D-FRONT show that SceneNAT outperforms state-of-the-art autoregressive and
diffusion baselines in both semantic compliance and spatial arrangement accuracy while using substantially lower computational cost, enabling high-throughput generation of diverse scenes at scale.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.CR
BlindU: Blind Machine Unlearning without Revealing Erasing Data
Weiqi Wang, Zhiyi Tian, Chenhan Zhang et al. (+1 more)
Machine unlearning enables data holders to remove the contribution of their specified samples from trained models to protect their
privacy. However, it is paradoxical that most unlearning methods require the unlearning requesters to firstly upload their data to the server as a prerequisite for unlearning. These methods are infeasible in many
privacy-preserving scenarios where servers are prohibited from accessing users' data, such as
federated learning (FL). In this paper, we explore how to implement unlearning under the condition of not uncovering the erasing data to the server. We propose Blind Unlearning (BlindU), which carries out unlearning using compressed representations instead of original inputs. BlindU only involves the server and the unlearning user: the user locally generates
privacy-preserving representations, and the server performs unlearning solely on these representations and their labels. For the FL model training, we employ the information bottleneck (IB) mechanism. The encoder of the IB-based FL model learns representations that distort maximum task-irrelevant information from inputs, allowing FL users to generate compressed representations locally. For effective unlearning using compressed representation, BlindU integrates two dedicated unlearning modules tailored explicitly for IB-based models and uses a multiple gradient descent
algorithm to balance forgetting and utility retaining. While IB compression already provides protection for task-irrelevant information of inputs, to further enhance the
privacy protection, we introduce a noise-free differential
privacy (DP) masking method to deal with the raw erasing data before compressing. Theoretical analysis and extensive experimental results illustrate the superiority of BlindU in
privacy protection and unlearning effectiveness compared with the best existing
privacy-preserving unlearning
benchmarks.
Read the full abstract ▾
All authors Weiqi Wang, Zhiyi Tian, Chenhan Zhang, Shui Yu
Computer Vision
SIRR-LMM: Single-image Reflection Removal via Large Multimodal Model
Yu Guo, Zhiqiang Lao, Xiyun Song et al. (+2 more)
Glass surfaces create complex interactions of reflected and transmitted light, making single-image reflection removal (SIRR) challenging. Existing
datasets suffer from limited physical realism in synthetic data or insufficient scale in real captures. We introduce a synthetic
dataset generation framework that path-traces 3D glass models over real background imagery to create physically accurate reflection scenarios with varied glass properties, camera settings, and post-processing effects. To leverage the capabilities of Large
Multimodal Model (LMM), we concatenate the image layers into a single composite input, apply joint captioning, and fine-tune the model using task-specific LoRA rather than full-parameter training. This enables our approach to achieve improved reflection removal and separation performance compared to state-of-the-art methods.
Read the full abstract ▾
All authors Yu Guo, Zhiqiang Lao, Xiyun Song, Yubin Zhou, Heather Yu
Computer Vision
ShowUI-Aloha: Human-Taught GUI Agent
Yichun Zhang, Xiangwu Guo, Yauhong Goh et al. (+5 more)
Graphical User Interfaces (GUIs) are central to human-computer interaction, yet automating complex GUI tasks remains a major challenge for autonomous
agents, largely due to a lack of scalable, high-quality training data. While recordings of human demonstrations offer a rich data source, they are typically long, unstructured, and lack annotations, making them difficult for
agents to learn from.To address this, we introduce ShowUI-Aloha, a comprehensive pipeline that transforms unstructured, in-the-wild human screen recordings from desktop environments into structured, actionable tasks. Our framework includes four key components: A recorder that captures screen
video along with precise user interactions like mouse clicks, keystrokes, and scrolls. A learner that semantically interprets these raw interactions and the surrounding visual context, translating them into descriptive natural
language captions. A planner that reads the parsed demonstrations, maintains task states, and dynamically formulates the next high-level action plan based on contextual
reasoning. An executor that faithfully carries out these action plans at the OS level, performing precise clicks, drags, text inputs, and window operations with safety checks and real-time feedback. Together, these components provide a scalable solution for collecting and parsing real-world human data, demonstrating a viable path toward building general-purpose GUI
agents that can learn effectively from simply observing humans.
Read the full abstract ▾
All authors Yichun Zhang, Xiangwu Guo, Yauhong Goh, Jessica Hu, Zhiheng Chen, Xin Wang, Difei Gao, Mike Zheng Shou
Computer Vision
DIVER: Dynamic Iterative Visual Evidence Reasoning for Multimodal Fake News Detection
Weilin Zhou, Zonghao Ying, Chunlei Meng et al. (+6 more)
Multimodal fake news detection is crucial for mitigating
adversarial misinformation. Existing methods, relying on static fusion or LLMs, face computational redundancy and
hallucination risks due to weak visual foundations. To address this, we propose DIVER (Dynamic Iterative Visual Evidence
Reasoning), a framework grounded in a progressive, evidence-driven
reasoning paradigm. DIVER first establishes a strong text-based baseline through
language analysis, leveraging intra-modal consistency to filter unreliable or hallucinated claims. Only when textual evidence is insufficient does the framework introduce visual information, where inter-modal
alignment verification adaptively determines whether deeper visual inspection is necessary. For samples exhibiting significant cross-modal semantic discrepancies, DIVER selectively invokes fine-grained visual tools (e.g., OCR and dense captioning) to extract task-relevant evidence, which is iteratively aggregated via uncertainty-aware fusion to refine
multimodal reasoning. Experiments on Weibo, Weibo21, and GossipCop demonstrate that DIVER outperforms state-of-the-art baselines by an average of 2.72%, while optimizing inference efficiency with a reduced latency of 4.12 s.
Read the full abstract ▾
All authors Weilin Zhou, Zonghao Ying, Chunlei Meng, Jiahui Liu, Hengyang Zhou, Quanchen Zou, Deyue Zhang, Dongdong Yang, Xiangzheng Zhang
Computer Vision
Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification
Shu Shen, C. L. Philip Chen and Tong Zhang
Reliable learning of
multimodal data (e.g., multi-omics) is a widely concerning issue, especially in safety-critical applications such as medical diagnosis. However, low-quality data induced by
multimodal noise poses a major challenge in this domain, causing existing methods to suffer from two key limitations. First, they struggle to handle heterogeneous data noise, hindering robust
multimodal representation learning. Second, they exhibit limited adaptability and generalization when encountering previously unseen noise. To address these issues, we propose Test-time Adaptive Hierarchical Co-enhanced Denoising Network (TAHCD). On one hand, TAHCD introduces the Adaptive Stable Subspace
Alignment and Sample-Adaptive Confidence
Alignment to reliably remove heterogeneous noise. They account for noise at both global and instance levels and enable jointly removal of modality-specific and cross-modality noise, achieving robust learning. On the other hand, TAHCD introduces Test-Time Cooperative Enhancement, which adaptively updates the model in response to input noise in a label-free manner, thus improving generalization. This is achieved by collaboratively enhancing the joint removal process of modality-specific and cross-modality noise across global and instance levels according to sample noise. Experiments on multiple
benchmarks demonstrate that the proposed method achieves superior classification performance, robustness, and generalization compared with state-of-the-art reliable
multimodal learning approaches.
Read the full abstract ▾
Computer Vision
Motion Focus Recognition in Fast-Moving Egocentric Video
Si-En Hong, James Tribble, Alexander Lake et al. (+8 more)
From
Vision-
Language-Action (VLA) systems to robotics, existing
egocentric datasets primarily focus on action recognition tasks, while largely overlooking the inherent role of motion analysis in sports and other fast-movement scenarios. To bridge this gap, we propose a real-time motion focus recognition method that estimates the subject's locomotion intention from any
egocentric video. We leverage the foundation model for camera pose estimation and introduce system-level optimizations to enable efficient and scalable inference. Evaluated on a collected
egocentric action
dataset, our method achieves real-time performance with manageable
memory consumption through a sliding batch inference strategy. This work makes motion-centric analysis practical for edge deployment and offers a complementary perspective to existing
egocentric studies on sports and fast-movement activities.
Read the full abstract ▾
All authors Si-En Hong, James Tribble, Alexander Lake, Hao Wang, Chaoyi Zhou, Ashish Bastola, Siyu Huang, Eisa Chaudhary, Brian Canada, Ismahan Arslan-Ari, Abolfazl Razi
Computer Vision · Cross-listed from cs.CR
Proof of Reasoning for Privacy Enhanced Federated Blockchain Learning at the Edge
James Calo and Benny Lo
Consensus mechanisms are the core of any blockchain system. However, the majority of these mechanisms do not target
federated learning directly nor do they aid in the aggregation step. This paper introduces Proof of
Reasoning (PoR), a novel consensus mechanism specifically designed for
federated learning using blockchain, aimed at preserving data
privacy, defending against malicious attacks, and enhancing the validation of participating networks. Unlike generic blockchain consensus mechanisms commonly found in the literature, PoR integrates three distinct processes tailored for
federated learning. Firstly, a masked autoencoder (MAE) is trained to generate an encoder that functions as a feature map and obfuscates input data, rendering it resistant to human reconstruction and model inversion attacks. Secondly, a downstream classifier is trained at the edge, receiving input from the trained encoder. The downstream network's weights, a single encoded datapoint, the network's output and the ground truth are then added to a block for
federated aggregation. Lastly, this data facilitates the aggregation of all participating networks, enabling more complex and verifiable aggregation methods than previously possible. This three-stage process results in more robust networks with significantly reduced computational complexity, maintaining high accuracy by training only the downstream classifier at the edge. PoR scales to large IoT networks with low latency and storage growth, and adapts to evolving data, regulations, and network conditions.
Read the full abstract ▾