Computer Vision · Cross-listed from cs.RO
DuetHOI: Language-Guided Bimanual Hand–Object Motion Generation with Articulation Planning and Contact Refinement
Zhi Wang, Yuyan Liu, Liu liu et al. (+2 more)
Bimanual articulated-object interaction generation requires a model to capture the
evolution of object articulation, coordination between the two hands, and fine-grained hand–object contact. Existing methods typically encode object and hand motion as a unified high-dimensional sequence, making it difficult to explicitly accommodate the different scales of manipulation progress, relative bimanual motion, and local hand pose. We propose DuetHOI, a structured framework for bimanual articulated-object interaction generation. DuetHOI first uses ContactVAE to predict contact intent on the object surface from
language and object geometry. Conditioned on this spatial intent, ArtPlanner provides a trajectory-level articulation reference, and DualFormer generates the global interaction using object states, object-centered hand positions, and compact hand-pose
tokens learned by ManiVAE. After global generation, ProxiRefine fixes the object trajectory and residually corrects the two hands using current hand–object proximity, improving local surface
alignment. Experiments across articulated and rigid objects and single- and bimanual settings show that DuetHOI outperforms three adapted baselines on most contact and hand–object consistency metrics, with particularly strong results in bimanual articulated interaction generation.
Read the full abstract ▾
All authors Zhi Wang, Yuyan Liu, Liu liu, Ruonan Liu, Ruixuan Liu
Computer Vision
AULLM++: Structured-Token-Conditioned Large Language Models for Micro-Expression Action Unit Detection
Zhishu Liu, Kaishen Yuan, Bo Zhao et al. (+2 more)
Micro-expression Action Unit (AU) detection identifies localized AUs from subtle facial muscle activations, providing a foundation for decoding affective cues. Previous methods face three key limitations: (1) heavy reliance on low-density visual information, rendering discriminative evidence vulnerable to background noise; (2) coarse-grained feature processing that misaligns with the demand for fine-grained representations; and (3) neglect of inter-AU correlations, restricting the parsing of complex expression patterns. We propose AULLM++, a
reasoning-oriented framework leveraging Large
Language Models (LLMs), which injects visual features into textual
prompts as actionable semantic premises to guide inference. It formulates AU prediction into three stages: evidence construction, structure modeling, and deduction-based prediction. Specifically, a Multi-Granularity Evidence-Enhanced Fusion Projector (MGE-EFP) fuses mid-level texture cues with high-level semantics, distilling them into a compact Content
Token (CT). Furthermore, inspired by micro- and macro-expression AU correspondence, we encode AU relationships as a sparse structural prior and learn interaction strengths via a Relation-Aware AU Graph Neural Network (R-AUGNN), producing an Instruction
Token (IT). We then fuse CT and IT into a structured textual
prompt and introduce Counterfactual Consistency Regularization (CCR) to construct counterfactual samples, enhancing the model's generalization. Extensive experiments demonstrate AULLM++ achieves state-of-the-art performance on standard
benchmarks and exhibits superior cross-domain generalization.
Read the full abstract ▾
All authors Zhishu Liu, Kaishen Yuan, Bo Zhao, Hui Ma, Zitong Yu
Computer Vision
Real-Time Drone Detection in Event Cameras via Per-Pixel Frequency Analysis
Michael Bezick and Majid Sahin
Detecting fast-moving objects, such as unmanned aerial vehicle (
UAV), from event camera data is challenging due to the sparse, asynchronous nature of the input. Traditional Discrete
Fourier Transforms (DFT) are effective at identifying periodic signals, such as spinning rotors, but they assume uniformly sampled data, which event cameras do not provide. We propose a novel per-pixel temporal analysis framework using the Non-uniform Discrete
Fourier Transform (NDFT), which we call Drone Detection via Harmonic Fingerprinting (DDHF). Our method uses purely analytical techniques that identify the frequency signature of drone rotors, as characterized by frequency combs in their power spectra, enabling a tunable and generalizable
algorithm that achieves accurate real-time localization of
UAV. We compare against a YOLO detector under equivalent conditions, demonstrating improvement in accuracy and latency across a difficult array of drone speeds, distances, and scenarios. DDHF achieves an average localization F1 score of 90.89% and average latency of 2.39ms per frame, while YOLO achieves an F1 score of 66.74% and requires 12.40ms per frame. Through utilization of purely analytic techniques, DDHF is quickly tuned on small data, easily interpretable, and achieves competitive accuracies and latencies to deep learning alternatives.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Rectified flow-based prediction of post-treatment brain MRI from pre-radiotherapy priors for patients with glioma
Selena Huisman, Nordin Belkacemi, Vera C. Keil et al. (+2 more)
Brain tumors result in 20 years of lost life on average. Standard therapies induce complex structural changes in the
brain that are monitored through MRI. Recent developments in artificial intelligence (AI) enable conditional
multimodal image generation from clinical data. In this study, we investigate AI-driven generation of follow-up MRI in patients with intracranial tumors through conditional image generation. This approach enables realistic modeling of post-radiotherapy changes, allowing for treatment optimization. The public SAILOR
dataset of 25 patients was used to create a 2D rectified flow model conditioned on axial slices of pre-treatment MRI and RT dose maps. Cross-
attention conditioning was used to incorporate temporal and chemotherapy data. The resulting images were validated with structural similarity index measure (SSIM), peak signal-to-noise ratio (PSNR), Dice scores and Jacobian determinants. The resulting model generates realistic follow-up MRI for any time point, while integrating treatment information. Comparing real versus predicted images, SSIM is 0.88, and PSNR is 22.82. Tissue
segmentations from real versus predicted MRI result in a mean Dice-Sørensen coefficient (DSC) of 0.91. The rectified flow (RF) model enables up to 250x faster inference than Denoising
Diffusion Probabilistic Models (DDPM). The proposed model generates realistic follow-up MRI in real-time, preserving both semantic and visual fidelity as confirmed by image quality metrics and tissue
segmentations. Conditional generation allows counterfactual simulations by varying treatment parameters, producing predicted morphological changes. This capability has potential to support adaptive treatment dose planning and personalized outcome prediction for patients with intracranial tumors. Code will be available upon peer-reviewed publication at: https://github.com/SelenaIHuisman/RF-GlioPREDICT
Read the full abstract ▾
All authors Selena Huisman, Nordin Belkacemi, Vera C. Keil, Joost Verhoeff, Szabolcs David
Computer Vision
This Looks Distinctly Like That: Grounding Interpretable Recognition in Stiefel Geometry against Neural Collapse
Junhao Jia, Jiaqi Wang, Yunyou Liu et al. (+4 more)
Prototype networks provide an intrinsic case based explanation mechanism, but their interpretability is often undermined by prototype collapse, where multiple prototypes degenerate to highly redundant evidence. We attribute this failure mode to the terminal dynamics of Neural Collapse, where cross
entropy optimization suppresses intra class variance and drives class conditional features toward a low dimensional limit. To mitigate this, we propose Adaptive
Manifold Prototypes (AMP), a framework that leverages Riemannian optimization on the Stiefel
manifold to represent class prototypes as orthonormal bases and make rank one prototype collapse infeasible by construction. AMP further learns class specific effective rank via a proximal gradient update on a nonnegative capacity vector, and introduces spatial regularizers that reduce rotational ambiguity and encourage localized, non overlapping part evidence. Extensive experiments on fine-grained
benchmarks demonstrate that AMP achieves state-of-the-art classification accuracy while significantly improving causal faithfulness over prior interpretable models.
Read the full abstract ▾
All authors Junhao Jia, Jiaqi Wang, Yunyou Liu, Haodong Jing, Yueyi Wu, Xian Wu, Yefeng Zheng
Computer Vision
Diffusion-Based Data Augmentation for Image Recognition: A Systematic Analysis and Evaluation
Zekun Li, Yinghuan Shi, Yang Gao et al. (+1 more)
Diffusion-based data augmentation (DiffDA) has emerged as a promising approach to improving classification performance under data scarcity. However, existing works vary significantly in task configurations, model choices, and experimental pipelines, making it difficult to fairly compare methods or assess their effectiveness across different scenarios. Moreover, there remains a lack of systematic understanding of the full DiffDA workflow. In this work, we introduce UniDiffDA, a unified analytical framework that decomposes DiffDA methods into three core components: model fine-tuning, sample generation, and sample utilization. This perspective enables us to identify key differences among existing methods and clarify the overall design space. Building on this framework, we develop a comprehensive and fair evaluation protocol, benchmarking representative DiffDA methods across diverse low-data classification tasks. Extensive experiments reveal the relative strengths and limitations of different DiffDA strategies and offer practical insights into method design and deployment. All methods are re-implemented within a unified codebase, with full release of code and configurations to ensure reproducibility and to facilitate future research.
Read the full abstract ▾
All authors Zekun Li, Yinghuan Shi, Yang Gao, Dong Xu
Computer Vision
$\Delta$VLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variation
Yijie Zhu, Jie He, Rui Shao et al. (+4 more)
Recent
vision-
language-action (VLA) models have significantly advanced robotic manipulation by unifying perception,
reasoning, and control. To achieve such integration, recent studies adopt a predictive paradigm that models future visual states or world knowledge to guide action generation. However, these models emphasize
forecasting outcomes rather than
reasoning about the underlying process of change, which is essential for determining how to act. To address this, we propose $\Delta$VLA, a prior-guided framework that models world-knowledge variations relative to an explicit current-world knowledge prior for action generation, rather than regressing absolute future world states. Specifically, 1) to construct the current world knowledge prior, we propose the Prior-Guided WorldKnowledge Extractor (PWKE). It extracts manipulable regions, spatial relations, and semantic cues from the visual input, guided by auxiliary heads and prior pseudo labels, thus reducing redundancy. 2) Building upon this, to represent how world knowledge evolves under actions, we introduce the Latent World Variation Quantization (LWVQ). It learns a discrete latent space via a VQ-VAE objective to encode world knowledge variations, shifting prediction from full modalities to compact latent. 3)Moreover, to mitigate interference during variation modeling, we design the Conditional Variation
Attention (CV-Atten), whichpromotes disentangled learning and preserves the independence of knowledge representations. Extensive experiments on both simulated
benchmarks and real-world robotic tasks demonstrate $\Delta$VLA achieves state-of-the-art performance while improving efficiency. Code and real-world execution
videos are available at https://github.com/JiuTian-VL/DeltaVLA.
Read the full abstract ▾
All authors Yijie Zhu, Jie He, Rui Shao, Kaishen Yuan, Tao Tan, Xiaochen Yuan, Zitong Yu
Computer Vision
Local-Global Prompt Learning via Sparse Optimal Transport
Deniz Kizaroğlu, Ülku Tuncer Küçüktas, Emre Çakmakyurdu et al. (+1 more)
Few-shot adaptation of
vision-
language models (VLMs) like CLIP typically relies on learning textual
prompts matched to global image
embeddings. Recent works extend this paradigm by incorporating local image-text
alignment to capture fine-grained visual cues, yet these approaches often select local regions independently for each
prompt, leading to redundant local feature usage and
prompt overlap. We propose SOT-GLP, which introduces a shared sparse patch support and balanced optimal transport allocation to explicitly partition salient visual regions among class-specific local
prompts while preserving global
alignment. Our method learns shared global
prompts and class-specific local
prompts. The global branch maintains standard image-text matching for robust category-level
alignment. The local branch constructs a class-conditioned sparse patch set using V-V
attention and aligns it to multiple class-specific
prompts via balanced entropic optimal transport, yielding a soft partition of patches that prevents
prompt overlap and collapse. We evaluate our method on two complementary objectives: (i) few-shot classification accuracy on 11 standard
benchmarks and (ii) out-of-distribution (OOD) detection. On the standard 11-
dataset benchmark with 16-shot ViT-B/16, SOT-GLP achieves 85.1% average accuracy, outperforming prior
prompt-learning methods. We identify a distinct accuracy-robustness trade-off in
prompt learning: while learnable projections optimize in-distribution fit, they alter the foundational feature space. We demonstrate that a projection-free local
alignment preserves the native geometry of the CLIP
manifold, yielding state-of-the-art OOD detection performance (94.2% AUC) that surpasses fully adapted models. Implementation available at: https://github.com/Deniz2304988/SOT-GLP
Read the full abstract ▾
All authors Deniz Kizaroğlu, Ülku Tuncer Küçüktas, Emre Çakmakyurdu, Alptekin Temizel
Computer Vision
Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathology
Mina Jamshidi Idaji, Julius Hense, Tom Neuhäuser et al. (+12 more)
Multiple instance learning (MIL) has enabled substantial progress in computational histopathology, where a large amount of patches from gigapixel whole slide images are aggregated into slide-level predictions. Heatmaps are widely used to validate MIL models and to discover tissue biomarkers. Yet, the validity of these heatmaps has barely been investigated. In this work, we introduce a general framework for evaluating the quality of MIL heatmaps without requiring additional labels. We conduct a large-scale
benchmark experiment to assess six explanation methods across histopathology task types (classification, regression, survival), MIL model architectures (
Attention-,
Transformer-, Mamba-based), and patch encoder backbones (UNI2, Virchow2). Our results show that explanation quality mostly depends on MIL model architecture and task type, with perturbation ("Single"), layer-wise relevance propagation (LRP), and integrated gradients (IG) consistently outperforming
attention-based and gradient-based saliency heatmaps, which often fail to reflect model decision mechanisms. We further demonstrate the advanced capabilities of the best-performing explanation methods: (i) We provide a proof-of-concept that MIL heatmaps of a bulk gene expression prediction model can be correlated with spatial transcriptomics for biological validation, and (ii) showcase the discovery of distinct model strategies for predicting human papillomavirus (HPV) infection from head and neck cancer slides. Our work highlights the importance of validating MIL heatmaps and establishes that improved explainability can enable more reliable model validation and yield biological insights, making a case for a broader adoption of explainable AI in digital pathology. Our code is provided in a public GitHub repository: https://github.com/bifold-pathomics/xMIL/tree/xmil-journal
Read the full abstract ▾
All authors Mina Jamshidi Idaji, Julius Hense, Tom Neuhäuser, Augustin Krause, Yanqing Luo, Oliver Eberle, Thomas Schnake, Laure Ciernik, Farnoush Rezaei Jafari, Reza Vahidimajd, Jonas Dippel, Christoph Walz, Frederick Klauschen, Andreas Mock, Klaus-Robert Müller
Computer Vision
Human-AI Divergence in Ego-centric Action Recognition under Spatial and Spatiotemporal Manipulations
Sadegh Rahmaniboldaji, Filip Rybansky, Quoc C. Vuong et al. (+3 more)
Humans consistently outperform state-of-the-art AI models in action recognition, particularly in challenging real-world conditions involving low resolution, occlusion, and visual clutter. Understanding the sources of this performance gap is essential for developing more robust and human-aligned models. In this paper, we present a large-scale human-AI comparative study of
egocentric action recognition using Minimal Identifiable Recognition Crops (MIRCs), defined as the smallest spatial or spatiotemporal regions sufficient for reliable human recognition. We used our previously introduced, Epic ReduAct, a systematically spatially reduced and temporally scrambled
dataset derived from 36 EPIC KITCHENS
videos, spanning multiple spatial reduction levels and temporal conditions. Recognition performance is evaluated using over 3,000 human participants and the Side4
Video model. Our analysis combines quantitative metrics, Average Reduction Rate and Recognition Gap, with qualitative analyses of spatial (high-, mid-, and low-level visual features) and spatiotemporal factors, including a categorisation of actions into Low Temporal Actions (LTA) and High Temporal Actions (HTA). Results show that human performance exhibits sharp declines when transitioning from MIRCs to subMIRCs, reflecting a strong reliance on sparse, semantically critical cues such as hand-object interactions. In contrast, the model degrades more gradually and often relies on contextual and mid- to low-level features, sometimes even exhibiting increased confidence under spatial reduction. Temporally, humans remain robust to scrambling when key spatial cues are preserved, whereas the model often shows insensitivity to temporal disruption, revealing class-dependent temporal sensitivities.
Read the full abstract ▾
All authors Sadegh Rahmaniboldaji, Filip Rybansky, Quoc C. Vuong, Anya C. Hurlbert, Frank Guerin, Andrew Gilbert
Computer Vision · Cross-listed from cs.CR
SlowBA: An efficiency backdoor attack towards VLM-based GUI agents
Junxian Li, Tu Lan, Haozhen Tan et al. (+2 more)
Modern
vision-
language-model (VLM) based graphical user interface (GUI)
agents are expected not only to execute actions accurately but also to respond to user instructions with low latency. While existing research on GUI-
agent security mainly focuses on manipulating action correctness, the security risks related to response efficiency remain largely unexplored. In this paper, we introduce SlowBA, a novel
backdoor attack that targets the responsiveness of VLM-based GUI
agents. The key idea is to manipulate response latency by inducing excessively long
reasoning chains under specific trigger patterns. To achieve this, we propose a two-stage reward-level
backdoor injection (RBI) strategy that first aligns the long-response format and then learns trigger-aware activation through reinforcement learning. In addition, we design realistic pop-up windows as triggers that naturally appear in GUI environments, improving the stealthiness of the attack. Extensive experiments across multiple
datasets and baselines demonstrate that SlowBA can significantly increase response length and latency while largely preserving task accuracy. The attack remains effective even with a small poisoning ratio and under several defense settings. These findings reveal a previously overlooked security vulnerability in GUI
agents and highlight the need for defenses that consider both action correctness and response efficiency. Code can be found in https://github.com/tu-tuing/SlowBA.
Read the full abstract ▾
All authors Junxian Li, Tu Lan, Haozhen Tan, Yan Meng, Haojin Zhu
Computer Vision
HDR-NSFF: High Dynamic Range Neural Scene Flow Fields
Shin Dong-Yeon, Kim Jun-Seong, Kwon Byung-Ki et al. (+1 more)
Radiance of real-world scenes typically spans a much wider dynamic range than what standard cameras can capture. While conventional HDR methods merge alternating-exposure frames, these approaches are inherently constrained to 2D pixel-level
alignment, often leading to ghosting artifacts and temporal inconsistency in dynamic scenes. To address these limitations, we present HDR-NSFF, a paradigm shift from 2D-based merging to 4D spatio-temporal modeling. Our framework reconstructs dynamic HDR radiance fields from alternating-exposure
monocular videos by representing the scene as a continuous function of space and time, and is compatible with both neural radiance field and 4D Gaussian Splatting (4DGS) based dynamic representations. This unified end-to-end pipeline explicitly models HDR radiance, 3D scene flow, geometry, and tone-mapping, ensuring physical plausibility and global coherence. We further enhance robustness by (i) extending semantic-based optical flow with DINO features to achieve exposure-invariant motion estimation, and (ii) incorporating a generative prior as a regularizer to compensate for limited observation in
monocular captures and saturation-induced information loss. To evaluate HDR space-time view synthesis, we present the first real-world HDR-GoPro
dataset specifically designed for dynamic HDR scenes. Experiments demonstrate that HDR-NSFF recovers fine radiance details and coherent dynamics even under challenging exposure variations, thereby achieving state-of-the-art performance in novel space-time view synthesis. Project page: https://shin-dong-yeon.github.io/HDR-NSFF/
Read the full abstract ▾
All authors Shin Dong-Yeon, Kim Jun-Seong, Kwon Byung-Ki, Tae-Hyun Oh
Computer Vision
Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness
Yehonatan Elisha, Oren Barkan and Noam Koenigstein
Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations, such as background cues, rather than semantically meaningful features. Existing regularization methods, typically relying on simple foreground-background masks, which fail to capture the fine-grained semantic concepts that define an object (e.g., “long beak” and “wings” for a “bird”). As a result, these methods provide limited robustness to distribution shifts. To address this limitation, we introduce a novel finetuning framework that steers model
reasoning toward concept-level semantics. Our approach optimizes the model's internal relevance maps to align with spatially grounded concept masks. These masks are generated automatically, without manual annotation: class-relevant concepts are first proposed using an LLM-based, label-free method, and then segmented using a VLM. The finetuning objective aligns relevance with these concept regions while simultaneously suppressing focus on spurious background areas. Notably, this process requires only a minimal set of images and uses half of the
dataset classes. Extensive experiments on five out-of-distribution
benchmarks demonstrate that our method improves robustness across multiple ViT-based models. Furthermore, we show that the resulting relevance maps exhibit stronger
alignment with semantic object parts, offering a scalable path toward more robust and interpretable
vision models. Finally, we confirm that concept-guided masks provide more effective supervision for model robustness than conventional
segmentation maps, supporting our central hypothesis.
Read the full abstract ▾
Computer Vision
Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation
Daniele Molino, Camillo Maria Caruso, Paolo Soda et al. (+1 more)
Text-conditioned generative models for volumetric medical imaging provide semantic control but lack explicit anatomical guidance, often resulting in outputs that are spatially ambiguous or anatomically inconsistent. In contrast, structure-driven methods ensure strong anatomical consistency but typically assume access to ground-truth annotations, which are unavailable when the target image is to be synthesized. We propose a
retrieval-augmented approach for Text-to-CT generation that integrates semantic and anatomical information under a realistic inference setting. Given a radiology report, our method retrieves a semantically related clinical case using a 3D
vision-
language encoder and leverages its associated anatomical annotation as a structural proxy. This proxy is injected into a text-conditioned latent
diffusion model via a ControlNet branch, providing coarse anatomical guidance while maintaining semantic flexibility. Experiments on the CT-RATE
dataset show that
retrieval-augmented generation improves image fidelity and clinical consistency compared to text-only baselines, while additionally enabling explicit spatial controllability, a capability inherently absent in such approaches. Further analysis highlights the importance of
retrieval quality, with semantically aligned proxies yielding consistent gains across all evaluation axes. This work introduces a principled and scalable mechanism to bridge semantic conditioning and anatomical plausibility in volumetric medical image synthesis. Code is available at https://github.com/arco-group/RAGText2CT.
Read the full abstract ▾
All authors Daniele Molino, Camillo Maria Caruso, Paolo Soda, Valerio Guarrasi
Computer Vision
Novel Semantic Prompting for Zero-Shot Action Recognition
Salman Iqbal and Waheed Rehman
Zero-shot action recognition relies on transferring knowledge from
vision-
language models to unseen actions using semantic descriptions. While recent methods focus on temporal modeling or architectural adaptations to handle
video data, we argue that semantic prompting alone provides a strong and underexplored signal for zero-shot action understanding. We introduce SP-CLIP, a lightweight framework that augments frozen
vision-
language models with structured semantic
prompts describing actions at multiple levels of abstraction, such as intent, motion, and object interaction. Without modifying the visual encoder or learning additional parameters, SP-CLIP aligns
video representations with enriched textual semantics through
prompt aggregation and consistency scoring. Experiments across standard
benchmarks show that semantic prompting substantially improves zero-shot action recognition, particularly for fine-grained and compositional actions, while preserving the efficiency and generalization of pretrained models.
Read the full abstract ▾
Computer Vision
OSCAR: Occupancy-based Shape Completion via Acoustic Neural Implicit Representations
Magdalena Wysocki, Kadir Burak Buldu, Miruna-Alexandra Gafencu et al. (+2 more)
Accurate 3D reconstruction of vertebral anatomy from ultrasound is important for guiding minimally invasive spine interventions, but it remains challenging due to acoustic shadowing and view-dependent signal variations. We propose an occupancy-based shape completion method that reconstructs complete 3D anatomical geometry from partial ultrasound observations. Crucially for intra-operative applications, our approach extracts the anatomical surface directly from the image, avoiding the need for anatomical labels during inference. This label-free completion relies on a coupled latent space representing both the image appearance and the underlying anatomical shape. By leveraging a Neural Implicit Representation (NIR) that jointly models both spatial occupancy and acoustic interactions, the method uses acoustic parameters to become implicitly aware of the unseen regions without explicit shadowing labels through tracking acoustic signal transmission. We show that this method outperforms state-of-the-art shape completion for B-mode ultrasound by 80% in HD95 score. We validate our approach both in-silico and on phantom US images with registered mesh models from CT labels, demonstrating accurate reconstruction of occluded anatomy and robust generalization across diverse imaging conditions. Code and data will be released on publication.
Read the full abstract ▾
All authors Magdalena Wysocki, Kadir Burak Buldu, Miruna-Alexandra Gafencu, Mohammad Farid Azampour, Nassir Navab
Computer Vision
Prototype-Guided Concept Erasure in Diffusion Models
Yuze Cai, Jiahao Lu, Hongxiang Shi et al. (+2 more)
Concept erasure is extensively utilized in image generation to prevent text-to-image models from generating undesired content. Existing methods can effectively erase narrow concepts that are specific and concrete, such as distinct intellectual properties (e.g. Pikachu) or recognizable characters (e.g. Elon Musk). However, their performance degrades on broad concepts such as “sexual” or “violent”, whose wide scope and multi-faceted nature make them difficult to erase reliably. To overcome this limitation, we exploit the model's intrinsic
embedding geometry to identify latent
embeddings that encode a given concept. By clustering these
embeddings, we derive a set of concept prototypes that summarize the model's internal representations of the concept, and employ them as negative conditioning signals during inference to achieve precise and reliable erasure. Extensive experiments across multiple
benchmarks show that our approach achieves substantially more reliable removal of broad concepts while preserving overall image quality, marking a step towards safer and more controllable image generation.
Read the full abstract ▾
All authors Yuze Cai, Jiahao Lu, Hongxiang Shi, Yichao Zhou, Hong Lu
Computer Vision
Event-based Motion & Appearance Fusion for 6D Object Pose Tracking
Zhichao Li, Chiara Bartolozzi, Lorenzo Natale et al. (+1 more)
Object pose tracking is a fundamental and essential task for robotics to perform tasks in the home and industrial settings. The most commonly used sensors to do so are RGB-D cameras, which can hit limitations in highly dynamic environments due to motion blur and frame-rate constraints. Event cameras have remarkable features such as high temporal resolution and low latency, which make them a potentially ideal
vision sensors for object pose tracking at high speed. Even so, there are still only few works on 6D pose tracking with event cameras. In this work, we take advantage of the high temporal resolution and propose a method that uses both a propagation step fused with a pose correction strategy. Specifically, we use 6D object velocity obtained from event-based optical flow for pose propagation, after which, a template-based local pose correction module is utilized for pose correction. Our learning-free method has comparable performance to the state-of-the-art
algorithms, and in some cases out performs them for fast-moving objects. The results indicate the potential for using event cameras in highly-dynamic scenarios where the use of deep network approaches are limited by low update rates.
Read the full abstract ▾
All authors Zhichao Li, Chiara Bartolozzi, Lorenzo Natale, Arren Glover
Computer Vision
WaDi: Weight Direction-aware Distillation for One-step Image Synthesis
Lei Wang, Yang Cheng, Senmao Li et al. (+3 more)
Despite the impressive performance of
diffusion models such as Stable
Diffusion (SD) in image generation, their slow inference limits practical deployment. Recent works accelerate inference by distilling multi-step
diffusion into one-step generators. To better understand the
distillation mechanism, we analyze U-Net/DiT weight changes between one-step students and their multi-step teacher counterparts. Our analysis reveals that changes in weight direction significantly exceed those in weight norm, highlighting it as the key factor during
distillation. Motivated by this insight, we propose the Low-rank Rotation of weight Direction (LoRaD), a parameter-efficient adapter tailored to one-step
diffusion distillation. LoRaD is designed to model these structured directional changes using learnable low-rank rotation
matrices. We further integrate LoRaD into Variational Score
Distillation (VSD), resulting in Weight Direction-aware
Distillation (WaDi)-a novel one-step
distillation framework. WaDi achieves state-of-the-art FID scores on COCO 2014 and COCO 2017 while using only approximately 10% of the trainable parameters of the U-Net/DiT. Furthermore, the distilled one-step model demonstrates strong versatility and scalability, generalizing well to various downstream tasks such as controllable generation, relation inversion, and high-resolution synthesis.
Read the full abstract ▾
All authors Lei Wang, Yang Cheng, Senmao Li, Ge Wu, Yaxing Wang, Jian Yang
Computer Vision
DynamicVGGT: Learning Dynamic Point Maps for 4D Scene Reconstruction in Autonomous Driving
Zhuolin He, Jing Li, Guanghao Li et al. (+9 more)
Dynamic scene reconstruction in autonomous driving remains a fundamental challenge due to significant temporal variations, moving objects, and complex scene dynamics. Existing feed-forward 3D models have demonstrated strong performance in static reconstruction but still struggle to capture dynamic motion. To address these limitations, we propose DynamicVGGT, a unified feed-forward framework that extends VGGT from static 3D perception to dynamic 4D reconstruction. Our goal is to model point motion within feed-forward 3D models in a dynamic and temporally coherent manner. To this end, we jointly predict the current and future point maps within a shared reference coordinate system, allowing the model to implicitly learn dynamic point representations through temporal correspondence. To efficiently capture temporal dependencies, we introduce a Motion-aware Temporal
Attention (MTA) module that learns motion continuity. Furthermore, we design a Dynamic 3D Gaussian Splatting Head that explicitly models point motion by predicting Gaussian velocities using learnable motion
tokens under scene flow supervision. It refines dynamic geometry through continuous 3D Gaussian optimization. Extensive experiments on autonomous driving
datasets demonstrate that DynamicVGGT significantly outperforms existing methods in reconstruction accuracy, achieving robust feed-forward 4D dynamic scene reconstruction under complex driving scenarios.
Read the full abstract ▾
All authors Zhuolin He, Jing Li, Guanghao Li, Xiaolei Chen, Jiacheng Tang, Siyang Zhang, Zhounan Jin, Feipeng Cai, Bin Li, Jian Pu, Jia Cai, Xiangyang Xue
Computer Vision · Cross-listed from cs.CG
Topologically Stable Hough Transform
Stefan Huber, Kristóf Huszár, Michael Kerber et al. (+1 more)
We propose an alternative formulation of the well-known Hough transform to detect lines in point clouds. Replacing the discretized voting scheme of the classical Hough transform by a continuous score function, its persistent features in the sense of persistent homology give a set of candidate lines. We also devise and implement an
algorithm to efficiently compute these candidate lines.
Read the full abstract ▾
All authors Stefan Huber, Kristóf Huszár, Michael Kerber, Martin Uray
Computer Vision
SiMO: Single-Modality-Operable Multimodal Collaborative Perception
Jiageng Wen, Shengjie Zhao, Bing Li et al. (+3 more)
Collaborative perception integrates multi-
agent perspectives to enhance the sensing range and overcome occlusion issues. While existing
multimodal approaches leverage complementary sensors to improve performance, they are highly prone to failure–especially when a key sensor like
LiDAR is unavailable. The root cause is that feature fusion leads to semantic mismatches between single-modality features and the downstream modules. This paper addresses this challenge for the first time in the field of collaborative perception, introducing Single-Modality-Operable
Multimodal Collaborative Perception (SiMO). By adopting the proposed Length-Adaptive Multi-Modal Fusion (LAMMA), SiMO can adaptively handle remaining modal features during modal failures while maintaining consistency of the semantic space. Additionally, leveraging the innovative "Pretrain-Align-Fuse-RD" training strategy, SiMO addresses the issue of modality competition–generally overlooked by existing methods–ensuring the independence of each individual modality branch. Experiments demonstrate that SiMO effectively aligns
multimodal features while simultaneously preserving modality-specific features, enabling it to maintain optimal performance across all individual modalities. The implementation details can be found in https://github.com/dempsey-wen/SiMO.
Read the full abstract ▾
All authors Jiageng Wen, Shengjie Zhao, Bing Li, Jiafeng Huang, Kenan Ye, Hao Deng
Computer Vision
Exploring Deep Learning and Ultra-Widefield Imaging for Diabetic Retinopathy and Macular Edema
Pablo Jimenez-Lizcano, Sergio Romero-Tapiador, Ruben Tolosana et al. (+4 more)
Diabetic retinopathy (DR) and diabetic macular edema (DME) are leading causes of preventable blindness among working-age adults. Traditional approaches in the literature focus on standard color fundus photography (CFP) for the detection of these conditions. Nevertheless, recent ultra-widefield imaging (UWF) offers a significantly wider field of view in comparison to CFP. Motivated by this, the present study explores state-of-the-art deep learning (DL) methods and UWF imaging on three clinically relevant tasks: i) image quality assessment for UWF, ii) identification of referable diabetic retinopathy (RDR), and iii) identification of DME. Using the publicly available UWF4DR Challenge
dataset, released as part of the MICCAI 2024 conference, we
benchmark DL models in the spatial (RGB) and frequency domains, including popular convolutional neural networks (CNNs) as well as recent
vision transformers (ViTs) and foundation models. In addition, we explore a final feature-level fusion to increase robustness. Finally, we also analyze the decisions of the DL models using Grad-CAM, increasing the explainability. Our proposal achieves consistently strong performance across all architectures, underscoring the competitiveness of emerging ViTs and foundation models and the promise of feature-level fusion and frequency-domain representations for UWF analysis.
Read the full abstract ▾
All authors Pablo Jimenez-Lizcano, Sergio Romero-Tapiador, Ruben Tolosana, Aythami Morales, Guillermo González de Rivera, Ruben Vera-Rodriguez, Julian Fierrez
Computer Vision
GarmentPainter: Efficient 3D Garment Texture Synthesis with Character-Guided Diffusion Model
Jinbo Wu, Xiaobo Gao, Xing Liu et al. (+2 more)
Generating high-fidelity, 3D-consistent garment textures remains a challenging problem due to the inherent complexities of garment structures and the stringent requirement for detailed, globally consistent texture synthesis. Existing approaches either rely on 2D-based
diffusion models, which inherently struggle with 3D consistency, require expensive multi-step optimization or depend on strict spatial
alignment between 2D reference images and 3D meshes, which limits their flexibility and scalability. In this work, we introduce GarmentPainter, a simple yet efficient framework for synthesizing high-quality, 3D-aware garment textures in UV space. Our method leverages a UV position map as the 3D structural guidance, ensuring texture consistency across the garment surface during texture generation. To enhance control and adaptability, we introduce a type selection module, enabling fine-grained texture generation for specific garment components based on a character reference image, without requiring
alignment between the reference image and the 3D mesh. GarmentPainter efficiently integrates all guidance signals into the input of a
diffusion model in a spatially aligned manner, without modifying the underlying UNet architecture. Extensive experiments demonstrate that GarmentPainter achieves state-of-the-art performance in terms of visual fidelity, 3D consistency, and computational efficiency, outperforming existing methods in both qualitative and quantitative evaluations.
Read the full abstract ▾
All authors Jinbo Wu, Xiaobo Gao, Xing Liu, Chen Zhao, Jialun Liu
Computer Vision
SRNeRV: A Scale-wise Recursive Framework for Neural Video Representation
Jia Wang, Jun Zhu and Xinfeng Zhang
Implicit Neural Representations (INRs) have emerged as a promising paradigm for
video representation and compression. However, existing multi-scale INR generators often suffer from significant parameter redundancy by stacking independent processing blocks for each scale. Inspired by the principle of scale self-similarity in the generation process, we propose SRNeRV, a novel scale-wise recursive framework that replaces this stacked design with a parameter-efficient shared architecture. The core of our approach is a hybrid sharing scheme derived from decoupling the processing block into a scale-specific spatial mixing module and a scale-invariant channel mixing module. We recursively apply the same shared channel mixing module, which contains the majority of the parameters, across all scales, significantly reducing the model size while preserving the crucial capacity to learn scale-specific spatial patterns. Extensive experiments demonstrate that SRNeRV achieves a significant rate-distortion performance boost, especially in INR-friendly scenarios, validating that our sharing scheme successfully amplifies the core strengths of the INR paradigm.
Read the full abstract ▾
Computer Vision
Ruixiang Zhao, Zhihao Xu, Bangxiang Lan et al. (+3 more)
For
video-text
retrieval, the use of CLIP has been a de facto choice. Since CLIP provides only image and text encoders, this consensus has led to a biased paradigm that entirely ignores the sound track of
videos. While several attempts have been made to reintroduce audio – typically by incorporating an audio encoder and fusing its output with visual features – these methods face two challenges: ineffective representation of
speech content and suboptimal
vision-audio fusion. To address these issues jointly, we propose SAVE, a
Speech Aware
Video rEpresentation learning method. SAVE improves upon AVIGATE, a SOTA audiovisual method, with a dedicated
speech branch for more effective
speech embedding. Furthermore, we introduce soft-ALBEF for early
vision-audio
alignment that facilitates fusion. Extensive experiments on five
benchmarks show that SAVE compares favorably against the SOTA, outperforming AVIGATE by +4.1% on MSRVTT-9k, +1.9% on MSRVTT-7k, +2.5% on VATEX, +9.8% on Charades, and +2.1% on LSMDC, in light of the SumR metric.
Read the full abstract ▾
All authors Ruixiang Zhao, Zhihao Xu, Bangxiang Lan, Zijie Xin, Jingyu Liu, Xirong Li
Computer Vision
Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRA
Zexi Wu, Baolu Li, Jing Dai et al. (+5 more)
Achieving semantic
alignment across diverse
video generation conditions remains a significant challenge. Methods that rely on explicit structural guidance often enforce rigid spatial constraints that limit semantic flexibility, whereas models tailored for individual control types lack interoperability and adaptability. These design bottlenecks hinder progress toward flexible and efficient semantic
video generation. To address this, we propose
Video2LoRA, a scalable and generalizable framework for semantic-controlled
video generation that conditions on a reference
video.
Video2LoRA employs a lightweight hypernetwork to predict personalized LoRA weights for each semantic input, which are combined with auxiliary
matrices to form adaptive LoRA modules integrated into a frozen
diffusion backbone. This design enables the model to generate
videos consistent with the reference semantics while preserving key style and content variations, eliminating the need for any per-condition training. Notably, the final model weights less than 150MB, making it highly efficient for storage and deployment.
Video2LoRA achieves coherent, semantically aligned generation across diverse conditions and exhibits strong zero-shot generalization to unseen semantics.
Read the full abstract ▾
All authors Zexi Wu, Baolu Li, Jing Dai, Yiming Zhang, Yue Ma, Qinghe Wang, Xu Jia, Hongming Xu
Computer Vision
Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors
Ishrat Jahan, Molla E Majid, M Murugappan et al. (+5 more)
Reliable unmanned aerial vehicle (
UAV) detection is critical for autonomous airspace monitoring but remains challenging when integrating sensor streams that differ substantially in resolution, perspective, and field of view. Conventional fusion methods-such as wavelet-,
Laplacian-, and decision-level approaches-often fail to preserve spatial correspondence across modalities and suffer from annotation of inconsistencies, limiting their robustness in real-world settings. This study introduces two fusion strategies, Registration-aware Guided Image Fusion (RGIF) and Reliability-Gated Modality-
Attention Fusion (RGMAF), designed to overcome these limitations. RGIF employs Enhanced Correlation Coefficient (ECC)-based affine registration combined with guided filtering to maintain thermal saliency while enhancing structural detail. RGMAF integrates affine and optical-flow registration with a reliability-weighted
attention mechanism that adaptively balances thermal contrast and visual sharpness. Experiments were conducted on the Multi-Sensor and Multi-View Fixed-Wing (MMFW)-
UAV dataset comprising 147,417 annotated air-to-air frames collected from infrared, wide-angle, and zoom sensors. Among single-modality detectors, YOLOv10x demonstrated the most stable cross-domain performance and was selected as the detection backbone for evaluating fused imagery. RGIF improved the visual baseline by 2.13% mAP@50 (achieving 97.65%), while RGMAF attained the highest recall of 98.64%. These findings show that registration-aware and reliability-adaptive fusion provides a robust framework for integrating heterogeneous modalities, substantially enhancing
UAV detection performance in
multimodal environments.
Read the full abstract ▾
All authors Ishrat Jahan, Molla E Majid, M Murugappan, Muhammad E. H. Chowdhury, N.B.Prakash, Saad Bin Abul Kashem, Balamurugan Balusamy, Amith Khandakar
Computer Vision
MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data
Siarhei Sheludzko, Dhimitrios Duka, Bernt Schiele et al. (+2 more)
Contrastive learning has become a fundamental approach in both uni-modal and multi-modal frameworks. This learning paradigm pulls positive pairs of samples closer while pushing negatives apart. In the uni-modal setting (e.g., image-based learning), previous research has shown that the strength of these forces can be controlled through the temperature parameter. In this work, we propose Multi-Modal Temperature and Margin Schedules (MM-TS), extending the concept of uni-modal temperature scheduling to multi-modal contrastive learning. Our method dynamically adjusts the temperature in the contrastive loss during training, modulating the attraction and repulsion forces in the multi-modal setting. Additionally, recognizing that standard multi-modal
datasets often follow imbalanced, long-tail distributions, we adapt the temperature based on the local distribution of each training sample. Specifically, samples from dense clusters are assigned a higher temperature to better preserve their semantic structure. Furthermore, we demonstrate that temperature scheduling can be effectively integrated within a max-margin framework, thereby unifying the two predominant approaches in multi-modal contrastive learning: InfoNCE loss and max-margin objective. We evaluate our approach on four widely used image- and
video-
language datasets, Flickr30K, MSCOCO, EPIC-KITCHENS-100, and YouCook2, and show that our dynamic temperature and margin schedules improve performance and lead to new state-of-the-art results in the field.
Read the full abstract ▾
All authors Siarhei Sheludzko, Dhimitrios Duka, Bernt Schiele, Hilde Kuehne, Anna Kukleva
Computer Vision
Fusion-Poly: A Polyhedral Framework Based on Spatial-Temporal Fusion for 3D Multi-Object Tracking
Xian Wu, Yitao Wu, Xiaoyu Li et al. (+3 more)
LiDAR-camera 3D multi-object tracking (MOT) combines rich visual semantics with accurate depth cues to improve trajectory consistency and tracking reliability. In practice, however,
LiDAR and cameras operate at different sampling rates. To maintain temporal
alignment, existing data pipelines usually synchronize heterogeneous sensor streams and annotate them at a reduced shared frequency, forcing most prior methods to perform spatial fusion only at synchronized timestamps through projection-based or learnable cross-sensor association. As a result, abundant asynchronous observations remain underexploited, despite their potential to support more frequent association and more robust trajectory estimation over short temporal intervals. To address this limitation, we propose Fusion-Poly, a spatial-temporal fusion framework for 3D MOT that integrates asynchronous
LiDAR and camera data. Fusion-Poly associates trajectories with multi-modal observations at synchronized timestamps and with single-modal observations at asynchronous timestamps, enabling higher-frequency updates of motion and existence states. The framework contains three key components: a frequency-aware cascade matching module that adapts to synchronized and asynchronous frames according to available detection modalities; a frequency-aware trajectory estimation module that maintains trajectories through high-frequency motion prediction, differential updates, and confidence-calibrated lifecycle management; and a full-state observation
alignment module that improves cross-modal consistency at synchronized timestamps by optimizing image-projection errors. On the nuScenes test set, Fusion-Poly achieves 76.5% AMOTA, establishing a new state of the art among tracking-by-detection 3D MOT methods. Extensive ablation studies further validate the effectiveness of each component. Code will be released.
Read the full abstract ▾
All authors Xian Wu, Yitao Wu, Xiaoyu Li, Zijia Li, Lijun Zhao, Lining Sun