Computer Vision
From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understanding
Wenzhao Xiang, Yue Wu, Hongyang Yu et al. (+3 more)
Self-supervised visual pre-training methods face an inherent tension: contrastive learning (CL) captures global semantics but loses fine-grained detail, while masked image modeling (MIM) preserves local textures but suffers from "
attention drift" due to semantically-agnostic random masking. We propose C2FMAE, a coarse-to-fine masked autoencoder that resolves this tension by explicitly learning hierarchical visual representations across three data granularities: semantic masks (scene-level), instance masks (object-level), and RGB images (pixel-level). Two synergistic innovations enforce a strict top-down learning principle. First, a cascaded decoder sequentially reconstructs from scene semantics to object instances to pixel details, establishing explicit cross-granularity dependencies that parallel decoders cannot capture. Second, a progressive masking curriculum dynamically shifts the training focus from semantic-guided to instance-guided and finally to random masking, creating a structured learning path from global context to local features. To support this framework, we construct a large-scale multi-granular
dataset with high-quality pseudo-labels for all 1.28M ImageNet-1K images. Extensive experiments show that C2FMAE achieves significant performance gains on image classification, object detection, and semantic
segmentation, validating the effectiveness of our hierarchical design in learning more robust and generalizable representations.
Read the full abstract ▾
All authors Wenzhao Xiang, Yue Wu, Hongyang Yu, Feng Gao, Fan Yang, Xilin Chen
Computer Vision
Leveraging whole slide difficulty in Multiple Instance Learning to improve prostate cancer grading
Marie Arrivat, Rémy Peyret, Elsa Angelini et al. (+1 more)
Multiple Instance Learning (MIL) has been widely applied in histopathology to classify Whole Slide Images (WSIs) with slide-level diagnoses. While the ground truth is established by expert pathologists, the slides can be difficult to diagnose for non-experts and lead to disagreements between the annotators. In this paper, we introduce the notion of Whole Slide Difficulty (WSD), based on the disagreement between an expert and a non-expert pathologist. We propose two different methods to leverage WSD, a multi-task approach and a weighted classification loss approach, and we apply them to Gleason grading of prostate cancer slides. Results show that integrating WSD during training consistently improves the classification performance across different feature encoders and MIL methods, particularly for higher Gleason grades (i.e. worse diagnosis).
Read the full abstract ▾
All authors Marie Arrivat, Rémy Peyret, Elsa Angelini, Pietro Gori
Computer Vision
No Image, No Problem: End-to-End Multi-Task Cardiac Analysis from Undersampled k-Space
Yundi Zhang, Sevgi Gokce Kafali, Niklas Bubeck et al. (+2 more)
Conventional clinical CMR pipelines rely on a sequential "reconstruct-then-analyze" paradigm, forcing an ill-posed intermediate step that introduces avoidable artifacts and information bottlenecks. This creates a fundamental mathematical paradox: it attempts to recover high-dimensional pixel arrays (i.e., images) from undersampled k-space, rather than directly extracting the low-dimensional physiological labels actually required for diagnosis. To unlock the direct diagnostic potential of k-space, we propose k-MTR (k-space Multi-Task Representation), a k-space representation learning framework that aligns undersampled k-space data and fully-sampled images into a shared semantic
manifold. Leveraging a large-scale controlled simulation of 42,000 subjects, k-MTR forces the k-space encoder to restore anatomical information lost to undersampling directly within the latent space, bypassing the explicit inverse problem for downstream analysis. We demonstrate that this latent
alignment enables the dense latent space embedded with high-level physiological semantics directly from undersampled frequencies. Across continuous phenotype regression, disease classification, and anatomical
segmentation, k-MTR achieves highly competitive performance against state-of-the-art image-domain baselines. By showcasing that precise spatial geometries and multi-task features can be successfully recovered directly from the k-space representations, k-MTR provides a robust architectural blueprint for task-aware cardiac MRI workflows.
Read the full abstract ▾
All authors Yundi Zhang, Sevgi Gokce Kafali, Niklas Bubeck, Daniel Rueckert, Jiazhen Pan
Computer Vision
Unsupervised Domain Adaptation with Target-Only Margin Disparity Discrepancy
Gauthier Miralles, Loïc Le Folgoc, Vincent Jugnon et al. (+1 more)
In interventional radiology, Cone-Beam Computed
Tomography (CBCT) is a helpful imaging modality that provides guidance to practicians during minimally invasive procedures. CBCT differs from traditional Computed
Tomography (CT) due to its limited reconstructed field of view, specific artefacts, and the intra-arterial administration of contrast medium. While CT benefits from abundant publicly available annotated
datasets, interventional CBCT data remain scarce and largely unannotated, with existing
datasets focused primarily on radiotherapy applications. To address this limitation, we leverage a proprietary collection of unannotated interventional CBCT scans in conjunction with annotated CT data, employing domain adaptation techniques to bridge the modality gap and enhance liver
segmentation performance on CBCT. We propose a novel unsupervised domain adaptation (UDA) framework based on the formalism of Margin Disparity Discrepancy (MDD), which improves target domain performance through a reformulation of the original MDD optimization framework. Experimental results on CT and CBCT
datasets for liver
segmentation demonstrate that our method achieves state-of-the-art performance in UDA, as well as in the few-shot setting.
Read the full abstract ▾
All authors Gauthier Miralles, Loïc Le Folgoc, Vincent Jugnon, Pietro Gori
Computer Vision
Adaptive Clinical-Aware Latent Diffusion for Multimodal Brain Image Generation and Missing Modality Imputation
Rong Zhou, Houliang Zhou, Yao Su et al. (+4 more)
Multimodal neuroimaging provides complementary insights for Alzheimer's disease diagnosis, yet clinical
datasets frequently suffer from missing modalities. We propose ACADiff, a framework that synthesizes missing
brain imaging modalities through adaptive clinical-aware
diffusion. ACADiff learns mappings between incomplete
multimodal observations and target modalities by progressively denoising latent representations while attending to available imaging data and clinical metadata. The framework employs adaptive fusion that dynamically reconfigures based on input availability, coupled with semantic clinical guidance via GPT-4o-encoded
prompts. Three specialized generators enable bidirectional synthesis among sMRI, FDG-PET, and AV45-PET. Evaluated on ADNI subjects, ACADiff achieves superior generation quality and maintains robust diagnostic performance even under extreme 80% missing scenarios, outperforming all existing baselines. To promote reproducibility, code is available at https://github.com/rongzhou7/ACADiff
Read the full abstract ▾
All authors Rong Zhou, Houliang Zhou, Yao Su, Brian Y. Chen, Yu Zhang, Lifang He, Alzheimer's Disease Neuroimaging Initiative
Computer Vision
Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction
Yao Zhang, Zhuchenyang Liu, Yalan He et al. (+2 more)
Text-motion
retrieval aims to learn a semantically aligned latent space between natural
language descriptions and 3D human motion skeleton sequences, enabling bidirectional search across the two modalities. Most existing methods use a dual-encoder framework that compresses motion and text into global
embeddings, discarding fine-grained local correspondences, and thus reducing accuracy. Additionally, these global-
embedding methods offer limited interpretability of the
retrieval results. To overcome these limitations, we propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image, compatible with pre-trained
Vision Transformers. For text-to-motion
retrieval, we employ MaxSim, a
token-wise late interaction mechanism, and enhance it with Masked
Language Modeling regularization to foster robust, interpretable text-motion
alignment. Extensive experiments on HumanML3D and KIT-ML show that our method outperforms state-of-the-art text-motion
retrieval approaches while offering interpretable fine-grained correspondences between text and motion. The code is available in the supplementary material.
Read the full abstract ▾
All authors Yao Zhang, Zhuchenyang Liu, Yalan He, Thomas Ploetz, Yu Xiao
Computer Vision
On the Structural Failure of Chamfer Distance in 3D Shape Optimization
Chang-Yong Song and David Hyde
Chamfer distance is the standard training loss for point cloud reconstruction, completion, and generation, yet directly optimizing it can produce worse Chamfer values than not optimizing it at all. We show that this paradoxical failure is gradient-structural. The per-point Chamfer gradient creates a many-to-one collapse that is the unique attractor of the forward term and cannot be resolved by any local regularizer, including repulsion, smoothness, and density-aware re-weighting. We derive a necessary condition for collapse suppression: coupling must propagate beyond local neighborhoods. In a controlled 2D setting, shared-basis deformation suppresses collapse by providing global coupling; in 3D shape morphing, a differentiable MPM prior instantiates the same principle, consistently reducing the Chamfer gap across 20 directed pairs with a 2.5$\times$ improvement on the topologically complex dragon. The presence or absence of non-local coupling determines whether Chamfer optimization succeeds or collapses. This provides a practical design criterion for any pipeline that optimizes point-level distance metrics.
Read the full abstract ▾
Computer Vision
WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition
Shan Ning, Longtian Qiu, Jiaxuan Sun et al. (+1 more)
Open-domain visual entity recognition (VER) seeks to associate images with entities in encyclopedic knowledge bases such as Wikipedia. Recent generative methods tailored for VER demonstrate strong performance but incur high computational costs, limiting their scalability and practical deployment. In this work, we revisit the contrastive paradigm for VER and introduce WikiCLIP, a simple yet effective framework that establishes a strong and efficient baseline for open-domain VER. WikiCLIP leverages large
language model
embeddings as knowledge-rich entity representations and enhances them with a
Vision-Guided Knowledge Adaptor (VGKA) that aligns textual semantics with visual cues at the patch level. To further encourage fine-grained discrimination, a Hard Negative Synthesis Mechanism generates visually similar but semantically distinct negatives during training. Experimental results on popular open-domain VER
benchmarks, such as OVEN, demonstrate that WikiCLIP significantly outperforms strong baselines. Specifically, WikiCLIP achieves a 16% improvement on the challenging OVEN unseen set, while reducing inference latency by nearly 100 times compared with the leading generative model, AutoVER. The project page is available at https://artanic30.github.io/project_pages/WikiCLIP/
Read the full abstract ▾
All authors Shan Ning, Longtian Qiu, Jiaxuan Sun, Xuming He
Computer Vision
Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports
Yuchen Yang, Yuqing Shao, Duxiu Huang et al. (+11 more)
Sports have long attracted broad
attention as they push the limits of human physical and cognitive capabilities. Amid growing interest in spatial intelligence for
vision-
language models (VLMs), sports provide a natural testbed for understanding high-intensity human motion and dynamic object interactions. To this end, we present CourtSI, the first large-scale spatial intelligence
dataset tailored to sports scenarios. CourtSI contains over 1M QA pairs, organized under a holistic taxonomy that systematically covers spatial counting, distance measurement, localization, and relational
reasoning, across representative net sports including badminton, tennis, and table tennis. Leveraging well-defined court geometry as metric anchors, we develop a semi-automatic data engine to reconstruct sports scenes, enabling scalable curation of CourtSI. In addition, we introduce CourtSI-Bench, a high-quality evaluation
benchmark comprising 3,686 QA pairs with rigorous human verification. We evaluate 25 proprietary and open-source VLMs on CourtSI-Bench, revealing a remaining human-AI performance gap and limited generalization from existing spatial intelligence
benchmarks. These findings indicate that sports scenarios expose limitations in spatial intelligence capabilities captured by existing
benchmarks. Further, fine-tuning Qwen3-VL-8B on CourtSI improves accuracy on CourtSI-Bench by 23.5 percentage points. The adapted model also generalizes effectively to CourtSI-Ext, an evaluation set built on a similar but unseen sport, and demonstrates enhanced spatial-aware commentary generation. Together, these findings demonstrate that CourtSI provides a scalable pathway toward advancing spatial intelligence of VLMs in sports.
Read the full abstract ▾
All authors Yuchen Yang, Yuqing Shao, Duxiu Huang, Linfeng Dong, Yifei Liu, Suixin Tang, Xiang Zhou, Yuanyuan Gao, Wei Wang, Yue Zhou, Xue Yang, Yanfeng Wang, Xiao Sun, Zhihang Zhong
Computer Vision
DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary
Jiazhi Guan, Quanwei Yang, Luying Huang et al. (+7 more)
Human-centric
video generation has advanced rapidly, yet existing methods struggle to produce controllable and physically consistent Human-Object Interaction (HOI)
videos. Existing works rely on dense control signals, template
videos, or carefully crafted text
prompts, which limit flexibility and generalization to novel objects. We introduce a framework, namely DISPLAY, guided by Sparse Motion Guidance, composed only of wrist joint coordinates and a shape-agnostic object bounding box. This lightweight guidance alleviates the imbalance between human and object representations and enables intuitive user control. To enhance fidelity under such sparse conditions, we propose an Object-Stressed
Attention mechanism that improves object robustness. To address the scarcity of high-quality HOI data, we further develop a Multi-Task Auxiliary Training strategy with a dedicated data curation pipeline, allowing the model to benefit from both reliable HOI samples and auxiliary tasks. Comprehensive experiments show that our method achieves high-fidelity, controllable HOI generation across diverse tasks. The project page can be found at \href{https://mumuwei.github.io/DISPLAY/}.
Read the full abstract ▾
All authors Jiazhi Guan, Quanwei Yang, Luying Huang, Junhao Liang, Borong Liang, Haocheng Feng, Wei He, Kaisiyuan Wang, Hang Zhou, Jingdong Wang
Computer Vision
InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing
Changyao Tian, Danni Yang, Guanzhou Chen et al. (+26 more)
Unified
multimodal models (UMMs) that integrate understanding,
reasoning, generation, and editing face inherent trade-offs between maintaining strong semantic comprehension and acquiring powerful generation capabilities. In this report, we present InternVL-U, a lightweight 4B-parameter UMM that democratizes these capabilities within a unified framework. Guided by the principles of unified contextual modeling and modality-specific modular design with decoupled visual representations, InternVL-U integrates a state-of-the-art
Multimodal Large
Language Model (MLLM) with a specialized MMDiT-based visual generation head. To further bridge the gap between aesthetic generation and high-level intelligence, we construct a comprehensive data synthesis pipeline targeting high-semantic-density tasks, such as text rendering and scientific
reasoning, under a
reasoning-centric paradigm that leverages Chain-of-Thought (CoT) to better align abstract user intent with fine-grained visual generation details. Extensive experiments demonstrate that InternVL-U achieves a superior performance - efficiency balance. Despite using only 4B parameters, it consistently outperforms unified baseline models with over 3x larger scales such as BAGEL (14B) on various generation and editing tasks, while retaining strong
multimodal understanding and
reasoning capabilities.
Read the full abstract ▾
All authors Changyao Tian, Danni Yang, Guanzhou Chen, Erfei Cui, Zhaokai Wang, Yuchen Duan, Penghao Yin, Sitao Chen, Ganlin Yang, Mingxin Liu, Zirun Zhu, Ziqian Fan, Leyao Gu, Haomin Wang, Qi Wei, Jinhui Yin, Xue Yang, Zhihang Zhong, Qi Qin, Yi Xin, Bin Fu, Yihao Liu, Jiaye Ge, Qipeng Guo, Gen Luo, Hongsheng Li, Yu Qiao, Kai Chen, Hongjie Zhang
Computer Vision
MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities
Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le et al. (+1 more)
Multimodal affective computing underpins key tasks such as sentiment analysis and
emotion recognition. Standard evaluations, however, often assume that textual, acoustic, and visual modalities are equally available. In real applications, some modalities are systematically more fragile or expensive, creating imbalanced missing rates and training biases that task-level metrics alone do not reveal. We introduce MissBench, a
benchmark and framework for
multimodal affective tasks that standardizes both shared and imbalanced missing-rate protocols on four widely used sentiment and
emotion datasets. MissBench also defines two diagnostic metrics. The Modality Equity Index (MEI) measures how fairly different modalities contribute across missing-modality configurations. The Modality Learning Index (MLI) quantifies optimization imbalance by comparing modality-specific gradient norms during training, aggregated across modality-related modules. Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions. These findings position MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing
multimodal affective models in realistic incomplete-modality settings.For reproducibility, we release our code at: https://anonymous.4open.science/r/MissBench-4098/
Read the full abstract ▾
All authors Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le, Cam-Van Thi Nguyen
Computer Vision · Cross-listed from eess.IV
CycleULM: A unified label-free deep learning framework for ultrasound localisation microscopy
Su Yan, Clara Rodrigo Gonzalez, Vincent C. H. Leung et al. (+11 more)
Super-resolution ultrasound via microbubble (MB) localisation and tracking, also known as ultrasound localisation microscopy (ULM), can resolve microvasculature beyond the acoustic diffraction limit. However, significant challenges remain in localisation performance and data acquisition and processing time. Deep learning methods for ULM have shown promise to address these challenges, however, they remain limited by in vivo label scarcity and the simulation-to-reality domain gap. We present CycleULM, the first unified label-free deep learning framework for ULM. CycleULM learns a physics-emulating translation between the real contrast-enhanced ultrasound (CEUS) data domain and a simplified MB-only domain, leveraging the power of CycleGAN without requiring paired ground truth data. With this translation, CycleULM removes dependence on high-fidelity simulators or labelled data, and makes MB localisation and tracking substantially easier. Deployed as modular plug-and-play components within existing pipelines or as an end-to-end processing framework, CycleULM delivers substantial performance gains across both in silico and in vivo
datasets. Specifically, CycleULM improves image contrast (contrast-to-noise ratio) by up to 15.3 dB and sharpens CEUS resolution with a 2.5{\times} reduction in the full width at half maximum of the point spread function. CycleULM also improves MB localisation performance, with up to +40% recall, +46% precision, and a -14.0 {\mu}m mean localisation error, yielding more faithful vascular reconstructions. Importantly, CycleULM achieves real-time processing throughput at 18.3 frames per second with order-of-magnitude speed-ups (up to ~14.5{\times}). By combining label-free learning, performance enhancement, and computational efficiency, CycleULM provides a practical pathway toward robust, real-time ULM and accelerates its translation to clinical applications.
Read the full abstract ▾
All authors Su Yan, Clara Rodrigo Gonzalez, Vincent C. H. Leung, Herman Verinaz-Jadan, Jiakang Chen, Matthieu Toulemonde, Kai Riemer, Jipeng Yan, Clotilde Vié, Qingyuan Tan, Peter D. Weinberg, Pier Luigi Dragotti, Kevin G. Murphy, Meng-Xing Tang
Computer Vision
Kangsan Kim, Yanlai Yang, Suji Kim et al. (+4 more)
As
embodied models become powerful, humans will collaborate with multiple
embodied AI
agents at their workplace or home in the future. To ensure better communication between human users and the multi-
agent system, it is crucial to interpret incoming information from
agents in parallel and refer to the appropriate context for each query. Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of
video and correctly aggregating multiple
egocentric videos to construct system-level
memory. In this work, we first formally define a novel problem of understanding multiple long-horizon
egocentric videos simultaneously collected from
embodied agents. To facilitate research in this direction, we introduce MultiAgent-EgoQA (MA-EgoQA), a
benchmark designed to systemically evaluate existing models in our scenario. MA-EgoQA provides 1.7k questions unique to multiple
egocentric streams, spanning five categories: social interaction, task coordination, theory-of-mind, temporal
reasoning, and environmental interaction. We further propose a simple baseline model for MA-EgoQA named EgoMAS, which leverages shared
memory across
embodied agents and
agent-wise dynamic
retrieval. Through comprehensive evaluation across diverse baselines and EgoMAS on MA-EgoQA, we find that current approaches are unable to effectively handle multiple
egocentric streams, highlighting the need for future advances in system-level understanding across the
agents. The code and
benchmark are available at https://ma-egoqa.github.io.
Read the full abstract ▾
All authors Kangsan Kim, Yanlai Yang, Suji Kim, Woongyeong Yeo, Youngwan Lee, Mengye Ren, Sung Ju Hwang
Computer Vision
VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models
Shuhao Kang, Youqi Liao, Peijie Wang et al. (+5 more)
Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural
language descriptions, reflecting how humans perceive and communicate spatial layouts through
language. However, existing methods largely rely on shallow text-point cloud correspondence without effective spatial
reasoning, limiting their accuracy in complex environments. To address this limitation, we propose VLM-Loc, a framework that leverages the spatial
reasoning capability of large
vision-
language models (VLMs) for T2P localization. Specifically, we transform point clouds into bird's-eye-view (BEV) images and scene graphs that jointly encode
geometric and semantic context, providing structured inputs for the VLM to learn cross-modal representations bridging linguistic and spatial semantics. On top of these representations, we introduce a partial node assignment mechanism that explicitly associates textual cues with scene graph nodes, enabling interpretable spatial
reasoning for accurate localization. To facilitate systematic evaluation across diverse scenes, we present CityLoc, a
benchmark built from multi-source point clouds for fine-grained T2P localization. Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods. Our code, model, and
dataset are available at \href{https://github.com/MCG-NKU/nku-3d-
vision}{repository}.
Read the full abstract ▾
All authors Shuhao Kang, Youqi Liao, Peijie Wang, Wenlong Liao, Qilin Zhang, Benjamin Busam, Xieyuanli Chen, Yun Liu
Computer Vision
BrainSTR: Spatio-Temporal Contrastive Learning for Interpretable Dynamic Brain Network Modeling
Guiliang Guo, Guangqi Wen, Lingwen Liu et al. (+6 more)
Dynamic functional connectivity captures time-varying
brain states for better neuropsychiatric diagnosis and spatio-temporal interpretability, i.e., identifying when discriminative disease signatures emerge and where they reside in the connectivity
topology. Reliable interpretability faces major challenges: diagnostic signals are often subtle and sparsely distributed across both time and
topology, while nuisance fluctuations and non-diagnostic connectivities are pervasive. To address these issues, we propose BrainSTR, a spatio-temporal contrastive learning framework for interpretable dynamic
brain network modeling. BrainSTR learns state-consistent phase boundaries via a data-driven Adaptive Phase Partition module, identifies diagnostically critical phases with
attention, and extracts disease-related connectivity within each phase using an Incremental Graph Structure Generator regularized by binarization, temporal smoothness, and sparsity. Then, we introduce a spatio-temporal supervised contrastive learning approach that leverages diagnosis-relevant spatio-temporal patterns to refine the similarity metric between samples and capture more discriminative spatio-temporal features, thereby constructing a well-structured semantic space for coherent and interpretable representations. Experiments on ASD, BD, and MDD validate the effectiveness of BrainSTR, and the discovered critical phases and subnetworks provide interpretable evidence consistent with prior neuroimaging findings. Our code: https://anonymous.4open.science/r/BrainSTR1.
Read the full abstract ▾
All authors Guiliang Guo, Guangqi Wen, Lingwen Liu, Ruoxian Song, Peng Cao, Jinzhu Yang, Fei Wang, Xiaoli Liu, Osmar R. Zaiane
Computer Vision
ConfCtrl: Enabling Precise Camera Control in Video Diffusion via Confidence-Aware Interpolation
Liudi Yang, George Eskandar, Fengyi Shen et al. (+5 more)
We address the challenge of novel view synthesis from only two input images under large viewpoint changes. Existing regression-based methods lack the capacity to reconstruct unseen regions, while camera-guided
diffusion models often deviate from intended trajectories due to noisy point cloud projections or insufficient conditioning from camera poses. To address these issues, we propose ConfCtrl, a confidence-aware
video interpolation framework that enables
diffusion models to follow prescribed camera poses while completing unseen regions. ConfCtrl initializes the
diffusion process by combining a confidence-weighted projected point cloud latent with noise as the conditioning input. It then applies a Kalman-inspired predict-update mechanism, treating the projected point cloud as a noisy measurement and using learned residual corrections to balance pose-driven predictions with noisy
geometric observations. This allows the model to rely on reliable projections while down-weighting uncertain regions, yielding stable, geometry-aware generation. Experiments on multiple
datasets show that ConfCtrl produces geometrically consistent and visually plausible novel views, effectively reconstructing occluded regions under large viewpoint changes.
Read the full abstract ▾
All authors Liudi Yang, George Eskandar, Fengyi Shen, Mohammad Altillawi, Yang Bai, Chi Zhang, Ziyuan Liu, Abhinav Valada
Computer Vision
RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding
Muyi Sun, Yixuan Wang, Hong Wang et al. (+5 more)
Audio-Visual Learning (AVL) is one fundamental task of multi-modality learning and
embodied intelligence, displaying the vital role in scene understanding and interaction. However, previous researchers mostly focus on exploring downstream tasks from a coarse-grained perspective (e.g., audio-visual correspondence, sound source localization, and audio-visual event localization). Considering providing more specific scene perception details, we newly define a fine-grained Audio-Visual Learning task, termed Region-Aware Sound Source Understanding (RA-SSU), which aims to achieve region-aware, frame-level, and high-quality sound source understanding. To support this goal, we innovatively construct two corresponding
datasets, i.e. fine-grained Music (f-Music) and fine-grained Lifescene (f-Lifescene), each containing annotated sound source masks and frame-by-frame textual descriptions. The f-Music
dataset includes 3,976 samples across 22 scene types related to specific application scenarios, focusing on music scenes with complex instrument mixing. The f-Lifescene
dataset contains 6,156 samples across 61 types representing diverse sounding objects in life scenarios. Moreover, we propose SSUFormer, a Sound-Source Understanding
TransFormer benchmark that facilitates both the sound source
segmentation and sound region description with a multi-modal input and multi-modal output architecture. Specifically, we design two modules for this framework, Mask Collaboration Module (MCM) and Mixture of Hierarchical-prompted Experts (MoHE), to respectively enhance the accuracy and enrich the elaboration of the sound source description. Extensive experiments are conducted on our two
datasets to verify the feasibility of the task, evaluate the availability of the
datasets, and demonstrate the superiority of the SSUFormer, which achieves SOTA performance on the Sound Source Understanding
benchmark.
Read the full abstract ▾
All authors Muyi Sun, Yixuan Wang, Hong Wang, Chen Su, Man Zhang, Xingqun Qi, Qi Li, Zhenan Sun
Computer Vision
Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency
Zhaofeng Shi, Heqian Qiu, Lanxiao Wang et al. (+4 more)
Efficient adaptation between
Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as human-
robot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE$^{2}$A$^{3}$) task, which aims to adjust the source-view-trained model online during test time to anticipate target-view actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multi-action candidates and significant temporal-spatial inter-view gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise
memory banks, which are updated by an
entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn
benchmarks show the effectiveness of our method, which outperforms related state-of-the-art methods by a large margin. Code is available at \href{https://github.com/ZhaofengSHI/DCPGN}{https://github.com/ZhaofengSHI/DCPGN}.
Read the full abstract ▾
All authors Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Qingbo Wu, Fanman Meng, Lili Pan, Hongliang Li
Computer Vision
What is Missing? Explaining Neurons Activated by Absent Concepts
Robin Hesse, Simone Schaub-Meyer, Janina Hesse et al. (+2 more)
Explainable artificial intelligence (XAI) aims to provide human-interpretable insights into the behavior of deep neural networks (DNNs), typically by estimating a simplified causal structure of the model. In existing work, this causal structure often includes relationships where the presence of a concept is associated with a strong activation of a
neuron. For example, attribution methods primarily identify input pixels that contribute most to a prediction, and feature visualization methods reveal inputs that cause high activation of a target
neuron - the former implicitly assuming that the relevant information resides in the input, and the latter that
neurons encode the presence of concepts. However, a largely overlooked type of causal relationship is that of encoded absences, where the absence of a concept increases neural activation. In this work, we show that such missing but relevant concepts are common and that mainstream XAI methods struggle to reveal them when applied in their standard form. To address this, we propose two simple extensions to attribution and feature visualization techniques that uncover encoded absences. Across experiments, we show how mainstream XAI methods can be used to reveal and explain encoded absences, how ImageNet models exploit them, and that debiasing can be improved when considering them.
Read the full abstract ▾
All authors Robin Hesse, Simone Schaub-Meyer, Janina Hesse, Bernt Schiele, Stefan Roth
Computer Vision
Removing the Trigger, Not the Backdoor: Alternative Triggers and Latent Backdoors
Gorka Abad, Ermes Franch, Stefanos Koffas et al. (+1 more)
Current
backdoor defenses assume that neutralizing a known trigger removes the
backdoor. We show this trigger-centric view is incomplete: alternative triggers, patterns perceptually distinct from training triggers, reliably activate the same
backdoor. We estimate the alternative trigger
backdoor direction in feature space by contrasting clean and triggered representations, and then develop a feature-guided attack that jointly optimizes target prediction and directional
alignment. First, we theoretically prove that alternative triggers exist and are an inevitable consequence of
backdoor training. Then, we verify this empirically. Additionally, defenses that remove training triggers often leave
backdoors intact, and alternative triggers can exploit the latent
backdoor feature-space. Our findings motivate defenses targeting
backdoor directions in representation space rather than input-space triggers.
Read the full abstract ▾
All authors Gorka Abad, Ermes Franch, Stefanos Koffas, Stjepan Picek
Computer Vision
Soroush Seifi, Simon Gardier, Vaggelis Dorovatas et al. (+2 more)
AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in
multimodal language models. A key challenge lies in overcoming the generic nature of these models to deliver personalized experiences. Existing approaches to personalizing large
vision language models often rely on additional training stages, which limit generality and scalability, or on engineered pipelines with external pre-trained modules, which hinder deployment efficiency. In this work, we propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts. Specifically, we extract visual
tokens that predominantly represent the target concept by utilizing the model's internal
attention mechanisms. These
tokens serve as a
memory of that specific concept, enabling the model to recall and describe it when it appears in test images. We conduct a comprehensive and unified evaluation of our approach and SOTA methods across various personalization settings including single-concept, multi-concept, and
video personalization, demonstrating strong performance gains with minimal personalization overhead.
Read the full abstract ▾
All authors Soroush Seifi, Simon Gardier, Vaggelis Dorovatas, Daniel Olmeda Reino, Rahaf Aljundi
Computer Vision
PanoAffordanceNet: Towards Holistic Affordance Grounding in 360{\deg} Indoor Environments
Guoliang Zhu, Wanjun Jia, Caoyang Shao et al. (+3 more)
Global perception is essential for
embodied agents in 360{\deg} spaces, yet current affordance grounding remains largely object-centric and restricted to perspective views. To bridge this gap, we introduce a novel task: Holistic Affordance Grounding in 360{\deg} Indoor Environments. This task faces unique challenges, including severe
geometric distortions from Equirectangular Projection (ERP), semantic dispersion, and cross-scale
alignment difficulties. We propose PanoAffordanceNet, an end-to-end framework featuring a Distortion-Aware Spectral Modulator (DASM) for latitude-dependent calibration and an Omni-Spherical Densification Head (OSDH) to restore topological continuity from sparse activations. By integrating multi-level constraints comprising pixel-wise, distributional, and region-text contrastive objectives, our framework effectively suppresses semantic drift under low supervision. Furthermore, we construct 360-AGD, the first high-quality panoramic affordance grounding
dataset. Extensive experiments demonstrate that PanoAffordanceNet significantly outperforms existing methods, establishing a solid baseline for scene-level perception in
embodied intelligence. The source code and
benchmark dataset will be made publicly available at https://github.com/GL-ZHU925/PanoAffordanceNet.
Read the full abstract ▾
All authors Guoliang Zhu, Wanjun Jia, Caoyang Shao, Yuheng Zhang, Zhiyong Li, Kailun Yang
Computer Vision
LogoDiffuser: Training-Free Multilingual Logo Generation and Stylization via Letter-Aware Attention Control
Mingyu Kang, Hyein Seo, Yuna Jeong et al. (+2 more)
Recent advances in text-to-image generation have been remarkable, but generating
multilingual design logos that harmoniously integrate visual and textual elements remains a challenging task. Existing methods often distort character geometry when applying creative styles and struggle to support
multilingual text generation without additional training. To address these challenges, we propose LogoDiffuser, a training-free method that synthesizes
multilingual logo designs using the
multimodal diffusion transformer. Instead of using textual
prompts, we input the target characters as images, enabling robust character structure control regardless of
language. We first analyze the joint
attention mechanism to identify core
tokens, which are
tokens that strongly respond to textual structures. With this observation, our method integrates character structure and visual design by injecting the most informative
attention maps. Furthermore, we perform layer-wise aggregation of
attention maps to mitigate
attention shifts across layers and obtain consistent core
tokens. Extensive experiments and user studies demonstrate that our method achieves state-of-the-art performance in
multilingual logo generation.
Read the full abstract ▾
All authors Mingyu Kang, Hyein Seo, Yuna Jeong, Junhyeong Park, Yong Suk Choi
Computer Vision
LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos
Lei Shi, Victor Aregbede, Andreas Persson et al. (+3 more)
Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional
videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that
language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce
Language-Aware Planning (LAP), a novel method that leverages the expressiveness of
language to bridge visual observation and planning. LAP uses a finetuned
Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text
embeddings. These text
embeddings are more distinctive than visual
embeddings and are used in a
diffusion model for planning action sequences. We evaluate LAP on three procedure planning
benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of
language-aware planning.
Read the full abstract ▾
All authors Lei Shi, Victor Aregbede, Andreas Persson, Martin Längkvist, Amy Loutfi, Stephanie Lowry
Computer Vision
ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios
Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto et al. (+6 more)
Understanding human behavior from complementary
egocentric (ego) and exocentric (exo) points of view enables the development of systems that can support workers in industrial environments and enhance their safety. However, progress in this area is hindered by the lack of
datasets capturing both views in realistic industrial scenarios. To address this gap, we propose ENIGMA-360, a new ego-exo
dataset acquired in a real industrial scenario. The
dataset is composed of 180
egocentric and 180 exocentric procedural
videos temporally synchronized offering complementary information of the same scene. The 360
videos have been labeled with temporal and spatial annotations, enabling the study of different aspects of human behavior in industrial domain. We provide baseline experiments for 3 foundational tasks for human behavior understanding: 1) Temporal Action
Segmentation, 2) Keystep Recognition and 3)
Egocentric Human-Object Interaction Detection, showing the limits of state-of-the-art approaches on this challenging scenario. These results highlight the need for new models capable of robust ego-exo understanding in real-world environments. We publicly release the
dataset and its annotations at https://fpv-iplab.github.io/ENIGMA-360/.
Read the full abstract ▾
All authors Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Daniele Di Mauro, Camillo Quattrocchi, Alessandro Passanisi, Irene D'Ambra, Antonino Furnari, Giovanni Maria Farinella
Computer Vision · Cross-listed from cs.RO
Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environments
Haoyuan Li, Rui Liu, Hehe Fan et al. (+1 more)
Vision-
Language Navigation in Continuous Environments (VLN-CE) requires
agents to learn complex
reasoning from long-horizon human interactions. While Multi-modal Large
Language Models (MLLMs) have driven recent progress, current training paradigms struggle to balance generalization capability, error recovery and training stability. Specifically, (i) policies derived from SFT suffer from compounding errors, struggling to recover from out-of-distribution states, and (ii) Reinforcement Fine-Tuning (RFT) methods e.g. GRPO are bottlenecked by sparse outcome rewards. Their binary feedback fails to assign credit to individual steps, leading to gradient signal collapse in failure dominant batches. To address these challenges, we introduce Step-Aware Contrastive
Alignment (SACA), a framework designed to extract dense supervision from imperfect trajectories. At its core, the Perception-Grounded Step-Aware auditor evaluates progress step-by-step, disentangling failed trajectories into valid prefixes and exact divergence points. Leveraging these signals, Scenario-Conditioned Group Construction mechanism dynamically routes batches to specialized resampling and optimization strategies. Extensive experiments on VLN-CE
benchmarks demonstrate that SACA achieves state-of-the-art performance.
Read the full abstract ▾
All authors Haoyuan Li, Rui Liu, Hehe Fan, Yi Yang
Computer Vision
$M^2$-Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs
Kaixin Lin, Kunyu Peng, Di Wen et al. (+3 more)
Semantic occupancy prediction enables dense 3D
geometric and semantic understanding for autonomous driving. However, existing camera-based approaches implicitly assume complete surround-view observations, an assumption that rarely holds in real-world deployment due to occlusion, hardware malfunction, or communication failures. We study semantic occupancy prediction under incomplete multi-camera inputs and introduce $M^2$-Occ, a framework designed to preserve
geometric structure and semantic coherence when views are missing. $M^2$-Occ addresses two complementary challenges. First, a Multi-view Masked Reconstruction (MMR) module leverages the spatial overlap among neighboring cameras to recover missing-view representations directly in the feature space. Second, a Feature
Memory Module (FMM) introduces a learnable
memory bank that stores class-level semantic prototypes. By retrieving and integrating these global priors, the FMM refines ambiguous voxel features, ensuring semantic consistency even when observational evidence is incomplete. We introduce a systematic missing-view evaluation protocol on the nuScenes-based SurroundOcc
benchmark, encompassing both deterministic single-view failures and
stochastic multi-view dropout scenarios. Under the safety-critical missing back-view setting, $M^2$-Occ improves the IoU by 4.36%. As the number of missing cameras increases, the robustness gap further widens; for instance, under the setting with five missing views, our method boosts the IoU by 6.67%. These gains are achieved without compromising full-view performance. The source code will be publicly released at https://github.com/qixi7up/M2-Occ.
Read the full abstract ▾
All authors Kaixin Lin, Kunyu Peng, Di Wen, Yufan Chen, Ruiping Liu, Kailun Yang
Computer Vision
FetalAgents: A Multi-Agent System for Fetal Ultrasound Image and Video Analysis
Xiaotian Hu, Junwei Huang, Mingxuan Liu et al. (+9 more)
Fetal ultrasound (US) is the primary imaging modality for prenatal screening, yet its interpretation relies heavily on the expertise of the clinician. Despite advances in deep learning and foundation models, existing automated tools for fetal US analysis struggle to balance task-specific accuracy with the whole-process versatility required to support end-to-end clinical workflows. To address these limitations, we propose FetalAgents, the first multi-
agent system for comprehensive fetal US analysis. Through a lightweight, agentic coordination framework, FetalAgents dynamically orchestrates specialized
vision experts to maximize performance across diagnosis, measurement, and
segmentation. Furthermore, FetalAgents advances beyond static image analysis by supporting end-to-end
video stream summarization, where keyframes are automatically identified across multiple anatomical planes, analyzed by coordinated experts, and synthesized with patient metadata into a structured clinical report. Extensive multi-center external evaluations across eight clinical tasks demonstrate that FetalAgents consistently delivers the most robust and accurate performance when compared against specialized models and
multimodal large
language models (MLLMs), ultimately providing an auditable, workflow-aligned solution for fetal ultrasound analysis and reporting.
Read the full abstract ▾
All authors Xiaotian Hu, Junwei Huang, Mingxuan Liu, Kasidit Anmahapong, Yifei Chen, Yitong Luo, Yiming Huang, Xuguang Bai, Zihan Li, Yi Liao, Haibo Qu, Qiyuan Tian
Computer Vision
EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning
Chengjun Yu, Xuhan Zhu, Chaoqun Du et al. (+4 more)
Multimodal large
language models (MLLMs) are increasingly considered as a foundation for
embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an
egocentric viewpoint. We study this gap through a new task,
Egocentric Scene Prediction with LOng-horizon
REasoning: given an initial-scene image and a sequence of atomic action descriptions, a model is asked to predict the final scene after all actions are executed. To enable systematic evaluation, we introduce EXPLORE-Bench, a
benchmark curated from real first-person
videos spanning diverse scenarios. Each instance pairs long action sequences with structured final-scene annotations, including object categories, visual attributes, and inter-object relations, which supports fine-grained, quantitative assessment. Experiments on a range of proprietary and open-source MLLMs reveal a significant performance gap to humans, indicating that long-horizon
egocentric reasoning remains a major challenge. We further analyze test-time scaling via stepwise
reasoning and show that decomposing long action sequences can improve performance to some extent, while incurring non-trivial computational overhead. Overall, EXPLORE-Bench provides a principled testbed for measuring and advancing long-horizon
reasoning for
egocentric embodied perception.
Read the full abstract ▾
All authors Chengjun Yu, Xuhan Zhu, Chaoqun Du, Pengfei Yu, Wei Zhai, Yang Cao, Zheng-Jun Zha