Computer Vision · Cross-listed from cs.CR
SpooFL: Spoofing Federated Learning
Isaac Baglin, Xiatian Zhu and Simon Hadfield
Traditional defenses against Deep Leakage (DL) attacks in
Federated Learning (FL) primarily focus on obfuscation, introducing noise, transformations or encryption to degrade an attacker's ability to reconstruct private data. While effective to some extent, these methods often still leak high-level information such as class distributions or feature representations, and are frequently broken by increasingly powerful denoising attacks. We propose a fundamentally different perspective on FL defense: framing it as a spoofing problem.We introduce SpooFL (Figure 1), a spoofing-based defense that deceives attackers into believing they have recovered the true training data, while actually providing convincing but entirely synthetic samples from an unrelated task. Unlike prior synthetic-data defenses that share classes or distributions with the private data and thus still leak semantic information, SpooFL uses a state-of-the-art generative model trained on an external
dataset with no class overlap. As a result, attackers are misled into recovering plausible yet completely irrelevant samples, preventing meaningful data leakage while preserving FL training integrity. We implement the first example of such a spoofing defense, and evaluate our method against state-of-the-art DL defenses and demonstrate that it successfully misdirects attackers without compromising model performance significantly.
Read the full abstract ▾
Computer Vision
Deep Leakage with Generative Flow Matching Denoiser
Isaac Baglin, Xiatian Zhu and Simon Hadfield
Federated Learning (FL) has emerged as a powerful paradigm for decentralized model training, yet it remains vulnerable to deep leakage (DL) attacks that reconstruct private client data from shared model updates. While prior DL methods have demonstrated varying levels of success, they often suffer from instability, limited fidelity, or poor robustness under realistic FL settings. We introduce a new DL attack that integrates a generative Flow Matching (FM) prior into the reconstruction process. By guiding optimization toward the distribution of realistic images (represented by a flow matching foundation model), our method enhances reconstruction fidelity without requiring knowledge of the private data. Extensive experiments on multiple
datasets and target models demonstrate that our approach consistently outperforms state-of-the-art attacks across pixel-level, perceptual, and feature-based similarity metrics. Crucially, the method remains effective across different training epochs, larger client batch sizes, and under common defenses such as noise injection, clipping, and sparsification. Our findings call for the development of new defense strategies that explicitly account for adversaries equipped with powerful generative priors.
Read the full abstract ▾
Computer Vision
Federated Transformer-GNN for Privacy-Preserving Brain Tumor Localization with Modality-Level Explainability
Andrea Protani, Riccardo Taiello, Marc Molina Van Den Bosch et al. (+1 more)
Deep learning models for
brain tumor analysis require large and diverse
datasets that are often siloed across healthcare institutions due to
privacy regulations. We present a
federated learning framework for
brain tumor localization that enables multi-institutional collaboration without sharing sensitive patient data. Our method extends a hybrid
Transformer-Graph Neural Network architecture derived from prior decoder-free supervoxel GNNs and is deployed within CAFEIN\textsuperscript{\textregistered}, CERN's
federated learning platform designed for healthcare environments. We provide an explainability analysis through
Transformer attention mechanisms that reveals which MRI modalities drive the model predictions. Experiments on the BraTS
dataset demonstrate a key finding: while isolated training on individual client data triggers early stopping well before reaching full training capacity,
federated learning enables continued model improvement by leveraging distributed data, ultimately matching centralized performance. This result provides strong justification for
federated learning when dealing with complex tasks and high-dimensional input data, as aggregating knowledge from multiple institutions significantly benefits the learning process. Our explainability analysis, validated through rigorous statistical testing on the full test set (paired t-tests with Bonferroni correction), reveals that deeper network layers significantly increase
attention to T2 and FLAIR modalities ($p<0.001$, Cohen's $d$=1.50), aligning with clinical practice.
Read the full abstract ▾
All authors Andrea Protani, Riccardo Taiello, Marc Molina Van Den Bosch, Luigi Serio
Computer Vision · Cross-listed from cs.RO
ExPrIS: Knowledge-Level Expectations as Priors for Object Interpretation from Sensor Data
Marian Renz, Martin Günther, Felix Igelbrink et al. (+2 more)
While deep learning has significantly advanced robotic object recognition, purely data-driven approaches often lack semantic consistency and fail to leverage valuable, pre-existing knowledge about the environment. This report presents the ExPrIS project, which addresses this challenge by investigating how knowledge-level expectations can serve as to improve object interpretation from sensor data. Our approach is based on the incremental construction of a 3D Semantic Scene Graph (3DSSG). We integrate expectations from two sources: contextual priors from past observations and semantic knowledge from external graphs like ConceptNet. These are embedded into a heterogeneous Graph Neural Network (GNN) to create an expectation-biased inference process. This method moves beyond static, frame-by-frame analysis to enhance the robustness and consistency of scene understanding over time. The report details this architecture, its evaluation, and outlines its planned integration on a mobile robotic platform.
Read the full abstract ▾
All authors Marian Renz, Martin Günther, Felix Igelbrink, Oscar Lima, Martin Atzmueller
Computer Vision · Cross-listed from cs.LG
Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
Adam Rokah, Daniel Veress, Caleb Caulk et al. (+1 more)
Mixture-of-Experts (MoE) architectures enable conditional computation by routing inputs to multiple expert subnetworks and are often motivated as a mechanism for scaling large
language models. In this project, we instead study MoE behavior in an image classification setting, focusing on predictive performance, expert utilization, and generalization. We compare dense, SoftMoE, and SparseMoE classifier heads on the CIFAR10
dataset under comparable model capacity. Both MoE variants achieve slightly higher validation accuracy than the dense baseline while maintaining balanced expert utilization through regularization, avoiding expert collapse. To analyze generalization, we compute Hessian-based sharpness metrics at convergence, including the largest
eigenvalue and trace of the loss Hessian, evaluated on both training and test data. We find that SoftMoE exhibits higher sharpness by these metrics, while Dense and SparseMoE lie in a similar
curvature regime, despite all models achieving comparable generalization performance. Complementary loss surface perturbation analyses reveal qualitative differences in non-local behavior under finite parameter perturbations between dense and MoE models, which help contextualize
curvature-based measurements without directly explaining validation accuracy. We further evaluate empirical inference efficiency and show that naively implemented conditional routing does not yield inference speedups on modern hardware at this scale, highlighting the gap between theoretical and realized efficiency in sparse MoE models.
Read the full abstract ▾
All authors Adam Rokah, Daniel Veress, Caleb Caulk, Sourav Sharan
Computer Vision
SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generation
Yanan Wang, Linjie Ren, Zihao Li et al. (+2 more)
While
video-to-audio generation has achieved remarkable progress in semantic and temporal
alignment, most existing studies focus solely on these aspects, paying limited
attention to the spatial perception and immersive quality of the synthesized audio. This limitation stems largely from current models' reliance on mono audio
datasets, which lack the binaural spatial information needed to learn visual-to-spatial audio mappings. To address this gap, we introduce two key contributions: we construct BinauralVGGSound, the first large-scale
video-binaural audio
dataset designed to support spatially aware
video-to-audio generation; and we propose a end-to-end spatial audio generation framework guided by visual cues, which explicitly models spatial features. Our framework incorporates a visual-guided audio spatialization module that ensures the generated audio exhibits realistic spatial attributes and layered spatial depth while maintaining semantic and temporal
alignment. Experiments show that our approach substantially outperforms state-of-the-art models in spatial fidelity and delivers a more immersive auditory experience, without sacrificing temporal or semantic consistency. The demo page can be accessed at https://github.com/renlinjie868-web/SpatialV2A.
Read the full abstract ▾
All authors Yanan Wang, Linjie Ren, Zihao Li, Junyi Wang, Tian Gan
Computer Vision
LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understanding
Xiaodong Wang, Langling Huang, Zhirong Wu et al. (+4 more)
The development of
multimodal large
language models (MLLMs) has advanced general
video understanding. However, existing
video evaluation
benchmarks primarily focus on non-interactive
videos, such as movies and recordings. To fill this gap, this paper proposes the first omnimodal
benchmark for interactive livestream
videos, LiViBench. It features a diverse set of 24 tasks, highlighting the perceptual,
reasoning, and livestream-specific challenges. To efficiently construct the
dataset, we design a standardized semi-automatic annotation workflow that incorporates the human-in-the-loop at multiple stages. The workflow leverages multiple MLLMs to form a multi-
agent system for comprehensive
video description and uses a seed-question-driven method to construct high-quality annotations. All interactive
videos in the
benchmark include audio,
speech, and real-time comments modalities. To enhance models' understanding of interactive
videos, we design tailored two-stage instruction-tuning and propose a
Video-to-Comment
Retrieval (VCR) module to improve the model's ability to utilize real-time comments. Based on these advancements, we develop LiVi-LLM-7B, an MLLM with enhanced knowledge of interactive livestreams. Experiments show that our model outperforms larger open-source models with up to 72B parameters, narrows the gap with leading proprietary models on LiViBench, and achieves enhanced performance on general
video benchmarks, including VideoMME, LongVideoBench, MLVU, and VideoEval-Pro.
Read the full abstract ▾
All authors Xiaodong Wang, Langling Huang, Zhirong Wu, Xu Zhao, Teng Xu, Xuhong Xia, Peixi Peng
Computer Vision · Cross-listed from eess.IV
Filtered 2D Contour-Based Reconstruction of 3D STL Model from CT-DICOM Images
K.Punnam Chandar and Y.Ravi Kumar
Reconstructing a 3D Stereo-lithography (STL) Model from 2D Contours of scanned structure in Digital Imaging and Communication in Medicine (DICOM) images is crucial to understand the geometry and deformity. Computed
Tomography (CT) images are processed to enhance the contrast, reduce the noise followed by smoothing. The processed CT images are segmented using thresholding technique. 2D contour data points are extracted from segmented CT images and are used to construct 3D STL Models. The 2D contour data points may contain outliers as a result of
segmentation of low resolution images and the geometry of the constructed 3D structure deviate from the actual. To cope with the imperfections in
segmentation process, in this work we propose to use filtered 2D contour data points to reconstruct 3D STL Model. The filtered 2D contour points of each image are delaunay triangulated and joined layer-by-layer to reconstruct the 3D STL model. The 3D STL Model reconstruction is verified on i) 2D Data points of basic shapes and ii) Region of Interest (ROI) of human pelvic bone and are presented as case studies. The 3D STL model constructed from 2D contour data points of ROI of segmented pelvic bone with and without filtering are presented. The 3D STL model reconstructed from filtered 2D data points improved the geometry of model compared to the model reconstructed without filtering 2D data points.
Read the full abstract ▾
Computer Vision
Unified Multi-Dataset Training for TBPS
Nilanjana Chatterjee, Sidharatha Garg, A V Subramanyam et al. (+1 more)
Text-Based Person Search (TBPS) has seen significant progress with
vision-
language models (VLMs), yet it remains constrained by limited training data and the fact that VLMs are not inherently pre-trained for pedestrian-centric recognition. Existing TBPS methods therefore rely on
dataset-centric fine-tuning to handle distribution shift, resulting in multiple independently trained models for different
datasets. While synthetic data can increase the scale needed to fine-tune VLMs, it does not eliminate
dataset-specific adaptation. This motivates a fundamental question: can we train a single unified TBPS model across multiple
datasets? We show that naive joint training over all
datasets remains sub-optimal because current training paradigms do not scale to a large number of unique person identities and are vulnerable to noisy image-text pairs. To address these challenges, we propose Scale-TBPS with two contributions: (i) a noise-aware unified
dataset curation strategy that cohesively merges diverse TBPS
datasets; and (ii) a scalable discriminative identity learning framework that remains effective under a large number of unique identities. Extensive experiments on CUHK-PEDES, ICFG-PEDES, RSTPReid, IIITD-20K, and UFine6926 demonstrate that a single Scale-TBPS model outperforms
dataset-centric optimized models and naive joint training.
Read the full abstract ▾
All authors Nilanjana Chatterjee, Sidharatha Garg, A V Subramanyam, Brejesh Lall
Computer Vision
Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers
Xinyu Peng, Han Li, Yuyang Huang et al. (+7 more)
Existing
video frame interpolation (VFI) methods often adopt a frame-centric approach, processing
videos as independent short segments (e.g., triplets), which leads to temporal inconsistencies and motion artifacts. To overcome this, we propose a holistic,
video-centric paradigm named Local
Diffusion Forcing for
Video Frame Interpolation (LDF-VFI). Our framework is built upon an auto-regressive
diffusion transformer that models the entire
video sequence to ensure long-range temporal coherence. To mitigate error accumulation inherent in auto-regressive generation, we introduce a novel skip-concatenate sampling strategy that effectively maintains temporal stability. Furthermore, LDF-VFI incorporates sparse, local
attention and tiled VAE encoding, a combination that not only enables efficient processing of long sequences but also allows generalization to arbitrary spatial resolutions (e.g., 4K) at inference without retraining. An enhanced conditional VAE decoder, which leverages multi-scale features from the input
video, further improves reconstruction fidelity. Empirically, LDF-VFI achieves state-of-the-art performance on challenging VFI
benchmarks, demonstrating superior per-frame quality and temporal consistency, especially in scenes with large motion. The source code is available at https://github.com/xypeng9903/LDF-VFI.
Read the full abstract ▾
All authors Xinyu Peng, Han Li, Yuyang Huang, Ziyang Zheng, Yaoming Wang, Xin Chen, Wenrui Dai, Chenglin Li, Junni Zou, Hongkai Xiong
Computer Vision
TempViz: On the Evaluation of Temporal Knowledge in Text-to-Image Models
Carolin Holtermann, Nina Krebs and Anne Lauscher
Time alters the visual appearance of entities in our world, like objects, places, and animals. Thus, for accurately generating contextually-relevant images, knowledge and
reasoning about time can be crucial (e.g., for generating a landscape in spring vs. in winter). Yet, although substantial work exists on understanding and improving temporal knowledge in natural
language processing, research on how temporal phenomena appear and are handled in text-to-image (T2I) models remains scarce. We address this gap with TempViz, the first data set to holistically evaluate temporal knowledge in image generation, consisting of 7.9k
prompts and more than 600 reference images. Using TempViz, we study the capabilities of five T2I models across five temporal knowledge categories. Human evaluation shows that temporal competence is generally weak, with no model exceeding 75% accuracy across categories. Towards larger-scale studies, we also examine automated evaluation methods, comparing several established approaches against human judgments. However, none of these approaches provides a reliable assessment of temporal cues - further indicating the pressing need for future research on temporal knowledge in T2I.
Read the full abstract ▾
Computer Vision
Erosion Attack for Adversarial Training to Enhance Semantic Segmentation Robustness
Yufei Song, Ziqi Zhou, Menghao Deng et al. (+4 more)
Existing
segmentation models exhibit significant vulnerability to
adversarial attacks.To improve robustness,
adversarial training incorporates
adversarial examples into model training. However, existing attack methods consider only global semantic information and ignore contextual semantic relationships within the samples, limiting the effectiveness of
adversarial training. To address this issue, we propose EroSeg-AT, a vulnerability-aware
adversarial training framework that leverages EroSeg to generate
adversarial examples. EroSeg first selects sensitive pixels based on pixel-level confidence and then progressively propagates perturbations to higher-confidence pixels, effectively disrupting the semantic consistency of the samples. Experimental results show that, compared to existing methods, our approach significantly improves attack effectiveness and enhances model robustness under
adversarial training.
Read the full abstract ▾
All authors Yufei Song, Ziqi Zhou, Menghao Deng, Yifan Hu, Shengshan Hu, Minghui Li, Leo Yu Zhang
Computer Vision
SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA
Xinyi Zheng, Yunze Liu, Chi-Hao Wu et al. (+5 more)
SpatialMem is a
memory-based system which integrates 3D geometry, semantics, and
language with one another in one combined, queryable representation. Starting with casually recorded
egocentric RGB
video, SpatialMem recreates metrically scaled interior scenes, recognizes structural 3D anchors (walls, doors, windows) as the first-layer scaffold, and associates open-vocabulary object nodes with evidence patches, visual
embeddings, and two-layer descriptions to 3D coordinates, enabling spatial-relation
reasoning,
language-directed
retrieval, and QA without special sensors. We test at scale on public
benchmarks. On VSI-Bench ($288$
videos, $5{,}130$ questions) SpatialMem scores $37.5$ overall with RGB alone, and its margin is precisely where a metric
memory should earn it (room size $41.4$ vs.\ $24.4$ with a same-size frame model). On the episodic-
memory task of OpenEQA ($953$ questions) the
memory serialised to text is lower than the blind control, but as evidence-frame
retrieval it recovers frame-level accuracy, matching uniform sampling at an equal eight-frame budget and pulling ahead as the budget tightens, with every answer grounded in linked 3D evidence. On $12$ ScanNet scenes evaluated using instance-level official annotations, the anchor-first hierarchy assigns the retrieved object to the proper structural parent where open-vocabulary 3D scene graphs essentially cannot ($0.24$ compared to $0.01$ with matched input), a capability difference instead of an overall win. The queryable index has a cost of $14$\,KB per scene without the linked evidence frames, and we make available the suites,
memories and code.
Read the full abstract ▾
All authors Xinyi Zheng, Yunze Liu, Chi-Hao Wu, Fan Zhang, Hao Zheng, Wenqi Zhou, Walterio W. Mayol-Cuevas, Junxiao Shen
Computer Vision
GAT-NeRF: Geometry-Aware-Transformer Enhanced Neural Radiance Fields for High-Fidelity 4D Facial Avatars
Zhe Chang, Haodong Jin, Ying Sun et al. (+2 more)
High-fidelity 4D dynamic facial avatar reconstruction from
monocular video is a critical yet challenging task, driven by increasing demands for immersive virtual human applications. While Neural Radiance Fields (NeRF) have advanced scene representation, their capacity to capture high-frequency facial details, such as dynamic wrinkles and subtle textures from information-constrained
monocular streams, requires significant enhancement. To tackle this challenge, we propose a novel hybrid neural radiance field framework, called Geometry-Aware-
Transformer Enhanced NeRF (GAT-NeRF) for high-fidelity and controllable 4D facial avatar reconstruction, which integrates the
Transformer mechanism into the NeRF pipeline. GAT-NeRF synergistically combines a coordinate-aligned Multilayer Perceptron (MLP) with a lightweight
Transformer module, termed as Geometry-Aware-
Transformer (GAT) due to its processing of multi-modal inputs containing explicit
geometric priors. The GAT module is enabled by fusing multi-modal input features, including 3D spatial coordinates, 3D Morphable Model (3DMM) expression parameters, and learnable latent codes to effectively learn and enhance feature representations pertinent to fine-grained geometry. The
Transformer's effective feature learning capabilities are leveraged to significantly augment the modeling of complex local facial patterns like dynamic wrinkles and acne scars. Comprehensive experiments unequivocally demonstrate GAT-NeRF's state-of-the-art performance in visual fidelity and high-frequency detail recovery, forging new pathways for creating realistic dynamic digital humans for multimedia applications.
Read the full abstract ▾
All authors Zhe Chang, Haodong Jin, Ying Sun, Yan Song, Hui Yu
Computer Vision
MTFlow: Time-Conditioned Flow Matching for Microtubule Segmentation in Noisy Microscopy Images
Sidi Mohamed Sid El Moctar, Achraf Ait Laydi, Yousef El Mourabit et al. (+1 more)
Microtubules are cytoskeletal filaments that play essential roles in many cellular processes and are key therapeutic targets in several diseases. Accurate
segmentation of microtubule networks is critical for studying their organization and dynamics but remains challenging due to filament
curvature, dense crossings, and image noise. We present MTFlow, a novel time-conditioned flow-matching model for microtubule
segmentation. Unlike conventional U-Net variants that predict masks in a single pass, MTFlow learns vector fields that iteratively transport noisy masks toward the ground truth, enabling interpretable, trajectory-based refinement. Our architecture combines a U-Net backbone with temporal
embeddings, allowing the model to capture the dynamics of uncertainty resolution along filament boundaries. We trained and evaluated MTFlow on synthetic and real microtubule
datasets and assessed its generalization capability on public biomedical
datasets of curvilinear structures such as retinal blood vessels and nerves. MTFlow achieves competitive
segmentation accuracy comparable to state-of-the-art models, offering a powerful and time-efficient tool for filamentous structure analysis with more precise annotations than manual or semi-automatic approaches.
Read the full abstract ▾
All authors Sidi Mohamed Sid El Moctar, Achraf Ait Laydi, Yousef El Mourabit, Hélène Bouvrais
Computer Vision
Multimodal system for skin cancer detection
Volodymyr Sydorskyi, Igor Krashenyi and Oleksii Yakubenko
Melanoma detection is vital for early diagnosis and effective treatment. While deep learning models on dermoscopic images have shown promise, they require specialized equipment, limiting their use in broader clinical settings. This study introduces a multi-modal melanoma detection system using conventional photo images, making it more accessible and versatile. Our system integrates image data with tabular metadata, such as patient demographics and lesion characteristics, to improve detection accuracy. It employs a multi-modal neural network combining image and metadata processing and supports a two-step model for cases with or without metadata. A three-stage pipeline further refines predictions by boosting
algorithms and enhancing performance. To address the challenges of a highly imbalanced
dataset, specific techniques were implemented to ensure robust training. An ablation study evaluated recent
vision architectures, boosting
algorithms, and loss functions, achieving a peak Partial ROC AUC of 0.18068 (0.2 maximum) and top-15
retrieval sensitivity of 0.78371. Results demonstrate that integrating photo images with metadata in a structured, multi-stage pipeline yields significant performance improvements. This system advances melanoma detection by providing a scalable, equipment-independent solution suitable for diverse healthcare environments, bridging the gap between specialized and general clinical practices.
Read the full abstract ▾
Computer Vision
POTR: Post-Training 3DGS Compression
Bert Ramlot, Martijn Courteaux, Peter Lambert et al. (+1 more)
3D Gaussian Splatting (3DGS) has recently emerged as a promising contender to Neural Radiance Fields (NeRF) in 3D scene reconstruction and real-time novel view synthesis. 3DGS outperforms NeRF in training and inference speed but has substantially higher storage requirements. To remedy this downside, we propose POTR, a post-training 3DGS codec built on two novel techniques. First, POTR introduces a novel
pruning approach that uses a modified 3DGS rasterizer to efficiently calculate every splat's individual removal effect simultaneously. This technique results in 2-4x fewer splats than other post-training
pruning techniques and as a result also significantly accelerates inference with experiments demonstrating 1.5-2x faster inference than other compressed models. Second, we propose a novel method to recompute lighting coefficients, significantly reducing their
entropy without using any form of training. Our fast and highly parallel approach especially increases AC lighting coefficient sparsity, with experiments demonstrating increases from 70% to 97%, with minimal loss in quality. Finally, we extend POTR with a simple fine-tuning scheme to further enhance
pruning, inference, and rate-distortion performance. Experiments demonstrate that POTR, even without fine-tuning, consistently outperforms all other post-training compression techniques in both rate-distortion performance and inference speed.
Read the full abstract ▾
All authors Bert Ramlot, Martijn Courteaux, Peter Lambert, Glenn Van Wallendael
Computer Vision
Symmetry Informative and Agnostic Feature Disentanglement for 3D Shapes
Tobias Weißberg, Weikang Wang, Paul Roetzer et al. (+2 more)
Shape descriptors, i.e., per-vertex features of 3D meshes or point clouds, are fundamental to shape analysis. Historically, various handcrafted geometry-aware descriptors and feature refinement techniques have been proposed. Recently, several studies have initiated a new research direction by leveraging features from image foundation models to create semantics-aware descriptors, demonstrating advantages across tasks like shape matching, editing, and
segmentation.
Symmetry, another key concept in shape analysis, has also attracted increasing
attention. Consequently, constructing
symmetry-aware shape descriptors is a natural progression. Although the recent method $\chi$ (Wang et al., 2025) successfully extracted
symmetry-informative features from semantic-aware descriptors, its features are only one-dimensional, neglecting other valuable semantic information. Furthermore, the extracted
symmetry-informative feature is usually noisy and yields small misclassified patches. To address these gaps, we propose a feature disentanglement approach which is simultaneously
symmetry informative and
symmetry agnostic. Further, we propose a feature refinement technique to improve the robustness of predicted
symmetry informative features. Extensive experiments, including intrinsic
symmetry detection, left/right classification, and shape matching, demonstrate the effectiveness of our proposed framework compared to various state-of-the-art methods, both qualitatively and quantitatively.
Read the full abstract ▾
All authors Tobias Weißberg, Weikang Wang, Paul Roetzer, Nafie El Amrani, Florian Bernard
Computer Vision
LocBAM: Advancing 3D Patch-Based Image Segmentation by Integrating Location Contex
Donnate Hooft, Stefan M. Fischer, Cosmin Bercea et al. (+2 more)
Patch-based methods are widely used in 3D medical image
segmentation to address
memory constraints in processing high-resolution volumetric data. However, these approaches often neglect the patch's location within the global volume, which can limit
segmentation performance when anatomical context is important. In this paper, we investigate the role of location context in patch-based 3D
segmentation and propose a novel
attention mechanism, LocBAM, that explicitly processes spatial information. Experiments on BTCV, AMOS22, and KiTS23 demonstrate that incorporating location context stabilizes training and improves
segmentation performance, particularly under low patch-to-volume coverage where global context is missing. Furthermore, LocBAM consistently outperforms classical coordinate encoding via CoordConv. Code is publicly available at https://github.com/compai-lab/2026-ISBI-hooft
Read the full abstract ▾
All authors Donnate Hooft, Stefan M. Fischer, Cosmin Bercea, Jan C. Peeken, Julia A. Schnabel
Computer Vision
UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking
Qihua Liang, Liang Chen, Yaozong Zheng et al. (+3 more)
Multi-modal object tracking has attracted considerable
attention by integrating multiple complementary inputs (e.g., thermal, depth, and event data) to achieve outstanding performance. Although current general-purpose multi-modal trackers primarily unify various modal tracking tasks (i.e., RGB-Thermal infrared, RGB-Depth or RGB-Event tracking) through
prompt learning, they still overlook the effective capture of spatio-temporal cues. In this work, we introduce a novel multi-modal tracking framework based on a mamba-style state space model, termed UBATrack. Our UBATrack comprises two simple yet effective modules: a Spatio-temporal Mamba Adapter (STMA) and a Dynamic Multi-modal Feature Mixer. The former leverages Mamba's long-sequence modeling capability to jointly model cross-modal dependencies and spatio-temporal visual cues in an adapter-tuning manner. The latter further enhances multi-modal representation capacity across multiple feature dimensions to improve tracking robustness. In this way, UBATrack eliminates the need for costly full-parameter fine-tuning, thereby improving the training efficiency of multi-modal tracking
algorithms. Experiments show that UBATrack outperforms state-of-the-art methods on RGB-T, RGB-D, and RGB-E tracking
benchmarks, achieving outstanding results on the LasHeR, RGBT234, RGBT210, DepthTrack, VOT-RGBD22, and VisEvent
datasets.
Read the full abstract ▾
All authors Qihua Liang, Liang Chen, Yaozong Zheng, Jian Nong, Zhiyi Mo, Bineng Zhong
Computer Vision
UniRoute: Unified Routing Mixture-of-Experts for Modality-Adaptive Remote Sensing Change Detection
Qingling Shu, Sibao Chen, Wei Lu et al. (+2 more)
Current remote sensing change detection (CD) methods mainly rely on specialized models, which limits the scalability toward modality-adaptive Earth observation. For homogeneous CD, precise boundary delineation relies on fine-grained spatial cues and local pixel interactions, whereas heterogeneous CD instead requires broader contextual information to suppress speckle noise and
geometric distortions. Moreover, difference operator (e.g., subtraction) works well for aligned homogeneous images but introduces artifacts in cross-modal or geometrically misaligned scenarios. Across different modality settings, specialized models based on static backbones or fixed difference operations often prove insufficient. To address this challenge, we propose UniRoute, a unified framework for modality-adaptive learning by reformulating feature extraction and fusion as conditional routing problems. We introduce an Adaptive Receptive Field Routing MoE (AR2-MoE) module to disentangle local spatial details from global semantic context, and a Modality-Aware Difference Routing MoE (MDR-MoE) module to adaptively select the most suitable fusion primitive at each pixel. In addition, we propose a Consistency-Aware Self-
Distillation (CASD) strategy that stabilizes unified training under data-scarce heterogeneous settings by enforcing multi-level consistency. Extensive experiments on five public
datasets demonstrate that UniRoute achieves strong overall performance, with a favorable accuracy-efficiency trade-off under a unified deployment setting.
Read the full abstract ▾
All authors Qingling Shu, Sibao Chen, Wei Lu, Zhihui You, Chengzhuang Liu
Computer Vision
Synthetic Data Augmentation for Multi-Task Chinese Porcelain Classification: A Stable Diffusion Approach
Ziyao Ling, Silvia Mirri, Paola Salomoni et al. (+1 more)
The scarcity of training data presents a fundamental challenge in applying deep learning to archaeological artifact classification, particularly for the rare types of Chinese porcelain. This study investigates whether synthetic images generated through Stable
Diffusion with Low-Rank Adaptation (LoRA) can effectively augment limited real
datasets for multi-task CNN-based porcelain classification. Using MobileNetV3 with transfer learning, we conducted controlled experiments comparing models trained on pure real data against those trained on mixed real-synthetic
datasets (95:5 and 90:10 ratios) across four classification tasks: dynasty, glaze, kiln and type identification. Results demonstrate task-specific benefits: type classification showed the most substantial improvement (5.5% F1-macro increase with 90:10 ratio), while dynasty and kiln tasks exhibited modest gains (3-4%), suggesting that synthetic augmentation effectiveness depends on the
alignment between generated features and task-relevant visual signatures. Our work contributes practical guidelines for deploying generative AI in archaeological research, demonstrating both the potential and limitations of synthetic data when archaeological authenticity must be balanced with data diversity.
Read the full abstract ▾
All authors Ziyao Ling, Silvia Mirri, Paola Salomoni, Giovanni Delnevo
Computer Vision
Reconstruction-Anchored Diffusion Model for Text-to-Motion Generation
Yifei Liu, Changxing Ding, Ling Guo et al. (+2 more)
Diffusion models have seen widespread adoption for text-driven human motion generation and related tasks due to their impressive generative capabilities and flexibility. However, current motion
diffusion models face two major limitations: a representational gap caused by pre-trained text encoders that lack motion-specific information, and error propagation during the iterative denoising process. This paper introduces Reconstruction-Anchored
Diffusion Model (RAM) to address these challenges. First, RAM leverages a motion latent space as intermediate supervision for text-to-motion generation. To this end, RAM co-trains a motion reconstruction branch with two key objective functions: self-regularization to enhance the discrimination of the motion space and motion-centric latent
alignment to enable accurate mapping from text to the motion latent space. Second, we propose Reconstructive Error Guidance (REG), a testing-stage guidance mechanism that exploits the motion
diffusion model's inherent self-correction ability to mitigate error propagation. At each denoising step, REG uses the motion reconstruction branch to reconstruct the previous estimate, reproducing the prior error patterns. By amplifying the residual between the current prediction and the reconstructed estimate, REG highlights the improvements in the current prediction. Extensive experiments demonstrate that RAM achieves significant improvements and state-of-the-art performance. Our code will be released.
Read the full abstract ▾
All authors Yifei Liu, Changxing Ding, Ling Guo, Huaiguang Jiang, Qiong Cao
Computer Vision
FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes
Jiaxuan Liu, Yang Xiang, Han Zhao et al. (+2 more)
Movie dubbing is the task of synthesizing
speech from scripts conditioned on
video scenes, requiring accurate lip sync, faithful timbre transfer, and proper modeling of character identity and
emotion. However, existing methods face two major limitations: (1) high-quality
multimodal dubbing
datasets are limited in scale, suffer from high word error rates, contain sparse annotations, rely on costly manual labeling, and are restricted to monologue scenes, all of which hinder effective model training; (2) existing dubbing models rely solely on the lip region to learn audio-visual
alignment, which limits their applicability to complex live-action cinematic scenes, and exhibit suboptimal performance in lip sync,
speech quality, and emotional expressiveness. To address these issues, we propose FunCineForge, which comprises an end-to-end production pipeline for large-scale dubbing
datasets and an MLLM-based dubbing model designed for diverse cinematic scenes. Using the pipeline, we construct the first Chinese television dubbing
dataset with rich annotations, and demonstrate the high quality of these data. Experiments across monologue, narration, dialogue, and multi-speaker scenes show that our dubbing model consistently outperforms SOTA methods in audio quality, lip sync, timbre transfer, and instruction following. Code and demos are available at https://anonymous.4open.science/w/FunCineForge.
Read the full abstract ▾
All authors Jiaxuan Liu, Yang Xiang, Han Zhao, Xiangang Li, Zhenhua Ling
Computer Vision
M2I2HA: Multi-modal Object Detection Based on Intra- and Inter-Modal Hypergraph Attention
Xiaofan Yang, Yubin Liu, Wei Pan et al. (+5 more)
Recent advances in multi-modal detection have significantly improved detection accuracy in challenging environments (e.g., low light, overexposure). By integrating RGB with modalities such as thermal and depth, multi-modal fusion increases data redundancy and system robustness. However, significant challenges remain in effectively extracting task-relevant information both within and across modalities, as well as in achieving precise cross-modal
alignment. While CNNs excel at feature extraction, they are limited by constrained receptive fields, strong inductive biases, and difficulty in capturing long-range dependencies.
Transformer-based models offer global context but suffer from quadratic computational complexity and are confined to pairwise correlation modeling. Mamba and other State Space Models (SSMs), on the other hand, are hindered by their sequential scanning mechanism, which flattens 2D spatial structures into 1D sequences, disrupting topological relationships and limiting the modeling of complex higher-order dependencies. To address these issues, we propose a multi-modal perception network based on hypergraph theory called M2I2HA. Our architecture includes an Intra-Hypergraph Enhancement module to capture global many-to-many high-order relationships within each modality, and an Inter-Hypergraph Fusion module to align, enhance, and fuse cross-modal features by bridging configuration and spatial gaps between data sources. We further introduce a M2-FullPAD module to enable adaptive multi-level fusion of multi-modal enhanced features within the network, meanwhile enhancing data distribution and flow across the architecture. Extensive object detection experiments on multiple public
datasets against baselines demonstrate that M2I2HA achieves state-of-the-art performance in multi-modal object detection tasks.
Read the full abstract ▾
All authors Xiaofan Yang, Yubin Liu, Wei Pan, Guoqing Chu, Junming Zhang, Jie Zhao, Zhuoqi Man, Xuanming Cao
Computer Vision
Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysis
Keita Takeda and Tomoya Sakai
This study investigates the feature representations produced by publicly available open source medical
vision-
language models (VLMs). While medical VLMs are expected to capture diagnostically relevant features, their learned representations remain underexplored, and standard evaluations like classification accuracy do not fully reveal if they acquire truly discriminative, lesion-specific features. Understanding these representations is crucial for revealing medical image structures and improving downstream tasks in medical image analysis. This study aims to investigate the feature distributions learned by medical VLMs and evaluate the impact of medical specialization. We analyze the feature distribution of multiple image modalities extracted by some representative medical VLMs across lesion classification
datasets on multiple modalities. These distributions were compared them with non-medical VLMs to assess the domain-specific medical training. Our experiments showed that medical VLMs can extract discriminative features that are effective for medical classification tasks. Moreover, it was found that non-medical VLMs with recent improvement with contextual enrichment such as LLM2CLIP produce more refined feature representations. Our results imply that enhancing text encoder is more crucial than training intensively on medical images when developing medical VLMs. Notably, non-medical models are particularly vulnerable to biases introduced by overlaied text strings on images. These findings underscore the need for careful consideration on model selection according to downstream tasks besides potential risks in inference due to background biases such as textual information in images.
Read the full abstract ▾
Computer Vision
Using Multi-Instance Learning to Identify Unique Polyps in Colon Capsule Endoscopy Images
Puneet Sharma, Kristian Dalsbø Hindberg, Eibe Frank et al. (+2 more)
Identifying unique polyps in colon capsule endoscopy (CCE) images is a critical yet challenging task for medical personnel due to the large volume of images, the cognitive load it creates for clinicians, and the ambiguity in labeling specific frames. This paper formulates this problem as a multi-instance learning (MIL) task, where a query polyp image is compared with a target bag of images to determine uniqueness. We employ a multi-instance verification (MIV) framework that incorporates
attention mechanisms, such as variance-excited multi-head
attention (VEMA) and distance-based
attention (DBA), to enhance the model's ability to extract meaningful representations. Additionally, we investigate the impact of self-supervised learning using SimCLR to generate robust
embeddings. Experimental results on a
dataset of 1912 polyps from 754 patients demonstrate that
attention mechanisms significantly improve performance, with DBA L1 achieving the highest test accuracy of 86.26% and a test AUC of 0.928 using a ConvNeXt backbone with SimCLR pretraining. This study underscores the potential of MIL and self-supervised learning in advancing automated analysis of Colon Capsule Endoscopy images, with implications for broader medical imaging applications.
Read the full abstract ▾
All authors Puneet Sharma, Kristian Dalsbø Hindberg, Eibe Frank, Benedicte Schelde-Olesen, Ulrik Deding
Computer Vision
ReinPath: A Multimodal Reinforcement Learning Approach for Pathology
Kangcheng Zhou, Jun Jiang, Qing Zhang et al. (+3 more)
Interpretability is significant in computational pathology, leading to the development of
multimodal information integration from histopathological image and corresponding text data.However, existing
multimodal methods have limited interpretability due to the lack of high-quality
dataset that support explicit
reasoning and inference and simple
reasoning process.To address the above problems, we introduce a novel
multimodal pathology large
language model with strong
reasoning capabilities.To improve the generation of accurate and contextually relevant textual descriptions, we design a semantic reward strategy integrated with group relative policy optimization.We construct a high-quality pathology visual question answering (VQA)
dataset, specifically designed to support complex
reasoning tasks.Comprehensive experiments conducted on this
dataset demonstrate that our method outperforms state-of-the-art methods, even when trained with only 20% of the data.Our method also achieves comparable performance on downstream zero-shot image classification task compared with CLIP.
Read the full abstract ▾
All authors Kangcheng Zhou, Jun Jiang, Qing Zhang, Shuang Zheng, Qingli Li, Shugong Xu
Computer Vision · Cross-listed from cs.CL
Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning
Yifan Wang, Shiyu Li, Peiming Li et al. (+3 more)
Chain-of-Thought (CoT) prompting has achieved remarkable success in unlocking the
reasoning capabilities of Large
Language Models (LLMs). Although CoT prompting enhances
reasoning, its verbosity imposes substantial computational overhead. Recent works often focus exclusively on outcome
alignment and lack supervision on the intermediate
reasoning process. These deficiencies obscure the analyzability of the latent
reasoning chain. To address these challenges, we introduce Render-of-Thought (RoT), the first framework to reify the
reasoning chain by rendering textual steps into images, making the latent rationale explicit and traceable. Specifically, we leverage the
vision encoders of existing
Vision Language Models (VLMs) as semantic anchors to align the
vision embeddings with the textual space. This design ensures plug-and-play implementation without incurring additional pre-training overhead. Extensive experiments on mathematical and logical
reasoning benchmarks demonstrate that our method achieves 3-4x
token compression and substantial inference acceleration compared to explicit CoT. Furthermore, it maintains competitive performance against other methods, validating the feasibility of this paradigm. Our code is available at https://github.com/TencentBAC/RoT
Read the full abstract ▾
All authors Yifan Wang, Shiyu Li, Peiming Li, Xiaochen Yang, Yang Tang, Zheng Wei
Computer Vision
SimD3: A Synthetic drone Dataset with Payload and Bird Distractor Modeling for Robust Detection
Ami Pandat, Kanyala Muvva, Punna Rajasekhar et al. (+2 more)
Reliable drone detection is challenging due to limited annotated real-world data, large appearance variability, and the presence of visually similar distractors such as birds. To address these challenges, this paper introduces SimD3, a large-scale high-fidelity synthetic
dataset designed for robust drone detection in complex aerial environments. Unlike existing synthetic drone
datasets, SimD3 explicitly models drones with heterogeneous payloads, incorporates multiple bird species as realistic distractors, and leverages diverse Unreal Engine 5 environments with controlled weather, lighting, and flight trajectories captured using a 360 six-camera rig. Using SimD3, we conduct an extensive experimental evaluation within the YOLOv5 detection framework, including an
attention-enhanced variant termed Yolov5m+C3b, where standard bottleneck-based C3 blocks are replaced with C3b modules. Models are evaluated on synthetic data, combined synthetic and real data, and multiple unseen real-world
benchmarks to assess robustness and generalization. Experimental results show that SimD3 provides effective supervision for small-object drone detection and that Yolov5m+C3b consistently outperforms the baseline across in-domain and cross-
dataset evaluations. These findings highlight the utility of SimD3 for training and benchmarking robust drone detection models under diverse and challenging conditions.
Read the full abstract ▾
All authors Ami Pandat, Kanyala Muvva, Punna Rajasekhar, Gopika Vinod, Rohit Shukla