Computer Vision
Not all Blends are Equal: The BLEMORE Dataset of Blended Emotion Expressions with Relative Salience Annotations
Tim Lachmann, Alexandra Israelsson, Christina Tornberg et al. (+4 more)
Humans often experience not just a single basic
emotion at a time, but rather a blend of several
emotions with varying salience. Despite the importance of such blended
emotions, most
video-based
emotion recognition approaches are designed to recognize single
emotions only. The few approaches that have attempted to recognize blended
emotions typically cannot assess the relative salience of the
emotions within a blend. This limitation largely stems from the lack of
datasets containing a substantial number of blended
emotion samples annotated with relative salience. To address this shortcoming, we introduce BLEMORE, a novel
dataset for
multimodal (
video, audio) blended
emotion recognition that includes information on the relative salience of each
emotion within a blend. BLEMORE comprises over 3,000 clips from 58 actors, performing 6 basic
emotions and 10 distinct blends, where each blend has 3 different salience configurations (50/50, 70/30, and 30/70). Using this
dataset, we conduct extensive evaluations of state-of-the-art
video classification approaches on two blended
emotion prediction tasks: (1) predicting the presence of
emotions in a given sample, and (2) predicting the relative salience of
emotions in a blend. Our results show that unimodal classifiers achieve up to 29% presence accuracy and 13% salience accuracy on the validation set, while
multimodal methods yield clear improvements, with ImageBind + WavLM reaching 35% presence accuracy and HiCMAE 18% salience accuracy. On the held-out test set, the best models achieve 33% presence accuracy (VideoMAEv2 + HuBERT) and 18% salience accuracy (HiCMAE). In sum, the BLEMORE
dataset provides a valuable resource to advancing research on
emotion recognition systems that account for the complexity and significance of blended
emotion expressions.
Read the full abstract ▾
All authors Tim Lachmann, Alexandra Israelsson, Christina Tornberg, Teimuraz Saghinadze, Michal Balazia, Philipp Müller, Petri Laukka
Computer Vision
ObjectVisA-120: Object-based Visual Attention Prediction in Interactive Street-crossing Environments
Igor Vozniak, Philipp Mueller, Nils Lipp et al. (+6 more)
The object-based nature of human visual
attention is well-known in cognitive science, but has only played a minor role in computational visual
attention models so far. This is mainly due to a lack of suitable
datasets and evaluation metrics for object-based
attention. To address these limitations, we present ObjectVisA-120 – a novel 120-participant
dataset of spatial street-crossing
navigation in virtual reality specifically geared to object-based
attention evaluations. The uniqueness of the presented
dataset lies in the ethical and safety affiliated challenges that make collecting comparable data in real-world environments highly difficult. ObjectVisA-120 not only features accurate gaze data and a complete state-space representation of objects in the virtual environment, but it also offers variable scenario complexities and rich annotations, including panoptic
segmentation, depth information, and vehicle keypoints. We further propose object-based similarity (oSIM) as a novel metric to evaluate the performance of object-based visual
attention models, a previously unexplored performance characteristic. Our evaluations show that explicitly optimising for object-based
attention not only improves oSIM performance but also leads to an improved model performance on common metrics. In addition, we present SUMGraph, a Mamba U-Net-based model, which explicitly encodes critical scene objects (vehicles) in a graph representation, leading to further performance improvements over several state-of-the-art visual
attention prediction methods. The
dataset, code and models will be publicly released.
Read the full abstract ▾
All authors Igor Vozniak, Philipp Mueller, Nils Lipp, Janis Sprenger, Konstantin Poddubnyy, Davit Hovhannisyan, Christian Mueller, Andreas Bulling, Philipp Slusallek
Computer Vision
Rethinking Skip Connections: Additive U-Net for Robust and Interpretable Denoising
Vikram R Lakkavalli
Skip connections are central to U-Net architectures for image denoising, but standard concatenation doubles channel dimensionality and obscures information flow, allowing uncontrolled noise transfer. We propose the Additive U-Net, which replaces concatenative skips with gated additive connections. Each skip pathway is scaled by a learnable non-negative scalar, offering explicit and interpretable control over encoder contributions while avoiding channel inflation. Evaluations on the Kodak-17 denoising
benchmark show that Additive U-Net achieves competitive PSNR/SSIM at noise levels {\sigma} = 15, 25, 50, with robustness across kernel schedules and depths. Notably, effective denoising is achieved even without explicit down/up-sampling or forced hierarchies, as the model naturally learns a progression from high-frequency to band-pass to low-frequency features. These results position additive skips as a lightweight and interpretable alternative to concatenation, enabling both efficient design and a clearer understanding of multi-scale information transfer in reconstruction networks.
Read the full abstract ▾
Computer Vision
GTPred: Benchmarking MLLMs for Interpretable Geo-localization and Time-of-capture Prediction
Jinnao Li, Tingzhu Chen and Changbo Wang
Geo-localization aims to infer the geographic location where an image was captured using observable visual evidence. Traditional methods achieve impressive results through large-scale training on massive image corpora. With the emergence of multi-modal large
language models (MLLMs), recent studies have explored their applications in geo-localization, benefiting from improved accuracy and interpretability. However, existing
benchmarks largely ignore the temporal information inherent in images, which can further constrain the location. To bridge this gap, we introduce GTPred, a novel
benchmark for geo-temporal prediction. GTPred comprises 370 globally distributed images spanning over 120 years. We evaluate MLLM predictions by jointly considering year and hierarchical location sequence matching, and further assess intermediate
reasoning chains using meticulously annotated ground-truth
reasoning processes. Experiments on 8 proprietary and 7 open-source MLLMs show that, despite strong visual perception, current models remain limited in world knowledge and geo-temporal
reasoning. Results also demonstrate that incorporating temporal information significantly enhances location inference performance.
Read the full abstract ▾
Computer Vision
From 100,000+ images to winning the first brain MRI foundation model challenges: Sharing lessons and models
Pedro M. Gordaliza, Jaume Banus, Benoît Gérin et al. (+4 more)
Developing Foundation Models for medical image analysis is essential to overcome the unique challenges of radiological tasks. The first challenges of this kind for 3D
brain MRI, SSL3D and FOMO25, were held at MICCAI 2025. Our solution ranked first in tracks of both contests. It relies on a U-Net CNN architecture combined with strategies leveraging anatomical priors and neuroimaging domain knowledge. Notably, our models trained 1-2 orders of magnitude faster and were 10 times smaller than competing
transformer-based approaches. Models are available here: https://github.com/jbanusco/BrainFM4Challenges.
Read the full abstract ▾
All authors Pedro M. Gordaliza, Jaume Banus, Benoît Gérin, Maxence Wynen, Nataliia Molchanova, Jonas Richiardi, Meritxell Bach Cuadra
Computer Vision
ICo3D: An Interactive Conversational 3D Virtual Human
Richard Shaw, Youngkyoon Jang, Athanasios Papaioannou et al. (+4 more)
This work presents Interactive Conversational 3D Virtual Human (ICo3D), a method for generating an interactive, conversational, and photorealistic 3D human avatar. Based on multi-view captures of a subject, we create an animatable 3D face model and a dynamic 3D body model, both rendered by splatting Gaussian primitives. Once merged together, they represent a lifelike virtual human avatar suitable for real-time user interactions. We equip our avatar with an LLM for conversational ability. During conversation, the audio
speech of the avatar is used as a driving signal to animate the face model, enabling precise synchronization. We describe improvements to our dynamic Gaussian models that enhance photorealism: SWinGS++ for body reconstruction and HeadGaS++ for face reconstruction, and provide as well a solution to merge the separate face and body models without artifacts. We also present a demo of the complete system, showcasing several use cases of real-time conversation with the 3D avatar. Our approach offers a fully integrated virtual avatar experience, supporting both oral and written form interactions in immersive environments. ICo3D is applicable to a wide range of fields, including gaming, virtual assistance, and personalized education, among others. Project page: https://ico3d.github.io/
Read the full abstract ▾
All authors Richard Shaw, Youngkyoon Jang, Athanasios Papaioannou, Arthur Moreau, Helisa Dhamo, Zhensong Zhang, Eduardo Pérez-Pellitero
Computer Vision
TVWorld: Foundations for Remote-Control TV Agents
Zhantao Ma, Quanfeng Lu, Shuai Zhong et al. (+3 more)
Recent large
vision-
language models (LVLMs) have demonstrated strong potential for device control. However, existing research has primarily focused on point-and-click (PnC) interaction, while remote-control (RC) interaction commonly encountered in everyday TV usage remains largely underexplored. To fill this gap, we introduce TVWorld, an offline graph-based abstraction of real-world TV
navigation that enables reproducible and deployment-free evaluation. On this basis, we derive two complementary
benchmarks that comprehensively assess TV-use capabilities: TVWorld-N for
topology-aware
navigation and TVWorld-G for focus-aware grounding. These
benchmarks expose a key limitation of existing
agents: insufficient
topology awareness for focus-based, long-horizon TV
navigation. Motivated by this finding, we propose a
Topology-Aware Training framework that injects
topology awareness into LVLMs. Using this framework, we develop TVTheseus, a foundation model specialized for TV
navigation. TVTheseus achieves a success rate of $68.3\%$ on TVWorld-N, surpassing strong closed-source baselines such as Gemini 3 Flash and establishing state-of-the-art (SOTA) performance. Additional analyses further provide valuable insights into the development of effective TV-use
agents.
Read the full abstract ▾
All authors Zhantao Ma, Quanfeng Lu, Shuai Zhong, Dahai Yu, Ping Luo, Michael K. Ng
Computer Vision · Cross-listed from cs.SE
Earth Embeddings as Products: Taxonomy, Ecosystem, and Standardized Access
Heng Fang, Adam J. Stewart, Isaac Corley et al. (+2 more)
Geospatial Foundation Models (GFMs) provide powerful representations, but high compute costs hinder their widespread use. Pre-computed
embedding data products offer a practical "frozen" alternative, yet they currently exist in a fragmented ecosystem of incompatible formats and resolutions. This lack of standardization creates an engineering bottleneck that prevents meaningful model comparison and reproducibility. We formalize this landscape through a three-layer taxonomy: Data, Tools, and Value. We survey existing products to identify interoperability barriers. To bridge this gap, we extend TorchGeo with a unified API that standardizes the loading and querying of diverse
embedding products. By treating
embeddings as first-class geospatial
datasets, we decouple downstream analysis from model-specific engineering, providing a roadmap for more transparent and accessible Earth observation workflows.
Read the full abstract ▾
All authors Heng Fang, Adam J. Stewart, Isaac Corley, Xiao Xiang Zhu, Hossein Azizpour
Computer Vision
CLIP-Guided Adaptable Self-Supervised Learning for Human-Centric Visual Tasks
Mingshuang Luo, Ruibing Hou, Bo Chao et al. (+4 more)
Human-centric visual analysis plays a pivotal role in diverse applications, including surveillance, healthcare, and human-computer interaction. With the emergence of large-scale unlabeled human image
datasets, there is an increasing need for a general unsupervised pre-training model capable of supporting diverse human-centric downstream tasks. To achieve this goal, we propose CLASP (CLIP-guided Adaptable Self-suPervised learning), a novel framework designed for unsupervised pre-training in human-centric visual tasks. CLASP leverages the powerful
vision-
language model CLIP to generate both low-level (e.g., body parts) and high-level (e.g., attributes) semantic pseudo-labels. These multi-level semantic cues are then integrated into the learned visual representations, enriching their expressiveness and generalizability. Recognizing that different downstream tasks demand varying levels of semantic granularity, CLASP incorporates a
Prompt-Controlled Mixture-of-Experts (MoE) module. MoE dynamically adapts feature extraction based on task-specific
prompts, mitigating potential feature conflicts and enhancing transferability. Furthermore, CLASP employs a multi-task pre-training strategy, where part- and attribute-level pseudo-labels derived from CLIP guide the representation learning process. Extensive experiments across multiple
benchmarks demonstrate that CLASP consistently outperforms existing unsupervised pre-training methods, advancing the field of human-centric visual analysis.
Read the full abstract ▾
All authors Mingshuang Luo, Ruibing Hou, Bo Chao, Hong Chang, Zimo Liu, Yaowei Wang, Shiguang Shan
Computer Vision
SplatReasoner: Enhancing Embodied Reasoning and Grounding by Novel View Synthesis
Kim Yu-Ji, Dahye Lee, Kim Jun-Seong et al. (+6 more)
Vision-
Language Models (VLMs) have demonstrated strong
reasoning capabilities over images and
videos, yet their application to
embodied scene understanding often constrained by the fixed viewpoints stored in episodic RGB-D
memories. These observations may fail to capture query-relevant evidence due to occlusions, object truncation, restricted fields of view, or suboptimal view composition. We present SplatReasoner, a framework that introduces novel view synthesis into the VLM
reasoning process by leveraging 3D Gaussian Splatting (3DGS). Given a user query about a 3D scene, SplatReasoner retrieves relevant observations and synthesizes query-conditioned viewpoints that reveal the visual evidence needed to answer the query and ground the referred entities in 3D. Experiments show that query-conditioned novel view synthesis improves both
embodied reasoning and 3D grounding over fixed-view
memory and
language-embedded 3DGS baselines.
Read the full abstract ▾
All authors Kim Yu-Ji, Dahye Lee, Kim Jun-Seong, Nam Hyeon-Woo, GeonU Kim, Yongjin Kwon, Yu-Chiang Frank Wang, Jaesung Choe, Tae-Hyun Oh
Computer Vision
PhaseMark: A Post-hoc, Optimization-Free Watermarking of AI-generated Images in the Latent Frequency Domain
Sung Ju Lee and Nam Ik Cho
The proliferation of hyper-realistic images from Latent
Diffusion Models (LDMs) demands robust watermarking, yet existing post-hoc methods are prohibitively slow due to iterative optimization or inversion processes. We introduce PhaseMark, a single-shot, optimization-free framework that directly modulates the phase in the VAE latent frequency domain. This approach makes PhaseMark thousands of times faster than optimization-based techniques while achieving state-of-the-art resilience against severe attacks, including regeneration, without degrading image quality. We analyze four modulation variants, revealing a clear performance-quality trade-off. PhaseMark demonstrates a new paradigm where efficient, resilient watermarking is achieved by exploiting intrinsic latent properties.
Read the full abstract ▾
Computer Vision
A Streamlined Attention-Based Network for Descriptor Extraction
Mattia D'Urso, Emanuele Santellani, Christian Sormann et al. (+3 more)
We introduce SANDesc, a Streamlined
Attention-Based Network for Descriptor extraction that aims to improve on existing architectures for keypoint description. Our descriptor network learns to compute descriptors that improve matching without modifying the underlying keypoint detector. We employ a revised U-Net-like architecture enhanced with Convolutional Block
Attention Modules and residual paths, enabling effective local representation while maintaining computational efficiency. We refer to the building blocks of our model as Residual U-Net Blocks with
Attention. The model is trained using a modified triplet loss in combination with a curriculum learning-inspired hard negative mining strategy, which improves training stability. Extensive experiments on HPatches, MegaDepth-1500, and the Image Matching Challenge 2021 show that training SANDesc on top of existing keypoint detectors leads to improved results on multiple matching tasks compared to the original keypoint descriptors. At the same time, SANDesc has a model complexity of just 2.4 million parameters. As a further contribution, we introduce a new urban
dataset featuring 4K images and pre-calibrated intrinsics, designed to evaluate feature extractors. On this
benchmark, SANDesc achieves substantial performance gains over the existing descriptors while operating with limited computational resources.
Read the full abstract ▾
All authors Mattia D'Urso, Emanuele Santellani, Christian Sormann, Mattia Rossi, Andreas Kuhn, Friedrich Fraundorfer
Computer Vision · Cross-listed from cs.RO
LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System
Muhayy Ud Din, Waseem Akram, Ahsan B. Bakht et al. (+1 more)
Maritime port inspection plays a critical role in ensuring safety, regulatory compliance, and operational efficiency in complex maritime environments. However, existing inspection methods often rely on manual operations and conventional computer
vision techniques that lack scalability and contextual understanding. This study introduces a novel integrated engineering framework that utilizes the synergy between Large
Language Models (LLMs) and
Vision Language Models (VLMs) to enable autonomous maritime port inspection using cooperative aerial and surface robotic platforms. The proposed framework replaces traditional state-machine mission planners with LLM-driven symbolic planning and improved perception pipelines through VLM-based semantic inspection, enabling context-aware and adaptive monitoring. The LLM module translates natural
language mission instructions into executable symbolic plans with dependency graphs that encode operational constraints and ensure safe
UAV-USV coordination. Meanwhile, the VLM module performs real-time semantic inspection and compliance assessment, generating structured reports with contextual
reasoning. The framework was validated using the extended MBZIRC Maritime Simulator with realistic port infrastructure and further assessed through real-world robotic inspection trials. The lightweight on-board design ensures suitability for resource-constrained maritime platforms, advancing the development of intelligent, autonomous inspection systems. Project resources (code and
videos) can be found here: https://github.com/Muhayyuddin/llm-vlm-fusion-port-inspection
Read the full abstract ▾
All authors Muhayy Ud Din, Waseem Akram, Ahsan B. Bakht, Irfan Hussain
Computer Vision
Patient-Conditioned Adaptive Offsets for Reliable Diagnosis across Subgroups
Gelei Xu, Yuying Duan, Jun Xia et al. (+3 more)
AI models for medical diagnosis often exhibit uneven performance across patient populations due to heterogeneity in disease prevalence, imaging appearance, and clinical risk profiles. Existing algorithmic
fairness approaches typically seek to reduce such disparities by suppressing sensitive attributes. However, in medical settings these attributes often carry essential diagnostic information, and removing them can degrade accuracy and reliability, particularly in high-stakes applications. In contrast, clinical decision making explicitly incorporates patient context when interpreting diagnostic evidence, suggesting a different design direction for subgroup-aware models. In this paper, we introduce HyperAdapt, a patient-conditioned adaptation framework that improves subgroup reliability while maintaining a shared diagnostic model. Clinically relevant attributes such as age and sex are encoded into a compact
embedding and used to condition a hypernetwork-style module, which generates small residual modulation parameters for selected layers of a shared backbone. This design preserves the general medical knowledge learned by the backbone while enabling targeted adjustments that reflect patient-specific variability. To ensure efficiency and robustness, adaptations are constrained through low-rank and bottlenecked parameterizations, limiting both model complexity and computational overhead. Experiments across multiple public medical imaging
benchmarks demonstrate that the proposed approach consistently improves subgroup-level performance without sacrificing overall accuracy. On the PAD-UFES-20
dataset, our method outperforms the strongest competing baseline by 4.1% in recall and 4.4% in F1 score, with larger gains observed for underrepresented patient populations.
Read the full abstract ▾
All authors Gelei Xu, Yuying Duan, Jun Xia, Ruining Deng, Wei Jin, Yiyu Shi
Computer Vision
Prototype Learning-Based Few-Shot Segmentation for Low-Light Crack on Concrete Structures
Yulun Guo
Crack detection is critical for concrete infrastructure safety, but real-world cracks often appear in low-light environments like tunnels and bridge undersides, degrading computer
vision segmentation accuracy. Pixel-level annotation of low-light crack images is extremely time-consuming, yet most deep learning methods require large, well-illuminated
datasets. We propose a dual-branch prototype learning network integrating Retinex theory with few-shot learning for low-light crack
segmentation. Retinex-based reflectance components guide illumination-invariant global representation learning, while metric learning reduces dependence on large annotated
datasets. We introduce a cross-similarity prior mask generation module that computes high-dimensional similarities between query and support features to capture crack location and structure, and a multi-scale feature enhancement module that fuses multi-scale features with the prior mask to alleviate spatial inconsistency. Extensive experiments on multiple
benchmarks demonstrate consistent state-of-the-art performance under low-light conditions. Code: https://github.com/YulunGuo/CrackFSS.
Read the full abstract ▾
Computer Vision
GridNet-HD: A High-Resolution Multi-Modal Dataset for LiDAR-Image Fusion on Power Line Infrastructure
Antoine Carreaud, Shanci Li, Malo De Lacour et al. (+3 more)
This paper presents GridNet-HD, a multi-modal
dataset for 3D semantic
segmentation of overhead electrical infrastructures, pairing high-density
LiDAR with high-resolution oblique imagery. The
dataset comprises 7,694 images and 2.5 billion points annotated into 11 classes, with predefined splits and mIoU metrics. Unimodal (
LiDAR-only, image-only) and multi-modal fusion baselines are provided. On GridNet-HD, fusion models outperform the best unimodal baseline by +5.55 mIoU, highlighting the complementarity of geometry and appearance. As reviewed in Sec. 2, no public
dataset jointly provides high-density
LiDAR and high-resolution oblique imagery with 3D semantic labels for power-line assets.
Dataset, baselines, and codes are available: https://huggingface.co/collections/heig-vd-geo/gridnet-hd.
Read the full abstract ▾
All authors Antoine Carreaud, Shanci Li, Malo De Lacour, Digre Frinde, Jan Skaloud, Adrien Gressin
Computer Vision
Think3D: Thinking with Space for Spatial Reasoning
Zaibin Zhang, Yuhan Wu, Lianjie Jia et al. (+10 more)
While
Vision-
Language Models (VLMs) excel at 2D visual understanding, they remain constrained by 2D-centric paradigm that severely limits genuine 3D spatial
reasoning. To bridge this gap, we introduce Think3D, a novel framework that equips VLM
agents with interactive, 3D chain-of-thought
reasoning capabilities. By integrating a suite of 3D manipulation tools, Think3D transforms perception into active spatial exploration, mirroring human
geometric reasoning. Think3D consistently improves proprietary models, including GPT-4.1 and Gemini 2.5 Pro, across BLINK Multi-view, MindCube-1K, and VSI-Bench-Tiny. We further propose Think3D-RL to teach smaller open-weight models how to manipulate 3D space effectively. Using only final-answer rewards, without process supervision or handcrafted exploration trajectories, Think3D-RL enables Qwen3-VL-4B to autonomously learn effective 3D exploration strategies. After training, the model exhibits tool-use patterns similar to those of stronger proprietary models, while shifting the effect of 3D tool use on MindCube-1K from a performance drop to a substantial improvement. These results show that active exploration in 3D space provides an effective and general paradigm for improving spatial
reasoning in
multimodal agents. Code, models, and data are available at https://github.com/zhangzaibin/spagent.
Read the full abstract ▾
All authors Zaibin Zhang, Yuhan Wu, Lianjie Jia, Yifan Wang, Zhongbo Zhang, Yijiang Li, Binghao Ran, Fuxi Zhang, Zhuohan Sun, Yizhuang Peng, Zhenfei Yin, Lijun Wang, Huchuan Lu
Computer Vision
AsyncBEV: Cross-modal Flow Alignment in Asynchronous 3D Object Detection
Shiming Wang, Holger Caesar, Liangliang Nan et al. (+1 more)
In autonomous driving, multi-modal perception tasks like 3D object detection typically rely on well-synchronized sensors, both at training and inference. However, despite the use of hardware- or software-based synchronization
algorithms, perfect synchrony is rarely guaranteed: Sensors may operate at different frequencies, and real-world factors such as network latency, hardware failures, or processing bottlenecks often introduce time offsets between sensors. Such asynchrony degrades perception performance, especially for dynamic objects. To address this challenge, we propose AsyncBEV, a trainable, lightweight and generic module to improve the robustness of 3D Bird's Eye View (BEV) object detection models against sensor asynchrony. Inspired by scene flow estimation, AsyncBEV predicts a dense 2D flow field directly from asynchronous multi-modal BEV features, taking into account the known time offset between these sensor measurements. The predicted feature flow is then used to warp and spatially align the feature maps, which we show can easily be integrated into different current BEV detector architectures (e.g., BEV grid-based and
token-based). Extensive experiments demonstrate AsyncBEV improves robustness against both small and large asynchrony between
LiDAR or camera sensors in both the
token-based CMT and grid-based UniBEV, especially for dynamic objects. We significantly outperform the ego motion compensated CMT and UniBEV baselines, notably by $16.6$ % and $11.9$ % NDS on dynamic objects in the worst-case scenario of a $0.5 s$ time offset. Code is available at https://github.com/tudelft-iv/AsyncBEV.
Read the full abstract ▾
All authors Shiming Wang, Holger Caesar, Liangliang Nan, Julian F. P. Kooij
Computer Vision
Early Prediction of Type 2 Diabetes Using Multimodal data and Tabular Transformers
Sulaiman Khan, Md. Rafiul Biswas and Zubair Shah
This study introduces a novel approach for early Type 2 Diabetes Mellitus (T2DM) risk prediction using a tabular
transformer (TabTrans) architecture to analyze longitudinal patient data. By processing patients` longitudinal health records and bone-related tabular data, our model captures complex, long-range dependencies in disease progression that conventional methods often overlook. We validated our TabTrans model on a retrospective Qatar BioBank (QBB) cohort of 1,382 subjects, comprising 725 men (146 diabetic, 579 healthy) and 657 women (133 diabetic, 524 healthy). The study integrated electronic health records (EHR) with dual-energy X-ray absorptiometry (DXA) data. To address class imbalance, we employed SMOTE and SMOTE-ENN resampling techniques. The proposed model`s performance is evaluated against conventional machine learning (ML) and generative AI models, including Claude 3.5 Sonnet (Anthropic`s constitutional AI), GPT-4 (OpenAI`s generative pre-trained
transformer), and Gemini Pro (Google`s
multimodal language model). Our TabTrans model demonstrated superior predictive performance, achieving ROC AUC $\geq$ 79.7 % for T2DM prediction compared to both generative AI models and conventional ML approaches. Feature interpretation analysis identified key risk indicators, with visceral adipose tissue (VAT) mass and volume, ward bone mineral density (BMD) and bone mineral content (BMC), T and Z-scores, and L1-L4 scores emerging as the most important predictors associated with diabetes development in Qatari adults. These findings demonstrate the significant potential of TabTrans for analyzing complex tabular healthcare data, providing a powerful tool for proactive T2DM management and personalized clinical interventions in the Qatari population. Index Terms: tabular
transformers,
multimodal data, DXA data, diabetes, T2DM, feature interpretation, tabular data
Read the full abstract ▾
Computer Vision
Cross-Scale Pretraining: Enhancing Self-Supervised Learning for Low-Resolution Satellite Imagery for Semantic Segmentation
John Waithaka, Gustave Bwirayesu and Moise Busogi
Self-supervised pretraining in remote sensing is mostly done using mid-spatial resolution (MR) image
datasets due to their high availability. Given the release of high-resolution (HR)
datasets, we ask how HR
datasets can be included in self-supervised pretraining to enhance MR image representation learning and downstream
segmentation performance on MR tasks. We design a spatial affinity component that can be added to existing self-supervised learning frameworks and that uses HR imagery to learn better representations of MR imagery. We test the spatial affinity component on two self-supervised learning frameworks and show that it outperforms models pretrained on HR or MR images alone.
Read the full abstract ▾
Computer Vision
StyMam: A Mamba-Based Generator for Artistic Style Transfer
Zhou Hong, Ning Dong, Yicheng Di et al. (+8 more)
Image style transfer aims to integrate the visual patterns of a specific artistic style into a content image while preserving its content structure. Existing methods mainly rely on the generative
adversarial network (GAN) or stable
diffusion (SD). GAN-based approaches using CNNs or
Transformers struggle to jointly capture local and global dependencies, leading to artifacts and disharmonious patterns. SD-based methods reduce such issues but often fail to preserve content structures and suffer from slow inference. To address these issues, we revisit GAN and propose a mamba-based generator, termed as StyMam, to produce high-quality stylized images without introducing artifacts and disharmonious patterns. Specifically, we introduce a mamba-based generator with a residual dual-path strip scanning mechanism and a channel-reweighted spatial
attention module. The former efficiently captures local texture features, while the latter models global dependencies. Finally, extensive qualitative and quantitative experiments demonstrate that the proposed method outperforms state-of-the-art
algorithms in both quality and speed.
Read the full abstract ▾
All authors Zhou Hong, Ning Dong, Yicheng Di, Xiaolong Xu, Rongsheng Hu, Yihua Shao, Run Ling, Yun Wang, Juqin Wang, Zhanjie Zhang, Ao Ma
Computer Vision
GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation
Riccardo Catalini, Davide Di Nucci, Guido Borghi et al. (+5 more)
We introduce GazeD, a new 3D gaze estimation method that jointly provides 3D gaze and human pose from a single RGB image. Leveraging the ability of
diffusion models to deal with uncertainty, it generates multiple plausible 3D gaze and pose hypotheses based on the 2D context information extracted from the input image. Specifically, we condition the denoising process on the 2D pose, the surroundings of the subject, and the context of the scene. With GazeD we also introduce a novel way of representing the 3D gaze by positioning it as an additional body joint at a fixed distance from the eyes. The rationale is that the gaze is usually closely related to the pose, and thus it can benefit from being jointly denoised during the
diffusion process. Evaluations across three
benchmark datasets demonstrate that GazeD achieves state-of-the-art performance in 3D gaze estimation, even surpassing methods that rely on temporal information. Project details will be available at https://aimagelab.ing.unimore.it/go/gazed.
Read the full abstract ▾
All authors Riccardo Catalini, Davide Di Nucci, Guido Borghi, Davide Davoli, Lorenzo Garattoni, Gianpiero Francesca, Yuki Kawana, Roberto Vezzani
Computer Vision · Cross-listed from cs.CY
AI-generated data contamination erodes pathological variability and diagnostic reliability
Hongyu He, Shaowen Xiang, Ye Zhang et al. (+15 more)
Generative artificial intelligence (AI) is rapidly populating medical records with synthetic content, creating a feedback loop where future models are increasingly at risk of training on uncurated AI-generated data. However, the clinical consequences of this AI-generated data contamination remain unexplored. Here, we show that in the absence of mandatory human verification, this self-referential cycle drives a rapid erosion of pathological variability and diagnostic reliability. By analysing more than 800,000 synthetic data points across clinical text generation,
vision-
language reporting, and medical image synthesis, we find that models progressively converge toward generic phenotypes regardless of the model architecture. Specifically, rare but critical findings, including pneumothorax and effusions, vanish from the synthetic content generated by AI models, while demographic representations skew heavily toward middle-aged male phenotypes. Crucially, this degradation is masked by false diagnostic confidence; models continue to issue reassuring reports while failing to detect life-threatening pathology, with false reassurance rates tripling to 40%. Blinded physician evaluation confirms that this decoupling of confidence and accuracy renders AI-generated documentation clinically useless after just two generations. We systematically evaluate three mitigation strategies, finding that while synthetic volume scaling fails to prevent collapse, mixing real data with quality-aware filtering effectively preserves diversity. Ultimately, our results suggest that without policy-mandated human oversight, the deployment of generative AI threatens to degrade the very healthcare data ecosystems it relies upon.
Read the full abstract ▾
All authors Hongyu He, Shaowen Xiang, Ye Zhang, Yingtao Zhu, Jin Zhang, Hao Deng, Emily Alsentzer, Yun Liu, Qingyu Chen, Kun-Hsing Yu, Andrew Marshall, Tingting Chen, Srinivas Anumasa, Daniel Ebner, Dean Ho, Kee Yuan Ngiam, Ching-Yu Cheng, Dianbo Liu
Computer Vision
QASA: Quality-Guided K-Adaptive Slot Attention for Unsupervised Object-Centric Learning
Tianran Ouyang, Xingping Dong, Jing Zhang et al. (+3 more)
Slot
Attention, an approach that binds different objects in a scene to a set of "slots", has become a leading method in unsupervised object-centric learning. Most methods assume a fixed slot count K, and to better accommodate the dynamic nature of object cardinality, a few works have explored K-adaptive variants. However, existing K-adaptive methods still suffer from two limitations. First, they do not explicitly constrain slot-binding quality, so low-quality slots lead to ambiguous feature attribution. Second, adding a slot-count penalty to the reconstruction objective creates conflicting optimization goals between reducing the number of active slots and maintaining reconstruction fidelity. As a result, they still lag significantly behind strong K-fixed baselines. To address these challenges, we propose Quality-Guided K-Adaptive Slot
Attention (QASA). First, we decouple slot selection from reconstruction, eliminating the mutual constraints between the two objectives. Then, we propose an unsupervised Slot-Quality metric to assess per-slot quality, providing a principled signal for fine-grained slot–object binding. Based on this metric, we design a Quality-Guided Slot Selection scheme that dynamically selects a subset of high-quality slots and feeds them into our newly designed gated decoder for reconstruction during training. At inference,
token-wise competition on slot
attention yields a K-adaptive outcome. Experiments show that QASA substantially outperforms existing K-adaptive methods on both real and synthetic
datasets. Moreover, on real-world
datasets QASA surpasses K-fixed methods.
Read the full abstract ▾
All authors Tianran Ouyang, Xingping Dong, Jing Zhang, Mang Ye, Kaihao Zhang, Bo Du
Computer Vision
Membership Inference Test: Auditing Training Data in Object Classification Models
Gonzalo Mancera, Daniel DeAlcala, Aythami Morales et al. (+2 more)
In this research, we analyze the performance of Membership Inference Tests (MINT), focusing on determining whether given data were utilized during the training phase, specifically in the domain of object recognition. Within the area of object recognition, we propose and develop architectures tailored for MINT models. These architectures aim to optimize performance and efficiency in data utilization, offering a tailored solution to tackle the complexities inherent in the object recognition domain. We conducted experiments involving an object detection model, an
embedding extractor, and a MINT module. These experiments were performed in three public databases, totaling over 174K images. The proposed architecture leverages convolutional layers to capture and model the activation patterns present in the data during the training process. Through our analysis, we are able to identify given data used for testing and training, achieving precision rates ranging between 70% and 80%, contingent upon the depth of the detection module layer chosen for input to the MINT module. Additionally, our studies entail an analysis of the factors influencing the MINT Module, delving into the contributing elements behind more transparent training processes.
Read the full abstract ▾
All authors Gonzalo Mancera, Daniel DeAlcala, Aythami Morales, Ruben Tolosana, Julian Fierrez
Computer Vision
Dual-Stream Collaborative Transformer for Image Captioning
Jun Wan, Jun Liu, Zhihui lai et al. (+1 more)
Current region feature-based image captioning methods have progressed rapidly and achieved remarkable performance. However, they are still prone to generating irrelevant descriptions due to the lack of contextual information and the over-reliance on generated partial descriptions for predicting the remaining words. In this paper, we propose a Dual-Stream Collaborative
Transformer (DSCT) to address this issue by introducing the
segmentation feature. The proposed DSCT consolidates and then fuses the region and
segmentation features to guide the generation of caption sentences. It contains multiple Pattern-Specific Mutual
Attention Encoders (PSMAEs) and Dynamic Nomination Decoders (DNDs). The PSMAE effectively highlights and consolidates the private information of two representations by querying each other. The DND dynamically searches for the most relevant learning blocks to the input textual representations and exploits the homogeneous features between the consolidated region and
segmentation features to generate more accurate and descriptive caption sentences. To the best of our knowledge, this is the first study to explore how to fuse different pattern-specific features in a dynamic way to bypass their semantic inconsistencies and spatial misalignment issues for image captioning. The experimental results from popular
benchmark datasets demonstrate that our DSCT outperforms the state-of-the-art image captioning models in the literature.
Read the full abstract ▾
All authors Jun Wan, Jun Liu, Zhihui lai, Jie Zhou
Computer Vision
Supervision-by-Hallucination-and-Transfer: A Weakly-Supervised Approach for Robust and Precise Facial Landmark Detection
Jun Wan, Yuanzhi Yao, Zhihui Lai et al. (+3 more)
High-precision facial landmark detection (FLD) relies on high-resolution deep feature representations. However, low-resolution face images or the compression (via pooling or strided convolution) of originally high-resolution images hinder the learning of such features, thereby reducing FLD accuracy. Moreover, insufficient training data and imprecise annotations further degrade performance. To address these challenges, we propose a weakly-supervised framework called Supervision-by-
Hallucination-and-Transfer (SHT) for more robust and precise FLD. SHT contains two novel mutually enhanced modules: Dual
Hallucination Learning Network (DHLN) and Facial Pose Transfer Network (FPTN). By incorporating FLD and face
hallucination tasks, DHLN is able to learn high-resolution representations with low-resolution inputs for recovering both facial structures and local details and generating more effective landmark heatmaps. Then, by transforming faces from one pose to another, FPTN can further improve landmark heatmaps and faces hallucinated by DHLN for detecting more accurate landmarks. To the best of our knowledge, this is the first study to explore weakly-supervised FLD by integrating face
hallucination and facial pose transfer tasks. Experimental results of both face
hallucination and FLD demonstrate that our method surpasses state-of-the-art techniques.
Read the full abstract ▾
All authors Jun Wan, Yuanzhi Yao, Zhihui Lai, Jie Zhou, Xianxu Hou, Wenwen Min
Computer Vision
TwoHead-SwinFPN: A Unified DL Architecture for Synthetic Manipulation, Detection and Localization in Identity Documents
Chan Naseeb, Adeel Ashraf Cheema, Hassan Sami et al. (+3 more)
The proliferation of sophisticated generative AI models has significantly escalated the threat of synthetic manipulations in identity documents, particularly through face swapping and text inpainting attacks. This paper presents TwoHead-SwinFPN, a unified deep learning architecture that simultaneously performs binary classification and precise localization of manipulated regions in ID documents. Our approach integrates a Swin
Transformer backbone with Feature Pyramid Network (FPN) and UNet-style decoder, enhanced with Convolutional Block
Attention Module (CBAM) for improved feature representation. The model employs a dual-head architecture for joint optimization of detection and
segmentation tasks, utilizing uncertainty-weighted multi-task learning. Extensive experiments on the FantasyIDiap
dataset demonstrate superior performance with 84.31% accuracy, 90.78% AUC for classification, and 57.24% mean Dice score for localization. The proposed method achieves an F1-score of 88.61% for binary classification while maintaining computational efficiency suitable for real-world deployment through FastAPI implementation. Our comprehensive evaluation includes ablation studies, cross-device generalization analysis, and detailed performance assessment across 10
languages and 3 acquisition devices.
Read the full abstract ▾
All authors Chan Naseeb, Adeel Ashraf Cheema, Hassan Sami, Tayyab Afzal, Muhammad Omair, Usman Habib
Computer Vision · Cross-listed from cs.RO
Sparse ActionGen: Accelerating Diffusion Policy with Real-time Pruning
Kangye Ji, Jianbo Zhou, Yuan Meng et al. (+3 more)
Diffusion Policy has dominated action generation due to its strong capabilities for modeling multi-modal action distributions, but its multi-step denoising processes make it impractical for real-time visuomotor control. Existing caching-based acceleration methods typically rely on $\textit{static}$ schedules that fail to adapt to the $\textit{dynamics}$ of
robot-environment interactions, thereby leading to suboptimal performance. In this paper, we propose $\underline{\textbf{S}}$parse $\underline{\textbf{A}}$ction$\underline{\textbf{G}}$en ($\textbf{SAG}$) for extremely sparse action generation. To accommodate the iterative interactions, SAG customizes a rollout-adaptive prune-then-reuse mechanism that first identifies prunable computations globally and then reuses cached activations to substitute them during action
diffusion. To capture the rollout dynamics, SAG parameterizes an observation-conditioned
diffusion pruner for environment-aware adaptation and instantiates it with a highly parameter- and inference-efficient design for real-time prediction. Furthermore, SAG introduces a one-for-all reusing strategy that reuses activations across both timesteps and blocks in a zig-zag manner, minimizing the global redundancy. Extensive experiments on multiple robotic
benchmarks demonstrate that SAG achieves up to 4$\times$ generation speedup without sacrificing performance. Project Page: https://sparse-actiongen.github.io.
Read the full abstract ▾
All authors Kangye Ji, Jianbo Zhou, Yuan Meng, Ye Li, Hanyun Cui, Zhi Wang
Computer Vision
Simultaneous Detection of LSD and FMD in Cattle Using Ensemble Deep Learning
Nazibul Basar Ayon, Abdul Hasib, Md. Faishal Ahmed et al. (+4 more)
Lumpy Skin Disease (LSD) and Foot-and-Mouth Disease (FMD) are highly contagious viral diseases affecting cattle, causing significant economic losses and welfare challenges. Their visual diagnosis is complicated by significant symptom overlap with each other and with benign conditions like insect bites or chemical burns, hindering timely control measures. Leveraging a comprehensive
dataset of 10,516 expert-annotated images from 18 farms across India, Brazil, and the USA, this study presents a novel Ensemble Deep Learning framework integrating VGG16, ResNet50, and InceptionV3 with optimized weighted averaging for simultaneous LSD and FMD detection. The model achieves a state-of-the-art accuracy of 98.2%, with macro-averaged precision of 98.2%, recall of 98.1%, F1-score of 98.1%, and an AUC-ROC of 99.5%. This approach uniquely addresses the critical challenge of symptom overlap in multi-disease detection, enabling early, precise, and automated diagnosis. This tool has the potential to enhance disease management, support global agricultural sustainability, and is designed for future deployment in resource-limited settings.
Read the full abstract ▾
All authors Nazibul Basar Ayon, Abdul Hasib, Md. Faishal Ahmed, Md. Sadiqur Rahman, Kamrul Islam, T. M. Mehrab Hasan, A. S. M. Ahsanul Sarkar Akib