Computer Vision
PMCE: Probabilistic Multi-Granularity Semantics with Caption-Guided Enhancement for Few-Shot Learning
Jiaying Wu, Can Gao, Jinglu Hu et al. (+3 more)
Few-shot learning aims to identify novel categories from only a handful of labeled samples, where prototypes estimated from scarce data are often biased and generalize poorly. Semantic-based methods alleviate this by introducing coarse class-level information, but they are mostly applied on the support side, leaving query representations unchanged. In this paper, we present PMCE, a Probabilistic few-shot framework that leverages Multi-granularity semantics with Caption-guided Enhancement. PMCE constructs a nonparametric knowledge bank that stores visual statistics for each category as well as CLIP-encoded class name
embeddings of the base classes. At meta-test time, the most relevant base classes are retrieved based on the similarities of class name
embeddings for each novel category. These statistics are then aggregated into category-specific prior information and fused with the support set prototypes via a simple MAP update. Simultaneously, a frozen BLIP captioner provides label-free instance-level image descriptions, and a lightweight enhancer trained on base classes optimizes both support prototypes and query features under an inductive protocol with a consistency regularization to stabilize noisy captions. Experiments on four
benchmarks show that PMCE consistently improves over strong baselines, achieving up to 7.71% absolute gain over the strongest semantic competitor on MiniImageNet in the 1-shot setting. Our code is available at https://anonymous.4open.science/r/PMCE-275D
Read the full abstract ▾
All authors Jiaying Wu, Can Gao, Jinglu Hu, Hui Li, Xiaofeng Cao, Jingcai Guo
Computer Vision · Cross-listed from cs.RO
When Backdoors Meet Partial Observability: Attacking Real-World Reinforcement Learning
Tairan Huang, Qingqing Ye, Yulin Jin et al. (+4 more)
Backdoor attacks can cause reinforcement learning (RL) policies to behave normally under clean inputs while executing malicious behaviors when triggers are present. Existing RL
backdoor attacks are primarily studied in simulation and often assume that attackers can reliably manipulate the observations driving policy decisions. This assumption becomes fragile in real-world deployment, where RL policies commonly rely on
multimodal observations. Attackers can manipulate visual inputs through physical triggers, but auxiliary states such as
LiDAR and odometry signals remain uncontrollable and vary across trajectories. We study this overlooked challenge and propose a
diffusion-guided
backdoor attack framework (DGBA) for real-world RL. DGBA uses small printable visual patches as triggers and learns a
stochastic trigger distribution via conditional
diffusion to maintain consistent attack activation under varying uncontrollable states. We further introduce an advantage-based poisoning strategy that injects triggers only at decision-critical training states. Experiments on a physical TurtleBot3 platform show that DGBA consistently outperforms prior RL
backdoor attacks while preserving normal task performance. Demo
videos and code are available in the supplementary material.
Read the full abstract ▾
All authors Tairan Huang, Qingqing Ye, Yulin Jin, Jiawei Lian, Yaxin Xiao, Yi Wang, Haibo Hu
Computer Vision
Interp3D: Correspondence-aware Interpolation for Generative Textured 3D Morphing
Xiaolu Liu, Yicong Li, Qiyuan He et al. (+4 more)
Textured 3D morphing seeks to generate smooth and plausible transitions between two 3D assets, preserving both structural coherence and fine-grained appearance. This ability is crucial not only for advancing 3D generation research but also for practical applications in animation, editing, and digital content creation. Existing approaches either operate directly on geometry, limiting them to shape-only morphing while neglecting textures, or extend 2D interpolation strategies into 3D, which often causes semantic ambiguity, structural misalignment, and texture blurring. These challenges underscore the necessity to jointly preserve
geometric consistency, texture
alignment, and robustness throughout the transition process. To address this, we propose Interp3D, a novel training-free framework for textured 3D morphing. It harnesses generative priors and adopts a progressive
alignment principle to ensure both
geometric fidelity and texture coherence. Starting from semantically aligned interpolation in condition space, Interp3D enforces structural consistency via SLAT (Structured Latent)-guided structure interpolation, and finally transfers appearance details through fine-grained texture fusion. For comprehensive evaluations, we construct a dedicated
dataset, Interp3DData, with graded difficulty levels and assess generation results from fidelity, transition smoothness, and plausibility. Both quantitative metrics and human studies demonstrate the significant advantages of our proposed approach over previous methods. Source code is available at https://github.com/xiaolul2/Interp3D.
Read the full abstract ▾
All authors Xiaolu Liu, Yicong Li, Qiyuan He, Jiayin Zhu, Wei Ji, Angela Yao, Jianke Zhu
Computer Vision
Curriculum-Based Strategies for Efficient Cross-Domain Action Recognition
Emily Kim, Allen Wu and Jessica Hodgins
Despite significant progress in human action recognition, generalizing to diverse viewpoints remains a challenge. Most existing
datasets are captured from ground-level perspectives, and models trained on them often struggle to transfer to drastically different domains such as aerial views. This paper examines how curriculum-based training strategies can improve generalization to unseen real aerial-view data without using any real aerial data during training. We explore curriculum learning for cross-view action recognition using two out-of-domain sources: synthetic aerial-view data and real ground-view data. Our results on the evaluation on order of training (fine-tuning on synthetic aerial data vs. real ground data) shows that fine-tuning on real ground data but differ in how they transition from synthetic to real. The first uses a two-stage curriculum with direct fine-tuning, while the second applies a progressive curriculum that expands the
dataset in multiple stages before fine-tuning. We evaluate both methods on the REMAG
dataset using SlowFast (CNN-based) and MViTv2 (
Transformer-based) architectures. Results show that combining the two out-of-domain
datasets clearly outperforms training on a single domain, whether real ground-view or synthetic aerial-view. Both curriculum strategies match the top-1 accuracy of simple
dataset combination while offering efficiency gains. With the two-step fine-tuning method, SlowFast achieves up to a 37% reduction in iterations and MViTv2 up to a 30% reduction compared to simple combination. The multi-step progressive approach further reduces iterations, by up to 9% for SlowFast and 30% for MViTv2, relative to the two-step method. These findings demonstrate that curriculum-based training can maintain comparable performance (top-1 accuracy within 3% range) while improving training efficiency in cross-view action recognition.
Read the full abstract ▾
Computer Vision
Two-Stream temporal transformer for video action classification
Nattapong Kurpukdee and Adrian G. Bors
Motion representation plays an important role in
video understanding and has many applications including action recognition,
robot and autonomous guidance or others. Lately,
transformer networks, through their self-
attention mechanism capabilities, have proved their efficiency in many applications. In this study, we introduce a new two-stream
transformer video classifier, which extracts spatio-temporal information from content and optical flow representing movement information. The proposed model identifies self-
attention features across the joint optical flow and temporal frame domain and represents their relationships within the
transformer encoder mechanism. The experimental results show that our proposed methodology provides excellent classification results on three well-known
video datasets of human activities.
Read the full abstract ▾
Computer Vision
DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning
Abdurrahim Yilmaz, Ozan Erdem, Ece Gokyayla et al. (+5 more)
Vision-
language models (VLMs) are increasingly important in medical applications; however, their evaluation in dermatology remains limited by
datasets that focus primarily on image-level classification tasks such as lesion recognition. While valuable for recognition, such
datasets cannot assess the full visual understanding,
language grounding, and clinical
reasoning capabilities of
multimodal models. Visual question answering (VQA)
benchmarks are required to evaluate how models interpret dermatological images, reason over fine-grained morphology, and generate clinically meaningful descriptions. We introduce DermaBench, a clinician-annotated dermatology VQA
benchmark built on the Diverse Dermatology Images (DDI)
dataset. DermaBench comprises 656 clinical images from 570 unique patients spanning Fitzpatrick skin types I-VI. Using a hierarchical annotation schema with 22 main questions (single-choice, multi-choice, and open-ended), expert dermatologists annotated each image for diagnosis, anatomic site, lesion morphology, distribution, surface features, color, and image quality, together with open-ended narrative descriptions and summaries, yielding approximately 14.474 VQA-style annotations. DermaBench is released as a metadata-only
dataset to respect upstream licensing and is publicly available at Harvard Dataverse.
Read the full abstract ▾
All authors Abdurrahim Yilmaz, Ozan Erdem, Ece Gokyayla, Ayda Acar, Burc Bugra Dagtas, Dilara Ilhan Erdil, Gulsum Gencoglan, Burak Temelkuran
Computer Vision
VENI: Variational Encoder for Natural Illumination
Paul Walker, James A. D. Gardner, Andreea Ardelean et al. (+2 more)
Inverse rendering is an ill-posed problem, but priors such as illumination priors can help simplify it. Existing work either disregards the spherical and rotation-equivariant nature of illumination environments or does not provide a well-behaved latent space. We propose a rotation-equivariant variational autoencoder that models natural illumination on the sphere without relying on 2D projections. To preserve the SO(2)-equivariance of environment maps, we use a novel Vector
Neuron Vision Transformer (VN-ViT) as encoder and a rotation-equivariant conditional neural field as decoder. In the encoder, we reduce the equivariance from SO(3) to SO(2) using a novel SO(2)-equivariant fully connected layer, an extension of Vector
Neurons. We show that our SO(2)-equivariant fully connected layer outperforms standard Vector
Neurons when used in our SO(2)-equivariant model. Compared to previous methods, our variational autoencoder enables smoother interpolation in latent space and offers a more well-behaved latent space.
Read the full abstract ▾
All authors Paul Walker, James A. D. Gardner, Andreea Ardelean, William A. P. Smith, Bernhard Egger
Computer Vision · Cross-listed from q-bio.NC
MooneyMaker: A Python package to create ambiguous two-tone images
Lars C. Reining, Thabo Matthies, Luisa Haussner et al. (+2 more)
Mooney images are high-contrast, two-tone visual stimuli, created by thresholding photographic images. They allow researchers to separate image content from image understanding, making them valuable for studying visual perception. An ideal Mooney image for this purpose achieves a specific balance: it initially appears unrecognizable but becomes fully interpretable to the observer after seeing the original template. Researchers traditionally created these stimuli manually using subjective criteria, which is labor-intensive and can introduce inconsistencies across studies. Automated generation techniques now offer an alternative to this manual approach. Here, we present MooneyMaker, an open-source Python package that automates the generation of ambiguous Mooney images using several complementary approaches. Users can choose between various generation techniques that range from approaches based on image statistics to deep learning models. These models strategically alter edge information to increase initial ambiguity. The package lets users create two-tone images with multiple methods and directly compare the results visually. In an experiment, we validate MooneyMaker by generating Mooney images using different techniques and assess their recognizability for human observers before and after disambiguating them by presenting the template images. Our results reveal that techniques with lower initial recognizability are associated with higher post-template recognition (i.e. a larger disambiguation effect). To help
vision scientists build effective databases of Mooney stimuli, we provide practical guidelines for technique selection. By standardizing the generation process, MooneyMaker supports more consistent and reproducible visual perception research.
Read the full abstract ▾
All authors Lars C. Reining, Thabo Matthies, Luisa Haussner, Rabea Turon, Thomas S. A. Wallis
Computer Vision
Unsupervised Video Class-Incremental Learning via Deep Embedded Clustering Management
Nattapong Kurpukdee and Adrian G. Bors
Unsupervised
video class incremental learning (uVCIL) represents an important learning paradigm for learning
video information without forgetting, and without considering any data labels. Prior approaches have focused on supervised class-incremental learning, relying on using the knowledge of labels and task boundaries, which is costly, requires human annotation, or is simply not a realistic option. In this paper, we propose a simple yet effective approach to address the uVCIL. We first consider a deep feature extractor network, providing a set of representative
video features during each task without assuming any class or task information. We then progressively build a series of deep clusters from the extracted features. During the successive task learning, the model updated from the previous task is used as an initial state in order to transfer knowledge to the current learning task. We perform in-depth evaluations on three standard
video action recognition
datasets, including UCF101, HMDB51, and Something-to-Something V2, by ignoring the labels from the supervised setting. Our approach significantly outperforms other baselines on all
datasets.
Read the full abstract ▾
Computer Vision
VERIDAH: Solving Enumeration Anomaly Aware Vertebra Labeling across Imaging Sequences
Hendrik Möller, Hanna Schoen, Robert Graf et al. (+14 more)
The human spine commonly consists of seven cervical, twelve thoracic, and five lumbar vertebrae. However, enumeration anomalies may result in individuals having eleven or thirteen thoracic vertebrae and four or six lumbar vertebrae. Although the identification of enumeration anomalies has potential clinical implications for chronic back pain and operation planning, the thoracolumbar junction is often poorly assessed and rarely described in clinical reports. Additionally, even though multiple deep-learning-based vertebra labeling
algorithms exist, there is a lack of methods to automatically label enumeration anomalies. Our work closes that gap by introducing "Vertebra Identification with Anomaly Handling" (VERIDAH), a novel vertebra labeling
algorithm based on multiple classification heads combined with a weighted vertebra sequence prediction
algorithm. We show that our approach surpasses existing models on T2w TSE sagittal (98.30% vs. 94.24% of subjects with all vertebrae correctly labeled, p < 0.001) and CT imaging (99.18% vs. 77.26% of subjects with all vertebrae correctly labeled, p < 0.001) and works in arbitrary field-of-view images. VERIDAH correctly labeled the presence 2 Möller et al. of thoracic enumeration anomalies in 87.80% and 96.30% of T2w and CT images, respectively, and lumbar enumeration anomalies in 94.48% and 97.22% for T2w and CT, respectively. Our code and models are available at: https://github.com/Hendrik-code/spineps.
Read the full abstract ▾
All authors Hendrik Möller, Hanna Schoen, Robert Graf, Matan Atad, Nathan Molinier, Anjany Sekuboyina, Bettina K. Budai, Fabian Bamberg, Steffen Ringhof, Christopher Schlett, Tobias Pischon, Thoralf Niendorf, Josua A. Decker, Marc-André Weber, Bjoern Menze, Daniel Rueckert, Jan S. Kirschke
Computer Vision
Fine-Grained Zero-Shot Composed Image Retrieval with Complementary Visual-Semantic Integration
Yongcong Ye, Kai Zhang, Yanghai Zhang et al. (+3 more)
Zero-shot composed image
retrieval (ZS-CIR) is a rapidly growing area with significant practical applications, allowing users to retrieve a target image by providing a reference image and a relative caption describing the desired modifications. Existing ZS-CIR methods often struggle to capture fine-grained changes and integrate visual and semantic information effectively. They primarily rely on either transforming the
multimodal query into a single text using image-to-text models or employing large
language models for target image description generation, approaches that often fail to capture complementary visual information and complete semantic context. To address these limitations, we propose a novel Fine-Grained Zero-Shot Composed Image
Retrieval method with Complementary Visual-Semantic Integration (CVSI). Specifically, CVSI leverages three key components: (1) Visual Information Extraction, which not only extracts global image features but also uses a pre-trained mapping network to convert the image into a pseudo
token, combining it with the modification text and the objects most likely to be added. (2) Semantic Information Extraction, which involves using a pre-trained captioning model to generate multiple captions for the reference image, followed by leveraging an LLM to generate the modified captions and the objects most likely to be added. (3) Complementary Information
Retrieval, which integrates information extracted from both the query and database images to retrieve the target image, enabling the system to efficiently handle
retrieval queries in a variety of situations. Extensive experiments on three public
datasets (e.g., CIRR, CIRCO, and FashionIQ) demonstrate that CVSI significantly outperforms existing state-of-the-art methods. Our code is available at https://github.com/yyc6631/CVSI.
Read the full abstract ▾
All authors Yongcong Ye, Kai Zhang, Yanghai Zhang, Enhong Chen, Longfei Li, Jun Zhou
Computer Vision
POCI-Diff: 3D-Layout Guided Diffusion for Controllable Synthetic Surveillance Data Generation
Andrea Rigo, Luca Stornaiuolo, Weijie Wang et al. (+3 more)
Training robust visual surveillance models requires large-scale
datasets with precise spatial annotations, yet collecting real surveillance data is costly,
privacy-sensitive, and often legally constrained. Synthetic data generation offers a compelling alternative, but existing methods lack fine-grained 3D control over object placement and appearance, limiting
geometric consistency across camera viewpoints. We introduce POCI-Diff (Positioning Objects Consistently and Interactively), a framework that generates annotated synthetic scenes from explicit 3D bounding-box layouts with per-object semantic control. By integrating Blended Latent
Diffusion with depth-conditioned ControlNet, POCI-Diff synthesises complex multi-object scenes in a single forward pass, binding individual text descriptions to specific 3D locations. We further propose a warping-free editing pipeline supporting object insertion, removal, and transformation via regeneration, enabling efficient scene variation for data augmentation. Object identity across edits is preserved by conditioning on reference images via IP-Adapter, ensuring appearance consistency throughout interactive scene manipulation. Experiments show that POCI-Diff outperforms state-of-the-art 3D layout-guided generation methods in visual fidelity and layout adherence, while eliminating warping-induced
geometric artifacts.
Read the full abstract ▾
All authors Andrea Rigo, Luca Stornaiuolo, Weijie Wang, Mauro Martino, Bruno Lepri, Nicu Sebe
Computer Vision
Decoder-Free Supervoxel GNN for Accurate Brain-Tumor Localization in Multi-Modal MRI
Andrea Protani, Marc Molina Van Den Bosch, Lorenzo Giusti et al. (+6 more)
Modern
vision backbones for 3D medical imaging typically process dense voxel grids through parameter-heavy encoder-decoder structures, a design that allocates a significant portion of its parameters to spatial reconstruction rather than feature learning. Our approach introduces SVGFormer, a decoder-free pipeline built upon a content-aware grouping stage that partitions the volume into a semantic graph of supervoxels. Its hierarchical encoder learns rich node representations by combining a patch-level
Transformer with a supervoxel-level Graph
Attention Network, jointly modeling fine-grained intra-region features and broader inter-regional dependencies. This design concentrates all learnable capacity on feature encoding and provides inherent, dual-scale explainability from the patch to the region level. To validate the framework's flexibility, we trained two specialized models on the BraTS
dataset: one for node-level classification and one for tumor proportion regression. Both models achieved strong performance, with the classification model achieving a F1-score of 0.875 and the regression model a MAE of 0.028, confirming the encoder's ability to learn discriminative and localized features. Our results establish that a graph-based, encoder-only paradigm offers an accurate and inherently interpretable alternative for 3D medical image representation.
Read the full abstract ▾
All authors Andrea Protani, Marc Molina Van Den Bosch, Lorenzo Giusti, Heloisa Barbosa Da Silva, Paolo Cacace, Albert Sund Aillet, Miguel Angel Gonzalez Ballester, Friedhelm Hummel, Luigi Serio
Computer Vision · Cross-listed from cs.LG
LLMOrbit: A Circular Taxonomy of Large Language Models -From Scaling Walls to Agentic AI Systems
Badri N. Patro and Vijay S. Agneeswaran
The field of artificial intelligence has undergone a revolution from foundational
Transformer architectures to
reasoning-capable systems approaching human-level performance. We present LLMOrbit, a comprehensive circular taxonomy navigating the landscape of large
language models spanning 2019-2025. This survey examines over 50 models across 15 organizations through eight interconnected orbital dimensions, documenting architectural innovations, training methodologies, and efficiency patterns defining modern LLMs, generative AI, and agentic systems. We identify three critical crises: (1) data scarcity (9-27T
tokens depleted by 2026-2028), (2) exponential cost growth ($3M to $300M+ in 5 years), and (3) unsustainable energy consumption (22x increase), establishing the scaling wall limiting brute-force approaches. Our analysis reveals six paradigms breaking this wall: (1) test-time compute (o1, DeepSeek-R1 achieve GPT-4 performance with 10x inference compute), (2) quantization (4-8x compression), (3) distributed edge computing (10x cost reduction), (4) model merging, (5) efficient training (ORPO reduces
memory 50%), and (6) small specialized models (Phi-4 14B matches larger models). Three paradigm shifts emerge: (1) post-training gains (RLHF, GRPO, pure RL contribute substantially, DeepSeek-R1 achieving 79.8% MATH), (2) efficiency revolution (MoE routing 18x efficiency, Multi-head Latent
Attention 8x KV cache compression enables GPT-4-level performance at $<$$0.30/M
tokens), and (3) democratization (open-source Llama 3 88.6% MMLU surpasses GPT-4 86.4%). We provide insights into techniques (RLHF, PPO, DPO, GRPO, ORPO), trace
evolution from passive generation to tool-using
agents (ReAct, RAG, multi-
agent systems), and analyze post-training innovations.
Read the full abstract ▾
Computer Vision
Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model
Haoran Xu, Yanlin Liu, Zizhao Tong et al. (+8 more)
Out-of-Distribution (OOD) detection is a critical task that has garnered significant
attention. The emergence of CLIP has spurred extensive research into zero-shot OOD detection, often employing a training-free approach. Current methods leverage expert knowledge from large
language models (LLMs) to identify potential outliers. However, these approaches tend to over-rely on knowledge in the text space, neglecting the inherent challenges involved in detecting out-of-distribution samples in the image space. In this paper, we propose a novel pipeline, MM-OOD, which leverages the
multimodal reasoning capabilities of MLLMs and their ability to conduct multi-round conversations for enhanced outlier detection. Our method is designed to improve performance in both near OOD and far OOD tasks. Specifically, (1) for near OOD tasks, we directly feed ID images and corresponding text
prompts into MLLMs to identify potential outliers; and (2) for far OOD tasks, we introduce the sketch-generate-elaborate framework: first, we sketch outlier exposure using text
prompts, then generate corresponding visual OOD samples, and finally elaborate by using
multimodal prompts. Experiments demonstrate that our method achieves significant improvements on widely used
multimodal datasets such as Food-101, while also validating its scalability on ImageNet-1K.
Read the full abstract ▾
All authors Haoran Xu, Yanlin Liu, Zizhao Tong, Jiaze Li, Kexue Fu, Yuyang Zhang, Longxiang Gao, Shuaiguang Li, Xingyu Li, Yanran Xu, Changwei Wang
Computer Vision
Weather-R1: Logically Consistent Reinforcement Fine-Tuning for Multimodal Reasoning in Meteorology
Kaiyu Wu, Pucheng Han, Hualong Zhang et al. (+2 more)
While
Vision Language Models (VLMs) show advancing
reasoning capabilities, their application in meteorology is constrained by a domain gap and a
reasoning faithfulness gap. Specifically, mainstream Reinforcement Fine-Tuning (RFT) can induce Self-Contradictory
Reasoning (Self-Contra), where the model's
reasoning contradicts its final answer, which is unacceptable in such a high-stakes domain. To address these challenges, we construct WeatherQA, a novel
multimodal reasoning benchmark in meteorology. We also propose Logically Consistent Reinforcement Fine-Tuning (LoCo-RFT), which resolves Self-Contra by introducing a logical consistency reward. Furthermore, we introduce Weather-R1, the first
reasoning VLM with logical faithfulness in meteorology, to the best of our knowledge. Experiments demonstrate that Weather-R1 improves performance on WeatherQA by 9.8 percentage points over the baseline, outperforming Supervised Fine-Tuning and RFT, and even surpassing the original Qwen2.5-VL-32B. These results highlight the effectiveness of our LoCo-RFT and the superiority of Weather-R1. Our
benchmark and code are available at https://github.com/Marcowky/Weather-R1.
Read the full abstract ▾
All authors Kaiyu Wu, Pucheng Han, Hualong Zhang, Naigeng Wu, Keze Wang
Computer Vision
Jiaze Li, Haoran Xu, Wanyi Wu et al. (+9 more)
Federated learning is a paradigm of joint learning in which clients collaborate by sharing model parameters instead of data. However, in the non-iid setting, the global model experiences client drift, which can seriously affect the final performance of the model. Previous methods tend to correct the global model that has already deviated based on the loss function or gradient, overlooking the impact of the client samples. In this paper, we rethink the role of the client side and propose
Federated Balanced Learning, i.e., FBL, to prevent this issue from the beginning through sample balance on the client side. Technically, FBL allows unbalanced data on the client side to achieve sample balance through knowledge filling and knowledge sampling using edge-side generation models, under the limitation of a fixed number of data samples on clients. Furthermore, we design a Knowledge
Alignment Strategy to bridge the gap between synthetic and real data, and a Knowledge Drop Strategy to regularize our method. Meanwhile, we scale our method to real and complex scenarios, allowing different clients to adopt various methods, and extend our framework to further improve performance. Numerous experiments show that our method outperforms state-of-the-art baselines. The code is released upon acceptance.
Read the full abstract ▾
All authors Jiaze Li, Haoran Xu, Wanyi Wu, Changwei Wang, Shuaiguang Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, Youyang Qu, Longxiang Gao, Xudong Yang, Lumin Xing
Computer Vision
Generalizing Abstention for Noise-Robust Learning in Medical Image Segmentation
Wesam Moustafa, Hossam Elsafty, Helen Schneider et al. (+2 more)
Label noise is a critical problem in medical image
segmentation, often arising from the inherent difficulty of manual annotation. Models trained on noisy data are prone to overfitting, which degrades their generalization performance. While a number of methods and strategies have been proposed to mitigate noisy labels in the
segmentation domain, this area remains largely under-explored. The abstention mechanism has proven effective in classification tasks by enhancing the capabilities of Cross
Entropy, yet its potential in
segmentation remains unverified. In this paper, we address this gap by introducing a universal and modular abstention framework capable of enhancing the noise-robustness of a diverse range of loss functions. Our framework improves upon prior work with two key components: an informed regularization term to guide abstention behaviour, and a more flexible power-law-based auto-tuning
algorithm for the abstention penalty. We demonstrate the framework's versatility by systematically integrating it with three distinct loss functions to create three novel, noise-robust variants: GAC, SAC, and ADS. Experiments on the CaDIS and DSAD medical
datasets show our methods consistently and significantly outperform their non-abstaining baselines, especially under high noise levels. This work establishes that enabling models to selectively ignore corrupted samples is a powerful and generalizable strategy for building more reliable
segmentation models. Our code is publicly available at https://github.com/wemous/abstention-for-
segmentation.
Read the full abstract ▾
All authors Wesam Moustafa, Hossam Elsafty, Helen Schneider, Lorenz Sparrenberg, Rafet Sifa
Computer Vision
Correcting and Quantifying Systematic Errors in 3D Box Annotations for Autonomous Driving
Alexandre Justo Miro, Ludvig af Klinteberg, Bogdan Timus et al. (+5 more)
Accurate ground truth annotations are critical to supervised learning and evaluating the performance of autonomous vehicle systems. These vehicles are typically equipped with active sensors, such as
LiDAR, which scan the environment in predefined patterns. 3D box annotation based on data from such sensors is challenging in dynamic scenarios, where objects are observed at different timestamps, hence different positions. Without proper handling of this phenomenon, systematic errors are prone to being introduced in the box annotations. Our work is the first to discover such annotation errors in widely used, publicly available
datasets. Through our novel offline estimation method, we correct the annotations so that they follow physically feasible trajectories and achieve spatial and temporal consistency with the sensor data. For the first time, we define metrics for this problem; and we evaluate our method on the Argoverse 2, MAN TruckScenes, and our proprietary
datasets. Our approach increases the quality of box annotations by more than 17% in these
datasets. Furthermore, we quantify the annotation errors in them and find that the original annotations are misplaced by up to 2.5 m, with highly dynamic objects being the most affected. Finally, we test the impact of the errors in benchmarking and find that the impact is larger than the improvements that state-of-the-art methods typically achieve with respect to the previous state-of-the-art methods; showing that accurate annotations are essential for correct interpretation of performance. Our code is available at https://github.com/alexandre-justo-miro/annotation-correction-3D-boxes.
Read the full abstract ▾
All authors Alexandre Justo Miro, Ludvig af Klinteberg, Bogdan Timus, Aron Asefaw, Ajinkya Khoche, Thomas Gustafsson, Sina Sharif Mansouri, Masoud Daneshtalab
Computer Vision
Likelihood-Separable Diffusion Inference for Multi-Image MRI Super-Resolution
Samuel W. Remedios, Zhangxing Bian, Shuwen Wei et al. (+3 more)
Diffusion models are the current state-of-the-art for solving inverse problems in imaging. Their impressive generative capability allows them to approximate sampling from a prior distribution, which alongside a known likelihood function permits posterior sampling without retraining the model. While recent methods have made strides in advancing the accuracy of posterior sampling, the majority focuses on single-image inverse problems. However, for modalities such as
magnetic resonance imaging (MRI), it is common to acquire multiple complementary measurements, each low-resolution along a different axis. In this work, we generalize common
diffusion-based inverse single-image problem solvers for multi-image super-resolution (MISR) MRI. We show that the DPS likelihood correction allows an exactly-separable gradient decomposition across independently acquired measurements, enabling MISR without constructing a joint operator, modifying the
diffusion model, or increasing network function evaluations. We derive MISR versions of DPS, DMAP, DPPS, and
diffusion-based PnP/ADMM, and demonstrate substantial gains over SISR across $4\times/8\times/16\times$ anisotropic degradations. Our results achieve state-of-the-art super-resolution of anisotropic MRI volumes and, critically, enable reconstruction of near-isotropic anatomy from routine 2D multi-slice acquisitions, which are otherwise highly degraded in orthogonal views.
Read the full abstract ▾
All authors Samuel W. Remedios, Zhangxing Bian, Shuwen Wei, Aaron Carass, Jerry L. Prince, Blake E. Dewey
Computer Vision · Cross-listed from eess.IV
SHARE: A Fully Unsupervised Framework for Single Hyperspectral Image Restoration
Jiangwei Xie, Zhang Wen, Mike Davies et al. (+1 more)
Hyperspectral image (HSI) restoration is a fundamental challenge in computational imaging and computer
vision. It involves ill-posed inverse problems, such as inpainting and super-resolution. Although deep learning methods have transformed the field through data-driven learning, their effectiveness hinges on access to meticulously curated ground-truth
datasets. This fundamentally restricts their applicability in real-world scenarios where such data is unavailable. This paper presents SHARE (Single Hyperspectral Image Restoration with Equivariance), a fully unsupervised framework that unifies
geometric equivariance principles with low-rank spectral modelling to eliminate the need for ground truth. SHARE's core concept is to exploit the intrinsic invariance of hyperspectral structures under differentiable
geometric transformations (e.g. rotations and scaling) to derive self-supervision signals through equivariance consistency constraints. Our novel Dynamic Adaptive Spectral
Attention (DASA) module further enhances this paradigm shift by explicitly encoding the global low-rank property of HSI and adaptively refining local spectral-spatial correlations through learnable
attention mechanisms. Extensive experiments on HSI inpainting and super-resolution tasks demonstrate the effectiveness of SHARE. Our method outperforms many state-of-the-art unsupervised approaches and achieves performance comparable to that of supervised methods. We hope that our approach will shed new light on HSI restoration and broader scientific imaging scenarios. The code will be released at https://github.com/xuwayyy/SHARE.
Read the full abstract ▾
All authors Jiangwei Xie, Zhang Wen, Mike Davies, Dongdong Chen
Computer Vision
Equivariant Learning for Unsupervised Image Dehazing
Zhang Wen, Jiangwei Xie and Dongdong Chen
Image Dehazing (ID) aims to produce a clear image from an observation contaminated by haze. Current ID methods typically rely on carefully crafted priors or extensive haze-free ground truth, both of which are expensive or impractical to acquire, particularly in the context of scientific imaging. We propose a new unsupervised learning framework called Equivariant Image Dehazing (EID) that exploits the
symmetry of image signals to restore clarity to hazy observations. By enforcing haze consistency and systematic equivariance, EID can recover clear patterns directly from raw, hazy images. Additionally, we propose an
adversarial learning strategy to model unknown haze physics and facilitate EID learning. Experiments on two scientific image dehazing
benchmarks (including
cell microscopy and medical endoscopy) and on natural image dehazing have demonstrated that EID significantly outperforms state-of-the-art approaches. By unifying equivariant learning with modelling haze physics, we hope that EID will enable more versatile and effective haze removal in scientific imaging. Code and
datasets will be published.
Read the full abstract ▾
Computer Vision
Jing Zuo, Lingzhou Mu, Fan Jiang et al. (+3 more)
Achieving human-level performance in
Vision-and-
Language Navigation (VLN) requires an
embodied agent to jointly understand
multimodal instructions and visual-spatial context while
reasoning over long action sequences. Recent works, such as NavCoT and NavGPT-2, demonstrate the potential of Chain-of-Thought (CoT)
reasoning for improving interpretability and long-horizon planning. Moreover,
multimodal extensions like OctoNav-R1 and CoT-VLA further validate CoT as a promising pathway toward human-like
navigation reasoning. However, existing approaches face critical drawbacks: purely textual CoTs lack spatial grounding and easily overfit to sparse annotated
reasoning steps, while
multimodal CoTs incur severe
token inflation by generating imagined visual observations, making real-time
navigation impractical. In this work, we propose FantasyVLN, a unified implicit
reasoning framework that preserves the benefits of CoT
reasoning without explicit
token overhead. Specifically, imagined visual
tokens are encoded into a compact latent space using a pretrained Visual AutoRegressor (VAR) during CoT
reasoning training, and the model jointly learns from textual, visual, and
multimodal CoT modes under a unified multi-CoT strategy. At inference, our model performs direct instruction-to-action mapping while still enjoying
reasoning-aware representations. Extensive experiments on LH-VLN show that our approach achieves
reasoning-aware yet real-time
navigation, improving success rates and efficiency while reducing inference latency by an order of magnitude compared to explicit CoT methods.
Read the full abstract ▾
All authors Jing Zuo, Lingzhou Mu, Fan Jiang, Chengcheng Ma, Mu Xu, Yonggang Qi
Computer Vision
Harmonizing the Deep: A Unified Information Pipeline for Robust Marine Biodiversity Assessment Across Heterogeneous Domains
Marco Piccolo, Qiwei Han, Astrid van Toor et al. (+1 more)
Marine biodiversity monitoring requires scalability and reliability across complex
underwater environments to support conservation and invasive-species management. Yet existing detection solutions often exhibit a pronounced deployment gap, with performance degrading sharply when transferred to new sites. This work establishes the foundational detection layer for a multi-year invasive species monitoring initiative targeting Arctic and Atlantic marine ecosystems. We address this challenge by developing a Unified Information Pipeline that standardises heterogeneous
datasets into a comparable information flow and evaluates a fixed, deployment-relevant detector under controlled cross-domain protocols. Across multiple domains, we find that structural factors, such as scene composition, object density, and contextual redundancy, explain cross-domain performance loss more strongly than visual degradation such as turbidity, with sparse scenes inducing a characteristic "Context Collapse" failure mode. We further validate operational feasibility by benchmarking inference on low-cost edge hardware, showing that runtime optimisation enables practical sampling rates for remote monitoring. The results shift emphasis from image enhancement toward structure-aware reliability, providing a democratised tool for consistent marine ecosystem assessment.
Read the full abstract ▾
All authors Marco Piccolo, Qiwei Han, Astrid van Toor, Joachim Vanneste
Computer Vision
STEC: A Reference-Free Spatio-Temporal Entropy Coverage Metric for Evaluating Sampled Video Frames
Shih-Yao Lin
Frame sampling is a fundamental component in
video understanding and
video–
language model pipelines, yet evaluating the quality of sampled frames remains challenging. Existing evaluation metrics primarily focus on perceptual quality or reconstruction fidelity, and are not designed to assess whether a set of sampled frames adequately captures informative and representative
video content. We propose Spatio-Temporal
Entropy Coverage (STEC), a simple and non-reference metric for evaluating the effectiveness of
video frame sampling. STEC builds upon Spatio-Temporal Frame
Entropy (STFE), which measures per-frame spatial information via
entropy-based structural complexity, and evaluates sampled frames based on their temporal coverage and redundancy. By jointly modeling spatial information strength, temporal dispersion, and non-redundancy, STEC provides a principled and lightweight measure of sampling quality. Experiments on the MSR-VTT test-1k
benchmark demonstrate that STEC clearly differentiates common sampling strategies, including random, uniform, and content-aware methods. We further show that STEC reveals robustness patterns across individual
videos that are not captured by average performance alone, highlighting its practical value as a general-purpose evaluation tool for efficient
video understanding. We emphasize that STEC is not designed to predict downstream task accuracy, but to provide a task-agnostic diagnostic signal for analyzing frame sampling behavior under constrained budgets.
Read the full abstract ▾
Computer Vision
DExTeR: Weakly Semi-Supervised Object Detection with Class and Instance Experts for Medical Imaging
Adrien Meyer, Didier Mutter and Nicolas Padoy
Detecting anatomical landmarks in medical imaging is essential for diagnosis and intervention guidance. However, object detection models rely on costly bounding box annotations, limiting scalability. Weakly Semi-Supervised Object Detection (WSSOD) with point annotations proposes annotating each instance with a single point, minimizing annotation time while preserving localization signals. A Point-to-Box teacher model, trained on a small box-labeled subset, converts these point annotations into pseudo-box labels to train a student detector. Yet, medical imagery presents unique challenges, including overlapping anatomy, variable object sizes, and elusive structures, which hinder accurate bounding box inference. To overcome these challenges, we introduce DExTeR (DETR with Experts), a
transformer-based Point-to-Box regressor tailored for medical imaging. Built upon Point-DETR, DExTeR encodes single-point annotations as object queries, refining feature extraction with the proposed class-guided deformable
attention, which guides
attention sampling using point coordinates and class labels to capture class-specific characteristics. To improve discrimination in complex structures, it introduces CLICK-MoE (CLass, Instance, and Common Knowledge Mixture of Experts), decoupling class and instance representations to reduce confusion among adjacent or overlapping instances. Finally, we implement a multi-point training strategy which promotes prediction consistency across different point placements, improving robustness to annotation variability. DExTeR achieves state-of-the-art performance across three
datasets spanning different medical domains (endoscopy, chest X-rays, and endoscopic ultrasound) highlighting its potential to reduce annotation costs while maintaining high detection accuracy.
Read the full abstract ▾
Computer Vision
VTONGuard: Automatic Detection and Authentication of AI-Generated Virtual Try-On Content
Shengyi Wu, Yan Hong, Shengyao Chen et al. (+5 more)
With the rapid advancement of generative AI, virtual try-on (VTON) systems are becoming increasingly common in e-commerce and digital entertainment. However, the growing realism of AI-generated try-on content raises pressing concerns about authenticity and responsible use. To address this, we present VTONGuard, a large-scale
benchmark dataset containing over 775,000 real and synthetic try-on images. The
dataset covers diverse real-world conditions, including variations in pose, background, and garment styles, and provides both authentic and manipulated examples. Based on this
benchmark, we conduct a systematic evaluation of multiple detection paradigms under unified training and testing protocols. Our results reveal each method's strengths and weaknesses and highlight the persistent challenge of cross-paradigm generalization. To further advance detection, we design a multi-task framework that integrates auxiliary
segmentation to enhance boundary-aware feature learning, achieving the best overall performance on VTONGuard. We expect this
benchmark to enable fair comparisons, facilitate the development of more robust detection models, and promote the safe and responsible deployment of VTON technologies in practice.
Read the full abstract ▾
All authors Shengyi Wu, Yan Hong, Shengyao Chen, Zheng Wang, Xianbing Sun, Jiahui Zhan, Jun Lan, Jianfu Zhang
Computer Vision
Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learning
Hongbo Bai, Yujin Zhou, Yile Wu et al. (+5 more)
Large
Multimodal Models (LMMs) have achieved remarkable success in visual understanding, yet they struggle with knowledge-intensive queries involving long-tail entities or evolving information due to static parametric knowledge. Recent search-augmented approaches attempt to address this limitation, but existing methods rely on indiscriminate whole-image
retrieval that introduces substantial visual redundancy and noise, and lack deep iterative reflection, limiting their effectiveness on complex visual queries. To overcome these challenges, we propose Glance-or-Gaze (GoG), a fully autonomous framework that shifts from passive perception to active visual planning. GoG introduces a Selective Gaze mechanism that dynamically chooses whether to glance at global context or gaze into high-value regions, filtering irrelevant information before
retrieval. We design a dual-stage training strategy: Reflective GoG Behavior
Alignment via supervised fine-tuning instills the fundamental GoG paradigm, while Complexity-Adaptive Reinforcement Learning further enhances the model's capability to handle complex queries through iterative
reasoning. Experiments across six
benchmarks demonstrate state-of-the-art performance. Ablation studies confirm that both Selective Gaze and complexity-adaptive RL are essential for effective visual search.
Read the full abstract ▾
All authors Hongbo Bai, Yujin Zhou, Yile Wu, Chi-Min Chan, Pengcheng Wen, Kunhao Pan, Sirui Han, Yike Guo
Computer Vision
TrackletGPT: A Language-like GPT Framework for White Matter Tract Segmentation
Anoushkrit Goel, Simroop Singh, Ankita Joshi et al. (+4 more)
White Matter Tract
Segmentation is imperative for studying
brain structural connectivity, neurological disorders and neurosurgery. This task remains complex, as tracts differ among themselves, across subjects and conditions, yet have similar 3D structure across hemispheres and subjects. To address these challenges, we propose TrackletGPT, a
language-like GPT framework which reintroduces sequential information in
tokens using tracklets. TrackletGPT generalises seamlessly across
datasets, is fully automatic, and encodes granular sub-streamline segments, Tracklets, scaling and refining GPT models in Tractography
Segmentation. Based on our experiments, TrackletGPT outperforms state-of-the-art methods on average DICE, Overlap and Overreach scores on TractoInferno and HCP
datasets, even on inter-
dataset experiments.
Read the full abstract ▾
All authors Anoushkrit Goel, Simroop Singh, Ankita Joshi, Ranjeet Ranjan Jha, Chirag Ahuja, Aditya Nigam, Arnav Bhavsar
Computer Vision · Cross-listed from cs.CL
HyperWalker: Dynamic Hypergraph-Based Deep Diagnosis for Multi-Hop Clinical Modeling across EHR and X-Ray in Medical VLMs
Yuezhe Yang, Hao Wang, Yige Peng et al. (+2 more)
Automated clinical diagnosis remains a core challenge in medical AI, which usually requires models to integrate multi-modal data and reason across complex, case-specific contexts. Although recent methods have advanced medical report generation (MRG) and visual question answering (VQA) with medical
vision-
language models (VLMs), these methods, however, predominantly operate under a sample-isolated inference paradigm, as such processing cases independently without access to longitudinal electronic health records (EHRs) or structurally related patient examples. This paradigm limits
reasoning to image-derived information alone, which ignores external complementary medical evidence for potentially more accurate diagnosis. To overcome this limitation, we propose HyperWalker, a Deep Diagnosis framework that reformulates clinical
reasoning via dynamic hypergraphs and test-time training. First, we construct a dynamic hypergraph, termed iBrochure, to model the structural heterogeneity of EHR data and implicit high-order associations among
multimodal clinical information. Within this hypergraph, a reinforcement learning
agent, Walker, navigates to and identifies optimal diagnostic paths. To ensure comprehensive coverage of diverse clinical characteristics in test samples, we incorporate a linger mechanism, a multi-hop orthogonal
retrieval strategy that iteratively selects clinically complementary neighborhood cases reflecting distinct clinical attributes. Experiments on MRG with MIMIC and medical VQA on EHRXQA demonstrate that HyperWalker achieves state-of-the-art performance. Code is available at: https://github.com/Bean-Young/HyperWalker
Read the full abstract ▾
All authors Yuezhe Yang, Hao Wang, Yige Peng, Jinman Kim, Lei Bi