Computer Vision
Effect of Patch Size on Fine-Tuning Vision Transformers in Two-Dimensional and Three-Dimensional Medical Image Classification
Massoud Dehghan, Ramona Woitek and Amirreza Mahbod
Vision Transformers (ViTs) and their variants have become state-of-the-art in many computer
vision tasks and are widely used as backbones in large-scale
vision and
vision-
language foundation models. While substantial research has focused on architectural improvements, the impact of patch size, a crucial initial design choice in ViTs, remains underexplored, particularly in medical domains where both two-dimensional (2D) and three-dimensional (3D) imaging modalities exist. In this study, using 12 medical imaging
datasets from various imaging modalities (including seven 2D and five 3D
datasets), we conduct a thorough evaluation of how different patch sizes affect ViT classification performance. Using a single graphical processing unit (GPU) and a range of patch sizes (1, 2, 4, 7, 14, 28), we fine-tune ViT models and observe consistent improvements in classification performance with smaller patch sizes (1, 2, and 4), which achieve the best results across nearly all
datasets. More specifically, our results indicate improvements in balanced accuracy of up to 12.78% for 2D
datasets (patch size 2 vs. 28) and up to 23.78% for 3D
datasets (patch size 1 vs. 14), at the cost of increased computational expense. Moreover, by applying a straightforward ensemble strategy that fuses the predictions of the models trained with patch sizes 1, 2, and 4, we demonstrate a further boost in performance in most cases, especially for the 2D
datasets. Our implementation is publicly available on GitHub: https://github.com/HealMaDe/MedViT
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
OVerSeeC: Open-Vocabulary Costmap Generation from Satellite Images and Natural Language
Rwik Rana, Jesse Quattrociocchi, Dongmyeong Lee et al. (+5 more)
Aerial imagery provides essential global context for autonomous
navigation, enabling route planning at scales inaccessible to onboard sensing. We address the problem of generating global costmaps for long-range planning directly from satellite imagery when entities and mission-specific traversal rules are expressed in natural
language at test time. This setting is challenging since mission requirements vary, terrain entities may be unknown at deployment, and user
prompts often encode compositional traversal logic. Existing approaches relying on fixed ontologies and static cost mappings cannot accommodate such flexibility. While foundation models excel at
language interpretation and open-vocabulary perception, no single model can simultaneously parse nuanced mission directives, locate arbitrary entities in large-scale imagery, and synthesize them into an executable cost function for planners. We therefore propose OVerSeeC, a zero-shot modular framework that decomposes the problem into Interpret-Locate-Synthesize: (i) an LLM extracts entities and ranked preferences, (ii) an open-vocabulary
segmentation pipeline identifies these entities from high-resolution imagery, and (iii) the LLM uses the user's natural
language preferences and masks to synthesize executable costmap code. Empirically, OVerSeeC handles novel entities, respects ranked and compositional preferences, and produces routes consistent with human-drawn trajectories across diverse regions, demonstrating robustness to distribution shifts. This shows that modular composition of foundation models enables open-vocabulary, preference-aligned costmap generation for scalable, mission-adaptive global planning.
Read the full abstract ▾
All authors Rwik Rana, Jesse Quattrociocchi, Dongmyeong Lee, Christian Ellis, Amanda Adkins, Adam Uccello, Garrett Warnell, Joydeep Biswas
Computer Vision · Cross-listed from eess.IV
DM4CT: Benchmarking Diffusion Models for Computed Tomography Reconstruction
Jiayang Shi, Daniel M. Pelt and K. Joost Batenburg
Diffusion models have recently emerged as powerful priors for solving inverse problems. While computed
tomography (CT) is theoretically a linear inverse problem, it poses many practical challenges. These include correlated noise, artifact structures, reliance on system geometry, and misaligned value ranges, which make the direct application of
diffusion models more difficult than in domains like natural image generation. To systematically evaluate how
diffusion models perform in this context and compare them with established reconstruction methods, we introduce DM4CT, a comprehensive
benchmark for CT reconstruction. DM4CT includes
datasets from both medical and industrial domains with sparse-view and noisy configurations. To explore the challenges of deploying
diffusion models in practice, we additionally acquire a high-resolution CT
dataset at a high-energy synchrotron facility and evaluate all methods under real experimental conditions. We
benchmark ten recent
diffusion-based methods alongside seven strong baselines, including model-based, unsupervised, and supervised approaches. Our analysis provides detailed insights into the behavior, strengths, and limitations of
diffusion models for CT reconstruction. The real-world
dataset is publicly available at zenodo.org/records/15420527, and the codebase is open-sourced at github.com/DM4CT/DM4CT.
Read the full abstract ▾
Computer Vision
BloomNet: Exploring Single vs. Multiple Object Annotation for Flower Recognition Using YOLO Variants
Safwat Nusrat and Prithwiraj Bhattacharjee
Precise localization and recognition of flowers are crucial for advancing automated agriculture, particularly in plant phenotyping, crop estimation, and yield monitoring. This paper
benchmarks several YOLO architectures such as YOLOv5s, YOLOv8n/s/m, and YOLOv12n for flower object detection under two annotation regimes: single-image single-bounding box (SISBB) and single-image multiple-bounding box (SIMBB). The FloralSix
dataset, comprising 2,816 high-resolution photos of six different flower species, is also introduced. It is annotated for both dense (clustered) and sparse (isolated) scenarios. The models were evaluated using Precision, Recall, and Mean Average Precision (mAP) at IoU thresholds of 0.5 (mAP@0.5) and 0.5-0.95 (mAP@0.5:0.95). In SISBB, YOLOv8m (SGD) achieved the best results with Precision 0.956, Recall 0.951, mAP@0.5 0.978, and mAP@0.5:0.95 0.865, illustrating strong accuracy in detecting isolated flowers. With mAP@0.5 0.934 and mAP@0.5:0.95 0.752, YOLOv12n (SGD) outperformed the more complicated SIMBB scenario, proving robustness in dense, multi-object detection. Results show how annotation density, IoU thresholds, and model size interact: recall-optimized models perform better in crowded environments, whereas precision-oriented models perform best in sparse scenarios. In both cases, the
Stochastic Gradient Descent (SGD) optimizer consistently performed better than alternatives. These density-sensitive sensors are helpful for non-destructive crop analysis, growth tracking, robotic pollination, and stress evaluation.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry
Guanghui Min, Tianhao Huang, Ke Wan et al. (+1 more)
Targeted data selection has emerged as a crucial paradigm for efficient instruction tuning, aiming to identify a small yet influential subset of training examples for a specific target task. In practice, influence is often measured through the effect of an example on parameter updates. To make selection scalable, many approaches leverage optimizer statistics (e.g., Adam states) as an axis-aligned surrogate for update geometry (i.e., diagonal precondition), implicitly treating parameters as coordinate-wise independent. We show that this assumption breaks down in parameter-efficient fine-tuning (PEFT) methods such as LoRA. In this setting, the induced optimization geometry exhibits strong cross-parameter coupling with non-trivial off-diagonal interactions, while the task-relevant update directions are confined to a low-dimensional subspace. Motivated by this mismatch, we propose GIST (Gradient Isometric Subspace Transformation), a simple yet principled alternative that replaces axis-aligned scaling with robust subspace
alignment. GIST recovers a task-specific subspace from validation gradients via singular value decomposition (SVD), projects training gradients into this coupled subspace, and scores examples by their
alignment with target directions. Extensive experiments have demonstrated that GIST matches or outperforms the state-of-the-art baseline with only 0.29% of the storage and 25% of the computational time under the same selection budget.
Read the full abstract ▾
All authors Guanghui Min, Tianhao Huang, Ke Wan, Chen Chen
Computer Vision · Cross-listed from eess.IV
4D-UNet improves clutter rejection in human transcranial contrast enhanced ultrasound
Tristan Beruard, Armand Delbos, Arthur Chavignon et al. (+2 more)
Transcranial ultrasound imaging is limited by high skull absorption, limiting vascular imaging to only the largest vessels. Traditional clutter filters struggle with low signal-to-noise ratio (SNR) ultrasound
datasets, where blood and tissue signals cannot be easily separated, even when the echogenicity of the blood is improved with contrast
agents. Here, we present a novel 4D U-Net approach for clutter filtering in transcranial 3D Contrast Enhanced Ultrasound (CEUS) exploiting spatial and temporal information via a 4D-UNet implementation to enhance microbubble detection in transcranial data acquired in human adults. Our results show that the 4D-UNet improves temporal clutter filters. By integrating deep learning into CEUS, this study advances neurovascular imaging, offering improved clutter rejection and visualization. The findings underscore the potential of AI-driven approaches to enhance ultrasound-based medical imaging, paving the way for more accurate diagnostics and broader clinical applications.
Read the full abstract ▾
All authors Tristan Beruard, Armand Delbos, Arthur Chavignon, Maxence Reberol, Vincent Hingot
Computer Vision
Rodent-Bench
Thomas Heap, Laurence Aitchison, Emma Cahill et al. (+1 more)
We present Rodent-Bench, a novel
benchmark designed to evaluate the ability of
Multimodal Large
Language Models (MLLMs) to annotate rodent behaviour footage. We evaluate state-of-the-art MLLMs, including Gemini-2.5-Pro, Gemini-2.5-Flash and Qwen-VL-Max, using this
benchmark and find that none of these models perform strongly enough to be used as an assistant for this task. Our
benchmark encompasses diverse
datasets spanning multiple behavioral paradigms including social interactions, grooming, scratching, and freezing behaviors, with
videos ranging from 10 minutes to 35 minutes in length. We provide two
benchmark versions to accommodate varying model capabilities and establish standardized evaluation metrics including second-wise accuracy, macro F1, mean average precision, mutual information, and Matthew's correlation coefficient. While some models show modest performance on certain
datasets (notably grooming detection), overall results reveal significant challenges in temporal
segmentation, handling extended
video sequences, and distinguishing subtle behavioral states. Our analysis identifies key limitations in current MLLMs for scientific
video annotation and provides insights for future model development. Rodent-Bench serves as a foundation for tracking progress toward reliable automated behavioral annotation in neuroscience research.
Read the full abstract ▾
All authors Thomas Heap, Laurence Aitchison, Emma Cahill, Adriana Casado Rodriguez
Computer Vision · Cross-listed from eess.IV
Triggering hallucinations in model-based MRI reconstruction via adversarial perturbations
Suna Buğday, Yvan Saeys and Jonathan Peck
Generative models are increasingly used to improve the quality of medical imaging, such as reconstruction of
magnetic resonance images and computed
tomography. However, it is well-known that such models are susceptible to
hallucinations: they may insert features into the reconstructed image which are not actually present in the original image. In a medical setting, such
hallucinations may endanger patient health as they can lead to incorrect diagnoses. In this work, we aim to quantify the extent to which state-of-the-art generative models suffer from
hallucinations in the context of
magnetic resonance image reconstruction. Specifically, we craft
adversarial perturbations resembling random noise for the unprocessed input images which induce
hallucinations when reconstructed using a generative model. We perform this evaluation on the
brain and knee images from the fastMRI data set using UNet and end-to-end VarNet architectures to reconstruct the images. Our results show that these models are highly susceptible to small perturbations and can be easily coaxed into producing
hallucinations. This fragility may partially explain why
hallucinations occur in the first place and suggests that a carefully constructed
adversarial training routine may reduce their prevalence. Moreover, these
hallucinations cannot be reliably detected using traditional image quality metrics. Novel approaches will therefore need to be developed to detect when
hallucinations have occurred.
Read the full abstract ▾
Computer Vision
Morphological Addressing of Identity Basins in Text-to-Image Diffusion Models
Andrew Fraser
We demonstrate that morphological pressure creates navigable gradients at multiple levels of the text-to-image generative pipeline. In Study~1, identity basins in Stable
Diffusion 1.5 can be navigated using morphological descriptors – constituent features like platinum blonde,” beauty mark,” and 1950s glamour” – without the target's name or photographs. A self-
distillation loop (generating synthetic images from descriptor
prompts, then training a LoRA on those outputs) achieves consistent convergence toward a specific identity as measured by ArcFace similarity. The trained LoRA creates a local coordinate system shaping not only the target identity but also its inverse: maximal away-conditioning produces eldritch” structural breakdown in base SD1.5, while the LoRA-equipped model produces “uncanny valley” outputs – coherent but precisely wrong. In Study~2, we extend this to
prompt-level morphology. Drawing on phonestheme theory, we generate 200 novel nonsense words from English sound-symbolic clusters (e.g., cr-, sn-, -oid, -ax) and find that phonestheme-bearing candidates produce significantly more visually coherent outputs than random controls (mean Purity@1 = 0.371 vs.\ 0.209, p<0.00001p < 0.00001 p<0.00001, Cohen's d=0.55d = 0.55 d=0.55). Three candidates – snudgeoid, crashax, and broomix – achieve perfect visual consistency (Purity@1 = 1.0) with zero training data contamination, each generating a distinct, coherent visual identity from phonesthetic structure alone. Together, these studies establish that morphological structure – whether in feature descriptors or
prompt-level phonological form – creates systematic navigational gradients through
diffusion model latent spaces. We document phase transitions in identity basins, CFG-invariant identity stability, and novel visual concepts emerging from sub-lexical sound patterns.
Read the full abstract ▾
Computer Vision
VLANeXt Family: A Systematic Study of VLA Models from Core Recipes to Emerging Paradigms
Xiao-Ming Wu, Kang Liao, Yihang Luo et al. (+6 more)
Following the rise of large foundation models,
Vision-
Language-Action models (VLAs) emerged, leveraging strong visual and
language understanding from
Vision-
Language Models for general-purpose policy learning. Yet, the current VLA landscape remains fragmented and exploratory. Although many groups have proposed their own VLA models, inconsistencies in training protocols and evaluation settings make it difficult to identify which design choices truly matter. To bring structure to this evolving space, we reexamine the VLA design space under a unified framework and evaluation setup. Starting from a simple VLA baseline similar to RT-2, which is the origin of VLA, we systematically dissect design choices along three dimensions: foundational components, perception essentials, and action modeling perspectives. From this study, we distill 12 key findings that together form a practical recipe for building strong VLA models. The outcome of this exploration is a simple yet effective model, VLANeXt. It outperforms the state-of-the-art methods on the LIBERO and LIBERO-plus
benchmarks and demonstrates strong performance in real-world experiments. Beyond identifying the core recipe, we further ask how far these design principles extend to the emerging paradigms in VLAs. We thus expand VLANeXt along several emerging directions, including model scaling, latent-action pretraining, latent predictive representation learning, and world action modeling. These studies give rise to the VLANeXt family, spanning compact and scaled VLA variants, latent-action models, JEPA-style predictive models, and World Action Models. Our results show that the core recipe provides a strong foundation across different model scales and emerging paradigms.
Read the full abstract ▾
All authors Xiao-Ming Wu, Kang Liao, Yihang Luo, Bin Fan, Jian-Jian Jiang, Runze Yang, Zhonghua Wu, Wei-Shi Zheng, Chen Change Loy
Computer Vision
Image-Based Classification of Olive Varieties Native to Turkiye Using Multiple Deep Learning Architectures: Analysis of Performance, Complexity, and Generalization
Hatice Karatas and Irfan Atabas
This study compares multiple deep learning architectures for the automated, image-based classification of five locally cultivated black table olive varieties in Turkey: Gemlik, Ayvalik, Uslu, Erkence, and Celebi. Using a
dataset of 2500 images, ten architectures - MobileNetV2, EfficientNetB0, EfficientNetV2-S, ResNet50, ResNet101, DenseNet121, InceptionV3, ConvNeXt-Tiny, ViT-B16, and Swin-T - were trained using transfer learning. Model performance was evaluated using accuracy, precision, recall, F1-score, Matthews Correlation Coefficient (MCC), Cohen's Kappa, ROC-AUC, number of parameters, FLOPs, inference time, and generalization gap. EfficientNetV2-S achieved the highest classification accuracy (95.8%), while EfficientNetB0 provided the best trade-off between accuracy and computational complexity. Overall, the results indicate that under limited data conditions, parametric efficiency plays a more critical role than model depth alone.
Read the full abstract ▾
Computer Vision
JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
Zhan Liu, Changli Tang, Yuxin Wang et al. (+7 more)
Current audio-visual large
language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB
video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial
reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and
reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a
benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and
reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints, and
datasets are available at https://github.com/liuzhan22/JAEGER.
Read the full abstract ▾
All authors Zhan Liu, Changli Tang, Yuxin Wang, Zhiyuan Zhu, Youjun Chen, Yiwen Shao, Tianzi Wang, Lei Ke, Zengrui Jin, Chao Zhang
Computer Vision
Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexity
Vasile Marian, Yong-Bin Kang and Alexander Buddery
Synthetic images are increasingly used to augment object-detection training sets, but reliably evaluating a synthetic
dataset before training remains difficult: standard global generative metrics (e.g., FID) often do not predict downstream detection mAP. We present a controlled evaluation of synthetic augmentation for YOLOv11 across three single-class detection regimes – Traffic Signs (sparse/near-saturated), Cityscapes Pedestrian (dense/occlusion-heavy), and COCO PottedPlant (multi-instance/high-variability). We
benchmark six GAN-,
diffusion-, and hybrid-based generators over augmentation ratios from 10% to 150% of the real training split, and train YOLOv11 both from scratch and with COCO-pretrained initialization, evaluating on held-out real test splits (mAP@0.50:0.95). For each
dataset-generator-augmentation configuration, we compute pre-training
dataset metrics under a matched-size bootstrap protocol, including (i) global feature-space metrics in both Inception-v3 and DINOv2
embeddings and (ii) object-centric distribution distances over bounding-box statistics. Synthetic augmentation yields substantial gains in the more challenging regimes (up to +7.6% and +30.6% relative mAP in Pedestrian and PottedPlant, respectively) but is marginal in Traffic Signs and under pretrained fine-tuning. To separate metric signal from augmentation quantity, we report both raw and augmentation-controlled (residualized) correlations with multiple-testing correction, showing that metric-performance
alignment is strongly regime-dependent and that many apparent raw associations weaken after controlling for augmentation level.
Read the full abstract ▾
Computer Vision
Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory
Vatsal Agarwal, Saksham Suri, Matthew Gwilliam et al. (+2 more)
Streaming
video understanding requires models to robustly encode, store, and retrieve information from a continuous
video stream to support accurate
video question answering (VQA). Existing state-of-the-art approaches rely on key-value caching to accumulate frame-level information over time, but use a limited number of
tokens per frame, leading to the loss of fine-grained visual details. In this work, we propose scaling the
token budget to enable more granular spatiotemporal understanding and
reasoning. First, we find that current methods are ill-equipped to handle dense streams: their feature encoding causes query-frame similarity scores to increase over time, biasing
retrieval toward later frames. To address this, we introduce an adaptive selection strategy that reduces
token redundancy while preserving local spatiotemporal information. We further propose a training-free
retrieval mixture-of-experts that leverages external models to better identify relevant frames. Our method, MemStream, achieves +8.0% on CG-Bench, +8.5% on LVBench, and +2.4% on VideoMME (Long) over ReKV with Qwen2.5-VL-7B.
Read the full abstract ▾
All authors Vatsal Agarwal, Saksham Suri, Matthew Gwilliam, Pulkit Kumar, Abhinav Shrivastava
Computer Vision
SARAH: Spatially Aware Real-time Agentic Humans
Evonne Ng, Siwei Zhang, Zhang Chen et al. (+2 more)
As
embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond
speech-aligned gestures:
agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with
speech while orienting the
agent according to the user. Our architecture combines a causal
transformer-based VAE with interleaved latent
tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial
alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D
dataset, our method achieves state-of-the-art motion quality at over 300 FPS – 3x faster than non-causal baselines – while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational
agents to real-time deployment. Please see https://evonneng.github.io/sarah/ for details.
Read the full abstract ▾
All authors Evonne Ng, Siwei Zhang, Zhang Chen, Michael Zollhoefer, Alexander Richard
Computer Vision · Cross-listed from cs.LG
The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning
Mojtaba Sahraee-Ardakan, Mauricio Delbracio and Peyman Milanfar
Autonomous (noise-agnostic) generative models, such as
Equilibrium Matching and blind
diffusion, challenge the standard paradigm by learning a single, time-invariant vector field that operates without explicit noise-level conditioning. While recent work suggests that high-dimensional concentration allows these models to implicitly estimate noise levels from corrupted observations, a fundamental paradox remains: what is the underlying landscape being optimized when the noise level is treated as a random variable, and how can a bounded, noise-agnostic network remain stable near the data
manifold where gradients typically diverge? We resolve this paradox by formalizing Marginal Energy, $E_{\text{marg}}(\mathbf{u}) = -\log p(\mathbf{u})$, where $p(\mathbf{u}) = \int p(\mathbf{u}|t)p(t)dt$ is the marginal density of the noisy data integrated over a prior distribution of unknown noise levels. We prove that generation using autonomous models is not merely blind denoising, but a specific form of Riemannian gradient flow on this Marginal Energy. Through a novel relative energy decomposition, we demonstrate that while the raw Marginal Energy landscape possesses a $1/t^p$ singularity normal to the data
manifold, the learned time-invariant field implicitly incorporates a local conformal metric that perfectly counteracts the
geometric singularity, converting an infinitely deep potential well into a stable attractor. We also establish the structural stability conditions for sampling with autonomous models. We identify a “Jensen Gap” in noise-prediction parameterizations that acts as a high-gain amplifier for estimation errors, explaining the catastrophic failure observed in deterministic blind models. Conversely, we prove that velocity-based parameterizations are inherently stable because they satisfy a bounded-gain condition that absorbs posterior uncertainty into a smooth
geometric drift.
Read the full abstract ▾
Computer Vision · Cross-listed from astro-ph.GA
Spatio-Spectroscopic Representation Learning using Unsupervised Convolutional Long-Short Term Memory Networks
Kameswara Bharadwaj Mantha, Lucy Fortson, Ramanakumar Sankar et al. (+8 more)
Integral Field Spectroscopy (IFS) surveys offer a unique new landscape in which to learn in both spatial and spectroscopic dimensions and could help uncover previously unknown insights into
galaxy evolution. In this work, we demonstrate a new unsupervised deep learning framework using Convolutional Long-Short Term
Memory Network Autoencoders to encode generalized feature representations across both spatial and spectroscopic dimensions spanning $19$ optical emission lines (3800A $< \lambda <$ 8000A) among a sample of $\sim 9000$
galaxies from the MaNGA IFS survey. As a demonstrative exercise, we assess our model on a sample of $290$ Active
Galactic Nuclei (AGN) and highlight scientifically interesting characteristics of some highly anomalous AGN.
Read the full abstract ▾
All authors Kameswara Bharadwaj Mantha, Lucy Fortson, Ramanakumar Sankar, Claudia Scarlata, Chris Lintott, Sandor Kruk, Mike Walmsley, Hugh Dickinson, Karen Masters, Brooke Simmons, Rebecca Smethurst
Computer Vision
CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigation
Xia Su, Ruiqi Chen, Benlin Liu et al. (+4 more)
Vision-
Language Models (VLMs) have shown remarkable progress in
Vision-
Language Navigation (VLN), offering new possibilities for
navigation decision-making that could benefit both robotic platforms and human users. However, real-world
navigation is inherently conditioned by the
agent's mobility constraints. For example, a sweeping
robot cannot traverse stairs, while a quadruped can. We introduce Capability-Conditioned
Navigation (CapNav), a
benchmark designed to evaluate how well VLMs can navigate complex indoor spaces given an
agent's specific physical and operational capabilities. CapNav defines five representative human and
robot agents, each described with physical dimensions, mobility capabilities, and environmental interaction abilities. CapNav provides 45 real-world indoor scenes, 473
navigation tasks, and 2365 QA pairs to test if VLMs can traverse indoor environments based on
agent capabilities. We evaluate 13 modern VLMs and find that current VLM's
navigation performance drops sharply as mobility constraints tighten, and that even state-of-the-art models struggle with obstacle types that require
reasoning on spatial dimensions. We conclude by discussing the implications for capability-aware
navigation and the opportunities for advancing
embodied spatial
reasoning in future VLMs. The
benchmark is available at https://github.com/makeabilitylab/CapNav
Read the full abstract ▾
All authors Xia Su, Ruiqi Chen, Benlin Liu, Jingwei Ma, Zonglin Di, Ranjay Krishna, Jon Froehlich
Computer Vision
Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control
Linxi Xie, Lisong C. Sun, Ashley Neall et al. (+3 more)
Extended reality (XR) demands generative models that respond to users' tracked real-world motion, yet current
video world models accept only coarse control signals such as text or keyboard input, limiting their utility for
embodied interaction. We introduce a human-centric
video world model that is conditioned on both tracked head pose and joint-level hand poses. For this purpose, we evaluate existing
diffusion transformer conditioning strategies and propose an effective mechanism for 3D head and hand control, enabling dexterous hand–object interactions. We train a bidirectional
video diffusion model teacher using this strategy and distill it into a causal, interactive system that generates
egocentric virtual environments. We evaluate this generated reality system with human subjects and demonstrate improved task performance as well as a significantly higher level of perceived amount of control over the performed actions compared with relevant baselines.
Read the full abstract ▾
All authors Linxi Xie, Lisong C. Sun, Ashley Neall, Tong Wu, Shengqu Cai, Gordon Wetzstein
Computer Vision
Latent Equivariant Operators for Robust Object Recognition: Promises and Challenges
Minh Dinh and Stéphane Deny
Despite the successes of deep learning in computer
vision, difficulties persist in recognizing objects that have undergone group-symmetric transformations rarely seen during training$\unicode{x2013}$for example objects seen in unusual poses, scales, positions, or combinations thereof. Equivariant neural networks are a solution to the problem of generalizing across symmetric transformations, but require knowledge of transformations a priori. An alternative family of architectures proposes to learn equivariant operators in a latent space, from examples of symmetric transformations. Here, using simple
datasets of rotated and translated noisy MNIST, we illustrate how such architectures can successfully be harnessed for out-of-distribution classification, thus overcoming the limitations of both traditional and equivariant networks. While conceptually enticing, we discuss challenges ahead on the path of scaling these architectures to more complex
datasets. Our code is available at https://github.com/
BRAIN-Aalto/equivariant_operator.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Exploiting Completeness Perception with Diffusion Transformer for Unified 3D MRI Synthesis
Junkai Liu, Nay Aung, Theodoros N. Arvanitis et al. (+3 more)
Missing data problems, such as missing modalities in multi-modal
brain MRI and missing slices in cardiac MRI, pose significant challenges in clinical practice. Existing methods rely on external guidance to supply detailed missing-state information for instructing generative models to synthesize missing MRIs. However, manual indicators are not always available or reliable in real-world scenarios due to the unpredictable nature of clinical environments. Moreover, these explicit masks are not informative enough to provide guidance for improving semantic consistency. In this work, we argue that generative models should infer and recognize missing states in a self-perceptive manner, enabling them to better capture subtle anatomical and pathological variations. Towards this goal, we propose CoPeDiT, a shared completeness-perception framework for 3D MRI synthesis, following a common conditioning strategy with task-specific instantiations for different missing-data scenarios. Specifically, we incorporate dedicated pretext tasks into our tokenizer, CoPeVAE, empowering it to learn completeness-aware discriminative
prompt tokens, and design MDiT3D, a specialized
diffusion transformer architecture for 3D MRI synthesis that effectively uses the completeness-aware
prompt tokens as guidance to enhance semantic consistency in 3D space. Comprehensive evaluations on three large-scale MRI
datasets demonstrate that CoPeDiT consistently improves upon state-of-the-art methods across diverse missing patterns, yielding high-fidelity and structurally consistent MRI synthesis. Our code is available at https://github.com/JK-Liu7/CoPeDiT.
Read the full abstract ▾
All authors Junkai Liu, Nay Aung, Theodoros N. Arvanitis, Joao A. C. Lima, Steffen E. Petersen, Le Zhang
Computer Vision
Self-Aware Object Detection via Degradation Manifolds
Stefan Becker, Simon Weiss, Wolfgang Hübner et al. (+1 more)
Object detectors achieve strong performance under nominal imaging conditions but can fail silently when exposed to blur, noise, compression, adverse weather, or resolution changes. In safety-critical settings, it is therefore insufficient to produce predictions without assessing whether the input remains within the detector's nominal operating regime, which we refer to as self-aware object detection. We introduce a degradation-aware framework based on degradation
manifolds, which structure a detector's feature space according to image degradation rather than semantic content. Our method augments a standard detection backbone with a lightweight
embedding head trained via multi-layer contrastive learning, where images sharing the same degradation composition are pulled together while differing configurations are separated. Unlike prior contrastive IQA approaches, our method performs detector-aligned input-fidelity monitoring: it uses task-backbone features and a nominal prototype to detect degradation-induced shifts, rather than estimating perceptual image quality in a standalone representation. To anchor the representation, we estimate a pristine prototype from clean training
embeddings, defining a nominal operating point in feature space. Self-awareness is then derived as
geometric deviation from this reference, providing an intrinsic image-level signal of degradation-induced shift that is independent of detection confidence. Experiments on synthetic corruptions, cross-
dataset zero-shot transfer, and natural weather-induced shifts show strong pristine-degraded separability, consistent behavior across detector architectures, and robust generalization across
datasets with substantial content variation. These results suggest that degradation-aware detector representations provide a reliable signal for monitoring input-fidelity shifts.
Read the full abstract ▾
All authors Stefan Becker, Simon Weiss, Wolfgang Hübner, Michael Arens
Computer Vision · Cross-listed from quant-ph
Quantum-enhanced satellite image classification
Qi Zhang, Anton Simen, Carlos Flores-Garrigós et al. (+7 more)
We demonstrate the application of a
quantum feature extraction method to enhance multi-class image classification for space applications. By harnessing the dynamics of many-body
spin Hamiltonians, the method generates expressive
quantum features that, when combined with classical processing, lead to
quantum-enhanced classification accuracy. Using a strong and well-established ResNet50 baseline, we achieved a maximum classical accuracy of 83%, which can be improved to 84% with a transfer learning approach. In contrast, applying our
quantum-classical method the performance is increased to 87% accuracy, demonstrating a clear and reproducible improvement over robust classical approaches. Implemented on several of IBM's
quantum processors, our hybrid
quantum-classical approach delivers consistent gains of 2-3% in absolute accuracy. These results highlight the practical potential of current and near-term
quantum processors in high-stakes, data-driven domains such as satellite imaging and remote sensing, while suggesting broader applicability in real-world machine learning tasks.
Read the full abstract ▾
All authors Qi Zhang, Anton Simen, Carlos Flores-Garrigós, Gabriel Alvarado Barrios, Paolo A. Erdman, Enrique Solano, Aaron C. Kemp, Vincent Beltrani, Vedangi Pathak, Hamed Mohammadbagherpoor
Computer Vision
G-LoG Bi-filtration for Medical Image Classification
Qingsong Wang, Jiaxing He, Bingzhe Hou et al. (+3 more)
Building practical filtrations on objects to detect topological and
geometric features is an important task in the field of Topological Data Analysis (TDA). In this paper, leveraging the ability of the
Laplacian of Gaussian operator to enhance the boundaries of medical images, we define the G-LoG (Gaussian-
Laplacian of Gaussian) bi-filtration to generate the features more suitable for multi-parameter persistence module. By modeling volumetric images as bounded functions, then we prove the interleaving distance on the persistence modules obtained from our bi-filtrations on the bounded functions is stable with respect to the maximum norm of the bounded functions. Finally, we conduct experiments on the MedMNIST
dataset, comparing our bi-filtration against single-parameter filtration and the established deep learning baselines, including Google AutoML
Vision, ResNet, AutoKeras and auto-sklearn. Experiments results demonstrate that our bi-filtration significantly outperforms single-parameter filtration. Notably, a simple Multi-Layer Perceptron (MLP) trained on the topological features generated by our bi-filtration achieves performance comparable to complex deep learning models trained on the original
dataset.
Read the full abstract ▾
All authors Qingsong Wang, Jiaxing He, Bingzhe Hou, Tieru Wu, Yang Cao, Cailing Yao
Computer Vision
Unifying Color and Lightness Correction with View-Adaptive Curve Adjustment for Robust 3D Novel View Synthesis
Ziteng Cui, Shuhong Liu, Xiaoyu Dong et al. (+4 more)
High-quality image acquisition in real-world environments remains challenging due to complex illumination variations and inherent limitations of camera imaging pipelines. These issues are exacerbated in multi-view capture, where differences in lighting, sensor responses, and image signal processor (ISP) configurations introduce photometric and chromatic inconsistencies that violate the assumptions of photometric consistency underlying modern 3D novel view synthesis (NVS) methods, including Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), leading to degraded reconstruction and rendering quality. We propose Luminance-GS++, a 3DGS-based framework for robust NVS under diverse illumination conditions. Our method combines a globally view-adaptive lightness adjustment with a local pixel-wise residual refinement for precise color correction. We further design unsupervised objectives that jointly enforce lightness correction and multi-view
geometric and photometric consistency. Extensive experiments demonstrate state-of-the-art performance across challenging scenarios, including low-light, overexposure, and complex luminance and chromatic variations. Unlike prior approaches that modify the underlying representation, our method preserves the explicit 3DGS formulation, improving reconstruction fidelity while maintaining real-time rendering efficiency.
Read the full abstract ▾
All authors Ziteng Cui, Shuhong Liu, Xiaoyu Dong, Xuangeng Chu, Lin Gu, Ming-Hsuan Yang, Tatsuya Harada
Computer Vision
Diff2DGS: Reliable Reconstruction of Occluded Surgical Scenes via 2D Gaussian Splatting
Tianyi Song, Danail Stoyanov, Evangelos Mazomenos et al. (+1 more)
Real-time reconstruction of deformable
surgical scenes is vital for advancing robotic surgery, improving intraoperative guidance, and enabling automation. Recent methods achieve dense reconstructions from da Vinci robotic surgery
videos, with Gaussian Splatting (GS) offering real-time performance via graphics acceleration. However, reconstruction quality in occluded regions remains limited, and depth accuracy has not been fully assessed, as
benchmarks like EndoNeRF and StereoMIS lack 3D ground truth. We propose Diff2DGS, a two-stage framework for reliable 3D reconstruction of occluded
surgical scenes. First, a
diffusion-based
video module with temporal priors inpaints tissue occluded by instruments with high spatiotemporal consistency. Second, we adapt 2D Gaussian Splatting (2DGS) with a Learnable Deformation Model (LDM) to capture dynamic tissue deformation and anatomical geometry, and introduce adaptive depth weight to improve
geometric fidelity. We further extend evaluation beyond image-quality metrics by performing quantitative depth analysis on the SCARED
dataset. Diff2DGS outperforms state-of-the-art methods in both appearance and geometry, reaching 38.02 dB PSNR on EndoNeRF and 34.40 dB on StereoMIS. Our experiments also show that optimizing image quality alone does not necessarily ensure accurate 3D geometry.The code is available at https://diff2dgs.github.io/.
Read the full abstract ▾
All authors Tianyi Song, Danail Stoyanov, Evangelos Mazomenos, Francisco Vasconcelos
Computer Vision
Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation
Ziyue Liu, Davide Talon, Federico Girella et al. (+5 more)
Sketches offer designers a concise yet expressive medium for early-stage fashion ideation by specifying structure, silhouette, and spatial relationships, while textual descriptions complement sketches to convey material, color, and stylistic details. Effectively combining textual and visual modalities requires adherence to the sketch visual structure when leveraging the guidance of localized attributes from text. We present LOcalized Text and Sketch with multi-level guidance (LOTS), a framework that enhances fashion image generation by combining global sketch guidance with multiple localized sketch-text pairs. LOTS employs a Multi-level Conditioning Stage to independently encode local features within a shared latent space while maintaining global structural coordination. Then, the
Diffusion Pair Guidance stage integrates both local and global conditioning via
attention-based guidance within the
diffusion model's multi-step denoising process. To validate our method, we develop Sketchy, the first fashion
dataset where multiple text-sketch pairs are provided per image. Sketchy provides high-quality, clean sketches with a professional look and consistent structure. To assess robustness beyond this setting, we also include an "in the wild" split with non-expert sketches, featuring higher variability and imperfections. Experiments demonstrate that our method strengthens global structural adherence while leveraging richer localized semantic guidance, achieving improvement over state-of-the-art. The
dataset, platform, and code are publicly available.
Read the full abstract ▾
All authors Ziyue Liu, Davide Talon, Federico Girella, Zanxi Ruan, Mattia Mondo, Loris Bazzani, Yiming Wang, Marco Cristani
Computer Vision
DEIG: Detail-Enhanced Instance Generation with Fine-Grained Semantic Control
Shiyan Du, Conghan Yue, Xinyu Cheng et al. (+1 more)
Multi-Instance Generation has advanced significantly in spatial placement and attribute binding. However, existing approaches still face challenges in fine-grained semantic understanding, particularly when dealing with complex textual descriptions. To overcome these limitations, we propose DEIG, a novel framework for fine-grained and controllable multi-instance generation. DEIG integrates an Instance Detail Extractor (IDE) that transforms text encoder
embeddings into compact, instance-aware representations, and a Detail Fusion Module (DFM) that applies instance-based masked
attention to prevent attribute leakage across instances. These components enable DEIG to generate visually coherent multi-instance scenes that precisely match rich, localized textual descriptions. To support fine-grained supervision, we construct a high-quality
dataset with detailed, compositional instance captions generated by VLMs. We also introduce DEIG-Bench, a new
benchmark with region-level annotations and multi-attribute
prompts for both humans and objects. Experiments demonstrate that DEIG consistently outperforms existing approaches across multiple
benchmarks in spatial consistency, semantic accuracy, and compositional generalization. Moreover, DEIG functions as a plug-and-play module, making it easily integrable into standard
diffusion-based pipelines.
Read the full abstract ▾
All authors Shiyan Du, Conghan Yue, Xinyu Cheng, Dongyu Zhang
Computer Vision · Cross-listed from cs.RO
RoEL: Robust Event-based 3D Line Reconstruction
Gwangtak Bae, Jaeho Shin, Seunggu Kang et al. (+3 more)
Event cameras in motion tend to detect object boundaries or texture edges, which produce lines of brightness changes, especially in man-made environments. While lines can constitute a robust intermediate representation that is consistently observed, the sparse nature of lines may lead to drastic deterioration with minor estimation errors. Only a few previous works, often accompanied by additional sensors, utilize lines to compensate for the severe domain discrepancies of event sensors along with unpredictable noise characteristics. We propose a method that can stably extract tracks of varying appearances of lines using a clever algorithmic process that observes multiple representations from various time slices of events, compensating for potential adversaries within the event data. We then propose
geometric cost functions that can refine the 3D line maps and camera poses, eliminating projective distortions and depth ambiguities. The 3D line maps are highly compact and can be equipped with our proposed cost function, which can be adapted for any observations that can detect and extract line structures or projections of them, including 3D point cloud maps or image observations. We demonstrate that our formulation is powerful enough to exhibit a significant performance boost in event-based mapping and pose refinement across diverse
datasets, and can be flexibly applied to
multimodal scenarios. Our results confirm that the proposed line-based formulation is a robust and effective approach for the practical deployment of event-based perceptual modules. Project page: https://gwangtak.github.io/roel/
Read the full abstract ▾
All authors Gwangtak Bae, Jaeho Shin, Seunggu Kang, Junho Kim, Ayoung Kim, Young Min Kim
Computer Vision
On the Adversarial Robustness of Discrete Image Tokenizers
Rishika Bhagwatkar, Irina Rish, Nicolas Flammarion et al. (+1 more)
Discrete image tokenizers encode visual inputs as sequences of
tokens from a finite vocabulary and are gaining popularity in
multimodal systems, including encoder-only, encoder-decoder, and decoder-only models. However, unlike CLIP encoders, their vulnerability to
adversarial attacks has not been explored. Ours being the first work studying this topic, we first formulate attacks that aim to perturb the features extracted by discrete tokenizers, and thus change the extracted
tokens. These attacks are computationally efficient, application-agnostic, and effective across classification,
multimodal retrieval, and captioning tasks. Second, to defend against this vulnerability, inspired by recent work on robust CLIP encoders, we fine-tune popular tokenizers with unsupervised
adversarial training, keeping all other components frozen. While unsupervised and task-agnostic, our approach significantly improves robustness to both unsupervised and end-to-end supervised attacks and generalizes well to unseen tasks and data. Unlike supervised
adversarial training, our approach can leverage unlabeled images, making it more versatile. Overall, our work highlights the critical role of tokenizer robustness in downstream tasks and presents an important step in the development of safe
multimodal foundation models.
Read the full abstract ▾
All authors Rishika Bhagwatkar, Irina Rish, Nicolas Flammarion, Francesco Croce