Computer Vision
Evaluating the Robustness of Anti-UAV Detection under Controlled Fog Degradation: Fog-Aware Training and Clear-Sky Tradeoff
Gur Levy Birkental, Seyed Sahand Mohammadi Ziabari and Ali Mohammed Mansoor Alsahag
Vision-based anti-
UAV systems must function in poor visibility, yet most
benchmarks use only clear-sky footage, and previous robustness studies treat adverse weather as a simple present/absent condition. As a result, the impact of fog severity on ground-to-air
UAV detection remains poorly understood. This work presents the first severity-controlled fog
benchmark for this task: synthetic fog at ten severity levels is applied to the RGB modality of the Anti-
UAV300
dataset, comparing a clear-trained YOLOv5m baseline to a fog-aware model trained on both clear and foggy images. Detection performance drops sharply and non-linearly: degradation is front-loaded across light-to-moderate fog (beta approximately 0.05-0.10), with a 96% reduction in mAP@0.5:0.95 from clear to thickest fog, mainly due to lost recall and confidence. On the comparable metric (mAP@0.5), this collapse exceeds the most extreme rain degradation reported in the closest prior
benchmark. Fog-aware training boosts detection across all severities (up to +0.320 mAP@0.5:0.95) with only a 9.1% drop in clear-sky accuracy, raising the threshold for reliable detection while not preventing collapse under extreme fog. Since reliability is lost within a narrow visibility range, simple clear vs adverse tests underestimate operational risk.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
FIDAL: Diversity-Aware Federated Active Learning Under Real-World Distribution Shifts
David Dueñas Gaviria and Shadi Albarqouni
Federated learning enables collaborative model training across institutions without centralizing data, yet high annotation costs, domain shifts, and class imbalance remain major obstacles, especially when irrelevant out-of-distribution (OOD) samples dilute the labeled data. Existing active learning methods target uncertainty or diversity within in-distribution (ID) data and overlook unknown samples in
federated clinical settings. We propose FIDAL, an open-set
federated active learning framework that combines calibrated global-local evidential uncertainty, support-set diversity weighting, and adaptive OOD rejection. The rejection gate thresholds a foundation-model Gaussian-coverage signal per client and per round with Otsu's criterion, so that highly informative ID samples are queried while irrelevant outliers are excluded without any hand-tuned threshold. Evaluated on three multi-center medical imaging
benchmarks (dermatology, histopathology, and mammography with organically occurring artifacts) in realistic open-set scenarios, FIDAL outperforms detector-based open-set methods by up to about 12 percentage points of balanced accuracy and is the only method on the accuracy-ID purity Pareto front of all three
benchmarks. At an equal query budget it spends at least 1.3 times fewer annotations on OOD samples than every accuracy-matched baseline, saving an estimated 7-29 hours of expert reading on the mammography
benchmark. By labeling only a fraction of the data pool, it matches or exceeds fully supervised performance across modalities. These results highlight the value of integrating uncertainty, diversity, and OOD rejection in open-set
federated active learning for medicine.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Grounding Vision-Language Models in Driving Semantics: A Multi-Dataset Predicate Framework for Explainable Reasoning
Mohamed Chouai, Fazli Faruk Okumus and Stefan Kugele
Vision-
language models are increasingly used for driving-scene understanding, yet the semantic relations expressed in their outputs are often difficult to verify against the underlying traffic situation. This paper introduces a deterministic multi-
dataset predicate framework that derives driving-scene semantics from measurable
geometric, kinematic, temporal, map, and traffic-control evidence.
Dataset-specific interfaces are used only to recover the required scene information, while predicate definitions remain unchanged across nuPlan and nuScenes and are materialised in a common Predicate Knowledge Graph. Quantitative semantic validation against manually annotated predicate relations on 200 scenarios from each
dataset yields macro F1 scores of 0.94 on nuPlan and 0.93 on nuScenes, with an average cross-
dataset difference of 0.02 across the shared predicates. The Predicate KG is further evaluated using a frozen LLaVA-OneVision-7B model on the nine NuPlanQA subtasks. Predicate grounding achieves the highest accuracy among the evaluated visual-input conditions in seven of nine NuPlanQA subtasks, including Traffic Light (53.2% to 71.5%), Situation Assessment (76.2% to 86.1%), and Action Recommendation (82.9% to 89.0%). Weather/Lighting remains essentially unchanged (89.4% vs. 88.8%), consistent with the absence of corresponding predicates, while Predicate KG only input outperforms metadata-only input in eight of nine subtasks. The results show that deterministic predicates provide a consistent and traceable semantic representation and, under oracle grounding, can reduce visual dependence for
reasoning tasks covered by the predicate vocabulary.
Read the full abstract ▾
Computer Vision
Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
Christian Gapp, Elias Tappeiner, Martin Welk et al. (+8 more)
Multimodal stroke recurrence prediction requires effective integration of heterogeneous clinical and imaging data, yet modality imbalance often causes models to over-rely on dominant modalities and underutilize complementary information. While self-supervised pretraining and selective parameter freezing are commonly employed to improve representation learning and fine-tuning stability, their effect on modality contributions and cross-modal behavior in
multimodal medical models remains largely unexplored. In this work, we investigate whether image pretraining on 3D CTA scans reduces modality imbalance and improves cross-modal integration for stroke recurrence prediction, a clinically critical task we recently addressed. To this end, two
multimodal neural networks are pretrained in a self-supervised manner and subsequently fine-tuned using two distinct freezing strategies. Their performance and modality utilization are compared against both the baseline model from our previous work and models trained entirely from scratch in this study. Our results demonstrate that self-supervised pretraining enables more effective utilization of the
multimodal image-tabular
dataset, outperforming both the prior baseline and all non-pretrained models. Notably, the best-performing
Vision Transformer based neural network successfully overcomes unimodal collapse. Synergy analysis reveals significant interactions between
vision and both gender and CHD, suggesting clinically relevant patterns for stroke recurrence. Overall, our findings demonstrate that self-supervised pretraining and strategic fine-tuning support more balanced modality utilization and enable meaningful cross-modal interactions. Code is publicly available at https://github.com/ChristianGappGit/SSL_Pretraining.
Read the full abstract ▾
All authors Christian Gapp, Elias Tappeiner, Martin Welk, Karl Fritscher, Stephanie Mangesius, Constantin Eisenschink, Philipp Deisl, Michael Knoflach, Astrid E. Grams, Elke R. Gizewski, Rainer Schubert
Computer Vision · Cross-listed from cs.GR
3D Field Data Reduction with Adaptive Sample-Based Gaussian-Encoded Reconstruction
Michael R. Martin, Joseph Insley, Victor A. Mateevitsi et al. (+2 more)
In scientific simulation, regular grids, unstructured meshes, and particle-based formats are chosen to represent field data for computational efficiency, geometry/adaptive flexibility, and following motion/deformation, respectively. Each of these field data formats is often handled through separate data-specific processing pipelines. We present a unified sample-based Gaussian encoding method that represents these data forms under a single fixed-budget formulation. The method initializes and refines Gaussian primitives directly from the input samples while preserving a prescribed primitive count and encoded size to achieve a desired level of data reduction. Across structured, unstructured, and particle data, the sample-based formulation improves reconstruction accuracy with measurably fewer primitives in comparison to prior formulations, achieving up to 4.8 dB higher PSNR with an approximate 44x reduction in primitive count. For time-varying data, warm-starting from the previous timestep reduces the optimization required to reach independently trained reconstruction quality. Together, these results demonstrate a unified fixed-budget Gaussian encoding framework for structured, particle, unstructured, and time-varying scientific data with predictable storage, higher reconstruction accuracy, and improved temporal encoding efficiency.
Read the full abstract ▾
All authors Michael R. Martin, Joseph Insley, Victor A. Mateevitsi, Silvio Rizzi, Kwan-Liu Ma
Computer Vision · Cross-listed from cs.RO
Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioning
Changbo Yan, Zhongbo Zhang, Zaibin Zhang et al. (+3 more)
3D point-cloud observations are inherently ambiguous in complex, cluttered manipulation scenes, where target objects may be partially occluded or tightly intermingled with visually similar distractors. As a result, standard 3D
diffusion policies often struggle to localize and exploit task-relevant geometry as scene complexity grows. We propose
Attention-DP3, a spatially object-aware 3D
diffusion policy that injects object-level
geometric cues via
attention while keeping the DP3
diffusion backbone unchanged. Our pipeline performs open-vocabulary 2D
segmentation on RGB images, then lifts predicted target masks into 3D using calibrated camera geometry to obtain object-centric
geometric priors. We incorporate these cues through Tri-field Attentional Conditioning, which constructs three complementary fields: (i) a targetness field to anchor the target object, (ii) an intra-target saliency field to emphasize task-relevant geometry within the target, and (iii) a backgroundness field to suppress distractors and clutter. Experiments on Adroit, DexArt, MetaWorld, and the real-world SO101 platform show consistent improvements over DP3, achieving state-of-the-art performance across
benchmarks. Notably, as distractor objects increase, DP3 drops sharply, whereas
Attention-DP3 remains stable and outperforms DP3 by up to 31% under heavy clutter. The code is publicly available at https://github.com/zhangzhongbo2213/
Attention-DP3.
Read the full abstract ▾
All authors Changbo Yan, Zhongbo Zhang, Zaibin Zhang, Yifan Wang, Lijun Wang, Huchuan Lu
Computer Vision
Task-Based CT Protocol Optimization Using Reinforcement Learning and Virtual Imaging Trials
Jiaqi Zou, David Fenwick, Vahid Tarokh et al. (+6 more)
Protocol optimization in computed
tomography (CT) aims to improve diagnostic image quality while reducing radiation dose, but the interdependence of acquisition and reconstruction parameters makes exhaustive testing impractical. We propose a virtual imaging trial framework with reinforcement learning for efficient CT protocol optimization. Sixty-three computational human models with liver lesions were imaged using a validated CT simulator across 468 combinations of acquisition and reconstruction parameters, including tube voltage, tube current, reconstruction kernel, slice thickness, and pixel size. The optimization objective balanced liver lesion detectability, quantified by detectability index d-
prime, against radiation dose. A Proximal Policy Optimization
agent was trained and conditioned on patient-specific CT localizer
embeddings derived from a pretrained
vision transformer. On held-out patients, evaluating only 8 protocols per patient, about 2% of exhaustive testing, recovered 98.2% of the exhaustive-search oracle objective. With no patient-specific simulation, surrogate scoring alone achieved 89.7% recovery. Conditioning on the localizer improved zero-simulation recovery by 10.7 percentage points over the localizer-blind policy (paired 95% CI 2.9-19.5; p=0.02). These results show that the proposed framework can substantially reduce exhaustive protocol testing while enabling task-based, dose-aware protocol selection before the diagnostic scan.
Read the full abstract ▾
All authors Jiaqi Zou, David Fenwick, Vahid Tarokh, Nicholas Felice, Jayasai Rajagopal, Anuj Kapadia, Ehsan Samei, Navid NaderiAlizadeh, Ehsan Abadi
Computer Vision
GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures
Sarthak Sattigeri
A companion evaluation found that naming the target part in a manipulation
prompt increased action accuracy by 0.32-0.63 across eight
vision-
language models, with no model outperforming a constant baseline until the part was named. However, naming the part supplies information that a real system must infer, confounding visual grounding, mechanical
reasoning, and category-to-action association. We introduce GroundBench, a diagnostic
benchmark that separates these explanations through six branch-and-merge conditions, each adding a controlled information bundle, and a counterfactual re-ask targeting a real alternate part visible in the same image. Across three OpenAI models and 1,068 predictions, supplying the target region without its identity leaves action accuracy at or below the 0.53 majority baseline (0.26, 0.26, and 0.53), although the models largely reproduce the supplied region. Supplying identity without location instead yields 0.74, 0.68, and 0.68. Every above-baseline gain in this curated set occurs where the supplied part category itself determines the action. A no-
vision control leaves GPT-5's scores unchanged or improved, providing evidence consistent with substantial category-to-action association. GPT-4o mini declines on one condition, so this interpretation is not universal. Adding joint type and motion axis does not improve accuracy across six model-stratum comparisons. On 74 counterfactual pairs from 32 objects, GPT-5 achieves 0.86 pair-weighted compliance with a 0.07 shortcut rate but fails all observed push-to-lift-vertical cases. GroundBench identifies which supplied information changes affordance behavior and tests whether apparently grounded performance can be reproduced through textual shortcuts.
Read the full abstract ▾
Computer Vision
HGSQ: Heatmap-Guided Sparse Query Detector for Real-Time Aerial Small Object Detection
Yangchen Zeng
Real-time aerial small object detection is an important visual signal and image processing problem, requiring a detector to preserve fine-grained localization while avoiding redundant computation on large background regions. This paper focuses on this deployment-oriented aerial/
UAV setting rather than claiming a universal detector for all object detection scenarios. Existing
Transformer-based detectors provide strong global modeling, but their dense query initialization and multi-layer decoder still spend substantial computation on background
tokens, which is inefficient when small objects occupy only sparse image regions. To address this problem, this paper proposes HGSQ, a Heatmap-Guided Sparse Query Detector for real-time aerial small object detection. HGSQ uses a lightweight Heatmap Budget Predictor (HBP) to predict a foreground budget map in a single forward pass. The predicted heatmap is then used by three fixed components: Heatmap-Guided Sparse Query Selection (HSQS), which initializes decoder queries from high-confidence foreground positions; Heatmap-Gated Lite Snake Convolution (HGLSConv), which performs local shape refinement only on heatmap-activated small-object regions; and Adaptive Query-Decoder Budgeting (AQDB), which adjusts the query budget and decoder depth according to the estimated object density. Unlike post-hoc heatmap generation, HGSQ treats the heatmap as a real-time computation budget rather than a visualization map during deployment. Experiments on NWPU VHR-10 and VisDrone2019 show that HGSQ achieves 95.10 mAP50 on NWPU VHR-10 and 54.8 mAP50 on VisDrone2019, while reducing GFLOPs to 48.6 and running at 96.0 FPS on an RTX 4070 under our TensorRT FP16 deployment protocol.
Read the full abstract ▾
Computer Vision
Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classification
Saibal Ghosh, Samarup Bhattacharya, Sanjoy Kumar Saha et al. (+2 more)
Medical image classification is frequently complicated by transitional categories whose feature distributions overlap those of adjacent classes, producing ambiguous decision boundaries. Conformal prediction returns uncertainty-aware prediction sets, but these are not directly actionable in clinical screening, where a single decision is required. This work proposes adaptive conformal redistribution (AdaConRed), a label-free post-conformal decision rule that converts ambiguous prediction sets into refined class assignments. A five-stage pipeline is developed.
Vision-
language generative augmentation addresses minority-class scarcity; a frozen DermFoundation encoder provides
embeddings; a lightweight multi-layer perceptron performs classification; an
entropy-modulated, margin-aware nonconformity score constructs adaptive prediction sets; samples predicted as transitional with multi-label sets are reassigned to the most probable alternative class within the set, using only model outputs at inference. Evaluation uses the OSCC oral lesion and ISIC skin lesion
benchmarks at a miscoverage level of 0.2. On the 3-class OSCC
benchmark, overall accuracy improves from 73.54% to 77.38%, with oral cancer accuracy rising from 64.29% to 82.14% and benign accuracy from 56.57% to 70.20%. Reassignment of transitional samples reduces OPMD accuracy from 84.78% to 80.16%, consistent with the asymmetric cost of missed malignancy. On ISIC, overall accuracy improves from 85.83% to 87.19%, melanoma accuracy rising from 66.04% to 68.34%. AdaConRed outperforms LAC, APS and RAPS under an identical backbone and redistribution rule. Conformal prediction can be extended beyond uncertainty quantification toward actionable decision support where transitional disease categories are present, with gains concentrated in the clinically critical malignant categories. Code repository: https://github.com/saibal436ghosh/AdaConRed.
Read the full abstract ▾
All authors Saibal Ghosh, Samarup Bhattacharya, Sanjoy Kumar Saha, Umapada Pal, Tapabrata Chakraborti
Computer Vision
Variational Template Matching with Statistical Fusion for Anomaly Detection in Patterned Structures
Qinwu Xu and Yifan Jiang
Anomaly detection in structured images is challenging in small-data settings where deep learning approaches are costly or impractical. Classical template matching is simple and interpretable but lacks robustness to
geometric variations such as scale, rotation, and perspective. We propose a variational template matching framework that represents anomaly templates as a family of transformed instances and performs detection via normalized cross-correlation over this transformation space. To further improve robustness, we introduce a density-based statistical anomaly score derived from local intensity distributions using kernel density estimation (KDE). This produces a smooth representation that captures distributional concentration and tail behavior more robustly than histogram-based methods. The structural and statistical signals are integrated through a unified fusion formulation, enabling complementary modeling of
geometric similarity and distributional deviation. Experiments on biological
cell images demonstrate that the proposed method outperforms classical baselines and achieves competitive performance with ResNet-50 under a fully training-free setting, while providing explicit localization. The approach offers an efficient, interpretable, and practical solution for anomaly detection in structured image domains.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
Long Giang Vu, Chengkai Yao, Yuxin Liu et al. (+2 more)
Vision-
Language Navigation (VLN) in unseen indoor environments is useful in real-world robotics, where an
agent must follow natural-
language instructions, locate objects, and answer spatial questions without a pre-built map or fixed object vocabulary.
Multimodal vision-
language models (VLMs) provide strong open-vocabulary grounding and zero-shot
reasoning, but struggle to emit reliable metric quantities such as range, bearing, and comparative spatial relations directly from images. Existing approaches address this by folding geometry into hand-engineered pipelines or asking models to output waypoints, requiring changes to the control stack for different
robots, tasks, or vocabularies. We introduce AnchorVLN, an open-vocabulary VLN system built on a simple rule: the VLM proposes semantics; geometry decides metrics. It is realised as
EMBODIED-NAV-MCP, a Model Context Protocol (MCP) server driven by a VLM
agent through a compact set of callable tools. Since no tool accepts distance in metres or bearing in radians, the schema enforces the semantic-geometry boundary without modifying the downstream autonomy stack. We
benchmark both tasks of the CMU
Vision-
Language Navigation Challenge 2026: 30 instruction-following questions over 15 scenes and a frozen 45-question object-reference set. The full system achieves 64.4 percent on instruction following, dropping by 13.3 percentage points without controller modeling (t = 2.77). On object reference,
geometric anchoring clears the challenge overlap threshold on 10 of 45 questions, versus 0 of 45 for direct coordinate estimation, reducing median center error from 3.37 m to 2.48 m.
Read the full abstract ▾
All authors Long Giang Vu, Chengkai Yao, Yuxin Liu, FNU Aryan, Rajath Chandrashekar Aralikatti
Computer Vision
Revisiting Multi-Object Tracking Baselines: Hyperparameter Optimization with Multi-Fidelity Greedy Coordinate Search
Momir Adžemović
Multi-object tracking (MOT) is dominated by the tracking-by-detection paradigm, whose methods typically rely on a small set of hyperparameters that are conventionally chosen by hand. Tuning them requires repeated expert-guided experimentation, while the procedures used to select reported values are often not systematically evaluated or fully documented. Hyperparameter optimization (HPO) automates this process, yet it remains rarely used in MOT, and existing studies applying HPO to MOT predate modern deep-detector-based trackers and HOTA evaluation. We systematically apply HPO across two
datasets and four tracking-by-detection methods. We also propose Multi-Fidelity Greedy Coordinate Search (MFGCS), which optimizes one hyperparameter at a time by first evaluating candidate values on a small subset of scenes and re-evaluating only promising candidates on the full
dataset. Across all eight tracker-
dataset combinations, the Tree-structured Parzen
Estimator (TPE) and MFGCS outperform both our hand-tuned configurations and the corresponding published results, with improvements of up to 4.38 and 16.05 HOTA points, respectively. MFGCS also reaches a predefined HOTA target faster than TPE in seven of the eight combinations. Within each tracker-
dataset pair, all optimizers share the same search space and evaluation pipeline, isolating the effect of the search strategy. We release the code and tuned configurations to enable future work to compare against systematically optimized rather than default or manually tuned baselines.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
PLSP (Pre-hoc Liminal Space Profiling): OOD Prediction over Detection – An Anticipatory Approach for Machine Learning Model Reliability
Vipul Bansal, Himanshu Buckchash, Balasubramanian Raman et al. (+1 more)
Out-of-Distribution (OOD) data poses a significant threat to machine learning models, often leading to model failure during deployment. All existing OOD detection methods are post-hoc, relying on evaluation metrics such as accuracy and AUC-ROC during inference to indirectly assess the model's response to OOD data by measuring deviations. In contrast to existing approaches, the proposed work shifts the paradigm from OOD detection to OOD prediction by proposing a pre-hoc anticipatory framework called PLSP for OOD prediction. We make several key contributions: (a) a
dataset-independent metric called the CREDibility Score (CREDS) is proposed for OOD prediction; (b) credibility curves are introduced to study the maximum credibility a model can attain; and (c) credibility heat maps (and volume under surface) are introduced to characterize pre-hoc model behavior across different
datasets. This work provides a novel perspective on signal processing under distributional shifts. Experiments across multiple
datasets demonstrate that the proposed metric serves as a valuable measure for improving the robustness of machine learning models toward OOD prediction.
Read the full abstract ▾
All authors Vipul Bansal, Himanshu Buckchash, Balasubramanian Raman, Deepak Dhungana
Computer Vision
QuPAINT: Physics-Aware Multimodal Reasoning for Quantum Material Characterization
Sankalp Pandey, Xuan-Bac Nguyen, Hoang-Quan Nguyen et al. (+4 more)
Characterizing two-dimensional (2D)
quantum materials by optical microscopy requires localizing exfoliated flakes and determining their layer thickness from subtle optical contrast and interference color to select suitable flakes for device fabrication. However, models face synthetic-to-real domain shifts and variation across materials, substrates, laboratories, and imaging conditions. We present QuPAINT, a physics-aware
multimodal framework for transferable
quantum flake characterization. The Synthetic Materials Framework (Synthia) generates diverse synthetic microscopy images while preserving layer-dependent optical behavior. Using these images, we construct QMat-Instruct, a
multimodal instruction
dataset with image-specific
reasoning traces generated from verified annotations and constrained to observable optical cues. QuPAINT integrates these signals through Physics-Informed
Attention (PIA), which injects substrate-relative optical priors into the visual representation to support grounded
multimodal reasoning. For evaluation, we introduce QF-Bench, to our knowledge, the largest real-world
benchmark for this problem, spanning diverse microscopy and substrate conditions. Using its verified annotations, we study counting, visual grounding,
reasoning quality, confidence calibration, and transfer to an unseen material. QuPAINT-8B substantially outperforms prior methods and establishes state-of-the-art performance for both general and monolayer flake detection. Additional experiments show that image-grounded supervision improves strict spatial grounding and confidence calibration while preserving robust general flake detection on the unseen material.
Read the full abstract ▾
All authors Sankalp Pandey, Xuan-Bac Nguyen, Hoang-Quan Nguyen, Tim Faltermeier, Nicholas Borys, Hugh Churchill, Khoa Luu
Computer Vision
Physics as the label for measuring and correcting materials reasoning in multimodal models
Hasan Kurban, Rasul Khanbayov and Mustafa Kurban
Vision-
language and
language models increasingly interpret materials data, yet
benchmarks report that they hallucinate invalid properties and violate physical law. Evaluation matches final answers to scarce human labels, while discovery
agents verify final proposals or density functional theory (DFT) execution. Neither measures the physical consistency of a model's
reasoning chain. Materials data carries its own physics, making a large class of materials
reasoning verifiable without annotation. We introduce MatPCR, a label-free
benchmark whose programmatic oracles check diffraction geometry through Bragg's law, scale bars, spectral peaks, and Materials Project-grounded checks of near-hull stability, computed band-gap class, and net magnetization. We define the Physical-Consistency Rate over image and structure inputs; introduce Constraint-Grounded Self-Verification, an agentic loop whose gain survives self-refinement and equal-compute re-prompting controls; release an open verifier useful in distribution but near chance on all six held-out constraint types; and derive an exact identity for how oracle error displaces the reported rate.
Read the full abstract ▾
Computer Vision
When Ground-Truth Fidelity Matters: An Orchestrated UAS Framework for Wheat Streak Mosaic Virus Detection Using Vision Transformers and Machine Learning
Dewi Endah Kharismawati, Sandeep Dhakal, Courtney E. McCusker et al. (+3 more)
Wheat streak mosaic virus (WSMV) is a destructive pathogen of sweet corn and other cereal crops, causing yield losses and complicating early detection because symptoms are spatially variable and subtle. In sweet corn seed production, WSMV also has regulatory importance, as phytosanitary regulations from countries such as New Zealand and Chile require seed lots to be certified virus-free. Visual scouting is unreliable because symptoms can resemble abiotic stress, while enzyme-linked immunosorbent assay (ELISA) is accurate but expensive, labor-intensive, and difficult to scale. We present an automated pipeline for plant-level WSMV detection using unmanned aircraft systems (UAS) multispectral imagery. The framework integrates orthomosaic reconstruction, geospatial
alignment, plant extraction, and classification using a
Vision Transformer with seven-channel inputs (five spectral bands, NDVI, and NDRE). Using treatment-based labels, the model achieved 89% accuracy on over 6,500 test patches across multiple growth stages. However, ELISA-based ground truth revealed substantial label noise: only a small fraction of sampled plants in inoculated plots were infected. Treatment labels therefore did not reliably represent infection status, and the high accuracy was largely driven by label bias rather than disease detection. Performance decreased markedly against row-level symptom severity and plant-level ELISA labels. Under these higher-fidelity but smaller-sample conditions, both deep learning and classical machine learning showed limited generalization and weak separability between ELISA-confirmed mock-inoculated and infected plants. These results show that UAS-based disease detection is constrained by label fidelity and data availability, emphasizing biologically grounded labels and models aligned with real-world conditions.
Read the full abstract ▾
All authors Dewi Endah Kharismawati, Sandeep Dhakal, Courtney E. McCusker, Jennifer R. Wilson, Erik W. Ohlson, Sami Khanal
Computer Vision
USPLIT-VQA: U-Shaped Split Learning for Visual Question Answering with Contribution-Aware Weighted Aggregation
Md Khalid Syfullah and Alvi Ataur Khalil
Visual Question Answering (VQA) systems, jointly interpreting images and natural
language queries, hold significant promise across many domains, yet the
privacy-sensitive nature of user data creates a fundamental barrier. Centralized training requires access to all data, while
federated learning requires each client to host the full model. We propose USPLIT-VQA, a U-shaped split learning framework for
privacy-preserving VQA in which each client retains the initial layers and the classification head while the server hosts the computationally heavy intermediate layers, keeping raw inputs and labels on the client device. We further introduce Contribution-Aware Weighted Aggregation (CAWA), a gradientsimilarity-based client scoring mechanism designed to reduce the influence of malicious updates. Experiments on four VQA
datasets (VQA-RAD, SLAKE, PathVQA, and VizWiz) with two backbones show accuracy gains over
Federated Learning for the Custom model and reduced accuracy for BiomedCLIP under the evaluated fixed split, alongside client
memory reductions of up to 5.8X and communication reductions of up to 10.8X. With one malicious client, CAWA reduces the attacker's influence by over 98%, while experiments at higher corruption levels identify its limitations. Reconstruction experiments further show lower inversion quality under the evaluated attacks.
Read the full abstract ▾
Computer Vision
Single-Query Person-Centric Bimanual Hand-Object Interaction Detection
Jonghyun Kim, Junho Roh, Yubin Yoon et al. (+5 more)
Understanding person-level bi-manual interactions requires not only detecting hands, but also identifying which two hands belong to the same person and what each hand interacts with. Existing hand–object interaction methods are mostly hand-centric: they treat each hand as an independent instance, which can lead to ambiguous ownership in multi-person scenes. We propose a person-centric formulation in which a single query predicts a structured output for one person, including the human box, body pose, hand boxes and states, and interaction targets. We introduce part-aware deformable
attention to allocate
attention across human, hand, and pose-specific reference regions, enabling one query to capture the full person structure. We further unify detection and interaction
reasoning with a hand-to-query relationship
matrix, where each hand selects its interaction target from the detected query set plus a learnable off
token, directly recovering the target's box and class without separate object regression. We build a COCO-based
dataset with person-centric bi-manual interaction annotations and define structured metrics for evaluating hand states and complete hand–object tuples. Experiments with a
transformer-based detector show that our formulation improves person-level bi-manual interaction parsing and provides an effective unified framework for joint detection, pose estimation, and hand
reasoning.
Read the full abstract ▾
All authors Jonghyun Kim, Junho Roh, Yubin Yoon, Hyotae Lee, Jongkuk Park, Taehwan Hwang, Jaechul Kim, Jungho Lee
Computer Vision
HSI-Road Relabeled: Surface-Aware Road-Scene Segmentation
Imad Ali Shah, Imran Mehmood, Enda Ward et al. (+3 more)
The HSI-Road
dataset provides paired RGB and 25-channel NIR (680-960nm) images with binary masks but no surface-level labels. This paper introduces a manually labeled six-class taxonomy: Background, Asphalt, Concrete, Dirt, Water, and Grass, and an RGB-to-NIR registration pipeline with corresponding annotations. Six semantic-
segmentation models are evaluated under four input configurations: original-resolution RGB, registered low-resolution RGB (RGB$_{\text{reg}}$), NIR, and channel-stacked RGB$_{\text{reg}}$-NIR (RGBN$_{\text{stk}}$). The comparison quantifies the effect of spatial-resolution reduction on RGB, along with evaluation of NIR and RGBN$_{\text{stk}}$, with results reported using per-class and mean IoU and F1 scores. The original-resolution RGB achieves the highest overall performance but contains 12$\times$ more pixels and incurs a 15.5-20.6$\%$ latency penalty compared to the reduced-resolution inputs. At the common 192$\times$384 resolution, RGBN$_{\text{stk}}$ outperforms NIR for all six models and RGB$_{\text{reg}}$ for five of six, with the most consistent gains for the Water class. These results highlight the importance of spatial resolution while showing that NIR provides complementary information to RGB.
Read the full abstract ▾
All authors Imad Ali Shah, Imran Mehmood, Enda Ward, Martin Glavin, Edward Jones, Brian Deegan
Computer Vision
Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation
Silas Kwabla Gah and Ebenezer Owusu
Classical classifier-combination work distinguishes score-level fusion from hard decision-level voting. We revisit this distinction where independently pretrained, frozen foundation models are composed at inference time for generalized few-shot 3D
segmentation. We ask: how much useful semantic information is lost when heterogeneous sources are collapsed to a single class before they can interact? We answer with a same-input semantic-retention intervention. Dense RegionPLC and sparse cross-view SAM3 evidence, model weights, masks, geometry, vocabularies, and fusion rules are frozen; only the number of semantic alternatives retained before interaction is varied via a matched top-k ladder. On 156 held-out ScanNet200 scenes, top-1 reaches 28.47 harmonic-mean (HM) IoU while full distribution fusion reaches 34.87 HM (+6.40, 95% CI [+5.24,+7.64]). The pattern replicates on 50 ScanNet++ scenes: 23.02 vs. 26.50 HM (+3.48, 95% CI [+1.64,+5.93]). The conclusion is robust: full-distribution HM is stable across sparse-source weights 0.3–0.7; alternative operators (max,
geometric pooling) also outperform top-1; and a GroundingDINO–SAM2.1 source-replacement diagnostic shows monotonic HM increase from 14.77 to 18.75 with full retention. Calibration diagnostics reveal opposite miscalibration of the two sources, yet correcting calibration does not eliminate the retention advantage. Across
datasets and source stacks, most information is recovered by retaining a compact set of plausible alternatives. The contribution is a controlled diagnosis of premature semantic collapse as a repeatable information bottleneck in heterogeneous frozen-model composition.
Read the full abstract ▾
Computer Vision
Does Video Memory Use What It Retrieves? A Causal Audit of Memory Specificity
Aditi Tiwari, Akshit Bhalla, Darshan Prasad et al. (+1 more)
Video models increasingly use
memory to preserve information over long sequences, with the assumption that gains come from retrieving and using the correct past content. Standard
memory ablations test whether
memory helps, but not whether the retrieved content is responsible. We test this directly with read-time
memory substitution, which replaces the consumed
memory value while leaving the rest of the computation unchanged. This separates
memory benefit from
memory specificity, the extent to which the gain depends on retrieved content. Across frozen
video world models, identity-free controls containing no evaluation-specific content recover essentially the full benefit on Ego-Exo4D and 7-Scenes and about 70% on TUM. In the Ego-Exo4D dose response, recovery falls from 102% to 1% as these values move away from observed training-
memory representations, supporting representation repair as the best-supported explanation in this setting. WorldMem shows graded dependence. A wrong
memory from the same trajectory recovers 94.1% of the PSNR benefit relative to zero content, while a donor from a disjoint trajectory and biome recovers 43.7%. SAM 2 shows strong content dependence. On DAVIS, replacing the correct spatial
memory with a valid wrong
memory reduces mean region and boundary score from 0.926 to 0.182. At MOSEv2 reappearance, it falls from 0.459 to 0.000. These results show that
memory gains can depend on generic representation support, broader context, or exact episodic content. Read-time substitution provides a direct way to distinguish them.
Read the full abstract ▾
All authors Aditi Tiwari, Akshit Bhalla, Darshan Prasad, Heng Ji
Computer Vision · Cross-listed from cs.RO
MoPA: Coordinated Mobile Manipulation via Subsystem-Specific Perception Alignment
Guangyu Chen, Qiwei Liang, Shaolong Zhu et al. (+7 more)
Mobile manipulation requires perceptual evidence at different spatial scales for base motion and arm control, while the two action modalities remain kinematically coupled. Existing policies often employ specialized action generation for different subsystems but condition heterogeneous action branches on a shared perceptual representation, leaving subsystem-specific perception-action correspondence implicit. We present MoPA, a framework that aligns perceptual conditioning with mobility and manipulation while preserving coordination at the action level. Dual Perceptual Streams employ two mutually masked query banks to extract separate perceptual representations from a shared
vision-
language context. Perception2Action Adaptation jointly updates each query bank and its corresponding action stream at every layer of a structured Mixture-of-
Transformers decoder, while enabling information exchange between the two action streams. Coupled conditional flow matching learns a joint vector field for coordinated generation of both action chunks. On the ManiSkill-HAB
benchmark, MoPA achieves state-of-the-art performance across all three task suites. Across four real-world tasks, MoPA achieves a mean full-task success rate of 76.3%, outperforming the best baseline by 12.5 percentage points. Ablation studies and further analyses validate the effectiveness of the proposed design. Website is available at: https://mopa-policy.github.io/.
Read the full abstract ▾
All authors Guangyu Chen, Qiwei Liang, Shaolong Zhu, Tianxing Chen, Zikuan Xiao, Yifan Xie, Lingfeng Zhang, Ping Luo, Renjing Xu, Wenbo Ding
Computer Vision
Feature Recovery for Object Understanding After Irreversible Fire Damage
Aditi Tiwari, Sofia Stoica, Savya Khosla et al. (+2 more)
Objects in post-fire environments often undergo irreversible physical transformations that change their geometry, material state, and visual appearance. Detecting and identifying these remnants is critical for locating hazards, reconstructing pre-incident contents, and inventorying losses. Unlike standard image corruptions, these degradations affect the physical structure of the object itself. To study this setting, we introduce TRACE, a transformation-aware
benchmark for post-fire object understanding. TRACE contains 21.4K real-image-grounded synthetic scenes and paired object-level pristine-to-degraded progressions spanning 499 object identities across 189 categories. We define five tasks targeting localization and pre-degradation understanding: degraded-object detection, pristine-state recovery and
retrieval, original material recovery, pristine description generation, and functional
reasoning. Existing models degrade sharply with severity. From the least to the most severe level, RF-DETR mAP decreases by 71% relative, while InternVL3.5
retrieval R@1 falls from 93.85 to 28.11. To address this, we propose the Feature Recovery Module (FRM), a plug-and-play module that maps degraded encoder features to pristine-aligned representations while keeping the host frozen. Trained only with paired feature supervision, FRM improves scene-level detection, CLIP/SigLIP2 feature recovery, and all four object-level VLM tasks, with larger gains under more severe degradation. Across VLM hosts and severity levels, relative gains average 12.5% for
retrieval, 20.1% for material recovery, 13.2% for description generation, and 12.4% for functional
reasoning.
Read the full abstract ▾
All authors Aditi Tiwari, Sofia Stoica, Savya Khosla, David Forsyth, Heng Ji
Computer Vision
SenseNova-U1.5: Towards Native Unified Visual Intelligence
Haiwen Diao, Jiahao Wang, Chenjing Ding et al. (+62 more)
We launch SenseNova-U1.5, an 8B-MoT native unified
multimodal model that understands, reasons about, and generates visual content within an encoder-free and VAE-free architecture. We strengthen its visual interface through spatially coherent patch reconstruction and scale its training with carefully curated generation and editing data, improved task formulation, structural
prompt enhancement, and native resolutions of up to 4K. For post-training, we optimize specialized experts for visual aesthetics, bilingual text rendering, infographic generation, and image editing, and consolidate their capabilities through multi-expert on-policy
distillation. Across extensive evaluations, SenseNova-U1.5 largely advances image fidelity, text rendering, complex composition, multi-reference editing, and interleaved generation, while improving instruction following and preserving subject identity, geometry, and unmodified regions. Despite limited exposure to structured formats in its generation data, SenseNova-U1.5 generalizes effectively to long, complex, and structured visual instructions, further proving that
multimodal understanding can transfer to visual planning and creation. Together, these findings position native unified modelling as a promising path towards systems that perceive, reason and create within a fully end-to-end framework. We will open-source training code, including supervised fine-tuning, reinforcement learning, and on-policy
distillation.
Read the full abstract ▾
All authors Haiwen Diao, Jiahao Wang, Chenjing Ding, Hanming Deng, Jiangnan Chen, Ruixi Zhang, Ruohui Wang, Wenwen Tong, Xiangyu Fan, Yubo Wang, Yue Zhu, Yuwei Niu, Zhengqi Bai, Zhiqian Lin, Zhitao Yang, Zhongang Cai, Bo Yang, Chen Feng, Chengguang Lv, Guangjia Liu, Guanlin Wang, Hanyu Zhang, Haojia Yu, Hongcan Xiao, Hongli Wang, Huan Wu, Huaping Zhong, Jian Fang, Jianan Fan, Jiaqi Li, Jiefan Lu, Jing Zuo, Jingcheng Ni, Junxiang Xu, Linjun Dai, Mutian Xu, Peishen Yan, Penghao Wu, Ruijie Mao, Ruisi Wang, Shihao Bai, Shuang Yang, Shuya Yang, Shuyan Zheng, Silei Wu, Siying Li, Tao Chu, Tianbo Zhong, Tongxi Zhou, Weichao Luo, Weichen Fan, Wenhao Jia, Wenjie Gao, Xiangli Kong, Yan Li, Yang Yong, Zimo Wen, Zixuan Qian, Wenxiu Sun, Ruihao Gong, Quan Wang, Lewei Lu, Lei Yang, Ziwei Liu, Dahua Lin
Computer Vision · Cross-listed from cs.AI
MindTopo: Can Foundation Models Reason in Topological Space?
Yunfei Ge, Anbang Liu, Qineng Wang et al. (+9 more)
Spatial
reasoning depends not only on metric properties such as distance, angle, and shape, but also on topological relations that remain invariant under continuous deformation. Cognitive science identifies these relations as foundational to spatial understanding, yet foundation-model evaluations largely focus on metric or viewpoint-dependent relations. We introduce MindTopo, a
benchmark of topological intuition across five properties grounded in cognitive science and formal
topology: continuity, separation, order, enclosure, and
knots. MindTopo evaluates each property at two cognitive levels.
Reasoning asks a model to identify topological relations or infer how they change. Planning instantiates a foundation model as a closed-loop
agent whose policy selects environment actions. MindTopo contains 11,030 instances across 13 procedurally generated task types with controllable difficulty. We
benchmark 14 MLLMs and study
agent configurations augmented with image and
video generation, including 3
video generative models in planning settings. Every MLLM performs better on
reasoning than on planning, and the best-performing model remains far below observed human performance. On Qwen3-VL-2B-Instruct, supervised fine-tuning and reinforcement learning improve
reasoning more than planning. Generated observations retain local cues and reach plausible endpoints, but audited rollouts do not reliably follow environment dynamics or preserve
topology across transitions. Our website is at https://mind-topo.github.io/
Read the full abstract ▾
All authors Yunfei Ge, Anbang Liu, Qineng Wang, Johnalbert Garnica, Jianwen Lyu, Zihan Wang, Reuben Tan, Jianfeng Gao, Ruohan Zhang, Yining Hong, Jiajun Wu, Manling Li