Computer Vision
Reducing Hallucination in Multimodal Large Language Models through Hard Grounding Preference Supervision
Qinwu Xu
Hallucination remains a major challenge in
vision-
language models (VLMs), particularly when linguistically plausible responses are unsupported by visual evidence. We study whether
multimodal hallucination can be reduced by concentrating post-training supervision on hard grounding boundaries, where preferred and rejected responses are semantically close but differ in their support from observable visual evidence. Under a frozen visual encoder and cross-modal
alignment pathway, we first use supervised fine-tuning (SFT) to establish broad decoder-side
multimodal behavior and then construct hard grounding preference pairs for Direct Preference Optimization (DPO). These pairs target evidence utilization, calibration, and grounding consistency across fine-grained recognition, spatial
reasoning, OCR, ambiguous or insufficient evidence, and false-premise queries. The resulting DPO model consistently improves over its SFT initialization on DocVQA, TextVQA, MMBench, and VQAv2. Across approximately 839 additional
multimodal examples, it further achieves 6.2–8.2 percentage-point higher pairwise win rates under three independent LLM judges. We additionally release HardVQA-DPO, a curated and growing resource with more than 3K hard-grounding SFT examples and an initial set of DPO preference pairs.https://huggingface.co/
datasets/vlmgrounding/hardvqa-sft-dpo These results show that decoder-side adaptation alone can improve a meaningful subset of grounding failures under a fixed
multimodal representation, while preserving broader
multimodal capabilities.
Read the full abstract ▾
Computer Vision
Test-Time Hinting for Black-Box Vision-Language Models
Kaihua Hou, Abhijith Varma Mudunuri, Jiaxing Qiu et al. (+3 more)
Test-time scaling (TTS) methods have proven highly effective for LLMs, yet their application to
vision-
language models (VLMs) remains relatively underexplored. Existing VLM TTS methods largely require open-weight model access or expensive repeated sampling, and are evaluated primarily on
multimodal mathematical and scientific
reasoning benchmarks rather than natural-image understanding. In this paper, we propose Test-Time Hinting, a method that improves VLM performance with a single call to the target VLM and requires only black-box API access, which makes it broadly applicable to frontier closed-weight models. Our method is motivated by the observation that VLM errors tend to cluster around recurring failure patterns that are shared across models and predictable from the input alone. We therefore train a lightweight hint generator model to predict, for a given test input, which "hint" should be prepended to the
prompt, providing targeted contextual or procedural guidance that steers the VLM away from its characteristic failure modes without revealing the answer. We show that Test-Time Hinting significantly improves the accuracy of multiple closed-weight VLMs on natural-image VQA
benchmarks and that these gains generalize to unseen
benchmarks and VLMs without retraining the hint generator.
Read the full abstract ▾
All authors Kaihua Hou, Abhijith Varma Mudunuri, Jiaxing Qiu, Roxana Daneshjou, Thomas Hartvigsen, Ahmed Alaa
Computer Vision
Qinwu Xu, Yifan Jiang and Haoyu Ren
Optical character recognition (OCR) and
multilingual scene-text understanding remain challenging for
multimodal large
language models (MLLMs), particularly in real-world images containing small or degraded text, cluttered layouts, occlusion, handwriting, and complex typography. We present an OCR-aware
multilingual post-training framework that improves visual-text grounding in a general-purpose MLLM without requiring an external OCR engine, OCR-extracted text, or text bounding boxes at inference time. The framework combines large-scale
multilingual OCR supervision, approximately 5M additional
multilingual training samples, controlled synthetic OCR generation and in-image text translation, LoRA-based supervised fine-tuning (SFT), and lightweight OCR-oriented Chain-of-Thought prompting. On a held-out real-world
multilingual OCR
benchmark, OCR-SFT improves OCR completeness from 71.3 to 84.6, reduces
hallucination rate from 18.3% to 5.5%, and improves translation BLEU-1 from 52.3 to 80.2, with substantial
hallucination reductions under blur and rotation. Evaluation on public
benchmarks further shows gains on OCR-intensive tasks while largely preserving broader
multimodal capabilities; ablations show that SFT provides the primary improvement, with prompting offering smaller complementary gains. These results demonstrate that data-centric OCR-aware post-training provides a practical and scalable approach to improving
multilingual visual-text grounding in general-purpose MLLMs.
Read the full abstract ▾
Computer Vision
Visual Search Patterns in 3D Pancreatic Imaging: An Eye Tracking Study
Anna Anikina, Leila Khaertdinova, Trine Balschmidt et al. (+6 more)
Eye tracking has emerged as a powerful tool for examining visual perception and search strategies in various domains, including medicine. While it is relatively straightforward to apply in 2D settings, its use in 3D medical imaging remains challenging and not yet well explored. This gap is particularly relevant for radiology, where volumetric images such as computed
tomography (CT) scans are routinely read by medical experts. Radiologists typically interpret these images by navigating through hundreds of 2D slices, most often viewed in the axial projection. A taxonomy of eye movement data during
navigation through a CT volume could be valuable to understand how radiologists approach diagnostic tasks. As an example of the derived taxonomy, we asked two radiologists to search abdominal CTs of the pancreas. We collect eye tracking data and align eye gaze movements with slice
navigation to visualize the representation of the pancreas through volume and analyze clinicians' gaze behavior in both space and time.
Read the full abstract ▾
All authors Anna Anikina, Leila Khaertdinova, Trine Balschmidt, Michael B Andersen, Christoph F Müller, Erik GS Brandt, Henrik S Thomsen, Claudia Mello-Thoms, Bulat Ibragimov
Computer Vision
Mitigating Illumination-Induced Domain Shift in Night-Time Pedestrian Detection for Intelligent Vehicles using Annotation-Preserving Diffusion Augmentation
Franky George, Muhammad Khalid, Adil Khan et al. (+1 more)
Night-time pedestrian detection remains challenging because labelled night-time data are limited and large illumination differences make daytime-only trained detectors unreliable. Latent
diffusion models (LDMs) provide a powerful basis for image-to-image translation and cross-domain augmentation, but their effectiveness in safety-critical perception depends on whether detector-relevant objects and local semantic structure are preserved when translating between source and target domains. In this work, we present Contrastive-SDXL, a day-to-night augmentation framework for night-time pedestrian detection built on SDXL-Turbo and fine-tuned using Low-Rank Adaptation (LoRA). To preserve semantic correspondence between daytime inputs and translated night-time images, we introduce a patch-wise semantic contrastive loss guided by a pretrained DINOv2 encoder rather than generator encoder features. Multi-level DINOv2 self-
attention maps enforce both local and global semantic consistency, while an object consistency loss explicitly encourages pedestrian preservation. Contrastive-SDXL produces realistic night-time images, achieving a Frechet Inception Distance (FID) of 22.5. Detectors trained with our synthetic images obtain a 6-7% reduction in miss rate compared with a daytime-only baseline, approaching the performance of detectors trained on real night-time data. These results demonstrate that consistency-driven
diffusion augmentation can effectively support safety-critical night-time pedestrian detection.Specific
Read the full abstract ▾
All authors Franky George, Muhammad Khalid, Adil Khan, Koorosh Aslansefat
Computer Vision
Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations
Nicola Debole, Andrea Passerini, Stefano Teso et al. (+2 more)
Concept-bottleneck models (CBMs) are neural classifiers that compute predictions from high-level concepts extracted from the input. CBMs ensure stakeholders can understand the concepts – and the predictions they entail – by learning these from concept-level annotations, which are however seldom available. Recent CBM architectures work around this issue by obtaining annotations from
Vision-
Language Models (VLMs). While greatly broadening applicability, doing so can yield lower quality concepts and therefore less interpretable models. We strike for a middle ground by introducing
Vision-plus-Human-guided CBM (VH-CBM), a hybrid approach that exploits both VLMs and a small amount of dense annotations. VH-CBM employs a Gaussian Process in the VLM's
embedding space, which captures useful global information about the target domain, to propagate the expert's supervision to any target data point. Our empirical evaluation shows how VH-CBM predicts more accurate concepts than VLM-guided CBMs even when annotating as little as 1% of the data, while sporting better concept calibration and supporting active learning.
Read the full abstract ▾
All authors Nicola Debole, Andrea Passerini, Stefano Teso, Andrea Pugnana, Emanuele Marconato
Computer Vision
Hybrid Quantum-MambaVision: A Quantum-enhanced State Space Model for Calibrated Mixed-type Wafer Defect Detection
Satwik Sai Prakash Sahoo, Jyoti Prakash Sahoo, Ting Wang et al. (+1 more)
Extracting actionable knowledge from industrial visual data is fundamentally bottlenecked by extreme class imbalance and the prohibitive computational complexity of modern foundation models. In semi-conductor manufacturing, identifying multi-label wafer defects is a complex spatial data mining task where overlapping patterns obscure critical root-cause signals. While
Vision Transformers (ViTs) excel at global dependency extraction, their quadratic scaling renders them inefficient for high-throughput, real-time anomaly detection. To overcome these computational barriers, this paper introduces Hybrid
Quantum-MambaVision, a highly efficient architecture tailored for spatial knowledge discovery. We integrate a linear-complexity State-Space Model (SSM) backbone with a Parameterized
Quantum Context Adapter (QCA) and Low-Rank Adaptation (LoRA). The Mamba backbone efficiently captures long-range spatial dependencies, while the
quantum adapter maps compressed latent features into a high-dimensional
Hilbert space to disentangle complex, overlapping signatures. On the highly imbalanced MixedWM38
dataset, Hybrid
Quantum-MambaVision achieves exceptional multi-label classification performance, significantly reducing the error rate on complex multi-defect
topologies compared to classical baselines. The
quantum regularizer acts as a profound uncertainty calibrator, substantially reducing Maximum Calibration Error (MCE) and minimizing expected false-positive costs. This work establishes a scalable
Quantum-Classical hybrid paradigm for efficient representation learning in industrial data mining.
Read the full abstract ▾
All authors Satwik Sai Prakash Sahoo, Jyoti Prakash Sahoo, Ting Wang, Subrota Kumar Mondal
Computer Vision
When Vision Speaks for Sound
Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu et al. (+6 more)
Despite rapid progress in
video-capable MLLMs, we find that their apparent audio understanding in
videos is often
vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage
alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general
video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general
video and audio-visual QA
benchmarks.
Read the full abstract ▾
All authors Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai, Tinghui Zhu, Wendi Li, Yanan Xie, Muhao Chen, Peng Qi
Computer Vision
WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments
Haoren Zhao, Tianyi Chen and Zhen Wang
Multimodal Large
Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces. This paper identifies a critical reliability gap: state-of-the-art
agents face distinct robustness challenges in real-world desktop environments characterized by multi-window stacking, occlusion, and visual clutter. To address this, we introduce WinDeskGround, a novel
benchmark and synthesis framework tailored for evaluating GUI grounding robustness. Unlike static
datasets, our framework parametrically generates complex desktop scenarios by controlling window occlusion, layout density, and semantic similarity, thereby simulating the distribution shifts of authentic workflows. We construct a diverse meta-
dataset of 1,356 high-fidelity instruction-target pairs and conduct comprehensive evaluations of five leading MLLMs. Our results demonstrate that while top-tier
agents excel in simplified settings, their accuracy declines under partial occlusion. WinDeskGround provides a valuable
benchmark to facilitate the assessment and advancement of GUI
agent robustness in realistic environments. The code is available at https://github.com/ZZZhr-1/WinDeskGround.
Read the full abstract ▾
Computer Vision
FSCM: Frequency-Enhanced Spatial-Spectral Coupled Mamba for Infrared Hyperspectral Image Colorization
Tingting Liu, Yuan Liu, Guiping Chen et al. (+2 more)
Thermal infrared imaging is robust to illumination variations and smoke interference, making it important for all-weather perception. However, the lack of natural color and fine texture limits target recognition, human visual interpretation, and the transfer of visible-light models. Existing infrared colorization methods mainly rely on single-band images, where insufficient spectral cues may lead to structural distortion and semantic confusion. Although infrared hyperspectral images provide rich spectral responses and material information, existing single-band frameworks remain limited in modeling spatial-spectral coupling and weak texture details. To address these issues, this paper presents FSCM, a spectral-information-guided GAN framework. Within FSCM, a frequency-enhanced spatial-spectral state-space generator composed of cascaded FSB units is constructed. Each FSB integrates three complementary components: state-space modeling captures global spatial-spectral dependencies; the frequency enhancement module (FEM) combines multi-level wavelet decomposition and
Fourier gating to recover structural contours, directional high-frequency details, and global frequency responses; and the dual-stream hybrid gating module (DGM) integrates deformation-aware sampling with sparse
attention to enhance effective local structures and suppress background interference. Additionally, an online semantic
segmentation-guided loss is introduced to constrain the generated results, improving semantic consistency in complex road scenes. Experiments show that FSCM outperforms existing infrared colorization methods in visual quality and semantic fidelity.
Read the full abstract ▾
All authors Tingting Liu, Yuan Liu, Guiping Chen, Xiubao Sui, Qian Chen
Computer Vision · Cross-listed from cs.LG
Mask-Morph Graph U-Net: A Generalisable Mesh-Based Surrogate for Crashworthiness Field Prediction under Large Geometric Variation
Haoran Li, Tobias Lehrer, Yingxue Zhao et al. (+5 more)
Nonlinear finite element crash simulations are accurate but computationally expensive, limiting their use in iterative design optimisation. Machine-learning surrogate models based on graph neural networks (GNNs) offer a faster alternative. Message-passing GNNs are widely used for mesh simulation, and their shared node and edge update functions are relatively generalisable across varying graph structures. By contrast, non-shareable edge-specific aggregation layers can capture nonlinear relationships more accurately but usually require fixed graph connectivity, which limits generalisability. This paper presents Mask-Morph Graph U-Net (MMGUNet), a practical approach to addressing the limitation of hierarchical Graph U-Net architectures that use edge-specific downsampling and upsampling layers. Fixed coarse graph connectivity is required for edge-specific layers. To retain this while improving spatial correspondence, the proposed method morphs the coarsened graph hierarchy to each input mesh using feature-aligned barycentric parameterisation before constructing cross-graph edges. It further applies node masking during supervised pretraining, followed by parameter-efficient fine-tuning in which high-parameter edge-specific layers are frozen. The proposed approach is evaluated in in-distribution, out-of-distribution, and cross-component transfer settings using mean Euclidean distance and maximum intrusion percentage error. Results show that coarse-graph morphing improves test accuracy relative to a fixed-coarse-graph baseline, while masked supervised pretraining reduces the train-test discrepancy and improves data efficiency during transfer. The proposed model also achieves lower prediction error compared with external baselines. These results demonstrate a practical route toward reusable, data-efficient mesh-based surrogate modelling for crashworthiness design exploration.
Read the full abstract ▾
All authors Haoran Li, Tobias Lehrer, Yingxue Zhao, Haosu Zhou, Philipp Stocker, Tobias Pfaff, Marcus Wagner, Nan Li
Computer Vision · Cross-listed from cs.RO
MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving
Rajeev Yasarla, Deepti Hegde, Hsin-Pai Cheng et al. (+9 more)
Vision-
language-action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings due to being trained under traditional imitation learning framework. Existing closed-loop supervision approaches lack scalability and fail to completely model a reactive environment. We propose MAPLE, a novel framework for reactive, multi-
agent rollout of a dynamic driving scenario in the latent space of the VLA model. The ego vehicle and nearby traffic
agents are independently controlled over multi-step horizons, while being reactive to other
agents in the scene, enabling closed-loop training. MAPLE consists of two training stages: (1) supervised fine-tuning on the latent rollouts based on ground-truth trajectories, followed by (2) reinforcement learning with global and
agent -specific rewards that encourage safety, progress, and interaction realism. We further propose diversity rewards that encourage the model to generate planning behaviors that may not be present in logged driving data. Notably, our closed-loop training framework is scalable and does not require external simulators, which can be computationally expensive to run and have limited visual fidelity to the real-world. MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-
agent play for robust E2E autonomous driving systems.
Read the full abstract ▾
All authors Rajeev Yasarla, Deepti Hegde, Hsin-Pai Cheng, Shizhong Han, Yunxiao Shi, Meysam Sadeghigooghari, Hanno Ackermann, Litian Liu, Pranav Desai, Fatih Porikli, Mohammad Ghavamzadeh, Hong Cai
Computer Vision
CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers
Zhuojin Li, Hsin-Pai Cheng, Hong Cai et al. (+2 more)
Diffusion Transformers (DiTs) deliver remarkable image and
video generation quality but incur high computational cost, limiting scalability and on-device deployment. We introduce CoReDiT, a structured
token pruning framework for DiTs across
vision tasks. CoReDiT uses a linear-time spatial coherence score to estimate local redundancy in the latent
token lattice and skips high coherence (redundant)
tokens in self-
attention. To maintain a dense representation and avoid visual discontinuities, we reconstruct skipped
attention outputs via coherence-guided aggregation of spatially neighboring retained
tokens. We further introduce a progressive, block-adaptive
pruning schedule that increases
pruning gradually and allocates larger budgets to blocks and denoising steps with higher redundancy. Across state-of-the-art
diffusion backbones including PixArt-{\alpha} and MagicDrive-V2, CoReDiT achieves up to 55% self-
attention FLOPs reduction and inference speedups of 1.33x on cloud GPUs and 1.72x on mobile NPUs, while maintaining high visual quality. Notably, CoReDiT also increases on-device
memory head-room, enabling higher-resolution generation.
Read the full abstract ▾
All authors Zhuojin Li, Hsin-Pai Cheng, Hong Cai, Shizhong Han, Fatih Porikli
Computer Vision
You Only Landmark Once: Lightweight U-Net Face Super Resolution with YOLO-World Landmark Heatmaps
Riccardo Carraro, Anna Briotto, Endi Hysa et al. (+2 more)
Face image super-resolution aims to recover high-resolution facial images from severely degraded inputs. Under extreme upscaling factors, fine facial details are often lost, making accurate reconstruction challenging. Existing methods typically rely on heavy network architectures,
adversarial training schemes, or separate
alignment networks, increasing model complexity and computational cost. To address these issues, we propose a lightweight U-Net based-architecture designed to reconstructs $128{ \times }128$ facial images from severely degraded $16{ \times }16$ inputs, achieving an $8 \times $ magnification. A key contribution is a novel auxiliary-training-free supervision strategy that leverages heatmaps generated by YOLO-World, an open-vocabulary object detector, to localize key facial features such as eyes, nose, and mouth. These heatmaps are converted into spatial weights to form a heatmap-guided loss that emphasizes reconstruction errors in semantically important regions. Unlike prior methods that require dedicated landmark or
alignment networks, our approach directly reuses detector outputs as supervision, maintaining an efficient training and inference pipeline. Experiments on the aligned CelebA
dataset demonstrate that the proposed loss consistently improves quantitative metrics and produces sharper, more realistic reconstructions. Overall, our results show that lightweight networks can effectively exploit detection-driven priors for perceptually convincing extreme upscaling, without
adversarial training or increased computational cost.
Read the full abstract ▾
All authors Riccardo Carraro, Anna Briotto, Endi Hysa, Marco Fiorucci, Lamberto Ballan
Computer Vision
Rethinking the Good Enough Embedding for Easy Few-Shot Learning
Michael Karnes and Alper Yilmaz
The field of deep visual recognition is undergoing a paradigm shift toward universal representations. The Platonic Representation Hypothesis suggests that diverse architectures trained on massive
datasets are converging toward a shared, "ideal" latent space. This again raises a critical question: is a "Good
Embedding All You Need?" In this paper, we leverage this convergence to demonstrate that off-the-shelf
embeddings are inherently "good enough" for complex tasks, rendering intensive task-specific fine-tuning unnecessary. We explore this hypothesis within the few-shot learning framework, proposing a straightforward, non-parametric pipeline that entirely bypasses backpropagation. By utilizing a k-Nearest Neighbor classifier on frozen DINOv2-L features, we conduct a layer-wise characterization to identify an optimal feature extraction. We further demonstrate that
manifold refinement via PCA and ICA provides a beneficial regularizing effect. Our results across four major
benchmarks demonstrate that our approach consistently surpasses sophisticated meta-learning
algorithms, achieving state-of-the-art performance.
Read the full abstract ▾
Computer Vision
TeDiO: Temporal Diagonal Optimization for Training-Free Coherent Video Diffusion
Nurislam Tursynbek, Zhiqiang Lao, Heather Yu et al. (+2 more)
Recent text-to-
video diffusion transformers generate visually compelling frames, yet still struggle with temporal coherence, often producing flickering, drifting, or unstable motion. We show that these failures leave a clear imprint inside the model: incoherent
videos consistently exhibit irregular, fragmented temporal diagonals in their intermediate self-
attention maps, whereas stable motion corresponds to smooth, band-diagonal patterns. Building on this observation, we introduce TeDiO, a training-free, inference-time method that reinforces temporal consistency by regularizing these internal
attention patterns. TeDiO estimates diagonal smoothness, identifies unstable regions, and performs lightweight latent updates that promote coherent frame-to-frame dynamics, without modifying model weights or using external motion supervision. Across multiple
video diffusion models (e.g., Wan2.1, CogVideoX), TeDiO delivers markedly smoother motion while preserving per-frame visual quality, offering an efficient plug-and-play approach to improving dynamic realism in modern
video generation systems.
Read the full abstract ▾
All authors Nurislam Tursynbek, Zhiqiang Lao, Heather Yu, Gedas Bertasius, Marc Niethammer
Computer Vision
PanoPlane: Plane-Aware Panoramic Completion for Sparse-View Indoor 3D Gaussian Splatting
Adil Qureshi, Dongki Jung, Jaehoon Choi et al. (+1 more)
We present PanoPlane, an approach for high-fidelity sparse-view indoor novel view synthesis that reconstructs closed room geometry via panoramic scene completion. Unlike perspective-based methods that generate training views from limited fields of view, PanoPlane leverages $360^{\circ}$ panoramic completion to condition the generative process on the full spatial layout. We propose Layout Anchored
Attention Steering, a training-free mechanism that steers
attention within the
diffusion model's internal representation toward scene's detected planar surfaces at inference time. By directing each unobserved region's
attention toward geometrically consistent observed content, our method replaces unconstrained
hallucination with grounded surface extrapolation. The resulting panoramic completions provide supervision for 3D Gaussian Splatting, enabling accurate novel-view synthesis across unobserved regions from as few as three input views. Experiments on Replica, ScanNet++, and Matterport3D demonstrate state-of-the-art novel view synthesis quality across 3, 6, and 9 input views, achieving up to $+17.8\%$ improvement in PSNR over the current state-of-the-art baseline without any training or fine-tuning of the
diffusion model.
Read the full abstract ▾
All authors Adil Qureshi, Dongki Jung, Jaehoon Choi, Dinesh Manocha
Computer Vision · Cross-listed from eess.IV
Keyed Nonlinear Transform: Lightweight Privacy-Enhancing Feature Sharing for Medical Image Analysis
Haebom Lee and Gyeongjung Kim
Feature sharing via split inference offers a lightweight alternative to
federated learning for resource-constrained hospitals, but transmitted features still leak patient identity information and lack practical mechanisms for controlled feature sharing. We propose Keyed Nonlinear Transform (KNT), a drop-in feature transformation that applies key-conditioned obfuscation to intermediate representations. KNT reduces re-identification AUC from 0.635 to 0.586, corresponding to a 36% reduction in above-chance identity signal, while introducing only 0.15 ms CPU overhead, without backbone retraining, and preserving classification performance within 1.0 pp. Our analysis shows that KNT's nonlinear transform prevents closed-form inversion and shifts recovery to iterative gradient-based optimization under full key compromise, substantially increasing inversion difficulty. The same transform generalizes to dense prediction tasks, incurring only a 4.4 pp Dice reduction on skin-lesion
segmentation without retraining. These results position KNT as a practical and efficient
privacy layer for split inference deployments.
Read the full abstract ▾
Computer Vision
ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflows
Alvaro Lopez Pellicer, Plamen Angelov, Marwan Bukhari et al. (+3 more)
While interpretable prototype networks offer compelling case-based
reasoning for clinical diagnostics, their raw continuous outputs lack the semantic structure required for medical documentation. Bridging this gap via standard
Retrieval-Augmented Generation (RAG) routinely triggers “
retrieval sycophancy,” where Large
Language Models (LLMs) hallucinate post-hoc rationalizations to align with visual predictions. We introduce ProtoMedAgent, a framework that formalizes
multimodal clinical reporting as an iterative, zero-gradient test-time optimization problem over a strict neuro-symbolic bottleneck. Operating on a frozen prototype backbone, we distill latent visual and tabular features into a discrete semantic
memory. Online generation is strictly constrained by exact set-theoretic differentials and a reflective Scribe-Critic loop, mathematically precluding unsupported narrative claims. To safely bound data disclosure, we introduce a semantic
privacy gate governed by $k$-anonymity and $\ell$-diversity. Evaluated on a 4,160-patient clinical cohort, ProtoMedAgent achieves 91.2% Comparison Set Faithfulness where it fundamentally outperforms standard RAG (46.2%). ProtoMedAgent additionally leverages a binding $\ell$-diversity phase transition to systematically reduce artifact-level membership inference risks by an absolute 9.8%.
Read the full abstract ▾
All authors Alvaro Lopez Pellicer, Plamen Angelov, Marwan Bukhari, Yi Li, Eduardo Soares, Jemma Kerns
Computer Vision
SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection
Sandro Papais, Lezhou Feng, Charles Cossette et al. (+1 more)
Vision Transformers (ViTs) enable strong multi-view 3D detection but are limited by high inference latency from dense
token and query processing across multiple views and large 3D regions. Existing sparsity methods, designed mainly for 2D
vision, prune or merge image
tokens but do not extend to full-model sparsity or address 3D object queries. We introduce SToRe3D, a relevance-aligned sparsity framework that jointly selects 2D image
tokens and 3D object queries while storing filtered features for reactivation. Mutual 2D-3D relevance heads allocate compute to driving-critical content and preserve other
embeddings. Evaluated on nuScenes and our new nuScenes-Relevance
benchmark, SToRe3D achieves up to 3x faster inference with marginal accuracy loss, establishing real-time large-scale ViT-based 3D detection while maintaining accuracy on planning-critical
agents.
Read the full abstract ▾
All authors Sandro Papais, Lezhou Feng, Charles Cossette, Lingting Ge
Computer Vision
A Cascaded Edge-Cloud Architecture for Automated Diabetic Retinopathy Screening
Nishi Doshi and Shrey Shah
Diabetic Retinopathy (DR) is one of the leading causes of preventable blindness, and automated screening can help extend specialist capacity in resource-constrained clinical workflows. Cloud-based deep learning systems can provide strong grading performance, but they require image upload and reliable connectivity. We evaluate a two-tier edge-cloud cascade on the public APTOS 2019 Blindness Detection
dataset. Tier 1 runs a lightweight MobileNetV3-small model locally to triage Referable DR (Classes 2-4) versus Non-referable DR (Classes 0-1). Tier 2 runs a RETFound-DINOv2 model in the cloud for ordinal severity grading only on images flagged as referable by Tier 1. On a stratified APTOS test split of 733 images, Tier 1 reaches 98.99% sensitivity and 84.37% specificity at a validation-tuned high-sensitivity threshold. The deployment-oriented safety-floor cascade forwards 49.52% of test images to Tier 2, reducing image-count cloud calls by 50.48% relative to a cloud-only model. In the deployed 4-class output space (Class 0-1 / Class 2 / Class 3 / Class 4), the safety-floor cascade obtains 77.49% accuracy and 0.7938 quadratic weighted kappa, while the cloud-only baseline obtains 80.76% accuracy and 0.8184 kappa. A Tier-2-override comparator improves accuracy to 80.49% and kappa to 0.8167, but downgrades 12 true referable cases after Tier 1 correctly forwarded them. We also report repeated stratified bootstrap variability, a prevalence-dependent cloud-call curve, raw-byte upload estimates, and CPU latency for the local tier. Index Terms: Diabetic retinopathy screening, retinal image analysis, edge-cloud computing, cascaded inference, teleophthalmology, clinical decision support, medical artificial intelligence, resource-constrained healthcare
Read the full abstract ▾
Computer Vision
DUET: Dual-Paradigm Adaptive Expert Triage with Single-cell Inductive Prior for Spatial Transcriptomics Prediction
Junchao Zhu, Ruining Deng, Junlin Guo et al. (+11 more)
Inferring spatially resolved gene expression from histology images offers a cost-effective complement to spatial transcriptomics (ST). However, existing methods reduce this task to a simple morphology-to-expression mapping, where visual similarity does not guarantee molecular consistency. Meanwhile, single-
cell data has amassed rich resources far surpassing the scale of ST data, yet it remains underexplored in
vision-omics modeling. Furthermore, current approaches commit to a monolithic paradigm with bottlenecks, unable to balance expressive flexibility with biological fidelity. To bridge these gaps, we propose DUET, a novel dual-paradigm framework that synergizes parametric prediction and
memory-based
retrieval under cellular inductive priors. DUET implements a parallel regression-
retrieval paradigm, adaptively reconciling the outputs of its complementary pathways. To mitigate aleatoric
vision ambiguity, we incorporate large-scale single-
cell references to impose molecular states as biological constraints for faithful learning. Building upon structural refinement, we further design a lightweight adapter to dynamically assign branch preference across spatial contexts to achieve optimal performance. Extensive experiments on three public
datasets across varied gene scales demonstrate that DUET achieves SOTA performance, with consistent gains contributed by each proposed component. Code is available at https://github.com/Junchao-Zhu/DUET
Read the full abstract ▾
All authors Junchao Zhu, Ruining Deng, Junlin Guo, Tianyuan Yao, Chongyu Qu, Juming Xiong, Zhengyi Lu, Yanfan Zhu, Marilyn Lionts, Yuechen Yang, Yu Wang, Shilin Zhao, Haichun Yang, Yuankai Huo
Computer Vision
Venus-DeFakerOne: Unified Fake Image Detection & Localization
GuangJian Team
In recent years, the rapid
evolution of generative AI has fundamentally reshaped the paradigm of image forgery, breaking the traditional boundaries between document editing, natural image manipulation, DeepFake generation, and full-image AIGC synthesis. Despite this shift toward unified forgery generation, existing research in Fake Image Detection and Localization (FIDL) remains fragmented. This creates a mismatch between increasingly unified forgery generation mechanisms and the domain-specific detection paradigm. Bridging this mismatch poses two key challenges for FIDL: understanding cross-domain artifacts transfer and interference, and building a high-capacity unified foundation model for joint detection and localization. To address these challenges, we propose DeFakerOne, a data-centric, unified FIDL foundation model integrating InternVL2 and SAM2. DeFakerOne enables simultaneous image-level detection and pixel-level forgery localization across diverse scenarios. Extensive experiments demonstrate that DeFakerOne achieves state-of-the-art performance, outperforming baselines on 39 forgery detection
benchmarks and 9 localization
benchmarks. Furthermore, the model exhibits superior robustness against real-world perturbations and state-of-the-art generators such as GPT-Image-2. Finally, we provide a systematic analysis of data scaling laws, cross-domain artifacts transfer-interference patterns, the necessity of fine-grained supervision, and the original resolution artifacts preservation, highlighting the design principles for scalable, robust, and unified FIDL.
Read the full abstract ▾
Computer Vision
CurveBench: A Benchmark for Exact Topological Reasoning over Nested Jordan Curves
Amirreza Mohseni, Mona Mohammadi, Morteza Saghafian et al. (+1 more)
We introduce CurveBench, a
benchmark for hierarchical topological
reasoning from visual input. CurveBench consists of 756 images of pairwise non-intersecting Jordan curves across easy, polygonal, topographic-inspired, maze-like, and dense counting configurations. Each image is annotated with a rooted tree encoding the containment relations between planar regions. We formulate the task as structured prediction: given an image, a model must recover the full rooted containment tree induced by the curves. Despite the visual simplicity of the task, the strongest evaluated model, Gemini 3.1 Pro, achieves only 71.1% tree-generation accuracy on CurveBench-Easy and 19.1% on CurveBench-Hard. We further demonstrate
benchmark utility through RLVR-style fine-tuning of open-weight
vision-
language models. Our trained Qwen3-VL-8B model improves over Qwen-3-VL-8B-Thinking from 2.8% to 33.3% tree-generation accuracy on CurveBench-Easy, exceeding GPT-5.4 and Claude Opus 4.5 under our evaluation protocol. The remaining gap, especially on CurveBench-Hard, shows that exact
topology-aware visual
reasoning remains far from solved.
Read the full abstract ▾
All authors Amirreza Mohseni, Mona Mohammadi, Morteza Saghafian, Naser Talebizadeh Sardari
Computer Vision · Cross-listed from cs.AI
Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
Haozhe Wang, Qixin Xu, Changpeng Wang et al. (+4 more)
Achieving robust perception-
reasoning synergy is a central goal for advanced
Vision-
Language Models (VLMs). Recent advancements have pursued this goal via architectural designs or agentic workflows. However, these approaches are often limited by static textual
reasoning or complicated by the significant compute and engineering burden of external agentic complexity. Worse, this heavy investment does not yield proportional gains, often witnessing a "seesaw effect" on perception and
reasoning. This motivates a fundamental rethinking of the true bottleneck. In this paper, we argue that the root cause of this trade-off is an ambiguity in modality credit assignment: when a VLM fails, is it due to flawed perception ("bad seeing") or flawed logic ("bad thinking")? To resolve this, we introduce a reinforcement learning framework that improves perception-
reasoning synergy by reliably rewarding the perception fidelity. We explicitly decompose the generation process into interleaved perception and
reasoning steps. This decoupling enables targeted supervision on perception. Crucially, we introduce Perception Verification (PV), leveraging a "blindfolded
reasoning" proxy to reward perceptual fidelity independently of
reasoning outcomes. Furthermore, to scale training across free-form VL tasks, we propose Structured Verbal Verification, which replaces high-variance LLM judging with structured algorithmic execution. These techniques are integrated into a Modality-Aware Credit Assignment (MoCA) mechanism, which routes rewards to the specific source of error – either bad seeing or bad thinking – enabling a single VLM to achieve simultaneous performance gains across a wide task spectrum.
Read the full abstract ▾
All authors Haozhe Wang, Qixin Xu, Changpeng Wang, Taofeng Xue, Chong Peng, Wenhu Chen, Fangzhen Lin
Computer Vision
Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation
Kieran Carrigg, Sigur de Vries, Amirhossein Sadough et al. (+1 more)
Vision Transformers (ViTs) achieve state-of-the-art performance on challenging
vision tasks, but their deployment on edge devices is hindered by the computational complexity and global reduction bottleneck imposed by layer normalization. Recent methods attempt to bypass this by replacing normalization layers with hardware-friendly scalar approximations. However, these homogeneous replacements do not optimally fit to all layers' behaviour and rely on expensive model retraining. In this work, we propose a highly efficient, hardware-aware framework that utilizes genetic programming (GP) to evolve heterogeneous, layer-specific scalar functions directly from pre-trained weights. Coupled with a novel post-training re-
alignment strategy, our approach eliminates the need to retrain models from scratch entirely. Our evolved expressions accurately approximate the target normalization behaviours, capturing $90$-$93\%$ of the variance ($R^2$) compared to only $70$-$76\%$ for homogeneous baselines, allowing our modified architecture to recover $84.32\%$ Top-1 ImageNet-1K accuracy on ViT-B and $85.74\%$ on ViT-L in only 20 epochs. By retaining near-baseline accuracy while eliminating the global reduction bottleneck, our approach achieves a strict reduction in both arithmetic complexity and off-chip
memory traffic compared to standard LayerNorm, removing a primary barrier to the efficient deployment of ViTs on edge accelerators.
Read the full abstract ▾
All authors Kieran Carrigg, Sigur de Vries, Amirhossein Sadough, Marcel van Gerven
Computer Vision
PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
Wei Dong, Han Zhou, Terry Ji et al. (+6 more)
Adverse weather removal (AWR) in real-world images remains challenging due to heterogeneous and unseen degradations, while distortion-driven training often yields overly smooth results. We propose PVRF, a unified framework that integrates zero-shot soft weather perceptions with velocity-constrained rectified-flow refinement. PVRF introduces an AWR-specific question answering module (AWR-QA) that uses frozen
vision–
language models (VLMs) to estimate soft probabilities of weather types and low-level attribute scores. These perceptions condition restoration networks via attribute-modulated normalization (AMN) and weather-weighted adapters (WWA), producing an anchor estimate for refinement. We then learn a terminal-consistent residual rectified flow with perception-adaptive source perturbation and a terminal-consistent velocity parameterization to stabilize learning near the terminal regime. Extensive experiments show that PVRF improves both fidelity and perceptual quality over state-of-the-art baselines, with strong cross-
dataset generalization on single and combined degradations. Code will be released at https://github.com/dongw22/PVRF.
Read the full abstract ▾
All authors Wei Dong, Han Zhou, Terry Ji, Guanhua Zhao, Shahab Asoodeh, Yulun Zhang, Guangtao Zhai, Jun Chen, Xiaohong Liu
Computer Vision · Cross-listed from cs.SD
Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study
Wuao Liu, Mustafa Chasmai, Subhransu Maji et al. (+1 more)
Bioacoustic recognition requires fine-grained acoustic understanding to distinguish similar-sounding species. However, many large-scale data repositories such as iNaturalist are weakly annotated, often with only a single positive species label per recording, making supervised learning particularly challenging. Inspired by advances in computer
vision, recent approaches have shifted toward self-supervised learning to capture the underlying structure of audio without relying on exhaustive annotations. In particular, masked autoencoders (MAE) have shown strong transferability on massive audio corpora, yet their effectiveness in more modest bioacoustic settings remains underexplored. In this work, we conduct a systematic study of MAE pretraining for species classification on iNatSounds, analyzing the impacts of pretraining data scale, domain specificity, data curation, and transfer strategies. Consistent with prior work, we find that models pretrained on diverse general audio data achieve the best transfer performance on iNatSounds. Contrary to observations from large-scale audio
benchmarks, we find that (1) additional masked reconstruction pretraining on domain-specific data provides limited benefits and may even degrade performance relative to off-the-shelf models, and (2) selective data filtering offers a negligible advantage when the overall data scale is limited. Our results indicate that, in moderate-sized fine-grained bioacoustic settings, pretraining scale dominates objective design. These findings further clarify when MAE-based pretraining is effective and provide practical guidance for model selection under limited supervision.
Read the full abstract ▾
All authors Wuao Liu, Mustafa Chasmai, Subhransu Maji, Grant Van Horn
Computer Vision
Unified Pix Token And Word Token Generative Language Model
Haun Leung and ZiNan Wang
Since the emergence of
Vision Transformer (ViT), it has been widely used in generative
language model and generative visual model. Especially in the current state-of-art open source
multimodal models, ViT obtained by CLIP or SigLIP method serves as the
vision encoder backbone to help them acquire visual understanding capabilities. But this method leads to limitations in visual understanding for details, such as difficulty in recognizing small text or numbers in images. To address these issues, we propose a new model to unify pix
token and word
token into the generative
language model. The new model also features with each pix of image having its own
token embedding, color folding, global conditional
attention approximation and image unsupervised pretraining. We conducted image unsupervised pretraining experiments using our new model to explore its potential. The experimental results show that it has good performance even in small model and with limited training data. We believe our model also conforms to the scaling law, as long as model parameters and training data increased, its performance will continue to improve.
Read the full abstract ▾
Computer Vision
CineMesh4D: Personalized 4D Whole Heart Reconstruction from Sparse Cine MRI
Xiaoyue Liu, Xiaohan Yuan, Mark Y Chan et al. (+2 more)
Accurate 3D+t whole-heart mesh reconstruction from cine MRI is a clinically crucial yet technically challenging task. The difficulty of this task arises from two coupled factors: inherently sparse sampling of 3D cardiac anatomy by 2D image slices and the tight coupling between cardiac shape and motion. Current cardiac image-to-mesh approaches typically reconstruct only a subset of cardiac chambers or a single phase of the cardiac cycle. In this work, we propose CineMesh4D, a novel end-to-end 4D (3D+t) pipeline that directly reconstructs patient-specific whole-heart mesh from multi-view 2D cine MRI via cross-domain mapping. Specifically, we introduce a differentiable rendering loss that enables supervision of 3D+t whole-heart mesh from multi-view sparse contours of cine MRI. Furthermore, we develop a dual-context temporal block that fuses global and local cardiac temporal information to capture high-dimensional sequential patterns. In quantitative and qualitative evaluations, CineMesh4D outperforms existing approaches in terms of reconstruction quality and motion consistency, providing a practical pathway for personalized real-time cardiac assessment. The code will be publicly released once the manuscript is accepted.
Read the full abstract ▾
All authors Xiaoyue Liu, Xiaohan Yuan, Mark Y Chan, Ching-Hui Sia, Lei Li