Computer Vision · Cross-listed from cs.CL
Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answering
Jongha Kim, Byungoh Ko, Jeehye Na et al. (+2 more)
Despite the remarkable capabilities of Large
Vision Language Models (LVLMs), they still lack detailed knowledge about specific entities.
Retrieval-augmented Generation (RAG) is a widely adopted solution that enhances LVLMs by providing additional contexts from an external Knowledge Base. However, we observe that previous decoding methods for RAG are sub-optimal as they fail to sufficiently leverage multiple relevant contexts and suppress the negative effects of irrelevant contexts. To this end, we propose Relevance-aware Multi-context Contrastive Decoding (RMCD), a novel decoding method for RAG. RMCD outputs a final prediction by combining outputs predicted with each context, where each output is weighted based on its relevance to the question. By doing so, RMCD effectively aggregates useful information from multiple relevant contexts while also counteracting the negative effects of irrelevant ones. Experiments show that RMCD consistently outperforms other decoding methods across multiple LVLMs, achieving the best performance on three knowledge-intensive visual question-answering
benchmarks. Also, RMCD can be simply applied by replacing the decoding method of LVLMs without additional training. Analyses also show that RMCD is robust to the
retrieval results, consistently performing the best across the weakest to the strongest
retrieval results. Code is available at https://github.com/mlvlab/RMCD.
Read the full abstract ▾
All authors Jongha Kim, Byungoh Ko, Jeehye Na, Jinsung Yoon, Hyunwoo J. Kim
Computer Vision · Cross-listed from cs.LG
NavFormer: IGRF Forecasting in Moving Coordinate Frames
Yoontae Hwang, Dongwoo Lee, Minseok Choi et al. (+4 more)
Triad magnetometer components change with sensor attitude even when the IGRF total intensity target stays invariant. NavFormer forecasts this invariant target with rotation invariant scalar features and a Canonical SPD module that stabilizes the spectrum of window level second moments of the triads without sign discontinuities. The module builds a canonical frame from a Gram
matrix per window and applies state dependent spectral scaling in the original coordinates. Experiments across five flights show lower error than strong baselines in standard training, few shot training, and zero shot transfer. The code is available at: https://anonymous.4open.science/r/NavFormer-Robust-IGRF-
Forecasting-for-Autonomous-Navigators-0765
Read the full abstract ▾
All authors Yoontae Hwang, Dongwoo Lee, Minseok Choi, Heechan Park, Yong Sup Ihn, Daham Kim, Deok-Young Lee
Computer Vision
IMSAHLO: Integrating Multi-Scale Attention and Hybrid Loss Optimization Framework for Robust Neuronal Brain Cell Segmentation
Ujjwal Jain, Oshin Misra, Roshni Chakraborty et al. (+1 more)
Accurate
segmentation of neuronal
cells in fluorescence microscopy is a fundamental task for quantitative analysis in computational neuroscience. However, it is significantly impeded by challenges such as the coexistence of densely packed and sparsely distributed
cells, complex overlapping morphologies, and severe class imbalance. Conventional deep learning models often fail to preserve fine topological details or accurately delineate boundaries under these conditions. To address these limitations, we propose a novel deep learning framework, IMSAHLO (Integrating Multi-Scale
Attention and Hybrid Loss Optimization), for robust and adaptive neuronal
segmentation. The core of our model features Multi-Scale Dense Blocks (MSDBs) to capture features at various receptive fields, effectively handling variations in
cell density, and a Hierarchical
Attention (HA) mechanism that adaptively focuses on salient morphological features to preserve Region of Interest (ROI) boundary details. Furthermore, we introduce a novel hybrid loss function synergistically combining Tversky and Focal loss to combat class imbalance, alongside a
topology-aware Centerline Dice (clDice) loss and a Contour-Weighted Boundary loss to ensure topological continuity and precise separation of adjacent
cells. Large-scale experiments on the public Fluorescent Neuronal
Cells (FNC)
dataset demonstrate that our framework outperforms state-of-the-art architectures, achieving precision of 81.4%, macro F1 score of 82.7%, micro F1 score of 83.3%, and balanced accuracy of 99.5% on difficult dense and sparse cases. Ablation studies validate the synergistic benefits of multi-scale
attention and hybrid loss terms. This work establishes a foundation for generalizable
segmentation models applicable to a wide range of biomedical imaging modalities, pushing AI-assisted analysis toward high-throughput neurobiological pipelines.
Read the full abstract ▾
All authors Ujjwal Jain, Oshin Misra, Roshni Chakraborty, Mahua Bhattacharya
Computer Vision
Muhammad Imran and Yugyung Lee
Vision-
Language Models (VLMs) demonstrate impressive capabilities across
multimodal tasks, yet exhibit systematic spatial
reasoning failures, achieving only 49% (CLIP) to 54% (BLIP-2) accuracy on basic directional relationships. For safe deployment in robotics and autonomous systems, we need to predict when to trust VLM spatial predictions rather than accepting all outputs. We propose a
vision-based confidence estimation framework that validates VLM predictions through independent
geometric verification using object detection. Unlike text-based approaches relying on self-assessment, our method fuses four signals via gradient boosting:
geometric alignment between VLM claims and coordinates, spatial ambiguity from overlap, detection quality, and VLM internal uncertainty. We achieve 0.674 AUROC on BLIP-2 (34.0% improvement over text-based baselines) and 0.583 AUROC on CLIP (16.1% improvement), generalizing across generative and classification architectures. Our framework enables selective prediction: at 60% target accuracy, we achieve 61.9% coverage versus 27.6% baseline (2.2x improvement) on BLIP-2. Feature analysis reveals
vision-based signals contribute 87.4% of model importance versus 12.7% from VLM confidence, validating that external
geometric verification outperforms self-assessment. We demonstrate reliable scene graph construction where confidence-based
pruning improves precision from 52.1% to 78.3% while retaining 68.2% of edges.
Read the full abstract ▾
Computer Vision
PSSF: Early osteoarthritis detection using physical synthetic knee X-ray scans and AI radiomics models
Abbas Alzubaidi and Ali Al-Bayaty
Knee osteoarthritis (OA) is a major cause of disability worldwide and is still largely assessed using subjective radiographic grading, most commonly the Kellgren-Lawrence (KL) scale. Artificial intelligence (AI) and radiomics offer quantitative tools for OA assessment but depend on large, well-annotated image
datasets, mainly X-ray scans, that are often difficult to obtain because of
privacy, governance and resourcing constraints. In this research, we introduce a physics-based synthetic simulation framework (PSSF) to fully generate controllable X-ray scans without patients' involvement and violating their
privacy and institutional constraints. This PSSF is a 2D X-ray projection simulator of anteroposterior knee radiographs from a parametric anatomical model of the distal femur and proximal tibia. Using PSSF, we create a virtual cohort of 180 subjects (260 knees), each is imaged under three protocols (reference, low-dose, and geometry-shift). Medial joint regions are automatically localized, preprocessed, and processed with the Image Biomarker Standardisation Initiative (IBSI). Practically, three machine learning (ML) models are utilized, logistic regression, random forest, and gradient boosting, to train binary (KL-like "0" vs. "2") and three-class (0-2) prediction radiographic images. Robustness is assessed within IBSI protocol, cross-protocol, and multi-protocol scenarios. Finally, features stability is then evaluated using intraclass correlation coefficients across acquisition changes.
Read the full abstract ▾
Computer Vision
Yuxi Liu, Yipeng Hu, Zekun Zhang et al. (+2 more)
While
Diffusion Transformers (DiTs) have achieved notable progress in
video generation, this long-sequence generation task remains constrained by the quadratic complexity inherent to self-
attention mechanisms, creating significant barriers to practical deployment. Although sparse
attention methods attempt to address this challenge, existing approaches either rely on oversimplified static patterns or require computationally expensive sampling operations to achieve dynamic sparsity, resulting in inaccurate pattern predictions and degraded generation quality. To overcome these limitations, we propose a \underline{M}ixture-\underline{O}f-\underline{D}istribution DiT (MOD-DiT), a novel sampling-free dynamic
attention framework that accurately models evolving
attention patterns through a two-stage process. First, MOD-DiT leverages prior information from early denoising steps and adopts a {distributed mixing approach} to model an efficient linear approximation model, which is then used to predict mask patterns for a specific denoising interval. Second, an online block masking strategy dynamically applies these predicted masks while maintaining historical sparsity information, eliminating the need for repetitive sampling operations. Extensive evaluations demonstrate consistent acceleration and quality improvements across multiple
benchmarks and model architectures, validating MOD-DiT's effectiveness for efficient, high-quality
video generation while overcoming the computational limitations of traditional sparse
attention approaches.
Read the full abstract ▾
All authors Yuxi Liu, Yipeng Hu, Zekun Zhang, Kunze Jiang, Kun Yuan
Computer Vision
Confident Learning for Object Detection under Model Constraints
Yingda Yu, Jiaqi Xuan, Shuhui Shi et al. (+3 more)
Agricultural weed detection on edge devices is subject to strict constraints on model capacity, computational resources, and real-time inference latency, which prevent performance improvements through model scaling or ensembling. This paper proposes Model-Driven Data Correction (MDDC), a data-centric framework that enhances detection performance by iteratively diagnosing and correcting data quality deficiencies. An automated error analysis procedure categorizes detection failures into four types: false negatives, false positives, class confusion, and localization errors. These error patterns are systematically addressed through a structured train-fix-retrain pipeline with version-controlled data management. Experimental results on multiple weed detection
datasets demonstrate consistent improvements of 5-25 percent in mAP at 0.5 using a fixed lightweight detector (YOLOv8n), indicating that systematic data quality optimization can effectively alleviate performance bottlenecks under fixed model capacity constraints.
Read the full abstract ▾
All authors Yingda Yu, Jiaqi Xuan, Shuhui Shi, Xuanyu Teng, Shuyang Xu, Guanchao Tong
Computer Vision
Evaluating Self-Correcting Vision Agents Through Quantitative and Qualitative Metrics
Aradhya Dixit
Recent progress in
multimodal foundation models has enabled
Vision-
Language Agents (VLAs) to decompose complex visual tasks into executable tool-based plans. While recent
benchmarks have begun to evaluate iterative self-correction, its quantitative limits and dominant
reasoning bottlenecks remain poorly characterized. This work introduces a Diagnostic Micro-
Benchmark. Our analysis decouples Task Success Rate (TSR = 62 percent) from Correction Success Rate (CSR = 25 to 33 percent), revealing that initial competence does not predict repair ability. We explicitly quantify the diminishing returns of correction, which saturates after three retries. Our Failure Taxonomy reveals a frequent factor is Semantic Drift (about 28 percent of failures), a loss of contextual state. By isolating this
reasoning bottleneck, this
benchmark defines a reproducible framework toward stateful, trustworthy
multimodal agents.
Read the full abstract ▾
Computer Vision
Now You See Me, Now You Don't: A Unified Framework for Expression Consistent Anonymization in Talking Head Videos
Anil Egin, Andrea Tangherloni and Antitza Dantcheva
Face
video anonymization is aimed at
privacy preservation while allowing for the analysis of
videos in a number of computer
vision downstream tasks such as expression recognition, people tracking, and action recognition. We propose here a novel unified framework referred to as Anon-NET, streamlined to de-identify facial
videos, while preserving age, gender, race, pose, and expression of the original
video. Specifically, we inpaint faces by a
diffusion-based generative model guided by high-level attribute recognition and motion-aware expression transfer. We then animate deidentified faces by
video-driven animation, which accepts the de-identified face and the original
video as input. Extensive experiments on the
datasets VoxCeleb2, CelebV-HQ, and HDTF, which include diverse facial dynamics, demonstrate the effectiveness of AnonNET in obfuscating identity while retaining visual realism and temporal consistency. The code of AnonNet will be publicly released.
Read the full abstract ▾
Computer Vision
When Rules Fall Short: Agent-Driven Discovery of Emerging Content Issues in Short Video Platforms
Chenghui Yu, Hongwei Wang, Junwen Chen et al. (+5 more)
Trends on short-
video platforms evolve at a rapid pace, with new content issues emerging every day that fall outside the coverage of existing annotation policies. However, traditional human-driven discovery of emerging issues is too slow, which leads to delayed updates of annotation policies and poses a major challenge for effective content governance. In this work, we propose an automatic issue discovery method based on
multimodal LLM
agents. Our approach automatically recalls short
videos containing potential new issues and applies a two-stage clustering strategy to group them, with each cluster corresponding to a newly discovered issue. The
agent then generates updated annotation policies from these clusters, thereby extending coverage to these emerging issues. Our
agent has been deployed in the real system. Both offline and online experiments demonstrate that this
agent-based method significantly improves the effectiveness of emerging-issue discovery (with an F1 score improvement of over 20%) and enhances the performance of subsequent issue governance (reducing the view count of problematic
videos by approximately 15%). More importantly, compared to manual issue discovery, it greatly reduces time costs and substantially accelerates the iteration of annotation policies.
Read the full abstract ▾
All authors Chenghui Yu, Hongwei Wang, Junwen Chen, Zixuan Wang, Bingfeng Deng, Zhuolin Hao, Hongyu Xiong, Yang Song
Computer Vision
Beyond Accuracy: Evaluating Grounded Visual Evidence in Thinking with Images
Xuchen Li, Xuzhao Li, Renjie Pi et al. (+3 more)
Despite the remarkable progress of
Vision-
Language Models (VLMs) in adopting "Thinking-with-Images" capabilities, accurately evaluating the authenticity of their
reasoning process remains a critical challenge. Existing
benchmarks mainly rely on outcome-oriented accuracy, lacking the capability to assess whether models can accurately leverage fine-grained visual cues for multi-step
reasoning. To address these limitations, we propose ViEBench, a process-verifiable
benchmark designed to evaluate faithful visual
reasoning. Comprising 200 multi-scenario high-resolution images with expert-annotated visual evidence, ViEBench uniquely categorizes tasks by difficulty into perception and
reasoning dimensions, where
reasoning tasks require utilizing localized visual details with prior knowledge. To establish comprehensive evaluation criteria, we introduce a dual-axis
matrix that provides fine-grained metrics through four diagnostic quadrants, enabling transparent diagnosis of model behavior across varying task complexities. Our experiments yield several interesting observations: (1) VLMs can sometimes produce correct final answers despite grounding on irrelevant regions, and (2) they may successfully locate the correct evidence but still fail to utilize it to reach accurate conclusions. Our findings demonstrate that ViEBench can serve as a more explainable and practical
benchmark for comprehensively evaluating the effectiveness agentic VLMs. The codes will be released at: https://github.com/Xuchen-Li/ViEBench.
Read the full abstract ▾
All authors Xuchen Li, Xuzhao Li, Renjie Pi, Shiyu Hu, Jian Zhao, Jiahui Gao
Computer Vision
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering
Zhiyang Li, Ao Ke, Yukun Cao et al. (+1 more)
Multi-modal Large
Language Models (MLLMs) for Visual Question Answering (VQA) often suffer from dual limitations: knowledge
hallucination and insufficient fine-grained visual perception. Crucially, we identify that commonsense graphs and scene graphs provide precisely complementary solutions to these respective deficiencies by providing rich external knowledge and capturing fine-grained visual details. However, prior works typically treat them in isolation, overlooking their synergistic potential. To bridge this gap, we propose KG-ViP, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs. The core of the KG-ViP framework is a novel
retrieval-and-fusion pipeline that utilizes the query as a semantic bridge to progressively integrate both graphs, synthesizing a unified structured context that facilitates reliable multi-modal
reasoning. Extensive experiments on FVQA 2.0+ and MVQA
benchmarks demonstrate that KG-ViP significantly outperforms existing VQA methods.
Read the full abstract ▾
All authors Zhiyang Li, Ao Ke, Yukun Cao, Xike Xie
Computer Vision
Compress to Focus: Efficient Coordinate Compression for Policy Optimization in Multi-Turn GUI Agents
Yurun Song, Jiong Yin, Rongjunchen Zhang et al. (+1 more)
Multi-turn GUI
agents enable complex task completion through sequential decision-making, but suffer from severe context inflation as interaction history accumulates. Existing strategies either sacrifice long-term context via truncation or compromise spatial structure through
token pruning. In this paper, we propose Coordinate Compression Policy Optimization (CCPO), an efficient policy optimization framework that couples visual compression with policy optimization for multi-turn GUI
agents. CCPO introduces Coordinate-Aware Spatial Compression (CASC), which aggregates coordinates from multiple rollouts to capture target-relevant regions and progressively narrow historical
attention around key visual areas. From interactions across rollouts, CASC adaptively constructs
attention boundaries that concentrate computation on the most informative regions of the scene. We further design a Distance-Based Advantage that provides fine-grained learning signals based on distance rather than binary correctness, improving both grounding accuracy and compression quality. Extensive experiments demonstrate that CCPO achieves SOTA performance across four
benchmarks with up to 55%
token compression and 3.8$\times$ training speedup.
Read the full abstract ▾
All authors Yurun Song, Jiong Yin, Rongjunchen Zhang, Ian G. Harris
Computer Vision
A one-step generation model with a Single-Layer Transformer: Layer number re-distillation of FreeFlow
Haonan Wei, Linyuan Wang, Nuolin Sun et al. (+3 more)
Currently, Flow matching methods aim to compress the iterative generation process of
diffusion models into a few or even a single step, with MeanFlow and FreeFlow being representative achievements of one-step generation based on Ordinary Differential Equations (ODEs). We observe that the 28-layer
Transformer architecture of FreeFlow can be characterized as an Euler discretization scheme for an ODE along the depth axis, where the layer index serves as the discrete time step. Therefore, we distill the number of layers of the FreeFlow model, following the same derivation logic as FreeFlow, and propose SLT (Single-Layer
Transformer), which uses a single shared DiT block to approximate the depth-wise feature
evolution of the 28-layer teacher. During training, it matches the teacher's intermediate features at several depth patches, fuses those patch-level representations, and simultaneously aligns the teacher's final velocity prediction. Through
distillation training, we compress the 28 independent
Transformer Blocks of the teacher model DiT-XL/2 into a single
Transformer Block, reducing the parameter count from 675M to 4.3M. Furthermore, leveraging its minimal parameters and rapid sampling speed, SLT can screen more candidate points in the noise space within the same timeframe, thereby selecting higher-quality initial points for the teacher model FreeFlow and ultimately enhancing the quality of generated images. Experimental results demonstrate that within a time budget comparable to two random samplings of the teacher model, our method performs over 100 noise screenings and produces a high-quality sample through the teacher model using the selected points. Quality fluctuations caused by low-quality initial noise under a limited number of FreeFlow sampling calls are effectively avoided, substantially improving the stability and average generation quality of one-step generation.
Read the full abstract ▾
All authors Haonan Wei, Linyuan Wang, Nuolin Sun, Zhizhong Zheng, Lei Li, Bin Yan
Computer Vision
A continental-scale dataset of ground beetles with high-resolution images and validated morphological trait measurements
S M Rayeed, Mridul Khurana, Alyson East et al. (+18 more)
Despite the ecological significance of invertebrates, global trait databases remain heavily biased toward vertebrates and plants, limiting comprehensive ecological analyses of high-diversity groups like ground beetles. Ground beetles (Coleoptera: Carabidae) serve as critical bioindicators of ecosystem health, providing valuable insights into biodiversity shifts driven by environmental changes. While the National Ecological Observatory Network (NEON) maintains an extensive collection of carabid specimens from across the United States, these primarily exist as physical collections, restricting widespread research access and large-scale analysis. To address these gaps, we present a
multimodal dataset digitizing over 13,200 NEON carabids from 30 sites spanning the continental US and Hawaii through high-resolution imaging, enabling broader access and computational analysis. The
dataset includes digitally measured elytra length and width of each specimen, establishing a foundation for automated trait extraction using AI. Validated against manual measurements, our digital trait extraction achieves sub-millimeter precision, ensuring reliability for ecological and computational studies. By addressing invertebrate under-representation in trait databases, this work supports AI-driven tools for automated species identification and trait-based research, fostering advancements in biodiversity monitoring and conservation.
Read the full abstract ▾
All authors S M Rayeed, Mridul Khurana, Alyson East, Isadora E. Fluck, Elizabeth G. Campolongo, Samuel Stevens, Iuliia Zarubiieva, Scott C. Lowe, Michael W. Denslow, Evan D. Donoso, Jiaman Wu, Michelle Ramirez, Benjamin Baiser, Charles V. Stewart, Paula Mabee, Tanya Berger-Wolf, Anuj Karpatne, Hilmar Lapp, Robert P. Guralnick, Graham W. Taylor, Sydne Record
Computer Vision · Cross-listed from cs.HC
The Algorithmic Gaze of Image Quality Assessment: An Audit and Trace Ethnography of the LAION-Aesthetics Predictor
Jordan Taylor, William Agnew, Maarten Sap et al. (+2 more)
Visual generative AI models are trained using a one-size-fits-all measure of aesthetic appeal. However, what is deemed "aesthetic" is inextricably linked to personal taste and cultural values, raising the question of whose taste is represented in visual generative AI models. In this work, we study an aesthetic evaluation model–LAION-Aesthetics Predictor (LAP)–that is widely used to curate
datasets to train visual generative image models, like Stable
Diffusion, and evaluate the quality of AI-generated images. To understand what LAP measures, we audited the model across three
datasets. First, we examined the impact of aesthetic filtering on the LAION-Aesthetics
Dataset (approximately 1.2B images), which was curated from LAION-5B using LAP. We find that the LAP disproportionally filters in images with captions mentioning women, while filtering out images with captions mentioning men or LGBTQ+ people. Then, we used LAP to score approximately 330k images across two art
datasets, finding the model rates realistic images of landscapes, cityscapes, and portraits from western and Japanese artists most highly. In doing so, the algorithmic gaze of this aesthetic evaluation model reinforces the imperial and male gazes found within western art history. In order to understand where these biases may have originated, we performed a digital ethnography of public materials related to the creation of LAP. We find that the development of LAP reflects the biases we found in our audits, such as the aesthetic scores used to train LAP primarily coming from English-speaking photographers and western AI-enthusiasts. In response, we discuss how aesthetic evaluation can perpetuate representational harms and call on AI developers to shift away from prescriptive measures of "aesthetics" toward more pluralistic evaluation.
Read the full abstract ▾
All authors Jordan Taylor, William Agnew, Maarten Sap, Sarah E. Fox, Haiyi Zhu
Computer Vision
Transition Matching Distillation for Fast Video Generation
Weili Nie, Julius Berner, Nanye Ma et al. (+3 more)
Large
video diffusion and flow models have achieved remarkable success in high-quality
video generation, but their use in real-time interactive applications remains limited due to their inefficient multi-step sampling process. In this work, we present Transition Matching
Distillation (TMD), a novel framework for distilling
video diffusion models into efficient few-step generators. The central idea of TMD is to match the multi-step denoising trajectory of a
diffusion model with a few-step probability transition process, where each transition is modeled as a lightweight conditional flow. To enable efficient
distillation, we decompose the original
diffusion backbone into two components: (1) a main backbone, comprising the majority of early layers, that extracts semantic representations at each outer transition step; and (2) a flow head, consisting of the last few layers, that leverages these representations to perform multiple inner flow updates. Given a pretrained
video flow model, we first introduce a flow head to the model, and adapt it into a conditional flow map. We then apply distribution matching
distillation to the student model with flow head rollout in each transition step. Extensive experiments on distilling Wan2.1 1.3B and 14B text-to-
video models demonstrate that TMD provides a flexible and strong trade-off between generation speed and visual quality. In particular, TMD outperforms existing distilled models under comparable inference costs in terms of visual fidelity and
prompt adherence. Project page: https://research.nvidia.com/labs/genair/tmd
Read the full abstract ▾
All authors Weili Nie, Julius Berner, Nanye Ma, Chao Liu, Saining Xie, Arash Vahdat
Computer Vision
Yang Xing, Jiong Wu, Savas Ozdemir et al. (+4 more)
Recent progress in medical
vision-
language models (VLMs) has achieved strong performance on image-level text-centric tasks such as report generation and visual question answering (VQA). However, achieving fine-grained visual grounding and volumetric spatial
reasoning in 3D medical VLMs remains challenging, particularly when aiming to unify these capabilities within a single, generalizable framework. To address this challenge, we proposed MedVL-SAM2, a unified 3D medical
multimodal model that concurrently supports report generation, VQA, and multi-paradigm
segmentation, including semantic, referring, and interactive
segmentation. MedVL-SAM2 integrates image-level
reasoning and pixel-level perception through a cohesive architecture tailored for 3D medical imaging, and incorporates a SAM2-based volumetric
segmentation module to enable precise multi-granular spatial
reasoning. The model is trained in a multi-stage pipeline: it is first pre-trained on a large-scale corpus of 3D CT image-text pairs to align volumetric visual features with radiology-
language embeddings. It is then jointly optimized with both
language-understanding and
segmentation objectives using a comprehensive 3D CT
segmentation dataset. This joint training enables flexible interaction via
language, point, or box
prompts, thereby unifying high-level visual
reasoning with spatially precise localization. Our unified architecture delivers state-of-the-art performance across report generation, VQA, and multiple 3D
segmentation tasks. Extensive analyses further show that the model provides reliable 3D visual grounding, controllable interactive
segmentation, and robust cross-modal
reasoning, demonstrating that high-level semantic
reasoning and precise 3D localization can be jointly achieved within a unified 3D medical VLM.
Read the full abstract ▾
All authors Yang Xing, Jiong Wu, Savas Ozdemir, Ying Zhang, Yang Yang, Wei Shao, Kuang Gong
Computer Vision
VibrantSR: Sub-Meter Canopy Height Models from Sentinel-2 Using Generative Flow Matching
Kiarie Ndegwa, Andreas Gros, Tony Chang et al. (+6 more)
We present VibrantSR (Vibrant Super-Resolution), a generative super-resolution framework for estimating 0.5 meter canopy height models (CHMs) from 10 meter Sentinel-2 imagery. Unlike approaches based on aerial imagery that are constrained by infrequent and irregular acquisition schedules, VibrantSR leverages globally available Sentinel-2 seasonal composites, enabling consistent monitoring at a seasonal-to-annual cadence. Evaluated across 22 EPA Level 3 eco-regions in the western United States using spatially disjoint validation splits, VibrantSR achieves a Mean Absolute Error of 4.39 meters for canopy heights >= 2 m, outperforming Meta (4.83 m), LANDFIRE (5.96 m), and ETH (7.05 m) satellite-based
benchmarks. While aerial-based VibrantVS (2.71 m MAE) retains an accuracy advantage, VibrantSR enables operational forest monitoring and carbon accounting at continental scales without reliance on costly and temporally infrequent aerial acquisitions.
Read the full abstract ▾
All authors Kiarie Ndegwa, Andreas Gros, Tony Chang, David Diaz, Vincent A. Landau, Nathan E. Rutenbeck, Luke J. Zachmann, Guy Bayes, Scott Conway
Computer Vision
Tune-Up Open-Weight CLIP: Optimization Framework for Self-Supervised Fine-tuning of CLIP
Anant Mehta, Xingyu Chen, Xiyuan Wei et al. (+1 more)
CLIP has become a cornerstone of
multimodal representation learning, yet improving its performance typically requires a prohibitively costly process of training from scratch on billions of samples. We ask a different question: Can we improve the performance of open-weight CLIP models across various downstream tasks using only existing self-supervised
datasets? Unlike supervised fine-tuning, which adapts a pretrained model to a single downstream task, our setting seeks to improve general performance across various tasks. However, as both our experiments and prior studies reveal, simply applying standard training protocols starting from an open-weight CLIP model often fails, leading to performance degradation. In this paper, we introduce TuneCLIP, a self-supervised fine-tuning framework that overcomes the performance degradation. TuneCLIP has two key components: (1) a warm-up stage of recovering optimization statistics to reduce cold-start bias, inspired by theoretical analysis, and (2) a fine-tuning stage of optimizing a new contrastive loss to mitigate the penalization on false negative pairs. Our extensive experiments show that TuneCLIP consistently improves performance across model architectures and scales. Notably, it elevates leading open-weight models like SigLIP (ViT-B/16), achieving gains of up to +2.5% on ImageNet and related out-of-distribution
benchmarks, and +1.2% on the highly competitive DataComp
benchmark, setting a new strong baseline for efficient post-pretraining adaptation. The code is available at https://github.com/Optimization-AI/TuneCLIP.
Read the full abstract ▾
All authors Anant Mehta, Xingyu Chen, Xiyuan Wei, Tianbao Yang
Computer Vision
ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioning
Po-han Li, Shenghui Chen, Ufuk Topcu et al. (+1 more)
Multimodal video captioning condenses dense footage into a structured format of keyframes and natural
language. By creating a cohesive
multimodal summary, this approach anchors generative AI in rich semantic evidence and serves as a lightweight proxy for high-efficiency
retrieval. However, traditional metrics like BLEU or ROUGE fail to quantify information coverage across disparate modalities, such as comparing a paragraph of text to a sequence of keyframes. To address this, we propose the
Video Summary Information Loss (ViSIL) score, an information-theoretic framework that quantifies the
video information not captured by a summary via
vision-
language model (VLM) inference. By measuring the information loss, ViSIL is a unified metric that enables direct comparison across
multimodal summary formats despite their structural discrepancies. Our results demonstrate that ViSIL scores show a statistically significant correlation with both human and VLM performance on
Video Question Answering (VQA) tasks. ViSIL also enables summary selection to optimize the trade-off between information loss and processing speed, establishing a Pareto-optimal frontier that outperforms text summaries by $7\%$ in VQA accuracy without increasing processing load.
Read the full abstract ▾
All authors Po-han Li, Shenghui Chen, Ufuk Topcu, Sandeep Chinchali
Computer Vision
UniHash: Unifying Pointwise and Pairwise Hashing Paradigms
Xiaoxu Ma, Runhao Li, Xiangbo Zhang et al. (+1 more)
Effective
retrieval across both seen and unseen categories is crucial for modern image
retrieval systems.
Retrieval on seen categories ensures precise recognition of known classes, while
retrieval on unseen categories promotes generalization to novel classes with limited supervision. However, most existing deep hashing methods are confined to a single training paradigm, either pointwise or pairwise, where the former excels on seen categories and the latter generalizes better to unseen ones. To overcome this limitation, we propose Unified Hashing (UniHash), a dual-branch framework that unifies the strengths of both paradigms to achieve balanced
retrieval performance across seen and unseen categories. UniHash consists of two complementary branches: a center-based branch following the pointwise paradigm and a pairwise branch following the pairwise paradigm. A novel hash code learning method is introduced to enable bidirectional knowledge transfer between branches, improving hash code discriminability and generalization. It employs a mutual learning loss to align hash representations and introduces a Split-Merge Mixture of Hash Experts (SM-MoH) module to enhance cross-branch exchange of hash representations. Theoretical analysis substantiates the effectiveness of UniHash, and extensive experiments on CIFAR-10, MSCOCO, and ImageNet demonstrate that UniHash consistently achieves state-of-the-art performance in both seen and unseen image
retrieval scenarios.
Read the full abstract ▾
All authors Xiaoxu Ma, Runhao Li, Xiangbo Zhang, Zhenyu Weng
Computer Vision
NanoSD: Edge Efficient Foundation Model for Real Time Image Restoration
Subhajit Sanyal, Srinivas Soumitri Miriyala, Akshay Janardan Bankar et al. (+7 more)
Latent
diffusion models such as Stable
Diffusion 1.5 offer strong generative priors that are highly valuable for image restoration, yet their full pipelines remain too computationally heavy for deployment on edge devices. Existing lightweight variants predominantly compress the denoising U-Net or reduce the
diffusion trajectory, which disrupts the underlying latent
manifold and limits generalization beyond a single task. We introduce NanoSD, a family of Pareto-optimal
diffusion foundation models distilled from Stable
Diffusion 1.5 through network surgery, feature-wise generative
distillation, and structured architectural scaling jointly applied to the U-Net and the VAE encoder-decoder. This full-pipeline co-design preserves the generative prior while producing models that occupy distinct operating points along the accuracy-latency-size frontier (e.g., 130M-315M parameters, achieving real-time inference down to 20ms on mobile-class NPUs). We show that parameter reduction alone does not correlate with hardware efficiency, and we provide an analysis revealing how architectural balance, feature routing, and latent-space preservation jointly shape true on-device latency. When used as a drop-in backbone, NanoSD enables state-of-the-art performance across image super-resolution, image deblurring, face restoration, and
monocular depth estimation, outperforming prior lightweight
diffusion models in both perceptual quality and practical deployability. NanoSD establishes a general-purpose
diffusion foundation model family suitable for real-time visual generation and restoration on edge devices.
Read the full abstract ▾
All authors Subhajit Sanyal, Srinivas Soumitri Miriyala, Akshay Janardan Bankar, Manjunath Arveti, Sowmya Vajrala, Shreyas Pandith, Sravanth Kodavanti, Abhishek Ameta, Harshit, Amit Satish Unde
Computer Vision
Explainable Deep Learning for Pediatric Pneumonia Detection in Chest X-Ray Images
Adil O. Khadidos, Aziida Nanyonga, Alaa O. Khadidos et al. (+2 more)
Background: Pneumonia remains a leading cause of morbidity and mortality among children worldwide, emphasizing the need for accurate and efficient diagnostic support tools. Deep learning has shown strong potential in medical image analysis, particularly for chest X-ray interpretation. This study compares two state-of-the-art convolutional neural network (CNN) architectures for automated pediatric pneumonia detection. Methods: A publicly available
dataset of 5,863 pediatric chest X-ray images was used. Images were preprocessed through normalization, resizing, and data augmentation to enhance generalization. DenseNet121 and EfficientNet-B0 were fine-tuned using pretrained ImageNet weights under identical training settings. Performance was evaluated using accuracy, F1-score, Matthews Correlation Coefficient (MCC), and recall. Model explainability was incorporated using Gradient-weighted Class Activation Mapping (Grad-CAM) and Local Interpretable Model-agnostic Explanations (LIME) to visualize image regions influencing predictions. Results: EfficientNet-B0 outperformed DenseNet121, achieving an accuracy of 84.6%, F1-score of 0.8899, and MCC of 0.6849. DenseNet121 achieved 79.7% accuracy, an F1-score of 0.8597, and MCC of 0.5852. Both models demonstrated high recall values above 0.99, indicating strong sensitivity to pneumonia detection. Grad-CAM and LIME visualizations showed consistent focus on clinically relevant lung regions, supporting the reliability of model decisions. Conclusions: EfficientNet-B0 provided a more balanced and computationally efficient performance compared to DenseNet121, making it a strong candidate for clinical deployment. The integration of explainability techniques enhances transparency and trustworthiness in AI-assisted pediatric pneumonia diagnosis.
Read the full abstract ▾
All authors Adil O. Khadidos, Aziida Nanyonga, Alaa O. Khadidos, Olfat M. Mirza, Mustafa Tahsin Yilmaz
Computer Vision
LCF3D: A Robust and Real-Time Late-Cascade Fusion Framework for 3D Object Detection in Autonomous Driving
Carlo Sgaravatti, Riccardo Pieroni, Matteo Corno et al. (+3 more)
Accurately localizing 3D objects like pedestrians, cyclists, and other vehicles is essential in Autonomous Driving. To ensure high detection performance, Autonomous Vehicles complement RGB cameras with
LiDAR sensors, but effectively combining these data sources for 3D object detection remains challenging. We propose LCF3D, a novel sensor fusion framework that combines a 2D object detector on RGB images with a 3D object detector on
LiDAR point clouds. By leveraging
multimodal fusion principles, we compensate for inaccuracies in the
LiDAR object detection network. Our solution combines two key principles: (i) late fusion, to reduce
LiDAR False Positives by matching
LiDAR 3D detections with RGB 2D detections and filtering out unmatched
LiDAR detections; and (ii) cascade fusion, to recover missed objects from
LiDAR by generating new 3D frustum proposals corresponding to unmatched RGB detections. Experiments show that LCF3D is beneficial for domain generalization, as it turns out to be successful in handling different sensor configurations between training and testing domains. LCF3D achieves significant improvements over
LiDAR-based methods, particularly for challenging categories like pedestrians and cyclists in the KITTI
dataset, as well as motorcycles and bicycles in nuScenes. Code can be downloaded from: https://github.com/CarloSgaravatti/LCF3D.
Read the full abstract ▾
All authors Carlo Sgaravatti, Riccardo Pieroni, Matteo Corno, Sergio M. Savaresi, Luca Magri, Giacomo Boracchi
Computer Vision
Diffusion-Driven Deceptive Patches: Adversarial Manipulation and Forensic Detection in Facial Identity Verification
Shahrzad Sayyafzadeh, Hongmei Chi and Shonda Bernadin
This work presents an end-to-end pipeline for generating, refining, and evaluating
adversarial patches to compromise facial biometric systems, with applications in forensic analysis and security testing. We utilize FGSM to generate
adversarial noise targeting an identity classifier and employ a
diffusion model with reverse
diffusion to enhance imperceptibility through Gaussian smoothing and adaptive brightness correction, thereby facilitating synthetic
adversarial patch evasion. The refined patch is applied to facial images to test its ability to evade recognition systems while maintaining natural visual characteristics. A
Vision Transformer (ViT)-GPT2 model generates captions to provide a semantic description of a person's identity for
adversarial images, supporting forensic interpretation and documentation for identity evasion and recognition attacks. The pipeline evaluates changes in identity classification, captioning results, and vulnerabilities in facial identity verification and expression recognition under
adversarial conditions. We further demonstrate effective detection and analysis of
adversarial patches and
adversarial samples using perceptual hashing and
segmentation, achieving an SSIM of 0.95.
Read the full abstract ▾
Computer Vision
Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning
Chi-Pin Huang, Yunze Man, Zhiding Yu et al. (+4 more)
Vision-
Language-Action (VLA) tasks require
reasoning over complex visual scenes and executing adaptive actions in dynamic environments. While recent studies on
reasoning VLAs show that explicit chain-of-thought (CoT) can improve generalization, they suffer from high inference latency due to lengthy
reasoning traces. We propose Fast-ThinkAct, an efficient
reasoning framework that achieves compact yet performant planning through verbalizable latent
reasoning. Fast-ThinkAct learns to reason efficiently with latent CoTs by distilling from a teacher, driven by a preference-guided objective to align manipulation trajectories that transfers both linguistic and visual planning capabilities for
embodied control. This enables
reasoning-enhanced policy learning that effectively connects compact
reasoning to action execution. Extensive experiments across diverse
embodied manipulation and
reasoning benchmarks demonstrate that Fast-ThinkAct achieves strong performance with up to 89.3% reduced inference latency over state-of-the-art
reasoning VLAs, while maintaining effective long-horizon planning, few-shot adaptation, and failure recovery.
Read the full abstract ▾
All authors Chi-Pin Huang, Yunze Man, Zhiding Yu, Min-Hung Chen, Jan Kautz, Yu-Chiang Frank Wang, Fu-En Yang
Computer Vision
SAM3-DMS: Decoupled Memory Selection for Multi-target Video Segmentation of SAM3
Ruiqi Shen, Chang Liu and Henghui Ding
Segment Anything 3 (SAM3) has established a powerful foundation that robustly detects, segments, and tracks specified targets in
videos. However, in its original implementation, its group-level collective
memory selection is suboptimal for complex multi-object scenarios, as it employs a synchronized decision across all concurrent targets conditioned on their average performance, often overlooking individual reliability. To this end, we propose SAM3-DMS, a training-free decoupled strategy that utilizes fine-grained
memory selection on individual objects. Experiments demonstrate that our approach achieves robust identity preservation and tracking stability. Notably, our advantage becomes more pronounced with increased target density, establishing a solid foundation for simultaneous multi-target
video segmentation in the wild.
Read the full abstract ▾
Computer Vision
COMPOSE: Hypergraph Cover Optimization for Multi-view 3D Human Pose Estimation
Tony Danjun Wang, Tolga Birdal, Nassir Navab et al. (+1 more)
3D human pose estimation from sparse multi-view camera rigs is an essential task for numerous applications, including action recognition, sports analysis, and human-
robot interaction. While learned methods dominate the field on
benchmarks, they require large annotated
datasets; training-free optimization-based methods remain promising as they circumvent 3D supervision by solving a correspondence problem across views from 2D detections. Existing combinatorial formulations rely on pairwise associations to model this correspondence problem and enforce global consistency across views only as a downstream constraint. However, reconciling locally plausible pairwise matches becomes brittle under occlusion and noisy detections, where local errors propagate globally. We propose COMPOSE, which recasts multi-view 3D human pose estimation as a weighted exact-cover optimization over a hypergraph of person hypotheses. Our formulation replaces pairwise association and post-hoc consistency enforcement with a single global combinatorial objective. To address the exponentially large candidate space, we introduce a
geometric pruning strategy alongside two complementary solvers: an exact Integer Linear Programming formulation and a scalable relaxation via Belief Propagation. Without any 3D supervision, COMPOSE improves average precision by up to 31 points over the best optimization-based method and 13 points over self-supervised learned methods, demonstrating the effectiveness of higher-order combinatorial association for training-free multi-view 3D human pose estimation.
Read the full abstract ▾
All authors Tony Danjun Wang, Tolga Birdal, Nassir Navab, Lennart Bastian