Computer Vision · Cross-listed from eess.IV
TopoGate: Quality-Aware Topology-Stabilized Gated Fusion for Longitudinal Low-Dose CT New-Lesion Prediction
Seungik Cho
Longitudinal low-dose CT follow-ups vary in noise, reconstruction kernels, and registration quality. These differences destabilize subtraction images and can trigger false new lesion alarms. We present TopoGate, a lightweight model that combines the follow-up appearance view with the subtraction view and controls their influence through a learned, quality-aware gate. The gate is driven by three case-specific signals: CT appearance quality, registration consistency, and stability of anatomical
topology measured with topological metrics. On the NLST–New-Lesion–LongCT cohort comprising 152 pairs from 122 patients, TopoGate improves discrimination and calibration over single-view baselines, achieving an area under the ROC curve of 0.65 with a standard deviation of 0.05 and a Brier score of 0.14. Removing corrupted or low-quality pairs, identified by the quality scores, further increases the area under the ROC curve from 0.62 to 0.68 and reduces the Brier score from 0.14 to 0.12. The gate responds predictably to degradation, placing more weight on appearance when noise grows, which mirrors radiologist practice. The approach is simple, interpretable, and practical for reliable longitudinal LDCT triage.
Read the full abstract ▾
Computer Vision
On the Evaluation Protocol of Gesture Recognition for UAV-based Rescue Operation based on Deep Learning: A Subject-Independence Perspective
Domonkos Varga
This paper presents a methodological analysis of the gesture-recognition approach proposed by Liu and Szirányi, with a particular focus on the validity of their evaluation protocol. We show that the reported near-perfect accuracy metrics result from a frame-level random train-test split that inevitably mixes samples from the same subjects across both sets, causing severe data leakage. By examining the published confusion
matrix, learning curves, and
dataset construction, we demonstrate that the evaluation does not measure generalization to unseen individuals. Our findings underscore the importance of subject-independent data partitioning in
vision-based gesture-recognition research, especially for applications - such as
UAV-human interaction - that require reliable recognition of gestures performed by previously unseen people.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Neural Prior Estimation: Learning Class Priors from Latent Representations
Masoud Yavari and Payman Moallem
Logit adjustment corrects class imbalance using the empirical class prior. We study whether a comparable class-frequency signal can instead be learned from the network representation, without explicitly supplying class counts to the correction rule. We introduce the Neural Prior
Estimator (NPE), which attaches one or more lightweight Prior Estimation Modules (PEMs) to the latent representation. Each PEM is trained with a one-way logistic objective on the ground-truth coordinate. The resulting frequency-dependent outputs are combined into an NPE estimate and used as a learned logit correction, forming NPE-LA. In a simplified scalar model, the optimum of the PEM objective is monotone in the class count and grows asymptotically as $\log N_c$, up to a slower $\log \log N_c$ term. Experiments on long-tailed CIFAR-10 and CIFAR-100 show that NPE-LA is competitive with standard logit adjustment and improves minority-class performance over CE and cRT in the reported settings. Experiments on STARE and ADE20K further show that the same idea can be used as a lightweight recalibration mechanism for dense prediction.
Read the full abstract ▾
Computer Vision
VQPP: Video Query Performance Prediction Benchmark
Adrian Catalin Lutu, Eduard Poesina and Radu Tudor Ionescu
Query performance prediction (QPP) is an important and actively studied information
retrieval task, having various applications, such as query reformulation, query expansion, and
retrieval system selection, among many others. The task has been primarily studied in the context of text and image
retrieval, whereas QPP for content-based
video retrieval (CBVR) remains largely underexplored. To this end, we propose the first
benchmark for
video query performance prediction (VQPP), comprising two text-to-
video retrieval datasets and two CBVR systems, respectively. VQPP contains a total of 56K text queries and 51K
videos, and comes with official training, validation and test splits, fostering direct comparisons and reproducible results. We explore multiple pre-
retrieval and post-
retrieval performance predictors, creating a representative
benchmark for future exploration of QPP in the
video domain. Our results show that pre-
retrieval predictors obtain competitive performance, enabling applications before performing the
retrieval step. We also demonstrate the applicability of VQPP by employing the best performing pre-
retrieval predictor as reward model for training a large
language model (LLM) on the query reformulation task via direct preference optimization (DPO). We release our
benchmark and code at https://github.com/AdrianLutu/VQPP.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Promptable segmentation with region exploration enables minimal-effort expert-level prostate cancer delineation
Junqing Yang, Natasha Thorley, Ahmed Nadeem Abbasi et al. (+4 more)
Purpose: Accurate
segmentation of prostate cancer on
magnetic resonance (MR) images is crucial for planning image-guided interventions such as targeted biopsies, cryoablation, and radiotherapy. However, subtle and variable tumour appearances, differences in imaging protocols, and limited expert availability make consistent interpretation difficult. While automated methods aim to address this, they rely on large expertly-annotated
datasets that are often inconsistent, whereas manual delineation remains labour-intensive. This work aims to bridge the gap between automated and manual
segmentation through a framework driven by user-provided point
prompts, enabling accurate
segmentation with minimal annotation effort. Methods: The framework combines reinforcement learning (RL) with a region-growing
segmentation process guided by user
prompts. Starting from an initial point
prompt, region-growing generates a preliminary
segmentation, which is iteratively refined through RL. At each step, the RL
agent observes the image and current
segmentation to predict a new point, from which region growing updates the mask. A reward, balancing
segmentation accuracy and voxel-wise uncertainty, encourages exploration of ambiguous regions, allowing the
agent to escape local optima and perform sample-specific optimisation. Despite requiring fully supervised training, the framework bridges manual and fully automated
segmentation at inference by substantially reducing user effort while outperforming current fully automated methods. Results: The framework was evaluated on two public prostate MR
datasets (PROMIS and PICAI, with 566 and 1090 cases). It outperformed the previous best automated methods by 9.9% and 8.9%, respectively, with performance comparable to manual radiologist
segmentation, reducing annotation time tenfold.
Read the full abstract ▾
All authors Junqing Yang, Natasha Thorley, Ahmed Nadeem Abbasi, Shonit Punwani, Zion Tse, Yipeng Hu, Shaheer U. Saeed
Computer Vision
VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
Narges Norouzi, Idil Esen Zulfikar, Niccolò Cavagnero et al. (+4 more)
Existing online
video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain
Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image
segmentation without requiring specialized modules. Motivated by this observation, we propose the
Video Encoder-only Mask
Transformer (VidEoMT), a simple encoder-only
video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/
Read the full abstract ▾
All authors Narges Norouzi, Idil Esen Zulfikar, Niccolò Cavagnero, Tommie Kerssies, Bastian Leibe, Gijs Dubbelman, Daan de Geus
Computer Vision
Enabling Training-Free Text-Based Remote Sensing Segmentation
Jose Sosa, Danila Rukhovich, Anis Kacem et al. (+1 more)
Recent advances in
Vision Language Models (VLMs) and
Vision Foundation Models (VFMs) have opened new opportunities for zero-shot text-guided
segmentation of remote sensing imagery. However, most existing approaches still rely on additional trainable components, limiting their generalisation and practical applicability. In this work, we investigate to what extent text-based remote sensing
segmentation can be achieved without additional training, by relying solely on existing foundation models. We propose a simple yet effective approach that integrates contrastive and generative VLMs with the Segment Anything Model (SAM), enabling a fully training-free or lightweight LoRA-tuned pipeline. Our contrastive approach employs CLIP as mask selector for SAM's grid-based proposals, achieving state-of-the-art open-vocabulary semantic
segmentation (OVSS) in a completely zero-shot setting. In parallel, our generative approach enables
reasoning and referring
segmentation by generating click
prompts for SAM using GPT-5 in a zero-shot setting and a LoRA-tuned Qwen-VL model, with the latter yielding the best results. Extensive experiments across 19 remote sensing
benchmarks, including open-vocabulary, referring, and
reasoning-based tasks, demonstrate the strong capabilities of our approach. Code will be released at https://github.com/josesosajs/trainfree-rs-
segmentation.
Read the full abstract ▾
All authors Jose Sosa, Danila Rukhovich, Anis Kacem, Djamila Aouada
Computer Vision · Cross-listed from eess.IV
Deep Learning for Dermatology: An Innovative Framework for Approaching Precise Skin Cancer Detection
Mohammad Tahmid Noor, B. M. Shahria Alam, Tasmiah Rahman Orpa et al. (+3 more)
Skin cancer can be life-threatening if not diagnosed early, a prevalent yet preventable disease. Globally, skin cancer is perceived among the finest prevailing cancers and millions of people are diagnosed each year. For the allotment of benign and malignant skin spots, an area of critical importance in dermatological diagnostics, the application of two prominent deep learning models, VGG16 and DenseNet201 are investigated by this paper. We evaluate these CNN architectures for their efficacy in differentiating benign from malignant skin lesions leveraging enhancements in deep learning enforced to skin cancer spotting. Our objective is to assess model accuracy and computational efficiency, offering insights into how these models could assist in early detection, diagnosis, and streamlined workflows in dermatology. We used two deep learning methods DenseNet201 and VGG16 model on a binary class
dataset containing 3297 images. The best result with an accuracy of 93.79% achieved by DenseNet201. All images were resized to 224x224 by rescaling. Although both models provide excellent accuracy, there is still some room for improvement. In future using new
datasets, we tend to improve our work by achieving great accuracy.
Read the full abstract ▾
All authors Mohammad Tahmid Noor, B. M. Shahria Alam, Tasmiah Rahman Orpa, Shaila Afroz Anika, Mahjabin Tasnim Samiha, Fahad Ahammed
Computer Vision
LGD-Net: Latent-Guided Dual-Stream Network for HER2 Scoring with Task-Specific Domain Knowledge
Peide Zhu, Linbin Lu, Zhiqin Chen et al. (+1 more)
It is a critical task to evalaute HER2 expression level accurately for breast cancer evaluation and targeted treatment therapy selection. However, the standard multi-step Immunohistochemistry (IHC) staining is resource-intensive, expensive, and time-consuming, which is also often unavailable in many areas. Consequently, predicting HER2 levels directly from H&E slides has emerged as a potential alternative solution. It has been shown to be effective to use virtual IHC images from H&E images for automatic HER2 scoring. However, the pixel-level virtual staining methods are computationally expensive and prone to reconstruction artifacts that can propagate diagnostic errors. To address these limitations, we propose the Latent-Guided Dual-Stream Network (LGD-Net), a novel framework that employes cross-modal feature
hallucination instead of explicit pixel-level image generation. LGD-Net learns to map morphological H&E features directly to the molecular latent space, guided by a teacher IHC encoder during training. To ensure the hallucinated features capture clinically relevant phenotypes, we explicitly regularize the model training with task-specific domain knowledge, specifically
nuclei distribution and membrane staining intensity, via lightweight auxiliary regularization tasks. Extensive experiments on the public BCI
dataset demonstrate that LGD-Net achieves state-of-the-art performance, significantly outperforming baseline methods while enabling efficient inference using single-modality H&E inputs.
Read the full abstract ▾
All authors Peide Zhu, Linbin Lu, Zhiqin Chen, Xiong Chen
Computer Vision
Multi-Modal Monocular Endoscopic Depth and Pose Estimation with Edge-Guided Self-Supervision
Xinwei Ju, Rema Daher, Danail Stoyanov et al. (+2 more)
Monocular depth and pose estimation play an important role in the development of colonoscopy-assisted
navigation, as they enable improved screening by reducing blind spots, minimizing the risk of missed or recurrent lesions, and lowering the likelihood of incomplete examinations. However, this task remains challenging due to the presence of texture-less surfaces, complex illumination patterns, deformation, and a lack of in-vivo
datasets with reliable ground truth. In this paper, we propose **PRISM** (Pose-Refinement with Intrinsic Shading and edge Maps), a self-supervised learning framework that leverages anatomical and illumination priors to guide
geometric learning. Our approach uniquely incorporates edge detection and luminance decoupling for structural guidance. Specifically, edge maps are derived using a learning-based edge detector (e.g., DexiNed or HED) trained to capture thin and high-frequency boundaries, while luminance decoupling is obtained through an intrinsic decomposition module that separates shading and reflectance, enabling the model to exploit shading cues for depth estimation. Experimental results on multiple real and synthetic
datasets demonstrate state-of-the-art performance. We further conduct a thorough ablation study on training data selection to establish best practices for pose and depth estimation in colonoscopy. This analysis yields two practical insights: (1) self-supervised training on real-world data outperforms supervised training on realistic phantom data, underscoring the superiority of domain realism over ground truth availability; and (2)
video frame rate is an extremely important factor for model performance, where
dataset-specific
video frame sampling is necessary for generating high quality training data.
Read the full abstract ▾
All authors Xinwei Ju, Rema Daher, Danail Stoyanov, Sophia Bano, Francisco Vasconcelos
Computer Vision
CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild
Balamurugan Thambiraja, Omid Taheri, Radek Danecek et al. (+3 more)
Hands play a central role in daily life, yet modeling natural hand motions remains underexplored. Existing methods that tackle text-to-hand-motion generation or hand animation captioning rely on studio-captured
datasets with limited actions and contexts, making them costly to scale to "in-the-wild" settings. Further, contemporary models and their training schemes struggle to capture animation fidelity with text-motion
alignment. To address this, we (1) introduce '3D Hands in the Wild' (3D-HIW), a
dataset of 32K 3D hand-motion sequences and aligned text, and (2) propose CLUTCH, an LLM-based hand animation system with two critical innovations: (a) SHIFT, a novel VQ-VAE architecture to tokenize hand motion, and (b) a
geometric refinement stage to finetune the LLM. To build 3D-HIW, we propose a data annotation pipeline that combines
vision-
language models (VLMs) and state-of-the-art 3D hand trackers, and apply it to a large corpus of
egocentric action
videos covering a wide range of scenarios. To fully capture motion in-the-wild, CLUTCH employs SHIFT, a part-modality decomposed VQ-VAE, which improves generalization and reconstruction fidelity. Finally, to improve animation quality, we introduce a
geometric refinement stage, where CLUTCH is co-supervised with a reconstruction loss applied directly to decoded hand motion parameters. Experiments demonstrate state-of-the-art performance on text-to-motion and motion-to-text tasks, establishing the first
benchmark for scalable in-the-wild hand motion modelling. Code, data and models will be released.
Read the full abstract ▾
All authors Balamurugan Thambiraja, Omid Taheri, Radek Danecek, Giorgio Becherini, Gerard Pons-Moll, Justus Thies
Computer Vision
KPM-Bench: A Kinematic Parsing Motion Benchmark for Fine-grained Motion-centric Video Understanding
Boda Lin, Yongjie Zhu, Xiaocheng Gong et al. (+2 more)
Despite recent advancements,
video captioning models still face significant limitations in accurately describing fine-grained motion details and suffer from severe
hallucination issues. These challenges become particularly prominent when generating captions for motion-centric
videos, where precise depiction of intricate movements and limb dynamics is crucial yet often neglected. To alleviate this gap, we introduce an automated annotation pipeline that integrates kinematic-based motion computation with linguistic parsing, enabling detailed decomposition and description of complex human motions. Based on this pipeline, we construct and release the Kinematic Parsing Motion
Benchmark (KPM-Bench), a novel open-source
dataset designed to facilitate fine-grained motion understanding. KPM-Bench consists of (i) fine-grained
video-caption pairs that comprehensively illustrate limb-level dynamics in complex actions, (ii) diverse and challenging question-answer pairs focusing specifically on motion understanding, and (iii) a meticulously curated evaluation set specifically designed to assess
hallucination phenomena associated with motion descriptions. Furthermore, to address
hallucination issues systematically, we propose the linguistically grounded Motion Parsing and Extraction (MoPE)
algorithm, capable of accurately extracting motion-specific attributes directly from textual captions. Leveraging MoPE, we introduce a precise
hallucination evaluation metric that functions independently of large-scale
vision-
language or
language-only models. By integrating MoPE into the GRPO post-training framework, we effectively mitigate
hallucination problems, significantly improving the reliability of motion-centric
video captioning models.
Read the full abstract ▾
All authors Boda Lin, Yongjie Zhu, Xiaocheng Gong, Wenyu Qin, Meng Wang
Computer Vision · Cross-listed from eess.AS
Detection and Classification of Cetacean Echolocation Clicks using Image-based Object Detection Methods applied to Advanced Wavelet-based Transformations
Christopher Hauer
A challenge in marine bioacoustic analysis is the detection of animal signals, like calls, whistles and clicks, for behavioral studies. Manual labeling is too time-consuming to process sufficient data to get reasonable results. Thus, an automatic solution to overcome the time-consuming data analysis is necessary. Basic mathematical models can detect events in simple environments, but they struggle with complex scenarios, like differentiating signals with a low signal-to-noise ratio or distinguishing clicks from echoes. Deep Learning Neural Networks, such as ANIMAL-SPOT, are better suited for such tasks. DNNs process audio signals as image representations, often using spectrograms created by Short-Time
Fourier Transform. However, spectrograms have limitations due to the uncertainty principle, which creates a tradeoff between time and frequency resolution. Alternatives like the wavelet, which provides better time resolution for high frequencies and improved frequency resolution for low frequencies, may offer advantages for feature extraction in complex bioacoustic environments. This thesis shows the efficacy of CLICK-SPOT on Norwegian Killer whale
underwater recordings provided by the cetacean biologist Dr. Vester. Keywords: Bioacoustics, Deep Learning, Wavelet Transformation
Read the full abstract ▾
Computer Vision
OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents
Akashah Shabbir, Muhammad Umer Sheikh, Muhammad Akhtar Munir et al. (+8 more)
Recent progress in
multimodal reasoning has enabled
agents that interpret imagery, connect it with
language, and execute structured analytical tasks. Extending these capabilities to remote sensing remains challenging, as models must reason over spatial scale, geographic structures, and multispectral indices while maintaining coherent multi-step logic. To address this gap, we introduce OpenEarthAgent, a unified framework for tool-augmented geospatial
reasoning trained on satellite imagery, natural-
language queries, and structured
reasoning traces. Beyond serving as a
benchmark, OpenEarthAgent establishes a cohesive agentic architecture built around a unified executable tool registry and trajectory-based policy learning. The framework standardizes heterogeneous visual, spectral, GIS, and georeferenced raster operations under a consistent callable schema, enabling modular orchestration and deterministic execution. Training is performed via supervised fine-tuning on structured
reasoning trajectories with deterministic replay validation to ensure executability and spatial correctness. The accompanying corpus comprises 14,538 training and 1,169 evaluation instances with over 107K
reasoning steps, spanning urban, environmental, disaster, and infrastructure domains and incorporating GIS operations alongside index analyses such as NDVI, NBR, and NDBI. Grounded in explicit
reasoning traces, the learned
agent demonstrates structured
reasoning, stable spatial understanding, and interpretable tool-driven behaviour across diverse EO scenarios. We report consistent improvements over a strong baseline and competitive performance against recent open and closed-source models. Our code, data and trained models are publicly available: https://github.com/mbzuai-oryx/OpenEarthAgent
Read the full abstract ▾
All authors Akashah Shabbir, Muhammad Umer Sheikh, Muhammad Akhtar Munir, Hiyam Debary, Mustansar Fiaz, Muhammad Zaigham Zaheer, Paolo Fraccaro, Fahad Shahbaz Khan, Muhammad Haris Khan, Xiao Xiang Zhu, Salman Khan
Computer Vision
When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs
Yu Fang, Yuchun Feng, Dong Jing et al. (+5 more)
Vision-
Language-Action models (VLAs) promise to ground
language instructions in
robot control, yet in practice often fail to follow
language faithfully. When presented with instructions that lack strong scene-specific supervision, VLAs suffer from counterfactual failures: they act based on
vision shortcuts induced by
dataset biases, repeatedly executing well-learned behaviors and selecting objects frequently seen during training regardless of
language intent. To support systematic study, we introduce LIBERO-CF, the first counterfactual
benchmark for VLAs that evaluates
language following capability by assigning alternative, feasible task instructions under the same LIBERO scene layouts. Our evaluation reveals that counterfactual failures are prevalent yet underexplored across state-of-the-art VLAs. To mitigate these failures, we propose Counterfactual Action Guidance (CAG), a simple yet effective dual-branch inference scheme that explicitly strengthens
language conditioning in VLAs. CAG combines a standard VLA policy with a parallel
language-unconditioned branch, amplifying the difference between their action predictions induced by
language instructions. This design reduces reliance on
vision shortcuts, improves robustness on counterfactual tasks, and requires neither additional demonstrations nor modifications to the base VLA architecture or its pretrained weights. Extensive experiments demonstrate consistent improvements across diverse VLAs. On LIBERO-CF, our training-free variant, TF-CAG, improves $\pi_{0.5}$ by 9.7% in
language grounding and 3.6% in task success. When using a separate
Vision-Action (VA) module, VA-CAG achieves larger gains of 15.5% and 8.5%, respectively. In real-world evaluations, VA-CAG improves
language grounding by 9.4% and task success by 17.2% on average.
Read the full abstract ▾
All authors Yu Fang, Yuchun Feng, Dong Jing, Jiaqi Liu, Yue Yang, Zhenyu Wei, Daniel Szafir, Mingyu Ding
Computer Vision
Human-level 3D shape perception emerges from multi-view learning
Tyler Bonnen, Jitendra Malik and Angjoo Kanazawa
Humans can infer the three-dimensional structure of objects from two-dimensional visual inputs. Modeling this ability has been a longstanding goal for the science and engineering of visual intelligence, yet decades of computational methods have fallen short of human performance. Here we develop a modeling framework that predicts human 3D shape inferences for arbitrary objects, directly from experimental stimuli. We achieve this with a novel class of neural networks trained using a visual-spatial objective over naturalistic sensory data; given a set of images taken from different locations within a natural scene, these models learn to predict spatial information related to these images, such as camera location and visual depth, without relying on any object-related inductive biases. Notably, these visual-spatial signals are analogous to sensory cues readily available to humans. We design a zero-shot evaluation approach to determine the performance of these 'multi-view' models on a well established 3D perception task, then compare model and human behavior. Our modeling framework is the first to match human accuracy on 3D shape inferences, even without task-specific training or fine-tuning. Remarkably, independent readouts of model responses predict fine-grained measures of human behavior, including error patterns and reaction times, revealing a natural correspondence between model dynamics and human perception. Taken together, our findings indicate that human-level 3D perception can emerge from a simple, scalable learning objective over naturalistic visual-spatial data. Code, images, and human data needed to reproduce all analyses can be found at https://tzler.github.io/human_multiview/
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting
Xiaohan Zhao, Zhaoyi Li, Yaxin Luo et al. (+2 more)
Black-box
adversarial attacks on Large
Vision-
Language Models (LVLMs) are challenging due to missing gradients and complex
multimodal boundaries. Prior state-of-the-art transfer-based approaches such as M-Attack match local crops between source and target images, but this induces high-variance, nearly orthogonal gradients across iterations. We attribute this instability to (i) ViT translation sensitivity, which yields spike-like gradients, and (ii) a structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and introduce M-Attack-V2 based on this formulation. Multi-Crop
Alignment (MCA) averages gradients from multiple local views per iteration, while Auxiliary Target
Alignment (ATA) replaces aggressive target augmentation with a small set of semantically correlated auxiliary images. Patch Momentum replays historical crop gradients, and a patch-diverse surrogate ensemble (PE+) strengthens transferable directions. M-Attack-V2 raises attack success rates over M-Attack from 40% to 61% on GPT-6.1 Sol, from 40% to 60% on GPT-6 Astra, and from 85% to 97% on Claude Sonnet 5.5. Code and data are publicly available at: https://github.com/VILA-Lab/M-Attack-V2.
Read the full abstract ▾
All authors Xiaohan Zhao, Zhaoyi Li, Yaxin Luo, Jiacheng Cui, Zhiqiang Shen
Computer Vision
IntRec: Intent-based Retrieval with Contrastive Refinement
Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger et al. (+1 more)
Retrieving user-specified objects from complex scenes remains a challenging task, especially when queries are ambiguous or involve multiple similar objects. Existing open-vocabulary detectors operate in a one-shot manner, lacking the ability to refine predictions based on user feedback. To address this, we propose IntRec, an interactive object
retrieval framework that refines predictions based on user feedback. At its core is an Intent State (IS) that maintains dual
memory sets for positive anchors (confirmed cues) and negative constraints (rejected hypotheses). A contrastive
alignment function ranks candidate objects by maximizing similarity to positive cues while penalizing rejected ones, enabling fine-grained disambiguation in cluttered scenes. Our interactive framework provides substantial improvements in
retrieval accuracy without additional supervision. On LVIS, IntRec achieves 35.4 AP, outperforming OVMR, CoDet, and CAKE by +2.3, +3.7, and +0.5, respectively. On the challenging LVIS-Ambiguous
benchmark, it improves performance by +7.9 AP over its one-shot baseline after a single corrective feedback, with less than 30 ms of added latency per interaction.
Read the full abstract ▾
All authors Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, Yue Lu
Computer Vision
CORAL: Correspondence Alignment for Improved Virtual Try-On
Jiyoung Kim, Youngjin Shin, Siyoon Jin et al. (+6 more)
Existing methods for Virtual Try-On (VTON) often struggle to preserve fine garment details, especially in unpaired settings where accurate person-garment correspondence is required. These methods do not explicitly enforce person-garment
alignment and fail to explain how correspondence emerges within
Diffusion Transformers (DiTs). In this paper, we first analyze full 3D
attention in DiT-based architecture and reveal that the person-garment correspondence critically depends on precise person-garment query-key matching within the full 3D
attention. Building on this insight, we then introduce CORrespondence
ALignment (CORAL), a DiT-based framework that explicitly aligns query-key matching with robust external correspondences. CORAL integrates two complementary components: a correspondence
distillation loss that aligns reliable matches with person-garment
attention, and an
entropy minimization loss that sharpens the
attention distribution. We further propose a VLM-based evaluation protocol to better reflect human preference. CORAL consistently improves over the baseline, enhancing both global shape transfer and local detail preservation. Extensive ablations validate our design choices.
Read the full abstract ▾
All authors Jiyoung Kim, Youngjin Shin, Siyoon Jin, Dahyun Chung, Jisu Nam, Tongmin Kim, Jongjae Park, Hyeonwoo Kang, Seungryong Kim
Computer Vision
Adapting Actively on the Fly: Relevance-Guided Online Meta-Learning with Latent Concepts for Geospatial Discovery
Jowaria Khan, Anindya Sarkar, Yevgeniy Vorobeychik et al. (+1 more)
In environmental monitoring, data collection is often costly, sparse, and shaped by urgent public-health needs. This is particularly true for cancer-causing PFAS (Per- and polyfluoroalkyl substances) contamination, where discussions with domain experts and environmental organizations highlight the need to strategically identify high-risk, under-observed regions under tight sampling budgets. More broadly, similar challenges arise in disaster response and public health settings, where dynamic environments make it essential to efficiently uncover hidden targets from limited ground truth. Yet sparse and biased geospatial labels limit the applicability of existing learning-based methods, such as reinforcement learning. To address this, we propose a unified geospatial discovery framework that integrates active learning, online meta-learning, and concept-guided
reasoning. Our approach introduces two key innovations built on a shared notion of *concept relevance*, capturing how domain-specific factors influence target presence: a *concept-weighted uncertainty sampling strategy*, where uncertainty is modulated by learned relevance from readily available concepts such as land cover and source proximity; and a *relevance-aware meta-batch formation strategy* that promotes semantic diversity during online-meta updates, improving generalization in dynamic environments. We evaluate our framework on PFAS contamination discovery as a real-world inspired environmental monitoring task, demonstrating robust target discovery under limited data and changing conditions.
Read the full abstract ▾
All authors Jowaria Khan, Anindya Sarkar, Yevgeniy Vorobeychik, Elizabeth Bondi-Kelly
Computer Vision
Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment
Ivan Rinaldi, Matteo Mendula, Nicola Fanelli et al. (+4 more)
Music generation has advanced markedly through
multimodal deep learning, enabling models to synthesize audio from text and, more recently, from images. However, existing image-conditioned systems suffer from two fundamental limitations: (i) they are typically trained on natural photographs, limiting their ability to capture the richer semantic, stylistic, and cultural content of artworks; and (ii) most rely on an image-to-text conversion stage, using
language as a semantic shortcut that simplifies conditioning but prevents direct visual-to-audio learning. Motivated by these gaps, we introduce ArtSound, a large-scale
multimodal dataset of 105,884 artwork-music pairs enriched with dual-modality captions, obtained by extending ArtGraph and the Free Music Archive. We further propose ArtToMus, the first framework explicitly designed for direct artwork-to-music generation, which maps digitized artworks to music without image-to-text translation or
language-based semantic supervision. The framework projects visual
embeddings into the conditioning space of a latent
diffusion model, enabling music synthesis guided solely by visual information. Experimental results show that ArtToMus generates musically coherent and stylistically consistent outputs that reflect salient visual cues of the source artworks. While absolute
alignment scores remain lower than those of text-conditioned systems-as expected given the substantially increased difficulty of removing linguistic supervision-ArtToMus achieves competitive perceptual quality and meaningful cross-modal correspondence. This work establishes direct visual-to-music generation as a distinct and challenging research direction, and provides resources that support applications in multimedia art, cultural heritage, and AI-assisted creative practice. Code and
dataset will be publicly released upon acceptance.
Read the full abstract ▾
All authors Ivan Rinaldi, Matteo Mendula, Nicola Fanelli, Florence Levé, Matteo Testi, Giovanna Castellano, Gennaro Vessio
Computer Vision · Cross-listed from cs.RO
FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations
Konstantinos Foteinos, Georgios Angelidis, Aggelos Psiris et al. (+3 more)
The ever increasing intensity and number of disasters make even more difficult the work of First Responders (FRs). Artificial intelligence and robotics solutions could facilitate their operations, compensating these difficulties. To this end, we propose a
dataset for gesture-based UGV control by FRs, introducing a set of 12 commands, drawing inspiration from existing gestures used by FRs and tactical hand signals and refined after incorporating feedback from experienced FRs. Then we proceed with the data collection itself, resulting in 3312 RGBD pairs captured from 2 viewpoints and 7 distances. To the best of our knowledge, this is the first
dataset especially intended for gesture-based UGV guidance by FRs. Finally we define evaluation protocols for our RGBD
dataset, termed FR-GESTURE, and we perform baseline experiments, which are put forward for improvement. We have made data publicly available to promote future research on the domain: https://doi.org/10.5281/zenodo.18131333.
Read the full abstract ▾
All authors Konstantinos Foteinos, Georgios Angelidis, Aggelos Psiris, Vasileios Argyriou, Panagiotis Sarigiannidis, Georgios Th. Papadopoulos
Computer Vision
RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward
Qiucheng Wu, Jing Shi, Simon Jenni et al. (+4 more)
Recent advances in
multimodal large
language models (MLLMs) have shown great potential for extending
vision-
language reasoning to professional tool-based image editing, enabling intuitive and creative editing. A promising direction is to use reinforcement learning (RL) to enable MLLMs to reason about and execute optimal tool-use plans within professional image-editing software. However, training remains challenging due to the lack of reliable, verifiable reward signals that can reflect the inherently subjective nature of creative editing. In this work, we introduce RetouchIQ, a framework that performs instruction-based executable image editing through MLLM
agents guided by a generalist reward model. RetouchIQ interprets user-specified editing intentions and generates corresponding, executable image adjustments, bridging high-level aesthetic goals with precise parameter control. To move beyond conventional, rule-based rewards that compute similarity against a fixed reference image using handcrafted metrics, we propose a generalist reward model, an RL fine-tuned MLLM that evaluates edited results through a set of generated metrics on a case-by-case basis. Then, the reward model provides scalar feedback through
multimodal reasoning, enabling reinforcement learning with high-quality, instruction-consistent gradients. We curate an extended
dataset with 190k instruction-
reasoning pairs and establish a new
benchmark for instruction-based image editing. Experiments show that RetouchIQ substantially improves both semantic consistency and perceptual quality over previous MLLM-based and
diffusion-based editing systems. Our findings demonstrate the potential of generalist reward-driven MLLM
agents as flexible, explainable, and executable assistants for professional image editing.
Read the full abstract ▾
All authors Qiucheng Wu, Jing Shi, Simon Jenni, Kushal Kafle, Tianyu Wang, Shiyu Chang, Handong Zhao
Computer Vision · Cross-listed from q-bio.NC
Probability-Invariant Random Walk Learning on Gyral Folding-Based Cortical Similarity Networks for Alzheimer's and Lewy Body Dementia Diagnosis
Minheng Chen, Tong Chen, Chao Cao et al. (+4 more)
Alzheimer's disease (AD) and Lewy body dementia (LBD) present overlapping clinical features yet require distinct diagnostic strategies. While neuroimaging-based
brain network analysis is promising, atlas-based representations may obscure individualized anatomy. Gyral folding-based networks using three-hinge gyri provide a biologically grounded alternative, but inter-individual variability in cortical folding results in inconsistent landmark correspondence and highly irregular network sizes, violating the fixed-
topology and node-
alignment assumptions of most existing graph learning methods, particularly in clinical
datasets where pathological changes further amplify anatomical heterogeneity. We therefore propose a probability-invariant random-walk-based framework that classifies individualized gyral folding networks without explicit node
alignment. Cortical similarity networks are built from local morphometric features and represented by distributions of anonymized random walks, with an anatomy-aware encoding that preserves permutation invariance. Experiments on a large clinical cohort of AD and LBD subjects show consistent improvements over existing gyral folding and atlas-based models, demonstrating robustness and potential for dementia diagnosis.
Read the full abstract ▾
All authors Minheng Chen, Tong Chen, Chao Cao, Jing Zhang, Tianming Liu, Li Su, Dajiang Zhu
Computer Vision · Cross-listed from eess.SP
Neural Implicit Representations for 3D Synthetic Aperture Radar Imaging
Nithin Sugavanam and Emre Ertin
Synthetic aperture radar (SAR) is a tomographic sensor that measures 2D slices of the 3D spatial
Fourier transform of the scene. In many operational scenarios, the measured set of 2D slices does not fill the 3D space in the
Fourier domain, resulting in significant artifacts in the reconstructed imagery. Traditionally, simple priors, such as sparsity in the image domain, are used to regularize the inverse problem. In this paper, we review our recent work that achieves state-of-the-art results in 3D SAR imaging employing neural structures to model the surface scattering that dominates SAR returns. These neural structures encode the surface of the objects in the form of a signed distance function learned from the sparse scattering data. Since estimating a smooth surface from a sparse and noisy point cloud is an ill-posed problem, we regularize the surface estimation by sampling points from the implicit surface representation during the training step. We demonstrate the model's ability to represent target scattering using measured and simulated data from single vehicles and a larger scene with a large number of vehicles. We conclude with future research directions calling for methods to learn complex-valued neural representations to enable synthesizing new collections from the volumetric neural implicit representation.
Read the full abstract ▾
Computer Vision
GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinking
Zixu Cheng, Da Li, Jian Hu et al. (+4 more)
Video reasoning requires a fine-grained understanding of the temporal dependencies and event-level relations between objects and events in
videos. Current
Multimodal Large
Language Models (MLLMs) are prone to severe temporal
hallucinations in
video reasoning. An underlying cause of these
hallucinations is weak visual-temporal grounding and the lack of explicit structure for modelling event relations. Models often rely on auxiliary text, such as dense captions, rather than explicitly anchoring their
reasoning in actual visual evidence. However, these textual representations are inherently unstructured and fail to provide explicit causal constraints needed to guide the model's
reasoning. In this work, we propose GraphThinker, a reinforcement finetuning method that constructs a structured event representation of a
video and enforces visual grounding to jointly reduce
reasoning hallucinations. Specifically, we employ an MLLM to construct an Event-based
Video Scene Graph (EVSG) that captures both intra- and inter-event relations, guiding a structured
video reasoning process. Moreover, we address the weak grounding issue by introducing a novel visual
attention reward during reinforcement finetuning that encourages the model to actively attend to reliable visual cues. On the RexTime
dataset, GraphThinker achieves an over 4% improvement in IoU=0.3 for moment localisation. On the VidHalluc
dataset, GraphThinker achieves a 9.8% improvement in reducing temporal sequence
hallucination and a 7.6% gain in Binary QA in reducing action
hallucination, compared to the state-of-the-art methods.
Read the full abstract ▾
All authors Zixu Cheng, Da Li, Jian Hu, Yuhang Zang, Ziquan Liu, Shaogang Gong, Wei Li
Computer Vision
LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs
Behzad Bozorgtabar, Dwarikanath Mahapatra, Sudipta Roy et al. (+3 more)
Medical
vision-
language models (VLMs) are strong zero-shot recognizers for medical imaging, but their reliability under domain shift hinges on calibrated uncertainty with guarantees. Split conformal prediction (SCP) offers finite-sample coverage, yet prediction sets often become large (low efficiency) and class-wise coverage unbalanced-high class-conditioned coverage gap (CCV), especially in few-shot, imbalanced regimes; moreover, naively adapting to calibration labels breaks exchangeability and voids guarantees. We propose \texttt{LATA} (
Laplacian-Assisted Transductive Adaptation), a training- and label-free refinement that operates on the joint calibration and test pool by smoothing zero-shot probabilities over an image-image k-NN graph using a small number of CCCP mean-field updates, preserving SCP validity via a deterministic transform. We further introduce a failure-aware conformal score that plugs into the
vision-
language uncertainty (ViLU) framework, providing instance-level difficulty and label plausibility to improve prediction set efficiency and class-wise balance at fixed coverage. \texttt{LATA} is black-box (no VLM updates), compute-light (windowed transduction, no backprop), and includes an optional prior knob that can run strictly label-free or, if desired, in a label-informed variant using calibration marginals once. Across three medical VLMs and nine downstream tasks, \texttt{LATA} consistently reduces set size and CCV while matching or tightening target coverage, outperforming prior transductive baselines and narrowing the gap to label-using methods, while using far less compute. Comprehensive ablations and qualitative analyses show that \texttt{LATA} sharpens zero-shot predictions without compromising exchangeability.
Read the full abstract ▾
All authors Behzad Bozorgtabar, Dwarikanath Mahapatra, Sudipta Roy, Muzammal Naseer, Imran Razzak, Zongyuan Ge
Computer Vision
Depth Augmented and FE Free 3D/2D Liver Registration for Laparoscopic Liver AR
Hanyuan Zhang, Lucas He, Runlong He et al. (+6 more)
Augmented reality (AR) guidance in laparoscopic liver surgery requires accurate registration of preoperative 3D models to intraoperative 2D
video, but remains challenging due to partial visibility, specularities, and tissue deformation. Existing methods often rely on contour-based rigid initialization and finite-element (FE) models for deformable registration, increasing modeling and engineering complexity. We present a depth-augmented, FE-free 3D–2D registration pipeline that combines robust rigid initialization with patient-specific non-rigid refinement. For rigid
alignment, we adapt the RefineNet module of FoundationPose to laparoscopic liver scenes by using multi-class contour maps and
monocular depth for relative pose refinement. For deformable
alignment, we construct a patient-specific statistical deformation model from non-rigid ICP (NICP) correspondences and optimize pose and shape parameters using a coarse-to-fine L-BFGS-B strategy. On a public clinical laparoscopic liver
dataset, the proposed method achieves a mean target registration error (TRE) of 14.73\,mm under a controlled manual-contour setting designed to isolate registration performance. Ablation studies show that
monocular depth improves rigid initialization over contour-only inputs, while tumor-mapping analysis indicates that good surface
alignment does not necessarily translate into lower target localization error. On an external
dataset without ground truth, the method produces visually plausible overlays for qualitative assessment. These results suggest that depth-augmented pose refinement and FE-free statistical deformation modeling provide a promising alternative to FE-based pipelines for controlled 3D–2D liver registration in
surgical AR.
Read the full abstract ▾
All authors Hanyuan Zhang, Lucas He, Runlong He, Weixi Yi, Abdolrahim Kadkhodamohammadi, Danail Stoyanov, Brian R. Davidson, Evangelos B. Mazomenos, Matthew J. Clarkson
Computer Vision
Tracing Copied Pixels and Regularizing Patch Affinity in Copy Detection
Yichen Lu, Siwei Nie, Minlong Lu et al. (+3 more)
Image Copy Detection (ICD) aims to identify manipulated content between image pairs through robust feature representation learning. While self-supervised learning (SSL) has advanced ICD systems, existing view-level contrastive methods struggle with sophisticated edits due to insufficient fine-grained correspondence learning. We address this limitation by exploiting the inherent
geometric traceability in edited content through two key innovations. First, we propose PixTrace - a pixel coordinate tracking module that maintains explicit spatial mappings across editing transformations. Second, we introduce CopyNCE, a geometrically-guided contrastive loss that regularizes patch affinity using overlap ratios derived from PixTrace's verified mappings. Our method bridges pixel-level traceability with patch-level similarity learning, suppressing supervision noise in SSL training. Extensive experiments demonstrate not only state-of-the-art performance (88.7% uAP / 83.9% RP90 for matcher, 72.6% uAP / 68.4% RP90 for descriptor on DISC21
dataset) but also better interpretability over existing methods.
Read the full abstract ▾
All authors Yichen Lu, Siwei Nie, Minlong Lu, Xudong Yang, Xiaobo Zhang, Peng Zhang
Computer Vision
QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery
Xuan-Bac Nguyen, Hoang-Quan Nguyen, Sankalp Pandey et al. (+4 more)
Characterizing two-dimensional
quantum materials from optical microscopy images is challenging due to the subtle layer-dependent contrast, limited labeled data, and significant variation across laboratories and imaging setups. Existing
vision models struggle in this domain since they lack physical priors and cannot generalize to new materials or hardware conditions. This work presents a new physics-aware
multimodal framework that addresses these limitations from both the data and model perspectives. We first present Synthia, a physics-based synthetic data generator that simulates realistic optical responses of
quantum material flakes under thin-film interference. Synthia produces diverse and high-quality samples, helping reduce the dependence on expert manual annotation. We introduce QMat-Instruct, the first large-scale instruction
dataset for
quantum materials, comprising
multimodal, physics-informed question-answer pairs designed to teach
Multimodal Large
Language Models (MLLMs) to understand the appearance and thickness of flakes. Then, we propose Physics-Aware Instruction Tuning (QuPAINT), a
multimodal architecture that incorporates a Physics-Informed
Attention module to fuse visual
embeddings with optical priors, enabling more robust and discriminative flake representations. Finally, we establish QF-Bench, a comprehensive
benchmark spanning multiple materials, substrates, and imaging settings, offering standardized protocols for fair and reproducible evaluation.
Read the full abstract ▾
All authors Xuan-Bac Nguyen, Hoang-Quan Nguyen, Sankalp Pandey, Tim Faltermeier, Nicholas Borys, Hugh Churchill, Khoa Luu