Computer Vision
DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrieval
Kaleem Ullah, Altaf Hussain, Muhammad Munsif et al. (+1 more)
In today's media-driven world, the exponential growth of
video content across domains such as surveillance, education, and entertainment has made retrieving semantically relevant
videos via natural
language queries increasingly critical. Early
video retrieval systems relied on handcrafted features or shallow cross-modal mappings, limiting their ability to capture complex semantics and temporal dynamics. While large-scale
vision-
language models have improved cross-modal
alignment, challenges remain in modeling fine-grained temporal dependencies and nuanced linguistic structures. In this paper, we introduce DREAM: Dual-path Representation Enhancement and
Alignment Model, a novel
multimodal framework that addresses these limitations through enhanced visual and textual encoding. DREAM incorporates a hybrid
language modeling strategy that combines masked and permuted
language modeling objectives to capture both local and global linguistic semantics. On the visual side, we design a hierarchical
vision encoder with cascaded group
attention, which integrates spatial and temporal information through multi-stage
token interaction and coarse-to-fine
attention refinement. We validate DREAM through comprehensive evaluations on the widely-used MSRVTT, MSVD and LSMDC
benchmark datasets, where it achieves new state-of-the-art R1 scores of 49.4%, 49.7% and 27.3%, respectively. Qualitative analyses further show the model's ability to maintain coherent
attention across frames and align complex queries with dynamic
video content. These findings underscore the effectiveness of hierarchical
attention and dual-objective textual modeling in enabling robust, context-aware
video retrieval, and pave the way for future research in advancing cross-modal representation learning.
Read the full abstract ▾
All authors Kaleem Ullah, Altaf Hussain, Muhammad Munsif, Sung Wook Baik
Computer Vision
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
Hong-Tao Yu, Chen-Wei Xie, Yuxin Peng et al. (+2 more)
Recent advancements in Large
Vision-
Language Models (LVLMs) have demonstrated remarkable
multimodal perception and
reasoning capabilities. While numerous
benchmarks have evaluated LVLMs from holistic or task-specific perspectives, their capabilities on fine-grained image tasks-fundamental to computer
vision-remain insufficiently understood. To address this gap, we introduce FG-BMK, a comprehensive fine-grained evaluation
benchmark containing 1.01 million questions and 0.28 million images, covering diverse scenarios from common object-centric domains to specialized domains. FG-BMK jointly evaluates dialogue-level fine-grained semantic recognition and feature-level visual discriminability through human-oriented and machine-oriented paradigms, enabling diagnostic analysis of whether LVLM failures arise from insufficient visual representations, weak visual-to-semantic grounding, or limited fine-grained knowledge. Through extensive experiments on a diverse set of representative LVLMs/VLMs, we find that current LVLMs remain inadequate fine-grained recognizers, with failures arising from intertwined bottlenecks in visual representations, semantic grounding, modality
alignment, and category-level knowledge. We further analyze training design factors for improving fine-grained capabilities and examine how visual and linguistic perturbations affect LVLM predictions. These findings provide diagnostic insights into the limitations of current LVLMs and offer guidance for future data construction and model design in developing more reliable LVLMs for fine-grained visual tasks. Our code is open-source and available at https://fg-bmk.github.io/.
Read the full abstract ▾
All authors Hong-Tao Yu, Chen-Wei Xie, Yuxin Peng, Serge Belongie, Xiu-Shen Wei
Computer Vision
Low-Rank Tensor Completion Based on Fractional Regularization with Ky Fan p-k Norm
Shan Fan, Feng Zhang, Jianjun Wang et al. (+2 more)
This paper addresses low-rank tensor completion (LRTC) by proposing a novel nonconvex surrogate, namely the ratio of the tensor
nuclear norm to the tensor Ky Fan p-k norm (TNPK), to accurately approximate the tensor tubal rank. The TNPK possesses appealing properties, including scale invariance, parameter flexibility, and the existence of closed-form solutions under specific choices of p and k. With specific parameter settings of p and k, it reduces to the ratio of the tensor
nuclear norm to the tensor Ky Fan k norm (TNK) or the ratio of the tensor
nuclear norm to the tensor Frobenius norm (TNF). We construct a LRTC model and, under the tensor null space property (NSP), prove that low-rank tensors are local minimizers of the proposed model. Moreover, we derive the proximal operator of the Ky Fan p-k inverse-norm and further develop an efficient alternating direction method of multipliers (ADMM)
algorithm with guaranteed subsequential convergence under mild conditions. Extensive experiments on synthetic and real-world
datasets validate the superior performance of our method against state-of-the-art competitors.
Read the full abstract ▾
All authors Shan Fan, Feng Zhang, Jianjun Wang, Xi-Le Zhao, Tingwen Huang
Computer Vision
FlowObject: Flow Steering for Bridging Generative Priors and Reconstruction Fidelity
Yuchen Rao, Xuqian Ren, Yinyu Nie et al. (+4 more)
Recovering complete 3D representations of objects from few casual image captures remains a significant challenge. Recent 3D generative models, particularly those based on Flow-Matching (FM), can synthesize high-quality textured assets; however, they often suffer from ”synthetic bias” where learned priors override observational evidence, alongside a lack of
alignment with the observed instance. Conversely, optimization-based methods like 3D Gaussian Splatting (3DGS) provide high fidelity on visible surfaces but fail to reason about unobserved geometry. In this paper, we present FlowObject, a framework that reformulates sparse-view 3D reconstruction as a training-free, guided inverse problem. Our approach applies a dual-space guidance strategy to steer the Ordinary Differential Equation (ODE) trajectory of a flow-matching model, enabling the completion of unseen regions through learned generative priors while enforcing strict consistency with real-world observations. By integrating a 3DGS refinement stage, FlowObject further bridges the gap between ”synthetic-looking” generative outputs and photorealistic reconstructions. Comprehensive
benchmarks on synthetic and real-world
datasets demonstrate that current state-of-the-art methods often struggle to achieve
geometric completeness and observational consistency simultaneously, especially under severe occlusions. In contrast, our method significantly outperforms state-of-the-art generative models and optimization-based frameworks in both
geometric completeness and view-dependent appearance fidelity.
Read the full abstract ▾
All authors Yuchen Rao, Xuqian Ren, Yinyu Nie, Sayan Deb Sarkar, Biao Zhang, Vincent Lepetit, Friedrich Fraundorfer
Computer Vision
Show, Don't Ask: Generative Visual Disambiguation for Composed Image Retrieval with Turn-Valid Coverage
Amsisan Tran, Baogh Le, Tuan Kiet Pham et al. (+1 more)
Composed image
retrieval (CIR) uses a reference image and a text modification to search for a target image. However, such queries often describe several possible images rather than one exact target, making the user's intent ambiguous. Recent methods address this by using conformal prediction to estimate ambiguity and by asking users clarifying text questions. However, these methods have two limitations: their coverage guarantee only holds at the first interaction, and text questions are often insufficient for resolving fine-grained visual differences such as appearance, attributes, or viewpoint. We propose CLARA, a clarification framework that resolves ambiguity by showing users a small panel of visual alternatives. Instead of answering text questions, the user simply selects the prototype image closest to the intended target. This provides a direct visual signal and avoids relying on a model to predict the user's answer. To maintain valid conformal guarantees across multiple interaction rounds, CLARA reweights calibration using the likelihood ratio induced by the user's selection. The displayed prototypes are also constrained to represent the current candidate set and are snapped to real corpus images, ensuring that generated images cannot artificially improve coverage. Experiments on open-domain and fashion
benchmarks show that CLARA matches single-turn state-of-the-art
retrieval performance, maintains nominal coverage across interaction rounds, and finds the intended target in fewer rounds than strong text-question baselines. Its advantage is especially clear when ambiguity involves viewpoint or fine-grained attributes, where visual clarification is more effective than textual questioning.
Read the full abstract ▾
All authors Amsisan Tran, Baogh Le, Tuan Kiet Pham, Sui Yang Guang
Computer Vision
Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning
Pengyu Li, Zhitao Gao, Lingling Zhang et al. (+5 more)
Unified
multimodal models (UMMs) interleave generated ”visual thoughts” (VTs) with text
reasoning to improve spatial tasks. This incurs roughly an order-of-magnitude inference cost from multi-step
diffusion. We find this cost yields limited direct benefit. On ThinkMorph, removing or noising VTs barely changes accuracy across nine
benchmarks. Once rendered,
attention concentrates on the VT regardless of content. Yet a KL diagnostic shows that conditioning on a privileged VT trace shifts the model's completion distribution. This suggests the generation pathway encodes useful
reasoning beyond the rendered pixels. Motivated by this gap, we propose Visual On-Policy Self-
Distillation(Visual-OPSD). Teacher and student share identical weights but differ in context: the teacher sees privileged VTs while the student sees only the question.
Token-level JSD
distillation on on-policy student trajectories transfers the teacher's
reasoning to a text-only student. Across nine
benchmarks, Visual-OPSD improves over its generative teacher by $+3.40$pp with $14.3\times$ speedup (10.0s vs. 142.8s per sample) and outperforms same-scale VLMs by $+63.83$pp on VSP. A Gaussian-noise control ($+0.40$pp vs. $+10.28$pp for real VTs) and $58.4\%$ closure of the KL gap confirm that gains come from the semantic content of the generation pathway.
Read the full abstract ▾
All authors Pengyu Li, Zhitao Gao, Lingling Zhang, Muye Huang, Yuanming Li, Zesheng Yang, Fangzhi Xu, Jun Liu
Computer Vision · Cross-listed from cs.LG
A Controlled Benchmark of Quantum-Latent GAN Augmentation for Brain MRI
Syed Mujtaba Haider and Silvia Figini
Medical image classification is often constrained by limited labeled data, motivating generative augmentation; recently,
quantum generative models have been proposed for this purpose, frequently reporting accuracy gains. However, such claims are typically based on single training runs, do not match the parameter budgets of the
quantum and classical generators, and do not characterize the data regime in which any benefit appears. We present a controlled
benchmark that isolates the contribution of a
quantum generator to
brain-MRI augmentation. Images are encoded into a KL-regularized latent space in which a conditional Wasserstein GAN with gradient penalty is trained using either a variational
quantum generator or a classical generator of near-identical parameter count (1648 vs. 1632). Synthetic samples are decoded and used to augment a pretrained classifier across labeled data fractions from 5% to 100%, evaluated over eight random seeds with paired significance testing (with multiple-comparison correction) and with intraset diversity and latent-distribution analyses. Across all fractions, no augmentation variant significantly outperforms real-data-only training, and the
quantum and classical generators are statistically indistinguishable. Any low-data benefit behaves as regularization rather than faithful data expansion:synthetic samples are off distribution and severely mode collapsed precisely where data is scarce, and the
quantum generator is no more diverse thanits classical counterpart. We release the protocol as a testbed for rigorous evaluation of
quantum generative augmentation in medical imaging.
Read the full abstract ▾
Computer Vision
Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation
Zirui Zheng, Jiaqian Yu, Xiongfeng Peng et al. (+7 more)
Action-conditioned world models have emerged as a promising paradigm for
robot learning, offering a scalable alternative to costly real-world experimentation by generating action-consistent
video rollouts. However, persistent world modeling remains challenging in manipulation: frequent end-effector occlusions and rapid wrist-camera motion make the current observation insufficient for predicting future views, causing models to forget or hallucinate scene details seen in earlier frames. Existing
memory retrieval strategies often fail to identify informative history in dynamic manipulation scenarios. To address this limitation, we propose Mem-World, a
memory-augmented multi-view action-conditioned world model. At its core, we present W-VMem, a 4D wrist-view-centered surfel-indexed
memory that anchors historical observations to temporally evolving surface elements. By explicitly modeling when and where scene elements are observed, W-VMem enables geometry-aware
retrieval of relevant history frames conditioned on future actions. During generation, relevant history frames are selected via surfel-based rendering and scoring, providing informative and non-redundant context for prediction. Extensive experiments show that Mem-World generates persistent rollouts in complex manipulation scenarios, enables more reliable policy evaluation than Ctrl-World, improving the Pearson correlation with real-world performance by 14.5%, and supports effective policy improvement through synthetic data generation, increasing success rates from 58% to 72% on long-horizon tasks.
Read the full abstract ▾
All authors Zirui Zheng, Jiaqian Yu, Xiongfeng Peng, jun shi, Mingyi Li, Chao Zhang, Weiming Li, Dong Wang, Huchuan Lu, Xu Jia
Computer Vision
Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos
Runze Xu, Yiluo Zhang, Jian Wang et al. (+2 more)
Training generalist
Vision-
Language-Action(VLA) models typically requires massive, diverse robotic
datasets with high-fidelity action annotations. While
egocentric human manipulation
videos are abundant and capture significant environmental diversity, the absence of action labels makes them difficult to use in conventional training paradigms. To address this, we propose a latent-action-based framework designed to extract general action priors from unlabeled human
videos. The architecture features a Hybrid Disentangled VQ-VAE that decouples motion dynamics from environmental backgrounds through physical masks, enabling the construction of a cross-embodiment action codebook. By pre-training on human
videos with the codebook, the VLM backbone learns deep representations of action intent. For adaptation to specific embodiments, we introduce an intent-perception decoupling strategy where the VLM predicts the action intent while a separate frozen visual encoder provides state-specific features to the action expert, thereby reducing action
hallucinations. Results in simulation and real-world environments show that our method, pre-trained exclusively on unlabeled human
videos, performs competitively with state-of-the-art VLA models trained on massive annotated
datasets, requiring only 50 trajectories for downstream adaptation.
Read the full abstract ▾
All authors Runze Xu, Yiluo Zhang, Jian Wang, Yu Wang, Jincheng Yu
Computer Vision
Physics-IQ Verified
Tim Rädsch, Yuki M Asano, Hilde Kuehne et al. (+4 more)
Video generative models ( VGMs) have become a new frontier that can be used not just for
video generation but for a multitude of downstream tasks, including world modeling. To advance these tasks, a good
video model must understand the physical reality of the world. Evaluating this understanding is an emerging field and has led to the Physics-IQ
benchmark, which quantifies this explicitly by comparing model-generated
videos to real-world
videos of physical experiments. In this work, we present a systematic audit of the Physics-IQ
benchmark, expose shortcomings and propose three solutions that sharpen how we can measure physical understanding of VGMs. Specifically, we improve
prompt and ground-truth quality to reduce the influence of confounding factors and further introduce a sample-level scoring system that weights each sample and metric equally. Our resulting
benchmark, Physics-IQ Verified, refines 57.6% of all samples and improves over 34.8% of
prompts. In a comparison study using six image-to-
video generative models, we observe moderate but meaningful ranking changes (Kendall's $\tau = 0.46$). We hope Physics-IQ Verified advances the community by providing a more reliable signal toward physically accurate VGMs. The code for the
benchmark can be accessed at https://github.com/google-deepmind/physics-iq-
benchmark
Read the full abstract ▾
All authors Tim Rädsch, Yuki M Asano, Hilde Kuehne, Stefan Bauer, Priyank Jaini, Robert Geirhos, Carsten T. Lüth
Computer Vision
BindEdit: Taming Attention Leakage for Precise Multi-Object Image Editing
Chaewon Park, Soyoon Lee, Naeun Lee et al. (+3 more)
Real image editing enables precise manipulation of visual content, yet existing methods often fail in complex multi-object scenarios, causing semantic blending, object duplication, or incomplete edits. We attribute these failures to
attention leakage, where signals across spatial regions and text
tokens become entangled during the denoising process. Specifically, we identify two distinct forms of leakage: Edit-
Token Leakage, where ambiguous
token-region
alignment leads to object blending, and Source Dominance Leakage, where
tokens of unchanged source objects overwhelm the
attention intended for target entities. To resolve these leakages, we propose BindEdit, which enforces
attention-level constraints within a single
diffusion trajectory. To suppress Edit-
Token Leakage, BindEdit jointly regularizes cross- and self-
attention so that each target
token group is bound to its corresponding spatial region while maintaining instance-level separation. To suppress Source Dominance Leakage, a cross-
attention re-balancing mechanism amplifies target
token influence and attenuates residual source semantics within editable regions. Moreover, a region fidelity term ensures that each target concept is expressed coherently across the entire editing mask. Additionally, we propose a comprehensive multi-object
benchmark encompassing diverse object counts and categories. Extensive experiments demonstrate that BindEdit consistently outperforms existing methods within a single
diffusion trajectory, maintaining robust performance across both single- and multi-object editing scenarios.
Read the full abstract ▾
All authors Chaewon Park, Soyoon Lee, Naeun Lee, Minjung Shin, Seogkyu Jeon, Kibeom Hong
Computer Vision
Automatic ply-specific analyses of CFRP micrographs using shortest-path-based ply distinction
Jonas Naumann, Jonas P. Appels, Julius Biermann et al. (+3 more)
We present an automated approach to distinguish between ply instances in semantic
segmentation masks of high-resolution carbon-fiber reinforced
polymer micrographs. Interpreting the
segmentation mask as a graph with pixels as vertices, enables us to use a shortest-path
algorithm yielding the ply-separating paths. Thereby, we bridge the gap between semantic
segmentation and ply instance
segmentation using global information. We successfully apply our approach on high-resolution micrographs featuring a broad range of characteristics like artificially added gaps in single or multiple plies, different stacking sequences and ply traversing cracks. Assigning each fiber pixel to a ply based on the calculated paths, allows for a comprehensive, quantitative ply analysis with respect to its microstructural properties like the local fiber volume fraction as well as locally resolved ply and interleaf layer thickness. These insights help to reveal manufacturing-induced inhomogeneities, draw conclusions on manufacturing parameters and link mechanical properties to underlying microstructural imperfections.
Read the full abstract ▾
All authors Jonas Naumann, Jonas P. Appels, Julius Biermann, Christopher Gorsky, Timo de Wolff, Christoph Brauer
Computer Vision
DINO-Med3D: Bridging Dimension and Domain Gaps in Volumetric Segmentation via Progressive Adaptation
Haoyu Hu, Xiyao Ma, Shiqi Liu et al. (+4 more)
Although DINOv3 has demonstrated remarkable semantic discrimination in natural imagery, its direct application to volumetric medical
segmentation is hindered by inherent dimension and domain disparities. To resolve these issues, we propose DINO-Med3D, a two-stage progressive framework that repurpose the pre-trained DINOv3 encoder for 3D medical tasks. In the first stage, we mitigate the dimension gap by introducing a multi-slice
embedding module that incorporates pseudo-3D context, while simultaneously employing a
segmentation proxy task to adapt representations learned from natural scenes to the medical domain. Subsequently, we further enhance volumetric understanding by adding lightweight 3D adapters into the frozen backbone to enforce global inter-slice continuity. Finally, to compensate for the spatial information loss inherent in the
embedding process, we design a parallel detail recovery stream to explicitly preserve high-frequency boundary cues. Extensive experiments on five public
datasets demonstrate that our approach successfully adapts DINOv3 to the medical domain and significantly outperforms state-of-the-art baselines.
Read the full abstract ▾
All authors Haoyu Hu, Xiyao Ma, Shiqi Liu, Linsen Zhang, Xiaoliang Xie, Xiaohu Zhou, Zeng-Guang Hou
Computer Vision
LARE: Low-Attention Region Encoding for Text-Image Retrieval
Abdulmalik Alquwayfili, Faisal Almeshal, Jumanah Almajnouni et al. (+8 more)
Image
retrieval in crowded scenes is particularly challenging due to the salience bias of conventional visual encoders, which tend to focus on dominant objects while neglecting low-
attention regions that are often crucial for fine-grained
retrieval. We propose LARE (Low-
Attention Region Encoding), a framework that explicitly models these overlooked regions. LARE adopts a dual-encoding strategy that encodes low-
attention regions of an image and the full image in parallel, leading to more diverse and informative image
embeddings. To evaluate image
retrieval performance in challenging crowded scenes, we introduce Dense-Set, a challenging subset derived from COCO and Flickr30K. In this subset, images are re-captioned to provide richer descriptions of low-
attention or previously overlooked regions. This
dataset highlights the limitations of existing
retrieval models and enables a more rigorous evaluation under densely crowded scene conditions. Experimental results demonstrate that the proposed framework improves
retrieval performance by preserving subtle, non-dominant visual cues within the shared latent space.
Read the full abstract ▾
All authors Abdulmalik Alquwayfili, Faisal Almeshal, Jumanah Almajnouni, Leena Alotaibi, Faisal Alhajari, Mohammed Alkhrashi, Alreem Almuhrij, Abdullah Aldwyish, Raied Aljadaany, Huda Alamri, Muhammad Kamran J. Khan
Computer Vision
Performance Gap Analysis between Latin and Arabic Scripts HTR
Sana Al-azzawi, Elisa Barney and Marcus Liwicki
Recent studies have shown that handwritten text recognition (HTR) systems perform worse on Arabic-script
datasets than on Latin-script data. However, the reasons for this gap are still not well understood due to the lack of controlled comparisons. In this work, we present a comprehensive study of Arabic and Latin scripts HTR using a unified CRNN model for line-level HTR across nine
datasets (including KHATT (Arabic), Muharaf (Arabic), NUST-UHWR (Urdu), PHTD (Persian), IAM (English), READ-2016 (German), and others) and di ferent training sizes (K in {100, 500, 1000, 2000, ..., Kfull}). Our results show the performance gap remains: it is large in low-resource settings, decreases with more data, but remains even at full scale, with a consistent difference of 5-7 CER points. We show that annotation quality matters, as many
datasets contain labeling errors. Cleaning reduces error rates and narrows the gap, but does not eliminate it. In addition, we find that a fixed number of training samples provides less effective coverage in Arabic due to higher visual variability, requiring more data to learn similar representations. We compare recognition across
datasets in terms of the number of text lines and the number of characters, showing an equivalence trade-off. We compare character frequency distributions across scripts and show that Arabic is significantly more heavy-tailed than Latin. Our error analysis reveals that around 30 percent of substitution errors in Arabic
datasets (e.g., KHATT) are caused by confusion between visually similar characters, compared to about 15 percent in Latin-script
datasets such as IAM.
Read the full abstract ▾
Computer Vision
Test-Time Adaptation in Optical Coherence Tomography Using Trajectory-Aligned Time-Independent Flow
Veit Hucke, Thomas Pinetz, Gregor Reiter et al. (+2 more)
Optical coherence
tomography (OCT) is essential in ophthalmology, but inconsistent image quality especially in low-cost devices hinders automated analysis. To address this, we introduce a flow-matching-based test-time adaptation method that generates high-quality surrogate images from noisy inputs. Typically, domain gaps between test and training data cause pixel distribution mismatches during the denoising process. We overcome this by matching the test image's histogram to synthetic reference trajectories, successfully aligning the input with expected distributions. Additionally, we remove the network's time conditioning to account for slight deviations in real-world noise distributions. Our approach achieves state-of-the-art performance in segmenting critical biomarkers for two stages of Age-related Macular Degeneration (AMD). Code is available: https://github.com/Veit21/tta-flow.
Read the full abstract ▾
All authors Veit Hucke, Thomas Pinetz, Gregor Reiter, Ursula Schmidt-Erfurth, Hrvoje Bogunović
Computer Vision
Bridging Single Distortion Artifacts and Multifactorial Clinical Quality: Few-shot Biparametric MRI Quality Assessment via Distortion-trained Prototypical Networks
Yucheng Tang, Alexander Ng, Wen Yan et al. (+11 more)
Clinical prostate multi-parametric MRI relies heavily on high-quality
diffusion-weighted imaging (DWI), yet reading DWI is frequently compromised by
geometric distortion, often caused by rectal air. Assessing quality via the PI-QUAL scoring system is an emerging clinical standard, but it is subjective, time-consuming and suffers from a class imbalance where low-quality cases are diverse and relatively scarce. Using the
PRIME clinical trial as an example, there are $6\%$ images with PI-QUAL scores lower than 4, $87\%$ of DWI issues are due to distortion. Many of the other clinical quality issues are under-represented. To address this common dual-scarcity of annotated clinical data, we propose a few-shot biparametric prototypical network for automated image quality assessment (IQA). Our framework utilizes a dual-branch 3D ResNet to fuse T2-weighted and DWI features, providing anatomical context to distinguish true morphology from distortion. To handle real-world heterogeneity, we introduce feature-wise linear modulation (FiLM) and a gradient reversal layer (GRL) to align feature distributions conditioned on varying b-values while suppressing acquisition-related biases. We demonstrate that a model meta-trained solely on comparatively objective, readily obtainable distortion labels can effectively adapt to predicting complex, multi-factorial clinical quality scores such as PI-QUAL using only five representative samples. Experimental results on two
datasets show that our method significantly outperforms few-shot learning baselines for this challenging IQA task, offering a practically feasible and data-efficient solution for standardizing prostate MRI quality control in clinical workflows.
Read the full abstract ▾
All authors Yucheng Tang, Alexander Ng, Wen Yan, Natasha Thorley, Pawel Rajwa, Yipei Wang, Aqua Asif, Clare Allen, Louise Dickinson, Francesco Giganti, Shonit Punwani, Daniel Alexander, Veeru Kasivisvanathan, Yipeng Hu
Computer Vision
Learning to Distort: Weakly-Supervised Image Quality Transfer for Prostate DWI Correction
YuCheng Tang, Wen Yan, Alexander Ng et al. (+13 more)
Single-shot echo-planar prostate
diffusion-weighted imaging (DWI) is frequently complicated by
geometric distortions, which impact the ability to derive reliable diagnoses from such images. Developing automated correction methods is challenged by the absence of paired distorted and undistorted clinical scans. In this paper, we first propose a novel weakly-supervised image quality transfer (IQT) framework from undistorted to distorted images that utilizes image quality assessment (IQA) signals to supervise the transfer process. Unlike traditional methods that require expensive, voxel-wise paired data or resort to developing unpaired
algorithms, our approach utilizes image-level quality labels (here, distorted vs. undistorted) to establish latent quality prototypes within a pre-trained feature space. Recognizing that simulating realistic distortions is more reliable than direct unpaired correction, we describe a weakly-supervised prototype flow matching
algorithm to explicitly regularize generative trajectories towards distorted prototypes, producing realistic susceptibility artifacts that mimic clinical degradations. By synthesizing these realistic pairs, we enable a second IQT model to be trained in the forward direction for distortion correction. Experimental results demonstrate that our generated images successfully mimic the diagnostic interference of real-world artifacts, which leads to more capable distortion correction IQT models. In addition to qualitative comparisons, we also conduct exhaustive quantitative evaluations that compare our approach with existing unpaired approaches (e.g., CycleGAN, UNIT-DDPM, and OT-FM) - as either forward or reverse alternatives - by assessing clinical downstream task performance in PI-RADS and Gleason score classification, using both in-distribution and external data sets.
Read the full abstract ▾
All authors YuCheng Tang, Wen Yan, Alexander Ng, Natasha Thorley, Pawel Rajwa, Yipei Wang, Aqua Asif, Clare Allen, Louise Dickinson, Francesco Giganti, David Atkinson, Shonit Punwani, Daniel Alexander, Shaheer Ullah Saeed, Veeru Kasivisvanathan, Yipeng Hu
Computer Vision
URDF Synthesis from RGB-D Sequences via Differentiable Joint Inference and Energy-Consistent Verification
Xinze Zhang
Reconstructing simulation-ready digital twins of articulated objects from sensor observations remains constrained by two persistent gaps: (i) part-level
geometric reconstruction is decoupled from kinematic-parameter estimation, and (ii) the recovered models often violate basic dynamic invariants such as energy conservation, leading to drift when the URDF is replayed in physics simulators. We present KinemaForge, a constraint-driven pipeline that jointly infers part-level shape, joint
topology, and joint parameters from short RGB-D sequences and validates the result against an energy-consistent verifier built on differentiable rigid-body dynamics. The pipeline introduces three components: a kinematic constraint graph that encodes joint-part incidences as soft edges; a differentiable screw-axis solver that backpropagates from rendered observations through Featherstone's articulated-body
algorithm to joint parameters; and an energy residual loss that penalises non-physical free responses of the reconstructed model. Across five PartNet-Mobility categories and an internal RGB-D
benchmark, KinemaForge reduces the average joint-axis error from 4.52 degrees to 2.83 degrees (-37.4%) over the strongest
geometric baseline (PARIS) and from 5.30 degrees to 2.83 degrees (-46.6%) over the interaction-based Ditto baseline, lowers long-horizon simulation drift by 64% (vs. PARIS) over 50 s rollouts, and yields URDFs whose closed-loop manipulation success rate improves by 14.6 percentage points over Ditto in our preliminary evaluation. Code and reconstruction data will be released upon acceptance.
Read the full abstract ▾
Computer Vision
Quantification of Uncertainty with Adversarial Models in Medical Image Segmentation
Hana Jebril, Thomas Pinetz, Günter Klambauer et al. (+1 more)
Reliable pixel-level uncertainty quantification holds the potential to transform clinical workflows by enabling high-fidelity longitudinal monitoring and distinguishing true pathological changes from artifacts. Ideally, these models provide the stability required for critical treatment planning and
surgical intervention. However, standard deep learning models often suffer from miscalibration, yielding overconfident predictions that mask underlying vulnerabilities at subtle pathological boundaries. To address this, we propose QUAM-SM, a post-hoc framework using targeted
adversarial search to identify "adversarially fragile" pixels. By actively seeking perturbations that expose predictive instability, our method highlights regions where decisions are most vulnerable to being flipped. Importantly, the framework disentangles epistemic uncertainty from aleatoric uncertainty. Experiments on two public
datasets with multiple expert annotations demonstrate that QUAM-SM outperforms both standard and recent uncertainty estimation approaches in terms of reliability and boundary sensitivity. Code is available at https://github.com/HanaJebril/quam_sm
Read the full abstract ▾
All authors Hana Jebril, Thomas Pinetz, Günter Klambauer, Hrvoje Bogunović
Computer Vision
From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models
Like Zhang, Runliang Niu, Shiqi Wang et al. (+5 more)
Vision-
language models (VLMs) are rapidly advancing toward sophisticated grounded structured visual
reasoning. Training models for such advanced capabilities demands a new genre of data that seamlessly unifies spatial coordinates, open-vocabulary descriptions, structured attributes, and topological relationships into a singular representation. However, existing data annotation tools fundamentally fail to meet these intricate demands, suffering from three systematic bottlenecks: limited expressiveness, severe annotation-training decoupling, and poor data reusability. To bridge this infrastructure gap, we introduce an open-source annotation tool, ScreenAnnotator. First, we define a unified annotation atom schema that binds spatial, semantic, and structural primitives into a single unit. Second, we implement an on-policy annotation loop embedded with a
Bayesian Annotation Verifier (BAV). Finally, we design a template-driven multi-task data synthesis process dynamically transforms static atoms into diverse multi-dimensional
reasoning tasks, eliminating redundant re-annotation. The on-policy loop drives the annotation accept rate to nearly 100% on flowcharts and 77% on GUI screenshots, while steadily reducing per-image annotation time as labeled data accumulate. In the flowchart scenario, fine-tuning a VLM yields 76.1% average accuracy, which is a 35.1% point absolute gain. Our code is available at: https://github.com/WnQinm/Annotator.
Read the full abstract ▾
All authors Like Zhang, Runliang Niu, Shiqi Wang, Xiyu Hu, Qianli Xing, Pan Wang, Qingzu He, Qi Wang
Computer Vision
Rethinking Air-Ground Collaboration: A Progressive Cross-Task Benchmark and Socialized Learning Framework
Zhoupeng Guo, Yunqi Zhu, Zhihe Fan et al. (+6 more)
Air-ground collaborative perception is crucial for robust visual understanding in real-world dynamic environments. However, existing studies typically formulate collaboration as single-task cross-view fusion, overlooking the functional dependencies among localization, target association, and fine-grained parsing. In addition, the heterogeneous nature of aerial and ground views introduces substantial
geometric, scale, and occlusion discrepancies, making uniform feature sharing vulnerable to negative transfer. To tackle these issues, we model air-ground perception as a progressive cross-task collaboration task and construct the Air-Ground Progressive Collaboration (AGPC)
benchmark, a spatio-temporally aligned
benchmark comprising more than 745K raw
video frames. Built upon this
benchmark, we propose Socialized Co-Perception (SCP), a coarse-to-fine framework that organizes collaboration progressively from aerial global localization to ground target association and identity-aware parsing. Its core module, the Dual-Layer Router (DLR), decouples input-side multi-scale expert selection from output-side task-conditioned modulation, enabling selective cross-view and cross-task interaction while suppressing harmful interference. Extensive experiments demonstrate the effectiveness of SCP. It achieves a 3.73% coevolutionary gain and a 7.86% improvement in average downstream performance. These results show that task-conditioned collaboration is more effective than uniform fusion for heterogeneous air-ground perception. The code is available at https://github.com/g1136639260-spec/AGSCP.
Read the full abstract ▾
All authors Zhoupeng Guo, Yunqi Zhu, Zhihe Fan, Xinjie Yao, Ruipu Zhao, Boan Tao, Yiming Sun, Zhen Wang, Pengfei Zhu
Computer Vision · Cross-listed from cs.LG
Semantic Robustness Certification for Vision-Language Models
Peiyu Yang, Paul Montague, Feng Liu et al. (+4 more)
Vision-
language models (VLMs) are now widely used in downstream tasks. However, real-world applications often expose VLMs to distribution shifts induced by semantic variation (e.g., shape, size, and style). Robustness certification determines if a model's prediction changes when transformations are applied to its input. While most certification frameworks study
geometric or pixel-level transformations over inputs, this work proposes a novel framework that enables certifying VLM robustness under semantic-level transformations. Leveraging the open-vocabulary capability of VLMs, we use text
prompts as semantic proxies to construct transformations parameterized by an extent that controls the degree of semantic variation. By characterizing the VLM decision boundary in closed form, our framework quantitatively certifies extent intervals for which the predicted class remains unchanged under the semantic transformation. Our framework is the first to certify VLM robustness under semantic-level variations without requiring additional data for each variation, making it practical to apply. Experiments on both synthetic and real-world data show that our framework enables certifying robustness under diverse semantic variations across scenarios.
Read the full abstract ▾
All authors Peiyu Yang, Paul Montague, Feng Liu, Andrew C. Cullen, Amardeep Kaur, Christopher Leckie, Sarah M. Erfani
Computer Vision · Cross-listed from physics.optics
EDoF-NeRF: extended depth-of-field neural radiance fields using a coded aperture camera
Yoshiyuki Shirasaki and Ryoichi Horisaki
We propose a method for extending the depth-of-field (DoF) to construct high-fidelity neural radiance fields (NeRF) – an emerging technique for rendering photorealistic novel views from a
dataset of images captured at different viewpoints, based on implicit neural representations. The trade-off between DoF and light quantity is inherent not only in conventional cameras but also in NeRF, since the
datasets used by NeRF are captured by these cameras. To address this issue, we introduce a coded aperture placed at the camera pupil, preserving spatial frequency components under defocused conditions. We develop a camera model incorporating coded apertures into NeRF, allowing direct input of coded images and enabling the generation of novel views with an extended DoF. We validate the proposed method, termed extended DoF-NeRF (EDoF-NeRF), through simulations and experiments, demonstrating its superior performance compared to conventional aperture cameras.
Read the full abstract ▾
Computer Vision
DreamReg: Belief-Driven World Model for 2D-3D Ultrasound Registration
Luoyao Kang, Yuelin Zhang, Jiwei Shan et al. (+3 more)
Ultrasound (US) is widely used for
surgical navigation, yet real-time registration between intraoperative 2D slices and preoperative 3D volumes remains challenging due to partial observability, speckle noise, and the action-dependent US acquisition. Existing methods are one-shot or short-horizon, making it hard for them to gather evidence over time or capture how surgeons adjust probe motion based on on-screen feedback. We propose DreamReg, a belief-driven world-model framework that formulates 2D-3D registration as belief updating over rigid transformations. DreamReg maintains a latent belief state that summarizes past observations and poses information, and continuously refines the transformation through learned dynamics as new slices arrive. During training, DreamReg is exposed to probe-motion trajectories that mimic clinical scanning behavior and learns to update its belief by conditioning pose refinement on the current US observation. During inference, DreamReg refines registration via internal imagination: it rolls out the learned world model to simulate candidate probe motions and their predicted observations, and integrates these imagined outcomes to converge to an accurate rigid transformation. Experiments on CAMUS and u-RegPro
datasets demonstrate improved robustness and competitive registration accuracy for real-time guidance compared with state-of-the-art methods.
Read the full abstract ▾
All authors Luoyao Kang, Yuelin Zhang, Jiwei Shan, Haifan Gong, Qingpeng Ding, Shing Shin Cheng
Computer Vision
Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videos
Yuxuan Xie, Nicolas Pugeault, Chongfeng Wei et al. (+2 more)
Pedestrian trajectory prediction from an on-board ego-centric camera is challenging since it depends on complex interactions with vehicles and scene context, as well as the intention of the pedestrian. The task becomes even more challenging since pedestrian intention is often ambiguous from historical observations alone, leading to an inherently
multimodal distribution over future trajectories. Existing CVAE-based predictors learn a latent representation by aligning the prior conditioned on trajectory history with the posterior conditioned on both history and future trajectories. This
alignment may suppress multimodality in the latent space, leading to mixed-mode predictions that interpolate between distinct future behaviours. In this paper, we propose MMPM, a mode-aware framework that separately models future trajectory distributions into semantically meaningful modes based on the pedestrian's crossing behavior. MMPM consists of two modules: behavior-aware Pedestrian Interaction Module (PIM) that jointly captures pedestrian-vehicle and pedestrian-environment interactions by introducing gaze and hand gestures, and a CVAE-based Mode-aware Trajectory Predictor (MTP) module to model the future trajectory distributions in two modes, crossing and non-crossing the road, separately. A query-based decoder further enforces mode consistency during decoding. Experiments on PIE and JAAD
datasets show that our method surpasses state-of-the-art baselines. Our proposed MTP is model-agnostic, which can be integrated into existing frameworks such as BiTrap and SGNet to further improve future trajectory prediction performance. We additionally introduce a data-driven validation protocol that retrieves spatio-temporally consistent real-world trajectories from the same scene and evaluates predictions against these retrieved examples, demonstrating improved frame-wise displacement errors over previous work.
Read the full abstract ▾
All authors Yuxuan Xie, Nicolas Pugeault, Chongfeng Wei, Hubert P. H. Shum, Edmond S. L. Ho
Computer Vision
Fuzzy-Geometric Branch-Point Modeling for Structure-Aware Augmentation of Handwritten Chinese Characters
Dongbin Jiao, Yibo Lyu, Qiulu Wei et al. (+3 more)
Data scarcity and structural distortion significantly limit handwriting recognition in high-security authentication. Existing augmentation methods often cause topological and morphological damage, particularly when processing complex Chinese characters where stroke intersections, ligatures, and sharp turns render traditional branch-point detection unreliable. To address this, this paper proposes a fuzzy geometry-driven structure-aware (FGSA) augmentation framework. We model branch points as fuzzy sets within the skeleton space, constructing a continuous branch-point membership field by integrating topological neighborhood evidence with direction field divergence. This membership field is adaptively optimized via an unsupervised surrogate objective, enabling robust stroke decoupling without manual annotation. Finally, kinematically-aligned samples are synthesized through parameterized cubic Bézier reconstruction and multi-strategy perturbations, ensuring a balance between structural fidelity and sample diversity. Moreover, we establish LZUSig, a large-scale, highly challenging
dataset specifically dedicated to fine-grained structural degradation in Chinese handwritten signatures. Extensive experiments on CASIA-HWDB1.1, ChiSig, and LZUSig demonstrate that FGSA significantly reduces the word-level error rate ($\Delta$WER), achieving optimal recognition gains over the compared baselines. More importantly, it strikes a robust trade-off among task gain, structural fidelity, and discriminative feature preservation, offering a highly controllable solution for handwriting augmentation.
Read the full abstract ▾
All authors Dongbin Jiao, Yibo Lyu, Qiulu Wei, Fuxiang Lu, Shengcai Liu, Shi Yan
Computer Vision
HandwritingAgent: Language-Driven Handwriting Synthesis in Scalable Vector Space
Jaward Sesay, Yue Yu and Börje F. Karlsson
Teaching machines to emulate natural handwriting styles remains an open challenge, as it requires synthesizing stroke sequences that dynamically vary in shape, texture, pressure and script - not only across individuals, but also within a single person's handwriting. Attempts at this challenge have largely explored deep learning methods in both online and offline settings. However, these approaches are often constrained by style-specific architectural choices, heavy reliance on large
datasets, high compute costs, and a lack of flexible control over writing styles through natural
language. To this end, we introduce HandwritingAgent, a
language-driven
agent that can synthesize natural handwriting sequences directly in Scalable Vector Graphics (SVG) format with no need for style-specific training. The
agent leverages a large
reasoning model to geometrically analyse and autoregressively generate target handwritten glyphs as stroke sequences in a discrete grid canvas environment. Generation is conditioned on texts provided in either conversational or non-conversational mode, along with a reference handwriting-style image. Experiments on diverse handwriting tasks spanning imitation, recognition, multi-lingual handwriting synthesis, and generation of complex handwritten maths and science expressions indicate substantial improvement in performance, with HandwritingAgent matching or surpassing state-of-the-art generative handwriting models, while providing a more efficient, controllable, and generalizable synthesis method.
Read the full abstract ▾
Computer Vision
Learned Radius Estimation for UDF-Based Point Cloud Reconstruction
Eito Ogawa and Hiroshi Watanabe
Surface reconstruction from point clouds is important for consumer-grade 3D capture, including AR/VR and indoor scanning. Local-patch Unsigned Distance Field (UDF) methods are lightweight and generalizable, but their accuracy depends on the support radius, traditionally fixed or selected by a one-dimensional
curvature heuristic that cannot capture heterogeneous local geometry. We propose a learned per-query radius selector that predicts a continuous support radius and plugs into a frozen LoSF-UDF backbone. The selector is trained using off-grid target radii obtained by
parabolic interpolation of cached UDF error curves. Experiments show improved fine-scale reconstruction accuracy.
Read the full abstract ▾
Computer Vision
SCR-Guided Difficulty-Aware Optimization for Infrared Small Target Detection
Yunus Sevim and Behçet Uğur Töreyin
Infrared small target detection remains challenging due to severe background clutter, low contrast, and weak spatial responses where
geometric overlap alone is insufficient to characterize detection quality. In this work, we propose REEM (Reweighted Explicit-visibility Enhanced Modulation), a lightweight SCR-guided difficulty-aware optimization framework that incorporates Signal-to-Clutter Ratio (SCR) as a physically meaningful visibility prior during training. Instead of modifying the network architecture or directly optimizing SCR, REEM computes a ground-truth local SCR from the input image and applies a differentiable modulation to the soft-IoU learning signal, emphasizing low-visibility targets while preserving stable optimization and identical inference behavior. REEM is integrated into a U-Net-based MSHNet without introducing additional parameters, architectural modifications, or inference-time overhead. Extensive experiments demonstrate consistent improvements over the baseline, achieving higher IoU and detection probability (Pd) together with substantially reduced false alarms (FA), particularly under challenging low-visibility conditions. These results suggest that SCR-guided difficulty-aware optimization provides an effective and physically grounded complement to conventional overlap-based objectives for infrared small target detection. The code is available at https://github. com/yall-in-one/Reemm.
Read the full abstract ▾