Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Electrical Engineering and Systems Science Image and Video Processing

eess.IV Watch Image and Video Processing

Wednesday 9 September

Image and Video Processing

XPos3R: Cross-Modal Transformer for Intraoperative 2D/3D Registration

Shiyan Su, Ruyi Zha, Hongdong Li et al. (+2 more)

Intraoperative 2D/3D registration, which aligns live X-ray images with preoperative volumes, is essential for image-guided interventions. Previous regression-based methods suffer from limited generalization, thus requiring time-consuming patient-specific retraining. Inspired by recent geometry foundation models such as DUSt3R, we propose XPos3R, a generalizable pose regression method that eliminates preoperative preparation. Unlike existing geometry models designed for homogeneous inputs, XPos3R extends this paradigm to multi-modal inputs, namely 2D X-rays and 3D volumes. Specifically, we introduce an asymmetric encoder-decoder architecture that improves cross-modal feature alignment while maintaining computational efficiency. To scale training under limited medical data, we adopt an anatomy-specific data curation strategy and construct million-scale synthetic datasets. Evaluated on real-world benchmarks, a single pretrained XPos3R surpasses patient-specific methods in both accuracy and robustness. With test-time optimization completed in seconds, it further reduces the 3D error to <4 mm and the reprojection error to <1 mm. The strong generalization, accuracy, and efficiency of XPos3R highlight its clinical potential, while its asymmetric framework may inspire broader cross-modal vision geometry tasks.
Read the full abstract

All authors Shiyan Su, Ruyi Zha, Hongdong Li, Xuelian Cheng, Zongyuan Ge

Image and Video Processing

Physics-Informed Sequential Reconstruction of Scanning Probe Microscopy Images with Calibrated Uncertainty

Surya Prakash Reddy and Sergei V. Kalinin

Scanning probe microscopy (SPM) has become a mainstay of materials science, condensed matter physics, nanotechnology, biology, and semiconductor metrology. An SPM image, however, is acquired sequentially as the probe traverses the surface under closed-loop feedback while thermal drift, feedback dynamics, tip state, mechanical disturbances, and electronic noise evolve during the experiment. Image correction has therefore become an intrinsic part of SPM data analysis. Classical approaches rely on line leveling, filtering, registration, interpolation, and explicit models of tip or scanner distortions, whereas recent work increasingly uses neural networks trained to map corrupted images to corrected ones. Here, we explore a different formulation in which SPM reconstruction is treated as a Bayesian sequential state-estimation problem. The hidden state represents surface height and its low-order evolution along the slow-scan direction; an artifact-dependent quality score continuously modifies the observation likelihood; a forward Kalman pass provides the causal estimate and innovation diagnostics; and a Rauch-Tung-Striebel (RTS) backward pass incorporates information from the complete acquisition. The result is a reconstructed surface, a relative posterior uncertainty map, and line-resolved diagnostic signals. The recursion is computationally lightweight and scales linearly with the number of image pixels, while the physical parameters can in principle be initialized from instrument characterization or from priors learned during previous operation, an extension not exercised here. We organize the method as a multi-axis family of models so that the contributions of state representation, artifact recognition, lateral coupling, and acquisition redundancy can be evaluated independently.
Read the full abstract

Image and Video Processing · Cross-listed from cs.LG

Storage-Scalable Progressive Semantic Communication via Knowledge-Base Reuse

Heng Zhu, Ye Liu, Kun Zhu et al. (+1 more)

Existing knowledge-base-assisted semantic communication schemes commonly adopt either single knowledge-base quantization (SKBQ) or multi-knowledge-base residual quantization (MKBQ). SKBQ incurs limited storage overhead but has restricted quantization capacity, whereas MKBQ supports progressive refinement by assigning an independent knowledge base (KB) to each stage, causing the KB storage to grow linearly with the transmission depth. To address this problem, we propose storage-scalable knowledge-base reuse quantization (SSKBQ), which reuses a compact set of KBs across multiple residual refinement stages and thereby decouples the number of transmission stages from the number of maintained KBs. A stage-aware residual supervision mechanism is further introduced to regularize intermediate quantized representations and encourage progressive refinement. Experimental results demonstrate that KB reuse provides an effective solution to the storage scalability problem while maintaining competitive progressive reconstruction performance.
Read the full abstract

All authors Heng Zhu, Ye Liu, Kun Zhu, Feifei Song

Image and Video Processing

Cost-Aware Vision–Language Model Arbitration for Fabric Structure Recognition A Deployable Multi-Agent System

Chenwei Wang, Haochen Li, Shuk Ching Tang et al. (+3 more)

Recognizing a fabric's structure is a prerequisite for translating textile-specific material information into structured digital form for downstream supply-chain systems. Pure CNN classifiers are cost-efficient but fail on visually ambiguous categories; vision–language models (VLMs) generalize more broadly but cost much more per image and are unstable on specialist domains. We present a multi-agent system in which a CNN cascade handles the easy majority and a VLM is invoked only as a selective arbiter, constrained to a top-3 taxonomy-consistent choice. The fabric taxonomy performs as a constraint for the whole recognition process to increase the accuracy and reduce the VLM calls. Meanwhile, the CNN cascade is distilled to a small parameter size to reduce the inference time and meet the needs of practical deployment. On a newly curated 14-class benchmark, a flat ConvNeXt-Tiny baseline reaches $90.45\,\%$ top-1 and $76.9\,\%$ on the four hardest classes; \method's hierarchical cascade reaches $93.94\,\%$ top-1 and $94.50\,\%$ hard ($+17.6$\,pp). Tightening the VLM trigger from $60\,\%$ to $<\!10\,\%$ cuts API cost by ${\sim}90\,\%$ with no measurable accuracy loss. CPU inference is $\le\!93$\,ms without a VLM call ($9.3$\,ms distilled). Each prediction carries a machine-readable reasoning record, offered as an entry point for future supply-chain documentation.
Read the full abstract

All authors Chenwei Wang, Haochen Li, Shuk Ching Tang, Misbah Iqbal, Carman Lee, Elif Ozden-Yenigun

Image and Video Processing

Morphological Decoupling-Based Skeletal Classification for Clinical Assessment of Malocclusion

Zhichun Jin, Zhicheng He, Hao Xu et al. (+4 more)

Malocclusion skeletal grading is a fundamental task in orthodontics, critical for diagnosis and treatment planning. Traditionally, cone-beam computed tomography (CBCT) is used for visual measurement, and the reconstructed lateral cephalograms are handed over to expert dentists for diagnosis. However, manual review is time-consuming, labor-intensive, and subject to inter-operator variability. Therefore, an automatic CBCT-based system is needed for reliable malocclusion skeletal grading. In this case, we develop TeethGNN, a novel graph-based framework designed to combine CBCT image features with morphological information for accurate and efficient malocclusion grading. TeethGNN utilizes a decoupled learnable decoder to directly predict key morphological indicators from CBCT images, eliminating the need for manual measurements. These morphological features are then fused with image features using a graph neural network (GNN), which effectively models the relationships between the modalities. To further enhance robustness and calibration, we introduce a collaborative calibration strategy. This strategy combines multi-scale graph adversarial perturbation for explicit calibration and nonlinear topological graph calibration for implicit confidence adjustment. Extensive experiments and ablation studies on our collected clinical dataset demonstrate that our malocclusion measurement system achieves 77.08% in accuracy and 89.61% in AUC, outperforming the compared state-of-the-art methods. These results validate the effectiveness of graph-based multimodal fusion and collaborative calibration in improving malocclusion grading performance. Our system shows strong potential for advancing computer-aided orthodontic diagnosis, providing an accurate and reliable solution for vision-based clinical measurement and diagnosis.
Read the full abstract

All authors Zhichun Jin, Zhicheng He, Hao Xu, Dongyang Li, Lin Wang, Hongliang Ren, Long Bai

Image and Video Processing · Cross-listed from cs.CV

Recovering Biomechanical Signals from Missing Keypoints Using Temporal Interpolation in Monocular Gait Analysis

Shubham Jariwala

Monocular pose estimation enables low-cost gait analysis but is sensitive to missing keypoints caused by occlusion, detection errors, or efficiency-driven model reduction. While prior work on recovering missing joints focuses on complex learned models, the effectiveness of simple temporal methods remains underexplored. We evaluate knee-angle estimation under a missing-ankle-keypoint condition and test a first-order temporal interpolation scheme as a recovery mechanism. Across 527 frames of monocular walking video (428 with valid baseline detections), removing the ankle keypoint increased mean angular error to 23.4{\deg} +/- 46.7{\deg} and collapsed signal variance to near zero. Temporal interpolation reduced error to 1.1{\deg} +/- 6.7{\deg} and restored variance and smoothness to within a few percent of baseline. These results indicate that gait signals possess sufficient temporal redundancy for a simple, computationally trivial interpolation scheme to recover a critical missing joint, without resorting to learned reconstruction models. The findings support low-complexity, real-time-compatible designs for gait analysis in resource-constrained or occlusion-prone monocular settings.
Read the full abstract

Image and Video Processing · Cross-listed from cs.NI

Automated Mobile Video Objective Testing System

Eric Petajan, Jonathan Lynam, Morey Antebi et al. (+5 more)

Applying QoE analysis to optimize usage of cellular spectrum is of high interest to mobile network operators. A key challenge is to be able to perform QoE measurement across very different types of apps, from DASH VoD to interactive applications such as Video Conferencing and Cloud Gaming. This paper presents AMVOTS, a QoE measurement system developed by AT&T, which is flexible enough to support a large range of application types and network conditions. We also discuss using AMVOTS as part of a closed loop to prototype QoE-aware radio resource allocation.
Read the full abstract

All authors Eric Petajan, Jonathan Lynam, Morey Antebi, Hessam Moeini, David Lindero, Lars Ernstrom, Gyanesh Patra, Szilveszter Nadas

Image and Video Processing

Myocardial Strain Drift Correction in Deep Learning Based Ultrasound Tracking

Thierry Judge, Nicolas Duchateau, Andreas Østvik et al. (+6 more)

Myocardial strain from echocardiography is a key biomarker for cardiac function. Recent deep learning methods show strong performance for myocardial motion tracking but often lack physiological constraints, leading to temporal drift across the cardiac cycle. Consequently, tracked points may not return to their relative initial positions at the end of each cardiac cycle, producing inaccurate strain estimates and even divergence in some cases. We propose a deep learning framework that compensates for drift during myocardial tracking. We extend a state-of-the-art echocardiographic tracking method (TAS-Net) with persistent memory tokens that share information across sliding windows over full cardiac cycles. A teacher-student fine-tuning strategy on real echocardiographic data then enforces physiologically consistent cyclic motion while preserving tracking accuracy. Experiments show reduced global and regional strain drift, improved agreement with clinical references, and better test-retest reproducibility, supporting more reliable myocardial strain estimation in clinical practice.
Read the full abstract

All authors Thierry Judge, Nicolas Duchateau, Andreas Østvik, Havard Dalen, Bjørnar Grenne, Pierre-Yves Courand, Lasse Lovstakken, Pierre-Marc Jodoin, Olivier Bernard

Tuesday 8 September

Image and Video Processing · Cross-listed from cs.CV

The MODA General Attribute Suite: A Four-Track Evaluation Benchmark for Fashion Attribute Extraction

Arkid Mitra

Fashion attribute extraction is evaluated inconsistently: results are reported as single aggregate numbers across image types that pose different problems, fields that are not visible in an image are scored as ordinary negatives, and the effect of vocabulary mismatch between datasets is acknowledged but not measured. We release the MODA General Attribute Suite, a four-track benchmark that keeps these problems separate by construction. Each track (localized garment crops, catalogue product images, applicability-aware full-body photographs, and product text) carries its own frozen test set, input contract, metric, and leakage unit, and the tracks are never averaged. The protocol requires label-blind prediction, SHA-256 commitment of prediction files before any label is opened, a fail-closed scorer, and a 10,000-sample paired cluster bootstrap at each track's natural leakage unit; promotion requires a positive interval on every track rather than a favourable mean. We release the scorers, the split builders, our own prediction files with their hashes for every image-track system in the main results and for runs we lose, and the MODA_NER(V) model checkpoints for three of the four tracks. The text model is not distributed; its benchmark is. We report baseline results, six interventions that did not improve them, and the limitations of the evidence. On the garment-crop track we also measure how far the headline depends on the benchmark's own design: the order of the two strongest systems depends on whether fields are micro- or macro-averaged, two thirds of the cells the protocol judges carry no annotation, and disabling the applicability decision cuts attribute micro-F1 from 0.63 to 0.27, so most of that score rests on absence decisions the source labels cannot adjudicate. This version also corrects one figure in an external comparison that an inference preprocessing defect had affected.
Read the full abstract

Image and Video Processing

CHIMERA Challenge Task 2 and 3: Response Subtypes Classification and Progression Survival Prediction in Bladder Cancer Patients using Multimodal Datasets

Catherine Chia

High-risk non-muscle-invasive bladder cancer (HR-NMIBC) carries substantial risks of recurrence and progression, while current clinical risk stratification remains limited. CHIMERA was established as a multimodal AI challenge to benchmark prediction in HR-NMIBC under standardized evaluation. Task BRS predicts RNA-seq-defined BCG Response Subtypes from histopathology and structured clinicopathological data, whereas Task Progression models time-to-progression using histopathology, structured data, and RNA sequencing. A multimodal dataset of 368 patients was divided into public training and hidden validation and test sets. In total, 159 submissions were made, and 13 top-performing models were selected for benchmarking. The best models achieved a weighted F1 score of 0.73 for Task BRS and a C-index of 0.68 for Task Progression. Post-challenge analyses revealed task-dependent modality contributions, cohort-dependent performance degradation, and sensitivity to missing structured data. In Task BRS, histopathology partly compensated for pathology-derived structured variables, whereas progression models showed greater dependence on complementary inputs. Cross-model error analysis further identified patients that were consistently difficult across different architectures, with T1 substage associated with prediction difficulty. These findings highlight barriers to transportability and the importance of missingness-aware modeling and independent multi-institutional validation. CHIMERA provides a standardized multimodal benchmark for bladder cancer and a framework for studying not only model performance, but also robustness, information sufficiency, and patient-level prediction failure.
Read the full abstract

Image and Video Processing · Cross-listed from cs.NI

Prototyping QoE-Aware Rate Adaptation in Cellular Networks with Commercial Applications

Szilveszter Nádas, Lars Ernström, Dan Druta et al. (+4 more)

Prior work has shown that QoE-aware resource sharing for real-time interactive video can support up to three times more simultaneous sessions at acceptable quality compared to rate-fair allocation. However, the required capabilities (QoE-targeted encoding, runtime spatial complexity estimation, and rich application-network APIs) are not yet available in commercial deployments. In this paper, we take an evolutionary approach: we design a system that delivers QoE-aware resource allocation using only capabilities that can be assembled in a lab today. We extend the utility-based allocation framework to the radio resource domain by introducing composite spatial complexity, which combines a session's video spatial complexity with its time-variant spectral efficiency into a single resource demand function. To operate with commercial real-time video streaming applications that use rate-based congestion control and lack capability to measure QoE, we use external tooling for QoE measurements. We develop an incremental reallocation algorithm with per-interval limits that encode both the congestion control algorithm's speed constraint and that spatial complexity estimates are reliable only near the current rate. The resulting prototype combines external QoE measurements with congestion-signal-based rate steering and does not require modification to commercial applications. We chart an evolution path from this prototype toward full QoE-aware resource sharing, mapping emerging standards (IETF SCONE, CAMARA, Media over QUIC) to the progressive capabilities they enable.
Read the full abstract

All authors Szilveszter Nádas, Lars Ernström, Dan Druta, Igor Pruzhansky, David Lindero, Jonathan Lynam, Eric Petajan

Image and Video Processing · Cross-listed from cs.CV

Spheriverse: 3D Scene Understanding from Spherical Observations in the Wild

Fei Teng, Sheng Wu, Mengfei Duan et al. (+7 more)

Spherical observations provide global visual context for 3D scene understanding. However, visual information is encoded in an angular domain, whereas the physical world is represented in Cartesian coordinates. This cross-space representation gap complicates geometric correspondence and semantic evidence aggregation. To delve into this challenge, we introduce Spheriverse, comprising 64,400 temporally aligned spherical image-LiDAR pairs organized into 644 sequences. The dataset spans diverse scenes, illumination, and weather conditions, with fine-grained semantic classes. We further establish benchmarks for semantic occupancy prediction, semantic mapping, and 3D object detection, evaluating 30+ methods through overall and scene-wise comparisons. For dense prediction, we propose SphereOcc, an occupancy framework that couples spherical geometry modeling with semantic evidence retrieval. Cartesian-Spherical Representation Remodeling (CSRR) incorporates spherical range-azimuth geometry into Cartesian voxel features through region-wise modulation. Spherical Evidence Re-querying (SER) then conditions queries on voxel content and range-height-azimuth geometry to adaptively retrieve relevant semantic evidence from source spherical image features. SphereOcc achieves 13.91% mIoU and 24.65% GeoIoU, yielding relative improvements of 13.9% and 9.3% over the respective best-performing methods, TPVFormer and SurroundOcc. It also ranks first in both metrics across all five scene categories, with consistent advantages across the evaluated spatial partitions and reduced fields of view. The established benchmark and source code will be available at https://feit-feiteng.github.io/Spheriverse.
Read the full abstract

All authors Fei Teng, Sheng Wu, Mengfei Duan, Guoqiang Zhao, Junhui Ma, Kai Luo, Siyu Li, Hao Shi, Zhiyong Li, Kailun Yang

Image and Video Processing · Cross-listed from cs.LG

Leveraging Cardiac Imaging to Improve ECG-Based Detection of Chagas Disease in Resource-Constrained Settings

Laura Alvarez-Florez, Daniel Uyterlinde, Samuel Ruipérez-Campillo et al. (+3 more)

Chagas disease is a major cause of cardiomyopathy in Latin America. Cardiac magnetic resonance (CMR) imaging can characterize its structural abnormalities, but scanners and expert readers remain scarce in endemic regions. Electrocardiography (ECG) is inexpensive and widely available, yet structural disease must be inferred indirectly from electrical signals. We propose to transfer CMR-derived structural knowledge to ECG through contrastive pre-training. Using 63,193 paired ECG-CMR examinations from the UK Biobank, we align an ECG encoder with a clinically grounded CMR embedding space using an asymmetric InfoNCE objective. Despite seeing no Chagas cases during pre-training, the resulting representation improves ECG-based Chagas detection. Across CODE-15% and SaMi-Trop, a frozen linear probe achieves an AUROC of 0.851 and sensitivity at the top 5% of predicted risk (Top5%-TPR) of 0.427 in five-fold cross-validation, compared with 0.827 and 0.377 for an unaligned ECG-FM baseline. On the PhysioNet/CinC 2025 Challenge test set, our model obtains the highest AUROC on SaMi-Trop-3 and the best ELSA-Brasil challenge score among the three top-performing methods, indicating that imaging-supervised ECG representations can generalize to populations and settings beyond the pre-training distribution.
Read the full abstract

All authors Laura Alvarez-Florez, Daniel Uyterlinde, Samuel Ruipérez-Campillo, Lukas P. A. Arts, Folkert W. Asselbergs, Fleur V. Y. Tjong

Image and Video Processing

Benchmarking Regularization Methods For 3D Radio Tomographic Imaging

Wanqin Ma, Yijun Chen, Alikhan Umirbayev et al. (+2 more)

Integrated Sensing and Communication (ISAC) is an important technology for 6G, enabling wireless systems to perceive their physical environment. Radio Tomographic Imaging (RTI) is a potential technique for device-free sensing in ISAC, reconstructing object locations and shapes from Received Signal Strength (RSS) measurements. It can complement other techniques, such as radar and LiDAR, by providing a narrowband modality with wavelength resolution. Recently, an extended version of RTI, known as the extended phaseless Rytov approximation (x3DPRA), has been developed to enhance RTI reconstruction quality, estimate material parameters, and extend RTI from two to three dimensions (3D). However, the 3D formulation is even more ill-posed than the 2D form, as the number of measurements is far fewer than the number of unknown voxels. In this work, we systematically evaluate and compare three distinct regularization approaches (Ridge, Total Variation, and Tensor Nuclear) within the x3DPRA framework. We provide a detailed performance analysis by evaluating both reconstruction quality and computational runtime across the three methods. Our findings show that Total Variation regularization provides the best reconstruction quality but not the fastest runtime, providing guidance for developing high-resolution 3D RTI systems for future ISAC applications.
Read the full abstract

All authors Wanqin Ma, Yijun Chen, Alikhan Umirbayev, Jichen Zhang, Ross Murch

Image and Video Processing

GC360IQ: Generic-to-Individualized Quality Assessment for Stitched 360-Degree Panoramas

Jinghan Zhou and Zhou Wang

Existing image quality assessment (IQA) methods typically predict mean opinion scores (MOSs) but struggle to capture variations in individual subject behaviors. This limitation is highly pronounced in immersive visual applications such as stitched 360-degree panoramas, where user opinions diverge drastically based on personal sensitivity to blending-induced luminance inconsistency, detail loss, and geometric misalignment. Here we propose GC360IQ, a novel Generic-to-Individualized IQA framework that establishes a learned explicit feature space to characterize individual subject behaviors. First, we construct a specialized 360-degree panorama database focusing on blending-induced luminance and detail degradation while minimizing geometric misalignment, providing multidimensional quality ratings alongside complete individual scores. Second, we develop a generic quality model that utilizes unblended views as a perceptual reference. Dual feature extraction branches capture gradient and structural information specifically around stitching regions to predict baseline quality. Third, we construct a compact preference embedding space that acts as an explicit feature domain to model each subject's deviation from generic quality perceptions. We also introduce a maximum a posteriori (MAP) adaptation mechanism. By leveraging a preference prior learned within our explicit feature space, a new subject's unique behavioral embedding is mapped progressively with an increasing number of anchor ratings. Experiments demonstrate that the generic model provides accurate stitching quality predictions and that subject adaptation further improves individual score predictions. Deeper analysis of the collected ratings and learned embeddings reveals that observer differences contain structured variation related to scoring tendencies and sensitivity to stitching artifacts, rather than merely random rating noise.
Read the full abstract

Image and Video Processing

Reliability assessment and multicenter clinical application of magnetic resonance methods for knee cartilage quantification

Binbin Yang, Yongmei Jian, Chenglei Liu et al. (+10 more)

Background: This study evaluated interreader agreement and longitudinal performance of MRI methods for knee cartilage volume, thickness, and defect-area quantification. Methods: AI-presegmented masks from 1,189 phase III examinations underwent independent correction by two readers and adjudication. Cartilage volume, three-dimensional ray-tracing thickness (3D-RT), and ray-based defect area (3D-RBA), defined by a 1.5-mm thickness threshold, were calculated. Agreement was assessed using segmentation metrics, intraclass correlation coefficients (ICCs), repeated-measures Bland-Altman analysis, and minimal detectable change at 95% confidence (MDC95). The 3D-RBA framework was evaluated in 120 digital-phantom experiments from 40 participants. Longitudinal analyses included 374 participants, alternative-method comparisons included 65, and retrospective phase II analysis included 24 participants with four visits. Results: Overall AI-to-adjudicated-mask Dice was 0.964 +/- 0.029. Interreader ICCs for volume, thickness, and defect area were 0.956, 0.904, and 0.932; corresponding MDC95 values were 1,596.9 mm^3, 0.227 mm, and 147.4 mm^2. Geometric mean absolute percentage error for defect area was 5.62%, with spatial Dice of 0.961. In 374 participants, volume changes correlated positively with thickness changes (rho=0.431) and negatively with defect-area changes (rho=-0.221). Within-participant phase II correlations followed the same directions in both groups. Conclusions: The workflow demonstrated good interreader agreement. Controlled geometric results and longitudinal associations supported the feasibility of threshold-based defect-area estimation. Volume, thickness, and defect area provide complementary measures of cartilage structure.
Read the full abstract

All authors Binbin Yang, Yongmei Jian, Chenglei Liu, Rui Huang, Hongda Shao, Linjun Tong, Yuanjing Xu, Jingshu Wu, Chengzhang He, Suting Peng, Ming Xiao, Yinan Chen, Qi Duan

Monday 7 September

Image and Video Processing · Cross-listed from cs.LG

Foundation Models for Generalizable Semantic and Goal-Oriented Communication

Boliang Liu, Wint Yi Poe, Riccardo Trivisonno et al. (+1 more)

Semantic and goal-oriented communication is increasingly studied for 6G, but generalization beyond seen data remains a key weakness under tight rate budgets. Many existing systems overfit their training data and degrade sharply at very low bit rates because they attempt to compress the entire signal. We introduce Foundation Model-Guided Semantic and Goal-Oriented Communication (FMSGOC), a framework that uses broad visual-linguistic Foundation Model priors to mitigate overfitting. It further improves rate efficiency by concentrating bits on sparse, goal-aligned anchors and relying on generative foundation-model priors to reconstruct the masked regions. By decoupling what to send from how to reconstruct, a vision-language foundation model selects and transmits a sparse set of semantic anchors, while a pretrained diffusion model, fine-tuned for masked completion, reconstructs the image at the receiver. In our experiments, FMSGOC reaches 0.039 bits per pixel (BPP), maintains high semantic fidelity (cosine similarity 0.87-0.90 on CIFAR-10), remains robust on previously unseen inputs (0.83-0.86 on ImageNet), and shows good perceptual similarity (0.1278/0.1558, CIFAR-10/ImageNet), outperforming strong end-to-end baselines at lower bit rates.
Read the full abstract

All authors Boliang Liu, Wint Yi Poe, Riccardo Trivisonno, Giuseppe Caire

Image and Video Processing · Cross-listed from cs.CV

Cross-modal learning for SAR target recognition using optical vision foundation models

Lucas Hirsch, James R. Hopgood, Javid Khan et al. (+2 more)

Synthetic Aperture Radar (SAR) is an important modality in a wide range of imaging applications due to its versatile, long range and near all weather operating capabilities. However, Automatic Target Recognition (ATR) remains a challenging problem due to limited labelled data, the strong speckle in SAR images and the significant domain gap between SAR and more abundant optical imagery. In contrast, electro-optical (EO) imagery benefits from massive datasets, clearer visual structure and powerful foundation models. In this work, we investigate how vision foundation models trained on optical data can provide class level supervision for SAR classification. We propose a cross-modal EO to SAR prototype alignment framework in which a frozen EO encoder, based on a DINOv3 vision foundation model, is used to construct class level optical prototypes without requiring strict EO/SAR pairs. A SAR model is then trained to classify SAR images while aligning its embeddings to the corresponding EO class prototype. At inference time, the SAR model operates independently, without access to optical imagery. We evaluate our approach on the UNICORNv2 dataset, an EO and SAR dataset of civilian vehicles with heavily speckled images and severe class imbalance. EO prototype alignment improves SAR classification accuracy over frozen DINOv3, SAR only finetuning and unpaired distribution alignment baselines, and t-SNE visualizations provide qualitative evidence of clearer separation among classes in the trained SAR embedding space. These results suggest that optical vision foundation models, despite being trained on visible spectrum imagery, provide transferable information for SAR image classification, offering a practical method for using large scale pretrained vision foundation models across challenging sensing modalities.
Read the full abstract

All authors Lucas Hirsch, James R. Hopgood, Javid Khan, Yoann Altmann, Mike E. Davies

Image and Video Processing

Spatial Feature-wise Linear Modulation (SpFiLM) for Contrast Agent-Aware Brain Parcellation

Pushpendra Singh, Joshua R. Astley, Roman Rodionov et al. (+3 more)

Most automated brain parcellation tools are developed and validated on T1-weighted (T1w) MRI. Yet, some clinical workflows for which parcellation is relevant only use contrast-enhanced T1w (T1ce) MRI, on which T1w-trained models are less accurate. We present a unified network that parcellates both pre- and post-contrast agent T1w MRI reliably, trained on a combination of the two with conditioning that spatially modulates its response differently for each. Feature-wise Linear Modulation (FiLM) is a known approach for input-based modulation in networks. It applies a per-channel scale and shift uniformly across the input. However, the appearance change between pre- and post-contrast varies locally across the brain, making FiLM suboptimal for our use case. In this work, we introduce Spatial FiLM (SpFiLM), a conditioning layer whose modulation varies spatially, assembling a voxel-wise scale and shift from image-derived spatial patterns. Using a cohort of 134 patients with paired T1w and T1ce MRI parcellated into 106 classes, the addition of SpFiLM layers in a UNet increased the mean Dice on the test set of 25 patients from 80.2% to 84.1%, a 4.9% relative improvement. Adding SpFiLM layers led to the best performance on both pre- and post-contrast MRI, even when controlling for network parameter counts.
Read the full abstract

All authors Pushpendra Singh, Joshua R. Astley, Roman Rodionov, John Duncan, Tom Vercauteren, Rachel Sparks

Image and Video Processing

Computer vision enabled oxygen sensing

Nikolaos Salaris, Evangelos Mazomenos, Adrien Desjardins et al. (+1 more)

Luminescence-based chemical sensors are almost universally read as point detectors with the signal inverted through a single calibration model. Here we reframe optical oxygen sensing as a computer vision problem, in which the sensing film acts as a spatially heterogeneous encoder and a pretrained Temporal Vision Transformer (TViT) as its decoder. The heterogeneity in film thickness, diffusion path length and illumination translate to pixels that provide complementary information on the underlying diffusion dynamics. We achieved up to 54% lower mean absolute error (MAE) by increasing the internal diversity in performance of a pixel group; a gain that linear models cannot reproduce. Using a low-cost platform comprising a Raspberry Pi camera, a UV LED and a porous PtOEP/polystyrene film in tandem with a TViT architecture yielded an MAE of ~6.7 {\mu}mol/L, a 96% reduction from the two-site Stern-Volmer (SV) model. We showed that this framework can computationally mitigate the fundamental trade-off between mechanical robustness and temporal response in diffusion limited oxygen sensing by reducing T90 response times by 91%, while exhibiting physically plausible dynamics under a Rauch-Tung-Striebel smoother (2.3% flag rate). The framework was applied across setups, environments and biofilm states, establishing an IoT-compatible paradigm for computationally compensated diffusion-limited chemical sensing.
Read the full abstract

All authors Nikolaos Salaris, Evangelos Mazomenos, Adrien Desjardins, Manish K. Tiwari

Image and Video Processing

AstraMoE-SR: Trajectory-Guided Diffusion for Blind Satellite Jitter Deblurring and Super-Resolution

Yi-Chung Lai, Chin-Tien Wu and Yu-Chih Chen

Pushbroom satellite imaging couples limited spatial resolution with platform attitude instability. Platform jitter produces spatially varying motion blur because each scan line is acquired under a different instantaneous attitude, while perspective geometry causes the same perturbation to induce different pixel displacements across the field of view. Existing blind restoration methods that assume a spatially invariant kernel and satellite jitter correction methods that rely on auxiliary observations are therefore not directly applicable. We present AstraMoE-SR, a single-image framework that jointly restores motion blur and spatial resolution without auxiliary measurements. Rather than estimating a blur kernel, we infer how the camera moved by reparameterizing degradation as a local exposure trajectory under pushbroom geometry. A conditional diffusion model estimates the trajectory distribution, mitigating the over-smoothing of high-frequency jitter by deterministic point estimation. The predicted trajectory conditions a pretrained latent diffusion backbone through trajectory-guided geometric alignment and spatially adaptive reconstruction. We further show that the remaining point-wise trajectory error is consistent with intrinsic jitter-phase ambiguity that is not resolved by increasing estimator capacity. On all 1,411 DOTA-v1.0 images degraded using our physically motivated forward model, AstraMoE-SR is the only evaluated method to outperform the no-restoration baseline across every fidelity metric, improving on StableSR by 0.64 dB PSNR, 15.2% LPIPS, and 0.091 DINO feature similarity. Reconstructions conditioned on predicted trajectories differ negligibly from those using ground-truth trajectories, indicating that the estimates retain the degradation information required for effective restoration.
Read the full abstract

Sunday 6 September

Image and Video Processing

MedGSSR: Generalizable Medical Image Super-Resolution 3D Reconstruction via Hierarchical Feed-forward Gaussian Splatting

Chengkai Wang, Luoyu Hong, Yiting Zhao et al. (+7 more)

High-resolution volumetric medical imaging is critical for clinical diagnosis, yet acquisition is often limited by scanner hardware, scan time, and for CT, radiation dose. Medical 3D Super-Resolution (Med3DSR) offers a computational alternative, but existing methods commonly rely on per-subject optimization, pretrained priors, or coordinate-based implicit representations, which compromise anatomical fidelity and limit efficiency. To address these limitations, we present MedGSSR, a fully end-to-end feed-forward framework that represents volumes as an explicit 3D Gaussian field for Med3DSR. Unlike coordinate-based implicit functions, our explicit 3D Gaussian representation naturally enhances signal continuity and local high-frequency fidelity. Specifically, MedGSSR explicitly decouples the reconstruction process into coarse-grained structural preservation and fine-grained textural refinement through the proposed Pyramid Anatomical Encoder and a Hierarchical Gaussian Projector. To support arbitrary-scale super-resolution, we introduce sub-voxel Gaussian decomposition and a Differentiable Gaussian Voxelizer that directly queries the continuous 3D intensity field, reducing discretization artifacts. Extensive experiments on MRI and CT benchmarks demonstrate that MedGSSR significantly outperforms state-of-the-art methods. Notably, our framework exhibits robust generalizability across unseen datasets without requiring per-subject optimization, enabling fast inference and high-fidelity volumetric super-resolution in practical clinical settings. Our project webpage, including code, is at https://william2ai.github.io/medgssr
Read the full abstract

All authors Chengkai Wang, Luoyu Hong, Yiting Zhao, Jiamin Wang, Xiang Feng, Feiwei Qin, Zhenzhong Kuang, Xuefei Yin, Ali Bashashati, Yanming Zhu

Image and Video Processing

Drones as Annotators: Amodal 3D Auto-Labeling for Ground LiDAR with Aerial Priors

Tianheng Zhu, Zhenhao Wang and Yiheng Feng

Scaling perception data in autonomous driving is hindered by manual 3D bounding box annotation, a costly and labor intensive process requiring substantial domain expertise. Existing auto-labeling methods reduce this burden, but most of them rely on onboard sensors, where a single ground-level viewpoint yields occluded and sparse observations and inaccurate object geometry. We introduce DAA (Drones as Annotators), a drone-assisted training-free framework for amodal 3D auto-labeling. DAA augments ground LiDAR with an aerial agent that provides less occluded vehicle detections with continuous tracks. Ground LiDAR, in turn, provides precise metric 3D geometry unavailable from aerial imagery alone. DAA exploits the two complementary yet heterogeneous modalities through a two-stage framework: (i) air-ground coordinate alignment unifies aerial detections and ground LiDAR scans into a shared map frame and bootstraps coarse 3D bounding boxes from the aligned aerial detections; (ii) EM-like amodal box refinement alternates between identifying the vehicle's foreground points and updating the coarse box from the identified foreground and aerial priors. We evaluate DAA on an air-ground cooperative perception dataset, where it consistently outperforms existing auto-labeling baselines. LiDAR detectors trained on DAA-generated labels remain competitive with those trained on manual annotations at moderate IoU thresholds. Evaluation against a reference vehicle with known poses and dimensions further confirms its accuracy. We also demonstrate that DAA transfers, without parameter retuning, from onboard to stationary roadside LiDAR and to multi-agent fused point clouds. Code and dataset will be released upon publication.
Read the full abstract

Saturday 5 September

Image and Video Processing · Cross-listed from cs.CV

Spatial Attention Supervision for Defect Localization: Exploiting Ground-Truth Masks as Training Signal in Diffusion-Augmented Defect Detection

Sajjad Rezvani Boroujeni, Muskan Saraf, Gnana Tulasi Makineni et al. (+2 more)

Ground-truth defect masks in industrial inspection datasets are typically reserved for evaluation. This paper repurposes them as spatial supervision signals during training of classification networks, teaching a model not just what to predict but where to look. The method adds an activation-based attention alignment loss that steers convolutional feature maps toward defect regions, in a mixed-supervision formulation that also accommodates samples without masks, such as diffusion-generated images. Combined with DDPM augmentation, synthetic images contribute quantity while masks contribute spatial precision. We evaluate 85 models (four CNN backbones under a 2x2 data/training factorial over five seeds, plus a Swin-V2-T transformer baseline) on the MVTec-AD bottle benchmark, with localization measured on held-out defect images excluded from classifier gradient updates. Main findings: (1) attention-guided training improves activation-based localization (Pixel-AUROC) by +18.0% for EfficientNetB0 with augmentation (p=0.005, Cohen's d=2.6) and +18.7% for ResNet50 (p=0.008), significant in four of eight CNN settings (uncorrected for multiple comparisons) with no significant change in classification; (2) for EfficientNetB0 a data x training-mode interaction is significant (p=0.002), consistent with a super-additive effect (+13.6% combined vs +1.6% summed individual effects); (3) architectures with weaker spatial representations benefit most, whereas ConvNeXt-T shows no effect, apparently because its depthwise-convolution activations yield spatially uninformative channel-mean maps; (4) unsupervised PatchCore remains the strongest localizer (Pixel-AUROC=0.983), contextualizing the supervised gains. These results show that existing evaluation masks can act as practical training signals that measurably and reproducibly improve where defect classifiers attend.
Read the full abstract

All authors Sajjad Rezvani Boroujeni, Muskan Saraf, Gnana Tulasi Makineni, Tom Bush, Hossein Abedi

Friday 4 September

Image and Video Processing · Cross-listed from cs.CV

ViT3Flow: A Test-Time Training Transformer MeanFlow for Postoperative Radiograph Synthesis in Scoliosis

Rui Tang, Sicheng Yang, Moxin Zhao et al. (+6 more)

Predicting postoperative spinal morphology from preoperative radiographs could provide valuable support for scoliosis surgical planning, but remains challenging because surgical correction induces large spatial changes while anatomical structures must be faithfully retained. We formulate this problem as postoperative scoliosis radiograph synthesis and construct ScoliSurg, the first paired dataset for this task, comprising 632 preoperative–postoperative whole-spine radiograph pairs with structured morphology information. We further propose ViT$^{3}$Flow, a single-NFE conditional MeanFlow framework for efficient postoperative radiograph synthesis. ViT$^{3}$Flow models surgical correction as finite-interval generative transport and replaces conventional self-attention with test-time-training token mixers that perform sample-specific inner adaptation to the anatomy and deformity pattern of each case. In addition, a Spinal Morphology Extraction Agent extracts distributions of dominant-curve region and direction from the preoperative radiograph. These distributions guide Diagnosis-Routed Interval Cross-Attention (DRICA), which performs interval-dependent vertical, horizontal, joint, and global retrieval from a separate preoperative token stream. This design enables the evolving postoperative representation to incorporate spatially corresponding anatomical evidence throughout the transport process. Extensive experiments on ScoliSurg demonstrate that ViT$^{3}$Flow achieves the best performance among the compared methods in perceptual image quality, anatomical fidelity, and clinically relevant geometric accuracy, while requiring only a single network evaluation. These results highlight the potential of ViT$^{3}$Flow for efficient and anatomically faithful postoperative radiograph synthesis in scoliosis surgical planning.
Read the full abstract

All authors Rui Tang, Sicheng Yang, Moxin Zhao, Hongqiu Wang, Guankun Wang, Lei Zhu, Hongliang Ren, Menglin Cong, Nan Meng

Image and Video Processing

Real-World Multi-Modal and Longitudinal Lung Cancer Dataset

Rita Cordeiro Mendes, Maria Rita Fonseca Verdelho, Carlos Santiago et al. (+1 more)

Multi-modal learning has demonstrated strong potential in medical applications by integrating heterogeneous data sources such as medical imaging, clinical records, and genomics to improve predictive performance and support clinical decision-making. However, advances in this area are often constrained by two key challenges: the limited availability of well-curated, ready-to-use datasets that accurately reflect real-world conditions, where medical data are frequently collected inconsistently and are often incomplete; and the inherent difficulty of integrating heterogeneous data modalities. In this work, we introduce a newly curated multi-center, multi-modal, and longitudinal dataset designed to support the evaluation of a wide range of learning pipelines under realistic conditions. The dataset comprises a total of 1,365 lung cancer patients and has three imaging modalities (whole-slide images, CT scans, and PET scans), structured clinical data, transcriptomic, and longitudinal follow-up and treatment information. For each imaging modality the dataset contains more than one instance. Moreover, the dataset exhibits substantial and non-uniform missingness across modalities, making it well-suited for studying robust multi-modal fusion strategies. We further provide both uni-modal and multi-modal benchmarks on the task of 12-month overall survival prediction, disease-specific survival, as well as longitudinal benchmark of hazard prediction under severe missing data. Our results show that, despite high levels of missingness, integrating complementary modalities consistently improves predictive performance over uni-modal approaches, highlighting the value of multi-modal fusion in realistic clinical settings. The dataset and benchmark code are available at https://github.com/ritacmendes/MMIST-LUNG.
Read the full abstract

All authors Rita Cordeiro Mendes, Maria Rita Fonseca Verdelho, Carlos Santiago, Catarina Barata

Image and Video Processing

Cross-dataset transportability of pediatric chest X-ray deep learning across three countries: discrimination, calibration, operating-point failure, and limited-label recovery

Nazim-E-Alam

Background and Objective: External evaluation of medical-imaging AI is often collapsed into discrimination. We evaluated a computational protocol that separately tests discrimination, probability calibration, fixed operatingpoint transport, shortcut-associated signal, and limited-label recoverability for pediatric pneumonia classification across datasets from three countries. Methods: After exact-duplicate removal, 5,824 Guangzhou radiographs supported leakage-controlled source development and internal testing. A frozen three-seed DenseNet121 dual-view ensemble was evaluated zero-shot on BDCXR-3257 from Bangladesh (n = 3, 257) and an untouched harmonized VinDr-PCXR/PediCXR test cohort from Vietnam (n = 1, 077). Matched seed-42 variants tested architectural robustness. Secondary BDCXR analyses used a fixed 651-image adaptation pool and 2,606-image hold-out; 163, 326, and 651 labels represented 5%, 10%, and 20% of complete BDCXR. Results: Internal AUROC was 0.976 with 95.1% sensitivity. BDCXR and VinDr-PCXR AUROC were 0.798 and 0.742, while frozen-threshold sensitivity fell to 6.2% and 0%. Source-to-BDCXR AUROC degradation occurred for a full-image baseline (0.961 to 0.749), ungated dual-view model (0.977 to 0.766), and gated MixStyle model (0.966 to 0.789). With 163 BDCXR labels, Platt recalibration preserved AUROC while increasing held-out sensitivity to 88.3%, but specificity was 47.9% and the alert rate was 78.5%. Two hundred repeated 163-label fits confirmed sensitivity recovery but substantial specificity variability. Conclusions: Cross-dataset shifts across countries affected ranking, probability alignment, and source-defined decision behavior differently. Transport studies should evaluate these components separately and quantify the operational burden of apparent recovery.
Read the full abstract

Image and Video Processing · Cross-listed from cs.AR

A Piecewise-Linear Approximation-based Energy-Efficient Error-Optimized Unsigned Square Rooter for Accuracy-Critical Applications

Prateek Goyal and Sujit Kumar Sahoo

Approximate computing improves energy efficiency in error-resilient applications, but square root units remain challenging due to the trade-off between hardware cost and computational accuracy. This paper presents an energy-efficient, error-optimized, piecewise-linear approximation-based unsigned square rooter (EOSQR) for 2n-bit inputs that achieves high accuracy with low hardware complexity, using only simple arithmetic and shift operations. The EOSQR design is implemented in Verilog-HDL and evaluated on a 16-bit benchmark synthesized on an Artix-7 FPGA. Compared to representative state-of-the-art approximate square rooters, EOSQR achieves the lowest error among accuracy-critical designs while delivering 61.91 percent resource savings, 77.54 percent power savings, and 53.11 percent latency reduction relative to a precise restoring array-based square rooter. To enable holistic evaluation, a Composite Efficiency Metric (CEM) is introduced to jointly capture accuracy and energy efficiency. EOSQR is further validated across representative image-processing workloads, including Sobel edge detection, K-means colour quantization, and K-nearest-neighbour (KNN) classification. Experimental results demonstrate that EOSQR achieves high computational accuracy with a superior CEM-based accuracy-hardware efficiency trade-off while maintaining visual quality and classification performance, making it well-suited for real-time edge-embedded systems.
Read the full abstract

Thursday 3 September

Image and Video Processing

Explainable Multimodal Deep Learning Integrating Imaging and Clinical Data for Oral Potentially Malignant Disorder Detection

Ruilin You, Yihan Wang, Jiabin Chen et al. (+4 more)

Oral potentially malignant disorders (OPMDs) are critical precursors to oral cancer, yet clinical detection remains challenging because of substantial phenotypic heterogeneity and overlap with benign conditions. Although image-based deep learning shows promise for automated screening, visual information alone may be insufficient in real-world settings, where diagnostic decisions also rely on patient-specific risk factors. We developed M2-OPMDNet, a multimodal deep learning framework that integrates co-registered white-light and autofluorescence intraoral images with structured clinical information for OPMD detection. A customized questionnaire was designed to capture clinically relevant risk factors and symptoms in a standardized, reproducible format for integration with image-derived features. Multiple image encoders, including conventional convolutional neural networks and foundation model-based architectures, were evaluated using a prospectively collected dataset reflecting real-world screening conditions. Model interpretability was assessed using SHapley Additive exPlanations (SHAP) to quantify feature- and modality-level contributions. M2-OPMDNet achieved an AUC of 0.952, outperforming unimodal approaches and showing improved performance for visually subtle lesions. SHAP analysis demonstrated that structured clinical variables contributed substantially to risk estimation and complemented imaging features. These results demonstrate that explainable multimodal learning combining white-light and autofluorescence imaging with structured clinical data can provide accurate, transparent, and clinically grounded OPMD detection. M2-OPMDNet offers a scalable framework for real-world oral cancer screening and decision support.
Read the full abstract

All authors Ruilin You, Yihan Wang, Jiabin Chen, Cherie Wink, Petra Wilder-Smith, Rongguang Liang, Bofan Song

Image and Video Processing

Fractional-Order Adaptive Motion Magnification: Phase-Reliability Weighting for Noise-Constrained Video Amplification

Alejandro Garnung Menéndez

Eulerian video amplification boosts sub-pixel motion by band-pass filtering per-pixel intensity traces and applying a uniform gain. That gain ignores local structure, so sensor noise is amplified together with the signal, especially in textureless regions where the monogenic phase is unreliable. We propose FrAM (Fractional-order Adaptive Motion Magnification), a pipeline developed first offline and then as a causal stream. It replaces the constant temporal gain with a Grünwald–Letnikov derivative of fractional order, giving continuous control over high-frequency emphasis, and replaces the uniform spatial gain with a per-pixel weight derived from the local amplitude of the monogenic signal. On a controlled synthetic sequence split into textured and flat halves, FrAM matches the amplification of the Eulerian baseline while keeping flat-region temporal noise at the input level. The reduction holds across an eightfold range of input noise levels. Real videos show improved spatial selectivity and lower background noise in every case. The causal reformulation cuts the per-frame cost by two orders of magnitude, reaching 69\,fps at 640$\times$480.
Read the full abstract