Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Electrical Engineering and Systems Science Image and Video Processing

eess.IV Watch Image and Video Processing

Friday 9 October

Image and Video Processing

The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging

Fabian Drexel, Bernhard Kainz, Jakob N. Kather et al. (+4 more)

Joint-embedding predictive architectures (JEPAs) learn by predicting teacher representations of masked regions from visible context. Point prediction summarizes the conditional target distribution and cannot reproduce variation that the context does not resolve. We call this constraint predictability bias. In computed tomography (CT), recurrent anatomy is predictable from context, whereas the sporadic, localized pathology that determines clinical interpretation often is not. We introduce TAP-JEPA, a ViT-B/3D trained on 113,611 CT volumes that is competitive with state-of-the-art encoders (COLIPRI, Merlin, VoxelFM) in scan-level classification. We then audit the learned predictive pathway by replaying the exact masking, teacher, and predictor computations used during training. Prediction lowers the mean Skill (prevalence-normalized average precision) from 0.388 to 0.271 for pathology, but only from 0.921 to 0.873 for anatomy. Along the teacher direction, separating pathology from matched control regions in the same organ, prediction moves pathology 34.7% toward the controls and controls 13.6% toward pathology, with larger shifts farther from visible context. Replacing visible matched-control representations increases surrounding-target prediction loss more than fully replacing pathology representations across four pathology categories. Downstream, removing all final-layer pathology tokens reduces the decision margin by at most 0.45% in four of seven scan-level probes. Together, these results show that contextual predictability is not a sufficient proxy for clinical relevance in medical representation learning. Training objectives should therefore retain diagnostic features regardless of their predictability from surrounding anatomy.
Read the full abstract

All authors Fabian Drexel, Bernhard Kainz, Jakob N. Kather, Daniel Rueckert, Lisa C. Adams, Keno K. Bressem, Cosmin I. Bercea

Image and Video Processing

Stochastic Finite-Transition Learning and Exploration for Accelerated MRI Reconstruction

Muhammad Usama Mirza, Esin Öztürk Iş{\i}k and Tolga Çukur

Generative MRI reconstruction maps an initial source distribution to the target image distribution along a learned recovery trajectory. Common generative methods, such as diffusion and flow, typically learn fine-grained local dynamics whose integration defines this global mapping, hence requiring many network evaluations at inference. Existing few-step sampling approaches can lower computational burden by approximating large transformations from local dynamics, albeit at the expense of reconstruction fidelity. FLEX-MRI instead partitions recovery into a small number of finite intervals between retained states defined by progressive changes in k-space support, and directly learns the reconstruction change across each interval. Because such finite transitions can admit multiple plausible outcomes, FLEX-MRI combines support-relaxed finite-transition learning with stochastic exploration. During training, randomized k-space sampling masks define finite-transition targets with varying support, while the prediction loss remains unrestricted by each individual mask. At inference, FLEX-MRI perturbs a random subset of currently estimated k-space coefficients, allowing the learned predictor to stochastically explore alternative transition paths. Each predicted state is then reconciled with the acquired measurements through data consistency. We theoretically show that this stochastic exploration induces nonzero transition-level variation through the learned recovery map and data consistency without requiring an auxiliary generative sampler. Experiments on multi-coil MRI datasets show that FLEX-MRI outperforms competing methods while maintaining computational efficiency comparable to few-step baselines.
Read the full abstract

Thursday 8 October

Image and Video Processing

Temporal Diffusion Bridge Model for Vessel Extraction Using Physics-informed Synthetic Data

Jinkui Hao, Donald R. Cantrell, Ramez Abdalla et al. (+2 more)

Accurate artery extraction from X-ray coronary angiography (XCA) is essential for coronary artery disease diagnosis and intervention. However, low vessel visibility, respiratory and cardiac motion, and limited annotations challenge existing methods. Moreover, most deep learning approaches produce binary vessel masks without preserving clinically relevant intra-arterial contrast information. We propose a Temporal Diffusion Bridge Model (TDBM) that formulates vessel extraction as image translation and learns from physics-informed synthetic data without manual vessel annotations. A physics-based XCA synthesis engine generates diverse, perfectly aligned pairs of contrast-enhanced and contrast-free images. Using these pairs, TDBM learns cross-domain mappings to recover high-fidelity vessel information. Temporal conditioning incorporates adjacent frames to improve consistency, while a vessel-aware contrastive loss exploits background-invariant synthetic pairs to enhance vascular discrimination. We evaluate TDBM on two XCA vessel segmentation datasets and conduct a reader study using two real-patient XCA sequence datasets. Zero-shot segmentation is further assessed on retinal imaging and fluorescence microscopy datasets. Results demonstrate that TDBM consistently outperforms existing methods and generalizes effectively across imaging modalities. Code and trained weights will be available at: https://github.com/Advanced-AI-in-Medicine-and-Physics-Lab/TDBM
Read the full abstract

All authors Jinkui Hao, Donald R. Cantrell, Ramez Abdalla, Sameer A. Ansari, Bo Zhou

Image and Video Processing

RePrior: Measurement-Reconciled Prior Correction for Sparse-View CBCT Reconstruction

Mengrui Yin, Tianxingjian Ding, Wufan Zhao et al. (+2 more)

Sparse-view cone-beam computed tomography (CBCT) is severely under-constrained by limited X-ray projections. Learned reconstruction can provide complementary anatomical information, but using it as a prior in patient-specific Gaussian reconstruction introduces two challenges. First, the discrepancy between the learned prior and current reconstruction is spatially non-uniform: some local differences contain useful missing 3D information, whereas others reflect prior bias. Second, even a volumetrically beneficial correction can conflict with the observed X-ray measurements. Since X-ray measurements are line integrals, spatially separated corrections may affect overlapping rays, with their signed projection responses reinforcing or partially canceling in the measurement residual, so their feasibility cannot always be assessed independently. In this paper, we introduce RePrior, a multi-stage progressive prior-correction framework that addresses correction actionability, measurement feasibility, and representation expressivity. Selective Correction Proposal (SCP) first identifies actionable signed local corrections from prior–reconstruction discrepancies. Measurement-Constrained Reconciliation (MCR) then jointly enforces their consistency with patient-specific X-ray measurements, while Hybrid Residual Enhancement (HRE) recovers additional correction capacity through complementary voxel- and Gaussian-domain degrees of freedom around the reconciled anchor. RePrior requires neither test-time ground truth nor case-specific network fine-tuning. RePrior gains approximately 2.6~dB in 3D PSNR over R$^2$-Gaussian across LUNA16, PANORAMA, and PENGWIN. Beyond prior-assisted reconstruction, it delivers additional gains in volumetric quality and observed-view projection fidelity, with consistent gains in held-out-view fidelity in the backbone-matched evaluation.
Read the full abstract

All authors Mengrui Yin, Tianxingjian Ding, Wufan Zhao, Ziyang Yan, Yu Tian

Image and Video Processing · Cross-listed from cs.RO

When pose graphs are not enough: a failure analysis of pose-graph-based camera estimation for Gaussian splatting

Senthil Palanisamy, Azwad Sabik, Alan Dolhasz et al. (+1 more)

Exterior Gaussian-splat twins for real-estate listings are captured to a fixed recipe: a consumer drone orbits a single property in several overlapping passes at stepped altitudes, with occasional vertical ascents, so the imagery is dense, sequential, GPS- and time-tagged, and revisits each facade from many angles. This structure is tantalizing: it suggests camera poses could be recovered by a dense pose-graph optimization (PGO) whose only unknowns are the poses, far fewer than the joint pose-and-point problem bundle adjustment (BA) solves. PGO is mature in robotics, but almost always as post-processing: it redistributes the drift of an already-decent trajectory once loop closures tie it together. Splatting captures push that regime to its limit, every frame closing loops with dozens of others, raising a clean question: does pose-only optimization reach BA accuracy? We built a system specialized to this capture structure, not a general-purpose SfM tool, to test it. It beat COLMAP on speed but never on quality; this paper is the candid post-mortem. The shortfall arises from several interacting causes, hand-set weights, greedy merging and a noisy GPS ruler, together with two limits inherent to the formulation: SE(3) edges carry no scale degree of freedom, so independently scaled sub-maps drift when merged, and a two-view edge stops short of the coupling one 3D point imposes on every frame that sees it. A viable design must add back Sim(3) constraints, absolute GPS priors, retained structure, and uncertainty-aware weighting
Read the full abstract

All authors Senthil Palanisamy, Azwad Sabik, Alan Dolhasz, Mira Slavcheva

Image and Video Processing

Ultrasound and Magnetic Resonance Elastography of the Neck: Biomechanical Basis, Clinical Applications and Methodological Challenges

Rashedul Hasan, Anna Yang, Kishani Kandasamy et al. (+4 more)

Many neck disorders, from chronic neck pain and headache to thyroid and nodal malignancy and radiation-induced fibrosis, both arise from and alter the mechanical properties of neck tissues, yet clinical assessment still relies on morphology, palpation and symptoms. Ultrasound and magnetic resonance elastography (MRE) now allow these properties to be measured in vivo, but the neck literature differs widely in methodology and remains dispersed across modalities and tissues. This review brings together the biomechanical basis of neck tissue behavior, the clinical conditions that alter it, and the principles and applications of ultrasound elastography and MRE in the neck. We show that ultrasound elastography has demonstrated diagnostic value for thyroid nodules and cervical lymph nodes, while neck MRE has been shown to be feasible but remains at an early stage. Across both modalities, reported values depend strongly on fiber orientation, posture, muscle activation and acquisition conditions. We therefore propose reporting recommendations and discuss how measurements beyond stiffness, including anisotropy, viscosity and nonlinearity, could improve disease management and inform computational models of the neck.
Read the full abstract

All authors Rashedul Hasan, Anna Yang, Kishani Kandasamy, Hossein Bahreinizad, Md Ashikuzzaman, Suman K. Chowdhury, Shahed Mohammed

Image and Video Processing · Cross-listed from cs.CV

LVS: Local View Synthesis from Relative Camera Pose by Reusing Previous Views

Qizhou Huo, Xuan Sun, Yongfei Guo et al. (+2 more)

Interactive scene exploration requires frequent view updates, although small camera motions preserve much of the visible content. Conventional 3D Gaussian Splatting nevertheless renders each target view, leaving this image overlap unexploited. Reusing rendered images offers an alternative. Geometric warping alone cannot recover newly exposed content and remains sensitive to depth errors. We propose a per-scene framework that replaces repeated scene rendering for nearby views with relative-pose-guided RGB-D image reuse. Geometric warping uses depth and relative pose to transport source content, while a lightweight multiscale network predicts RGB residuals to correct artifacts and infer missing appearance. Cached source features further reduce repeated computation. On GS-render, residual refinement improves PSNR by 0.72~dB over pure warping; evaluations on captured and rendered scenes demonstrate low query latency. This separation of scene rendering from local view updates supports responsive scene exploration, with potential applications in augmented and virtual reality.
Read the full abstract

All authors Qizhou Huo, Xuan Sun, Yongfei Guo, Zhipeng Wang, Yuanhao Gong

Image and Video Processing · Cross-listed from cs.CV

Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi et al. (+2 more)

Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) labels each video segment with an event class, including classes that were never seen during training. The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual-text and audio-text. Existing methods then collapse this pair into a single scalar score with a fixed rule (geometric mean, weighted average) before taking the argmax. Instead, we compute complex-valued similarities and learn their fusion using a complex-valued neural network (CVNN). Each modality's standard representation becomes the real part of our pipeline, and a paired companion stream supplies the imaginary part. We use imaginary part of iHSV for visual modality and CycleGAN-translated phase spectrogram for audio modality as these companion streams. This results in two complex similarities, which are then fused. While the vision and audio encoders remain frozen, only the temporal-attention blocks and the fusion CVNN are trained. The four-stream complex architecture sets a new state of the art on both OV-AVEL benchmarks. On the open (unseen-class) split of OV-AVEBench we reach 66.5/59.1/54.1% Acc/Seg-F1/Event-F1 (+1.6/+4.1/+6.6 over the previously reported fine-tuned baseline), with consistent gains for seen classes as well. We also modify AVE dataset for this task and observe that our architecture reaches 60.7/51.9/50.4% Acc/Seg-F1/Event-F1, achieving state-of-the-art OV-AVEL results on it as well. We also propose a two-stream alternative, which also sees great improvements over the baseline.
Read the full abstract

All authors Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi, Aveen Dayal, Neela Sawant

Image and Video Processing · Cross-listed from cond-mat.mtrl-sci

Localization of Candidate Kikuchi Regions in RHEED Images: Visibility and Annotation Boundaries

Lumou Weng, Hanshan Huang, Zhenhan Zhang et al. (+1 more)

Kikuchi lines and bands in reflection high-energy electron diffraction (RHEED) carry information on crystal geometry and electron scattering, but are often obscured by intense diffraction streaks. We combine multiscale convolution with spatial detail from skip connections and independent supervision that allows overlapping regions to localize streaks and candidate Kikuchi regions separately. Using manual polygon annotations made before model-assisted editing, the Kikuchi intersection-over-union (IoU) on 29 laboratory test images from separate growth batches was $0.6290 \pm 0.0137$. We then performed supervised adaptation to public chalcogenide images and stratified images by Kikuchi visibility using the median ratings of three observers who rated the images separately, two of whom rated with predictions hidden. With all 244 training images, IoU for the clear-feature group of 10 test images was $0.5014 \pm 0.0336$, whereas ambiguous images gave lower values. With 100 training images, clear-group IoU was $0.5171 \pm 0.0073$; expanding the training set also reduced responses on images without target features. Reported uncertainties are sample standard deviations across four laboratory runs or three adaptation runs. Public-image adaptation used reference masks of mixed provenance, including prediction-derived drafts. The method converts visual cues into inspectable spatial regions, providing a basis for analysis of line positions, intersections, and local intensity.
Read the full abstract

All authors Lumou Weng, Hanshan Huang, Zhenhan Zhang, Gan Wang

Image and Video Processing · Cross-listed from cs.CV

A Unified Score Matching Paradigm for Video Anomaly Detection and Anticipation

Congqi Cao, Zhenhe Liang, Hanwen Zhang et al. (+4 more)

Video anomaly detection (VAD) is a fundamental and safety-critical task in computer vision. Recent generative approaches detect anomalies from a distributional perspective, but remain limited by local anomaly modes. Meanwhile, video anomaly anticipation (VAA), as a proactive extension beyond post-hoc detection, introduces additional challenges. In particular, the contrastive inference paradigm in VAD, which relies on ground-truth frames, is not applicable to VAA, hindering its development. To address these challenges, we propose a unified score-driven framework, termed Uni-DSM, based on denoising score matching (DSM), which models anomaly patterns through likelihood estimation and score functions over the learned data distribution. Within this unified framework, we adopt a shared noise-conditioned score transformer backbone with scene-dependent embeddings and motion-aware weighting for distribution-level modeling. Instead of introducing separate architectures, Uni-DSM unifies VAD and VAA through different inference and supervision paradigms built upon the same score-based formulation. For VAD, we instantiate an autoregressive denoising score matching (ADSM) mechanism, which progressively accumulates anomalous evidence via autoregressive denoising, enabling enhanced perception of local modes beyond visual cues. For VAA, we extend the same architecture by incorporating a lightweight auxiliary decoder and a novel self-distilled denoising score matching (SDSM) mechanism. By constructing supervision from output discrepancies instead of relying on unavailable future ground truth, our method achieves efficient training suitable or early anomaly anticipation. Extensive experiments on multiple benchmark datasets demonstrate state-of-the-art performance in both VAD and VAA while maintaining high efficiency, establishing a unified and scalable pipeline from anomaly detection to anticipation.
Read the full abstract

All authors Congqi Cao, Zhenhe Liang, Hanwen Zhang, Yifan Zhao, Qinyi Lv, Lingtong Min, Yanning Zhang

Image and Video Processing · Cross-listed from cs.CV

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

Jingbo Yue, Bruce Coburn, Jinge Ma et al. (+2 more)

Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.
Read the full abstract

All authors Jingbo Yue, Bruce Coburn, Jinge Ma, Jui-Feng Chi, Fengqing Zhu

Image and Video Processing · Cross-listed from cs.CV

Skeleton-Guided Progressive Test-Time Adaptation for Thin Curvilinear Structures

Boa Jang, JunGyu Lee, Gwanho Lee et al. (+2 more)

Accurate segmentation of thin curvilinear structures is vital for various real-world applications, from vessel analysis to road extraction. Yet their intricate geometry makes even minor pixel-wise errors enough to break the global topology, and this structural fragility turns severe domain shifts into catastrophic failures. The difficulty is most acute under cross-modality gaps, where the imaging process itself differs fundamentally between source and target. While test-time adaptation (TTA) offers a practical source-free remedy, existing methods adapt feature statistics and confidence, neither of which constrains connectivity, and thus degrade under such extreme gaps. To address this, we propose Skeleton-Guided Progressive Test-Time Adaptation (SGP-TTA). Progressive Batch Normalization (ProgBN) shifts normalization from frozen source statistics toward current target estimates under a sample-count schedule, so that the source-target balance follows the stage of adaptation rather than a fixed coefficient. Consensus Skeleton Recall (CSR) then derives a structural target from geometrically aligned multi-view predictions and updates only the BN affine parameters to preserve connected structures. Extensive experiments show that SGP-TTA consistently outperforms existing TTA methods in topological connectivity, with the largest margins under cross-modality shift. The project page is available at https://boa-jang.github.io/SGP-TTA.
Read the full abstract

All authors Boa Jang, JunGyu Lee, Gwanho Lee, Jinwook Choi, Young-Gon Kim

Image and Video Processing

Adapting Appearance-Based Gaze Estimation to Narrow-Range, Long-Duration Screen Viewing

Jordan Prescott, Kleanthis Avramidis and Shrikanth Narayanan

Appearance-based gaze estimation offers a low-cost alternative to infrared eye tracking for screen-based behavioral and clinical applications, however existing models are typically developed for wide ranges of gaze angle and head pose. Prolonged screen viewing presents a distinct regime in which gaze remains near the screen center, head motion is limited, and calibration drift accumulates over time. In this work, we benchmark six published estimators along with a proposed differential-gaze model on 140 long-duration facial video recordings with synchronized eye tracking under subject-disjoint evaluation and a common budget of calibration frames. Differential estimation was the only static method that significantly improved upon a baseline predictor of each recording's mean gaze, and was further boosted by addition of temporal context. It also yielded the strongest agreement with reference fixations and saccades, demonstrating that low angular error alone could not validate eye movement reconstruction. These findings establish differential estimation as a promising foundation for reliable gaze tracking in long-duration, narrow-range settings.
Read the full abstract

Wednesday 7 October

Image and Video Processing · Cross-listed from cs.SD

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing

William Chen, Prem Seetharaman, Ke Chen et al. (+10 more)

Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.
Read the full abstract

All authors William Chen, Prem Seetharaman, Ke Chen, Oriol Nieto, Kevin Duarte, Siddharth Srinivasan Iyer, Mamshad Nayeem Rizve, Zhiwen Cao, Shinji Watanabe, Yuanjun Xiong, Jianming Zhang, Zeyu Jin, Justin Salamon

Image and Video Processing

Diagnosing Diversity Collapse and Validating Mask-Conditioned Diffusion for Labeled Microtubule Microscopy

Mario Koddenbrock, Frederic Rapp, Simone Reber et al. (+1 more)

Mask-conditioned diffusion models have become standard for generating labeled training data in biomedical imaging. But the tools used to evaluate them were built for a different problem. Standard metrics like FID and KID score images through ImageNet-trained features. Those features do not transfer to microscopy, leaving the metrics poorly calibrated for specialized domains. Even with a domain-appropriate feature space, a single blended number can still hide a real failure: a model can produce visually plausible images that still transfer poorly to downstream tasks. We identify a specific failure mode: Late in training, realism keeps improving while texture diversity collapses. The model settles onto a fixed color palette, converging to a high-similarity, low-diversity state that standard metrics do not penalize. Therefore, we propose a diversity-aware diagnostic in a domain-validated feature space. It combines a realism axis based on inter-similarity to real images with a diversity axis based on intra-similarity among generated samples. This diagnostic selects a checkpoint that domain experts cannot reliably distinguish from real recordings in a forced-choice study. It also yields useful downstream segmentation: a segmenter trained only on synthetically labeled data reaches a median skeletonised IoU comparable to one trained on real data, at lower cross-fold variance, and clearly ahead of the best available public alternative in this domain, a parametric renderer. We also reproduce a data-efficient hyperparameter-transfer experiment from prior work, tuning several foundation segmenters on a small labeled subset and evaluating on real images: transfer is stronger with our data. We release DiffuMT on HuggingFace, including the triplet dataset, code to reproduce the downstream-utility validation, and a standalone diagnostic tool for evaluating mask-conditioned diffusion models.
Read the full abstract

All authors Mario Koddenbrock, Frederic Rapp, Simone Reber, Erik Rodner

Tuesday 6 October

Image and Video Processing

On-Device Super-Resolution Imaging for a Low-Cost SPAD Array on a 640-KB-SRAM Microcontroller

Zhenya Zang, Istvan Gyongy and Mike Davies

This work presents a super-resolution (SR) deep-learning (DL) architecture to simultaneously upscale low-resolution (LR) depth and intensity images acquired by a consumer-grade VL53L9CX time-of-flight (ToF) sensor, a compact single-photon avalanche diode (SPAD) ranging device, while targeting deployment on a resource-constrained microcontroller (MCU) with only 640 KB SRAM and 2 MB Flash. The VL53L9CX provides 54 $\times$ 42 depth and intensity measurements. The proposed framework performs $\times$4 spatial SR to reconstruct 216 $\times$ 168 depth and intensity images simultaneously. The network employs separate reconstruction branches for intensity and high-resolution (HR) depth. We train the network with efficient loss functions and investigate the performance by testing multiple compact, hardware-oriented backbones, including Spatially Adaptive Feature Modulation (SAFM), Swift Parameter-Free Attention Network (SPAN), and Residual Local Feature Network (RLFN). We evaluate the network on both synthetic and real measurements, with particular attention to constrained activation and memory storage. We export the selected network to ONNX, quantize it to INT8, generate C inference code for an STM32H563ZI Arm Cortex-M33 MCU, and integrate it with our simplified sensor's firmware. This work is the first to demonstrate a compact DL SR model on a low-cost bare-metal MCU for a consumer-grade LR SPAD sensor, integrated with customized MCU firmware to enable on-device SR inference.
Read the full abstract

Image and Video Processing

Hybrid++: The Bridge between PDE Models and Deep Learning for Gamma Noise Removal

Mahipal Jetta and Sujato Dutta

Multiplicative gamma noise is one of the dominant noise factors in Synthetic Aperture Radar (SAR) and medical ultrasound images. They are dependent on pixel level noises due to which they are highly varying across the image and harder to handle as compared to additive noise. The denoising methods to address this noise currently include classical partial differential equation (PDE) methods which are good for interpretation but lack the restoration ability as compared to the state-of-the-art models while deep convolutional networks like DnCNN achieve high performance but at the cost of transparency due to which practitioners are skeptical to use them in high-risk critical fields like medicine. This paper presents Hybrid++, a novel trainable nonlinear reaction-diffusion architecture that addresses both the concerns - staying interpretable while offering performance close to huge black-box models. It combines a fully learnable PDE initialization with a 3-stage reaction-diffusion network having 64-channel multiscale filter banks, 4-layer Squeeze-and-Excitation attention-based influence functions and a 64-dimensional noise level embedding. It uses a two-phase training strategy, stage-wise optimization followed by joint end-to-end refinement which enables co-adaptation of all learnable parameters. On the FoE benchmark, Hybrid++ substantially improves over classical PDE, BM3D and the original TNRD baselines. In the severe-noise setting L=1, it comes within 0.23 dB PSNR of a separately trained DnCNN while using only about 8% of its parameters. We therefore position Hybrid++ not as a universal state-of-the-art image restoration backbone, but as a compact, physically structured reaction-diffusion model for multiplicative gamma noise.
Read the full abstract

Image and Video Processing · Cross-listed from cs.CV

RACE-FPP: A Robust AI-assisted Characterisation Enhancement for Fringe Projection Profilometry

Osman Ali, Xiangjun Kong, Tibebe Yalew et al. (+2 more)

Fringe Projection Profilometry (FPP) requires precise system characterisation to achieve reliable three-dimensional (3D) reconstructions; however, characterisation accuracy strongly depends on robust checkerboard feature localisation, which can deteriorate under challenging imaging conditions such as lens blur and characterisation target orientations. Existing deep learning-based corner detectors are typically assessed using detection metrics and camera reprojection error alone, without considering their wider impact on projector characterisation, camera-projector stereo characterisation consistency, or overall measurement accuracy. In this work, we introduce a complete FPP characterisation pipeline that incorporates deep learning-based corner detection into the standard camera characterisation workflow. We also characterise the projector by sampling phase values at the centres of the white squares in the characterisation target. Rather than treating corner detection as an isolated task, the proposed framework explicitly analyses how localisation errors propagate throughout the entire FPP characterisation chain. Performance is evaluated using detection metrics (e.g., precision and recall), camera and projector reprojection errors, and the camera and projector stereo characterisation. Across a mixed dataset of clean and degraded images, the camera reprojection error is reduced from 1.237 pixels to 0.259 pixels, while the projector reprojection error is reduced by roughly 50%. Dimensional evaluation of reconstructed artefacts shows improved geometric accuracy compared with those resulting from the conventional pipeline. Overall, the findings indicate increased robustness of system-level characterisation under challenging imaging conditions, thereby enabling more reliable industrial FPP measurements.
Read the full abstract

All authors Osman Ali, Xiangjun Kong, Tibebe Yalew, Waiel Elmadih, Samanta Piano

Image and Video Processing · Cross-listed from physics.med-ph

Laplace-Domain Beamforming for Ultrafast Plane-Wave Imaging

Martin F. Schiffner

Fourier-domain beamforming methods, such as filtered backpropagation (FBP), are beneficial in ultrafast plane-wave imaging due to their computational efficiency and high image quality. These methods use the Fourier diffraction theorem (FDT) for steered plane waves to exactly invert linear scattering models. However, these models rely on simplifying assumptions that degrade image quality. The use of linear transducer arrays, in particular, requires: (i) a two-dimensional space, and (ii) lossless tissues. Herein, Laplace-domain beamforming is proposed. This method retains the computational efficiency of the Fourier-domain methods while improving image quality for linear arrays. Specifically, the method uses a generalization of the FDT, here called Laplace diffraction theorem, to account for three spatial dimensions, finite array element heights, and lossy tissues. A phantom experiment showed that the proposed method improves image uniformity, spatial resolution, and contrast compared to FBP. Using three steering angles, the lateral and axial -6 dB-widths of wires reduced by up to 12% and 25%, respectively, while the contrast of anechoic regions improved by up to 9%.
Read the full abstract

Image and Video Processing · Cross-listed from math.NA

Learned Adaptive Multiresolution Diffusion Imaging

Christian Tantardini, Stig Rune Jensen, Roberto Di Remigio Eikås et al. (+1 more)

Adaptive multiresolution methods reduce representation cost by concentrating fine-scale degrees of freedom where needed, but their tree updates are usually governed by fixed local criteria. We introduce Learned Adaptive Multiresolution Diffusion Imaging (Learned AMDI), which preserves the AMDI fixed-tree propagator and hierarchy constraints while replacing the post-propagation selector with a shared local policy trained by proximal policy optimization. Regression tests reproduce deterministic AMDI trajectories to machine precision when identical trees are used. In the Haar implementation studied here, the deterministic one-step selector accepts no refinements in 54 decisions. Across nine held-out cases, Learned AMDI executes 393 refinements and reduces the mean terminal reference discrepancy from $0.17496$ to $0.13657$, while occupancy rises from $0.13737$ to $0.26660$. Step-resolved diagnostics reveal occasional small adaptation-energy increases; fixed-tree energy stability therefore does not guarantee monotonicity of the learned outer iteration. At comparable occupancy, a validation-tuned observed-detail threshold reaches a discrepancy of $0.13792$ with slightly better RMSE and SSIM, placing both methods on essentially the same accuracy–occupancy tradeoff. A decision-1-only control reaches $0.13742$, indicating that most of the improvement on this static benchmark arises from the initial allocation. The shared actor transfers without retraining to $64\times64$ and $128\times128$ images, improving reference discrepancy, RMSE, and SSIM relative to deterministic AMDI, while the frozen threshold rule remains competitive. Learned AMDI thus provides a hierarchy-constrained, resolution-transferable mechanism for adaptive allocation and clarifies the contribution of sequential decisions.
Read the full abstract

All authors Christian Tantardini, Stig Rune Jensen, Roberto Di Remigio Eikås, Joakim Henrik Beck

Image and Video Processing · Cross-listed from cs.CV

CETUS: How Far Do Representations Trained on Earth Transfer to Cassini SAR of Titan?

Kevin Lee

Cassini synthetic aperture radar (SAR) images reveal the dunes, plains, and lake basins of Titan, providing an instance of representations learned from Earth imagery for planetary terrain classification. Cross-domain Evaluation of Earth-to-Titan Transfer Using SAR (CETUS) compares features from DINOv2, DOFA and CROMA with classical image measurements and features from an untrained vision transformer on the U.S. Geological Survey's Cassini SAR mosaic. The classifiers learn terrain labels from an expert geomorphological map and predict those labels in geographically separate Titan regions. Under logistic regression settings, pretrained encoders achieve higher mean macro F1 than the combined classical features. Encoder rankings change when feature scaling, optimization, and regularization change together. Further training on Titan improves DINOv2 performance, degrades DOFA performance, and leads to mixed results for CROMA under the tested settings. Architectural and input processing differences prevent these comparisons from isolating the effect of pretraining. Classifier fitting and performance on individual terrain classes matter when assessing representation transfer for planetary mapping. Since the map draws partly on the same radar observations, the scores measure agreement with expert interpretation.
Read the full abstract

Image and Video Processing · Cross-listed from cs.CV

LARK: A Low-Cost, Accurate, Occlusion-Resilient, Kalman Filter-Assisted Tracking System for Image-Guided Surgery

George Sideris, Justin Cree, Andrew Stirling et al. (+3 more)

Image-guided surgery (IGS) depends on accurate tracking of surgical instruments to provide real-time navigation relative to anatomical structures. Commercial stereo infrared trackers are accurate but prone to occlusion and cost-prohibitive for many settings. This work presents LARK, a multi-camera optical tracking system using commodity RGB hardware and multi-view redundancy and fusion. We develop and evaluate two complete tracking methods: multi-view monocular pose fusion and multi-view triangulation. Both methods are assessed under varying occlusion levels using a precision-machined grid and an anatomical head phantom, and compared against a gold-standard stereo infrared system. With five cameras and adaptive Kalman filtering, LARK achieves median target registration errors of 0.64 mm for point localization with triangulation and 0.73 mm for trajectory tracking with pose fusion on the machined grid. Camera-subset experiments show graceful degradation in adaptive pose-fusion accuracy as fewer views remain available. With tracking hardware costing under $1,000 USD, LARK provides a low-cost platform for image-guided surgery research. Hardware designs and software are publicly available at https://nist.mni.mcgill.ca/software/ , and datasets at https://nist.mni.mcgill.ca/data/ .
Read the full abstract

All authors George Sideris, Justin Cree, Andrew Stirling, Mamadou Ly, Étienne Léger, D. Louis Collins

Monday 5 October

Image and Video Processing

Geometry-Aware Diffusion Approximate Posterior Sampling for Sparse-View and Limited-Angle CT

Honglei Brinkmann, Carola-Bibiane Schönlieb and Ander Biguri

Sparse-view computed tomography (CT) reduces radiation dose and acquisition time and may mitigate motion artifacts. However, angular undersampling provides insufficient information to determine the image uniquely and stably. Limited-angle CT, arising from restricted angular coverage, produces strongly directional information loss associated with the missing angular range. In both settings, image directions may be strongly observed, weakly constrained, or unobservable, leading to severe ill-posedness and reconstruction ambiguity. Existing diffusion-based approaches incorporate measurement information through likelihood guidance, data-consistency operations, or range-null-space corrections. However, they do not generally use the continuously varying measurement sensitivity of the acquisition to jointly shape both reconstruction updates and stochastic exploration. We propose a geometry-aware diffusion-guided stochastic reconstruction framework for sparse-view and limited-angle CT. Its central component is a regularized noise-weighted pullback metric constructed from the CT forward operator and measurement-noise covariance. This metric continuously adapts both the measurement-aware update and stochastic exploration according to directional measurement sensitivity, suppressing changes along strongly constrained directions while permitting greater exploration along weakly constrained and unobservable directions. We complement this geometry-aware update with a regularized data-consistency correction and approximately null-space-restricted stochastic perturbations, implemented matrix-free using forward and backprojection operations together with conjugate-gradient solves. Experiments on sparse-view, noisy, and limited-angle CT demonstrate competitive reconstruction quality, strong measurement consistency, and spatially resolved empirical uncertainty estimates.
Read the full abstract

Image and Video Processing · Cross-listed from cs.CV

Multitask Conditional Generative Adversarial Network Enables Automatic Whole Knee Cartilage and Menisci Segmentation and Reliable $T_{1\rho}$ and $T_2$ Quantification Without High-Resolution Morphological Images

Ahmed Tahseen Minhaz, Richard Lartey, Zhiyuan Zhang et al. (+8 more)

Early osteoarthritis detection through quantitative MRI (qMRI) requires accurate cartilage and meniscus segmentation, traditionally necessitating time-consuming, costly 3D high-resolution Double Echo Steady-State (DESS) MRI scans. This study developed a multi-task conditional generative adversarial network (MT-cGAN) to simultaneously synthesize DESS-like images and segment tissues directly from qMRI echo images. This retrospective study evaluated 508 knee MRI volumes from 361 subjects (mean age: $40.4 \pm 12.2$ years; 179 female) across three cohorts. Ground truth segmentation masks were generated from DESS images using a pretrained model with manual correction, and $T_{1\rho}$ and $T_2$ maps were computed from magnetization-prepared angle-modulated partitioned $k$-space spoiled gradient echo snapshots (MAPSS) echo images. MT-cGAN was trained to jointly synthesize DESS-like images and segment cartilage and meniscus directly from echo images. Model performance was evaluated using Dice score for segmentation accuracy and coefficient of variation (CV) for $T_{1\rho}$ and $T_2$ quantification. MT-cGAN achieved the highest segmentation performance, mean Dice score 0.84 (range: 0.80–0.86) across all cartilage and meniscus compartments and significantly outperformed the state-of-the-art conditional GAN model with transfer learning (mean Dice, 0.82; $p < 0.001$, Wilcoxon signed-rank test). For relaxometry quantification, MT-cGAN demonstrated the highest consistency with the reference DESS protocol, yielding the lowest CV ($T_{1\rho}$: 1.84%, $T_2$: 1.81%). The proposed MT-cGAN accurately segmented cartilage and menisci while providing reliable $T_{1\rho}$ and $T_2$ quantification directly from echo images. By eliminating the need for separate morphological DESS scans, this workflow reduces required scan times to facilitate the clinical translation of qMRI.
Read the full abstract

All authors Ahmed Tahseen Minhaz, Richard Lartey, Zhiyuan Zhang, Jeehun Kim, Kunio Nakamura, Mingrui Yang, Jiasen Zhang, Weihong Guo, Naveen Subhas, Carl S. Winalski, Xiaojuan Li

Image and Video Processing · Cross-listed from cs.CV

Analysis of SWIR Imaging Detection Performance Under Adverse Environmental Conditions for Autonomous Driving Systems

Rohan Mehra, Alexandre Riffard, Yannis Loumouamou et al. (+1 more)

Short-wave infrared (SWIR) imaging has emerged as a promising modality for autonomous driving, yet its practical benefits over RGB remain poorly characterized across diverse conditions. This paper presents a systematic comparative study of paired RGB and SWIR object detection on the RASMD dataset, covering four weather conditions and two real-time detection architectures, with various fine-tunings evaluated against a unified ground truth. Overall, RGB demonstrates comparable or superior performance in most scenarios, while RF-DETR exhibits greater robustness across varying conditions. Beyond aggregate metrics, we propose a sensor-dominance mining framework that combines multi-model agreement with targeted manual inspection to identify scenarios where one sensing modality provides more reliable detections using largely unannotated paired data. This analysis reveals that SWIR offers clear advantages in four safety-critical situations, including windshield glare, water droplets on the windshield, low-contrast object visibility, and long-range vehicle detection. The findings suggest that SWIR should be viewed as a complementary modality that enhances perception in rare but challenging conditions. The datasets will be available upon request, and all code and trained model weights are publicly released at https://github.com/comsee-research/swir-adverse-env-analysis.
Read the full abstract

All authors Rohan Mehra, Alexandre Riffard, Yannis Loumouamou, Mathieu Labussière

Image and Video Processing

Right Bregman proximal gradient with application to Poisson inverse problems *

Thibaut Modrzyk, Elie Bretin and Voichita Maxim

We introduce a novel Bregman proximal algorithm for convex composite optimization by applying the standard Bregman proximal gradient (BPG) method to a mirror-coordinate reparameterization of the objective. In primal variables, the resulting algorithm alternates between a preconditioned gradient step followed by a right Bregman proximal update. Our analysis relies on relative smoothness holding in the mirror coordinates instead of primal coordinates. Under this condition, we prove monotonic decrease of the objective in the general convex setting. We then specialize the method to Poisson inverse problems using weighted negative entropy as the potential. The resulting scheme recovers the classical Richardson-Lucy multiplicative updates and extends them to general convex regularizers. Building on a recent convergence analysis of multiplicative updates, we establish a sublinear convergence rate in function values for the Poisson setting. Finally, we demonstrate the performance of the regularized algorithm on several imaging inverse problems with Poisson-distributed observations. The code is publicly available at https://github.com/Tmodrzyk/MU-Bregman.
Read the full abstract

Image and Video Processing

Diffusion Meets Unrolling: Compressive SAR Image Reconstruction with Interleaved Learned Corrections

Odysseas Pappas, Andrew C. M. Austin, Perla Mayo et al. (+2 more)

Compressive Synthetic Aperture Radar (SAR) imaging, typically formulated as an inverse problem and solved with traditional iterative optimisation methods, can be very computationally expensive. We investigate the use of denoising diffusion probabilistic models (DDPMs) for compressive SAR image reconstruction, where the diffusion model is guided by a poor initial reconstruction from sub-sampled data obtained via standard imaging methods. We augment this data-driven method with model-driven interleaved refinement processes, inspired by traditional compressed sensing (CS) methods, to enhance sparsity and heavy tails in SAR images. Experimental results on real ERS-1 SAR datasets demonstrate consistent improvements over baseline DDPM and state-of-the-art methods, with notable gains in reconstruction fidelity and structural similarity, while incurring only modest computational overhead. The proposed hybrid framework effectively combines the representational power and efficiency of diffusion models with the interpretability and robustness of model-based optimisation, enabling accurate compressive SAR imaging.
Read the full abstract

All authors Odysseas Pappas, Andrew C. M. Austin, Perla Mayo, Mohammad Golbabaee, Alin Achim

Image and Video Processing

AI-Driven XR Situational Awareness Platform for Urban Crisis Management and Smart Mobility Operations

Dimitris Spyridonidis, Gerasimos Arvanitis and Konstantinos Moustakas

Urban environments are increasingly exposed to complex, dynamic, and interdependent risks, ranging from traffic incidents and infrastructure failures to large-scale crises such as extreme weather events and emergency response scenarios. In such conditions, decision-makers and operators are required to act under time-critical constraints while relying on fragmented, heterogeneous, and often incomplete information. The lack of unified situational awareness, combined with limited visibility and disconnected systems, significantly affects response time, coordination efficiency, and overall operational effectiveness. In parallel, modern cities have deployed extensive sensing infrastructures, including surveillance camera networks, IoT devices, connected vehicles, and satellite-based observation systems. Although these technologies generate vast amounts of data, their exploitation remains limited due to the absence of integrated platforms capable of real time data fusion, intelligent interpretation, and intuitive visualization. As a result, a substantial gap persists between data availability and actionable intelligence, particularly in safety-critical and crisis management applications. To address this challenge, this paper presents an AI-driven XR situational awareness and operational platform designed for urban crisis management and smart mobility operations. The proposed system integrates data from city infrastructure, connected vehicles, and VRUs, enabling a comprehensive and real-time understanding of the urban environment. Through a unified operational dashboard and AR interfaces, the platform supports both centralized monitoring and field-level interaction. By enhancing perception, enabling cooperative awareness, and delivering context-aware information, the proposed approach improves decision-making, coordination, and safety across diverse urban scenarios.
Read the full abstract

Image and Video Processing

How well do routinely collected demographic and clinical variables aid point-of-care lung ultrasound TB classification

Joshua M. Jansen van Vüren, Christiaan M. Geldenhuys, Devendra S. Parihar et al. (+7 more)

We consider the fusion of lung ultrasound images with routinely-collected clinical and demographic data for the purpose of automated tuberculosis (TB) screening using deep-learning. Such deep-learning based screening tools for TB could meaningfully support the health care system in Africa, where the burden of disease is severe and resources are constrained. Beginning with an established ResNet baseline for classification of lung ultrasound images, which achieves an area under the receiver operating characteristic (AUROC) curve of 0.91 [0.86,0.96] (95% CI), we consider the incorporation of the clinical and demographic data using three fusion approaches. We find that a simple average-based fusion of the output scores of separately-trained image and clinical data classifiers consistently matches or outperforms a more complex approach where the data is fused earlier and a combined classifier is trained. Fusing the image and the clinical classifiers in this way leads to a classifier with an overall AUROC of 0.95 [0.91,0.99] (specificity of 0.76 at sensitivity 0.93) which is an improvement of 4% absolute over the image-only baseline. We also find that greedy feature selection can be used to reduce the number of clinical and demographic inputs without sacrificing classification performance. Finally, when we differentiate between clinical and demographic data that are self-reported, that require some basic measurement or calculation, and that require a point-of-care (POC) test, we find the inclusion of the POC tests included in this study to be of minimal benefit to classification performance. We conclude that the incorporation of routinely-collected clinical and demographic data is a promising way to improve the performance of lung ultrasound based automatic classification.
Read the full abstract

All authors Joshua M. Jansen van Vüren, Christiaan M. Geldenhuys, Devendra S. Parihar, Véronique Suttels, Trevor Brokowski, Ablo P. Wachinou, Mary-Anne Hartley, Rensu P. Theart, Grant Theron, Thomas R. Niesler

Sunday 4 October

Image and Video Processing

Distortion-Free High-Resolution PROPELLER-DWI: Technical Advances and Initial Demonstration for the Prostate

Jingjia Chen, Kun Zhou, Haoyang Pei et al. (+7 more)

Purpose: To develop and evaluate an integrated acquisition and reconstruction framework for distortion-free, high-resolution prostate diffusion-weighted imaging (DWI). Methods: PROPELLER-DWI combines pulsed-gradient spin-echo diffusion preparation, an interleaved gradient-and-spin-echo PROPELLER acquisition with golden-angle blade rotation, and self-supervised joint reconstruction and denoising. It was compared with conventional NUFFT reconstruction and MPPCA denoising in a NIST/QIBA diffusion phantom, 25 volunteers, and 4 clinical patients undergoing 3T prostate MRI across three spatial-resolution settings and two b-values (50 and 1000 s/mm2). Reader studies compared geometric fidelity, image quality, and overall preference with clinical single-shot EPI-DWI. Results: PROPELLER-DWI preserved prostate geometry in cases where EPI-DWI showed susceptibility-induced distortion, signal loss, and metal-related artifacts. Self-supervised reconstruction improved noise suppression, with the greatest benefit at higher spatial resolution. Across all resolutions, PROPELLER-DWI had significantly higher geometric fidelity than EPI-DWI, and radiologist preference increased with the severity of EPI distortion. ADC measurements remained consistent across resolutions and comparable to clinical EPI-DWI, whereas EPI-DWI ADC varied with increasing resolution. Retrospective reduced-shot reconstruction also supported shorter scan times. Conclusion: PROPELLER-DWI enables distortion-free, high-resolution prostate DWI while preserving quantitative diffusion measurements, providing a practical alternative for technically challenging prostate MRI and applications requiring improved geometric fidelity.
Read the full abstract

All authors Jingjia Chen, Kun Zhou, Haoyang Pei, Angela Tong, Sherif Elsherif, Luke Ginocchio, Mahesh Bharath Keerthivasan, Qiuting Wen, Hersh Chandarana, Li Feng