Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Electrical Engineering and Systems Science Image and Video Processing

eess.IV Watch Image and Video Processing

Sunday 20 September

Image and Video Processing · Cross-listed from cs.AR

FINN-Tro: Exploiting Verification Gaps in Dataflow Inference Accelerators

Qazi Arbab Ahmed, Suraj Karki and Thorsten Jungeblut

The growing adoption of dataflow accelerators for neural network inference introduces new attack surfaces that existing verification methodologies fail to address. Inference ac- celeration frameworks such as FINN, which transform quantized neural networks into FPGA-deployable dataflow architectures, implicitly assume semantic equivalence between the software model and the synthesized hardware. In this work, we in- troduce the FINN-Tro attack, which identifies and exploits a critical verification gap in the FINN compilation pipeline that enables stealthy hardware Trojan insertion without modifying the original quantized model. The Trojan is placed in the last Matrix-Vector Activation Unit (MVAU) layer and supports two counter-based trigger modes, periodic and persistent, and three payload types: bias addition, logit swapping, and bias subtraction, resulting in six different configurations. FINN-Tro is evaluated on an MNIST feed-forward network and a CIFAR-10 convolutional neural network deployed on a PYNQ-Z1 board. Across the evaluated configurations, accuracy reductions range from 0.90% to 82.84%, while throughput and runtime remain close to the corresponding baseline designs. The most severe configuration, persistent Bias Addition, reduces accuracy from 92.96% to 10.12% on MNIST and from 84.19% to 10.00% on CIFAR-10. The inserted logic introduces modest implementation overhead, with maximum LUT and FF increases of 6.71% and 7.49% for MNIST, and 2.50% and 3.98% for CIFAR-10, respectively. Our findings reveal that widely used pre- and post- compilation verification flows are insufficient for detecting such temporally delayed hardware manipulations, motivating the need for stronger verification mechanisms in accelerator toolchains.
Read the full abstract

Image and Video Processing · Cross-listed from q-bio.QM

WILSON - a pathology foundation model framework for patient-level analysis and diagnostic text generation

Saghir Alfasly, Wataru Uegami, Sobhan Hemati et al. (+13 more)

Pathologists integrate morphology across magnifications and across the slides of a patient case, whereas pathology foundation models encode thousands of tiles from single slides and aggregate their features. Here we present WILSON, a vision–language foundation model that represents whole-slide images and multi-slide cases as single multi-magnification composite images, trained on approximately 189k slides from Mayo Clinic spanning 42 organs and 829 diagnostic entities using pathology reports as supervision. Without task-specific training, WILSON exceeded a dedicated case-level model on all internal cohorts (macro-F1 0.52 versus 0.38) and matched slide-level models up to 9.4 times larger at 272- to 2,155-fold lower compute. End-to-end fine-tuning on 508 triple-negative breast cancer cases improved histologic subtyping and stromal tumor-infiltrating lymphocyte grading by 0.16 and 0.11 macro-F1. WILSON retrieved matching diagnostic text at 75.6% recall@1 (PRISM, 58.1%) and generated captions closer to report-derived references than PRISM and PRISM2 on the internal cohort and on most external comparisons. Composite images thus offer a compact, clinically aligned computational unit for pathology.
Read the full abstract

All authors Saghir Alfasly, Wataru Uegami, Sobhan Hemati, Wenchao Han, Xiaojia Tang, Kevin Thompson, Daniel Stone, Ghazal Alabtah, Saba Yasir, Michael R. Lucas, Eric W. Klee, Cheryl L. Willman, Judy C. Boughey, Matthew P. Goetz, Krishna R. Kalari, H.R. Tizhoosh

Image and Video Processing

ORION-CMR: On-scanner Reporting with Integrated Foundation Model for End-to-End Cardiac MRI Analysis and Interpretation

Omer Burak Demirel, Kelly K. Horst, Alessio Perazzolo et al. (+15 more)

Cardiovascular magnetic resonance (CMR) provides comprehensive cardiac assessment but remains underutilized because of the complexity of acquisition, post-processing, and interpretation. Existing artificial intelligence (AI) methods address isolated tasks, limiting clinical integration. We present ORION-CMR (On-scanner Reporting with Integrated fOunda-tioN Model), the first clinically evaluated scanner-native end-to-end CMR foundation model. Pretrained on 12,896,733 CMR images from 9,258 studies, ORION-CMR performs sequence classification, ventricular function assessment, late gadolinium enhancement (LGE) detection, binary and multiclass disease classification, and local large language model-based report generation in approximately 90 seconds. The framework. was evaluated on public benchmarks and clinically validated in a multi-vendor cohort of 68 subjects with normal examinations, congenital heart disease, dilated cardiomyopathy, and myocardial infarction. ORION-CMR outperformed supervised baselines and the previously published CMR foundation model (CMR-FM), achieving state-of-the-art performance for LGE classification and scar segmentation. Clinical evaluation achieved an AUC of 0.96 for normal-versus abnormal classification and 0.88 for multiclass disease classification, while generated reports demonstrated 81.4% agreement with expert interpretation. These results demonstrate the feasibility of real-time scanner-native AI-assisted CMR analysis and automated report generation.
Read the full abstract

All authors Omer Burak Demirel, Kelly K. Horst, Alessio Perazzolo, Elisa Bruno, Kenan Kaya, Rongzhen Ouyang, Enas Ahmed, Jouke Smink, Spencer L. Waddle, Zainudeen Kallumpurath, Tzu Cheng Chao, Dinghui Wang, Steve G. Langer, Timothy L. Kline, Panagiotis Korfiatis, Jacinta Browne, Ivana Isgum, Tim Leiner

Image and Video Processing

VGG16-MCA UNet: Whole-Tumor Segmentation in 2D FLAIR MRI with Decoder-Side Channel Attention

Shubham Gajjar, Deep Joshi, Avi Poptani et al. (+1 more)

Automated brain tumor segmentation supports diagnosis, treatment planning, and monitoring of disease progression, but building models that generalize across heterogeneous tumors and limited annotated data remains difficult. We present VGG16-MCA UNet, a hybrid architecture pairing an ImageNet-pretrained VGG16 encoder with a decoder in which a Multi-Channel Attention (MCA) module recalibrates features after each skip-connection fusion, trained with the Focal Tversky loss to counter severe foreground-background imbalance. We evaluate the model as a 2D, FLAIR-only, whole-tumor segmenter on tumor-positive slices from two public datasets: the BraTS 2020 benchmark and the LGG MRI Segmentation dataset. Using 5-fold cross-validation and a single network formed by averaging the weights of the five fold models, the method attains an aggregate pixel-level Dice (F1) of 95.10% on our held-out BraTS 2020 split and 88.32% on LGG. These scores are computed over all test pixels pooled into a single confusion matrix rather than averaged per case, and are therefore not directly comparable to the per-case mean Dice used in the BraTS challenge protocol. All partitions were drawn over individual slices rather than over patients, so every patient contributes slices to both training and test; the figures above therefore measure interpolation within known patients and should be read as an upper bound rather than as generalization to new ones. The model segments a 256x256 slice in 66.32 ms on a single 6 GB NVIDIA RTX 2060, approximately 8 ms more than an equivalent VGG16-UNet without MCA. We release the split records and report the protocol in full, with the aim of providing a precisely specified and reproducible 2D FLAIR baseline.
Read the full abstract

All authors Shubham Gajjar, Deep Joshi, Avi Poptani, Vishal Barot

Image and Video Processing

Learning-Based 3D Reconstruction of Power Networks from Aerial Point Clouds

Rishabh Jain, Anuja Saini and Vishal Jain

This paper presents an end-to-end framework for reconstructing overhead power utility network topology and extracting span-level physical metadata from large-scale aerial LiDAR. The pipeline begins with semantic segmentation of the input point cloud using an improved KPConv-based model, in which data sampling and loss functions are adapted to emphasize pole and conductor (wire) classes. Network topology inference then proceeds in two stages: (i) pole instances are obtained by clustering pole-class points and validating candidates using geometric criteria, including height and verticality estimated via PCA, and (ii) candidate pole pairs are evaluated using a heuristic method and a lightweight ResNet-based classifier on 2D top-view projections of pole and wire point distributions to determine whether a physical conductor span exists. By explicitly classifying candidate spans, the approach mitigates common failure modes of heuristic connectivity rules in dense or cluttered scenes and under partial wire observation. For each validated wire, attributes regarding utility infrastructure geometry are computed, including endpoint conductor heights, ground elevation, sag-related lowest-point features, conductor arrangement, and wire width. Evaluation on multiple real-world aerial LiDAR datasets demonstrates decimeter-level endpoint height accuracy and approximately 9% relative improvement in recall for topology reconstruction compared to heuristic nearest-neighbor baselines, with larger gains in complex layouts.
Read the full abstract

Image and Video Processing

Hierarchical Filter Band Selection for Multispectral Object Classification

Katja Kossira, Jürgen Seiler and André Kaup

Multispectral camera arrays capture image data in various spectral bands, enabling image acquisition beyond human perception. These systems are widely used in medical, agricultural, environmental, and remote sensing applications. However, not all recorded bands are needed for classification tasks, thus reducing them can lower hardware complexity and cost. The conditional filter band selection algorithm addresses this by selecting low-noise, non-redundant bands to minimize the number of filters and cameras. This paper improves the approach by introducing a second classification stage that estimates object material in addition to the object label. This information is merged by a decision-tree based band selection strategy. The proposed method achieves a 28.9% relative reduction in classification error on the SMM50 dataset compared to the state-of-the-art. Moreover, for the same classification accuracy, the required number of cameras is reduced from 7 to 4, demonstrating that the proposed approach improves performance while significantly lowering hardware requirements.
Read the full abstract

Image and Video Processing

LiteSC: Lightweight Semantic Communication for Robust Wireless Telesurgical Video Transmission

Zexin Deng, Zhenhui Yuan, Xiang Ji et al. (+2 more)

Reliable laparoscopic video is essential for telesurgery, yet fixed-rate digital transmission can degrade abruptly under poor channel conditions. We propose LiteSC, a lightweight semantic communication framework combining a frozen pretrained extractor, a compact joint source–channel coding pair, and a receiver-adapted surgical renderer. The extractor produces a latent with approximately 97.9% fewer scalar values than the red–green–blue (RGB) input, while inference requires neither segmentation masks nor temporal reference frames. Using separate CholecSeg8k source videos for training and testing, we evaluate LiteSC over an additive white Gaussian noise (AWGN) channel at symbol signal-to-noise ratios $E_s/N_0$ from 0 to 25 dB. At a channel bandwidth ratio of $\rho=0.0208$, equivalent to 0.0208 complex channel symbols per RGB source value, peak signal-to-noise ratio (PSNR) rises from 28.7 to 31.9 dB, structural similarity index measure (SSIM) from 0.869 to 0.919, and learned perceptual image patch similarity (LPIPS) falls from 0.102 to 0.048. Against H.265 transmission protected by low-density parity-check (LDPC) coding under the same channel-symbol budget, LiteSC achieves higher SSIM and lower LPIPS, although H.265+LDPC attains a higher peak PSNR. These results show graceful laparoscopic frame reconstruction under the evaluated AWGN conditions.
Read the full abstract

All authors Zexin Deng, Zhenhui Yuan, Xiang Ji, Yi Han, Gaofeng Li

Image and Video Processing

Resolution-Flexible Decoding for Hybrid Neural Video Representations

Taiga Hayami, Masaya Takabe and Hiroshi Watanabe

Neural video representations (NVRs) represent videos using neural network parameters and, in hybrid formulations, frame-wise latent embeddings. Although hybrid NVRs can improve reconstruction quality by using content-adaptive latent embeddings, their latent spatial sizes and decoder upsampling schedules are tied to the target frame resolution. For high-resolution videos, this dependency may require large and non-uniform upsampling factors and can affect the parameter allocation between the latent embeddings and the decoder. In this paper, we propose a resolution-flexible decoder framework for hybrid NVRs. The decoder is constructed from uniform \(2\times\) upsampling stages, whose target feature sizes are obtained by tracing the spatial resolution backward from the final output resolution. After each upsampling stage, the feature map is aligned with the target size by minimal padding or cropping when necessary. To support this progressive decoding process, we further use intermediate reconstruction supervision and a reconstruction-difficulty-aware frame sampling strategy based on recent frame-wise losses. The framework preserves the basic representation format of hybrid NVRs and can therefore be applied to different backbones. Experiments on the UVG dataset show that the proposed approach improves reconstruction quality over the corresponding NVR baselines.
Read the full abstract

Saturday 19 September

Image and Video Processing

UQMIA: An Open, Hands-On Tutorial on Uncertainty Quantification in Medical Imaging Analysis with Large Language Model-Based Assessment of Educational Content

Benyamin Gheiji, Danial Elyassirad, Mahsa Vatanparast et al. (+2 more)

Uncertainty quantification (UQ) is increasingly recognized as an important component of reliable machine learning in medical imaging, yet practical resources connecting UQ theory, implementation, and evaluation remain limited. We developed Uncertainty Quantification in Medical Imaging Analysis (UQMIA), an open-access, hands-on tutorial comprising 19 sessions covering major UQ approaches, including variational inference, Monte Carlo dropout, deep ensembles, evidential deep learning, and conformal prediction, together with methods for evaluating uncertainty reliability. The tutorial is designed for researchers and practitioners with experience in medical imaging, progressing from foundational concepts to implementation and evaluation, with notebooks executable through Kaggle. Beyond presenting the tutorial, we introduce a framework using large language models (LLMs) to evaluate whether technical educational resources contain retrievable and usable knowledge. Using 100 four-option multiple-choice questions derived from primary methodological literature, we evaluated 20 instruction-tuned LLMs from six model families with and without retrieved tutorial context. Retrieval of UQMIA improved accuracy in 18 of 20 models, increasing mean accuracy from 0.680 to 0.742 (+0.062; Holm-adjusted P=0.00032), and improved area under the receiver operating characteristic curve (AUROC) in 18 of 20 models, increasing from 0.725 to 0.788 (+0.063; Holm-adjusted P=0.00032). UQMIA provides an accessible resource for learning UQ in medical imaging and demonstrates an LLM-based approach for evaluating technical educational material. The tutorial is available at https://benyamin-gheiji.github.io/Uncertainty-Quantification-Medical-Imaging-Analysis/ and its source code and notebooks at https://github.com/benyamin-gheiji/Uncertainty-Quantification-Medical-Imaging-Analysis/.
Read the full abstract

All authors Benyamin Gheiji, Danial Elyassirad, Mahsa Vatanparast, Meysam Tavakoli, Shahriar Faghani

Image and Video Processing

DA-Lion: Efficient Neural Video Representation via Direction-Aware Optimization

Qingyu Mao, Jiacong Chen, Shuai Liu et al. (+2 more)

Implicit neural video representation (NVR) encodes a video as the parameters of an overfitted neural network, where training and testing data are identical and the objective is instance-specific signal fitting. In this deterministic regime, optimization dynamics directly determine reconstruction quality under a fixed budget, yet existing methods universally adopt general-purpose optimizers designed for stochastic gradient training. Adam-family optimizers descend steadily but converge slowly, while Lion achieves faster early progress but oscillates markedly in later stages when gradient directions become unstable. We trace this instability to the insensitivity of the sign-based update to gradient–momentum alignment. To address this, we propose Direction-Aware Lion (DA-Lion), a task-driven optimizer tailored for NVR. DA-Lion introduces (i) a direction-consistency criterion that switches between sign update and momentum update based on gradient–momentum alignment, and (ii) a learning-rate-aware magnitude modulation that stabilizes effective step sizes across training phases. DA-Lion modifies only the parameter update rule, preserving model architecture, parameter count, FLOPs, and decoding throughput. Experiments on three datasets with four NeRV-family backbones show that DA-Lion improves convergence stability and boosts PSNR/MS-SSIM across all backbones. The source code will be made available at https://github.com/maoqingyu1996/DA-Lion.
Read the full abstract

All authors Qingyu Mao, Jiacong Chen, Shuai Liu, Yongsheng Liang, Youneng Bao

Image and Video Processing

Signal recovery in the polychromatic computed tomography model: Injectivity and complexity

Xuanzhou Chen and Ashwin Pananjady

We consider a nonlinear model motivated by polychromatic computed tomography (CT). Here, $W$ distinct $d$-dimensional signals $x^*_1, \ldots, x^*_W \in \mathbb{R}^d$ must be recovered from $n$ measurements $(a_i, y_i)_{i = 1}^n$ that obey the nonlinear model $\mathbb{E}[y_i|a_i] = h(\langle a_i, x^*_1 \rangle, \ldots, \langle a_i, x^*_W \rangle)$, where $h: \mathbb{R}^W \to \mathbb{R}$ is a known nonlinearity that models a certain type of exponential attenuation law. Even when there is no noise in the measurements, the sample size $n$ (for any measurement ensemble $\{a_i\}_{i = 1}^n$) must exceed the number of unknowns $Wd$ to guarantee that the underlying signals are identifiable. We construct a measurement ensemble that ensures perfect signal recovery almost surely provided $n \geq Wd + W - 1$, thereby isolating the injectivity threshold up to an additive factor $W - 1$. We also study computational complexity of signal recovery in this model with a general measurement ensemble $\{a_i\}$. We show that if $W \geq 2$, then there is a measurement ensemble for which deciding whether there exist signals consistent with the measurements is NP-hard. In particular, this implies that polychromatic, multimaterial CT reconstruction is NP-hard in general. This finding stands in sharp contrast to the single-material setting, for which a polynomial-time algorithm can provably perform signal recovery for any measurement ensemble.
Read the full abstract

Image and Video Processing

Quality Assessment of 3D Gaussian Splatting: Distortions, Benchmarks, and Open Challenges

Shuai Liu, Binqiang Liu, Qingyu Mao et al. (+3 more)

3D Gaussian Splatting (3DGS) has become a practical scene representation for real-time novel-view rendering, compression, and immersive content delivery. However, its quality assessment still largely follows rendered-view proxy protocols that sample camera poses, render images or videos, and apply inherited image and video quality metrics. While convenient, this practice does not fully capture 3DGS-native degradations that originate from Gaussian primitive distributions, splatting and visibility behavior, and trajectory-dependent artifacts. This survey reviews recent 3DGS quality assessment studies from four perspectives, covering distortion characteristics, subjective benchmarks, objective metric reliability, and emerging directions for native 3DGS evaluation. Across the literature, we find that different benchmarks construct different notions of quality, and that metric effectiveness is highly protocol dependent, varying with distortion sources, stimulus formats, and view and trajectory sampling. Finally, we summarize open challenges and outline practical guidelines for building more comparable and representative 3DGS-QA evaluations.
Read the full abstract

All authors Shuai Liu, Binqiang Liu, Qingyu Mao, Jiacong Chen, Yongsheng Liang, Youneng Bao

Image and Video Processing

SomaNet: Weakly Supervised Learning for Instance Soma Segmentation in 3D Electron Microscopy with Partial Annotations

Mohammad Khateri, Morteza Ghahremani, Jussi Tohka et al. (+1 more)

Soma instance segmentation, i.e., identifying and delineating individual cell somas as distinct instances, is crucial for cellular analysis and connectomic reconstruction. Three-dimensional electron microscopy (3D EM) provides nanometer-scale resolution for capturing fine-grained soma morphology. However, dense instance-level manual annotation is prohibitively costly, limiting the scalability of fully supervised methods. To address this challenge, we propose SomaNet, a weakly supervised framework for 3D EM soma instance segmentation under partial annotation constraints. SomaNet adopts a teacher–student learning paradigm tailored to partial labels. The teacher is trained using partially annotated data to generate pseudo-labels, while the student jointly learns from the partial ground-truth annotations and the generated pseudo-labels, progressively recovering dense instance segmentations. To accurately delineate cell somas under limited supervision with varying instance counts, SomaNet incorporates affinity learning, which encourages high similarity within instances and low similarity across instance boundaries. Semantic-guided affinity decoding and 2D-to-3D reconstruction then produce volumetrically consistent 3D soma instances while preserving the large receptive fields of 2D backbones. The framework is architecture-flexible and supports diverse backbones, including vision transformers and foundation models, enabling direct transfer of pretrained visual representations to volumetric EM segmentation. Experiments on 3D EM brain datasets demonstrate that SomaNet achieves accurate and robust soma instance segmentation across regions with diverse soma morphologies under partial annotation. Code is available at https://github.com/mkhateri/SomaNet.
Read the full abstract

All authors Mohammad Khateri, Morteza Ghahremani, Jussi Tohka, Alejandra Sierra

Image and Video Processing

Scalable SSIM Estimation from PSNR for Per-Title and Context-Adaptive Encoding Workflows

Luc Trudeau and Maria G. Martini

Modern streaming pipelines run hundreds of candidate encodes per asset to support per-title encoding, shot-based optimization, and context-adaptive ABR ladder construction. These techniques have moved perceptual quality metrics into the critical path: SSIM and VMAF now guide encoding decisions rather than passively monitor them. We measure that SSIM evaluation accounts for 7-35% of x264 encode time at production speed presets, with the cost ratio rising as encoders run faster. We propose ApproxSSIMate, a low-complexity method for estimating SSIM from PSNR combined with reference-sequence statistics computed once per sequence and reused across every candidate encode. This decouples quality estimation from the encode-decode-compare loop, enabling perceptual quality feedback in live encoding and amortizing quality measurement across candidate encodes in per-title workflows. We validate the approach across H.264/AVC, H.265/HEVC, and AV1 on the Objective-1-fast dataset and release the implementation as free and open-source software.
Read the full abstract

Image and Video Processing

A Controlled Evaluation of Quantum Correlation Refinement for Few-Shot Semantic Segmentation: Resource Cost and IBM Quantum Hardware Validation

Hina Shakir, Muhammad Irfan Memon, Asia Samreen et al. (+4 more)

PQCs are increasingly proposed as trainable components in classical ML pipelines, but rarely characterized alongside a controlled measurement of task-level benefit. We report such an evaluation, using few shot segmentation as testbed. We integrate a Quantum Correlation Refiner (QCR), a six-qubit variational circuit with amplitude embedding and a gated residual connection into a classical correlation-based architecture, under the four fold PASCAL 5i protocol, three seeds per fold, against an unrefined baseline and a matched MLP refiner. Across 12 fold-seed comparisons, QCR changes mIoU by only +0.0001 (t(11)=0.10, p=0.92, dz=0.03); the MLP control and an expanded Pauli-measurement variant show similarly no improvement. QCR adds just 473 parameters but roughly doubles per epoch training time. A trained circuit on IBM hardware (18 patches) agrees closely with the noiseless simulator (r=0.954, MAE=0.101), a fidelity check, not an accuracy gain. The module is functional, trainable, and hardware deployable, yet shows no task-level advantage over a matched classical alternative under this regime, a template for evaluating quantum components jointly via matched controls, repeated seeds, resource measurement, and hardware validation, rather than any one alone.
Read the full abstract

All authors Hina Shakir, Muhammad Irfan Memon, Asia Samreen, Javeed Hussain, Syed Rizwan Ali, Mohammad Mohatram, Muhammad Hussain

Friday 18 September

Image and Video Processing

AWR-Net: Decoupling Anatomy and Appearance for 3D Fetal Brain Ultrasound Synthesis

Yuhuan Lu, Sergio Valencia, Yuanji Zhang et al. (+4 more)

Three-dimensional fetal brain ultrasound offers non-ionizing, cost-effective imaging with rich spatial information for comprehensive anatomical assessment, yet its development remains limited by scarce data and annotations. In contrast, fetal brain magnetic resonance imaging has advanced further, supported by larger datasets and mature analysis methods. To leverage these resources, anatomical label maps provide a promising modality-invariant bridge for transferring knowledge from fetal brain magnetic resonance imaging to ultrasound. However, generating ultrasound images from label maps remains challenging because anatomy is tightly entangled with ultrasound appearance. To address this challenge, we separate anatomical correspondence learning from ultrasound appearance adaptation at both the data and model levels. Specifically, we propose the anatomy wavelet residual network, a two-stage framework combining wavelet diffusion and residual refinement. The first stage learns to generate volumes from label maps using atlas pairs in the wavelet domain, while the second stage learns bounded residual corrections from real clinical ultrasound in the image domain. This separation enables realistic synthesis with consistent preservation of normal and abnormal anatomy. Experiments on real fetal brain ultrasound show that our method outperforms representative synthesis methods, with normalized cross correlation of 0.518 versus 0.482 and Fréchet Inception Distance of 8.905 versus 13.319 for the strongest baseline. Beyond synthesis quality, volumes generated from fetal brain magnetic resonance imaging label maps also improve downstream segmentation, particularly for severe abnormal cases. Overall, these results highlight the potential of the proposed framework to leverage rich fetal brain magnetic resonance imaging resources for advancing three-dimensional ultrasound analysis.
Read the full abstract

All authors Yuhuan Lu, Sergio Valencia, Yuanji Zhang, Yuhao Huang, Camilo Jaimes, P. Ellen Grant, Davood Karimi

Image and Video Processing · Cross-listed from cs.RO

Tracker-Free Robotic Ultrasound Calibration with a Spherical-Marker Phantom and Threshold-Free Center Localization

Kyoungmo Koo, Guangshen Ma, Xueding Wang et al. (+1 more)

Robotic ultrasound (US) calibration is essential for accurately relating US images to the robot coordinate system, but accurate and automated calibration remains challenging because existing methods often require complex phantoms or external 3D trackers. In this work, we develop a tracker-free robotic US calibration framework using a spherical-marker phantom and a highly automated perception pipeline for sphere-center localization. The proposed threshold-free image-processing method localizes the spherical feature based on each US image's intensity distribution, eliminating hand-tuned intensity thresholds and improving robustness across imaging settings without per-system retuning. Multi-pose observations of the spherical fiducial are then used to estimate the US-to-EE transformation without external tracking or prior localization of the sphere center in the robot base frame. We validate the proposed framework on a robotic US platform through repeated sphere scans and further assess the calibrated system using geometrically distinct phantoms with known CAD models. Across 12 cross-validation folds, single-marker calibration achieved sphere-center accuracy and precision of $1.75\pm0.51$ mm and $0.85\pm0.11$ mm, respectively, compared with $1.72\pm0.51$ mm and $0.84\pm0.12$ mm for three-marker calibration. Across the three reconstruction sets, the single-marker calibration yielded pooled post-registration point-to-surface MAE$\pm$SD values of $0.41\pm0.35$ mm for the cone and $0.50\pm0.41$ mm for the triangular prism, closely matching the three-marker results of $0.40\pm0.35$ mm and $0.48\pm0.40$ mm, respectively.
Read the full abstract

All authors Kyoungmo Koo, Guangshen Ma, Xueding Wang, Mark Draelos

Image and Video Processing

AutoRASOR: Autonomous Rapid Scanning Electron Microscope Operator

Kevin Zhang, Mohammad Taha, Rafael Espinosa Castañeda et al. (+7 more)

Scanning Electron Microscopy (SEM) is a foundational technique for characterizing material microstructure, which dictates many fundamental physical and chemical properties. With the rise of Self-Driving Labs (SDLs), samples are now synthesized in large batches, demanding equally high-throughput, autonomous characterization. Existing automated electron microscopy pipelines are task-specific: they detect pre-defined features, optimize known properties, or require prior knowledge of the sample, restricting each pipeline to the material system it was built for. We introduce AutoRASOR, a task-agnostic autonomous SEM pipeline that captures the multi-scale morphology of an unknown sample without domain-specific pre-training, fine-tuning, or human prompting. AutoRASOR embeds micrographs in real time with a vision foundation model (VFM), DINOv3, and selects regions of interest (ROIs) through two complementary policies: Latent Farthest Point Sampling (LFPS) and active learning on morphological ambiguity, which is the conditional variance of unresolved fine-scale features given lower magnification appearance. Active learning on ambiguity consistently captures more diverse and rare morphologies than random ROI selection, while LFPS reliably recovers the specimen's morphological distribution within a limited capture budget, tested both on real SEM micrographs and synthetic phase-field images. By shifting autonomous characterization from pre-defined, task-specific targeting of features to morphological survey, AutoRASOR generates information-rich multi-scale datasets for diverse downstream tasks. Without the need for domain-specific pre-training or prior material assumptions, this framework can be deployed out-of-the-box to characterize novel materials produced by SDLs.
Read the full abstract

All authors Kevin Zhang, Mohammad Taha, Rafael Espinosa Castañeda, Yutong Liu, Yin Zhu, Lotan Portal, Robert A. Mcleod, Vahid Attari, Jane Y. Howe, Jason Hattrick-Simpers

Image and Video Processing

Multicentre Bi-atrial Segmentation from LGE-MRI for Atrial Fibrillation with a 2D and 3D Framework

Malitha Gunawardhana, Gregory B. Sands, Mark L. Trew et al. (+1 more)

Accurate delineation of bi-atrial structures from late gadolinium enhancement MRI (LGE-MRI) is an important prerequisite for structural analysis and future fibrosis-quantification workflows in atrial fibrillation (AF). However, automated segmentation is challenging due to thin-walled anatomy, domain shifts across imaging centres, and limited benchmarking of existing methods. This study presents a two-stage segmentation framework and benchmarking platform for evaluating how ROI localisation, encoder design, 2D/3D dimensionality, and ensemble fusion affect bi-atrial wall and cavity segmentation across multicentre LGE-MRI datasets. The framework integrates 3D localisation and fine segmentation using 2D and 3D U-Net variants with ResNeXt encoders and compares them with convolutional, transformer-based, and state-space architectures. { Evaluation across three independent cohorts assessed accuracy and cross-domain transfer without target-domain fine-tuning. Cavity segmentation transferred more consistently across centres than atrial wall segmentation, while wall performance remained sensitive to domain shift, particularly in the Kobe cohort.} By quantifying how 2D, 3D, and ensemble architectures behave across centres and between walls and cavities, this work provides a reproducible benchmark for future methodological development and clinical validation.
Read the full abstract

All authors Malitha Gunawardhana, Gregory B. Sands, Mark L. Trew, Jichao Zhao

Image and Video Processing

Lesion-Gated Hybrid Synthesis for Virtual Contrast-Enhanced Breast MRI: A MAMA-SYNTH Challenge Solution

Shohei Yoshimoto

Purpose: Contrast-enhanced breast MRI depends on intravenous gadolinium-based contrast agents, motivating methods that synthesise post-contrast appearance from pre-contrast images alone. The MAMA-SYNTH Challenge (MICCAI 2026 Deep-Breath Workshop) evaluates such synthesis across four metric groups: image fidelity, tumour region of interest, downstream classification, and downstream segmentation. The four group ranks are averaged, so optimising a single objective is insufficient. Materials and Methods: We developed a lesion-gated hybrid synthesis pipeline. A lesion probability map, estimated from the pre-contrast slice alone by an ensemble of pre-contrast-only segmentation networks, spatially coordinates a tumour-focused regression pathway and a background-focused Pix2PixHD synthesis pathway, followed by a region-dependent calibration of the predicted enhancement. Training used the public MAMA-MIA collection with a patient-level split. Inference consumes a single pre-contrast 2D slice, with no mask, no post-contrast image, and no auxiliary metadata beyond image geometry. Results: On internal validation (n = 120 patients) the method reached MSE 0.523, LPIPS 0.185, and tumour-region SSIM 0.495. It was submitted to the hidden external 300-case test cohort as a self-contained inference container. The method ranked sixth in the official MAMA-SYNTH Challenge leaderboard. Conclusion: A test-compatible lesion probability map derived from the pre-contrast image can coordinate complementary tumour-focused regression and background-focused perceptual synthesis, enabling balanced virtual contrast enhancement under a multi-metric challenge setting.
Read the full abstract

Image and Video Processing

Anatomically Faithful Artifact Suppression in SENSE Accelerated Brain MRI

Changjing Chai, Bin Huang, Libo Xu et al. (+5 more)

Background: Four-fold accelerated sensitivity encoding (SENSE4) can shorten brain MRI acquisition time but may amplify noise and result in residual aliasing artifacts after conventional reconstruction. Purpose: To evaluate whether an image-domain refinement framework can improve the quality of SENSE4 brain MRI while preserving anatomical information for quantitative measurements. Methods: In this prospective paired study, 80 participants underwent fully sampled and four-fold accelerated SENSE T1-weighted MRI. We developed an Anatomy-aware Residual Attention Network (ART-Net) to refine accelerated reconstructions through generalized self-attention and correlation-based residual artifact regularization. Participant-level splitting yielded training, validation, and independent test cohort (45/5/30 participants). The independent test cohort underwent quantitative, segmentation-based, and blinded radiologist assessments of image quality and anatomical preservation. Results: ART-Net demonstrated highly competitive reconstruction performance, achieving the highest peak signal-to-noise ratio (31.03 +/- 2.88 dB) and structural similarity index (0.963 +/- 0.022) among evaluated methods. It also demonstrated improved anatomical fidelity, with numerically highest Dice coefficients for medial temporal structures relevant to atrophy assessment (0.8824 +/- 0.0827) and whole-brain regions (0.8857 +/- 0.0885). Moreover, ART-Net improved gradient fidelity, regional contrast preservation, and radiologist-rated structural quality. Conclusion: ART-Net improved agreement between SENSE4 and fully sampled T1-weighted images in a single-center, held-out test cohort while maintaining segmentation-derived anatomical measurements. These findings suggest that ART-Net may support accelerated brain MRI by improving image fidelity and enabling reliable downstream anatomical analysis.
Read the full abstract

All authors Changjing Chai, Bin Huang, Libo Xu, Jian Zhou, Boyang Pan, Kristen W Yeom, Qiyong Gong, Nan-Jie Gong

Image and Video Processing

Model-Based Iterative Reconstruction with View-Dependent Detector Displacements for Cone-Beam CT

Doga Topcicek, Sharmin Majumder, Whitney A. Poling et al. (+4 more)

Detector shifting is used in industrial cone-beam computed tomography to reduce ring artifacts, but the resulting detector motion can vary nonuniformly across projection views and produce detector positions that cannot be represented by fixed offsets. Although analytical reconstruction can accommodate this motion, approaches that use projection interpolation or rebinning may limit spatial resolution and introduce artifacts. MBIR provides an alternative by representing the acquisition geometry explicitly within the reconstruction model. Existing separable distance-driven MBIR formulations assume fixed detector geometry across projection views and therefore cannot directly represent nonuniform motion in both detector dimensions, as this motion changes the voxel-to-detector mapping. We extend the separable distance-driven MBIR model to incorporate recorded horizontal and vertical detector displacements into the distance-driven overlap computation while leaving measured projections unchanged. Forward-projection validation achieved an NRMSE of 0.0338 across 1080 views. The proposed model achieved a 3D SSIM of 0.9404 and NRMSE of 0.0406, compared with 0.5396 and 0.1740 for fixed-detector MBIR. A second industrial dataset showed reduced structured artifacts compared with FDK.
Read the full abstract

All authors Doga Topcicek, Sharmin Majumder, Whitney A. Poling, Dmitriy D. Bruder, Zhili Feng, Dali Wang, Xiao Wang

Image and Video Processing

Opportunistic Conditional Entropy Coding with Frozen Analysis and Synthesis Transforms

Vincent Corlay, Maxime Rousselot and Andriy Enttsel

In many delivery settings, a receiver may already hold a lower-quality or lower-resolution representation of an image, obtained through an independent transmission. Conventional codecs encode a subsequently requested higher-quality representation without exploiting this incidental side information, whereas conditional codecs generally assume a prescribed source of side information that is always available. We instead consider an opportunistic setting in which side information may or may not be present. We introduce a single entropy model that conditions on a previously decoded latent when available and falls back to a standard hyperprior otherwise. The proposed adapter maps the side-information latent to the prior signal required by the entropy model, allowing the same model to support multiple target and side-information quality combinations. The analysis and synthesis transforms remain frozen, enabling retrofitting of an existing learned codec while preserving its latent representation and reconstruction path. When the receiver holds the quality immediately below the target, the proposed method reduces the rate of the subsequent transmission by up to 46%, or by 52% when an additional hyper-latent is transmitted. In the absence of side information, the rate penalty remains below 4%, and the reconstructions are bit-identical across the conditional and fallback modes.
Read the full abstract

Image and Video Processing · Cross-listed from eess.AS

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Haolin He, Yunfei Chu, Qi Chen et al. (+15 more)

We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, good replies often depend on multimodal context and can be phrased in many ways, making keyword matching unreliable for evaluation. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. Replies are judged by a large language model based on explicit scoring criteria. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Read the full abstract

All authors Haolin He, Yunfei Chu, Qi Chen, Wen Huang, Yuan Feng, Muzhi Zhu, Zheqi Dai, Haoning Xu, Dongchao Yang, Chunyat Wu, Zining Liang, Zhengxi Liu, Xiquan Li, Xie Chen, Xize Cheng, Qize Yang, Jin Xu, Qiuqiang Kong

Image and Video Processing

WS-NeRF: A Mamba-Driven World-State-Aware Adaptive Deblurring Neural Radiance Field

Hang Jiang, Jinghao Wang, Yiming Zhang et al. (+3 more)

Neural Radiance Fields (NeRF) have attracted extensive attention in recent years due to their strong capability for high-quality 3D reconstruction and novel view synthesis from multi-view images. Existing methods usually rely on high-quality sharp inputs, while real-world image acquisition is highly susceptible to blur degradation, which severely affects the reconstruction quality of NeRF. In this paper, we propose a novel Mamba-driven world-state-aware adaptive deblurring neural radiance field, termed WS-NeRF, to address image degradation and 3D inconsistency. We formulate the alternating optimization of radiance fields as a dynamic evolution process with temporal memory, and jointly exploit comprehensive multi-dimensional world states and a mixture-of-experts mechanism to dynamically adjust the confidence of deblurring priors. Experimental results show that WS-NeRF significantly improves blurry radiance field reconstruction quality, achieving better performance on PSNR, SSIM, and LPIPS, while exhibiting more stable iterative recovery behavior.
Read the full abstract

All authors Hang Jiang, Jinghao Wang, Yiming Zhang, Xinhong Wang, Luwei Ran, Yinfeng Yu

Image and Video Processing

Adaptive Color Grading

Trevor D. Canham, Abhijith Punnappurath and Michael S. Brown

Independent control of tonescale regions (e.g., shadows, highlights) is essential for painters, photographers and cinematographers to bring 2D images to life. In image manipulation software this is most directly addressed by color grading modules, which use intensity thresholds to segment distinct illumination regions for local manipulation. In this work we develop an open source color grading tool and use it to annotate a large dataset of video frames with tonescale region thresholds. Using these thresholds we conduct modeling experiments with strategies based on both practitioners' conventional wisdom and machine learning. Results show that K-nearest neighbors is an effective prediction strategy, outperforming state-of-the-art end-to-end methods for image enhancement. This outcome demonstrates the benefit of focusing on a compact set of core parameters when modeling creative stylization processes. Our adaptive color grading interface and data are available at https://github.com/SamsungLabs/adaptive-color-grading.
Read the full abstract

Thursday 17 September

Image and Video Processing · Cross-listed from cs.LG

Uncertainty-Aware RL-Controlled Adaptive 3D Mapping

Alpay Ozkan, Tunc Ozan Aydin, Marc Pollefeys et al. (+2 more)

Voxel-based volumetric mapping is fundamental to 3D reconstruction, yet fixed-resolution grids remain inherently inefficient - wasting memory in uniform regions and losing detail in complex ones. Existing adaptive methods, such as MAP-ADAPT, partially address this by varying resolution based on geometry and user-defined semantic class lists, but these heuristics require expert tuning, lack generalization to unseen objects, and provide no explicit mechanism to control memory usage. We propose an adaptive framework that refines voxels based on semantic entropy, which captures label uncertainty, together with geometric curvature and texture richness as scene complexity cues, yielding principled resolution allocation without reliance on semantic taxonomies. To make the accuracy-memory trade-off explicit and user-controlled, we further introduce a reinforcement learning agent that learns voxel subdivision policies under a user-specified target memory budget, replacing hand-tuned thresholds with a single intuitive control parameter. The resulting multi-resolution TSDF achieves higher geometric accuracy, better semantic consistency, and improved memory-accuracy trade-offs compared to MAP-ADAPT and fixed-resolution baselines on both synthetic and real-world datasets. Our code and models are available at https://github.com/alpayozkan/UnRL.
Read the full abstract

All authors Alpay Ozkan, Tunc Ozan Aydin, Marc Pollefeys, Jelena Trisovic, Daniel Barath

Image and Video Processing

Forecasting Brain Tracer Distribution from Pre-Contrast MRI: A Comparison of Direct Regression and Flow Matching

Qinghui Liu, Jon André Ottesen, Thu Nguyen et al. (+6 more)

Intrathecal contrast-enhanced MRI shows how a tracer injected into the cerebrospinal fluid spreads through the brain, but requires repeated scans over 24–48\,h. We therefore ask how much of the later tracer distribution can be predicted from the pre-contrast scan and elapsed time alone. Using 104 training patients (878 MRI volumes), we compared a direct image-to-image regression model with a more complex generative model based on conditional flow matching, and tested both on 23 held-out patients and on 51 patients from a separate cohort with idiopathic normal pressure hydrocephalus (INPH). The direct model reduced the error of simply copying the pre-contrast scan by about 60% in held-out patients and 25% in the INPH cohort, and its predicted change maps agreed with the observed tracer enhancement rather than only reproducing anatomy. The generative model was less accurate in both cohorts. A substantial part of the later tracer distribution is therefore predictable from the patient's brain anatomy as seen on pre-contrast MRI, and the added model complexity gave no benefit at this data scale. With prospective validation, such forecasts could be studied for patient stratification and treatment planning in intrathecal therapies.
Read the full abstract

All authors Qinghui Liu, Jon André Ottesen, Thu Nguyen, Geir Ringstad, Ingrid Mossige, Siri Fløgstad Svensson, Atle Bjørnerud, Per Kristian Eide, Kyrre Eeg Emblem

Image and Video Processing

Recasting the Destroy Step of Large Neighborhood Search as Dense Segmentation

Yang Liu, Yulin Huang, Jianshen Zhang et al. (+1 more)

Large neighborhood search improves an incumbent by releasing selected variables and repairing the resulting subproblem under a time limit. FOVEA casts variable selection as dense semantic segmentation on a fixed $128\times128$ canvas. A small U-Net reads twelve semantic channels and predicts cell scores, which are decoded into a variable set subject to a fixed variable budget. Family-specific layouts connect the search state to the canvas, while one set of network weights serves four problem families. The FP32 network input occupies $786$\,kB across instance sizes. Our analysis bounds constraint coupling for spatially coherent regions, gives a rank certificate for rounds with zero possible improvement, and bounds the coupling advantage attainable on an expander family. On the tested families and hardware, FOVEA trails the strongest graph encoder at $10^{4}$ variables and overtakes it near $1.5\times10^{4}$; the cost model provides an approximate crossover estimate. At larger sizes, where the graph-encoder baselines exceed device memory, FOVEA reduces the primal integral by $12$ to $16\%$ relative to the best runnable baselines. On the expander family, FOVEA performs comparably to random selection, with a coupling advantage close to one.
Read the full abstract

All authors Yang Liu, Yulin Huang, Jianshen Zhang, Yongzhi Qi

Image and Video Processing · Cross-listed from physics.optics

RTLViT: real-time lensless reconstruction with a lightweight vision transformer

Leyla A. Kabuli, Vasilisa Ponomarenko and Laura Waller

Mask-based lensless imagers capture measurements using simple, compact hardware and recover images using a reconstruction algorithm. The reconstruction algorithm affects the image quality, inference speed, and computational requirements of the imaging system, often trading off image quality against computational efficiency. Advancing toward integrated lensless imagers, where encoding and reconstruction occur within a single device (e.g., a mobile phone), requires a fast, high-quality, and practical reconstruction method. We introduce the Real-Time Lensless Vision Transformer (RTLViT), a lightweight, purely data-driven reconstruction architecture for high-quality and real-time lensless reconstruction. With only 1.09 million learnable parameters, RTLViT provides higher-quality reconstructions than both real-time baselines and substantially larger attention-based architectures, including improvements of up to 3.46 dB in peak signal-to-noise ratio over architectures with $15 \times$ more learnable parameters. We demonstrate that RTLViT maintains high-quality reconstructions across a wide range of training dataset sizes and two different mask designs (lenslets and a diffuser). Finally, we implement real-time reconstruction with RTLViT on laptops and smartphones, supporting future integrated lensless imagers for real-time applications.
Read the full abstract