Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Electrical Engineering and Systems Science Image and Video Processing

eess.IV Watch Image and Video Processing

Sunday 4 October

Image and Video Processing

TIRMamba: A Thermal-Prior-Modulated State-Space Network for Sub-Million-Parameter Infrared Image Super-Resolution

Chun-An Lin, Tsung-Jung Liu and Yen-Chieh Ouyang

Infrared image super-resolution is currently led by Mamba-based networks with 26 to 37 million parameters, which are difficult to deploy on the airborne and handheld platforms where thermal imaging is most needed. This paper presents TIRMamba, a network with 896K to 910K parameters for single-channel thermal imagery. A Thermal Prior Highway computes gradient, local-contrast and spectral cues once at the input and, through one adapter per residual group, modulates a weight-tied bidirectional state-space trunk and gates its dual-scale detail branch; a tri-path reconstruction adds the learned residual to a bicubic radiometric baseline. Because the standard benchmark provides only 265 infrared training images and evaluates fusion products on full images, we train with a replay strategy: grayscale DIV2K pre-training followed by fine-tuning on 64-pixel patches drawn with equal probability from the infrared and natural corpora. At scale factor 4, TIRMamba matches the strongest protocol-trained methods on both official test sets with 29 to 40 times fewer parameters and 2.8 to 9.4 times lower latency; at scale factor 2 it gives the highest SSIM on both. A variant with prior-conditioned selectivity, TIRMamba-Rad, corrects a 3 dB raw-thermal failure of an intermediate size-invariant design and gives the best results at scale factor 4 on raw-thermal, unmanned-aerial-vehicle and independent-sensor test sets. Code and trained models will be released at https://github.com/julian135707/TIRMamba upon acceptance.
Read the full abstract

Saturday 3 October

Image and Video Processing

FloVMos: Optical Flow-based Medical Video Mosaicking

Jinyang Liu, Sandesh Ghimire, Chaman Singh et al. (+5 more)

Biomedical imaging modalities often require a trade-off among resolution, field of view (FOV), and acquisition speed. Video mosaicking offers a strategy to overcome this limitation by computationally stitching sequential high-resolution frames into a wide-FOV composite. However, existing methods struggle with non-rigid deformations, and modality-specific artifacts arising in clinical and research imaging. Here, we present FloVMos, a generalizable, optical-flow-based deep learning framework for real-time video mosaicking across diverse biomedical imaging modalities. FloVMos achieves robust, pixel-level registration by fine-tuning an optical flow model on synthetic training data with ground-truth deformation fields. We introduce a pipeline for generating this training data, simulating realistic tissue motion and imaging distortions from existing mosaics or raw videos. Our automated synthetic data generation and optical flow model training based on this data allow users to adapt FloVMos to different imaging modalities. To demonstrate this, we applied FloVMos to seven diverse imaging modalities: reflection confocal microscopy, open-top light-sheet microscopy, fetoscopy, laparoscopy, dermoscopy, sparse spectral microscopy, and endoscopy. FloVMos outperforms conventional baselines in accuracy, robustness, and speed for all the tested modalities. This adaptable and training-efficient framework enables large-area visualization with real-time performance and may support broader use of video-based biomedical imaging in research and clinical workflows.
Read the full abstract

All authors Jinyang Liu, Sandesh Ghimire, Chaman Singh, Jennifer Dy, Milind Rajadhyaksha, Dana H. Brooks, Octavia Camps, Kivanc Kose

Friday 2 October

Image and Video Processing

Abdominal Ultrasound Simulation from Semantic Labels using Paired Label-to-Physics-Based Image Translation

Santiago Vitale, Duilio Deangeli, Ignacio Larrabide et al. (+1 more)

Purpose: Current abdominal ultrasound (US) simulation methods often require CT-based anatomical references for ray-casting, limiting deformation and pathology variability. We propose a learning-based pipeline trained to predict physics-based images derived from CT scans from semantic labels, enabling controlled simulations without patient-specific CT volumes at inference time. Methods: We introduce a two-stage pipeline that maps anatomical segmentations to realistic US images through a simplified US image. Stage~I synthesizes this image from semantic labels using models trained on CT-based ray-casting outputs. Stage~II refines it into a realistic US scan using anatomically guided unpaired translation. Deformations and pathologies are generated by editing anatomical maps. Results: We evaluated Pix2Pix and the Semantic Diffusion Model (SDM) in Stage~I, followed by segmentation-guided CycleGAN (SG-CycleGAN) refinement in Stage~II. SDM significantly outperformed Pix2Pix in morphological metrics, including MAE (19.85 vs. 21.65), SSIM (0.28 vs. 0.24), and mIoU (0.43 vs. 0.29), whereas Pix2Pix yielded better perceptual point estimates (LPIPS: 0.17 vs. 0.19; FID: 0.32 vs. 0.37; KID: 0.25 vs. 0.48). Conclusion: Training paired generative models with physics-based supervision enables approximation of CT-derived ray-casting outputs at inference time directly from semantic labels. Although the pipeline does not require patient-specific CT volumes at inference time, CT-derived segmentations and ray-casting simulations remain necessary to train Stage~I. Once trained, the framework enables controllable healthy and pathological simulations through semantic-map modification.
Read the full abstract

All authors Santiago Vitale, Duilio Deangeli, Ignacio Larrabide, José Ignacio Orlando

Image and Video Processing

STRIDE: Spatial-Temporal Representation for Interval-conditioned Disease Evolution in Longitudinal Glioblastoma MRI

Wenhao Guo, Changchang Yin, Pierre Giglio et al. (+3 more)

Glioblastoma (GBM), an aggressive primary brain tumor, is routinely monitored with longitudinal MRI after treatment. Distinguishing stable disease (SD), pseudoprogression (PsP), and true progression (TP) remains challenging because these states can show overlapping MRI appearances despite different temporal trajectories. Existing longitudinal methods still face challenges in modeling scan-specific spatial variability, variable follow-up intervals, and complementary information from the observed follow-up state and its longitudinal change. We propose STRIDE, a framework for spatial-temporal representation of interval-conditioned disease evolution that takes paired post-treatment MRI scans and their inter-scan interval as input and predicts SD, PsP, or TP. The lesion-prior-guided spatial representation combines SoftGate and an Adaptive-window Hierarchical Transformer (AWHT) to emphasize lesion-related regions while preserving surrounding context. The time-conditioned latent transition uses pair-level context and the actual inter-scan interval to estimate interval-dependent representation changes between visits. The observed–transition fusion integrates the transition-estimated follow-up representation with the directly observed follow-up representation to jointly characterize the follow-up state and its longitudinal change. BraTS2024 is used to develop and evaluate the lesion-prior generator, while longitudinal pretraining on LUMIERE supports transfer before downstream adaptation to Burdenko. On the Burdenko three-class task, STRIDE achieves a macro ROC–AUC of 0.816 and a macro F1-score of 0.796. These results support its potential for more reliable longitudinal post-treatment GBM state assessment.
Read the full abstract

All authors Wenhao Guo, Changchang Yin, Pierre Giglio, Weidan Cao, Ping Zhang, Golrokh Mirzaei

Image and Video Processing · Cross-listed from cs.LG

What Should World Models Forget? Stratified Retention for Continual Adaptation

Nishit Anand, Ramani Duraiswami and Dinesh Manocha

Continual learning treats degradation on previously seen data as evidence of failure, a convention inherited from settings with a stationary prediction target, where a correct label remains correct indefinitely. World models do not satisfy this condition. Their prediction target is the environment, which changes, so knowledge that was accurate when acquired may later become false, and discarding it is required behavior rather than a defect. Non-stationary ground truth is well studied in the concept drift literature and in the temporal factuality of language models, but has not been formulated for world models, which are distinctive in that they also encode knowledge that must never be revised. We argue that continual world models require retention stratified by invariance timescale, separating invariants such as physics and object permanence, which must never be revised, from instance-level facts that should be revised as soon as the environment changes. Standard forgetting metrics cannot distinguish a world model that has correctly revised outdated knowledge from one that has suffered catastrophic forgetting, and consequently rank a frozen model highest, while existing physical-reasoning benchmarks evaluate only frozen checkpoints. We propose differential retention, which reports invariant regression testing across the adaptation stream jointly with revision latency, without aggregation.
Read the full abstract

Image and Video Processing

Wrong Organ, Right Physics: Transferring Echocardiography Pretraining to Lung Ultrasound for Tuberculosis Screening

Christiaan M. Geldenhuys, Joshua M. Jansen van Vüren, Véronique Suttels et al. (+6 more)

Lung ultrasound (LUS) is attractive for tuberculosis (TB) screening at primary-care level, but labelled cohorts are small. Echocardiography carries no such constraint, while sharing the same underlying ultrasound imaging physics, signal processing and B-mode appearance as LUS. We ask whether an encoder pretrained on that high-resource ultrasound domain carries representations that remain usable in the low-resource one. Only the encoder varies, across seventeen encoders spanning three architecture families. Among them, a latent-predictive video encoder pretrained on generic video (V-JEPA2-L) and its echocardiography counterpart (EchoJEPA-L) differ in pretraining corpus alone. The choice among these encoders does not resolve the classification, the whole family spanning 2.50 percentage points against a measurement resolution of 2.71. What moves the task instead is feature conditioning. Standardising the features between the encoder and the classifier improves all seventeen encoders by a mean of +1.23 percentage points at $p=1.5\times10^{-5}$. On the held-out test set every encoder selected on the development folds stands above the baseline system by up to +2.57 percentage points of area under the receiver operating characteristic curve (AUROC), and specificity at 90% sensitivity reaches 79.3% against 60.3%. The contrast specified in advance, EchoJEPA-L against V-JEPA2-L, measures -0.16 percentage points at $p=0.926$. We therefore find no evidence that shared ultrasonic physics alone makes echocardiography a more productive pretraining corpus than generic video, and any advantage, if present, is smaller than this cohort can resolve. The video encoders receive replicated still images, however, so whether this absence of an effect reflects the pretraining domain or a video encoder applied to static frames cannot be separated. The limiting factor is the labelled cohort rather than the encoder.
Read the full abstract

All authors Christiaan M. Geldenhuys, Joshua M. Jansen van Vüren, Véronique Suttels, Trevor Brokowski, Ablo P. Wachinou, Mary-Anne Hartley, Rensu P. Theart, Grant Theron, Thomas R. Niesler

Image and Video Processing · Cross-listed from cs.CV

FUSEye: Training-Light Fisheye Detection with Overlapping Views and Zero-Initialized Adapters

Wenya Su, Kai Luo, Di Wen et al. (+5 more)

Fisheye cameras give mobile robots a single-sensor, low-cost view of their surroundings, yet the COCO-pretrained detectors that practitioners routinely reuse fail on them: strong radial distortion warps local image structure, while boundary compression shrinks objects to near-invisible sizes. Full fine-tuning closes much of the gap but requires abundant fisheye labels and compute. We present FUSEye, a training-light framework that turns a frozen-backbone COCO-pretrained extra-large YOLO26 detector (YOLO26-x) into a fisheye detector. FUSEye adds roughly 227k new parameters while updating the inserted modules and the pretrained detection head. It addresses the transfer gap at three causally linked levels. At the input level, overlapping grid view generation and box remapping (GridViews) enlarge compressed boundary regions. At the feature level, zero-initialized residual adapters (Z-Adapters) correct distortion-induced feature misalignment. At the decision level, learned cross-projection agreement fusion (AgreeFusion) promotes low-confidence detections only when they are supported by consistent evidence across multiple views. On the WoodScape surround-view fisheye benchmark, FUSEye raises YOLO26-x from 0.148 to 0.266 mAP50 and retains 84.3% fully fine-tuned accuracy. Moreover, randomly using only 25% of the labeled training images, FUSEye achieves 0.2597 mAP50, retaining 97.6% of its full-label performance. FUSEye also consistently improves YOLOv8-11 detectors, showing that the recipe is architecture-agnostic. Source code will be available at https://github.com/Su-wenya/FUSEye.
Read the full abstract

All authors Wenya Su, Kai Luo, Di Wen, Ruiping Liu, Yufan Chen, Junwei Zheng, Kunyu Peng, Kailun Yang

Image and Video Processing

One Photon, Many Worlds: Posteriors and Predictions with Single-Photon Cameras

Haejoon Lee, Mohit Gupta, Vijayakumar Bhagavatula et al. (+1 more)

Single-photon avalanche diode (SPAD) cameras operate fundamentally differently from conventional cameras due to their photon-counting nature. Each frame produces a binary image: pixels report zero if no photons arrived during exposure, and one if one or more photons arrived. Reconstructing a scene or inferring its properties from a single binary frame is difficult because many different images could produce the same measurement; thus, the inverse problem is fundamentally one-to-many. As we gather more binary measurements, the inherent uncertainty associated with the inverse problem and any associated inference diminishes. With sufficient photon counts, photon noise becomes negligible relative to the signal mean, enabling near-deterministic scene recovery and inference. This work characterizes the transition from stochastic to near-deterministic scene understanding as photon budget increases, analyzing how the stochasticity in photon arrival affects downstream inference tasks. Technically, we develop a conditional generative framework based on a Hypergeometric frame-thinning process for accumulated binary SPAD measurements. Generative models capture the one-to-many nature of photon-starved inverse problems, enabling empirical characterization of how this ambiguity diminishes with increasing measurements and its impact on downstream tasks like character recognition, QR code decoding, and facial analysis.
Read the full abstract

All authors Haejoon Lee, Mohit Gupta, Vijayakumar Bhagavatula, Aswin C. Sankaranarayanan

Thursday 1 October

Image and Video Processing

Image-Based Breast Implant Detection for Mammography Dataset Curation and Near-Real-Time Deployment: Comparing Foundation Models and Task-Specific Convolutional Models

Vasisht Ishwar, Hari Trivedi, Young Seok Jeon et al. (+5 more)

Purpose: To evaluate the performance-feasibility tradeoffs of foundation models (FMs) and task-specific convolutional neural networks (CNNs) trained from scratch for breast implant classification in 2D mammography, with emphasis on suitability for near real-time clinical deployment. Methods: We evaluated four models: two FMs (RAD-DINO and MammoCLIP) and two CNNs trained from scratch for implant prediction (ResNet18 and our lightweight ResNetLite). Using the Emory Breast Imaging Dataset, 5,000 unilateral screening mammograms were used for training/validation and 1,000 manually reviewed unilateral images were held out for testing. For the FMs, global image embeddings from the pretrained encoder were classified using a support vector machine (SVM). The CNNs were trained end-to-end on 2D mammograms, with ResNetLite optimized via grid search over depth and width to balance accuracy and efficiency. Performance was evaluated using AUROC, sensitivity, specificity, accuracy, embedding visualization, and inference-latency. Results: All models demonstrated strong performance on held-out test data (n = 1,000). MammoCLIP achieved the highest AUROC (0.999) with the quickest training time of 493 seconds. RAD-DINO achieved the highest sensitivity (0.980; accuracy 0.989) but had the slowest inference and training times. ResNet18 and MammoCLIP achieved comparable accuracy (0.985). ResNetLite showed no statistically significant difference from ResNet18 (AUROC 0.993; accuracy 0.976) despite using only 1.4% of ResNet18's parameters, and had the fastest inference time. Conclusion: FMs and task-specific CNN models reliably detect breast implants on 2D mammography. Model selection is best guided by deployment context: MammoCLIP for GPU-equipped hospital settings requiring scalable integration, and lightweight CNNs such as ResNetLite for resource-constrained or edge deployments.
Read the full abstract

All authors Vasisht Ishwar, Hari Trivedi, Young Seok Jeon, Beatrice Brown-Mulry, Frank Li, Rohan Satya Isaac, Mohammadreza Chavoshi, Judy Wawira Gichoya

Image and Video Processing

Reliability Stress Tests and Decision-Time Routing for Chest X-ray Vision-Language Models

Xinye Yang, Zhusi Zhong, Scott Collins et al. (+3 more)

Medical vision-language model (VLM) evaluation is sensitive to workflow design, prompting strategy, and benchmark construction, yet most studies treat these factors in isolation. We introduce a reliability stress test for chest X-ray interpretation built on two balanced datasets (a private report-backed set and a curated MIMIC subset). Three medical VLMs (CheXagent, MedGemma-4B, and MedGemma-27B) are evaluated across three prompt styles and two workflows (single-VLM and multi-agent), producing 36 configurations. We show that exact-match accuracy alone can overstate the effectiveness of conservative models that default to "Normal" predictions. Diagnostic reliability also depends heavily on model family and scale: multi-agent reasoning helps some configurations but hurts others. Building on these observations, we propose a decision-time routing framework that selectively escalates to multi-agent inference only when beneficial, improving the cost-quality trade-off over fixed workflows. Our results highlight the need for evaluation protocols that jointly consider prompt sensitivity, failure-mode diversity, and workflow choice before clinical deployment.
Read the full abstract

All authors Xinye Yang, Zhusi Zhong, Scott Collins, Grayson Baird, Xuyu Wang, Zhicheng Jiao

Image and Video Processing

Confidence-Gated Cloud-Edge Cascade Triage via Variational Risk Minimization for Medical Imaging

Xinye Yang, Zhusi Zhong, Scott Collins et al. (+7 more)

Emergency chest X-ray (CXR) triage has a structural modality gap: reports arrive after triage decisions, yet multimodal foundation models require image-text inputs. We present Variational Risk Minimization (VRM), a distillation framework that treats LVLM-generated report variants as Monte Carlo samples of latent clinical interpretations. Rather than distilling from a single teacher target, VRM learns from a variationally marginalized teacher distribution, enabling uncertainty-aware supervision under missing-modality constraints. Under matched encoder families, VRM outperforms direct fine-tuning baselines and improves calibration with strong recovery from hallucinated supervision. Marginalized supervision reduces report-selection instability. In our compact edge-student instantiation, a confidence-gated cascade reaches AUC 0.941 at 103ms average latency with 20.3% cloud escalation, yielding an explicit reliability-latency operating point for cloud-edge clinical workflows.
Read the full abstract

All authors Xinye Yang, Zhusi Zhong, Scott Collins, Michael Bernstein, Grayson Baird, Terrence Healey, Michael Atalay, Mahesh Jayaraman, Xuyu Wang, Zhicheng Jiao

Image and Video Processing

Event-guided Neural Video Compression

Jiyun Kong, Jungwoo Kim, Enes Eray Demirtas et al. (+2 more)

Neural video codecs derive motion and temporal contexts mainly from RGB frames, leaving room for cross-modal guidance from complementary temporal observations. Event streams can provide such observations by recording brightness changes between frames. In this work, we propose an Event-guided Neural Video Codec (ENVC) that uses events shared by the encoder and decoder to improve RGB compression efficiency. For motion coding, ENVC forms an event-guided motion prior and codes the remaining motion residual. For frame coding, an event-conditioned predictor supplies multi-scale features for gated temporal context refinement. To support training and evaluation on standard video datasets, we synthesize paired RGB-event data and assess its predictive utility through comparisons with real events. Across six benchmarks, ENVC achieves average BD-rate savings of 39.13% using PSNR-RGB and 67.63% using LPIPS relative to DCMVC. Further analyses show that our gains persist on large-motion sequences and that ENVC effectively learns to integrate event information. These results demonstrate the potential of events as a complementary modality for reducing the RGB coding rate. Our model and code are available at https://github.com/kjungwoo03/ENVC.
Read the full abstract

All authors Jiyun Kong, Jungwoo Kim, Enes Eray Demirtas, Touradj Ebrahimi, Jong-Seok Lee

Image and Video Processing

ZAGNet: Zone-Aware Graph Aggregation Network for Patient-Level Lung Ultrasound Diagnosis

Li Chen, Shubham Patil, Rashid Al Mukaddim et al. (+3 more)

Patient-level lung ultrasound (LUS) diagnosis requires integrating findings acquired across multiple anatomical zones, yet clinical examinations frequently involve variable and incomplete scanning protocols with missing zones. Existing diagnostic AI methods primarily analyze individual frames or video loops, relying on heuristic aggregation strategies such as max or mean pooling that ignore inter-zone relationships for patient-level inference. This paper presents ZAGNet, a Zone- Aware Graph Neural Network that represents temporally tracked pathology findings as graph nodes connected by anatomical zone adjacency. A graph transformer network propagates contextual information across neighboring lung regions, while a virtual global node aggregates graph-level features to predict patientlevel consolidation and pleural effusion using only patient-level supervision. ZAGNet accommodates missing zones by computing on a graph structure without fixed input format or size. We evaluate ZAGNet on a multicenter dataset of 714 subjects (20,256 LUS video loops) with exams varying from 4 to 16 zones across anterior, posterior, and lateral thoracic regions. For consolidation diagnosis, ZAGNet achieved an AUC of 0.803 compared to 0.677 (max pooling) and 0.674 (mean pooling). For pleural effusion, AUC increased to 0.893 from 0.804 (max pooling) and 0.815 (mean pooling). These represent improvements of up to 19% and 11% for consolidation and pleural effusion respectively. The results demonstrate that graph-based inter-zone reasoning provides an effective and clinically consistent framework for automated patient-level LUS assessment.
Read the full abstract

All authors Li Chen, Shubham Patil, Rashid Al Mukaddim, Jochen Kruecker, Balasundar Raju, Alvin Chen

Image and Video Processing · Cross-listed from cs.CV

MIRTO: a registration-gated, multiverse-tested evaluation protocol for unsupervised anomaly segmentation in brain MRI

Negin Kafee Hernashki and Soumick Chatterjee

Unsupervised anomaly detection (UAD) methods for brain MRI are ranked by a single score, yet that score rests on choices that are rarely reported: how each anomaly map is aligned with the reference, how and on which data the threshold is set, and which false-positive budget, metric, aggregation and lesion definition are used. We present MIRTO, an evaluation protocol that makes these choices explicit and measures their effect. It gates the geometry of every comparison with a registration check and label-free diagnostics of known power, sets thresholds on validation data alone and reports the false-positive volume actually realised on test, repeats each comparison over 15,552 defensible evaluation pipelines, and attaches paired subject-bootstrap intervals with multiplicity control. Applied to four UAD methods trained on the same healthy data and tested on 312 BraTS 2020 subjects, MIRTO showed that an axis-order mismatch between stored maps and the reference lowered a diffusion model's voxel AUROC from 0.873 to 0.583 whilst barely moving its slice-level AUROC. Within each metric, the method explained at least 0.95 of the variance in voxel AUROC and AUPRC and 0.77 in Dice, but only 0.14 in lesion sensitivity, where the lesion definition and hit criterion dominated. A Dice advantage that was significant at validation thresholds vanished at equal realised false-positive burden, and an exact identity attributes it to threshold transfer. A training-free change to REFLECT's latent aggregation raised Dice at equal burden by 0.052. Nine hypotheses were tested against explicit criteria; because the same cohort served to develop the protocol, all inference is exploratory.
Read the full abstract

Image and Video Processing

Binary Phase Retrieval of Cosine Transforms via Local Curvature Minimization

Ronald Ogden, Shwetadwip Chowdhury, Takashi Tanaka et al. (+1 more)

Cosine transforms see frequent use in image and video compression due to their ease of computation and high energy compaction. This has sparked interest within the optics community in computing cosine transforms optically for compression. When imaging in the Fourier plane to capture a cosine transform, one encounters a phase retrieval problem: optical fields have both magnitude and phase, but cameras only capture the magnitude of the field. Cosine transforms restrict the phase retrieval problem to a binary domain as opposed to the unit circle, but general phase retrieval solutions do not leverage this. In this work, we demonstrate that sign errors in phase retrieval for cosine transforms result in a significant increase in the magnitude of the Hessian of the transform at the error. Motivated by this, we develop an algorithm to solve this binary phase retrieval problem by minimizing the curvature of the reconstructed cosine transform. We demonstrate via computational experiments that given the magnitude of the cosine transform of an image, we can consistently reconstruct the original image within a multiscale structural similarity index of 0.95. We show that the solve time grows approximately linearly with the number of pixels solved.
Read the full abstract

All authors Ronald Ogden, Shwetadwip Chowdhury, Takashi Tanaka, David Fridovich-Keil

Image and Video Processing · Cross-listed from cs.LG

A foundation for systematic analysis of transformers and RNNs for tractography

Emmanuelle Renauld, Philippe Poulin, Hugo Larochelle et al. (+2 more)

Machine learning (ML) has emerged as a promising approach for improving diffusion MRI (dMRI) tractography, a task that remains limited by the intrinsic tension between local diffusion information and global anatomical plausibility. In this work, we systematically evaluate recurrent neural networks (RNNs) and Transformer models for iterative tractography, with particular attention to training strategies, input representations (including convolutional neural network (CNN)-based embeddings and end-of-sequence (EOS) tokens), and hyperparameter selection. We introduce a generation-validation phase enabling supervision at the streamline level during training, allowing supervision despite the mismatch between local loss functions and global streamline quality. Using the ISMRM2015 tractography challenge dataset, our models achieve the highest reported performance to date. Through controlled experiments, we quantify the impact of missing bundles, noisy or imperfect training streamlines, and invalid fibers in the training set. Finally, we demonstrate the applicability of our best-performing models for in vivo data from the Tractoinferno database. Overall, our results highlight both the potential and the limits of sequence-based deep learning models such as Transformers and RNNs for tractography, and emphasize the need for improved phantoms and evaluation methods for in vivo validation. We provide takeaways and recommendations for future researchers training and validating sequence-based supervised methods for tractography.
Read the full abstract

All authors Emmanuelle Renauld, Philippe Poulin, Hugo Larochelle, Antoine Théberge, Maxime Descoteaux

Image and Video Processing

MDIRNET: Multi-Degradation Image Restoration Network via Deep Unfolding

Talha Nadeem, Arslan Majal, Muhammad Tahir et al. (+1 more)

Real images often exhibit unknown and mixed degradations, making restoration substantially more challenging than single-task image restoration because multiple distortion types interact within the same observation. Consequently, existing methods often rely on prior knowledge of the degradation type or separate task-specific models, which may oversmooth fine structures or leave residual artifacts, motivating a compact model-driven alternative. We propose the Multi-Degradation Image Restoration Network (MDIRNET), a unified framework that combines a model-driven low-rank prior with end-to-end learning. Here, unified refers to joint training on three degradation types: noise, rain, and blur. A single MDIRNET model restores all three without requiring task-specific models, modules, or branches at inference. The low-rank prior exploits the redundancy and compact structure of natural image patches. To identify this underlying low-dimensional representation, we formalize restoration via Orthogonal Variational PCA (OVPCA) and translate its iterative inference into a deep unfolding network. To handle spatially non-uniform corruption and local content variability, we further introduce a learnable patch-partitioning strategy and a lightweight dynamic rank-allocation module that predicts the appropriate subspace dimension for each region. Spatially adaptive reconstruction refinement is performed using a supervised attention module. Extensive experiments on standard denoising, deblurring, and deraining benchmarks show that MDIRNET achieves competitive or superior performance over strong baselines across most metrics, while controlled mixed-degradation experiments demonstrate consistent performance across the evaluated synthetic degradation combinations. The code is available at https://github.com/ScholarForge/mdirnet.git.
Read the full abstract

All authors Talha Nadeem, Arslan Majal, Muhammad Tahir, Khurram Ali

Image and Video Processing

RC-aware nnU-Netv2 for Pre-treatment and Post-treatment Glioma Segmentation Using Multimodal MRI

Lin Qu, Ziqi Chen, Anqi Wu et al. (+2 more)

BraTS 2025 Lighthouse Challenge Task 1 (BraTS-GLI 2025) evaluates glioma segmentation in pre-treatment and post-treatment multimodal MRI. The resection cavity (RC) is applicable only to post-treatment cases, creating different target definitions across the two cohorts. We developed RC-aware nnU-Netv2, a framework that combines lesion- and boundary-aware single-cohort training with an RC-aware joint objective and treatment-status-guided routing. During pooled training, the joint objective applies both standard and boundary-weighted binary cross-entropy to all four region channels, while using standard Dice supervision for channels with a non-empty target in the current mini-batch and a controlled false-positive penalty for channels that are empty across the mini-batch. This empty-target handling is particularly relevant to the all-zero RC target in pre-treatment cases. Beyond the standard nnU-Netv2 augmentation pipeline, the submission does not use synthetic tumor generation, on-the-fly GliGAN augmentation, model-level probability averaging, voting, multi-fold fusion, or multi-architecture ensembling. Each case is routed to exactly one specialized model. Our submission ranked second in BraTS-GLI 2025. On the official blind test set, our method achieved mean lesion-wise Dice scores of 0.7878, 0.8709, 0.7923, and 0.8715 and mean NSD@1.0 scores of 0.8309, 0.8712, 0.7980, and 0.8336 for enhancing tumor, RC, tumor core, and whole tumor, respectively. In paired post-treatment analysis, the RC-aware joint model improved mean lesion-wise Dice by 0.042 (95% CI: [0.028, 0.057]) and mean NSD@1.0 by 0.043 (95% CI: [0.028, 0.059]) compared with joint baseline training.
Read the full abstract

All authors Lin Qu, Ziqi Chen, Anqi Wu, Jinyao Shen, Hai Shu

Image and Video Processing · Cross-listed from cs.CV

HierGF: Hierarchical Gaussian Fields via Geometry-perception Message Passing for Sparse-view 3D Reconstruction

Bi'an Du, Zhimin Zhang, Daizong Liu et al. (+2 more)

Sparse view 3D reconstruction is an important and common scenario in multimedia applications, such as augmented reality/virtual reality (AR/VR) content creation, cultural heritage digitization, and certain robotic applications, where only a limited number of randomly captured views may be available. However, sparse views contain only limited 3D information, posing two major challenges:1) too few images are available for matching, making it difficult to build multi-view consistency; 2) insufficient view coverage leads to a lack of information in under-sampled regions, resulting in missing parts of object structure. Existing methods mostly still rely on limited reprojection errors and regularization terms, which are prone to overfitting to a single view and inconsistent appearances across views. In geometrically under-sampled regions, they often rely on heuristic density control, lacking reliable guidance and often resulting in blurring and structural holes.To address these issues, this paper proposes Hierarchical Gaussian Fields (HierGF), which revisits sparse-view reconstruction from a hierarchical geometry-perception perspective and converts limited observations into reliable self-generated supervision beyond fixed priors and heuristic density control. In particular, we transform coarse 3D geometric information and additional 2D generative priors into structured pseudo-supervision through a two-stage geometry-perception backbone network, thereby enhancing multi-view consistency with very few input views. In addition, we introduce a learnable confidence network to guide gradients toward cross-view consistent content, and a geometrically consistent densification module to improve the reconstruction of multi-view alignment and under-sampled regions.
Read the full abstract

All authors Bi'an Du, Zhimin Zhang, Daizong Liu, Baoquan Chen, Wei Hu

Image and Video Processing · Cross-listed from cs.RO

UniTrackPLA: Unified Panorama-Language-Action Model for Instruction-Guided Navigation and Dynamic Person Tracking

Pengfei Qi, Haoran Lin, Sizhuang Chen et al. (+7 more)

General-purpose embodied robots should support both navigation toward language-specified destinations and dynamic person tracking under arbitrary initial target azimuths. However, existing methods typically rely on forward-facing observations and address these tasks with separate policies, limiting omnidirectional perception and unified closed-loop control. We present UniTrackPLA, a unified panorama-language-action model for instruction-guided navigation and dynamic person tracking. Its Panoramic-Aware Encoding (PAE) preserves the temporal and azimuthal structure of perspective views projected from each panorama, enabling perspective-pretrained visual encoders to process omnidirectional observations. A shared vision-language backbone grounds instructions in the panoramic context and predicts continuous robot-centric waypoint chunks for both tasks. World-Action Consistency (WAC) further predicts action-conditioned future visual states and verifies waypoint prefixes online, allowing reliable actions to be reused while triggering replanning upon inconsistency. We also introduce OmniTrackNav-Bench, comprising 5,000 simulated tracking trajectories, 10,000 simulated VLN routes, and 96 verified real-world routes, providing 919,978 waypoint-supervision instances. UniTrackPLA improves overall tracking SR from 23.50% to 35.00% and Omni-VLN SR/SPL from 13.00%/12.77% to 19.75%/19.29%. Incorporating 76 real-world routes further improves held-out EP@0.2m from 42.92% to 92.08%. Closed-loop experiments on a Go2-W robot demonstrate unified panoramic tracking and navigation across indoor and outdoor environments. The project page is at https://tw5775.github.io/UniTrackPLA.
Read the full abstract

All authors Pengfei Qi, Haoran Lin, Sizhuang Chen, Kai Luo, Sirui Zhang, Xinqi Liu, Fei Cheng, Wenrui Chen, Liming Yin, Kailun Yang

Image and Video Processing

MorphoBranch: A Fine-Structure-Preserving Workbench for Morphometric Analysis of Branched Cellular Structures

Song Zhiying, Ling Hanyi, Wu Junyi et al. (+1 more)

Background and Objectives: Fluorescence-labeled cellular arbors provide readouts of neuronal and microglial morphology, but fine and weakly labeled processes are prone to fragmentation and false connections that bias skeleton-based measurements. We present MorphoBranch, a fine-structure-preserving, human-reviewable workbench for morphometry of branched cellular structures. Methods: MorphoBranch combines a deterministic Morphometry Engine with an LLM-assisted Refinement Engine. The Mor- phometry Engine implements an image-to-graph workflow integrating multiscale structural evidence extraction, hysteresis segmen- tation, evidence-constrained skeleton refinement, and graph-based morphometry. The Refinement Engine maps natural-language requests to registered actions for parameter adjustment, preview execution, metric reporting, and unsupported-request handling, while image processing and quantitative computation remain deterministic and reviewable. Results: MorphoBranch was evaluated on two public neuronal axon datasets, AxonMIP and AxonStack, and the in-house Cell- Morph dataset of microglial fluorescence images. It achieved the highest Skeleton F1 and clDice and the lowest length-estimation error among the evaluated methods on all three datasets, while also achieving the highest Dice and IoU on AxonMIP and Axon- Stack. Across 150 natural-language tasks, the Refinement Engine achieved a 94.0% end-to-end success rate. Conclusions: These results demonstrate that MorphoBranch provides a reproducible, human-reviewable workflow for mor- phometric analysis of branched cellular structures. It supports fine-structure-preserving quantification across neuronal axon and microglial fluorescence images while maintaining inspectable and reproducible analysis workflows.
Read the full abstract

All authors Song Zhiying, Ling Hanyi, Wu Junyi, Jiang Yangbo

Wednesday 30 September

Image and Video Processing · Cross-listed from q-bio.QM

Optimization of Deep Learning Model for Denoising Dose Profiles in Proton Therapy

Junaid Jawaid, Matteo Spezialetti, Gabriele Libardi et al. (+2 more)

Proton therapy offers a dosimetric advantage over conventional radiotherapy through the Bragg peak, enabling precise dose delivery to the tumour while sparing surrounding tissue. Monte Carlo (MC) simulation provides the most accurate dose calculation but is computationally expensive for clinical workflows. A promising compromise is to simulate with a reduced number of primary histories and recover a high-quality dose distribution through a deep learning denoising step. In this work, we take a 3D multiscale U-Net from recent literature as a case study and investigate two improvements critical for clinical deployment. First, we replace the standard MSE training objective with a differentiable $\gamma$-index-based loss, extended to native 3D operation, and demonstrate that directly optimising the clinical evaluation metric yields a mean Gamma Passing Rate of 99.47% at 2mm/2% and 95.67% at 1mm/1% on the test set, compared to 92.10% and 71.13% for MSE. Second, we apply L1-norm structured filter pruning to reduce the model's memory footprint, achieving a 13.3% parameter reduction with negligible performance loss under partial pruning, and 20% reduction under full-block pruning after fine-tuning.
Read the full abstract

All authors Junaid Jawaid, Matteo Spezialetti, Gabriele Libardi, Giovanni Stilo, Filippo Mignosi

Image and Video Processing

Spatially Gated Diffusion for Localized Counterfactual Chest Radiograph Editing

Kamran Ullah Afaq and Basit Raza

Editing a chest radiograph requires completing the requested change while preserving unrelated content. We study a latent diffusion editor with an instruction-independent source trajectory and an instruction-conditioned editing trajectory. A learned gate mixes their post-sampler candidates at each executed step, and a separate image-space mask composites the decoded proposal with the source. On 2,400 MIMIC-derived requests, 2,244 outputs met the joint target, preservation, quality, and coverage rubric (93.5%), and target completion was 97.4%. Joint validity exceeded a matched composition-only control by 1.7 percentage points (paired patient-cluster 95% interval, 0.6–2.8). At a fixed learned mask, the learned-gate proposal improved joint validity by 1.6 points (0.8–2.4); at a fixed learned proposal, the learned mask improved it by 2.6 points (1.7–3.5). Across three training seeds, mean joint validity was 93.5% with a 0.3-point sample standard deviation. A blinded 240-request assessment yielded adjudicated joint validity of 93.3% for the full editor and 91.7% for composition only. Protected-region mean absolute error decreased from 0.0190 in the raw proposal to 0.0075 after composition. These findings distinguish recurrent-gating effects on the proposal from preservation through final composition in the assessed cohort.
Read the full abstract

Image and Video Processing · Cross-listed from eess.AS

MAV-C: A Framework for the Joint Objective Estimation of Audio-Visual Complexity in Immersive Virtual Environments

Luca Resti, Amelia Gully, Michael McLoughlin et al. (+2 more)

We present the Motion-Aware Audio-Visual Complexity Metric (MAV-C), a reference-free framework for the joint objective estimation of audio-visual complexity. The metric combines entropy-based audio features (temporal, spectral, and spatial) with visual features (Sobel gradient magnitude, chromatic uniqueness, and optical flow) via a parametric fusion stage, producing a continuous joint complexity score CAV (t) [0,1]. We validate MAV-C on two datasets: a controlled synthetic corpus (SYN) of stimuli with known signal characteristics and a naturalistic gameplay corpus (GAM) of 60 clips drawn from the SAFEPLAY-X dataset. On SYN, the metric exhibits strong validity: the audio score CA and visual score CV are each insensitive to changes in the opposite modality (CoV < 0.003), the joint score CAV spans [0.00,0.90] across all parameter combinations, and single-axis feature sweeps produce monotone trajectories (Spearman up to 0.995). On GAM, CV differs significantly across content categories (Kruskal-Wallis p = 0.021) while CA does not, and the two sub-scores are uncorrelated (r = 0.03), confirming they operate on independent signal dimensions. OFAT sensitivity analysis identifies a two-tier parameter hierarchy, with modality balance (wa) and visual regularization (v) as most significant tunable parameters. Full subjective calibration is planned as future work.
Read the full abstract

All authors Luca Resti, Amelia Gully, Michael McLoughlin, Gavin Kearney, Alena Denisova

Image and Video Processing

Synthetic-to-Real Transfer in Cerebral Microbleed Generation and Segmentation

To-Liang Hsu, Ting-Yu Lai, Ching-Ting Lin et al. (+2 more)

The development of automated cerebral microbleed (CMB) detection models is hindered by the low prevalence of CMBs and the high cost of expert annotation. To address this limitation, we developed a synthetic CMB generation pipeline and investigated the effectiveness of synthetic lesions for training deep learning detectors. Models trained solely on synthetic data achieved substantial detection performance, reaching approximately 87% of the lesion sensitivity of their real-trained counterparts. We further investigated the complementary roles of synthetic and real data under different training paradigms, revealing that a significant performance gap remains. Moreover, we found that successful synthetic-to-real transfer is strongly dependent on the downstream detection architecture, providing new insight into both the potential and limitations of synthetic data for CMB detection.
Read the full abstract

All authors To-Liang Hsu, Ting-Yu Lai, Ching-Ting Lin, Chun-Hao Huang, Wei-Chun Wang

Image and Video Processing

Nonparametric Distribution Matching for Self-Supervised Whole-Slide Image Condensation

Duong M. Nguyen, Trong Nghia Hoang, Hang Thi Nguyen et al. (+3 more)

Histological whole-slide images (WSIs) are central to computational pathology but pose severe computational challenges due to their extremely high resolution, often spanning several gigabytes per slide. To enable scalable learning, existing methods apply self-supervised data condensation to reduce computational cost, but typically rely on heuristic prototype learning and do not explicitly preserve learning-relevant feature distributions for downstream tasks. In response, we introduce a principled reformulation of WSI condensation as a distribution-matching problem under a fixed representational lens, and develop NICER, a tractable approximation framework based on a nonparametric prior with slide-adaptive capacity. Experiments on five histopathology datasets, together with clinical evaluation from a board-certified pathologist, show that NICER consistently outperforms prior methods, achieving an average accuracy improvement of 7.44% while offering improved efficiency-accuracy trade-offs, highlighting the benefits of principled, distribution-aware condensation for scalable histological representation learning. Source codes are available in https://github.com/nmduonggg/NICER.
Read the full abstract

All authors Duong M. Nguyen, Trong Nghia Hoang, Hang Thi Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Minh N. Do

Image and Video Processing

CPathOGen: Spatially and Morphologically Controlled H&E Counterfactuals for Probing Pathology Models

Samarth Singhal and Varang Rai

Computational pathology models infer biologically and clinically meaningful outcomes from histology, but their predictions are shaped by complex, intertwined tissue signals whose roles are important to understand. Common pixel- and feature-space perturbations can produce implausible tissue, making model responses difficult to interpret. We introduce CPathOGen, a conditional latent-diffusion framework for generating paired H&E counterfactuals with explicit controls over cellular spatial organization, nuclear morphology, and stain appearance. Cellular maps condition spatial structure through a spatial encoder, while a morphology/appearance vector modulates denoising through blockwise feature-wise linear modulation (FiLM). On held-out H&E tiles, CPathOGen generates visually plausible tissue with improved distributional agreement after spatially guided selection, as reflected by lower Fréchet Inception Distance (FID) and Kernel Inception Distance (KID); generated cells track requested abundance and position, and measured morphology and color vary monotonically with their controls. We use these verified interventions to probe pathology encoders with endpoint heads, task-specific classifiers, and survival models. Responses are quantified using total variation distance and prediction-flip rate. We further introduce the Biology-Nuisance Sensitivity Ratio, a metric that contrasts model sensitivity to biologically motivated morphology and spatial factors with sensitivity to non-biological, stain-related nuisance variation. CPathOGen provides a practical, fidelity-audited framework for evaluating robustness and controlled feature sensitivity in computational pathology model. \href{https://github.com/a12dongithub/PathOGen}{GitHub} and \href{https://huggingface.co/a12donhf/CPathOGen}{Hugging~Face}.
Read the full abstract

Image and Video Processing

Hetero-modal learning and corruption-resistant hetero-modal inference for joint segmentation of white matter hyperintensities and ischaemic stroke lesions in MRI

Jesse Phitidis, William N. Whiteley, Joanna M. Wardlaw et al. (+3 more)

White matter hyperintensities (WMH) and ischaemic stroke lesions (ISL) are visually confounding, co-occurring pathologies that require large, diverse datasets for robust deep learning segmentation. However, assembling such datasets is hindered by cohort samples that lack reference segmentations from both features and complete sets of MRI structural sequences (i.e., "modalities"). To maximise data utility, we investigate hetero-modal learning using a dataset of 206 vascular disease patients across four MRI sequences (T1-weighted, T2-weighted, fluid-attenuated inversion recovery, and diffusion-weighted imaging) with expert annotations of both WMH and ISL. We demonstrate that hetero-modal learning outperforms models trained using a single imaging modality in scenarios with substantial missing data, including a split where only 10% of the training data contains all four modalities while the remainder is uni-modal, and a split relying on a single shared "anchor" modality with zero overlap between the remaining modalities. Furthermore, models trained under this second split successfully perform inference on unseen combinations of modalities. Yet, while standard hetero-modal networks handle missing sequences, clinical deployment introduces the additional challenge of silent data degradation - where modalities are present but severely corrupted. To bridge this gap, we introduce the Multimodal Attention Router (MMAR) block. Our experiments demonstrate that, when trained with a "corruption augmentation" strategy, the MMAR effectively dynamically weights the encoded features of each modality, maintaining strong performance during hetero-modal inference even in the presence of unflagged catastrophically corrupted modalities.
Read the full abstract

All authors Jesse Phitidis, William N. Whiteley, Joanna M. Wardlaw, Miguel O. Bernabeu, Francesco Dalla Serra, Maria del C. Valdés Hernández

Image and Video Processing

RIQE: a NIQE-style reference model for Computed Tomography

Fabio Mattiussi

The Natural Image Quality Evaluator (NIQE) scores an image by its statistical distance from a model fitted on pristine images, and its distributed model is fitted on photographs. We release the Radiology Image Quality Evaluator (RIQE), a NIQE-style model fitted on 3,792 full-dose slices from 158 patients of the public LDCT-and-Projection-data collection, with a declared intensity mapping, a manifest of every slice and a script that reproduces the fit. On 40 held-out patients, RIQE ranks reduced-dose reconstructions, simulated by projection-domain noise insertion, worse than the full-dose reconstruction of the same slice in 240 of 240 chest and 230 of 240 abdominal pairs, and ranks images with 20% more noise worse than their source in 97.5-100% of cases. Its preferences among filtered images, however, do not follow lesion signal. With a 4 mm, +10 HU lesion inserted in noisy abdominal slices, RIQE prefers bilateral filtering to the unfiltered image in every image up to a 32 HU residual, at which 29% of the lesion's matched-filter signal remains and its detectability index falls from 0.51 to 0.33; it never prefers Gaussian smoothing, which at the same 32 HU residual leaves 70% of the signal and a detectability index of 0.48. Fitted on photographs with the parameters published for NIQE, the same code ranks every simulated reduced-dose abdominal image better than its full-dose counterpart. RIQE is suited to ranking a degraded image against its source; under the conditions tested it should not be the sole criterion for selecting, comparing or tuning denoisers.
Read the full abstract

Image and Video Processing

The Effect of Tissue Detection on False Positives of Diffusion-Based Artifact Detection in Histopathology

Konstantinos Moutselos and Ilias Maglogiannis

One-class artifact detectors for whole-slide images learn normal tissue from a clean training pool and flag departures from it. The pool is built by a preprocessing pipeline whose tissue-detection step is usually treated as neutral. We tested whether it is. On 16 annotated slides from The Cancer Genome Atlas, we rebuilt the clean pool of a diffusion-based detector with different tissue detection methods and compared the resulting models in a four-fold cross-validation. Per-slide saturation-Otsu detection excluded normal tissue, chiefly tissue with large clear spaces such as adipose tissue and alveolar parenchyma, and on slides with thick marker ink kept the ink while excluding ordinary tissue. Replacing it with entropy-based detection reduced the false-positive fraction on held-out clean slides from 0.102 to 0.016, in every fold and with a second training seed, without loss of sensitivity; the gain came from the composition of the pool, not its size. Across three tissue detection methods, false positives followed the fraction of such clear-space tissue in the pool, a statistic that needs no labels or training (0.103, 0.016 and 0.009). The effect did not carry over at the same size to a nearest-neighbour detector on foundation-model features. On an external cohort, the curated pool lowered clean-control false positives by about 20%, far less than on the development slides, and the remaining cross-center loss was not explained by stain differences. For one-class quality control, tissue detection decides what the model learns as normal and should be chosen and reported accordingly.
Read the full abstract