Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Electrical Engineering and Systems Science Image and Video Processing

eess.IV Watch Image and Video Processing

Monday 17 August

Image and Video Processing

Decoupling Parcellation from Classification: Systematic Benchmark of Fast Brain Segmentation Methods for Alzheimer's Disease Detection

Jiadao Zou, Hongyu Guo and Wei Xi

Brain parcellation and classification are typically evaluated in isolation, yet downstream AD detection performance depends on their interaction. We decouple these components and systematically benchmark fast deep learning parcellation methods (SynthSeg+, OpenMAP-T1) against the FreeSurfer (FS-HV) clinical baseline through down- stream AD classification on OASIS-1. Our factorial design evaluates three parcellation methods, two volumetry strategies (hard vs. soft), and four classifier paradigms (clinical thresholds, supervised feedforward networks, ensemble methods, and foundation models with zero/few-shot prompting), with all results quantified using BCa Bootstrap 95% confidence intervals.
Read the full abstract

Sunday 16 August

Image and Video Processing

TractoGraphVLM: A Unified Vision-Language Framework for White Matter Tractography

Gurucharan Marthi Krishna Kumar, Janine Dale Mendola and Amir Shmuel

Vision language models have transformed 2D medical imaging, yet extending them to 3D white matter tractography remains challenging due to the complex topology of fiber bundles. We introduce TractoGraphVLM, a unified framework for four tasks, bundle classification, text-to-tract retrieval, anatomical captioning, and visual question answering, built on a shared GPS architecture, training procedure, and read-out design. Fiber bundles are represented as streamline graphs whose nodes encode 3D position and tangent orientation. A General, Powerful, Scalable (GPS) graph transformer produces bundle embeddings aligned with a frozen BiomedBERT text encoder via contrastive learning, while a BioGPT decoder with visual prefix tokens generates captions and answers. A single shared encoder and decoder is trained jointly across all four tasks and evaluated from one checkpoint. Trained on HCP Young Adult subjects, TractoGraphVLM achieves 91.8% bundle classification accuracy, 84.7% retrieval R@1, BLEU-4=20.1, ROUGE-L=66.8, and 66.4% VQA accuracy on a held-out test set. The same checkpoints transfer zero-shot to HCP Aging subjects, with a modest drop on discriminative tasks and a larger drop on generative tasks, showing robustness to age and acquisition shift. Language supervision yields richer representations than label-only training, recovering structure like hemisphere and fiber family, carried by captions but never given as a label. Swapping only the visual encoder, graphs preserving fiber orientation outperform volumetric baselines, with GPS giving the best balance. Generative metrics measure consistency with a structured knowledge base rather than independent clinical text; even so, TractoGraphVLM shows that classifying, retrieving, describing, and answering questions about a white matter bundle can be served by one jointly trained model that learns transferable neuroanatomy from language alone.
Read the full abstract

Image and Video Processing

MagViT: Interpretable Multi-Magnification Transformers with Patient-Level Model Selection for Breast Histopathology

Nabil Ashab, Soumit Kumar Kundu, Saif Mahmud Parvez et al. (+3 more)

Breast cancer is one of the most common types of cancer among women around the world. Rapid detection and early treatment can hinder its progress to more complex stages and can impede its spread to other parts of the body. Histopathological image classification is the most common task in cancer detection due to its robustness in analyzing cellular data. Breast histopathology classification requires handling both multi-scale tissue morphology and clinically relevant generalization beyond the source domain. This paper presents MagViT, an interpretable multi-magnification transformer framework with scale-gated fusion and patient-level model selection. The model uses four BreakHis magnifications (40X, 100X, 200X, 400X) and extracts per-scale representations with a ViT backbone, and combines them via a learnable gate that masks missing scales. Patient-level five-fold cross-validation with a fixed seed has been run and compared with three architectural branches. The most accurate branch is then selected as the final model due to the strongest patient-level accuracy while retaining the simplest fusion pathway. On BreakHis, our architecture achieves a mean image accuracy of 0.9191, a mean patient accuracy of 0.9643, and a mean macro-F1 of 0.9042. External transfer experiments provide preliminary evidence of cross-dataset generalization under controlled adaptation settings on BUSI (image accuracy 0.8306, macro-F1 0.7480, patient accuracy 0.8291) and IDC (image accuracy 0.8577, macro-F1 0.8191, patient accuracy 0.8372). Grad-CAM visualization indicates that the model focuses on diagnostically significant and meaningful regions across magnifications. Relative to prior ViT-centered BreakHis work, this study emphasizes patient-level selection and cross-dataset robustness under a reproducible protocol.
Read the full abstract

All authors Nabil Ashab, Soumit Kumar Kundu, Saif Mahmud Parvez, Shahadat Hossain Sohag, Bidhan Biswas, Nazmus Subha

Image and Video Processing

ORViT-DR: Ordinally-Robust Hybrid ViT for Low-Resolution Diabetic Retinopathy Grading

Soumit Kumar Kundu, Nabil Ashab, Bidhan Biswas et al. (+4 more)

Diabetic retinopathy (DR) is one of the main causes of impaired vision. A good and reliable automated grading system can make the screening process safer and more accurate. Because DR stages progress gradually, the task of grading disease severity naturally follows an ordinal structure in which neighboring classes share similar visual characteristics. In this study, ORViT-DR, a hybrid deep learning framework, is designed to improve DR grading from low-resolution retinal images. The proposed approach combines convolutional feature extraction with transformer-based global context modeling through a pre-trained ViT-Hybrid backbone, which integrates BiT-ResNetv2 with a Vision Transformer architecture. The approach is tested on the RetinaMNIST subset of the MedMNISTv2 dataset, which contains 28x28 retinal fundus images annotated with five levels of disease severity. To promote stable training and better feature learning, the training strategy applies progressive layer unfreezing, layer-wise learning rate decay, exponential moving average (EMA) parameter updates, and ensemble-based prediction during inference. Experimental results on the official RetinaMNIST test set show that the proposed method achieves 57.00% classification accuracy, along with a quadratic weighted kappa score of 0.5963 and a macro-F1 score of 0.4293. These results suggest that hybrid CNN-Transformer architectures can provide effective representations for ordinal retinal image analysis.
Read the full abstract

All authors Soumit Kumar Kundu, Nabil Ashab, Bidhan Biswas, Shahadat Hossain Sohag, Saif Mahmud Parvez, Souvik Kumar Kundu, Zunayed Ahmed Rafi

Image and Video Processing

Deep learning-based computed tomography (CT) derived body composition classifier for colorectal cancer patients

Eve Harling, Chattarin Pumtako, Bernd Porr et al. (+2 more)

Background: Accurate body composition analysis using Computed Tomography (CT) scans is essential for assessing skeletal muscle area (SMA) and skeletal muscle density (SMD), key markers of nutritional status in cancer patients. Conventional manual methods are labour-intensive and require specialist expertise, limiting their routine clinical use. Therefore, this study serves as a feasibility and pilot investigation to explore the potential of deep learning-based automated regression for body composition analysis within a clinical workflow. Methods: Four deep learning architectures (AlexNet, UNet, GoogLeNet, and ResNet34) were trained to predict SMA, SMD, subcutaneous fat area (SFA), and visceral fat area (VFA) from CT scans of colorectal cancer patients. Systematic hyperparameter optimization identified the most accurate models, which were subsequently implemented in a web application for clinical use. Results: GoogLeNet achieved the best performance, with a mean percentage error (PE) of 4.96% for SMA prediction, while AlexNet reached 8.12% for SMD. Independent testing demonstrated robust accuracy, correctly classifying body composition metrics in 80% of cases. The web application delivered rapid and consistent outputs, supporting integration into clinical workflows. Conclusion: Optimized deep learning models, particularly GoogLeNet and AlexNet, can automate CT-derived body composition analysis with a Mean Percentage Error (PE) of 4.96% for SMA and 8.12% for SMD. These tools have the potential to streamline clinical practice by reducing the time and expertise required for manual segmentation. Further validation in larger, more diverse datasets is warranted.
Read the full abstract

All authors Eve Harling, Chattarin Pumtako, Bernd Porr, Donald C McMillan, Ross D Dolan

Image and Video Processing

Underwater Color Restoration with Vanishing Uncertainty

Grigory Solomatov and Derya Akkaynak

Underwater color restoration promises to unlock color as a reliable signal for aquatic sciences, but achieving this with scientific confidence remains out of reach. Current methods are validated almost exclusively on an empirical basis, which provides confidence only to the extent that the vast diversity of possible visibility conditions is covered with end-to-end testing using a known ground truth. This is exacerbated by color restoration being a fatally ill-posed problem when considered in full mathematical generality, requiring additional constraints to narrow the solution to a finite uncertainty interval. The gap between which constraints suffice in theory and which constraints are satisfied by real-world data is poorly understood, making it unclear whether existing methods are solving a problem that is actually solvable. In this article, we investigate the theoretical side of this gap, identifying idealized conditions which guarantee bounded uncertainty that converges to zero as the spatial resolution of the camera increases.
Read the full abstract

Saturday 15 August

Image and Video Processing

Dual-Branch State-Displacement Network for Sea Surface Temperature Super-Resolution

Wankun Chen, Feng Gao, Yanhai Gan et al. (+4 more)

Sea surface temperature (SST) is a critical indicator of global climate change, yet satellite-derived SST imagery often suffers from coarse spatial resolution, limiting the ability to capture fine-scale thermal structures such as ocean fronts. To address this, we propose a Dual-Branch State-Displacement Network (DBSD-Net) for SST super-resolution. DBSD-Net adopts a dual-branch architecture: a wavelet frequency branch that explicitly separates low and high-frequency components via discrete wavelet transform for targeted processing, and a VGGUNet branch that extracts multi-scale semantic features from a frozen pre-trained VGG backbone. Within the wavelet branch, we introduce a Structural State Space Module (SSSM) with a Gated Structure Refinement (GSR) unit to efficiently capture long-range dependencies and enhance structural integrity, and a Displacement Gate Module (DGM) that learns a displacement field for geometry-aware modulation of high-frequency details, thereby mitigating spatially varying degradation. Experiments on multiple public SST datasets demonstrate that DBSD-Net outperforms existing state-of-the-art methods.
Read the full abstract

All authors Wankun Chen, Feng Gao, Yanhai Gan, Chuanzheng Gong, Xun Gong, Junyu Dong, Qian Du

Image and Video Processing · Cross-listed from cs.CV

ENAF: A Multi-Exit Network with an Adaptive Patch Fusion for Large Image Super Resolution

Duong M. Nguyen, Tuan Nghia Nguyen and Xuan Truong Nguyen

To accelerate single image super-resolution (SISR) networks on large images (2K-8K), many recent approaches decompose an image into small patches and dynamically determine an execution path according to its difficulty (referred to as a dynamic network). To quantify the hardness of a patch, they mainly rely on a handcrafted assessment score, e.g., edge, which weakly associates a patch's texture with the computational complexity of a SISR model. To address the problem, we introduce ENAF - a dynamic network for SISR with an adaptive patch fusion. Built on top of a backbone, ENAF incorporates multiple early exits (EEs) to tackle the over-parameterized SISR model. More importantly, ENAF plugs a tiny network that estimates PSNR to associate data texture with a computation cost at an EE. Based on the scores, ENAF effectively assigns image patches to an exit, enhancing the quality-complexity trade-off. Extensive experiments on common datasets with popular SISR backbones demonstrate the effectiveness of ENAF in various settings. The source code is provided in https://github.com/nmduonggg/ENAF
Read the full abstract

Image and Video Processing

Multi-Channel Feature Fusion and Monte Carlo Dropout for Uncertainty-Aware Diabetic Retinopathy Grading

Saksham Kumar

Automated five-stage diabetic retinopathy (DR) grading requires more than high accuracy alone. Medical-grade deployment calls for lesion-aware preprocessing, ordinal predictions, calibrated uncertainty, and explainability to support reliable diagnostic systems. We present a unified pipeline that addresses these requirements using a Ben-Graham-green-channel CLAHE feature representation, an EfficientNetV2-L ordinal regressor, and Monte Carlo dropout for uncertainty-driven referral. Grad-CAM provides visual explanations aligned with clinically relevant lesions. The proposed method achieves a QWK of 91.31% on the APTOS-2019 official test split, placing it within the near-perfect agreement band (>80%). At a 20% referral rate, 293 of 366 images are automatically graded with a QWK of 90.40%. More complex cases are referred for specialist assessment, demonstrating a practical trade-off among grading quality, automation, and patient safety in robust, reliable, and deployment-ready medical diagnostic systems.
Read the full abstract

Image and Video Processing · Cross-listed from cs.CV

MODAL: Multi-Modal Object Re-ID via Model-Driven Sparse Decoupling and Text-Image Differential Filtering

Chengbo Huang, Jun-Jie Huang, Long Lan et al. (+5 more)

Multi-modal object re-identification (Re-ID) aims to facilitate cross-camera object retrieval in complex environments by leveraging complementary information from visual (e.g., RGB, NIR, TIR) and textual modalities. However, existing approaches often lack principled feature disentanglement and coherent multi-modal integration, leading to entangled representations that introduce cross-modal conflicts, obscure discriminative cues, and suffer distribution shift under modality-missing conditions. To tackle these challenges, we propose MODAL, a novel multi-modal object re-identification framework, grounded in coupled sparse coding theory and differential suppression principles. A core component of MODAL is a Multi-modal Feature Sparse Decoupling module, developed in a model-driven deep unrolling manner based on multi-modal coupled sparse coding. It explicitly decomposes multi-modal features into uni-modal specific, bi-modal and tri-modal shared representations, thereby achieving more transparent and effective feature disentanglement. Benefiting from the principled feature disentanglement, MODAL naturally mitigates performance degradation in incomplete-modality scenarios via a Modality-Aware Subspace Activation that selectively activates only the consistently shared subspaces. Moreover, we propose a Text-Image Differential Filtering module that leverages coarse-grained textual semantics to adaptively suppress task-irrelevant responses in the decoupled visual representations, thereby enhancing discriminative information. Extensive experiments on four datasets demonstrate that MODAL achieves state-of-the-art performance with superior transparency.
Read the full abstract

All authors Chengbo Huang, Jun-Jie Huang, Long Lan, Tianrui Liu, Xueqiong Li, Yuanxi Peng, Xinwang Liu, Meng Wang

Image and Video Processing · Cross-listed from eess.SP

Flexible Deep Joint Source-Channel Coding: A Vibrotactile Example

Shuijie Li, Kemi Chen, Runjie Wang et al. (+2 more)

The increasing demand for real-time tactile communication in multimedia systems has exposed the limitations of existing Joint Source-Channel Coding (JSCC) techniques. While current JSCC models facilitate end-to-end optimization, they typically operate at fixed coding rates and require separate model instances for different rate settings. This results in significant storage overhead and limited adaptability to dynamic bandwidth conditions. To address these challenges, we propose the Flexible Deep Joint Source-Channel Coding (FD-JSCC) framework for vibrotactile signals, which supports flexible-rate transmission without the need for model switching. The FD-JSCC integrates a flexible-rate encoder-decoder enhanced with Hierarchical Gain Adaptation Module (HGAM) and Rate-Switchable Residual Module (RSRM), enabling bitrate-aware compression by selectively preserving salient vibrotactile features. Additionally, we introduce a Channel Feature Processing Module (CFPM), which leverages real-time SNR information to enhance robustness against channel noise and signal degradation. Trained on the IEEE 1918.1.1 vibrotactile dataset, FD-JSCC achieves reconstruction performance comparable to fixed-rate baselines (e.g., DeepSC-S), while reducing storage requirements by 61.1% when supporting four rates. These results underscore its potential for scalable, low-latency tactile communication in next-generation networks.
Read the full abstract

All authors Shuijie Li, Kemi Chen, Runjie Wang, Tiesong Zhao, Xiaoming Tao

Image and Video Processing · Cross-listed from eess.SP

ParaJSCC: A Parameterized Framework for Reusable Multimodal Joint Source-Channel Coding

Kemi Chen, Mingkai Chen, Youjia Chen et al. (+3 more)

Multimodal signals, such as visual, audio, and tactile data, are increasingly maintained as persistent digital assets in immersive communication systems and digital twins. In these settings, the same multimodal content is repeatedly accessed by heterogeneous receivers with varying modality and bandwidth requirements. Existing compression and Joint Source-Channel Coding (JSCC) methods typically follow a per-request encoding paradigm, resulting in redundant computation and low efficiency during repeated access. To address this issue, we propose ParaJSCC, a multimodal JSCC framework designed for reusable representation serving. ParaJSCC converts each multimodal sample offline at the cloud/content server into a compact, quantized parameter package, which is then stored at the edge serving node for low-latency access. During serving, only the subset required by the current request is transmitted over the wireless channel, followed by lightweight decoding at the receiver. The framework employs a progressive shared-private parameterization to support modality-selective transmission and scalable reconstruction under varying bandwidth constraints. Experiments on multimodal datasets show that ParaJSCC significantly reduces online latency (e.g., from 17.18~ms to 4.34~ms for image-only requests and from 43.96~ms to 11.21~ms for full multimodal requests) and transmission rate (by 47.8%–51.2% for selective requests), while maintaining strong reconstruction quality under noisy channels.
Read the full abstract

All authors Kemi Chen, Mingkai Chen, Youjia Chen, Qian Liu, Wei Gao, Tiesong Zhao

Image and Video Processing · Cross-listed from cs.LG

PathFinder: Joint Decompositions of Linked Multimodal Datasets

Ying-Qiu Zheng, Alex Fung, Stephen M Smith et al. (+2 more)

Low-rank matrix decompositions can uncover patterns and structure in data and have a number of different applications across many disciplines. Extensions to "joint" low-rank decompositions have been proposed to link datasets from different modalities. While these methods enable the discovery of common patterns across modalities, they require that all the multimodal data share one or more dimensions. We propose a new analysis method, PathFinder, that enables co-analysis of datasets that do not necessarily all share a dimension. The key insight is that as long as pairs or subgroups of matrices do share some dimension, and that there are one or more paths that link across the data matrices, a global joint decomposition can be sought out. This enables the joint estimation of common patterns across different modalities, species, or scales, where a one-to-one mapping across all data along some dimension is not necessarily available. We show that PathFinder is a general umbrella under which many matrix decomposition methods fall as special cases. It can be used to discover common patterns across disparate datasets and to make predictions for missing data or modalities.
Read the full abstract

All authors Ying-Qiu Zheng, Alex Fung, Stephen M Smith, Rogier B Mars, Saad Jbabdi

Friday 14 August

Image and Video Processing

Modality-Invariant Coarse-to-Fine Retinal Image Registration

Bo Wen, Nehal Nailesh Mehta, Melanie Tran et al. (+3 more)

Retinal image registration is essential for ophthalmic diagnosis, longitudinal disease monitoring, and multimodal retinal image analysis. Existing retinal registration methods are typically modality-dependent: they are designed or optimized either for a single imaging modality in mono-modal registration or for a fixed pair of modalities in cross-modal registration. This limits their flexibility and applicability in practical scenarios involving diverse retinal imaging modalities and different combinations of them. In this work, we propose a generalizable two-stage, modality-invariant framework for retinal image registration. First, we introduce a sparse feature-matching model driven by a universal retinal vessel segmentation to achieve robust coarse global alignment across modalities. Second, we develop a modality-invariant optical flow estimation network, termed MI-RAFT, to refine the alignment through dense local registration. Extensive experiments demonstrate that the proposed method can handle diverse combinations of commonly used retinal imaging modalities, exhibiting strong modality invariance while outperforming state-of-the-art modality-dependent registration methods.
Read the full abstract

All authors Bo Wen, Nehal Nailesh Mehta, Melanie Tran, Dirk-Uwe Bartsch, William Freeman, Truong Nguyen

Image and Video Processing

Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

Yuhao Huang, Yuanji Zhang, Yuhuan Lu et al. (+3 more)

Assessment of ventriculomegaly (VM) on fetal brain ultrasound relies primarily on measuring lateral ventricular atrial width on standard planes, which is operator-dependent and may not fully reflect the overall ventricular enlargement. Fetal brain MRI provides more reliable volumetric information but is costly and less accessible for routine use. To address these limitations, we propose VIFBA, an ultrasound video-based framework for fetal brain assessment that predicts MRI-derived lateral ventricular volume, classifies VM severity, and identifies potential non-VM fetal brain abnormalities. Our contribution is three-fold. First, we introduce a joint-embedding predictive architecture (JEPA)-inspired tube latent prediction objective that leverages spatio-temporal coherence in ultrasound videos to enhance representation learning. Second, we develop a contrastive cross-modal alignment strategy that transfers structural information from MRI to ultrasound during training, while requiring ultrasound alone at inference. Third, we augment VIFBA with a training-free vision-language model and retrieval augmentation to verify uncertain predictions and identify potential non-VM fetal brain abnormalities. We validated VIFBA on a large dataset comprising 857 cases (3,196 videos) with paired fetal brain ultrasound and MRI examinations. On held-out test data, VIFBA achieved an MAE of 0.5909 mL and Pearson correlation coefficient of 0.9907 for ventricular volume regression, 0.9400 accuracy for VM severity classification, and an F1 score of 0.7764 for multi-abnormality classification, substantially outperforming single-task baselines, video-based strong competitors, and state-of-the-art foundation models. By enabling MRI-informed volumetric assessment from routine ultrasound alone, VIFBA offers a practical and potentially broadly deployable pathway toward accurate and affordable prenatal brain screening.
Read the full abstract

All authors Yuhao Huang, Yuanji Zhang, Yuhuan Lu, Dong Ni, P. Ellen Grant, Davood Karimi

Image and Video Processing

Test-Time Instance Selection for Improved Whole Slide Image Analysis

Quoc Anh Nguyen, Sunhong Park and Jin Tae Kwak

Whole Slide Image (WSI) analysis has been widely studied for cancer diagnosis. Conventionally, a gigapixel WSI is divided into small patches and processed by Multiple Instance Learning (MIL) models. However, existing MIL models typically process all patches, many of which contain redundant or non-informative tissue patterns. Although recent approaches have focused on instance selection to identify discriminative patches and reduce redundancy, these selection modules still require additional training. In this work, we propose Test-Time Instance Selection (TTIS), a training-free, plug-and-play framework that selects compact yet representative patches during inference. TTIS further incorporates a multi-view ensemble strategy to integrate distinct facets of tissue morphology, enhancing robustness. Importantly, TTIS can be seamlessly integrated into existing MIL models without retraining or architectural changes, enabling flexible deployment. Extensive evaluations across multiple benchmarks demonstrate that our approach improves or matches baseline MIL performance across a range of classification and subtyping tasks. Our implementation code is available at https://github.com/QuIIL/TTIS
Read the full abstract

Image and Video Processing

Synthesizing Post-Acetazolamide Cerebral Blood Flow Maps from Baseline MRI in Moyamoya Using 3D Generative AI

Julia Huang, Camila Gonzalez, Rydham Goyal et al. (+5 more)

For patients with Moyamoya disease, impaired cerebrovascular reserve (CVR) is an important hemodynamic criterion for recommending extracranial-to-intracranial bypass surgery. Standard CVR assessment in this cohort uses paired arterial spin labeling (ASL) perfusion MRI acquired before and after acetazolamide (ACZ). When ACZ is contraindicated or avoided, the post-ACZ cerebral blood flow (CBF) map needed for hemodynamic assessment is unavailable. We propose CAE3D, a deterministic 3D conditional autoencoder that synthesizes post-ACZ CBF maps directly from pre-ACZ ASL input. We evaluated CAE3D against ten comparators, including deterministic and diffusion-style 3D baselines, a 2D contextual baseline, and frozen-encoder foundation-model adapters. CAE3D achieved the lowest held-out MAE (0.066), with SSIM 0.80 and PSNR 24.0 dB, and near-zero full-brain mean bias. Its MAE advantage was statistically significant over seven of eight trained-from-scratch baselines, excluding the 2D CAE_2D comparator; its SSIM and PSNR advantages were significant over all eight. Regional delta-CBF predictions compressed the dynamic range in high-response territories. These results establish the retrospective feasibility of post-ACZ CBF synthesis in patients who completed the standard two-scan protocol. Extension to ACZ-contraindicated patients, who were not represented in this cohort, requires external and prospective validation.
Read the full abstract

All authors Julia Huang, Camila Gonzalez, Rydham Goyal, Aja Zou, Sasha Alexander, Michael Moseley, Moss Y. Zhao, Gary K. Steinberg

Image and Video Processing

KHiM-Mamba: Injecting Pathology Knowledge into Mamba via Hidden-State Modulation for Whole Slide Image Analysis

Qixiang Zhang, Yi Li, Tianqi Xiang et al. (+4 more)

Whole slide image analysis is commonly formulated as multiple instance learning (MIL), where instance features are contextually updated and aggregated into a slide representation, a process we term slide encoding dynamics. Recently, selective state-space models (SSM) have emerged as promising MIL architectures due to their long-sequence modeling capability and linear complexity. However, existing SSM-based MIL methods rely solely on visual features during MIL. Meanwhile, in large-scale WSIs, where sparse diagnostically decisive regions are surrounded by abundant irrelevant information, such purely vision-driven selective dynamics can misallocate state updates and readouts, causing the evolving SSM state to accumulate task-irrelevant evidence and dilute critical diagnostic cues over long scan trajectories. In this work, we propose the Knowledge-Aware Hidden-State Modulation architecture (KHiM-Mamba), which innovatively regulates Mamba's core selective state-space mechanism with explicit knowledge priors, steering slide encoding dynamics toward diagnostically meaningful evidence accumulation. Specifically, we redesign the original SSM layer to perform knowledge modulation operations during the evolution of hidden states, thereby guiding what visual evidence is accumulated and retrieved from the hidden state at each encoding step. Furthermore, we additionally introduce a local-adaptive vocabulary retrieval module that uses large language models to assign each patch fine-grained, tissue-specific semantic descriptions, enabling precise modulation across diverse tasks. Experiments on 11 public benchmarks across 4 tasks show that KHiM-Mamba consistently achieves state-of-the-art performance.
Read the full abstract

All authors Qixiang Zhang, Yi Li, Tianqi Xiang, Haonan Wang, Mengjiao Wei, Bo Xu, Xiaomeng Li

Image and Video Processing

A Unified DINOv2-Based Framework for LVEF Estimation, GLS Dysfunction Classification, and Early Cardiotoxicity Prediction

Xiaotong Zhang, Mingyue Cui, Qing Cao et al. (+1 more)

Left ventricular ejection fraction (LVEF) estimation (Task 1), global longitu-dinal strain (GLS)-based dysfunction classification (Task 2), and early cardi-otoxicity prediction (Task 3) provide complementary information for cardio-oncology assessment. LVEF reflects macroscopic ventricular volume chang-es as the clinical standard, whereas GLS captures subtle myocardial defor-mation, indicating subclinical cardiotoxicity before overt LVEF decline. Fur-thermore, predicting cardiotoxicity from baseline echocardiography prior to treatment enables preventive interventions at an early stage. To address these three tasks, we employ a DINOv2-based framework with task-specific adap-tation and prediction heads. Built upon a frozen foundation encoder, the framework incorporates parameter-efficient Low-Rank Adaptation (LoRA) and temporal aggregation to learn task-specialized representations, ensuring robust generalization. Crucially, during inference, it operates in a fully cycle-detection-free and phase-free manner, requiring neither cardiac cycle seg-mentation nor explicit End-Diastolic/End-Systolic (ED/ES) annotations. Ad-ditionally, we introduce an ED/ES-guided 2D/3D hybrid multi-view regres-sion model specifically to optimize Task 1. On a patient-level split containing 1,203 training videos from 237 patients and 300 validation videos from 59 independent patients, the DINOv2-based framework achieved a mean abso-lute error (MAE) of 5.03% for Task 1, an AUC-ROC of 76.48% for Task 2, and an AUC-ROC of 70.26% for Task 3. For Task 1, the specialized ED/ES-guided model further improves performance, achieving an MAE of 4.64%. This framework demonstrates the effectiveness of foundation model repre-sentations across diverse cardio-oncology tasks and the additional benefit of physiology-guided modeling for accurate LVEF estimation.
Read the full abstract

All authors Xiaotong Zhang, Mingyue Cui, Qing Cao, Jingming Xia

Image and Video Processing

Incision trajectory tracing for electrosurgical navigation by CNN-based knife contacting frames extraction method

Yu Chun Wang, Kaixu Chen, Naoto Ienaga et al. (+1 more)

Background and Objective: Image-guided surgical navigation has been actively studied because of its advantage of identifying subsurface targets and critical structures, whereas it requires incision trajectories to update the preoperative three-dimensional model dynamically during the surgery. The novelty of this study is the thermal feature distinguishment of whether the electric tools contacting the tissue by Convolutional Neural Network (CNN), and the extraction of the knife contacting frames, to form incision trajectories which can meet with the requirement during the surgery. Methods: This study firstly verified that CNN can classify the thermal images of electric knife and ultrasonic cutter operations separately, and can raise the accuracy of the incision trajectories derived from the connection of the thermal intensity centroid of the frames predicted by CNN as contacting. Results: Our results obtained by employing the electric knife not only reveal a remarkably high accuracy 97.2 % in CNNs identification, but also can achieve an error reduction as high as more than 2.5 times of the incision trajectory prediction as compared to those proceeded in the conventional method. Besides electric knife, the results obtained by employing another electric tool, ultrasonic cutter, reveal a high accuracy up to 93.7 %. Conclusion: In this study, we ensured the possibility of CNN in distinguishing electric tools contacting with the tissue, and confirmed that the proposed method has not only overcome the problem of missing trajectories which usually occurs in the convolutional long-short term memory method but also achieved a remarkable improvement of the accuracy with less limitation.
Read the full abstract

All authors Yu Chun Wang, Kaixu Chen, Naoto Ienaga, Yoshihiro Kuroda

Image and Video Processing

UMPIRE-Net: Unrolled Magnitude-Phase Regularization Network for Accelerated MRI

Mahdi Saberi, Toygan Kiliç and Mehmet Akçakaya

MRI reconstruction from undersampled k-space measurements is an ill-posed inverse problem. Physics-driven deep learning (PD-DL) methods have shown strong performance for this task by combining the MRI forward model with learned image regularization within algorithm-unrolling frameworks. However, most existing PD-DL methods reconstruct complex-valued images directly, thereby implicitly coupling magnitude and phase within a single learned representation. This coupled regularization may be suboptimal in reconstruction settings where accurate phase modeling plays an important role, such as partial Fourier (PF) imaging, where recovery of the omitted asymmetric k-space measurements depends on the underlying image phase. In such scenarios, explicit modeling of magnitude and phase as separate components may reduce the reliance on externally estimated or predefined phase information. To this end, we propose UMPIRE-Net (Unrolled Magnitude-Phase In REgularization Network), a PD-DL method that introduces separate learned regularizers for magnitude and phase components, together with a novel data-fidelity formulation that enforces measurements consistency. We evaluate UMPIRE-Net for accelerated MRI with PF across different datasets and acceleration factors. Experimental results demonstrate that our proposed method improves reconstruction quality compared with a conventional complex-valued PD-DL baseline, yielding sharper images and reduced artifacts. Code available at: https://github.com/MahdiSaberii/UMPIRE-Net
Read the full abstract

Image and Video Processing

Personalized Digital Semantic Communication for Image Transmission with Vision-Language Models

Nan Li, Li Zhou, Haijun Wang et al. (+3 more)

Semantic communication (SC) enables bandwidth-efficient wireless image transmission, but most existing SC schemes are user-agnostic and ignore receiver-dependent semantics. To address this issue, we propose a personalized digital semantic communication (PDSC) framework that integrates a vision-language model (VLM)-based semantic encoder with a latent diffusion model (LDM)-based semantic decoder. Specifically, the semantic encoder extracts source-aware personalized semantic tokens from both the source image and the receiver's historical interactions. These tokens are vector-quantized into discrete semantic indices and further encoded into a compact fixed-length bitstream, enabling compatibility with digital transmission. At the receiver, the semantic decoder reconstructs a personalized image conditioned on the recovered semantic tokens. Furthermore, we formulate a capacity-constrained personalized semantic rate-distortion problem and introduce a semantic distortion metric that jointly characterizes source-semantic fidelity and user-preference alignment. Experiments show that PDSC achieves superior source-semantic consistency and personalization over state-of-the-art SC baselines, including CDDM and MoS, under bandwidth-limited wireless transmission.
Read the full abstract

All authors Nan Li, Li Zhou, Haijun Wang, Jun Xiong, Haitao Zhao, Jibo Wei

Image and Video Processing

A Subjective Study on a New Sharpness Informed Class of Metrics

Uditangshu Aurangabadkar, Vibhoothi Vibhoothi, Darren Ramsook et al. (+1 more)

Perceptual loss functions in Deep Neural Network (DNN) deblurring architectures improve the overall quality of restored images. However, few focus on explicitly targeting sharpness in the restorations. We conduct a subjective study of models trained with and without losses which explicitly target sharpness using a four-protocol approach, exploring preferred sharpness levels and effects on image quality. We introduce a novel dataset of images with uniform sharpness increments along with Difference Mean Opinion Scores (DMOS). Additionally, we propose a novel class of Sharpness Informed (SI) Image Quality Assessment (IQA) metrics which properly penalize over-sharpening. Our new SI-PSNR metric outperforms all other PSNR variants in terms of correlation statistics on IQA benchmarking datasets. We show that, on average, images restored using a sharpness-aware composite loss are preferred in 67% of binarized comparisons, as opposed to losses that do not explicitly target sharpness.
Read the full abstract

All authors Uditangshu Aurangabadkar, Vibhoothi Vibhoothi, Darren Ramsook, Anil Kokaram

Image and Video Processing

Practical Lossless Volumetric Medical Image Compression via Tri-plane Context Tree Learning

Yuanchao Bai, Yifan Zhao, Kai Wang et al. (+5 more)

Lossless compression of volumetric medical images is of paramount importance for clinical and research applications where data fidelity is essential. Traditional compression methods are often limited in efficiency due to rigid, handcrafted models. Conversely, deep neural network (DNN)-based compression methods, while effective, demand substantial computational resources, hindering deployment in resource-constrained settings. To address these challenges, we propose a novel tri-plane context tree (TCT)-based method for lossless volumetric medical image compression that delivers high performance without relying on DNNs or external training data. To exploit intra-slice and inter-slice redundancies, we introduce a compact tri-plane context representation that decomposes complex 3D context modeling into efficient 2D modeling on three orthogonal planes. By integrating this representation with a context tree framework, we develop an input-specific TCT model employing an adaptive binary tree structure. At each tree node, the model dynamically selects from a suite of tri-plane based predictors and contextual feature extractors, enabling data-adaptive context modeling tailored to local structural characteristics. Instead of offline training, we sample a subset of the input volume to learn the TCT model by optimizing the minimum description length (MDL) through iterative construction and pruning. With the learned TCT model, each pixel retrieves its corresponding context, computes the prediction residual using the predictor dictated by the context, and performs entropy encoding based on the associated histograms. Experimental results demonstrate that the proposed method achieves compression performance on par with recent DNN-based methods on multiple datasets, while maintaining low computational cost and fast coding speeds, making it highly applicable in practice.
Read the full abstract

All authors Yuanchao Bai, Yifan Zhao, Kai Wang, Yuanbo Du, Jie Cheng, Teng Fang, Xianming Liu, Wen Gao

Image and Video Processing

From crown candidates to neighborhood screening: integrating optical GeoAI and spatial modeling for urban-canopy assessment in Davis, California

Mohammadreza Narimani, Shreyan Mitra and Parastoo Farajpoor

Timely urban-canopy information is essential for linking remote sensing with heat, mobility, and neighborhood planning. We developed an optical GeoAI workflow for Davis, California, using 2022 National Agriculture Imagery Program imagery (0.6 m RGB+NIR). DeepForest generated crown candidates; an NDVI threshold, non-maximum suppression, and box-prompted Segment Anything Model (ViT-B) produced a crown-anchored canopy surface. Analyses used the 25.92 km2 Census TIGER municipal boundary and a 100 m grid. The workflow retained 11,741 candidate crowns and mapped 7.71 km2 of canopy (29.8% of the city). On the identical extent, pixel precision was 0.804, pixel recall was 0.873, and 97.4% of candidate centers agreed with the 2022 USDA/CAL FIRE LiDAR-assisted canopy product (IoU 0.719; Dice 0.837; area recovery 108.5%). Approximately 49% of candidates occurred within 15 m of a road. Canopy was inversely associated with Landsat land-surface temperature (Spearman rho = -0.477; partial rho = -0.551 controlling for built probability), and spatial-lag modeling confirmed clear neighborhood structure. Two transparent attention surfaces combined canopy need with thermal and contextual indicators. The framework provides a reproducible, updateable screening layer that complements structural canopy products and municipal inventories while retaining assumptions, data provenance, and spatial diagnostics for planning interpretation.
Read the full abstract

Thursday 13 August

Image and Video Processing

VLM- and LLM-Driven Multi-Agent System for PET Image Denoising

Boxiao Yu, Savas Ozdemir, Yang Xing et al. (+8 more)

Positron emission tomography (PET) imaging suffers from limited spatial resolution and low signal-to-noise ratio, which can compromise quantitative accuracy and lesion detectability. Deep learning-based denoising methods have demonstrated strong potential for improving PET image quality. However, their practical deployment in real-world settings remains challenging, often requiring multiple specialized models and expert interventions, such as identifying motion-induced misregistration artifacts, estimating noise levels to select an appropriate denoiser, and performing lesion-focused quantitative assessment after denoising. Recent advances in vision-language models (VLMs) for image quality understanding and large language models (LLMs) for contextual reasoning provide new opportunities for automated, decision-driven workflows. Inspired by expert workflows for PET image quality enhancement, we propose an VLM- and LLM-driven multi-agent PET denoising framework that dynamically assesses image quality and lesion status, autonomously selects optimal denoising models and parameters, and enables closed-loop feedback with rollback mechanisms. Experiments were conducted on Siemens Biograph Vision Quadra PET/CT data with 1/20 and 1/50 low-dose settings. Individual module evaluations demonstrated the reliability of the agentic components, while the complete framework achieved higher PSNR and SSIM than UNet, GAN, and DDPM baselines at both dose levels. These preliminary results demonstrate the feasibility of using a closed-loop multi-agent framework to adapt PET denoising strategies to different image conditions.
Read the full abstract

All authors Boxiao Yu, Savas Ozdemir, Yang Xing, Fumio Hashimoto, Jiong Wu, Yizhou Chen, Axel Rominger, Ruogu Fang, Kuangyu Shi, Tinsu Pan, Kuang Gong

Image and Video Processing

TRUE-Colon: Exposing a Consistent Transfer Asymmetry in Real-Time Polyp Detection

Sebastian Doerrich, Andreas Franz Schwab, Francesco Di Salvo et al. (+3 more)

Computer-aided detection (CADe) systems for colonoscopy promise to reduce clinical miss rates, yet reliable real-world deployment remains elusive. This translational gap stems in part from a structural flaw in model development: the reliance on curated datasets that under-represent the long negative stretches and procedure-related artifacts characteristic of routine examinations. Training and evaluating architectures strictly on these lesion-centric benchmarks creates an illusion of success, since such benchmarks cannot capture clinically crucial metrics. To expose this gap, we establish TRUE-Colon, a standardized benchmarking protocol that measures key deployment characteristics alongside localization accuracy, and evaluate four real-time architectures (Faster R-CNN, YOLOv8, YOLOv11, RT-DETR) across curated benchmarks (SUN, PICCOLO) and 60 unedited, full-length procedures (REAL-Colon). We observe a consistent transfer asymmetry: models trained strictly on curated clips suffer a severe performance collapse when evaluated on full procedures, whereas procedure-trained models substantially improve rejection of non-polyp content on REAL-Colon, and largely retain their accuracy on curated benchmarks. Beyond transferability, we find that the Transformer detector attains the strongest sensitivity and the earliest, most persistent detections, while the convolutional detectors stay competitive at a higher throughput. Together, these results indicate that both training and benchmarking for deployable CADe should shift from curated, lesion-centric clips toward full-procedure data and deployment-relevant operating points. Source code is available at https://github.com/sdoerrich97/true-colon.
Read the full abstract

All authors Sebastian Doerrich, Andreas Franz Schwab, Francesco Di Salvo, Shyam Nandan Rai, Hanh Huyen My Nguyen, Christian Ledig

Image and Video Processing · Cross-listed from cs.IT

Resource-efficient Semantic Coding Schemes with Manifold-constrained Hyper-connections

Jingwen Fu and Ming Xiao

Semantic communication (SemCom) and task-oriented communication (TOC) can reduce wireless resource consumption by focusing on transmitting semantic or task-relevant information instead of raw messages. In practice, a main challenge is to make transmitting information robust to channel noise and fading while keeping it compact. Existing learning-based transceivers often improve reliability by using larger encoders or higher-dimensional channel features, which increase computation complexity and channel uses. Therefore, optimized system design needs explicit rate control to balance performance and transmitting resources e.g., bandwidth and power. For this purpose, we propose a manifold-constrained hyper-connection (mHC) coding scheme with an entropy bottleneck (EB) for resource-efficient SemCom and TOC over wireless channels. Instead of using a single residual path of existing encoders, the proposed mHC-based semantic encoder applies multiple residual streams and constrains their interaction by doubly stochastic (DS) mixing matrices. The new structure improves representation diversity and training stability with negligible parameter and floating-point overhead. The EB quantizes the channel features and estimates the entropy-coded rate, enabling end-to-end rate–distortion/task optimization under bandwidth and transmit-power constraints. We further show that DS-constrained stream mixing does not increase the differential entropy of the transmitted features. This implies no increase in the ideal EB coding length. Experiments on SemCom and TOC under additive white Gaussian noise (AWGN), Rayleigh fading, Rician fading, and imperfect channel state information (CSI) show that the proposed scheme improves semantic/task performance, communication robustness, and convergence stability over residual and unconstrained HC baselines, while requiring no additional channel uses.
Read the full abstract

Image and Video Processing

SoM-MTM: Synesthesia of Machines (SoM)-Driven Masked Token Model for Cooperative Perception over Packet Loss Channel

Haozhen Li, Rongqing Zhang and Xiang Cheng

To support the large-scale and heterogeneous visual cooperative perception (CP) demands in next-generation mobile networks, intelligent and efficient sensory data transmission is a critical challenge. Under the emerging convergence of communication networks and agentic artificial intelligence (AI), existing research emphasizes utilizing end-to-end neural networks to simplify communication modules, which has shown promising potential for CP. However, these studies are still limited to specific channel models, cooperation modes, and perception tasks, failing to fully leverage powerful visual processing approaches to enhance universality. To address this, we propose a Synesthesia of Machines (SoM)-driven Masked Token Model, referred to as SoM-MTM, as a plug-and-play paradigm for generic visual CP. Inspired by masked image modeling methods such as MAE, it possesses great perceptual context learning capabilities to recover distorted features over packet loss channels, thereby improving information carrying efficiency. Building upon Swin Transformer, SoM-MTM further embeds prior masked information through an External Routing MoE mechanism, maximally repairing and enhancing environmental perception features during cooperation. Comprehensive experimental results confirm that SoM-MTM can consistently enhance perception performances on various tasks, especially strong generalization to unseen scenarios, while maintaining favorable model cost and scalability.
Read the full abstract

Image and Video Processing · Cross-listed from cs.LG

CardioState-JEPA: Delay-Aware Cross-Modal Learning of a Shared Cardiac Representation

Hamza Shafiq, Hung Manh Pham, Bin Zhu et al. (+3 more)

Electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) provide complementary views of the same cardiac cycle, yet existing cardiac foundation models are trained for a single sensing modality, leaving the shared physiology across sensors unexploited. We introduce CardioState-JEPA, a cardiac foundation model to learn a single shared representation jointly across ECG, PPG, and PCG, built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance. To handle the temporal offsets between electrical, mechanical, and hemodynamic events, cross-modal prediction uses a learned delay aligner that matches signals at the corresponding cardiac time. Because synchronized multi-sensor recordings are scarce, CardioState-JEPA first learns within-modality structure from abundant unimodal data and then uses paired data to align modalities in latent cardiac time. Evaluated as a frozen encoder across 25 downstream tasks spanning ECG, PPG, and PCG, our encoder improves average PPG classification by 8.2 AUROC points, PCG murmur detection by 18.8 AUROC points, and ECG classification by 15.5 AUROC points over the best self-supervised signal baseline and matches or exceeds cardiac models trained with privileged clinical text or supervised labels on several ECG benchmarks. These results establish that heterogeneous cardiac signals can mutually supervise a single foundation model of cardiac physiology.
Read the full abstract

All authors Hamza Shafiq, Hung Manh Pham, Bin Zhu, Pan Zhou, Jun Hu, Aaqib Saeed