Computer Vision
NucFuseRank: Dataset Fusion and Performance Ranking for Nuclei Instance Segmentation
Nima Torbati, Anastasia Meshcheryakova, Ramona Woitek et al. (+3 more)
Nuclei instance
segmentation in hematoxylin and eosin (H&E)-stained images plays an important role in automated histological image analysis, with various applications in downstream tasks. While several machine learning and deep learning approaches have been proposed for
nuclei instance
segmentation, most research in this field focuses on developing new
segmentation algorithms and benchmarking them on a limited number of arbitrarily selected public
datasets. In this work, rather than focusing on model development, we focused on the
datasets used for this task. Based on an extensive literature review, we identified manually annotated, publicly available
datasets of H&E-stained images for
nuclei instance
segmentation and standardized them into a unified input and annotation format. Using two state-of-the-art
segmentation models, one based on convolutional neural networks (CNNs) and one based on a hybrid CNN and
vision transformer architecture, we systematically evaluated and ranked these
datasets based on their
nuclei instance
segmentation performance. Furthermore, we proposed a unified test set (NucFuse-test) for fair cross-
dataset evaluation and a unified training set (NucFuse-train) for improved
segmentation performance by merging images from multiple
datasets. By evaluating and ranking the
datasets, performing comprehensive analyses, generating fused
datasets, conducting external validation, and making our implementation publicly available, we provided a new
benchmark for training, testing, and evaluating
nuclei instance
segmentation models on H&E-stained histological images.
Read the full abstract ▾
All authors Nima Torbati, Anastasia Meshcheryakova, Ramona Woitek, Sepideh Hatamikia, Diana Mechtcheriakova, Amirreza Mahbod
Computer Vision
Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
Chuan Qin, Constantin Venhoff, Sonia Joseph et al. (+2 more)
Contrastive
Language-Image Pre-training (CLIP) has become a cornerstone in
vision-
language representation learning, powering diverse downstream tasks and serving as the default
vision backbone in
multimodal large
language models (MLLMs). Despite its success, CLIP's dense and opaque latent representations pose significant interpretability challenges. A common assumption is that interpretability and performance are in tension: enforcing sparsity during training degrades accuracy, motivating recent post-hoc approaches such as Sparse Autoencoders (SAEs). However, these post-hoc approaches often suffer from degraded downstream performance and loss of CLIP's inherent
multimodal capabilities, with most learned features remaining unimodal. We propose a simple yet effective approach that integrates sparsity directly into CLIP training, yielding representations that are both interpretable and performant. Compared to SAEs, our Sparse CLIP representations preserve strong downstream task performance, achieve superior interpretability, and retain
multimodal capabilities. We show that
multimodal sparse features enable straightforward semantic concept
alignment and reveal training dynamics of how cross-modal knowledge emerges. Finally, as a proof of concept, we train a
vision-
language model on sparse CLIP representations that enables interpretable,
vision-based steering capabilities. Our findings challenge conventional wisdom that interpretability requires sacrificing accuracy and demonstrate that interpretability and performance can be co-optimized, offering a promising design principle for future models.
Read the full abstract ▾
All authors Chuan Qin, Constantin Venhoff, Sonia Joseph, Fanyi Xiao, Stefan Scherer
Computer Vision
Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Data
Atik Faysal, Mohammad Rostami, Reihaneh Gh. Roshan et al. (+2 more)
We address the challenge of training
Vision Transformers (ViTs) when labeled data is scarce but unlabeled data is abundant. We propose Semi-Supervised Masked Autoencoder (SSMAE), a framework that jointly optimizes masked image reconstruction and classification using both unlabeled and labeled samples with dynamically selected pseudo-labels. SSMAE introduces a validation-driven gating mechanism that activates pseudo-labeling only after the model achieves reliable, high-confidence predictions that are consistent across both weakly and strongly augmented views of the same image, reducing confirmation bias. On CIFAR-10 and CIFAR-100, SSMAE consistently outperforms supervised ViT and fine-tuned MAE, with the largest gains in low-label regimes (+9.24% over ViT on CIFAR-10 with 10% labels). Our results demonstrate that when pseudo-labels are introduced is as important as how they are generated for data-efficient
transformer training. Codes are available at https://github.com/atik666/ssmae.
Read the full abstract ▾
All authors Atik Faysal, Mohammad Rostami, Reihaneh Gh. Roshan, Nikhil Muralidhar, Huaxia Wang
Computer Vision
DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation
Zhen Yao, Xin Li, Taotao Jing et al. (+2 more)
Open-vocabulary semantic
segmentation aims to assign labels to every pixel in an image based on text labels. Existing approaches typically utilize
vision-
language models (VLMs), such as CLIP, for dense prediction. However, VLMs, pre-trained on image-text pairs, are biased toward salient, object-centric regions and exhibit two critical limitations when adapted to
segmentation: (i) Foreground Bias, which tends to ignore background regions, and (ii) Limited Spatial Localization, resulting in blurred object boundaries. To address these limitations, we introduce DiSa, a novel saliency-aware foreground-background disentangled framework. By explicitly incorporating saliency cues in our designed Saliency-aware Disentanglement Module (SDM), DiSa separately models foreground and background ensemble features in a divide-and-conquer manner. Additionally, we propose a Hierarchical Refinement Module (HRM) that leverages pixel-wise spatial contexts and enables channel-wise feature refinement through multi-level updates. Extensive experiments on six
benchmarks demonstrate that DiSa consistently outperforms state-of-the-art methods.
Read the full abstract ▾
All authors Zhen Yao, Xin Li, Taotao Jing, Shuai Zhang, Mooi Choo Chuah
Computer Vision · Cross-listed from cs.AR
Primitive-Driven Acceleration of Hyperdimensional Computing for Real-Time Image Classification
Dhruv Parikh, Jebacyril Arockiaraj and Viktor Prasanna
Hyperdimensional Computing (HDC) represents data using extremely high-dimensional, low-precision vectors, termed hypervectors (HVs), and performs learning and inference through lightweight, noise-tolerant operations. However, the high dimensionality, sparsity, and repeated data movement involved in HDC make these computations difficult to accelerate efficiently on conventional processors. As a result, executing core HDC operations: binding, permutation, bundling, and similarity search: on CPUs or GPUs often leads to suboptimal utilization,
memory bottlenecks, and limits on real-time performance. In this paper, our contributions are two-fold. First, we develop an image-encoding
algorithm that, similar in spirit to convolutional neural networks, maps local image patches to hypervectors enriched with spatial information. These patch-level hypervectors are then merged into a global representation using the fundamental HDC operations, enabling spatially sensitive and robust image encoding. This encoder achieves 95.67% accuracy on MNIST and 85.14% on Fashion-MNIST, outperforming prior HDC-based image encoders. Second, we design an end-to-end accelerator that implements these compute operations on an FPGA through a pipelined architecture that exploits parallelism both across the hypervector dimensionality and across the set of image patches. Our Alveo U280 implementation delivers 0.09ms inference latency, achieving up to 1300x and 60x speedup over state-of-the-art CPU and GPU baselines, respectively.
Read the full abstract ▾
Computer Vision
Size Matters: Reconstructing Real-Scale 3D Models from Monocular Images for Food Portion Estimation
Gautham Vinod, Bruce Coburn, Siddeshwar Raghavan et al. (+2 more)
The rise of chronic diseases related to diet, such as obesity and diabetes, emphasizes the need for accurate monitoring of food intake. While AI-driven dietary assessment has made strides in recent years, the ill-posed nature of recovering size (portion) information from
monocular images for accurate estimation of “how much did you eat?” is a pressing challenge. Some 3D reconstruction methods have achieved impressive
geometric reconstruction but fail to recover the crucial real-world scale of the reconstructed object, limiting its usage in precision nutrition. In this paper, we bridge the gap between 3D computer
vision and digital health by proposing a method that recovers a true-to-scale 3D reconstructed object from a
monocular image. Our approach leverages rich visual features extracted from models trained on large-scale
datasets to estimate the scale of the reconstructed object. This learned scale enables us to convert single-view 3D reconstructions into true-to-life, physically meaningful models. Extensive experiments and ablation studies on two publicly available
datasets show that our method consistently outperforms existing techniques, achieving nearly a 30% reduction in mean absolute volume-estimation error, showcasing its potential to enhance the domain of precision nutrition. Code: https://gitlab.com/viper-purdue/size-matters
Read the full abstract ▾
All authors Gautham Vinod, Bruce Coburn, Siddeshwar Raghavan, Jiangpeng He, Fengqing Zhu
Computer Vision · Cross-listed from cs.LG
MeanCache: From Instantaneous to Average Velocity for Accelerating Flow Matching Inference
Huanlin Gao, Ping Chen, Fuyuan Shi et al. (+11 more)
We present MeanCache, a training-free caching framework for efficient Flow Matching inference. Existing caching methods reduce redundant computation but typically rely on instantaneous velocity information (e.g., feature caching), which often leads to severe trajectory deviations and error accumulation under high acceleration ratios. MeanCache introduces an average-velocity perspective: by leveraging cached Jacobian–vector products (JVP) to construct interval average velocities from instantaneous velocities, it effectively mitigates local error accumulation. To further improve cache timing and JVP reuse stability, we develop a trajectory-stability scheduling strategy as a practical tool, employing a Peak-Suppressed Shortest Path under budget constraints to determine the schedule. Experiments on FLUX.1, Qwen-Image, and HunyuanVideo demonstrate that MeanCache achieves 4.12X and 4.56X and 3.59X acceleration, respectively, while consistently outperforming state-of-the-art caching baselines in generation quality. We believe this simple yet effective approach provides a new perspective for Flow Matching inference and will inspire further exploration of stability-driven acceleration in commercial-scale generative models.
Read the full abstract ▾
All authors Huanlin Gao, Ping Chen, Fuyuan Shi, Ruijia Wu, Li YanTao, Qiang Hui, Yuren You, Ting Lu, Chao Tan, Shaoan Zhao, Zhaoxiang Liu, Fang Zhao, Kai Wang, Shiguo Lian
Computer Vision
DuwatBench: Bridging Language and Visual Heritage through an Arabic Calligraphy Benchmark for Multimodal Understanding
Shubham Patle, Sara Ghaboura, Hania Tariq et al. (+4 more)
Arabic calligraphy represents one of the richest visual traditions of the Arabic
language, blending linguistic meaning with artistic form. Although
multimodal models have advanced across
languages, their ability to process Arabic script, especially in artistic and stylized calligraphic forms, remains largely unexplored. To address this gap, we present DuwatBench, a
benchmark of 1,272 curated samples containing about 1,475 unique words across six classical and modern calligraphic styles, each paired with sentence-level detection annotations. The
dataset reflects real-world challenges in Arabic writing, such as complex stroke patterns, dense ligatures, and stylistic variations that often challenge standard text recognition systems. Using DuwatBench, we evaluated 13 leading Arabic and
multilingual multimodal models and showed that while they perform well on clean text, they struggle with calligraphic variation, artistic distortions, and precise visual-text
alignment. By publicly releasing DuwatBench and its annotations, we aim to advance culturally grounded
multimodal research, foster fair inclusion of the Arabic
language and visual heritage in AI systems, and support continued progress in this area. Our
dataset (https://huggingface.co/
datasets/MBZUAI/DuwatBench) and evaluation suit (https://github.com/mbzuai-oryx/DuwatBench) are publicly available.
Read the full abstract ▾
All authors Shubham Patle, Sara Ghaboura, Hania Tariq, Mohammad Usman Khan, Omkar Thawakar, Rao Muhammad Anwer, Salman Khan
Computer Vision
VGGT-SLAM 2.0: Real-time Dense Feed-forward Scene Reconstruction
Dominic Maggio and Luca Carlone
We present VGGT-SLAM 2.0, a real-time RGB feed-forward SLAM system which substantially improves upon VGGT-SLAM for incrementally aligning submaps created from VGGT. Firstly, we remove high-dimensional 15-degree-of-freedom drift and planar degeneracy from VGGT-SLAM by creating a new factor graph design while still addressing the reconstruction ambiguity of VGGT given unknown camera intrinsics. Secondly, by studying the
attention layers of VGGT, we show that one of the layers is well suited to assist in image
retrieval verification for free without additional training, which enables both rejecting false positive matches and allows for completing more loop closures. Finally, we conduct a suite of experiments which includes showing VGGT-SLAM 2.0 can easily be adapted for open-set object detection and demonstrating real-time performance while running online onboard a ground
robot using a Jetson Thor. We test in environments ranging from cluttered indoor apartments and office scenes to a 4,200 square foot barn, and we also demonstrate VGGT-SLAM 2.0 achieves the highest accuracy on the TUM
dataset with about 23 percent less pose error than VGGT-SLAM. Code will be released upon publication.
Read the full abstract ▾
Computer Vision
SONIC: Spectral Oriented Neural Invariant Convolutions
Gijs Joppe Moens, Regina Beets-Tan and Eduardo H. P. Pooch
Convolutional Neural Networks (CNNs) rely on fixed-size kernels scanning local patches, which limits their ability to capture global context or long-range dependencies without very deep architectures.
Vision Transformers (ViTs), in turn, provide global connectivity but lack spatial inductive bias, depend on explicit positional encodings, and remain tied to the initial patch size. Bridging these limitations requires a representation that is both structured and global. We introduce SONIC (Spectral Oriented Neural Invariant Convolutions), a continuous spectral parameterisation that models convolutional operators using a small set of shared, orientation-selective components. These components define smooth responses across the full frequency domain, yielding global receptive fields and filters that adapt naturally across resolutions. Across synthetic
benchmarks, large-scale image classification, and 3D medical
datasets, SONIC shows improved robustness to
geometric transformations, noise, and resolution shifts, and matches or exceeds convolutional,
attention-based, and prior spectral architectures with an order of magnitude fewer parameters. These results demonstrate that continuous, orientation-aware spectral parameterisations provide a principled and scalable alternative to conventional spatial and spectral operators.
Read the full abstract ▾
Computer Vision
EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning
Binzhu Xie, Shi Qiu, Sicheng Zhang et al. (+5 more)
Robust 3D hand reconstruction in
egocentric vision is challenging due to depth ambiguity, self-occlusion, and complex hand-object interactions. Prior methods mitigate these issues by scaling training data or adding auxiliary cues, but they often struggle in unseen contexts. We present EgoHandICL, the first in-context learning (ICL) framework for 3D hand reconstruction that improves semantic
alignment, visual consistency, and robustness under challenging
egocentric conditions. EgoHandICL introduces complementary exemplar
retrieval guided by
vision-
language models (VLMs), an ICL-tailored tokenizer for
multimodal context, and a masked autoencoder (MAE)-based architecture trained with hand-guided
geometric and perceptual objectives. Experiments on ARCTIC and EgoExo4D show consistent gains over state-of-the-art methods. We also demonstrate real-world generalization and improve EgoVLM hand-object interaction
reasoning by using reconstructed hands as visual
prompts. Code and data: https://github.com/Nicous20/EgoHandICL
Read the full abstract ▾
All authors Binzhu Xie, Shi Qiu, Sicheng Zhang, Yinqiao Wang, Hao Xu, Muzammal Naseer, Chi-Wing Fu, Pheng-Ann Heng
Computer Vision
Haya Alyoussef, Ahmad Bdeir, Diego Coello de Portugal Mecke et al. (+3 more)
Data across modalities such as images, text, and graphs often contains hierarchical and relational structures, which are challenging to model within Euclidean geometry.
Hyperbolic geometry provides a natural framework for representing such structures. Building on this property, this work introduces HexFormer, a
hyperbolic vision transformer for image classification that incorporates exponential map aggregation within its
attention mechanism. Two designs are explored: a
hyperbolic ViT (HexFormer) and a hybrid variant (HexFormer-Hybrid) that combines a
hyperbolic encoder with an Euclidean linear classification head. HexFormer incorporates a novel
attention mechanism based on exponential map aggregation, which yields more accurate and stable aggregated representations than standard centroid based averaging, showing that simpler approaches retain competitive merit. Experiments across multiple
datasets demonstrate consistent performance improvements over Euclidean baselines and prior
hyperbolic ViTs, with the hybrid variant achieving the strongest overall results. Additionally, this study provides an analysis of gradient stability in
hyperbolic transformers. The results reveal that
hyperbolic models exhibit more stable gradients and reduced sensitivity to warmup strategies compared to Euclidean architectures, highlighting their robustness and efficiency in training. Overall, these findings indicate that
hyperbolic geometry can enhance
vision transformer architectures by improving gradient stability and accuracy. In addition, relatively simple mechanisms such as exponential map aggregation can provide strong practical benefits.
Read the full abstract ▾
All authors Haya Alyoussef, Ahmad Bdeir, Diego Coello de Portugal Mecke, Tom Hanika, Niels Landwehr, Lars Schmidt-Thieme
Computer Vision
Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering
Kun Li, Michael Ying Yang and Sami Sebastian Brandt
Audio–Visual Question Answering (AVQA) is a challenging
multimodal task that requires jointly
reasoning over audio, visual, and textual information in a given
video to answer natural
language questions. Inspired by recent advances in
Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to
video analysis, and the textual question information contributes minimally to audio–visual understanding, as it is typically integrated only in the final stages of
reasoning. To address these limitations, we propose a novel Query-guided Spatial–Temporal–Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio–visual understanding. Furthermore, we introduce a Query Context
Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on two AVQA
benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA,
Video QA, and AVQA approaches. The code is released under https://github.com/lik1996/QSTar.
Read the full abstract ▾
Computer Vision
Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision
Zhixiang Wei, Yi Li, Zhehan Kan et al. (+38 more)
Despite the significant advancements represented by
Vision-
Language Models (VLMs), current architectures often exhibit limitations in retaining fine-grained visual information, leading to coarse-grained
multimodal comprehension. We attribute this deficiency to a suboptimal training paradigm inherent in prevailing VLMs, which exhibits a text-dominant optimization bias by conceptualizing visual signals merely as passive conditional inputs rather than supervisory targets. To mitigate this, we introduce Youtu-VL, a framework leveraging the
Vision-
Language Unified Autoregressive Supervision (VLUAS) paradigm, which fundamentally shifts the optimization objective from “
vision-as-input” to “
vision-as-target.” By integrating visual
tokens directly into the prediction stream, Youtu-VL applies unified autoregressive supervision to both visual details and linguistic content. Furthermore, we extend this paradigm to encompass
vision-centric tasks, enabling a standard VLM to perform
vision-centric tasks without task-specific additions. Extensive empirical evaluations demonstrate that Youtu-VL achieves competitive performance on both general
multimodal tasks and
vision-centric tasks, establishing a robust foundation for the development of comprehensive generalist visual
agents.
Read the full abstract ▾
All authors Zhixiang Wei, Yi Li, Zhehan Kan, Xinghua Jiang, Zuwei Long, Shifeng Liu, Hongze Shen, Wei Liu, Xiaoyu Tan, Haojia Lin, Yubo Zhu, Qianyu Li, Di Yin, Haoyu Cao, Weibo Gu, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, Yunsheng Wu, Mingkong Tang, Shuangyin Liu, Lexiang Tang, Haodong Lin, Junru Lu, Jiarui Qin, Lingfeng Qiao, Ruizhi Qiao, Bo Ke, Jianfeng He, Ke Li, Yangning Li, Yunhang Shen, Mengdan Zhang, Peixian Chen, Kun Yin, Bing Liu, Yunfei Wu, Huang Chen, Zhongpeng Cai, Xiaotian Li
Computer Vision
Diffusion for De-Occlusion: Accessory-Aware Diffusion Inpainting for Robust Ear Biometric Recognition
Deeksha Arun, Kevin W. Bowyer and Patrick Flynn
Ear occlusions (arising from the presence of ear accessories such as earrings and earphones) can negatively impact performance in ear-based biometric recognition systems, especially in unconstrained imaging circumstances. In this study, we assess the effectiveness of a
diffusion-based ear inpainting technique as a pre-processing aid to mitigate the issues of ear accessory occlusions in
transformer-based ear recognition systems. Given an input ear image and an automatically derived accessory mask, the inpainting model reconstructs clean and anatomically plausible ear regions by synthesizing missing pixels while preserving local
geometric coherence along key ear structures, including the helix, antihelix, concha, and lobule. We evaluate the effectiveness of this pre-processing aid in
transformer-based recognition systems for several
vision transformer models and different patch sizes for a range of
benchmark datasets. Experiments show that
diffusion-based inpainting can be a useful pre-processing aid to alleviate ear accessory occlusions to improve overall recognition performance.
Read the full abstract ▾
Computer Vision
GeoDiff3D: Self-Supervised 3D Scene Generation with Geometry-Constrained 2D Diffusion Guidance
Haozhi Zhu, Miaomiao Zhao, Dingyao Liu et al. (+4 more)
3D scene generation is a core technology for gaming, film/VFX, and VR/AR. Growing demand for rapid iteration, high-fidelity detail, and accessible content creation has further increased interest in this area. Existing methods broadly follow two paradigms - indirect 2D-to-3D reconstruction and direct 3D generation - but both are limited by weak structural modeling and heavy reliance on large-scale ground-truth supervision, often producing structural artifacts,
geometric inconsistencies, and degraded high-frequency details in complex scenes. We propose GeoDiff3D, an efficient self-supervised framework that uses coarse geometry as a structural anchor and a geometry-constrained 2D
diffusion model to provide texture-rich reference images. Importantly, GeoDiff3D does not require strict multi-view consistency of the
diffusion-generated references and remains robust to the resulting noisy, inconsistent guidance. We further introduce voxel-aligned 3D feature aggregation and dual self-supervision to maintain scene coherence and fine details while substantially reducing dependence on labeled data. GeoDiff3D also trains with low computational cost and enables fast, high-quality 3D scene generation. Extensive experiments on challenging scenes show improved generalization and generation quality over existing baselines, offering a practical solution for accessible and efficient 3D scene construction.
Read the full abstract ▾
All authors Haozhi Zhu, Miaomiao Zhao, Dingyao Liu, Runze Tian, Yan Zhang, Jie Guo, Fenggen Yu
Computer Vision
PaW-ViT: A Patch-based Warping Vision Transformer for Robust Ear Verification
Deeksha Arun, Kevin W. Bowyer and Patrick Flynn
The rectangular
tokens common to
vision transformer methods for visual recognition can strongly affect performance of these methods due to incorporation of information outside the objects to be recognized. This paper introduces PaW-ViT, Patch-based Warping
Vision Transformer, a preprocessing approach rooted in anatomical knowledge that normalizes ear images to enhance the efficacy of ViT. By accurately aligning
token boundaries to detected ear feature boundaries, PaW-ViT obtains greater robustness to shape, size, and pose variation. By aligning feature boundaries to natural ear
curvature, it produces more consistent
token representations for various morphologies. Experiments confirm the effectiveness of PaW-ViT on various ViT models (ViT-T, ViT-S, ViT-B, ViT-L) and yield reasonable
alignment robustness to variation in shape, size, and pose. Our work aims to solve the disconnect between ear biometric morphological variation and
transformer architecture positional sensitivity, presenting a possible avenue for authentication schemes.
Read the full abstract ▾
Computer Vision
WaterClear-GS: Optical-Aware Gaussian Splatting for Underwater Reconstruction and Restoration
Xinrui Zhang, Yufeng Wang, Zesheng Wang et al. (+3 more)
Underwater 3D reconstruction and appearance restoration remain challenging due to the complex optical properties of water, such as wavelength-dependent attenuation and scattering. Existing Neural Radiance Fields (NeRF)-based approaches often suffer from slow rendering and limited practicality, while vanilla 3D Gaussian Splatting (3DGS) lacks an effective mechanism to account for
underwater degradation. To address this, we propose WaterClear-GS, a physics-informed reformulation of
underwater Gaussian splatting that models degradation as intrinsic Gaussian attributes rather than external medium fields. Furthermore, we adopt a dual-branch optimization strategy that preserves
underwater photometric consistency while encouraging a practical separation between restoration-oriented latent clean appearance and degradation. This strategy is enhanced by depth-guided geometry regularization and perception-driven image supervision, together with exposure constraints, spatially adaptive regularization, and physics-informed spectral regularization, which collectively promote spatial coherence and plausible visual appearance. Extensive experiments on standard
benchmarks and our collected
dataset demonstrate that WaterClear-GS achieves strong performance on both novel view synthesis (NVS) and
underwater image restoration (UIR) tasks, while maintaining 160+ FPS real-time rendering. The code will be available at https://sherii888.github.io/WaterClear-GS/.
Read the full abstract ▾
All authors Xinrui Zhang, Yufeng Wang, Zesheng Wang, Dacheng Qi, Wenrui Ding, Shuangkang Fang
Computer Vision · Cross-listed from cs.MM
Benchmarking Multimodal Large Language Models for Missing Modality Completion in Product Catalogues
Junchen Fu, Wenhao Deng, Kaiwen Zheng et al. (+5 more)
Missing-modality information on e-commerce platforms, such as absent product images or textual descriptions, often arises from annotation errors or incomplete metadata, impairing both product presentation and downstream applications such as recommendation systems. Motivated by the
multimodal generative capabilities of recent
Multimodal Large
Language Models (MLLMs), this work investigates a fundamental yet underexplored question: can MLLMs generate missing modalities for products in e-commerce scenarios? We propose the Missing Modality Product Completion
Benchmark (MMPCBench), which consists of two sub-
benchmarks: a Content Quality Completion
Benchmark and a Recommendation
Benchmark. We further evaluate six state-of-the-art MLLMs from the Qwen2.5-VL and Gemma-3 model families across nine real-world e-commerce categories, focusing on image-to-text and text-to-image completion tasks. Experimental results show that while MLLMs can capture high-level semantics, they struggle with fine-grained word-level and pixel- or patch-level
alignment. In addition, performance varies substantially across product categories and model scales, and we observe no trivial correlation between model size and performance, in contrast to trends commonly reported in mainstream
benchmarks. We also explore Group Relative Policy Optimization (GRPO) to better align MLLMs with this task. GRPO improves image-to-text completion but does not yield gains for text-to-image completion. Overall, these findings expose the limitations of current MLLMs in real-world cross-modal generation and represent an early step toward more effective missing-modality product completion.
Read the full abstract ▾
All authors Junchen Fu, Wenhao Deng, Kaiwen Zheng, Ioannis Arapakis, Yu Ye, Yongxin Ni, Joemon M. Jose, Xuri Ge
Computer Vision · Cross-listed from eess.IV
Interpretable and backpropagation-free Green Learning for efficient multi-task echocardiographic segmentation and classification
Jyun-Ping Kao, Jiaxin Yang, C.-C. Jay Kuo et al. (+1 more)
Echocardiography is a cornerstone for managing heart failure (HF), with Left Ventricular Ejection Fraction (LVEF) being a critical metric for guiding therapy. However, manual LVEF assessment suffers from high inter-observer variability, while existing Deep Learning (DL) models are often computationally intensive and data-hungry "black boxes" that impede clinical trust and adoption. Here, we propose a backpropagation-free multi-task Green Learning (MTGL) framework that performs simultaneous Left Ventricle (LV)
segmentation and LVEF classification. Our framework integrates an unsupervised VoxelHop encoder for hierarchical spatio-temporal feature extraction with a multi-level regression decoder and an XG-Boost classifier. On the EchoNet-Dynamic
dataset, our MTGL model achieves state-of-the-art classification and
segmentation performance, attaining a classification accuracy of 94.3% and a Dice Similarity Coefficient (DSC) of 0.912, significantly outperforming several advanced 3D DL models. Crucially, our model achieves this with over an order of magnitude fewer parameters, demonstrating exceptional computational efficiency. This work demonstrates that the GL paradigm can deliver highly accurate, efficient, and interpretable solutions for complex medical image analysis, paving the way for more sustainable and trustworthy artificial intelligence in clinical practice.
Read the full abstract ▾
All authors Jyun-Ping Kao, Jiaxin Yang, C.-C. Jay Kuo, Jonghye Woo
Computer Vision
DiffStyle3D: Consistent 3D Gaussian Stylization via Attention Optimization
Yitong Yang, Xuexin Liu, Yinglin Wang et al. (+4 more)
3D style transfer enables the creation of visually expressive 3D content, enriching the visual appearance of 3D scenes and objects. However, existing VGG- and CLIP-based methods struggle to model multi-view consistency within the model itself, while
diffusion-based approaches can capture such consistency but rely on denoising directions, leading to unstable training. To address these limitations, we propose DiffStyle3D, a novel
diffusion-based paradigm for 3DGS style transfer that directly optimizes in the latent space. Specifically, we introduce an
Attention-Aware Loss that performs style transfer by aligning style features in the self-
attention space, while preserving original content through content feature
alignment. Inspired by the
geometric invariance of 3D stylization, we propose a Geometry-Guided Multi-View Consistency method that integrates
geometric information into self-
attention to enable cross-view correspondence modeling. Based on
geometric information, we additionally construct a geometry-aware mask to prevent redundant optimization in overlapping regions across views, which further improves multi-view consistency. Extensive experiments show that DiffStyle3D outperforms state-of-the-art methods, achieving higher stylization quality and visual realism.
Read the full abstract ▾
All authors Yitong Yang, Xuexin Liu, Yinglin Wang, Jing Wang, Hao Dou, Changshuo Wang, Shuting He
Computer Vision
Self-Supervised Weight Templates for Scalable Vision Model Initialization
Yucheng Xie, Fu Feng, Ruixiao Shi et al. (+3 more)
The increasing scale and complexity of modern model parameters underscore the importance of pre-trained models. However, deployment often demands architectures of varying sizes, exposing limitations of conventional pre-training and fine-tuning. To address this, we propose SWEET, a self-supervised framework that performs constraint-based pre-training to enable scalable initialization in
vision tasks. Instead of pre-training a fixed-size model, we learn a shared weight template and size-specific weight scalers under Tucker-based factorization, which promotes modularity and supports flexible adaptation to architectures with varying depths and widths. Target models are subsequently initialized by composing and reweighting the template through lightweight weight scalers, whose parameters can be efficiently learned from minimal training data. To further enhance flexibility in width expansion, we introduce width-wise
stochastic scaling, which regularizes the template along width-related dimensions and encourages robust, width-invariant representations for improved cross-width generalization. Extensive experiments on classification, detection,
segmentation and generation tasks demonstrate the state-of-the-art performance of SWEET for initializing variable-sized
vision models.
Read the full abstract ▾
All authors Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang, Yong Rui, Xin Geng
Computer Vision
DSVM-UNet : Enhancing VM-UNet with Dual Self-distillation for Medical Image Segmentation
Renrong Shao, Dongyang Li, Dong Xia et al. (+4 more)
Vision Mamba models have been extensively researched in various fields, which address the limitations of previous models by effectively managing long-range dependencies with a linear-time overhead. Several prospective studies have further designed
Vision Mamba based on UNet(VM-UNet) for medical image
segmentation. These approaches primarily focus on optimizing architectural designs by creating more complex structures to enhance the model's ability to perceive semantic features. In this paper, we propose a simple yet effective approach to improve the model by Dual Self-
distillation for VM-UNet (DSVM-UNet) without any complex architectural designs. To achieve this goal, we develop double self-
distillation methods to align the features at both the global and local levels. Extensive experiments conducted on the ISIC2017, ISIC2018, and Synapse
benchmarks demonstrate that our approach achieves state-of-the-art performance while maintaining computational efficiency. Code is available at https://github.com/RoryShao/DSVM-UNet.git.
Read the full abstract ▾
All authors Renrong Shao, Dongyang Li, Dong Xia, Lin Shao, Jiangdong Lu, Fen Zheng, Lulu Zhang
Computer Vision
Ziyue Wang, Sheng Jin, Zhongrong Zuo et al. (+5 more)
Reinforcement learning (RL) has shown strong potential for enhancing
reasoning in
multimodal large
language models, yet existing
video reasoning methods often rely on coarse sequence-level rewards or single-factor
token selection, neglecting fine-grained links among visual inputs, temporal dynamics, and linguistic outputs, limiting both accuracy and interpretability. We propose
Video-KTR, a modality-aware policy shaping framework that performs selective,
token-level RL by combining three attribution signals: (1) visual-aware
tokens identified via counterfactual masking to reveal perceptual dependence; (2) temporal-aware
tokens detected through frame shuffling to expose temporal sensitivity; and (3) high-
entropy tokens signaling predictive uncertainty. By reinforcing only these key
tokens,
Video-KTR focuses learning on semantically informative, modality-sensitive content while filtering out low-value
tokens. Across five challenging
benchmarks,
Video-KTR achieves state-of-the-art or highly competitive results, achieving 42.7% on
Video-Holmes (surpassing GPT-4o) with consistent gains on both
reasoning and general
video understanding tasks. Ablation studies verify the complementary roles of the attribution signals and the robustness of targeted
token-level updates. Overall,
Video-KTR improves accuracy and interpretability, offering a simple, drop-in extension to RL for complex
video reasoning. Our code and models are available at https://github.com/zywang0104/
Video-KTR.
Read the full abstract ▾
All authors Ziyue Wang, Sheng Jin, Zhongrong Zuo, Jiawei Wu, Han Qiu, Qi She, Hao Zhang, Xudong Jiang
Computer Vision
SharpNet: Enhancing MLPs to Represent Functions with Controlled Non-differentiability
Hanting Niu, Junkai Deng, Fei Hou et al. (+2 more)
Multi-layer perceptrons (MLPs) are a standard tool for learning and function approximation, but they inherently produce globally smooth outputs. Consequently, they struggle to represent functions that are continuous yet intentionally non-differentiable (i.e., functions with prescribed $C^0$ sharp features) without ad hoc post-processing. We present SharpNet, a modified MLP architecture that encodes user-specified sharp features by augmenting the network with an auxiliary feature function defined as the solution to Poisson's equation with jump Neumann boundary conditions. This feature function is evaluated via an efficient local integral and is fully differentiable with respect to the feature locations, allowing us to jointly optimize both the feature locations and the MLP parameters to recover the target function or geometry. This construction provides precise control over where non-differentiability occurs, enforcing the desired $C^0$ behavior at feature locations while preserving smoothness elsewhere. We validate SharpNet on 2D problems and 3D CAD reconstruction, and compare it with several state-of-the-art baselines. In both settings, SharpNet accurately recovers sharp edges and corners while remaining smooth away from them, whereas existing methods tend to blur gradient discontinuities. Qualitative and quantitative results demonstrate the effectiveness of our approach. Our project page, code and models are publicly available at https://sharpnettech.github.io.
Read the full abstract ▾
All authors Hanting Niu, Junkai Deng, Fei Hou, Wencheng Wang, Ying He
Computer Vision
A new Image Similarity Metric for a Perceptual and Transparent Geometric and Chromatic Assessment
Antonio Di Marino, Vincenzo Bevilacqua, Emanuel Di Nardo et al. (+3 more)
In the literature, several studies have shown that state-of-the-art image similarity metrics are not perceptual metrics; moreover, they have difficulty evaluating images, especially when texture distortion is also present. In this work, we propose a new perceptual metric composed of two terms. The first term evaluates the dissimilarity between the textures of two images using Earth Mover's Distance. The second term evaluates the chromatic dissimilarity between two images in the Oklab perceptual color space. We evaluated the performance of our metric on a non-traditional
dataset, called Berkeley-Adobe Perceptual Patch Similarity, which contains a wide range of complex distortions in shapes and colors. We have shown that our metric outperforms the state of the art, especially when images contain shape distortions, confirming also its greater perceptiveness. Furthermore, although deep black-box metrics could be very accurate, they only provide similarity scores between two images, without explaining their main differences and similarities. Our metric, on the other hand, provides visual explanations to support the calculated score, making the similarity assessment transparent and justified.
Read the full abstract ▾
All authors Antonio Di Marino, Vincenzo Bevilacqua, Emanuel Di Nardo, Angelo Ciaramella, Ivanoe De Falco, Giovanna Sannino
Computer Vision
KeepLoRA: Continual Learning with Residual Gradient Adaptation
Mao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang et al. (+3 more)
Continual learning for pre-trained
vision-
language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.
Read the full abstract ▾
All authors Mao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang, Yuanyu Wan, Tong Wei, Min-Ling Zhang
Computer Vision
Focus on What Really Matters in Low-Altitude Governance: A Management-Centric Multi-Modal Benchmark with Implicitly Coordinated Vision-Language Reasoning Framework
Hao Chang, Zhihui Wang, Lingxiang Wu et al. (+5 more)
Low-altitude
vision systems are becoming a critical infrastructure for smart city governance. However, existing object-centric perception paradigms and loosely coupled
vision-
language pipelines are still difficult to support management-oriented anomaly understanding required in real-world urban governance. To bridge this gap, we introduce GovLA-10K, the first management-oriented multi-modal
benchmark for low-altitude intelligence, along with GovLA-Reasoner, a unified
vision-
language reasoning framework tailored for governance-aware aerial perception. Unlike existing studies that aim to exhaustively annotate all visible objects, GovLA-10K is deliberately designed around functionally salient targets that directly correspond to practical management needs, and further provides actionable management suggestions grounded in these observations. To effectively coordinate the fine-grained visual grounding with high-level contextual
language reasoning, GovLA-Reasoner introduces an efficient Spatially-aware Grounding Adapter (SGA) that implicitly coordinates discriminative representation sharing between the visual detector and the large
language model (LLM). Different from existing adapters that primarily focus on global
embedding alignment, our SGA is specifically designed to compress and aggregate multi-stream grounding-aware representations, thereby preserving fine-grained spatial cues while enabling their effective integration into the
language reasoning process. Extensive experiments indicate that our GovLA-Reasoner effectively improves performance while avoiding the need of fine-tuning for any task-specific individual components. We believe our work offers a new perspective and foundation for future studies on management-aware low-altitude
vision-
language systems. The code and
dataset will be publicly released after further organization.
Read the full abstract ▾
All authors Hao Chang, Zhihui Wang, Lingxiang Wu, Wei An, Boyang Li, Zaiping Lin, Weidong Sheng, Jinqiao Wang
Computer Vision
The pretraining domain outweighs the training objective in setting the privacy-utility trade-off of differentially private medical image analysis
Soroosh Tayebi Arasteh, Mina Farajiamiri, Mahshad Lotfinia et al. (+7 more)
Differential
privacy limits how much each training image can influence a medical imaging model but lowers its diagnostic accuracy. Pretrained weights reduce this loss. It is unknown whether the pretraining objective or domain improves accuracy more. Here we train chest radiograph classifiers under differential
privacy from five starting points on 591,884 radiographs from five
datasets. We show that supervised pretraining on chest radiographs ranks first in 24 of 25 combinations of
dataset and
privacy setting. Measured by the area under the receiver operating characteristic curve, its lead over supervised pretraining on natural images is 2.6% without
privacy and 15.3% at the strictest budget. From that starting point, changing the domain improves this area 2.2 to 3.7 times more than changing the objective. Private pretraining on chest radiographs lowers this area by about 5% under
privacy but outperforms the three starting points without patient data at the strictest budget. Under
privacy, supervised chest radiograph pretraining leads in the worst-performing demographic subgroup.
Read the full abstract ▾
All authors Soroosh Tayebi Arasteh, Mina Farajiamiri, Mahshad Lotfinia, Behrus Hinrichs-Puladi, Jonas Bienzeisler, Mohamed Alhaskir, Mirabela Rusu, Christiane Kuhl, Sven Nebelung, Daniel Truhn
Computer Vision
GMS-CAVP: Improving Audio-Video Correspondence with Multi-Scale Contrastive and Generative Pretraining
Shentong Mo, Zehua Chen and Jun Zhu
Recent advances in
video-audio (V-A) understanding and generation have increasingly relied on joint V-A
embeddings, which serve as the foundation for tasks such as cross-modal
retrieval and generation. While prior methods like CAVP effectively model semantic and temporal correspondences between modalities using contrastive objectives, their performance remains suboptimal. A key limitation is the insufficient modeling of the dense, multi-scale nature of both
video and audio signals, correspondences often span fine- to coarse-grained spatial-temporal structures, which are underutilized in existing frameworks. To this end, we propose GMS-CAVP, a novel framework that combines Multi-Scale
Video-Audio
Alignment and Multi-Scale Spatial-Temporal
Diffusion-based pretraining objectives to enhance V-A correspondence modeling. First, GMS-CAVP introduces a multi-scale contrastive learning strategy that captures semantic and temporal relations across varying granularities. Second, we go beyond traditional contrastive learning by incorporating a
diffusion-based generative objective, enabling modality translation and synthesis between
video and audio. This unified discriminative-generative formulation facilitates deeper cross-modal understanding and paves the way for high-fidelity generation. Extensive experiments on VGGSound, AudioSet, and Panda70M demonstrate that GMS-CAVP outperforms previous methods in generation and
retrieval.
Read the full abstract ▾