Computer Vision
Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility
Honglin Lin, Zheng Liu, Chonghan Qin et al. (+7 more)
While synthetic data has proven effective for improving scientific
reasoning in the text domain,
multimodal reasoning remains constrained by the difficulty of synthesizing scientifically rigorous images. Existing Text-to-Image (T2I) models often produce outputs that are visually plausible yet scientifically incorrect, resulting in a persistent visual-logic divergence that limits their value for downstream
reasoning. Motivated by recent advances in next-generation T2I models, we conduct a systematic study of scientific image synthesis across generation paradigms, evaluation, and downstream use. We analyze both direct pixel-based generation and programmatic synthesis, and propose ImgCoder, a logic-driven framework that follows an explicit "understand - plan - code" workflow to improve structural precision. To rigorously assess scientific correctness, we introduce SciGenBench, which evaluates generated images based on information utility and logical validity. Our evaluation reveals systematic failure modes in pixel-based models and highlights a fundamental expressiveness-precision trade-off. Finally, we show that fine-tuning Large
Multimodal Models (LMMs) on rigorously verified synthetic scientific images yields consistent
reasoning gains, with potential scaling trends analogous to the text domain, validating high-fidelity scientific synthesis as a viable path to unlocking massive
multimodal reasoning capabilities.
Read the full abstract ▾
All authors Honglin Lin, Zheng Liu, Chonghan Qin, Qizhi Pei, Yu Li, Zhanping Zhong, Xin Gao, Yanfeng Wang, Conghui He, Lijun Wu
Computer Vision
Shaunak Halbe, Bhagyashree Puranik, Jayakrishnan Unnikrishnan et al. (+3 more)
Modern
video retrieval systems are expected to handle diverse tasks ranging from corpus-level
retrieval, fine-grained moment localization to flexible
multimodal querying. Specialized architectures achieve strong
retrieval performance by training modality-specific encoders on massive
datasets, but they lack the ability to process composed
multimodal queries. In contrast,
multimodal LLM (MLLM)-based methods support rich
multimodal search but their
retrieval performance remains well below that of specialized systems. We present VeRVE, an MLLM-based versatile
video retrieval framework that integrates corpus and moment-level
retrieval capabilities while accommodating composed
multimodal queries within a single architecture. We use contrastive
alignment of visual and textual
embeddings generated using a shared MLLM backbone to facilitate efficient
embedding-based candidate search. Our
embedding model, trained efficiently using low-rank adaptation (LoRA) on 700K paired visual-text data samples, surpasses other MLLM-based methods on zero-shot
video retrieval tasks. Additionally, we demonstrate that the same model can be adapted without further training to achieve competitive results on zero-shot moment
retrieval, and state of the art results for zero-shot composed
video retrieval. With additional training for reranking candidates identified in the
embedding-based search, our model substantially outperforms existing MLLM-based
retrieval systems and achieves
retrieval performance comparable to state of the art specialized models.
Read the full abstract ▾
All authors Shaunak Halbe, Bhagyashree Puranik, Jayakrishnan Unnikrishnan, Kushan Thakkar, Vimal Bhat, Toufiq Parag
Computer Vision
Inverse Rendering for High-Genus 3D Surface Meshes from Multi-view Images with Persistent Homology Priors
Xiang Gao, Xinmu Wang, Yuanpeng Liu et al. (+4 more)
Reconstructing 3D objects from images is inherently an ill-posed problem due to ambiguities in geometry, appearance, and
topology. This paper introduces collaborative inverse rendering with persistent homology priors, a novel strategy that leverages topological constraints to resolve these ambiguities. By incorporating priors that capture critical features such as tunnel loops and handle loops, our approach directly addresses the difficulty of reconstructing high-genus surfaces. The collaboration between photometric consistency from multi-view images and homology-based guidance enables recovery of complex high-genus geometry while circumventing catastrophic failures such as collapsing tunnels or losing high-genus structure. Instead of neural networks, our method relies on gradient-based optimization within a mesh-based inverse rendering framework to highlight the role of topological priors. Experimental results show that incorporating persistent homology priors leads to lower Chamfer Distance (CD) and higher Volume IoU compared to state-of-the-art mesh-based methods, demonstrating improved
geometric accuracy and robustness against topological failure.
Read the full abstract ▾
All authors Xiang Gao, Xinmu Wang, Yuanpeng Liu, Yue Wang, Junqi Huang, Wei Chen, Xianfeng Gu
Computer Vision
Enhanced Diagnostic Performance via Large-Resolution Inference Optimization for Pathology Foundation Models
Mengxuan Hu, Zihan Guan, John Kang et al. (+2 more)
Despite their prominent performance on tasks such as ROI classification and
segmentation, many pathology foundation models remain constrained by a specific input size e.g. 224 x 224, creating substantial inefficiencies when applied to whole-slide images (WSIs), which span thousands of resolutions. A naive strategy is to either enlarge inputs or downsample the WSIs. However, enlarging inputs results in prohibitive GPU
memory consumption, while downsampling alters the microns-per-pixel resolution and obscures critical morphological details. To overcome these limitations, we propose an space- and time- efficient inference strategy that sparsifies
attention using spatially aware neighboring blocks and filters out non-informative
tokens through global
attention scores. This design substantially reduces GPU
memory and runtime during high-resolution WSI inference while preserving and even improving the downstream performance, enabling inference at higher resolutions under the same GPU budget. The experimental results show that our method can achieves up to an 7.67% improvement in the ROI classification and compatible results in
segmentation.
Read the full abstract ▾
All authors Mengxuan Hu, Zihan Guan, John Kang, Sheng Li, Zhongliang Zhou
Computer Vision
Principal Component Analysis-Based Terahertz Self-Supervised Denoising and Deblurring Deep Neural Networks
Pengfei Zhu, Stefano Sfarra, Hai Zhang et al. (+4 more)
Terahertz (THz) systems inherently introduce frequency-dependent degradation effects, resulting in low-frequency blurring and high-frequency noise in amplitude images. Conventional image processing techniques cannot simultaneously address both issues, and manual intervention is often required due to the unknown boundary between denoising and deblurring. To tackle this challenge, we propose a principal component analysis (PCA)-based THz self-supervised denoising and deblurring network (THz-SSDD). The network employs a Recorrupted-to-Recorrupted self-supervised learning strategy to capture the intrinsic features of noise by exploiting invariance under repeated corruption. PCA decomposition and reconstruction are then applied to restore images across both low and high frequencies. The performance of the THz-SSDD network was evaluated on four types of samples. Training requires only a small set of unlabeled noisy images, and testing across samples with different material properties and measurement modes demonstrates effective denoising and deblurring. Quantitative analysis further validates the network feasibility, showing improvements in image quality while preserving the physical characteristics of the original signals.
Read the full abstract ▾
All authors Pengfei Zhu, Stefano Sfarra, Hai Zhang, Carlo Santulli, Elana Pivarciova, Fabrizio Sarasini, Xavier Maldague
Computer Vision
Segment and Matte Anything in a Unified Model
Zezhong Fan, Xiaohan Li, Topojoy Biswas et al. (+2 more)
Segment Anything (SAM) has recently pushed the boundaries of
segmentation by demonstrating zero-shot generalization and flexible prompting after training on over one billion masks. Despite this, its mask prediction accuracy often falls short of the precision required in real-world applications. While several refinement modules have been proposed to boost SAM's
segmentation quality, achieving highly accurate object delineation within a single, unified framework remains an open challenge. Furthermore, interactive image matting, which aims to generate fine-grained alpha mattes guided by diverse user hints, has not yet been explored in the context of SAM. Insights from recent studies highlight strong correlations between
segmentation and matting, suggesting the feasibility of a unified model capable of both tasks. In this paper, we introduce Segment And Matte Anything (SAMA), a lightweight extension of SAM that delivers high-quality interactive image
segmentation and matting with minimal extra parameters. Our Multi-View Localization Encoder (MVLE) captures detailed features from local views, while the Localization Adapter (Local-Adapter) refines mask outputs by recovering subtle boundary details. We also incorporate two prediction heads for each task into the architecture to generate
segmentation and matting masks, simultaneously. Trained on a diverse
dataset aggregated from publicly available sources, SAMA achieves state-of-the-art performance across multiple
segmentation and matting
benchmarks, showcasing its adaptability and effectiveness in a wide range of downstream tasks.
Read the full abstract ▾
All authors Zezhong Fan, Xiaohan Li, Topojoy Biswas, Kaushiki Nag, Kannan Achan
Computer Vision · Cross-listed from cs.LG
EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts
Anzhe Cheng, Shukai Duan, Shixuan Li et al. (+5 more)
The relentless scaling of deep learning models has led to unsustainable computational demands, positioning Mixture-of-Experts (MoE) architectures as a promising path towards greater efficiency. However, MoE models are plagued by two fundamental challenges: 1) a load imbalance problem known as the“rich get richer" phenomenon, where a few experts are over-utilized, and 2) an expert homogeneity problem, where experts learn redundant representations, negating their purpose. Current solutions typically employ an auxiliary load-balancing loss that, while mitigating imbalance, often exacerbates homogeneity by enforcing uniform routing at the expense of specialization. To resolve this, we introduce the Eigen-Mixture-of-Experts (EMoE), a novel architecture that leverages a routing mechanism based on a learned orthonormal eigenbasis. EMoE projects input
tokens onto this shared eigenbasis and routes them based on their
alignment with the principal components of the feature space. This principled,
geometric partitioning of data intrinsically promotes both balanced expert utilization and the development of diverse, specialized experts, all without the need for a conflicting auxiliary loss function. Our code is publicly available at https://github.com/Belis0811/EMoE.
Read the full abstract ▾
All authors Anzhe Cheng, Shukai Duan, Shixuan Li, Chenzhong Yin, Mingxi Cheng, Shahin Nazarian, Paul Thompson, Paul Bogdan
Computer Vision
CARLA-Round: A Multi-Factor Simulation Dataset for Roundabout Trajectory Prediction
Xiaotong Zhou, Zhenhui Yuan, Yi Han et al. (+2 more)
Accurate trajectory prediction of vehicles at roundabouts is critical for reducing traffic accidents, yet it remains highly challenging due to their circular road geometry, continuous merging and yielding interactions, and absence of traffic signals. Developing accurate prediction
algorithms relies on reliable,
multimodal, and realistic
datasets; however, such
datasets for roundabout scenarios are scarce, as real-world data collection is often limited by incomplete observations and entangled factors that are difficult to isolate. We present CARLA-Round, a systematically designed simulation
dataset for roundabout trajectory prediction. The
dataset varies weather conditions (five types) and traffic density levels (spanning Level-of-Service A-E) in a structured manner, resulting in 25 controlled scenarios. Each scenario incorporates realistic mixtures of driving behaviors and provides explicit annotations that are largely absent from existing
datasets. Unlike randomly sampled simulation data, this structured design enables precise analysis of how different conditions influence trajectory prediction performance. Validation experiments using standard baselines (LSTM, GCN, GRU+GCN) reveal traffic density dominates prediction difficulty with strong monotonic effects, while weather shows non-linear impacts. The best model achieves 0.312m ADE on real-world rounD
dataset, demonstrating effective sim-to-real transfer. This systematic approach quantifies factor impacts impossible to isolate in confounded real-world
datasets. Our CARLA-Round
dataset is available at https://github.com/Rebecca689/CARLA-Round.
Read the full abstract ▾
All authors Xiaotong Zhou, Zhenhui Yuan, Yi Han, Tianhua Xu, Laurence T. Yang
Computer Vision
RCDN: Real-Centered Detection Network for Robust Face Forgery Identification
Wyatt McCurdy, Xin Zhang, Yuqi Song et al. (+1 more)
Image forgery has become a critical threat with the rapid proliferation of AI-based generation tools, which make it increasingly easy to synthesize realistic but fraudulent facial content. Existing detection methods achieve near-perfect performance when training and testing are conducted within the same domain, yet their effectiveness deteriorates substantially in crossdomain scenarios. This limitation is problematic, as new forgery techniques continuously emerge and detectors must remain reliable against unseen manipulations. To address this challenge, we propose the Real-Centered Detection Network (RCDN), a frequency spatial convolutional neural networks(CNN) framework with an Xception backbone that anchors its representation space around authentic facial images. Instead of modeling the diverse and evolving patterns of forgeries, RCDN emphasizes the consistency of real images, leveraging a dual-branch architecture and a real centered loss design to enhance robustness under distribution shifts. Extensive experiments on the DiFF
dataset, focusing on three representative forgery types (FE, I2I, T2I), demonstrate that RCDN achieves both state-of-the-art in-domain accuracy and significantly stronger cross-domain generalization. Notably, RCDN reduces the generalization gap compared to leading baselines and achieves the highest cross/in-domain stability ratio, highlighting its potential as a practical solution for defending against evolving and unseen image forgery techniques.
Read the full abstract ▾
All authors Wyatt McCurdy, Xin Zhang, Yuqi Song, Min Gao
Computer Vision
Energy-Aware Ensemble Learning for Coffee Leaf Disease Classification
Larissa Ferreira Rodrigues Moreira, Rodrigo Moreira and Leonardo Gabriel Ferreira Rodrigues
Coffee yields are contingent on the timely and accurate diagnosis of diseases; however, assessing leaf diseases in the field presents significant challenges. Although Artificial Intelligence (AI)
vision models achieve high accuracy, their adoption is hindered by the limitations of constrained devices and intermittent connectivity. This study aims to facilitate sustainable on-device diagnosis through knowledge
distillation: high-capacity Convolutional Neural Networks (CNNs) trained in data centers transfer knowledge to compact CNNs through Ensemble Learning (EL). Furthermore, dense tiny pairs were integrated through simple and optimized ensembling to enhance accuracy while adhering to strict computational and energy constraints. On a curated coffee leaf
dataset, distilled tiny ensembles achieved competitive with prior work with significantly reduced energy consumption and carbon footprint. This indicates that lightweight models, when properly distilled and ensembled, can provide practical diagnostic solutions for Internet of Things (IoT) applications.
Read the full abstract ▾
Computer Vision
Detecting 3D Line Segments for 6DoF Pose Estimation with Limited Data
Matej Mok, Lukáš Gajdošech, Michal Mesároš et al. (+2 more)
The task of 6DoF object pose estimation is one of the fundamental problems of 3D
vision with many practical applications such as industrial automation. Traditional deep learning approaches for this task often require extensive training data or CAD models, limiting their application in real-world industrial settings where data is scarce and object instances vary. We propose a novel method for 6DoF pose estimation focused specifically on bins used in industrial settings. We exploit the cuboid geometry of bins by first detecting intermediate 3D line segments corresponding to their top edges. Our approach extends the 2D line segment detection network LeTR to operate on structured point cloud data. The detected 3D line segments are then processed using a simple
geometric procedure to robustly determine the bin's 6DoF pose. To evaluate our method, we extend an existing
dataset with a newly collected and annotated
dataset, which we make publicly available. We show that incorporating synthetic training data significantly improves pose estimation accuracy on real scans. Moreover, we show that our method significantly outperforms current state-of-the-art 6DoF pose estimation methods in terms of the pose accuracy (3 cm translation error, 8.2$^\circ$ rotation error) while not requiring instance-specific CAD models during inference.
Read the full abstract ▾
All authors Matej Mok, Lukáš Gajdošech, Michal Mesároš, Martin Madaras, Viktor Kocur
Computer Vision
Conditional Random Fields for Interactive Refinement of Histopathological Predictions
Tiffanie Godelaine, Maxime Zanella, Karim El Khoury et al. (+3 more)
Assisting pathologists in the analysis of histopathological images has high clinical value, as it supports cancer detection and staging. In this context, histology foundation models have recently emerged. Among them,
Vision-
Language Models (VLMs) provide strong yet imperfect zero-shot predictions. We propose to refine these predictions by adapting Conditional Random Fields (CRFs) to histopathological applications, requiring no additional model training. We present HistoCRF, a CRF-based framework, with a novel definition of the pairwise potential that promotes label diversity and leverages expert annotations. We consider three experiments: without annotations, with expert annotations, and with iterative human-in-the-loop annotations that progressively correct misclassified patches. Experiments on five patch-level classification
datasets covering different organs and diseases demonstrate average accuracy gains of 16.0% without annotations and 27.5% with only 100 annotations, compared to zero-shot predictions. Moreover, integrating a human in the loop reaches a further gain of 32.6% with the same number of annotations. The code will be made available on https://github.com/tgodelaine/HistoCRF.
Read the full abstract ▾
All authors Tiffanie Godelaine, Maxime Zanella, Karim El Khoury, Saïd Mahmoudi, Benoît Macq, Christophe De Vleeschouwer
Computer Vision
Toward Real-World High-Precision Image Matting and Segmentation
Haipeng Zhou, Zhaohu Xing, Hongqiu Wang et al. (+3 more)
High-precision scene parsing tasks, including image matting and dichotomous
segmentation, aim to accurately predict masks with extremely fine details (such as hair). Most existing methods focus on salient, single foreground objects. While interactive methods allow for target adjustment, their class-agnostic design restricts generalization across different categories. Furthermore, the scarcity of high-quality annotation has led to a reliance on inharmonious synthetic data, resulting in poor generalization to real-world scenarios. To this end, we propose a Foreground Consistent Learning model, dubbed as FCLM, to address the aforementioned issues. Specifically, we first introduce a Depth-Aware
Distillation strategy where we transfer the depth-related knowledge for better foreground representation. Considering the data dilemma, we term the processing of synthetic data as domain adaptation problem where we propose a domain-invariant learning strategy to focus on foreground learning. To support interactive prediction, we contribute an Object-Oriented Decoder that can receive both visual and
language prompts to predict the referring target. Experimental results show that our method quantitatively and qualitatively outperforms SOTA methods.
Read the full abstract ▾
All authors Haipeng Zhou, Zhaohu Xing, Hongqiu Wang, Jun Ma, Ping Li, Lei Zhu
Computer Vision
EmoLat: Text-driven Image Sentiment Transfer via Emotion Latent Space
Jing Zhang, Bingjie Fan, Jixiang Zhu et al. (+1 more)
We propose EmoLat, a novel
emotion latent space that enables fine-grained, text-driven image sentiment transfer by modeling cross-modal correlations between textual semantics and visual
emotion features. Within EmoLat, an
emotion semantic graph is constructed to capture the relational structure among
emotions, objects, and visual attributes. To enhance the discriminability and transferability of
emotion representations, we employ
adversarial regularization, aligning the latent
emotion distributions across modalities. Building upon EmoLat, a cross-modal sentiment transfer framework is proposed to manipulate image sentiment via joint
embedding of text and EmoLat features. The network is optimized using a multi-objective loss incorporating semantic consistency,
emotion alignment, and
adversarial regularization. To support effective modeling, we construct EmoSpace Set, a large-scale
benchmark dataset comprising images with dense annotations on
emotions, object semantics, and visual attributes. Extensive experiments on EmoSpace Set demonstrate that our approach significantly outperforms existing state-of-the-art methods in both quantitative metrics and qualitative transfer fidelity, establishing a new paradigm for controllable image sentiment editing guided by textual input. The EmoSpace Set and all the code are available at http://github.com/JingVIPLab/EmoLat.
Read the full abstract ▾
All authors Jing Zhang, Bingjie Fan, Jixiang Zhu, Zhe Wang
Computer Vision
CroBIM-V: Memory-Quality Controlled Remote Sensing Referring Video Object Segmentation
H. Jiang, Y. Sun, Z. Dong et al. (+2 more)
Remote sensing
video referring object
segmentation (RS-RVOS) is challenged by weak target saliency and severe visual information truncation in dynamic scenes, making it extremely difficult to maintain discriminative target representations during
segmentation. Moreover, progress in this field is hindered by the absence of large-scale dedicated
benchmarks, while existing models are often affected by biased initial
memory construction that impairs accurate instance localization in complex scenarios, as well as indiscriminate
memory accumulation that encodes noise from occlusions or misclassifications, leading to persistent error propagation. This paper advances RS-RVOS research through dual contributions in data and methodology. First, we construct RS-RVOS Bench, the first large-scale
benchmark comprising 111
video sequences, about 25,000 frames, and 213,000 temporal referring annotations. Unlike common RVOS
benchmarks where many expressions are written with access to the full
video context, our
dataset adopts a strict causality-aware annotation strategy in which linguistic references are generated solely from the target state in the initial frame. Second, we propose a
memory-quality-aware online referring
segmentation framework, termed
Memory Quality Control with Segment Anything Model (MQC-SAM). MQC-SAM introduces a temporal motion consistency module for initial
memory calibration, leveraging short-term motion trajectory priors to correct structural deviations and establish accurate
memory anchoring. Furthermore, it incorporates a decoupled
attention-based
memory integration mechanism with dynamic quality assessment, selectively updating high-confidence semantic features while filtering unreliable information, thereby effectively preventing error accumulation and propagation. Extensive experiments on RS-RVOS Bench demonstrate that MQC-SAM achieves state-of-the-art performance.
Read the full abstract ▾
All authors H. Jiang, Y. Sun, Z. Dong, T. Liu, Y. Gu
Computer Vision
ARMARecon: An ARMA Convolutional Filter based Graph Neural Network for Neurodegenerative Dementias Classification
VSS Tejaswi Abburi, Ananya Singhal, Saurabh J. Shigwan et al. (+1 more)
Early detection of neurodegenerative diseases such as Alzheimer's Disease (AD) and Frontotemporal Dementia (FTD) is essential for reducing the risk of progression to severe disease stages. As AD and FTD propagate along white-matter regions in a global, graph-dependent manner, graph-based neural networks are well suited to capture these patterns. Hence, we introduce ARMARecon, a unified graph learning framework that integrates Autoregressive Moving Average (ARMA) graph filtering with a reconstruction-driven objective to enhance feature representation and improve classification accuracy. ARMARecon effectively models both local and global connectivity by leveraging 20-bin Fractional Anisotropy (FA) histogram features extracted from white-matter regions, while mitigating over-smoothing. Overall, ARMARecon achieves superior performance compared to state-of-the-art methods on the multi-site dMRI
datasets ADNI and NIFD.
Read the full abstract ▾
All authors VSS Tejaswi Abburi, Ananya Singhal, Saurabh J. Shigwan, Nitin Kumar
Computer Vision
Learning Stochastic Bridges for Video Object Removal via Video-to-Video Translation
Zijie Lou, Xiangwei Feng, Jiaxin Wang et al. (+7 more)
Existing
video object removal methods predominantly rely on
diffusion models following a noise-to-data paradigm, where generation starts from uninformative Gaussian noise. This approach discards the rich structural and contextual priors present in the original input
video. Consequently, such methods often lack sufficient guidance, leading to incomplete object erasure or the synthesis of implausible content that conflicts with the scene's physical logic. In this paper, we reformulate
video object removal as a
video-to-
video translation task via a
stochastic bridge model. Unlike noise-initialized methods, our framework establishes a direct
stochastic path from the source
video (with objects) to the target
video (objects removed). This bridge formulation effectively leverages the input
video as a strong structural prior, guiding the model to perform precise removal while ensuring that the filled regions are logically consistent with the surrounding environment. To address the trade-off where strong bridge priors hinder the removal of large objects, we propose a novel adaptive mask modulation strategy. This mechanism dynamically modulates input
embeddings based on mask characteristics, balancing background fidelity with generative flexibility. Extensive experiments demonstrate that our approach significantly outperforms existing methods in both visual quality and temporal consistency. The project page is https://bridgeremoval.github.io/.
Read the full abstract ▾
All authors Zijie Lou, Xiangwei Feng, Jiaxin Wang, Jiangtao Yao, Fei Che, Tianbao Liu, Chengjing Wu, Xiaochao Qu, Luoqi Liu, Ting Liu
Computer Vision
Learning Language-Driven Sequence-Level Modal-Invariant Representations for Video-Based Visible-Infrared Person Re-Identification
Xiaomei Yang, Antai Liu, Xizhan Gao et al. (+3 more)
The core of
video-based visible-infrared person re-identification (VVI-ReID) lies in learning sequence-level modal-invariant representations across different modalities. Recent research tends to use modality-shared
language prompts generated by CLIP to guide the learning of modal-invariant representations. Despite achieving optimal performance, such methods still face limitations in efficient spatial-temporal modeling, sufficient cross-modal interaction, and explicit modality-level loss guidance. To address these issues, we propose the
language-driven sequence-level modal-invariant representation learning (LSMRL) method, which includes spatial-temporal feature learning (STFL) module, semantic
diffusion (SD) module and cross-modal interaction (CMI) module. To enable parameter- and computation-efficient spatial-temporal modeling, the STFL module is built upon CLIP with minimal modifications. To achieve sufficient cross-modal interaction and enhance the learning of modal-invariant features, the SD module is proposed to diffuse modality-shared
language prompts into visible and infrared features to establish preliminary modal consistency. The CMI module is further developed to leverage bidirectional cross-modal self-
attention to eliminate residual modality gaps and refine modal-invariant representations. To explicitly enhance the learning of modal-invariant representations, two modality-level losses are introduced to improve the features' discriminative ability and their generalization to unseen categories. Extensive experiments on large-scale VVI-ReID
datasets demonstrate the superiority of LSMRL over AOTA methods.
Read the full abstract ▾
All authors Xiaomei Yang, Antai Liu, Xizhan Gao, Fa Zhu, Sijie Niu, Giancarlo Fortino
Computer Vision
Automating Parameter Selection in Deep Image Prior for Fluorescence Microscopy Image Denoising via Similarity-Based Parameter Transfer
Lina Meyer, Felix Wissel, Tobias Knopp et al. (+8 more)
Unsupervised deep image prior (DIP) addresses shortcomings of training data requirements and limited generalization associated with supervised deep learning. The performance of DIP depends on the network architecture and the stopping point of its iterative process. Optimizing these parameters for a new image requires time, restricting DIP application in domains where many images need to be processed. Focusing on fluorescence microscopy data, we hypothesize that similar images share comparable optimal parameter configurations for DIP-based denoising, potentially enabling optimization-free DIP for fluorescence microscopy. We generated a calibration (n=110) and validation set (n=55) of semantically different images from an open-source
dataset for a network architecture search targeted towards ideal U-net architectures and stopping points. The calibration set represented our transfer basis. The validation set enabled the assessment of which image similarity criterion yields the best results. We then implemented AUTO-DIP, a pipeline for automatic parameter transfer, and compared it to the originally published DIP configuration (baseline) and a state-of-the-art image-specific variational denoising approach. We show that a parameter transfer from the calibration
dataset to a test image based on only image metadata similarity (e.g., microscope type, imaged specimen) leads to similar and better performance than a transfer based on quantitative image similarity measures. AUTO-DIP outperforms the baseline DIP (DIP with original DIP parameters) as well as the variational denoising approaches for several open-source test
datasets of varying complexity, particularly for very noisy inputs. Applications to locally acquired fluorescence microscopy images further proved superiority of AUTO-DIP.
Read the full abstract ▾
All authors Lina Meyer, Felix Wissel, Tobias Knopp, Susanne Pfefferle, Ralf Fliegert, Maximilian Sandmann, Liana Uebler, Franziska Möckl, Björn-Philipp Diercks, David Lohr, René Werner
Computer Vision
Task-Driven Prompt Learning: A Joint Framework for Multi-modal Cloud Removal and Segmentation
Zaiyan Zhang, Jie Li, Shaowei Shi et al. (+1 more)
Optical remote sensing imagery is indispensable for Earth observation, yet persistent cloud occlusion limits its downstream utility. Most cloud removal (CR) methods are optimized for low-level fidelity and can over-smooth textures and boundaries that are critical for analysis-ready data (ARD), leading to a mismatch between visually plausible restoration and semantic utility. To bridge this gap, we propose TDP-CR, a task-driven
multimodal framework that jointly performs cloud removal and land-cover
segmentation. Central to our approach is a
Prompt-Guided Fusion (PGF) mechanism, which utilizes a learnable degradation
prompt to encode cloud thickness and spatial uncertainty. By combining global channel context with local
prompt-conditioned spatial bias, PGF adaptively integrates Synthetic Aperture Radar (SAR) information only where optical data is corrupted. We further introduce a parameter-efficient two-phase training strategy that decouples reconstruction and semantic representation learning. Experiments on the LuojiaSET-OSFCR
dataset demonstrate the superiority of our framework: TDP-CR surpasses heavy state-of-the-art baselines by 0.18 dB in PSNR while using only 15% of the parameters, and achieves a 1.4% improvement in mIoU consistently against multi-task competitors, effectively delivering analysis-ready data.
Read the full abstract ▾
All authors Zaiyan Zhang, Jie Li, Shaowei Shi, Qiangqiang Yuan
Computer Vision
A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models
Weixin Ye, Wei Wang, Yahui Liu et al. (+5 more)
In
federated learning,
Transformer, as a popular architecture, faces critical challenges in defending against gradient attacks and improving model performance in both Computer
Vision (CV) and Natural
Language Processing (NLP) tasks. It has been revealed that the gradient of Position
Embeddings (PEs) in
Transformer contains sufficient information, which can be used to reconstruct the input data. To mitigate this issue, we introduce a Masked Jigsaw Puzzle (MJP) framework. MJP starts with random
token shuffling to break the
token order, and then a learnable unknown (unk) position
embedding is used to mask out the PEs of the shuffled
tokens. In this manner, the local spatial information which is encoded in the position
embeddings is disrupted, and the models are forced to learn feature representations that are less reliant on the local spatial information. Notably, with the careful use of MJP, we can not only improve models' robustness against gradient attacks, but also boost their performance in both
vision and text application scenarios, such as classification for images (e.g., ImageNet-1K) and sentiment analysis for text (e.g., Yelp and Amazon). Experimental results suggest that MJP is a unified framework for different
Transformer-based models in both
vision and
language tasks. Code is publicly available via https://github.com/ywxsuperstar/transformerattack
Read the full abstract ▾
All authors Weixin Ye, Wei Wang, Yahui Liu, Yue Song, Bin Ren, Wei Bi, Rita Cucchiara, Nicu Sebe
Computer Vision
FocaLogic: Logic-Based Interpretation of Visual Model Decisions
Chenchen Zhao, Muxi Chen and Qiang Xu
Interpretability of modern visual models is crucial, particularly in high-stakes applications. However, existing interpretability methods typically suffer from either reliance on white-box model access or insufficient quantitative rigor. To address these limitations, we introduce FocaLogic, a novel model-agnostic framework designed to interpret and quantify visual model decision-making through logic-based representations. FocaLogic identifies minimal interpretable subsets of visual regions-termed visual focuses-that decisively influence model predictions. It translates these visual focuses into precise and compact logical expressions, enabling transparent and structured interpretations. Additionally, we propose a suite of quantitative metrics, including focus precision, recall, and divergence, to objectively evaluate model behavior across diverse scenarios. Empirical analyses demonstrate FocaLogic's capability to uncover critical insights such as training-induced concentration, increasing focus accuracy through generalization, and anomalous focuses under biases and
adversarial attacks. Overall, FocaLogic provides a systematic, scalable, and quantitative solution for interpreting visual models.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.HC
Multimodal Feedback for Handheld Tool Guidance: Combining Wrist-Based Haptics with Augmented Reality
Yue Yang, Christoph Leuze, Brian Hargreaves et al. (+2 more)
We investigate how vibrotactile wrist feedback can enhance spatial guidance for handheld tool movement in optical see-through augmented reality (AR). While AR overlays are widely used to support
surgical tasks, visual occlusion, lighting conditions, and interface ambiguity can compromise precision and confidence. To address these challenges, we designed a
multimodal system combining AR visuals with a custom wrist-worn haptic device delivering directional and state-based cues. A formative study with experienced surgeons and residents identified key tool maneuvers and preferences for reference mappings, guiding our cue design. In a cue identification experiment (N=21), participants accurately recognized five vibration patterns under visual load, with higher recognition for full-actuator states than spatial direction cues. In a guidance task (N=27), participants using both AR and haptics achieved significantly higher spatial precision (5.8 mm) and usability (SUS = 88.1) than those using either modality alone, despite having modest increases in task time. Participants reported that haptic cues provided reassuring confirmation and reduced cognitive effort during
alignment. Our results highlight the promise of integrating wrist-based haptics into AR systems for high-precision, visually complex tasks such as
surgical guidance. We discuss design implications for
multimodal interfaces supporting confident, efficient tool manipulation.
Read the full abstract ▾
All authors Yue Yang, Christoph Leuze, Brian Hargreaves, Bruce Daniel, Fred M Baik
Computer Vision
DIAMOND-SSS: Diffusion-Augmented Multi-View Optimization for Data-efficient SubSurface Scattering
Guillermo Figueroa-Araneda, Iris Diana Jimenez, Florian Hofherr et al. (+3 more)
Subsurface scattering (SSS) gives translucent materials – such as wax, jade, marble, and skin – their characteristic soft shadows, color bleeding, and diffuse glow. Modeling these effects in neural rendering remains challenging due to complex light transport and the need for densely captured multi-view, multi-light
datasets (often more than 100 views and 112 OLATs). We present DIAMOND-SSS, a data-efficient framework for high-fidelity translucent reconstruction from extremely sparse supervision – even as few as ten images. We fine-tune
diffusion models for novel-view synthesis and relighting, conditioned on estimated geometry and trained on less than 7 percent of the
dataset, producing photorealistic augmentations that can replace up to 95 percent of missing captures. To stabilize reconstruction under sparse or synthetic supervision, we introduce illumination-independent
geometric priors: a multi-view silhouette consistency loss and a multi-view depth consistency loss. Across all sparsity regimes, DIAMOND-SSS achieves state-of-the-art quality in relightable Gaussian rendering, reducing real capture requirements by up to 90 percent compared to SSS-3DGS.
Read the full abstract ▾
All authors Guillermo Figueroa-Araneda, Iris Diana Jimenez, Florian Hofherr, Manny Ko, Hector Andrade-Loarca, Daniel Cremers