Computer Vision · Cross-listed from cs.IR
WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata
Prasanna Sridhar, Horace Lee, David M. S. Pinto et al. (+2 more)
In this paper, we present WISE, an open-source audiovisual search engine which integrates a range of
multimodal retrieval capabilities into a single, practical tool accessible to users without machine learning expertise. WISE supports natural-
language and reverse-image queries at both the scene level (e.g. empty street) and object level (e.g. horse) across images and
videos; face-based search for specific individuals; audio
retrieval of acoustic events using text (e.g. wood creak) or an audio file; search over automatically transcribed
speech; and filtering by user-provided metadata. Rich insights can be obtained by combining queries across modalities – for example, retrieving German trains from a historical archive by applying the object query "train" and the metadata query "Germany", or searching for a face in a place. By employing vector search techniques, WISE can scale to support efficient
retrieval over millions of images or thousands of hours of
video. Its modular architecture facilitates the integration of new models. WISE can be deployed locally for private or sensitive collections, and has been applied to various real-world use cases. Our code is open-source and available at https://gitlab.com/vgg/wise/wise.
Read the full abstract ▾
All authors Prasanna Sridhar, Horace Lee, David M. S. Pinto, Andrew Zisserman, Abhishek Dutta
Computer Vision
GSM-GS: Geometry-Constrained Single and Multi-view Gaussian Splatting for Surface Reconstruction
Xiao Ren, Yu Liu, Ning An et al. (+3 more)
Recently, 3D Gaussian Splatting has emerged as a prominent research direction owing to its ultrarapid training speed and high-fidelity rendering capabilities. However, the unstructured and irregular nature of Gaussian point clouds poses challenges to reconstruction accuracy. This limitation frequently causes high-frequency detail loss in complex surface microstructures when relying solely on routine strategies. To address this limitation, we propose GSM-GS: a synergistic optimization framework integrating single-view adaptive sub-region weighting constraints and multi-view spatial structure refinement. For single-view optimization, we leverage image gradient features to partition scenes into texture-rich and texture-less sub-regions. The reconstruction quality is enhanced through adaptive filtering mechanisms guided by depth discrepancy features. This preserves high-weight regions while implementing a dual-branch constraint strategy tailored to regional texture variations, thereby improving
geometric detail characterization. For multi-view optimization, we introduce a geometry-guided cross-view point cloud association method combined with a dynamic weight sampling strategy. This constructs 3D structural normal constraints across adjacent point cloud frames, effectively reinforcing multi-view consistency and reconstruction fidelity. Extensive experiments on public
datasets demonstrate that our method achieves both competitive rendering quality and
geometric reconstruction. See our interactive project page
Read the full abstract ▾
All authors Xiao Ren, Yu Liu, Ning An, Jian Cheng, Xin Qiao, He Kong
Computer Vision
Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting
Xiaowen Zhang, Zijie Yue, Yong Luo et al. (+3 more)
Object counting is a fundamental task in computer
vision, with broad applicability in many real-world scenarios. Fully-supervised counting methods require costly point-level annotations per object. Few weakly-supervised methods leverage only image-level object counts as supervision and achieve fairly promising results. They are, however, often limited to counting a single category, e.g. person. In this paper, we propose WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting. Instead of directly fine-tuning MLLMs to predict object counts, which can be challenging due to the modality gap, we incorporate three simple yet effective strategies to bootstrap the counting paradigm in both training and testing: First, a divide-and-discern dialogue tuning strategy is proposed to guide the MLLM to determine whether the object count falls within a specific range and progressively break down the range through multi-round dialogue. Second, a compare-and-rank count optimization strategy is introduced to train the MLLM to optimize the relative ranking of multiple images according to their object counts. Third, a global-and-local counting enhancement strategy aggregates and fuses local and global count predictions to improve counting performance in dense scenes. Extensive experiments on FSC-147, CARPK, PUCPR+, and ShanghaiTech show that WS-COC matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs. Code is available at https://github.com/viscom-tongji/WS-COC.
Read the full abstract ▾
All authors Xiaowen Zhang, Zijie Yue, Yong Luo, Cairong Zhao, Qijun Chen, Miaojing Shi
Computer Vision
PixelRush: Ultra-Fast, Training-Free High-Resolution Image Generation via One-step Diffusion
Hong-Phuc Lai, Phong Nguyen and Anh Tran
Pre-trained
diffusion models excel at generating high-quality images but remain inherently limited by their native training resolution. Recent training-free approaches have attempted to overcome this constraint by introducing interventions during the denoising process; however, these methods incur substantial computational overhead, often requiring more than five minutes to produce a single 4K image. In this paper, we present PixelRush, the first tuning-free framework for practical high-resolution text-to-image generation. Our method builds upon the established patch-based inference paradigm but eliminates the need for multiple inversion and regeneration cycles. Instead, PixelRush enables efficient patch-based denoising within a low-step regime. To address artifacts introduced by patch blending in few-step generation, we propose a seamless blending strategy. Furthermore, we mitigate over-smoothing effects through a noise injection mechanism. PixelRush delivers exceptional efficiency, generating 4K images in approximately 20 seconds representing a 10$\times$ to 35$\times$ speedup over state-of-the-art methods while maintaining superior visual fidelity. Extensive experiments validate both the performance gains and the quality of outputs achieved by our approach.
Read the full abstract ▾
Computer Vision
Towards complete digital twins in cultural heritage with ART3mis 3D artifacts annotator
Dimitrios Karamatskos, Vasileios Arampatzakis, Vasileios Sevetlidis et al. (+5 more)
Archaeologists, as well as specialists and practitioners in cultural heritage, require applications with additional functions, such as the annotation and attachment of metadata to specific regions of the 3D digital artifacts, to go beyond the simplistic three-dimensional (3D) visualization. Different strategies addressed this issue, most of which are excellent in their particular area of application, but their capacity is limited to their design's purpose; they lack generalization and interoperability. This paper introduces ART3mis, a general-purpose, user-friendly, feature-rich, interactive web-based textual annotation tool for 3D objects. Moreover, it enables the communication, distribution, and reuse of information as it complies with the W3C Web Annotation Data Model. It is primarily designed to help cultural heritage conservators, restorers, and curators who lack technical expertise in 3D imaging and graphics, handle, segment, and annotate 3D digital replicas of artifacts with ease.
Read the full abstract ▾
All authors Dimitrios Karamatskos, Vasileios Arampatzakis, Vasileios Sevetlidis, Stavros Nousias, Athanasios Kalogeras, Christos Koulamas, Aris Lalos, George Pavlidis
Computer Vision · Cross-listed from eess.IV
VineetVC: Adaptive Video Conferencing Under Severe Bandwidth Constraints Using Audio-Driven Talking-Head Reconstruction
Vineet Kumar Rakesh, Soumya Mazumdar, Tapas Samanta et al. (+3 more)
Intense bandwidth depletion within consumer and constrained networks has the potential to undermine the stability of real-time
video conferencing: encoder rate management becomes saturated, packet loss escalates, frame rates deteriorate, and end-to-end latency significantly increases. This work delineates an adaptive conferencing system that integrates WebRTC media delivery with a supplementary audio-driven talking-head reconstruction pathway and telemetry-driven mode regulation. The system consists of a WebSocket signaling service, an optional SFU for multi-party transmission, a browser client capable of real-time WebRTC statistics extraction and CSV telemetry export, and an AI REST service that processes a reference face image and recorded audio to produce a synthesized MP4; the browser can substitute its outbound camera track with the synthesized stream with a median bandwidth of 32.80 kbps. The solution incorporates a bandwidth-mode switching strategy and a client-side mode-state logger.
Read the full abstract ▾
All authors Vineet Kumar Rakesh, Soumya Mazumdar, Tapas Samanta, Hemendra Kumar Pandey, Amitabha Das, Sarbajit Pal
Computer Vision
Towards reconstructing experimental sparse-view X-ray CT data with diffusion models
Nelas J. Thomsen, Xinyuan Wang, Felix Lucka et al. (+1 more)
Diffusion-based image generators are promising priors for ill-posed inverse problems like sparse-view X-ray Computed
Tomography (CT). As most studies consider synthetic data, it is not clear whether training data mismatch (“domain shift”) or forward model mismatch complicate their successful application to experimental data. We measured CT data from a physical phantom resembling the synthetic Shepp-Logan phantom and trained
diffusion priors on synthetic image data sets with different degrees of domain shift towards it. Then, we employed the priors in a Decomposed
Diffusion Sampling scheme on sparse-view CT data sets with increasing difficulty leading to the experimental data. Our results reveal that domain shift plays a nuanced role: while severe mismatch causes model collapse and
hallucinations, diverse priors match or exceed well-matched but narrow priors. Forward model mismatch pulls the image samples away from the prior
manifold, which causes artifacts but can be mitigated with annealed likelihood weight schedules that also increase computational efficiency. Overall, we demonstrate that performance gains do not immediately translate from synthetic to experimental data, and future development must validate against real-world
benchmarks.
Read the full abstract ▾
All authors Nelas J. Thomsen, Xinyuan Wang, Felix Lucka, Ezgi Demircan-Tureyen
Computer Vision
ReBA-Pred-Net: Weakly-Supervised Regional Brain Age Prediction on MRI
Shuai Shao, Yan Wang, Shu Jiang et al. (+6 more)
Brain age has become a prominent biomarker of
brain health. Yet most prior work targets whole
brain age (WBA), a coarse paradigm that struggles to support tasks such as disease characterization and research on development and aging patterns, because relevant changes are typically region-selective rather than
brain-wide. Therefore, robust regional
brain age (ReBA) estimation is critical, yet a widely generalizable model has yet to be established. In this paper, we propose the Regional
Brain Age Prediction Network (ReBA-Pred-Net), a Teacher-Student framework designed for fine-grained
brain age estimation. The Teacher produces soft ReBA to guide the Student to yield reliable ReBA estimates with a clinical-prior consistency constraint (regions within the same function should change similarly). For rigorous evaluation, we introduce two indirect metrics: Healthy Control Similarity (HCS), which assesses statistical consistency by testing whether regional
brain-age-gap (ReBA minus chronological age) distributions align between training and unseen HC; and Neuro Disease Correlation (NDC), which assesses factual consistency by checking whether clinically confirmed patients show elevated
brain-age-gap in disease-associated regions. Experiments across multiple backbones demonstrate the statistical and factual validity of our method.
Read the full abstract ▾
All authors Shuai Shao, Yan Wang, Shu Jiang, Shiyuan Zhao, Xinzhe Luo, Di Yang, Jiangtao Wang, Yutong Bai, Jianguo Zhang
Computer Vision · Cross-listed from eess.IV
Lung nodule classification on CT scan patches using 3D convolutional neural networks
Volodymyr Sydorskyi
Lung cancer remains one of the most common and deadliest forms of cancer worldwide. The likelihood of successful treatment depends strongly on the stage at which the disease is diagnosed. Therefore, early detection of lung cancer represents a critical medical challenge. However, this task poses significant difficulties for thoracic radiologists due to the large number of studies to review, the presence of multiple nodules within the lungs, and the small size of many nodules, which complicates visual assessment. Consequently, the development of automated systems that incorporate highly accurate and computationally efficient lung nodule detection and classification modules is essential. This study introduces three methodological improvements for lung nodule classification: (1) an advanced CT scan cropping strategy that focuses the model on the target nodule while reducing computational cost; (2) target filtering techniques for removing noisy labels; (3) novel augmentation methods to improve model robustness. The integration of these techniques enables the development of a robust classification subsystem within a comprehensive Clinical Decision Support System for lung cancer detection, capable of operating across diverse acquisition protocols, scanner types, and upstream models (
segmentation or detection). The multiclass model achieved a Macro ROC AUC of 0.9176 and a Macro F1-score of 0.7658, while the binary model reached a Binary ROC AUC of 0.9383 and a Binary F1-score of 0.8668 on the LIDC-IDRI
dataset. These results outperform several previously reported approaches and demonstrate state-of-the-art performance for this task.
Read the full abstract ▾
Computer Vision
Synthetic Craquelure Generation for Unsupervised Painting Restoration
Jana Cuch-Guillén, Antonio Agudo and Raül Pérez-Gonzalo
Cultural heritage preservation increasingly demands non-invasive digital methods for painting restoration, yet identifying and restoring fine craquelure patterns from complex brushstrokes remains challenging due to scarce pixel-level annotations. We propose a fully annotation-free framework driven by a domain-specific synthetic craquelure generator, which simulates realistic branching and tapered fissure geometry using Bézier trajectories. Our approach couples a classical morphological detector with a learning-based refinement module: a SegFormer backbone adapted via Low-Rank Adaptation (LoRA). Uniquely, we employ a detector-guided strategy, injecting the morphological map as an input spatial prior, while a masked hybrid loss and logit adjustment constrain the training to focus specifically on refining candidate crack regions. The refined masks subsequently guide an Anisotropic
Diffusion inpainting stage to reconstruct missing content. Experimental results demonstrate that our pipeline significantly outperforms state-of-the-art photographic restoration models in zero-shot settings, while faithfully preserving the original paint brushwork.
Read the full abstract ▾
Computer Vision
SPRig: Self-Supervised Pose-Invariant Rigging from Mesh Sequences
Ruipeng Wang, Langkun Zhong and Miaowei Wang
State-of-the-art rigging methods typically assume a predefined canonical rest pose. However, this assumption does not hold for dynamic mesh sequences such as DyMesh or DT4D, where no canonical T-pose is available. When applied independently frame-by-frame, existing methods lack pose invariance and often yield temporally inconsistent
topologies. To address this limitation, we propose SPRig, a general fine-tuning framework that enforces cross-frame consistency across a sequence to learn pose-invariant rigs on top of existing models, covering both skeleton and skinning generation. For skeleton generation, we introduce novel consistency regularization in both
token space and geometry space. For skinning, we improve temporal stability through an articulation-invariant consistency loss combined with consistency
distillation and structural regularization. Extensive experiments show that SPRig achieves superior temporal coherence and significantly reduces artifacts in prior methods, without sacrificing and often even enhancing per-frame static generation quality. The code is available in the supplemental material and will be made publicly available upon publication.
Read the full abstract ▾
Computer Vision
VimRAG: Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph
Qiuchen Wang, Shihang Wang, Yu Zeng et al. (+9 more)
Effectively retrieving,
reasoning, and understanding
multimodal information remains a critical challenge for agentic systems. Traditional
Retrieval-augmented Generation (RAG) methods rely on linear interaction histories, which struggle to handle long-context tasks, especially those involving information-sparse yet
token-heavy visual data in iterative
reasoning scenarios. To bridge this gap, we introduce VimRAG, a framework tailored for
multimodal Retrieval-augmented
Reasoning across text, images, and
videos. Inspired by our systematic study, we model the
reasoning process as a dynamic directed acyclic graph that structures the
agent states and retrieved
multimodal evidence. Building upon this structured
memory, we introduce a Graph-Modulated Visual
Memory Encoding mechanism, with which the significance of
memory nodes is evaluated via their topological position, allowing the model to dynamically allocate high-resolution
tokens to pivotal evidence while compressing or discarding trivial clues. To implement this paradigm, we propose a Graph-Guided Policy Optimization strategy. This strategy disentangles step-wise validity from trajectory-level rewards by
pruning memory nodes associated with redundant actions, thereby facilitating fine-grained credit assignment. Extensive experiments demonstrate that VimRAG consistently achieves state-of-the-art performance on diverse
multimodal RAG
benchmarks. The code is available at https://github.com/Alibaba-NLP/VRAG.
Read the full abstract ▾
All authors Qiuchen Wang, Shihang Wang, Yu Zeng, Qiang Zhang, Fanrui Zhang, Zhuoning Guo, Bosi Zhang, Wenxuan Huang, Lin Chen, Zehui Chen, Pengjun Xie, Ruixue Ding
Computer Vision
ART3mis: Ray-Based Textual Annotation on 3D Cultural Objects
Vasileios Arampatzakis, Vasileios Sevetlidis, Fotis Arnaoutoglou et al. (+7 more)
Beyond simplistic 3D visualisations, archaeologists, as well as cultural heritage experts and practitioners, need applications with advanced functionalities. Such as the annotation and attachment of metadata onto particular regions of the 3D digital objects. Various approaches have been presented to tackle this challenge, most of which achieve excellent results in the domain of their application. However, they are often confined to that specific domain and particular problem. In this paper, we present ART3mis - a general-purpose, user-friendly, interactive textual annotation tool for 3D objects. Primarily attuned to aid cultural heritage conservators, restorers and curators with no technical skills in 3D imaging and graphics, the tool allows for the easy handling, segmenting and annotating of 3D digital replicas of artefacts. ART3mis applies a user-driven, direct-on-surface approach. It can handle detailed 3D cultural objects in real-time and store textual annotations for multiple complex regions in JSON data format.
Read the full abstract ▾
All authors Vasileios Arampatzakis, Vasileios Sevetlidis, Fotis Arnaoutoglou, Athanasios Kalogeras, Christos Koulamas, Aris Lalos, Chairi Kiourt, George Ioannakis, Anestis Koutsoudis, George Pavlidis
Computer Vision · Cross-listed from cs.CL
MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs
Baorong Shi, Bo Cui, Boyuan Jiang et al. (+17 more)
We present MedXIAOHE, a medical
vision-
language foundation model designed to advance general-purpose medical understanding and
reasoning in real-world clinical applications. MedXIAOHE achieves state-of-the-art performance across diverse medical
benchmarks and surpasses leading closed-source
multimodal systems on multiple capabilities. To achieve this, we propose an entity-aware continual pretraining framework that organizes heterogeneous medical corpora to broaden knowledge coverage and reduce long-tail gaps (e.g., rare diseases). For medical expert-level
reasoning and interaction, MedXIAOHE incorporates diverse medical
reasoning patterns via reinforcement learning and tool-augmented agentic training, enabling multi-step diagnostic
reasoning with verifiable decision traces. To improve reliability in real-world use, MedXIAOHE integrates user-preference rubrics, evidence-grounded
reasoning, and low-
hallucination long-form report generation, with improved adherence to medical instructions. We release this report to document our practical design choices, scaling insights, and evaluation framework, hoping to inspire further research.
Read the full abstract ▾
All authors Baorong Shi, Bo Cui, Boyuan Jiang, Deli Yu, Fang Qian, Haihua Yang, Huichao Wang, Jiale Chen, Jianfei Pan, Jieqiong Cao, Jinghao Lin, Kai Wu, Lin Yang, Shengsheng Yao, Tao Chen, Xiaojun Xiao, Xiaozhong Ji, Xu Wang, Yijun He, Zhixiong Yang
Computer Vision
Channel-Aware Probing for Multi-Channel Imaging
Umar Marikkar, Syed Sameed Husain, Muhammad Awais et al. (+1 more)
Training and evaluating
vision encoders on Multi-Channel Imaging (MCI) data remains challenging as channel configurations vary across
datasets, preventing fixed-channel training and limiting reuse of pre-trained encoders on new channel settings. Prior work trains MCI encoders but typically evaluates them via full fine-tuning, leaving probing with frozen pre-trained encoders comparatively underexplored. Existing studies that perform probing largely focus on improving representations, rather than how to best leverage fixed representations for downstream tasks. Although the latter problem has been studied in other domains, directly transferring those strategies to MCI yields weak results, even worse than training from scratch. We therefore propose Channel-Aware Probing (CAP), which exploits the intrinsic inter-channel diversity in MCI
datasets by controlling feature flow at both the encoder and probe levels. CAP uses Independent Feature Encoding (IFE) to encode each channel separately, and Decoupled Pooling (DCP) to pool within channels before aggregating across channels. Across three MCI
benchmarks, CAP consistently improves probing performance over the default probing protocol, matches fine-tuning from scratch, and largely reduces the gap to full fine-tuning from the same MCI pre-trained checkpoints. Code can be found in https://github.com/umarikkar/CAP.
Read the full abstract ▾
All authors Umar Marikkar, Syed Sameed Husain, Muhammad Awais, Sara Atito
Computer Vision
Motion Prior Distillation in Time Reversal Sampling for Generative Inbetweening
Wooseok Jeon, Seunghyun Shin, Dongmin Shin et al. (+1 more)
Recent progress in image-to-
video (I2V)
diffusion models has significantly advanced the field of generative inbetweening, which aims to generate semantically plausible frames between two keyframes. In particular, inference-time sampling strategies, which leverage the generative priors of large-scale pre-trained I2V models without additional training, have become increasingly popular. However, existing inference-time sampling, either fusing forward and backward paths in parallel or alternating them sequentially, often suffers from temporal discontinuities and undesirable visual artifacts due to the misalignment between the two generated paths. This is because each path follows the motion prior induced by its own conditioning frame. In this work, we propose Motion Prior
Distillation (MPD), a simple yet effective inference-time
distillation technique that suppresses bidirectional mismatch by distilling the motion residual of the forward path into the backward path. Our method can deliberately avoid denoising the end-conditioned path which causes the ambiguity of the path, and yield more temporally coherent inbetweening results with the forward motion prior. We not only perform quantitative evaluations on standard
benchmarks, but also conduct extensive user studies to demonstrate the effectiveness of our approach in practical scenarios.
Read the full abstract ▾
All authors Wooseok Jeon, Seunghyun Shin, Dongmin Shin, Hae-Gon Jeon
Computer Vision · Cross-listed from cs.LG
SLA2: Sparse-Linear Attention with Learnable Routing and Quantization-aware Training
Jintao Zhang, Haoxu Wang, Kai Jiang et al. (+6 more)
Sparse-Linear
Attention (SLA) combines sparse and linear
attention to accelerate
diffusion models and has shown strong performance in
video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on
attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the
attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear
attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each
attention computation should use sparse or linear
attention, (II) a more faithful and direct sparse-linear
attention formulation that uses a learnable ratio to combine the sparse and linear
attention branches, and (III) a sparse + low-bit
attention design, where low-bit
attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on
video diffusion models, SLA2 can achieve 97%
attention sparsity and deliver an 18.6x
attention speedup while preserving generation quality.
Read the full abstract ▾
All authors Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, Youhe Jiang, Ion Stoica, Jianfei Chen, Jun Zhu, Joseph E. Gonzalez
Computer Vision
IndicFairFace: Balanced Indian Face Dataset for Auditing and Mitigating Geographical Bias in Vision-Language Models
Aarish Shah Mohsin, Mohammed Tayyab Ilyas Khan, Mohammad Nadeem et al. (+3 more)
Vision-
Language Models (VLMs) are known to inherit and amplify societal biases from their web-scale training data with Indian being particularly misrepresented. Existing
fairness-aware
datasets have significantly improved demographic balance across global race and gender groups, yet they continue to treat Indian as a single monolithic category. The oversimplification ignores the vast intra-national diversity across 28 states and 8 Union Territories of India and leads to representational and geographical bias. To address the limitation, we present IndicFairFace, a novel and balanced face
dataset comprising 14,400 images representing geographical diversity of India. Images were sourced ethically from Wikimedia Commons and open-license web repositories and uniformly balanced across states and gender. Using IndicFairFace, we quantify intra-national geographical bias in prominent CLIP-based VLMs and reduce it using post-hoc Iterative Nullspace Projection debiasing approach. We also show that the adopted debiasing approach does not adversely impact the existing
embedding space as the average drop in
retrieval accuracy on
benchmark datasets is less than 1.5 percent. Our work establishes IndicFairFace as the first
benchmark to study geographical bias in VLMs for the Indian context.
Read the full abstract ▾
All authors Aarish Shah Mohsin, Mohammed Tayyab Ilyas Khan, Mohammad Nadeem, Shahab Saquib Sohail, Erik Cambria, Jiechao Gao
Computer Vision
CBEN – A Multimodal Machine Learning Dataset for Cloud Robust Remote Sensing Image Understanding
Marco Stricker, Masakazu Iwamura and Koichi Kise
Clouds are a common phenomenon that distorts optical satellite imagery, which poses a challenge for remote sensing. However, in the literature cloudless analysis is often performed where cloudy images are excluded from machine learning
datasets and methods. Such an approach cannot be applied to time sensitive applications, e.g., during natural disasters. A possible solution is to apply cloud removal as a preprocessing step to ensure that cloudfree solutions are not failing under such conditions. But cloud removal methods are still actively researched and suffer from drawbacks, such as generated visual artifacts. Therefore, it is desirable to develop cloud robust methods that are less affected by cloudy weather. Cloud robust methods can be achieved by combining optical data with radar, a modality unaffected by clouds. While many
datasets for machine learning combine optical and radar data, most researchers exclude cloudy images. We identify this exclusion from machine learning training and evaluation as a limitation that reduces applicability to cloudy scenarios. To investigate this, we assembled a
dataset, named CloudyBigEarthNet (CBEN), of paired optical and radar images with cloud occlusion for training and evaluation. Using average precision (AP) as the evaluation metric, we show that state-of-the-art methods trained on combined clear-sky optical and radar imagery suffer performance drops of 23-33 percentage points when evaluated on cloudy images. We then adapt these methods to cloudy optical data during training, achieving relative improvement of 17.2-28.7 percentage points on cloudy test cases compared with the original approaches. Code and
dataset are publicly available at: https://github.com/mstricker13/CBEN
Read the full abstract ▾
Computer Vision
Multi-Task Learning with Additive U-Net for Image Denoising and Classification
Vikram Lakkavalli and Neelam Sinha
We investigate additive skip fusion in U-Net architectures for image denoising and denoising-centric multi-task learning (MTL). By replacing concatenative skips with gated additive fusion, the proposed Additive U-Net (AddUNet) constrains shortcut capacity while preserving fixed feature dimensionality across depth. This structural regularization induces controlled encoder-decoder information flow and stabilizes joint optimization. Across single-task denoising and joint denoising-classification settings, AddUNet achieves competitive reconstruction performance with improved training stability. In MTL, learned skip weights exhibit systematic task-aware redistribution: shallow skips favor reconstruction, while deeper features support discrimination. Notably, reconstruction remains robust even under limited classification capacity, indicating implicit task decoupling through additive fusion. These findings show that simple constraints on skip connections act as an effective architectural regularizer for stable and scalable multi-task learning without increasing model complexity.
Read the full abstract ▾
Computer Vision
ImageRAGTurbo: Towards One-step Text-to-Image Generation with Retrieval-Augmented Diffusion Models
Peijie Qiu, Hariharan Ramshankar, Arnau Ramisa et al. (+4 more)
Diffusion models have emerged as the leading approach for text-to-image generation. However, their iterative sampling process, which gradually morphs random noise into coherent images, introduces significant latency that limits their applicability. While recent few-step
diffusion models reduce the number of sampling steps to as few as one to four steps, they often compromise image quality and
prompt alignment, especially in one-step generation. Additionally, these models require computationally expensive training procedures. To address these limitations, we propose ImageRAGTurbo, a novel approach to efficiently finetune few-step
diffusion models via
retrieval augmentation. Given a text
prompt, we retrieve relevant text-image pairs from a database and use them to condition the generation process. We argue that such retrieved examples provide rich contextual information to the UNet denoiser that helps reduce the number of denoising steps without compromising image quality. Indeed, our initial investigations show that using the retrieved content to edit the denoiser's latent space ($\mathcal{H}$-space) without additional finetuning already improves
prompt fidelity. To further improve the quality of the generated images, we augment the UNet denoiser with a trainable adapter in the $\mathcal{H}$-space, which efficiently blends the retrieved content with the target
prompt using a cross-
attention mechanism. Experimental results on fast text-to-image generation demonstrate that our approach produces high-fidelity images without compromising latency compared to existing methods.
Read the full abstract ▾
All authors Peijie Qiu, Hariharan Ramshankar, Arnau Ramisa, René Vidal, Amit Kumar K C, Vamsi Salaka, Rahul Bhagat
Computer Vision · Cross-listed from cs.LG
Formalizing the Sampling Design Space of Diffusion-Based Generative Models via Adaptive Solvers and Wasserstein-Bounded Timesteps
Sangwoo Jo and Sungjoon Choi
Diffusion-based generative models have achieved remarkable performance across various domains, yet their practical deployment is often limited by high sampling costs. While prior work focuses on training objectives or individual solvers, the broader sampling design problem, specifically solver selection and scheduling, remains largely governed by static heuristics. We propose SDM, a principled, training-free sampling framework that adapts both the numerical solver and the timestep schedule to the intrinsic properties of the
diffusion trajectory. By analyzing the PF-ODE dynamics, we show that velocity variation is small in high-noise stages and increases near the data
manifold, identifying intervals where solver order is most consequential. In parallel, we introduce an offline-calibrated adaptive scheduling method that explicitly controls the local Wasserstein discretization error and projects the calibrated trajectory to a prescribed NFE budget. We further extend the formulation to a mixed-transition Wasserstein error bound, providing a unified error-propagation view of adaptive scheduling and solver selection within the overall SDM framework. Across standard
benchmarks, with extensions to modern ODE samplers, high-resolution synthesis, and text-to-image generation, SDM achieves improved sample quality compared to baseline methods, attaining an FID of 1.93 on CIFAR-10, 2.41 on FFHQ, and 1.98 on AFHQv2, with a reduced number of function evaluations compared to existing samplers. Our code is available at https://github.com/aiimaginglab/sdm.
Read the full abstract ▾
Computer Vision
Omer Faruk Deniz, Ruiyu Mao, Ruochen Li et al. (+2 more)
Multimodal Large
Language Models (MLLMs) incur significant computational cost from processing numerous
vision tokens through all LLM layers. Prior
pruning methods operate either before the LLM, limiting generality due to diverse encoder-projector designs or within the LLM using heuristics that are incompatible with FlashAttention. We take a different approach: rather than identifying unimportant
tokens, we treat the LLM itself as the optimal guide for compression. Observing that deeper layers naturally transmit
vision-to-text information, we introduce
Attention-Driven Self-Compression (ADSC), a simple, broadly applicable method that progressively reduces
vision tokens using only the LLM's
attention mechanism. Our method applies uniform
token downsampling at selected layers, forming bottlenecks that encourage the model to reorganize and compress information into the remaining
tokens. It requires no score computation, auxiliary modules, or
attention modification, and remains fully compatible with FlashAttention. Applied to LLaVA-1.5, ADSC reduces FLOPs by 53.7% and peak KV-cache
memory by 56.7%, while preserving 98.2% of the original model performance. Across multiple
benchmarks, it outperforms prior
pruning approaches in both efficiency and accuracy. Crucially, under high compression ratios, our method remains robust while heuristic-based techniques degrade sharply.
Read the full abstract ▾
All authors Omer Faruk Deniz, Ruiyu Mao, Ruochen Li, Yapeng Tian, Latifur Khan
Computer Vision
QuEPT: Quantized Elastic Precision Transformers with One-Shot Calibration for Multi-Bit Switching
Ke Xu, Yixin Wang, Zhongcheng Li et al. (+3 more)
Elastic precision quantization enables multi-bit deployment via a single optimization pass, fitting diverse quantization scenarios.Yet, the high storage and optimization costs associated with the
Transformer architecture, research on elastic quantization remains limited, particularly for large
language models.This paper proposes QuEPT, an efficient post-training scheme that reconstructs block-wise multi-bit errors with one-shot calibration on a small data slice. It can dynamically adapt to various predefined bit-widths by cascading different low-rank adapters, and supports real-time switching between uniform quantization and mixed precision quantization without repeated optimization. To enhance accuracy and robustness, we introduce Multi-Bit
Token Merging (MB-ToMe) to dynamically fuse
token features across different bit-widths, improving robustness during bit-width switching. Additionally, we propose Multi-Bit Cascaded Low-Rank adapters (MB-CLoRA) to strengthen correlations between bit-width groups, further improve the overall performance of QuEPT. Extensive experiments demonstrate that QuEPT achieves comparable or better performance to existing state-of-the-art post-training quantization methods.Our code is available at https://github.com/xuke225/QuEPT
Read the full abstract ▾
All authors Ke Xu, Yixin Wang, Zhongcheng Li, Hao Cui, Jinshui Hu, Xingyi Zhang
Computer Vision
Unbiased Gradient Estimation for Event Binning via Functional Backpropagation
Jinze Chen, Wei Zhai, Han Han et al. (+4 more)
Event-based
vision encodes dynamic scenes as asynchronous spatio-temporal spikes called events. To leverage conventional image processing pipelines, events are typically binned into frames. However, binning functions are discontinuous, which truncates gradients at the frame level and forces most event-based
algorithms to rely solely on frame-based features. Attempts to directly learn from raw events avoid this restriction but instead suffer from biased gradient estimation due to the discontinuities of the binning operation, ultimately limiting their learning efficiency. To address this challenge, we propose a novel framework for unbiased gradient estimation of arbitrary binning functions by synthesizing weak derivatives during backpropagation while keeping the forward output unchanged. The key idea is to exploit integration by parts: lifting the target functions to functionals yields an integral form of the derivative of the binning function during backpropagation, where the cotangent function naturally arises. By reconstructing this cotangent function from the sampled cotangent vector, we compute weak derivatives that provably match long-range finite differences of both smooth and non-smooth targets. Experimentally, our method improves simple optimization-based egomotion estimation with 3.2% lower RMS error and 1.57$\times$ faster convergence. On complex downstream tasks, we achieve 9.4% lower EPE in self-supervised optical flow, and 5.1% lower RMS error in SLAM, demonstrating broad benefits for event-based visual perception. Source code can be found at https://github.com/chjz1024/EventFBP.
Read the full abstract ▾
All authors Jinze Chen, Wei Zhai, Han Han, Tiankai Ma, Yang Cao, Bin Li, Zheng-Jun Zha
Computer Vision
The Constant Eye: Benchmarking and Bridging Appearance Robustness in Autonomous Driving
Jiabao Wang, Hongyu Zhou, Yuanbo Yang et al. (+2 more)
Despite rapid progress, autonomous driving
algorithms remain notoriously fragile under Out-of-Distribution (OOD) conditions. We identify a critical decoupling failure in current research: the lack of distinction between appearance-based shifts, such as weather and lighting, and structural scene changes. This leaves a fundamental question unanswered: Is the planner failing because of complex road geometry, or simply because it is raining? To resolve this, we establish navdream, a high-fidelity robustness
benchmark leveraging generative pixel-aligned style transfer. By creating a visual stress test with negligible
geometric deviation, we isolate the impact of appearance on driving performance. Our evaluation reveals that existing planning
algorithms often show significant degradation under OOD appearance conditions, even when the underlying scene structure remains consistent. To bridge this gap, we propose a universal perception interface leveraging a frozen visual foundation model (DINOv3). By extracting appearance-invariant features as a stable interface for the planner, we achieve exceptional zero-shot generalization across diverse planning paradigms, including regression-based,
diffusion-based, and scoring-based models. Our plug-and-play solution maintains consistent performance across extreme appearance shifts without requiring further fine-tuning. The
benchmark and code will be made available.
Read the full abstract ▾
All authors Jiabao Wang, Hongyu Zhou, Yuanbo Yang, Jiahao Shao, Yiyi Liao
Computer Vision
PLLM: Pseudo-Labeling Large Language Models for CAD Program Synthesis
Yuanbo Li, Dule Shu, Yanying Chen et al. (+2 more)
Recovering Computer-Aided Design (CAD) programs from 3D geometries is a widely studied problem. Recent advances in large
language models (LLMs) have enabled progress in CAD program synthesis, but existing methods rely on supervised training with paired shape-program data, which is often unavailable. We introduce PLLM, a self-training framework for CAD program synthesis from unlabeled 3D shapes. Given a pre-trained CAD-capable LLM and a shape
dataset, PLLM iteratively samples candidate programs, selects high-fidelity executions, and augments programs to construct synthetic program-shape pairs for fine-tuning. We experiment on adapting CAD-Recode from DeepCAD to the unlabeled ABC
dataset show consistent improvements in
geometric fidelity and program diversity.
Read the full abstract ▾
All authors Yuanbo Li, Dule Shu, Yanying Chen, Matt Klenk, Daniel Ritchie
Computer Vision
Self-Supervised JEPA-based World Models for LiDAR Occupancy Completion and Forecasting
Haoran Zhu and Anna Choromanska
Autonomous driving, as an
agent operating in the physical world, requires the fundamental capability to build world models that capture how the environment evolves spatiotemporally in order to support long-term planning. At the same time, scalability demands learning such models in a self-supervised manner; joint-
embedding predictive architecture (JEPA) enables learning world models via leveraging large volumes of unlabeled data without relying on expensive human annotations. In this paper, we propose AD-LiST-JEPA, a self-supervised world model for autonomous driving that predicts future spatiotemporal
evolution from
LiDAR data using a JEPA framework. We evaluate the quality of the learned representations through a downstream
LiDAR-based occupancy completion and
forecasting (OCF) task, which jointly assesses perception and prediction. Proof of concept experiments show better OCF performance with pretrained encoder after JEPA-based world model learning.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models
Bowen Ping, Chengyou Jia, Minnan Luo et al. (+2 more)
Reinforcement learning has emerged as a promising paradigm for aligning
diffusion and flow-matching models with human preferences, yet practitioners face fragmented codebases, model-specific implementations, and engineering complexity. We introduce Flow-Factory, a unified framework that decouples
algorithms, models, and rewards through through a modular, registry-based architecture. This design enables seamless integration of new
algorithms and architectures, as demonstrated by our support for GRPO, DiffusionNFT, and AWM across Flux, Qwen-Image, and WAN
video models. By minimizing implementation overhead, Flow-Factory empowers researchers to rapidly prototype and scale future innovations with ease. Flow-Factory provides production-ready
memory optimization, flexible multi-reward training, and seamless distributed training support. The codebase is available at https://github.com/X-GenGroup/Flow-Factory.
Read the full abstract ▾
All authors Bowen Ping, Chengyou Jia, Minnan Luo, Hangwei Qian, Ivor Tsang
Computer Vision
Geometric Stratification for Singular Configurations of the P3P Problem via Local Dual Space
Xueying Sun, Zijia Li and Nan Li
This paper investigates singular configurations of the P3P problem. Using local dual space, a systematic algebraic-computational framework is proposed to give a complete
geometric stratification for the P3P singular configurations with respect to the multiplicity $\mu$ of the camera center $O$: for $\mu\ge 2$, $O$ lies on the “danger cylinder”, for $\mu\ge 3$, $O$ lies on one of three generatrices of the danger cylinder associated with the first Morley triangle or the circumcircle, and for $\mu\ge 4$, $O$ lies on the circumcircle which indeed corresponds to infinite P3P solutions. Furthermore, a
geometric stratification for the complementary configuration $O^\prime$ associated with a singular configuration $O$ is studied as well: for $\mu\ge 2$, $O^\prime$ lies on a deltoidal surface associated with the danger cylinder, and for $\mu\ge 3$, $O^\prime$ lies on one of three cuspidal curves of the deltoidal surface.
Read the full abstract ▾