Computer Vision · Cross-listed from cs.LG
Faster Inference of Flow-Based Generative Models via Improved Data-Noise Coupling
Aram Davtyan, Leello Tadesse Dadi, Volkan Cevher et al. (+1 more)
Conditional Flow Matching (CFM), a simulation-free method for training continuous normalizing flows, provides an efficient alternative to
diffusion models for key tasks like image and
video generation. The performance of CFM in solving these tasks depends on the way data is coupled with noise. A recent approach uses minibatch optimal transport (OT) to reassign noise-data pairs in each training step to streamline sampling trajectories and thus accelerate inference. However, its optimization is restricted to individual minibatches, limiting its effectiveness on large
datasets. To address this shortcoming, we introduce LOOM-CFM (Looking Out Of Minibatch-CFM), a novel method to extend the scope of minibatch OT by preserving and optimizing these assignments across minibatches over training time. Our approach demonstrates consistent improvements in the sampling speed-quality trade-off across multiple
datasets. LOOM-CFM also enhances
distillation initialization and supports high-resolution synthesis in latent space training.
Read the full abstract ▾
All authors Aram Davtyan, Leello Tadesse Dadi, Volkan Cevher, Paolo Favaro
Computer Vision
Dataset Diversity Metrics and Impact on Classification Models
Théo Sourget, Niclas Claßen, Jack Junchi Xu et al. (+2 more)
The diversity of training
datasets is usually perceived as an important aspect to obtain a robust model. However, the definition of diversity is often not defined or differs across papers, and while some metrics exist, the quantification of this diversity is often overlooked when developing new
algorithms. In this work, we study the behaviour of multiple
dataset diversity metrics for image, text and metadata using MorphoMNIST, a toy
dataset with controlled perturbations, and PadChest, a publicly available chest X-ray
dataset. We evaluate whether these metrics correlate with each other but also with the intuition of a clinical expert. We also assess whether they correlate with downstream-task performance and how they impact the training dynamic of the models. We find limited correlations between the AUC and image or metadata reference-free diversity metrics, but higher correlations with the FID and the semantic diversity metrics. Finally, the clinical expert indicates that scanners are the main source of diversity in practice. However, we find that the addition of another scanner to the training set leads to shortcut learning. The code used in this study is available at https://github.com/TheoSourget/
dataset_diversity_evaluation
Read the full abstract ▾
All authors Théo Sourget, Niclas Claßen, Jack Junchi Xu, Rob van der Goot, Veronika Cheplygina
Computer Vision
Flash-Unified: A Training-Free and Task-Aware Acceleration Framework for Native Unified Models
Junlong Ke, Zichen Wen, Boxue Yang et al. (+6 more)
Native unified
multimodal models, which integrate both generative and understanding capabilities, face substantial computational overhead that hinders their real-world deployment. Existing acceleration techniques typically employ a static, monolithic strategy, ignoring the fundamental divergence in computational profiles between iterative generation tasks (e.g., image generation) and single-pass understanding tasks (e.g., VQA). In this work, we present the first systematic analysis of unified models, revealing pronounced parameter specialization, where distinct
neuron sets are critical for each task. This implies that, at the parameter level, unified models have implicitly internalized separate inference pathways for generation and understanding within a single architecture. Based on these insights, we introduce a training-free and task-aware acceleration framework, FlashU, that tailors optimization to each task's demands. Across both tasks, we introduce Task-Specific Network
Pruning and Dynamic Layer Skipping, aiming to eliminate inter-layer and task-specific redundancy. For visual generation, we implement a time-varying control signal for the guidance scale and a temporal approximation for the
diffusion head via
Diffusion Head Cache. For
multimodal understanding, building upon the pruned model, we introduce Dynamic
Token Pruning via a V-Norm Proxy to exploit the spatial redundancy of visual inputs. Extensive experiments on Show-o2 demonstrate that FlashU achieves 1.78$\times$ to 2.01$\times$ inference acceleration across both understanding and generation tasks while maintaining SOTA performance, outperforming competing unified models and validating our task-aware acceleration paradigm. Our code is publicly available at https://github.com/Rirayh/FlashU.
Read the full abstract ▾
All authors Junlong Ke, Zichen Wen, Boxue Yang, Yantai Yang, Xuyang Liu, Chenfei Liao, Zhaorun Chen, Shaobo Wang, Linfeng Zhang
Computer Vision
Self-Supervised ImageNet Representations for In Vivo Confocal Microscopy: Tortuosity Grading without Segmentation Maps
Kim Ouan, Noémie Moreau and Katarzyna Bozek
The tortuosity of corneal nerve fibers are used as indication for different diseases. Current state-of-the-art methods for grading the tortuosity heavily rely on expensive
segmentation maps of these nerve fibers. In this paper, we demonstrate that self-supervised pretrained features from ImageNet are transferable to the domain of in vivo confocal microscopy. We show that DINO should not be disregarded as a deep learning model for medical imaging, although it was superseded by two later versions. After careful fine-tuning, DINO improves upon the state-of-the-art in terms of accuracy (84,25%) and sensitivity (77,97%). Our fine-tuned model focuses on the key morphological elements in grading without the use of
segmentation maps.
Read the full abstract ▾
Computer Vision
Exemplar Diffusion: Improving Medical Object Detection with Opportunistic Labels
Victor Wåhlstrand, Jennifer Alvén and Ida Häggström
We present a framework to take advantage of existing labels at inference, called exemplars, in order to improve the performance of object detection in medical images. The method, exemplar
diffusion, leverages existing
diffusion methods for object detection to enable a training-free approach to adding information of known bounding boxes at test time. We demonstrate that for medical image
datasets with clear spatial structure, the method yields an across-the-board increase in average precision and recall, and a robustness to exemplar quality, enabling non-expert annotation. Moreover, we demonstrate how our method may also be used to quantify predictive uncertainty in
diffusion detection methods. Source code and data splits openly available online: https://github.com/waahlstrand/ExemplarDiffusion
Read the full abstract ▾
Computer Vision
IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learning
Konstantinos Almpanakis and Anna Kreshuk
Self-supervised learning (SSL) has revolutionized representation learning, with Joint-
Embedding Architectures (JEAs) emerging as an effective approach for capturing semantic features. Existing JEAs rely on implicit or explicit batch interaction – via negative sampling or statistical regularization – to prevent representation collapse. This reliance becomes problematic in regimes where batch sizes must be small, such as high-dimensional scientific data, where
memory constraints and class imbalance make large, well-balanced batches infeasible. We introduce IConE (Instance-Contrasted
Embeddings), a framework that decouples collapse prevention from the training batch size. Rather than enforcing diversity through batch statistics, IConE maintains a global set of learnable auxiliary instance
embeddings regularized by an explicit diversity objective. This transfers the anti-collapse mechanism from the transient batch to a
dataset-level
embedding space, allowing stable training even when batch statistics are unreliable, down to batch size 1. Across diverse 2D and 3D biomedical modalities, IConE outperforms strong contrastive and non-contrastive baselines throughout the small-batch regime (from B=1 to B=64) and demonstrates marked robustness to severe class imbalance.
Geometric analysis shows that IConE preserves high intrinsic dimensionality in the learned representations, preventing the collapse observed in existing JEAs as batch sizes shrink.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecasting
Jing Wu, Yang Liu, Lin Zhang et al. (+13 more)
Accurate weather
forecasting is more than grid-wise regression: it must preserve coherent synoptic structures and physical consistency of meteorological fields, especially under autoregressive rollouts where small one-step errors can amplify into structural bias. Existing physics-priors approaches typically impose global, once-for-all constraints via architectures, regularization, or NWP coupling, offering limited state-adaptive and sample-specific controllability at deployment. To bridge this gap, we propose
Agent-Guided Cross-modal Decoding (AGCD), a plug-and-play decoding-time prior-injection paradigm that derives state-conditioned physics-priors from the current multivariate atmosphere and injects them into forecasters in a controllable and reusable way. Specifically, We design a multi-
agent meteorological narration pipeline to generate state-conditioned physics-priors, utilizing MLLMs to extract various meteorological elements effectively. To effectively apply the priors, AGCD further introduce cross-modal region interaction decoding that performs region-aware multi-scale tokenization and efficient physics-priors injection to refine visual features without changing the backbone interface. Experiments on WeatherBench demonstrate consistent gains for 6-hour
forecasting across two resolutions (5.625 degree and 1.40625 degree) and diverse backbones (generic and weather-specialized), including strictly causal 48-hour autoregressive rollouts that reduce early-stage error accumulation and improve long-horizon stability.
Read the full abstract ▾
All authors Jing Wu, Yang Liu, Lin Zhang, Junbo Zeng, Jiabin Wang, Zi Ye, Guowen Li, Shilei Cao, Jiashun Cheng, Fang Wang, Meng Jin, Yerong Feng, Hong Cheng, Yutong Lu, Haohuan Fu, Juepeng Zheng
Computer Vision
HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning
Kuniaki Saito, Risa Shinoda, Shohei Tanaka et al. (+3 more)
Hallucination detection in captions (HalDec) assesses a
vision-
language model's ability to correctly align image content with text by identifying errors in captions that misrepresent the image. Beyond evaluation, effective
hallucination detection is also essential for curating high-quality image-caption pairs used to train VLMs. However, the generalizability of VLMs as
hallucination detectors across different captioning models and
hallucination types remains unclear due to the lack of a comprehensive
benchmark. In this work, we introduce HalDec-Bench, a
benchmark designed to evaluate
hallucination detectors in a principled and interpretable manner. HalDec-Bench contains captions generated by diverse VLMs together with human annotations indicating the presence of
hallucinations, detailed
hallucination-type categories, and segment-level labels. The
benchmark provides tasks with a wide range of difficulty levels and reveals performance differences across models that are not visible in existing
multimodal reasoning or
alignment benchmarks. Our analysis further uncovers two key findings. First, detectors tend to recognize sentences appearing at the beginning of a response as correct, regardless of their actual correctness. Second, our experiments suggest that
dataset noise can be substantially reduced by using strong VLMs as filters while employing recent VLMs as caption generators. Our project page is available at https://dahlian00.github.io/HalDec-Bench-Page/.
Read the full abstract ▾
All authors Kuniaki Saito, Risa Shinoda, Shohei Tanaka, Tosho Hirasawa, Fumio Okura, Yoshitaka Ushiku
Computer Vision
Multi-turn Physics-informed Vision-language Model for Physics-grounded Anomaly Detection
Yao Gu, Xiaohao Xu and Yingna Wu
Vision-
Language Models (VLMs) demonstrate strong general-purpose
reasoning but remain limited in physics-grounded anomaly detection, where causal understanding of dynamics is essential. Existing VLMs, trained predominantly on appearance-centric correlations, fail to capture kinematic constraints, leading to poor performance on anomalies such as irregular rotations or violated mechanical motions. We introduce a physics-informed instruction tuning framework that explicitly encodes object properties, motion paradigms, and dynamic constraints into structured
prompts. By delivering these physical priors through multi-turn dialogues, our method decomposes causal
reasoning into incremental steps, enabling robust internal representations of normal and abnormal dynamics. Evaluated on the Phys-AD
benchmark, our approach achieves 96.7% AUROC in
video-level detection–substantially outperforming prior SOTA (66.9%)–and yields superior causal explanations (0.777 LLM score). This work highlights how structured physics priors can transform VLMs into reliable detectors of dynamic anomalies.
Read the full abstract ▾
Computer Vision
HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understanding
Xuerui Qiu, Yutao Cui, Guozhen Zhang et al. (+9 more)
Unified
Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches typically compromise by employing decoupled encoders, stacking representation encoder atop VAEs, or utilizing discrete quantization. However, these methods often disrupt information coherence and lead to optimization conflicts. To this end, we introduce HYDRA-TOK, a representation-harmonized pure ViT in the insight that visual modeling should evolve from generation to understanding. HYDRA-TOK reformulates the standard backbone into a progressive learner that transitions from a Gen-ViT, which captures structure-preserving primitives, to a Sem-ViT for semantic encoding. Crucially, this transition is mediated by a Generation-Semantic Bottleneck (GSB), which compresses features into a low-dimensional space to filter noise for robust synthesis, then restores dimensionality to empower complex semantic comprehension. Built upon this foundation, we present HYDRA, a native unified framework integrating perception and generation within a single parameter space. Extensive experiments establish HYDRA as a new state-of-the-art. It sets a
benchmark in visual reconstruction (rFID 0.08) and achieves top-tier generation performance on GenEval (0.86), DPG-Bench (86.4), and WISE (0.53), while simultaneously outperforming previous native UMMs by an average of ~10.0 points across eight challenging understanding
benchmarks.
Read the full abstract ▾
All authors Xuerui Qiu, Yutao Cui, Guozhen Zhang, Junzhe Li, Yaqi Zhao, Xiao Zhang, Yang Li, Songtao Liu, Miles Yang, Yu Shi, Zhao Zhong, Liefeng Bo
Computer Vision
Tracking the Discriminative Axis: Dual Prototypes for Test-Time OOD Detection Under Covariate Shift
Wooseok Lee, Jin Mo Yang, Saewoong Bahk et al. (+1 more)
For reliable deployment of deep-learning systems, out-of-distribution (OOD) detection is indispensable. In the real world, where test-time inputs often arrive as streaming mixtures of in-distribution (ID) and OOD samples under evolving
covariate shifts, OOD samples are domain-constrained and bounded by the environment, and both ID and OOD are jointly affected by the same
covariate factors. Existing methods typically assume a stationary ID distribution, but this assumption breaks down in such settings, leading to severe performance degradation. We empirically discover that, even under
covariate shift,
covariate-shifted ID (csID) and OOD (csOOD) samples remain separable along a discriminative axis in feature space. Building on this observation, we propose DART, a test-time, online OOD detection method that dynamically tracks dual prototypes – one for ID and the other for OOD – to recover the drifting discriminative axis, augmented with multi-layer fusion and flip correction for robustness. Extensive experiments on a wide range of challenging
benchmarks, where all
datasets are subjected to 15 common corruption types at severity level 5, demonstrate that our method significantly improves performance, yielding 15.32 percentage points (pp) AUROC gain and 49.15 pp FPR@95TPR reduction on ImageNet-C vs. Textures-C compared to established baselines. These results highlight the potential of the test-time discriminative axis tracking for dependable OOD detection in dynamically changing environments.
Read the full abstract ▾
All authors Wooseok Lee, Jin Mo Yang, Saewoong Bahk, Hyung-Sin Kim
Computer Vision · Cross-listed from cs.CL
Efficient Document Parsing via Parallel Token Prediction
Lei Li, Ze Zhao, Meng Li et al. (+6 more)
Document parsing, as a fundamental yet crucial
vision task, is being revolutionized by
vision-
language models (VLMs). However, the autoregressive (AR) decoding inherent to VLMs creates a significant bottleneck, severely limiting parsing speed. In this paper, we propose Parallel-
Token Prediction (PTP), a plugable, model-agnostic and simple-yet-effective method that enables VLMs to generate multiple future
tokens in parallel with improved sample efficiency. Specifically, we insert some learnable
tokens into the input sequence and design corresponding training objectives to equip the model with parallel decoding capabilities for document parsing. Furthermore, to support effective training, we develop a comprehensive data generation pipeline that efficiently produces large-scale, high-quality document parsing training data for VLMs. Extensive experiments on OmniDocBench and olmOCR-bench demonstrate that our method not only significantly improves decoding speed (1.6x-2.2x) but also reduces model
hallucinations and exhibits strong generalization abilities.
Read the full abstract ▾
All authors Lei Li, Ze Zhao, Meng Li, Zhongwang Lun, Yi Yuan, Xingjing Lu, Zheng Wei, Jiang Bian, Zang Li
Computer Vision · Cross-listed from cs.RO
What Matters for Scalable and Robust Learning in End-to-End Driving Planners?
David Holtz, Niklas Hanselmann, Simon Doll et al. (+2 more)
End-to-end autonomous driving has gained significant
attention for its potential to learn robust behavior in interactive scenarios and scale with data. Popular architectures often build on separate modules for perception and planning connected through latent representations, such as bird's eye view feature grids, to maintain end-to-end differentiability. This paradigm emerged mostly on open-loop
datasets, with evaluation focusing not only on driving performance, but also intermediate perception tasks. Unfortunately, architectural advances that excel in open-loop often fail to translate to scalable learning of robust closed-loop driving. In this paper, we systematically re-examine the impact of common architectural patterns on closed-loop performance: (1) high-resolution perceptual representations, (2) disentangled trajectory representations, and (3) generative planning. Crucially, our analysis evaluates the combined impact of these patterns, revealing both unexpected limitations as well as underexplored synergies. Building on these insights, we introduce BevAD, a novel lightweight and highly scalable end-to-end driving architecture. BevAD achieves 72.7% success rate on the Bench2Drive
benchmark and demonstrates strong data-scaling behavior using pure imitation learning. Our code and models are publicly available here: https://dmholtz.github.io/bevad/
Read the full abstract ▾
All authors David Holtz, Niklas Hanselmann, Simon Doll, Marius Cordts, Bernt Schiele
Computer Vision
Multimodal Connectome Fusion via Cross-Attention for Autism Spectrum Disorder Classification Using Graph Learning
Ansar Rahman, Hassan Shojaee-Mend and Sepideh Hatamikia
Autism spectrum disorder (ASD) is a complex neurodevelopmental condition characterized by atypical functional
brain connectivity and subtle structural alterations. rs-fMRI has been widely used to identify disruptions in large-scale
brain networks, while structural MRI provides complementary information about morphological organization. Despite their complementary nature, effectively integrating these heterogeneous imaging modalities within a unified framework remains challenging. This study proposes a
multimodal graph learning framework that preserves the dominant role of functional connectivity while integrating structural imaging and phenotypic information for ASD classification. The proposed framework is evaluated on ABIDE-I
dataset. Each subject is represented as a node within a population graph. Functional and structural features are extracted as modality-specific node attributes, while inter-subject relationships are modeled using a pairwise association encoder (PAE) based on phenotypic information. Two Edge Variational GCNs are trained to learn subject-level
embeddings. To enable effective
multimodal integration, we introduce a novel asymmetric
transformer-based cross-
attention mechanism that allows functional
embeddings to selectively incorporate complementary structural information while preserving functional dominance. The fused
embeddings are then passed to a MLP for ASD classification. Using stratified 10-fold cross-validation, the framework achieved an AUC of 87.3% and an accuracy of 84.4%. Under leave-one-site-out cross-validation (LOSO-CV), the model achieved an average cross-site accuracy of 82.0%, outperforming existing methods by approximately 3% under 10-fold cross-validation and 7% under LOSO-CV. The proposed framework effectively integrates heterogeneous
multimodal data from the multi-site ABIDE-I
dataset, improving automated ASD classification across imaging sites.
Read the full abstract ▾
Computer Vision
Question-guided Visual Compression with Memory Feedback for Long-Term Video Understanding
Sosuke Yamao, Natsuki Miyahara, Yuankai Qi et al. (+1 more)
In the context of long-term
video understanding with large
multimodal models, many frameworks have been proposed. Although
transformer-based visual compressors and
memory-augmented approaches are often used to process long
videos, they usually compress each frame independently and therefore fail to achieve strong performance on tasks that require understanding complete events, such as temporal ordering tasks in MLVU and VNBench. This motivates us to rethink the conventional one-way scheme from perception to
memory, and instead establish a feedbackdriven process in which past visual contexts stored in the context
memory can benefit ongoing perception. To this end, we propose Question-guided Visual Compression with
Memory Feedback (QViC-MF), a framework for long-term
video understanding. At its core is a Question-guided
Multimodal Selective
Attention (QMSA), which learns to preserve visual information related to the given question from both the current clip and the past related frames from the
memory. The compressor and
memory feedback work iteratively for each clip of the entire
video. This simple yet effective design yields large performance gains on longterm
video understanding tasks. Extensive experiments show that our method achieves significant improvement over current state-of-the-art methods by 6.1% on MLVU test, 8.3% on LVBench, 18.3% on VNBench Long, and 3.7% on VideoMME Long. The code will be released publicly.
Read the full abstract ▾
All authors Sosuke Yamao, Natsuki Miyahara, Yuankai Qi, Shun Takeuchi
Computer Vision
DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer
Zhengxu He, Jun Li and Zhijian Wu
Large-scale
Vision-
Language Models (VLMs) encode rich
multimodal semantics that are highly beneficial for fine-grained visual categorization (FGVC). However, their prohibitive computational cost hinders practical deployment in resource-constrained environments. Although knowledge
distillation contributes to transferring VLMs capacity to lightweight classifiers, conventional
distillation mechanisms, which directly transfer from a generic VLM to a compact student, often yield suboptimal results due to severe architectural misalignment and introducing task-irrelevant information. To alleviate this limitation, we propose
Distillation with Adaptive Intermediate Teacher transfer (DAIT) in this study, facilitating adaptive knowledge transfer from VLMs to lightweight students. DAIT introduces a trainable intermediate teacher that learns to transfer frozen VLMs representations under explicit supervision from the target fine-grained task. This intermediate teacher adaptively enhances discriminative visual cues, thereby producing compact and task-aligned knowledge that can be reliably distilled into lightweight models. Extensive evaluations on multiple FGVC
benchmarks with diverse student architectures demonstrate that our method achieves respective performance gains of 12.63% and 8.34% on FGVC-Aircraft and CUB-200-2011
datasets, establishing DAIT as a principled paradigm for transferring from general-purpose VLMS to deployable fine-grained recognition models.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
Vision-Language Model Based Multi-Expert Fusion for CT Image Classification
Jianfa Bai, Kejin Lu, Runtian Yuan et al. (+5 more)
Robust detection of COVID-19 from chest CT remains challenging in multi-institutional settings due to substantial source shift, source imbalance, and hidden test-source identities. In this work, we propose a three-stage source-aware multi-expert framework for multi-source COVID-19 CT classification. First, we build a lung-aware 3D expert by combining original CT volumes and lung-extracted CT volumes for volumetric classification. Second, we develop two MedSigLIP-based experts: a slice-wise representation and probability learning module, and a
Transformer-based inter-slice context modeling module for capturing cross-slice dependency. Third, we train a source classifier to predict the latent source identity of each test scan. By leveraging the predicted source information, we perform model fusion and voting based on different experts. On the validation set covering all four sources, the Stage 1 model achieves the best macro-F1 of 0.9711, ACC of 0.9712, and AUC of 0.9791. Stage~2a and Stage~2b achieve the best AUC scores of 0.9864 and 0.9854, respectively. Stage~3 source classifier reaches 0.9107 ACC and 0.9114 F1. These results demonstrate that source-aware expert modeling and hierarchical voting provide an effective solution for robust COVID-19 CT classification under heterogeneous multi-source conditions.
Read the full abstract ▾
All authors Jianfa Bai, Kejin Lu, Runtian Yuan, Qingqiu Li, Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng
Computer Vision
TextOVSR: Text-Guided Real-World Opera Video Super-Resolution
Hua Chang, Xin Xu, Wei Liu et al. (+4 more)
Many classic opera
videos exhibit poor visual quality due to the limitations of early filming equipment and long-term degradation during storage. Although real-world
video super-resolution (RWVSR) has achieved significant advances in recent years, directly applying existing methods to degraded opera
videos remains challenging. The difficulties are twofold. First, accurately modeling real-world degradations is complex: simplistic combinations of classical degradation kernels fail to capture the authentic noise distribution, while methods that extract real noise patches from external
datasets are prone to style mismatches that introduce visual artifacts. Second, current RWVSR methods, which rely solely on degraded image features, struggle to reconstruct realistic and detailed textures due to a lack of high-level semantic guidance. To address these issues, we propose a Text-guided Dual-Branch Opera
Video Super-Resolution (TextOVSR) network, which introduces two types of textual
prompts to guide the super-resolution process. Specifically, degradation-descriptive text, derived from the degradation process, is incorporated into the negative branch to constrain the solution space. Simultaneously, content-descriptive text is incorporated into a positive branch and our proposed Text-Enhanced Discriminator (TED) to provide semantic guidance for enhanced texture reconstruction. Furthermore, we design a Degradation-Robust Feature Fusion (DRF) module to facilitate cross-modal feature fusion while suppressing degradation interference. Experiments on our OperaLQ
benchmark show that TextOVSR outperforms state-of-the-art methods both qualitatively and quantitatively. The code is available at https://github.com/ChangHua0/TextOVSR.
Read the full abstract ▾
All authors Hua Chang, Xin Xu, Wei Liu, Jiayi Wu, Kui Jiang, Fei Ma, Qi Tian
Computer Vision
SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generation
Shufan Li, Jiuxiang Gu, Kangning Liu et al. (+3 more)
Recent advancements in discrete image generation showed that scaling the VQ codebook size significantly improves reconstruction fidelity. However, training generative models with a large VQ codebook remains challenging, typically requiring larger model size and a longer training schedule. In this work, we propose
Stochastic Neighbor Cross
Entropy Minimization (SNCE), a novel training objective designed to address the optimization challenges of large-codebook discrete image generators. Instead of supervising the model with a hard one-hot target, SNCE constructs a soft categorical distribution over a set of neighboring
tokens. The probability assigned to each
token is proportional to the proximity between its code
embedding and the ground-truth image
embedding, encouraging the model to capture semantically meaningful
geometric structure in the quantized
embedding space. We conduct extensive experiments across class-conditional ImageNet-256 generation, large-scale text-to-image synthesis, and image editing tasks. Results show that SNCE significantly improves convergence speed and overall generation quality compared to standard cross-
entropy objectives.
Read the full abstract ▾
All authors Shufan Li, Jiuxiang Gu, Kangning Liu, Zhe Lin, Aditya Grover, Jason Kuen
Computer Vision · Cross-listed from eess.IV
Clinical Priors Guided Lung Disease Detection in 3D CT Scans
Kejin Lu, Jianfa Bai, Qingqiu Li et al. (+5 more)
Accurate classification of lung diseases from chest CT scans plays an important role in computer-aided diagnosis systems. However, medical imaging
datasets often suffer from severe class imbalance, which may significantly degrade the performance of deep learning models, especially for minority disease categories. To address this issue, we propose a gender-aware two-stage lung disease classification framework. The proposed approach explicitly incorporates gender information into the disease recognition pipeline. In the first stage, a gender classifier is trained to predict the patient's gender from CT scans. In the second stage, the input CT image is routed to a corresponding gender-specific disease classifier to perform final disease prediction. This design enables the model to better capture gender-related imaging characteristics and alleviate the influence of imbalanced data distribution. Experimental results demonstrate that the proposed method improves the recognition performance for minority disease categories, particularly squamous
cell carcinoma, while maintaining competitive performance on other classes.
Read the full abstract ▾
All authors Kejin Lu, Jianfa Bai, Qingqiu Li, Runtian Yuan, Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng
Computer Vision
Context-Aware Sensor Modeling for Asynchronous Multi-Sensor Tracking in Stone Soup
Martin Vonheim Larsen and Kim Mathiassen
Multi-sensor tracking in the real world involves asynchronous sensors with partial coverage and heterogeneous detection performance. Although probabilistic tracking methods permit detection probability and clutter intensity to depend on state and sensing context, many practical frameworks enforce globally uniform observability assumptions. Under multi-rate and partially overlapping sensing, this simplification causes repeated non-detections from high-rate sensors to erode tracks visible only to low-rate sensors, potentially degrading fusion performance. We introduce DetectorContext, an abstraction for the open-source multi-target tracking framework Stone Soup. DetectorContext exposes detection probability and clutter intensity as state-dependent functions evaluated during hypothesis formation. The abstraction integrates with existing probabilistic trackers without modifying their update equations. Experiments on asynchronous radar-
lidar data demonstrate that context-aware modeling restores stable fusion and significantly improves HOTA and GOSPA performance without increasing false tracks.
Read the full abstract ▾
Computer Vision
WiT: Waypoint Diffusion Transformers for Alleviating Trajectory Conflict in Pixel-Space Image Generation
Hainuo Wang, Mingjia Li and Xiaojie Guo
While recent Flow Matching models avoid the reconstruction bottlenecks of latent autoencoders by operating directly in pixel space, the raw pixel
manifold provides little explicit semantic organization, making target-specific transport directions difficult for a shared finite-capacity vector field to distinguish. When different transport requirements are locally entangled in this way, their learning signals can interfere and hinder optimization, a phenomenon we refer to as trajectory conflict. To alleviate trajectory conflict while preserving direct generation in pixel space, we propose Waypoint
Diffusion Transformers (WiT), which introduces explicit semantic routing into pixel-space generation. WiT structures pixel-space prediction through compact semantic waypoints projected from pre-trained
vision representations, providing a semantically organized intermediate representation that complements direct pixel prediction. During ODE integration, a lightweight Waypoints Predictor dynamically infers the semantic waypoint from the current noisy state, and the predicted waypoint continuously conditions the primary Pixel Space Generator through our Just-Pixel AdaLN mechanism. This spatially varying semantic modulation reorganizes the finite-capacity learning problem and helps the model preserve target-specific transport directions while retaining generation entirely in pixel space. Experiments on ImageNet 2012 demonstrate consistent improvements over strong pixel-space baselines across model scales, with WiT-H/16 achieving an FID of 1.79. Code will be released at https://hainuo-wang.github.io/WiT.
Read the full abstract ▾
Computer Vision
Low-light Image Enhancement with Retinex Decomposition in Latent Space
Bolun Zheng, Qingshan Lei, Quan Chen et al. (+4 more)
Retinex theory provides a principled foundation for low-light image enhancement, inspiring numerous learning-based methods that integrate its principles. However, existing methods exhibits limitations in accurately decomposing reflectance and illumination components. To address this, we propose a Retinex-Guided
Transformer~(RGT) model, which is a two-stage model consisting of decomposition and enhancement phases. First, we propose a latent space decomposition strategy to separate reflectance and illumination components. By incorporating the log transformation and 1-pixel offset, we convert the intrinsically multiplicative relationship into an additive formulation, enhancing decomposition stability and precision. Subsequently, we construct a U-shaped component refiner incorporating the proposed guidance fusion
transformer block. The component refiner refines reflectance component to preserve texture details and optimize illumination distribution, effectively transforming low-light inputs to normal-light counterparts. Experimental evaluations across four
benchmark datasets validate that our method achieves competitive performance in low-light enhancement and a more stable training process.
Read the full abstract ▾
All authors Bolun Zheng, Qingshan Lei, Quan Chen, Qianyu Zhang, Kainan Yu, Xu Jia, Lingyu Zhu
Computer Vision
Next-Frame Decoding for Ultra-Low-Bitrate Image Compression with Video Diffusion Priors
Yunuo Chen, Chuqin Zhou, Jiangchuan Li et al. (+5 more)
We present a novel paradigm for ultra-low-bitrate image compression (ULB-IC) that exploits the “temporal”
evolution in generative image compression. Specifically, we define an explicit intermediate state during decoding: a compact anchor frame, which preserves the scene geometry and semantic layout while discarding high-frequency details. We then reinterpret generative decoding as a virtual temporal transition from this anchor to the final reconstructed image. To model this progression, we leverage a pretrained
video diffusion model (VDM) as a temporal prior: the anchor frame serves as the initial frame and the original image as the target frame, transforming the decoding process into a next-frame prediction task. In contrast to image
diffusion-based ULB-IC models, our decoding proceeds from a visible, semantically faithful anchor, which improves both fidelity and realism for perceptual image compression. Extensive experiments demonstrate that our method achieves superior rate-distortion performance. On the CLIC2020 test set, our method achieves over 50% bitrate savings across LPIPS, DISTS, FID, and KID compared to DiffC, while also delivering a significant decoding speedup of up to $\times$5. Code will be released at https://github.com/UnoC-727/NeFIC.
Read the full abstract ▾
All authors Yunuo Chen, Chuqin Zhou, Jiangchuan Li, Xiaoyue Ling, Bing He, Jincheng Dai, Li Song, Guo Lu
Computer Vision · Cross-listed from cs.RO
A Novel Camera-to-Robot Calibration Method for Vision-Based Floor Measurements
Jan Andre Rudolph, Dennis Haitz and Markus Ulrich
A novel hand-eye calibration method for ground-observing mobile
robots is proposed. While cameras on mobile
robots are common, they are rarely used for ground-observing measurement tasks.
Laser trackers are increasingly used in robotics for precise localization. A referencing plate is designed to combine the two measurement modalities of
laser-tracker 3D metrology and camera-based 2D imaging. It incorporates reflector nests for pose acquisition using a
laser tracker and a camera calibration target that is observed by the
robot-mounted camera. The procedure comprises estimating the plate pose, the plate-camera pose, and the
robot pose, followed by computing the
robot-camera transformation. Experiments indicate sub-millimeter repeatability.
Read the full abstract ▾
Computer Vision
A Tutorial on ALOS2 SAR Utilization: Dataset Preparation, Self-Supervised Pretraining, and Semantic Segmentation
Nevrez Imamoglu, Ali Caglayan and Toru Kouyama
Masked auto-encoders (MAE) and related approaches have shown promise for satellite imagery, but their application to synthetic aperture radar (SAR) remains limited due to challenges in semantic labeling and high noise levels. Building on our prior work with SAR-W-MixMAE, which adds SAR-specific intensity-weighted loss to standard MixMAE for pretraining, we also introduce SAR-W-SimMIM; a weighted variant of SimMIM applied to ALOS-2 single-channel SAR imagery. This method aims to reduce the impact of speckle and extreme intensity values during self-supervised pretraining. We evaluate its effect on semantic
segmentation compared to our previous trial with SAR-W-MixMAE and random initialization, observing notable improvements. In addition, pretraining and fine-tuning models on satellite imagery pose unique challenges, particularly when developing region-specific models. Imbalanced land cover distributions such as dominant water, forest, or desert areas can introduce bias, affecting both pretraining and downstream tasks like land cover
segmentation. To address this, we constructed a SAR
dataset using ALOS-2 single-channel (HH polarization) imagery focused on the Japan region, marking the initial phase toward a national-scale foundation model. This
dataset was used to pretrain a
vision transformer-based autoencoder, with the resulting encoder fine-tuned for semantic
segmentation using a task-specific decoder. Initial results demonstrate significant performance improvements compared to training from scratch with random initialization. In summary, this work provides a guide to process and prepare ALOS2 observations to create
dataset so that it can be taken advantage of self-supervised pretraining of models and finetuning downstream tasks such as semantic
segmentation.
Read the full abstract ▾
Computer Vision
VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents
Udi Barzelay, Ophir Azulai, Inbar Shapira et al. (+4 more)
We introduce VAREX (VARied-schema EXtraction), a
benchmark for evaluating
multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The
benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing
benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy – a capability absent from prior
benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular
attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance – not extraction capability – is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the
benchmark most effectively discriminates models in the 60-95% accuracy band.
Dataset and evaluation code are publicly available.
Read the full abstract ▾
All authors Udi Barzelay, Ophir Azulai, Inbar Shapira, Idan Friedman, Foad Abo Dahood, Madison Lee, Abraham Daniels
Computer Vision · Cross-listed from cs.LG
Sampling-guided exploration of active feature selection policies
Gabriel Bernardino, Anders Jonsson, Patrick Clarysse et al. (+1 more)
Determining the most appropriate features for machine learning predictive models is challenging regarding performance and feature acquisition costs. In particular, global feature choice is limited given that some features will only benefit a subset of instances. In previous work, we proposed a reinforcement learning approach to sequentially recommend which modality to acquire next to reach the best information/cost ratio, based on the instance-specific information already acquired. We formulated the problem as a Markov Decision Process where the state's dimensionality changes during the episode, avoiding data imputation, contrary to existing works. However, this only allowed processing a small number of features, as all possible combinations of features were considered. Here, we address these limitations with two contributions: 1) we expand our framework to larger
datasets with a heuristic-based strategy that focuses on the most promising feature combinations, and 2) we introduce a post-fit regularisation strategy that reduces the number of different feature combinations, leading to compact sequences of decisions. We tested our method on four binary classification
datasets (one involving high-dimensional variables), the largest of which had 56 features and 4500 samples. We obtained better performance than state-of-the-art methods, both in terms of accuracy and policy complexity.
Read the full abstract ▾
All authors Gabriel Bernardino, Anders Jonsson, Patrick Clarysse, Nicolas Duchateau
Computer Vision
PAKAN: Pixel Adaptive Kolmogorov-Arnold Network Modules for Pansharpening
Haoyu Zhang, Haojing Chen, Zhen Zhong et al. (+1 more)
Pansharpening aims to fuse high-resolution spatial details from panchromatic images with the rich spectral information of multispectral images. Existing deep neural networks for this task typically rely on static activation functions, which limit their ability to dynamically model the complex, non-linear mappings required for optimal spatial-spectral fusion. While the recently introduced Kolmogorov-Arnold Network (KAN) utilizes learnable activation functions, traditional KANs lack dynamic adaptability during inference. To address this limitation, we propose a Pixel Adaptive Kolmogorov-Arnold Network framework. Starting from KAN, we design two adaptive variants: a 2D Adaptive KAN that generates spline summation weights across spatial dimensions and a 1D Adaptive KAN that generates them across spectral channels. These two components are then assembled into PAKAN 2to1 for feature fusion and PAKAN 1to1 for feature refinement. Extensive experiments demonstrate that our proposed modules significantly enhance network performance, proving the effectiveness and superiority of pixel-adaptive activation in pansharpening tasks.
Read the full abstract ▾
All authors Haoyu Zhang, Haojing Chen, Zhen Zhong, Liangjian Deng
Computer Vision · Cross-listed from cs.AI
PrototypeNAS: Rapid Design of Deep Neural Networks for Microcontroller Units
Mark Deutel, Simon Geis and Axel Plinge
Enabling efficient deep neural network (DNN) inference on edge devices with different hardware constraints is a challenging task that typically requires DNN architectures to be specialized for each device separately. To avoid the huge manual effort, one can use neural architecture search (NAS). However, many existing NAS methods are resource-intensive and time-consuming because they require the training of many different DNNs from scratch. Furthermore, they do not take the resource constraints of the target system into account. To address these shortcomings, we propose PrototypeNAS, a zero-shot NAS method to accelerate and automate the selection, compression, and specialization of DNNs to different target microcontroller units (MCUs). We propose a novel three-step search method that decouples DNN design and specialization from DNN training for a given target platform. First, we present a novel search space that not only cuts out smaller DNNs from a single large architecture, but instead combines the structural optimization of multiple architecture types, as well as optimization of their
pruning and quantization configurations. Second, we explore the use of an ensemble of zero-shot proxies during optimization instead of a single one. Third, we propose the use of Hypervolume subset selection to distill DNN architectures from the Pareto front of the multi-objective optimization that represent the most meaningful tradeoffs between accuracy and FLOPs. We evaluate the effectiveness of PrototypeNAS on 12 different
datasets in three different tasks: image classification, time series classification, and object detection. Our results demonstrate that PrototypeNAS is able to identify DNN models within minutes that are small enough to be deployed on off-the-shelf MCUs and still achieve accuracies comparable to the performance of large DNN models.
Read the full abstract ▾