Computer Vision · Cross-listed from cs.AI
Concept-based Visual Counterfactual Explanations with Diffusion Models
Yassine Oueslati, Daniil Kirilenko, Martin Gjoreski et al. (+1 more)
Visual counterfactual explanations aim to answer "what minimal change to this image would flip the model's prediction?", and are increasingly important as
vision models are deployed in safety-critical domains (e.g., medicine). Existing
diffusion-based methods can produce realistic edits, but they rely on external classifiers that must work reliably on noisy images, which makes them fragile and hard to deploy for robust explanations. We introduce C-VCE, a new
diffusion framework that builds the classifier directly into the generative model via a concept bottleneck layer, so that counterfactuals are guided by human-interpretable features (concepts) instead of a separate noise robust classifier that works with pixel-level edits. Our model lets users to toggle on/off semantic concepts during sampling, then minimally adjusts relevant image regions, while preserving the rest of the image, respecting feature correlations. To keep edits small and controlled, we add a simple probabilistic regularizer that balances "change the prediction" against "stay close to the original", plus a gradient-based mask that confines modifications to the most relevant regions. On
benchmarks such as CelebA, C-VCE matches or improves flip rates while producing counterfactuals that are visually closer to the input and less distorted than baselines that depend on separate noisy-image classifiers. These properties make C-VCE a practical tool for
vision systems where users need concrete "what-if" images without having to trust an additional, noise-robust classifier. More broadly, our results suggest that exposing and controlling an internal concept layer is a promising way to make powerful generative models easier to understand and safer to use.
Read the full abstract ▾
All authors Yassine Oueslati, Daniil Kirilenko, Martin Gjoreski, Marc Langheinrich
Computer Vision · Cross-listed from cs.GR
Semantic-Structural Alignment for Generative Pictorial Charts
Zhida Sun, Yulin Zhang, Zheng Gu et al. (+4 more)
Traditional statistical graphics are precise but often lack the visual appeal, memorability, and engagement of pictorial charts. We present a generative framework for the automated synthesis of pictorial charts that bridges the gap between semantic expression and structural faithfulness. Rather than treating charts merely as images to be stylized, we frame the problem as a dual-conditioned generation task guided by two parallel external control signals: a text
prompt capturing the semantic context of the editing intent, and a context image providing the abstract statistical chart's global structure. To reinforce these controls within a Multi-Modal
Diffusion Transformer, we introduce two complementary feature-level mechanisms: structural
alignment to anchor spatial layouts to the input chart, and semantic
alignment to transfer expressive textures from reference images. Generalizing across major visual channels (i.e., length, area, angle, and position) and diverse semantic domains, our method produces pictorial charts that are both artistically compelling and structurally consistent. Extensive quantitative evaluations and perceptual user studies demonstrate that our framework outperforms traditional controllable generation and image editing baselines, providing a foundation for high-fidelity, data-driven generative modeling in expressive visual storytelling. Project page: https://ssalign.github.io/.
Read the full abstract ▾
All authors Zhida Sun, Yulin Zhang, Zheng Gu, Min Lu, Bongshin Lee, Daniel Cohen-Or, Hui Huang
Computer Vision · Cross-listed from cs.LG
Weakly Supervised Concept Learning for Object-centric Visual Reasoning
Sparsh Tiwari, Bettina Finzel and Gesina Schwalbe
Neurosymbolic systems promise to combine deep neural network's (DNN) processing of raw sensor inputs with few-shot performance of symbolic artificial intelligence. Two-stage approaches explicitly decouple DNN based perception from subsequent rule based
reasoning. This avoids optimization and interpretability issues of end to end differentiable approaches, but requires costly labels for the perception output. This paper introduces an efficient weak supervision scheme for the perception stage to ground its output symbols for logical induction in object-centric
reasoning tasks. It combines a slot-based architecture for object-centricity with a Variational Autoencoder (VAE) for self-supervision, competing with concept guidance on latent dimensions for human interpretable grounding. The resulting predictions are translated into symbolic background knowledge for
reasoning frameworks, such as Inductive Logic Programming (ILP), Decision Trees, and
Bayesian Networks. Our extensive empirical evaluation on synthetic and real world
datasets shows that our approach can discover complex, abstract rules for object centric
reasoning whilst reducing supervision to as little as 1% of labels, and being robust even under substantial domain shift. Notably, at 1% supervision it even outperforms state of the art foundation model baselines in domain generalization
Read the full abstract ▾
Computer Vision · Cross-listed from cs.AI
Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits
Logan Mann, Ajit Saravanan, Ishan Dave et al. (+4 more)
A pervasive intuition holds that
vision-
language models (VLMs) are most trustworthy when their
attention maps look sharp: concentrated
attention on the queried region should imply a confident, calibrated answer. We test this
Attention-Confidence Assumption directly. We instrument three open-weight VLM families (LLaVA-1.5, PaliGemma, Qwen2-VL; 3-7B parameters) with a unified mechanistic pipeline – the VLM Reliability Probe (VRP) – that compares
attention structure, generation dynamics, and hidden-state geometry against a single correctness label. Three results emerge. (i)
Attention structure is a near-zero predictor of correctness (R_pb(C_k,y)=0.001, 95% CI [-0.034,0.036]; R_pb(H_s,y)=-0.012, [-0.047,0.024] on a pooled n=3,090 split), even though
attention remains causally necessary for feature extraction (top-30% patch masking drops accuracy by 8.2-11.3 pp, p<0.001). (ii) Reliability becomes legible later in the computation: a single hidden-state linear probe reaches AUROC>0.95 on POPE for two of three families, and self-consistency at K=10 is the strongest behavioral predictor we measure at 10x inference cost (R_pb=0.43). (iii) Causal
neuron-level ablations expose a sharp architectural split with direct monitor-design implications: late-fusion LLaVA concentrates reliability in a fragile late bottleneck (-8.3 pp object-identification accuracy after top-5 probe-
neuron ablation), whereas early-fusion PaliGemma and Qwen2-VL distribute it widely and absorb destruction of ~50% of their peak-layer hidden dimension with <=1 pp degradation. The takeaway is narrow but consequential: in 3-7B VLMs, reliability is read more reliably off hidden-state geometry, layer-wise margin formation, and sparse late-layer
circuits than off
attention-map sharpness.
Read the full abstract ▾
All authors Logan Mann, Ajit Saravanan, Ishan Dave, Shikhar Shiromani, Saadullah Ismail, Yi Xia, Emily Huang
Computer Vision
Survey on Disaster Management Datasets for Remote Sensing Based Emergency Applications
Alain P. Ndigande, Josiah Wiggins and Sedat Ozer
Recent natural disasters have highlighted the urgent need for efficient data-driven approaches to disaster management. Machine learning (ML) and deep learning (DL) techniques have shown considerable promise in enhancing the key phases of disaster management including mitigation, preparedness, detection, response, and recovery. A critical enabler of successful ML or DL based applications in remote sensing, however, is the accessibility and quality of annotated
datasets. With the growing availability of high-resolution imagery from unmanned aerial vehicles (
UAVs) and satellites, computer
vision and remote sensing
algorithms have become essential tools for rapid detection, situational assessment, and decision-making in disaster scenarios. This survey provides a comprehensive overview of publicly available image-based
datasets relevant to ML/DL-based disaster management pipelines. Emphasis is placed on
datasets that support computer
vision and remote sensing tasks across all phases of disaster events including pre-disaster, during, and post-disaster. The goal of this work is to serve as a centralized reference for researchers and practitioners seeking high-quality
datasets for rapid development and deployment of remote sensing-driven disaster response solutions.
Read the full abstract ▾
Computer Vision
Normalization Equivariance for Arbitrary Backbones, with Application to Image Denoising
Youssef Saied and François Fleuret
Normalization Equivariance (NE) is a structural prior that improves robustness to distribution shift in image-to-image tasks. A function $f$ is normalization equivariant iff $f(a y + b\mathbf{1}) = a f(y) + b\mathbf{1}$ for all $a>0$ and $b\in\mathbb{R}$. Existing NE methods constrain every internal layer to NE-compatible operations. These constraints add runtime cost and exclude standard
transformer components such as softmax
attention and LayerNorm. We introduce Wrapped Normalization Equivariance (WNE), a parameter-free wrapper that normalizes the input, applies any backbone, and denormalizes the output. We prove every NE function admits this factorization, so the wrapper exactly parameterizes the class of NE functions. On blind denoising, wrapping CNN and
transformer architectures improves robustness under noise-level mismatch with no measurable GPU overhead, while architectural NE baselines are up to $1.6\times$ slower.
Read the full abstract ▾
Computer Vision
A Robust Out-of-Distribution Detection Framework via Synergistic Smoothing
Maria Stoica, Abdelrahman Hekal and Alessio Lomuscio
Reliable out-of-distribution (OOD) detection is a critical requirement for the safe deployment of machine learning systems. Despite recent progress, state-of-the-art OOD detectors are highly susceptible to
adversarial attacks, which undermines their trustworthiness in automated systems. To address this vulnerability, we apply median smoothing to baseline OOD detection scores, balancing clean and
adversarial accuracies. Our key insight is that the noisy samples generated for median smoothing can be repurposed to quantify the local instability of the base score. We observe that OOD samples exhibit higher instability under perturbation. Based on this, we propose ROSS, a novel and robust post-hoc OOD detector that leverages the instability of baseline scores to further distinguish between in-distribution (ID) and OOD samples. ROSS achieves symmetric robustness, performing strongly against both score-minimising and score-maximising attacks, unlike prior work. This symmetric defence leads to state-of-the-art robustness, outperforming prior methods by up to 40 AUROC points. We demonstrate ROSS's effectiveness on extensive experiments across CIFAR-10, CIFAR-100, and ImageNet. Code is available at: https://github.com/Abdu-Hekal/ROSS.
Read the full abstract ▾
Computer Vision
Neuroscience-Inspired Analyses of Visual Interestingness in Multimodal Transformers
Mathis Immertreu, Fitim Abdullahu, Thomas Kinfe et al. (+3 more)
Human
attention is the gateway to conscious perception,
memory and decision-making. However, its role in modern
transformer models remains largely unexplored. As these systems increasingly influence what people see, prefer and buy, the question arises as to whether they encode principles of human interest or merely exploit large-scale correlations. Addressing this issue is crucial for understanding cognition and ensuring the responsible use of AI in communication and marketing. In order to address this issue, the concept of visual interest was examined within the
multimodal vision-
language-model Qwen3-VL-8B, using a pre-defined Common Interestingness (CI) score derived from large-scale human engagement data on the photo-sharing platform Flickr. Here, we analyzed internal representations across
vision and
language components using methods from the neurosciences. Our analyses revealed that CI information is linearly decodable from final-layer
embeddings, indicating that it is aligned with human-derived measures of visual interestingness. Dimensionality reduction and Generalized Discrimination Value (GDV) analyses demonstrate that CI-related hidden representations emerge in intermediate
vision transformer layers and becomes progressively more distinguishable across
language model layers. Concept vectors derived using
geometric, probe, and Sparse Auto-Encoder based methods converge in higher layers, as confirmed by representational similarity analysis. This indicates a robust and structured encoding of visual interestingness without explicit supervision. Future work will seek to identify shared computational principles linking human
brain dynamics and
transformer architectures, with the ultimate goal of uncovering the organizing mechanisms that give rise to
attention and interest in both biological and artificial systems.
Read the full abstract ▾
All authors Mathis Immertreu, Fitim Abdullahu, Thomas Kinfe, Helmut Grabner, Patrick Krauss, Achim Schilling
Computer Vision
Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery
Bo Ye, Kai Gan, Tong Wei et al. (+1 more)
Generalized Category Discovery (GCD) seeks to identify novel categories from unlabeled data while retaining the classification ability of seen categories. Prior GCD methods commonly leverage transferable representations from pre-trained models, adapting to downstream
datasets via partial fine-tuning (updating only the final ViT block) and visual
prompt tuning (appending learnable vectors to inputs). However, conventional partial fine-tuning offers limited flexibility, as it fails to adapt the entire model; meanwhile, visual
prompt tuning is prone to overfitting, due to its sensitivity to initialization and inherently constrained capacity. To address these limitations, we propose LAGCD, a simple yet effective GCD approach that embeds a residual linear adapter into each ViT block. From the perspective of feature sparsity, we systematically show that non-linearity in conventional adapters impairs performance, whereas our linear adapter enhances it by enabling more flexible model capacity. We further introduce an auxiliary distribution
alignment loss to mitigate the negative impact of biased predictions between seen and novel categories. Extensive experiments on both generic and fine-grained
datasets confirm that LAGCD consistently improves performance over many sophisticated baselines. The source code is available at https://github.com/yebo0216best/LAGCD
Read the full abstract ▾
All authors Bo Ye, Kai Gan, Tong Wei, Min-Ling Zhang
Computer Vision
Text-Guided Multi-Scale Frequency Representation Adaptation
Weicai Yan, Xinhua Ma, Wang Lin et al. (+1 more)
Parameter-efficient fine-tuning methods introduce a small number of training parameters, enabling pre-trained models to adapt rapidly to new data distributions. While these methods have shown promising results, they exhibit notable limitations. First, most existing methods operate in the signal space domain, which results in substantial information redundancy. Second, most existing methods utilize fixed
prompts or adaptation layers, failing to fully account for the multi-scale characteristics of signals. To address these challenges, we propose the Multi-Scale Frequency Adapter (FreqAdapter), which integrates textual information and performs multi-scale fine-tuning of signals in the frequency domain. Additionally, we introduce a multi-scale adaptation strategy to optimize receptive fields across different frequency ranges, further enhancing the model's representational capacity. Extensive experiments on
multimodal models, including CLIP and LLaVA, demonstrate that FreqAdapter significantly improves both performance and efficiency. FreqAdapter improves performance with minimal cost and fast convergence within one epoch. Code is available at https://github.com/Kelvin-ywc/FreqAdapter.
Read the full abstract ▾
All authors Weicai Yan, Xinhua Ma, Wang Lin, Tao Jin
Computer Vision
KARMA-MV: A Benchmark for Causal Question Answering on Music Videos
Archishman Ghosh, Abhinaba Roy and Dorien Herremans
While significant progress has been made in
Video Question Answering and cross-modal understanding, causal
reasoning about how visual dynamics drive musical structure in music
videos remains under-explored. We introduce KARMA-MV, a large-scale multiple-choice QA
dataset derived from 2,682 YouTube music
videos, designed to test models' ability to integrate temporal audio-visual cues and reason about visual-to-musical influence across
reasoning, prediction, and counterfactual questions. Unlike traditional
datasets requiring manual annotation, KARMA-MV leverages LLM
reasoning for scalable generation and validation, yielding 37,737 MCQs. We propose a causal knowledge graph (CKG) approach that augments
vision-
language models (VLMs) with structured
retrieval of cross-modal dependencies. Experiments on state-of-the-art VLMs and LLMs show consistent gains from CKG grounding – especially for smaller models – establishing the value of explicit causal structure for music-
video reasoning. KARMA-MV provides a new
benchmark for advancing causal audio-visual understanding beyond correlation.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
CERSA: Cumulative Energy-Retaining Subspace Adaptation for Memory-Efficient Fine-Tuning
Jingze Ge, Xue Geng, Yun Liu et al. (+6 more)
To mitigate the
memory constraints associated with fine-tuning large pre-trained models, existing parameter-efficient fine-tuning (PEFT) methods, such as LoRA, rely on low-rank updates. However, such updates fail to fully capture the rank characteristics of the weight modifications observed in full-parameter fine-tuning, resulting in a performance gap. Furthermore, LoRA and other existing PEFT methods still require substantial
memory to store the full set of frozen weights, limiting their efficiency in resource-constrained settings. To addres these limitations, we introduce Cumulative Energy-Retaining Subspace Adaptation (CERSA), a novel fine-tuning paradigm that leverages singular value decomposition (SVD) to retain only the principal components responsible for 90% to 95% of the spectral energy. By fine-tuning low-rank representations derived from this principal subspace, CERSA significantly reduces
memory consumption. We conduct extensive evaluations of CERSA across models of varying scales and domains, including image recognition, text-to-image generation, and natural
language understanding. Empirical results demonstrate that CERSA consistently outperforms state-of-the-art PEFT methods while achieving substantially lower
memory requirements. The code will be publicly released.
Read the full abstract ▾
All authors Jingze Ge, Xue Geng, Yun Liu, Wanqi Dong, Wang Zhe Mark, Min Wu, Ngai-Man Cheung, Bharadwaj Veeravalli, Xulei Yang
Computer Vision
CASISR: Circular Arbitrary-Scale Image Super-Resolution
Honggui Li, Zhengyang Zhang, Dingtai Li et al. (+11 more)
The generalization performance (GP) of deep learning-based arbitrary-scale image super-resolution (ASISR) methods is subject to limited training
datasets and unlimited testing
datasets. It is vitally significant to enhance the GP of the pretrained ASISR models by making full use of the testing samples. The ASISR models usually employ an open-loop architecture from low-resolution (LR) images to super-resolution (SR) images. The degradation model from SR samples to LR samples is known bicubic down-sampling for the classical ASISR, is supposed down-sampling with additive random noise for the blind ASISR, and is learnable for the real-world ASISR. Combining the ASISR and degradation models, it is potentially possible to adopt a closed-loop architecture based on the automatic control theory for strengthening the GP of the ASISR methods. Therefore, this paper proposes a closed-loop architecture, circular ASISR (CASISR), to lift the capability of image reconstruction. A mathematical nonlinear loop equation is established to describe the CASISR, the reasonability of the CASISR is proven by conditional probability theory, and the stability of the CASISR is proven by Taylor series approximation. The first-order and second-order absolute difference images are defined to compare the image reconstruction performance of the ASISR and the CASISR methods. Comprehensive simulation experiments show that the proposed CASISR approach outperforms the eight state-of-the-art ASISR approaches in the quality of image reconstruction. Especially, the proposed CASISR is extraordinarily suitable for fractional SR scale factors and is extremely effective for text and stripe images with drastically changed edges.
Read the full abstract ▾
All authors Honggui Li, Zhengyang Zhang, Dingtai Li, Sinan Chen, Nahid Md Lokman Hossain, Xinfeng Xu, Yinlu Qin, Ruobing Wang, Hantao Lu, Yuting Feng, Maria Trocan, Dimitri Galayko, Amara Amara, Mohamad Sawan
Computer Vision · Cross-listed from eess.IV
Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
Daniel Mensing, Jan Kapar, Jochen G. Hirsch et al. (+3 more)
We propose a
multimodal latent
diffusion model that jointly synthesizes volumetric
magnetic resonance imaging (MRI) and tabular clinical data within a shared latent space via cross-
attention. This approach enables coherent joint representation learning of MRI and tabular modalities for generative modeling. Our model utilizes a variational autoencoder to fuse the two modalities before
diffusion-based synthesis, allowing modality-appropriate reconstruction with separate decoders for MRI and tabular data. We evaluated the framework on data from the German National Cohort (NAKO Gesundheitsstudie), comprising over 10,000 participants with MRI scans and clinical tabular features such as age, sex, body measurements, and ethnicity. The generated MRI volumes exhibited anatomical plausibility and body composition consistent with the synthesized tabular attributes. Quantitative evaluation using Fréchet distance and precision-recall metrics confirmed high-fidelity image generation. In the tabular modality, our model outperformed CTGAN across standard evaluation metrics and achieved results comparable to TVAE, demonstrating competitive performance relative to established unimodal baselines. This work is, to our knowledge, the first to demonstrate the feasibility of jointly modeling MRI and mixed-type tabular data in a single latent
diffusion framework, offering a proof-of-concept for generating coherent synthetic
multimodal patient data and aligning with the broader goal of developing digital twins in healthcare.
Read the full abstract ▾
All authors Daniel Mensing, Jan Kapar, Jochen G. Hirsch, Matthias Günther, Horst Hahn, Marvin N. Wright
Computer Vision
Intermediate Representations are Strong AI-Generated Image Detectors
Zhenhan Huang, Pin-Yu Chen, Tejaswini Pedapati et al. (+1 more)
The rapid advancement in generative AI models has enabled the creation of photorealistic images. At the same time, there are growing concerns about the potential misuse and dangers of generated content, as well as a pressing need for effective AI-generated image detectors. However, current training-based detection techniques are typically computationally costly and can hardly be generalized to unseen data domains, while training-free methods fall short in detection performance. To bridge this gap, we propose a search-based method employing data
embedding sensitivity in intermediate layers to detect AI-generated images. Given a set of real and AI-generated images, our method examines the similarity between original image
embeddings and perturbed image
embeddings, and detects AI-generated images based on the similarity. We examine the proposed method on two comprehensive
benchmarks: GenImage and Forensics Small. Our method exhibits improved performance across different
datasets compared to both training-free and training-based state-of-the-art methods. On average, our method achieves the largest performance gain on the Forensics Small
benchmark by 39.61% compared to the best training-free method and 5.14% compared to the best training-based method in AUROC score.
Read the full abstract ▾
All authors Zhenhan Huang, Pin-Yu Chen, Tejaswini Pedapati, Jianxi Gao
Computer Vision
InterFuserDVS: Event-Enhanced Sensor Fusion for Safe RL-Based Decision Making
Mustafa Sakhaia, Kaung Sithua, Min Khant Soe Okea et al. (+1 more)
Autonomous driving systems rely heavily on robust sensor fusion to perceive complex envi- ronments. Traditional setups using RGB cameras and
LiDAR often struggle in high-dynamic- range scenes or high-speed scenarios due to motion blur and latency. Dynamic
Vision Sensors (DVS), or event cameras, offer a paradigm shift by capturing asynchronous brightness changes with microsecond temporal resolution and high dynamic range. In this paper, we propose an extended architecture of the state-of-the-art InterFuser model, integrating DVS as an additional modality to enhance perception reliability. We introduce a novel
token-based fusion strategy that incorporates accumulated event frames into the
transformer-based backbone of InterFuser. Our method leverages the complementary nature of RGB,
LiDAR, and DVS data. We evaluate our approach on the Car Learning to Act (CARLA) Leaderboard
benchmarks, demonstrating that the inclusion of DVS improves the robustness of the driving
agent, achieving a competitive Driving Score of 77.2 and a superior Route Completion of 100%. The results indicate that event-based
vision is a promising direction for improving safety and performance in adverse lighting and dynamic conditions.
Read the full abstract ▾
All authors Mustafa Sakhaia, Kaung Sithua, Min Khant Soe Okea, Maciej Wielgosza
Computer Vision · Cross-listed from cs.LG
Covariance-Aware Goodness for Scalable Forward-Forward Learning
Xiaoyi Jiang, Bashir M. Al-Hashimi and Kai Xu
The Forward-Forward
algorithm eliminates global gradient flow and full network activations storage. However, in convolutional settings, existing BP-free FF methods significantly under-perform backpropagation on complex
benchmarks such as ImageNet-100 and Tiny-ImageNet. We identify this gap as a structural bottleneck in goodness extraction: standard sum-of-squares formulation collapses feature volumes into channel-wise activation energies which omits critical second-order dependencies. To address this, we propose a framework centered on three key components. First, Bi-axis Covariance Goodness(BiCovG) explicitly augments the standard goodness function with structured second-order information along two axes: cross-channel projections that model inter-feature covariance, and nested multi-scale aggregation that encodes spatial correlation statistics. This provides a tractable approximation to covariance-aware goodness without the prohibitive O(C^2) complexity of explicit
matrix estimation. Second, a lightweight Logistic Fusion module aggregates layer-wise predictions, amplifying the contribution of deeper representations. Third, the Feature
Alignment Layer(FAL) introduces a zero-initialized correction at block boundaries to mitigate representation misalignment in deep locally trained networks. By introducing these three components, we effectively double the depth of viable Forward-Forward learning, extending robust layer utilization from shallow baselines to 16 layer architectures like VGG-16. The resulting BP-free model achieves 73.01% on ImageNet-100 and 50.30% on Tiny-ImageNet. As a practical extension, Hybrid Goodness Blocks control the scope of gradient propagation via configurable block sizes, further narrowing the ImageNet-100 gap to 3.6% and matching BP on Tiny-ImageNet, while still reducing peak
memory by approximately 50% relative to BP.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Learning-based Statistical Refinement for Denoising
Rihuan Ke
This work proposes a learning-based statistical refinement method for improving the denoising results of a given denoiser without knowing the precise noise distribution or accessing clean images or calibration data. While there are many existing successful denoising approaches for handling different kinds of noise, they typically require accurate modelling of the images and the noise (implicitly or explicitly), and hence the denoising results can be suboptimal due to different practical factors such as imperfect models, unreliable noise assumptions, or low quality data. In particular, when clean image samples are not available and there is a lack of knowledge of the underlying noise distribution, which is the case in various practical situations, the results may not well align with the noise statistics. The unawareness of the useful statistical information leads to suboptimal results. This work aims to make the best use of the statistical information to improve the consistency between the given denoising results and the noise statistics, under the assumption that the noise is conditionally pixel-wise independent given the clean signal. A method, based on a
Bayesian formulation of an auxiliary signal in the noisy data, is proposed for evaluating the consistency of the denoising results, without precise information on noise distribution. By leveraging the statistical information from noisy data, the method enhances the statistical noise consistency and improves denoising quality.
Read the full abstract ▾
Computer Vision
Hierarchical Visual Agent: Managing Contexts in Joint Image-Text Space for Advanced Chart Reasoning
Qihua Dong, Ruozhen He, Junwen Chen et al. (+4 more)
Advanced chart question answering requires both precise perception of small visual elements and multi-step
reasoning across several subplots. While existing MLLMs are strong at understanding single plots, they often struggle with multi-step
reasoning across multiple subplots. We propose HierVA, a hierarchical visual
agent framework for chart
reasoning that iteratively constructs and updates a working context in a joint image–text space. A high-level manager generates plans and maintains a compact context containing only key information, while specialized workers perform
reasoning, gather evidence, and return results. In particular, the
agent maintains separate visual and textual contexts, using a zoom-in tool to restrict the visual context. Experiments on the CharXiv
reasoning subset demonstrate consistent improvements over strong
multimodal baselines, and ablation studies verify that hierarchical architecture, scoped visual context, and distilled context contribute complementary gains.
Read the full abstract ▾
All authors Qihua Dong, Ruozhen He, Junwen Chen, Yizhou Wang, Xu Ma, Songyao Jiang, Yun Fu
Computer Vision
Beyond Fixed Thresholds and Domain-Specific Benchmarks for Explainable Multi-Task Classification in Autonomous Vehicles
Maryam Sadat Hosseini Azad and Shahriar Baradaran Shokouhi
Scene understanding is a vital part of autonomous driving systems, which requires the use of deep learning models. Deep learning methods are intrinsically black box models, which lack transparency and safety in autonomous driving. To make these systems transparent, multi-task visual understanding has become crucial for explainable autonomous driving perception systems, where simultaneous prediction of multiple driving behaviors and their underlying explanations is essential for safe
navigation and human trust in autonomous vehicles. In order to design an accurate and cross-cultural explainable autonomous driving system, we introduce a comprehensive confidence threshold sensitivity analysis that evaluates various threshold values to identify optimal decision boundaries for different tasks. Our analysis demonstrates that traditional fixed threshold approaches are suboptimal for multi-task scenarios. Through extensive evaluation, we demonstrate that our adaptive threshold selection methodology improves F1-scores across different tasks. In addition, we introduce IUST-XAI-AD, a novel
dataset consisting of 958 images with human annotations for driving decisions and corresponding
reasoning. This
dataset addresses the critical gap in domain-specific evaluation
benchmarks for distinct driving contexts and provides a more challenging test environment compared to existing
datasets. Experimental results demonstrate that confidence threshold sensitivity analysis can significantly improve model performance, while the introduction of the IUST-XAI-AD
dataset reveals important insights about cross-cultural driving behavior patterns. The combined contributions of this work provide both methodological advances and practical evaluation tools that can accelerate the development of more reliable, explainable, and culturally-adaptive autonomous driving systems for global deployment.
Read the full abstract ▾
Computer Vision
Imagery Dataset for Remaining Useful Life Estimation of Synthetic Fibre Ropes
Anju Rani, Daniel Ortiz-Arroyo and Petar Durdevic
Remaining useful life (RUL) estimation of synthetic fibre ropes (SFRs) is critical for safe operation in offshore-crane, wind turbine installation, and heavy-load handling applications, where rope failure can result in catastrophic safety incidents and costly downtime. Despite growing research interest in data-driven condition monitoring, there is no publicly available image
dataset that captures the complete degradation lifecycle of SFRs under controlled cyclic fatigue loading. To address this gap, we present a novel image
dataset comprising approximately 34,700 high-resolution images of eleven Dyneema SK75/78 high-modulus polyethylene (HMPE) rope samples subjected to cyclic fatigue on a sheave-bend test stand at seven distinct axial load levels ranging from 60 kN to 280 kN. Ropes were loaded until mechanical failure, with fatigue lifetimes ranging from 695 cycles to 8,340 cycles. After every fixed number of sheave cycles (an inspection burst), ten images were captured at different cross-sectional positions along the rope, providing spatially representative sampling of surface degradation throughout the rope's entire service life. The images obtained from each load are annotated with the corresponding elapsed cycle count, enabling a direct computation of RUL for any rope in the sequence. This
dataset aims to support a broad range of machine learning (ML) tasks including RUL regression, damage progression modelling, anomaly detection, and load-conditioned prognostics. The
dataset is intended to serve as a
benchmark resource for the development and comparison of
vision-based condition monitoring (CM) and prognostics
algorithms for SFRs.
Read the full abstract ▾
Computer Vision
Physics-Guided Regime Unmixing
Paula Pacheco, Pablo Granitto and Juan B. Cabral
The Linear Mixing Model (LMM) dominates spectral unmixing for its simplicity, but fails under multiple scattering; existing nonlinear models compensate by applying a fixed regime uniformly across entire scenes. We propose Physics-Guided Regime Unmixing (PGRU), which estimates a pixel-wise scalar $\xi_i \in [0,1]$ from observable physical features to activate nonlinear mixing only where justified. Residuals from the Generalized Bilinear Model (GBM), the Post-Nonlinear Mixing Model (PPNM), and Hapke are combined via learned
attention, yielding interpretable regime maps. Experiments on Samson, Jasper Ridge, and Urban show consistent improvements over baselines, with physical coherence $\rho > 0.90$.
Read the full abstract ▾
Computer Vision
Densification and forecasting of Sentinel-2 time series from multimodal SAR and Optical satellite data using deep generative models
Véronique Defonte, Dawa Derksen, Alexandre Constantin et al. (+1 more)
Optical satellite image time series are extensively used in many Earth observation applications, including agriculture,
climate monitoring, and land surface analysis. However, clouds and swath edges result in irregular sampling along the temporal dimension, limiting continuous monitoring. To address this issue, a growing body of work has focused on temporal densification and reconstruction of satellite image time series, with the objective of filling missing or cloud-contaminated observations within the temporal extent of the available data. While these approaches improve temporal continuity, they are inherently restricted to the reconstruction of the gaps within the observed time periods, and do not address the prediction of future observations. This work proposes a probabilistic deep learning framework for the densification and
forecasting of Sentinel-2 time series by generating optical images at arbitrary past or future dates. The approach leverages
multimodal satellite data by jointly exploiting Sentinel-2 optical and Sentinel-1 SAR observations. Unlike most existing works, we propose to focus on the uncertainty of the generated images. Experimental results demonstrate effective densification and
forecasting, on sparse and temporally misaligned time series.
Read the full abstract ▾
All authors Véronique Defonte, Dawa Derksen, Alexandre Constantin, Bastien Nespoulous
Computer Vision
Disentangled Learning Improves Implicit Neural Representations for Medical Reconstruction
Qing Wu, Xuanyu Tian, Chenhe Du et al. (+4 more)
Implicit neural representations (INRs) have emerged as a powerful paradigm for medical imaging via physics-informed unsupervised learning. Classical INRs optimize an entire network from scratch for each subject, leading to inefficient training and suboptimal imaging quality. Recent initialization-based approaches attempt to inject population priors into pre-trained networks, yet they rely on high-quality images and often suffer from catastrophic forgetting during fine-tuning. We present DisINR, a novel INR framework that explicitly disentangles shared and subject-specific representations. DisINR introduces a shared encoder-decoder pair and subject-specific encoders, whose features are jointly decoded for image reconstruction. By integrating differentiable forward models, it pre-trains the shared modules directly from limited raw measurements, removing the need for pre-acquired high-quality images. During test-time adaptation, only the subject-specific encoder is optimized, while the shared pair remains frozen, effectively preserving learned priors. Extensive evaluations on three representative medical imaging tasks show that DisINR significantly outperforms state-of-the-art INRs in both reconstruction accuracy and efficiency.
Read the full abstract ▾
All authors Qing Wu, Xuanyu Tian, Chenhe Du, Haonan Zhang, Xiao Wang, Le Lu, Yuyao Zhang
Computer Vision
Anatomy of a failure: When, how, and why deep vision fails in scientific domains
Ji-Hun Oh, Dou Hoon Kwark, Kianoush Falahkheirkhah et al. (+4 more)
Mirroring its ubiquity in popular media and all human activities, the use of deep learning (DL) is rapidly growing in scientific imaging modalities. However, unlike everyday RGB pictures, pixels encode precise physicochemical properties in scientific imaging across potentially thousands of channels. While DL is well validated on human-centric RGB perceptual tasks, its effectiveness for scientific imaging remains uncertain. Here, we show that the naive application of DL frameworks to scientific images can lead to critical failures. We evaluate the use of DL for pathology, comparing RGB images of stained tissue with the quantitative and information-rich biochemical signatures of infrared (IR) imaging. Despite this informational advantage, DL models trained on IR data paradoxically underperform. We investigate this discrepancy to find that IR data priors interact poorly with the simplicity bias of DL, causing models to collapse to one-dimensional predictions. This constitutes a catastrophic DL failure because the model's representational capacity remains largely unused, while furthermore raising AI safety concerns and undermining the advantages of such scientific modalities. Notably, this problem persists even with state-of-the-art DL robustification strategies, which are primarily designed and validated for RGB imagery and thus inherit the same prior-bias mismatch. This work establishes a framework for understanding the limitations of generic DL in science and advocates for the study of modality-specific failure modes to guide the development of specialized, safe AI
algorithms.
Read the full abstract ▾
All authors Ji-Hun Oh, Dou Hoon Kwark, Kianoush Falahkheirkhah, Kevin Yeh, John Cheville, Volodymyr Kindratenko, Rohit Bhargava
Computer Vision
Topology-Constrained Quantized nnUNet for Efficient and Anatomically Accurate 3D Tooth Segmentation
Paarth Prasad and Ruchika Malhotra
We propose a
topology-constrained quantized nnUNet framework for efficient and anatomically accurate 3D tooth
segmentation, addressing the challenges of spatial distortion introduced by quantization in deep learning models. The proposed method integrates a novel tooth-specific topological loss into quantization-aware training, preserving critical anatomical structures such as tooth count, adjacency relationships, and cavity integrity while maintaining computational efficiency. The system employs an 8-bit quantized nnUNet backbone, where weights and activations are dynamically calibrated to minimize precision loss during inference. Furthermore, the topological loss combines connected-component analysis, adjacency consistency, and hole detection penalties, ensuring anatomical fidelity without modifying the underlying network architecture. The joint optimization objective harmonizes cross-
entropy loss, quantization regularization, and topological constraints, enabling end-to-end training with gradient approximations for persistent homology terms. Experiments demonstrate that our approach significantly reduces topological errors compared to conventional quantized models, achieving clinically plausible
segmentations on dental CBCT scans. The method retains the hardware efficiency of integer-only inference, making it suitable for deployment in resource-constrained clinical environments. This work bridges the gap between computational efficiency and anatomical precision in medical image
segmentation, offering a practical solution for real-world dental applications.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.GR
JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation
Lin Song, Wenbo Li, Guoqing Ma et al. (+16 more)
We present JoyAI-Image, a unified
multimodal foundation model for visual understanding, text-to-image generation, and instruction-guided image editing. JoyAI-Image couples a spatially enhanced
Multimodal Large
Language Model (MLLM) with a
Multimodal Diffusion Transformer (MMDiT), allowing perception and generation to interact through a shared
multimodal interface. Around this architecture, we build a scalable training recipe that combines unified instruction tuning, long-text rendering supervision, spatially grounded data, and both general and spatial editing signals. This design gives the model broad
multimodal capability while strengthening geometry-aware
reasoning and controllable visual synthesis. Experiments across understanding, generation, long-text rendering, and editing
benchmarks show that JoyAI-Image achieves state-of-the-art or highly competitive performance. More importantly, the bidirectional loop between enhanced understanding, controllable spatial editing, and novel-view-assisted
reasoning enables the model to move beyond general visual competence toward stronger spatial intelligence. These results suggest a promising path for unified visual models in downstream applications such as
vision-
language-action systems and world models.
Read the full abstract ▾
All authors Lin Song, Wenbo Li, Guoqing Ma, Wei Tang, Bo Wang, Yuan Zhang, Yijun Yang, Yicheng Xiao, Jianhui Liu, Yanbing Zhang, Guohui Zhang, Wenhu Zhang, Hang Xu, Nan Jiang, Xin Han, Haoze Sun, Maoquan Zhang, Haoyang Huang, Nan Duan
Computer Vision
Audio-Visual Intelligence in Large Foundation Models
You Qin, Kai Liu, Shengqiong Wu et al. (+12 more)
Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate, and interact in the
multimodal real world. In the era of large foundation models, joint modeling of audio and
vision has become increasingly crucial, i.e., not only for understanding but also for controllable generation and
reasoning across dynamic, temporally grounded signals. Recent advances, such as Meta MovieGen and Google Veo-3, highlight the growing industrial and academic focus on unified audio-
vision architectures that learn from massive
multimodal data. However, despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent taxonomies, and heterogeneous evaluation practices that impede systematic comparison and knowledge integration. This survey provides the first comprehensive review of AVI through the lens of large foundation models. We establish a unified taxonomy covering the broad landscape of AVI tasks, ranging from understanding (e.g.,
speech recognition, sound localization) to generation (e.g., audio-driven
video synthesis,
video-to-audio) and interaction (e.g., dialogue,
embodied, or agentic interfaces). We synthesize methodological foundations, including modality tokenization, cross-modal fusion, autoregressive and
diffusion-based generation, large-scale pretraining, instruction
alignment, and preference optimization. Furthermore, we curate representative
datasets,
benchmarks, and evaluation metrics, offering a structured comparison across task families and identifying open challenges in synchronization, spatial
reasoning, controllability, and safety. By consolidating this rapidly expanding field into a coherent framework, this survey aims to serve as a foundational reference for future research on large-scale AVI.
Read the full abstract ▾
All authors You Qin, Kai Liu, Shengqiong Wu, Kai Wang, Shijian Deng, Yapeng Tian, Junbin Xiao, Yazhou Xing, Yinghao Ma, Bobo Li, Roger Zimmermann, Lei Cui, Furu Wei, Jiebo Luo, Hao Fei
Computer Vision
UniCorrn: Unified Correspondence Transformer Across 2D and 3D
Prajnan Goswami, Tianye Ding, Feng Liu et al. (+1 more)
Visual correspondence across image-to-image (2D-2D), image-to-point cloud (2D-3D), and point cloud-to-point cloud (3D-3D)
geometric matching forms the foundation for numerous 3D
vision tasks. Despite sharing a similar problem structure, current methods use task-specific designs with separate models for each modality combination. We present UniCorrn, the first correspondence model with shared weights that unifies
geometric matching across all three tasks. Our key insight is that
Transformer attention naturally captures cross-modal feature similarity. We propose a dual-stream decoder that maintains separate appearance and positional feature streams. This design enables end-to-end learning through stack-able layers while supporting flexible query-based correspondence estimation across heterogeneous modalities. Our architecture employs modality-specific backbones followed by shared encoder and decoder components, trained jointly on diverse data combining pseudo point clouds from depth maps with real 3D correspondence annotations. UniCorrn achieves competitive performance on 2D-2D matching and surpasses prior state-of-the-art by 8% on 7Scenes (2D-3D) and 10% on 3DLoMatch (3D-3D) in registration recall. Project website: https://neu-vi.github.io/UniCorrn
Read the full abstract ▾
All authors Prajnan Goswami, Tianye Ding, Feng Liu, Huaizu Jiang
Computer Vision
Large Language Models are Universal Reasoners for Visual Generation
Sucheng Ren, Chen Chen, Zhenbang Wang et al. (+5 more)
Text-to-image generation has advanced rapidly with
diffusion models, progressing from CLIP and T5 conditioning to unified systems where a single LLM backbone handles both visual understanding and generation. Despite the architectural unification, these systems frequently fail to faithfully align complex
prompts during synthesis, even though they remain highly accurate at verifying whether an image satisfies those same
prompts. We formalize this as the understanding-generation gap and propose UniReasoner, a framework that leverages the LLM as a universal reasoner to convert its understanding strength into direct generation guidance. Given a
prompt, the LLM first produces a coarse visual draft composed of discrete
vision tokens. It then performs a self-critique by evaluating the draft for
prompt consistency, producing a grounded textual evaluation that pinpoints what needs to be corrected. Finally, a
diffusion model is conditioned jointly on the
prompt, the visual draft, and the evaluation, ensuring that generation is guided by explicit corrective signals. Each signal addresses a limitation of the other: the draft provides a concrete, scene-level anchor that reduces under-specification in text-only conditioning, while the evaluation turns verification into grounded, actionable constraints that correct omissions,
hallucinations, and relational errors. Experiments show that UniReasoner improves compositional
alignment and semantic faithfulness under the same
diffusion backbone while maintaining image quality, demonstrating a practical way to exploit LLM
reasoning to close the understanding-generation gap.
Read the full abstract ▾
All authors Sucheng Ren, Chen Chen, Zhenbang Wang, Liangchen Song, Xiangxin Zhu, Alan Yuille, Liang-Chieh Chen, Jiasen Lu