Machine Learning
Knowledge-Embedded Latent Projection for Robust Representation Learning
Weijing Tang, Ming Yuan, Zongqi Xia et al. (+1 more)
Latent space models are widely used for analyzing high-dimensional discrete data
matrices, such as patient-feature
matrices in electronic health records (EHRs), by capturing complex dependence structures through low-dimensional
embeddings. However, estimation becomes challenging in the imbalanced regime, where one
matrix dimension is much larger than the other. In EHR applications, cohort sizes are often limited by disease prevalence or data availability, whereas the feature space remains extremely large due to the breadth of medical coding system. Motivated by the increasing availability of external semantic
embeddings, such as pre-trained
embeddings of clinical concepts in EHRs, we propose a knowledge-embedded latent projection model that leverages semantic side information to regularize representation learning. Specifically, we model column
embeddings as smooth functions of semantic
embeddings via a mapping in a reproducing kernel
Hilbert space. We develop a computationally efficient two-step estimation procedure that combines semantically guided subspace construction via kernel principal component analysis with scalable projected gradient descent. We establish estimation error bounds that characterize the trade-off between statistical error and approximation error induced by the kernel projection. Furthermore, we provide local convergence guarantees for our non-convex optimization procedure. Extensive simulation studies and a real-world EHR application demonstrate the effectiveness of the proposed method.
Read the full abstract ▾
All authors Weijing Tang, Ming Yuan, Zongqi Xia, Tianxi Cai
Machine Learning
Causality is Key for Interpretability Claims to Generalise
Shruti Joshi, Aaron Mueller, David Klindt et al. (+3 more)
Interpretability research on large
language models (LLMs) has yielded important insights into model behaviour, yet recurring pitfalls persist: findings that do not generalise, and causal interpretations that outrun the evidence. Our position is that causal inference specifies what constitutes a valid mapping from model activations to invariant high-level structures, the data or assumptions needed to achieve it, and the inferences it can support. Specifically, Pearl's causal hierarchy clarifies what an interpretability study can justify. Observations establish associations between model behaviour and internal components. Interventions (e.g., ablations or activation patching) support claims how these edits affect a behavioural metric (e.g., average change in
token probabilities) over a set of
prompts. However, counterfactual claims – i.e., asking what the model output would have been for the same
prompt under an unobserved intervention – remain largely unverifiable without controlled supervision. We show how causal representation learning (CRL) operationalises this hierarchy, specifying which variables are recoverable from activations and under what assumptions. Together, these motivate a diagnostic framework that helps practitioners select methods and evaluations matching claims to evidence such that findings generalise.
Read the full abstract ▾
All authors Shruti Joshi, Aaron Mueller, David Klindt, Wieland Brendel, Patrik Reizinger, Dhanya Sridhar
Machine Learning
Protecting the Undeleted in Machine Unlearning
Aloni Cohen, Refael Kohen, Kobbi Nissim et al. (+1 more)
Machine unlearning aims to remove specific data points from a trained model, often striving to emulate "perfect retraining", i.e., producing the model that would have been obtained had the deleted data never been included. We demonstrate that this approach, and security definitions that enable it, carry significant
privacy risks for the remaining (undeleted) data points. We present a reconstruction attack showing that for certain tasks, which can be computed securely without deletions, a mechanism adhering to perfect retraining allows an adversary controlling merely $\omega(1)$ data points to reconstruct almost the entire
dataset merely by issuing deletion requests. We survey existing definitions for machine unlearning, showing they are either susceptible to such attacks or too restrictive to support basic functionalities like exact summation. To address this problem, we propose a new security definition that specifically safeguards undeleted data against leakage caused by the deletion of other points. We show that our definition permits several essential functionalities, such as bulletin boards, summations, and statistical learning.
Read the full abstract ▾
All authors Aloni Cohen, Refael Kohen, Kobbi Nissim, Uri Stemmer
Machine Learning · Cross-listed from q-bio.GN
Simple models outperform single-cell foundation models on downstream benchmarks
Huan Souza and Pankaj Mehta
Single-
cell RNA sequencing (scRNA-seq) data exhibit strong and reproducible statistical structure. This has motivated the development of large-scale foundation models, such as TranscriptFormer, that use
transformer-based architectures to learn a generative model for gene expression by
embedding genes into a latent vector space. These
embeddings have been used to obtain state-of-the-art (SOTA) performance on downstream tasks such as
cell-type classification, disease-state prediction, and cross-species learning. Here, we ask whether similar performance can be achieved without utilizing computationally intensive deep learning-based representations. Using simple, interpretable pipelines that rely on careful normalization and linear methods, we obtain SOTA or near SOTA performance across multiple
benchmarks commonly used to evaluate single-
cell foundation models, including outperforming foundation models on out-of-distribution tasks involving novel
cell types and organisms absent from the training data. Our findings highlight the need for rigorous benchmarking and suggest that the biology of
cell identity can be captured by simple linear representations of single
cell gene expression data.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ME
Synthetic-Powered Multiple Testing with FDR Control
Yonghoon Lee, Meshi Bashari, Edgar Dobriban et al. (+1 more)
Multiple hypothesis testing with false discovery rate (FDR) control is a fundamental problem in statistical inference, with broad applications in genomics, drug screening, and outlier detection. In many such settings, researchers may have access not only to real experimental observations but also to auxiliary or synthetic data – from past, related experiments or generated by generative models – that can provide additional evidence about the hypotheses of interest. We introduce SynthBH, a synthetic-powered multiple testing procedure that safely leverages such synthetic data. We prove that SynthBH guarantees finite-sample, distribution-free FDR control under a mild PRDS-type positive dependence condition, without requiring the pooled-data p-values to be valid under the null. The proposed method adapts to the (unknown) quality of the synthetic data: it enhances the sample efficiency and may boost the power when synthetic data are of high quality, while controlling the FDR at a user-specified level regardless of their quality. We demonstrate the empirical performance of SynthBH on tabular outlier detection
benchmarks and on genomic analyses of drug-cancer sensitivity associations, and further study its properties through controlled experiments on simulated data.
Read the full abstract ▾
All authors Yonghoon Lee, Meshi Bashari, Edgar Dobriban, Yaniv Romano
Machine Learning · Cross-listed from cs.CV
Are Object-Centric Representations Better At Compositional Generalization?
Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Maximilian Seitzer et al. (+4 more)
Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning. Object-centric (OC) representations, which encode a scene as a set of objects, are often argued to support such generalization, but systematic evidence in visually rich settings is limited. We introduce a Visual Question Answering
benchmark across three controlled visual worlds (CLEVRTex, Super-CLEVR, and MOVi-C) to measure how well
vision encoders, with and without object-centric biases, generalize to unseen combinations of object properties. To ensure a fair and comprehensive comparison, we carefully account for training data diversity, sample size, representation size, downstream model capacity, and compute. We use DINOv2 and SigLIP2, two widely used
vision encoders, as the foundation models and their OC counterparts. Our key findings reveal that (1) OC approaches are superior in harder compositional generalization settings; (2) original dense representations surpass OC only on easier settings and typically require substantially more downstream compute; and (3) OC models are more sample efficient, achieving stronger generalization with fewer images, whereas dense encoders catch up or surpass them only with sufficient data and diversity. Overall, object-centric representations offer stronger compositional generalization when any one of
dataset size, training data diversity, or downstream compute is constrained.
Read the full abstract ▾
All authors Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Maximilian Seitzer, Karl Henrik Johansson, Carsten Marr, Stefan Bauer, Andrea Dittadi
Machine Learning · Cross-listed from cs.CC
On the Hardness of Approximation of the Fair k-Center Problem
Suhas Thejaswi
In this work, we study the hardness of approximation of the fair $k$-center problem. In this problem, we are given a set of data points in a metric space that is partitioned into groups and the task is to choose a subset of $k$-data points, called centers, such that a prescribed number of data points from each group are chosen while minimizing the maximum distance from any point to its closest center. Although a
polynomial-time $3$-approximation is known for fair $k$-center in general metrics, it has remained open whether this approximation guarantee is tight or could be further improved, especially since the classical unconstrained $k$-center problem admits a
polynomial-time factor-$2$ approximation. We resolve this open question by proving that, assuming $\mathsf{P} \neq \mathsf{NP}$, for any $\epsilon>0$, no
polynomial-time
algorithm can approximate fair $k$-center to $(3-\epsilon)$-factor. Our inapproximability results hold even when only two disjoint groups are present and at least one center must be chosen from each group. Further, it extends to the canonical one-per-group setting with $k$-groups (for arbitrary $k$), where exactly one center must be selected from each group. Consequently, the factor-$3$ barrier for fair $k$-center in general metric spaces is inherent, and existing $3$-approximation
algorithms are optimal up to lower-order terms even in these restricted regimes. This result stands in sharp contrast to the $k$-supplier formulation, where both the unconstrained and fair variants admit factor-$3$ approximation in
polynomial time.
Read the full abstract ▾
Machine Learning
Retrieval-Augmented Foundation Models for Matched Molecular Pair Transformations to Recapitulate Medicinal Chemistry Intuition
Bo Pan, Peter Zhiping Zhang, Hao-Wei Pang et al. (+4 more)
Matched molecular pairs (MMPs) capture the local chemical edits that medicinal chemists routinely use to design analogs, but existing ML approaches either operate at the whole-molecule level with limited edit controllability or learn MMP-style edits from restricted settings and small models. We propose a variable-to-variable formulation of analog generation and train a foundation model on large-scale MMP transformations (MMPTs) to generate diverse variables conditioned on an input variable. To enable practical control, we develop prompting mechanisms that let the users specify preferred transformation patterns during generation. We further introduce MMPT-RAG, a
retrieval-augmented framework that uses external reference analogs as contextual guidance to steer generation and generalize from project-specific series. Experiments on general chemical corpora and patent-specific
datasets demonstrate improved diversity, novelty, and controllability, and show that our method recovers realistic analog structures in practical discovery scenarios.
Read the full abstract ▾
All authors Bo Pan, Peter Zhiping Zhang, Hao-Wei Pang, Alex Zhu, Xiang Yu, Liying Zhang, Liang Zhao
Machine Learning
Neighborhood Stability as a Measure of Nearest Neighbor Searchability
Thomas Vecchiato and Sebastian Bruch
Clustering-based Approximate Nearest Neighbor Search (ANNS) organizes a set of points into partitions, and searches only a few of them to find the nearest neighbors of a query. Despite its popularity, there are virtually no analytical tools to determine the suitability of clustering-based ANNS for a given
dataset – what we call "searchability." To address that gap, we present two measures for flat clusterings of high-dimensional points in Euclidean space. First is Clustering-Neighborhood Stability Measure (clustering-NSM), an internal measure of clustering quality – a function of a clustering of a
dataset – that we show to be predictive of ANNS accuracy. The second, Point-Neighborhood Stability Measure (point-NSM), is a measure of clusterability – a function of the
dataset itself – that is predictive of clustering-NSM. The two together allow us to determine whether a
dataset is searchable by clustering-based ANNS given only the data points. Importantly, both are functions of nearest neighbor relationships between points, not distances, making them applicable to various distance functions including inner product.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Towards a Science of AI Agent Reliability
Stephan Rabanser, Sayash Kapoor, Peter Kirgis et al. (+3 more)
AI
agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard
benchmarks suggest rapid progress, many
agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing
agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether
agents behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity. Grounded in safety-critical engineering, we provide a holistic performance profile by proposing twelve concrete metrics that decompose
agent reliability along four key dimensions: consistency, robustness, predictability, and safety. Evaluating 15 models across two complementary
benchmarks, we find that recent capability gains have only yielded small improvements in reliability. By exposing these persistent limitations, our metrics complement traditional evaluations while offering tools for
reasoning about how
agents perform, degrade, and fail.
Read the full abstract ▾
All authors Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, Arvind Narayanan
Machine Learning · Cross-listed from cs.CL
Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment
Yuyan Bu, Xiaohao Liu, ZhaoXing Ren et al. (+2 more)
The widespread deployment of large
language models (LLMs) across linguistic communities necessitates reliable
multilingual safety
alignment. However, recent efforts to extend
alignment to other
languages often require substantial resources, either through large-scale, high-quality supervision in the target
language or through pairwise
alignment with high-resource
languages, which limits scalability. In this work, we propose a resource-efficient method for improving
multilingual safety
alignment. We introduce a plug-and-play Multi-Lingual Consistency (MLC) loss that can be integrated into existing monolingual
alignment pipelines. By improving collinearity between
multilingual representation vectors, our method encourages directional consistency at the
multilingual semantic level in a single update. This allows simultaneous
alignment across multiple
languages using only
multilingual prompt variants without requiring additional response-level supervision in low-resource
languages. We validate the proposed method across different model architectures and
alignment paradigms, and demonstrate its effectiveness in enhancing
multilingual safety with limited impact on general model utility. Further evaluation across
languages and tasks indicates improved cross-lingual generalization, suggesting the proposed approach as a practical solution for
multilingual consistency
alignment under limited supervision.
Read the full abstract ▾
All authors Yuyan Bu, Xiaohao Liu, ZhaoXing Ren, Yaodong Yang, Juntao Dai
Machine Learning · Cross-listed from astro-ph.SR
Investigating Nonlinear Quenching Effects on Polar Field Buildup in the Sun Using Physics-Informed Neural Networks
Jithu J. Athalathil, Mohammed H. Talafha and Bhargav Vaidya
The
solar dynamo relies on the regeneration of the poloidal
magnetic field through processes strongly modulated by nonlinear feedbacks such as tilt quenching (TQ) and latitude quenching (LQ). These mechanisms play a decisive role in regulating the buildup of the Sun's polar field and, in turn, the amplitude of future
solar cycles. In this work, we employ Physics-Informed Neural Networks (PINN) to solve the surface flux transport (SFT) equation,
embedding physical constraints directly into the neural network framework. By systematically varying transport parameters, we isolate the relative contributions of TQ and LQ to polar dipole buildup. We use the residual dipole moment as a diagnostic for cycle-to-cycle amplification and show that TQ suppression strengthens with increasing diffusivity, while LQ dominates in advection-dominated regimes. The ratio $\Delta D_{\mathrm{LQ}}/\Delta D_{\mathrm{TQ}}$ exhibits a smooth inverse-square dependence on the dynamo effectivity range, refining previous empirical fits with improved accuracy and reduced scatter. The results further reveal that the need for a decay term is not essential for PINN set-up due to the training process. Compared with the traditional 1D SFT model, the PINN framework achieves significantly lower error metrics and more robust recovery of nonlinear trends. Our results suggest that the nonlinear interplay between LQ and TQ can naturally produce alternations between weak and strong cycles, providing a physical explanation for the observed even-odd cycle modulation. These findings demonstrate the potential of PINN as an accurate, efficient, and physically consistent tool for
solar cycle prediction.
Read the full abstract ▾
Machine Learning
Factorization Machine with Quadratic-Optimization Annealing for RNA Inverse Folding and Evaluation of Binary-Integer Encoding and Nucleotide Assignment
Shuta Kikuchi and Shu Tanaka
The RNA inverse folding problem aims to identify nucleotide sequences that preferentially adopt a given target secondary structure. While various heuristic and machine learning-based approaches have been proposed, many require a large number of sequence evaluations, which limits their applicability when experimental validation is costly. We propose a method to solve the problem using a factorization machine with quadratic-optimization annealing (FMQA). FMQA is a discrete black-box optimization method reported to obtain high-quality solutions with a limited number of evaluations. Applying FMQA to the problem requires converting nucleotides into binary variables. However, the influence of integer-to-nucleotide assignments and binary-integer encoding on the performance of FMQA has not been thoroughly investigated, even though such choices determine the structure of the surrogate model and the search landscape, and thus can directly affect solution quality. Therefore, this study aims both to establish a novel FMQA framework for RNA inverse folding and to analyze the effects of these assignments and encoding methods. We evaluated all 24 possible assignments of the four nucleotides to the ordered integers (0-3), in combination with four binary-integer encoding methods. Our results demonstrated that one-hot and domain-wall encodings outperform binary and unary encodings in terms of the normalized ensemble defect value. In domain-wall encoding, nucleotides assigned to the boundary integers (0 and 3) appeared with higher frequency. In the RNA inverse folding problem, assigning guanine and cytosine to these boundary integers promoted their enrichment in stem regions, which led to more thermodynamically stable secondary structures than those obtained with one-hot encoding.
Read the full abstract ▾
Machine Learning
Optimizer choice matters for the emergence of Neural Collapse
Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk et al. (+1 more)
Neural Collapse (NC) refers to the emergence of highly symmetric
geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods. In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay in adaptive optimizers, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics. Also, we show the accelerating effect of momentum on NC (beyond convergence of train loss) when trained with SGD, being the first result concerning momentum in the context of NC. Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various
datasets, architectures, optimizers, and hyperparameters, confirming our theoretical results. This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights the overlooked role of weight-decay coupling in shaping the implicit biases of optimizers.
Read the full abstract ▾
All authors Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk, Aurelien Lucchi
Machine Learning · Cross-listed from stat.ML
Enhanced Diffusion Sampling: Efficient Rare Event Sampling and Free Energy Calculation with Diffusion Models
Yu Xie, Ludwig Winkler, Lixin Sun et al. (+10 more)
The rare-event sampling problem has long been the central limiting factor in molecular dynamics (MD), especially in biomolecular simulation. Recently,
diffusion models such as BioEmu have emerged as powerful
equilibrium samplers that generate independent samples from complex molecular distributions, eliminating the cost of sampling rare transition events. However, a sampling problem remains when computing observables that rely on states which are rare in
equilibrium, for example folding free energies. Here, we introduce enhanced
diffusion sampling, enabling efficient exploration of rare-event regions while preserving unbiased thermodynamic
estimators. The key idea is to perform quantitatively accurate steering protocols to generate biased ensembles and subsequently recover
equilibrium statistics via exact reweighting. We instantiate our framework in three
algorithms: UmbrellaDiff (umbrella sampling with
diffusion models), MetaDiff (a batchwise analogue for metadynamics), and $\Delta$G-Diff (free-energy differences via tilted ensembles). Across toy systems,
protein folding landscapes and folding free energies, our methods achieve fast, accurate, and scalable estimation of
equilibrium properties within GPU-minutes to hours per system-closing the rare-event sampling gap that remained after the advent of
diffusion-model
equilibrium samplers.
Read the full abstract ▾
All authors Yu Xie, Ludwig Winkler, Lixin Sun, Sarah Lewis, Adam E. Foster, José Jiménez Luna, Tim Hempel, Michael Gastegger, Yaoyi Chen, Iryna Zaporozhets, Cecilia Clementi, Christopher M. Bishop, Frank Noé
Machine Learning
Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
Ethan Blaser, Jiuqi Wang and Shangtong Zhang
The average reward is a fundamental performance metric in reinforcement learning (RL) focusing on the long-run performance of an
agent. Differential temporal difference (TD) learning
algorithms are a major advance for average reward RL as they provide an efficient online method to learn the value functions associated with the average reward in both on-policy and off-policy settings. However, existing convergence guarantees require a local clock in learning rates tied to state visit counts, which practitioners do not use and does not extend beyond tabular settings. We address this limitation by proving the almost sure convergence of on-policy $n$-step differential TD for any $n$ using standard diminishing learning rates without a local clock. We then derive three sufficient conditions under which off-policy $n$-step differential TD also converges without a local clock. These results strengthen the theoretical foundations of differential TD and bring its convergence analysis closer to practical implementations.
Read the full abstract ▾
Machine Learning
A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models
SungJun Cho, Chetan Gohil, Rukuang Huang et al. (+2 more)
Recent success in natural
language processing has motivated growing interest in large-scale foundation models for neuroimaging data. Such models often require discretization of continuous neural time series data, a process referred to as 'tokenization'. However, the impact of different tokenization strategies for neural data is currently poorly understood. In this work, we present a systematic evaluation of sample-level tokenization strategies for
transformer-based large neuroimaging models (LNMs) applied to magnetoencephalography (MEG) data. We compare learnable and non-learnable tokenizers by examining their signal reconstruction fidelity and their impact on subsequent foundation modeling performance (
token prediction, biological plausibility of generated data, preservation of subject-specific information, and performance on downstream tasks). For the learnable tokenizer, we introduce a novel approach based on an autoencoder. Experiments were conducted on three publicly available MEG
datasets spanning different acquisition sites, scanners, and experimental paradigms. Our results show that both learnable and non-learnable discretization schemes achieve high reconstruction accuracy and broadly comparable performance across most evaluation criteria, suggesting that simple fixed sample-level tokenization strategies can be used in the development of neural foundation models. The code is available at https://github.com/OHBA-analysis/Cho2026_Tokenizer.
Read the full abstract ▾
All authors SungJun Cho, Chetan Gohil, Rukuang Huang, Oiwi Parker Jones, Mark W. Woolrich
Machine Learning · Cross-listed from cs.CL
Who can we trust? LLM-as-a-jury for Comparative Assessment
Mengjie Qian, Guangzhi Sun, Mark J.F. Gales et al. (+1 more)
Large
language models (LLMs) are increasingly applied as automatic evaluators for natural
language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and evaluation aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-asa-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on
benchmark NLG evaluation
datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminators strongly correlate with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.
Read the full abstract ▾
All authors Mengjie Qian, Guangzhi Sun, Mark J.F. Gales, Kate M. Knill
Machine Learning · Cross-listed from cs.CL
Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models
Melkamu Abay Mersha and Jugal Kalita
Transformer models achieve state-of-the-art performance across domains and tasks, yet their deeply layered representations make their predictions difficult to interpret. Existing explainability methods rely on final-layer attributions, capture either local
token-level attributions or global
attention patterns without unification, and lack context-awareness of inter-
token dependencies and structural components. They also fail to capture how relevance evolves across layers and how structural components shape decision-making. To address these limitations, we proposed the Context-Aware Layer-wise Integrated Gradients (CA-LIG) Framework, a unified hierarchical attribution framework that computes layer-wise Integrated Gradients within each
Transformer block and fuses these
token-level attributions with class-specific
attention gradients. This integration yields signed, context-sensitive attribution maps that capture supportive and opposing evidence while tracing the hierarchical flow of relevance through the
Transformer layers. We evaluate the CA-LIG Framework across diverse tasks, domains, and
transformer model families, including sentiment analysis and long and multi-class document classification with BERT, hate
speech detection in a low-resource
language setting with XLM-R and AfroLM, and image classification with Masked Autoencoder
vision Transformer model. Across all tasks and architectures, CA-LIG provides more faithful attributions, shows stronger sensitivity to contextual dependencies, and produces clearer, more semantically coherent visualizations than established explainability methods. These results indicate that CA-LIG provides a more comprehensive, context-aware, and reliable explanation of
Transformer decision-making, advancing both the practical interpretability and conceptual understanding of deep neural models.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Quantifying Error Propagation and Model Collapse in Diffusion Models
Nail B. Khelifa, Richard E. Turner and Ramji Venkataramanan
Machine learning models are increasingly trained or fine-tuned on synthetic data. Recursively training on such data has been observed to significantly degrade performance in a wide range of tasks, often characterized by a progressive drift away from the target distribution. In this work, we theoretically analyze this phenomenon in the setting of score-based
diffusion models. For a realistic pipeline where each training round uses a combination of synthetic data and fresh samples from the target distribution, we obtain upper and lower bounds on the accumulated divergence between the generated and target distributions. Notably, to the best of our knowledge, this is the first lower bound on the divergence between the learned and target distributions, even for standard
diffusion models. Our results allow us to characterize different regimes of drift, depending on the score estimation error and the proportion of fresh data used in each generation. In a certain regime, the accumulated divergence after several retraining rounds can be expressed as a discounted sum of score estimation errors made at each generation. We also provide empirical results on synthetic data and images to illustrate the theory.
Read the full abstract ▾
Machine Learning
Predicting The Cop Number Using Machine Learning
Meagan Mann, Christian Muise and Erin Meger
Cops and Robbers is a pursuit evasion game played on a graph, first introduced independently by Quilliot \cite{quilliot1978jeux} and Nowakowski and Winkler \cite{NOWAKOWSKI1983235} over four decades ago. A main interest in recent the literature is identifying the cop number of graph families. The cop number of a graph, $c(G)$, is defined as the minimum number of cops required to guarantee capture of the robber. Determining the cop number is computationally difficult and exact
algorithms for this are typically restricted to small graph families. This paper investigates whether classical machine learning methods and graph neural networks can accurately predict a graph's cop number from its structural properties and identify which properties most strongly influence this prediction. Of the classical machine learning models, tree-based models achieve high accuracy in prediction despite class imbalance, whereas graph neural networks achieve comparable results without explicit feature engineering. The interpretability analysis shows that the most predictive features are related to node connectivity, clustering, clique structure, and width parameters, which aligns with known theoretical results. Our findings suggest that machine learning approaches can be used in complement with existing cop number
algorithms by offering scalable approximations where computation is infeasible.
Read the full abstract ▾
Machine Learning
Sequential Membership Inference Attacks
Thomas Michel, Debabrota Basu and Emilie Kaufmann
Modern AI models are not static. They go through multiple updates in their lifecycles. We propose to design Sequential Membership Inference (SeMI) attacks leading to tighter
privacy audits by exploiting the sequence of models and injecting a target canary at a controlled insertion time. First, for empirical mean computation, we develop SeMI*, an optimal SeMI attack to identify the presence of a target inserted at a specific insertion step. We derive the power of SeMI* to show that accessing the model sequence yields more powerful MI attacks than scrutinising only the final model. SeMI* exhibits an isolation property. Its power depends on the statistics obtained right before and after insertion of the target. Leveraging this insight, we develop practical white-box (accessing model gradients) and black-box (accessing loss) SeMI attacks against models trained with (DP-)SGD. Across
datasets and models trained with (DP-)SGD, our experiments show that SeMI attacks achieve higher power than snapshot-independent baselines, and yield tighter
privacy audits thanks to (a) control over the insertion time and (b) observations across the model sequence. Code is available at https://github.com/Thomick/pysemi.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
A Contrastive Learning Framework Empowered by Attention-based Feature Adaptation for Street-View Image Classification
Qi You, Yitai Cheng, Zichao Zeng et al. (+1 more)
Street-view image attribute classification is a vital downstream task of image classification, enabling applications such as autonomous driving, urban analytics, and high-definition map construction. It remains computationally demanding whether training from scratch, initialising from pre-trained weights, or fine-tuning large models. Although pre-trained
vision-
language models such as CLIP offer rich image representations, existing adaptation or fine-tuning methods often rely on their global image
embeddings, limiting their ability to capture fine-grained, localised attributes essential in complex, cluttered street scenes. To address this, we propose CLIP-MHAdapter, a variant of the current lightweight CLIP adaptation paradigm that appends a bottleneck MLP equipped with multi-head self-
attention operating on patch
tokens to model inter-patch dependencies. With approximately 1.4 million trainable parameters, CLIP-MHAdapter achieves superior or competitive accuracy across eight attribute classification tasks on the Global StreetScapes
dataset, attaining new state-of-the-art results while maintaining low computational cost. The code is available at https://github.com/SpaceTimeLab/CLIP-MHAdapter.
Read the full abstract ▾
All authors Qi You, Yitai Cheng, Zichao Zeng, James Haworth
Machine Learning
AIFL: A Global Daily Streamflow Forecasting Model Using a Deterministic LSTM Pre-trained on ERA5-Land and Fine-tuned on IFS
Maria Luisa Taccari, Kenza Tazi, Oisín M. Morrison et al. (+7 more)
Reliable global streamflow
forecasting is essential for flood preparedness and water resource management, yet data-driven models often suffer from a performance gap when transitioning from historical reanalysis to operational forecast products. This paper introduces AIFL (Artificial Intelligence for Floods), a deterministic LSTM-based model designed for global daily streamflow
forecasting. Trained on 18,588 basins curated from the Caravan
dataset, AIFL utilises a two-stage transfer-learning strategy to bridge the reanalysis-to-forecast domain shift. The model is first pre-trained on 40 years of ERA5-Land reanalysis (1980-2019) to capture robust hydrological processes, then fine-tuned on operational Integrated
Forecasting System (IFS) forecasts (2016-2019) to adapt to the specific error structures and biases of operational numerical weather prediction. Ablation experiments confirm that this two-stage approach outperforms both a naive IFS-only baseline and a mixed-forcing single-stage alternative. To our knowledge, this is the first global model trained end-to-end within the Caravan ecosystem. On an independent temporal test set (2021-2024), AIFL achieves high predictive skill with a median modified Kling-Gupta Efficiency (KGE') of 0.66 and a median Nash-Sutcliffe Efficiency (NSE) of 0.53. Benchmarking results show that AIFL achieves comparable accuracy to current state-of-the-art global systems. The model provides a streamlined and operationally robust baseline for the global hydrological community.
Read the full abstract ▾
All authors Maria Luisa Taccari, Kenza Tazi, Oisín M. Morrison, Andreas Grafberger, Juan Colonese, Corentin Carton de Wiart, Christel Prudhomme, Cinzia Mazzetti, Matthew Chantry, Florian Pappenberger
Machine Learning
MoDE-Boost: Boosting Shared Mobility Demand with Edge-Ready Prediction Models
Antonios Tziorvas, George S. Theodoropoulos and Yannis Theodoridis
Urban demand
forecasting plays a critical role in optimizing routing, dispatching, and congestion management within Intelligent Transportation Systems. By leveraging data fusion and analytics techniques, traffic demand
forecasting serves as a key intermediate measure for identifying emerging spatial and temporal demand patterns. In this paper, we tackle this challenge by proposing two gradient boosting model variations, one for classiffication and one for regression, both capable of generating demand forecasts at various temporal horizons, from 5 minutes up to one hour. Our overall approach effectively integrates temporal and contextual features, enabling accurate predictions that are essential for improving the efficiency of shared (micro-) mobility services. To evaluate its effectiveness, we utilize open shared mobility data derived from e-scooter and e-bike networks in five metropolitan areas. These real-world
datasets allow us to compare our approach with state-of-the-art methods as well as a Generative AI-based model, demonstrating its effectiveness in capturing the complexities of modern urban mobility. Ultimately, our methodology offers novel insights on urban micro-mobility management, helping to tackle the challenges arising from rapid urbanization and thus, contributing to more sustainable, efficient, and livable cities.
Read the full abstract ▾
Machine Learning
Steering diffusion models with quadratic rewards: a fine-grained analysis
Ankur Moitra, Andrej Risteski and Dhruv Rohatgi
Inference-time
algorithms are an emerging paradigm in which pre-trained models are used as subroutines to solve downstream tasks. Such
algorithms have been proposed for tasks ranging from inverse problems and guided image generation to
reasoning. However, the methods currently deployed in practice are heuristics with a variety of failure modes – and we have very little understanding of when these heuristics can be efficiently improved. In this paper, we consider the task of sampling from a reward-tilted
diffusion model – that is, sampling from $p^{\star}(x) \propto p(x) \exp(r(x))$ – given a reward function $r$ and pre-trained
diffusion oracle for $p$. We provide a fine-grained analysis of the computational tractability of this task for quadratic rewards $r(x) = x^\top A x + b^\top x$. We show that linear-reward tilts are always efficiently sampleable – a simple result that seems to have gone unnoticed in the literature. We use this as a building block, along with a conceptually new ingredient – the Hubbard-Stratonovich transform – to provide an efficient
algorithm for sampling from low-rank positive-definite quadratic tilts, i.e. $r(x) = x^\top A x$ where $A$ is positive-definite and of rank $O(1)$. For negative-definite tilts, i.e. $r(x) = - x^\top A x$ where $A$ is positive-definite, we prove that the problem is intractable even if $A$ is of rank 1 (albeit with exponentially-large entries).
Read the full abstract ▾
Machine Learning · Cross-listed from math.ST
Separating Oblivious and Adaptive Models of Variable Selection
Ziyun Chen, Jerry Li, Kevin Tian et al. (+1 more)
Sparse recovery is among the most well-studied problems in learning theory and high-dimensional statistics. In this work, we investigate the statistical and computational landscapes of sparse recovery with $\ell_\infty$ error guarantees. This variant of the problem is motivated by variable selection tasks, where the goal is to estimate the support of a $k$-sparse signal in $\mathbb{R}^d$. Our main contribution is a provable separation between the oblivious (“for each”) and adaptive (“for all”) models of $\ell_\infty$ sparse recovery. We show that under an oblivious model, the optimal $\ell_\infty$ error is attainable in near-linear time with $\approx k\log d$ samples, whereas in an adaptive model, $\gtrsim k^2$ samples are necessary for any
algorithm to achieve this bound. This establishes a surprising contrast with the standard $\ell_2$ setting, where $\approx k \log d$ samples suffice even for adaptive sparse recovery. We conclude with a preliminary examination of a partially-adaptive model, where we show nontrivial variable selection guarantees are possible with $\approx k\log d$ measurements.
Read the full abstract ▾
All authors Ziyun Chen, Jerry Li, Kevin Tian, Yusong Zhu
Machine Learning
A Scalable Approach to Solving Simulation-Based Network Security Games
Michael Lanier and Yevgeniy Vorobeychik
We introduce MetaDOAR, a lightweight meta-controller that augments the Double Oracle / PSRO paradigm with a learned, partition-aware filtering layer and Q-value caching to enable scalable multi-
agent reinforcement learning on very large cyber-network environments. MetaDOAR learns a compact state projection from per node structural
embeddings to rapidly score and select a small subset of devices (a top-k partition) on which a conventional low-level actor performs focused beam search utilizing a critic
agent. Selected candidate actions are evaluated with batched critic forwards and stored in an LRU cache keyed by a quantized state projection and local action identifiers, dramatically reducing redundant critic computation while preserving decision quality via conservative k-hop cache invalidation. Empirically, MetaDOAR attains higher player payoffs than SOTA baselines on large network
topologies, without significant scaling issues in terms of
memory usage or training time. This contribution provide a practical, theoretically motivated path to efficient hierarchical policy learning for large-scale networked decision problems.
Read the full abstract ▾
Machine Learning
Illustration of Barren Plateaus in Quantum Computing
Gerhard Stenzel, Tobias Rohe, Michael Kölle et al. (+3 more)
Variational
Quantum Circuits (VQCs) have emerged as a promising paradigm for
quantum machine learning in the NISQ era. While parameter sharing in VQCs can reduce the parameter space dimensionality and potentially mitigate the barren plateau phenomenon, it introduces a complex trade-off that has been largely overlooked. This paper investigates how parameter sharing, despite creating better global optima with fewer parameters, fundamentally alters the optimization landscape through deceptive gradients – regions where gradient information exists but systematically misleads optimizers away from global optima. Through systematic experimental analysis, we demonstrate that increasing degrees of parameter sharing generate more complex solution landscapes with heightened gradient magnitudes and measurably higher deceptiveness ratios. Our findings reveal that traditional gradient-based optimizers (Adam, SGD) show progressively degraded convergence as parameter sharing increases, with performance heavily dependent on hyperparameter selection. We introduce a novel gradient deceptiveness detection
algorithm and a quantitative framework for measuring optimization difficulty in
quantum circuits, establishing that while parameter sharing can improve
circuit expressivity by orders of magnitude, this comes at the cost of significantly increased landscape deceptiveness. These insights provide important considerations for
quantum circuit design in practical applications, highlighting the fundamental mismatch between classical optimization strategies and
quantum parameter landscapes shaped by parameter sharing.
Read the full abstract ▾
All authors Gerhard Stenzel, Tobias Rohe, Michael Kölle, Leo Sünkel, Jonas Stein, Claudia Linnhoff-Popien
Machine Learning · Cross-listed from math.OC
Learning Distributed Equilibria in Linear-Quadratic Stochastic Differential Games: An $\alpha$-Potential Approach
Philipp Plank and Yufei Zhang
We analyze independent policy-gradient (PG) learning in $N$-player linear-quadratic (LQ)
stochastic differential games. Each player employs a distributed policy that depends only on its own state and updates the policy independently using the gradient of its own objective. We establish global linear convergence of these methods to an
equilibrium by showing that the LQ game admits an $\alpha$-potential structure, with $\alpha$ determined by the degree of pairwise interaction asymmetry. For pairwise-symmetric interactions, we construct an affine distributed
equilibrium by minimizing the potential function and show that independent PG methods converge globally to this
equilibrium, with complexity scaling linearly in the population size and logarithmically in the desired accuracy. For asymmetric interactions, we prove that independent projected PG
algorithms converge linearly to an approximate
equilibrium, with suboptimality proportional to the degree of asymmetry. Numerical experiments confirm the theoretical results across both symmetric and asymmetric interaction networks.
Read the full abstract ▾