Machine Learning
Entropy-Based Dimension-Free Convergence and Loss-Adaptive Schedules for Diffusion Models
Ahmad Aghapour, Erhan Bayraktar and Ziqing Zhang
Diffusion generative models synthesize samples by discretizing reverse-time dynamics driven by a learned score (or denoiser). Existing convergence analyses of
diffusion models typically scale at least linearly with the ambient dimension, and sharper rates often depend on intrinsic-dimension assumptions or other
geometric restrictions on the target distribution. We develop an alternative, information-theoretic approach to dimension-free convergence that avoids any
geometric assumptions. Under mild assumptions on the target distribution, we bound KL divergence between the target and generated distributions by $O(H^2/K)$ (up to endpoint factors), where $H$ is the Shannon
entropy and $K$ is the number of sampling steps. Moreover, using a reformulation of the KL divergence, we propose a Loss-Adaptive Schedule (LAS) for efficient discretization of reverse SDE which is lightweight and relies only on the training loss, requiring no post-training heavy computation. Empirically, LAS improves sampling quality over common heuristic schedules.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Lev Fedorov, Michaël E. Sander, Romuald Elie et al. (+2 more)
Transformers have revolutionized deep learning across various domains but understanding the precise
token dynamics remains a theoretical challenge. Existing theories of deep
Transformers with layer normalization typically predict that
tokens cluster to a single point; however, these results rely on deterministic weight assumptions, which fail to capture the standard initialization scheme in
Transformers. In this work, we show that accounting for the intrinsic stochasticity of random initialization alters this picture. More precisely, we analyze deep
Transformers where noise arises from the random initialization of value
matrices. Under
diffusion scaling and
token-wise RMS normalization, we prove that, as the number of
Transformer layers goes to infinity, the discrete
token dynamics converge to an interacting-particle system on the sphere where
tokens are driven by a common
matrix-valued Brownian noise. In this limit, we show that initialization noise prevents the collapse to a single cluster predicted by deterministic models. For two
tokens, we prove a phase transition governed by the interaction strength and the
token dimension: unlike deterministic
attention flows, antipodal configurations become attracting with positive probability. Numerical experiments confirm the predicted transition, reveal that antipodal formations persist for more than two
tokens, and demonstrate that suppressing the intrinsic noise degrades accuracy.
Read the full abstract ▾
All authors Lev Fedorov, Michaël E. Sander, Romuald Elie, Pierre Marion, Mathieu Laurière
Machine Learning
Robust Multimodal Representation Learning in Healthcare
Xiaoguang Zhu, Linxiao Gong, Lianlong Sun et al. (+3 more)
Medical
multimodal representation learning aims to integrate heterogeneous data into unified patient representations to support clinical outcome prediction. However, real-world medical
datasets commonly contain systematic biases from multiple sources, which poses significant challenges for medical
multimodal representation learning. Existing approaches typically focus on effective
multimodal fusion, neglecting inherent biased features that affect the generalization ability. To address these challenges, we propose a Dual-Stream Feature Decorrelation Framework that identifies and handles the biases through structural causal analysis introduced by latent confounders. Our method employs a causal-biased decorrelation framework with dual-stream neural networks to disentangle causal features from spurious correlations, utilizing generalized cross-
entropy loss and mutual information minimization for effective decorrelation. The framework is model-agnostic and can be integrated into existing medical
multimodal learning methods. Comprehensive experiments on MIMIC-IV, eICU, and ADNI
datasets demonstrate consistent performance improvements.
Read the full abstract ▾
All authors Xiaoguang Zhu, Linxiao Gong, Lianlong Sun, Yang Liu, Haoyu Wang, Jing Liu
Machine Learning
LoRIF: Pushing the Scalability Frontier of Training Data Attribution
Shuangqi Li, Hieu Le, Jingyi Xu et al. (+1 more)
Training data attribution (TDA) identifies which training examples most influenced a model's prediction. Influence function methods are a theoretically grounded family of TDA methods and exploit gradients. To overcome the scalability challenge arising from gradient computation, the most popular strategy is random projection (e.g., TRAK, LoGRA). However, this still faces two bottlenecks when scaling to large training sets and high-quality attribution: (i) storing and loading projected per-example gradients for all $N$ training examples, where query latency is dominated by I/O; and (ii) forming the $D \times D$ inverse Hessian approximation, which costs $O(D^2)$
memory. Both bottlenecks scale with the projection dimension $D$, yet increasing $D$ is necessary for attribution quality – creating a quality–scalability tradeoff. We introduce LoRIF (Low-Rank Influence Functions), which exploits low-rank structures of gradient to address both bottlenecks. First, we store rank-$c$ factors of projected per-example gradients rather than full
matrices, reducing storage and query-time I/O from $O(D)$ to $O(c\sqrt{D})$ per layer per sample. Second, we use truncated SVD with the Woodbury identity to approximate the inverse Hessian term in an $r$-dimensional subspace, reducing
memory from $O(D^2)$ to $O(Dr)$. On models from 0.1B to 70B parameters trained on
datasets with millions of examples, LoRIF achieves up to 20$\times$ storage reduction and query-time speedup compared to LoGRA, while matching or exceeding its attribution quality. LoRIF makes gradient-based TDA practical at frontier scale. Code is available at https://github.com/doub7e/LoRIF.
Read the full abstract ▾
All authors Shuangqi Li, Hieu Le, Jingyi Xu, Mathieu Salzmann
Machine Learning
One-Step Bellman Alignment Enables Provably Efficient Transfer in Online RL
Elynn Chen, Enpei Zhang, Jinhang Chai et al. (+1 more)
We study online transfer reinforcement learning (RL) in episodic Markov decision processes, where experience from related source tasks is available during learning on a target task. A fundamental difficulty is that task similarity is typically defined in terms of rewards or transitions, whereas online RL
algorithms operate on Bellman regression targets. As a result, naively reusing source Bellman updates introduces systematic bias and invalidates regret guarantees. We identify one-step Bellman
alignment as the correct abstraction for transfer in online RL and propose re-weighted targeting (RWT), an operator-level correction that retargets continuation values and compensates for transition mismatch via a change of measure. RWT reduces task mismatch to a fixed one-step correction and enables statistically sound reuse of source data. This
alignment yields a two-stage RWT $Q$-learning framework that separates variance reduction from bias correction. Under RKHS function approximation, we establish regret bounds that scale with the complexity of the task shift rather than the target MDP. We further show the required density ratios admit a constructive RKHS
estimator with finite-sample guarantees, and empirically validate robustness to estimated and mis-specified ratios. Empirical results in both tabular and neural network settings demonstrate consistent improvements over single-task learning and naïve pooling, highlighting Bellman
alignment as a model-agnostic transfer principle for online RL.
Read the full abstract ▾
All authors Elynn Chen, Enpei Zhang, Jinhang Chai, Yujun Yan
Machine Learning · Cross-listed from math.OC
On Approximate Computation of Critical Points
Amir Ali Ahmadi and Georgina Hall
We show that computing even very coarse approximations of critical points is intractable for simple classes of nonconvex functions. More concretely, we prove that if there exists a
polynomial-time
algorithm that takes as input a
polynomial in $n$ variables of constant degree (as low as three) and outputs a point whose gradient has Euclidean norm at most $2^n$ whenever the
polynomial has a critical point, then P=NP. The
algorithm is permitted to return an arbitrary point when no critical point exists. We also prove hardness results for approximate computation of critical points under additional structural assumptions, including settings in which existence and uniqueness of a critical point are guaranteed, the function is lower bounded, and approximation is measured in terms of distance to a critical point. Overall, our results stand in contrast to the commonly-held belief that, in nonconvex optimization, approximate computation of critical points is a tractable task.
Read the full abstract ▾
Machine Learning
Hardware-Triggered Backdoors
Jonas Möller, Erik Imgrund, Thorsten Eisenhofer et al. (+1 more)
Machine learning models are routinely deployed on a wide range of computing hardware. Although such hardware is typically expected to produce identical results, differences in its design can lead to small numerical variations during inference. In this work, we show that these variations can be exploited to create
backdoors in machine learning models. The core idea is to shape the model's decision function such that it yields different predictions for the same input when executed on different hardware. This effect is achieved by locally moving the decision boundary close to a target input and then refining numerical deviations to flip the prediction on selected hardware. We empirically demonstrate that these hardware-triggered
backdoors can be created reliably across common GPU accelerators. Our findings reveal a novel attack vector affecting the use of third-party models, and we investigate different defenses to counter this threat.
Read the full abstract ▾
All authors Jonas Möller, Erik Imgrund, Thorsten Eisenhofer, Konrad Rieck
Machine Learning
Breaking the Regional Barrier: Inductive Semantic Topology Learning for Worldwide Air Quality Forecasting
Zhiqing Cui, Siru Zhong, Ming Jin et al. (+3 more)
Global air quality
forecasting grapples with extreme spatial heterogeneity and the poor generalization of existing transductive models to unseen regions. To tackle this, we propose OmniAir, a semantic
topology learning framework tailored for global station-level prediction. By encoding invariant physical environmental attributes into generalizable station identities and dynamically constructing adaptive sparse
topologies, our approach effectively captures long-range non-Euclidean correlations and physical
diffusion patterns across unevenly distributed global networks. We further curate WorldAir, a massive
dataset covering over 7,800 stations worldwide. Extensive experiments show that OmniAir achieves state-of-the-art performance against 18 baselines, maintaining high efficiency and scalability with speeds nearly 10 times faster than existing models, while effectively bridging the monitoring gap in data-sparse regions.
Read the full abstract ▾
All authors Zhiqing Cui, Siru Zhong, Ming Jin, Shirui Pan, Qingsong Wen, Yuxuan Liang
Machine Learning
A Low-Complexity Plug-and-Play Deep Learning Model for Generalizable Massive MIMO Precoding
Ali Hasanzadeh Karkan, Ahmed Ibrahim, Jean-François Frigon et al. (+1 more)
Massive multiple-input multiple-output (mMIMO) downlink precoding offers high spectral efficiency but remains challenging to deploy in practice because near-optimal
algorithms such as the weighted minimum mean squared error (WMMSE) are computationally expensive, and sensitive to SNR and channel-estimation quality, while existing deep learning (DL)-based solutions often lack robustness and require retraining for each deployment site. This paper proposes a plug-and-play precoder (PaPP), a DL framework with a backbone that can be trained for either fully digital (FDP) or hybrid beamforming (HBF) precoding and reused across sites, transmit-power levels, and with varying amounts of channel estimation error, avoiding the need to train a new model from scratch at each deployment. PaPP combines a high-capacity teacher and a compact student with a self-supervised loss that balances teacher imitation and normalized sum-rate, trained using meta-learning domain-generalization and transmit-power-aware input normalization. Numerical results on ray-tracing data from three unseen sites show that the PaPP FDP and HBF models both outperform conventional and deep learning baselines, after fine-tuning with a small set of local unlabeled samples. Across both architectures, PaPP achieves more than 21$\times$ reduction in modeled computation energy and maintains good performance under channel-estimation errors, making it a practical solution for energy-efficient mMIMO precoding.
Read the full abstract ▾
All authors Ali Hasanzadeh Karkan, Ahmed Ibrahim, Jean-François Frigon, François Leduc-Primeau
Machine Learning
Not All Code Is Equal: A Data-Centric Study of Code Complexity and LLM Reasoning
Lukas Twist, Shu Yang, Hanqi Yan et al. (+4 more)
Large
Language Models (LLMs) increasingly exhibit strong
reasoning abilities, often attributed to their capacity to generate chain-of-thought-style intermediate
reasoning. Recent work suggests that exposure to code can further enhance these skills, but existing studies largely treat code as a generic training signal, leaving open the question of which properties of code actually contribute to improved
reasoning. To address this gap, we study the structural complexity of code, which captures control flow and compositional structure that may shape how models internalise multi-step
reasoning during fine-tuning. We examine two complementary settings: solution-driven complexity, where complexity varies across multiple solutions to the same problem, and problem-driven complexity, where complexity reflects variation in the underlying tasks. Using cyclomatic complexity and logical lines of code to construct controlled fine-tuning
datasets, we evaluate a range of open-weight LLMs on diverse
reasoning benchmarks. Our findings show that although code can improve
reasoning, structural properties strongly determine its usefulness. In 83% of experiments, restricting fine-tuning data to a specific structural complexity range outperforms training on structurally diverse code, pointing to a data-centric path for improving
reasoning beyond scaling.
Read the full abstract ▾
All authors Lukas Twist, Shu Yang, Hanqi Yan, Jingzhi Gong, Di Wang, Helen Yannakoudakis, Jie M. Zhang
Machine Learning · Cross-listed from eess.SP
VSE: Variational state estimation of complex model-free process
Gustav Norén, Anubhab Ghosh, Fredrik Cumlin et al. (+1 more)
We design a variational state estimation (VSE) method that provides a closed-form Gaussian posterior of an underlying complex dynamical process from (noisy) nonlinear measurements. The complex process is model-free. That is, we do not have a suitable physics-based model characterizing the temporal
evolution of the process state. The closed-form Gaussian posterior is provided by a recurrent neural network (RNN). The use of RNN is computationally simple in the inference phase. For learning the RNN, an additional RNN is used in the learning phase. Both RNNs help each other learn better based on variational inference principles. The VSE is demonstrated for a tracking application - state estimation of a
stochastic Lorenz system (a
benchmark process) using a 2-D camera measurement model. The VSE is shown to be competitive against a particle filter that knows the Lorenz system model and a recently proposed data-driven state estimation method that does not know the Lorenz system model.
Read the full abstract ▾
All authors Gustav Norén, Anubhab Ghosh, Fredrik Cumlin, Saikat Chatterjee
Machine Learning
Managing Solution Stability in Decision-Focused Learning with Cost Regularization
Victor Spitzer and Francois Sanson
Decision-focused learning integrates predictive modeling and combinatorial optimization by training models to directly improve decision quality rather than prediction accuracy alone. Differentiating through combinatorial optimization problems represents a central challenge, and recent approaches tackle this difficulty by introducing perturbation-based approximations. In this work, we focus on estimating the objective function coefficients of a combinatorial optimization problem. Our study demonstrates that fluctuations in perturbation intensity occurring during the learning phase can lead to ineffective training, by establishing a theoretical link to the notion of solution stability in combinatorial optimization. We propose addressing this issue by introducing a regularization of the estimated cost vectors which improves the robustness and reliability of the learning process, as demonstrated by extensive numerical experiments.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.LO
How Expressive Are Graph Neural Networks in the Presence of Node Identifiers?
Arie Soeteman, Michael Benedikt, Martin Grohe et al. (+1 more)
Graph neural networks (GNNs) are a widely used class of machine learning models for graph-structured data, based on local aggregation over neighbors. GNNs have close connections to logic. In particular, their expressive power is linked to that of modal logics and bounded-variable logics with counting. In many practical scenarios, graphs processed by GNNs have node features that act as unique identifiers. In this work, we study how such identifiers affect the expressive power of GNNs. We initiate a study of the key-invariant expressive power of GNNs, inspired by the notion of order-invariant definability in finite model theory: which node queries that depend only on the underlying graph structure can GNNs express on graphs with unique node identifiers? We provide answers for various classes of GNNs with local max- or sum-aggregation.
Read the full abstract ▾
All authors Arie Soeteman, Michael Benedikt, Martin Grohe, Balder ten Cate
Machine Learning
Transfer Denoising of Low-Rank Plus Sparse Matrices under Ambient and Rank Growth
Elynn Chen, Jinhang Chai, Xuyuan Liu et al. (+1 more)
We study transfer learning for low-rank plus sparse
matrix denoising when both ambient and latent dimensions grow. A data-rich source task provides subspace information for a higher-dimensional, data-scarce target with a small rank increment and localized sparse edits. We develop an anchored alternating-projection
estimator that preserves transferred subspaces while learning target-specific coefficients, additional latent directions, and sparse modifications. We establish deterministic error bounds that separate target noise, representation growth, and source-estimation error, identifying improved rates through small structural increments and reduced noise within inherited subspaces. We further extend it to approximate anchoring quantifies robustness to subspace mismatch. We establish application-specific guarantees for Markov transition
matrix estimation from single trajectories with dependent noise and for structured covariance estimation, with the latter analysis provided in the appendix. Experiments demonstrate consistent improvements over non-transfer baselines on synthetic data and an approximately 45% reduction in principal-angle error on a NYC taxi transition-estimation task with expanding state spaces.
Read the full abstract ▾
All authors Elynn Chen, Jinhang Chai, Xuyuan Liu, Yujun Yan
Machine Learning · Cross-listed from stat.ML
On Forgetting and Stability of Score-based Generative models
Stanislas Strasman, Gabriel Cardoso, Sylvain Le Corff et al. (+2 more)
Understanding the stability and long-time behavior of generative models is a fundamental problem in modern machine learning. This paper provides quantitative bounds on the sampling error of score-based generative models by leveraging stability and forgetting properties of the Markov chain associated with the reverse-time dynamics. Under weak assumptions, we provide the two structural properties to ensure the propagation of initialization and discretization errors of the backward process: a Lyapunov drift condition and a Doeblin-type minorization condition. A practical consequence is quantitative stability of the sampling procedure, as the reverse
diffusion dynamics induces a contraction mechanism along the sampling trajectory. Our results clarify the role of
stochastic dynamics in score-based models and provide a principled framework for analyzing propagation of errors in such approaches.
Read the full abstract ▾
All authors Stanislas Strasman, Gabriel Cardoso, Sylvain Le Corff, Vincent Lemaire, Antonio Ocello
Machine Learning
MoHETS: Long-term Time Series Forecasting with Mixture-of-Heterogeneous-Experts
Evandro S. Ortigossa, Guy Lutsker and Eran Segal
Real-world multivariate time series can exhibit intricate multi-scale structures, including global trends, local periodicities, and non-stationary regimes, which makes long-horizon
forecasting challenging. Although sparse Mixture-of-Experts (MoE) approaches improve scalability and specialization, they typically rely on homogeneous MLP experts that poorly capture the diverse temporal dynamics of time series data. We address these limitations with MoHETS, an encoder-only
Transformer that integrates sparse Mixture-of-Heterogeneous-Experts (MoHE) layers. MoHE routes temporal patches to a small subset of expert networks, combining a shared depthwise-convolution expert for sequence-level continuity with routed
Fourier-based experts for patch-level periodic structures. MoHETS further improves robustness to non-stationary dynamics by incorporating exogenous information via cross-
attention over
covariate patch
embeddings. Finally, we replace parameter-heavy linear projection heads with a lightweight convolutional patch decoder, improving parameter efficiency, reducing training instability, and allowing a single model to generalize across arbitrary forecast horizons. We validate across seven multivariate
benchmarks and multiple horizons, with MoHETS consistently achieving state-of-the-art performance, reducing the average MSE by $12\%$ compared to strong recent baselines, demonstrating effective heterogeneous specialization for long-term
forecasting.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IR
LEMUR: Learned Multi-Vector Retrieval
Elias Jääsaari, Ville Hyvönen and Teemu Roos
Multi-vector representations generated by late interaction models, such as ColBERT, enable superior
retrieval quality compared to single-vector representations in information
retrieval applications. In multi-vector
retrieval systems, both queries and documents are encoded using one
embedding per
token, and similarity between queries and documents is measured by the MaxSim similarity measure. However, the improved quality of multi-vector
retrieval comes at the expense of significantly increased search latency. In this work, we introduce LEMUR, a simple yet efficient framework for multi-vector similarity search. LEMUR consists of two consecutive problem reductions: First, we formulate multi-vector similarity search as a supervised learning problem that can be solved using a one-hidden-layer neural network. Second, we reduce inference under this model to single-vector similarity search in its latent space, enabling the use of existing single-vector search indexes to accelerate
retrieval. LEMUR is an order of magnitude faster than prior multi-vector similarity search methods. Our code is available at https://github.com/ejaasaari/lemur
Read the full abstract ▾
Machine Learning
Visual Disentangled Diffusion Autoencoders: Scalable Counterfactual Generation for Foundation Models
Sidney Bender and Marco Morik
Foundation models, despite their robust zero-shot capabilities, remain vulnerable to spurious correlations and 'Clever Hans' strategies. Existing mitigation methods often rely on unavailable group labels or computationally expensive gradient-based
adversarial optimization. To address these limitations, we propose Visual Disentangled
Diffusion Autoencoders (DiDAE), a novel framework integrating frozen foundation models with disentangled dictionary learning for efficient, gradient-free counterfactual generation directly for the foundation model. DiDAE first edits foundation model
embeddings in interpretable disentangled directions of the disentangled dictionary and then decodes them via a
diffusion autoencoder. This allows the generation of multiple diverse, disentangled counterfactuals for each factual, much faster than existing baselines, which generate single entangled counterfactuals. When paired with Counterfactual Knowledge
Distillation, DiDAE-CFKD achieves state-of-the-art performance in mitigating shortcut learning, improving downstream performance on unbalanced
datasets.
Read the full abstract ▾
Machine Learning
READY: Reward Discovery for Meta-Black-Box Optimization
Zechuan Huang, Zhiguang Cao, Hongshu Guo et al. (+2 more)
Meta-Black-Box Optimization (MetaBBO) is an emerging avenue within Optimization community, where
algorithm design policy could be meta-learned by reinforcement learning to enhance optimization performance. So far, the reward functions in existing MetaBBO works are designed by human experts, introducing certain design bias and risks of reward hacking. In this paper, we use Large
Language Model~(LLM) as an automated reward discovery tool for MetaBBO. Specifically, we consider both effectiveness and efficiency sides. On effectiveness side, we borrow the idea of
evolution of heuristics, introducing tailored
evolution paradigm in the iterative LLM-based program search process, which ensures continuous improvement. On efficiency side, we additionally introduce multi-task
evolution architecture to support parallel reward discovery for diverse MetaBBO approaches. Such parallel process also benefits from knowledge sharing across tasks to accelerate convergence. Empirical results demonstrate that the reward functions discovered by our approach could be helpful for boosting existing MetaBBO works, underscoring the importance of reward design in MetaBBO. We provide READY's project at https://anonymous.4open.science/r/ICML_READY-747F.
Read the full abstract ▾
All authors Zechuan Huang, Zhiguang Cao, Hongshu Guo, Yue-Jiao Gong, Zeyuan Ma
Machine Learning
Constrained Meta Reinforcement Learning with Provable Test-Time Safety
Tingting Ni and Maryam Kamgarpour
Meta reinforcement learning (RL) allows
agents to leverage experience across a distribution of tasks on which the
agent can train at will, enabling faster learning of optimal policies on new test tasks. Despite its success in improving sample complexity on test tasks, many real-world applications, such as robotics and healthcare, impose safety constraints during testing. Constrained meta RL provides a promising framework for integrating safety into meta RL. An open question in constrained meta RL is how to ensure safety of the policy on the real-world test task, while reducing the sample complexity and thus, enabling faster learning of optimal policies. To address this gap, we propose an
algorithm that refines policies learned during training, with provable safety and sample complexity guarantees for learning a near optimal policy on the test tasks. We further derive a matching lower bound, showing that this sample complexity is tight.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CY
Test-Time Compute Games
Ander Artola Velasco, Dimitrios Rontogiannis, Stratis Tsirtsis et al. (+1 more)
Test-time compute has emerged as a promising strategy to enhance the
reasoning abilities of large
language models (LLMs). However, this strategy has in turn increased how much users pay cloud-based providers offering LLM-as-a-service, since providers charge users for the amount of test-time compute they use to generate an output. In our work, we show that the
market of LLM-as-a-service is socially inefficient: providers have a financial incentive to increase the amount of test-time compute, even if this increase contributes little to the quality of the outputs. To address this inefficiency, we introduce a reverse second-price auction mechanism where providers bid their offered price and (expected) quality for the opportunity to serve a user, and users pay proportionally to the marginal value generated by the winning provider relative to the second-highest bidder. To illustrate and complement our theoretical results, we conduct experiments with multiple instruct models from the $\texttt{Llama}$ and $\texttt{Qwen}$ families, as well as
reasoning models distilled from $\texttt{DeepSeek-R1}$, on math and science
benchmark datasets.
Read the full abstract ▾
All authors Ander Artola Velasco, Dimitrios Rontogiannis, Stratis Tsirtsis, Manuel Gomez-Rodriguez
Machine Learning
Scalable Linearized Laplace Approximation via Surrogate Neural Kernel
Luis A. Ortega, Simón Rodríguez-Santana and Daniel Hernández-Lobato
We introduce a scalable method to approximate the kernel of the Linearized Laplace Approximation (LLA). For this, we use a surrogate deep neural network (DNN) that learns a compact feature representation whose inner product replicates the Neural Tangent Kernel (NTK). This avoids the need to compute large Jacobians. Training relies solely on efficient Jacobian-vector products, allowing to compute predictive uncertainty on large-scale pre-trained DNNs. Experimental results show similar or improved uncertainty estimation and calibration compared to existing LLA approximations. Notwithstanding, biasing the learned kernel significantly enhances out-of-distribution detection. This remarks the benefits of the proposed method for finding better kernels than the NTK in the context of LLA to compute prediction uncertainty given a pre-trained DNN.
Read the full abstract ▾
Machine Learning
Goal-Driven Adaptive Sampling Strategies for Machine Learning Models Predicting Fields
Jigar Parekh and Philipp Bekemeyer
Machine learning models are widely regarded as a way forward to tackle multi-query challenges that arise once expensive black-box simulations such as computational fluid dynamics are investigated. However, ensuring the desired level of accuracy for a certain task at minimal computational cost, e.g. as few black-box samples as possible, remains a challenges. Active learning strategies are used for scalar quantities to overcome this challenges and different so-called infill criteria exists and are commonly employed in several scenarios. Even though needed in various field an extension of active learning strategies towards field predictions is still lacking or limited to very specific scenarios and/or model types. In this paper we propose an active learning strategy for machine learning models that are capable if predicting field which is agnostic to the model architecture itself. For doing so, we combine a well-established Gaussian process model for a scalar reference value and simultaneously aim at reducing the epistemic model error and the difference between scalar and field predictions. Different specific forms of the above-mentioned approach are introduced and compared to each other as well as only scalar-valued based infill. Results are presented for the NASA common research model for an uncertainty propagation task showcasing high level of accuracy at significantly smaller cost compared to an approach without active learning.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Generative Modeling of Discrete Data Using Geometric Latent Subspaces
Daniel Gonzalez-Alvarado, Jonas Cassel, Stefania Petra et al. (+1 more)
We propose a
geometric latent-subspace framework for generative modeling of discrete data. Specifically, we introduce latent subspaces in the exponential parameter space of product
manifolds of categorical distributions as a novel approach to learning low-dimensional representations of high-dimensional discrete data. The resulting low-dimensional latent space captures statistical dependencies and removes redundant degrees of freedom among the categorical variables. We equip the parameter domain with a Riemannian geometry such that the latent subspace and induced data
manifold are related isometrically, enabling consistent flow matching. Exploiting this structure, we propose a geometry-aware dimensionality reduction objective, called
geometric PCA (GPCA), which we formulate as a regularized cross-
entropy minimization that encourages small Riemannian distances between the data and their reconstructions. In particular, under the induced geometry, geodesics correspond to straight lines in the latent parameter space, allowing flow matching to be performed directly in reduced coordinates. Empirical results show that low-dimensional latent representations suffice to accurately model high-dimensional discrete data and enable substantially more computationally efficient flow matching.
Read the full abstract ▾
All authors Daniel Gonzalez-Alvarado, Jonas Cassel, Stefania Petra, Christoph Schnörr
Machine Learning
DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training
Xinwei Qiang, Hongmin Chen, Shixuan Sun et al. (+3 more)
Determinism is indispensable for reproducibility in large
language model (LLM) training, yet it often exacts a steep performance cost. In widely used
attention implementations such as FlashAttention-3, the deterministic backward pass can incur up to a 37.9% throughput reduction relative to its non-deterministic counterpart, primarily because gradient accumulation operations must be serialized to guarantee numerical consistency. This performance loss stems from suboptimal scheduling of compute and gradient-reduction phases, leading to significant hardware underutilization. To address this challenge, we formulate the backward pass of deterministic
attention as a scheduling problem on a Directed Acyclic Graph (DAG) and derive schedules that minimize the critical path length. Building on this formulation, we present DASH (Deterministic
Attention Scheduling for High-Throughput), which encapsulates two complementary scheduling strategies: (i) Descending Q-Tile Iteration, a reversed query-block traversal that shrinks pipeline stalls in causal
attention, and (ii) Shift Scheduling, a theoretically optimal schedule within our DAG model that reduces pipeline stalls for both full and causal masks. Our empirical evaluations on NVIDIA H800 GPUs demonstrate that DASH narrows the performance gap of deterministic
attention. The proposed strategies improve the throughput of the
attention backward pass by up to 1.28$\times$ compared to the baseline, significantly advancing the efficiency of reproducible LLM training. Our code is open-sourced at https://github.com/SJTU-
Liquid/deterministic-FA3.
Read the full abstract ▾
All authors Xinwei Qiang, Hongmin Chen, Shixuan Sun, Jingwen Leng, Xin Liu, Minyi Guo
Machine Learning · Cross-listed from stat.ML
A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
Mingyuan Xu, Xinzi Tan, Jiawei Wu et al. (+1 more)
Evaluating large
language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates. More data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood
estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public
benchmarks and a newly collected
dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.
Read the full abstract ▾
All authors Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou
Machine Learning
Nonparametric LLM Evaluation from Preference Data
Dennis Frauen, Athiya Deviyani, Mihaela van der Schaar et al. (+1 more)
Evaluating the performance of large
language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards. However, many existing approaches either rely on restrictive parametric assumptions or lack valid uncertainty quantification when flexible machine learning methods are used. In this paper, we propose a nonparametric statistical framework, called DMLRank, for comparing and ranking LLMs from preference data using debiased machine learning (DML). For this, we introduce generalized average ranking scores (GARS), which generalize commonly used ranking models, including the Bradley-Terry model or PageRank/ Rank centrality, with complex human responses such as ties. DMLRank comes with the following advantages: (i)~It produces statistically efficient estimates of GARS ranking scores. (ii) It naturally allows the incorporation of black-box machine learning methods for estimation. (iii) It can be combined with pre-trained LLM evaluators (e.g., using LLM-as-a-judge). (iv) It suggests optimal policies for collecting preference data under budget constraints. We demonstrate these advantages both theoretically and empirically using both synthetic and real-world preference
datasets. In summary, our framework provides practitioners with powerful, state-of-the-art methods for comparing or ranking LLMs for leaderboards.
Read the full abstract ▾
All authors Dennis Frauen, Athiya Deviyani, Mihaela van der Schaar, Stefan Feuerriegel
Machine Learning · Cross-listed from stat.ML
A Decomposable Forward Process in Diffusion Models for Time-Series Forecasting
Francisco Caldas, Sahil Kumar and Cláudia Soares
We introduce a model-agnostic forward
diffusion process for time-series
forecasting that decomposes signals into spectral components, preserving structured temporal patterns such as seasonality more effectively than standard
diffusion. Unlike prior work that modifies the network architecture or diffuses directly in the frequency domain, our proposed method alters only the
diffusion process itself, making it compatible with existing
diffusion backbones (e.g., DiffWave, TimeGrad, CSDI). By staging noise injection according to component energy, it maintains high signal-to-noise ratios for dominant frequencies throughout the
diffusion trajectory, thereby improving the recoverability of long-term patterns. This strategy enables the model to maintain the signal structure for a longer period in the forward process, leading to improved forecast quality. Across standard
forecasting benchmarks, we show that applying spectral decomposition strategies, such as the
Fourier or Wavelet transform, consistently improves upon
diffusion models using the baseline forward process, with negligible computational overhead. The code for this paper is available at https://anonymous.4open.science/r/D-FDP-4A29.
Read the full abstract ▾
Machine Learning
Effective LoRA Adapter Routing using Task Representations
Akash Dhasade, Anne-Marie Kermarrec, Igor Pavlovic et al. (+4 more)
Low-rank adaptation (LoRA) enables parameter efficient specialization of large
language models (LLMs) through modular adapters, resulting in rapidly growing public adapter pools spanning diverse tasks. Effectively using these adapters requires routing: selecting and composing the appropriate adapters for a query. We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics. Unlike existing approaches that map queries directly to adapters, LORAUTER routes queries via task
embeddings derived from small validation sets and does not require adapter training data. By operating at the task level, LORAUTER achieves efficient routing that scales with the number of tasks rather than the number of adapters. Experiments across multiple tasks show that LORAUTER consistently outperforms baseline routing approaches, matching Oracle performance (101.2%) when task-aligned adapters exist and achieving state-of-the-art results on unseen tasks (+5.2 points). We further demonstrate the robustness of LORAUTER to very large, noisy adapter pools by scaling it to over 1500 adapters.
Read the full abstract ▾
All authors Akash Dhasade, Anne-Marie Kermarrec, Igor Pavlovic, Diana Petrescu, Rafael Pires, Mathis Randl, Martijn de Vos
Machine Learning
Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Models
Yejin Kim, Dongjun Hwang, Sungmin Cha et al. (+1 more)
Large
Vision-
Language Models (LVLMs) are widely adopted for their strong
multimodal capabilities, yet they raise serious concerns such as
privacy leakage and harmful content generation. Machine unlearning has emerged as a promising solution for removing the influence of specific data from trained models. However, existing approaches largely rely on gradient-based optimization, incurring substantial computational costs for large-scale LVLMs. To address this limitation, we propose Knowledge Vector Weakening (KVW), a training-free unlearning method that directly intervenes in the full model without gradient computation. KVW identifies knowledge vectors that are activated during the model's output generation on the forget set and progressively weakens their contributions, thereby preventing the model from exploiting undesirable knowledge. Experiments on the MLLMU and CLEAR
benchmarks demonstrate that KVW achieves a stable forget-retain trade-off while significantly improving computational efficiency over gradient-based and LoRA-based unlearning methods.
Read the full abstract ▾
All authors Yejin Kim, Dongjun Hwang, Sungmin Cha, Junsuk Choe