Machine Learning
How Understanding Forecast Uncertainty Resolves the Explainability Problem in Machine Learning Models
Joseph L. Breeden
For applications of machine learning in critical decisions, explainability is a primary concern, and often a regulatory requirement. Local linear methods for generating explanations, such as LIME and SHAP, have been criticized for being unstable near decision boundaries. In this paper, we explain that such concerns reflect a misunderstanding of the problem. The forecast uncertainty is high at decision boundaries, so consequently, the explanatory instability is high. The correct approach is to change the sequence of events and questions being asked. Nonlinear models can be highly predictive in some regions while having little or no predictability in others. Therefore, the first question is whether a usable forecast exists. When there is a forecast with low enough uncertainty to be useful, an explanation can be sought via a local linear approximation. In such cases, the explanatory instability is correspondingly low. When no usable forecast exists, the decision must fall to a simpler overall model such as traditional logistic regression. Additionally, these results show that some methods that purport to be explainable everywhere, such as ReLU networks or any piecewise linear model, have only an illusory explainability, because the forecast uncertainty at the segment boundaries is too high to be useful. Explaining an unusable forecast is pointless.
Read the full abstract ▾
Machine Learning
The Illusion of Forgetting: Attack Unlearned Diffusion via Initial Latent Variable Optimization
Manyi Li, Yufan Liu, Lai Jiang et al. (+3 more)
Text-to-image
diffusion models (DMs) are frequently abused to produce harmful or copyrighted content, violating public interests. Concept erasure (unlearning) is a promising paradigm to alleviate this issue. However, there exists a peculiar forgetting illusion phenomenon with unclear cause. Based on empirical analysis, we formally explain this cause: most unlearning partially disrupt the mapping between linguistic symbols and the underlying internal knowledge, leaving the knowledge intact as dormant
memories. We further demonstrate that distributional discrepancy in the denoising process serves as a measurable indicator of how much of the mapping is retained, also reflecting unlearning strength. Inspired by this, we propose IVO (Initial Latent Variable Optimization), a novel attack framework designed to assess the robustness of current unlearning methods. IVO optimizes initial latent variables to realign the noise distribution of unlearned models with that of their vanilla counterparts, which reconstructs the fractured mappings and consequently revives dormant
memories. Extensive experiments covering 11 unlearning techniques and 3 concept scenarios show that IVO outperforms state-of-the-art baselines, exposing fundamental flaws in current unlearning mechanisms. Warning: This paper has unsafe images that may offend some readers.
Read the full abstract ▾
All authors Manyi Li, Yufan Liu, Lai Jiang, Bing Li, Yuming Li, Weiming Hu
Machine Learning
Learning Robust Reasoning through Guided Adversarial Self-Play
Shuozhe Li, Vaishnav Tadiparthi, Kwonjoon Lee et al. (+6 more)
Reinforcement learning from verifiable rewards (RLVR) produces strong
reasoning models, yet they can fail catastrophically when the conditioning context is fallible (e.g., corrupted chain-of-thought, misleading partial solutions, or mild input perturbations), since standard RLVR optimizes final-answer correctness only under clean conditioning. We introduce GASP (Guided
Adversarial Self-Play), a robustification method that explicitly trains detect-and-repair capabilities using only outcome verification. Without human labels or external teachers, GASP forms an
adversarial self-play game within a single model: a polluter learns to induce failure via locally coherent corruptions, while an
agent learns to diagnose and recover under the same corrupted conditioning. To address the scarcity of successful recoveries early in training, we propose in-distribution repair guidance, an imitation term on self-generated repairs that increases recovery probability while preserving previously acquired capabilities. Across four open-weight models (1.5B–8B), GASP transforms strong-but-brittle reasoners into robust ones that withstand misleading and perturbed context while often improving clean accuracy. Further analysis shows that
adversarial corruptions induce an effective curriculum, and in-distribution guidance enables rapid recovery learning with minimal representational drift.
Read the full abstract ▾
All authors Shuozhe Li, Vaishnav Tadiparthi, Kwonjoon Lee, Nakul Agarwal, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Lizhang Chen, Amy Zhang, Liu Leqi
Machine Learning · Cross-listed from stat.ML
Neuron Block Dynamics for XOR Classification with Zero-Margin
Guillaume Braun and Masaaki Imaizumi
The ability of neural networks to learn useful features through
stochastic gradient descent (SGD) is a cornerstone of their success. Most theoretical analyses focus on regression or on classification tasks with a positive margin, where worst-case gradient bounds suffice. In contrast, we study zero-margin nonlinear classification by analyzing the Gaussian XOR problem, where inputs are Gaussian and the XOR decision boundary determines labels. In this setting, a non-negligible fraction of data lies arbitrarily close to the boundary, breaking standard margin-based arguments. Building on Glasgow's (2024) analysis, we extend the study of training dynamics from discrete to Gaussian inputs and develop a framework for the dynamics of
neuron blocks. We show that
neurons cluster into four directions and that block-level signals evolve coherently, a phenomenon essential in the Gaussian setting where individual
neuron signals vary significantly. Leveraging this block perspective, we analyze generalization without relying on margin assumptions, adopting an average-case view that distinguishes regions of reliable prediction from regions of persistent error. Numerical experiments confirm the predicted two-phase block dynamics and demonstrate their robustness beyond the Gaussian setting.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals
Mathew Chandy, Michael Johnson, Judong Shen et al. (+4 more)
Multimodal learning combines information from multiple data modalities to improve predictive performance. However, modalities often contribute unequally and in a data dependent way, making it unclear which data modalities are genuinely informative and to what extent their contributions can be trusted. Quantifying modality level importance together with uncertainty is therefore central to interpretable and reliable
multimodal learning. We introduce conformal Shapley intervals, a framework that combines Shapley values with conformal inference to construct uncertainty-aware importance intervals for each modality. Building on these intervals, we propose a modality selection procedure with a provable optimality guarantee: conditional on the observed features, the selected subset of modalities achieves performance close to that of the optimal subset. We demonstrate the effectiveness of our approach on multiple
datasets, showing that it provides meaningful uncertainty quantification and strong predictive performance while relying on only a small number of informative modalities.
Read the full abstract ▾
All authors Mathew Chandy, Michael Johnson, Judong Shen, Devan V. Mehrotra, Hua Zhou, Jin Zhou, Xiaowu Dai
Machine Learning
The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective
Qiyao Liang, Jinyeop Song, Yizhou Liu et al. (+4 more)
Weight-perturbation
evolution strategies (ES) can fine-tune billion-parameter
language models with surprisingly small populations (e.g., $N\!\approx\!30$), contradicting classical zeroth-order curse-of-dimensionality intuition. We also observe a second seemingly separate phenomenon: under fixed hyperparameters, the
stochastic fine-tuning reward often rises, peaks, and then degrades in both ES and GRPO. We argue that both effects reflect a shared
geometric property of fine-tuning landscapes: they are low-dimensional in
curvature. A small set of high-
curvature dimensions dominates improvement, producing (i) heterogeneous time scales that yield rise-then-decay under fixed stochasticity, as captured by a minimal quadratic
stochastic-ascent model, and (ii) degenerate improving updates, where many random perturbations share similar components along these directions. Using ES as a
geometric probe on fine-tuning reward landscapes of GSM8K, ARC-C, and WinoGrande across Qwen2.5-Instruct models (0.5B–7B), we show that reward-improving perturbations remain empirically accessible with small populations across scales. Together, these results reconcile ES scalability with non-monotonic training dynamics and suggest that high-dimensional fine-tuning may admit a broader class of viable optimization methods than worst-case theory implies.
Read the full abstract ▾
All authors Qiyao Liang, Jinyeop Song, Yizhou Liu, Jeff Gore, Ila Fiete, Risto Miikkulainen, Xin Qiu
Machine Learning · Cross-listed from cs.CV
VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation
Hongyang Du, Junjie Ye, Xiaoyan Cong et al. (+7 more)
While recent
video diffusion models (VDMs) produce visually impressive results, they fundamentally struggle to maintain 3D structural consistency, often resulting in object deformation or spatial drift. We hypothesize that these failures arise because standard denoising objectives lack explicit incentives for
geometric coherence. To address this, we introduce VideoGPA (
Video Geometric Preference
Alignment), a data-efficient self-supervised framework that leverages a geometry foundation model to automatically derive dense preference signals that guide VDMs via Direct Preference Optimization (DPO). This approach effectively steers the generative distribution toward inherent 3D consistency without requiring human annotations. VideoGPA significantly enhances temporal stability,
geometric plausibility, and motion coherence using minimal preference pairs, consistently outperforming state-of-the-art baselines in extensive experiments.
Read the full abstract ▾
All authors Hongyang Du, Junjie Ye, Xiaoyan Cong, Runhao Li, Jingcheng Ni, Aman Agarwal, Zeqi Zhou, Zekun Li, Randall Balestriero, Yue Wang
Machine Learning · Cross-listed from cs.RO
End-to-end Optimization of Belief and Policy Learning in Shared Autonomy Paradigms
MH Farhadi, Ali Rabiee, Sima Ghafoori et al. (+3 more)
Shared autonomy systems require principled methods for inferring user intent and determining appropriate assistance levels. This is a central challenge in human-
robot interaction, where systems must be successful while being mindful of user agency. Previous approaches relied on static blending ratios or separated goal inference from assistance arbitration, leading to suboptimal performance in unstructured environments. We introduce BRACE (
Bayesian Reinforcement Assistance with Context Encoding), a novel framework that fine-tunes
Bayesian intent inference and context-adaptive assistance through an architecture enabling end-to-end gradient flow between intent inference and assistance arbitration. Our pipeline conditions collaborative control policies on environmental context and complete goal probability distributions. We provide analysis showing (1) optimal assistance levels should decrease with goal uncertainty and increase with environmental constraint severity, and (2) integrating belief information into policy learning yields a quadratic expected regret advantage over sequential approaches. We validated our
algorithm against SOTA methods (IDA, DQN) using a three-part evaluation progressively isolating distinct challenges of end-effector control: (1) core human-interaction dynamics in a 2D human-in-the-loop cursor task, (2) non-linear dynamics of a robotic arm, and (3) integrated manipulation under goal ambiguity and environmental constraints. We demonstrate improvements over SOTA, achieving 6.3% higher success rates and 41% increased path efficiency, and 36.3% success rate and 87% path efficiency improvement over unassisted control. Our results confirmed that integrated optimization is most beneficial in complex, goal-ambiguous scenarios, and is generalizable across robotic domains requiring goal-directed assistance, advancing the SOTA for adaptive shared autonomy.
Read the full abstract ▾
All authors MH Farhadi, Ali Rabiee, Sima Ghafoori, Anna Cetera, Andrew Fisher, Reza Abiri
Machine Learning
Decoupled Diffusion Sampling for Inverse Problems on Function Spaces
Thomas Y.L. Lin, Jiachen Yao, Lufang Chiang et al. (+2 more)
We propose a data-efficient, physics-aware generative framework in function space for inverse PDE problems. Existing plug-and-play
diffusion posterior samplers represent physics implicitly through joint coefficient-solution modeling, requiring substantial paired supervision. In contrast, our Decoupled
Diffusion Inverse Solver (DDIS) employs a decoupled design: an unconditional
diffusion learns the coefficient prior, while a neural operator explicitly models the forward PDE for guidance. This decoupling enables superior data efficiency and effective physics-informed learning, while naturally supporting Decoupled Annealing Posterior Sampling (DAPS) to avoid over-smoothing in
Diffusion Posterior Sampling (DPS). Theoretically, we prove that DDIS avoids the guidance attenuation failure of joint models when training data is scarce. Empirically, DDIS achieves state-of-the-art performance under sparse observation, improving $l_2$ error by 11% and spectral error by 54% on average; when data is limited to 1%, DDIS maintains accuracy with 40% advantage in $l_2$ error compared to joint models.
Read the full abstract ▾
All authors Thomas Y.L. Lin, Jiachen Yao, Lufang Chiang, Julius Berner, Anima Anandkumar
Machine Learning
FOCUS: DLLMs Know How to Tame Their Compute Bound
Kaihua Liang, Xin Tan, An Zhong et al. (+2 more)
Diffusion Large
Language Models (DLLMs) offer a compelling alternative to Auto-Regressive models, but their deployment is constrained by high decoding cost. In this work, we identify a key inefficiency in DLLM decoding: while computation is parallelized over
token blocks, only a small subset of
tokens is decodable at each
diffusion step, causing most compute to be wasted on non-decodable
tokens. We further observe a strong correlation between
attention-derived
token importance and
token-wise decoding probability. Based on this insight, we propose FOCUS, an inference system designed for DLLMs. By dynamically focusing computation on decodable
tokens and evicting non-decodable ones on-the-fly, FOCUS increases the effective batch size, alleviating compute limitations and enabling scalable throughput. Empirical evaluations demonstrate that FOCUS achieves up to 3.52$\times$ throughput improvement over the production-grade engine LMDeploy in large-batch settings, while preserving or improving generation quality across multiple
benchmarks.
Read the full abstract ▾
All authors Kaihua Liang, Xin Tan, An Zhong, Hong Xu, Marco Canini
Machine Learning · Cross-listed from astro-ph.IM
Denoising the Deep Sky: Physics-Based CCD Noise Formation for Astronomical Imaging
Shuhong Liu, Xining Ge, Ziying Gu et al. (+7 more)
Astronomical imaging remains noise-limited under practical observing conditions. Standard calibration pipelines remove structured artifacts but largely leave
stochastic noise unresolved. Although learning-based denoising has shown strong potential, progress is constrained by scarce paired training data and the requirement for physically interpretable models in scientific workflows. We propose a physics-based noise synthesis framework tailored to CCD noise formation in the
telescope. The pipeline models
photon shot noise, photo-response non-uniformity, dark-current noise, readout effects, and localized outliers arising from
cosmic-ray hits and hot pixels. To obtain low-noise inputs for synthesis, we stack multiple unregistered exposures to produce high-SNR bases. Realistic noisy counterparts synthesized from these bases using our noise model enable the construction of abundant paired
datasets for supervised learning. Extensive experiments on our real-world multi-band
dataset curated from two ground-based
telescopes demonstrate the effectiveness of our framework in both photometric and scientific accuracy.
Read the full abstract ▾
All authors Shuhong Liu, Xining Ge, Ziying Gu, Quanfeng Xu, Lin Gu, Ziteng Cui, Xuangeng Chu, Jun Liu, Dong Li, Tatsuya Harada
Machine Learning
Particle-Guided Diffusion Models for Partial Differential Equations
Andrew Millard, Fredrik Lindsten and Zheng Zhao
We introduce a guided
stochastic sampling method that augments sampling from
diffusion models with physics-based guidance derived from partial differential equation (PDE) residuals and observational constraints, ensuring generated samples remain physically admissible. We embed this sampling procedure within a new Sequential Monte Carlo (SMC) framework, yielding a scalable generative PDE solver. Across multiple
benchmark PDE systems as well as multiphysics and interacting PDE systems, our method produces solution fields with lower numerical error than existing state-of-the-art generative methods.
Read the full abstract ▾
Machine Learning
TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training
Ruijie Zhang, Yequan Zhao, Ziyue Liu et al. (+5 more)
The
Muon optimizer has demonstrated strong empirical performance in pre-training large
language models by performing
matrix-level gradient (or momentum) orthogonalization in each layer independently. In this work, we propose TEON, a principled generalization of
Muon that extends orthogonalization beyond individual layers by modeling the gradients of a neural network as a structured higher-order tensor. We present TEON's improved convergence guarantee over layer-wise
Muon, and further develop a practical instantiation of TEON based on the theoretical analysis with corresponding ablation. We evaluate our approach on two widely adopted architectures: GPT-style models, ranging from 130M to 774M parameters, and LLaMA-style models, ranging from 60M to 1B parameters. Experimental results show that TEON consistently improves training and validation perplexity across model scales and exhibits strong robustness under various approximate SVD schemes.
Read the full abstract ▾
All authors Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, Dongyang Li, Yupeng Su, Sijia Liu, Zheng Zhang
Machine Learning
Agnostic Language Identification and Generation
Mikael Møller Høgsgaard and Chirag Pabbaraju
Recent works on
language identification and generation have established tight statistical rates at which these tasks can be achieved. These works typically operate under a strong realizability assumption: that the input data is drawn from an unknown distribution necessarily supported on some
language in a given collection. In this work, we relax this assumption of realizability entirely, and impose no restrictions on the distribution of the input data. We propose objectives to study both
language identification and generation in this more general "agnostic" setup. Across both problems, we obtain novel interesting characterizations and nearly tight rates.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Training-Free Test-Time Adaptation with Brownian Distance Covariance in Vision-Language Models
Yi Zhang, Chun-Wun Cheng, Angelica I. Aviles-Rivero et al. (+2 more)
Vision-
language models suffer performance degradation under domain shift, limiting real-world applicability. Existing test-time adaptation methods are computationally intensive, rely on back-propagation, and often focus on single modalities. To address these issues, we propose Training-free Test-Time Adaptation with Brownian Distance Covariance (TaTa). TaTa leverages Brownian Distance Covariance-a powerful statistical measure that captures both linear and nonlinear dependencies via pairwise distances-to dynamically adapt VLMs to new domains without training or back-propagation. This not only improves efficiency but also enhances stability by avoiding disruptive weight updates. TaTa further integrates attribute-enhanced prompting to improve
vision-
language inference with descriptive visual cues. Combined with dynamic clustering and pseudo-label refinement, it effectively recalibrates the model for novel visual contexts. Experiments across diverse
datasets show that TaTa significantly reduces computational cost while achieving state-of-the-art performance in domain and cross-
dataset generalization.
Read the full abstract ▾
All authors Yi Zhang, Chun-Wun Cheng, Angelica I. Aviles-Rivero, Zhihai He, Liang-Jie Zhang
Machine Learning · Cross-listed from stat.CO
Nested Slice Sampling: Vectorized Nested Sampling for GPU-Accelerated Inference
David Yallup, Namu Kroupa and Will Handley
Model comparison and calibrated uncertainty quantification often require integrating over parameters, but scalable inference can be challenging for complex,
multimodal targets. Nested Sampling is a robust alternative to standard MCMC, yet its typically sequential structure and hard constraints make efficient accelerator implementations difficult. This paper introduces Nested Slice Sampling (NSS), a GPU-friendly, vectorized formulation of Nested Sampling that uses Hit-and-Run Slice Sampling for constrained updates. A tuning analysis yields a simple near-optimal rule for setting the slice width, improving high-dimensional behavior and making per-step compute more predictable for parallel execution. Experiments on challenging synthetic targets, high dimensional
Bayesian inference, and Gaussian process hyperparameter marginalization show that NSS maintains accurate evidence estimates and high-quality posterior samples, and is particularly robust on difficult
multimodal problems where current state-of-the-art methods such as tempered SMC baselines can struggle. An open-source implementation is released to facilitate adoption and reproducibility.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Graph Attention Network for Node Regression on Random Geometric Graphs with Erdős–Rényi contamination
Somak Laha, Suqi Liu and Morgane Austern
Graph
attention networks (GATs) are widely used and often appear robust to noise in node
covariates and edges, yet rigorous statistical guarantees demonstrating a provable advantage of GATs over non-
attention graph neural networks~(GNNs) are scarce. We partially address this gap for node regression with graph-based errors-in-variables models under simultaneous
covariate and edge corruption: responses are generated from latent node-level
covariates, but only noise-perturbed versions of the latent
covariates are observed; and the sample graph is a random
geometric graph created from the node
covariates but contaminated by independent Erdős–Rényi edges. We propose and analyze a carefully designed, task-specific GAT that constructs denoised proxy features for regression. We prove that regressing the response variables on the proxies achieves lower error asymptotically in (a) estimating the regression coefficient compared to the ordinary least squares (OLS)
estimator on the noisy node
covariates, and (b) predicting the response for an unlabelled node compared to a vanilla graph convolutional network~(GCN) – under mild growth conditions. Our analysis leverages high-dimensional
geometric tail bounds and concentration for neighbourhood counts and sample covariances. We verify our theoretical findings through experiments on synthetically generated data. We also perform experiments on real-world graphs and demonstrate the effectiveness of the
attention mechanism in several node regression tasks.
Read the full abstract ▾
Machine Learning
How well do generative models solve inverse problems? A benchmark study
Patrick Krüger, Patrick Materne, Werner Krebs et al. (+1 more)
Generative learning generates high dimensional data based on low dimensional conditions, also called
prompts. Therefore, generative learning
algorithms are eligible for solving (
Bayesian) inverse problems. In this article we compare a traditional
Bayesian inverse approach based on a forward regression model and a prior sampled with the Markov Chain Monte Carlo method with three state of the art generative learning models, namely conditional Generative
Adversarial Networks, Invertible Neural Networks and Conditional Flow Matching. We apply them to a problem of gas turbine combustor design where we map six independent design parameters to three performance labels. We propose several metrics for the evaluation of this inverse design approaches and measure the accuracy of the labels of the generated designs along with the diversity. We also study the performance as a function of the training
dataset size. Our
benchmark has a clear winner, as Conditional Flow Matching consistently outperforms all competing approaches.
Read the full abstract ▾
All authors Patrick Krüger, Patrick Materne, Werner Krebs, Hanno Gottschalk
Machine Learning
YuriiFormer: A Suite of Nesterov-Accelerated Transformers
Aleksandr Zimin, Yury Polyanskiy and Philippe Rigollet
We propose a variational framework that interprets
transformer layers as iterations of an optimization
algorithm acting on
token embeddings. In this view, self-
attention implements a gradient step of an interaction energy, while MLP layers correspond to gradient updates of a potential energy. Standard GPT-style
transformers emerge as vanilla gradient descent on the resulting composite objective, implemented via Lie–Trotter splitting between these two energy functionals. This perspective enables principled architectural design using classical optimization ideas. As a proof of concept, we introduce a Nesterov-style accelerated
transformer that preserves the same
attention and MLP oracles. The resulting architecture consistently outperforms a nanoGPT baseline on TinyStories and OpenWebText, demonstrating that optimization-theoretic insights can translate into practical gains.
Read the full abstract ▾
Machine Learning
Sequence Diffusion Model for Temporal Link Prediction in Continuous-Time Dynamic Graph
Nguyen Minh Duc and Viet Cuong Ta
Temporal link prediction in dynamic graphs is a fundamental problem in many real-world systems. Existing temporal graph neural networks mainly focus on learning representations of historical interactions. Despite their strong performance, these models are still purely discriminative, producing point estimates for future links and lacking an explicit mechanism to capture the uncertainty and sequential structure of future temporal interactions. In this paper, we propose SDG, a novel sequence-level
diffusion framework that unifies dynamic graph learning with generative denoising. Specifically, SDG injects noise into the entire historical interaction sequence and jointly reconstructs all interaction
embeddings through a conditional denoising process, thereby enabling the model to capture more comprehensive interaction distributions. To align the generative process with temporal link prediction, we employ a cross-
attention denoising decoder to guide the reconstruction of the destination sequence and optimize the model in an end-to-end manner. Extensive experiments on various temporal graph
benchmarks show that SDG consistently achieves state-of-the-art performance in the temporal link prediction task.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.IV
Solving Inverse Problems with Flow-based Models via Model Predictive Control
George Webber, Alexander Denker, Riccardo Barbano et al. (+1 more)
Flow-based generative models provide strong unconditional priors for inverse problems, but guiding their dynamics for conditional generation remains challenging. Recent work casts training-free conditional generation in flow models as an optimal control problem; however, solving the resulting trajectory optimisation is computationally and
memory intensive, requiring differentiation through the flow dynamics or adjoint solves. We propose MPC-Flow, a model predictive control framework that formulates inverse problem solving with flow-based generative models as a sequence of control sub-problems, enabling practical optimal control-based guidance at inference time. We provide theoretical analysis linking MPC-Flow to the underlying optimal control objective and show how different algorithmic choices yield a spectrum of guidance
algorithms, including regimes that avoid backpropagation through the generative model trajectory. We evaluate MPC-Flow on
benchmark image restoration tasks, spanning linear and non-linear settings such as in-painting, deblurring, and super-resolution, and demonstrate strong performance and scalability to massive state-of-the-art architectures via training-free guidance of FLUX.2 (32B) in a quantised setting on consumer hardware.
Read the full abstract ▾
All authors George Webber, Alexander Denker, Riccardo Barbano, Andrew J Reader
Machine Learning
Agile Reinforcement Learning through Separable Neural Architecture and Applications
Rajib Mostakim, Reza T. Batley and Sourav Saha
Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, yet go-to function approximators - multilayer perceptrons (MLPs) - are often parameter-inefficient due to an imperfect inductive bias for the smooth structure of many value functions. This mismatch can also hinder sample efficiency and slow policy learning in this capacity-limited regime. Although model compression techniques exist, they operate post-hoc and do not improve learning efficiency. Spline-based architectures such as Kolmogorov-Arnold Networks (KANs) have been shown to offer parameter efficiency but are widely reported to exhibit significant computational overhead, especially at scale. In seeking to address these limitations, this work introduces SPAN (SPline-based Adaptive Networks) for RL. SPAN adapts the KHRONOS framework with a learnable preprocessing layer. SPAN is evaluated across discrete (PPO) and high-dimensional continuous (SAC) control tasks, offline settings (Minari/D4RL) and a real-world datacenter HVAC control application. SPAN achieves a 30-50% improvement in sample efficiency and 1.3-9 times higher success rates across
benchmarks compared to MLP baselines. Despite incurring a per-step evaluation overhead of 1.2-1.8x, SPAN's superior convergence reliability yields an expected total training cost 1.3-6.3x lower than MLP baselines when accounting for convergence failures. In the HVAC application, SPAN reduces energy consumption in 9 of 12 months relative to MLP while simultaneously achieving a 1.1-3.4x reduction in thermal comfort violations across the evaluation year, demonstrating generalization to real-world engineering control. Furthermore, SPAN demonstrates superior anytime performance and robustness to hyperparameter variations, suggesting it as a viable, high-performance alternative for learning efficient policies in resource-limited settings.
Read the full abstract ▾
Machine Learning
Optimal Fair Aggregation of Crowdsourced Noisy Labels using Demographic Parity Constraints
Gabriel Singer, Samuel Gruffaz, Olivier Vo Van et al. (+2 more)
As acquiring reliable ground-truth labels is usually costly, or infeasible, crowdsourcing and aggregation of noisy human annotations is the typical resort. Aggregating subjective labels, though, may amplify individual biases, particularly regarding sensitive features, raising
fairness concerns. Nonetheless,
fairness in crowdsourced aggregation remains largely unexplored, with no existing convergence guarantees and only limited post-processing approaches for enforcing $\varepsilon$-
fairness under demographic parity. We address this gap by analyzing the
fairness s of crowdsourced aggregation methods within the $\varepsilon$-
fairness framework, for Majority Vote and Optimal
Bayesian aggregation. In the small-crowd regime, we derive an upper bound on the
fairness gap of Majority Vote in terms of the
fairness gaps of the individual annotators. We further show that the
fairness gap of the aggregated consensus converges exponentially fast to that of the ground-truth under interpretable conditions. Since ground-truth itself may still be unfair, we generalize a state-of-the-art multiclass
fairness post-processing
algorithm from the continuous to the discrete setting, which enforces strict demographic parity constraints to any aggregation rule. Experiments on synthetic and real
datasets demonstrate the effectiveness of our approach and corroborate the theoretical insights.
Read the full abstract ▾
All authors Gabriel Singer, Samuel Gruffaz, Olivier Vo Van, Nicolas Vayatis, Argyris Kalogeratos
Machine Learning
Tackling air quality with SAPIENS
Marcella Bona, Nathan Heatley, Jia-Chen Hua et al. (+7 more)
Air pollution is a chronic problem in large cities worldwide and awareness is rising as the long-term health implications become clearer. Vehicular traffic has been identified as a major contributor to poor air quality. In a lot of cities the publicly available air quality measurements and forecasts are coarse-grained both in space and time. However, in general, real-time traffic intensity data is openly available in various forms and is fine-grained. In this paper, we present an in-depth study of pollution sensor measurements combined with traffic data from Mexico City. We analyse and model the relationship between traffic intensity and air quality with the aim to provide hyper-local, dynamic air quality forecasts. We developed an innovative method to represent traffic intensities by transforming simple colour-coded traffic maps into concentric ring-based descriptions, enabling improved characterisation of traffic conditions. Using Partial Least Squares Regression, we predict pollution levels based on these newly defined traffic intensities. The model was optimised with various training samples to achieve the best predictive performance and gain insights into the relationship between pollutants and traffic. The workflow we have designed is straightforward and adaptable to other contexts, like other cities beyond the specifics of our
dataset.
Read the full abstract ▾
All authors Marcella Bona, Nathan Heatley, Jia-Chen Hua, Adriana Lara, Valeria Legaria-Santiago, Alberto Luviano Juarez, Fernando Moreno-Gomez, Jocelyn Richardson, Natan Vilchis, Xiwen Shirley Zheng
Machine Learning · Cross-listed from stat.ML
A Random Matrix Theory of Masked Self-Supervised Regression
Arie Wortsman Zurich, Federica Gerace, Bruno Loureiro et al. (+1 more)
In the era of
transformer models, masked self-supervised learning (SSL) has become a foundational training paradigm. A defining feature of masked SSL is that training aggregates predictions across many masking patterns, giving rise to a joint,
matrix-valued predictor rather than a single vector-valued
estimator. This object encodes how coordinates condition on one another and poses new analytical challenges. We develop a precise high-dimensional analysis of masked modeling objectives in the proportional regime where the number of samples scales with the ambient dimension. Our results provide explicit expressions for the generalization error and characterize the spectral structure of the learned predictor, revealing how masked modeling extracts structure from data. For spiked covariance models, we show that the joint predictor undergoes a Baik–Ben Arous–Péché (BBP)-type phase transition, identifying when masked SSL begins to recover latent signals. Finally, we identify structured regimes in which masked self-supervised learning provably outperforms PCA, highlighting potential advantages of SSL objectives over classical unsupervised methods
Read the full abstract ▾
All authors Arie Wortsman Zurich, Federica Gerace, Bruno Loureiro, Yue M. Lu
Machine Learning
Learning to Execute Graph Algorithms Exactly with Graph Neural Networks
Muhammad Fetrat Qharabagh, Artur Back de Luca, George Giapitzakis et al. (+1 more)
Understanding what graph neural networks can learn, especially their ability to learn to execute
algorithms, remains a central theoretical challenge. In this work, we prove exact learnability results for graph
algorithms under bounded-degree and finite-precision constraints. Our approach follows a two-step process. First, we train an ensemble of multi-layer perceptrons (MLPs) to execute the local instructions of a single node. Second, during inference, we use the trained MLP ensemble as the update function within a graph neural network (GNN). Leveraging Neural Tangent Kernel (NTK) theory, we show that local instructions can be learned from a small training set, enabling the complete graph
algorithm to be executed during inference without error and with high probability. To illustrate the learning power of our setting, we establish a rigorous learnability result for the LOCAL model of distributed computation. We further demonstrate positive learnability results for widely studied
algorithms such as message flooding, breadth-first and depth-first search, and Bellman-Ford.
Read the full abstract ▾
All authors Muhammad Fetrat Qharabagh, Artur Back de Luca, George Giapitzakis, Kimon Fountoulakis
Machine Learning · Cross-listed from eess.IV
Scale-Cascaded Diffusion Models for Super-Resolution in Medical Imaging
Darshan Thaker, Mahmoud Mostapha, Radu Miron et al. (+2 more)
Diffusion models have been increasingly used as strong generative priors for solving inverse problems such as super-resolution in medical imaging. However, these approaches typically utilize a
diffusion prior trained at a single scale, ignoring the hierarchical scale structure of image data. In this work, we propose to decompose images into
Laplacian pyramid scales and train separate
diffusion priors for each frequency band. We then develop an
algorithm to perform super-resolution that utilizes these priors to progressively refine reconstructions across different scales. Evaluated on
brain, knee, and prostate MRI data, our approach both improves perceptual quality over baselines and reduces inference time through smaller coarse-scale networks. Our framework unifies multiscale reconstruction and
diffusion priors for medical image super-resolution.
Read the full abstract ▾
All authors Darshan Thaker, Mahmoud Mostapha, Radu Miron, Shihan Qiu, Mariappan Nadar
Machine Learning
Ensuring Semantics in Weights of Implicit Neural Representations through the Implicit Function Theorem
Tianming Qiu, Christos Sonis and Hao Shen
Weight Space Learning (WSL), which frames neural network weights as a data modality, is an emerging field with potential for tasks like meta-learning or transfer learning. Particularly, Implicit Neural Representations (INRs) provide a convenient testbed, where each set of weights determines the corresponding individual data sample as a mapping from coordinates to contextual values. So far, a precise theoretical explanation for the mechanism of encoding semantics of data into network weights is still missing. In this work, we deploy the Implicit Function
Theorem (IFT) to establish a rigorous mapping between the data space and its latent weight representation space. We analyze a framework that maps instance-specific
embeddings to INR weights via a shared hypernetwork, achieving performance competitive with existing baselines on downstream classification tasks across 2D and 3D
datasets. These findings offer a theoretical lens for future investigations into network weights.
Read the full abstract ▾
Machine Learning
TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification
Haoyun Jiang, Junqi He, Feng Hong et al. (+9 more)
Inference efficiency in Large
Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as
reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain
tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35% speedup over standard SD, with up to 50% fewer target model invocations while maintaining comparable accuracy.
Read the full abstract ▾
All authors Haoyun Jiang, Junqi He, Feng Hong, Xinlong Yang, Jianwei Zhang, Zhengyang Zhuge, Zheng Li, Xiaofeng Cao, Zhiyong Chen, Bo Han, Junyang Lin, Jiangchao Yao
Machine Learning
MeshGraphNet-Transformer: Scalable Mesh-based Learned Simulation for Solid Mechanics
Mikel M. Iparraguirre, Iciar Alfaro, David Gonzalez et al. (+1 more)
We present MeshGraphNet-
Transformer (MGN-T), a novel architecture that combines the global modeling capabilities of
Transformers with the
geometric inductive bias of MeshGraphNets, while preserving a mesh-based graph representation. MGN-T overcomes a key limitation of standard MGN, the inefficient long-range information propagation caused by iterative message passing on large, high-resolution meshes. A physics-
attention Transformer serves as a global processor, updating all nodal states simultaneously while explicitly retaining node and edge attributes. By directly capturing long-range physical interactions, MGN-T eliminates the need for deep message-passing stacks or hierarchical, coarsened meshes, enabling efficient learning on high-resolution meshes with varying geometries,
topologies, and boundary conditions at an industrial scale. We demonstrate that MGN-T successfully handles industrial-scale meshes for impact dynamics, a setting in which standard MGN fails due message-passing under-reaching. The method accurately models self-contact, plasticity, and multivariate outputs, including internal, phenomenological plastic variables. Moreover, MGN-T outperforms state-of-the-art approaches on classical
benchmarks, achieving higher accuracy while maintaining practical efficiency, using only a fraction of the parameters required by competing baselines.
Read the full abstract ▾
All authors Mikel M. Iparraguirre, Iciar Alfaro, David Gonzalez, Elias Cueto