Peer Revenue Past citations are no guarantee of future returns

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Wednesday 11 February

Machine Learning (Statistics) · Cross-listed from cs.LG

Bootstrapping-based Regularisation for Reducing Individual Prediction Instability in Clinical Risk Prediction Models

Clinical prediction models are increasingly used to support patient care, yet many deep learning-based approaches remain unstable, as their predictions can vary substantially when trained on different samples from the same population. Such instability undermines reliability and limits clinical adoption. In this study, we propose a novel bootstrapping-based regularisation framework that embeds the bootstrapping process directly into the training of deep neural networks. This approach constrains prediction variability across resampled datasets, producing a single model with inherent stability properties. We evaluated models constructed using the proposed regularisation approach against conventional and ensemble models using simulated data and three clinical datasets: GUSTO-I, Framingham, and SUPPORT. Across all datasets, our model exhibited improved prediction stability, with lower mean absolute differences (e.g., 0.019 vs. 0.059 in GUSTO-I; 0.057 vs. 0.088 in Framingham) and markedly fewer significantly deviating predictions. Importantly, discriminative performance and feature importance consistency were maintained, with high SHAP correlations between models (e.g., 0.894 for GUSTO-I; 0.965 for Framingham). While ensemble models achieved greater stability, we show that this came at the expense of interpretability, as each constituent model used predictors in different ways. By regularising predictions to align with bootstrapped distributions, our approach allows prediction models to be developed that achieve greater robustness and reproducibility without sacrificing interpretability. This method provides a practical route toward more reliable and clinically trustworthy deep learning models, particularly valuable in data-limited healthcare settings.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from econ.EM

Cross-Fitting-Free Debiased Machine Learning with Multiway Dependence

This paper develops an asymptotic theory for two-step debiased machine learning (DML) estimators in generalised method of moments (GMM) models with general multiway clustered dependence, without relying on cross-fitting. While cross-fitting is commonly employed, it can be statistically inefficient and computationally burdensome when first-stage learners are complex and the effective sample size is governed by the number of independent clusters. We show that valid inference can be achieved without sample splitting by combining orthogonal moments with a localisation-based empirical process approach, allowing for an arbitrary number of clustering dimensions. The resulting debiased GMM estimators are shown to be asymptotically linear and asymptotically normal under multiway clustered dependence. A central technical contribution of the paper is the derivation of novel global and local maximal inequalities for empirical processes indexed by general classes of functions for separately exchangeable arrays, which underpin our theoretical arguments and are of independent interest. Python and R packages are available for implementing the proposed procedures.
Read the full abstract

Machine Learning (Statistics)

Amortised and provably-robust simulation-based inference

Complex simulator-based models are now routinely used to perform inference across the sciences and engineering, but existing inference methods are often unable to account for outliers and other extreme values in data which occur due to faulty measurement instruments or human error. In this paper, we introduce a novel approach to simulation-based inference grounded in generalised Bayesian inference and a neural approximation of a weighted score-matching loss. This leads to a method that is both amortised and provably robust to outliers, a combination not achieved by existing approaches. Furthermore, through a carefully chosen conditional density model, we demonstrate that inference can be further simplified and performed without the need for Markov chain Monte Carlo sampling, thereby offering significant computational advantages, with complexity that is only a small fraction of that of current state-of-the-art approaches.
Read the full abstract

All authors Ayush Bharti, Charita Dellaporta, Yuga Hikida, François-Xavier Briol

Statistics Theory

Entropic vector quantile regression: Duality and Gaussian case

Vector quantile regression (VQR) is an optimal transport (OT) problem subject to a mean-independence constraint that extends classical linear quantile regression to vector response variables. Motivated by computational considerations, prior work has considered entropic relaxation of VQR, but its fundamental structural and approximation properties are still much less understood than entropic OT. The goal of this paper is to address some of these gaps. First, we study duality theory for entropic VQR and establish strong duality and dual attainment for marginals with possibly unbounded supports. In addition, when all marginals are compactly supported, we show that dual potentials are real analytic. Second, building on our duality theory, when all marginals are Gaussian, we show that entropic VQR has a closed-form optimal solution, which is again Gaussian, and establish the precise approximation rate toward unregularized VQR.
Read the full abstract

Statistics Theory

A New Look at Bayesian Testing

We identify the critical deviation scale governing Bayesian evidence accumulation in regular parametric testing. Under integrated Bayes risk with zero-one loss, the risk-optimal rejection boundary lies in a moderate deviation regime, with a square-root logarithmic inflation relative to the usual local asymptotic normal scale. Under Cramer regularity, local prior smoothness at the null, and symmetric loss, we derive the sharp threshold and show that its leading logarithmic term is universal across regular priors, while lower-order constants depend on the local prior density, Fisher information, and prior model odds. The result extends to one-parameter exponential families through local asymptotic normality and places Jeffreys' testing threshold, the Bayesian information criterion penalty, and Chernoff-Stein type error-exponent arguments within a common asymptotic moderate deviation framework.
Read the full abstract

All authors Jyotishka Datta, Nicholas G. Polson, Vadim Sokolov, Daniel Zantedeschi

Statistics Theory · Cross-listed from physics.soc-ph

Formalization of the generalized Pareto principle and structural typicality of the 20/80-rule

We formalize a generalized form of the Pareto principle - “fraction $p$ of inputs yields fraction $1-p$ of outputs” - as a property of non-negative gain densities $\ell \in L^1([0,1])$, working with the decreasing rearrangement to obtain a unique characterization. For probability distributions, the resulting $p$ coincides with $1 - k_F$, where $k_F$ is the Kolkata index of the corresponding Lorenz curve. Within this framework we analyze both constructed gain densities and commonly encountered distribution families. We derive closed-form expressions for $p$ for truncated power-law, exponential, and normal distribution families. Combining these with estimates of the truncation parameter as a function of sample size $N$, we predict that datasets of size $N \in [10^2, 10^5]$ from exponential and normal families concentrate $p$ near $[0.15, 0.26]$ and $[0.20, 0.29]$ - values close to the canonical 0.2/0.8-rule, and strictly below the saturation $k \approx 0.865$ conjectured earlier by Ghosh and Chakrabarti. We discuss the implications of the structural ubiquity of Pareto-type imbalances for their use as prescriptive targets.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

Information-Theoretic Thresholds for Bipartite Latent-Space Graphs under Noisy Observations

We study information-theoretic phase transitions for the detectability of latent geometry in bipartite random geometric graphs RGGs with Gaussian d-dimensional latent vectors while only a subset of edges carries latent information determined by a random mask with i.i.d. Bern(q) entries. For any fixed edge density p in (0,1) we determine essentially tight thresholds for this problem as a function of d and q. Our results show that the detection problem is substantially easier if the mask is known upfront compared to the case where the mask is hidden. Our analysis is built upon a novel Fourier-analytic framework for bounding signed subgraph counts in Gaussian random geometric graphs that exploits cancellations which arise after approximating characteristic functions by an appropriate power series. The resulting bounds are applicable to much larger subgraphs than considered in previous work which enables tight information-theoretic bounds, while the bounds considered in previous works only lead to lower bounds from the lens of low-degree polynomials. As a consequence we identify the optimal information-theoretic thresholds and rule out computational-statistical gaps. Our bounds further improve upon the bounds on Fourier coefficients of random geometric graphs recently given by Bangachev and Bresler [STOC'24] in the dense, bipartite case. The techniques also extend to sparser and non-bipartite settings, at least if the considered subgraphs are sufficiently small. We furhter believe that they might help resolve open questions for related detection problems.
Read the full abstract

Statistical Methodology

A Doubly Robust Machine Learning Approach for Disentangling Treatment Effect Heterogeneity with Functional Outcomes

Causal inference is paramount for understanding the effects of interventions, yet extracting personalized insights from increasingly complex data remains a significant challenge for modern machine learning. This is the case, in particular, when considering functional outcomes observed over a continuous domain (e.g., time, or space). Estimation of heterogeneous treatment effects, known as CATE, has emerged as a crucial tool for personalized decision-making, but existing meta-learning frameworks are largely limited to scalar outcomes, failing to provide satisfying results in scientific applications that leverage the rich, continuous information encoded in functional data. Here, we introduce FOCaL (Functional Outcome Causal Learning), a novel, doubly robust meta-learner specifically engineered to estimate a functional heterogeneous treatment effect (F-CATE). FOCaL integrates advanced functional regression techniques for both outcome modeling and functional pseudo-outcome reconstruction, thereby enabling the direct and robust estimation of F-CATE. We provide a rigorous theoretical derivation of FOCaL, demonstrate its performance and robustness compared to existing non-robust functional methods through comprehensive simulation studies, and illustrate its practical utility on diverse real-world functional datasets. FOCaL advances the capabilities of machine intelligence to infer nuanced, individualized causal effects from complex data, paving the way for more precise and trustworthy AI systems in personalized medicine, adaptive policy design, and fundamental scientific discovery.
Read the full abstract

Statistical Computation

Large Scale High-Dimensional Reduced-Rank Linear Discriminant Analysis

Reduced-rank linear discriminant analysis (RRLDA) is a foundational method of dimension reduction for classification that has been useful in a wide range of applications. The goal is to identify an optimal subspace to project the observations onto that simultaneously maximizes between-group variation while minimizing within-group differences. The solution is straight forward when the number of observations is greater than the number of features but computational difficulties arise in both the high-dimensional setting, where there are more features than there are observations, and when the data are very large. Many works have proposed solutions for the high-dimensional setting and frequently involve additional assumptions or tuning parameters. We propose a fast and simple iterative algorithm for both classical and high-dimensional RRLDA on large data that is free from these additional requirements and that comes with guarantees. We also explain how RRLDA-RK provides implicit regularization towards the least norm solution without explicitly incorporating penalties. We demonstrate our algorithm on real data and highlight some results.
Read the full abstract

Statistical Methodology

Renet: Principled and Efficient Relaxation for the Elastic Net via Dynamic Objective Selection

We introduce Renet, a principled generalization of the Relaxed Lasso to the Elastic Net family of estimators. While, on the one hand, $\ell_1$-regularization is a standard tool for variable selection in high-dimensional regimes and, on the other hand, the $\ell_2$ penalty provides stability and solution uniqueness through strict convexity, the standard Elastic Net nevertheless suffers from shrinkage bias that frequently yields suboptimal prediction accuracy. We propose to address this limitation through a framework called relaxation. Existing relaxation implementations rely on naive linear interpolations of penalized and unpenalized solutions, which ignore the non-linear geometry that characterizes the entire regularization path and risk violating the Karush-Kuhn-Tucker conditions. Renet addresses these limitations by enforcing sign consistency through an adaptive relaxation procedure that dynamically dispatches between convex blending and efficient sub-path refitting. Furthermore, we identify and formalize a unique synergy between relaxation and the “One-Standard-Error” rule: relaxation serves as a robust debiasing mechanism, allowing practitioners to leverage the parsimony of the 1-SE rule without the traditional loss in predictive fidelity. Our theoretical framework incorporates automated stability safeguards for ultra-high dimensional regimes and is supported by a comprehensive benchmarking suite across 20 synthetic and real-world datasets, demonstrating that Renet consistently outperforms the standard Elastic Net and provides a more robust alternative to the Adaptive Elastic Net in high-dimensional, low signal-to-noise ratio and high-multicollinearity regimes. By leveraging an adaptive solver backend, Renet delivers these statistical gains while offering a computational profile that remains competitive with state-of-the-art coordinate descent implementations.
Read the full abstract

Statistical Computation · Cross-listed from cs.LG

Direct Learning of Calibration-Aware Uncertainty for Neural PDE Surrogates

Neural PDE surrogates are often deployed in data-limited or partially observed regimes where downstream decisions depend on calibrated uncertainty in addition to low prediction error. Existing approaches obtain uncertainty through ensemble replication, fixed stochastic noise such as dropout, or post hoc calibration. Cross-regularized uncertainty learns uncertainty parameters during training using gradients routed through a held-out regularization split. The predictor is optimized on the training split for fit, while low-dimensional uncertainty controls are optimized on the regularization split to reduce train-test mismatch, yielding regime-adaptive uncertainty without per-regime noise tuning. The framework can learn continuous noise levels at the output head, within hidden features, or within operator-specific components such as spectral modes. We instantiate the approach in Fourier Neural Operators and evaluate on APEBench sweeps over observed fraction and training-set size. Across these sweeps, the learned predictive distributions are better calibrated on held-out splits and the resulting uncertainty fields concentrate in high-error regions in one-step spatial diagnostics.
Read the full abstract

Statistical Methodology

Constrained Fiducial Inference for Gaussian Models

We propose a new fiducial Markov Chain Monte Carlo (MCMC) method for fitting parametric Gaussian models. We utilize the Cayley transform to decompose the parametric covariance matrix, which in turn allows us to formulate a general data generating algorithm for Gaussian data. Leveraging constrained generalized fiducial inference, we are able to create the basis of an MCMC algorithm, which can be specified to parametric models with minimal effort. The appeal of this novel approach is the wide class of models which it permits, ease of implementation and the posterior-like fiducial distribution without the need for a prior. We provide background information for the derivation of the relevant fiducial quantities, and a proof that the proposed MCMC algorithm targets the correct fiducial distribution. We need not assume independence nor identical distribution of the data, which makes the method attractive for application to time series and spatial data. Well-performing simulation results of the MA(1) and Matérn models are presented.
Read the full abstract

Machine Learning (Statistics)

A Gibbs posterior sampler for inverse problem based on prior diffusion model

This paper addresses the issue of inversion in cases where (1) the observation system is modeled by a linear transformation and additive error, (2) the problem is ill-posed and regularization relies on a Bayesian strategy, (3)~the prior is modeled by a diffusion process adjusted on an available large set of examples. In this context, it is known that the issue of posterior sampling is a thorny one and the paper introduces a Gibbs algorithm. It appears that this avenue has not been explored, and we show that it is particularly effective and remarkably simple. In addition, it provides clear elements regarding convergence guarantees in a specific case and arguments supporting such guarantees in practical cases. The results are clearly confirmed by numerical simulations based on a toy example.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories

This work addresses the problem of offline safe imitation learning (IL), where the goal is to learn safe and reward-maximizing policies from demonstrations that do not have per-timestep safety cost or reward information. In many real-world domains, online learning in the environment can be risky, and specifying accurate safety costs can be difficult. However, it is often feasible to collect trajectories that reflect undesirable or unsafe behavior, implicitly conveying what the agent should avoid. We refer to these as non-preferred trajectories. We propose a novel offline safe IL algorithm, OSIL, that infers safety from non-preferred demonstrations. We formulate safe policy learning as a Constrained Markov Decision Process (CMDP). Instead of relying on explicit safety cost and reward annotations, OSIL reformulates the CMDP problem by deriving a lower bound on reward maximizing objective and learning a cost model that estimates the likelihood of non-preferred behavior. Our approach allows agents to learn safe and reward-maximizing behavior entirely from offline demonstrations. We empirically demonstrate that our approach can learn safer policies that satisfy cost constraints without degrading the reward performance, thus outperforming several baselines.
Read the full abstract

Statistics Theory

Variational Optimality of Föllmer Processes in Generative Diffusions

We construct and analyze generative diffusions that transport a point mass to a prescribed target distribution over a finite time horizon using the stochastic interpolant framework. The drift is expressed as a conditional expectation that can be estimated from independent samples without simulating stochastic processes. We show that the diffusion coefficient can be tuned a~posteriori without changing the time-marginal distributions. Among all such tunings, we prove that minimizing the impact of estimation error on the path-space Kullback–Leibler divergence selects, in closed form, a Föllmer process – a diffusion whose path measure minimizes relative entropy with respect to a reference process determined by the interpolation schedules alone. This yields a new variational characterization of Föllmer processes, complementing classical formulations via Schrödinger bridges and stochastic control, and provides a conditional-expectation representation of the Föllmer drift that enables simulation-free estimation from data. We further establish that, under this optimal diffusion coefficient, the path-space Kullback–Leibler divergence becomes independent of the interpolation schedule, rendering different schedules statistically equivalent in this variational sense. We provide numerical experiments to illustrate the impact of path-space variational optimality of Föllmer's processes in probabilistic forecasting and data assimilation applications.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

A Jointly Efficient and Near-Optimal Algorithm for Heteroskedastic Generalized Linear Bandits with Adversarial Corruptions

We consider heteroskedastic generalized linear bandits (GLBs) with adversarial corruptions and propose HCW-GLB-OMD, a simple algorithm that combines an online mirror descent (OMD)-based estimator with Hessian-based confidence weights. Although both components are well studied, establishing guarantees for the combined algorithm requires a nontrivial analysis. We establish a corruption-robust confidence sequence using technical tools not required in prior analyses, including a weighted likelihood supermartingale and a self-bounding argument. For the regret analysis, we derive a new Hessian potential lemma to handle the interaction between adversarial corruption and adaptive confidence weighting. For self-concordant GLMs, we prove a high-probability regret bound of $\widetilde{\mathcal{O}}\bigl( d \sqrt{\sum_t g(\tau_t) \dot\mu_{t,\star}} + d^2 g_{\max} \kappa + d(1+\kappa)(C\vee1) \bigr)$, where $\dot\mu_{t,\star}$ is the slope of $\mu$ at the optimal arm at time $t$, $g(\tau_t)$ is the observed dispersion, $g_{\max}=\max_t g(\tau_t)$, $\kappa$ is the worst-case inverse slope of $\mu$, and $C$ is the corruption budget. The algorithm attains this bound with one OMD update per round and $\mathcal{O}(d^2)$ storage. We complement this guarantee with matching lower bounds for the leading $\sqrt{T}$ term and, up to $\kappa$, the $dC$ corruption term.
Read the full abstract

Statistical Methodology

Weighting-Based Identification and Estimation in Graphical Models of Missing Data

We propose a constructive algorithm for identifying complete data distributions in graphical models of missing data. The complete data distribution is unrestricted, while the missingness mechanism is assumed to factorize according to a conditional directed acyclic graph. Our approach follows an interventionist perspective in which missingness indicators are treated as variables that can be intervened on. A central challenge in this setting is that sequences of interventions on missingness indicators may induce and propagate selection bias, so that identification can fail even when a propensity score is invariant to available interventions. To address this challenge, we introduce a tree-based identification algorithm that explicitly tracks the creation and propagation of selection bias and determines whether it can be avoided through admissible intervention strategies. The resulting tree provides both a diagnostic and a constructive characterization of identifiability under a given missingness mechanism. Building on these results, we develop recursive inverse probability weighting procedures that mirror the intervention logic of the identification algorithm, yielding valid estimating equations for both the missingness mechanism and functionals of the complete data distribution. Simulation studies and a real-data application illustrate the practical performance of the proposed methods. An accompanying R package, flexMissing, implements all proposed procedures.
Read the full abstract

Statistical Computation · Cross-listed from q-fin.ST

Integrating granular data into a multilayer network: an interbank model of the euro area for systemic risk assessment

Micro-structural models of contagion and systemic risk emphasize that shock propagation is inherently multi-channel, spanning counterparty exposures, short-term funding and roll-over risk, securities cross-holdings, and common-asset (fire-sale) spillovers. Empirical implementations, however, often rely on stylized or simulated networks, or focus on a single exposure dimension, reflecting the practical difficulty of reconciling heterogeneous granular collections into a coherent representation with consistent identifiers and consolidation rules. We close part of this gap by constructing an empirically grounded multilayer network for euro area significant banking groups that integrates several supervisory and statistical datasets into layer-consistent exposure matrices defined on a common node set. Each layer corresponds to a distinct transmission channel, long- and short-term credit, securities cross-holdings, short-term secured funding, and overlapping external portfolios, and nodes are enriched with balance-sheet information to support model calibration. We document pronounced cross-layer heterogeneity in connectivity and centrality, and show that an aggregated (flattened) representation can mask economically relevant structure and misidentify the institutions that are systemically important in specific markets. We then illustrate how the resulting network disciplines standard systemic-risk analytics by implementing a centrality-based propagation measure and a micro-structural agent-based framework on real exposures. The approach provides a data-grounded basis for layer-aware systemic-risk assessment and stress testing across multiple dimensions of the banking network.
Read the full abstract

All authors Ilias Aarab, Thomas Gottron, Andrea Colombo, Jörg Reddig, Annalauro Ianiro

Statistical Methodology

Prior Smoothing for Multivariate Disease Mapping Models

To date, we have seen the emergence of a large literature on multivariate disease mapping. That is, incidence of (or mortality from) multiple diseases is recorded at the scale of areal units where incidence (mortality) across the diseases is expected to manifest dependence. The modeling involves a hierarchical structure: a Poisson model for disease counts (conditioning on the rates) at the first stage, and a specification of a function of the rates using spatial random effects at the second stage. These random effects are specified as a prior and introduce spatial smoothing to the rate (or risk) estimates. What we see in the literature is the amount of smoothing induced under a given prior across areal units compared with the observed/empirical risks. Our contribution here extends previous research on smoothing in univariate areal data models. Specifically, for three different choices of multivariate prior, we investigate both within prior smoothing according to hyperparameters and across prior smoothing. Its benefit to the user is to illuminate the expected nature of departure from perfect fit associated with these priors, providing measures of their smoothing behavior not captured by conventional measures of predictive performance. We propose both theoretical and empirical metrics for our investigation and illustrate with both simulated and real data.
Read the full abstract

All authors Garazi Retegui, María Dolores Ugarte, Jaione Etxeberria, Alan E. Gelfand

Machine Learning (Statistics)

Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networks

Effective initialization in deep networks requires an understanding of random neural networks. In this work, a rigorous probabilistic analysis of deep bias-free random Leaky ReLU networks is provided. We prove a Law of Large Numbers and a Central Limit Theorem for the logarithm of the norm of network activations, establishing that, as the number of layers increases, their growth is governed by a parameter called the Lyapunov exponent. This parameter characterizes a sharp phase transition between vanishing and exploding activations, and we calculate the Lyapunov exponent explicitly for Gaussian or orthogonal weight matrices. Our results reveal that standard methods, such as He initialization or orthogonal initialization, do not guarantee activation stability for deep networks of low width. Based on these theoretical insights, we propose a novel initialization method, referred to as Lyapunov initialization, which sets the Lyapunov exponent to zero and thereby ensures that the neural network is as stable as possible, leading empirically to improved learning.
Read the full abstract

Statistical Methodology

Non-centred Bayesian inference for discrete-valued state-transition models: the Rippler algorithm

Stochastic state-transition models of infectious disease transmission can be used to deduce relevant drivers of transmission when fitted to data using statistically principled methods. Fitting this individual-level data requires inference on individuals' unobserved disease statuses over time, which form a high-dimensional and highly correlated state space. We introduce a novel Bayesian (data-augmentation Markov chain Monte Carlo) algorithm for jointly estimating the model parameters and unobserved disease statuses, which we call the Rippler algorithm. This is a non-centred method that can be applied to any individual-based state-transition model. We compare the Rippler algorithm to the state-of-the-art inference methods for individual-based stochastic epidemic models and find that it performs better than these methods as the number of disease states in the model increases.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.CL

SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora

We present SoftMatcha 2, an ultra-fast and flexible search algorithm that enables search over trillion-scale natural language corpora in under 0.3 seconds while allowing semantic variations in the form of substitution, insertion, and deletion. Our approach employs string matching based on suffix arrays that scales well with corpus size, and represents words as vectors, which underpin its semantic flexibility. To mitigate the combinatorial explosion induced by the semantic relaxation of queries, our method is built on two key algorithmic ideas: dynamic corpus-aware pruning and fast exact lookup enabled by a disk-aware design. We theoretically analyze the efficiency of the proposed method, indicating that it can mitigate exponential growth in the search space. Empirically, on FineWeb-Edu (Lozhkov et al., 2024) (1.4T tokens), it attains substantially lower search latency than existing methods: infini-gram (Liu et al., 2024), infini-gram mini (Xu et al., 2025), and SoftMatcha (Deguchi et al., 2025). As a practical application, our method uncovers benchmark contamination in training corpora that existing approaches miss, and it also benefits information retrieval and paraphrase detection. We also provide an online demo of fast, soft search across corpora in seven languages.
Read the full abstract

All authors Masataka Yoneda, Yusuke Matsushita, Go Kamoda, Kohei Suenaga, Takuya Akiba, Masaki Waga, Sho Yokoi

Machine Learning (Statistics) · Cross-listed from cs.LG

Natural Hypergradient Descent: Algorithm Design, Convergence Analysis, and Parallel Implementation

In this work, we propose Natural Hypergradient Descent (NHGD), a new method for solving bilevel optimization problems. To address the computational bottleneck in hypergradient estimation–namely, the need to compute or approximate Hessian inverse–we exploit the statistical structure of the inner optimization problem and use the empirical Fisher information matrix as an asymptotically consistent surrogate for the Hessian. This design enables a parallel optimize-and-approximate framework in which the Hessian-inverse approximation is updated synchronously with the stochastic inner optimization, reusing gradient information at negligible additional cost. Our main theoretical contribution establishes high-probability error bounds and sample complexity guarantees for NHGD that match those of state-of-the-art optimize-then-approximate methods, while significantly reducing computational time overhead. Empirical evaluations on representative bilevel learning tasks further demonstrate the practical advantages of NHGD, highlighting its scalability and effectiveness in large-scale machine learning settings.
Read the full abstract

All authors Deyi Kong, Zaiwei Chen, Shuzhong Zhang, Shancong Mou

Machine Learning (Statistics)

Deep Learning of Compositional Targets with Hierarchical Spectral Methods

Why depth yields a genuine computational advantage over shallow methods remains a central open question in learning theory. We study this question in a controlled high-dimensional Gaussian setting, focusing on compositional target functions. We analyze their learnability using an explicit three-layer fitting model trained via layer-wise spectral estimators. Although the target is globally a high-degree polynomial, its compositional structure allows learning to proceed in stages: an intermediate representation reveals structure that is inaccessible at the input level. This reduces learning to simpler spectral estimation problems, well studied in the context of multi-index models, whereas any shallow estimator must resolve all components simultaneously. Our analysis relies on Gaussian universality, leading to sharp separations in sample complexity between two and three-layer learning strategies.
Read the full abstract

All authors Hugo Tabanelli, Yatin Dandi, Luca Pesce, Florent Krzakala

Statistics Applications

Integrating Unsupervised and Supervised Learning for the Prediction of Defensive Schemes in American football

Anticipating defensive coverage schemes is a crucial yet challenging task for offenses in American football. Because defenders' assignments are intentionally disguised before the snap, they remain difficult to recognize in real time. To address this challenge, we develop a statistical framework that integrates supervised and unsupervised learning using player tracking data. Our goal is to forecast the defensive coverage scheme – man or zone – through elastic net logistic regression and gradient-boosted decision trees with incrementally derived features. We first use features from the pre-motion situation, then incorporate players' trajectories during motion in a naive way, and finally include features derived from a hidden Markov model (HMM). Based on player movements, the non-homogeneous HMM infers latent defensive assignments between offensive and defensive players during motion and transforms decoded state sequences into informative features for the supervised models. These HMM-based features enhance predictive performance and are significantly associated with coverage outcomes. Moreover, estimated random effects offer interpretable insights into how different defenses and positions adjust their coverage responsibilities.
Read the full abstract

Statistics Theory

Nonparametric two sample test of spectral densities

A novel nonparametric test for the equality of the covariance matrices of two Gaussian stationary processes, possibly of different lengths, is proposed. The test translates to testing the equality of two spectral densities and is shown to be minimax rate-optimal. Test performance is validated in a simulation study, and the practical utility is demonstrated in the analysis of real electroencephalography data. The test is implemented in the R-package sdf.test.
Read the full abstract

Statistics Theory

Covariate-Adaptive Randomization in Clinical Trials without Inflated Variances

Covariate adaptive randomization (CAR) procedures are extensively used to reduce the likelihood of covariate imbalances occurring in clinical trials. In literatures, a lot of CAR procedures have been proposed so that the specified covariates are balanced well between treatments. However, the variance of the imbalance of the unspecified covariates may be inflated comparing to the one under the simple randomization. The inflation of the variance causes the usual test of treatment effects being not valid and adjusting the test being not an easy work. In this paper, we propose a new kind covariate adaptive randomization procedures to balance covariates between two treatments with a ratio $\rho:(1-\rho)$. Under this kind of CAR procedures, the convergence rate of the imbalance of the specified covariates is $o(n^{1/2})$, and at the same time the asymptotic variance of the imbalance of any unspecified (observed or unobserved) covariates does not exceed the one under the simple randomization. The “shift problem” found by Liu, Hu, and Ma (2025) will not appear under the new CAR procedures.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Exploring the impact of adaptive rewiring in Graph Neural Networks

This paper explores sparsification methods as a form of regularization in Graph Neural Networks (GNNs) to address high memory usage and computational costs in large-scale graph applications. Using techniques from Network Science and Machine Learning, including Erdős-Rényi for model sparsification, we enhance the efficiency of GNNs for real-world applications. We demonstrate our approach on N-1 contingency assessment in electrical grids, a critical task for ensuring grid reliability. We apply our methods to three datasets of varying sizes, exploring Graph Convolutional Networks (GCN) and Graph Isomorphism Networks (GIN) with different degrees of sparsification and rewiring. Comparison across sparsification levels shows the potential of combining insights from both research fields to improve GNN performance and scalability. Our experiments highlight the importance of tuning sparsity parameters: while sparsity can improve generalization, excessive sparsity may hinder learning of complex patterns. Our adaptive rewiring approach, particularly when combined with early stopping, proves promising by allowing the model to adapt its connectivity structure during training. This research contributes to understanding how sparsity can be effectively leveraged in GNNs for critical applications like power grid reliability analysis.
Read the full abstract

All authors Charlotte Cambier van Nooten, Christos Aronis, Yuliya Shapovalova, Lucia Cavallaro

Statistics Applications

The Dataset of Daily Air Quality for the Years 2013-2023 in Italy

Air quality and climate are major issues in Italian society and lie at the intersection of many research fields, including public health and policy planning. There is an increasing need for readily available, easily accessible, ready-to-use and well-documented datasets on air quality and climate. In this paper, we present the GRINS AQCLIM dataset, created under the GRINS project framework covering the Italian domain for an extensive time period. It includes daily statistics (e.g., minimum, quartiles, mean, median and maximum) for a collection of air pollutant concentrations and climate variables at the locations of the 700+ available monitoring stations. Input data are retrieved from the European Environmental Agency and Copernicus Programme and were subjected to multiple processing steps to ensure their reliability and quality. These steps include automatic procedures for fixing raw files, manual inspection of stations information, the detection and removal of anomalies, and the temporal harmonisation on a daily basis. Datasets are hosted on Zenodo under open-access principles.
Read the full abstract

Statistical Methodology

A closed form solution for Bayesian analysis of a simple linear mixed model

Linear mixed-effects models are a central analytical tool for modeling hierarchical and longitudinal data, as they allow simultaneous representation of fixed and random sources of variation. In practice, inference for such models is most often based on likelihood-based approximations, which are computationally efficient, but rely on numerical integration and may be unreliable example wise in small-sample settings. In this study, the somewhat obscure four-parameter generalized beta density is shown to be usable as a conjugate prior distribution for a simple linear mixed model. This leads to a closed-form Bayesian solution for a balanced mixed-model design, representing a methodological development beyond standard approximate or simulation-based Bayesian approaches. Although the derivation is restricted to a balanced setting, the proposed framework suggests a pathway toward analytically tractable Bayesian inference for more complex mixed-model structures. The method is evaluated through comparison with a standard frequentist solution based on likelihood estimation for linear mixed-effects models. Results indicate that the Bayesian approach performs just as well as the frequentist alternative, while yielding slightly reduced mean squared error. The study further discusses the use of empirical Bayes strategies for hyperparameter specification and outlines potential directions for extending the approach beyond the balanced case.
Read the full abstract