Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Tuesday 3 March

Machine Learning (Statistics)

Exact Functional ANOVA Decomposition for Categorical Inputs Models

Functional ANOVA offers a principled framework for interpretability by decomposing a model's prediction into main effects and higher-order interactions. For independent features, this decomposition is well-defined, strongly linked with SHAP values, and serves as a cornerstone of additive explainability. However, the lack of an explicit closed-form expression for general dependent distributions has forced practitioners to rely on costly sampling-based approximations. We completely resolve this limitation for categorical inputs. By bridging functional analysis with the extension of discrete Fourier analysis, we derive a closed-form decomposition without any assumption. Our formulation is computationally very efficient. It seamlessly recovers the classical independent case and extends to arbitrary dependence structures, including distributions with non-rectangular support. Furthermore, leveraging the intrinsic link between SHAP and ANOVA under independence, our framework yields a natural generalization of SHAP values for the general categorical setting.
Read the full abstract

All authors Baptiste Ferrere, Nicolas Bousquet, Fabrice Gamboa, Jean-Michel Loubes, Joseph Muré

Machine Learning (Statistics) · Cross-listed from cs.LG

HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization

Adam and AdamW are a class of default optimizers for training deep learning models in machine learning. These adaptive algorithms converge faster but generalize worse compared to SGD. In fact, their proved generalization error $O(\frac{1}{\sqrt{N}})$ also is larger than $O(\frac{1}{N})$ of SGD, where $N$ denotes training sample size. Recently, although some variants of Adam have been proposed to improve its generalization, their improved generalizations are still unexplored in theory. To fill this gap, in the paper, we restudy generalization of Adam and AdamW via algorithmic stability, and first prove that Adam and AdamW without square-root (i.e., Adam(W)-srf) have a generalization error $O(\frac{\hat{\rho}^{-2T}}{N})$, where $T$ denotes iteration number and $\hat{\rho}>0$ denotes the smallest element of second-order momentum plus a small positive number. To improve generalization, we propose a class of efficient clever Adam (i.e., HomeAdam(W)) algorithms via sometimes returning momentum-based SGD. Moreover, we prove that our HomeAdam(W) have a smaller generalization error $O(\frac{1}{N})$ than $O(\frac{\hat{\rho}^{-2T}}{N})$ of Adam(W)-srf, since $\hat{\rho}$ is generally very small. In particular, it is also smaller than the existing $O(\frac{1}{\sqrt{N}})$ of Adam(W). Meanwhile, we prove our HomeAdam(W) have a faster convergence rate of $O(\frac{1}{T^{1/4}})$ than $O(\frac{\breve{\rho}^{-1}}{T^{1/4}})$ of the Adam(W)-srf, where $\breve{\rho}\leq\hat{\rho}$ also is very small. Extensive numerical experiments demonstrate efficiency of our HomeAdam(W) algorithms.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Implicit Bias in Deep Linear Discriminant Analysis

While the Implicit Bias(or Implicit Regularization) of standard loss functions has been studied, the optimization geometry induced by discriminative metric-learning objectives remains largely unexplored.To the best of our knowledge, this paper presents an initial theoretical analysis of the implicit regularization induced by the Deep LDA,a scale invariant objective designed to minimize intraclass variance and maximize interclass distance. By analyzing the gradient flow of the loss on a L-layer diagonal linear network, we prove that under balanced initialization, the network architecture transforms standard additive gradient updates into multiplicative weight updates, which demonstrates an automatic conservation of the (2/L) quasi-norm.
Read the full abstract

Statistics Applications

Detecting Structural Heart Disease from Electrocardiograms via a Generalized Additive Model of Interpretable Foundation-Model Predictors

Structural heart disease (SHD) is a prevalent condition with many undiagnosed cases, and early detection is often limited by the high cost and accessibility constraints of echocardiography (ECHO). Recent studies show that artificial intelligence (AI)-based analysis of electrocardiograms (ECGs) can detect SHD, offering a scalable alternative. However, existing methods are fully black-box models, limiting interpretability and clinical adoption. To address these challenges, we propose an interpretable and effective framework that integrates clinically meaningful ECG foundation-model predictors within a generalized additive model, enabling transparent risk attribution while maintaining strong predictive performance. Using the EchoNext benchmark of over 80,000 ECG-ECHO pairs, the method demonstrates relative improvements of +0.98% in AUROC, +1.01% in AUPRC, and +1.41% in F1 score over the latest state-of-the-art deep-learning baseline, while achieving slightly better performance even with only 30% of the training data. Subgroup analyses confirm robust performance across heterogeneous populations, and the estimated entry-wise functions provide interpretable insights into the relationships between risks of traditional ECG diagnoses and SHD. This work illustrates a complementary paradigm between classical statistical modeling and modern AI, offering a pathway to interpretable, high-performing, and clinically actionable ECG-based SHD screening.
Read the full abstract

All authors Ya Zhou, Zhaohong Sun, Tianxiang Hao, Xiangjie Li

Statistical Methodology

A Bayesian Hierarchical Hurdle Beta-Binomial Model for Survey-Weighted Bounded Counts and Its Application to Childcare Enrollment

Bounded discrete proportions – counts out of known totals – present modeling challenges when data exhibit structural zeros, overdispersion, and hierarchical clustering. We develop a Bayesian hierarchical hurdle beta-binomial model with state-varying coefficients that addresses all four features. The framework makes three methodological contributions: (i) it studies cross-margin dependence via a cross-block covariance component and clarifies when and how this parameter is identified through the hierarchical layer rather than the conditional likelihood; (ii) it proposes a Cholesky-based sandwich variance calibration for pseudo-posterior inference under survey weights, guided by a parameter-specific design effect ratio diagnostic; and (iii) it introduces a log-scale marginal effect decomposition for hurdle models that translates regression coefficients into policy-relevant quantities. Applied to 6,785 childcare providers across 51 states from the 2019 National Survey of Early Care and Education, the model reveals a "poverty reversal": poverty reduces enrollment participation yet increases intensity among participants, with the extensive margin accounting for two-thirds of the total effect. Design-calibrated simulation shows that sandwich-corrected intervals substantially improve coverage, reaching 82–88.5% at the 90% nominal level for fixed effects. The R package hurdlebb implements all methods.
Read the full abstract

Machine Learning (Statistics)

Combinatorial Sparse PCA Beyond the Spiked Identity Model

Sparse PCA is one of the most well-studied problems in high-dimensional statistics. In this problem, we are given samples from a distribution with covariance $\Sigma$, whose top eigenvector $v \in R^d$ is $s$-sparse. Existing sparse PCA algorithms can be broadly categorized into (1) combinatorial algorithms (e.g., diagonal or elementwise covariance thresholding) and (2) SDP-based algorithms. While combinatorial algorithms are much simpler, they are typically only analyzed under the spiked identity model (where $\Sigma = I_d + \gamma vv^\top$ for some $\gamma > 0$), whereas SDP-based algorithms require no additional assumptions on $\Sigma$. We demonstrate explicit counterexample covariances $\Sigma$ against the success of standard combinatorial algorithms for sparse PCA, when moving beyond the spiked identity model. In light of this discrepancy, we give the first combinatorial method for sparse PCA that provably succeeds for general $\Sigma$ using $s^2 \cdot \mathrm{polylog}(d)$ samples and $d^2 \cdot \mathrm{poly}(s, \log(d))$ time, by providing a global convergence guarantee on a variant of the truncated power method of Yuan and Zhang (2013). We provide a natural generalization of our method to recovering a vector in a sparse leading eigenspace. Finally, we evaluate our method on synthetic and real-world sparse PCA datasets.
Read the full abstract

All authors Syamantak Kumar, Purnamrita Sarkar, Kevin Tian, Peiyuan Zhang

Machine Learning (Statistics)

Low-Degree Method Fails to Predict Robust Subspace Recovery

The low-degree polynomial framework has been highly successful in predicting computational versus statistical gaps for high-dimensional problems in average-case analysis and machine learning. This success has led to the low-degree conjecture, which posits that this method captures the power and limitations of efficient algorithms for a wide class of high-dimensional statistical problems. We identify a natural and basic hypothesis testing problem in $\mathbb{R}^n$ which is polynomial time solvable, but for which the low-degree polynomial method fails to predict its computational tractability even up to degree $k=n^{\Omega(1)}$. Moreover, the low-degree moments match exactly up to degree $k=O(\sqrt{\log n/\log\log n})$. Our problem is a special case of the well-studied robust subspace recovery problem. The lower bounds suggest that there is no polynomial time algorithm for this problem. In contrast, we give a simple and robust polynomial time algorithm that solves the problem (and noisy variants of it), leveraging anti-concentration properties of the distribution. Our results suggest that the low-degree method and low-degree moments fail to capture algorithms based on anti-concentration, challenging their universality as a predictor of computational barriers.
Read the full abstract

Statistical Computation

Composite Wavelet Matrix-Based Transforms and Applications

Orthogonal wavelet transforms are a cornerstone of modern signal and image denoising because they combine multiscale representation, energy preservation, and perfect reconstruction. In this paper, we show that these advantages can be retained and enhanced by moving beyond classical single-basis wavelet filterbanks to a broader class of composite wavelet-like matrices. By combining orthogonal wavelet matrices through products, Kronecker products, and block-diagonal constructions, we obtain new unitary transforms that generally fall outside the strict wavelet filterbank class, yet remain fully invertible and numerically stable. The central finding is that such composite transforms often induce stronger concentration of signal energy into fewer coefficients than conventional wavelets. This increased sparsity, quantified using Lorenz curve diagnostics, directly translates into improved denoising under identical thresholding rules. Extensive simulations on Donoho-Johnstone benchmark signals, complex-valued unitary examples, and adaptive block constructions demonstrate consistent reductions in mean-squared error relative to single-basis transforms. Applications to atmospheric turbulence measurements and image denoising of the Barbara benchmark further confirm that composite transforms better preserve salient structures while suppressing noise.
Read the full abstract

Statistics Applications

An Augmented Rating System for Test cricket: adapting the Glicko rating system

The International Cricket Council's (ICC) Test cricket ratings use match and series results alone, stating no allowance about the precision of a rating, or about home advantage and toss. We fold both into the Glicko rating system, which gives a probabilistic expected score, and recalibrate its scale for Test cricket. The two enter as covariates whose significance and weights are estimated from match data. Home advantage is worth approximately 13 rating points and the toss roughly 8. They add, with no significant interaction. Over the two completed World Test Championship cycles (2021-23 and 2023-25) the model predicts 77.6% of decisive matches correctly in the first, and matches whichever of standard Elo and unmodified Glicko rating system is more accurate, at a lower Brier score and log loss in both. Permuting each cycle's match order 1000 times leaves every final rating unchanged. The model shows robustness to ordering, not fairness towards the fixture list, which the unbalanced calendar prevents judging. Our ordering follows the ICC's (Spearman 0.979 and 0.983). We add not a different ranking but a calibrated win probability per match, a deviation on every rating, and home and toss adjustments of estimated size, none of which the ICC supplies.
Read the full abstract

Statistical Methodology

Modeling double bounded data based on correlated gamma random variables

Many types of bounded data defined on the unit interval arise naturally as ratios of the form $X/(X + Y)$. In the existing literature, the main statistical models proposed for this type of bounded data typically based on the assumption that the random variables $X$ and $Y$ are independent. However, this assumption is often unrealistic in practical applications, where $X$ and $Y$ tend to be correlated due to shared underlying mechanisms or common sources of variability. In this paper, we overcome such limitations and propose a model in which the marginal distributions of the two components are linked by a copula, leading to a more flexible and realistic representation of unit-interval data. In particular, in the proposed model, $X$ and $Y$ are dependent gamma random variables whose joint distribution is specified via Morgenstern's bivariate distribution}, allowing for positive and negative correlations between the components. The mathematical properties and practical applications are rigorously investigated. The resulting distribution exhibits a wide range of shapes, accommodating different degrees of skewness and, for some parameter configurations, more complex density structures. A Monte Carlo simulation study is carried out that shows the good performance of the maximum likelihood estimator in several scenarios of parameter choices. The potential and limitations of efficient likelihood-based computations are also discussed. We evaluate the effectiveness of the new model and its estimates in modeling real-world datasets related to economics.
Read the full abstract

Statistics Theory

Graph Disjointness with Applications to Reversible Markov Chains

The correspondence between weighted undirected graphs and reversible Markov chains via vertex random walks is simple and well known. Leveraging this correspondence and ideas from the theory of dynamical systems, we study the structural discordance of graphs and Markov chains by means of graph joinings. Informally, a joining of graphs $G$ and $H$ is a graph on the product of their vertex sets giving rise to a coupling of their random walks. Graphs $G$ and $H$ are strongly disjoint if their only joining is the tensor product, and they are weakly disjoint if the degree function of every joining is equal to the degree function of the tensor product. We establish close connections between graph joinings, disjointness, and graph factors. Our first principal result characterizes weak disjointness of graphs in terms of the spectral overlap of their Markov transition matrices. The second establishes that two graphs without self loops are strongly disjoint if and only if they are weakly disjoint and exactly one of the graphs is a tree. The third shows that the strong or weak disjointness of graphs is essentially determined by their vertex and edge sets, without regard to edge weights. Translating these results into the language of Markov chains yields new insights into the rigidity and structure of reversible couplings of reversible Markov chains.
Read the full abstract

Statistics Theory · Cross-listed from cs.IT

Functional Properties of the Focal-Entropy

The focal-loss has become a widely used alternative to cross-entropy in class-imbalanced classification problems, particularly in computer vision. Despite its empirical success, a systematic information-theoretic study of the focal-loss remains incomplete. In this work, we adopt a distributional viewpoint and study the focal-entropy, a focal-loss analogue of the cross-entropy. Our analysis establishes conditions for finiteness, convexity, and continuity of the focal-entropy, and provides various asymptotic characterizations. We prove the existence and uniqueness of the focal-entropy minimizer, describe its structure, and show that it can depart significantly from the data distribution. In particular, we rigorously show that the focal-loss amplifies mid-range probabilities, suppresses high-probability outcomes, and, under extreme class imbalance, induces an over-suppression regime in which very small probabilities are further diminished. These results, which are also experimentally validated, offer a theoretical foundation for understanding the focal-loss and clarify the trade-offs that it introduces when applied to imbalanced learning tasks.
Read the full abstract

Statistical Methodology

Semi-partitioned Generalized Method of Moments for Longitudinal Data with Lagged and Feedback Covariates

We propose a semi-partitioned Generalized Method of Moments (GMM) framework for analyzing longitudinal data with time-dependent covariates, within a marginal modeling paradigm. This approach addresses limitations of both aggregated and fully partitioned GMM models. Aggregated methods obscure temporal dynamics by assuming constant effects, while fully partitioned approaches offer temporal specificity at the cost of increased model complexity and instability–particularly with moderate sample sizes or deep lag structures. Our method distinguishes immediate from lagged effects by estimating contemporaneous coefficients separately and grouping lagged moment conditions into structured sets, while retaining flexibility in the lag-specific effects. This yields a model that is both statistically efficient and interpretable, capturing essential temporal variation while mitigating variance inflation and convergence challenges associated with full partitioning. The framework accommodates feedback, supports both continuous and binary outcomes, and utilizes the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm for reliable optimization. Through simulations, we demonstrate that the semi-partitioned GMM achieves coverage and competitive efficiency relative to fully partitioned models when the grouped-lag structure approximates the underlying lag pattern. Applications to clinical datasets on knee osteoarthritis and adolescent obesity confirm that the method recovers consistent, interpretable effects and avoids instability associated with finely grained partitioning.
Read the full abstract

Statistical Methodology

Tree-Embedded Bayesian Factor Models for Multidimensional Categorical Distributions

Analyzing data collected from multiple observational units to estimate common and heterogeneous structures through a hierarchical model is a central task in Bayesian inference, and to this end, Bayesian factor models are one of the most widely used tools for this purpose. In this paper, we propose a novel Bayesian latent factor model for categorical distributions from grouped data, providing a parsimonious model for describing many observed distributions through lower-dimensional structures. Grouped data arise in a wide range of applications in social science, for example, distributions of age composition and income observed across locations. In these contexts, standard mixture models can be inefficient because the distributions do not necessarily exhibit clear clustering structures, and the distributions can be more accurately approximated as a combination of lower-dimensional characteristics. To analyze distribution-valued data with the Bayesian factor analysis, we adopt a tree-based transformation that embeds distributions into a Euclidean space and construct a Bayesian latent factor model in the transformed space. We develop the hierarchical model by incorporating the infinite factor model, which can adaptively estimate the number of effective factors. In addition, we propose its generalization by incorporating a spatial dependence by introducing a prior based on a SAR model. The proposed model provides smooth estimates of multivariate distributional structures, because once a tree-based transformation is applied, both univariate and multivariate distributions are essentially treated as the same Euclidean vectors. Through numerical experiments using real population data, we demonstrate that the proposed model outperforms existing parametric and Bayesian nonparametric models in various scenarios involving smooth spatial variations, especially under small sample sizes.
Read the full abstract

All authors Naoki Awaya, Keisuke Sasaki, Genya Kobayashi, Shonosuke Sugasawa

Statistics Theory · Cross-listed from cs.IT

E-variables and tests of randomness for distribution classes

E-variables are a relatively new approach for testing statistical hypotheses that has been experiencing major development during the last several years. In this paper we introduce the method of e-variable-approximability and use it to develop a general approximation technique allowing us to construct e-variables for popular distribution classes important for applications. E-variables were originally based on a concept of Levin's (average-bounded) randomness tests from Algorithmic Information Theory. We show that our construction of e-variables can be used to provide an explicit construction for a randomness test with respect to a class of distributions.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty

As artificial agents become increasingly capable, what internal structure is necessary for an agent to act competently under uncertainty? Classical results show that optimal control can be implemented using belief states or world models, but not that such representations are required. We prove quantitative "selection theorems" showing that strong task performance (low average-case regret) forces world models, belief-like memory and – under task mixtures – persistent regime-tracking variables resembling functional primitives of emotion, along with informational modularity under block-structured tasks. Our results cover stochastic policies, partial observability, and evaluation under task distributions, without assuming optimality, determinism, or access to an explicit model. Technically, we reduce predictive modeling to binary "betting" decisions and show that regret bounds limit probability mass on suboptimal bets, enforcing the predictive distinctions needed to separate high-margin outcomes. In fully observed settings, this yields approximate recovery of the interventional transition kernel; under partial observability, it implies necessity of predictive state and belief-like memory, addressing an open question in prior world-model recovery work.
Read the full abstract

Statistical Methodology

A Decision Analysis Framework for High-fidelity and Low-fidelity Systems with Applications in Manufacturing Processes

Optimizing complex manufacturing processes often involves a trade-off between data accuracy and acquisition cost. High-fidelity data are accurate but limited, while low-fidelity data are abundant but often biased. Balancing these two sources is critical for efficient manufacturing optimization. To address this challenge, we develop a decision analysis framework based on multi-fidelity Gaussian process (GP) modeling based on the Kennedy-O'Hagan (KOH) framework. We propose a systematic Bayesian calibration approach using multi-fidelity GPs that explicitly quantifies the model discrepancy, and an algorithm that combines posterior sampling of calibration parameters with predictive sampling to characterize the distribution of optimal input settings and their associated uncertainty. These components are integrated into a five-stage practical workflow for the optimization of manufacturing processes. Through an illustrative example and two real-world applications in composite cure cycle optimization and injection molding process control, we demonstrate how the framework integrates information from both high-fidelity and low-fidelity data sources to support decision-making under parameter uncertainty.
Read the full abstract

All authors Fan Zhang, Qiong Zhang, Madhura Limaye, Dhanashree Shinde, Gang Li, Sai Aditya Pradeep, Srikanth Pilla

Machine Learning (Statistics)

Geometric structures and deviations on James' symmetric positive-definite matrix bicone domain

Symmetric positive-definite (SPD) matrix datasets play a central role across numerous scientific disciplines, including signal processing, statistics, finance, computer vision, information theory, and machine learning among others. The set of SPD matrices forms a cone which can be viewed as a global coordinate chart of the underlying SPD manifold. Rich differential-geometric structures may be defined on the SPD cone manifold. Among the most widely used geometric frameworks on this manifold are the affine-invariant Riemannian structure and the dual information-geometric log-determinant barrier structure, each associated with dissimilarity measures (distance and divergence, respectively). In this work, we introduce two new structures, a Finslerian structure and a dual information-geometric structure, both derived from James' bicone reparameterization of the SPD domain. Those structures ensure that geodesics correspond to straight lines in appropriate coordinate systems. The closed bicone domain includes the spectraplex (the set of positive semi-definite diagonal matrices with unit trace) as an affine subspace, and the Hilbert VPM distance is proven to generalize the Hilbert simplex distance which found many applications in machine learning. Finally, we discuss several applications of these Finsler/dual Hessian structures and provide various inequalities between the new and traditional dissimilarities.
Read the full abstract

Monday 2 March

Machine Learning (Statistics)

The Theory behind UMAP?

In 2018, McInnes et al. introduced a dimensionality reduction algorithm called UMAP, which enjoys wide popularity among data scientists. Their work introduces a finite variant of a functor called the metric realization, based on an unpublished draft by Spivak. This draft contains many errors, most of which are reproduced by McInnes et al. and subsequent publications. This article aims to repair these errors and provide a self-contained document with the full derivation of Spivak's functors and McInnes et al.'s finite variant. We contribute an explicit description of the metric realization and related functors. At the end, we discuss the UMAP algorithm, as well as claims about properties of the algorithm and the correspondence of McInnes et al.'s finite variant to the UMAP algorithm.
Read the full abstract

Statistical Methodology

Transportable inference using target population summary statistics under covariate shift

Transporting findings from a study population to a target population is central to evidence-based decision-making in real-world settings. Most existing methods require individual-level data from both populations to account for covariate shift. However, privacy regulations and data-sharing constraints often preclude access to such data from the target population, leaving only covariate summaries available for analysis. In this paper, we develop transportability methods that enable valid inference using source individual-level data and target covariate summaries. First, we apply entropy balancing to transportability, enabling source individual-level data to be adjusted to match the target covariate moments. We establish asymptotic normality for the entropy balancing estimator and propose a variance estimator to account for uncertainty in covariate summaries. Second, we develop a new transportability method that allows more flexible modeling of covariate shift, thereby accounting for covariate shift and uncertainty in covariate summaries simultaneously. Asymptotic normality for the proposed estimator is established and its asymptotic variance is consistently estimated. The proposed method offers greater flexibility in accounting for covariate shift and thus permits consistent estimation and valid inference when the validity of entropy balancing is not guaranteed. The proposed methods are evaluated by simulations and illustrated with an analysis of Surveillance, Epidemiology, and End Results breast cancer data.
Read the full abstract

Statistical Computation

CCMnet: A Software Package for Network Generation with Congruence Class Models

We introduce CCMnet, an R package designed to generate network ensembles that accurately reflect the uncertainty inherent in empirical data. While traditional network modeling often results in ensembles with fixed property values or model-determined levels of variability, CCMnet enables a continuous spectrum of variability for network properties, including edge counts, degree distribution, and mixing patterns. By defining probability distributions directly over congruence classes of networks, the package allows researchers to specify the uncertainty in network properties across the generated ensemble to match a specific sampling design or empirical distribution. Furthermore, this formulation provides a principled framework that encompasses several classic models (e.g., Erdős–Rényi model, stochastic block models, and certain exponential random graph models) that implicitly share this structural basis, while offering the flexibility to specify arbitrary, even non-parametric, distributions for network properties. CCMnet implements a Markov chain Monte Carlo (MCMC) framework to sample from these models. The utility of the package is illustrated by generating posterior predictive network ensembles representing school friendship networks.
Read the full abstract

All authors Ravi Goyal, Victor De Gruttola, Natasha K. Martin, Lior Rennert, Jukka-Pekka Onnela

Machine Learning (Statistics)

Conformal Graph Prediction with Z-Gromov-Wasserstein Distances

Supervised graph prediction addresses regression problems where the outputs are structured graphs. Although several approaches exist for graph-valued prediction, principled uncertainty quantification remains limited. We propose a conformal prediction framework for graph-valued outputs, providing distribution-free coverage guarantees in structured output spaces. Our method defines nonconformity via the Z-Gromov-Wasserstein distance, instantiated in practice through Fused Gromov-Wasserstein (FGW), enabling permutation invariant comparison between predicted and candidate graphs. To obtain adaptive prediction sets, we introduce Score Conformalized Quantile Regression (SCQR), an extension of Conformalized Quantile Regression (CQR) to handle complex output spaces such as graph-valued outputs. We evaluate the proposed approach on a synthetic task and a real problem of molecule identification.
Read the full abstract

All authors Gabriel Melo, Thibaut de Saivre, Anna Calissano, Florence d'Alché-Buc

Machine Learning (Statistics) · Cross-listed from cs.LG

Manifold Aware Denoising Score Matching (MAD)

A major focus in designing methods for learning distributions defined on manifolds is to alleviate the need to implicitly learn the manifold so that learning can concentrate on the data distribution within the manifold. However, accomplishing this often leads to compute-intensive solutions. In this work, we propose a simple modification to denoising score-matching in the ambient space to implicitly account for the manifold, thereby reducing the burden of learning the manifold while maintaining computational efficiency. Specifically, we propose a simple decomposition of the score function into a known component $s^{base}$ and a remainder component $s-s^{base}$ (the learning target), with the former implicitly including information on where the data manifold resides. We derive known components $s^{base}$ in analytical form for several important cases, including distributions over rotation matrices and discrete distributions, and use them to demonstrate the utility of this approach in those cases.
Read the full abstract

All authors Alona Levy-Jurgenson, Alvaro Prat, James Cuin, Yee Whye Teh

Statistical Computation

Leveraging Sparsity to Improve No-U-Turn Sampling Efficiency for Hierarchical Bayesian Models

Analysts routinely use Bayesian hierarchical models to understand natural processes. The no-U-turn sampler (NUTS) is the most widely used algorithm to sample high-dimensional, continuously differentiable models. But NUTS is slowed by high correlations, especially in high dimensions, limiting the complexity of applied analyses. Here we introduce Sparse NUTS (SNUTS), which preconditions (decorrelates and descales) posteriors using a sparse precision matrix ($Q$). We use Template Model Builder (TMB) to efficiently compute $Q$ from the mode of the Laplace approximation to the marginal posterior, then pass the preconditioned posterior to NUTS through the Bayesian software Stan for sampling. We apply SNUTS to seventeen diverse case studies to demonstrate that preconditioning with $Q$ converges one to two orders of magnitude faster than Stan's industry standard diagonal or dense preconditioners. SNUTS also outperforms preconditioning with the inverse of the covariance estimated with Pathfinder variational inference. SNUTS does not improve sampling efficiency for models with the highly varying curvature found in funnels, wide tails, or multiple modes. SNUTS is most advantageous, and can be scaled beyond $10^4$ parameters, in the presence of high dimensionality, sparseness, and high correlations, all of which are widespread in applied statistics. An open-source implementation of SNUTS is provided in the R package SparseNUTS.
Read the full abstract

All authors Cole C. Monnahan, Kasper Kristensen, James T. Thorson, Bob Carpenter

Machine Learning (Statistics) · Cross-listed from cs.LG

Dimension-Independent Convergence of Underdamped Langevin Monte Carlo in KL Divergence

Underdamped Langevin dynamics (ULD) is a widely-used sampler for Gibbs distributions $\pi\propto e^{-V}$, and is often empirically effective in high dimensions. However, existing non-asymptotic convergence guarantees for discretized ULD typically scale polynomially with the ambient dimension $d$, leading to vacuous bounds when $d$ is large. The main known dimension-free result concerns the randomized midpoint discretization in Wasserstein-2 distance (Liu et al.,2023), while dimension-independent guarantees for ULD discretizations in KL divergence have remained open. We close this gap by proving the first dimension-free KL divergence bounds for discretized ULD. Our analysis refines the KL local error framework (Altschuler et al., 2025) to a dimension-free setting and yields bounds that depend on $\mathrm{tr}(\mathbf{H})$, where $\mathbf{H}$ upper bounds the Hessian of $V$, rather than on $d$. As a consequence, we obtain improved iteration complexity for underdamped Langevin Monte Carlo relative to overdamped Langevin methods in regimes where $\mathrm{tr}(\mathbf{H})\ll d$.
Read the full abstract

All authors Shiyuan Zhang, Qiwei Di, Xuheng Li, Quanquan Gu

Statistics Applications

On the misuse of time-dependent models in assessing mask usage and excess mortality

The effectiveness of face masks as a population level intervention against respiratory viral transmission remains contested. While a large observational literature published during the COVID-19 pandemic reported beneficial effects, randomized controlled trials have consistently shown limited or no impact. An ecological analysis of European countries reported that average mask usage during the years 2020 and 2021 is positively associated with excess mortality in that same period in 24 European countries (Tausk and Spira, 2025). Such association remains after several attempts at controlling for confounding variables. This finding was later challenged by other authors and attributed to reverse causality (Cerqueira-Silva et al., 2026). In this paper, we reassess those criticisms in detail. We show that their analysis is fundamentally flawed, as the time-dependent regression framework used to refute the original findings yields spurious results partly due to the use of cumulative excess mortality as an outcome variable, thereby incorporating pre-intervention deaths and producing statistically significant effects even at impossible negative time lags. Diagnostic analyses further demonstrate that key assumptions of the model are violated, invalidating any association or causal interpretation. Finally, we present an original longitudinal analysis of mask usage designed to directly test the reverse causality hypothesis. By constructing multiple indices that capture mask adoption during distinct phases of pandemic waves, including interwave periods characterized by low mortality, we show that the association between mask usage and excess mortality persists and is not driven by reactive increases in masking. These findings provide substantial evidence that reverse causality provides, at most, a minor contribution to the observed association.
Read the full abstract

Statistics Applications

Contributions of geolocated weather and building related data for insurance assessment of flood risks

Floods rank among the costliest natural hazards, causing over USD 100 billion in insured losses between 2013 and 2023. In France, persistent deficits in the natural catastrophe scheme highlight the need for accurate, building-scale flood risk assessment. Insurers typically rely on frequency-severity models supported by hazard maps and regional climate indicators. However, previous studies show that such large-scale variables explain only a limited share of the variability in individual flood losses. This study evaluates the marginal contribution of multiple georeferenced data layers to modeling flood claim occurrence and severity in a large French home insurance portfolio. Starting from a baseline model based on standard underwriting information, we sequentially introduce climate-expert variables, extreme rainfall indicators, and fine-scale geolocated building and environmental attributes. The analysis focuses on a practical setting in which insurers cannot deploy full hydrological or hydraulic catastrophe models because of budgetary, licensing, or operational constraints. Results show that rainfall-based indicators, particularly a newly constructed metric capturing intense local precipitation, substantially improve claim modeling performance. Building and environmental variables further enhance occurrence prediction. Overall, the findings demonstrate how high-resolution geolocated data improve exposure and vulnerability assessment, complement official flood maps, and provide insurers with an operational framework for refining flood risk evaluation and pricing.
Read the full abstract

All authors Mulah Moriah, Franck Vermet, Pierre Ailliot, Philippe Naveau, Juliette Legrand

Machine Learning (Statistics)

Mini-Batch Covariance, Diffusion Limits, and Oracle Complexity in Stochastic Gradient Descent: A Sampling-Design Perspective

Stochastic gradient descent (SGD) is central to simulation optimization, stochastic programming, and online M-estimation, where sampling effort is a decision variable. We study the mini-batch gradient noise as a sampling-design object. Under exchangeable fresh-sampling mini-batches, the conditional covariance given the de Finetti directing measure mu is b^{-1} G_mu(theta), and under identifiability the projected population object is b^{-1} G*(theta) – projected Fisher information for correctly specified likelihoods, the sandwich partner of the Hessian otherwise. This identification fixes the noise matrix entering the diffusion analysis of constant-step SGD: the raw iterate path has a deterministic fluid limit, and the sqrt(b/eta)-scaled fluctuations satisfy a functional CLT with noise covariance G*; near a nondegenerate optimum the limit is Ornstein-Uhlenbeck, and its Lyapunov covariance scaled by eta/b matches the linearized discrete recursion at leading order. Under a curvature-noise compatibility condition mu_F > 0, we prove 1/N mean-square upper bounds and an i.i.d. parametric Fisher van Trees lower bound of the same rate order, with oracle-complexity guarantees depending on an effective dimension d_eff and condition number kappa_F. Numerical experiments verify the identification and confirm the Lyapunov predictions in direct SGD.
Read the full abstract

Other Statistics

Implications of the Pessimistic Lower Limit on the Drake Equation

The observation of life on Earth is generally accepted to be uninformative concerning the probability of life on other Earth-like planets, a belief first formalized by Brandon Carter and based on the selection effect of our existence. In a similar way, the Drake equation is either presented as estimate of the total number of active, communicative, extraterrestrial civilizations in our Galaxy ($n^g_{\rm civ}$), i.e. excluding humanity, or humanity is included in the estimate but judged to be an uninformative data point. Daniel Whitmire has recently challenged the Carter abiogenesis argument, claiming the logic behind it is flawed, as the conditional likelihoods used by Carter in Bayes' theorem are not evaluated prior to the occurrence of the evidence of life on Earth, but posterior. Doing so correctly, the anthropic selection effect is removed and the observation of life on Earth is informative after all. Following this argument, we treat the Drake equation as estimate of all technological civilizations in a statistical counting experiment and include the data point of humanity as informative evidence. This allows one to set a pessimistic lower limit on $n^o_{\rm civ}$ for the observable universe, $n^o_{\rm civ} > 0.051$ at 95% C.L., or $n^g_{\rm civ} > 8\times10^{-13}$ at 95% C.L. for the Galaxy. In particular, this excludes models that predict $n^o_{\rm civ}\ll 1$ for the observable universe and refines the allowable parameter space for hypotheses like Rare Earth. Our analysis substantially reduces the portion of the Drake equation parameter space that predicts humanity is alone; when applying the lower limit this study finds $P(n^o_{\rm civ}>1 |\, {\rm humanity}) = 97.6\%$, making solitude in the observable universe a disfavored outcome. For the low-end estimate of $n^o_{\rm civ}\! =\! 1$ we calculate a probability of 42% for the existence of other communicating civilizations.
Read the full abstract

Statistics Applications · Cross-listed from math.PR

"Game, Set, Match": Double Delight Watching a Grand Slam Tennis Match

Probabilistic properties of tennis scoring systems are examined and compared with best-of-K systems. A model, where each player has his/her own probability of winning his/her service point and which remains invariant for the duration of the match, and where outcomes of points played are independent of each other, is assumed. Probabilities of winning a game tie-breaker, a game, a set tie-breaker, a set, and the match are obtained. Since tennis scoring systems are unique, probability calculations require decomposing big and complicated problems into smaller and simpler constituent problems, solving these sub-problems, then combining to obtain the solution to the big problem. The problems that arise from tennis scoring systems offer excellent pedagogical venues for teaching probability, in particular, the use of the Theorem of Total Probability and the Iterated Rules for Mean, Variance, and Covariance. There are also many interesting questions in tennis, foremost of which is whether a tennis match under this assumption will actually end with probability one; or whether when two players of `equal abilities' play a match, the first server possesses an advantage. These questions are addressed in this work. Tennis scoring systems are technically statistical decision systems to determine the better player. Since such a decision system is based on a finite number of points played, erroneous decisions could arise, such as the inferior player winning the match. We compare different systems in terms of the probability of the better player winning, as well as the duration of the match in terms of the number of points played.
Read the full abstract