Peer Revenue Impact factor: volatile

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Monday 24 August

Statistics Applications · Cross-listed from econ.EM

Accelerating the Adoption of Residential Solar Power Systems: Policy Analysis using a Dynamic Structural Model

Problem definition: Solar electricity generation is a strategic component of energy portfolios designed to meet growing demand and reduce carbon emissions. Governments and municipalities encourage household photovoltaic (PV) adoption through upfront rebates and tax credits. Limited budgets require principled, data-driven policies that account for the drivers of adoption and the effects of incentives on adoption rates. Methodology/results: We develop a dynamic structural model of residential PV diffusion based on adoption decisions by forward-looking households that weigh the economic trade-offs between installing now and later. Adoption depends on return on investment and influence from neighboring adopters. The model segments households by home value and urbanization level, incorporates unobserved heterogeneity, and captures spatiotemporal installation dynamics. We estimate the model using Bayesian methods and detailed household-level data from Austin, Texas. In out-of-sample tests, it predicts installations more accurately than contemporary alternatives. We simulate counterfactual policies within the dynamic equilibrium of PV diffusion to evaluate rebate designs. The framework can also be adapted to study the adoption of other durable technologies. Managerial implications: A rebate offered for a limited period generates more adoption and emissions reductions than a prolonged, costlier program. This counterintuitive result arises from forward-looking behavior, neighbor influence, and accelerated adoption before the rebate expires. We also evaluate phased reductions and rebates differentiated by household segment. A two-step reduction outperforms multiple small reductions. Geographic differentiation improves policy performance, whereas differentiation by home value offers little advantage over a uniform rebate.
Read the full abstract

All authors Sebastián Souyris, Jason A. Duan, Anantaram Balakrishnan, Varun Rai

Statistical Methodology

Approximating the null distribution of generalized distance covariance

The null distribution of distance covariance is usually approximated by permutation, which is prohibitive when very small p-values are needed, or by matching a few moments to a parametric family, which is inaccurate in the tails. A third option is to approximate the limiting distribution, a weighted sum of chi-square variables, directly through the spectra of the doubly centred distance matrices. This is used for kernel-based tests but has lacked a rigorous justification. We prove that the empirical spectra give a uniformly consistent approximation of the limiting null distribution, and hence an asymptotically valid test, for a general class of distances of negative type on separable metric spaces. The result covers the Hilbert-Schmidt independence criterion as a special case. We also give an adaptive algorithm that brackets the p-value from a partial eigendecomposition, reducing the cost from $O(n^3)$ to $O(k n^2)$, and a shrinkage correction matching the first two moments. In simulations, the proposed tests are the only non-Monte-Carlo procedures whose empirical type I error converges to the nominal level.
Read the full abstract

Machine Learning (Statistics)

(Mis)Understanding Benign Overfitting in Equity Return Prediction

Highly overparameterized models often predict well despite interpolating training data in complex domains, challenging the classical bias–variance tradeoff. We investigate whether this “benign overfitting” phenomenon extends to equity return prediction. Consistent with recent statistical theory, we document two key phenomena: first, a double descent pattern in the ridgeless model's prediction risk; and second, that while the optimal ridge model consistently outperforms its ridgeless counterpart, this performance gap becomes negligible at large parameter-to-observation ratios. Ultimately, however, both models fail to outperform a simple historical average. This empirical evidence aligns with our asymptotic results under the null hypothesis of zero slope coefficients, suggesting that standard equity predictors lack true forecasting power—even within highly flexible, nonlinear machine learning architectures. These findings reconcile modern and classical machine learning in asset pricing: in the absence of a true signal, they asymptotically collapse to the historical average benchmark.
Read the full abstract

All authors Hui Guo, Jiawei Huang, Runze Li, Yan Yu

Statistical Methodology · Cross-listed from econ.EM

Principal Component Analysis for a Mix of Stationary and Nonstationary Variables

This paper develops a procedure for uncovering the common cyclical factors that drive a mix of stationary and nonstationary variables. The method does not require knowing which variables are nonstationary or the nature of the nonstationarity. An application to the FRED-MD macroeconomic dataset demonstrates that the approach offers similar benefits to those of traditional principal component analysis with some added advantages.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

From Causal Plausibility to Causal Reliability: Evaluating LLMs as Calibrated Direct Causal-Edge Classifiers

Large language models (LLMs) are increasingly used to provide prior causal knowledge for structural causal discovery, yet whether their direct-edge judgments and confidence can be trusted remains unclear. We systematically evaluate 12 instruction-tuned open-weight models across six benchmark causal graphs, five prompting strategies, and four confidence sources: verbalized, logit-based, cross-prompt agreement, and cross-model agreement. Under our language-only pairwise protocol, our evaluation yields three key findings. (i) LLM-based causal judgments are strongly recall-dominant: models predict overly dense graphs with many false-positive edges, while prompting mainly shifts the precision-recall trade-off rather than resolving overprediction. Gains from model scale diminish on the largest graphs and do not eliminate miscalibration. (ii) LLMs often capture causal relatedness without reliably identifying directness or orientation. Relative to published reference graphs, models misclassify 40.0% of indirect and 36.0% of reversed non-edges as direct edges, versus 28.2% of other non-edges. Moreover, 80.8% and 84.6% of these false positives receive verbalized confidence of at least 80%, revealing substantial overconfidence in structurally incorrect predictions. (iii) Conventional confidence estimates are unreliable, whereas agreement offers a more promising signal. Logit-based confidence frequently collapses near 1.0 regardless of correctness, while cross-prompt and cross-model agreement achieve better mean calibration and discrimination, though their advantages are not statistically significant after Holm correction. A benchmark-familiarity audit further identifies potential familiarity in five model-dataset pairs, all involving AsiaM. Overall, our results suggest LLMs are better viewed as sources of externally validated soft causal priors than as direct evidence of causal structure.
Read the full abstract

All authors Amit Kumar, Elnur Adl Zarabi, Suranjana Trivedy, Zhiqian Chen, Lei Zhang, Kaiqun Fu, Taoran Ji

Statistics Theory · Cross-listed from cs.LG

Provably adaptive sampling with uniform and remasking discrete diffusion models

Discrete diffusion models offer a promising alternative to autoregressive generation by enabling parallel updates, but their sampling efficiency can depend strongly on the choice of the forward process and the sampler. For the uniform forward process, existing lower bounds for the standard $\tau$-leaping sampler scale linearly with the ambient dimension $d$, raising the question of whether this dependence is intrinsic to the forward process. We answer this question in the negative. We consider a first-order sampler based on the leave-one-out denoiser for uniform and remasking processes whose coordinate updates can be performed in parallel. In both cases, the sampler can correct denoising mistakes during the sampling process, which becomes necessary when many coordinates are updated together. Our main result establishes an adaptive sampling guarantee: up to logarithmic factors, $N = O(\mathrm{DTC}(X_0) / \varepsilon)$ discretization steps suffice to achieve sampling error $O(\varepsilon_{\mathrm{score}}+\varepsilon)$, where $\varepsilon_{\mathrm{score}}$ is the error in score estimation. Thus, the sampling complexity is governed by the intrinsic dependence structure of the target distribution, as measured by its dual total correlation $\mathrm{DTC}(X_0)$, rather than directly by the ambient dimension $d$. Our analysis proceeds through a Bayes-optimal auxiliary sampler that separates discretization error from score-estimation error. We also derive an exact information-theoretic representation of the discretization error in terms of the mutual information between different coordinates of the forward process at different times. This representation applies to general forward processes and, in the uniform and remasking cases, can be controlled by $\mathrm{DTC}(X_0)$. Numerical experiments on structured synthetic distributions illustrate the predicted dimension-adaptive behavior.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.CL

ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings

Recent advances in continuous diffusion and flow-based language models (LMs) have achieved performance competitive with discrete LMs. However, existing continuous frameworks still rely on decoders supervised with cross entropy (CE) because the flow trajectories are not guaranteed to terminate at valid token embeddings. Motivated by this limitation, we introduce ConvergeFlow, an embedding-space flow-based LM, which constrains the data predictor to the convex hull of token embeddings and trains it solely with the mean squared error objective induced by flow matching. Under suitable regularity conditions, we prove that the resulting flow converges to valid token embeddings despite errors in the data predictor, enabling direct token prediction without a CE-supervised decoder. We further develop three sampling mechanisms for controlling the trade-off between the generative perplexity and entropy. Experiments on OpenWebText demonstrate that ConvergeFlow achieves performance competitive with existing continuous and discrete diffusion LMs. These findings demonstrate the potential of the flow-based paradigm for language modeling. Our code is available at https://github.com/Na-Li66/ConvergeFlow.
Read the full abstract

All authors Na Li, Yuchen Jiao, Changxiao Cai, Gen Li

Statistical Methodology

Interpretable AI with Local Distillation

Modern AI models such as tabular foundation models and gradient-boosted ensembles can outpredict classical methods, but provide little basis for reasoning about their predictions. High-stakes decisions call for models that are both accurate and interpretable as built. Local linear modeling offers a path forward: a smooth regression function is locally well approximated by a linear one, allowing a linear fit near each query point to achieve high accuracy without sacrificing transparency. The challenges lie in learning what is "local" and developing statistical tools for interpretation. Here, we propose local distillation, in which a black-box "teacher" guides a regularized linear "student" model at each query point. The teacher (1) defines locality by upweighting training observations with similar predicted outcomes, and (2) anchors the fit with its prediction at the query point, included as a pseudo-observation whose weight is estimated from the data. For interpretation, we add a small amount of Gaussian randomization to the local objective and use refits to assess stability: selection frequencies identify reliable features at a query point, and clustering the randomized fits identifies stable subgroups across the data. Under the lasso penalty, we prove that this randomization yields feature-selection probabilities that are stable under small perturbations of the training responses. Across 17 benchmark datasets, local distillation nearly matches its AI teacher's accuracy while producing a sparse linear model at each test point. In a high-dimensional cancer gene expression example, the framework identifies patient subgroups whose local models use different genes; this heterogeneity is invisible to a global linear model, and difficult to surface in a black-box model.
Read the full abstract

Statistics Applications · Cross-listed from econ.GN

The Measurement Revolution? Credible Measurement and Inference in the Age of AI

Artificial intelligence (AI) is transforming measurement in economics. AI models convert unstructured data, such as text and images, into structured variables at low cost, making previously prohibitive measurement feasible at scale. This shifts the bottleneck from finding any scalable measure of a phenomenon to choosing among many plausible ones, which may support different empirical conclusions. This review provides guidance for navigating that shift. We describe three stages at which AI enters the measurement pipeline—discovery, construct definition, and observation—and what each demands of researchers. We argue that credible inference with AI-generated variables requires appropriately designed validation: anchoring measurement to explicit criteria, rather than informal claims that a proxy is reasonable. We then examine how validation samples support valid inference even when AI predictions are arbitrarily biased, and what can be done when a random validation sample is unavailable.
Read the full abstract

Statistics Theory

Asymptotics for Model Selection in Probabilistic Principal Component Analysis

The probabilistic formulation of principal component analysis promises statistically grounded solutions to the problem of selecting the number of principal components. However, developing tractable model selection methods is complicated by the fact that the probabilistic principal component analysis (PPCA) model exhibits non-standard large sample asymptotics at model singularities, where the Fisher information matrix does not have full rank. In this work, tools from singular learning theory are used to provide a complete description of the marginal likelihood asymptotics for PPCA. The singular Bayesian information criterion (sBIC) along with our asymptotic results provides an effective procedure for selecting the number of principal components. In particular, the sBIC corrects for the overpenalization of the standard dimension-based BIC, which leads to too few principal components being selected, while also selecting the smallest true model with probability converging to one. Our framework extends beyond PPCA, where we also provide expressions for the sBIC that can be used to select between PPCA and factor analysis models. In both simulations and on real data the effectiveness of the proposed sBIC methodology is demonstrated.
Read the full abstract

Statistical Methodology

Penalized likelihood estimation of probability density functions using compositional splines

Probability density functions are commonly estimated through preliminary smoothing or aggregation procedures, e.g., histograms or kernel density estimation, before subsequent functional representation and functional data analyses. Such a two-stage approach can lead to additional approximation bias and weaken the direct connection between the observed data and the underlying distributional structure. In this paper, we propose a penalized maximum likelihood framework for direct estimation of probability density functions from raw observations within the framework of Bayes Hilbert spaces while preserving the compositional geometry of densities. The methodology is based on the centered log-ratio (clr) transformation, an isometric isomorphism between Bayes Hilbert spaces and the standard Lebesgue space of square integrable functions with zero integral, enabling efficient spline representations. The clr transformed densities are represented using ZB-spline basis functions, while their smoothness is controlled through quadratic penalties imposed on the spline coefficients. The proposed framework is developed for univariate and bivariate densities. In the latter case, it naturally incorporates the orthogonal decomposition into independent and interactive parts together with the corresponding geometric marginals. The performance is evaluated in a simulation study involving multiple complex scenarios and compared with kernel smoothing. Finally, the applicability of the framework is illustrated using empirical geochemical data.
Read the full abstract

Machine Learning (Statistics)

Primal–Dual Alternating Neural Learning for Timely Classification with Performance Guarantees

Timely risk classification is essential in many clinical monitoring settings, where decisions must balance the benefit of classifying patients early for subsequent intervention against the value of observing additional data. Yet most existing statistical and machine-learning methods are designed for fully observed trajectories and offer limited control over key operating characteristics such as sensitivity, specificity, and monitoring cost. We cast the sequential classification problem within a multi-objective optimization framework targeting these three criteria. We characterize the optimal decision rule through a value recursion that quantifies, at each time point, the trade-off between immediate classification and continued monitoring. To estimate the rule from data, we formulate a constrained optimization problem that maximizes specificity while enforcing prespecified sensitivity and monitoring-cost constraints. We then develop an estimation procedure that employs a recurrent neural network to approximate the evolving value processes and a primal–dual updating scheme to satisfy the performance constraints. Through simulation studies and an application to continuous glucose monitoring for hypoglycemia risk prediction, we demonstrate that the proposed method yields accurate and timely sequential decision rules that adhere to the desired operating characteristics.
Read the full abstract

Statistical Methodology

Evaluating the effects of policy interventions subject to early adoption: A case study of prescription drug monitoring programs and opioid dispensing

Policies that require organizations to use new systems, such as prescription drug monitoring programs (PDMPs), are often implemented in phases, with an initial period of voluntary access followed by mandated compliance. This allows the policy intervention to be adopted before compliance is required (early adoption), causing outcomes to change before the mandate takes effect. When early adoption is present, the no-anticipation assumption underlying synthetic control methods (SCM) is violated, leading to biased policy effect estimates. We formalize early adoption in a potential outcomes framework for staggered policy implementation and decompose the total policy effect into early adoption and mandate components. We then propose a two-stage, early adoption-aware SCM procedure that first estimates early adoption effects using an interactive fixed effects model fit to pre-mandate data and then residualizes outcomes before applying SCM variants to estimate mandate and total policy effects. Simulations, including settings with correlation between early adoption and latent factors, show reduced bias and improved uncertainty quantification relative to conventional SCM estimators. We apply the framework to state-level PDMP policies and per-capita opioid dispensing. After accounting for early adoption, estimates suggest reductions in opioid dispensing following PDMP availability and mandates; however, the estimates are imprecise and not statistically significant.
Read the full abstract

All authors Sarika Aggarwal, Brent A. Coull, Nima Hejazi, Rachel C. Nethery

Statistics Applications

From Daily Fluctuations to Annual Hydrological Cycles: A Wavelet-Based Analysis of Nonstationary Seasonality in Senegal River Hydropower Inflows

This study presents a reproducible framework combining Fourier and wavelet analysis to examine the seasonality of daily inflows at three sites on the Senegal River (Bafing Makana, Felou, Gouina), based on 65,631 daily observations spanning nearly 60 years (1961-2020). Using harmonic regression, Welch spectral analysis, stationary wavelet decomposition, Morlet continuous wavelet transforms, trend and change-point tests, and cross-wavelet coherence, the authors show that the annual cycle corresponds mainly to the D8 detail level (roughly 256-512 days), rather than a level-4 approximation. The Haar wavelet was selected as the best fit among four families tested. The D8 component accounts for 18.85-19.89% of total energy, while Fourier harmonic models explain 73.3-78.7% of daily inflow variability. The average seasonal peak occurs in September (around days 252-254), and annual coherence across the three sites is exceptionally high (0.988-0.999), indicating a highly synchronized regional hydrological cycle. No robust monotonic trend was found in seasonality strength, amplitude, or peak timing. However, Pettitt tests detected a statistically significant shift in amplitude or mean inflow around 1971-1976, consistent with a regional regime transition, though confirming this would require independent rainfall and dam-operation data. The framework offers a transparent basis for generating inflow scenarios within MOSSHOOS-Plan4RES and is transferable to other data-scarce hydropower systems.
Read the full abstract

All authors Steeven B. Affognon, Babacar M. Ndiaye, Pierre Mendy, Cheikh M. F. Kebe

Machine Learning (Statistics) · Cross-listed from math.PR

Strong Averaging Principle and Long-Time Dynamics for Fast-Slow SDEs with Increasing Time-Scale Separation and Degenerate Noise

We establish a strong averaging principle for fast-slow stochastic differential equations with a time-dependent scale-separation parameter $(\varepsilon_t)_{t \geq 0}$ satisfying $\varepsilon_t \to 0$ as $t \to \infty$. In contrast to approaches based on noise-induced smoothing or elliptic regularity, our approach relies on dissipativity of the frozen fast dynamics and therefore permits degenerate diffusion coefficients. We prove a maximal $L^p$-estimate between the slow variable and the averaged ODE at late times, with the classical strong convergence rate of order $1/2$. Under an additional decay condition on $(\varepsilon_t)_{t \ge 0}$, this estimate implies that the slow variable is almost surely an asymptotic pseudo-trajectory of the averaged ODE. As a consequence, we obtain criteria for the identification of possible limit points and for convergence toward asymptotically stable equilibria for the slow variable by analyzing the dynamical behavior of the averaged equation.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Traceable Spectral Inference via Influence Functions: Efficient Data Attribution and Error Proxies for the Ariel Mission

Interpretability is critical for machine learning models deployed in scientific space missions such as ESA's Ariel, where ground truth is unavailable during operations and physical plausibility must be assessed. While most explainable AI methods focus on feature attribution, this work investigates training data attribution through influence functions and introduces three key contributions for operational spectroscopy pipelines. First, influence is reformulated in terms of prediction rather than loss, enabling label-free deployment. Second, by leveraging the closed-form ridge solution of an Extreme Learning Machine, infinitesimal prediction influence is efficiently computed. Third, an influence-based conservative error proxy is derived by propagating training residuals through the influence sensitivities. Evaluated against simulated spectra, the proposed proxy correlates strongly with scale and shape-based spectral errors. Furthermore, influence functions enable the identification of the most influential samples and the approximation of the most harmful ones. Together, these results suggest that this approach can serve as an operational framework for scientific machine learning.
Read the full abstract

All authors Nikki Grens, Luís F. Simões, Kai Hou Yip, Theresa Lueftinger

Statistical Methodology

Transporting Randomized Trial Effects to Real-World Populations via Riesz-Calibrated Optimal Transport

Randomized trials support causal inference, but differences between trial and target populations can limit the transportability of treatment effects to real-world settings. Many existing approaches model the propensity of trial participation and can therefore be sensitive to model misspecification and weak overlap of the covariate distributions. Optimal Transport (OT) offers a different route by comparing the trial and target populations directly in covariate space. We develop RICOT, a Riesz-calibrated OT procedure transporting treatment effects to a treated target population. We consider a semi-unbalanced OT with entropic regularization where the source marginals are relaxed. We show that the uncalibrated OT introduces a bias which does not shrink with increasing sample size. RICOT removes this bias by imposing calibration equations directly within the transport problem. With a growing calibration sieve, the calibrated weight consistently estimates the target-to-trial density ratio, equivalently the Riesz representer of the target expectation functional, even when the entropic and source-relaxation parameters remain fixed and positive. Combined with outcome regression, the resulting estimator is doubly robust and attains the semiparametric efficiency bound under suitable rate conditions. Its variance is estimated directly from the influence function, without resampling or repeated OT optimization. Simulations show low bias and near-nominal coverage across a range of overlap and misspecification settings, including settings in which sampling-score methods perform poorly. We illustrate RICOT in a real-world application involving a rare progressive cardiomyopathy, comparing conventional IPW and AIPW estimators with our OT-based IPW and doubly robust estimators for transporting the randomized treatment effect to a real-world population receiving the same treatment.
Read the full abstract

All authors Anik Burman, Margaret Gamalo, Promit Ghosal, Prosenjit Kundu

Statistics Applications · Cross-listed from cs.AI

Modalities Should Talk to Each Other: Dual-Stream Multimodal Learning for Long-Horizon Influenza Forecasting

Forecasting long-range influenza-like illness (ILI) matters for public health readiness. Publicly available surveillance datasets typically pair numeric epidemiological signals with textual information that is noisy, loosely structured, only indirectly related to near-term trends, and often lagged relative to the numeric signal. Fusing the two therefore requires careful design. We propose Dual-Stream Attention (DSA), a multimodal deep learning framework that forecasts 12-week-ahead ILI activity from a 36-week multimodal history by letting the numerical and textual streams condition each other. Using the Time-MMD health-domain dataset, DSA separately encodes the two modalities with a Transformer-based numerical encoder and a domain-adapted headline encoder, then couples them through a bidirectional Cross-Modal Attention (CMA) mechanism: the text (news headlines) conditions the interpretation of the numeric signal and vice versa. The CMA output then passes to a causal temporal model for forecasting. Evaluated across ten random seeds, DSA achieves a median test MSE of 0.416, versus 0.668, 0.607, and 0.851 for iTransformer, TaTS, and GPT4MTS, corresponding to mean-error reductions of 54.95%, 37.29%, and 67.23%, with paired Cohen's d of 0.555, 0.337, and 0.345, respectively, and ranks first in 100% of bootstrap draws. It also has substantially lower worst-window error than all baselines. On an external-geography dataset, DSA again ranks first among nine evaluated baselines. Ablations show the advantage does not depend on text-encoder choice or language-model fine-tuning, and that bidirectional attention outperforms either direction alone. Finally, perturbation-based faithfulness analysis shows the learned CMA is functionally informative under targeted masking, with a stronger effect in the text-to-numerical direction.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.IR

Hierarchical Exponential-Gaussian Mixtures for Watch-Time Distribution Prediction

Accurate watch-time (WT) prediction is an important requirement for short-video recommendations. Yet WT distributions are near-zero-inflated, long-tailed and multimodal. The recent Exponential-Gaussian Mixture Network (EGMN) models the full conditional WT distribution rather than a single point estimate and achieves state-of-the-art performance. Our large-scale reproduction study reveals that EGMN is vulnerable to variance collapse, component redundancy, and inactive components. We propose a Hierarchical Exponential-Gaussian Mixture (HEGM) model that addresses these failure modes through a hierarchical skip-watch decomposition, KL-based variance regularization, structured initialization, removing the forced Gaussian shift and the entropy regularizer. Across public and large-scale industrial datasets, HEGM improves ranking accuracy and threshold-event prediction, while maintaining competitive point-estimation accuracy and substantially improving mixture stability and interpretability. A 1.5-month production A/B test confirms statistically significant engagement lifts. Our code and models are publicly released at https://github.com/rw404/HEGM.
Read the full abstract

All authors Sofia Gulevskaia, Mikhail Trapeznikov, Aleksandr Poslavsky, Alexander D'yakonov

Other Statistics

Becoming Good Stewards of Information: A framework for integrating ethical, civic, and professional formation throughout the statistics and data science curriculum

Recent recommendations in statistics and data science education emphasize goals that extend beyond content mastery, including statistical literacy, evidence-based decision-making, communication, ethics, responsible use of data, and civic responsibility. We argue these goals can all be understood through a common lens: helping students become better stewards of information. Stewardship of information provides a unifying framework for the formation of students as statisticians, data scientists, and citizens who are thoughtful producers and consumers of data and evidence, communicate findings with integrity and transparency, engage critically with data ethics, recognize the human beings behind the data, and harness data to make informed decisions, serve communities, and advance the common good. We present stewardship of information as a developmental model that can be introduced early and revisited with increasing sophistication across a curriculum, including in introductory, advanced, applied, and theoretical courses. To illustrate this approach, we provide concrete examples of course activities and curated weekly readings as a low-barrier implementation strategy and present preliminary evidence of student engagement with the framework. By offering both a conceptual foundation and practical implementation strategies, this paper contributes a coherent approach for integrating ethical, professional, and civic formation across the statistics and data science curriculum.
Read the full abstract

Statistical Methodology

Classification testing: A new framework for drawing qualitative conclusions from quantitative estimates

Social scientists rely on hypothesis testing to support their research conclusions, but the standard tests are designed for testing one hypothesis rather than adjudicating between rival possibilities. We develop a new framework, "classification testing", as an alternative. Instead of selecting one hypothesis to test, a researcher conducting a classification test decides what qualitative distinctions ("classes") are most substantively relevant; the test either assigns the estimand to a class with error control similar to that of a conventional hypothesis test, or declares the result inconclusive. We argue that classification testing is superior to current practice not just when the objective is to adjudicate between rival possibilities but also when there is one research hypothesis to be tested, because classification testing exposes that hypothesis to refutation. We illustrate the framework by applying it to a well-known media experiment and offer an R package to aid in implementation.
Read the full abstract

Statistics Applications · Cross-listed from math.MG

An Approach to Study the Structural Consistency of Triangle Badness Functions and Distance Metrics

Triangle-based measures, commonly referred to as badness functions, are widely employed to quantify the extent to which a distance matrix deviates from an ideal geometric configuration. Different formulations of these functions may capture distinct facets of local non-uniformity, and their behavior is often influenced by the underlying distance metric chosen for evaluation. In practical settings, although a canonical badness function may be conceptually preferred, factors such as computational cost, algorithmic constraints, or data-specific characteristics frequently necessitate the adoption of modified versions-for instance, approximate forms or alternatives defined under different distance metrics. This gives rise to a central question: to what degree do these variants retain the structural consistency properties of their original counterparts? To address this issue, we develop a systematic correlation-based framework for evaluating structural consistency. As an illustrative instantiation of this framework, we compute badness sequences from a set of representative distance matrices alongside randomly generated triangle configurations, which are designed to cover variants that may arise under diverse practical scenarios. We then assess pairwise similarities among these sequences using four correlation coefficients. The experimental outcomes indicate that certain badness variants exhibit a notably high degree of structural consistency, whereas others reveal complementary behavioral patterns; moreover, the choice of distance metric exerts a considerable influence on the observed trends. These findings offer practical insights for the informed selection of distance metrics and triangle badness function variants in tasks including geometric reconstruction, triangulation, and structural analysis of pairwise distance data.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.CL

Credal Large Language Models for Semantic Commitment under Uncertainty

Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence. A central limitation is that standard LLMs represent uncertainty through a single predictive distribution, conflating epistemic ignorance with genuine ambiguity. We introduce Credal Large Language Models (CLLMs): an ensemble of LoRA adapters induces a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions rather than collapsing to a single softmax output. From this representation, we derive a single commitment rule: the model commits to an answer only when its lower probability exceeds the upper probability of every alternative, and otherwise returns the set of answers that no plausible predictor rules out. We apply this commitment rule at two depths: Credal Token Commitment (CTC) applies it to answer tokens from one ensemble forward pass, which decides constrained answers without any generation; for open-ended answers, credal decoding extends a partial answer only when no completed answer dominates it, so that the completions produced are those the plausible predictors license, and Credal Semantic Commitment (CSC) applies the rule to their meaning clusters. We evaluate CLLMs with Gemma-2-9B, Llama-3.1-8B and Qwen2.5-7B on OpenBookQA, CoQA, TriviaQA and ARC-Challenge. On multiple choice, CTC commits on 73-91% of questions at 89-98% accuracy, returns sets of 1.1-1.5 options containing the gold one on 89-98%, and its intervals contain the observed accuracy in 24 of 30 confidence bins without calibration; corrupted context lowers commitment from 87-92% to 65-71%, and on Gemma the credal bound detects corruption better than every baseline. On open-ended QA, CLLM outperforms semantic entropy and Laplace-LoRA at a fixed coverage by up to 19% and 9.5% absolute accuracy on CoQA and TriviaQA with context, for every backbone.
Read the full abstract

Statistics Theory

Spectral stability of empirical metric-measure Laplacians

The variance of nonparametric estimators is typically insensitive to the regularity of the object being estimated. We establish such a property for the spectra of graph Laplacian matrices at a fixed bandwidth $h>0$. Specifically, given $n$ i.i.d. samples from a probability measure $\mu$ on a Polish metric space, we compare the eigenvalues of the empirical weighted Laplacian operator $\Delta_{\mu_n}^h$ to those of the population counterpart $\Delta_\mu^h$ under a spectral gap condition, bounding the relative error by $1/\sqrt{nv_\mu(h)}$ for eigenvalues of order smaller than $h^{-2}$, where $v_\mu(h)$ is the smallest mass of a ball of radius $h$. This bound requires very weak regularity conditions on $\mu$: it is satisfied if $\mu$ belongs to the class of coarse PI measures that we introduce. This class contains measures on metric graphs, spaces with sufficiently regular boundaries, corners, or branch points, together with discretizations or thickenings of these at scale $O(h)$. Even for measures having densities of regularity $s>2$ on manifolds (the only known case so far), our bound improves on the state-of-the-art by shaving off logarithmic factors.
Read the full abstract

Machine Learning (Statistics)

One Inverse Step is a Convex Program: Bayes-Limit Calibration of Diffusion Inversion

One implicit DDIM inversion step is the cheapest probe of whether a pretrained diffusion model encodes local manifold geometry. It is the stationarity condition of an explicit potential, $x-G(x)=\nabla\Psi_t(x)$, strongly convex at the Bayes limit with modulus exactly $e^{-h_t}$ for the step's log-SNR gap $h_t$ $-$ for every data law, schedule and point, with no manifold, reach or unimodality hypothesis. Three consequences must be kept apart. (i) The solution is unique at the Bayes limit; a second one requires the trained score to violate the posterior-covariance bound by $1/(1-e^{-h_t})$, a hypothesis-free certificate of model error; the same bound makes contraction a schedule constant, $\rho_g^{\star}=1-e^{-h_t}<0.326$ throughout the standard DDPM schedule. (ii) The solver can still fail: Picard iteration is unit-step gradient descent on $\Psi_t$, unstable wherever $\lambda_{\max}(\nabla^2\Psi_t)>2$, so oscillation certifies nothing; damping below $2/\lambda_{\max}$ cures it. (iii) The geometry lives in the convergence domain: on the scale-free depth $w=r\kappa_{\max}$ the oscillation shell sits at $w=\tfrac12$, schedule-free, and the divergence shell at $w=1/(1+\rho_g^{\star})$, with a measured finite-noise correction in $\|\mathrm{II}\|^2$. Exact scores reproduce both to within $0.54\%$ on three classes; no trained score we probe shows a shell $-$ a derived limitation, not a null result: the Fermi window conflicts with the model's own training support by $3.6$-$5.6\times$, and the trained Hessian-Lipschitz constant is $2$-$12\%$ of the curvature the law reads, $0$ on a ReLU net. Finally the unconditional ceiling $\sigma_t\lambda_{\max}(\mathrm{sym}\,J)\le1$, from $\mathrm{Cov}(x_0\mid x_t)\succeq0$ alone, holds for the exact score to $3\times10^{-7}$ but is violated in all DDPM CIFAR-10/CelebA-HQ-256 settings, by $1.26$-$4.66\times$.
Read the full abstract

Statistical Methodology

Sequentially valid inference for probabilistic inflation forecasts

Traditional statistical tests are poorly suited for the sequential evaluation of probabilistic forecast calibration. We address this limitation in macroeconomic forecasting by applying a new sequential testing method based on e-values. The e-value-based methodology enables anytime-valid inference. It allows practitioners to test against calibration continuously without invalidating statistical guarantees. To illustrate the framework's practical value, we apply it to probabilistic inflation forecasts for the United States, the Euro Area, and Switzerland. Our analysis shows that the sequential approach gives detailed insights into the timing and nature of forecast misspecification. We find these diagnostics are particularly insightful during major structural breaks. During these events, we find evidence against calibration that static, full-sample tests often miss. Therefore, this work shows that e-value-based tests are a practical method for the evaluation of forecast calibration in empirical macroeconomics.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from hep-ph

Neural Boltzmann Equations

The dynamics of particles in the early universe are described by Boltzmann equations, which involve high-dimensional phase-space integrals. Classical approaches use quadrature integration and evolve the system on a fixed momentum grid, which scales poorly to complicated systems and parameter scans, severely limiting the complexity of processes that can be studied. We introduce Neural Boltzmann Equations (NBEs), which combine three coupled concepts to overcome these limitations. First, particle properties are encoded in physics-inspired neural distribution functions, with parameters that can be predicted using neural networks, enabling efficient parameter scans. Second, phase-space integrals are evaluated with Monte Carlo, using importance sampling tools from collider physics. Third, we use the natural gradient method to evolve the system. After demonstrating the individual benefits of NBEs, we use the framework to perform the most numerically precise calculation to date of the effective number of relativistic neutrino degrees of freedom, $N_\mathrm{eff}$, in the Standard Model.
Read the full abstract

Statistics Applications · Cross-listed from q-bio.MN

Uncovering Cellular Resolution in scRNAseq via Unbiased Cell and Gene Network Analysis

Conventional annotation of single-cell RNA-sequencing (scRNA-seq) data relies heavily on manual, marker-based thresholding, an approach that can obscure subtle transcriptomic gradients and collapse functionally distinct cell states into broad, heterogeneous populations. Here we apply the Gaussian multi-Graphical Model (GmGM) framework, which jointly infers cell-cell and gene-gene dependency structure from a single scRNA-seq data matrix, to a 10x Genomics PBMC dataset. Ten independent GMGM-Leiden clustering runs were integrated into a robust consensus partition using a soft cluster ensemble approach and benchmarked against reference cell-type annotations. This strategy yielded stable cluster partitions that resolve biologically meaningful sub-populations not distinguished by the reference annotation. In parallel, for each cluster, gene co-expression modules were extracted from the fitted model via consensus Leiden clustering across resolutions, evaluated using standard network metrics, and validated functionally with the Network Enrichment Analysis Test (NEAT), which confirmed non-random enrichment signal. A module-scoring procedure linked network topology to per-cell, per-cluster expression signatures, and a novel extension of GmGM, recovering a shared cell-cell network together with population-specific gene networks in a single model run, was demonstrated in a case study on the CD4+ T-cell population. These results indicate that GmGM provides a unified, reproducible framework for joint cell clustering and gene-network inference, capable of revealing cellular structure beyond that captured by conventional pipelines.
Read the full abstract

All authors Olga lanzetta, Luisa Cutillo, Bailey Andrew, Claudia Angelini

Statistics Theory · Cross-listed from quant-ph

Batched and Complete U-Statistics for Trace-Polynomial Estimation from Classical Shadows

We study estimation of the trace polynomial $\operatorname{tr} p(P\rho P)$ from global classical shadows, where $\rho$ is an unknown quantum state and $P$ is a fixed projector. Disjoint batching and complete U-statistics yield unbiased estimators of the same trace moments, but assign different sample-size factors to the degenerate terms in their Hoeffding decompositions. Under the global Clifford protocol, exact degree-two variance formulas show that, on a null projected block of rank $s$, the quadratic degenerate term has order $s^2/N$ under batching and $s^2/N^2$ under complete symmetrization. For a logarithmic-degree polynomial used in entropy approximation, the quadratic coefficient raises the batched variance to at least order $s^2N\log^2N$ at the classical entropy cutoff. For complete U-statistics, we derive a cross-degree covariance identity and an exact variance decomposition for polynomial estimators. We also bound every Hoeffding order at a fixed degree and obtain a growing-dimensional risk bound for a small-spectrum entropy functional. The higher-order bounds retain a polynomial dependence on the ambient dimension and therefore do not cover logarithmically increasing degrees. Monte Carlo experiments confirm the degree-two formulas, and exact calculations illustrate the entropy risks.
Read the full abstract

Statistical Methodology

Identification and Inference for Causal Effects in Extremes under General Conditions

Understanding the propagation of extreme events is important in many economic and environmental applications, yet most econometric methods for causal inference focus on average effects rather than tail behavior. This paper studies the identification of causal relations in extremes and derives resulting estimators and their asymptotic inference. As measure of causal dependence between extreme realizations of variables, we analyze the asymptotic behavior of the Causal Tail Coefficient (CTC) within a linear structural causal model with heavy-tailed regularly varying innovations. In contrast to the existing literature, we allow the variables in the system to exhibit heterogeneous tail indices and consider the presence of potentially heavy-tailed confounders. We derive theoretical results assessing the limiting behavior of the CTC under these conditions and show how differences in tail behavior can help to reach identification of the causal structure. Light-tailed confounders are asymptotically negligible, but sufficiently heavy-tailed confounders can induce extremal dependence patterns that are observationally indistinguishable from direct causal effects. When suitable proxy information is available, identification can be recovered using an adjusted Causal Tail Coefficient. Based on these results, we develop estimation and inference procedures for causal relations in extremes under general conditions. We establish asymptotic properties of the proposed estimators and derive tests for the causal direction and heavy-tailed confounding. Simulation evidence examines their finite-sample performance and provides guidance on their implementation. Applications to climate and financial extremes illustrate how the proposed methods can uncover causal relations that may remain undetected by approaches targeting average dependence.
Read the full abstract