Statistics Applications
SCARFACE: a harmonized spatio-temporal dataset integrating socio-economic, environmental, and agricultural indicators for the Po Valley (Italy), 2011–2024
Paolo Maranzano, Pietro Colombo, Felicetta Carillo et al. (+5 more)
We present "Sequestering CARbon through Forests, AgriCulture, and land usE (SCARFACE)", a harmonized spatio-temporal
dataset that integrates
climate, air quality, airborne pollutant emissions, land cover, soil properties, agro-industry dynamics and socio-economic indicators, to jointly investigate interconnected processes linking agricultural systems, atmospheric dynamics, emissions, and socioeconomic conditions in the Po Valley, Northern Italy. The spatial reference unit is the Agrarian Sub Region (ASR), that is, groups of contiguous municipalities that are considered homogeneous with respect to physical geography, agronomic characteristics, and prevailing agricultural production systems. The
dataset adopts an annual panel structure from 2011 to 2024 defined over the 256 ASRs partitioning the Po Valley and comprises more than 2,700 indicators sourced from national and international public institutions. Heterogeneous data are harmonized within a processing workflow, tailored to the specific characteristics of each
dataset, that guarantee spatial and temporal consistency of the output
dataset. The resource supports reuse in applied econometrics, spatio-temporal modeling, clustering, and policy analysis focused on agriculture, air quality, and land use in a major European hotspot.
Read the full abstract ▾
All authors Paolo Maranzano, Pietro Colombo, Felicetta Carillo, Riccardo Borgoni, Riccardo Pajno, Matteo Borrotti, Luca Ferrero, Ezio Bolzacchini
Statistical Methodology
Robustifying and Selecting Cohort-Appropriate Prognostic Models under Distributional Shifts
Dimitris Bertsimas, Carol Gao, Angelos G. Koulouras et al. (+1 more)
External validation is widely regarded as the gold standard for prognostic model evaluation. In this study, we challenge the assumption that successful external calibration guarantees model generalizability and propose two complementary strategies to improve transportability of prognostic models across cohorts. Using six real-world
surgical cohorts from tertiary academic centers, we tested whether successful external calibration depends largely on similarity in
covariates and outcomes between training and validation cohorts, quantified using Kullback-Leibler (KL) divergence, with calibration assessed by the Integrated Calibration Index (ICI). From the model-developer's perspective, we trained the "best-on-average" prognostic model by tuning toward a meta-analysis-derived
covariate and outcome distribution as an approximation of the broader target population. From the end-user perspective, we proposed a simple measure for cohort outcome similarity to identify, among published models, the one most suitable for a given target cohort in terms of both calibration and clinical utility. External calibration worsened as distributional mismatch increased. Higher KL divergence was associated with higher ICI in both surgery-alone (Spearman $\rho=0.614$, $p=0.004$) and surgery + adjuvant chemotherapy cohorts (Spearman $\rho=0.738$, $p<0.001$). Meta-analysis-informed weighting improved calibration in most settings without materially affecting discrimination, with the clearest benefit when evaluated on the aggregated external population ($p=0.037$). Models developed in more similar cohorts achieved lower ICI in surgery-alone (Spearman $\rho=0.803$, $p<0.001$) and surgery + adjuvant chemotherapy cohorts (Spearman $\rho=0.737$, $p<0.001$), and provided greater clinical utility on DCA.
Read the full abstract ▾
All authors Dimitris Bertsimas, Carol Gao, Angelos G. Koulouras, Georgios Antonios Margonis
Statistics Applications
Practical Process Capability Indices Workflows
Fei Jiang and Lei Yang
This paper presents a comprehensive review of univariate process capability indices (PCIs), which are critical metrics for assessing how effectively a manufacturing process satisfies customer specifications based on a single quality characteristic. The primary objective of this review is to develop practical procedural workflows for conducting process capability analysis under various preconditions, including those less frequently addressed scenarios in existing literature. Key analytical components, such as outlier detection, normality test, and best distribution fitting, are integrated into the proposed framework to ensure accurate and robust capability assessments. By systematically evaluating a range of methodologies, this study offers guidance for researchers and practitioners in selecting the most appropriate PCIs for specific process conditions. Ultimately, the work aims to simplify the complexity of PCI analysis while enhancing its precision and utility in quality control and process improvement efforts.
Read the full abstract ▾
Machine Learning (Statistics)
PRIM-cipal components analysis
Tianhao Liu, Daniel Andrés Díaz-Pachón and J. Sunil Rao
Supervised No Free Lunch
Theorems (NFLTs) are well studied, yet unsupervised NFLTs remain underexplored. For elliptical distributions, we prove that there exist two equally optimal, scientifically meaningful bump-hunting strategies that are exact opposites, with no universal winner. Specifically, peeling $k$ orthogonal dimensions from $\mathbb{R}^d$ ($d \ge k$), retaining an inter-
quantile region of probability $1-\alpha$ per peeled dimension, maximizes total variance and Frobenius norm when the $k$ smallest principal components (called pettiest components) are selected, and minimizes them when the selected dimensions are the $k$ leading principal components. These optima inspire PRIM-based bump-hunting
algorithms either by minimizing variance or by minimizing volume, thereby motivating an NFLT. We test our results on the Fashion-MNIST database, showing that peeling the largest principal components captures multiplicity, while peeling the smallest principal components isolates popular styles.
Read the full abstract ▾
Statistical Methodology · Cross-listed from q-fin.ST
Spurious Predictability in Financial Machine Learning
Sotirios D. Nikolopoulos
Adaptive specification search generates statistically significant backtests even under martingale-difference nulls. We introduce a falsification audit testing complete predictive workflows against synthetic reference classes, including zero-predictability environments and microstructure placebos. Workflows generating significant walk-forward evidence in these environments are falsified. For passing workflows, we quantify selection-induced performance inflation using an absolute magnitude gap linking optimized in-sample evidence to disjoint walk-forward realizations, adjusted for effective multiplicity. Simulations validate extreme-value scaling under correlated searches and demonstrate detection power under genuine structure. Empirical case studies confirm that many apparent findings represent methodological artifacts rather than genuine predictability.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.SI
A Quasi-Experiment comparing the health of unhoused people who have and have not experienced an eviction in King County, WA
Ihsan Kahveci, Timothy A. Thomas, Nathalie E. Williams et al. (+5 more)
Home eviction poses a significant threat to housing stability, a critical determinant of health. This study examines the relationship between eviction and health and substance use within the unhoused population of King County, Washington. Using a sample of 1,106 individuals experiencing homelessness, we employed a quasi-experimental design to compare the health outcomes of those who have experienced eviction with those who have not. Our findings reveal eviction is associated with an 8.3% point increase (SE = 0.039) in the likelihood of reporting poor general health and an 9.5% increase (SE = 0.032) in substance use disorder. No significant effect was found for mental health outcomes. While these results highlight the severe health risks linked to eviction, further research with more precise estimates is necessary to better understand long-term effects. These findings contribute to the growing evidence of how home eviction undermines the well-being of vulnerable populations.
Read the full abstract ▾
All authors Ihsan Kahveci, Timothy A. Thomas, Nathalie E. Williams, Janelle Rothfolk, Cathea Carey, Paul Hebert, Amy Hagopian, Zack W. Almquist
Statistical Methodology
Sample continuation in Bayesian hierarchical model via variational inference
Yucong Liu, Zilai Si and Alexander Strang
Posterior distributions arising in ill-posed
Bayesian inverse problems are often both analytically intractable and highly sensitive to parameters of the chosen prior family. We aim to understand the sensitivity of intractable posterior distributions to changes in prior assumptions by tracking how a sample representation of the posterior changes as the prior parameters change. This enables sensitivity analysis for small perturbations in the prior, providing insights into the robustness of the posterior estimates under minor changes in assumptions. It also allows solution continuation when dealing with significant alterations in prior beliefs, facilitating a comprehensive understanding of how large shifts in assumptions affect the posterior distribution. We focus on a class of non-conjugate hierarchical models tailored to encourage sparsity in linear inverse problems. The specific hierarchical model of interest is chosen since it is parameterized by a small number of shape parameters, and includes most classical sparsity promoting priors as special cases. As the shape parameters change, the posterior can transition continuously from a tractable unimodal distribution to an intractable
multimodal distribution. To track the change in the posterior, we adopt particle based variational inference methods, specifically Stein Variational Gradient Descent (SVGD). SVGD iteratively updates a set of samples to minimize the KL-divergence away from a desired target distribution. We augment SVGD by Birth-Death sampling, which can efficiently exchange mass between separated modes, while simultaneously optimizing the kernel bandwidth used to derive the SVGD update. This method enables the discovery of new modes by tracing the modes as they branch out of a simpler, unimodal posterior, derived within the same family of priors.
Read the full abstract ▾
Statistical Methodology
Spatially continuous modelling of aggregated outcome data
Stephen Jun Villejo, Peter Diggle, Finn Lindgren et al. (+5 more)
This work develops a block aggregation approach to spatial estimation and prediction when the response is observed at a coarse spatial scale, for example as counts of events in administrative areas, or blocks, while
covariates are available at a finer spatial resolution, typically as raster images. Our approach specifies a linear predictor at the finer resolution as a combination of
covariate effects and a latent, spatially continuous Gaussian process. This linear predictor then determines the distribution of the response through an inverse link function and spatial integration. We use a simulation study to evaluate the performance of the proposed approach in comparison to two industry standard approaches: a traditional geostatistical model that associates each response with the centroid of its block; and a Markov random field (MRF) approach that aggregates
covariate data to block-level. As expected, the differences in performance among the three approaches are small with respect to block-level prediction. The rationale for, and advantage of, the block aggregation approach lies in its delivery of reliable inferences at whatever spatial resolution is required in a particular application. We describe two applications: a linear Gaussian sampling model of wastewater virus concentrations in England, using population density as
covariate; and log-linear Poisson model of cardiovascular hospitalisations in England using socio-demographic variables at fine-scale administrative units as
covariates.
Read the full abstract ▾
All authors Stephen Jun Villejo, Peter Diggle, Finn Lindgren, Haavard Rue, Guangquan Li, Ella White, Matthew Wade, Marta Blangiardo
Machine Learning (Statistics)
One-Shot Generative Flows: Existence and Obstructions
Panos Tsimpos, Daniel Sharp and Youssef Marzouk
We study dynamic measure transport for generative modeling, focusing on transport maps that connect a source measure $P_0$ to a target measure $P_1$ by integrating a velocity field of the form $v_t(x) = \mathbb{E}[\dot X_t \mid X_t = x]$, where $X_\bullet = (X_t)_t$ is a
stochastic process satisfying $(X_0,X_1)\sim{P_0}\otimes{P_1}$ and $\dot X_t$ is its time derivative. We investigate when $X_\bullet$ induces a straight-line flow: a flow whose pointwise acceleration vanishes and is therefore exactly integrable by any first-order method. First, we develop multiple characterizations of straight-line flows in terms of PDEs involving the conditional statistics of the process. Then, we prove that straight-line flows under endpoint independence exhibit a sharp dichotomy. On the one hand, we construct explicit, computable straight-line processes for arbitrary Gaussian endpoints. On the other hand, we show that straight-line processes do not exist for targets with sufficiently well-separated modes. We demonstrate this obstruction through a sequence of increasingly general impossibility
theorems that uncover a fundamental relationship between the sample-path behavior of a process with independent endpoints and the space-time geometry of this process' flow map. Taken together, these results provide a structural theory of when straight-line generative flows can, and cannot, exist.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks
Kang An, Chenhao Si, Shiqian Ma et al. (+1 more)
Physics-Informed Neural Networks (PINNs) often suffer from slow convergence, training instability, and reduced accuracy on challenging partial differential equations due to the anisotropic and rapidly varying geometry of their loss landscapes. We propose a lightweight
curvature-aware optimization framework that augments existing first-order optimizers with an adaptive predictive correction based on secant information. Consecutive gradient differences are used as a cheap proxy for local
geometric change, together with a step-normalized secant
curvature indicator to control the correction strength. The framework is plug-and-play, computationally efficient, and broadly compatible with existing optimizers, without explicitly forming second-order
matrices. Experiments on diverse PDE
benchmarks show consistent improvements in convergence speed, training stability, and solution accuracy over standard optimizers and strong baselines, including on the high-dimensional heat equation, Gray–Scott system, Belousov–Zhabotinsky system, and 2D Kuramoto–Sivashinsky system.
Read the full abstract ▾
All authors Kang An, Chenhao Si, Shiqian Ma, Ming Yan
Statistics Theory
Generalization of Pearl's Front-Door Criterion
Carol Wu and Elina Robeva
Pearl's front-door criterion provides a set of sufficient conditions for estimating the total causal effect from observational data in the presence of latent confounding, using the functional P(y | do(x := x*)) = \sum_z P(z | x*) \sum_x P(y | x, z) P(x). An open question is whether these conditions can be generalized to be both necessary and sufficient for the validity of this functional, similar to the generalization achieved for the back-door adjustment criterion by Shpitser. In this paper, we present a new, weakened set of graph-based conditions sufficient for the front-door formula to estimate the total causal effect, expanding the scope of problems amenable to front-door identification.
Read the full abstract ▾
Machine Learning (Statistics)
Structural interpretability in SVMs with truncated orthogonal polynomial kernels
Víctor Soto-Larrosa, Nuria Torrado and Edmundo J. Huertas
We study post-training interpretability for Support Vector Machines (SVMs) built from truncated orthogonal
polynomial kernels. Since the associated reproducing kernel
Hilbert space is finite-dimensional and admits an explicit tensor-product orthonormal basis, the fitted decision function can be expanded exactly in intrinsic RKHS coordinates. This leads to Orthogonal Representation Contribution Analysis (ORCA), a diagnostic framework based on normalized Orthogonal Kernel Contribution (OKC) indices. These indices quantify how the squared RKHS norm of the classifier is distributed across interaction orders, total
polynomial degrees, marginal coordinate effects, and pairwise contributions. The methodology is fully post-training and requires neither surrogate models nor retraining. We illustrate its diagnostic value on a synthetic double-spiral problem and on a real five-dimensional echocardiogram
dataset. The results show that the proposed indices reveal structural aspects of model complexity that are not captured by predictive accuracy alone.
Read the full abstract ▾
Statistics Theory · Cross-listed from quant-ph
Cloning is as Hard as Learning for Stabilizer States
Nikhil Bansal, Matthias C. Caro and Gaurav Mahajan
The impossibility of simultaneously cloning non-orthogonal states lies at the foundations of
quantum theory. Even when allowing for approximation errors, cloning an arbitrary unknown pure state requires as many initial copies as needed to fully learn the state. Rather than arbitrary unknown states, modern
quantum learning theory often considers structured classes of states and exploits such structure to develop learning
algorithms that outperform general-state
tomography. This raises the question: How do the sample complexities of learning and cloning relate for such structured classes? We answer this question for an important class of states. Namely, for $n$-
qubit stabilizer states, we show that the optimal sample complexity of cloning is $\Theta(n)$. Thus, also for this structured class of states, cloning is as hard as learning. To prove these results, we use representation-theoretic tools in the recently proposed Abelian State Hidden Subgroup framework and a new structured version of the recently introduced random purification channel to relate stabilizer state cloning to a variant of the sample amplification problem for probability distributions that was recently introduced in classical learning theory. This allows us to obtain our cloning lower bounds by proving new sample amplification lower bounds for classes of distributions with an underlying linear structure. Our results provide a more fine-grained perspective on No-Cloning
theorems, opening up connections from foundations to
quantum learning theory and
quantum cryptography.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Optimal last-iterate convergence in matrix games with bandit feedback using the log-barrier
Come Fiegel, Pierre Menard, Tadashi Kozuno et al. (+2 more)
We study the problem of learning minimax policies in zero-sum
matrix games. Fiegel et al. (2025) recently showed that achieving last-iterate convergence in this setting is harder when the players are uncoupled, by proving a lower bound on the exploitability gap of Omega(t^{-1/4}). Some online mirror descent
algorithms were proposed in the literature for this problem, but none have truly attained this rate yet. We show that the use of a log-barrier regularization, along with a dual-focused analysis, allows this O-tilde(t^{-1/4}) convergence with high-probability. We additionally extend our idea to the setting of extensive-form games, proving a bound with the same rate.
Read the full abstract ▾
All authors Come Fiegel, Pierre Menard, Tadashi Kozuno, Michal Valko, Vianney Perchet
Statistics Applications
On the robustness of Mann-Kendall tests used to forecast critical transitions
Tristan Gamot, Nils Thibeau–Sutre and Tom J.M. Van Dooren
Non-parametric approaches to test for trends in time series make use of the Mann-Kendall statistic. Based on asymptotic arguments, these tests assume that its distribution follows a Gaussian distribution, even for autocorrelated time series. Recent results on the lack of validity of this assumption urge a robustness analysis of these approaches. While the issue is relevant across a wide range of applications, we illustrate it here in the context of detecting early warning signals (EWS) of critical transitions, which are used across a variety of research domains, and where commonly applied methods generate autocorrelation. We present a broad analysis, covering all types of critical transitions commonly investigated in EWS studies. We compare empirical distributions of the Mann-Kendall statistic computed from classical EWS indicators preceding critical transitions to the theoretical distributions hypothesized by Mann-Kendall tests. We detect mismatches leading to inflated type I error rates, which would routinely lead to announcing a critical transition while it is not occurring. In contrast to a recent recommendation, we conclude that the use of Mann-Kendall tests for trend detection in the context of
forecasting critical transitions should be avoided. We point out several alternative methods available instead.
Read the full abstract ▾