Statistics Theory
Tweedie Calculus
Santiago Torres
Tweedie's formula, which under Gaussian noise expresses the posterior mean of a latent variable directly from the observed-data density, is a cornerstone of empirical Bayes and measurement-error analysis. No general theory, however, explains when analogous identities hold, how they are structured, or how to derive them for non-Gaussian noise and for posterior functionals other than the mean. This paper develops such a framework for additive-noise models. I characterize when conditional expectations of an unobserved latent variable, given the observed signal, admit direct expressions in terms of the observed density – identities I call Tweedie representations – and show that they are governed by a linear map, the Tweedie functional. Under general conditions, I prove that this functional exists, is unique, and is continuous. I provide a constructive method for its computation based on
Fourier analysis: the functional is obtained by extending the inverse
Fourier transform of an explicit tempered distribution. The theory yields posterior-mean formulas for non-Gaussian noise and provides new representations for nonlinear posterior functionals. Applications include Laplace mechanisms in differential
privacy and heteroskedastic Gaussian sequence models in compound decision problems.
Read the full abstract ▾
Statistics Theory · Cross-listed from math.NT
Arithmetic functions and learning theory
W. Burstein, A. Iosevich and A. Sant
We establish a connection between analytic number theory and computational learning theory by showing that the Möbius function belongs to a class of functions that is statistically hard to learn from random samples. Let $\mu_R$ denote the restriction of the Möbius function to the squarefree integers in $\{1,\dots,R\}$. Using a recent lower bound of Pandey and Radziwiłł for the $L^1$ norm of exponential sums with Möbius coefficients, we prove that \[ \FR(\mu_R) \gg R^{-1/4-\epsilon} \] for every $\epsilon>0$. We then show that, for a suitable absolute constant $c_0>0$, the class of $\{-1,1\}$-valued functions on the squarefree integers with
Fourier Ratio at least $c_0$ has Vapnik–Chervonenkis dimension at least $cR$. It follows that any distribution-independent learning
algorithm that succeeds uniformly on the class $\mathcal{H}_R(\eta_R)$ containing $\mu_R$, where $\eta_R \to 0$, requires at least $\Omega(R)$ samples. We also discuss a conditional improvement under a strong uniform bound for additive twists of the Möbius function, and we note that the same method applies to the Liouville function.
Read the full abstract ▾
Statistical Methodology
Propensity Score Weighting to Ensure Balance in Key Subgroups or Strata: A Practical Guide
Emma K. Mackay, Amol A. Verma, Fahad Razak et al. (+1 more)
Propensity score weighting approaches have been widely implemented in clinical research to estimate the effects of a treatment or exposure while mitigating the risk of confounding in the absence of random assignment. In practice, when working with large electronic health records (EHR) or administrative
datasets to evaluate health quality outcomes at the institutional level, or evaluate supportive care interventions for a wide range of hospitalized patients, it may be advisable to stratify the propensity score weighting approach by indication, reason for admission, or other clinical risk factors due to the potential for substantial heterogeneity across subgroups of patients with complex care needs. A stratified approach may be appropriate if (i) prognosis differs substantially between patient subgroups such that achieving balance in the composition of these strata between exposure/treatment groups should be prioritized, (ii) likelihood of exposure differs substantially across clinical subgroups, or (iii) the
covariate-exposure associations are expected to differ substantially between subgroups (i.e. there are
covariate-subgroup interactions in the exposure/treatment propensity model). For example, we may want to evaluate the impact of prophylactic anticoagulant use for venous thromboembolism prevention in elderly patients admitted to hospital for a wide array of conditions. The purpose of this article is to outline an approach to implementing propensity score weighting with stratification by clinical groups. We also provide guidance on best practices with particular focus on EHR and administrative medical data, and population health settings.
Read the full abstract ▾
All authors Emma K. Mackay, Amol A. Verma, Fahad Razak, Surain B. Roberts
Statistics Theory
Early-stopped aggregation: Adaptive inference with computational efficiency
Ilsang Ohn, Shitao Fan, Jungbin Jun et al. (+1 more)
When considering a model selection or, more generally, an aggregation approach for adaptive statistical inference, it is often necessary to compute
estimators over a wide range of model complexities including unnecessarily large models even when the true data-generating process is relatively simple, due to the lack of prior knowledge. This requirement can lead to substantial computational inefficiency. In this work, we propose a novel framework for efficient model aggregation called the early-stopped aggregation (ESA): instead of computing and aggregating
estimators for all candidate models, we compute only a small number of simpler ones using an early-stopping criterion and aggregate only these for final inference. Our framework is versatile and applies to both
Bayesian model selection, in particular, within the variational Bayes framework, and frequentist estimation, including a general penalized estimation setting. We investigate adaptive optimal property of the ESA approach across three learning paradigms. We first show that ESA achieves optimal adaptive contraction rates in the variational Bayes setting under mild conditions. We extend this result to variational empirical Bayes, where prior hyperparameters are chosen in a data-dependent manner. In addition, we apply the ESA approach to frequentist aggregation including both penalization-based and sample-splitting implementations, and establish corresponding theory. As we demonstrate, there is a clear unification between early-stopped Bayes and frequentist penalized aggregation, with a common "energy" functional comprising a data-fitting term and a complexity-control term that drives both procedures. We further present several applications and numerical studies that highlight the efficiency and strong performance of the proposed approach.
Read the full abstract ▾
All authors Ilsang Ohn, Shitao Fan, Jungbin Jun, Lizhen Lin
Statistics Theory · Cross-listed from econ.EM
Generalized Autoregressive Multivariate Models: From Binary to Poisson
Anna Bykhovskaya and Nour Meddahi
This paper presents a framework for binary autoregressive time series in which each observation is a Bernoulli variable whose success probability evolves with past outcomes and probabilities, in the spirit of GARCH-type dynamics, accommodating nonlinearities, network interactions, and cross-sectional dependence in the multivariate case. Existence and uniqueness of a stationary solution is established via a coupling argument tailored to the discontinuities inherent in binary data. A key theoretical result, further supported by our empirical illustration on S&P 100 data, shows that, under a rare-events scaling, aggregates of such binary processes converge to a Poisson autoregression, providing a micro-foundation for this widely used count model. Maximum likelihood estimation is proposed and illustrated empirically.
Read the full abstract ▾
Statistical Methodology
Deployment of AI-Assisted Interventions: Capacity Constraints and Noisy Compliance
Carri W. Chan, Yi Han, Hannah Li et al. (+1 more)
AI tools increasingly drive targeted interventions in various service settings, including healthcare, education, and public services.
Algorithms score individuals, trigger outreach to those above a threshold (e.g., high-risk or high-value), and encourage them to request service; then providers deliver service to those who request. Much of the work in this area has focused on improving predictive accuracy, implicitly assuming that better predictions lead to better outcomes. We show that predictions are only one component of a larger service system: when service capacity is limited and behavioral responses to outreach are probabilistic, system efficacy depends on operational forces that predictive accuracy does not capture. In such settings, the optimal score threshold must balance two effects: ensuring all capacity is filled (utilization) and, when capacity is constrained, preventing low-value requests from crowding out high-value ones (cannibalization). We characterize the optimal threshold and prove that thresholds based solely on predictive accuracy are generally suboptimal. Further,
algorithm selection metrics such as AUC can be misaligned with operational performance: they weight all thresholds equally, while optimal deployment uses a subset of thresholds that depends on both capacity and compliance behavior. We introduce a new metric, Operational AUC (OpAUC), and show that it identifies the efficacy-optimal
algorithm. Finally, we conduct a case study on sepsis early warning data that illustrates the magnitude of the improvement available from better threshold selection and shows that a predictor with lower AUC can achieve higher system efficacy under optimal deployment.
Read the full abstract ▾
All authors Carri W. Chan, Yi Han, Hannah Li, Benjamin L. Ranard
Statistics Applications
Joint Bayesian Inference of Genetic Effect Sizes and PK Parameters in Nonlinear Mixed-Effects Models
Julien Martinelli, Ibtissem Rebai, David W. Haas et al. (+1 more)
High-dimensional genetic
covariate selection in population pharmacokinetic (PK) models is challenging due to the cohort's restricted size and high correlation among single-nucleotide polymorphisms (SNPs). We propose a fully
Bayesian, single-stage framework that jointly infers nonlinear mixed effect model (NLMEM) parameters and SNP effect sizes, providing coherent posterior uncertainty and inclusion summaries within a single model fit. We compare five sparsity-inducing priors – Spike-and-Slab, Hierarchical Lasso, Regularized Horseshoe, R2–D2, and the $\ell_1$-ball – calibrated through effect-size and sparsity targets. In simulations, all priors showed low false-discovery rates around $0$–$0.08$ under the null, and recovered the causal signal under the alternative, with peak $F_1$ scores around $0.8$–$0.85$ under reasonable inclusion cutoffs. Spike-and-Slab was especially attractive because it provides analytical posterior inclusion probabilities directly, while among priors requiring tolerance-based proxy inclusion summaries, the $\ell_1$-ball combined similarly strong recovery with the most stable behavior across tolerance values. On genetic and PK data from the ANRS 12154 study in 129 Cambodians living with HIV and receiving nevirapine, posterior predictive checks indicated adequate calibration and PK parameter inference remained stable across priors. While the dominant signal was robust across priors, additional candidate SNPs showed only partial agreement in ranking and more prior-sensitive effect-size estimates. These results support
Bayesian variable selection within joint NLMEM as a principled approach for pharmacogenetic analyses when uncertainty quantification and regularization are central.
Read the full abstract ▾
All authors Julien Martinelli, Ibtissem Rebai, David W. Haas, Julie Bertrand
Statistical Methodology
PROXIMA: A Reliability Scoring Framework for Proxy Metrics in Online Controlled Experiments
Avinash Amudala
Online A/B testing at scale relies on proxy metrics – short-term, easily-measured signals used in place of slow-moving long-term outcomes. When the proxy-outcome relationship is heterogeneous across user segments, aggregate correlation can mask directional failures akin to Simpson's Paradox, leading to costly ship/no-ship errors. We introduce PROXIMA (Proxy Metric Validation Framework for Online Experiments), a lightweight diagnostic framework that scores proxy reliability through a composite of three complementary dimensions: normalised effect correlation, directional accuracy, and segment-level fragility rate. Unlike surrogate-index approaches that predict long-term treatment effects, PROXIMA directly audits whether a candidate proxy leads to correct launch decisions and flags the user segments where it fails. We validate PROXIMA on two public
datasets – the Criteo Uplift corpus (14M observations, advertising) and KuaiRec (7K users,
video recommendation) – using 80 simulated A/B tests. Early engagement metrics achieve a composite reliability of 0.80 on Criteo and 0.62 on KuaiRec, yielding 98.4% average decision agreement with an oracle policy. Fragility analysis reveals that recommendation domains exhibit substantially higher segment-level heterogeneity (68% fragility) than advertising (13%), yet directional accuracy remains above 96% in both cases. A sensitivity analysis over the weight space confirms that no single component suffices and that the composite provides substantially better discrimination between reliable and unreliable proxies than correlation alone. Code and reproduction scripts are available at: https://github.com/Avinash-Amudala/PROXIMA
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Search
Tianhao Qian, Jiayu Chen, Zhenyu Sun et al. (+1 more)
LLM-guided search explores multiple candidate trajectories, but at substantial test-time cost.
Pruning low-scoring frontier candidates can control this cost, yet it also turns potentially biased evaluator scores into irreversible decisions: systematic ranking errors can persist under repeated scoring and remove useful branches. We propose Probably Approximately Correct Conformal Filtering (PAC-CF). Its fixed-frontier analysis formulates elimination as an $(\varepsilon,\delta)$-PAC problem under bounded evaluator bias; its operational rule separately calibrates a score-gap threshold on held-out tasks by running the original controller without PAC-CF and using post-search verifier labels to measure the deficit of solution-preserving candidates relative to the frontier leader. Conditional on exchangeable native-controller tasks with nonempty protected exposure, conformal calibration gives finite-sample coverage for retaining at least one verifier-defined valid continuation at every protected frontier on the native trajectory. At deployment, PAC-CF removes only candidates whose gap from the highest frontier score exceeds the frozen threshold. We evaluate PAC-CF across three domains, five controllers, and four request budgets from B100 to B500. In the cross-domain/controller macro averages, the point estimates for all three workload measures are lower at every budget; the paired-bootstrap 95% confidence interval for utility excludes zero at B100 and B200. For
pruning-aware ToolTree, the full-test-set cross-domain utility difference is $+4.38$ points at each tested budget; on the natural-termination sensitivity cohort, physical requests decrease by $18.94$–$18.95\%$ and end-to-end
token usage by $23.57$–$23.76\%$.
Read the full abstract ▾
All authors Tianhao Qian, Jiayu Chen, Zhenyu Sun, Lixu Wang
Machine Learning (Statistics) · Cross-listed from cs.LG
Path-Sampled Integrated Gradients
Firuz Kamalov, Fadi Thabtah, R. Sivaraj et al. (+1 more)
We introduce path-sampled integrated gradients (PS-IG), a framework that generalizes feature attribution by computing the expected value over baselines sampled along the linear interpolation path. We prove that PS-IG is mathematically equivalent to path-weighted integrated gradients, provided the weighting function matches the cumulative distribution function of the sampling density. This equivalence allows the
stochastic expectation to be evaluated via a deterministic Riemann sum, improving the error convergence rate from $O(m^{-1/2})$ to $O(m^{-1})$ for smooth models. Furthermore, we demonstrate analytically that PS-IG functions as a variance-reducing filter against gradient noise - strictly lowering attribution variance by a factor of 1/3 under uniform sampling - while preserving key axiomatic properties such as linearity and implementation invariance.
Read the full abstract ▾
All authors Firuz Kamalov, Fadi Thabtah, R. Sivaraj, Neda Abdelhamid
Machine Learning (Statistics) · Cross-listed from cs.LG
Heat and Matérn Kernels on Matchings
Dmitry Eremeev, Salem Said and Viacheslav Borovitskiy
Applying kernel methods to matchings is challenging due to their discrete, non-Euclidean nature. In this paper, we develop a principled framework for constructing
geometric kernels that respect the natural geometry of the space of matchings. To this end, we first provide a complete characterization of stationary kernels, i.e. kernels that respect the inherent
symmetries of this space. Because the class of stationary kernels is too broad, we specifically focus on the heat and Matérn kernel families, adding an appropriate inductive bias of smoothness to stationarity. While these families successfully extend widely popular Euclidean kernels to matchings, evaluating them naively incurs a prohibitive super-exponential computational cost. To overcome this difficulty, we introduce and analyze a novel, sub-exponential
algorithm leveraging zonal
polynomials for efficient kernel evaluation. Finally, motivated by the known bijective correspondence between matchings and phylogenetic trees-a crucial data modality in biology-we explore whether our framework can be seamlessly transferred to the space of trees, establishing novel negative results and identifying a significant open problem.
Read the full abstract ▾
Machine Learning (Statistics)
Doubly Outlier-Robust Online Infinite Hidden Markov Model
Horace Yiu, Leandro Sánchez-Betancourt, Álvaro Cartea et al. (+1 more)
We derive a robust update rule for the online infinite hidden Markov model (iHMM) for when the streaming data contains outliers and the model is misspecified. Leveraging recent advances in generalised
Bayesian inference, we define robustness via the posterior influence function (PIF), and provide conditions under which the online iHMM has bounded PIF. Imposing robustness inevitably induces an adaptation lag for regime switching. Our method, which is called Batched Robust iHMM (BR-iHMM), balances adaptivity and robustness with two additional tunable parameters. Across limit order book data, hourly electricity demand, and a synthetic high-dimensional linear system, BR-iHMM reduces one-step-ahead
forecasting error by up to 67% relative to competing online
Bayesian methods. Together with theoretical guarantees of bounded PIF, our results highlight the practicality of our approach for both
forecasting and interpretable online learning.
Read the full abstract ▾
All authors Horace Yiu, Leandro Sánchez-Betancourt, Álvaro Cartea, Gerardo Duran-Martin
Statistical Methodology
Combining Bayesian and Frequentist Inference for Laboratory-Specific Performance Guarantees in Copy Number Variation Detection
Austin Talbot, Alex V. Kotlar and Yue Ke
Targeted amplicon panels are widely used in oncology diagnostics, but providing per-gene performance guarantees for copy number variant (CNV) detection remains challenging due to amplification artifacts, process-mismatch heterogeneity, and limited validation sample sizes. While
Bayesian CNV callers naturally quantify per-sample uncertainty, translating this into the frequentist population-level guarantees required for clinical validation, coverage rates, false-positive bounds, and minimum detectable copy-number changes, is a fundamentally different inferential problem. We show empirically that even robust
Bayesian credible intervals, including coarsened posteriors and sandwich-adjusted intervals, are severely miscalibrated on panels with small amplicon counts per gene. To address this, we propose a hybrid framework that evaluates
Bayesian posterior functionals on validation samples and models the resulting squared losses with a Gamma distribution, yielding tolerance intervals with valid frequentist coverage. Three components make the method practical under real-world constraints: (1) imputation that removes the influence of true CNV-positive samples without requiring known ground truth, (2) regularization to address small sample variability, and (3) evidence-based stratification on the log model evidence to accommodate non-exchangeable noise profiles arising from process mismatch. Evaluated on two targeted amplicon panels using leave-one-out cross-validation, the proposed method achieves single-digit mean absolute coverage error across all genes under both process-matched and unmatched conditions, whereas
Bayesian comparators exhibit mean absolute errors exceeding 60% on clinically relevant genes such as ERBB2.
Read the full abstract ▾
Statistics Applications · Cross-listed from econ.GN
Mapping the causal structure of price formation in Texas's transitioning electricity market
Shiva Madadkhani, Nils Sturma, Mathias Drton et al. (+1 more)
Renewable deployment and rising demand from electrification and large digital loads are transforming electricity
markets. However, how these developments reshape electricity price dynamics remains poorly understood, leaving system planners, capacity investors, and
market participants reliant on assumptions from a thermal-dominated era that may no longer hold. We use causal discovery to study the
evolution of wholesale electricity prices in Texas, which is undergoing rapid transformation. Our findings overturn the view of Texas as a gas-price-driven
market, demonstrating that wind generation has become the dominant causal driver of day-ahead prices, with effects more than three times greater than those of natural gas. Yet wind's price-suppressing effect is weakening during peak periods, and wind growth redistributes congestion costs to distant load centres. Furthermore, rising load in South and West Texas alters system prices and regional differentials. Uncovering the evolving spatiotemporal nature of causal drivers, our analysis reveals that the pace, geographic siting, and relative scale of new generation and large loads will be decisive for future electricity price risks, infrastructure needs, and investments.
Read the full abstract ▾
All authors Shiva Madadkhani, Nils Sturma, Mathias Drton, Svetlana Ikonnikova
Machine Learning (Statistics) · Cross-listed from cs.LG
Metric-Aware Principal Component Analysis (MAPCA):A Unified Framework for Scale-Invariant Representation Learning
Michael Leznik
We introduce Metric-Aware Principal Component Analysis (MAPCA), a unified framework for scale-invariant representation learning based on the generalised eigenproblem max Tr(W^T Sigma W) subject to W^T M W = I, where M is a symmetric positive definite metric
matrix. The choice of M determines the representation geometry. The canonical beta-family M(beta) = Sigma^beta, beta in [0,1], provides continuous spectral bias control between standard PCA (beta=0) and output whitening (beta=1), with condition number kappa(beta) = (lambda_1/lambda_p)^(1-beta) decreasing monotonically to isotropy. The diagonal metric M = D = diag(Sigma) recovers Invariant PCA (IPCA), a method rooted in Frisch (1928) diagonal regression, as a distinct member of the broader framework. We prove that scale invariance holds if and only if the metric transforms as M_tilde = CMC under rescaling C, a condition satisfied exactly by IPCA but not by the general beta-family at intermediate values. Beyond its classical interpretation, MAPCA provides a
geometric language that unifies several self-supervised learning objectives. Barlow Twins and ZCA whitening correspond to beta=1 (output whitening); VICReg's variance term corresponds to the diagonal metric. A key finding is that W-MSE, despite being described as a whitening-based method, corresponds to M = Sigma^{-1} (beta = -1), outside the spectral compression range entirely and in the opposite spectral direction to Barlow Twins. This distinction between input and output whitening is invisible at the level of loss functions and becomes precise only within the MAPCA framework.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.AI
Interpretable and Explainable Surrogate Modeling for Simulations: A State-of-the-Art Survey and Perspectives on Explainable AI for Decision-Making
Pramudita Satria Palar, Paul Saves, Muhammad Daffa Robani et al. (+6 more)
The simulation of complex systems increasingly relies on sophisticated but fundamentally opaque computational black-box simulators. Surrogate models play a central role in reducing the computational cost of complex systems simulations across a wide range of scientific and engineering domains. Notwithstanding, they inevitably inherit and often exacerbate this black-box nature, obscuring how input variables drive physical responses. Conversely, Explainable Artificial Intelligence (XAI) offers powerful tools to unpack these models. Yet, XAI methods struggle with engineering-specific constraints, such as highly correlated inputs, dynamical systems, and rigorous reliability requirements. Consequently, surrogate modeling and XAI have largely evolved as distinct fields of research, despite their strong complementarity. To reconnect these approaches, this state-of-the-art survey provides a structured perspective that maps existing XAI techniques onto the various stages of surrogate modeling workflows for design and exploration. To ground this synthesis, we draw upon illustrative applications across both equation-based simulations and
agent-based modeling. We survey a broad spectrum of techniques, highlighting their strengths for revealing interactions and supporting human comprehension. Finally, we identify pressing open challenges, including the explainability of dynamical systems and the handling of mixed-variable systems, and propose a research agenda to make explainability a core, embedded element of simulation-driven workflows from model construction through decision-making. By transforming opaque emulators into explainable tools, this agenda empowers practitioners to move beyond accelerating simulations to extracting actionable insights from complex system behaviors.
Read the full abstract ▾
All authors Pramudita Satria Palar, Paul Saves, Muhammad Daffa Robani, Nicolas Verstaevel, Moncef Garouani, Julien Aligon, Koji Shimoyama, Joseph Morlier, Benoit Gaudou
Machine Learning (Statistics) · Cross-listed from cs.LG
Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
Arseniy Andreyev, Advikar Ananthkumar, Marc Walden et al. (+2 more)
Recent work suggests that (
stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found. Momentum and mini-batch gradients are widely used in practical deep learning optimization, but it remains unclear whether they operate in a comparable regime of instability. We demonstrate that SGD with momentum exhibits an Edge of
Stochastic Stability (EoSS)-like regime with batch-size-dependent behavior that cannot be explained by a single momentum-adjusted stability threshold. Batch Sharpness (the expected directional mini-batch
curvature) stabilizes in two distinct regimes: at small batch sizes it converges to a lower plateau $2(1-\beta)/\eta$, reflecting amplification of
stochastic fluctuations by momentum and favoring flatter regions than vanilla SGD; at large batch sizes it converges to a higher plateau $2(1+\beta)/\eta$, where momentum recovers its classical stabilizing effect and favors sharper regions consistent with full-batch dynamics. We further show that this aligns with linear stability thresholds and discuss the implications for hyperparameter tuning and coupling.
Read the full abstract ▾
All authors Arseniy Andreyev, Advikar Ananthkumar, Marc Walden, Tomaso Poggio, Pierfrancesco Beneventano
Other Statistics
The Epidemiology of Artificial Intelligence
Harsh Parikh, Tyler McCormick, Emily Johnson et al. (+3 more)
Artificial intelligence (AI) systems increasingly shape how people access health information, make medical decisions, and receive care – yet epidemiology lacks frameworks for measuring AI exposure or studying its health effects at the population level. Here we argue that AI now functions as a determinant of health and propose a conceptual framework, borrowed from environmental epidemiology, for studying it. We distinguish ambient AI exposure – algorithmic curation and AI-mediated institutional decisions that affect populations regardless of individual choice – from personal AI exposure – direct, volitional use of AI tools. We characterize AI's possible causal roles in epidemiological models, show that existing experimental approaches are inadequate for capturing chronic, population-level effects, and illustrate these ideas with nationally representative US survey data. We discuss implications for study design, health equity, and AI governance.
Read the full abstract ▾
All authors Harsh Parikh, Tyler McCormick, Emily Johnson, Leo Hickey, Megan Ranney, Bhramar Mukherjee
Statistical Computation · Cross-listed from physics.comp-ph
Distributional Inverse Homogenization
Arnaud Vadeboncoeur, Mark Girolami, Kaushik Bhattacharya et al. (+1 more)
For many materials, macroscopic mechanical behavior is determined by an intricate microstructure. Understanding the relation between these two scales helps scientists and engineers design better materials. The relation which maps microstructure to bulk material properties can be understood via the well-established theory of homogenization. However inverting the homogenization process, to recover microstructural information from measured macroscopic properties, is fraught with difficulties because of the averaging processes that underlie homogenization. Therefore, scientists and engineers usually need recourse to more invasive, often highly localized, investigations to estimate the microstructure. In this work, we develop a noninvasive methodology by which one can leverage large collections of measured bulk material properties to infer information about the statistics of microstructure at a global level. We call this, distributional inverse homogenization. We study this problem in one and two dimensions, considering both periodic and
stochastic homogenization. We demonstrate the methodology in the context of 2D Voronoi constructions and underpin the observed empirical success with theory in 1D. We also show how the natural spatial variability of microstructure can be exploited to gather data that enables distributional inversion. And we concurrently learn a surrogate model, approximating the homogenization map, that accelerates the resulting computations in this setting. The work formulates a new class of inverse problems, bridging ideas from probability and homogenization to facilitate the inference of microstructural material variability from macroscopic measurements.
Read the full abstract ▾
All authors Arnaud Vadeboncoeur, Mark Girolami, Kaushik Bhattacharya, Andrew M. Stuart
Machine Learning (Statistics) · Cross-listed from math.OC
Multistage Conditional Compositional Optimization
Buse Şen, Yifan Hu and Daniel Kuhn
We introduce Multistage Conditional Compositional Optimization (MCCO) as a new paradigm for decision-making under uncertainty that combines aspects of multistage
stochastic programming and conditional
stochastic optimization. MCCO minimizes a nest of conditional expectations and nonlinear cost functions. It has numerous applications and arises, for example, in optimal stopping, linear-quadratic regulator problems, distributionally robust contextual bandits, as well as in problems involving dynamic risk measures. The naïve nested sampling approach for MCCO suffers from the curse of dimensionality familiar from scenario tree-based multistage
stochastic programming, that is, its scenario complexity grows exponentially with the number of nests. We develop new multilevel Monte Carlo techniques for MCCO whose scenario complexity grows only polynomially with the desired accuracy.
Read the full abstract ▾
Statistics Theory
Finite-Step Bounds for Iterated Correlation Matrices
Ishrak AlhajjHassan
We establish finite-step probabilistic upper bounds on the contraction ratios $\rho_k = \Delta_{k+1}/\Delta_k$ for iterated Pearson correlation dynamics. Let $(P_k)_{k\ge 0}$ be the sequence generated by the Pearson update. Define $\Delta_k := \|P_{k+1}-P_k\|_F$, $\rho_k := \Delta_{k+1}/\Delta_k$ for $\Delta_k > 0$, and $\delta_k := \Delta_k/n$. Although $\Delta_k \to 0$ along convergent trajectories, the ratios $\rho_k$ may exceed unity in finitely many steps. This behavior is invisible to local linearization. Our main contribution is a probabilistic bounding framework that captures these finite-step expansions. We initialize $P_0$ with i.i.d. $\mathcal{U}[-1,1]$ entries and let $\mathbb{P}$ be the induced measure. For $k \ge 2$, we construct state-dependent bounds $B_p : \mathbb{R}_+ \to \mathbb{R}_+$ satisfying $\mathbb{P}(\rho_k \le B_p(\delta_k)) \ge p$. The functions $B^{\mathrm{q}}_p(\delta)$ are empirical conditional $p$-
quantiles of $\log \rho_k$ given $\delta_k$ under logarithmic binning. Larger families $B^{\mathrm{TC}}_{p,\tau}(\delta)$ and $B^{\mathrm{tol}}_{p,\tau}(\delta)$ are obtained via multiplicative adjustments, yielding pointwise larger bounds that preserve the $\delta$-dependence. Validation on held-out trajectories confirms the bounds hold with empirical coverage matching nominal levels for all $n \in [3,2000]$. The baseline $0.95$-
quantile bound $B^{\mathrm{q}}_{0.95}(\delta)$ yields two concrete results: $\mathbb{P}(\rho \le 1 \mid \delta \le 0.03) \ge 0.95$ uniformly in $n$, and $\mathbb{P}(\rho \le 1.7) \ge 0.95$ for 21 of 22 dimensions. The exception $n = 69$ attains $2.35$, revealing a rare extreme upper tail discontinuity not captured by asymptotic analysis. These are the first finite-step probabilistic bounds for Pearson correlation dynamics. The framework is fully reproducible with provided code and data.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.IT
Two-Sided Bounds for Entropic Optimal Transport via a Rate-Distortion Integral
Jingbo Liu
We show that the maximum expected inner product between a random vector and the standard normal vector over all couplings subject to a mutual information constraint or regularization is equivalent to a truncated integral involving the rate-distortion function, up to universal multiplicative constants. The proof is based on a lifting technique, which constructs a Gaussian process indexed by a random subset of the type class of the probability distribution involved in the information-theoretic inequality, and then applying a form of the majorizing measure
theorem.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
BOAT: Navigating the Sea of In Silico Predictors for Antibody Design via Multi-Objective Bayesian Optimization
Jackie Rao, Ferran Gonzalez Hernandez, Leon Gerard et al. (+1 more)
Antibody lead optimization is inherently a multi-objective challenge in drug discovery. Achieving a balance between different drug-like properties is crucial for the development of viable candidates, and this search becomes exponentially challenging as desired properties grow. The ever-growing zoo of sophisticated in silico tools for predicting antibody properties calls for an efficient joint optimization procedure to overcome resource-intensive sequential filtering pipelines. We present BOAT, a versatile
Bayesian optimization framework for multi-property antibody engineering. Our `plug-and-play' framework couples uncertainty-aware surrogate modeling with a genetic
algorithm to jointly optimize various predicted antibody traits while enabling efficient exploration of sequence space. Through systematic benchmarking against genetic
algorithms and newer generative learning approaches, we demonstrate competitive performance with state-of-the-art methods for multi-objective
protein optimization. We identify clear regimes where surrogate-driven optimization outperforms expensive generative approaches and establish practical limits imposed by sequence dimensionality and oracle costs.
Read the full abstract ▾
All authors Jackie Rao, Ferran Gonzalez Hernandez, Leon Gerard, Alexandra Gessner
Statistical Methodology
Improving Treatment Effect Estimation in Trials through Adaptive Borrowing of External Controls
Qinwei Yang, Jingyi Li, Peng Wu et al. (+1 more)
Randomized controlled trials (RCTs) often suffer from limited inferential efficiency in estimating treatment effects due to their small sample sizes. In recent years, incorporating external controls (ECs) has gained increasing
attention as an effective way to augment small RCTs and thereby enhance estimation efficiency. However, ECs are not always comparable to RCTs, and direct borrowing without careful evaluation can introduce substantial bias and, paradoxically, undermine the accuracy of treatment effect estimation. In this paper, we propose a novel adaptive influence-based sample borrowing framework to improve average treatment effect (ATE) estimation in RCTs. The framework quantifies the “comparability” of each sample in ECs using influence functions and identifies the optimal subset of ECs that minimizes the mean squared error of the ATE
estimator. The proposed framework is assumption-lean regarding the distribution of ECs and is robust to outliers, making it broadly applicable across diverse settings. Moreover, we develop an outcome calibration method to improve the data utilization efficiency of ECs, further strengthening the adaptive influence-based sample-borrowing framework. We demonstrate the effectiveness of the proposed method using both simulated and real-world
datasets.
Read the full abstract ▾
All authors Qinwei Yang, Jingyi Li, Peng Wu, Shu Yang
Statistical Methodology
High-Dimensional Data Analysis for Elliptically Symmetric Distributions
Long Feng
High-dimensional data arise routinely in modern statistics, econometrics, finance, genomics, and machine learning. While a large body of existing methodology is developed under Gaussian or light-tailed assumptions, many real data sets exhibit heavy tails, heterogeneity, and departures from classical covariance-based models. This book provides a systematic treatment of high-dimensional data analysis under elliptically symmetric distributions, with an emphasis on robust inference based on spatial signs, spatial ranks, multivariate Kendall's tau
matrices, and related shape-based methods.The book covers the basic theory of elliptical
symmetry, high-dimensional location inference, estimation and testing for covariance and precision
matrices, sphericity and proportionality testing, high-dimensional alpha testing in factor pricing models, change-point analysis, white-noise and independence testing, high-dimensional discriminant analysis, and dimension reduction through principal component analysis and factor models. Throughout, we review classical low-dimensional and high-dimensional
benchmark methods and then develop robust alternatives tailored to elliptical models. Particular
attention is paid to the interplay between sum-type, max-type, and adaptive procedures, as well as to the role of scatter, shape, and rank-based dependence measures in heavy-tailed settings. This book is intended as a unified overview of robust high-dimensional methods under elliptical
symmetry and as a synthesis of the author's recent research contributions in this area. It is written for researchers and graduate students in statistics, econometrics, and related fields who are interested in modern high-dimensional inference beyond the Gaussian paradigm.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from physics.soc-ph
Sandpile Economics: Theory, Identification, and Evidence
Diego Vallarino
Why do capitalist economies recurrently generate crises whose severity is disproportionate to the size of the triggering shock? This paper proposes a structural answer grounded in the evolutionary geometry of production networks. As economies evolve through specialization, integration, and competitive selection, their inter-sectoral linkages drift toward configurations of increasing
geometric fragility, eventually crossing a threshold beyond which small disturbances generate disproportionately large cascades. We introduce Sandpile Economics, a formal framework that interprets macroeconomic instability as an emergent property of disequilibrium production networks. The key state variable is the Forman–
Ricci curvature of the input–output graph, capturing local substitution possibilities when supply chains are disrupted. We show that when
curvature falls below an endogenous threshold, the distribution of cascade sizes follows a power law with tail index $\alpha \in (1,2)$, implying a regime of unbounded amplification. The underlying mechanism is evolutionary: specialization reduces input substitutability, pushing the economy toward criticality, while crisis episodes induce endogenous network reconfiguration and path dependence. These dynamics are inherently non-ergodic and cannot be captured by representative-
agent frameworks. Empirically, using global input–output data, we document that production networks operate in persistently negative
curvature regimes and that
curvature robustly predicts medium-run output dynamics. A one-standard-deviation increase in
curvature is associated with higher cumulative growth over three-year horizons, and
curvature systematically outperforms standard network metrics in explaining cross-country differences in resilience.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.LG
Simulation-Based Optimisation of Batting Order and Bowling Plans in T20 Cricket
Tinniam V Ganesh
This paper develops a unified Markov Decision Process (MDP) framework for optimising two recurring in-match decisions in T20 cricket, namely batting order selection and bowling plan assignment, directly in terms of win and defend probability rather than expected runs. A three-phase player profile engine (Powerplay, Middle, Death) with James-Stein shrinkage (a technique that blends a player's individual statistics toward the league average when their phase-specific data is sparse) is estimated from 1,161 IPL ball-by-ball records (2008-2025). Win/defend probabilities are evaluated using vectorised Monte Carlo simulation over N = 50,000 innings trajectories. Batting orders are evaluated by comparing all feasible arrangements of the remaining players and selecting the one that maximises win probability. Bowling plans are optimised through a guided search over possible over assignments, progressively improving the allocation while respecting constraints such as the prohibition on consecutive overs by the same bowler. Applied to two 2026 IPL matches, the optimal batting order improves Mumbai Indians' win probability by 4.1 percentage points (52.4% to 56.5%), and the optimal Gujarat Titans bowling plan improves defend probability by 5.2 percentage points (39.1% to 44.3%). In both cases, the observed sub-optimality is consistent with phase-agnostic deployment: decisions that appear reasonable under aggregate metrics are shown to be costly when phase-specific profiles are applied.
Read the full abstract ▾
Statistics Theory
The Integer-valued Moving-Average Random Field
Angelika Silbernagel and Christian H. Weiß
An integer-valued moving average (INMA) model for count random fields is proposed and investigated. Closed-form expressions are derived for both its marginal distribution and spatial dependence structure, for arbitrary model order and also covering the multilateral case. In particular, general expressions for bivariate distributions and autocovariances are provided. It is shown that the INMA random field can be equipped (among others) with a Poisson marginal distribution. It is also demonstrated that different and well-interpretable dependence structures are possible. For illustration, we discuss a real-world data example and propose an INMA approximation to a given spatial dependence structure.
Read the full abstract ▾
Statistical Methodology
Testing Alpha in High-Dimensional Conditional Time-Varying Factor Models with Dependent Observations
Long Feng, Huifang Ma and Zhaojun Wang
This paper studies alpha testing in a high-dimensional conditional time-varying factor model with temporally dependent observations. Both factor loadings and alpha processes are allowed to vary smoothly over time, and the cross-sectional dimension may be comparable to or larger than the sample size. Using a B-spline sieve method, we develop a sum-type test for dense alternatives, a max-type test for sparse alternatives, and a Cauchy combination test for adaptive inference. On the theoretical side, we derive explicit
stochastic expansions for the estimated average alphas, establish asymptotic normality of the sum statistic, and develop the extreme-value limit theory for the max statistic by showing its Gumbel convergence under temporal dependence together with the validity of block-bootstrap calibration. We further prove asymptotic independence between the sum and max statistics and thereby justify the Cauchy combination test. Simulation results demonstrate that the proposed procedures achieve satisfactory size control and competitive power across a wide range of dense and sparse alternatives. An empirical application further illustrates the usefulness of the proposed methods in testing asset-pricing models with time-varying structure.
Read the full abstract ▾