Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics Statistical Methodology

stat.ME Watch Statistical Methodology (sign in to use)

Tuesday 6 October

Statistical Methodology · Cross-listed from math.ST

Asymptotic Null Distributions of Moran's $I$ and Assortativity in Large Networks

This study investigates the asymptotic behavior of two dependence measures defined on networks, Moran's $I$ statistic and Newman's assortativity, under the null hypothesis that a Gaussian node attribute $Y$ is independent of the network structure. We demonstrate that the structure of the network directly affects the convergence rate to normality of these measures as the size of the network increases. We further establish that, in some instances, the mean values of these dependence measures under the null hypothesis remain non-negligible asymptotically and must therefore be explicitly accounted for when calculating the test statistics. Applications to a variety of simulated and real networks also reveal that the normal approximation performs well only when the network is not strongly heterogeneous. Network topology determines both the convergence rate to normality and whether the limiting distribution is Gaussian. In dense networks whose degree heterogeneity does not vanish, we further show that assortativity can fail to be a valid test statistic even though Moran's $I$ remains well behaved, whereas a dominating node invalidates both.
Read the full abstract

Statistical Methodology

When does conformal calibration need censoring weights? Cause-of-failure prediction sets under competing risks

Split conformal prediction sets for competing-risks labels at a fixed horizon require calibration labels that right censoring can leave unobserved. Complete-case calibration guarantees coverage for the label-complete subpopulation, but its population coverage can deviate in either direction, even at the true class probabilities. We study how selection changes the score distribution near the population quantile. At the true score in our main simulation family, with independent draws, complete-case calibration covers 0.8723 at a nominal 0.900 when 22% of subjects are event-free at the horizon. In 4 of 48 further designs using per-draw normalisation, estimating cause incidences as one minus the exponential of the negative of the cause-specific Nelson-Aalen cumulative hazards, with the event-free probability as the clipped and renormalised remainder, puts complete-case coverage at least four standard errors below nominal. In these designs, weights from a correctly specified censoring model keep coverage near nominal. Extending the argument of Yi et al. (2025) to a cause label, we establish a finite-sample coverage lower bound with an explicit penalty for censoring-model error. Misspecifying the censoring model can also lead to under-coverage.
Read the full abstract

Statistical Methodology · Cross-listed from econ.EM

Measuring Gift Card Program Incrementality via Causal Data Fusion

Businesses regularly offer gift card programs to drive customer spending and increase engagement. A central question is how much incremental revenue these programs generate, and which channels drive it most efficiently. Measuring the incremental revenue associated with a gift card program is a challenging problem in causal inference, requiring a firm to infer how much each customer would have spent if they never received a gift card. Observational data on past customer purchasing behavior reveal possession of a gift card only when a customer makes a purchase, thus leaving a customer's treatment status systematically censored. In this paper, we develop a novel data fusion approach to overcome this missing data challenge. We identify and estimate incrementality by combining a large observational dataset with a smaller experimental dataset from a different population. Our approach relies on a mild transferability condition, which posits that the conditional relative treatment effect of gift card receipt on the decision to purchase is invariant across the two populations. We develop a flexible, machine learning-based estimator for the incremental revenue and establish its asymptotic normality. We apply our estimator across both first- and third-party channels through which Airbnb distributes gift cards, finding heterogeneity in incrementality across segments of the population. In particular, we find not only that third-party channels are more incremental than first-party ones, but also that "self-gifters" (i.e., customers likely to have purchased their own gift cards) are more incremental than the broader population.
Read the full abstract

All authors Justin Whitehouse, William Betz, Yan Zhang, Peter Coles, Ramesh Johari, Vasilis Syrgkanis

Statistical Methodology

Fridge: focused fine-tuning of ridge regression for personalize predictions

Statistical prediction methods typically require some form of fine-tuning of tuning parameter(s), with $K$-fold cross-validation as the canonical procedure. For ridge regression there exist numerous procedures, but common for all, including cross-validation, is that one single parameter is chosen for all future predictions. We propose instead to calculate a unique tuning parameter for each individual for which we wish to predict an outcome. This generates an individualized prediction by focusing on the vector of covariates of a specific individual. The focused ridge – fridge – procedure is introduced with a two-part contribution: 1) first we define an oracle tuning parameter minimizing the mean squared prediction error of a specific covariate vector, 2) then we propose to estimate this tuning parameter by using plug-in estimates of the regression coefficients and error variance parameter. The procedure is extended to logistic ridge regression by utilizing parametric bootstrap. For high-dimensional data, we propose to use ridge regression with cross-validation as the plug-in estimate, and simulations show that fridge gives smaller average prediction error than ridge with cross-validation for both simulated and real data. We illustrate the new concept for both linear and logistic regression models in two applications of personalized medicine: predicting individual risk and treatment response based on gene expression data. The method is implemented in the R package "fridge".
Read the full abstract

Statistical Methodology

Elastic kernel Ridge regression, with applications in phonetics

Predicting the shapes of entire curves requires accounting for nonlinear geometry and unknown alignments between curves. We develop elastic kernel ridge regression, a nonparametric method for planar curve responses with scalar, multivariate, or functional covariates. Using square-root velocity representations, we formulate penalized conditional Fréchet mean estimation in a shape space invariant to translation, rotation, scaling, and reparametrization. A vector-valued reproducing kernel Hilbert space provides a flexible nonlinear link through the spherical exponential map. We use an alternating algorithm to align each observed curve to its current fitted value, and updates the regression function. To facilitate this, we provide a new Euclidean quasi-Newton solver that exploits scale invariance of the reparametrization objective; this accelerates alignment while retaining accuracy in a numerical comparison. Simulations demonstrate the benefits of estimating alignment within the regression and respecting spherical geometry. Applied to vocal tract contours from a real-time magnetic resonance imaging recording, the method recovers missing frames and reconstructs shape trajectories from downsampled data. A speech inversion proof of concept on the same recording predicts tongue shapes from acoustic features, illustrating the method's potential. The method is implemented in the R package sphereg2.
Read the full abstract

All authors Beatrice Matteo, Almond Stoecker, Davide Pigoli, Shahin Tavakoli

Statistical Methodology · Cross-listed from stat.ML

High-Dimensional Statistical Inference for Sparse Support Vector Machines

Using a replica-symmetric high-dimensional characterization, we develop an inferential framework for sparse support vector machines when the sample size and number of features grow proportionally. The main challenge is the nonsmooth hinge loss, which prevents direct application of debiasing arguments developed for smooth classification losses. We overcome this difficulty by representing the $L_1$-penalized support vector machine (SVM) as a linear program and identifying the hinge-loss subgradient through its dual variables. This yields a computationally accessible debiased estimator whose coordinates are asymptotically Gaussian under the proportional asymptotic regime. The resulting distributional characterization provides confidence intervals and hypothesis tests for individual features and enables false-discovery-rate-controlled variable selection. Extensive simulations examine calibration, power, and variable-selection performance under a range of covariance structures, including strongly correlated designs. An analysis of high-dimensional breast cancer gene-expression data illustrates how the proposed inference can distinguish statistically significant features from variables selected by the original sparse SVM.
Read the full abstract

Statistical Methodology

Covariate-dependent Nonparametric $g$-modeling for regression via infinite Mixture-of-Expertizing class

Empirical Bayes $g$-modeling captures unit-level heterogeneity by estimating a latent prior distribution from observed data. In the existing formulations, however, the prior is shared by all units. In this paper, we develop a covariate-dependent g-modeling framework for regression in which the entire prior distribution of the regression coefficients is allowed to depend on covariates. We formulate the estimation of the prior as nonparametric maximum likelihood estimation (NPMLE) of the covariate-dependent prior, and show that the unrestricted problem is ill-posed. To resolve this, we introduce the infinite Mixture-of-Expertizing class of conditional priors, under which the NPMLE is precisely a softmax-gated Mixture of Experts (MoE) whose number of experts is not fixed in advance but is determined by the data. Building on a first-order optimality condition, we propose two exemplar-based estimation algorithms that select experts automatically, together with a post-hoc aggregation of experts for interpretation. On the theoretical side, we show that every conditional prior in the class is Lipschitz continuous in the covariates, and that aggregated softmax gates can approximate any continuous gate function. The effectiveness of the proposed NPMLE is shown through application to synthetic datasets and real datasets.
Read the full abstract

Statistical Methodology

Bayesian heterogeneous copula mixtures with nonparametric margins: consistency, identifiability and tail asymmetry in physical fitness data

Finite mixtures of Clayton, Gumbel, Frank and Gaussian copulas can describe dependence that differs between the upper and lower tails. We study a two-stage Bayesian analysis in which ranks or kernel estimates replace the margins and the copula likelihood is evaluated at the resulting pseudo-observations. This pseudo-posterior is strongly consistent for the copula density whenever the log density admits a logarithmic boundary envelope. Both stages may use the same data, margins may be standardized within observed strata, and neither smoothness nor identifiability is required. The envelope holds for finite mixtures of Gaussian, Student, Clayton, Gumbel and Frank copulas in any fixed dimension; tail-dependence coefficients and conditional tail probabilities are therefore consistently estimated. We further prove that Clayton, Gumbel, Frank and Gaussian copulas are jointly finitely linearly independent, which makes mixture weights and components identifiable and consistently estimated. In simulations the pseudo-posterior matches multi-start maximum pseudo-likelihood in large samples. It is more stable in small samples and near independence (every component close to the independence copula), where tail coefficients are learned long before the weights and a marginal Metropolis sampler is up to twice as efficient as data augmentation. Two physical fitness datasets show mirror-image asymmetries. Among 8772 university students, sprint and jump performance are coupled mainly at the top. Among 5336 adults in a national health survey, low grip strength and low daily activity cluster together, increasingly with age, whereas high values do not. A Gaussian copula misses both patterns in held-out data.
Read the full abstract

Statistical Methodology

Where Do Two Populations of Persistence Diagrams Differ? Calibrated Local Inference at a Fixed Budget

Many two-sample tests for populations of persistence diagrams assess global differences without identifying the regions of the birth-death plane that contribute to them. We study simultaneous inference for local mean contrasts when the number of available diagrams is fixed. They are differences in expected weighted feature mass within $\ell_\infty$ neighborhoods at several centers and radii. We estimate these contrasts using additive landmark responses. A Gaussian multiplier bootstrap calibrates simultaneous confidence intervals while allowing unequal group covariances. The neighborhoods whose intervals exclude zero form a map with approximate family-wise error control, and selecting a subset of original intervals for display preserves their joint coverage guarantee. On the simultaneous coverage event, every reported neighborhood lies within twice its radius of the support of the mean-measure difference. A geometric result gives sufficient radius conditions for a displaced feature to produce a nonzero contrast. A comparison of sufficient detection thresholds quantifies the tradeoff between reducing the number of tested coordinates and reserving observations for an independent pilot. In simulations with 40 to 120 diagrams per class, the bands achieved 94%-98% simultaneous coverage under both the strict null and equal means with unequal covariances. In the latter setting, a permutation maximum and the pooled-t implementation of the two-stage persistence-image test of Moon and Lazar rejected in up to 32% and 26% of runs, respectively. In the fixed-budget simulations, spending a third of the observations on a pilot to choose landmarks or radii located changes less often than a prespecified grid at a single radius. On the MUTAG benchmark, the localized region concentrates on rings of fused-ring systems, an exploratory reading.
Read the full abstract

All authors Pramita Bagchi, Edward Bae, Atish Mitra, Alexander D. Silberman, Žiga Virk, Sushovan Majhi

Statistical Methodology · Cross-listed from stat.ML

Anytime-valid simulation-based hypothesis testing

For a given data distribution $(X_t)_{t \in \mathbb{N}} \sim Q$ i.i.d., we investigate the hypothesis testing problem: $H_0: Q = P_0$ vs. $H_1: Q = P_1$, for two different model probability distributions $P_0$ and $P_1$. In contrast to the standard setting, where analytic densities $p_0$ and $p_1$ are given, here, we consider the density-free setting, where we only have access to i.i.d. simulations $(Z^0_t)_{t \in \mathbb{N}} \sim P_0$ and $(Z^1_t)_{t \in \mathbb{N}} \sim P_1$. For this simulation-based hypothesis testing setting, we construct an e-test martingale, resulting in a sequential test with anytime-valid type-I error guarantees, approximate growth optimality, geometrically decaying type-II error bounds, and asymptotic power one. Most ingredients used in our constructions are variants of well known concepts. The value of this paper lies in the compact presentation of an effective, anytime-valid solution for the density-free simulation-based sequential hypothesis testing case.
Read the full abstract

Statistical Methodology

The Missing Corner: When Sharp Causal Bounds Fail to Compose

Sharp bounds on two potential-outcome means need not combine into sharp bounds on their difference. We characterize exactly when composition is guaranteed across environments sharing a joint potential-outcome law. For binary treatment and outcome, fix strict selection-probability bounds and interior treatment propensities. Subtracting the full model's sharp mean intervals gives sharp average-treatment-effect bounds for every compatible observed law if and only if the absolute selection bounds coincide across environments. Under this alignment, the complete mean region is rectangular; its endpoints and attaining causal models require $O(K)$ arithmetic operations for $K$ environments. For every mismatch, failure occurs on a relatively open set of positive observed tables at the prescribed propensities and persists under small joint drift. We give computable perturbation certificates and an exact family whose gap is first order in the mismatch. A separate finite-confounding family shows when the lost joint information changes an identified causal sign. These results distinguish local marginal sharpness from joint attainability after combining environments.
Read the full abstract

Statistical Methodology · Cross-listed from econ.EM

The Noise Is the Signal: Correlated Sampling Error Is Rank-Informative for Proxy Metric Selection

North-star metrics such as customer lifetime value are often too slow and noisy to decide a short A/B test. Teams therefore rely on a proxy metric, commonly chosen by how closely its effects tracked the north star's across past experiments. Validating that choice, or any method for making it, is hard: the only benchmark is the noisy north star, and the number of available past experiments is limited. In addition, proxy and north-star effects are estimated on the same customers, so their sampling errors are correlated. Recent work at major experimentation platforms removes this shared error as contamination, improving estimates of the true-effect covariance. Choosing a proxy, however, is a ranking problem, and a better estimate need not give a better ranking. We measure agreement free of shared error by estimating the two effects on disjoint random halves of each experiment's customers. In an archive of 262 experiments and 69 candidate proxies, the shared error ranks the candidates in a similar order to this agreement (Spearman correlation 0.65): it carries information about proxy quality. The more of it a correction removes, the worse the ranking because removal discards part of the signal but leaves the main sources of ranking noise, the noisy north star and the limited number of experiments, untouched. Archive-calibrated simulations, in which the correct ranking is known, confirm this even when every correction receives the true sampling covariance. Held-out real experiments, evaluated on disjoint customer halves so that shared error cannot bias the comparison, closely reproduce the predicted ordering (Spearman correlation 0.93). Correction can still pay off with more experiments, but the number needed rises steeply with the north star's noise. We map this crossover and give platform teams three inexpensive checks for deciding from their own archive whether and how strongly to correct.
Read the full abstract

Statistical Methodology

TEASE: Targeted elastic spatial envelope

Envelope regression is a crucial part of multivariate linear modeling, leveraging separation into material and immaterial parts to provide parsimonious data reduction. Rekabdarkolaee et al. (2020) construct spatial envelope, a novel multivariate Gaussian process, extended by May et al. (2022) to linear coregionalization envelope. Our proposal (TEASE) combines spatial envelope and full-scale basis graphical lasso (LeDuc et al., 2025), leveraging targeted graphical elastic net (Kovács et al., 2021) for immaterial precision matrices. Material part is a multivariate multiresolution Gaussian Markov random field (Kleiber et al., 2019; Caringi and Secchi, 2026). Enhanced envelope penalty (Kwon and Zou, 2025) regularizes regression coefficients. Demonstrations are on climate data.
Read the full abstract

Statistical Methodology · Cross-listed from cs.DC

Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations

Concept drift threatens production machine learning, yet the empirical behavior of multivariate two-sample drift detectors at scale remains under-characterized. Existing benchmarks rarely address the hundreds of millions of rows and high-cardinality features typical of industrial-operational datasets. We evaluate five multi-column two-sample tests (marginal, projection-based, and kernel embedding methods) across three complementary environments: the Harvard Dataverse, a validated Failing Loudly reproduction (mean absolute error between 0.030 and 0.053), and a novel synthetic-injection benchmark on the 137.5-million-row Trendyol collection-ranking feature table. Testing four drift types across two severity-scope regimes, we demonstrate that distributed Maximum Mean Discrepancy with Random Fourier Features on Apache Spark scales robustly. Averaged over the four drift types in the strong regime and under a calibrated threshold, it achieves a Pearson correlation of r = 0.940 with expected drift magnitude, an 80.4% true positive rate, and a 3.2% false positive rate. Conversely, the per-dimension Kolmogorov-Smirnov test failed due to statistic saturation from ID-like columns under asymmetric sampling, establishing a critical constraint for large-scale sampling design. At weak configurations (realized-flip fractions of at most 0.57%), detectors struggled to reliably discriminate, highlighting the need for future intensity-grid power analyses to distinguish fundamental sensitivity bounds from scalable threshold shifts.
Read the full abstract

All authors Cagdas Pullu, Mahmut Emir Arslan, Bugra Balkac, Aylin Ondersev Balta, Cihangir Celal Palaci, Fikri Cem Yilmaz, Altan Cakir

Statistical Methodology

The Geometry of Existence and Uniqueness of Maximum Likelihood Estimation in Categorical Response Models

Nonexistence of the maximum likelihood estimate (MLE) under separation is treated as a solved problem for binary logistic regression and as a scattered collection of model-specific results everywhere else. We show that it is one phenomenon with one criterion. In a latent polyhedral categorical response model, every observed outcome corresponds to a polyhedral event in latent variables whose faces shift linearly with the parameter. Random-utility choice, cumulative-link, ranking, multivariate binary and ordinal, sequential, adjacent-category logit, and fixed-score stereotype models belong to this family. The likelihood sees each observed factor only through the columns of its threshold map, the structure vectors. A finite MLE exists if and only if the pooled structure vector set has overlap. Sufficiency requires only continuity, necessity requires only strictly threshold-increasing probabilities, and neither likelihood concavity, exchangeability, nor full design rank is needed. Positive strictly log-concave latent densities and threshold-identifiable polyhedra then yield uniqueness on the estimable span. A single linear program determines whether overlap holds, and convex cone geometry measures the dimension of separation. An application using cumulative-link models for willingness to share health data identifies observations and model terms associated with nonexistence and illustrates how the diagnostics inform model revision and sensitivity analysis.
Read the full abstract

Statistical Methodology

What Should We Measure Next? Finding Identification Strategies by Refining Mechanisms

Canonical approaches in causal inference treat model specification as fixed, assuming that researchers directly translate all relevant domain knowledge into a causal model, which can then be used to deduce its logical implications. Yet, in practical applications, model specification is often an iterative process, and involves exploration and introspection: of the many aspects of the phenomenon one could investigate, which ones actually matter for the identification of the causal effect of interest? In this paper we study the problem of iterative identification in partially specified causal models. We focus on determining where observing variables that intercept a direct effect or a confounding path between two variables could enable identification in semi-Markovian models. We give necessary and sufficient graphical conditions for when observing such variables can render an unidentifiable query identifiable, together with an efficient algorithm for locating all such opportunities in a given causal diagram. Our results can help analysts better navigate the model space by drawing attention to the parts of their substantive knowledge that could result in a successful identification strategy.
Read the full abstract

All authors Mikko Väänänen, Fanyu Cui, Carlos Cinelli, Santtu Tikka

Statistical Methodology

Collaborative representations for targeted causal inference under outcome selection

Estimating causal effects with missing outcomes requires learning outcome, exposure, and selection models. Flexible propensity learners can predict treatment or missingness well while worsening target estimation by emphasizing instruments, weak-overlap regions, or variation unrelated to outcome-regression bias. We propose a collaborative representation-based targeted learning method for recovered average treatment effects under outcome selection. The method learns low-dimensional exposure and selection representations from cross-fitted pseudo-outcomes that encode outcome-regression drift. A finite candidate library alternates targeted outcome updates with representation updates, and inner cross-validation selects representation complexity, collaboration strength, and regularization using a target-aware risk. We give high-level sufficient conditions for collaborative robustness, consistency, and asymptotic linearity after outer validation-fold targeting. Simulations show finite-sample bias reduction, with explicit bias-variance trade-offs across configurations and relative to competing estimators
Read the full abstract

Statistical Methodology

The Topp-Leone XLindley Distribution: Properties, Estimation and Applications to Lifetime Data

In this article, we use a family of distributions developed by Topp-Leone to create a novel lifetime two-parameter distribution. The Topp-Leone XLindley distribution is the name used for it. We study several types of statistical and mathematical characteristics of this distribution, such as reliability functions, moments, moment-generating function, quantile function, and the Renyi entropy. For the purpose of estimating parameters under the proposed distribution model, this simulation study is conducted to evaluate the performance of the maximum likelihood estimates for the parameters of the Topp-Leone XLindley distribution. A comprehensive simulation investigation is carried out to evaluate the performance of the proposed methods. Furthermore, a practical data set of bank customer waiting times has been investigated for the purpose of illustration.
Read the full abstract

Statistical Methodology

Scope-Restricted Backtracking Counterfactuals

Interventional counterfactuals evaluate statements such as "had $A$ been $a$" by replacing the assignment for $A$ while keeping the unit's exogenous background fixed. Backtracking counterfactuals reverse this logic: they preserve the structural assignments and instead vary background conditions across worlds. This semantics is natural for diagnostic explanation, but full backtracking may be too permissive because every exogenous coordinate can vary. We introduce scope-restricted backtracking counterfactuals, in which the modeler specifies which coordinates may vary while the remainder are shared across worlds. The scope determines the admissible carriers of the contrast, while a cross-world kernel governs the plausibility of their alternative values. We present model-based inference for general scope-restricted backtracking counterfactuals, derive sharp marginal bounds for a specified explanatory query, and establish frontier and compatibility conditions under which this explanatory query admits a modified-value policy representation. A linear-Gaussian example distinguishes local-noise, ancestor, irrelevant-noise, and infeasible explanations across scopes, while an application to the UCI Adult benchmark illustrates kernel specification, sensitivity, and how much of the explanation is carried by the cross-world coupling itself.
Read the full abstract

Statistical Methodology · Cross-listed from q-fin.RM

A Functional Representation of Credit Behavior for Probability of Default Modeling

This paper proposes a framework for modeling probability of default via functional data analysis. By representing a series of credit variables as functions, we investigate whether intra-monthly information improves default predictability in linear models. We further show how a range of widely used variables, among them available funds, utilization rate, and overdraft, can all be derived from three quantities observed over time. Namely, 1) the type of each account, 2) the balance of the account, and 3) the size of the credit limit on the account. Retaining these quantities as continuous-time processes, rather than reducing them to monthly aggregated values, yields a continuous faithful representation of the borrower. When analyzing these three processes, we discovered that recurring events associated with the ordinal position among banking days created strong cyclical patterns. We therefore develop a relative time framework that aligns the recurring events across borrowers, ensuring that borrowers possess the same cyclical pattern, regardless of real time. We assess the framework using functional logistic regression. This approach accommodates the continuous representation while remaining closely related to a logistic regression model commonly used in credit risk practice, due to strict regulatory constraints. We show that, when equipped with an effective functional representation of transactional trajectories, the proposed model attains predictive performance on par with XGBoost while consistently outperforming logistic regression. Importantly, the model balances predictive performance of a machine learning model with the interpretability of linear default models, potentially enabling financial institutions to use the model, even under strict regulation.
Read the full abstract

All authors Jonas Brunholm, Bjarne Højgaard, Thomas D. Nielsen, Orimar Sauri

Statistical Methodology

State-space representation of early-life child mortality dynamics: linking event history analysis and orbit-based models

Understanding early-life child mortality requires methods that capture both statistically significant risk factors and the structure of individual life trajectories. Event History Analysis (EHA) estimates associations between covariates and mortality risk but may obscure rare or high-dimensional configurations of risk. Orbit Theory (OT) represents individuals as exact trajectories in a combinatorial state space, preserving full multivariate structure. We analyse longitudinal data across 12 variables for 31,081 children from the Agincourt Health and Demographic Surveillance System (South Africa, 1998-2008) using both methods. We show three things EHA alone cannot. First, maternal migration is structurally pivotal in the transition dynamics leading to child death despite being statistically non-significant in EHA. Second, maternal refugee status, while statistically significant, generates no trajectory dynamics and is correctly excluded from any structural reduction. Third, a three-variable subsystem (child status, mother status, mother migration) recovers EHA's principal findings within a reduced state space of at most 48 states while exposing small high-risk clusters invisible to regression-based analysis. Maternal death is the dominant EHA predictor; maternal migration organises 87% of the observable sequential transitions into child-death states. These are not contradictory: maternal death rarely appears as a preceding state because its role is terminal rather than sequential. Early-life mortality can be read at two levels simultaneously - statistically, through effect estimates, and structurally, through observed transitions.
Read the full abstract

Statistical Methodology

Design-Driven Inference for Online Experiments: Jointly Optimal Evidence Collection, Testing, and Estimation

Online multifactorial experiments requires early stopping for practically negligible effects while controlling false stopping under continuous monitoring and retaining precise treatment-effect estimation. We develop a design-driven framework that chooses allocation and the evidence rule jointly, treating allocation as evidence collection and testing and estimation as complementary uses of the same evidence. For multifactor experiments with nuisance block effects and treatment-by-block interactions, we show that block-orthogonal designs remove nuisance contamination from the treatment score and form a complete class for worst-case evidence growth. Under a fixed information budget, an isotropic allocation and an explicit radial e-value jointly attain the minimax rate optimal for directionally unknown alternatives. The same allocation is universally optimal for maximum likelihood estimation of treatment effects, simultaneously achieving A-, D-, and E-optimality and establishing double optimality for testing and estimation. Batchwise replication yields an anytime-valid e-process, retains minimax rate optimality for cumulative conditional worst-case evidence growth, and preserves universal estimation optimality for the prespecified complete experiment. Simulations and a large language model prompt experiment illustrate gains in stopping efficiency and estimation precision.
Read the full abstract

All authors Jun Yu, Wenbiao, Zhao, Lixing, Zhu

Statistical Methodology · Cross-listed from stat.ML

Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects

Large language models (LLMs) are increasingly used as judges for automated AI evaluation. A common practice is to randomize prompt sequences and average the resulting scores, but its statistical validity remains unclear. We show that LLM evaluation mechanisms can be approximated by a class of Markov generalized linear mixed models (GLMMs), supported by out-of-sample predictions across three major commercial LLMs. Using a first-order Markov GLMM, we study leaderboard ranking and group comparison. For leaderboard ranking, randomize-and-average selection is consistent under a mild separation condition, and a Williams square design can improve efficiency when item qualities are close. For group comparison, naive averaging can yield inconsistent conclusions about differences in group-level quality because of the response model's nonlinearity. Empirical results further support the validity of the proposed model-based inference beyond the first-order theory, including settings with higher-order sequence memory. We illustrate the approach in an application where AI judges compare two graphical model estimation methods.
Read the full abstract

Statistical Methodology

Measuring Negotiation Traits with Collaborative Choice Data: A Two-Stage Latent Variable Model

Consider a collaborative decision-making setting in which team members with different reward structures need to make a joint decision. Repeated collaborative decisions can reflect individual differences in team members' negotiation ability. However, extracting such information is challenging because each observed response arises from a collaborative process involving multiple participants and their reward structures. In this paper, we propose a two-stage probabilistic measurement model for structured collaborative choice tasks. In the first stage, known member-specific rewards and latent participant negotiation traits jointly determine team-option utilities, from which a latent team choice is generated. In the second stage, individual responses are modeled conditional on the latent team choice, with possible deviations that depend on participants' own rewards. Participant-level covariates can also be incorporated through a structural model. A simulation study shows satisfactory recovery of the participant traits and model parameters across different sample sizes and item sizes. An application to data from simulated collaborative negotiation tasks shows choice prediction accuracy well above the random-choice baseline and meaningful associations between the estimated traits and external criterion variables. The proposed framework provides a psychometric approach to measuring participant-level negotiation traits with collaborative choice data.
Read the full abstract

All authors Tongxin Zhang, Yunxiao Chen, Jiangang Hao, Emily Kerzabi, Patrick Kyllonen, Irini Moustaki, Tao Xin

Monday 5 October

Statistical Methodology

Debiased Machine Learning for Count Data: a Partially Linear Poisson Model Based on Neural Networks

We develop a novel debiased machine learning (DML) estimator to analyze count data, which are common in clinical or biomedical studies. Specifically, we estimate the effect of a binary treatment on count outcomes using a partially linear Poisson model. A key advantage of this model is that it represents the complex covariate structure through a nuisance function, which is estimated flexibly using neural networks. We use a Neyman-orthogonal score function to construct the estimator, reducing its sensitivity to errors in estimating the nuisance function. Cross-fitting is used to mitigate overfitting bias in neural networks. Under mild regularity conditions, this DML estimator is asymptotically normal with root-$n$ convergence. We derive a closed-form variance estimator and construct a Wald confidence interval for the treatment effect. Extensive simulations demonstrate that the proposed estimator reduces bias and root mean square error compared with the generalized linear model and the augmented inverse probability weighting (AIPW) estimator in most settings, while achieving a coverage probability near the nominal level. The proposed procedure is implemented in the R package $\texttt{PoissonDML}$. We apply our approach to a synthetic HIV cohort data to investigate the effect of treatment on AIDS-defining event counts of 7,141 analyzed patients. Our approach provides practical and valid estimation and inference for treatment effects under the partially linear Poisson model while combining modern machine learning methods to estimate complex covariate structures.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

Active Feature Acquisition for Cost-Efficient Temporal Prediction with Reduced Participant Burden

Accurate forecasting of pathological outcomes is a central problem in psychology. To do so, psychologists often collect intensive longitudinal data. However, in such studies, the desire to acquire a large number of variables for the sake of accurate prediction is often counteracted by the need to minimize participant burden. Acquiring more variables per occasion can yield better predictions, but having too many acquisitions increase the risk of non-response and attrition. Longitudinal Active Feature Acquisition (LAFA) is a principled approach to resolve this conundrum. Instead of requiring responses to every item at every acquisition occasion, LAFA produces a policy that seeks to optimally select dynamic subsets of items to be acquired at each timepoint while preserving our ability to forecast a specific outcome. However, existing LAFA methods are mostly based on Neural Networks (NN) that are difficult to interpret in practice. In this work, we introduce a tree distillation method for learning an interpretable policy from NN-based LAFA networks. We validated our method through both a simulation and an empirical EMA dataset on forecasting daily alcohol consumption. In both cases, we find that we can meaningfully reduce the number of items acquired at each occasion with minimal loss in accuracy. Networks (NN) that are difficult to interpret in practice. In this work, we introduce a tree distillation method for learning an interpretable policy from NN-based LAFA networks. We validated our method through both a simulation and an empirical EMA dataset on forecasting daily alcohol consumption. In both cases, we find that we can meaningfully reduce the number of items acquired at each occasion with minimal loss in accuracy.
Read the full abstract

All authors Yunni Qu, Bing Cai Kok, Whitney Ringwald, Grant King, Aidan Wright, Kathleen Gates, Junier Oliva

Statistical Methodology

Large-scale linear hypothesis testing for high-dimensional online M-estimation

With the growing prevalence of streaming data, renewable estimation has become an important tool for updating statistical analyses without retaining historical raw observations. We study estimation and linear hypothesis testing in high-dimensional online M-estimation. Existing renewable procedures based on quadratic approximations can incur first-order approximation errors because the gradient of the unpenalized loss generally does not vanish at a preceding penalized estimator, posing a particular challenge for valid inference. To address this issue, we propose a corrected renewable loss that preserves historical first-order information and combine it with partial folded concave regularization and an iterative local linear approximation. We establish finite sample $\ell_1$- and $\ell_2$-error bounds, together with contraction bounds and strong oracle properties for both the constrained and unconstrained iterative estimators. Building on these results, we derive oracle Bahadur representations and establish asymptotically valid Wald and score tests for general linear hypotheses, allowing both the testing-oracle dimension and the number of restrictions to diverge. We further characterize their asymptotic power under local alternatives. The proposed procedure requires only recursively updated summaries while recovering the first-order inferential behavior of its pooled-data oracle counterpart. Simulation studies and real-data applications demonstrate the finite-sample performance of the proposed estimation and testing procedures.
Read the full abstract

Statistical Methodology

Multiple Change Point Detection in the Mean and Covariance of High-Dimensional Matrix Time Series

This paper studies simultaneous change point detection in the first- and second-order structures of high-dimensional matrix-valued time series. Specifically, we extend the framework of main effect factor models to accommodate multiple change points so that shifts in the mean and covariance structures are characterized by changes in the trend-stationary main effects main effects and in the factor-driven common components, respectively. Under this framework, the two types of changes are not necessarily aligned and can be arbitrarily large without masking the effects of each other, and each of the detected change points is identified either with the row or column categorizations which improves their interpretability. We propose a unified procedure based on moving-sum statistics to detect and localize both types of changes. Under general regularity conditions that allow for temporal and cross-sectional dependence, we derive detection and localization guarantees for the proposed method. The finite-sample performance of the method is demonstrated through extensive experiments, and its practical usefulness is illustrated by a real data application to New York City Yellow Taxi trip records.
Read the full abstract

Statistical Methodology

A perspective note on likelihood approximation and inference for complex simulation models using a chain of aggregated normalizing flows

We present a new perspective on the problem of likelihood approximation within the framework of simulation-based inference that promotes scalable and controllable simulation routines for large-scale data analysis, allows efficient parameter space exploration or smooth interpolation in high-dimensions and, thus, supports valid statistical treatments of hypothesis testings as well as uncertainty quantification. In particular, we consider a chain of $n$-aggregated normalizing flows for likelihood approximation scheme, where a set of upfront replicated observation datasets from the forward complex simulation model pass through the first set of bijective transformations, and then subsequently pass to the other sets of bijective transformations. Here, we assume that, for any $k \in \{1,\,2, \ldots, n\}$, the parameters corresponding to the first $k$ sets of bijective transformations are estimated sequentially, in some sense of optimality, for constructing flexible probability distributions, regardless of the remaining $(n-k)$ sets of bijective transformations. Moreover, our objects of interest are to highlight two complementary mathematical arguments that leverage an informatics-theoretic formalization, based-on empirical likelihood estimators under moment restrictions, and a sequential decision-making paradigm, with mixing distributions, for updating and aggregating the estimated parameters of the overall normalizing flows. As a by-product, the framework provides a reliable surrogate model, conditioned on the model parameters defining the forward computational simulation, that allows samples generation, with statistical powers, and facilitates computationally tractable scheme in the Bayesian paradigm for inference, hypothesis testings and uncertainty quantification.
Read the full abstract

Statistical Methodology · Cross-listed from stat.ML

HyperNSDE: Personalized Neural SDEs for Joint Static-Longitudinal Clinical Data Generation

Synthetic patient data generation is a promising solution to the dual challenge of data scarcity and privacy constraints in healthcare machine learning. Realistic synthesis of patient-level clinical data requires jointly modeling heterogeneous static covariates, irregularly sampled longitudinal trajectories, and informative observation times - three tightly coupled components in practice yet rarely addressed together. We propose HyperNSDE, a continuous-time generative model that conditions a latent Neural SDE on static patient representations through a hypernetwork, allowing baseline characteristics to shape trajectory evolution beyond the initial condition without requiring a trajectory encoder, while stochastic latent dynamics capture realistic variability in generated paths. Observation times are modeled jointly through a latent-state-dependent intensity process, and training on irregular stochastic paths is stabilized via a deterministic-stochastic path decomposition with a non-adversarial signature-kernel objective. Experiments on simulated and real clinical datasets show improved observation-time fidelity and competitive performance, while matched-grid analyses reveal that forecasting and correlation metrics are affected by observation-grid regularity and trajectory smoothness.
Read the full abstract