Statistical Methodology
Handling mild outliers and unobserved values in compositional datasets using finite mixtures of mean-parametrised Dirichlet models
Jason Pillay, Andriëtte Bekker, Cristina Tortora et al. (+1 more)
Heterogeneous compositional data may be simultaneously affected by missing values and atypical points, posing challenges for both clustering and outlier detection. We develop a mixture model for incomplete compositional data under Huber's contamination model, with contamination defined directly on the simplex and on observations that may be missing at random. The model provides a principled representation of outliers and allows the distribution of missing parts to be derived while accounting for contamination. We establish that maximisation of the observed log-likelihood constructed from contaminated, mean-parametrised Dirichlet densities is a convex optimisation problem. We then develop a tailored expectation-maximisation. The E-step incorporates the moments from the distribution of the missing parts of the data. Although the resulting parameter estimates are not available in closed form, the maximisation step admits tractable element-wise iterative updates. Numerical experiments demonstrate the performance of the proposed approach under varying percentage of missingness and contamination, and different sample size. An application to the American Time Use Survey identifies two interpretable clusters corresponding to work-intensive and sociable recreational days, while revealing atypical time-use compositions. In contrast, a conventional Dirichlet mixture model identifies four clusters, reflecting the influence of outliers and an artificial splitting of one cluster.
Read the full abstract ▾
All authors Jason Pillay, Andriëtte Bekker, Cristina Tortora, Antonio Punzo
Statistical Methodology
Efficient Bayesian Inference for Benter Models on Ranked Data
Michael Pearce
The Benter model for partial and complete rankings generalizes the well-known Plackett-Luce by attaching rank-level dampening parameters that permit some ranking stages to be noisier than others. This extension is empirically important for domains ranging from horse racing to ranked-choice elections. However, model fitting is made challenging by the fractional powers these dampening parameters introduce to the likelihood, breaking the conjugacy underlying existing Plackett-Luce samplers. This paper develops an efficient
Bayesian estimation procedure for the Benter model. We introduce a two-part augmentation scheme using positive $\alpha$-stable and exponential auxiliary variables that linearizes the intractable normalizers in the Benter likelihood and yields closed-form Gibbs updates. A simulation study confirms efficient estimation, accurate parameter recovery, and nominal credible-interval coverage across sample sizes and item counts. We illustrate the
algorithm on complete and partial rankings from survey preference and ranked-choice election
datasets.
Read the full abstract ▾
Statistical Methodology
Graph-dependent shrinkage priors for Bayesian trend filtering
Andrea Mascaretti and Daniel R. Kowal
Many common data dependencies can be characterized by graphs: time series data are sequential (chain graph), images appear as pixels (
lattice graph), areal data are defined by neighboring units (spatial adjacency graph), etc. Graph trend filtering seeks to smooth and predict such data. However, classical trend filtering only incorporates the graph for estimation of the trend, which limits its adaptivity, and is brittle in the presence of missing data. Further, it lacks uncertainty quantification and faces certain computing challenges. We address these limitations with a comprehensive
Bayesian framework for (graph-) dependent data. Our approach leverages the graph at three critical junctures: 1) the trend, to enable smoothing, imputation, and prediction; 2) the local shrinkage, to enhance adaptivity and precision; and 3) the MCMC sampling
algorithm, to deliver scalable posterior (predictive) inference via sparse and banded operations. For the proposed graph-dependent shrinkage priors, we study the local concentration and adaptivity properties and establish conditions for posterior propriety. Simulation studies demonstrate that, relative to state-of-the-art frequentist and
Bayesian alternatives, this framework provides more accurate point estimates, more precise interval estimates, and highly competitive computing. We apply our methods for spatio-temporal modeling and forecasting of local area unemployment data for every county in the continental U.S. during the 2020 COVID-19 unemployment shock.
Read the full abstract ▾
Statistical Methodology
Approximating the null distribution of generalized distance covariance
Dominic Edelmann
The null distribution of distance covariance is usually approximated by permutation, which is prohibitive when very small p-values are needed, or by matching a few moments to a parametric family, which is inaccurate in the tails. A third option is to approximate the limiting distribution, a weighted sum of chi-square variables, directly through the spectra of the doubly centred distance
matrices. This is used for kernel-based tests but has lacked a rigorous justification. We prove that the empirical spectra give a uniformly consistent approximation of the limiting null distribution, and hence an asymptotically valid test, for a general class of distances of negative type on separable metric spaces. The result covers the
Hilbert-Schmidt independence criterion as a special case. We also give an adaptive
algorithm that brackets the p-value from a partial eigendecomposition, reducing the cost from $O(n^3)$ to $O(k n^2)$, and a shrinkage correction matching the first two moments. In simulations, the proposed tests are the only non-Monte-Carlo procedures whose empirical type I error converges to the nominal level.
Read the full abstract ▾
Statistical Methodology · Cross-listed from stat.ML
(Mis)Understanding Benign Overfitting in Equity Return Prediction
Hui Guo, Jiawei Huang, Runze Li et al. (+1 more)
Highly overparameterized models often predict well despite interpolating training data in complex domains, challenging the classical bias–variance tradeoff. We investigate whether this “benign overfitting” phenomenon extends to equity return prediction. Consistent with recent statistical theory, we document two key phenomena: first, a double descent pattern in the ridgeless model's prediction risk; and second, that while the optimal ridge model consistently outperforms its ridgeless counterpart, this performance gap becomes negligible at large parameter-to-observation ratios. Ultimately, however, both models fail to outperform a simple historical average. This empirical evidence aligns with our asymptotic results under the null hypothesis of zero slope coefficients, suggesting that standard equity predictors lack true forecasting power—even within highly flexible, nonlinear machine learning architectures. These findings reconcile modern and classical machine learning in asset pricing: in the absence of a true signal, they asymptotically collapse to the historical average
benchmark.
Read the full abstract ▾
All authors Hui Guo, Jiawei Huang, Runze Li, Yan Yu
Statistical Methodology · Cross-listed from econ.EM
Principal Component Analysis for a Mix of Stationary and Nonstationary Variables
James D. Hamilton, Xinwei Ma and Jin Xi
This paper develops a procedure for uncovering the common cyclical factors that drive a mix of stationary and nonstationary variables. The method does not require knowing which variables are nonstationary or the nature of the nonstationarity. An application to the FRED-MD macroeconomic
dataset demonstrates that the approach offers similar benefits to those of traditional principal component analysis with some added advantages.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.LG
From Causal Plausibility to Causal Reliability: Evaluating LLMs as Calibrated Direct Causal-Edge Classifiers
Amit Kumar, Elnur Adl Zarabi, Suranjana Trivedy et al. (+4 more)
Large
language models (LLMs) are increasingly used to provide prior causal knowledge for structural causal discovery, yet whether their direct-edge judgments and confidence can be trusted remains unclear. We systematically evaluate 12 instruction-tuned open-weight models across six
benchmark causal graphs, five prompting strategies, and four confidence sources: verbalized, logit-based, cross-
prompt agreement, and cross-model agreement. Under our
language-only pairwise protocol, our evaluation yields three key findings. (i) LLM-based causal judgments are strongly recall-dominant: models predict overly dense graphs with many false-positive edges, while prompting mainly shifts the precision-recall trade-off rather than resolving overprediction. Gains from model scale diminish on the largest graphs and do not eliminate miscalibration. (ii) LLMs often capture causal relatedness without reliably identifying directness or orientation. Relative to published reference graphs, models misclassify 40.0% of indirect and 36.0% of reversed non-edges as direct edges, versus 28.2% of other non-edges. Moreover, 80.8% and 84.6% of these false positives receive verbalized confidence of at least 80%, revealing substantial overconfidence in structurally incorrect predictions. (iii) Conventional confidence estimates are unreliable, whereas agreement offers a more promising signal. Logit-based confidence frequently collapses near 1.0 regardless of correctness, while cross-
prompt and cross-model agreement achieve better mean calibration and discrimination, though their advantages are not statistically significant after Holm correction. A
benchmark-familiarity audit further identifies potential familiarity in five model-
dataset pairs, all involving AsiaM. Overall, our results suggest LLMs are better viewed as sources of externally validated soft causal priors than as direct evidence of causal structure.
Read the full abstract ▾
All authors Amit Kumar, Elnur Adl Zarabi, Suranjana Trivedy, Zhiqian Chen, Lei Zhang, Kaiqun Fu, Taoran Ji
Statistical Methodology
Interpretable AI with Local Distillation
Erin Craig, Yiling Huang and Snigdha Panigrahi
Modern AI models such as tabular foundation models and gradient-boosted ensembles can outpredict classical methods, but provide little basis for
reasoning about their predictions. High-stakes decisions call for models that are both accurate and interpretable as built. Local linear modeling offers a path forward: a smooth regression function is locally well approximated by a linear one, allowing a linear fit near each query point to achieve high accuracy without sacrificing transparency. The challenges lie in learning what is "local" and developing statistical tools for interpretation. Here, we propose local
distillation, in which a black-box "teacher" guides a regularized linear "student" model at each query point. The teacher (1) defines locality by upweighting training observations with similar predicted outcomes, and (2) anchors the fit with its prediction at the query point, included as a pseudo-observation whose weight is estimated from the data. For interpretation, we add a small amount of Gaussian randomization to the local objective and use refits to assess stability: selection frequencies identify reliable features at a query point, and clustering the randomized fits identifies stable subgroups across the data. Under the lasso penalty, we prove that this randomization yields feature-selection probabilities that are stable under small perturbations of the training responses. Across 17
benchmark datasets, local
distillation nearly matches its AI teacher's accuracy while producing a sparse linear model at each test point. In a high-dimensional cancer gene expression example, the framework identifies patient subgroups whose local models use different genes; this heterogeneity is invisible to a global linear model, and difficult to surface in a black-box model.
Read the full abstract ▾
Statistical Methodology
Penalized likelihood estimation of probability density functions using compositional splines
Stanislav Škorňa and Jitka Machalová
Probability density functions are commonly estimated through preliminary smoothing or aggregation procedures, e.g., histograms or kernel density estimation, before subsequent functional representation and functional data analyses. Such a two-stage approach can lead to additional approximation bias and weaken the direct connection between the observed data and the underlying distributional structure. In this paper, we propose a penalized maximum likelihood framework for direct estimation of probability density functions from raw observations within the framework of Bayes
Hilbert spaces while preserving the compositional geometry of densities. The methodology is based on the centered log-ratio (clr) transformation, an isometric isomorphism between Bayes
Hilbert spaces and the standard Lebesgue space of square integrable functions with zero integral, enabling efficient spline representations. The clr transformed densities are represented using ZB-spline basis functions, while their smoothness is controlled through quadratic penalties imposed on the spline coefficients. The proposed framework is developed for univariate and bivariate densities. In the latter case, it naturally incorporates the orthogonal decomposition into independent and interactive parts together with the corresponding
geometric marginals. The performance is evaluated in a simulation study involving multiple complex scenarios and compared with kernel smoothing. Finally, the applicability of the framework is illustrated using empirical geochemical data.
Read the full abstract ▾
Statistical Methodology · Cross-listed from stat.ML
Primal–Dual Alternating Neural Learning for Timely Classification with Performance Guarantees
Jiaming Qiu, Yingye Zheng and Ying-Qi Zhao
Timely risk classification is essential in many clinical monitoring settings, where decisions must balance the benefit of classifying patients early for subsequent intervention against the value of observing additional data. Yet most existing statistical and machine-learning methods are designed for fully observed trajectories and offer limited control over key operating characteristics such as sensitivity, specificity, and monitoring cost. We cast the sequential classification problem within a multi-objective optimization framework targeting these three criteria. We characterize the optimal decision rule through a value recursion that quantifies, at each time point, the trade-off between immediate classification and continued monitoring. To estimate the rule from data, we formulate a constrained optimization problem that maximizes specificity while enforcing prespecified sensitivity and monitoring-cost constraints. We then develop an estimation procedure that employs a recurrent neural network to approximate the evolving value processes and a primal–dual updating scheme to satisfy the performance constraints. Through simulation studies and an application to continuous glucose monitoring for hypoglycemia risk prediction, we demonstrate that the proposed method yields accurate and timely sequential decision rules that adhere to the desired operating characteristics.
Read the full abstract ▾
Statistical Methodology
Evaluating the effects of policy interventions subject to early adoption: A case study of prescription drug monitoring programs and opioid dispensing
Sarika Aggarwal, Brent A. Coull, Nima Hejazi et al. (+1 more)
Policies that require organizations to use new systems, such as prescription drug monitoring programs (PDMPs), are often implemented in phases, with an initial period of voluntary access followed by mandated compliance. This allows the policy intervention to be adopted before compliance is required (early adoption), causing outcomes to change before the mandate takes effect. When early adoption is present, the no-anticipation assumption underlying synthetic control methods (SCM) is violated, leading to biased policy effect estimates. We formalize early adoption in a potential outcomes framework for staggered policy implementation and decompose the total policy effect into early adoption and mandate components. We then propose a two-stage, early adoption-aware SCM procedure that first estimates early adoption effects using an interactive fixed effects model fit to pre-mandate data and then residualizes outcomes before applying SCM variants to estimate mandate and total policy effects. Simulations, including settings with correlation between early adoption and latent factors, show reduced bias and improved uncertainty quantification relative to conventional SCM
estimators. We apply the framework to state-level PDMP policies and per-capita opioid dispensing. After accounting for early adoption, estimates suggest reductions in opioid dispensing following PDMP availability and mandates; however, the estimates are imprecise and not statistically significant.
Read the full abstract ▾
All authors Sarika Aggarwal, Brent A. Coull, Nima Hejazi, Rachel C. Nethery
Statistical Methodology
Transporting Randomized Trial Effects to Real-World Populations via Riesz-Calibrated Optimal Transport
Anik Burman, Margaret Gamalo, Promit Ghosal et al. (+1 more)
Randomized trials support causal inference, but differences between trial and target populations can limit the transportability of treatment effects to real-world settings. Many existing approaches model the propensity of trial participation and can therefore be sensitive to model misspecification and weak overlap of the
covariate distributions. Optimal Transport (OT) offers a different route by comparing the trial and target populations directly in
covariate space. We develop RICOT, a Riesz-calibrated OT procedure transporting treatment effects to a treated target population. We consider a semi-unbalanced OT with entropic regularization where the source marginals are relaxed. We show that the uncalibrated OT introduces a bias which does not shrink with increasing sample size. RICOT removes this bias by imposing calibration equations directly within the transport problem. With a growing calibration sieve, the calibrated weight consistently estimates the target-to-trial density ratio, equivalently the Riesz representer of the target expectation functional, even when the entropic and source-relaxation parameters remain fixed and positive. Combined with outcome regression, the resulting
estimator is doubly robust and attains the semiparametric efficiency bound under suitable rate conditions. Its variance is estimated directly from the influence function, without resampling or repeated OT optimization. Simulations show low bias and near-nominal coverage across a range of overlap and misspecification settings, including settings in which sampling-score methods perform poorly. We illustrate RICOT in a real-world application involving a rare progressive cardiomyopathy, comparing conventional IPW and AIPW
estimators with our OT-based IPW and doubly robust
estimators for transporting the randomized treatment effect to a real-world population receiving the same treatment.
Read the full abstract ▾
All authors Anik Burman, Margaret Gamalo, Promit Ghosal, Prosenjit Kundu
Statistical Methodology
Classification testing: A new framework for drawing qualitative conclusions from quantitative estimates
Andrew C. Eggers and Zikai Li
Social scientists rely on hypothesis testing to support their research conclusions, but the standard tests are designed for testing one hypothesis rather than adjudicating between rival possibilities. We develop a new framework, "classification testing", as an alternative. Instead of selecting one hypothesis to test, a researcher conducting a classification test decides what qualitative distinctions ("classes") are most substantively relevant; the test either assigns the estimand to a class with error control similar to that of a conventional hypothesis test, or declares the result inconclusive. We argue that classification testing is superior to current practice not just when the objective is to adjudicate between rival possibilities but also when there is one research hypothesis to be tested, because classification testing exposes that hypothesis to refutation. We illustrate the framework by applying it to a well-known media experiment and offer an R package to aid in implementation.
Read the full abstract ▾
Statistical Methodology
Sequentially valid inference for probabilistic inflation forecasts
Amadeo Grob, Maurizio Daniele and Johanna Ziegel
Traditional statistical tests are poorly suited for the sequential evaluation of probabilistic forecast calibration. We address this limitation in macroeconomic forecasting by applying a new sequential testing method based on e-values. The e-value-based methodology enables anytime-valid inference. It allows practitioners to test against calibration continuously without invalidating statistical guarantees. To illustrate the framework's practical value, we apply it to probabilistic inflation forecasts for the United States, the Euro Area, and Switzerland. Our analysis shows that the sequential approach gives detailed insights into the timing and nature of forecast misspecification. We find these diagnostics are particularly insightful during major structural breaks. During these events, we find evidence against calibration that static, full-sample tests often miss. Therefore, this work shows that e-value-based tests are a practical method for the evaluation of forecast calibration in empirical macroeconomics.
Read the full abstract ▾
Statistical Methodology
Identification and Inference for Causal Effects in Extremes under General Conditions
Lisa Leimenstoll and Melanie Schienle
Understanding the propagation of extreme events is important in many economic and environmental applications, yet most econometric methods for causal inference focus on average effects rather than tail behavior. This paper studies the identification of causal relations in extremes and derives resulting
estimators and their asymptotic inference. As measure of causal dependence between extreme realizations of variables, we analyze the asymptotic behavior of the Causal Tail Coefficient (CTC) within a linear structural causal model with heavy-tailed regularly varying innovations. In contrast to the existing literature, we allow the variables in the system to exhibit heterogeneous tail indices and consider the presence of potentially heavy-tailed confounders. We derive theoretical results assessing the limiting behavior of the CTC under these conditions and show how differences in tail behavior can help to reach identification of the causal structure. Light-tailed confounders are asymptotically negligible, but sufficiently heavy-tailed confounders can induce extremal dependence patterns that are observationally indistinguishable from direct causal effects. When suitable proxy information is available, identification can be recovered using an adjusted Causal Tail Coefficient. Based on these results, we develop estimation and inference procedures for causal relations in extremes under general conditions. We establish asymptotic properties of the proposed
estimators and derive tests for the causal direction and heavy-tailed confounding. Simulation evidence examines their finite-sample performance and provides guidance on their implementation. Applications to
climate and financial extremes illustrate how the proposed methods can uncover causal relations that may remain undetected by approaches targeting average dependence.
Read the full abstract ▾
Statistical Methodology · Cross-listed from stat.AP
Data Shared Neighbourhood Selection for multi-condition network inference
Blanche Francheterre, Ruben Colindres Zuehlke, Vivian Viallon et al. (+2 more)
External stresses may affect both the circulating levels of specific biomarkers and disturb the correlation structures across molecular entities. The contribution of both types of dysregulations to the subsequent risk of disease are yet to be evaluated. We propose Data Shared Neighbourhood Selection (DSNS), a joint network inference method for estimating preserved and altered conditional association structures across related conditions. DSNS combines neighbourhood selection with the Data Shared Lasso decomposition, representing each nodewise regression coefficient as the sum of a shared component and a sparse condition-specific deviation. This provides an interpretable decomposition of molecular associations while retaining the computational advantages of neighbourhood selection. We also adapt the Stability Approach to Regularisation Selection (
StARS) to this two-parameter joint estimation setting. In simulations involving sparse, hub-based and rewiring perturbation mechanisms, DSNS matched the best joint estimation methods for two conditions and outperformed them as the number of conditions increased, while remaining substantially faster than graphical lasso frameworks. Applied to prediagnostic inflammatory proteomic data from future lung cancer cases and matched controls in the EPIC-Italy and NOWAC cohorts, DSNS highlighted altered associations involving CDCP1 and IL10, two established lung cancer risk markers, as well as differential associations involving
proteins not selected by risk models.
Read the full abstract ▾
All authors Blanche Francheterre, Ruben Colindres Zuehlke, Vivian Viallon, Marc Chadeau-Hyam, Julien Chiquet
Statistical Methodology
Randomization tests for model specification in causal inference under network interference
Supriya Tiwari and Pallavi Basu
Analysis of experimental data becomes challenging when the underlying population is connected by a network. Exposure mapping is a common tool in the literature for defining and estimating spillover effects. These mappings reduce the dimensionality of the estimand, thereby facilitating identifiability. It is assumed that this mapping is correctly specified, leaving the choice of the exposure mapping to the analyst. This makes
estimators of the spillover effect, such as the Horvitz-Thompson
estimator, vulnerable to bias from model misspecification. Although these
estimators have been shown to be robust to certain forms of controlled misspecification, there has been relatively little methodological progress in empirically investigating appropriate exposure mappings. In this paper, we propose a novel design-based model specification framework for causal inference. Building on this, we develop a randomization-testing procedure to assess the correct specification of an exposure-mapping model in the presence of network interference. We provide theoretical guarantees for the asymptotic validity of the proposed testing procedure. We establish the favorable power properties of our method through an extensive simulation study and illustrate it in a field experiment investigating the effect of anti-conflict norms among adolescents.
Read the full abstract ▾
Statistical Methodology · Cross-listed from q-fin.ST
From Exponential to Polynomial: An Exact Filter for High-Dimensional MSM Models
Daniyal Ali Hameedi
In this paper we propose a new formulation of the
Bayesian Filter as used in the discrete-time Markov-Switching-Multifractal (MSM) model of
volatility based on existing permutation
symmetry within the likelihood structure. We show both analytically and empirically that such a formulation leads to a reduction in time complexity from $O(D^k)$ to $O(k^D)$ thereby significantly reducing the computational bottleneck associated with dimensionality. We compare the agreement between the naive and sector filters and find that while there are significant disagreements, the ground-truth recovery of the latter seems to improve on the former.
Read the full abstract ▾
Statistical Methodology
An Anchored Logistic Family for Bounded Trait Measurement and Growth: Origin Before Unit
Jaehwa Choi
Latent trait models differ less in what they measure than in what they fix. Item response theory frees the trait from the items administered but, in doing so, surrenders the origin and unit of the scale to convention. The Cognitive Trait Model (CTM; Choi, 2022) restores both by bounding the trait on [0, 1], where 0 denotes an ignorance level and 1 a mastery level defined by the task domain. We generalize the derivation behind CTM into a family indexed by the mastery anchor $L$: the published model is the case $L = 1$, while the limit $L \to \infty$ gives a half-truncated link with an absolute origin but no ceiling. Origin before unit. Bounded support admits Gauss-Legendre quadrature without truncation error, so item and person parameters follow from Bayes modal EM rather than MCMC. On identical data and priors this reproduces the published estimates ($\theta$ correlation 0.9999) while running 35 times faster than a converged random-walk sampler and 73 times faster than NUTS; a single adaptive-testing update costs about five microseconds. Applied to the time axis, the same link becomes a four-parameter growth curve spanning accelerating, decelerating, plateauing and declining trajectories, though not non-monotone ones. The family reduces to a reparameterization of the logistic model only when all items share a slope, so a measurement advantage is possible in principle; three simulations find none, locating the contribution in interpretation and computation rather than in measurement itself. In the LSAT reanalysis, examinees with perfect scores read 67%, 75% or 100% of mastery depending on the
estimator – the criterion-referenced claim is meaningful only with its uncertainty attached, and the coordinate is not in general the proportion of the domain a person can perform.
Read the full abstract ▾
Statistical Methodology
Logistic Regression Equivalent Weights for Survey Inference: Construction and Asymptotic Properties
Seonghun Lee
Equivalent weights from regression models provide a bridge between design-based and model- based survey inference. We develop a frequentist survey-weighting framework for logistic regres- sion equivalent weights under categorical poststratification. Because the logistic model-based population
estimator is nonlinear in the observed outcomes, we define equivalent weights through a local first-order representation, with each weight given by the scaled derivative of the popula- tion
estimator with respect to the corresponding observed outcome. This construction yields an explicit closed-form expression in terms of the fitted logistic model and population
cell structure. We establish convergence of the effective sample size ratio, asymptotic linearity of the resulting weighted
estimator, and a consistent plug-in variance
estimator under a finite-
cell superpop- ulation framework. Simulation results examine population recovery, regression performance, effective sample size, and the finite-sample behavior of the asymptotic variance approximation. The results show that logistic equivalent weighting provides competitive finite-sample point esti- mation and approximately valid plug-in variance estimation, effective sample size is competitive with alternative model-based weighting in the settings considered. An application to the Fu- ture of Families and Child Wellbeing Study illustrates the construction and use of the proposed weights and variance
estimator in an empirical survey setting. We emphasize the relationship between logistic equivalent weighting and recent work on locally equivalent weights for non- linear multilevel regression and poststratification, while focusing specifically on the frequentist interpretation of the weights, their survey-weight properties, and their use for inference.
Read the full abstract ▾