Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics Statistical Methodology

stat.ME Watch Statistical Methodology (sign in to use)

Thursday 10 September

Statistical Methodology

Bayesian Variable Selection for High-Dimensional Predictors with Missing Psychometric Outcomes

High-dimensional, multimodal predictors and partially observed multivariate outcomes are common in psychometric research. However, existing regularization methods often do not accommodate hierarchical predictor structures and are primarily designed for univariate outcomes. We propose SHIM, a Bayesian framework for structured variable selection that combines hierarchical horseshoe shrinkage with a Bayesian treatment of missing outcomes. The framework jointly accommodates predictor hierarchies, dependence among outcomes, and incomplete multivariate responses. We establish theoretical properties of the proposed prior specification and evaluate SHIM through simulation studies. The results demonstrate that SHIM balances sensitivity with false-positive control while yielding accurate coefficient estimates and well-calibrated uncertainty quantification. We further apply SHIM to data from an Alzheimer's disease cohort to characterize associations between multimodal neuroimaging measures and multivariate neuropsychological outcomes and to generate posterior-based multiple imputations for downstream analyses of the relationships between fluid biomarkers and cognition. An R package, shim, is publicly available to facilitate implementation.
Read the full abstract

All authors Zongyue Teng, Shujie Ma, Timothy J. Hohman, Angela L. Jefferson, Panpan Zhang

Statistical Methodology

Discretization in covariate-adaptive randomization: gains and losses

Covariate-adaptive randomization(CAR) is widely implemented in clinical trials to balance prognostic covariates across treatment arms. Continuous covariates are often discretized into strata in practice, yet their consequences are not clearly understood. This paper provides a comprehensive study of the impact of discretization on both the CAR design process and the inferential results thereafter. We establish the asymptotic properties of both imbalance measures and treatment effect estimators under discretized and non-discretized settings. Practical recommendations are given on when and how discretization should be employed. We show that discretization in design is generally recommended, as it enhances robustness against model misspecification. However, if the true model is known, the most efficient strategy is to balance covariates according to that model in the design. The theoretical results are corroborated by extensive simulation studies and an empirical application to a diabetes trial dataset. Together, the results clarify the gains and losses of discretization in CAR and pave the way for learning impact of discretization to other designs and beyond.
Read the full abstract

Statistical Methodology

Generalized Ridge Refitting for the Lasso and Prediction Improvement Bounds

We study a class of Lasso based estimators obtained by applying a quadratic correction on the Lasso equicorrelation set. The penalty matrix determines both the magnitude and geometry of the correction and contains, among other cases, the isotropic Lasso–Ridge correction, least squares refitting, Gram proportional interpolation between the Lasso and least squares, and coordinate specific penalties. We first derive a closed form representation and isolate the positive gain component of the resulting prediction improvement. We then control the remaining stochastic linear term in expectation by localizing the random signed equicorrelation model around a deterministic reference support. This yields a finite sample expectation bound that explicitly accounts for the randomness induced by Lasso model selection. The resulting decomposition provides a unified framework for understanding when Lasso based quadratic corrections can improve prediction.
Read the full abstract

Statistical Methodology

Surprise Reduction and Nullification in Bayesian and Inverse Bayesian Inference under Ambiguous Prediction-Error Attribution

In non-stationary environments, prediction errors may signal environmental change or transient outliers, and adaptive systems must track such changes without overreacting to outliers. We distinguish surprise reduction, which updates beliefs to fit observations, from surprise nullification, which weakens constraints imposed by the predictive structure, and formalize both within Bayesian and inverse Bayesian (BIB) inference. Belief and likelihood updates are derived from variational objectives sharing a nullification strength, determined endogenously by minimizing surprise under the candidate post-update predictive distribution. In the Gaussian case, nullification expands belief and likelihood variances by a common factor relative to standard Bayesian updating, leaving the ratio unchanged. BIB thus defers attribution of the prediction error, committing to neither latent-state change nor observation-process uncertainty. The nullification strength is carried over as a candidate and is maintained or released according to the predictive surprise of the next observation. In a mean estimation task with outliers and changepoints, no scanned parameter setting of a Sage-Husa-type adaptive Kalman filter, fixed-strength BIB variant, or belief-forgetting-only variant outperforms BIB in both changepoint tracking and post-outlier stability. An oracle-informed reduced Bayesian model tracks changepoints better but is less stable after outliers. Although BIB maintains no explicit hypotheses about changepoints or outliers, it generates event-dependent dynamics. The learning rate increases after changepoints, whereas after outliers, nullification is released, and this increase is suppressed. Deferring attribution and letting subsequent observations differentiate the responses may constitute a principle of adaptive inference in non-stationary environments.
Read the full abstract

All authors Shuji Shinohara, Daiki Morita, Yoshihiro Nakajima, Takeshi Takano, Masakazu Higuchi, Ung-il Chunge, Yukio-Pegio Gunji

Wednesday 9 September

Statistical Methodology

Agnostic Model-Assisted Estimation with Machine Learning for Survey Data

Model-assisted estimation uses prediction rules to improve the efficiency of estimators of finite population parameters while retaining design-based inference. Although flexible prediction methods have been considered, existing theoretical results are largely method-specific. We develop a learner-agnostic framework that replaces separate analyses for individual learners with general conditions on the sampling design and prediction error. We connect design-aware and design-agnostic cross-fitting and characterize the sampling designs under which they yield conditional independence across folds. Under suitable conditions, conditional weighting gives exact design-unbiasedness. We establish first-order equivalence to oracle estimators, leading to design consistency and asymptotic normality, and clarify when conditional and original inclusion probabilities yield the same first-order behavior. We propose consistent variance estimators based on cross-fitted residuals and construct asymptotically valid confidence intervals. Under additional model and regularity conditions, we establish asymptotic optimality through attainment of the Godambe–Joshi lower bound. Simulations show that cross-fitting substantially reduces finite-sample bias and improves variance estimation and coverage with adaptive learners.
Read the full abstract

All authors Ziming An, Mehdi Dagdoug, David Haziza, Yves Tillé

Statistical Methodology · Cross-listed from math.ST

The Elliptically Optimal Confidence Interval: A Bivariate Extension of Wilson's Score Method

Constructing a confidence interval for the difference between two independent binomial proportions involves a nuisance direction that is not identified by the estimand. The one-sample Wilson score interval inverts a scalar score test, but has no direct bivariate analogue isolating the difference: inverting the joint normal approximation yields an elliptical region in the unit square, whereas the estimand \(p_1-p_2\) is one-dimensional. We define the Elliptically Optimal (EO) confidence interval as the range of \(p_1-p_2\) over this region and solve the resulting optimization problem in closed form, obtaining explicit bounds in six mutually exclusive and exhaustive cases. The solution admits a compact characterization: the EO interval is the score interval obtained by maximizing over the nuisance variance rather than estimating it. It is therefore the shortest interval obtained by projecting the elliptical region, and inherits its coverage guarantee. We derive the exact coverage excess, \(2[\Phi(z\mathcal{R})-\Phi(z)]\), where \(\mathcal{R}\) is the ratio of the least-favourable to the true standard deviation. The excess vanishes on an explicit line through the parameter space, is bounded by \(\alpha\), and is invariant under proportional scaling of the sample sizes. Exact enumeration of the binomial coverage shows that the Wald interval, whose variance estimator is downward biased by a factor \(1-1/n\) under balanced allocation, falls below nominal coverage almost everywhere. The EO interval never under-covers under the normal approximation and always yields admissible, non-degenerate bounds. Its price is over-coverage when both proportions are extreme, which we quantify exactly.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

Processing and classifying bird songs using wavelet techniques and supervised learning

This study proposes an integrated framework for the processing and classification of invasive bird species vocalizations within natural soundscapes, characterized by high levels of environmental noise. We address the challenge of signal degradation by employing a Bayesian wavelet shrinkage methodology based on the Epanechnikov kernel prior, which offers a closed form decision rule and high computational efficiency for processing large bioacoustic datasets. The methodology was applied to recordings of three species obtained from the iNaturalist platform: Euphonia violacea, Leiothrix lutea, and Passer domesticus. After signal denoising, we extracted a comprehensive set of features, including Mel-Frequency Cepstral Coefficients (MFCCs) and spectral indices such as entropy and zero-crossing rate. Several supervised learning models: Random Forest, Multinomial Logistic Regression and Support Vector Machine (SVM) were evaluated across different feature dimensionalities. Our results demonstrate that the proposed wavelet based preprocessing significantly enhances classification performance, with the SVM model achieving the highest accuracy (up to 0.9398) under a 10-dimensional MFCC configuration. This research provides a robust statistical tool for automated ecological monitoring and the management of biological invasions.
Read the full abstract

All authors Laura Lucia Dominguez Barrios, Fidel Aniano Causil Barrios, Alex Rodrigo dos Santos Sousa, Mariana Rodrigues Motta

Statistical Methodology

Mixture-based Nonparametric Estimation of Spatial Covariance Functions with Applications to HIV Key Population Size Estimation across Sub-Saharan Africa

Consistent data on the sizes of key populations, such as female sex workers (FSWs), are often scarce, particularly at the sub-national level. Accurate size estimates are critical to effectively allocate resources and achieve HIV targets. Since FSW population sizes may be spatially correlated across areas, models that account for spatial dependence can improve estimation. An important component of such models is the covariance function, which characterizes the spatial dependence structure of the underlying process. In this work, we study spatial covariance functions to estimate FSW population sizes in Sub-Saharan Africa (SSA). Many spatial models rely on parametric covariance functions. However, parametric estimation can suffer from model mis-specification, potentially leading to inefficient or biased predictions. We therefore develop a robust non-parametric approach for estimating the covariance function of a stationary isotropic process in $\mathbb{R}^d$. We focus on a class of covariance functions that are valid in all dimensions, which includes popular kernels such as the exponential and Matérn kernels. Leveraging the fact that such covariance functions can be represented as infinite mixtures of scaled Gaussian kernels, we propose two estimation methods: weighted least squares and nonparametric maximum likelihood estimation to estimate the mixing measure of scaled Gaussian kernels. We also develop computationally efficient methods to solve these optimization problems using non-negative least squares and second-order descent updates. We evaluate the proposed methods through simulations and apply them to estimate the FSW population sizes at the sub-national level in SSA.
Read the full abstract

All authors Manushi Siriwardana, Hyebin Song, Le Bao, Stephen Berg

Statistical Methodology

Likelihood-free inference with nuisance parameters through normalizing flows

We present a simple decomposition of a neural-network-based normalizing flow that naturally uncovers a pivotal statistic (or something close) in the presence of nuisance parameters, based only on a sample generator from the distribution of interest. We show that the statistic is near-pivotal in the sense of minimum average KL-divergence of its $p$-values versus uniform and we argue that it can be expected to have good power when the dimension of the statistic equals the dimension of the parameter. It is able to incorporate prior knowledge about group invariances such as translation and scale. It can discover the one-sample $t$-test almost exactly, outperforms the Welch test in terms of worst-case size over a constrained variance-ratio range and achieves good calibration on partial biserial correlations, while showing higher power (and being much faster) on small-to-moderate samples than profile likelihood-ratio techniques.
Read the full abstract

Statistical Methodology

Dynamic prediction intervals for survival times

Most work on survival prediction focuses on estimating survival probabilities rather than predicting individual event times. Recent conformal methods have made it possible to construct prediction intervals for survival times with right-censored outcomes, but existing approaches are restricted to settings with covariates only measured at baseline and do not address dynamic prediction with longitudinal data. We study prediction intervals for survival times in a dynamic prediction framework with longitudinal covariates. Our approach uses Penalized Regression Calibration (PRC) as a working dynamic prediction model, combining linear mixed models for the longitudinal histories with a Cox model for post-landmark survival, and then applies a conformal calibration step to obtain prediction intervals. We compare naive intervals obtained by direct inversion of the survival function estimated by PRC to our dynamic conformal method. A Monte Carlo simulation study evaluates empirical coverage and interval length across sample sizes, censoring levels, landmark times, and non-proportional hazards (NPH) scenarios. We illustrate the proposed methodology by computing dynamic prediction intervals for the time until a dementia diagnosis in the ADNI dataset. The results show that naive inversion is often unreliable, whereas the proposed dynamic conformal method yields more stable predictive performance.
Read the full abstract

Statistical Methodology

Semiparametric Inference for Conditional Shapley Feature Importance

Shapley values are widely used for post-hoc feature attribution, but most estimators return point quantities and do not quantify uncertainty, and popular implementations sample out-of-coalition features from their marginal distribution, which misattributes importance when features are dependent. This paper studies the conditional formulation, in which out-of-coalition features are integrated out under their true conditional distribution. The target is a global, loss-based importance that pairs a conditional value function with a SAGE-style loss aggregation. We propose a one-step estimator with K-fold cross-fitting and a U-statistic correction of the squared loss that removes the Monte Carlo bias of the naive plug-in; it is $\sqrt{n}$-consistent and asymptotically normal under double-robust rate conditions, and the resulting Wald interval attains nominal coverage. A Pinsker-type bound quantifies the bias from misspecifying the working copula class, while vine copulas keep conditional sampling tractable. In a Gaussian design study with n= 500, the empirical coverage of the 95% interval lies between 0.91 and 0.96 across all features, the test holds its Type-I rate at 0.05, and it reaches power one for moderate signals. Applied to the UCI Concrete and California Housing data, the method identifies the conditionally informative features with Bonferroni-controlled significance.
Read the full abstract

Statistical Methodology · Cross-listed from stat.AP

A spatiotemporal negative binomial model with dynamic dispersion: An application to Tuberculosis infections

Tuberculosis (TB) remains a critical public health concern in Brazil, characterized by pronounced spatial heterogeneity and fluctuating temporal volatility. In this paper, we study monthly TB notifications across 61 microregions of Sao Paulo state from 2001 to 2024. To do this, we introduce a negative binomial spatial integer-valued generalized autoregressive conditional heteroskedastic (INGARCH) model featuring jointly dynamic conditional means and time-varying dispersion. To capture inter-regional spillovers, we incorporate both discrete adjacency structures and a novel continuous distance-based formulation leveraging the Matern correlation function. Parameter estimation via conditional maximum likelihood employs a two-step profile-likelihood iterative scheme, demonstrating solid finite-sample performance in simulation studies. Applied to the Sao Paulo TB surveillance data, the framework substantially outperforms standard Poisson and fixed-dispersion spatiotemporal baselines in empirical fit and uncertainty quantification, maintaining nominal 95% predictive coverage across both dense metropolitan centers and rural microregions. Our results reveal marked spatial heterogeneity in baseline incidence, dynamic overdispersion driven by localized outbreaks, and short-range spatial interaction decay. By accurately modeling spatiotemporal volatility, the proposed methodology provides a robust statistical tool to support public health surveillance, policy-making, and resource allocation.
Read the full abstract

Statistical Methodology

Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation

Multidimensional item response theory (MIRT) is a statistical test theory that precisely estimates multiple latent skills of learners from the responses in a test. Both compensatory and non-compensatory models have been proposed for MIRT: the former assumes that each skill can complement other skills, whereas the latter assumes they cannot. This non-compensatory assumption is convincing in many tests that measure multiple skills; therefore, applying non-compensatory models to such data is crucial for achieving unbiased and accurate estimation. In contrast to tests, latent skills will change over time in daily learning. To monitor the growth of skills, dynamical extensions of MIRT models have been investigated. However, most of them assumed compensatory models, and a model that can reproduce continuous latent states of skills under the non-compensatory assumption has not been proposed thus far. To enable accurate skill tracing under the non-compensatory assumption, we propose a dynamical extension of non-compensatory MIRT models by combining a linear dynamical system and a non-compensatory model. This results in a complicated posterior of skills, which we approximate with a Gaussian distribution by minimizing the Kullback-Leibler divergence between the approximated posterior and the true posterior. The learning algorithm for the model parameters is derived through Monte Carlo expectation maximization. Simulation studies verify that the proposed method is able to reproduce latent skills accurately, whereas the dynamical compensatory model suffers from significant underestimation errors. Furthermore, experiments on an actual data set demonstrate that our dynamical non-compensatory model can infer practical skill tracing and clarify differences in skill tracing between non-compensatory and compensatory models.
Read the full abstract

Statistical Methodology

Cointegration by Parts: Locating Cointegration in Time

Tests for cointegration are typically applied to a single window spanning the entire sample, assuming that the long-run relationship holds throughout. When it holds over only a part of the sample, such tests lose power, because the stationary episode is diluted by periods without cointegration. We propose three statistics for testing whether two or more series cointegrate only over a part of the sample, each an infimum of the Engle-Granger statistic over recursive, backward-expanding, or doubly-flexible windows. We derive their limiting distributions and establish which alternatives each is consistent against. Only the doubly-flexible statistic has power against both break directions. Inspired by the seminal work of James G. MacKinnon, critical values are obtained by simulation and summarized through response surface regressions. We apply the tests to global mean sea level and global mean surface temperature anomalies. All three reject the null of no cointegration at the 5% level, locating it in a sub-period of the 1880–2019 record that coincides with documented discontinuities in sea surface temperature data collection.
Read the full abstract

Statistical Methodology · Cross-listed from stat.ML

Optimal Value Inference for Reinforcement Learning

We study offline inference for the optimal value in reinforcement learning under finite state and action spaces. Two new nuisances are derived as fixed points of a self-induced Bellman equation, in which we approximate the maximum Bellman operator by its softmax correspondence. We propose a debiased estimator through the Neyman orthogonality and establish its asymptotic normality under diverging horizons even when the behavior policy changes with time, as long as the nuisances have the statistical rates that can be achieved by many machine learning methods. We provide a concrete estimating procedure for these nuisances and show they can lead to valid inference. Synthetic experiments validate the numerical performance of our inference method, and we implement it in real-life decision-making problems, including bike repositioning and AI agentic tool use.
Read the full abstract

All authors Nan Lu, Ethan Lee, James M. Robins, David Simchi-Levi, Junwei Lu

Statistical Methodology · Cross-listed from math.ST

Proportional-limit asymptotics for Diaconis-Ylvisaker-penalised logistic regression with fitted intercept

This paper develops estimator-level asymptotic theory for maximum Diaconis-Ylvisaker prior penalised likelihood for logistic regression with a jointly fitted intercept and nonzero prior slope direction in the proportional-limit regime. For $\mathrm{N}(\mathbf{0}_p, p^{-1}\mathbf{I}_p)$ Gaussian covariates and $p/n\to\kappa\in(0,1)$, a conditional convex Gaussian min–max analysis yields almost-sure convergence of the fitted intercept and a pseudo-Lipschitz empirical law for the slope estimator. This estimator-level law gives asymptotic limits for out-of-sample scores, classification error, optimal thresholding and oracle calibration. It also yields oracle-adjusted fixed-block $Z$-statistics under isotropic Gaussian covariates, and yields the main ingredient in establishing the limiting distribution of the penalised likelihood-ratio test statistic and identifies the rescaling to recover the nominal chi-square distribution. We extend these results to Gaussian designs with arbitrary deterministic mean and positive-definite covariance via affine centering and whitening and discuss extensions to subgaussian covariates. Finally, we propose a consistent response-moment estimator of the oracle parameters entering the state equations that govern the slope limiting law and are required for feasible inference.
Read the full abstract

Statistical Methodology

Sliced $L^p$ Distributional Balancing

A popular class of causal inference methods addresses confounding through weighting, which reweights treated and control groups to balance their covariate distributions without using outcome information, thereby preserving a design-based perspective. In this paper, we propose sliced $L^p$ distributional balancing (SLDB), a family indexed by $p\in[1,\infty)$ that measures imbalance by averaging squared $L^p$ distances between the cumulative distribution functions of one-dimensional linear projections. The Cramér–Wold device ensures that this criterion identifies equality of multivariate distributions, while projection reduces its computation to sorting-based one-dimensional operations. Because our method lies outside the maximum mean discrepancy (MMD) framework underlying many existing distributional balancing methods, their theoretical and computational tools do not directly apply. We therefore develop a computationally efficient projected subgradient descent algorithm for estimating balancing weights, offering improved computational complexity over MMD-based methods. Furthermore, we establish a novel theoretical framework for SLDB-based causal effect estimation and prove, under suitable conditions, root-n consistency and asymptotic normality, with the asymptotic variance attaining the semiparametric efficiency bound. Finally, we develop inferential procedures that do not require augmentation with an outcome model, thereby retaining the design-based principle. Simulation studies and a real-world application demonstrate that SLDB performs competitively with existing methods.
Read the full abstract

Statistical Methodology

A binary factor model

The orthogonal factor model has been a very useful tool in uncovering covariance structures in a set of variables through a smaller set of underlying factors. This old model is suitable for continuous variables with unbounded support, since the most common assumption for the observables and the factors is multivariate normality. In this work, we propose a factor model for binary data. Factors are negative dependent, so they avoid each other. We study the theoretical properties of the model and carry out a full Bayesian inference. We illustrate the performance of our proposal with simulated and real data sets and compare with the traditional benchmark.
Read the full abstract

Tuesday 8 September

Statistical Methodology · Cross-listed from math.PR

Gumbel convergence for maximal packing distances from uniform random samples

Let $(X _n )_n$ be a sequence of i.i.d. $R^d$-valued random variables uniformly distributed on a compact subset $M$ of $R^d$ . In this work, we study the asymptotic behavior of maximal packings of this compact subset $M$. Under some assumptions on $M$ (in particular, that it is a smooth submanifold of $R^d$ without boundary), we establish a precise extreme value law (of Gumbel type) for the minimal distances from the sample $(X_1 , \cdots, X_ n)$ to the centers of maximal packings. This result leads to explicit asymptotic confidence bounds for this maximal packing and thus to the support $M$. A distinctive feature of our contribution is the explicit derivation of the scaling sequences in the Gumbel convergence, depending only on the geometry of the support M. These formulas provide interpretable confidence bounds for $M$, which represent a novel complement to previous approaches such as those by Fasy et al. (2014). Our approach bridges geometric and combinatorial probability arguments and relies on analytic tools such as the Lambert $W$ function. In addition, we provide examples of main sample spaces in directional and circular statistics (circle, sphere, torus), along with simulations that illustrate and support the theoretical findings.
Read the full abstract

Statistical Methodology · Cross-listed from econ.EM

When is statistical evidence strong enough? Using hypothesis tests to value data collection

Policy decisions often hinge on conventional p-value thresholds, which ignore economic costs and benefits of further data collection. This paper recasts statistical significance as a choice between making an immediate policy recommendation and deferring it until further evidence is collected. The welfare-optimal decision corresponds, under minimax regret, to a statistical test whose level depends on the cost and precision of additional evidence. Inverting this rule, we introduce and recommend reporting the abstention-value (A-value) to determine where additional data collection is most needed. The A-value defines the break-even welfare cost of abstaining and recommending further experimentation given the initial evidence. Computing the A-value only requires a point estimate and its standard error, imposes no prior assumptions on policy effects, and can be converted to a monetary research budget using inputs already frequently used by regulators and funding agencies. When experimentation capacity is limited, we show that prioritizing additional data collection where A-values are the largest yields strong finite-sample guarantees. Applications to anti-poverty programs and to a medical study illustrate the practical benefits of using A-values.
Read the full abstract

Statistical Methodology

Differentially Private Average Treatment Effect Estimation by Propensity Score Blocking

Average treatment effect (ATE) estimation in observational studies is a fundamental statistical tool used frequently in social science, medicine, and other fields. These fields often work with sensitive data where privacy protections are important, so a differentially private mechanism for ATE estimation is highly desirable. Here we present two propensity score-based algorithms for ATE estimation on observational data, one improving the inverse probability weighting (IPW) method used in prior work, and the other using blocking on the propensity score (BPS). Both show lower error and less bias than prior work, with the BPS-based algorithm frequently reducing error by 75% or more compared to prior work.
Read the full abstract

Statistical Methodology

Differential Privacy Guarantees in Small Area Estimation

Statistical agencies increasingly rely on small area estimation to produce reliable estimates for subpopulations with limited sample sizes. These estimates are built from individual survey responses, so agencies must ensure that releasing them does not reveal information about any single respondent. We show that when a single draw from the posterior distribution of the Bayesian Fay-Herriot model is released, pure $\varepsilon$-differential privacy is unattainable, but the release satisfies formal privacy guarantees under Rényi differential privacy and zero-concentrated differential privacy without any noise being added, provided we treat the variance components as fixed. The key insight is that the posterior draw equals the posterior mean plus the Gaussian noise whose variance equals the posterior variance. The guarantee is thus governed by the sensitivity of the direct survey estimate and the posterior variance, and applies equally to a release of the posterior mean with that amount of noise added. For binary outcomes estimated with the Hájek estimator, the sensitivity equals the largest survey weight in the area divided by the sum of the weights. For the intercept-only model we derive exact coefficients describing how a change in one record propagates to every area's posterior mean, giving finite-sample per-area guarantees and a joint guarantee for releasing all areas at once that exceeds the largest per-area guarantee by at most a few percent in our applications. Two applications, poverty prevalence across 2,462 Public Use Microdata Areas in the American Community Survey and smoking prevalence across 52 substrata in the Washington state Behavioral Risk Factor Surveillance System, show that the guarantee is driven far more by the inequality of the survey weights than by the sample size, and that the shrinkage of the model tightens it substantially.
Read the full abstract

Statistical Methodology

Covariate-localized False Discovery Rates

We introduce a flexible model for covariate-dependent multiple testing which can be encoded using a nonparametric Gaussian mixture model. Weight-localized predictive recursion (PRx), a new development in the methodology of Newton's predictive recursion algorithm, is then leveraged to estimate the components of this mixture model, allowing for recovery of the covariate-localized false discovery rate $\text{Pr}(H_i = 0|z_i,x_i)$ using a single, unified algorithm. This quantity represents the most direct extension of Efron's local false discovery rate to the covariate-dependent setting, and admits provable Bayesian FDR control properties under simple rejection rules. We introduce several procedures for estimating and thresholding the local false discovery rate, and show using various simulations and a real-data example that our procedures lead to increased power, tighter Bayesian FDR control, and more interpretable rejections. We furthermore show that this holds for fixed and randomized hypothesis labels, indicating that our proposed methods perform well under both frequentist and Bayesian interpretations of multiple testing.
Read the full abstract

Statistical Methodology · Cross-listed from math.ST

MiNCE: Nonparametric, Strongly Consistent Confidence Envelopes for Band-Limited Functions and their Smoothed Spectra

Minimum-norm confidence envelope strategies offer a nonparametric approach to constructing nonasymptotic, simultaneous confidence regions for band-limited functions, exploiting the theory of Reproducing Kernel Hilbert Spaces (RKHS). While the finite-sample coverage guarantees of these envelopes have been established, their consistency has not been analyzed so far. In this paper, we study this construction, here termed the Minimum-Norm Confidence Envelope (MiNCE) framework, and establish the strong uniform consistency of the resulting bands, both for noise-free and noisy observation models, under mild assumptions on the measurement noises. We further extend this formulation to the frequency domain, deriving nonasymptotic, simultaneous, strongly uniformly consistent confidence bands for the smoothed spectra. Numerical experiments in nonparametric regression and spectral estimation empirically confirm our theoretical results, illustrating the contraction of the confidence envelopes toward the target function as the sample size increases.
Read the full abstract

Statistical Methodology

Semiparametric Receiver Operating Characteristic Analysis in the Presence of an Imperfect Reference Standard via a Box-Cox Density Ratio Model

Receiver operating characteristic (ROC) analysis is commonly used to evaluate the diagnostic accuracy of continuous biomarkers. In practice, the true disease status may be unavailable and only a nominal disease status provided by an imperfect reference standard is observed. Existing nonparametric methods have been developed for ROC analysis in this setting, but may suffer from reduced estimation efficiency, numerical instability, or sensitivity to the choice of biomarker scale. We propose a semiparametric method based on a Box-Cox density ratio model, which links the biomarker distributions of the truly healthy and diseased populations while leaving the baseline distribution unspecified. A key feature of the proposed method is that the transformation parameter is estimated from the data rather than specified in advance, allowing the density-ratio structure to adapt to different transformation scales. We develop an empirical likelihood approach for estimation and an expectation-maximization algorithm for computation. We establish the asymptotic distributions of estimators of the ROC curve, area under the curve, Youden's index, and the sensitivity and specificity at the Youden-optimal cutoff, and develop bootstrap confidence intervals and a goodness-of-fit test. Simulation studies demonstrate that the proposed method provides accurate and numerically stable estimation and inference across a range of distributional settings without requiring the transformation scale to be specified in advance. The proposed method is illustrated using data from a malaria study.
Read the full abstract

All authors Yi Chang, Siyan Liu, Qinglong Tian, Pengfei Li

Statistical Methodology

A Counterfactual Framework for Estimating Infectious Disease Prevalence under Repeated Testing with Symptomatic and Contact-Tracing Components

This paper addresses the problem of estimating infectious disease prevalence under longitudinal testing programs that include scheduled, symptomatic, and contact-tracing testing. Our study is motivated by data from The Ohio State University, where a mandatory once-per-week COVID-19 testing and isolation program was implemented during the Fall 2020 semester, supplemented by additional testing for symptomatic individuals and identified contacts. In this setting, the probability of being tested depends on symptoms or contact-tracing status, creating a complex observation process. We develop a counterfactual framework that links the observation process to a hypothetical process in which infection is prevented. This formulation enables unbiased estimation of disease prevalence by modeling the testing process, possibly nonparametrically, without requiring explicit modeling of transmission dynamics, even though the testing and infection processes are jointly dependent.
Read the full abstract

All authors Jeongjin Lee, Junke Yang, Grzegorz A. Rempala, Patrick M. Schnell

Statistical Methodology

Estimating Causal Treatment Effects in Placebo-Controlled Randomized Clinical Trials When High Placebo Response is Anticipated

In placebo-controlled randomized clinical trials (RCTs), the placebo response significantly modifies treatment effects and diminishes the intention-to-treat (ITT) treatment effect, $\Delta_{ITT}$. This study presents a novel two-stage framework for estimating the standardized causal treatment effect, $\Delta_{STD}$, among the ITT population, under the assumption that their placebo responses are similar to the levels of self-administering medication at home. The first stage employs a real-world, pragmatic, single-blinded placebo lead-in to measure placebo responses to levels expected during routine at-home use. This is achieved by preserving the participants' expectations and controlling for trial-related factors that inflate the responses. In the second stage, a double-blinded randomized phase is used to estimate the conditional average treatment effect (CATE) as a function of placebo response levels and other important effect modifiers. To facilitate CATE estimation, the prognostic scores, defined as the expected placebo responses, are used for dimension reduction. The causal estimand $\Delta_{STD}$ is computed by integrating the CATE function over the distribution of the expected placebo response levels from stage one and other modifiers. We further derive theoretical values for $\Delta_{ITT}-\Delta_{STD}$ to quantify the underestimated treatment benefit due to high placebo responses. The validity and statistical performance of the proposed framework are evaluated through comprehensive simulations.
Read the full abstract

All authors Yang Song, Yunzhe Qian, Chanmin Kim, Gheorghe Doros

Statistical Methodology

Real-time and adaptive anomaly detection algorithm for cyclostationary models

This article introduces PeriodicCALM, an effective real-time anomaly detection framework designed for cyclostationary data streams. While classical cyclostationary processes feature periodically time-varying statistical properties, real-world signals often contain recurring impulsive components that conceal abnormal behavior. Existing real-time methods for struggle with these dynamics, frequently misinterpreting phase-dependent variability as non-cyclic anomalies and causing excessive false alarms. To address this, PeriodicCALM incorporates cycle-dependent variability to systematically ignore regular cyclic impulses while accurately isolating genuine anomalies. Operating in real time with continuous retraining capabilities, the method adapts dynamically to evolving signal characteristics. Comparative evaluations against the baseline CALM framework using simulated data demonstrate significant improvements in detection accuracy and training efficiency, alongside a reduction in prediction latency. Furthermore, the practical utility of PeriodicCALM is validated on real-world vibration signals collected from a compressor monitoring system.
Read the full abstract

All authors Justyna Witulska, Tomasz Barszcz, Ireneusz Jabłoński, Agnieszka Wyłomańska

Statistical Methodology

Kalman Filtering and Smoothing for Improving Precision in Horvitz–Thompson Estimation of Infectious Disease Prevalence

Horvitz–Thompson (HT) estimators can provide unbiased daily estimates of infectious disease prevalence under repeated surveillance by correcting for nonrandom testing induced by scheduled, symptom-based, and contact-tracing components. However, because each HT estimate is based on the testing data available for that day and may involve highly variable inverse probability weights, it can be noisy, have precision that varies over time, and become unavailable during temporary interruptions in testing. The daily HT estimator is modeled as a noisy observation of an underlying prevalence process, with day-specific observation variances estimated using a delete-a-group jackknife. Our primary specification is a joint local linear trend state-space model that extends the standard level-only random walk by adding a latent slope. The Kalman filter improves precision by borrowing information from past estimates. At each time $t$, the process variances are estimated or carried forward using only observations available through time $t$, so the resulting filtered estimate is available in real time. We also describe the corresponding Kalman smoother as a retrospective extension based on the full observed series. When daily HT estimates are missing, the Kalman filter proceeds through prediction-only updates, whereas the corresponding smoother retrospectively reconstructs those periods using later observations. In simulations, the joint Kalman filter substantially improves precision relative to the raw daily HT estimator while preserving the main temporal pattern, and the smoother provides a more stable retrospective summary. In The Ohio State University's fall 2020 SARS-CoV-2 surveillance data, the filter and smoother provide estimates on no-testing days, when the HT estimator provides neither point nor interval estimates, and yield narrower confidence intervals than HT intervals on observed days.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground

Sensor-based AI systems are rarely operated under the conditions under which they were trained: devices, personnel and recording epochs change, and each change degrades performance in ways a random train-test split cannot reveal. We propose a staged, accountable evaluation protocol that treats the evaluation of a deployed model as a measurement with declared reference levels and a quantified uncertainty. Four cumulative generalisation stages hold out devices, subjects and time. Each stage is judged on quantiles of repeated trainings against chance references with the correct class count, an out-of-present-scope rate exposes silent misdirection towards classes that are no longer present in deployment relative to training, and an explicit decision rule ties roll-out decisions not to means but to 5% quantiles. We demonstrate the protocol on infrastructure-free geomagnetic localisation with smartphone-based recurrent classifiers in two real underground mines, including a replication of the scheme's training stages at the second site. Unchanged models are re-evaluated on data recorded 34 months after the training campaign, on a device generation unknown at training time and with a held-out surveyor. The 5% quantile of their present-conditioned precision there is 0.39 over 299 repeated trainings, 16.5 times the chance level; across the composition of the 42 reachable location classes the figure varies by +/-0.08, several times the spread between repeated runs. Repeated trainings of a single configuration show why means mislead: a bimodal configuration passes a mean-based test decisively while its 5% quantile lies more than an order of magnitude below chance.
Read the full abstract

All authors Benny Platte, Rico Thomanek, Christian Roschke, Marc Ritter