Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics Statistics Applications

stat.AP Watch Statistics Applications (sign in to use)

Sunday 9 August

Statistics Applications

Risk-Aware Pump Control in Water Supply Systems Using Probabilistic Water Demand and Electricity Price Forecasts

Drinking water utilities face uncertain water demand and electricity prices, as well as requirements for reliable and cost-efficient operation. This paper investigates a sequential multi-stage pump scheduling problem under uncertainty for a drinking water supplier participating in the day-ahead auction with subsequent imbalance market settlement. We integrate probabilistic forecasts of water demand and electricity prices into a risk-aware stochastic optimization framework. Reliability and economic risk preferences are represented through a lexicographic objective incorporating Conditional Value-at-Risk and Exceedance Risk measures, where the framework allows to explicitly account for the imbalance market settlement. A numerical study demonstrates that improvements in forecasting performance generally translate into improved policy performance. Furthermore, we found that the primary benefit of stochastic optimization lies in improved operational robustness rather than lower expected costs. The water demand is found to be a dominant driver of policy quality, the marginal value of further improvements in electricity price forecasts appears comparatively limited, indicating that the economically exploitable component of the price information is already captured to a large extent. Compared with a price-invariant benchmark, the best stochastic policy achieves cost savings of up to $9\%$, indicating that substantial economic benefits can be realized by accepting a carefully controlled increase in imbalance exposure and operational risk.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Exact Conditional Distributions of Chi-Square-Family Statistics for Two-Way Contingency Tables, by Cell-Separable Dynamic Programming

Every chi-square-family statistic for a two-way contingency table (Pearson's X^2, the power-divergence members, the variance-stabilized T_root) is referred to an approximate null distribution (chi-square, moment-matched chi-square, saddlepoint, or bootstrap) that miscalibrates on sparse or heterogeneous tables, where the exact distribution is a lattice of atoms rather than a continuous curve. The exact reference conditions on both margins (the multivariate Fisher noncentral hypergeometric law) but is usually treated as uncomputable, because the number of tables sharing a margin is astronomical. We show that the exact conditional moments and the exact conditional distribution of any chi-square-family statistic are computable without enumerating tables, because the statistic is additive over cells and the margin-conditional law factorizes cell by cell: a dynamic program walks the table one cell at a time, carrying per row-capacity state either a few moment accumulators, a map from statistic value to probability, or one complex number (the characteristic function), at a cost set by the number of margin states rather than the number of tables. The moment engine computes the exact conditional moments of a 5x5 table at three per cell (about 79 billion tables on the margin fibre) in three seconds; the distribution engine returns the exact tail, verified against enumeration to machine precision, where a three-moment chi-square fit misses it by up to 0.28. The engine furnishes Monte-Carlo-free ground truth against which any approximate reference can be measured, extends to the non-null alternative, and is the computational substrate beneath the exact conditional interval and reporting tools of the companion papers.
Read the full abstract

2608.08576 · 9 Aug · 25 pages, 5 figures, 4 tables. Reproducible code, data, and the derivation companion (DH1-DH8) are openly archived on Zenodo: https://doi.org/10.5281/zenodo.21831680 . Part of a contingency-table methodology series (companions on the honest effect size, the exact conditional interval for Cramer's V, and reporting)

Saturday 8 August

Statistics Applications · Cross-listed from stat.ME

Three Routes to One Answer: Reconciling AIPW, TMLE, and Double Machine Learning for Applied Researchers

Augmented inverse-probability weighting (AIPW), targeted maximum likelihood estimation (TMLE), and double/debiased machine learning (DML) are three routes to the same efficient influence function for the average treatment effect — settled theory we treat as background. This tutorial's contribution is its worked, shared-nuisance reconciliation on real data: what a practitioner must actually match for the routes, and the software packages, to agree. Working the effect of smoking cessation on weight change in the open NHEFS data (n=1566) with one shared Super Learner library and identical cross-fitting folds, we build all three estimators by hand from one influence function, in full-sample, cross-fit, and double-cross-fit variants; the six resulting doubly-robust estimates span only 3.32–3.42 kg, consistent with the established benchmark. We then reconcile the same estimand across our engine and the tmle, AIPW, DoubleML, and tmle3 packages. At their defaults the estimates span 3.32–3.49 kg. Once the library and folds are matched and single-split noise is averaged out, the three library-sharing implementations agree to within 0.01 kg — so the residual spread traces to the nuisance library, folds, and repetitions, not to the estimator label. Because the estimators share one influence function, they agree under good overlap; under a positivity violation the pooled ATE is not identified without additional extrapolation assumptions, and their finite-sample estimates can then diverge sharply. We therefore place a positivity diagnosis ahead of estimator choice, illustrate the failure on a no-overlap example, and close with a reporting checklist. Open-source R code reproduces every number.
Read the full abstract

Statistics Applications

Real-time anomaly detection in base station testbeds via scalable kernel density estimation framework

Large-scale testing infrastructures are critical for validating telecommunication systems, yet their growing complexity makes efficient resource utilization and anomaly detection increasingly challenging. In reservation-based testbed environments, errors in resource allocation or preparation often manifest as abrupt spikes or regime changes in time-based metrics. This paper proposes a scalable, unsupervised framework for real-time anomaly detection in such environments. We introduce CALM (Continuous Anomaly Localization for univariate and Multivariate data), a nonparametric method based on kernel density estimation and bootstrap-based thresholding, designed for anomaly detection at the individual testbed level. To address system-wide visibility, we further propose AggCALM, an aggregation framework that consolidates local anomaly signals across multiple testbeds to detect statistically significant global anomalies while mitigating alarm fatigue. The methodology is evaluated using simulated multivariate data and real-world data from a large-scale base station testing platform. Results demonstrate that the proposed framework enables timely, flexible, and accurate anomaly detection without requiring labeled data, supporting reliable operation of complex test environments. Although the proposed methodology is presented within the context of a telecommunication testing labs, it can be effectively used to other applications, such as condition monitoring, where anomaly detection serves as a pivotal pre-processing step for diagnostic signals.
Read the full abstract

All authors Justyna Witulska, Marcin Szczukiewicz, Artur Tabaka, Rafał Sarniak, Ireneusz Jabłoński, Agnieszka Wyłomańska, Dominik Dulas

Statistics Applications

A Gaussian Process Model of 3D Udder Point Clouds for Teat Length Phenotyping in Dairy Cows

Phenotyping conformation traits is important for dairy cattle breeding and management. Although large-scale phenotyping is possible with 3D imaging technologies, manual annotation of anatomical landmarks and long run times prevent full pipeline automation. In particular, the morphological heterogeneity of cow udders makes automated detection of teat landmarks challenging. To address this limitation, we propose and evaluate a method for teat length estimation from udder point clouds with a Gaussian process. We model the vertical coordinates as the sum of a Gaussian process representing the udder floor and an unknown function representing the teat. Since the udder floor process is smooth and has a significantly wider dependence lengthscale than the teat function, this model allows separating the two terms needed for teat landmark definition. To ensure computational feasibility, we implement a low-rank approximation of the covariance matrix, reducing the computational complexity of the method from $\mathcal{O}(n^3)$ to $\mathcal{O}(n)$. This approach is both faster and more accurate than existing methods, reducing RMSE by a factor of two. It is also more robust to uncommon udder morphologies, making it better suited for automated phenotyping of large numbers of individuals.
Read the full abstract

Statistics Applications · Cross-listed from eess.SY

Source-Agnostic Sizing of Flexibility Reserves

Growing shares of variable renewable energy sources (VRES) increase forecast uncertainty and variability, requiring flexibility reserves that adapt to changing operating conditions and reflect the likelihood and severity of forecast deviations. Conventional fixed-rule or Gaussian approaches misrepresent asymmetric, heavy-tailed errors, leading to inefficient procurement or optimistic risk estimates. This paper presents a source-agnostic framework that constructs conditional error distributions for load, wind, and solar from historical deviations or probabilistic forecasts, combines them into a conditional net-load error distribution, and derives upward and downward reserves using coverage- and risk-based criteria. Nonparametric density estimation captures empirical error behavior without a parametric assumption, while a Conditional Value-at-Risk (CVaR)-based metric quantifies expected uncovered deviations. Experiments on a New York Independent System Operator (NYISO)-based synthetic dataset show that the framework meets target coverage with lower reserve volumes than static benchmarks and avoids the tail-risk distortion of Gaussian-based methods. Because reserves are built directly from the resource uncertainty distributions, the framework is transparent and interpretable, and it integrates seamlessly into existing production-cost models as deterministic reserve constraints, avoiding scenario-based stochastic optimization. It is implemented in the Electric Power Research Institute's (EPRI) DynADOR tool for operational reserve scheduling.
Read the full abstract

All authors Napoleon Costilla-Enriquez, Miguel A. Ortega-Vazquez, Aidan Tuohy, Erik Ela

Statistics Applications · Cross-listed from cs.DL

StatCite: A Large-scale Citation Network Dataset for Statistics and Data Science

In this paper, we introduce StatCite, a large-scale citation network dataset covering publications in statistics and data science from 1981 to 2025. The dataset contains 189,101 research articles collected from 62 representative journals and provides bibliographic metadata, including title, author list, publisher, published year, abstract, keywords, and reference list. Based on the collected publications, we construct four complementary citation-based networks, namely the paper citation network, the co-citation network, the bibliographic coupling network, and the journal citation network. To illustrate the utility of the dataset, we present descriptive analyses of the constructed networks and investigate the community structure of the paper citation network. The results show that StatCite preserves key structural characteristics commonly observed in large-scale citation networks and captures several major research areas in statistics and data science. By integrating multiple network representations with rich textual metadata, StatCite provides a valuable resource for statistical analysis, knowledge discovery, and data-driven studies of scientific literature.
Read the full abstract

All authors Tianang Deng, Tianchen Gao, Rui Pan, Yan Zhang

Statistics Applications

Evaluating for the long term: Learnings from industry

Online platforms prioritize long-term business outcomes, yet typical experiments are far too short to measure these outcomes directly. Our goal in this paper is to collect and share industry knowledge on how to make decisions from short-term experiments that are better aligned with long-term outcomes. Based on a daylong workshop with 26 experts from 15 online platforms and 4 universities, we formulate a series of propositions that reflect current industry knowledge. Participants largely agreed that reversals of sign from short-run to long-run treatment effects are rare, with reversals concentrating in specific cases such as treatments involving content quality signals, hyper-monetization, and pricing. Although the magnitude of treatment effects can shift over time, a "univariate autosurrogate", corresponding to the short-run treatment effect on the long-run metric of interest, is often hard to beat. A recurring theme was the importance of surrogates that are not only (or even primarily) unbiased for true long-run outcomes, but that improve decision-making. Thus, participants generally agreed that simple, interpretable surrogates were generally preferable to elaborate but hard-to-explain surrogate indices. Participants also agreed that, due to concerns about confounding and transportability, experimentally-learned surrogates are generally preferable to observationally-learned surrogates. However, the drawback is that learning good surrogates from experiments typically requires a large, representative portfolio of long-run experiments that few platforms possess. We conclude that there is no substitute for a well-run long-term experiment, whether for learning surrogates or validating them, and we highlight open challenges including evolving treatments, persistent treatments not fully mediated by short-term proxies, and mismatch between experimental samples and the target population.
Read the full abstract

All authors Leif Sigerson, Tom Cunningham, Winston Chou, Sana Pandey, Jonathan Stray, Lo-Hua Yuan, Eytan Bakshy, Timothy Chan, Molly Davies, Maria Dimakopoulou, Simon Ejdemyr, Kenneth Hung, Nathan Kallus, Madhav Kumar, Thu Le, A. Demetri Pananos, Lee Richardson, Brennan Schaffner, Rose Tan, Martin Tingley, Nadia Tomova, Panagiotis Toulis, Wenjing Zheng, Zander Arnao, Dean Eckles

Statistics Applications

From Annual Throughput to Vessel Schedules: A Stochastic Generator for Transshipment Hub Simulation

Simulating container transshipment hubs requires vessel arrivals reflecting cyclical liner schedules and origin-destination (OD) cargo pairing. Existing models relying on Poisson arrivals and aggregate transshipment volumes severely distort waiting-time and yard-occupancy predictions. We propose a three-phase schedule generator, calibrated entirely from public port statistics, eliminating the need for proprietary data. The framework introduces a two-level Gamma mathematical model that balances structured weekly services with operational perturbations. For cargo routing, we develop GreedyDwellFit (GDF), a fast allocation heuristic to pair transshipment batches to specific connecting services using a gravity model. Validated against Busan and Singapore mega-hubs, the model reproduces throughput and call frequencies with under 1% error. Our results show that replacing Poisson models with this Gamma-GDF framework eliminates significant distortions in terminal performance projections, offering a robust, generalisable foundation for port simulation.
Read the full abstract

All authors Qiaohong Li, Haobin Li, Tianhao Chen, Ek Peng Chew

Statistics Applications

Crowd-Sourced Geographies of Income: Using Google Maps Points of Interest as High-Frequency Proxies for Sub-Municipal Income Estimation in Sao Paulo, Brazil

Accurate, up-to-date income data at the sub-municipal scale is essential for social policy in middle-income countries, yet in Brazil it depends on a costly decennial census whose intercensal gap recently exceeded a decade. We test whether the composition of crowd-sourced Google Maps Points of Interest (POIs) can serve as a high-frequency, low-cost proxy for household income across the 26,625 census sectors of the municipality of Sao Paulo. Using a theoretically motivated set of POI categories retrieved from Google Places, we represent each sector by its POI counts, decompose these high-dimensional, sparse features with principal component analysis (PCA) and non-negative matrix factorization (NMF), and train a sweep of regression models to predict census-derived income. Under a data leakage-aware spatial validation design the best model (NMF with gradient boosting) attains a held-out R^2 of 0.65, with performance stable across feature-extraction methods. Interpretable decompositions reveal which POI types carry the income signal. These results suggest that commercial, crowd-sourced geospatial data can complement conventional income statistics during intercensal periods, and we discuss extensions toward multidimensional poverty and the capabilities framework.
Read the full abstract

All authors Adrienne C. Kinney, Anya Workman, Ademar Takeo Akabane, Jenna Barac, Paulo Fernando Braga Carvalho, Jeova Farias, Fernando Nascimento, Paulo Ricardo da Silva Oliveira

Statistics Applications · Cross-listed from stat.ME

Proximal Individualized Functional Treatment Regimes

Estimating individualized treatment regimes (ITRs) is fundamental in data-driven personalized decision-making problems, such as precision medicine. Most of the ITR literature either focuses on categorical/continuous treatments or assumes no unmeasured confounding. In this paper, we make the first attempt to estimate the optimal individualized functional treatment regime (IFTR) for observational data where the treatment is a function and unmeasured confounding is present. We establish an identification result for a class of IFTRs under the proximal causal inference framework. Based on the identification result, we develop an algorithm of finding the optimal IFTR. The appealing practical performance of the proposed method is demonstrated by a simulation study. The proposed method is applied to an accelerometry dataset collected by the US National Health and Nutrition Examination Survey to find the optimal physical activity distribution for the best of the Triglyceride-Glucose index.
Read the full abstract

Friday 7 August

Statistics Applications · Cross-listed from cs.LG

Machine Learning and ARIMA Model Averaging for Adaptive Public Health Forecasting: Comparative Evaluation and an Ontario COVID-19 Case Study

Public health forecasts must respond to abrupt changes in surveillance data without over-extrapolating noise, reporting artifacts, or temporary trends. We evaluated autoregressive integrated moving average (ARIMA), random forest, and extreme gradient boosting (XGBoost) models using 190 weekly observations of publicly available Ontario COVID-19 case counts from January 2020 to October 2023. Rolling-origin time-series cross-validation preserved temporal order during model tuning and evaluation. Performance was assessed across three operating dimensions: responsiveness following selected turning points, forecast horizons of one to six weeks, and the amount of historical training data. We also developed Machine Learning and ARIMA Model Averaging (MLAMA), a non-negative performance-weighted ensemble with weights that vary by forecast horizon and responsiveness setting. Retrospective comparisons showed that ARIMA adapted rapidly after turning points but its normalized error increased at longer horizons. Random forest and XGBoost were less responsive initially but maintained more stable normalized error over longer horizons. For two-week forecasts at the end of the study period, training on the most recent data outperformed using longer historical periods, particularly for XGBoost. MLAMA achieved the lowest normalized mean absolute percentage error across most forecast horizons and ranked among the best-performing methods across responsiveness settings. These findings support selecting forecasting models according to operating conditions rather than relying on a single universally preferred approach. MLAMA provides a practical framework for combining complementary statistical and machine-learning forecasts. The accompanying Python package is currently maintained in a private repository while software validation and reproducibility testing are completed.
Read the full abstract

All authors Yushu Zou, Ye Li, Johra Moosa, Martin Grunnill, Samir N. Patel, Venkata R. Duvvuri

Statistics Applications · Cross-listed from stat.ME

Small Area Bayesian Dynamic Borrowing: Adaptive Subgroup Estimation for Large-Scale Educational Assessments

Large-scale assessments suppress subgroup achievement estimates below minimum sample size thresholds, such as the National Assessment of Educational Progress (NAEP) rule of 62, disproportionately affecting historically underrepresented groups. This study introduces Small Area Bayesian Dynamic Borrowing (SABDB), a unit-level small area estimation method assigning each regression coefficient its own between-area variance, so cross-area borrowing adapts to each coefficient's heterogeneity. We compare SABDB against the unit-level Hierarchical Bayesian Small Area Estimation (HBSAE) model in a simulation study and an empirical case study. The simulation mimics the NAEP eighth-grade mathematics assessment, and the case study uses the PISA 2018 dataset. Across both studies, SABDB achieved near-nominal coverage, whereas HBSAE's intervals were narrow but severely miscalibrated.
Read the full abstract

Statistics Applications

SurroPilot: An LLM-Assisted Platform for Heterogeneous Surrogate Endpoint Evaluation in Clinical Trials

Surrogate endpoints are widely used in clinical trials to accelerate treatment evaluation, yet their validity may vary substantially across patient subgroups. Although recent advances in heterogeneous causal mediation analysis enable subgroup-specific surrogate evaluation, applying these methods requires substantial expertise in causal inference, statistical programming, and clinical trial methodology, limiting their accessibility to many biomedical researchers. We present SurroPilot, a large language model (LLM)-assisted platform for heterogeneous surrogate endpoint evaluation in clinical trials. Through natural-language interaction, SurroPilot supports the complete analytical workflow, including dataset understanding, data preprocessing, mediator and covariate selection, heterogeneous causal mediation analysis, subgroup interpretation, and automated report generation. To improve the reliability of AI-assisted statistical computing, the platform incorporates a shared context programmerinspector framework for iterative R code correction and automated validation of LLM-generated variable selections. Rather than replacing statistical methodology, SurroPilot integrates LLM with a validated heterogeneous mediation framework, allowing the LLM to assist with analytical reasoning while statistical inference is performed using established causal inference methods. Using the ACTG175 Phase III HIV clinical trial, we demonstrate that SurroPilot provides an end-to-end, reproducible workflow for heterogeneous surrogate endpoint evaluation and substantially lowers the technical barriers to applying advanced causal mediation methods in clinical trial research.
Read the full abstract

Statistics Applications

Evidence on Slowing Progress in Longevity – Is it Misleading?

A growing literature argues that progress in human longevity is slowing, and that this slowdown may signal an approaching biological limit to lifespan. The argument rests on visible flattening of cumulative life expectancy records, evaluated through linear regressions or through comparisons of average gains across decades. These summaries, however, provide only weak evidence for slowing improvements, because they cannot distinguish a true from an apparent slowdown. An apparent slowdown may arise due to periods of low variance around unchanged mean annual gains. Applying a Bayesian change-point framework to annual gains, both at the best-practice frontier and separately for each record-holding country, we find no evidence that the expected annual gain has changed across nearly two centuries, but strong evidence that its variance has moved through distinct historical regimes, with progress most regular in recent decades. The recent flattening is consistent with a low-variance regime and provides little evidence for a decline in the underlying mean pace of improvement. Claims that the best-practice frontier signals a biological ceiling should therefore not rest on the recent bend in the cumulative record alone. The frontier is better read as an empirical benchmark, the highest country-specific survival achieved under observed historical conditions.
Read the full abstract

Statistics Applications · Cross-listed from cond-mat.stat-mech

Statistical stability of random potentials to thermal and quantum activation

In numerous physical, chemical, and biological systems the dynamics can be reduced to the motion of state variables in a complex potential landscape. In case the manifold is known, the motion and response of the embedded object can be described deterministically up to stochastic effects usually associated with a noise. In contrast, if the manifold is unknown, the static and dynamic response of the state variable may be used as a spectroscopic tool to characterize the potential landscape. Inspired by a seminal work of L.\ Embon and co-workers, [Sci.\ Rep.\ 5, 7598 (2015)] we investigate the statistical properties of potential minima, in particular, their stability to thermal and quantum activation. For Gaussian random manifolds, we derive an algebraic expression to evaluate the statistical probability of the potential character (value, slope, curvature, ...). With this tool, we compute the expectation value for the rate of thermal and quantum activation and link these findings to the principal characteristics of the Gaussian potential, i.e., its Green's function. This link provides the opportunity to access information on the potential's Green's function by studying the activation behavior of an object in this manifold.
Read the full abstract

Statistics Applications

A Bayesian bivariate conditional Poisson regression for goal dependence in the English Premier League

Understanding the relationship between home and away goal counts in football provides valuable insights into match-level dynamics. While the influence of home advantage is well-established, with historical records indicating roughly 50% of matches are won by home teams (versus about 30% by the away team), properly determining the joint goal distribution while accounting for key match factors remains under-explored. In this paper, we develop a Bayesian bivariate Conditional Poisson (BCP) regression model to explicitly capture the dependence between home and away goal counts, addressing a core limitation of traditional football scoring models that assume independence or only allow for positive correlation. The BCP regression is applied to the English Premier League (EPL) data spanning three seasons, incorporating stadium attendance and committed fouls by both teams as regressors. Inference is conducted within a Bayesian framework, enabling interpretable uncertainty quantification and model validation through posterior predictive checks. Our results reveal a negative correlation between home and away goal counts and highlight an asymmetry in how match attendance influences home versus away scoring.
Read the full abstract

All authors Marcus Nolan, Wagner Barreto-Souza, Luiza S.C. Piancastelli, Raanju R. Sundararajan

Statistics Applications

How Common Are Estimated Latent-Distribution Departures From Normality? Evidence From 504 Item-Response Data Sets

Item response models usually assume a normal trait distribution, yet little is known about how often fitted distributions in real studies differ substantially from normality or which reported results are most affected. We analyzed 504 itemresponse data sets from 273 studies in the Item Response Warehouse, fitting each with the normal assumption and with a flexible distribution estimated from the responses, and we compared reliability, item estimates, predicted test responses, and person scores between the two calibrations. In more than half of the data sets the two fitted distributions differed by at least 10 percentage points of cumulative probability at some point on the trait scale, with a median maximum difference of about 11 points; about one-third reached 15 points, and nearly one-fifth reached 20 points. The estimated shapes included skewness, heavy tails, flat regions, and occasional multimodality. Differences were larger in several attitudinal, affective, and behavioral domains, although these contrasts weakened when data sets were compared within item-model families. Reliability usually changed little when item estimates were held fixed, whereas refitting the full model produced larger changes in some data sets, and item estimates, predicted test responses, and person scores did not change in parallel. Alternative flexible methods generally identified the same data sets as most unusual but disagreed somewhat about magnitude. Applied analyses should state the distributional assumption, examine a flexible alternative, and report sensitivity separately for each result used in interpretation or decision making.
Read the full abstract

Statistics Applications

How Reliable Are Psychological Measurements? The Distribution of Marginal Reliability Across 889 Item-Response Datasets

Nearly every quantitative study in psychology reports a reliability coefficient, so the field knows a great deal about the reliability that authors choose to publish. It knows much less about the reliability of the data psychology actually produces, because published coefficients pass through decisions about what to compute and what to report. We therefore measure reliability directly, applying the same estimators under the same rules to 889 datasets from the Item Response Warehouse, a public collection of item-response data that spans cognitive tests, clinical screeners, personality inventories, and attitude scales. Three findings emerge. Low reliability is common: even under a lenient definition of reliability, 30% of datasets fall below the conventional .80 threshold. The variation across datasets is real rather than statistical, since estimation noise accounts for only about one percent of it. Finally, the answer depends on the definition itself: under a strict definition the share below .80 rises to 52%, a difference large enough to change what one concludes about the field. We conclude that a reliability report should say which definition it uses, attach a measure of uncertainty, and give a strict coefficient alongside a lenient one.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Bridging Balancing Weights and Augmentation in Covariate-adjusted Analyses with Time-to-Event Endpoints: Theory and Practical Recommendations

Covariate adjustment improves the efficiency of treatment-effect analyses in randomized clinical trials, provided the adjustment targets the correct quantity. For time-to-event endpoints, two marginal targets are of primary interest: the log-rank test for the presence of a treatment effect and the marginal hazard ratio for its magnitude. Existing covariate adjustment approaches reach these targets by different ways. Augmentation adjusts the log-rank score by regressing derived outcomes on the baseline covariates within each arm. Weighting instead reweights the two arms to balance the covariates before the survival comparison is formed: inverse probability weighting does so through a fitted propensity model, while calibration weighting solves directly for weights that match covariate means. In this manuscript, we first develop balancing weighting for time-to-event endpoints, covering both calibration weights (stable balancing weights and entropy balancing) and propensity score weights, and prove that any balancing-regular weighting is first-order equivalent to the augmented log-rank score and to the root of the marginal Cox score. All three routes therefore deliver the same estimator to first order, and calibration reaches it without fitting any model. The weighted procedures thereby inherit the validity and guaranteed efficiency gain of the augmentation approach. In addition, we show that the efficiency gain grows with the prognostic strength of the adjustment covariates, while the practical caveat lies in variance estimation, for which we give recommendations to guard against finite-sample Type I error inflation. We further confirm our results through simulation studies and an analysis of the REWIND cardiovascular trial.
Read the full abstract

All authors Baoshan Zhang, Yi Chen, Yu Du, Tuo Wang

Statistics Applications

Semiparametric Functional Multistate Modeling of Alzheimer's Disease Progression with Imaging Biomarkers

Medical imaging provides rich information for predicting Alzheimer's disease progression, but existing imaging-based methods typically focus on a single survival endpoint and treat transition times as exactly observed or right-censored. Motivated by the Alzheimer's Disease Neuroimaging Initiative (ADNI), we develop a predictive framework that represents disease progression as an intermittently observed multistate process with interval-censored transition times and predicts future progression from any current disease state. We incorporate imaging biomarkers as functional covariates in a semiparametric proportional intensity model and combine functional principal component analysis with nonparametric maximum pseudo likelihood estimation. We further develop a profile score test for assessing the overall association between the imaging covariate and the multistate process. We establish the asymptotic properties of the proposed estimators and test statistic, and simulation studies demonstrate satisfactory finite-sample performance. In the ADNI application, baseline lateral ventricular morphology is strongly associated with Alzheimer's disease progression. The proposed functional multistate model also achieves the best overall predictive performance among the competing methods.
Read the full abstract

Thursday 6 August

Statistics Applications · Cross-listed from cs.LG

FarSky: Task-Aware Latent-Space Coupling for Generative Intra-Hour Solar Forecasting

Accurate solar irradiance forecasting is essential for the reliable integration of photovoltaic power into modern electricity grids. All-sky imagers (ASI) provide high-resolution observations of clouds, making them well suited for intra-hour forecasting. Recent deep learning approaches have substantially improved forecast accuracy but are often limited by deterministic predictions and a reduced capability to anticipate ramp events. This work proposes FarSky, a generative forecasting framework that leverages latent-space coupling to learn task-aware representations of sky images. A multi-task autoencoder first learns a shared latent representation for image reconstruction and irradiance estimation. A latent diffusion model then generates future latent states conditioned on recent observations, from which irradiance forecasts are directly decoded. Probabilistic forecasts are inherently obtained through stochastic sampling. The framework is developed using a multi-year ASI dataset acquired at the Plataforma Solar de Almería, Spain, and evaluated on two independent test datasets against persistence, state-of-the-art end-to-end, and generative forecasting approaches. FarSky achieves the best overall deterministic and probabilistic forecasting performance, improving forecast skill by up to 11 percentage points. Furthermore, it substantially improves ramp event detection over existing methods, achieving F1-scores above 60%. These results demonstrate the potential of combining generative models with task-aware latent-space coupling for solar forecasting.
Read the full abstract

All authors Yann Fabel, Bijan Nouri, Milon Miah, Niklas Blum, Luis F. Zarzalejo, Julia Kowalski, Robert Pitz-Paal

Statistics Applications

Hit Selection Using SSMD-Based Machine Learning Performance Metrics in High-Throughput Screening Assays

High-throughput screening (HTS) assays are central to early-stage drug discovery but are often limited by extreme data sparsity, as primary screens typically use only a single replicate per test substance. This sparsity makes conventional machine-learning performance metrics, such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC), difficult to estimate empirically because they require adequately sized labeled samples. Here, we introduce a model-based framework that derives these classification metrics from the strictly standardized mean difference (SSMD), a well-established HTS effect-size parameter. Under a Gaussian equal-variance assumption, we derive closed-form relationships linking SSMD to Youden-optimal sensitivity and specificity, and sensitivity at a preset specificity, yielding explicit estimators and exact confidence intervals from the noncentral t-distribution, even under single-replicate designs. Unlike classical statistical power, which approaches 1 as sample size grows regardless of how small the true non-zero difference between group means is, the SSMD-derived sensitivity converges to a finite population value that reflects the true degree of separation between two groups, making it a more meaningful and stable performance measure for hit selection. We demonstrate the utility of this framework in a hepatitis C virus primary siRNA screen comprising approximately 22,000 single-replicate measurements, showing that SSMD, AUROC, and sensitivity-based thresholds yield equivalent and interpretable hit sets. This work bridges classical HTS statistics and machine-learning evaluation theory, providing a statistically principled, reproducible way to estimate classification performance in ultra-low-replication screening workflows.
Read the full abstract

Statistics Applications

Modelling Athletic Ageing Relative to an Estimated Performance Envelope

Athletic careers yield sparse, irregular longitudinal series: few seasons per athlete, incomplete paths, and selection into continued play. Scientific interest often centres on proximity to peak attainable performance-a ceiling-rather than on the mean trajectory, and on how that proximity co-varies with the rate of decline. Standard tools address only pieces of this problem. Linear mixed models describe average ageing; functional principal components describe dominant modes of variation; shape-invariant models register curves about a mean template; growth charts estimate population centiles but stop short of individual latent trajectories. We develop RACE (Relative Aging Curves via Envelopes), a two-stage framework that estimates a population performance envelope as an age-conditional high centile and then models each athlete as a low-dimensional geometric transformation of that envelope. STAR (Shape Translation And Rotation) is the Stage 2 mixed model, with parameters for level, timing, and tempo. Envelope geometry determines which of these parameters are identifiable when careers are short: near-linear envelopes identify level and tempo only, whereas curved envelopes identify all three. Embedding STAR in a nonlinear mixed-effects hierarchy makes the level-tempo association a parameter of the random-effect covariance rather than a post-hoc correlation of separate fits. Simulations ask whether that association is recoverable under sparsity, how geometry governs identifiability, and how sensitive results are to envelope misspecification. Applied to Major League Baseball Statcast sprint speed and bolt rate, the analysis demonstrates how the proposed embedding separates athletic level and ageing tempo in Functional Ageing Space, and shows why timing is identifiable for bolt rate but not for sprint speed.
Read the full abstract

Statistics Applications

Authentic Multinational Federated Time-to-Event Analyses Among People with HIV in Latin America

Multinational HIV cohort studies face regulatory barriers to cross-border sharing of individual participant data, limiting centralized pooled analyses. Federated statistical methods, which exchange only aggregated information, offer a privacy-preserving alternative but have rarely been examined in real-world distributed environments for HIV research. Here, we evaluate the feasibility and analytical performance of a communication-efficient federated framework within the Caribbean, Central, and South America Network for HIV epidemiology. Virologic failure and major regimen change after antiretroviral therapy initiation were analyzed as separate outcomes; for each, we estimated cumulative incidence functions (CIFs) and fit stratified cause-specific Cox proportional hazards models via a surrogate likelihood-based federated implementation. Each site imputed missing data, conducted local analysis, and shared only summary statistics according to a coordinated computation protocol. The federated approach exactly reproduced centralized CIFs and closely approximated centralized Cox regression estimates, outperforming conventional meta-analysis for both outcomes. These findings demonstrate that authentic federated analysis is feasible for multinational HIV research and can yield results closely aligned with centralized analysis while preserving data privacy. Post-hoc feedback from local analysts, however, indicated that broader adoption will require managing the logistical and coordination overhead and ensuring harmonized data collection and quality control across sites.
Read the full abstract

All authors Kaixing Liu, Zhuohui J. Liang, Fabio Paredes, Ronaldo I. Moreira, Yanink Caro-Vega, Jiayi Tong, Zhuohang Li, Carina Cesar, Yong Chen, Jessica L. Castilho, Stephany N. Duda, Bradley A. Malin, Chao Yan, Bryan E. Shepherd, the CCASAnet

Statistics Applications

Longitudinal wearable monitoring and polygenic risk for incident major depressive disorder in the All of Us Research Program

Major depressive disorder (MDD) risk reflects both stable inherited liability and dynamic behavioral patterns, yet these dimensions are rarely examined together using long-term objective data in real-world settings. Here, we integrated genomic, electronic health record (EHR), and longitudinal Fitbit wearable data from 3,030 adults of genetically inferred European ancestry in the All of Us Research Program, 284 of whom developed EHR-recorded incident MDD after a 180-day baseline period. Time-varying Cox models examined associations of MDD polygenic risk scores (PRS), monthly wearable-derived physical activity and sleep features, and interactions between wearable features and MDD PRS with incident MDD. Higher MDD PRS, lower daily steps, lower light and vigorous physical activity, lower sleep efficiency, and greater sleep duration variability were associated with higher risk of EHR-recorded incident MDD. The associations of sedentary time and sleep duration variability with incident MDD differed across MDD PRS levels, with stronger risk associations among participants with higher MDD PRS. PRS-stratified hazard ratio curves further indicated that comparable estimated risk corresponded to more favorable behavioral levels (such as higher daily step counts and more stable sleep) among participants with higher MDD PRS than among those with lower MDD PRS. Sequentially integrating MDD PRS, baseline wearable features, monthly wearable features, and selected interactions between wearable features and MDD PRS increased model discrimination, with the C-index increasing from 0.637 to 0.705. These findings support the complementary value of inherited liability and longitudinal real-world behavioral monitoring for incident MDD risk characterization and may inform future work on genetically informed digital phenotyping for personalized risk monitoring and prevention.
Read the full abstract

All authors Yuezhou Zhang, Amos A. Folarin, Rongrong Zhong, Hyunju Kim, Shaoxiong Sun, Callum Stewart, Richard JB Dobson

Statistics Applications

Evaluating the influence of treatment-effect heterogeneity on discrimination

Analyzing the heterogeneity of treatment effects is crucial in personalized medicine to identify which patients will benefit from specific treatments. The performance of a conditional average treatment effects model to guide treatment decisions can be assessed in different ways, with an important one being the model's ability to effectively discriminate between individuals who benefit from the treatment and those who do not. While many methods and algorithms have been proposed to develop conditional average treatment effects models and individualized treatment rules, little is known about the discriminative ability that can be achieved according to the population's underlying distribution of treatment effects. In this work, we computed the discrimination that can be achieved under oracle CATE for a panel of 20 distributions with varying average treatment effects and levels of heterogeneity. The assessment included the following discrimination metrics: the c-statistic for benefit, the concentration of benefit, and the population average prescriptive effect (PAPE). Results showed that the three metrics employed in this study did not require the same levels of treatment effect heterogeneity to lead to high discrimination results. Notably, achieving high c-statistic for benefit and PAPE values required greater heterogeneity than obtaining high concentration of benefit values. The three metrics considered behave very differently across the distributions. For instance, the concentration of benefit can indicate perfect discrimination in settings with negligible treatment-effects heterogeneity.
Read the full abstract

All authors Florie Bouvier, Etienne Peyrot, Francois Petit, Raphaël Porcher

Statistics Applications

Information leakage from data revisions in retrospective forecasts

Aygün et al (2026, https://doi.org/10.1038/s41586-026-10658-6) claim that their AI-driven Empirical Research Assistance (ERA) system produces COVID-19 hospitalisation forecasts which outperform the state-of-the-art CDC ensemble by a considerable margin for the 2024/25 season. We demonstrate that the observed performance gain is attributable to information leakage in the retrospective forecasting setup, which resulted because data revisions were not taken into account. As similar mechanisms are at play in many other forecasting fields, our cautionary tale applies not just to epidemic forecasting, but is relevant to the entire emerging field of AI-assisted predictive modelling.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Fuzzy network jump models for soft dynamic clustering of graph-structured data

We introduce a fuzzy network jump model for clustering time-varying observations indexed by the nodes of a weighted graph. The framework allows flexible graph representations with spatial and temporal regularization promoting smooth soft cluster assignments across connected nodes and consecutive time points. Estimation is performed through an efficient alternating optimization scheme that exploits the quadratic structure of the regularization terms. A simulation study covering different levels of spatial dependence and cluster overlap shows that the proposed method accurately recovers the true membership probabilities and outperforms competing clustering methods. An application to traffic-network data for the city of San Francisco identifies interpretable traffic regimes and reveals their evolution over time and across connected road segments.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Two base rates, two weights: base-rate neglect has a second axis

Base-rate neglect is usually treated as one mistake: giving the prior too little weight. But turning co-occurrences into a judgment means correcting for two base rates, not one – the prior (how common the outcome is) and the cue's own frequency. Under-correcting the prior is classical base-rate neglect; under-correcting the cue is the cue-density effect, long studied but not usually framed as base-rate neglect's cue-side analogue. We write both corrections as two weights in one Bayesian equation. Response format fixes which weight a study can measure: a graded rating carries the cue-frequency weight in full; a forced choice between two outcomes cancels it and measures a companion weight in odds space instead, the same correction as it acts on the posterior odds rather than on the judged magnitude. The cancellation is exact, so a same-cue choice carries no information about the cue-frequency weight, not even its direction; a choice between two cues rather than two outcomes keeps the term and identifies the weight, and differing cue frequencies is necessary and sufficient for a choice to see it. The same weight appears in six standard learning-and-memory models, some as an exact identity and some only as a limiting or constructed correspondence; when an experiment keeps only the four cells of a cue x outcome contingency table, every model family rich enough to fit them projects onto one scalar coordinate, which alone cannot separate the accounts free to sit anywhere along it. The two neglects should be separately identifiable: an experimenter can move the regressor driving one weight without moving the regressor driving the other – a two-coefficient separation a one-parameter account cannot produce. This paper lays out the framework and the rating experiment that tests the coefficient separation; a companion paper fits the two weights to an existing colour-flavour dataset.
Read the full abstract