Statistics Applications
Tree statistics improve neural simulation-based inference of epidemiological parameters over case counts alone
Zena Lapp, Emma E. Goldberg, John R. Tipton et al. (+3 more)
During an infectious disease outbreak, accurate estimates of epidemiological parameters such as the reproduction number and infection duration can inform the public health response. Recently, neural simulation-based inference (NSBI) methods have become popular for estimating these parameters due to lower computational costs and more input data flexibility than traditional methods. Input data modalities that have been evaluated independently in an NSBI framework include case counts and transmission trees. Here, we evaluate whether combining information from these two data types outperforms using information from each individually with NSBI. To do so, we simulated
stochastic susceptible-infectious-recovered (SIR) outbreaks across a range of basic reproduction numbers (R0) and average infection durations (D) to generate training, validation, and testing
datasets. We then calculated, for each simulated outbreak, summary statistics from case count trajectories and from transmission trees. Using these features, individually and in combination, we performed a type of NSBI called neural ratio estimation (NRE) to estimate R0 and D. We also calculated maximum likelihood estimates of the parameters by fitting a deterministic SIR model using the case count trajectory. All three NRE models outperformed the baseline method. Furthermore, incorporating tree summary statistics improved the estimation of both R0 and D over case count summaries alone. This suggests that trees provide valuable additional information for estimating epidemiological parameters above and beyond case counts alone.
Read the full abstract ▾
All authors Zena Lapp, Emma E. Goldberg, John R. Tipton, Thea Sukianto, Dave Osthus, Lauren A. Castro
Statistics Applications
Estimating Optimal Dynamic Treatment Regimes for Personalized Education: A Tutorial and Applications with Machine Learning
Chenguang Pan, Yuxuan Li, Thomas Harris et al. (+1 more)
Optimal dynamic treatment regimes (ODTRs) are increasingly used to generate sequences of personalized recommendations that maximize a final outcome of interest. Although ODTRs are widely used in biostatistics and precision medicine, their applications in education remain limited, especially for longitudinal decision-making. This study translates ODTR methods for single- and multi-stage designs into individualized high school math course recommendations, providing step-by-step guidelines and demonstrations with data from the High School Longitudinal Study of 2009 (HSLS:09). We use Targeted Maximum Likelihood Estimation with ensemble machine learning to estimate ODTRs that maximize students' math achievement and college enrollment, imposing feasibility constraints based on domain knowledge and propensity-score thresholds to ensure recommendations are practically implementable. Results show that unconstrained regimes yield higher estimated values but infeasible recommendations, while feasible regimes produce realistic pathways with stronger empirical and expert-informed support. Finally, we discuss practical considerations for designing ODTRs in education.
Read the full abstract ▾
All authors Chenguang Pan, Yuxuan Li, Thomas Harris, Youmi Suk
Statistics Applications · Cross-listed from cs.SI
Some Twitch chats exhibit multifractal characteristics resembling stream-of-consciousness narrative
Marcin Ból, Stanisław Drożdż, Robert Kluszczyński et al. (+2 more)
Live-streaming platforms generate high-frequency records of collective human activity in which many loosely coordinated users react in real time to shared external stimuli and to one another. We investigate whether the temporal organization of Twitch chat messages exhibits multifractal signatures characteristic of complex dynamical systems. Selected high-activity chats were converted into time series by measuring the length of each consecutive message in words, with platform emotes treated as individual
tokens. The resulting signals display heterogeneous fluctuations, bursty organization, and heavy-tailed distributions that, in several cases, are better captured by q-Weibull forms than by conventional light-tailed models. Autocorrelation analysis reveals persistent temporal dependence over broad ranges of message lags, indicating that the variability is not produced by independent message generation alone. MFDFA shows that some chats possess well-developed multifractal spectra and generalized Hurst exponents strongly dependent on moment order, whereas others exhibit weak scaling, approximately monofractal behavior, or no well-defined scaling regime. Surrogate tests indicate that temporal correlations are essential for the observed multifractality, while heavy tails enhance its apparent strength. Under particular dynamical conditions, Twitch chat may therefore form an emergent collective signal whose fragmented, reactive, associative, and multiscale organization is structurally reminiscent of stream-of-consciousness narrative. This expression is used strictly as a structural and dynamical analogy, not as a claim of a unified collective mind or of semantic, cognitive, or phenomenological equivalence with literary stream-of-consciousness writing. The results position live-stream chat dynamics within the broader physics of complex, intermittent, human-generated signals.
Read the full abstract ▾
All authors Marcin Ból, Stanisław Drożdż, Robert Kluszczyński, Jarosław Kwapień, Marcin Wątorek
Statistics Applications
On the Benefit of Blocking for Online Experiments with Skewed Data
Abbas Zaidi, Matthew Reimherr, Rina Friedberg et al. (+2 more)
This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%) due to the limits of discretization, the authors demonstrate it offers two crucial structural benefits over post-hoc statistical adjustments. First, fixed-weight blocking correctly targets the true average treatment effect, avoiding the severe bias introduced by efficiency-weighted alternatives. Second, blocking localizes extreme outliers, enabling targeted within-block winsorization that effectively controls noise without destroying the tail-end treatment effect.
Read the full abstract ▾
All authors Abbas Zaidi, Matthew Reimherr, Rina Friedberg, Fabian Martinez, Richard Mudd
Statistics Applications
Estimating Uncoded Crash Factors with Tabular Foundation and System One Models: Kumo Tabular and Jev
Amir Rafe and Subasish Das
Road safety programs count the coded fields of police crash records, while the officer's narrative, which often records factors the fields omit, is rarely read. A safety office thus cannot tell how much its counts miss or where to review. This study develops and evaluates a system that joins both views of the 5,601,890 Texas crashes from 2017 to 2025 into population estimates with stated validity. An in-context tabular foundation model, Kumo Tabular, reads the coded record of every crash, a calibrated System One model, Jev, reads the narratives of two probability samples, and human judgments recalibrate its probabilities. A multiwave predict-then-debias
estimator joins the three tiers, and a second human tier drawn with recorded probabilities checks the estimates by design. For hydroplaning, medical episodes, fatigue, animals, and phone use, the narrative documents more injury crashes than the coded field, 15,074 against 7,340 for phone use, and the human check agrees with all fifteen estimates within its margin. A re-read list ranked by Kumo Tabular finds confirmed discordance 7 to 58 times as often as random reading. At the planning cost of human coding, one further round of human judgments would cut the root mean square relative half-width from 22.0 to 16.2 percent, against 21.2 for reading every narrative. Two calibrated readers of different views, joined by a sampling design, give a safety office counts, a discordance map, a validated re-read list, and a reading budget, with Kumo Tabular reading the table at 15 times the speed of TabPFN 3.5.
Read the full abstract ▾
Statistics Applications
Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches
Pablo Alcain, Jason Kang, Mia Garrard et al. (+3 more)
Evaluating "neutral launches" (e.g., infrastructure upgrades) using traditional confidence interval overlap is flawed: it is dangerously permissive with scarce data and excessively restrictive with abundant data. To resolve this, this paper introduces Expected
Bayesian Loss (EBL), a continuous metric that quantifies both the probability and expected severity of metric degradation. Computable directly from standard frequentist estimates, EBL explicitly penalizes empirical noise and high-variance experiments. Validated against expert decisions, EBL provides experimentation platforms with a rigorous, tunable guardrail that aligns statistical safety with institutional risk appetite.
Read the full abstract ▾
All authors Pablo Alcain, Jason Kang, Mia Garrard, Marine Veits, Houssam Nassif, Abbas Zaidi
Statistics Applications · Cross-listed from stat.ML
Conformal Prediction for Spatially Dependent Data via Sequential Whitening
Ayush Baran Sen and Arkajyoti Saha
Split conformal prediction uses prediction errors on held-out (calibration) data to determine how wide the prediction intervals should be. It guarantees distribution-free finite-sample coverage when these errors and the error at the target site are exchangeable. This assumption may fail under spatial dependence and nonrandom sampling geometry. Existing spatial methods use fitting residuals to remove the predictable part of spatial variation from calibration and target errors. However, the spatial variation that only the calibration residuals can predict remains in both the target and calibration errors, reducing the efficiency and stability of the interval. We address this by additionally conditioning on the calibration residuals sequentially, which scales to large networks through nearest-neighbour approximations. Under a correct working covariance and an elliptical residual law, the resulting interval has exact finite-sample coverage under any spatial design, and under further conditions it is asymptotically oracle efficient. We also bound coverage loss under covariance misspecification and develop a diagnostic that identifies regions at risk of undercoverage. In simulated data, our method produces narrower and more stable intervals than global and localized state-of-the-art alternatives. In a national PM2.5 application, it produces narrower intervals within the network and identifies regions at risk of coverage failure.
Read the full abstract ▾
Statistics Applications
Multivariate LSTM Modeling for Wastewater Quality Prediction
Sophie Laruelle and Innocentia Bénédicta Sossou
The proposed approach relies on high-frequency multivariate time series collected at the Seine aval (SIAAP 1 ) treatment plant, including pH, temperature, conductivity, and total suspended solids, as well as exogenous variables related to precipitation and measurements from an upstream plant. The objective is to forecast wastewater quality over a 24-hour horizon. A multivariate Long Short-Term
Memory (LSTM) recurrent neural network is implemented to capture complex temporal dependencies and nonlinear patterns in the data. The model is trained on one year of data and validated on four months of data (one per season), and is compared with a persistence model and SARIMA models. The evaluation shows an overall superiority of the LSTM model, particularly for variables exhibiting high levels of noise and nonlinearity.
Read the full abstract ▾
Statistics Applications · Cross-listed from stat.ML
Extreme Binary Classification: Extreme Value Theory for Extreme Constraint on False Negative
Samuel Gruffaz, Muhammad Fawad and Jaakko Nevalainen
While binary classification is one of the most extensively studied problems in machine learning, the regime in which the goal is to learn a classifier with an almost zero false negative rate remains largely unexplored. In this paper, we introduce the Extreme Binary Classification problem, where the objective is to learn a classifier whose false negative rate $\alpha$ is constrained by $\epsilon_{N_1}=o_{N_1\to\infty}(1/N_1)$, with $N_1$ denoting the number of positive examples in the training set. To address this problem, we propose a threshold adaptation method theoretically grounded in guarantees derived from Extreme Value Theory, together with a feature selection procedure based on a permutation test applied to sample maxima. Experimental results on four real-world
datasets of varying sizes demonstrate that our approach compares favorably with state-of-the-art methods. In addition, we illustrate its interpretability through an application to a cancer screening
dataset.
Read the full abstract ▾
Statistics Applications · Cross-listed from stat.CO
Overlapped We Stand, Separated We Fall: Software Approaches to Diagnosing Separation Phenomena in Categorical Response Models
Thomas Rusch, Lukas Sablica and Kurt Hornik
In statistical models for categorical responses, including binary, nominal and ordinal responses, a phenomenon called “separation” may occur which can render maximum likelihood estimate nonexistent and/or nonunique. Based on Sablica et al. (2026)'s general characterization of separation phenomena via structure vectors, in this article we discuss computational, numerical and practical aspects of separation phenomena in different software packages, including consequences, remedies and ways of diagnosing them. We further introduce the R package divoRce which provides the most comprehensive software suite to diagnose separation in existence. It allows to check for separation, to distinguish between types of separation, characterize the recession cone and identify variables and observations contributing to separation for almost any categorical response model in the literature, including models with baseline-category link, cumulative link, adjacent-category link and sequential link specifications. Diagnostics are based on methods for computational geometry and linear programming and can be done in floating-point or rational arithmetic with different solvers. The functionality is easily extendible by wrappers and S3 methods to accommodate implementations in third-party packages or of new categorical response models. We discuss our implementation in detail and show our software in action in a wide range of applications. We recommend to incorporate our checks for separation and existence of the MLE as a diagnostic routine in every categorical data analysis both in third-party software implementations and as part of a standard model diagnostic workflow.
Read the full abstract ▾
Statistics Applications
Learning joint probabilistic weather forecasts from station observations alone
Chaeyeon Yi and Yun Am Seo
Assessing compound weather risks requires forecasts representing dependence between variables. CLARA (Calibrated Advection-Routing
Attention) learns joint Gaussian predictive distributions of five surface variables from station observations alone, without numerical weather prediction or reanalysis; the approximately 28,000-parameter model supports CPU training and prediction. Across six multi-year folds on 96 stations, its lead-mean energy score is 4.9% lower than that of a learned comparator with matched temporal inputs (4.7% with a similar parameter count) and 11-65% lower than those of statistical baselines. Holding marginal variances fixed, removing learned correlations worsens joint negative log-likelihood by 1.0-2.8 nats per station. A covariance-scale
estimator, proved consistent under stated assumptions, improves short-lead calibration but over-corrects at long leads. Synthetic interventions show an
attention-bias coefficient alone does not measure forecast influence. Retrained in ten regions on six continents, CLARA outperforms persistence in all 60 multi-year region-lead comparisons and a similarly sized learned model in 57 of 60.
Read the full abstract ▾