Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics Statistics Applications

stat.AP Watch Statistics Applications (sign in to use)

Saturday 19 September

Statistics Applications

Modeling Transition Dynamics and Network Structure in Cross-National Process Data: A Hierarchical Multi-State Survival Framework

Process data from computer-based assessments record the sequence and timing of actions through which respondents solve a task, providing information about both the pace and structure of problem-solving behavior. Modeling such processes across countries is challenging because country-by-response-group cells are often small and unbalanced and the observed transition supports can differ substantially across countries. We propose a hierarchical framework that integrates a Bayesian multi-state survival model with a network-based representation of transition structure. Partial pooling across countries yields country-specific covariate and key-action effects, transition speed, and estimates of between-country heterogeneity. Posterior transition probability networks are embedded in a common latent space using a directed graph auto-encoder adapted to heterogeneous supports, and 1-Wasserstein distances between node-role distributions are evaluated across posterior draws to characterize global network structure while propagating estimation uncertainty. We apply the framework to two problem-solving items from the Programme for the International Assessment of Adult Competencies across 14 countries. The results reveal cross-country heterogeneity in transition speed, systematic response-group differences in global network organization, item-dependent variation in within-group dispersion across countries, and local differences in routing around shared intermediate actions.
Read the full abstract

All authors Doungjun Kim, Daeun Hwangbo, Minjeong Jeon, Ick Hoon Jin

Statistics Applications

Criminal Justice Risk Assessments with Unreliable Class Assignments

Quantitative risk assessments have been used for decades to help inform criminal justice decisions. A set of predictors and a categorical response variable are used to train a statistical classifier. The trained classifier can be employed to compute risk scores for a new, unlabeled case for which a forecast is needed, one risk score for each response variable class. In the spirit of Bayes classifiers, the class with the largest risk score conventionally becomes the forecasted outcome label for that case. The largest and next largest risk scores can be very similar or very different. However, when they are much the same, the outcome class is essentially being assigned by a coin flip; the classification is unreliable. Forecasting error becomes more likely, and concerns about fairness can arise. This paper shows how Mondrian conformal prediction sets can increase classification reliability when the classifier is indecisive and provide valid estimates of forecast uncertainty. Risk forecasts for a large dataset of offenders on probation are used to illustrate the issues.
Read the full abstract

Friday 18 September

Statistics Applications

Adapting Pairs Trading to Gambling Markets A Case Study of the U.S. Presidential Election

Pairs trading exploits mean reversion in the relationship between related assets. We adapt this idea to political betting markets by modelling the combined implied probability of the two major-party nominees with a latent Ornstein-Uhlenbeck process whose mean-reversion level varies over time and whose observations contain additive noise. Model parameters are estimated from regularly sampled odds data using a state-space likelihood, with consecutive repeated values represented by a single retained observation and the elapsed number of sampling intervals preserved in the continuous-time transition. Parametric-bootstrap upper prediction bounds identify signal times at which the combined implied probability is likely to decline, and a no-intercept Bradley-Terry-type model selects the candidate-specific odds quote. The candidate-selection model is trained on 2020 U.S. presidential-election data and evaluated out of sample on 2024 data. The 2024 analysis produced 130 signals, empirical one-step coverage of 95.1%, a mean synthetic odds-price return of 1.86%, and an unannualized per-trade Sharpe-type ratio of 1.12. These returns are frictionless descriptive quantities rather than executable betting-exchange profits. The results support the integrated framework as a proof of concept for two-candidate electoral markets.
Read the full abstract

All authors Haoyu Liu, Len Thomas, Benjamin Baer, Carl Donovan

Statistics Applications · Cross-listed from math.ST

Exact maximum likelihood inference for drifted multi-sub-fractional Brownian motion at discrete observation

Sub-fractional Brownian motion is self-similar and long-range dependent but has no stationary increments, so the increment covariance is not Toeplitz and no spectral density is available. We show that a complete finite-sample likelihood theory survives nonetheless. The model is a linear trend observed at $N$ equidistant times through a superposition of $m$ independent sub-fractional Brownian motions with known Hurst indices and a common scale. Nondegeneracy follows from realising the process as the even part of a two-sided fractional Brownian motion, and the maximum likelihood estimators of the trend and of the scale are explicit. The statistics on which inference rests are pivotal, their laws depending on the sample size alone, so intervals and tests of exact level are available at every $N\ge2$, together with complete sufficiency, minimum variance unbiasedness and attainment of the Cramér–Rao bound. An explicit variance bound gives strong consistency and asymptotic normality, and simulations confirm the exact coverage and the predicted effect of a misspecified Hurst vector.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Stochastic Epidemic Model Criticism with Neural Evidence Estimation

Neural techniques for simulation-based inference (SBI) have become popular in scientific domains featuring complex simulation models. These methods train neural networks on simulated data to infer unknown parameters; their reliability depends on the simulator faithfully representing reality. In infectious disease modeling, multiple models encoding different hypotheses often provide plausible explanations of observed data. Model selection is a difficult problem when likelihoods are unavailable, as is the case with many common stochastic transmission models. We propose a model criticism methodology designed for SBI, Neural Evidence Estimation (NEE), that integrates model selection and model misspecification detection. NEE learns a model's marginal likelihood (a.k.a. evidence) directly from simulations using normalizing flows, thereby avoiding noisy or complex Monte Carlo estimators. Through synthetic experiments, we demonstrate that NEE reliably selects the correct data generating process among multiple models and flags poorly specified models. We apply NEE to a real-world modeling problem concerning competing mechanisms for influenza reinfection in a multiple wave epidemic. This work argues for the importance of principled model criticism in SBI and infectious disease modeling.
Read the full abstract

Statistics Applications

County-Level Heterogeneity in Opioid Harm Reduction and Treatment Effects: A Simulation Modeling Analysis

Opioid overdose deaths remain a severe public health crisis in the US, with heterogeneous burden across counties that differ in epidemic trajectory, baseline resources, and local context. While harm reduction through naloxone distribution and buprenorphine treatment are both evidence-based strategies, limited information on county-level effects hinders the ability of policymakers to prioritize resources across counties. We developed a simulation model of opioid use disorder (OUD), calibrated separately to six Pennsylvania counties spanning large urban (Allegheny, Philadelphia), intermediate-sized (Erie, Dauphin), and rural (Clearfield, Columbia) settings. We projected county-specific overdose mortality trajectories under three levels of increase in buprenorphine dispensing and naloxone distribution (10%, 20%, and 30% above each county's baseline), over a five-year horizon from 2025 to 2029. A 30% increase in naloxone distribution above observed county baseline levels was projected to reduce 2029 overdose deaths by approximately 70% (95% uncertainty interval, UI: 55-81%) in Allegheny County, 11% (95% UI:4-17%) in Erie County, and 28% (95% UI:2-64%) in Clearfield County. Projected reductions in overdose deaths from increasing buprenorphine were consistently smaller (10%-23%), except that in Erie buprenorphine produced larger projected reduction by 20% vs 11% for naloxone. Heterogeneity in naloxone responsiveness was strongly associated with each county's historical naloxone dispensing variability. The same proportional increase in naloxone distribution yields substantially different projected mortality reductions across counties depending on each county's baseline distribution history, a pattern invisible from mortality statistics alone. County-level context is important for informing harm reduction and treatment prioritization at the county level.
Read the full abstract

All authors Abdulrahman A. Ahmed, M. Amin Rahimian, Qiushi Chen, Praveen Kumar

Statistics Applications · Cross-listed from stat.ME

A Novel Convolution-Based Stratified Attribute Estimator for QRE Determination in R&D Tax Credit Studies

The One Big, Beautiful Bill Act reinstates immediate expensing of domestic research expenses, reducing the tax burden and incentivizing reinvestment in U.S.-based research and development including software development across a wide range of industries. In this context, accurate and defensible estimation of qualified research expenses (QREs) is of high importance. Statistical sampling provides a practical framework for estimating QREs for a well-defined population of business components (sampling frame) documented in accordance with Internal Revenue Code Section 41 (Form 6765). IRS Revenue Procedure 2011-42 permits both attribute and variable statistical methods, although the latter (stratified mean and difference estimators) are often regarded as the standard approach to QRE estimation. In this paper, we compare attribute and variable statistical methods within a simulation study and demonstrate that attribute methods offer several theoretical and practical advantages. A key concern with the simple attribute estimator is the possibility of upward bias in QRE determination arising from a preponderance of high potential QRE (pQRE), non-qualified projects in the sampling frame. We address this concern by introducing a stratified sampling design that ensures adequate representation of high pQRE projects in the sample. We demonstrate that a convolution-based stratified attribute estimator produces a valid one-sided 95% lower confidence bound on total QREs across a range of sampling frame structures.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Integrating Multivariate Adaptive Regression Splines into Small Area Estimation for Nonlinear Poverty Modeling in Java Island

Small area estimation (SAE) modeling is often used to address the unreliability of estimates in areas with limited survey samples. However, standard SAE models (Fay-Herriot) can only accommodate linear patterns, whereas in real cases, including poverty, many patterns are nonlinear. This study proposes the implementation of a supervised learning model, multivariate adaptive regression splines (MARS), integrated within the SAE framework (MARS-SAE) to address the challenge of unreliable direct estimation, particularly in nonlinear cases. The hyperparameter tuning results yielded the best model with the following hyperparameter combinations: maximum basis function (Max BF) = 10, maximum interaction (MI) = 2, minimum observation (MO) = 1, penalty = 3, with general cross validation (GCV) = 6.128. MARS-SAE demonstrated very strong performance; empirically, the model proved more efficient at reducing estimation error due to its higher relative efficiency (RE) compared to the SAE Fay-Herriot model, as well as better reduction of relative standard error (RSE) than the comparison model, and it yielded a lower relative root mean squared error (RRMSE) compared to the SAE Fay-Herriot model (13.64% vs. 13.75%). MARS-SAE has been proven to capture nonlinear patterns very well while still maintaining interpretability, making it an effective tool for the predictive and diagnostic analyses of phenomena. Thus, this research can serve as an important reference for readers regionally and in Southeast Asia, given that Java Island is a strategic region for Indonesia, the largest economic power in the ASEAN region.
Read the full abstract

Statistics Applications

A Probabilistic Modeling Framework for Transient Debris Outcomes in Low Lunar Orbits

This work presents a novel probabilistic framework for the assessment of orbital debris fragment outcomes in the low lunar orbit (LLO) regime over time horizons ranging from a few hours to a hundred days after a debris generation event. The framework develops continuous distributions for the probability of sinking (lunar collision) and non-sinking over these transient time horizons, building upon NASA's Standard Breakup Model to provide insights into the variations in the likelihood of debris outcomes across the LLO regime and offering an alternative to computationally expensive Monte Carlo simulations. The effects of perturbative forces such as solar radiation pressure are used to assess the realization of debris outcomes over varying time horizons, providing an analytical framework that bounds the likelihood of each outcome. Results indicate variations in the probability of sinking over short time horizons depending on the originating location of the debris generation event, as well as a link between the physical characteristics of fragments and their likelihood of sinking over longer time horizons. The future incorporation of this framework into broader-scale orbital environment models, mission risk assessment procedures, and policy development are briefly discussed.
Read the full abstract

Statistics Applications · Cross-listed from cs.CR

A Framework to Quantify the Probability of Future Cyber Loss Events

Cybersecurity risk quantification remains challenging due to limited operational data and difficulties in quantifying Loss Event Frequency (LEF). This paper introduces the Loss Event Frequency Security Analyser (LEFSA), a probabilistic framework that reformulates LEF estimation as machine-level Cyber Loss Event (CLE) prediction combined with hierarchical infrastructure-level aggregation. LEFSA estimates calibrated machine-level CLE probabilities from operational cybersecurity telemetry and aggregates them across infrastructure layers while accounting for machine-level dependencies. This provides a foundation for scalable, explainable, and operationally applicable cyber risk estimation at the level of machines, services, business processes, and the entire organization. The framework was evaluated using proprietary Managed Detection & Response telemetry from 23 organizations using Microsoft Defender for Endpoint. XGBoost achieved the strongest predictive performance, with a mean area under the receiver operating characteristic curve of 0.90 and consistently low calibration error across evaluation periods. The results demonstrate that operational cybersecurity telemetry contains substantial predictive information for future CLE occurrence, supporting probabilistic machine-level modeling and hierarchical aggregation as a promising foundation for quantitative, data-driven cyber risk management.
Read the full abstract

Statistics Applications · Cross-listed from econ.EM

Match forecasts in UEFA club competitions: Elo ratings versus Transfermarkt valuations

The pre-season strengths of European football clubs are usually measured by two proxies in the literature. Football Club Elo Ratings provide strictly performance-based Elo ratings from the early days of the European Cups, while Transfermarkt valuations are crowd-based estimates of squad market values. This paper compares them by evaluating their ability to forecast the results of matches played in the UEFA Champions League and the UEFA Europa League between the seasons 2020/21 and 2024/25. The two indicators yield almost identical out-of-sample accuracy when used separately. Combining the two measures leads to a modest improvement, but the best aggregation procedure is sensitive to the forecast target. Our results suggest that seeding based on Elo ratings would be (closely) optimal.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Testing Conditional Stochastic Dominance via Copula Derivatives

Comparing two populations at the same physical covariate value requires more than conditional means or isolated target-point decisions: researchers may need evidence about an entire conditional-distribution ordering over a continuum, even when covariate margins differ. This paper makes that common-value comparison estimable under an explicit structure–flexibility tradeoff and turns the resulting surface into simultaneous evidence for first-order stochastic dominance. Population-specific margins map the common covariate value into each group, while a fitted copula-derivative representation links conditional distributions across the region. Uniform inference propagates uncertainty from both the margins and dependence model through a one-sided statistic with unknown binding locations. Under correct specification within a finite copula class, smoothness and trimming conditions, and a uniquely best candidate family, the procedure admits uniform control and consistent calibration. Simulations show increasing rejection as alternatives become more distinguishable, alongside model-selection sensitivity and small-sample size distortion. In a descriptive PSID application, the high–low parental-education comparison satisfies the two-direction criterion after multiplicity adjustment, whereas adjacent education-group comparisons remain inconclusive. The framework therefore supports region-wide distributional comparison while making its structural and inferential boundaries explicit.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Robust tests for log-normal lifetimes under cyclic-stress accelerated tests and interval monitoring

Some products are, by nature, designed to operate under stress conditions that repeatedly cycle between two levels, such as batteries undergoing repeated charge and discharge, or components exposed to recurring thermal or pressure cycles. When such products are tested under accelerated conditions, the applied stress is naturally elevated in the same cyclic manner rather than held constant or increased in a single step. Cyclic-stress accelerated life testing (CyALT) reflects this by alternating the applied stress between specified levels. Units are often inspected only at fixed times, so the data consist of interval failure counts rather than exact lifetimes. Hypothesis testing plays a crucial role in this setting. It can assess whether the applied stress significantly affects lifetime, or validate the parameter values used to design the experiment. Classical tests rely on the maximum likelihood estimator, which is highly sensitive to contamination. Since interval probabilities are estimated jointly across stress conditions, a single anomalous failure count can distort the entire parameter vector and severely affect the test's significance level and power. This paper develops Wald-type and Rao-type test statistics for log-normal CyALT lifetimes under interval monitoring, based on the weighted minimum density power divergence estimator. Both tests are derived for simple and composite null hypotheses. A Monte Carlo study shows that classical tests lose control of the significance level rapidly under contamination, while the robust tests stay close to nominal with only a modest loss of power under clean data. An application to air-conditioner evaporator lifetime data further illustrates the proposed tests.
Read the full abstract

Thursday 17 September

Statistics Applications · Cross-listed from stat.ML

Triply-Scalable Equivariant Gaussian Process Modeling

Gaussian processes (GPs) provide principled probabilistic predictions while encoding prior knowledge, including equivariances. Yet, their use in large-scale scientific problems is limited by computational cost. Equivariant neural networks are common but typically lack the uncertainty quantification offered by GPs, which is valuable in applications such as molecular research. High-dimensional inputs and large symmetry groups further demand scalability. We establish results pertaining to the interplay of GP equivariance and conditioning and leverage them to obtain equivariant sparse GPs through suitable mean functions and covariance kernels. We instantiate this framework with a flexible class of integration-free equivariant kernels, yielding scalable and data-efficient GP inference. In particular, we introduce triply scalable equivariant Gaussian processes. We employ equivariant sparse variational Gaussian processes for $\mathrm{SO}(2)$-equivariant vector fields and molecular property prediction. Alongside the SVGP, we develop a matrix-free equivariant full-GP implementation that combines an exact Kronecker reduction with preconditioned conjugate-gradient solves, enabling fast and scalable evaluation of the full joint predictive density. We further compare different approaches for selecting inducing points in the equivariant sparse GP models. Our test cases include synthetic $\mathrm{SO}(2)$-equivariant fields as well as the prediction of electric dipole moments of N-methylformamide based on quantum chemistry simulations, achieving accurate, uncertainty-aware predictions at a fraction of the computational cost of classical GP inference.
Read the full abstract

Statistics Applications · Cross-listed from cs.LG

Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies

Quasars are luminous objects in the universe that exhibit stochastic brightness variations encoding information about the supermassive black holes powering them, and modeling these variations from ground-based survey data time series, known as light curves, is a statistical challenge. This paper reviews how stochastic differential equations (SDEs) have been adapted with neural network parameterizations to overcome this challenge in history. We create the Continuous-Delayed-Memory Stochastic Gradient Descent which depend on the past state of the discrete iteration process. We performed the simulation on some 2-dimensional landscape and observed some wider-exploration and more precise convergent behavior compared to Vanilla SGD by adjusting hyperparameters. Besides, we proposed a reinforcement learning structure with continuous time policy gradients for exploratory policies without solving HJB PDE, and we show that its optimality conditions recover the Gibbs policy of previous works.
Read the full abstract

Statistics Applications · Cross-listed from stat.ML

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean. Direct estimators, including prediction-powered inference (PPI), use only a domain's own labels and are imprecise where labels are few. Small area estimation addresses this problem, and we build on it to develop an integrated workflow for estimation and validation. For estimation, we propose prediction-powered smoothing (PP-S), a Bayesian model fit to each domain's prediction-powered estimate, with an extension that borrows strength across a reporting taxonomy (PP-TS). For validation, we derive a new, approximately unbiased design-based cross-validation score for choosing among direct and smoothed estimators. We study a curated benchmark with verifiable grading and deployed agent traffic graded by humans, each with every outcome observed. In both, the proposed estimators improve on the direct estimators in point and interval estimation, with near-nominal coverage. At the same sampling budget, our score selects as well as an independent validation sample does and estimates the selected estimator's error far more accurately.
Read the full abstract

Statistics Applications · Cross-listed from econ.EM

Beyond Linearity: Semiparametric Solutions to Contamination Bias with Multi-valued Treatments

We examine semiparametric solutions to contamination bias for nonbinary treatments. Deepening the discussion by Goldsmith-Pinkham et al. (2024), we detail how spline functions approximate conditional expectation and propensity score functions under weak functional-form assumptions. Reanalyzing 18 regressions across 11 studies, we compare standard linear regressions against parametric and semiparametric versions of three contamination-robust estimators. We document large point-estimate discrepancies between parametric and semiparametric approaches and find that adopting flexible semiparametric solutions may not increase statistical uncertainty substantially. We recommend that researchers verify the robustness of their conclusions to the use of semiparametric tools that address contamination bias.
Read the full abstract

Statistics Applications

Emulation strategies for Bayesian inference of regional left ventricle material parameters

Patient-specific biomechanical models of the left ventricle can relate cardiac magnetic resonance imaging to regional myocardial material properties, but existing emulator-based studies typically treat the myocardium as mechanically homogeneous, limiting representation of localised dysfunction. We propose a Bayesian surrogate-modelling framework for inferring regional Holzapfel-Ogden material parameters in a left ventricle partitioned into five physiological zones derived from the American Heart Association 17-segment model. Eight emulator strategies spanning single- versus multi-output, local versus global, and Gaussian-process- versus neural-network-based architectures were screened using parameter point-estimation accuracy; the three retained models were evaluated using empirical marginal credible-interval coverage and posterior contraction. We found that models with comparable point accuracy nevertheless differed markedly in uncertainty. A multi-output variational Gaussian process provided the most favourable balance across these criteria and was retained for the subsequent analyses. In synthetic local and global stiffening scenarios, maximum a posteriori estimates generally distinguished stiffened from baseline zones, but the nonlinear-stiffening parameters were more difficult to identify from end-diastolic observations than the stiffness-magnitude parameters. A healthy-volunteer analysis demonstrates feasibility with an incomplete observation vector and jointly inferred strain-noise scales. These results suggest that the proposed framework provides a computationally feasible, uncertainty-aware approach to regional left ventricle parameter inference.
Read the full abstract

All authors Hongjin Ren, Vinny Davies, Hao Gao, Mu Niu, Benn Macdonald

Statistics Applications

Comparing statistical learning models in wastewater-based epidemiology: An application to norovirus

Wastewater-based epidemiology (WBE) is an increasingly important tool for infectious disease surveillance, but there has been limited direct comparison of modelling approaches for predicting pathogen concentrations across space and time. We compare the predictive performance of six modelling approaches using norovirus in England as a case study, considering 3,232 wastewater samples from 152 sewage treatment works collected between May 2021 and March 2022 as part of the UK Environmental Monitoring for Health Protection programme. The benchmark was a Bayesian spatio-temporal model using Integrated Nested Laplace Approximation (INLA) with the Stochastic Partial Differential Equation approach (SPDE), compared with Lasso regression, Generalised Additive Models (GAM), Bayesian GAM, Extreme Gradient Boosting (XGBoost), and Random Forest. Models were evaluated using 10-fold spatial-block cross-validation with metrics including mean squared error, bias, correlation, empirical coverage of 95% prediction intervals, interval score, and computational cost. Random Forest was the best overall performing model, achieving the best interval score and nearest to nominal empirical coverage (95.3%), while maintaining point prediction accuracy comparable to XGBoost. The INLA-SPDE model also performed well, with near-nominal empirical coverage, the third best interval score, and consistently low bias across all evaluated metrics. Comparing predicted spatio-temporal trends, both models yielded similar spatial patterns but notable differences in uncertainty estimation. Our findings highlight a trade-off between predictive accuracy, uncertainty quantification, and computational efficiency. Ensemble machine learning methods are well suited to rapid prediction, whereas Bayesian geostatistical models remain valuable when probabilistic decision support is a priority for public health surveillance
Read the full abstract

All authors Caelan McNamara, Fion Tan, Ella White, Elizaveta Semenova, Marta Blangiardo

Statistics Applications · Cross-listed from stat.ME

Gain-function optimisation of graphical multiple testing procedures for confirmatory clinical trials

Graphical multiple testing procedures are a flexible and transparent way to control the family-wise error rate when a confirmatory trial pursues several label claims, but they leave open the question of which graph to use. In practice sponsors often fall back on fixed-sequence or Holm procedures that may poorly reflect what the trial is actually trying to achieve. We propose to instead choose the graph that maximises an explicit gain function, which states what each possible set of rejected hypotheses is worth to the sponsor: typically nothing until a regulatory hurdle is cleared, then an incremental amount for each further claim. Conventional power criteria are recovered as special cases, and because the search is confined to graphical procedures, family-wise error rate control holds whichever graph is selected. We set out a structured procedure for eliciting the gain function from the clinical, commercial, and regulatory members of a trial team. The proposed framework also handles uncertainty in the treatment effects and correlations assumed at the design stage, by averaging performance over their plausible values rather than fixing a single assumption. For group sequential designs, we describe how it can further reward early claims. Five examples, drawn from real and hypothetical pharmaceutical trials, show that the best graph depends on how trial success is defined, how uncertain the design assumptions are, and, in a group sequential setting, when claims can be established.
Read the full abstract

All authors Alexander D. V. Spiers, Michael J. Grayling, Graham M. Wheeler, Adrian P. Mander

Statistics Applications

Identifying Damage Pathways Linking Sequence Composition to Storage Failure in DNA Data Storage via High-Dimensional Mediation Analysis

DNA data storage offers extraordinary information density and long-term durability, but its reliability is limited by sequence-dependent errors introduced during synthesis and accumulated during storage. It remains unclear how sequence composition is associated with storage failure through specific molecular damage components. We develop a high-dimensional semiparametric mediation framework for survival outcomes. GC content is treated as the exposure, a high-dimensional baseline damage spectrum (a vector of per-read damage counts stratified by trinucleotide context and error type) as the mediator, and storage-quality failure as the outcome. Nonlinear covariate effects in both the mediator and survival models are approximated using deep neural networks. A three-step procedure combining product-of-coefficients screening, Smoothly Clipped Absolute Deviation (SCAD) penalized estimation, and joint significance testing is developed for mediator selection and inference. Applied to an aging experiment on electrochemically synthesized DNA, the method identifies 14 significant mediators, all corresponding to single-base deletions, with estimated mediated effects concentrated in trinucleotide contexts ending in C. These results reveal deletion-type damage as a major pathway linking sequence composition to reduced archival reliability and suggest candidate sequence features for future optimization and error-control strategies. The proposed framework thus offers a mechanism-oriented statistical approach for understanding and improving the reliability of DNA data storage.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Mutual Information as a Tool for Optimal Classification: Application to Identifying Rapid-Responding Behaviour

Existing methods for identifying rapid-responding behaviour in large-scale assessments require parametric assumptions about the population. In this study, we propose a novel, non-parametric, mutual information-based framework of methods as an alternative. The methods within this framework compute the mutual information of the observed responses and discretised response times and maximise the information gain to determine a threshold that differentiates rapid responses from engaged responses. The only difference between the methods is the number of categories in the relevant variables. We present three methods explicitly. The first method uses response correctness and binarised response times. The second method uses correctness and categorises time into three groups. The third method uses raw responses and binarised times. We applied these methods to mathematics achievement data collected through the Programme for International Student Assessment in 2022. Furthermore, we examined the behaviour and usability of the first method in certain realistic conditions at the population and realised levels. The results indicated that the proposed framework is a viable alternative for identifying rapid responses. The framework's novelty lies in its non-parametric nature and its ability to utilise raw responses instead of correctness. Finally, we discuss some possible future research directions on this topic.
Read the full abstract

All authors Santeri Holopainen, Jari Metsämuuronen, Mikko-Jussi Laakso, Janne V. Kujala

2609.19781 · 17 Sep · Main text: 36 pages, 8 figures. Supplement: 12 pages, 1 figure. PISA 2022 data: https://www.oecd.org/en/data/datasets/pisa-2022-database.html. R codes: https://github.com/sajomaho-uni/Empirical-Results-of-MaxMI-for-RRB-Identification. Changes: added references, appendices, better description of AI use, discussion about the advantages of MaxMI, and new analyses

Statistics Applications · Cross-listed from eess.IV

Compression Hurts, Pooling Helps: Information Loss in Rayleigh-Scale Estimation from B-Mode Ultrasound

Clinical B-mode images are widely available as potential data sources for quantitative ultrasound (QUS) analysis for tissue characterization. However, standard clinical ultrasound devices apply unknown log-compression to RF envelope data before display and storage. Previous work has demonstrated estimation of the underlying RF envelope statistics in the presence of an unknown compression law. Using Fisher information analysis, we show that finite-offset log compression causes severe information loss when estimating the Rayleigh scale $\sigma$, which controls diffuse speckle. For a single image window, unknown compression raises the minimum achievable variance for unbiased estimation of $\sigma$ by a compression-independent factor of approximately 34.98. When $M$ equal-sized windows share the same unknown compression settings, the excess variance decays as $1/M$; even in the most favorable regime, reducing the variance inflation factor below 1.1 requires 340 windows. Our analysis treats the contrast parameter $a$ as unknown and the boundary offset $b$ as known; estimating $b$ experimentally shows even larger variance. We validate this theory using synthetic estimation experiments and demonstrate RF-scale recovery on real RF-envelope windows from the OASBUD dataset. Together, these results clarify the limitations of using routine B-mode images for QUS.
Read the full abstract

Wednesday 16 September

Statistics Applications · Cross-listed from stat.ME

Beyond Actor-Partner Associations: A Target-Trial Framework for Causal Spillover Effects in Longitudinal Dyadic Data

Actor-partner interdependence models (APIMs) describe how one member's exposure is associated with that member's and the partner's outcomes, but actor and partner coefficients are causal only under additional design and identification assumptions. We formulate APIM-style questions as dyadic target trials under within-dyad interference and make three links between the associational and causal formulations explicit. First, under joint exchangeability, positivity, and a homogeneous correctly specified treatment-related conditional mean, APIM actor, partner, interaction, and joint coefficients equal the corresponding causal contrasts. Second, for binary dyadic treatment, multiplying the true member-specific propensities induces an exact one-parameter discrepancy delta(H) = Cov(A_1, A_2 | H): the four joint-cell errors are (+delta, -delta, -delta, +delta), total variation is 2|delta|, and IPW contrast bias has a closed-form representation. Third, the same discrepancy affects AIPW through its product with outcome-regression error. A 36-scenario simulation and a true-marginal factorization diagnostic show that large separate-propensity bias under residual dependence is primarily factorization bias, while flexible AIPW reduces nonlinear-model bias but not weak-support instability. In the strongest dependence condition, factorizing even the true marginals reversed the spillover contrast and inflated the interaction to nearly five times its true value. We apply the framework to 2,400 Health and Retirement Study couples studying retirement transitions and later depressive symptoms. Joint retirement was rare, residual treatment dependence was substantial, and estimates were sensitive to support and propensity flooring. The contribution is a design-and-diagnostic bridge from APIM questions to causal dyadic analyses, not a new general interference or AIPW theory.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Earth and space observations meet complex algebras: from complex to octonions for multivariate autoregressive time series analysis

Many datasets arise as multivariate time series observed at irregular time intervals, particularly in Earth observation and astronomical measurements. Classical time-series models assume that time is discrete and observations occur at equally spaced intervals, limiting their ability to capture dynamics when observation gaps vary. Existing approaches to irregular sampling rely on continuous-time formulations, which assume that observation intervals are sufficiently small. We address this limitation by introducing a discrete-time framework that accommodates irregular observation gaps while preserving multivariate dependence structures. We introduce a framework for the analysis of irregularly observed multivariate time series based on hypercomplex autoregressive processes. The framework embeds multivariate observations into a hypercomplex algebraic structure, allowing multiple variables and their interactions to be represented within a single mathematical entity while incorporating irregular observation gaps. The resulting process admits a structured matrix representation, which enables the model to be expressed as a state-space system. This formulation provides an estimation methodology in which model parameters can be inferred using Kalman filtering techniques. The state-space representation allows recursive estimation and prediction in the presence of irregular sampling intervals. The performance of the methodology is illustrated through data and applications to remote sensing and astronomical datasets, where observations occur at nonuniform time intervals. The results demonstrate that the approach captures dynamics and cross-variable interactions difficult to model using traditional time-series techniques. The proposed framework provides a methodology for analyzing irregularly sampled multivariate time series and opens possibilities for modeling observational data across disciplines.
Read the full abstract

All authors Susana Eyheramendy, Felipe Elorrieta, Wilfredo Palma, Javier Lopatin

Statistics Applications · Cross-listed from physics.soc-ph

Ensuring proportionality: a logical model for compensatory seats added to multi-member constituencies

How many compensatory seats are needed to ensure (full) proportionality in two-tier electoral systems with multi-member constituencies? This question is answered by estimating the discrepancy between seats and votes rigorously, probabilistically and empirically. In 1919, Pólya showed that the Jefferson/D'Hondt method favours large parties. This seat surplus accumulates across constituencies. In some countries (e.g., Portugal, Poland, Spain and Turkey), this accumulation directly leads to quantifiable disproportionality in the final parliament. Elsewhere compensatory seats counteract this accumulated seat surplus. Using a logic-first approach, closed formulas for the required number of compensatory seats are derived given regional and national apportionment methods, the number of relevant parties $n$ and the size of the largest party $p$. If there are $c$ constituencies, it is derived logically that a two-tier system built on local D'Hondt apportionment in multi-member constituencies needs around $\frac{c}{2}(n-\frac{1}{p})$ compensatory seats to ensure proportionality. In words, that is half a compensatory seat per constituency times a number that increases with the number of parties and the size of the largest party. The validity of the derived formulas is analysed empirically using the extensive recent CLEA dataset. Many quantities do not depend much on the total parliament sizes, which means that the variance can be reduced by artificially varying the house sizes. Portugal, Poland, Spain and Turkey all employ a one-tier D'Hondt system and the analysis shows that if they were to add compensatory seats to the system, reasonable numbers would be 40, 75, 75 and 120, assuming that no other properties of the systems change.
Read the full abstract

Tuesday 15 September

Statistics Applications · Cross-listed from stat.ME

Coherent Hierarchical Forecasting for Proportion and Discrete Time Series

Hierarchical and grouped time series arise when a multivariate time series is forced to satisfy a set of aggregation constraints, motivating forecast reconciliation methods that ensure coherent forecasts of such hierarchical structures. Many real-world applications involve discrete or bounded supports, introducing additional challenges that are not addressed Gaussian-based reconciliation methods. We develop a post-hoc hierarchical forecasting approach to construct coherent forecast hierarchies for discrete and bounded time series. The method constructs coherent forecasts by convolution and exponential tilting, preserving the distributional properties and the underlying support throughout the hierarchy. We evaluate the proposed approach against state-of-the-art reconciliation methods for both discrete and continuous settings, demonstrating strong performance across a range of experiments. Through simulation studies and empirical applications in epidemiological and demographic data, we show that the method provides reliable and coherent distributional forecasts in challenging scenarios.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Online Bayesian Model Averaging with Joint Uncertainty Quantification for Models and Regression Coefficients in Binary Regression

Streaming binary response data arise in many applications, including virtual learning platforms where student responses are collected sequentially and estimated probabilities of correct responses may inform future question assignment. Logistic regression provides an interpretable framework for such analysis, but the data may support multiple plausible predictor subsets. Bayesian model averaging (BMA) accounts for such model uncertainty by averaging inference across competing models. Since repeatedly applying BMA to accumulating data can be computationally burdensome, we develop an online implementation based on renewable estimation. At each update, retained summaries and new observations are used to approximate the joint posterior of models and model-specific coefficients without revisiting historical data, enabling point and interval estimates that incorporate model uncertainty. Simulations show that online BMA closely approximates its offline counterpart in coefficient estimation, model and variable importance, prediction, and interval estimation of predictive probabilities, while substantially reducing computational cost. In a virtual learning application, we find substantial uncertainty across competing predictor subsets and large variability in credible interval widths across questions. Our method distinguishes questions with precise estimates from those with substantial uncertainty, providing information beyond point estimates for adaptive question assignment.
Read the full abstract

Statistics Applications · Cross-listed from cs.LG

Accelerating Diffusion Sampling via Speculative Draft Trees

Speculative sampling accelerates diffusion model generation by drafting inexpensive candidate states and correcting them under a coupling that preserves the target distribution exactly, reducing the number of expensive target evaluations. Existing diffusion samplers, notably those based on reflection maximal coupling, are topologically constrained: their lookahead drafts form a chain graph, a single linear sequence, which inherently limits the acceptance rate per target evaluation. We connect speculative sampling in diffusion models to relative entropy coding (REC). This perspective shows the lookahead need not be linear and motivates our central contribution, draft trees, which enrich the candidates considered per round and lower the target function evaluations. We further adopt greedy rejection sampling, an REC algorithm, as the draft-target coupling, improving acceptance while guaranteeing exact target samples. Experiments across diverse target and draft models demonstrate up to 8.3% acceleration over the reflection coupling baseline in practical settings.
Read the full abstract

All authors Marcello Bullo, Yanxiao Liu, Öykü S{\i}la Güner, Arpan Mukherjee, Deniz Gündüz

Statistics Applications · Cross-listed from astro-ph.CO

$B$-sure. Part II. Scattering transforms as robustness test for tensor-to-scalar ratio detection from CMB observations

Galactic foregrounds represent a major contamination to the measurement of primordial $B$-modes from observations of the Cosmic Microwave Background polarisation. Even after the application of component separation algorithms, foreground residuals may potentially still bias the estimate of the tensor-to-scalar ratio $r$, causing a false detection. In this work, we present the methodology of a robustness test for the validation of an eventual detection of primordial $B$-modes, as obtained by a future, LiteBIRD-like satellite experiment. The goal of the test is to identify the foreground residuals contamination by looking for non-Gaussian properties in the CMB $B$-modes map, recovered through blind component separation algorithms. We adopt scattering transforms (ST) as our summary statistics sensitive to the non-Gaussian features of foreground residuals and to their correlation with foregrounds tracer maps. We characterise and validate the methodology on realistic sky simulations with different levels of foregrounds complexity. The proposed test is able to identify a bias on the tensor-to-scalar ratio of $\gtrsim 10^{-3}$ in $\sim 90\%$ of our simulations, with this bias value being of the same order of the accuracy targeted by LiteBIRD. Additionally, for our particular experimental configuration, the test is passed when the bias is lower than the sensitivity on the $r$ parameter, and no warning is raised. These results provide an important step forward in the development of statistical tools for validating future measurement of cosmological parameters, against foregrounds contamination.
Read the full abstract

All authors Claudio Ranucci, Sébastien Pierre, Léo Vacher, Alessandro Carones, Nicoletta Krachmalnicoff, Erwan Allys, Jean-Marc Delouis, Paolo Campeti, Giuseppe Puglisi, Carlo Baccigalupi