Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics Statistics Applications

stat.AP Watch Statistics Applications (sign in to use)

Friday 4 September

Statistics Applications

Kumaraswamy-Teissier Distribution: Bayesian and Non-Bayesian Inference, Regression Modeling and Climate Application

This study introduces the three-parameter Kumaraswamy–Teissier distribution (KTD) by applying the Kumaraswamy-G construction to the Teissier distribution. The proposed distribution provides additional shape flexibility for modelling positive-valued observations and accommodates different density and hazard-rate shapes. Several statistical properties, including moments, the moment generating function, quantile function, and order statistics, are derived. Maximum likelihood, maximum product spacing, and Bayesian methods are considered for parameter estimation, and simulation results demonstrate improving estimation accuracy with increasing sample size. A median-based regression model is further developed for the KTD, with its finite-sample performance examined through simulation. The practical utility of the proposed regression model is demonstrated through an application to monthly gridded rainfall data from the Northwest Himalayas, incorporating annual and semi-annual harmonic terms to capture seasonal variation. Its performance is assessed against Kumaraswamy and beta regression using residual-based goodness-of-fit diagnostics and likelihood-based model selection. The results demonstrate effectiveness of the proposed distribution for modelling heterogeneous and skewed positive-valued rainfall data.
Read the full abstract

All authors Kamana Mishra, Neeraj Poonia, Tanmay Kayal, Sarita Azad

Statistics Applications

Estimating water levels in the High Plains Aquifer by synthesizing satellite data with groundwater well observations

The High Plains Aquifer (HPA) is a critical water resource in the Central United States, yet its depletion remains a major concern. While the Gravity Recovery and Climate Experiment (GRACE) satellite mission provides large-scale estimates of liquid water equivalent thickness (LWET), its coarse spatial resolution (approx. 24 km) limits local inference. In contrast, well observations from the National Ground-Water Monitoring Network (NGWMN) offer valuable but spatially sparse measurements of depth-to-groundwater. In this paper, we develop a downscaling framework for the satellite data that integrates the two sources and covariates using a Bayesian hierarchical framework. Our model uses a latent Gaussian Markov Random Field (GMRF) that describes the groundwater storage at a high spatial resolution. To address computational issues, we use a basis representation approach specified via the Moran basis. We find that fine-scale covariates like irrigation intensity and precipitation help refine spatial predictions. We thus provide, to our knowledge, the first statistically-rigorous approach for downscaling groundwater information based on GRACE satellite data and NGWMN groundwater measurements. Our approach yields high-resolution estimates of groundwater variations across the HPA from 2002–2022. The resulting fine-scale inference provides valuable insights into groundwater dynamics, highlighting the effects of land use and local extraction patterns.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Nonparametric Hypothesis Testing of High-dimensional Clustering With Application to Single-cell RNA Data

Single-cell RNA sequencing studies routinely use clustering to define putative cell types and cell states, yet the observed separation may arise from sampling variability rather than genuine biological heterogeneity. This paper studies formal significance testing of such clustering structure in high-dimensional data. Existing SigClust methods assess clustering significance through Monte Carlo simulation under a Gaussian single-cluster null, but this assumption can be unreliable for normalized gene expression data and other non-Gaussian settings. We propose SigClust-LCP, a nonparametric extension that models a single cluster by a log-concave distribution. To make this approach computationally feasible in moderate to high dimensions, we develop a score-matching estimator for log-concave projection inspired by recent generative modeling ideas. We establish theoretical guarantees for the estimator and for its use in clustering significance testing. Simulations show that SigClust-LCP controls Type-I error more reliably than existing methods across a range of unimodal and mixture distributions while retaining competitive power. In a single-cell RNA sequencing analysis of Hydra cells, the method avoids spurious subclusters within annotated cell populations and supports biologically meaningful separation across lineages and body-axis regions.
Read the full abstract

Statistics Applications · Cross-listed from physics.soc-ph

One equation allocates playing time in competitive team sports: the rulebook sets the arithmetic and the league sets the habit

Every team sport that lets a team replace players rations the replacements, and the ration settles who plays and for how long. Across eight codes, three in both sexes, we measure how playing time is allocated and find one equation behind it, complete in the one-way codes, its terms separating what the rulebook fixes from what a competition chooses. Under scarcity every league behaves alike: loaded starters are kept on at the same rate on four continents. Under freedom each expresses a preference, and the size of a league's departure is priced near one-for-one by what its own withdrawal minute prices (lambda = 0.87, 95% interval 0.54 to 1.21, twelve leagues), a law confirmed in four out-of-sample predictions and refused in one. The ordering of leagues by appetite persists across a rule change, and club wealth does not price it. Rules are the geometry of a competition; culture is its material.
Read the full abstract

2609.05584 · 4 Sep · 36 pages, 5 figures, 1 table, 5 Extended Data figures and 4 Extended Data tables; 75-page Supplementary Information included as an ancillary file. Replication record (manuscript, pipeline, tests and derived tables as one object): https://doi.org/10.5281/zenodo.22304792

Statistics Applications

Symptom clusters in Long COVID in the UK: prospective community-based cohort study using unsupervised machine learning

Long COVID is a condition usually defined by persisting symptoms following infection by the SARS-CoV-2 virus beyond the acute phase of infection. The condition has a significant impact on healthcare systems, the economy, and the individuals living with it. Due to the diverse and extensive symptomatology of long COVID, symptom co-occurrence tracking can be used to capture patient experiences and improve understanding, diagnosis, and management. Here, we leverage the UK Office for National Statistics (ONS) COVID-19 Infection Survey (CIS). The CIS was run between April 2020 and March 2023. On February 3, 2021, the ONS launched a new CIS question evaluating self-reported symptom persistence of 23 long COVID symptoms post self-reported COVID-19 infection. We use Jaccard symptom-by symptom distance matrices, derived from the binary survey responses of the presence of each symptom. Three methods are then used to visualise symptom co-occurrence: heatmaps, non-metric multidimensional scaling, and agglomerative hierarchical clustering with complete linkage. We split our analysis into two parts, first looking at all long COVID survey responses (n = 207,319) and then looking at responses stratified by time-since-onset of long COVID up to 24 months (n = 30,224 at zero months-since-onset). We find higher symptom co-occurrence in prolonged long COVID. We also find clusters of neurological/systemic symptoms, gastrointestinal symptoms, and respiratory symptoms, with separability in symptom co-occurrence by organ system becoming less pronounced as time-since-onset of long COVID increases. Shortness of breath, weakness/tiredness, and muscle ache present as core symptoms long COVID. We demonstrate that the symptom experience of long COVID evolves from early stages to late stages with higher symptom burden and multi-systemic presentation. We also find three possible phenotypes of early long COVID.
Read the full abstract

Statistics Applications

Multi-Fidelity Gaussian Processes for Translational Modelling of Clinical Outcomes

Bridging the gap between animal and human experiments remains a major challenge in translational medicine, particularly in early drug development. Progress is constrained by financial cost, the difficulty of integrating heterogeneous in vitro and in vivo data, and the desire to reduce the use of animal testing balanced against minimising the risk to human participants. We present a statistical machine learning framework using multi-fidelity Gaussian processes, in which animal studies are considered as lower fidelity but informative approximations to human experiments. This allows cross-species similarities and nonlinear exposure-response relationships to be learned simultaneously, enabling principled extrapolation between species while quantifying uncertainty. By leveraging information from multiple experimental fidelities, our method improves estimation of clinically relevant quantities of interest and supports the replacement, reduction, and refinement of in vivo testing. We first illustrate this approach in a simulated scenario, before validating it on real clinical data. We simulate data for drug-induced QT-interval prolongation, a key cardiac safety assessment required for regulatory approval. This framework provides a probabilistic surrogate capable of integrating in vitro pharmacology, animal experiments and human data within a unified statistical model. Crucially, it achieves this at no additional experimental cost while also enabling transfer learning across compounds. As a result, predictions and uncertainty quantification for new drugs can be generated from in vitro findings alone, providing additional efficiency gains and accelerating decision making. For validation, we use a clinical dataset measuring change in heart rate under autonomic blockade, which represents some of the challenges commonly found in multi-species datasets.
Read the full abstract

All authors Isaac S. Hayden, Alicia D'Souza, Steven Niederer, Sarah Filippi

Statistics Applications · Cross-listed from stat.ME

Time-dependent two-way partial AUC and partial Youden Index estimator for right censored data

In medical research, it is often of interest to evaluate the predictive performance of a biomarker. Statistical approaches based on the Receiver Operating Characteristic (ROC) curve and its summary measures, such as the area under the curve (AUC) and the Youden index, are widely used to evaluate the prognostic performance of these biomarkers. In time-to-event studies, ROC analysis poses additional challenges due to change in disease status over time and the presence of censored individuals. To address these issues, time-dependent ROC curves were introduced. In this paper, we propose a non-parametric estimator of the time-dependent two-way partial AUC for right-censored data. We also discuss the partial Youden index and the associated optimal biomarker cutoff estimator for the right-censored data. We conduct an extensive simulation study to investigate the finite sample performance of the proposed estimators. The simulation study indicates that the proposed non-parametric estimators efficiently account for right censoring. Finally, we illustrate the proposed methods using two real data sets, one from the Primary Biliary Cirrhosis study and the other from the Molecular Taxonomy of Breast Cancer International Consortium trial.
Read the full abstract

Statistics Applications

Pricing rides as option contracts: guarantees and memberships under travel-time uncertainty

Modern ride-share platforms must commit to a price before a trip is taken, yet the realized fare depends on travel time that is uncertain at the moment of sale, which can occur days in advance. The upfront price thus decomposes into the expected fare and the premium on an insurance claim whose payoff is the shortfall between the realized and promised prices. This work computes the fair premium for such a claim, following from contingent-claim pricing. Recent advances in travel-time distribution modeling cast the asymptotic distribution of travel time as a Brownian motion on a metric graph carrying a cyclostationary travel-time process. We thus derive closed-form premiums under both a population and a route-conditional travel-time law. A single pricing equation then covers products of increasing informational difficulty, such as on-demand and scheduled rides and capped-membership and commuter products. To price on sparse data, we develop route and travel-time samplers and validate them out of sample on 2014 Quebec City GPS trips, where they reproduce the empirical trip-time distribution. We show that guarantee premiums are inexpensive, typically under 5% of the fare. Conditioning on the realized route roughly halves the premium while reducing tail risk. Analyzing membership abuse, we show that adverse selection acts precisely along the dimension the price fails to condition on, yielding a design principle: subsidies additive in the fare, such as a fixed per-ride discount, leave marginal incentives unchanged and are abuse-proof by construction, with deterministic closed-form cost.
Read the full abstract

Statistics Applications

The Periodically Correlated Components of Measles in New York City using the Variable Band-pass Periodic Block Bootstrap

Measles, a highly contagious, deadly virus, is at risk of losing its eradication status in the Unites States. Understanding the pattern, including seasonality, of measles could provide great benefit for forecasting, prevention, and public health preparedness as the virus re-emerges. The novel Variable Band-pass Periodic Block Bootstrap (VBPBB), which suppresses noise and interfering signals, is more efficient and statistically powerful to find periodic components compared to other existing methods. Using this method, we have found several significant periodic components in historical New York City measles cases which other methods were incapable of identifying. These findings give us a reusable method for understanding measles and other diseases and greater insight into what may occur should measles vaccine rates continue to fall.
Read the full abstract

Thursday 3 September

Statistics Applications

Quantifying Impacts of Wildfire Smoke on Crop Yields in California

Climate change and increasingly severe wildfires pose significant threats to both people and ecosystems, yet little research has focused on the impacts of wildfires on crops. In addition to direct effects such as health risks to agricultural workers and crop damage, there is evidence that smoke may affect crop productivity and quality. This study aims to analyze the effects of wildfire smoke on crop production in Northern California. We quantified the relationship of particulate matter on grape yields using statistical regression models with a wide range of different climate and air quality predictor variables. Our analysis identifies an association between air quality deterioration due to wildfire smoke emissions and reduction in wine grape yields. For two of the three study sites, particulate matter emerged as a significant predictor variable in combination with temperature. Final models for Sonoma and Napa had R2 values of 0.79 and 0.66 respectively, suggesting a shift from climate only-controlled yields to wildfire and climate-controlled yields. This work sets the stage for the next generation of agronomic models that incorporate the growing threat of wildfire on agriculture production systems.
Read the full abstract

Statistics Applications

One-Shot Screening of Simulated Systems for Acceptability

We introduce a general-purpose framework for designing screening procedures for problems featuring a finite set of simulated systems, a.k.a. ranking-and-selection problems. The framework offers a novel perspective on screening in which decisions to retain or eliminate systems are based on confidence regions for the unknown problem instance rather than comparisons of estimated performances. Specifically, a system is retained if it has acceptable performance under some plausible configuration of response vectors contained in the confidence region. This perspective facilitates the design of procedures that guarantee to return either all acceptable systems, or each acceptable system, with high probability and accommodates many well-studied definitions of acceptability, including feasibility with respect to stochastic constraints and optimality with respect to one or more objectives. We further study a subclass of the framework that yields simple and computationally efficient screening procedures that often have lower-order time complexity than existing methods and naturally supports parallelization without loss of screening power. We demonstrate the effectiveness and efficiency of the procedures through numerical experiments.
Read the full abstract

Statistics Applications

Natural Disasters and the Nonprofit Sector

When natural disasters strike, individuals, communities, and even entire countries can suffer. Researchers have studied the impacts of disasters on various factors of interest, from mental health, to poverty, to economic activity. However, the impact of disasters on the nonprofit sector is understudied despite the nonprofit sector's perhaps surprising role in local or national economies as well as its role in disaster response and recovery. Thus, we study the effect of natural disaster damage on different county-level nonprofit outcomes using a panel dataset spanning 1991 to 2021 and causal inference methods tailored to panel data. Contrary to prior work, which found small but positive associations between disaster damage and nonprofit revenue or assets, we find no evidence of a causal effect.
Read the full abstract

Statistics Applications · Cross-listed from cs.LG

A Location-Invariant Estimator of Extremal Quantile Treatment Effects for Heavy-Tailed Distributions

Quantile treatment effects (QTEs) measure the effect of a treatment on the distribution of an outcome, and their estimation at extreme quantile levels is of central interest in applications where the target quantiles lie far beyond the range of the data. For heavy-tailed potential outcomes, existing extremal QTE estimators rely on extrapolation combined with a causal extreme value index (EVI) estimator, but the resulting estimator is not invariant under a common location shift of the potential outcome distributions, even though the population QTE is. We address this issue in two steps. First, we adapt the location-invariant Fraga estimator of the EVI to the causal setting using inverse propensity score weighting. Second, we replace the original extrapolation formula with a difference-based scheme, under which the location parameter cancels when quantile differences are taken. The resulting QTE estimator is therefore location invariant. We establish the consistency and asymptotic normality of the proposed extremal QTE estimators, and provide a consistent variance estimator, leading to asymptotically valid inference. A simulation study confirms the location invariance, the stability with respect to the threshold, and the coverage of the proposed methods.
Read the full abstract

All authors Xin Yu, Shuwei Huang, Jicheng Liu, Jielin Tang, Bolin Wang, Yunxiao Zhang, Tian Zhao

Statistics Applications · Cross-listed from stat.ME

Random mixtures in Bayes Hilbert spaces

We present a framework for the analysis and unmixing of random density mixtures in the Bayes Hilbert space. General identifiability results for mixtures in Hilbert spaces are established and applied to the Bayes Hilbert space setting. Building on these results, we propose a penalised maximum likelihood approach for the unmixing of Bayes Hilbert mixtures aimed at recovering the statistically space-efficient representation, together with a computationally efficient coordinate-wise maximisation algorithm for its implementation. The methodology is illustrated through a hyperspectral data application, where observations can be naturally embedded in the Bayes Hilbert space and analyzed in terms of distributional shape rather than amplitude. A complementary simulation study demonstrates the interpretability and practical performance of the proposed approach.
Read the full abstract

Statistics Applications · Cross-listed from cs.CV

SafeRestore: Detector-Relative Risk Certificates for Selective Industrial Image Restoration

Industrial inspection pipelines often restore a measured image before a detector acts on it, yet restoration can suppress detector-supported defect structure or create clean-region activations. We formulate restoration as a selective action problem over the measured display, five restored candidates, and review. SafeRestore ranks candidates with action-specific fitted scores, chooses a gate on threshold-tuning data, and evaluates the fixed gate on a disjoint certification sample with two one-sided exact binomial bounds: one for the positive-conditional evidence-loss incident rate and one for the all-accepted excess-activation incident rate. The guarantee is marginal for one policy fixed before its certification outcomes are observed, under an image-level i.i.d. working model. In a retrospective split-sample study of 4,591 public Carinthia-S images, the protocol yields auditable risk-coverage behavior. The primary all-action policy passes in one of five training repetitions (12.0% +/- 26.9% pass-gated test coverage when failures count as zero), whereas fixed bicubic and reduced-complexity variants pass more often. On reserved morphologies, evidence-loss incidence rises to 81.1-90.3%, and KolektorSDD lacks both detector competence and enough positive certification images for the stated target. The contribution is therefore an auditable, detector-relative framework for deciding when a transformed image may be returned automatically and when review remains necessary – not a claim that adaptive routing outperforms simpler policies on the present evidence.
Read the full abstract

2609.03475 · 3 Sep · 27 pages, 10 figures, 6 main-text tables, and 17 supplementary tables. Shares the Carinthia-S image identities with arXiv:2607.17401 by the same authors; the overlap and the distinct estimand are stated in Section 2.4 and Table 1

Wednesday 2 September

Statistics Applications · Cross-listed from cs.CL

Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent

Counterfactual fairness audits of clinical language-model agents report a flip rate: how often an action changes when only the patient's demographic descriptor changes. Part of that rate is not demographic. A stochastic agent also changes its own action when nothing changes, and a flip rate cannot be interpreted without knowing how often. We measured it. Re-running one condition ten times over sixteen synthetic vignettes at default sampling changed a clinical agent's action in 8.7 percent of replicate pairs, from 2.2 percent for intensive-care escalation to 17.9 percent for controlled-substance caution, an output given no operational criteria. Across six models from five vendors, pooled floors ranged from 2.5 to 23.7 percent; in this panel neither disclosed size, vendor, nor hosting ordered them. The floor depends on the decoding configuration: majority voting over five draws removed 39 percent of it (95 percent confidence interval (CI) 18 to 64); at temperature 0 three of four locally served models showed no disagreement, but a hosted model still did. We also show that, for a binary action, the flip rate expected under no demographic effect equals the floor and a real effect adds only its square, so a flip rate inside the floor is not evidence of fairness, and direction must be tested with a signed paired test. We give a four-step reporting procedure and release FairMedAgent, the harness, with its protocol, vignettes and analysis scripts, so any team can measure the floor for its own agent.
Read the full abstract

All authors Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar, Rahul Joshi

Statistics Applications

Uncertainty quantification of fatigue initiation life for powder bed fusion metal additive manufacturing

Predicting fatigue life with quantified uncertainties is essential for the qualification of critical components produced by laser-based powder bed fusion additive manufacturing. We present a framework that propagates microstructure and defect uncertainties directly to a fatigue initiation life distribution for a specific part. In particular, microstructure and defect characterizations are obtained from electron backscatter diffraction and micro-computed tomography scan data, which in turn inform three physics-based simulations yielding the fatigue-affecting quantities: the elastic energy release rate, the surface energy along the crack path, and the fatigue indicator parameter. Accounting for the uncertainties in these quantities and the high correlations among them due to the shared underlying microstructure, we derive a closed-form probability density function for the fatigue initiation life. This provides an analytical distribution instead of conservative deterministic predictions and enables more informed decision making for the qualification and deployment of additively manufactured components. Applying the framework to 316L stainless steel parts produced by an EOS M290 laser powder bed fusion machine, we find that both grain sizes and void distributions influence the fatigue initiation life distribution. Specifically, for a fixed total void volume fraction, larger grain sizes cause a marginal reduction in fatigue initiation life, and a population of many small voids is more favorable for fatigue life than fewer, larger voids of equivalent total volume.
Read the full abstract

Statistics Applications · Cross-listed from eess.SY

Dynamic Operational Reserve Margin Assessment from Risk-Constrained Unit Commitment States

We propose Dynamic Reserve Margin (DRM) as a time-varying operational adequacy metric derived from risk-constrained unit commitment (RCUC) states. DRM quantifies reserve adequacy using the additional generation capacity that committed generators can provide within a 5-minute response window relative to uncertainty and contingency reserve requirements. We introduce a complementary Reserve Risk Envelope (RRE) metric to quantify operational reserve headroom in directly interpretable MW terms. A low-margin duration metric is further developed to quantify the persistence of reserve stress over an operating horizon. We use an IEEE 14-bus example to illustrate these concepts, followed by large-scale RCUC case studies under multiple operating scenarios. Results demonstrate that reserve requirements and ramp-accessible reserve capability can vary substantially across operating conditions, and that commitment decisions adapt to maintain reserve adequacy under changing system conditions. The proposed DRM and RRE metrics provide an interpretable operational characterization of reserve adequacy, reveal reserve accessibility and stress persistence that are not captured by conventional reserve margin metrics.
Read the full abstract

All authors Pratishtha Shukla, Shaked Regev, Evan J. R. Brody, Charles Foltz, Teja Kuruganti

Statistics Applications

Learning CNN Filters via Generalized Stein's Method

Convolutional Neural Networks (CNNs) have undoubtedly revolutionized image data analysis and the field of computer vision. As the cornerstone of CNNs, the convolution operation enables the networks to extract abstract features and uncover hidden relationships in the image data. This paper considers the problem of estimating convolution filters from a statistical perspective using a classical tool — Stein's formula. We first formulate CNNs into a general index model with matrix-valued input, where convolution filters can be viewed as index vectors. Furthermore, we propose a novel singular value decomposition (SVD) based approach to accurately learn the convolution filters based on a generalized version of the first-order Stein's formula. Theoretical analysis suggests that our estimation achieves an optimal convergence rate, comparable to that of generalized linear models where the link function is known. Extensive simulation studies and real data analyses demonstrate that our approach outperforms popular deep learning algorithms, such as Adam. Notably, our method extends beyond filter estimation and can be applied to nonlinear dimension reduction, providing a viable pathway for representation learning.
Read the full abstract

All authors Guang Yang, Wei Shi, Yuan Cao, Long Feng

Statistics Applications · Cross-listed from stat.ME

DOMIC: Provably Calibrated Detection of Dependence-Structure Change Points via Density-Operator Mutual Information

Dependence can change between variable blocks without changing correlation; serial dependence complicates calibration. We propose DOMIC, a rank-based detector using density-operator mutual information (DOMI). Unit-norm random features make sample-level partial traces exact, so prefix sums compute segment statistics at per-split cost independent of segment length; a Gram form uses the exact kernel. Permutation tests are finite-sample exact under pair or block exchangeability for windows and schemes fixed in advance; validity after data-driven scheme selection is unproved. For i.i.d. pairs with feature dimension, frequency draw and bandwidth fixed, we prove a weighted chi-square limit at independence for the rank-based random-feature statistic. An additive entropy cost with Holevo gains drives segmentation. DOMI attains higher localised power than rank, copula, kernel and distance baselines in most non-Gaussian and correlation-free settings; the Gram form leads or ties in all 16 non-Gaussian settings where any statistic exceeds 0.10. On a three-break benchmark, Holevo partitioning returns exactly three breaks in 47.5% of replicates, against at most 19% for calibrated baselines. Block-calibrated primary scans reject in all three pre-specified pairs of a US financial panel, with strongest stage-two evidence for stocks and Treasury yields. One of 27 weather candidates passes the multiplicity-corrected re-test; candidate selection is unadjusted.
Read the full abstract

Statistics Applications

Did Mary Shelley Write Frankenstein? A Stylometric Analysis

The novel Frankenstein was published anonymously in 1818, and was first credited to Mary Shelley in a French translation of 1821. Since its publication, several claims - both contemporaneous and recent - have been made suggesting that Frankenstein was actually written by Mary's husband, Percy Bysshe Shelley. We review the background of this controversy and then apply modern techniques from computational stylometry to determine who the true author is. Based on our analysis, we find extremely substantial evidence that Mary Shelley is indeed the true author of Frankenstein, and that it is very improbable that Percy Bysshe Shelley played a heavy role in composing the text. While our finding confirms mainstream scholarly opinion regarding Frankenstein, our analysis is the first application of stylometric techniques to this question and provides strong objective grounds for favouring Shelley by freeing the question from some of the politics which have traditionally accompanied it.
Read the full abstract

Statistics Applications

Big data, differential privacy, and national statistical organisations

Differential privacy (DP) has emerged in the computer science literature as a measure of the impact on an individual's privacy resulting from the publication of a statistical output such as a frequency table. This paper provides an introduction to DP for official statisticians and discuss its relevance, benefits, and challenges from a National Statistical Organisation (NSO) perspective. We motivate our study by examining how privacy is evolving in the era of big data and how this might prompt a shift from traditional statistical disclosure techniques used in official statistics–which are generally applied on a cell-by-cell or table-by-table basis–to formal privacy methods, like DP, which are applied from a perspective encompassing the totality of the outputs generated from a given dataset. We identify an important interplay between DP's holistic privacy risk measure and the difficulty for NSOs in implementing DP, showing that DP's major advantage is also DP's major challenge. This paper provides new work addressing two key DP research areas for NSOs: DP's application to survey data and its incorporation within the Five Safes framework.
Read the full abstract

Statistics Applications

Sequential Certification of Threshold Decisions in Rare-Event Risk Prediction

Rare clinical outcomes pose a difficulty deeper than ordinary class imbalance: a penalized logistic model can return finite, stable-looking coefficients before the data support a reliable threshold decision. We formulate the accrual question as decision-targeted sequential certification. On a prespecified finite monitoring schedule and target set, asymptotic prediction bands are adjusted for simultaneous coverage, and certification is assessed only among profiles that might be referred, so that a large low-risk majority cannot trigger an uninformative stop. Under the working rare-event logistic model and stated regularity conditions, each certified decision is asymptotically model-conditionally correct with probability at least \(1-\alpha\) over the schedule, and the effective information scales with the number of genuine events. Simulations and a US linked birth/infant-death application illustrate the gap between a model that is merely estimable and one whose decisions are certifiable: the whole-population rule stopped while more than half of referral-relevant profiles remained ambiguous, whereas the decision-targeted rule did not certify by the 300{,}000-birth horizon despite stable temporal validation. Regularization makes a rare-event model estimable but does not substitute for genuine rare-event information.
Read the full abstract

Tuesday 1 September

Statistics Applications · Cross-listed from stat.ME

Bigraphical Matérn-Whittle (BMW) Processes for Fast Inference of Big Multivariate Spatial Data on General Domains

Large spatial data sets now record many correlated variables at many thousands of locations, often on domains where Euclidean distance misrepresents proximity. The central difficulty is modelling the cross-variable dependence jointly while retaining variable-level interpretation. We introduce the bigraphical Mat'ern-Whittle process, a multivariate Gaussian process that resolves this with two graphs. A spatial graph generates the Mat'ern structure of each variable through a fractional power of a graph Laplacian, so the process is valid on any topology, with per-variable range, smoothness and amplitude. A directed acyclic variable graph encodes the scientific structure: we prove that each absent edge yields an exact conditional independence between the corresponding fields. We further prove that the operator determinant does not involve the cross-dependence coefficients, which keeps matrix-free likelihood evaluation and Bayesian learning of the variable graph tractable at scale. Estimation requires only sparse matrix-vector products and scales to tens of millions of space-variable pairs. In simulations the method recovered parameters and graphs accurately, remained robust under misspecification, and halved held-out prediction error on a non-convex domain. In a spatial transcriptomics section with 19,809 cells and 1,122 genes, fitted in 75 minutes on a laptop, borrowing across the learned gene graph reduced held-out prediction error by 50 to 91 percent. Theoretical challenges, such as the achievable efficiency of estimating the variance of the nugget, are also explored.
Read the full abstract

Statistics Applications

Neural Networks Learning the Radon–Nikodym Derivative: Empirical Option Pricing in Incomplete Markets

In incomplete markets, no-arbitrage (NFLVR) guarantees the existence, not the uniqueness, of an equivalent local martingale measure (ELMM): unhedgeable risks (jumps, stochastic volatility) admit a whole family of equivalent measures, and asset dynamics alone cannot pin down the one the market selects. We characterize the identifiability of $Q$ from option data and propose a measure that is identifiable from data yet prices any claim consistently. The key boundary is an “identification wall”: European options identify only the terminal marginal, while out-of-sample tails and path/joint structure require instruments matched to the priced risk (variance or higher-moment swaps, path-dependent claims). Within this view, minimum-relative-entropy weighted Monte Carlo (WMC) is the optimal baseline for the marginal; we generalize it to a full path-space measure change parameterized by a neural network on physical scenarios—XiNet—which learns $\xi=\mathrm{d}\mathbb{Q}/\mathrm{d}\mathbb{P}$ directly from 8 model-free path features, with option prices as a soft constraint. On European (marginal) pricing XiNet matches but does not surpass WMC, and both are bound by the identification wall out-of-sample. On path-dependent claims the picture reverses: calibrated on the same European surface, per-marginal methods fail structurally (an ATM forward-start is mispriced by $\sim+100\%$), whereas XiNet's single self-consistent measure keeps the bias to $+0.3\%$, beating maximum-entropy WMC ($-24\%$), because $\xi=f_\theta(\text{path features})$ captures joint structure Europeans cannot constrain. Identification is thus risk-specific, and XiNet is a single measure that absorbs available instruments and prices all claims consistently.
Read the full abstract

Statistics Applications

Random Forest-Informed Cellular Automaton for Large-Scale Wildfire Spread Modelling

Accurate large-scale wildfire spread modelling requires models that capture both the environmental conditions associated with fire occurrence and the local dynamics of fire propagation. We propose a three-stage framework that combines a Random Forest (RF) model with a cellular automaton (CA). First, an RF model trained on the 2021 Canadian fire season estimates daily pixel-level fire-occurrence probabilities. Second, quantile gradient boosting models provide optional spread-rate priors for sensitivity analysis. Third, an RF-informed CA combines the RF probability layer with neighbourhood-driven spread on a 5 km grid. The RF model achieved AUC values of 0.725–0.795 on the 2022–2024 datasets, while the RF-informed CA achieved substantially higher spatial overlap than the evaluated CA-only baselines in the 2023 simulation. A higher-resolution simulation provides an additional qualitative assessment of local spatial errors. These results suggest that combining RF-derived probabilities with local CA spread can improve large-scale wildfire simulations under the tested conditions.
Read the full abstract

Statistics Applications · Cross-listed from physics.data-an

Multi-fidelity Monte Carlo estimation of floor response spectra under combined seismic and structural parameter uncertainties

Floor response spectra (FRS) are essential tools for the design of non-structural elements (such as equipment or components). Given the various physical phenomena influencing FRS, high-fidelity (HF) mechanical models of the primary structure may be required to estimate them. Since numerical simulations based on such models are generally computationally expensive, this paper proposes using a multi-fidelity Monte Carlo (MFMC) approach for the efficient estimation of FRS. The method relies on using observations from a fast low-fidelity (LF) model as control variables. If the absolute value of the correlation between LF and HF samples is close to 1, this approach reduces both variance and estimation error compared to a standard Monte Carlo estimate based solely on HF data samples. Through a case study involving the reactor building of the Kashiwazaki-Kariwa nuclear power plant, we demonstrate the suitability of this method for FRS estimation. It effectively reduces variance and estimation error, even when using a LF model as simple as a single-degree-of-freedom system. We also show that the method accounts for modeling uncertainties while maintaining comparable performance. Its ease of use makes it a valuable tool for practitioners.
Read the full abstract

All authors Nils Baillie, Baptiste Kerleguer, Cyril Feau, Josselin Garnier, Fan Wang

Statistics Applications · Cross-listed from physics.ao-ph

A Sensor-Adaptive Incremental Learning Framework for Artifact Detection in Satellite Precipitation Data

Historically, retrieving rainfall data from satellite imagery has been the domain of space agencies. However, in recent years, the development of cheaper, more compact satellites (SmallSats) capable of detecting rainfall proxies has led to a significant increase in private-sector initiatives for satellite launch and surface precipitation products. This rapid growth has yet to be matched by data validation efforts. Consequently, the need for a robust tool to detect anomalies in near-real-time data before it is disseminated to the public has become critical. In this paper, we present the development of an anomaly-detection system to identify artifacts in global satellite-based rainfall products. The developed framework leverages pre-trained computer vision models and incorporates scarce human-labeled data to detect specific anomalies. Our proposed anomaly detection strategy is tested on data from the Special Sensor Microwave Imager (SSMI) and the Special Sensor Microwave Imager/Sounder (SSMIS). Results demonstrate the efficacy of our approach at separating regular orbits from artifact-containing orbits for each satellite, with performance comparable to state-of-the-art in-place methods. Additionally, the framework offers explainability and the capacity for iterative refinement following false-positive or false-negative classifications.
Read the full abstract

Monday 31 August

Statistics Applications · Cross-listed from stat.ME

Generalized Bayesian Clustering with Regression for Unaligned Longitudinal Binary Data

We propose a generalized Bayesian clustering with regression model for unaligned longitudinal binary outcomes, motivated by seizure diary data from the Human Epilepsy Project. Seizure diaries are sparse, irregularly observed, and vary enormously across patients. A single fully-specified generative model tends to be either misspecified or computationally inefficient. We address the challenge by two strategies. We set up a regression by way of clustering as model-based clustering using a mixture model. For the latter, we take a generalized Bayesian perspective which replaces the full likelihood with a loss-based update using a generalized likelihood. We combine a trajectory similarity loss and a regression loss, so that clustering is informed by both trajectory similarity and the prediction of outcomes The trajectory similarity loss is constructed by representing each trajectory as an (empirical) distribution of subsequences, called reads, and then is defined based on the sliced Wasserstein distance between these empirical distributions. This loss allows alignment-free comparison of sequences that are irregularly observed or temporally misaligned, and it scales quasi-linearly in trajectory length. The regression loss is the negative log-likelihood of a probit regression. A prior on the cluster-specific parameters is defined by way of a Dirichlet process prior on the mixing measure.
Read the full abstract

Statistics Applications · Cross-listed from physics.ao-ph

How well is surface ocean carbon represented in observations and ocean models?

We introduce a general framework for quantifying the information content and representation quality of complex geophysical datasets based on the intrinsic dimension and differentiable information imbalance of data manifolds. We use it to derive and compare optimal representations of surface ocean carbon in the SOCAT database of observations and in global ocean biogeochemistry models (GOBMs) and to assess the robustness of the information we can extract from existing data. We find that within the most widely used feature set, the complexity of the data space of SOCAT observations is not fully captured by GOBMs, but the ranking and relative importance of variables learned through GOBMs are substantially correct. We observe that the learned representation of ocean carbon is less accurate in some regions, including the Southern Ocean, but doesn't appear to have evolved significantly over the last two decades. Finally, we show how the optimal representations can be used to improve the skill of distance-based machine learning models and demonstrate it for ocean carbon, and we propose two new metrics to compare models and observations that can be used to build more accurate weighted ensembles of estimates.
Read the full abstract

All authors Viviana Acquaviva, Romina Wild, Alessandro Laio, Amanda R. Fay, Thea H. Heimdal, Galen A. McKinley