Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics Statistics Applications

stat.AP Watch Statistics Applications (sign in to use)

Friday 11 September

Statistics Applications · Cross-listed from stat.ME

Bayesian Consensus Calibration of Continuously Evolving IRT Item Banks

AI-based item generation and NLP-based prediction of item parameters are producing item banks that are substantially larger, sparser, and more frequently updated than conventional banks. Hierarchical Bayesian item response theory (IRT) is a natural calibration framework for such banks, but the common practice of refitting the entire accumulated response history at each update is costly and can exceed available memory. We describe consensus calibration, a divide-and-conquer procedure that calibrates each time period independently and reconstructs the pooled posterior in two layers. First, the posterior draws of each period are mapped to a common metric by a robust characteristic-curve linking (Haebara) that is solved separately for each draw, which propagates the uncertainty of the linking transformation into the linked posteriors. Second, the linked item posteriors are combined as a product of Gaussian densities from which the population prior contributed by each period is removed and a single prior—obtained by consensus across the per-period population posteriors—is reinstated. The correction targets the posterior dispersion, not only its location. As evidence for consensus calibration, we compare it to a pooled single-run analysis on a large operational assessment in terms of item-parameter recovery, an uncertainty-by-exposure diagnostic, and the ability distributions.
Read the full abstract

All authors Paul A. Jewsbury, Steven W. Nydick, Manqian Liao, Siyuan Chen

Statistics Applications · Cross-listed from stat.ME

Fingerprint Analysis for Climate Change Detection and Attribution under a Latent Factor Model

Detection and attribution (D&A) analyses provide a statistical framework for quantifying the contribution of external forcings to observed climate change. Optimal fingerprinting, the primary approach for D&A, is formulated as an errors-in-variables regression with a high-dimensional covariance structure. Reliable inference is challenging because covariance matrices must be estimated from a limited number of control simulations, and climate models may exhibit variability patterns that differ from those of the observed climate system. We develop a spiked optimal fingerprinting framework that exploits the spiked structure of dominant climate variability modes while providing stable covariance estimation in high-dimensional settings. The proposed method develops bias-corrected spiked covariance estimation for constructing adaptive weight matrices and incorporates variability inflation adjustments to account for model–observation differences in internal variability. We further develop a valid uncertainty quantification procedure for the scaling factor estimators and a residual consistency test for assessing model adequacy. Numerical studies demonstrate improved estimation accuracy and uncertainty quantification compared with existing practical approaches. Applied to annual mean near-surface temperature data, the proposed framework produces shorter and better-calibrated confidence intervals and leads to different detection and attribution conclusions in several regions, providing new insights into the interpretation of climate attribution results.
Read the full abstract

Statistics Applications · Cross-listed from stat.ML

Beyond Point Forecasts: A Survey on Probabilistic Forecasting for Time Series and Spatiotemporal Data

Probabilistic forecasting is central to decision-making under uncertainty, yet its methodological landscape has become increasingly fragmented across temporal and spatiotemporal forecasting, statistical modeling, machine learning, and deep generative modeling. This survey develops a unified perspective by organizing probabilistic forecasting methods according to where and how uncertainty is introduced into the forecasting pipeline. Our taxonomy connects model-agnostic approaches including ensembles and distribution-free calibration, with model-intrinsic approaches spanning Bayesian modeling, parametric predictive distributions, distributional regression, and modern generative models, and further examines the emerging role of time series foundation models. Beyond methodological synthesis, we identify the assumptions, computational demands, and forms of uncertainty represented by different paradigms, and translate these distinctions into data-driven and domain-specific guidance for method selection. We complement the survey with a cross-paradigm empirical study on univariate, multivariate, and spatiotemporal forecasting tasks. The results reveal that no single uncertainty-quantification paradigm dominates across settings. Calibration, sharpness, predictive accuracy, and computational efficiency can lead to substantially different model preferences, while expressive generative models and zero-shot foundation models exhibit markedly different accuracy-efficiency trade-offs. Lastly, we identify unresolved challenges surrounding uncertainty in evolving dependency structures, physics-informed predictive distributions, forecasting extreme events, handling count-valued, directional, and continuous-time series, and the development of unified software resources. Our survey provides both a conceptual framework and a practical roadmap for probabilistic forecasting research.
Read the full abstract

All authors Donia Besher, Rajdeep Pathak, Madhurima Panja, Tanujit Chakraborty

Statistics Applications

Evaluating differential air quality impacts of prescribed fire and wildfire in the United States

Increasing wildfire activities in the United States have generated far-reaching effects on air pollution and human health. While prescribed fires are used as a preventive strategy to reduce wildfire risks, the health impacts of their smoke remain uncertain. Small-scale case studies indicate that prescribed fires generally produce lower pollutant emissions than uncontrolled wildfires, potentially benefiting air quality. However, the extent to which this translates into reduced population-level smoke exposure remains uncertain due to a lack of robust, large-scale quantitative evidence. Moreover, the factors driving differences in air quality impacts between prescribed fires and wildfires were underexplored. Using fire data in California, Florida, and Georgia (2006-2020) and a causal forest approach, we quantified and compared the impacts of prescribed fires and wildfires on air quality, assessed their differences in population-level smoke exposure, and identified key factors influencing their smoke exposure levels. On a per square-kilometer basis, prescribed fires resulted in over six times less population-level smoke exposure compared to wildfires. Additionally, prescribed fires were associated with lower smoke exposure in regions with higher relative humidity and precipitation. This work could enable the broad risk-benefit assessment to guide the judicious use of prescribed fires and optimize public health outcomes while mitigating wildfire threats.
Read the full abstract

All authors Ting-Hsuan Chang, Minghao Qiu, Yaguang Wei, Xiao Wu

Statistics Applications

MIMO and Multi-Group Comparison Problem in Process Control: A Multivariate Statistical Framework for Systems Represented with Frequency Response Functions

Characterizing complex Multi-Input Multi-Output (MIMO) systems presents two issues: feedback controllers mask fault variance, rendering single-variable monitoring ineffective, and repeated testing across multiple conditions inflates false positive rates due to the multiple comparisons problem. This study proposes a multivariate statistical framework to resolve these limitations. We extend a statistical library that identifies system dynamics via Frequency Response Functions (FRFs) by transforming them into time-domain Pseudo-Impulse Responses (PIRs). This functional representation captures the complete dynamic signature of the system, offering a richer diagnostic profile than traditional static scalar metrics. The framework, originally developed for SISO systems, is extended to the MIMO case by introducing supervectors that aggregate multiple PIRs, evaluated using Permutational Multivariate Analysis of Variance (PERMANOVA). This non-parametric approach handles the high dimensionality and complex correlation structures inherent in MIMO functional data. We demonstrate that the PIR Supervector is more powerful than single-variable analysis, and the MIMO PERMANOVA approach outperforms traditional SISO anomaly detection. By distinguishing between normal operation, external thermal disturbance effectively compensated within the investigated operating range, and severe parametric faults that standard univariate methods miss, the framework provides a single, rigorous metric for specific fault diagnosis ($p < 0.001$).
Read the full abstract

Statistics Applications · Cross-listed from cs.HC

ForkSCOPE: Charting the Agentic Garden of Forking Paths

Even with a fixed dataset and research question, data analysis involves many defensible decisions. Understanding how these choices influence the results is scientifically important but remains challenging. Crowdsourcing and agentic AI can generate hundreds of end-to-end analyses, but scaling generation alone can create a processing bottleneck and an analytic “black hole.” A common workaround is to impose a shared fixed decision taxonomy, which can limit insight and understate uncertainty. We present ForkSCOPE, a human-AI collaboration framework that induces structure bottom-up from the code corpus of end-to-end analyses, without a taxonomy fixed before or after generation, so the organization and evaluation of the garden can scale with the corpus. ForkSCOPE surfaces the charted garden of forking paths through a human-AI collaboration pipeline and an evidence-linked interactive viewer for steering and verification: it spotlights organically identified forks and structures and produces a derived taxonomy and decision map compatible with existing multiverse tools.
Read the full abstract

Statistics Applications

Optimization of ReaxFF parameters for the $\mathrm{Mo-S}$ system using random optimization and coordinate search

ReaxFF is a molecular dynamics method that can be considered a good approximation to quantum methods for investigating reactive molecular systems consisting of ten thousand to one hundred thousand atoms. While ReaxFF is usually a much faster alternative to quantum methods, the force field consists of nearly 100 parameters per element, which makes the force field development a high dimensional optimization problem. In addition to the high-dimensionality, non-convexity and non-continuity make it a hard problem to optimize. We use random optimization along with coordinate search strategies to optimize efficiently and sample new parameter points that yield good molecular properties close to predefined `reference values' obtained from quantum mechanical methods for the $\mathrm{Mo-S}$ system. We also provide empirical error guaranties starting from any random sample of inputs. We discover new points for the $\mathrm{Mo-S}$ system at adjusted error levels of $13{,}000$ as compared to Sengul et al. (2022) at $70{,}000$ levels under the same loss function, registering over $80\%$ improvement. We also extend our algorithm to an out-of-sample system, $\mathrm{W-S}$, with no training data to record over $70\%$ improvement over Sengul et al. (2021).
Read the full abstract

All authors Arun Ravichandran, Michael L. Stein, Mert Y. Sengul, Ying Hung, Tirthankar Dasgupta

Thursday 10 September

Statistics Applications

Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark

Large language models (LLMs) are proposed as synthetic respondents for pilot testing, but their usefulness depends on whether they supply the evidence assessment development requires. We calibrated rating scale models on 14 digital-use skill items from 6,245 adults and used the human item parameters to evaluate responses generated for 1,300 demographically matched personas. LLM responses had high internal consistency ($\alpha \approx .94$) but used the lowest category in 0.2-0.3% of responses versus 13.7-21.5% for humans, and no persona chose it on every item. These gaps changed the response-category judgment on both subscales and the PC targeting judgment; on the human-calibrated scales, 12 of 14 items had infit below 0.70, indicating responses more predictable than the Rasch model expects. Preregistered changes to the prompt, category order, and persona information did not restore the human lower range. High internal consistency is insufficient evidence that LLM responses can replace human pilot data.
Read the full abstract

Statistics Applications

Bayesian Semiparametric Hidden Markov Random Partition Fields for Factor Collapse on Graphs: A Study of Cortical Mapping of Fingertips

Understanding how the brain's cortical regions respond to stimuli like fingertip tapping is a significant challenge in neuroscience, especially with high-resolution functional magnetic resonance imaging data. To address this, we propose a new statistical method for evaluating how a factor's influence locally varies across a complex graph, such as the brain's cortex. Our approach is designed to handle the complexities of large graph sizes and computational demands. Our method uses a novel Bayesian hidden Markov random field to partition the factor's influence into collapsed states with similar effects on the outcome at each node. This unique model promotes sparsity in the partitions and penalizes large variations across adjacent nodes. The result is a highly detailed local influence map that captures subtle changes in the patterns and magnitudes of a factor's influence across the graph. To ensure efficient analysis of large datasets, we developed a specialized Markov chain Monte Carlo algorithm. Our simulation experiments demonstrate significant improvements over existing techniques in both accuracy and scalability. Ultimately, our method provides a powerful new framework for exploring the intricate relationship between stimuli and brain activity, offering a clearer picture of the cortical mapping of fingertips.
Read the full abstract

All authors Blake Moya, Kevin Sitek, Arkaprava Roy, Bharath Chandrasekaran, Abhra Sarkar

Statistics Applications · Cross-listed from econ.EM

Simultaneous Clustered Orthogonalization

Identification in vector autoregressions involves two distinct choices: how extensively to orthogonalize shocks and whether orthogonality is imposed sequentially or simultaneously. Generalized identification imposes no orthogonality; Sims (1980} imposes full orthogonality sequentially; Francis et al. (2026) impose full orthogonality simultaneously; and Buchwalter et al. (2026a) provide clustered partial orthogonalization sequentially. We fill the remaining case by developing simultaneous clustered orthogonalization (SCO). SCO preserves contemporaneous dependence within economically meaningful clusters while imposing orthogonality across clusters jointly, thereby eliminating dependence on cluster ordering. We formulate the associated correlation-maximizing identification problem and show that, in correlation space, it reduces to a quadratic matrix equation. This yields a closed-form solution, and we prove that the associated identification matrix is the unique global maximizer. SCO is order- and scale-invariant and nests generalized identification and full simultaneous orthogonalization as special cases, yielding a flexible family of structural decompositions indexed by the number and composition of clusters.
Read the full abstract

Statistics Applications · Cross-listed from q-bio.QM

Robust POMDP Framework for Lung Cancer Screening Problems

Lung cancer remains a leading cause of cancer mortality because many cases are diagnosed at advanced stages. Low-dose computed tomography (LDCT) screening can reduce mortality through earlier detection. Partially observable Markov decision process (POMDP) models can personalize screening by maintaining a belief over an individual's latent cancer state. However, cancer-state transition probabilities are often generated from clinical simulations and are subject to estimation error and model misspecification. We propose a robust POMDP framework using $\ell_1$-norm ambiguity sets around the nominal transition probability. The model optimizes screening decisions against the worst-case transition probability while keeping other components fixed at nominal values. Building on the piecewise-linear and convex structure of the robust value function, we adapt point-based value iteration to compute robust screening policies. We evaluate the policies using out-of-sample simulations that perturb selected cancer-progression parameters and compare them with the nominal ENGAGE policy for representative female and male heavy-smoker cohorts at age 50. Robust POMDP policies generally outperform nominal ENGAGE in mean out-of-sample quality-adjusted life-years (QALYs), with the best performance at a moderate ambiguity radius within the tested grid. Clinical analysis shows that the robust policy reduces lung cancer deaths (LCDs) in all evaluated settings for the female cohort and in most settings for the male cohort, with additional false positives (FPs). Screening-schedule analysis shows that the robust policy recommends more LDCT screens and detects more early-stage lung cancers. These findings show that incorporating transition-model uncertainty into data-driven screening models can improve out-of-sample reliability and provide more robust decision support when clinical simulation inputs are misspecified.
Read the full abstract

Statistics Applications

Geospatial Foundation Models Capture Health-Relevant Dimensions of Place Beyond Conventional Social Risk Indices

Area-based social risk indices summarize residents' socioeconomic conditions but incompletely capture physical features of place that may affect health. We evaluated whether numerical representations of physical place produced by four geospatial foundation model families from 2022 satellite data explained residual variance in tract-level associations between the Area Deprivation Index, Social Deprivation Index, and Social Vulnerability Index with health outcomes. We used LightGBM to predict variables from the American Community Survey and 40 chronic disease and health-behavior outcomes from CDC PLACES across 82,646 census tracts in the contiguous United States, evaluating performance across 10 held-out states. Among survey variables, models were moderately predictive of some variables including housing type (R-squared up to 0.54) but weak for disability, unemployment, and income disparity. For health outcomes, models explained up to 54% of variance left unexplained by social risk indices, with the largest gains for annual checkups, arthritis, and high blood pressure. Mean total variance explained by geospatial foundation models across the 40 health-related outcomes increased from 0.31 in the smallest tract-size decile to 0.39 in the largest. Geospatial foundation models capture health-relevant features of place not represented by conventional social risk indices and may usefully augment them in epidemiological analyses.
Read the full abstract

All authors Nathaniel Hendrix, Carl Y. Zhang, Chris Heitzig, Andrew Bazemore, David H. Rehkopf

Statistics Applications

Privacy-Preserving Causal Meta-Mediation Analysis with Survival Outcomes

Privacy and data-governance constraints often prevent pooling individual-level data across studies, limiting the use of conventional approaches for causal media- tion analysis in multicenter settings. We propose a federated causal meta-mediation framework for right-censored time-to-event outcomes that enables collaborative es- timation without sharing individual-level data. Our framework targets natural indirect effects in a prespecified population by combining information on mediator and outcome mechanisms across distributed data sources. A site-by-site identifi- cation strategy further allows heterogeneity across data sources to be character- ized, with a variance decomposition separating outcome-related, mediator-related, and interaction components. We develop federated one-step and targeted maxi- mum likelihood estimators that accommodate data-adaptive and machine-learning methods for nuisance-function estimation. The finite-sample performance of the proposed estimators is evaluated through numerical simulations. To illustrate the practical utility of the framework, we apply it on data from the French National Health Data System to evaluate the role of methotrexate coprescription in explain- ing the effect of TNFi versus IL-12/23 inhibitor therapy on treatment persistence among psoriatic patients.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Market-Informed Networks for Modeling and Forecast Evaluation of Financial Extremes

Modeling the joint distribution of extreme values in high-dimensional financial time series is challenging because extremes are sparse and locally extreme observations are not necessarily extreme relative to their full marginal distribution. To address this, we introduce a time-dependent network Hüsler-Reiss model in which market-informed adjacency matrices determine how strongly observations contribute to the estimation. We propose binary and weighted specifications, including the Joint Extremes Adjacency Matrix (JEAM) which combines information about individual extremeness with historical patterns of joint extreme movements. In the forecasting evaluation part, covering one-minute stock returns from three sectors of the S&P 100, JEAM achieves the best out-of-sample log scores for both tail directions; improving scores by 12.5-13.6% in the lower tail and 11.4-14.9% in the upper tail. The results show that incorporating market-informed network structures in the estimation, improves forecast evaluation of extremes across time series.
Read the full abstract

Statistics Applications · Cross-listed from eess.SP

Optimizing GEDI Simulator Configuration for European Temperate Forests

Accurate estimation of aboveground biomass density is essential for quantifying forest carbon stocks. NASA's GEDI mission provides valuable canopy structure data, but its sparse sampling necessitates the use of simulators to calibrate biomass models at field inventory locations. The widely used simulator of Hancock et al. (2019) emulates GEDI waveforms from airborne LiDAR point clouds, yet it has never been validated over European temperate forests. Here, we compare approximately 9,500 pairs of observed and simulated GEDI relative height (RH) profiles across French forests using the national airborne LiDAR program as input. We separate two sources of error: waveform modeling differences, assessed by referencing both simulated and real RH metrics to a common ALS-derived ground elevation, and ground detection bias, evaluated by comparing each GEDI L2A processing algorithm against the ALS reference. Under the baseline configuration, the mean absolute bias across the full RH profile reaches 0.69 m in leaf-on and 1.28 m in leaf-off conditions. Switching to intensity-based return weighting and selecting the a3 L2A algorithm reduces these biases to 0.44 m and 0.40 m respectively. The a3 algorithm also achieves near-unbiased ground detection (-0.03 m versus -0.88 m for the default), directly reducing a previously overlooked source of error. We also show that leaf-off acquisitions and low-sensitivity shots, both typically excluded from standard biomass products, are simulated as reliably as their counterparts, substantially expanding the potential calibration and inference datasets.
Read the full abstract

All authors Selim Behloul, Nikola Besic, Steven Hancock, Cedric Vega, Sylvie Durrieu, Jean-Pierre Renaud, Ibrahim Fayad, Philippe Ciais

Statistics Applications · Cross-listed from cs.CY

Buyer Artificial Intelligence-Enabled Environmental Governance and Supplier Environmental Controversies: An Organizational Information Processing and Signaling

Environmental controversies in global supply chains pose significant risks for global buyers. This study examines whether overseas suppliers' exposure to buyers' artificial intelligence (AI)-enabled environmental governance reduces supplier environmental controversies. Drawing on organizational information processing theory and signaling theory, we investigate how suppliers' exposure to AI-enabled governance influences their environmental controversies and the institutional contingencies under which this effect varies. Using text analysis to measure buyer AI-enabled environmental governance, we analyze panel data on 2,505 suppliers of U.S.-listed firms across 41 countries from 2020 to 2024 with multidimensional fixed-effects models. We find that suppliers' exposure to buyer AI-enabled environmental governance is negatively associated with supplier environmental controversies in the following year. This negative relationship is stronger in supplier countries with higher AI readiness and regulatory quality. The study contributes to research on AI-enabled sustainability governance and sustainable supply chain risk management.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Bayesian Variable Selection for High-Dimensional Predictors with Missing Psychometric Outcomes

High-dimensional, multimodal predictors and partially observed multivariate outcomes are common in psychometric research. However, existing regularization methods often do not accommodate hierarchical predictor structures and are primarily designed for univariate outcomes. We propose SHIM, a Bayesian framework for structured variable selection that combines hierarchical horseshoe shrinkage with a Bayesian treatment of missing outcomes. The framework jointly accommodates predictor hierarchies, dependence among outcomes, and incomplete multivariate responses. We establish theoretical properties of the proposed prior specification and evaluate SHIM through simulation studies. The results demonstrate that SHIM balances sensitivity with false-positive control while yielding accurate coefficient estimates and well-calibrated uncertainty quantification. We further apply SHIM to data from an Alzheimer's disease cohort to characterize associations between multimodal neuroimaging measures and multivariate neuropsychological outcomes and to generate posterior-based multiple imputations for downstream analyses of the relationships between fluid biomarkers and cognition. An R package, shim, is publicly available to facilitate implementation.
Read the full abstract

All authors Zongyue Teng, Shujie Ma, Timothy J. Hohman, Angela L. Jefferson, Panpan Zhang

Wednesday 9 September

Statistics Applications · Cross-listed from cs.GT

STV Audit Graphs: A Visual Tool to Measure Election Stability

The Single Transferable Vote (STV) is an algorithmic election rule. Round by round, a profile of ranked-choice ballots is reinterpreted to determine which decision to make next, and candidates are seated or eliminated until a full winner set emerges. This algorithmic nature makes STV theoretically more brittle than other election rules: uncertainty about an early round of the election might percolate irreversibly into the rest. For this reason, a generalized non-trivial Risk-Limiting Audit (RLA) framework has remained elusive for STV. Such a framework must concoct a set of null hypotheses, or assertions, whose rejection would bound the probability that the outcome of the election was incorrectly reported. We present audit graphs as a solution to design the assertions needed for RLAs of arbitrary STV elections, as well as quantitatively describe the uncertainty (or lack thereof) of their outcomes. These audit graphs explore election paths that are “close” to the recorded one by considering the alternative decisions the STV algorithm might have made if a small number of ballots were perturbed.
Read the full abstract

Statistics Applications · Cross-listed from physics.soc-ph

Estimating the Time Advantage of Split-Staggered Starts in Record-Breaking Middle-Distance Races

In July 2026, Josh Kerr broke the 26-year-old mile world record, the latest in an unusual concentration of recent middle-distance records that has led to widespread speculation about the role of advancing shoe and other technologies. However, like almost every other recent record-breaking performance, Kerr's race featured a split-staggered start – an often overlooked, relatively novel start configuration that places a subset of athletes in a front group in the outside lanes, reducing the geometric burden associated with accelerating on tight turns while also avoiding the bulk of the extra distance associated with outside positioning in a traditional distance race. Here, these advantages are quantified using empirical lane-performance data, OMEGA measurements of realized race distance, video reconstruction, and a force-based model of curved running. Applying the resulting corrections to record-breaking performances from 2026, estimated advantages range from approximately 0.1 seconds to more than 1 second, depending on the event, race configuration, and model assumptions. Although small in absolute terms, these advantages demonstrate that start configuration represents a meaningful and previously overlooked source of performance advantage in modern middle-distance racing – large enough to make the difference between a world record and an otherwise great performance.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Mixture-based Nonparametric Estimation of Spatial Covariance Functions with Applications to HIV Key Population Size Estimation across Sub-Saharan Africa

Consistent data on the sizes of key populations, such as female sex workers (FSWs), are often scarce, particularly at the sub-national level. Accurate size estimates are critical to effectively allocate resources and achieve HIV targets. Since FSW population sizes may be spatially correlated across areas, models that account for spatial dependence can improve estimation. An important component of such models is the covariance function, which characterizes the spatial dependence structure of the underlying process. In this work, we study spatial covariance functions to estimate FSW population sizes in Sub-Saharan Africa (SSA). Many spatial models rely on parametric covariance functions. However, parametric estimation can suffer from model mis-specification, potentially leading to inefficient or biased predictions. We therefore develop a robust non-parametric approach for estimating the covariance function of a stationary isotropic process in $\mathbb{R}^d$. We focus on a class of covariance functions that are valid in all dimensions, which includes popular kernels such as the exponential and Matérn kernels. Leveraging the fact that such covariance functions can be represented as infinite mixtures of scaled Gaussian kernels, we propose two estimation methods: weighted least squares and nonparametric maximum likelihood estimation to estimate the mixing measure of scaled Gaussian kernels. We also develop computationally efficient methods to solve these optimization problems using non-negative least squares and second-order descent updates. We evaluate the proposed methods through simulations and apply them to estimate the FSW population sizes at the sub-national level in SSA.
Read the full abstract

All authors Manushi Siriwardana, Hyebin Song, Le Bao, Stephen Berg

Statistics Applications · Cross-listed from eess.SY

Multivariate linear regression without prior assumptions

Recovering the linear relationships that govern a system from noisy measurements is a basic task across the physical and engineering sciences. Because every measured variable may carry an unknown amount of noise, classical regression must commit in advance to a set of structural assumptions: ordinary least squares requires a declared input-output partition with input variables being noise-free, total least squares assumes equal noise variance across all variables, and generalized total least squares additionally requires the noisy-variable partition and variances to be known beforehand. Kalman~\cite{Kalman:1982} showed that any procedure returning a unique linear model from inexact data must rest on such unverifiable a priori assumptions – “prejudices” – that cannot be checked against the data itself, and that removing them leaves the identification problem fundamentally indeterminate. Whether these prejudices can instead be resolved directly from the data has remain unresolved. Here we show that an iterative generalized-eigenvalue algorithm, QZ-IPCA, recovers the noisy-variable partition, noise variances, number of linear relations, and regression coefficients of a multivariate linear system simultaneously, using only the raw data. Across all possible exhaustive noise configurations of a five-variable benchmark network, QZ-IPCA correctly identifies model structure and recovers coefficients with error below 6.4%. It outperforms ordinary least squares even when given the best partition, and succeeds in rank identification precisely where standard total least squares falls once noise variances differ across variables. These results show that the assumptions conventionally required for multivariate regression are not necessary, recasting model identification as a problem solvable from data geometry alone.
Read the full abstract

All authors Mayank S.K. Gupta, Deepanjhan Das, Arun K. Tangirala, Shankar Narasimhan

Statistics Applications

The Impact of a Gridded Streamflow Measure on Drought Variation in the Conterminous United States

Models for droughts draw on a wide range of meteorological and hydrological inputs. Stakeholders classify droughts according to different purposes and priorities, and accordingly rely on different measures to explain and predict the onset of drought. While many meteorological inputs are available as gridded data products, hydrological streamflow measurements are often only available as point-referenced gauge measurements. This leads to an issue of misalignment for studies relying on both areal meteorological and point-referenced hydrological data. Such gauge data can also have notable spatial and/or temporal missingness. Many areas remain ungauged, and where gauges exist, equipment malfunctions cause temporal gaps. In this study, we document the value of an existing gridded streamflow measure for the conterminous United States, one which was specifically designed to match the spatio-temporal support of publicly available meteorological and ordinal drought measurements. We use this homogenized database to assess the relative importance of this streamflow measure in explaining US drought variability. This assessment first requires that we address autocorrelation and variability in the variance of observed droughts, two extensions to existing statistical methodology. After suitably controlling for both, our results show that the streamflow measure is often the most statistically important explanatory variable from among a wide set of meteorological variables. We explore the spatial variation and some drivers of this result.
Read the full abstract

All authors Rob Erhardt, Courtney Di Vittorio, Staci Hepler, Mostafa Shams, Wendy Wei, James Zhao

Statistics Applications

A spatiotemporal negative binomial model with dynamic dispersion: An application to Tuberculosis infections

Tuberculosis (TB) remains a critical public health concern in Brazil, characterized by pronounced spatial heterogeneity and fluctuating temporal volatility. In this paper, we study monthly TB notifications across 61 microregions of Sao Paulo state from 2001 to 2024. To do this, we introduce a negative binomial spatial integer-valued generalized autoregressive conditional heteroskedastic (INGARCH) model featuring jointly dynamic conditional means and time-varying dispersion. To capture inter-regional spillovers, we incorporate both discrete adjacency structures and a novel continuous distance-based formulation leveraging the Matern correlation function. Parameter estimation via conditional maximum likelihood employs a two-step profile-likelihood iterative scheme, demonstrating solid finite-sample performance in simulation studies. Applied to the Sao Paulo TB surveillance data, the framework substantially outperforms standard Poisson and fixed-dispersion spatiotemporal baselines in empirical fit and uncertainty quantification, maintaining nominal 95% predictive coverage across both dense metropolitan centers and rural microregions. Our results reveal marked spatial heterogeneity in baseline incidence, dynamic overdispersion driven by localized outbreaks, and short-range spatial interaction decay. By accurately modeling spatiotemporal volatility, the proposed methodology provides a robust statistical tool to support public health surveillance, policy-making, and resource allocation.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Cointegration by Parts: Locating Cointegration in Time

Tests for cointegration are typically applied to a single window spanning the entire sample, assuming that the long-run relationship holds throughout. When it holds over only a part of the sample, such tests lose power, because the stationary episode is diluted by periods without cointegration. We propose three statistics for testing whether two or more series cointegrate only over a part of the sample, each an infimum of the Engle-Granger statistic over recursive, backward-expanding, or doubly-flexible windows. We derive their limiting distributions and establish which alternatives each is consistent against. Only the doubly-flexible statistic has power against both break directions. Inspired by the seminal work of James G. MacKinnon, critical values are obtained by simulation and summarized through response surface regressions. We apply the tests to global mean sea level and global mean surface temperature anomalies. All three reject the null of no cointegration at the 5% level, locating it in a sub-period of the 1880–2019 record that coincides with documented discontinuities in sea surface temperature data collection.
Read the full abstract

Statistics Applications · Cross-listed from cs.CE

Geometric organization of olfactory descriptor data in the Poincaré disk

Odor quality is commonly represented using high dimensional descriptor profiles, yet their low dimensional organization remains unclear. We investigated whether a two-dimensional hyperbolic embedding can provide an interpretable representation of this structure. We applied hyperbolic metric multidimensional scaling to two complementary datasets: 480 Sagar rating profiles from three participants rating 160 odorants on 15 continuous descriptors, and 4983 GoodScents–Leffingwell molecules annotated with 138 binary descriptors. The embeddings substantially preserved pairwise descriptor distances, supporting subsequent analyses of radial and angular organization. In Sagar, rating profile entropy was strongly and negatively associated with hyperbolic radius, with diffuse profiles closer to the center and concentrated profiles closer to the boundary. This radial organization emerged primarily at the level of the full descriptor profile, rather than any individual descriptor, and remained robust across alternative descriptor representations, participant specific analyses, and averaged ratings. Sweet, musky, fruity, pleasantness showed the strongest directional trends. In GoodScents–Leffingwell, active label entropy, reflecting descriptor multiplicity, increased with radius, whereas orthogonalized descriptor entropy, reflecting spread across orthogonal modes, decreased with radius. Related binary descriptors occupied coherent localized high-density regions. These findings reveal complementary radial and angular organization in the hyperbolic representation of olfactory descriptor data. They support hyperbolic mapping as an interpretable descriptive framework in which radius summarizes global profile properties, while the angular component captures continuous descriptor gradients and categorical organization.
Read the full abstract

All authors Aniss Aiman Medbouhi, Farzaneh Taleb, Giovanni Luca Marchetti, Danica Kragic

Tuesday 8 September

Statistics Applications · Cross-listed from stat.ME

Differential Privacy Guarantees in Small Area Estimation

Statistical agencies increasingly rely on small area estimation to produce reliable estimates for subpopulations with limited sample sizes. These estimates are built from individual survey responses, so agencies must ensure that releasing them does not reveal information about any single respondent. We show that when a single draw from the posterior distribution of the Bayesian Fay-Herriot model is released, pure $\varepsilon$-differential privacy is unattainable, but the release satisfies formal privacy guarantees under Rényi differential privacy and zero-concentrated differential privacy without any noise being added, provided we treat the variance components as fixed. The key insight is that the posterior draw equals the posterior mean plus the Gaussian noise whose variance equals the posterior variance. The guarantee is thus governed by the sensitivity of the direct survey estimate and the posterior variance, and applies equally to a release of the posterior mean with that amount of noise added. For binary outcomes estimated with the Hájek estimator, the sensitivity equals the largest survey weight in the area divided by the sum of the weights. For the intercept-only model we derive exact coefficients describing how a change in one record propagates to every area's posterior mean, giving finite-sample per-area guarantees and a joint guarantee for releasing all areas at once that exceeds the largest per-area guarantee by at most a few percent in our applications. Two applications, poverty prevalence across 2,462 Public Use Microdata Areas in the American Community Survey and smoking prevalence across 52 substrata in the Washington state Behavioral Risk Factor Surveillance System, show that the guarantee is driven far more by the inequality of the survey weights than by the sample size, and that the shrinkage of the model tightens it substantially.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

A Counterfactual Framework for Estimating Infectious Disease Prevalence under Repeated Testing with Symptomatic and Contact-Tracing Components

This paper addresses the problem of estimating infectious disease prevalence under longitudinal testing programs that include scheduled, symptomatic, and contact-tracing testing. Our study is motivated by data from The Ohio State University, where a mandatory once-per-week COVID-19 testing and isolation program was implemented during the Fall 2020 semester, supplemented by additional testing for symptomatic individuals and identified contacts. In this setting, the probability of being tested depends on symptoms or contact-tracing status, creating a complex observation process. We develop a counterfactual framework that links the observation process to a hypothetical process in which infection is prevented. This formulation enables unbiased estimation of disease prevalence by modeling the testing process, possibly nonparametrically, without requiring explicit modeling of transmission dynamics, even though the testing and infection processes are jointly dependent.
Read the full abstract

All authors Jeongjin Lee, Junke Yang, Grzegorz A. Rempala, Patrick M. Schnell

Statistics Applications · Cross-listed from stat.ME

Kalman Filtering and Smoothing for Improving Precision in Horvitz–Thompson Estimation of Infectious Disease Prevalence

Horvitz–Thompson (HT) estimators can provide unbiased daily estimates of infectious disease prevalence under repeated surveillance by correcting for nonrandom testing induced by scheduled, symptom-based, and contact-tracing components. However, because each HT estimate is based on the testing data available for that day and may involve highly variable inverse probability weights, it can be noisy, have precision that varies over time, and become unavailable during temporary interruptions in testing. The daily HT estimator is modeled as a noisy observation of an underlying prevalence process, with day-specific observation variances estimated using a delete-a-group jackknife. Our primary specification is a joint local linear trend state-space model that extends the standard level-only random walk by adding a latent slope. The Kalman filter improves precision by borrowing information from past estimates. At each time $t$, the process variances are estimated or carried forward using only observations available through time $t$, so the resulting filtered estimate is available in real time. We also describe the corresponding Kalman smoother as a retrospective extension based on the full observed series. When daily HT estimates are missing, the Kalman filter proceeds through prediction-only updates, whereas the corresponding smoother retrospectively reconstructs those periods using later observations. In simulations, the joint Kalman filter substantially improves precision relative to the raw daily HT estimator while preserving the main temporal pattern, and the smoother provides a more stable retrospective summary. In The Ohio State University's fall 2020 SARS-CoV-2 surveillance data, the filter and smoother provide estimates on no-testing days, when the HT estimator provides neither point nor interval estimates, and yield narrower confidence intervals than HT intervals on observed days.
Read the full abstract

Statistics Applications · Cross-listed from stat.ME

Covariate Adjustment in Randomized Experiments: A Unified Framework for Decision and Practice

Should researchers adjust for covariates in randomized experiments, and if so, how? The literature offers three distinct prescriptions: do not adjust because randomization guarantees unbiasedness; adjust for outcome-prognostic covariates to improve precision; or adjust for covariates imbalanced between treatment arms. These competing prescriptions create confusion and uncertainty. We develop a unified framework for decision and practice. Given available information, we show that the optimal correction is what we call ex-post bias. The only relevant criterion for adjustment is prognosticity for ex-post bias; neither raw covariate imbalance nor outcome prognosticity is sufficient by itself. We also show that correcting imbalance and improving precision are two sides of the same decision problem. We develop two estimation approaches, one of which recovers familiar adjustment estimators and provides a new theoretical justification for them. Simulations compare alternative covariate-selection and adjustment strategies. Overall, our framework provides a unified foundation for covariate adjustment in randomized experiments.
Read the full abstract

Statistics Applications · Cross-listed from cs.LG

Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling

A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to data generation: frozen transformers can simulate iterative generative samplers from in-context samples. We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers. The construction identifies a concrete generative role for softmax attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained language models, we study semantic-topic sampling: prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.
Read the full abstract

All authors Arman Adibi, Alireza Jafari, Mohammad Ghavamzadeh, Hadi Daneshmand