Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Tuesday 4 November

Statistics Applications · Cross-listed from cs.CE

Physics-Informed Neural Network-based Reliability Analysis of Buried Pipelines

Buried pipelines transporting oil and gas across geohazard-prone regions are exposed to potential ground movement, leading to the risk of significant strain demand and structural failure. Reliability analysis, which determines the probability of failure after accounting for pertinent uncertainties, is essential for ensuring the safety of pipeline systems. However, traditional reliability analysis methods involving computationally intensive numerical models, such as finite element simulations of pipeline subjected to ground movement, have limited applications; this is partly because stochastic sampling approaches require repeated simulations over a large number of samples for the uncertain variables when estimating low probabilities. This study introduces Physics-Informed Neural Network for Reliability Analysis (PINN-RA) for buried pipelines subjected to ground movement, which integrates PINN-based surrogate model with Monte Carlo Simulation (MCS) to achieve efficient reliability assessment. To enable its application under uncertain variables associated with soil properties and ground movement, the PINN-based surrogate model is extended to solve a parametric differential equation system, namely the governing equation of pipelines embedded in soil with different properties. The findings demonstrate that PINN-RA significantly reduces the computational effort required and thus accelerates reliability analysis. By eliminating the need for repetitive numerical evaluations of pipeline subjected to permanent ground movement, the proposed approach provides an efficient and scalable tool for pipeline reliability assessment, enabling rapid decision-making in geohazard-prone regions.
Read the full abstract

Statistics Applications · Cross-listed from cs.CL

Sequential Bayesian Evaluation of Large Language Model Behavior

It is increasingly important to evaluate the characteristics of systems based on large language models (LLMs). Evaluations in this context often rely on a curated benchmark set of input prompts provided to the LLM, where the output for each prompt may be assigned a binary or ordinal score and the aggregation of scores across prompts is then used as a summary evaluation. In this paper, we develop a Bayesian approach for quantifying the uncertainty that arises in such evaluation metrics as a result of the stochasticity of the LLM-based systems; the same prompt may exhibit different outcomes on repeated runs. Our framework leads naturally to a sequential evaluation, in which we leverage the Bayesian model to preferentially select which prompts in the benchmark to use next, enabling more cost-effective LLM evaluations. We demonstrate this approach through four case studies: pairwise LLM preferences in interactive dialogue (MT-Bench), refusal rates on adversarial inputs (JailbreakBench), embodied agentic reasoning (ALFWorld), and agentic web-search Q&A (GAIA). Our results illustrate how a Bayesian approach can provide insightful uncertainty quantification about the behavior of blackbox LLM-based systems in a manner that is not accessible via deterministic or single-run evaluations.
Read the full abstract

All authors Saatvik Kher, Shang Wu, Rachel Longjohn, Catarina Belém, Padhraic Smyth

Statistics Applications

Centralized Health and Exposomic Resource (C-HER): Analytic and AI-Ready Data for External Exposomic Research

The Centralized Health and Exposomic Resource (C-HER) project has identified, profiled, spatially indexed, and stored over 30 external exposomic datasets. The resulting analytic and AI-ready data (AAIRD) provides a significant opportunity to develop an integrated picture of the exposome for health research. The exposome is a conceptual framework designed to guide the study of the complex environmental and genetic factors that together shape human health. Few composite measures of the exposome exist due to the high dimensionality of exposure data, multimodal data sources, and varying spatiotemporal scales. We develop a data engineering solution that overcomes the challenges of spatio-temporal linkage in this field. We provide examples of how environmental data can be combined to characterize a region, model air pollution, or provide indicators for cancer research. The development of AAIRD will allow future studies to use ML and deep learning methods to generate spatial and contextual exposure data for disease prediction.
Read the full abstract

All authors Heidi A. Hanson, Joemy Ramsay, Josh Grant, Maggie Davis, Janet O. Agbaje, Dakotah Maguire, Jeremy Logan, Marissa Taddie, Chad Melton, Midgie MacFarland, James VanDerslice

Statistics Applications · Cross-listed from cs.LG

Applying Time Series Deep Learning Models to Forecast the Growth of Perennial Ryegrass in Ireland

Grasslands, constituting the world's second-largest terrestrial carbon sink, play a crucial role in biodiversity and the regulation of the carbon cycle. Currently, the Irish dairy sector, a significant economic contributor, grapples with challenges related to profitability and sustainability. Presently, grass growth forecasting relies on impractical mechanistic models. In response, we propose deep learning models tailored for univariate datasets, presenting cost-effective alternatives. Notably, a temporal convolutional network designed for forecasting Perennial Ryegrass growth in Cork exhibits high performance, leveraging historical grass height data with RMSE of 2.74 and MAE of 3.46. Validation across a comprehensive dataset spanning 1,757 weeks over 34 years provides insights into optimal model configurations. This study enhances our understanding of model behavior, thereby improving reliability in grass growth forecasting and contributing to the advancement of sustainable dairy farming practices.
Read the full abstract

All authors Oluwadurotimi Onibonoje, Vuong M. Ngo, Andrew McCarre, Elodie Ruelle, Bernadette O-Briend, Mark Roantree

Machine Learning (Statistics) · Cross-listed from cs.AI

Epidemiology of Large Language Models: A Benchmark for Observational Distribution Knowledge

Artificial intelligence (AI) systems hold great promise for advancing various scientific disciplines, and are increasingly used in real-world applications. Despite their remarkable progress, further capabilities are expected in order to achieve more general types of intelligence. A critical distinction in this context is between factual knowledge, which can be evaluated against true or false answers (e.g., "what is the capital of England?"), and probabilistic knowledge, reflecting probabilistic properties of the real world (e.g., "what is the sex of a computer science graduate in the US?"). In this paper, our goal is to build a benchmark for understanding the capabilities of LLMs in terms of knowledge of probability distributions describing the real world. Given that LLMs are trained on vast amounts of text, it may be plausible that they internalize aspects of these distributions. Indeed, LLMs are touted as powerful universal approximators of real-world distributions. At the same time, classical results in statistics, known as curse of dimensionality, highlight fundamental challenges in learning distributions in high dimensions, challenging the notion of universal distributional learning. In this work, we develop the first benchmark to directly test this hypothesis, evaluating whether LLMs have access to empirical distributions describing real-world populations across domains such as economics, health, education, and social behavior. Our results demonstrate that LLMs perform poorly overall, and do not seem to internalize real-world statistics naturally. When interpreted in the context of Pearl's Causal Hierarchy (PCH), our benchmark demonstrates that language models do not contain knowledge on observational distributions (Layer 1 of PCH), and thus the Causal Hierarchy Theorem implies that interventional (Layer 2) and counterfactual (Layer 3) knowledge of these models is also limited.
Read the full abstract

All authors Drago Plecko, Patrik Okanovic, Shreyas Havaldar, Torsten Hoefler, Elias Bareinboim

Machine Learning (Statistics)

Precise asymptotic analysis of Sobolev training for random feature models

Gradient information is widely useful and available in applications, and is therefore natural to include in the training of neural networks. Yet little is known theoretically about the impact of Sobolev training – regression with both function and gradient data – on the generalization error of highly overparameterized predictive models in high dimensions. In this paper, we obtain a precise characterization of this training modality for random feature (RF) models in the limit where the number of trainable parameters, input dimensions, and training data tend proportionally to infinity. Our model for Sobolev training reflects practical implementations by sketching gradient data onto finite dimensional subspaces. By combining the replica method from statistical physics with linearizations in operator-valued free probability theory, we derive a closed-form description for the generalization errors of the trained RF models. For target functions described by single-index models, we demonstrate that supplementing function data with additional gradient data does not universally improve predictive performance. Rather, the degree of overparameterization should inform the choice of training method. More broadly, our results identify settings where models perform optimally by interpolating noisy function and gradient data.
Read the full abstract

All authors Katharine E Fisher, Matthew TC Li, Youssef Marzouk, Timo Schorlepp

Statistical Methodology

New sampling approaches for Shrinkage Inverse-Wishart distribution

In this paper, we propose new sampling approaches for the Shrinkage Inverse-Wishart (SIW) distribution, a generalized family of the Inverse-Wishart distribution originally proposed by Berger et al. (2020, Annals of Statistics). It offers a flexible prior for covariance matrices and remains conjugate to the Gaussian likelihood, similar to the classical Inverse-Wishart. Despite these advantages, sampling from SIW remains challenging. The existing algorithm relies on a nested Gibbs sampler, which is slow and lacks rigorous theoretical analysis of its convergence. We propose a new algorithm based on the Sampling Importance Resampling (SIR) method, which is significantly faster and comes with theoretical guarantees on convergence rates. A known issue with SIR methods is the large discrepancy in importance weights, which occurs when the proposal distribution has thinner tails than the target. In the case of SIW, certain parameter settings can lead to such discrepancies, reducing the robustness of the output samples. To sample from such SIW distributions, we robustify the proposed algorithm by including a clipping step to the SIR framework which transforms large importance weights. We provide theoretical results on the convergence behavior in terms of the clipping size, and discuss strategies for choosing this parameter via simulation studies. The robustified version retains the computational efficiency of the original algorithm.
Read the full abstract

Statistics Applications · Cross-listed from eess.SY

Quantifying Power Systems Resilience Using Statistical Analysis and Bayesian Learning

The increasing frequency and intensity of extreme weather events is significantly affecting the power grid, causing large-scale outages and impacting power system resilience. Yet limited work has been done on systematically modeling the impacts of weather parameters to quantify resilience. This study presents a framework using statistical and Bayesian learning approaches to quantitatively model the relationship between weather parameters and power system resilience metrics. By leveraging real-world publicly available outage and weather data, we identify key weather variables of wind speed, temperature, and precipitation influencing a particular region's resilience metrics. A case study of Cook County, Illinois, and Miami-Dade County, Florida, reveals that these weather parameters are critical factors in resiliency analysis and risk assessment. Additionally, we find that these weather variables have combined effects when studied jointly compared to their effects in isolation. This framework provides valuable insights for understanding how weather events affect power distribution system performance, supporting decision-makers in developing more effective strategies for risk mitigation, resource allocation, and adaptation to changing climatic conditions.
Read the full abstract

All authors Apsara Adhikari, Charlotte Wertz, Anamika Dubey, Arslan Ahmad, Ian Dobson

Machine Learning (Statistics) · Cross-listed from cs.LG

Discrete Bayesian Sample Inference for Graph Generation

Generating graph-structured data is crucial in applications such as molecular generation, knowledge graphs, and network analysis. However, their discrete, unordered nature makes them difficult for traditional generative models, leading to the rise of discrete diffusion and flow matching models. In this work, we introduce GraphBSI, a novel one-shot graph generative model based on Bayesian Sample Inference (BSI). Instead of evolving samples directly, GraphBSI iteratively refines a belief over graphs in the continuous space of distribution parameters, naturally handling discrete structures. Further, we state BSI as a stochastic differential equation (SDE) and derive a noise-controlled family of SDEs that preserves the marginal distributions via an approximation of the score function. Our theoretical analysis further reveals the connection to Bayesian Flow Networks and Diffusion models. Finally, in our empirical evaluation, we demonstrate state-of-the-art performance on molecular and synthetic graph generation, outperforming existing one-shot graph generative models on the standard benchmarks Moses and GuacaMol.
Read the full abstract

All authors Ole Petersen, Marcel Kollovieh, Marten Lienen, Stephan Günnemann

Machine Learning (Statistics)

Unifying Information-Theoretic and Pair-Counting Clustering Similarity

Comparing clusterings is central to evaluating unsupervised models, yet the many existing similarity measures can produce widely divergent, sometimes contradictory, evaluations. Clustering similarity measures are typically organized into two principal families, pair-counting and information-theoretic, reflecting whether they quantify agreement through element pairs or aggregate information across full cluster contingency tables. Prior work has uncovered parallels between these families and applied empirical normalization or chance-correction schemes, but their deeper analytical connection remains only partially understood. Here, we develop an analytical framework that unifies these families through two complementary perspectives. First, both families are expressed as weighted expansions of observed versus expected co-occurrences, with pair-counting arising as a quadratic, low-order approximation and information-theoretic measures as higher-order, frequency-weighted extensions. Second, we generalize pair-counting to k-tuple agreement and show that information-theoretic measures can be viewed as systematically accumulating higher-order co-assignment structure beyond the pairwise level. We illustrate the approaches analytically for the Rand index and Mutual Information, and show how other indices in each family emerge as natural extensions. Together, these views clarify when and why the two regimes diverge, relating their sensitivities directly to weighting and approximation order, and provide a principled basis for selecting, interpreting, and extending clustering similarity measures across applications.
Read the full abstract

Machine Learning (Statistics)

Scalable Single-Cell Gene Expression Generation with Latent Diffusion Models

Computational modeling of single-cell gene expression is crucial for understanding cellular processes, but generating realistic expression profiles remains a major challenge. This difficulty arises from the count nature of gene expression data and complex latent dependencies among genes. Existing generative models often impose artificial gene orderings or rely on shallow neural network architectures. We introduce a scalable latent diffusion model for single-cell gene expression data, which we refer to as scLDM, that respects the fundamental exchangeability property of the data. Our VAE uses fixed-size latent variables leveraging a unified Multi-head Cross-Attention Block (MCAB) architecture, which serves dual roles: permutation-invariant pooling in the encoder and permutation-equivariant unpooling in the decoder. We enhance this framework by replacing the Gaussian prior with a latent diffusion model using Diffusion Transformers and linear interpolants, enabling high-quality generation with multi-conditional classifier-free guidance. We show its superior performance in a variety of experiments for both observational and perturbational single-cell data, as well as downstream tasks like cell-level classification.
Read the full abstract

All authors Giovanni Palla, Sudarshan Babu, Payam Dibaeinia, James D. Pearce, Donghui Li, Aly A. Khan, Theofanis Karaletsos, Jakub M. Tomczak

Statistical Methodology

Constructing Large Orthogonal Minimally Aliased Response Surface Designs by Concatenating Two Definitive Screening Designs

Orthogonal minimally aliased response surface (OMARS) designs permit the study of quantitative factors at three levels using an economical number of runs. In these designs, the linear effects of the factors are neither aliased with each other nor with the quadratic effects and the two-factor interactions. Complete catalogs of OMARS designs with up to five factors have been obtained using an enumeration algorithm. However, the algorithm is computationally demanding for designs with many factors and runs. To overcome this issue, we propose a construction method for large OMARS designs that concatenates two definitive screening designs and improves the statistical features of its parent designs. The concatenation employs an algorithm that minimizes the aliasing among the second-order effects using foldover techniques and column permutations for one of the parent designs. We study the properties of the new OMARS designs and compare them with alternative designs in the literature.
Read the full abstract

Statistical Methodology

Detecting Conflicts in Evidence Synthesis Models Using Score Discrepancies

Evidence synthesis models combine multiple data sources to estimate latent quantities of interest, enabling reliable inference on parameters that are difficult to measure directly. However, shared parameters across data sources can induce conflicts both among the data and with the assumed model structure. Detecting and quantifying such conflicts remains a challenge in model criticism. Here we propose a general framework for conflict detection in evidence synthesis models based on score discrepancies, extending prior-data conflict diagnostics to more general conflict checks in the latent space of hierarchical models. Simulation studies in an exchangeable model demonstrate that the proposed approach effectively detects between-data inconsistencies. Application to an influenza severity model illustrates its use, complementary to traditional deviance-based diagnostics, in complex real-world hierarchical settings. The proposed framework thus provides a flexible and broadly applicable tool for consistency assessment in Bayesian evidence synthesis.
Read the full abstract

Statistical Methodology

Adaptive Orthogonalization for Stable Estimation of the Effects of Time-Varying Treatments

Inferring the causal effects of time-varying treatments is often hindered by highly variable inverse propensity weights, particularly in settings with limited covariate overlap. Building on the key framework of Imai and Ratkovic (2015), we establish sufficient balancing conditions for identification in longitudinal studies of treatment effects and propose a novel estimator that directly targets features of counterfactual or potential covariates. Instead of balancing observed covariates, our method balances the components of covariates that are orthogonal to their history, thereby isolating the new information at each time point. This strategy directly targets the joint distribution of potential covariates and prioritizes features that are most relevant to the outcome. We prove that the resulting estimator for the mean potential outcome is consistent and asymptotically normal, even in settings where standard inverse propensity weighting fails. Extensive simulations show that our estimator attains efficiency comparable to that of g-computation while providing superior robustness to model misspecification. We apply our method to a longitudinal study of private versus public schooling in Chile, demonstrating its stability and interpretability in estimating their effects on university admission scores.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Power Constrained Nonstationary Bandits with Habituation and Recovery Dynamics

A common challenge for decision makers is selecting actions whose rewards are unknown and evolve over time based on prior policies. For instance, repeated use may reduce an action's effectiveness (habituation), while inactivity may restore it (recovery). These nonstationarities are captured by the Reducing or Gaining Unknown Efficacy (ROGUE) bandit framework, which models real-world settings such as behavioral health interventions. While existing algorithms can compute sublinear regret policies to optimize these settings, they may not provide sufficient exploration due to overemphasis on exploitation, limiting the ability to estimate population-level effects. This is a challenge of particular interest in micro-randomized trials (MRTs) that aid researchers in developing just-in-time adaptive interventions that have population-level effects while still providing personalized recommendations to individuals. In this paper, we first develop ROGUE-TS, a Thompson Sampling algorithm tailored to the ROGUE framework, and provide theoretical guarantees of sublinear regret. We then introduce a probability clipping procedure to balance personalization and population-level learning, with quantified trade-off that balances regret and minimum exploration probability. Validation on two MRT datasets concerning physical activity promotion and bipolar disorder treatment shows that our methods both achieve lower regret than existing approaches and maintain high statistical power through the clipping procedure without significantly increasing regret. This enables reliable detection of treatment effects while accounting for individual behavioral dynamics. For researchers designing MRTs, our framework offers practical guidance on balancing personalization with statistical validity.
Read the full abstract

All authors Fengxu Li, Stephanie M. Carpenter, Matthew P. Buman, Yonatan Mintz

Statistical Computation

Optimal transport with a density-dependent cost function

A new pairwise cost function is proposed for the optimal transport barycenter problem, adopting the form of the minimal action between two points, with a Lagrangian that takes into account an underlying probability distribution. Under this notion of distance, two points can only be close if there exist paths joining them that do not traverse areas of small probability. A framework is proposed and developed for the numerical solution of the corresponding data-driven optimal transport problem. The procedure parameterizes the paths of minimal action through path dependent Chebyshev polynomials and enforces the agreement between the paths' endpoints and the given source and target distributions through an adversarial penalization. The methodology and its application to clustering and matching problems is illustrated through synthetic examples.
Read the full abstract

Other Statistics

From Hume to Jaynes: Induction as the Logic of Plausible Reasoning

The problem of induction has persisted since Hume exposed the logical gap between repeated observation and universal inference. Traditional attempts to resolve it have oscillated between two extremes: the probabilistic optimism of Laplace and Jeffreys, who sought to quantify belief through probability, and the critical skepticism of Popper, who replaced confirmation with falsification. Both approaches, however, assume that induction must deliver certainty or its negation. In this paper, I argue that the problem of induction dissolves when recast in terms of logical coherence (understood as internal consistency of credences under updating) rather than truth. Following E. T. Jaynes, probability is interpreted not as frequency or decision rule but as the extension of deductive logic to incomplete information. Under this interpretation, Bayes's theorem is not an empirical statement but a consistency condition that constrains rational belief updating. Induction thus emerges as the special case of deductive reasoning applied to uncertain premises. Falsification appears as the limiting form of Bayesian updating when new data drive posterior plausibility toward zero, while the Bayes Factor quantifies the continuous spectrum of evidential strength. Through analytical examples, including Laplace's sunrise problem, Jeffreys's mixed prior, and confidence-based reformulations, I show that only the logic of plausible reasoning unifies these perspectives without contradiction. Induction, properly understood, is not the leap from past to future but the discipline of maintaining coherence between evidence, belief, and information.
Read the full abstract

Statistics Theory

Curvature of high-dimensional data

We consider the problem of estimating curvature where the data can be viewed as a noisy sample from an underlying manifold. For manifolds of dimension greater than one there are multiple definitions of local curvature, each suggesting a different estimation process for a given data set. Recently, there has been progress in proving that estimates of “local point cloud curvature" converge to the related smooth notion of local curvature as the density of the point cloud approaches infinity. Herein we investigate practical limitations of such convergence theorems and discuss the significant impact of bias in such estimates as reported in recent literature. We provide theoretical arguments for the fact that bias increases drastically in higher dimensions, so much so that in high dimensions, the probability that a naive curvature estimate lies in a small interval near the true curvature could be near zero. We present a probabilistic framework that enables the construction of more accurate estimators of curvature for arbitrary noise models. The efficacy of our technique is supported with experiments on spheres of dimension as large as twelve.
Read the full abstract

Statistical Methodology

The Bias-Variance Tradeoff in Long-Term Experimentation

As we exhaust methods that reduces variance without introducing bias, reducing variance in experiments often requires accepting some bias, using methods like winsorization or surrogate metrics. While this bias-variance tradeoff can be optimized for individual experiments, bias may accumulate over time, raising concerns for long-term optimization. We analyze whether bias is ever acceptable when it can accumulate, and show that a bias-variance tradeoff persists in long-term settings. Improving signal-to-noise remains beneficial, even if it introduces bias. This implies we should shift from thinking there is a single “correct”, unbiased metric to thinking about how to make the best estimates and decisions when better precision can be achieved at the expense of bias. Furthermore, our model adds nuance to previous findings that suggest less stringent launch criterion leads to improved gains. We show while this is beneficial when the system is far from the optimum, more stringent launch criterion is preferable as the system matures.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models

Zeroth-order or derivative-free optimization (MeZO) is an attractive strategy for finetuning large language models (LLMs) because it eliminates the memory overhead of backpropagation. However, it converges slowly due to the inherent curse of dimensionality when searching for descent directions in the high-dimensional parameter space of billion-scale LLMs. We propose ConMeZO, a novel zeroth-order optimizer that accelerates convergence by adaptive directional sampling. Instead of drawing the direction uniformly at random, ConMeZO restricts the sampling to a cone centered around a momentum estimate. This concentrates the search in directions where the true gradient is more likely to lie and thus reduces the effect of high dimensions. We prove that ConMeZO achieves the same worst-case convergence rate as MeZO. Empirically, when finetuning LLMs on natural language tasks, ConMeZO is up to 2X faster than MeZO while retaining the low-memory footprint of zeroth-order methods.
Read the full abstract

All authors Lejs Deen Behric, Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil

Statistical Methodology

DANIEL: A Distributed and Scalable Approach for Global Representation Learning with EHR Applications

Classical probabilistic graphical models face fundamental challenges in modern data environments, which are characterized by high dimensionality, source heterogeneity, and stringent data-sharing constraints. In this work, we revisit the Ising model, a well-established member of the Markov Random Field (MRF) family, and develop a distributed framework that enables scalable and privacy-preserving representation learning from large-scale binary data with inherent low-rank structure. Our approach optimizes a non-convex surrogate loss function via bi-factored gradient descent, offering substantial computational and communication advantages over conventional convex approaches. We evaluate our algorithm on multi-institutional electronic health record (EHR) datasets from 58,248 patients across the University of Pittsburgh Medical Center (UPMC) and Mass General Brigham (MGB), demonstrating superior performance in global representation learning and downstream clinical tasks, including relationship detection, patient phenotyping, and patient clustering. These results highlight a broader potential for statistical inference in federated, high-dimensional settings while addressing the practical challenges of data complexity and multi-institutional integration.
Read the full abstract

All authors Zebin Wang, Ziming Gan, Weijing Tang, Zongqi Xia, Tianrun Cai, Tianxi Cai, Junwei Lu

Machine Learning (Statistics)

Optimizing Kernel Discrepancies via Subset Selection

Kernel discrepancies are a powerful tool for analyzing worst-case errors in quasi-Monte Carlo (QMC) methods. Building on recent advances in optimizing such discrepancy measures, we extend the subset selection problem to the setting of kernel discrepancies, selecting an m-element subset from a large population of size $n \gg m$. We introduce a novel subset selection algorithm applicable to general kernel discrepancies to efficiently generate low-discrepancy samples from both the uniform distribution on the unit hypercube, the traditional setting of classical QMC, and from more general distributions $F$ with known density functions by employing the kernel Stein discrepancy. We also explore the relationship between the classical $L_2$ star discrepancy and its $L_\infty$ counterpart.
Read the full abstract

All authors Deyao Chen, François Clément, Carola Doerr, Nathan Kirk

Statistics Applications

Extended Kalman Filtering on Stiefel Manifolds

A generalisation of the extended Kalman filter for Stiefel manifold-valued measurements is presented. We provide simulations on the 2-sphere and the space of orthogonal 4-by-2 matrices which show significant improvement of the Extended Kalman Filter compared to only relying on raw measurements.
Read the full abstract

Statistics Theory

On the Convergence of the Extended Kalman Filter on Stiefel Manifolds when Observing a Constant Particle with Measurement Errors

In this paper we first introduce the setting of filtering on Stiefel manifolds. Then, assuming the underlying system process is constant, the convergence of the extended Kalman filter with Stiefel manifold-valued observations is proved. This corresponds to the case where one has measurement errors that needs to be filtered. Finally, some simulations are presented for a selected few Stiefel manifolds and the speed of convergence is studied.
Read the full abstract

Statistics Theory

Spectral analysis of high-dimensional spot volatility matrix with applications

In random matrix theory, the spectral distribution of the covariance matrix has been well studied under the large dimensional asymptotic regime when the dimensionality and the sample size tend to infinity at the same rate. However, most existing theories are built upon the assumption of independent and identically distributed samples, which may be violated in practice. For example, the observational data of continuous-time processes at discrete time points, namely, the high-frequency data. In this paper, we extend the classical spectral analysis for the covariance matrix in large dimensional random matrix to the spot volatility matrix by using the high-frequency data. We establish the first-order limiting spectral distribution and obtain a second-order result, that is, the central limit theorem for linear spectral statistics. Moreover, we apply the results to design some feasible tests for the spot volatility matrix, including the identity and sphericity tests. Simulation studies justify the finite sample performance of the test statistics and verify our established theory.
Read the full abstract

All authors Qiang Liu, Yiming Liu, Zhi Liu, Wang Zhou

Statistics Applications · Cross-listed from physics.soc-ph

Stochastic Redistribution of Indistinguishable Items in Shared Habitation: A Multi-Agent Simulation Framework

This paper presents a discrete-event stochastic model for the redistribution of indistinguishable personal items, exemplified by socks, among multiple cohabitants sharing a communal laundry system. Drawing on concepts from ecological population dynamics, diffusion processes, and stochastic exchange theory, the model captures the probabilistic mechanisms underlying item mixing, recovery, and loss. Each cohabitant is represented as an autonomous agent whose belongings interact through iterative cycles of collective washing, sorting, and partial correction. The system's evolution is characterized by random mixing events, selective recollection, and attrition over time. Implemented using the SimPy discrete-event simulation framework, the model demonstrates that even minimal exchange probabilities can generate emergent asymmetries, quasi-equilibrium distributions, and long-term disorder. The findings illustrate how stochastic processes inherent to shared domestic systems can produce persistent imbalances, offering a quantitative perspective on an everyday social phenomenon.
Read the full abstract

Statistical Methodology

Synthetic Control with Weight Uncertainty: Robust Identification and Statistical Inference

The synthetic control method estimates causal effects by comparing a treated unit with weighted controls matched on its pre-treatment trajectory. However, validity can be compromised when highly correlated controls leave the synthetic weights weakly determined or when treated-control relationships shift after treatment. We propose a new estimand, the weight-robust treatment effect, defined as the optimizer of a worst-case optimization problem over an uncertainty class of synthetic weights compatible with the pre-treatment fit. We establish its connection to sensitivity analysis: the uncertainty class induces an interval of plausible treatment effects, and the proposed estimand is the point in the interval closest to zero. Under the classical identification conditions, the estimand coincides with the true treatment effect. When these conditions fail, the estimand remains point identified; if the uncertainty class contains the true post-treatment weight, it provides a conservative sign-preserving bound on the effect. The estimator of this target may have a non-normal limiting distribution, and we propose a novel perturbation-based method for constructing valid confidence intervals. Our proposal may be of independent interest for partial identification and sensitivity analysis, where estimators defined through constrained optimization can have non-standard limiting distributions.
Read the full abstract

Statistical Methodology

Bayesian copula-based spatial random effects models for inference with complex spatial data

In this article, we develop fully Bayesian, copula-based, spatial-statistical models for large, noisy, incomplete, and non-Gaussian spatial data. Our approach includes novel constructions of copulas that accommodate a spatial-random-effects structure, enabling low-rank representations and computationally efficient Bayesian inference. The spatial copula is used in a latent process model of the Bayesian hierarchical spatial-statistical model, and, conditional on the latent copula-based spatial process, the data model handles measurement errors and missing data. Our simulation studies show that a fully Bayesian approach delivers accurate and fast inference for both parameter estimation and spatial-process prediction, outperforming several benchmark methods, including fixed rank kriging (FRK). The new class of copula-based models is used to map atmospheric methane in the Bowen Basin, Queensland, Australia, from Sentinel 5P satellite data.
Read the full abstract

Statistical Methodology · Cross-listed from math-ph

Analysis of dissipative dynamics on noncommutative spaces and statistical inference of continuous time network stochastic processes

In this thesis, we analyse the generalisations of the Ornstein-Uhlenbeck (OU) semigroup and study them in both quantum and classical setups. In the first three chapters, we analyse the dissipative dynamics on noncommutative/quantum spaces, in particular, the systems with multiparticle interactions associated to CCR algebras. We provide various models where the dissipative dynamics are constructed using noncommutative Dirichlet forms. Some of our models decay to equilibrium algebraically and the Poincare inequality does not hold. Using the classical representation of generators of nilpotent Lie algebras, we provide the noncommutative representations of Lie algebras in terms of creation and annihilation operators and discuss the construction of corresponding Dirichlet forms. This introduces the opportunity to explore quantum stochastic processes related to Lie algebras and nilpotent Lie algebras. Additionally, these representations enable the investigation of the noncommutative analogue of hypoellipticity. In another direction, we explore the potential for introducing statistical models within a quantum framework. In this thesis, however, we present a classical statistical model of multivariate Graph superposition of OU (Gr supOU) process which allows for long(er) memory in the modelling of sparse graphs. We estimate these processes using generalised method of moments and show that it yields consistent estimators. We demonstrate the asymptotic normality of the moment estimators and validate these estimators through a simulation study.
Read the full abstract

Statistics Applications

Identification of Separable OTUs for Multinomial Classification in Compositional Data Analysis

High-throughput sequencing has transformed microbiome research, but it also produces inherently compositional data that challenge standard statistical and machine learning methods. In this work, we propose a multinomial classification framework for compositional microbiome data based on penalized log-ratio regression and pairwise separability screening. The method quantifies the discriminative ability of each OTU through the area under the receiver operating characteristic curve ($AUC$) for all pairwise log-ratios and aggregates these values into a global separability index $S_k$, yielding interpretable rankings of taxa together with confidence intervals. We illustrate the approach by reanalyzing the Baxter colorectal adenoma dataset and comparing our results with Greenacre's ordination-based analysis using Correspondence Analysis and Canonical Correspondence Analysis. Our models consistently recover a core subset of taxa previously identified as discriminant, thereby corroborating Greenacre's main findings, while also revealing additional OTUs that become important once demographic covariates are taken into account. In particular, adjustment for age, gender, and diabetes medication improves the precision of the separation index and highlights new, potentially relevant taxa, suggesting that part of the original signal may have been influenced by confounding. Overall, the integration of log-ratio modeling, covariate adjustment, and uncertainty estimation provides a robust and interpretable framework for OTU selection in compositional microbiome data. The proposed method complements existing ordination-based approaches by adding a probabilistic and inferential perspective, strengthening the identification of biologically meaningful microbial signatures.
Read the full abstract

All authors R. Alberich, N.A. Cruz, R. Fernández, I. García Mosquera, A. Mir, F. Rosselló