Statistics Applications · Cross-listed from cs.CE
Physics-Informed Neural Network-based Reliability Analysis of Buried Pipelines
Pouya Taraghi, Yong Li and Samer Adeeb
Buried pipelines transporting oil and gas across geohazard-prone regions are exposed to potential ground movement, leading to the risk of significant strain demand and structural failure. Reliability analysis, which determines the probability of failure after accounting for pertinent uncertainties, is essential for ensuring the safety of pipeline systems. However, traditional reliability analysis methods involving computationally intensive numerical models, such as finite element simulations of pipeline subjected to ground movement, have limited applications; this is partly because
stochastic sampling approaches require repeated simulations over a large number of samples for the uncertain variables when estimating low probabilities. This study introduces Physics-Informed Neural Network for Reliability Analysis (PINN-RA) for buried pipelines subjected to ground movement, which integrates PINN-based surrogate model with Monte Carlo Simulation (MCS) to achieve efficient reliability assessment. To enable its application under uncertain variables associated with soil properties and ground movement, the PINN-based surrogate model is extended to solve a parametric differential equation system, namely the governing equation of pipelines embedded in soil with different properties. The findings demonstrate that PINN-RA significantly reduces the computational effort required and thus accelerates reliability analysis. By eliminating the need for repetitive numerical evaluations of pipeline subjected to permanent ground movement, the proposed approach provides an efficient and scalable tool for pipeline reliability assessment, enabling rapid decision-making in geohazard-prone regions.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.CL
Sequential Bayesian Evaluation of Large Language Model Behavior
Saatvik Kher, Shang Wu, Rachel Longjohn et al. (+2 more)
It is increasingly important to evaluate the characteristics of systems based on large
language models (LLMs). Evaluations in this context often rely on a curated
benchmark set of input
prompts provided to the LLM, where the output for each
prompt may be assigned a binary or ordinal score and the aggregation of scores across
prompts is then used as a summary evaluation. In this paper, we develop a
Bayesian approach for quantifying the uncertainty that arises in such evaluation metrics as a result of the stochasticity of the LLM-based systems; the same
prompt may exhibit different outcomes on repeated runs. Our framework leads naturally to a sequential evaluation, in which we leverage the
Bayesian model to preferentially select which
prompts in the
benchmark to use next, enabling more cost-effective LLM evaluations. We demonstrate this approach through four case studies: pairwise LLM preferences in interactive dialogue (MT-Bench), refusal rates on
adversarial inputs (JailbreakBench),
embodied agentic
reasoning (ALFWorld), and agentic web-search Q&A (
GAIA). Our results illustrate how a
Bayesian approach can provide insightful uncertainty quantification about the behavior of blackbox LLM-based systems in a manner that is not accessible via deterministic or single-run evaluations.
Read the full abstract ▾
All authors Saatvik Kher, Shang Wu, Rachel Longjohn, Catarina Belém, Padhraic Smyth
Statistics Applications
Centralized Health and Exposomic Resource (C-HER): Analytic and AI-Ready Data for External Exposomic Research
Heidi A. Hanson, Joemy Ramsay, Josh Grant et al. (+8 more)
The Centralized Health and Exposomic Resource (C-HER) project has identified, profiled, spatially indexed, and stored over 30 external exposomic
datasets. The resulting analytic and AI-ready data (AAIRD) provides a significant opportunity to develop an integrated picture of the exposome for health research. The exposome is a conceptual framework designed to guide the study of the complex environmental and genetic factors that together shape human health. Few composite measures of the exposome exist due to the high dimensionality of exposure data,
multimodal data sources, and varying spatiotemporal scales. We develop a data engineering solution that overcomes the challenges of spatio-temporal linkage in this field. We provide examples of how environmental data can be combined to characterize a region, model air pollution, or provide indicators for cancer research. The development of AAIRD will allow future studies to use ML and deep learning methods to generate spatial and contextual exposure data for disease prediction.
Read the full abstract ▾
All authors Heidi A. Hanson, Joemy Ramsay, Josh Grant, Maggie Davis, Janet O. Agbaje, Dakotah Maguire, Jeremy Logan, Marissa Taddie, Chad Melton, Midgie MacFarland, James VanDerslice
Statistics Applications · Cross-listed from cs.LG
Applying Time Series Deep Learning Models to Forecast the Growth of Perennial Ryegrass in Ireland
Oluwadurotimi Onibonoje, Vuong M. Ngo, Andrew McCarre et al. (+3 more)
Grasslands, constituting the world's second-largest terrestrial carbon sink, play a crucial role in biodiversity and the regulation of the carbon cycle. Currently, the Irish dairy sector, a significant economic contributor, grapples with challenges related to profitability and sustainability. Presently, grass growth
forecasting relies on impractical mechanistic models. In response, we propose deep learning models tailored for univariate
datasets, presenting cost-effective alternatives. Notably, a temporal convolutional network designed for
forecasting Perennial Ryegrass growth in Cork exhibits high performance, leveraging historical grass height data with RMSE of 2.74 and MAE of 3.46. Validation across a comprehensive
dataset spanning 1,757 weeks over 34 years provides insights into optimal model configurations. This study enhances our understanding of model behavior, thereby improving reliability in grass growth
forecasting and contributing to the advancement of sustainable dairy farming practices.
Read the full abstract ▾
All authors Oluwadurotimi Onibonoje, Vuong M. Ngo, Andrew McCarre, Elodie Ruelle, Bernadette O-Briend, Mark Roantree
Machine Learning (Statistics) · Cross-listed from cs.AI
Epidemiology of Large Language Models: A Benchmark for Observational Distribution Knowledge
Drago Plecko, Patrik Okanovic, Shreyas Havaldar et al. (+2 more)
Artificial intelligence (AI) systems hold great promise for advancing various scientific disciplines, and are increasingly used in real-world applications. Despite their remarkable progress, further capabilities are expected in order to achieve more general types of intelligence. A critical distinction in this context is between factual knowledge, which can be evaluated against true or false answers (e.g., "what is the capital of England?"), and probabilistic knowledge, reflecting probabilistic properties of the real world (e.g., "what is the sex of a computer science graduate in the US?"). In this paper, our goal is to build a
benchmark for understanding the capabilities of LLMs in terms of knowledge of probability distributions describing the real world. Given that LLMs are trained on vast amounts of text, it may be plausible that they internalize aspects of these distributions. Indeed, LLMs are touted as powerful universal approximators of real-world distributions. At the same time, classical results in statistics, known as curse of dimensionality, highlight fundamental challenges in learning distributions in high dimensions, challenging the notion of universal distributional learning. In this work, we develop the first
benchmark to directly test this hypothesis, evaluating whether LLMs have access to empirical distributions describing real-world populations across domains such as economics, health, education, and social behavior. Our results demonstrate that LLMs perform poorly overall, and do not seem to internalize real-world statistics naturally. When interpreted in the context of Pearl's Causal Hierarchy (PCH), our
benchmark demonstrates that
language models do not contain knowledge on observational distributions (Layer 1 of PCH), and thus the Causal Hierarchy
Theorem implies that interventional (Layer 2) and counterfactual (Layer 3) knowledge of these models is also limited.
Read the full abstract ▾
All authors Drago Plecko, Patrik Okanovic, Shreyas Havaldar, Torsten Hoefler, Elias Bareinboim
Machine Learning (Statistics)
Precise asymptotic analysis of Sobolev training for random feature models
Katharine E Fisher, Matthew TC Li, Youssef Marzouk et al. (+1 more)
Gradient information is widely useful and available in applications, and is therefore natural to include in the training of neural networks. Yet little is known theoretically about the impact of Sobolev training – regression with both function and gradient data – on the generalization error of highly overparameterized predictive models in high dimensions. In this paper, we obtain a precise characterization of this training modality for random feature (RF) models in the limit where the number of trainable parameters, input dimensions, and training data tend proportionally to infinity. Our model for Sobolev training reflects practical implementations by sketching gradient data onto finite dimensional subspaces. By combining the replica method from statistical physics with linearizations in operator-valued free probability theory, we derive a closed-form description for the generalization errors of the trained RF models. For target functions described by single-index models, we demonstrate that supplementing function data with additional gradient data does not universally improve predictive performance. Rather, the degree of overparameterization should inform the choice of training method. More broadly, our results identify settings where models perform optimally by interpolating noisy function and gradient data.
Read the full abstract ▾
All authors Katharine E Fisher, Matthew TC Li, Youssef Marzouk, Timo Schorlepp
Statistical Methodology
New sampling approaches for Shrinkage Inverse-Wishart distribution
Yiye Jiang
In this paper, we propose new sampling approaches for the Shrinkage Inverse-Wishart (SIW) distribution, a generalized family of the Inverse-Wishart distribution originally proposed by Berger et al. (2020, Annals of Statistics). It offers a flexible prior for covariance
matrices and remains conjugate to the Gaussian likelihood, similar to the classical Inverse-Wishart. Despite these advantages, sampling from SIW remains challenging. The existing
algorithm relies on a nested Gibbs sampler, which is slow and lacks rigorous theoretical analysis of its convergence. We propose a new
algorithm based on the Sampling Importance Resampling (SIR) method, which is significantly faster and comes with theoretical guarantees on convergence rates. A known issue with SIR methods is the large discrepancy in importance weights, which occurs when the proposal distribution has thinner tails than the target. In the case of SIW, certain parameter settings can lead to such discrepancies, reducing the robustness of the output samples. To sample from such SIW distributions, we robustify the proposed
algorithm by including a clipping step to the SIR framework which transforms large importance weights. We provide theoretical results on the convergence behavior in terms of the clipping size, and discuss strategies for choosing this parameter via simulation studies. The robustified version retains the computational efficiency of the original
algorithm.
Read the full abstract ▾
Statistics Applications · Cross-listed from eess.SY
Quantifying Power Systems Resilience Using Statistical Analysis and Bayesian Learning
Apsara Adhikari, Charlotte Wertz, Anamika Dubey et al. (+2 more)
The increasing frequency and intensity of extreme weather events is significantly affecting the power grid, causing large-scale outages and impacting power system resilience. Yet limited work has been done on systematically modeling the impacts of weather parameters to quantify resilience. This study presents a framework using statistical and
Bayesian learning approaches to quantitatively model the relationship between weather parameters and power system resilience metrics. By leveraging real-world publicly available outage and weather data, we identify key weather variables of wind speed, temperature, and precipitation influencing a particular region's resilience metrics. A case study of Cook County, Illinois, and Miami-Dade County, Florida, reveals that these weather parameters are critical factors in resiliency analysis and risk assessment. Additionally, we find that these weather variables have combined effects when studied jointly compared to their effects in isolation. This framework provides valuable insights for understanding how weather events affect power distribution system performance, supporting decision-makers in developing more effective strategies for risk mitigation, resource allocation, and adaptation to changing climatic conditions.
Read the full abstract ▾
All authors Apsara Adhikari, Charlotte Wertz, Anamika Dubey, Arslan Ahmad, Ian Dobson
Machine Learning (Statistics) · Cross-listed from cs.LG
Discrete Bayesian Sample Inference for Graph Generation
Ole Petersen, Marcel Kollovieh, Marten Lienen et al. (+1 more)
Generating graph-structured data is crucial in applications such as molecular generation, knowledge graphs, and network analysis. However, their discrete, unordered nature makes them difficult for traditional generative models, leading to the rise of discrete
diffusion and flow matching models. In this work, we introduce GraphBSI, a novel one-shot graph generative model based on
Bayesian Sample Inference (BSI). Instead of evolving samples directly, GraphBSI iteratively refines a belief over graphs in the continuous space of distribution parameters, naturally handling discrete structures. Further, we state BSI as a
stochastic differential equation (SDE) and derive a noise-controlled family of SDEs that preserves the marginal distributions via an approximation of the score function. Our theoretical analysis further reveals the connection to
Bayesian Flow Networks and
Diffusion models. Finally, in our empirical evaluation, we demonstrate state-of-the-art performance on molecular and synthetic graph generation, outperforming existing one-shot graph generative models on the standard
benchmarks Moses and GuacaMol.
Read the full abstract ▾
All authors Ole Petersen, Marcel Kollovieh, Marten Lienen, Stephan Günnemann
Machine Learning (Statistics)
Unifying Information-Theoretic and Pair-Counting Clustering Similarity
Alexander J. Gates
Comparing clusterings is central to evaluating unsupervised models, yet the many existing similarity measures can produce widely divergent, sometimes contradictory, evaluations. Clustering similarity measures are typically organized into two principal families, pair-counting and information-theoretic, reflecting whether they quantify agreement through element pairs or aggregate information across full cluster contingency tables. Prior work has uncovered parallels between these families and applied empirical normalization or chance-correction schemes, but their deeper analytical connection remains only partially understood. Here, we develop an analytical framework that unifies these families through two complementary perspectives. First, both families are expressed as weighted expansions of observed versus expected co-occurrences, with pair-counting arising as a quadratic, low-order approximation and information-theoretic measures as higher-order, frequency-weighted extensions. Second, we generalize pair-counting to k-tuple agreement and show that information-theoretic measures can be viewed as systematically accumulating higher-order co-assignment structure beyond the pairwise level. We illustrate the approaches analytically for the Rand index and Mutual Information, and show how other indices in each family emerge as natural extensions. Together, these views clarify when and why the two regimes diverge, relating their sensitivities directly to weighting and approximation order, and provide a principled basis for selecting, interpreting, and extending clustering similarity measures across applications.
Read the full abstract ▾
Machine Learning (Statistics)
Scalable Single-Cell Gene Expression Generation with Latent Diffusion Models
Giovanni Palla, Sudarshan Babu, Payam Dibaeinia et al. (+5 more)
Computational modeling of single-
cell gene expression is crucial for understanding cellular processes, but generating realistic expression profiles remains a major challenge. This difficulty arises from the count nature of gene expression data and complex latent dependencies among genes. Existing generative models often impose artificial gene orderings or rely on shallow neural network architectures. We introduce a scalable latent
diffusion model for single-
cell gene expression data, which we refer to as scLDM, that respects the fundamental exchangeability property of the data. Our VAE uses fixed-size latent variables leveraging a unified Multi-head Cross-
Attention Block (MCAB) architecture, which serves dual roles: permutation-invariant pooling in the encoder and permutation-equivariant unpooling in the decoder. We enhance this framework by replacing the Gaussian prior with a latent
diffusion model using
Diffusion Transformers and linear interpolants, enabling high-quality generation with multi-conditional classifier-free guidance. We show its superior performance in a variety of experiments for both observational and perturbational single-
cell data, as well as downstream tasks like
cell-level classification.
Read the full abstract ▾
All authors Giovanni Palla, Sudarshan Babu, Payam Dibaeinia, James D. Pearce, Donghui Li, Aly A. Khan, Theofanis Karaletsos, Jakub M. Tomczak
Statistical Methodology
Constructing Large Orthogonal Minimally Aliased Response Surface Designs by Concatenating Two Definitive Screening Designs
Alan R. Vazquez, Peter Goos and Eric D. Schoen
Orthogonal minimally aliased response surface (OMARS) designs permit the study of quantitative factors at three levels using an economical number of runs. In these designs, the linear effects of the factors are neither aliased with each other nor with the quadratic effects and the two-factor interactions. Complete catalogs of OMARS designs with up to five factors have been obtained using an enumeration
algorithm. However, the
algorithm is computationally demanding for designs with many factors and runs. To overcome this issue, we propose a construction method for large OMARS designs that concatenates two definitive screening designs and improves the statistical features of its parent designs. The concatenation employs an
algorithm that minimizes the aliasing among the second-order effects using foldover techniques and column permutations for one of the parent designs. We study the properties of the new OMARS designs and compare them with alternative designs in the literature.
Read the full abstract ▾
Statistical Methodology
Detecting Conflicts in Evidence Synthesis Models Using Score Discrepancies
Fuming Yang, David J. Nott and Anne M. Presanis
Evidence synthesis models combine multiple data sources to estimate latent quantities of interest, enabling reliable inference on parameters that are difficult to measure directly. However, shared parameters across data sources can induce conflicts both among the data and with the assumed model structure. Detecting and quantifying such conflicts remains a challenge in model criticism. Here we propose a general framework for conflict detection in evidence synthesis models based on score discrepancies, extending prior-data conflict diagnostics to more general conflict checks in the latent space of hierarchical models. Simulation studies in an exchangeable model demonstrate that the proposed approach effectively detects between-data inconsistencies. Application to an influenza severity model illustrates its use, complementary to traditional deviance-based diagnostics, in complex real-world hierarchical settings. The proposed framework thus provides a flexible and broadly applicable tool for consistency assessment in
Bayesian evidence synthesis.
Read the full abstract ▾
Statistical Methodology
Adaptive Orthogonalization for Stable Estimation of the Effects of Time-Varying Treatments
Yige Li, María de los Angeles Resa and José R. Zubizarreta
Inferring the causal effects of time-varying treatments is often hindered by highly variable inverse propensity weights, particularly in settings with limited
covariate overlap. Building on the key framework of Imai and Ratkovic (2015), we establish sufficient balancing conditions for identification in longitudinal studies of treatment effects and propose a novel
estimator that directly targets features of counterfactual or potential
covariates. Instead of balancing observed
covariates, our method balances the components of
covariates that are orthogonal to their history, thereby isolating the new information at each time point. This strategy directly targets the joint distribution of potential
covariates and prioritizes features that are most relevant to the outcome. We prove that the resulting
estimator for the mean potential outcome is consistent and asymptotically normal, even in settings where standard inverse propensity weighting fails. Extensive simulations show that our
estimator attains efficiency comparable to that of g-computation while providing superior robustness to model misspecification. We apply our method to a longitudinal study of private versus public schooling in Chile, demonstrating its stability and interpretability in estimating their effects on university admission scores.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Power Constrained Nonstationary Bandits with Habituation and Recovery Dynamics
Fengxu Li, Stephanie M. Carpenter, Matthew P. Buman et al. (+1 more)
A common challenge for decision makers is selecting actions whose rewards are unknown and evolve over time based on prior policies. For instance, repeated use may reduce an action's effectiveness (habituation), while inactivity may restore it (recovery). These nonstationarities are captured by the Reducing or Gaining Unknown Efficacy (ROGUE) bandit framework, which models real-world settings such as behavioral health interventions. While existing
algorithms can compute sublinear regret policies to optimize these settings, they may not provide sufficient exploration due to overemphasis on exploitation, limiting the ability to estimate population-level effects. This is a challenge of particular interest in micro-randomized trials (MRTs) that aid researchers in developing just-in-time adaptive interventions that have population-level effects while still providing personalized recommendations to individuals. In this paper, we first develop ROGUE-TS, a Thompson Sampling
algorithm tailored to the ROGUE framework, and provide theoretical guarantees of sublinear regret. We then introduce a probability clipping procedure to balance personalization and population-level learning, with quantified trade-off that balances regret and minimum exploration probability. Validation on two MRT
datasets concerning physical activity promotion and bipolar disorder treatment shows that our methods both achieve lower regret than existing approaches and maintain high statistical power through the clipping procedure without significantly increasing regret. This enables reliable detection of treatment effects while accounting for individual behavioral dynamics. For researchers designing MRTs, our framework offers practical guidance on balancing personalization with statistical validity.
Read the full abstract ▾
All authors Fengxu Li, Stephanie M. Carpenter, Matthew P. Buman, Yonatan Mintz
Statistical Computation
Optimal transport with a density-dependent cost function
Zichu Wang and Esteban G. Tabak
A new pairwise cost function is proposed for the optimal transport barycenter problem, adopting the form of the minimal action between two points, with a Lagrangian that takes into account an underlying probability distribution. Under this notion of distance, two points can only be close if there exist paths joining them that do not traverse areas of small probability. A framework is proposed and developed for the numerical solution of the corresponding data-driven optimal transport problem. The procedure parameterizes the paths of minimal action through path dependent Chebyshev
polynomials and enforces the agreement between the paths' endpoints and the given source and target distributions through an
adversarial penalization. The methodology and its application to clustering and matching problems is illustrated through synthetic examples.
Read the full abstract ▾
Other Statistics
From Hume to Jaynes: Induction as the Logic of Plausible Reasoning
Tommaso Costa
The problem of induction has persisted since Hume exposed the logical gap between repeated observation and universal inference. Traditional attempts to resolve it have oscillated between two extremes: the probabilistic optimism of Laplace and Jeffreys, who sought to quantify belief through probability, and the critical skepticism of Popper, who replaced confirmation with falsification. Both approaches, however, assume that induction must deliver certainty or its negation. In this paper, I argue that the problem of induction dissolves when recast in terms of logical coherence (understood as internal consistency of credences under updating) rather than truth. Following E. T. Jaynes, probability is interpreted not as frequency or decision rule but as the extension of deductive logic to incomplete information. Under this interpretation, Bayes's
theorem is not an empirical statement but a consistency condition that constrains rational belief updating. Induction thus emerges as the special case of deductive
reasoning applied to uncertain premises. Falsification appears as the limiting form of
Bayesian updating when new data drive posterior plausibility toward zero, while the Bayes Factor quantifies the continuous spectrum of evidential strength. Through analytical examples, including Laplace's sunrise problem, Jeffreys's mixed prior, and confidence-based reformulations, I show that only the logic of plausible
reasoning unifies these perspectives without contradiction. Induction, properly understood, is not the leap from past to future but the discipline of maintaining coherence between evidence, belief, and information.
Read the full abstract ▾
Statistics Theory
Curvature of high-dimensional data
Jiayi Chen, Mohammad Javad Latifi Jebelli and Daniel N. Rockmore
We consider the problem of estimating
curvature where the data can be viewed as a noisy sample from an underlying
manifold. For
manifolds of dimension greater than one there are multiple definitions of local
curvature, each suggesting a different estimation process for a given data set. Recently, there has been progress in proving that estimates of “local point cloud
curvature" converge to the related smooth notion of local
curvature as the density of the point cloud approaches infinity. Herein we investigate practical limitations of such convergence
theorems and discuss the significant impact of bias in such estimates as reported in recent literature. We provide theoretical arguments for the fact that bias increases drastically in higher dimensions, so much so that in high dimensions, the probability that a naive
curvature estimate lies in a small interval near the true
curvature could be near zero. We present a probabilistic framework that enables the construction of more accurate
estimators of
curvature for arbitrary noise models. The efficacy of our technique is supported with experiments on spheres of dimension as large as twelve.
Read the full abstract ▾
Statistical Methodology
The Bias-Variance Tradeoff in Long-Term Experimentation
Daniel Ting and Kenneth Hung
As we exhaust methods that reduces variance without introducing bias, reducing variance in experiments often requires accepting some bias, using methods like winsorization or surrogate metrics. While this bias-variance tradeoff can be optimized for individual experiments, bias may accumulate over time, raising concerns for long-term optimization. We analyze whether bias is ever acceptable when it can accumulate, and show that a bias-variance tradeoff persists in long-term settings. Improving signal-to-noise remains beneficial, even if it introduces bias. This implies we should shift from thinking there is a single “correct”, unbiased metric to thinking about how to make the best estimates and decisions when better precision can be achieved at the expense of bias. Furthermore, our model adds nuance to previous findings that suggest less stringent launch criterion leads to improved gains. We show while this is beneficial when the system is far from the optimum, more stringent launch criterion is preferable as the system matures.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models
Lejs Deen Behric, Liang Zhang, Bingcong Li et al. (+1 more)
Zeroth-order or derivative-free optimization (MeZO) is an attractive strategy for finetuning large
language models (LLMs) because it eliminates the
memory overhead of backpropagation. However, it converges slowly due to the inherent curse of dimensionality when searching for descent directions in the high-dimensional parameter space of billion-scale LLMs. We propose ConMeZO, a novel zeroth-order optimizer that accelerates convergence by adaptive directional sampling. Instead of drawing the direction uniformly at random, ConMeZO restricts the sampling to a cone centered around a momentum estimate. This concentrates the search in directions where the true gradient is more likely to lie and thus reduces the effect of high dimensions. We prove that ConMeZO achieves the same worst-case convergence rate as MeZO. Empirically, when finetuning LLMs on natural
language tasks, ConMeZO is up to 2X faster than MeZO while retaining the low-
memory footprint of zeroth-order methods.
Read the full abstract ▾
All authors Lejs Deen Behric, Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil
Statistical Methodology
DANIEL: A Distributed and Scalable Approach for Global Representation Learning with EHR Applications
Zebin Wang, Ziming Gan, Weijing Tang et al. (+4 more)
Classical probabilistic graphical models face fundamental challenges in modern data environments, which are characterized by high dimensionality, source heterogeneity, and stringent data-sharing constraints. In this work, we revisit the Ising model, a well-established member of the Markov Random Field (MRF) family, and develop a distributed framework that enables scalable and
privacy-preserving representation learning from large-scale binary data with inherent low-rank structure. Our approach optimizes a non-convex surrogate loss function via bi-factored gradient descent, offering substantial computational and communication advantages over conventional convex approaches. We evaluate our
algorithm on multi-institutional electronic health record (EHR)
datasets from 58,248 patients across the University of Pittsburgh Medical Center (UPMC) and Mass General Brigham (MGB), demonstrating superior performance in global representation learning and downstream clinical tasks, including relationship detection, patient phenotyping, and patient clustering. These results highlight a broader potential for statistical inference in
federated, high-dimensional settings while addressing the practical challenges of data complexity and multi-institutional integration.
Read the full abstract ▾
All authors Zebin Wang, Ziming Gan, Weijing Tang, Zongqi Xia, Tianrun Cai, Tianxi Cai, Junwei Lu
Machine Learning (Statistics)
Optimizing Kernel Discrepancies via Subset Selection
Deyao Chen, François Clément, Carola Doerr et al. (+1 more)
Kernel discrepancies are a powerful tool for analyzing worst-case errors in quasi-Monte Carlo (QMC) methods. Building on recent advances in optimizing such discrepancy measures, we extend the subset selection problem to the setting of kernel discrepancies, selecting an m-element subset from a large population of size $n \gg m$. We introduce a novel subset selection
algorithm applicable to general kernel discrepancies to efficiently generate low-discrepancy samples from both the uniform distribution on the unit hypercube, the traditional setting of classical QMC, and from more general distributions $F$ with known density functions by employing the kernel Stein discrepancy. We also explore the relationship between the classical $L_2$
star discrepancy and its $L_\infty$ counterpart.
Read the full abstract ▾
All authors Deyao Chen, François Clément, Carola Doerr, Nathan Kirk
Statistics Applications
Extended Kalman Filtering on Stiefel Manifolds
Jordi-Lluís Figueras, Aron Persson and Lauri Viitasaari
A generalisation of the extended Kalman filter for Stiefel
manifold-valued measurements is presented. We provide simulations on the 2-sphere and the space of orthogonal 4-by-2
matrices which show significant improvement of the Extended Kalman Filter compared to only relying on raw measurements.
Read the full abstract ▾
Statistics Theory
On the Convergence of the Extended Kalman Filter on Stiefel Manifolds when Observing a Constant Particle with Measurement Errors
Jordi-Lluís Figueras, Aron Persson and Lauri Viitasaari
In this paper we first introduce the setting of filtering on Stiefel
manifolds. Then, assuming the underlying system process is constant, the convergence of the extended Kalman filter with Stiefel
manifold-valued observations is proved. This corresponds to the case where one has measurement errors that needs to be filtered. Finally, some simulations are presented for a selected few Stiefel
manifolds and the speed of convergence is studied.
Read the full abstract ▾
Statistics Theory
Spectral analysis of high-dimensional spot volatility matrix with applications
Qiang Liu, Yiming Liu, Zhi Liu et al. (+1 more)
In random
matrix theory, the spectral distribution of the covariance
matrix has been well studied under the large dimensional asymptotic regime when the dimensionality and the sample size tend to infinity at the same rate. However, most existing theories are built upon the assumption of independent and identically distributed samples, which may be violated in practice. For example, the observational data of continuous-time processes at discrete time points, namely, the high-frequency data. In this paper, we extend the classical spectral analysis for the covariance
matrix in large dimensional random
matrix to the spot
volatility matrix by using the high-frequency data. We establish the first-order limiting spectral distribution and obtain a second-order result, that is, the central limit
theorem for linear spectral statistics. Moreover, we apply the results to design some feasible tests for the spot
volatility matrix, including the identity and sphericity tests. Simulation studies justify the finite sample performance of the test statistics and verify our established theory.
Read the full abstract ▾
All authors Qiang Liu, Yiming Liu, Zhi Liu, Wang Zhou
Statistics Applications · Cross-listed from physics.soc-ph
Stochastic Redistribution of Indistinguishable Items in Shared Habitation: A Multi-Agent Simulation Framework
Syed Haseeb Shah
This paper presents a discrete-event
stochastic model for the redistribution of indistinguishable personal items, exemplified by socks, among multiple cohabitants sharing a communal laundry system. Drawing on concepts from ecological population dynamics,
diffusion processes, and
stochastic exchange theory, the model captures the probabilistic mechanisms underlying item mixing, recovery, and loss. Each cohabitant is represented as an autonomous
agent whose belongings interact through iterative cycles of collective washing, sorting, and partial correction. The system's
evolution is characterized by random mixing events, selective recollection, and attrition over time. Implemented using the SimPy discrete-event simulation framework, the model demonstrates that even minimal exchange probabilities can generate emergent asymmetries, quasi-
equilibrium distributions, and long-term disorder. The findings illustrate how
stochastic processes inherent to shared domestic systems can produce persistent imbalances, offering a quantitative perspective on an everyday social phenomenon.
Read the full abstract ▾
Statistical Methodology
Synthetic Control with Weight Uncertainty: Robust Identification and Statistical Inference
Taehyeon Koo and Zijian Guo
The synthetic control method estimates causal effects by comparing a treated unit with weighted controls matched on its pre-treatment trajectory. However, validity can be compromised when highly correlated controls leave the synthetic weights weakly determined or when treated-control relationships shift after treatment. We propose a new estimand, the weight-robust treatment effect, defined as the optimizer of a worst-case optimization problem over an uncertainty class of synthetic weights compatible with the pre-treatment fit. We establish its connection to sensitivity analysis: the uncertainty class induces an interval of plausible treatment effects, and the proposed estimand is the point in the interval closest to zero. Under the classical identification conditions, the estimand coincides with the true treatment effect. When these conditions fail, the estimand remains point identified; if the uncertainty class contains the true post-treatment weight, it provides a conservative sign-preserving bound on the effect. The
estimator of this target may have a non-normal limiting distribution, and we propose a novel perturbation-based method for constructing valid confidence intervals. Our proposal may be of independent interest for partial identification and sensitivity analysis, where
estimators defined through constrained optimization can have non-standard limiting distributions.
Read the full abstract ▾
Statistical Methodology
Bayesian copula-based spatial random effects models for inference with complex spatial data
Alan Pearse, David Gunawan and Noel Cressie
In this article, we develop fully
Bayesian, copula-based, spatial-statistical models for large, noisy, incomplete, and non-Gaussian spatial data. Our approach includes novel constructions of copulas that accommodate a spatial-random-effects structure, enabling low-rank representations and computationally efficient
Bayesian inference. The spatial copula is used in a latent process model of the
Bayesian hierarchical spatial-statistical model, and, conditional on the latent copula-based spatial process, the data model handles measurement errors and missing data. Our simulation studies show that a fully
Bayesian approach delivers accurate and fast inference for both parameter estimation and spatial-process prediction, outperforming several
benchmark methods, including fixed rank kriging (FRK). The new class of copula-based models is used to map atmospheric methane in the Bowen Basin, Queensland, Australia, from Sentinel 5P satellite data.
Read the full abstract ▾
Statistical Methodology · Cross-listed from math-ph
Analysis of dissipative dynamics on noncommutative spaces and statistical inference of continuous time network stochastic processes
Shreya Mehta
In this thesis, we analyse the generalisations of the Ornstein-Uhlenbeck (OU)
semigroup and study them in both
quantum and classical setups. In the first three chapters, we analyse the dissipative dynamics on noncommutative/
quantum spaces, in particular, the systems with multiparticle interactions associated to CCR
algebras. We provide various models where the dissipative dynamics are constructed using noncommutative Dirichlet forms. Some of our models decay to
equilibrium algebraically and the Poincare inequality does not hold. Using the classical representation of generators of nilpotent Lie
algebras, we provide the noncommutative representations of Lie
algebras in terms of creation and annihilation operators and discuss the construction of corresponding Dirichlet forms. This introduces the opportunity to explore
quantum stochastic processes related to Lie
algebras and nilpotent Lie
algebras. Additionally, these representations enable the investigation of the noncommutative analogue of hypoellipticity. In another direction, we explore the potential for introducing statistical models within a
quantum framework. In this thesis, however, we present a classical statistical model of multivariate Graph superposition of OU (Gr supOU) process which allows for long(er)
memory in the modelling of sparse graphs. We estimate these processes using generalised method of moments and show that it yields consistent
estimators. We demonstrate the asymptotic normality of the moment
estimators and validate these
estimators through a simulation study.
Read the full abstract ▾
Statistics Applications
Identification of Separable OTUs for Multinomial Classification in Compositional Data Analysis
R. Alberich, N.A. Cruz, R. Fernández et al. (+3 more)
High-throughput sequencing has transformed microbiome research, but it also produces inherently compositional data that challenge standard statistical and machine learning methods. In this work, we propose a multinomial classification framework for compositional microbiome data based on penalized log-ratio regression and pairwise separability screening. The method quantifies the discriminative ability of each OTU through the area under the receiver operating characteristic curve ($AUC$) for all pairwise log-ratios and aggregates these values into a global separability index $S_k$, yielding interpretable rankings of taxa together with confidence intervals. We illustrate the approach by reanalyzing the Baxter colorectal adenoma
dataset and comparing our results with Greenacre's ordination-based analysis using Correspondence Analysis and Canonical Correspondence Analysis. Our models consistently recover a core subset of taxa previously identified as discriminant, thereby corroborating Greenacre's main findings, while also revealing additional OTUs that become important once demographic
covariates are taken into account. In particular, adjustment for age, gender, and diabetes medication improves the precision of the separation index and highlights new, potentially relevant taxa, suggesting that part of the original signal may have been influenced by confounding. Overall, the integration of log-ratio modeling,
covariate adjustment, and uncertainty estimation provides a robust and interpretable framework for OTU selection in compositional microbiome data. The proposed method complements existing ordination-based approaches by adding a probabilistic and inferential perspective, strengthening the identification of biologically meaningful microbial signatures.
Read the full abstract ▾
All authors R. Alberich, N.A. Cruz, R. Fernández, I. García Mosquera, A. Mir, F. Rosselló