Machine Learning (Statistics) · Cross-listed from cs.AI
Foundation Priors
Sanjog Misra
Foundation models, and in particular large
language models, can generate highly informative responses, prompting growing interest in using these ”synthetic” outputs as data in empirical research and decision-making. This paper introduces the idea of a foundation prior, which shows that model-generated outputs are not as real observations, but draws from the foundation prior induced prior predictive distribution. As such synthetic data reflects both the model's learned patterns and the user's subjective priors, expectations, and biases. We model the subjectivity of the generative process by making explicit the dependence of synthetic outputs on the user's anticipated data distribution, the
prompt-engineering process, and the trust placed in the foundation model. We derive the foundation prior as an exponential-tilted, generalized
Bayesian update of the user's primitive prior, where a trust parameter governs the weight assigned to synthetic data. We then show how synthetic data and the associated foundation prior can be incorporated into standard statistical and econometric workflows, and discuss their use in applications such as refining complex models, informing latent constructs, guiding experimental design, and augmenting random-coefficient and partially linear specifications. By treating generative outputs as structured, explicitly subjective priors rather than as empirical observations, the framework offers a principled way to harness foundation models in empirical work while avoiding the conflation of synthetic ”facts” with real data.
Read the full abstract ▾
Machine Learning (Statistics)
Discriminative classification with generative features: bridging Naive Bayes and logistic regression
Zachary Terner, Alexander Petersen and Yuedong Wang
We introduce Smart Bayes, a new classification framework that bridges generative and discriminative modeling by integrating likelihood-ratio-based generative features into a logistic-regression-style discriminative classifier. From the generative perspective, Smart Bayes relaxes the fixed unit weights of Naive Bayes by allowing data-driven coefficients on density-ratio features. From a discriminative perspective, it constructs transformed inputs as marginal log-density ratios that explicitly quantify how much more likely each feature value is under one class than another, thereby providing predictors with stronger class separation than the raw
covariates. To support this framework, we develop a spline-based
estimator for univariate log-density ratios that is flexible, robust, and computationally efficient. Through extensive simulations and real-data studies, Smart Bayes often outperforms both logistic regression and Naive Bayes. Our results highlight the potential of hybrid approaches that exploit generative structure to enhance discriminative performance.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Bayesian dynamic scheduling of multipurpose batch processes under incomplete look-ahead information
Taicheng Zheng, Dan Li and Jie Li
Multipurpose batch processes become increasingly popular in manufacturing industries since they adapt to low-volume, high-value products and shifting demands. These processes often operate in a dynamic environment, which faces disturbances such as processing delays and demand changes. To minimise long-term cost and system nervousness (i.e., disruptive changes to schedules), schedulers must design rescheduling strategies to address such disturbances effectively. Existing methods often assume complete look-ahead information over the scheduling horizon. This assumption contrasts with realistic situations where schedulers can only access incomplete look-ahead information. Sticking with existing methods may lead to suboptimal long-term costs and high-level system nervousness. In this work we propose a
Bayesian dynamic scheduling method. Our method relies on learning a
Bayesian Network from the probability distribution of disturbances. Specifically, the
Bayesian Network represents how likely each operation will be impacted by disturbances. During the online execution, when new disturbances become observed, this method updates the posterior distribution and therefore guides the rescheduling strategy. We compare our method with the existing periodic rescheduling strategy (which generates new schedules from scratch at fixed intervals) on four
benchmark problems. Computational results show that our method achieves statistically better long-term costs and system nervousness. In the theoretical aspect, we prove that if disturbances are mutually independent, the impact-quantifying variables inherently satisfy the independence assumptions required by
Bayesian Networks. As an implication, practitioners can extend the method to other scheduling problems (such as job shop scheduling and continuous processes), given that they define the problem-specific dependencies between operations.
Read the full abstract ▾
Statistics Applications
COVID-19 Forecasting from U.S. Wastewater Surveillance Data: A Retrospective Multi-Model Study (2022-2024)
Faharudeen Alhassan, Hamed Karami, Amanda Bleichrodt et al. (+4 more)
Accurate and reliable
forecasting models are critical for guiding public health responses and policy decisions during pandemics such as COVID-19. Retrospective evaluation of model performance is essential for improving epidemic
forecasting capabilities. In this study, we used COVID-19 wastewater data from CDC's National Wastewater Surveillance System to generate sequential weekly retrospective forecasts for the United States from March 2022 through September 2024, both at the national level and for four major regions (Northeast, Midwest, South, and West). We produced 133 weekly forecasts using 11 models, including ARIMA, generalized additive models (GAM), simple linear regression (SLR), Prophet, and the n-sub-epidemic framework (top-ranked, weighted-ensemble, and unweighted-ensemble variants). Forecast performance was assessed using mean absolute error (MAE), mean squared error (MSE), weighted interval score (WIS), and 95% prediction interval coverage. The n-sub-epidemic unweighted ensembles outperformed all other models at 3-4-week horizons, particularly at the national level and in the Midwest and West. ARIMA and GAM performed best at 1-2-week horizons in most regions, whereas Prophet and SLR consistently underperformed across regions and horizons. These findings highlight the value of region-specific modeling strategies and demonstrate the utility of the n-sub-epidemic framework for real-time outbreak
forecasting using wastewater surveillance data.
Read the full abstract ▾
All authors Faharudeen Alhassan, Hamed Karami, Amanda Bleichrodt, James M. Hyman, Isaac C. H. Fung, Ruiyan Luo, Gerardo Chowell
Statistical Methodology · Cross-listed from math.DG
Covariance Estimation for Matrix-variate Data via Fixed-rank Core Covariance Geometry
Bongjung Sung
We study the geometry of the fixed-rank core covariance
manifold arising from the Kronecker-core decomposition of covariance
matrices. As shown in Hoff, McCormack, and Zhang (2023), every covariance
matrix $\Sigma$ of $p_1\times p_2$
matrix-variate data uniquely decomposes into a separable component $K$ and a core component $C$. Such a decomposition also exists for rank-$r$ $\Sigma$ if $p_1/p_2+p_2/p_1<r$, with $C$ sharing the same rank. If this core $C$ exhibits a partial-isotropy structure, then a partial-isotropy rank-$r$ core is a non-trivial convex combination of a rank-$r$ core and $I_p$ for $p:=p_1p_2$, where the weight on $I_p$ measures the deviation of $\Sigma$ from separability. This motivates studying the geometry of the space of rank-$r$ cores, $\mathcal{C}_{p_1,p_2,r}^+$. We show that $\mathcal{C}_{p_1,p_2,r}^+$ is a smooth
manifold, except for a measure-zero subset associated with canonical decomposability. When $r=p$, $\mathcal{C}_{p_1,p_2}^{++}:=\mathcal{C}_{p_1,p_2,p}^+$ is itself a smooth
manifold. The
geometric properties, including smoothness of the positive definite cone via separability and the Riemannian gradient and Hessian operator relevant to $\mathcal{C}_{p_1,p_2,r}^+$, are also derived. As an application, we propose a partial-isotropy core shrinkage
estimator for
matrix-variate data.
Read the full abstract ▾
Statistics Applications
pvEBayes: An R Package for Empirical Bayes Methods in Pharmacovigilance
Yihao Tan, Marianthi Markatou and Saptarshi Chakraborty
Monitoring the safety of medical products is a core concern of contemporary pharmacovigilance. To support drug safety assessment, Spontaneous Reporting Systems (SRS) collect reports of suspected adverse events of approved medical products offering a critical resource for identifying potential safety concerns that may not emerge during clinical trials. Modern nonparametric empirical Bayes methods are flexible statistical approaches that can accurately identify and estimate the strength of the association between an adverse event and a drug from SRS data. However, there is currently no comprehensive and easily accessible implementation of these methods. Here, we introduce the R package pvEBayes, which implements a suite of nonparametric empirical Bayes methods for pharmacovigilance, along with post-processing tools and graphical summaries for streamlining the application of these methods. Detailed examples are provided to demonstrate the application of the package through analyses of two real-world SRS
datasets curated from the publicly available FDA FAERS database.
Read the full abstract ▾
Statistics Applications
A Clinical Instrument to Measure Patient Anecdotes in Clinical Trials
Ian Miller, Ann Hyslop and Colin Decker
Clinical trials assessing neurological treatment are challenging due to the diversity of
brain function, and the difficulty in quantifying it. Traditional treatment studies in epilepsy use seizure frequency as the primary outcome measure, which may overlooking meaningful improvements in patients' quality of life. This paper introduces the Clinical Instrument for Measuring Patient Anecdotes in Clinical Trials (Clinical IMPACT), a novel tool designed to capture qualitative non-seizure improvement across neurological domains. The Clinical IMPACT incorporates open-ended inquiries that allow participants or caregivers to identify and select anecdotal evidence of their most significant treatment benefits. A blinded panel of experts ranks these anecdotes, facilitating a rigorous statistical analysis using the Wilcoxon Rank-Sum Test to detect treatment efficacy. The approach is resistant to type 1 error, yet comprehensive in its ability to capture real-world effects on quality of life. The potential of the Clinical IMPACT tool to enhance sensitivity while also providing qualitative insights that can inform patients, healthcare providers, and regulatory bodies about treatment effects makes it important to consider in any neurological trial. We describe how it can be used in epilepsy, and advocate for its inclusion as a key secondary endpoint to provide a perspective on non-seizure outcomes, which have previously been challenging to measure, let alone to interpret, even when the clinical trial is positive.
Read the full abstract ▾
Statistics Theory
Asymptotic inference in a stationary quantum time series
Michael Nussbaum and Arleta Szkoła
We consider a statistical model of a n-mode
quantum Gaussian state which is shift invariant and also gauge invariant. Such models can be considered analogs of classical Gaussian stationary time series, parametrized by their spectral density. Defining an appropriate
quantum spectral density as the parameter, we establish that the
quantum Gaussian time series model is asymptotically equivalent to a classical nonlinear regression model given as a collection of independent
geometric random variables. The asymptotic equivalence is established in the sense of the
quantum Le Cam distance between statistical models (experiments). The
geometric regression model has a further classical approximation as a certain Gaussian white noise model with a transformed
quantum spectral density as signal. In this sense, the result is a
quantum analog of the asymptotic equivalence of classical spectral density estimation and Gaussian white noise, which is known for Gaussian stationary time series. In a forthcoming version of this preprint, we will also identify a
quantum analog of the periodogram and provide optimal parametric and nonparametric estimates of the
quantum spectral density.
Read the full abstract ▾
Statistical Methodology
Correlated Confounding Variables Are Not Easily Controlled for in Large Survey Research
William H. Press
Results in epidemiology and social science often require the removal of confounding effects from measurements of the pairwise correlation of variables in survey data. This is typically accomplished by some variant of linear regression (e.g., “logistic" or “Cox proportional"). But, knowing whether all possible confounders have been identified, or are even visible (not latent), is in general impossible. Here, we exhibit two examples that frame the issue. The first example proposes a highly unlikely hypothesis on drug use, draws data from a large, respected survey, and succeeds in “proving" the implausible hypothesis, despite regressing out more than 20 confounding variables. The second constructs a “metamodel" in which a single (by hypothesis unmeasurable) latent variable affects many mutually correlated confounders. From simulations, we derive formulas for the magnitude of spurious association that persists even as increasing numbers of confounders are regressed out. The intent of these examples is for them to serve as cautionary tales.
Read the full abstract ▾
Statistical Methodology
The Dual Wavelet Spectra: An Alternative Perspective on Hurst Exponent Estimation with Application to Mammogram Classification
Raymond J. Hinton, Jr., Pepa Ramírez Cobo et al. (+1 more)
The wavelet spectra is a common starting point for estimating the Hurst exponent of a self-similar signal using wavelet-based techniques. The decay of the $\log_2$ average energy of the detail wavelet coefficients as a function of the level of signal decomposition can be used to construct
estimators for this parameter. In this paper, we expand on previous work which introduced the “dual" wavelet spectra, where decomposition levels are instead treated as a function of energy values, and propose a relationship between its slope and the Hurst exponent by inverting the standard wavelet spectra, thereby creating a new
estimator. The effectiveness of this
estimator and its sensitivity to several settings are demonstrated through a simulation study. Finally, we show how the technique performs as a feature extraction method by applying it to the task of detecting the presence of breast cancer in mammogram images. Dual spectra wavelet features had a statistically significant effect on the log-odds of Cancer.
Read the full abstract ▾
All authors Raymond J. Hinton, Jr., Pepa Ramírez Cobo, Brani Vidakovic
Machine Learning (Statistics)
An Approach to Variable Clustering: K-means in Transposed Data and its Relationship with Principal Component Analysis
Victor Saquicela, Kenneth Palacio-Baus and Mario Chifla
Principal Component Analysis (PCA) and K-means constitute fundamental techniques in multivariate analysis. Although they are frequently applied independently or sequentially to cluster observations, the relationship between them, especially when K-means is used to cluster variables rather than observations, has been scarcely explored. This study seeks to address this gap by proposing an innovative method that analyzes the relationship between clusters of variables obtained by applying K-means on transposed data and the principal components of PCA. Our approach involves applying PCA to the original data and K-means to the transposed data set, where the original variables are converted into observations. The contribution of each variable cluster to each principal component is then quantified using measures based on variable loadings. This process provides a tool to explore and understand the clustering of variables and how such clusters contribute to the principal dimensions of variation identified by PCA.
Read the full abstract ▾
Other Statistics · Cross-listed from cs.LG
Subgroup Validity in Machine Learning for Echocardiogram Data
Cynthia Feeney, Shane Williams, Benjamin S. Wessler et al. (+1 more)
Echocardiogram
datasets enable training deep learning models to automate interpretation of cardiac ultrasound, thereby expanding access to accurate readings of diagnostically-useful images. However, the gender, sex, race, and ethnicity of the patients in these
datasets are underreported and subgroup-specific predictive performance is unevaluated. These reporting deficiencies raise concerns about subgroup validity that must be studied and addressed before model deployment. In this paper, we show that current open echocardiogram
datasets are unable to assuage subgroup validity concerns. We improve sociodemographic reporting for two
datasets: TMED-2 and MIMIC-IV-ECHO. Analysis of six open
datasets reveals no consideration of gender-diverse patients and insufficient patient counts for many racial and ethnic groups. We further perform an exploratory subgroup analysis of two published aortic stenosis detection models on TMED-2. We find insufficient evidence for subgroup validity for sex, racial, and ethnic subgroups. Our findings highlight that more data for underrepresented subgroups, improved demographic reporting, and subgroup-focused analyses are needed to prove subgroup validity in future work.
Read the full abstract ▾
All authors Cynthia Feeney, Shane Williams, Benjamin S. Wessler, Michael C. Hughes
Machine Learning (Statistics)
Thompson Sampling for Multi-Objective Linear Contextual Bandit
Somangchan Park, Heesang Ann and Min-hwan Oh
We study the multi-objective linear contextual bandit problem, where multiple possible conflicting objectives must be optimized simultaneously. We propose MOL-TS, the first Thompson Sampling
algorithm with Pareto regret guarantees for this problem. Unlike standard approaches that compute an empirical Pareto front each round, MOL-TS samples parameters across objectives and efficiently selects an arm from a novel effective Pareto front, which accounts for repeated selections over time. Our analysis shows that MOL-TS achieves a worst-case Pareto regret bound of $\widetilde{O}(d^{3/2}\sqrt{T})$, where $d$ is the dimension of the feature vectors, $T$ is the total number of rounds, matching the best known order for randomized linear bandit
algorithms for single objective. Empirical results confirm the benefits of our proposed approach, demonstrating improved regret minimization and strong multi-objective performance.
Read the full abstract ▾
Machine Learning (Statistics)
Outcome-Aware Spectral Feature Learning for Instrumental Variable Regression
Dimitri Meunier, Jakub Wornbard, Vladimir R. Kostic et al. (+5 more)
We address the problem of causal effect estimation in the presence of hidden confounders using nonparametric instrumental variable (IV) regression. An established approach is to use
estimators based on learned spectral features, that is, features spanning the top singular subspaces of the operator linking treatments to instruments. While powerful, such features are agnostic to the outcome variable. Consequently, the method can fail when the true causal function is poorly represented by these dominant singular functions. To mitigate, we introduce Augmented Spectral Feature Learning, a framework that makes the feature learning process outcome-aware. Our method learns features by minimizing a novel contrastive loss derived from an augmented operator that incorporates information from the outcome. By learning these task-specific features, our approach remains effective even under spectral misalignment. We provide a theoretical analysis of this framework and validate our approach on challenging
benchmarks.
Read the full abstract ▾
All authors Dimitri Meunier, Jakub Wornbard, Vladimir R. Kostic, Antoine Moulin, Alek Fröhlich, Karim Lounici, Massimiliano Pontil, Arthur Gretton
Statistical Methodology
An Imbalance-Robust Evaluation Framework for Extreme Risk Forecasts
Sotirios D. Nikolopoulos
Evaluating rare-event forecasts is challenging because standard metrics collapse as event prevalence declines. Measures such as F1-score, AUPRC, MCC, and accuracy induce degenerate thresholds – converging to zero or one – and their values become dominated by class imbalance rather than tail discrimination. We develop a family of rare-event-stable (RES) metrics whose optimal thresholds remain strictly interior as the event probability approaches zero, ensuring coherent decision rules under extreme rarity. Simulations spanning event probabilities from 0.01 down to one in a million show that RES metrics maintain stable thresholds, consistent model rankings, and near-complete prevalence invariance, whereas traditional metrics exhibit statistically significant threshold drift and structural collapse. A credit-default application confirms these results: RES metrics yield interpretable probability-of-default cutoffs (4-9%) and remain robust under subsampling, while classical metrics fail operationally. The RES framework provides a principled, prevalence-invariant basis for evaluating extreme-risk forecasts.
Read the full abstract ▾
Statistical Methodology
Group Competition Test: A P-value-free False Discovery Rate Control Framework for Grouped Hypotheses
Mingzhou Deng and Yan Fu
Grouped multiple testing arises in many high-dimensional applications, where hypotheses may differ across groups and valid p-values are unavailable. Applying competition-based procedures separately within groups can inflate the global false discovery rate (FDR), while conservative corrections may substantially reduce power. We extend the competition test framework to the grouped setting and propose two procedures. The group competition (GC) test provides finite-sample FDR bounds under arbitrary inter-group dependence. For homogeneous groups, the required correction has logarithmic order in the number of groups, and this order is unavoidable for the GC test. The consistent group competition (CGC) filter instead couples group-specific thresholds through a single global normalization. Under the stated basic-independence conditions, it controls the FDR in finite samples without a correction that grows with the number of groups; under arbitrary inter-group dependence, the same guarantee is retained with a per-group correction that depends on the
symmetry parameter of each group. We further develop a parameterized-threshold analysis to establish asymptotic guarantees and to characterize when the per-group correction can be dropped, so that a simple filter still controls the FDR approximately. Simulation studies show that the GC test and the CGC filter can improve power over pooled competition testing in heterogeneous settings, whereas the pooled test may remain more powerful under homogeneity. A peptide-spectrum matching
benchmark involving post-translational modifications illustrates the practical relevance of the proposed methods.
Read the full abstract ▾
Statistical Methodology
A Scalable Variational Bayes Approach for Fitting Non-Conjugate Spatial Generalized Linear Mixed Models via Basis Expansions
Jin Hyung Lee and Ben Seiyon Lee
Large spatial
datasets with non-Gaussian responses are increasingly common in environmental monitoring, ecology, and remote sensing, yet scalable
Bayesian inference for such data remains challenging. Markov chain Monte Carlo methods are often prohibitive for large
datasets, and existing variational Bayes methods rely on conjugacy or strong approximations that limit their applicability and can underestimate posterior variances. A scalable variational framework that incorporates semi-implicit variational inference (SIVI) with basis representations of spatial generalized linear mixed models, which may not have conjugacy, is proposed. The proposed framework accommodates gamma, negative binomial, Poisson, Bernoulli, and Gaussian responses on continuous spatial domains. Across 20 simulation scenarios with 50,000 locations, SIVI achieves predictive accuracy and posterior distributions comparable to Metropolis–Hastings and
Hamiltonian Monte Carlo while providing notable computational speedups. Applications to remotely-sensed land surface temperature and blue jay abundance further demonstrate the utility of the approach for large non-Gaussian spatial
datasets.
Read the full abstract ▾
Statistical Computation · Cross-listed from cs.AI
ARCADIA: Scalable Causal Discovery for Corporate Bankruptcy Analysis Using Agentic AI
Fabrizio Maturo, Donato Riccio, Andrea Mazzitelli et al. (+3 more)
This paper introduces ARCADIA, an agentic AI framework for causal discovery that integrates large-
language-model
reasoning with statistical diagnostics to construct valid, temporally coherent causal structures. Unlike traditional
algorithms, ARCADIA iteratively refines candidate DAGs through constraint-guided prompting and causal-validity feedback, leading to stable and interpretable models for real-world high-stakes domains. Experiments on corporate bankruptcy data show that ARCADIA produces more reliable causal graphs than NOTEARS, GOLEM, and DirectLiNGAM while offering a fully explainable, intervention-ready pipeline. The framework advances AI by demonstrating how agentic LLMs can participate in autonomous scientific modeling and structured causal inference.
Read the full abstract ▾
All authors Fabrizio Maturo, Donato Riccio, Andrea Mazzitelli, Giuseppe Bifulco, Francesco Paolone, Iulia Brezeanu
Statistics Theory
Infinitely divisible privacy and beyond I: resolution of the $s^2=2k$ conjecture
Aaradhya Pandey, Arian Maleki and Sanjeev Kulkarni
Differential
privacy is increasingly formalized through the lens of hypothesis testing via the robust and interpretable $f$-DP framework, where
privacy guarantees are encoded by a baseline Blackwell trade-off function $f_{\infty} = T(P_{\infty}, Q_{\infty})$ involving a pair of distributions $(P_{\infty}, Q_{\infty})$. The problem of choosing the right
privacy metric in practice leads to a central question: what is a statistically appropriate baseline $f_{\infty}$ given some prior modeling assumptions? The special case of Gaussian differential
privacy (GDP) showed that, under compositions of nearly perfect mechanisms, these trade-off functions exhibit a central limit behavior with a Gaussian limit experiment. Inspired by Le Cam's theory of limits of statistical experiments, we answer this question in full generality in an infinitely divisible setting. We show that suitable composition experiments $(P_n^{\otimes n}, Q_n^{\otimes n})$ converge to a binary limit experiment $(P_{\infty}, Q_{\infty})$ whose log-likelihood ratio $L = \log(dQ_{\infty} / dP_{\infty})$ is infinitely divisible under $P_{\infty}$. Thus any limiting trade-off function $f_{\infty}$ is determined by an infinitely divisible law $P_{\infty}$, characterized by its Levy–Khintchine triplet, and its Esscher tilt defined by $dQ_{\infty}(x) = e^{x} dP_{\infty}(x)$. This characterizes all limiting baseline trade-off functions $f_{\infty}$ arising from compositions of nearly perfect differentially private mechanisms. Our framework recovers GDP as the purely Gaussian case and yields explicit non-Gaussian limits, including Poisson examples. It also positively resolves the empirical $s^2 = 2k$ phenomenon observed in the GDP paper and provides an optimal mechanism for count statistics achieving asymmetric Poisson differential
privacy.
Read the full abstract ▾