Machine Learning (Statistics)
FREESIA: Covariance-Aware Posterior Transport for Expressive and Scalable Data Assimilation
Shiwei Ni, Yangwen Zhang, Hang Qi et al. (+2 more)
Data assimilation aims to infer the state of complex dynamical systems based on observational data. However, accurate inference of the
multimodal posteriors induced by nonlinear or non-injective observation operators remains a key challenge under high-dimensional and sparse observation conditions. Ensemble filters scale to high dimensions but are confined by restrictive distributional assumptions, while training-free generative filters (e.g., EnSF, EnFF) alleviate this limitation but may introduce structural errors and hinder information propagation under sparse observations. To address these issues, we propose a training-free, asymptotically exact posterior transport method. Firstly, a covariance-aware posterior transport scheme is designed, which embeds the forecast cross-covariance into flow-based transport and accurately recovers unobserved states while preserving the non-Gaussian posterior structure. Furthermore, the method combines a tractable observation-adaptive proposal with posterior correction, ensuring accurate approximation of the nonlinear posterior distribution. Finally, we establish the corresponding posterior flow theory, from which the asymptotic exactness of the proposed method relative to finite-ensemble surrogates and the Wasserstein error bound are derived. Experiments on Double-Well, Lorenz-96, and Kolmogorov flow show that the proposed method captures complex posterior structure and remains accurate under sparse, nonlinear, and non-injective observations. In the sparse non-injective setting, it reduces RMSE by 56% relative to the best baseline.
Read the full abstract ▾
All authors Shiwei Ni, Yangwen Zhang, Hang Qi, Xiaofei Guan, Lili Ju
Statistical Methodology · Cross-listed from cs.LG
The Price of Self-Calibration: Exact Evidence Budgets and Manufactured Blind Sets in Adaptive Monitoring
Abdou-Raouf Atarmla
Self-calibrating monitors adapt their threshold online to guarantee a prescribed long-run false-alarm rate under arbitrary drift. We compute the price of that guarantee, stating every law with its exact domain of validity. First, the guarantee is an accounting identity, insensitive to what the monitor is meant to detect. Two evidence identities make the cost exact for the online
quantile tracker: a persistent step of height $\delta$ yields excess alarm mass within one alarm of $\delta/\eta$, and exactly $\delta/\eta$ pathwise when $\delta$ is a
lattice multiple of the gain $\eta$; a ramp of slope $c$ yields a stationary excess rate of exactly $c/\eta$, independent of accumulated size, up to a boundary $c=\eta(1-\alpha)$ coinciding with the alarm-rate cap. Second, the certificate's own fluctuation obeys an exact law: the windowed alarm rate has standard deviation of order $1/L$, not the binomial $1/\sqrt{L}$, since the windowed mass
telescopes to a difference of a tight internal state; the closed-form constant is validated with no fitted parameter. Detectors calibrated on the binomial scale are miscalibrated by $\sqrt{\eta\varphi(q_0)L}$, and correct calibration turns detection windows from quadratic to linear in the inverse fault speed. Third, any monitor required to tolerate a drift class $\mathcal{D}$ is blind, at any horizon and for any rule, to every fault in $\mathcal{D}-\mathcal{D}$; the proof is a deliberately elementary two-point argument and the contribution is the object it identifies: for speed-bounded classes the blind set is exactly the doubled-speed class, and the tracker absorbs a speed class fixed by its own gain, so that under a certification regime declaring absorbed drift normal, the monitor manufactures $\mathcal{D}$. An exact Gaussian projection bound, sharper than Pinsker and never vacuous, quantifies power outside it.
Read the full abstract ▾
Statistics Applications
Fear Moves Markets: Sentiment-Augmented POMP for Volatility Modeling of Bitcoin Returns
Abeyankar Giridharan, Chang Li, Suvrorup Mukherjee et al. (+1 more)
Cryptocurrency
markets exhibit extreme price swings and sentiment-driven regime shifts, which traditional
volatility models often fail to capture. To address this, we develop a partially observed Markov processes (POMP) model augmented with sentiment and heavy-tailed distributions. Specifically, we extend Breto's framework by including the Fear and Greed Index (FGI) as an exogenous regressor in the latent
volatility dynamics and replacing Gaussian measurement noise with a Student's t distribution. We fit the model via simulation-based inference using daily Bitcoin returns and FGI data from January 2020 to April 2025. Our approach significantly outperforms three
benchmark models in log-likelihood and filter stability. Moreover, our model yields interpretable parameters consistent with known features of financial
volatility. These results demonstrate that incorporating sentiment signals and heavy-tailed noise improves the modeling of
volatility and regime shifts in cryptocurrency
markets.
Read the full abstract ▾
All authors Abeyankar Giridharan, Chang Li, Suvrorup Mukherjee, Xinhe Wu
Statistics Applications
Auditing Bayesian Graph Alignment: Diagnostic Comparisons and Reference Failure
Melika Gorgi and Kourosh Mirsohi
Bayesian graph
alignment estimates correspondence probabilities, but convergence of an
alignment-score trace need not imply accurate correspondence marginals. We audit this gap on 240 new exact graph pairs from four source families, 240 larger pairs with 20-100 vertices, and a separate 60-case exact implementation check. Under an explicit edge-flip likelihood, we compare three samplers and score, marginal, indicator, categorical, and classifier-based diagnostics. Marginal disagreement improves error discrimination over score R-hat for the exact informed sampler, but its improvement for vanilla local sampling is uncertain. Assignment-based R* and short indicator panels are competitive; no diagnostic dominates across samplers and endpoints. At larger sizes, diagnostics predict subsequent marginal changes, not posterior error, and classification performance depends on the drift threshold. Disjoint-window and held-out-chain checks attenuate but preserve positive associations. Only 22 of 240 original reference sets pass an agreement screen. On forty failure-selected cases, eightfold SMC particle escalation does not resolve disagreement, whereas additional rejuvenation helps. Longer informed runs remain unstable. An elementary feasible-
alignment bound demonstrates severely unrepresentative SMC and informed-chain scores in concentrated 100-vertex cases, independently of approximate reference consensus. We also exhibit common-start chains with near-zero disagreement despite exact marginal error near .967. These results support assignment-sensitive auditing while identifying limits of finite budgets, diagnostic rankings, and reference agreement as evidence of accuracy.
Read the full abstract ▾
Statistical Methodology
Consistent Order Selection under Non-Identifiability and Dependence
Eduardo Fonseca Mendes
We provide sufficient conditions for the consistency of penalized least squares procedures that select the order (dimension) of a regression model from a sequence of nested classes, allowing for dependent, martingale-difference errors. The main contribution is to relax the classical identifiability requirement: parameters indexing classes larger than the true order need not be identified, provided the additional, excess directions admit a linear approximation to the truth in a neighbourhood of the true parameter. This relaxation lets the number of candidate models grow with the sample size, removing the usual need for a fixed upper bound. We verify the resulting high-level conditions for two classes of nonlinear regression models used in applied work: multiple-regime smooth transition regression and mixture-of-experts models with generic generalized linear model experts. Under a BIC-type penalty, the resulting order selection rule is consistent in either model class whenever the number of candidate models grows slower than the logarithm of the sample size.
Read the full abstract ▾
Statistical Computation · Cross-listed from cs.LG
Causal Inference with Unobserved Confounding: A Mixture Learning Perspective
Mansi Sood and Devavrat Shah
Unobserved confounding is a fundamental challenge in causal inference from observational data. This article develops a mixture-learning perspective, viewing latent confounders as sources of heterogeneity that induce mixture structure in observed data. Under suitable structural and identifiability assumptions, recovering the mixing distribution and component mechanisms enables estimation of interventional distributions and causal estimands. Using variants of Bernoulli mixtures as a running example, we contextualize mixture-learning techniques and their structural assumptions, and connect them to causal inference in panel-data settings, including latent factor models and synthetic interventions.We then consider high-dimensional exponential-family mixtures with dependent outcome trajectories, moving beyond counterfactual means to model counterfactual distributions. We situate this perspective relative to complementary approaches for unobserved confounding. Together, these ideas provide a bridge between mixture learning and causal inference, connecting recent advances in high-dimensional mixture learning to scalable identification and estimation of causal effects while raising new challenges for mixture learning.
Read the full abstract ▾
Machine Learning (Statistics)
Bayesian Deck-of-cards-based Ordinal Regression with Sequential Preference Elicitation
Marco Grillo and Silvano Zappalà
The Deck-of-cards-based Ordinal Regression (DOR) infers a value function from a ranking of reference alternatives in which the Decision Maker (DM) inserts blank cards between consecutive levels to express preference intensity. DOR, and its
stochastic extension (SMAA-DOR), treat these answers as hard constraints defining a set of compatible value functions. We propose B-DOR, a probabilistic reformulation of DOR in which each pair of adjacent levels yields an ordinal observation, the declared direction and the number of cards, modelled through a cumulative-link likelihood that relates the number of blank cards to the latent value difference between alternatives. Two
Bayesian inference
algorithms are proposed: BAYES-DOR samples the whole posterior distribution by
Hamiltonian Monte Carlo; FTRL-DOR tracks the maximum a posteriori estimate by constrained convex optimization. Moreover, through a multi-step elicitation process, elicitation can be spread over several short sessions reducing the cognitive burden on the DM. Both
algorithms enjoy logarithmic regret bounds for prediction that hold for any sequence of DM responses and that guide the choice of the prior hyperparameters. A Monte Carlo study over 768 configurations shows that accuracy grows with the number of sessions, that blank cards add significant information over preference directions alone, that both
algorithms maintain good performance under inconsistent answers, and that both outperform DOR and SMAA-DOR. An illustrative application to Italian regional healthcare performance demonstrates the practical applicability of the approach for building composite indicators.
Read the full abstract ▾
Statistical Methodology
Heterogeneous survivor average causal effects beyond monotonicity: Applications to a clinical trial evaluating mechanical ventilation strategies
Zihan Zhu, Guangyu Tong, Fernando Godinho Zampieri et al. (+3 more)
Clinical trials in critical care often evaluate outcomes that are truncated by death, such as time to discharge alive, for which treatment effects are not well defined among patients who would not survive under one or both treatment strategies. Principal stratification provides a natural framework for defining survivor causal effects, but existing approaches often rely on monotonicity assumptions that may be implausible in settings where treatment can affect survival in competing directions. This concern is illustrated by the ARDS Network trial of lower versus higher positive end-expiratory pressure (PEEP), in which higher PEEP may benefit some patients by improving alveolar recruitment while harming others through over-distention or hemodynamic compromise. Moreover, the largely null average findings of the original trial do not rule out the possibility of clinically meaningful subgroups that may benefit from or be harmed by higher PEEP. We propose a monotonicity-free framework for identifying conditional survivor average causal effects (CSACE) using an interpretable sensitivity parameter that characterizes latent principal-stratum membership. We then develop a flexible
Bayesian estimation strategy based on BART, a posterior-mean-based variable importance measure, and a distribution-aware conditional inference tree that uses the full posterior draws of individualized CSACEs. In simulation studies, the proposed BART-based approach improves individualized CSACE estimation and variable-importance recovery compared with linear modeling, especially under nonlinear treatment-effect heterogeneity. Applied to the ARDS PEEP trial, our method reveals clinically interpretable heterogeneity among estimated always-survivors, identifying subgroups with posterior evidence of benefit or harm that are obscured by the overall null trial results.
Read the full abstract ▾
All authors Zihan Zhu, Guangyu Tong, Fernando Godinho Zampieri, Snigdha Jain, Michael O. Harhay, Fan Li
Machine Learning (Statistics)
Conformal Robustness in Prediction-Driven Decision-Making
Lingjie Zhao, Hansheng Jiang and Wei Qi
Modern prediction-driven decision systems often rely on black-box predictors, but a point forecast alone does not provide the uncertainty scale required for robust downstream decision-making. We build a score-calibrated robustness framework that converts any fixed point predictor into a decision-relevant uncertainty representation through distribution-free conformal calibration. We use the conformal score, rather than a particular uncertainty set, as the primitive unit of robustness. The same score determines coverage-calibrated uncertainty sets for reliability-based robust optimization and normalizes target violations in a target-oriented formulation, Conformal Robust Satisficing. This formulation induces a conformal fragility measure that quantifies how rapidly performance deteriorates as the realized parameter departs from the forecast on the conformal score scale. For objective-uncertainty problems under standard convexity and duality conditions, we show that the reliability-based and target-oriented formulations parameterize the same score-calibrated robust decision frontier. This equivalence yields a data-driven mapping between reliability levels and acceptable targets and characterizes the marginal cost of robustness. Synthetic experiments validate the theoretical guarantees and illustrate the reliability-target correspondence. A real-data online-grocery case study demonstrates how the interface combines deep-learning demand forecasts with tractable inventory optimization, thereby improving reliability and reducing operational costs. Overall, our work shows that conformal scores endow fixed black-box predictors with an interpretable uncertainty scale for downstream decision-making while enabling reliability guarantees, acceptable-target selection, and fragility analysis within a unified framework.
Read the full abstract ▾
Other Statistics
See You at the Posterior Line: Learning Bayesian Modeling Through a Car Racing Game
Federica Zoe Ricci and Mine Dogucu
We present an interactive classroom activity designed to address a central challenge in teaching introductory
Bayesian statistics: how to formalize subjective knowledge and available information into prior distributions and then update them with empirical data. Role-playing as data analysts for a racing team, students evaluate candidate tires by converting qualitative engineering reports into prior distributions, collecting primary data via a virtual racing game, and using a Beta-Binomial model to inform team strategy. This discovery-based exercise allows small groups to observe directly how different prior choices and sample data jointly shape posterior inference. Student feedback ($n=32$) highlights high enjoyment, engagement and improved conceptual clarity. Open-access materials to implement the activity are provided, alongside recommendations for adapting it to other teaching contexts.
Read the full abstract ▾
Machine Learning (Statistics)
Toscani-Fourier Distance on Probability Measures: Wasserstein Control, Topological Equivalence on Model Classes, and Duality
Mehrdad Mohammadi
Comparing probability measures in machine learning trades transport geometry against computational cost: Wasserstein distances encode the geometry of $\mathbb R^d$ but require solving a transport problem, while kernel discrepancies are cheap to evaluate yet depend delicately on their test class. We study the Toscani–
Fourier family $\mathrm T_{s,p}$, the weighted $L^p$ norm of the difference of two characteristic functions, as a continuous
Fourier-side discrepancy on $\mathbb R^d$. For $1\le p<\infty$ we show that $d/p<s<1+d/p$ is exactly the window in which $\mathrm T_{s,p}$ is finite on $\mathcal P_p(\mathbb R^d)$, both endpoints already failing for a pair of Dirac measures, and we establish the metric,
embedding, and compactness structure of the resulting space, which we prove to be complete. Duality identifies $\mathrm T_{s,p}$ as an integral probability metric over a homogeneous
Fourier–Lebesgue ball, with an explicit extremizer when $1<p<\infty$. We prove the global bound $\mathrm T_{s,p}\lesssim W_p^{\,s-d/p}$, whose exponent is sharp, show that no global converse of any form can hold, and recover topological equivalence with $W_p$ on bounded-support and uniform-tail classes, together with explicit reverse moduli on bounded-support classes that improve the imported energy-kernel exponent at $p=2$. There, $\mathrm T_{s,2}$ is a constant multiple of the classical energy distance, which yields an exact finite-sample identity for the mean of the empirical discrepancy; the numerical experiments are otherwise diagnostic.
Read the full abstract ▾
Statistical Methodology
Modeling bounded well-being indices using Bayesian double generalized beta regression with spatial and temporal borrowing
Abhi Jain, Michael LaValley, Kimberly A. Dukes et al. (+1 more)
Health and well-being indices are widely used to assess population health outcomes and inform policy decisions. Individual-level assessment of well-being can be used to develop community-level indices that measure wellness for different geographical units. While many existing indices operate at coarse geographic levels such as counties or states, finer spatial resolution can offer more actionable insights. We present a novel
Bayesian double generalized beta regression framework to model a bounded individual-level well-being index (WBI) using annual survey data collected from 2021 to 2023 in Massachusetts. Although survey respondents may differ across years, responses are geotagged to ZIP Code Tabulation Areas (ZCTAs), enabling the integration of both spatial and temporal information. Our framework incorporates spatial dependencies via a graph
Laplacian matrix that encodes driving time-based ZCTA neighborhood structure, and leverages temporal borrowing by using posterior spatial effect estimates from one year to inform priors in the next. This dual-borrowing strategy within a
Bayesian double generalized beta regression framework enhances estimation precision, particularly in areas with sparse data, and improves inference for smaller geographic units. We demonstrate the utility of our method through a realistic simulation study that highlights improved estimation when borrowing spatial and temporal information. In the real data analysis, we model individual-level well-being for residents in Massachusetts and find that income, education status, and marital status are most associated with WBI. Additionally, we observe that ZCTAs in Western Massachusetts, Cape Cod, and those near Boston perform best with the highest spatial effects.
Read the full abstract ▾
All authors Abhi Jain, Michael LaValley, Kimberly A. Dukes, Shariq Mohammed
Statistical Methodology
Calibration of Time-Varying SEIR Models
Yisa Abolade
Mechanistic epidemic calibration can be sensitive to short-lived reporting anomalies that give a few observations high leverage under least squares (LSQ). We develop direct forward-solver least absolute deviations (LAD) calibration for time-varying susceptible-exposed-infectious-removed models fitted to reporting-interval incidence, with finite-horizon regularity and large-sample results. A phase-aware Monte Carlo study compares LAD and LSQ across three transmission drivers, four epidemic phases, five reporting mechanisms, and four forecast horizons. All 120,000 primary fits converged. LAD was favored in 56.7% of mean-absolute-error comparisons and 57.9% of weighted-interval-score comparisons overall; under isolated spikes, backlog release, or temporary underreporting, these proportions rose to 73.6% and 72.2%, with strong gains near the epidemic peak. Rolling-origin validation on four synthetic RAPIDD Ebola scenarios also identifies settings where LSQ is preferred, demonstrating the framework's ability to distinguish robustness gains from sustained-trend behavior. Multiple uncertainty and identifiability diagnostics, complete code and data, and an interactive application provide a reproducible computational workflow.
Read the full abstract ▾
Statistics Theory
Recovery Limits for Eigenvector Alignment in HDLSS Spiked Covariance Models
Hubeyb Gurdogan and Alex Shkolnik
In a spiked covariance model, let $\mathcal{H}$ and $\mathcal{B}$ be selected sample and population eigenvector
matrices. Their
alignment matrix $\Xi=\mathcal{H}^\top\mathcal{B}$ is a latent oracle diagnostic of direction-specific correspondence beyond subspace overlap. We study its uniform pathwise recovery in a nested high-dimensional, low-sample-size experiment with fixed sample size and growing dimension. At every fixed model, an observable spectral contrast consistently estimates the row Gram
matrix $\Xi\Xi^\top$. For two or more spikes, however, no measurable
estimator sequence achieves uniformly reliable almost-sure path recovery of $\Xi$ over a family having common spike strengths and a common signal subspace but different ordered eigenframes. For every $0<\epsilon<1/\sqrt{2}$, a finite orbit forces each
estimator to have some orbit point at which the approximate path-recovery probability is at most a variational bound $u(\epsilon)$, where $u(\epsilon)$ tends to zero as $\epsilon$ tends to zero. The worst-model exact-recovery probability over a countable orbit is zero. Thus the Gram summary is recoverable model by model, while the full
alignment matrix is not uniformly recoverable along the nested path.
Read the full abstract ▾