Statistics Applications
Diabetes Lifestyle Medicine Treatment Assistance Using Reinforcement Learning
Yuhan Tang
Type 2 diabetes prevention and treatment can benefit from personalized lifestyle prescriptions. However, the delivery of personalized lifestyle medicine prescriptions is limited by the shortage of trained professionals and the variability in physicians' expertise. We propose an offline contextual bandit approach that learns individualized lifestyle prescriptions from the aggregated NHANES profiles of 119,555 participants by minimizing the Magni glucose risk-reward function. The model encodes patient status and generates lifestyle medicine prescriptions, which are trained using a mixed-action Soft Actor-Critic
algorithm. The task is treated as a single-step contextual bandit. The model is validated against lifestyle medicine prescriptions issued by three certified physicians from Xiangya Hospital. These results demonstrate that offline mixed-action SAC can generate risk-aware lifestyle medicine prescriptions from cross-sectional NHANES data, warranting prospective clinical validation.
Read the full abstract ▾
Machine Learning (Statistics)
Learning Time-Varying Graphs from Incomplete Graph Signals
Chuansen Peng and Xiaojing Shen
This paper tackles the challenging problem of jointly inferring time-varying network
topologies and imputing missing data from partially observed graph signals. We propose a unified non-convex optimization framework to simultaneously recover a sequence of graph
Laplacian matrices while reconstructing the unobserved signal entries. Unlike conventional decoupled methods, our integrated approach facilitates a bidirectional flow of information between the graph and signal domains, yielding superior robustness, particularly in high missing-data regimes. To capture realistic network dynamics, we introduce a fused-lasso type regularizer on the sequence of
Laplacians. This penalty promotes temporal smoothness by penalizing large successive changes, thereby preventing spurious variations induced by noise while still permitting gradual topological
evolution. For solving the joint optimization problem, we develop an efficient Proximal Alternating Direction Method of Multipliers (PADMM)
algorithm, which leverages the problem's structure to yield closed-form solutions for both the graph and signal subproblems. This design ensures scalability to large-scale networks and long time horizons. On the theoretical front, despite the inherent non-convexity, we establish a convergence guarantee, proving that the proposed PADMM scheme converges to a stationary point. Furthermore, we derive non-asymptotic statistical guarantees, providing high-probability error bounds for the graph
estimator as a function of sample size, signal smoothness, and the intrinsic temporal variability of the graph. Extensive numerical experiments validate the approach, demonstrating that it significantly outperforms state-of-the-art baselines in both convergence speed and the joint accuracy of graph learning and signal recovery.
Read the full abstract ▾
Statistical Methodology
Functional principal component analysis for functional data with detection limits
Haiyan Liu and Jeanine Houwing-Duistermaat
When measurements fall below or above a detection threshold, the resulting data are missing not at random (MNAR), posing challenges for statistical analysis. For example, in longitudinal biomarker studies, observations may be subject to detection limits. Functional principal component analysis (FPCA) is commonly used method for dimension reduction of dense and sparse data measured along a continuum, but standard approaches typically ignore MNAR mechanisms by imputing detection limit values, leading to biased estimates of principal components and scores. Building on recent work by Liu and Houwing-Duistermaat (2022, 2023), who proposed
estimators for the mean and covariance functions under detection limits, we extend FPCA to accommodate functional data affected by such limits. We derive the asymptotic properties of the resulting
estimators and assess their performance through simulations, comparing them to standard methods. Finally, we illustrate our approach using longitudinal biomarker data subject to detection limits. Our method yields more accurate estimates of functional principal components and scores, enhancing the reliability of functional data analysis in the presence of detection limits.
Read the full abstract ▾
Machine Learning (Statistics)
When to Transfer: Adaptive Source Selection for Positive Transfer in Linear Models
Hamza Cherkaoui, Hélène Halconruy and Yohan Petetin
In many business settings, task-specific labeled data are scarce or costly to obtain, limiting supervised learning on a target task. A classical response is transfer learning (TL). Many TL works study how to transfer information from related sources. We study, for linear regression and classification, when to transfer via sample sharing: in a multi-source setting, we greedily decide from which sources and how many samples to incorporate into the target
dataset. Our method uses an accept/reject rule based on a data-dependent estimate of the transfer gain, i.e the marginal decrease in target predictive error, computed conditionally on the observed target samples. We analyze our approach and show that how the derived statistical test enforces positive transfer with high probability. Under additional standard conditions, we also study the transfer gain itself and characterize when transfer is beneficial. Experiments on synthetic and real data show consistent gains over classical and recent strong baselines while avoiding negative transfer.
Read the full abstract ▾
Statistical Methodology
Causal Variance Decompositions for Measuring Health Inequalities
Lin Yu, Zhihui Liu, Kathy Han et al. (+1 more)
Recent causal inference literature has introduced causal effect decompositions to quantify sources of observed inequalities or disparities in outcomes, but these approaches are typically limited to pairwise comparisons. In healthcare delivery settings, both the exposure of interest-hospital or healthcare unit-and sociodemographic group membership may be polytomous, making pairwise contrasts inadequate. We therefore take the observed variance in care delivery outcomes as the quantity of interest and develop a new causal variance decomposition framework for this setting. The proposed framework attributes the observed variation to eight components, including novel terms characterizing modification of hospital effects by sociodemographic group membership, hospital access or selection, and the correlation between these two sources of heterogeneity. We discuss the causal interpretation of these components, propose both parametric and nonparametric model-based
estimators, and study their performance through simulation. Finally, we illustrate the method using data from the SEER program in an application to cervical cancer care delivery.
Read the full abstract ▾
All authors Lin Yu, Zhihui Liu, Kathy Han, Olli Saarela
Machine Learning (Statistics) · Cross-listed from cs.LG
Differentially Private Linear Regression and Synthetic Data Generation with Statistical Guarantees
Shurong Lin, Aleksandra Slavković and Deekshith Reddy Bhoomireddy
In the social sciences, small- to medium-scale
datasets are common, and linear regression is canonical. In
privacy-aware settings, much work has focused on differentially private (DP) linear regression, but mostly on point estimation with limited
attention to uncertainty quantification. Meanwhile, synthetic data generation (SDG) is increasingly important for reproducibility studies, yet current DP linear regression methods do not readily support it. Mainstream DP-SDG approaches either are tailored to discrete or discretized data, making them less suitable for analyses involving continuous variables, or rely on deep learning models that require large
datasets, limiting their use for the smaller-scale data typical in social science. We propose a method for linear regression with valid inference under Gaussian DP. It includes a bias-corrected
estimator with asymptotic confidence intervals (CIs) and a general SDG procedure such that the corresponding regression on the synthetic data matches our DP linear regression procedure. Our approach is effective in small- to moderate-dimensional settings. Experiments show that our method (1) improves accuracy over existing methods for DP linear regression, (2) provides valid CIs, and (3) produces more reliable synthetic data for downstream statistical and machine learning tasks than current DP synthesizers.
Read the full abstract ▾
Statistics Theory
Low-Rank Tensor Estimation from Nonlinear Observations: A Unified Framework
Junren Chen, Lijun Ding, Dong Xia et al. (+1 more)
We consider the estimation of a $d_1\times d_2\times d_3$ tensor $X^\star$ of Tucker rank $(r_1,r_2,r_3)$ from the nonlinear observations $\{y_i=f_i(\langle A_i,X^\star\rangle)\}_{i=1}^n$. We develop a unified approach that first constructs a gradient map from the data and then establishes the tensor restricted approximate invertibility condition (T-RAIC), a condition that quantifies how well the gradient map aligns with the ideal descent step under a low-rank tensor dual norm. We show that T-RAIC yields local linear convergence guarantees for a Riemannian gradient descent (RGD)
algorithm, which may incorporate a normalization step if $\|X^\star\|_{\rm F}$ is known a priori. Under $O(r_1r_2r_3+\sum_{1\le i\le 3}r_id_i)$ Gaussian measurements, we establish T-RAICs for single-index models, logistic regression, phase
retrieval, ReLU regression, and one-bit compressed sensing. The RAICs imply that RGD locally converges to $X^\star$ exactly in phase
retrieval and ReLU regression, and up to near-optimal estimation errors in the remaining models. We further show that, in all these models except for phase
retrieval, a simple spectral initialization yields the desired initialization from $O(d^{3/2})$ measurements under $d_1=d_2=d_3=d$ (ignoring dependence on the Tucker rank and condition number of $X^\star$). This is also the best known sample complexity for
polynomial-time and end-to-end
algorithms in tensor linear regression and tensor completion. Numerical simulations are provided to corroborate our theoretical findings.
Read the full abstract ▾
All authors Junren Chen, Lijun Ding, Dong Xia, Ming Yuan
Machine Learning (Statistics)
Prediction-Augmented Trees for Reliable Statistical Inference
Vikram Kher, Argyris Oikonomou and Manolis Zampetakis
The remarkable success of machine learning (ML) in predictive tasks has led scientists to incorporate ML predictions as a core component of the scientific discovery pipeline. This was exemplified by the landmark achievement of AlphaFold (Jumper et al. (2021)). In this paper, we study how ML predictions can be safely used in statistical analysis of data towards scientific discovery. In particular, we follow the framework introduced by Angelopoulos et al. (2023). In this framework, we assume access to a small set of $n$ gold-standard labeled samples, a much larger set of $N$ unlabeled samples, and a ML model that can be used to impute the labels of the unlabeled data points. We introduce two new learning-augmented
estimators: (1) Prediction-Augmented Residual Tree (PART), and (2) Prediction-Augmented Quadrature (PAQ). Both
estimators have significant advantages over existing
estimators like PPI and PPI++ introduced by Angelopoulos et al. (2023) and Angelopoulos et al. (2024), respectively. PART is a decision-tree based
estimator built using a greedy criterion. We first characterize PART's asymptotic distribution and demonstrate how to construct valid confidence intervals. Then we show that PART outperforms existing methods in real-world
datasets from ecology, astronomy, and census reports, among other domains. This leads to
estimators with higher confidence, which is the result of using both the gold-standard samples and the machine learning predictions. Finally, we provide a formal proof of the advantage of PART by exploring PAQ, an estimation that arises when considering the limit of PART when the depth its tree grows to infinity. Under appropriate assumptions in the input data we show that the variance of PAQ shrinks at rate of $O(N^{-1} + n^{-4})$, improving significantly on the $O(N^{-1}+n^{-1})$ rate of existing methods.
Read the full abstract ▾
Statistics Theory
Filtering Problem for Functionals of Stationary Processes with Missing Observations
Mykhailo Moklyachuk and Maria Sidei
The problem of the mean-square optimal linear estimation of the functional $A\xi=\ \int\limits_{R^s}a(t)\xi(-t)dt,$ which depends on the unknown values of
stochastic stationary process $\xi(t)$ from observations of the process $\xi(t)+\eta(t)$ at points $t\in\mathbb{R} ^{-} \backslash S $, $S=\bigcup\limits_{l=1}^{s}[-M_{l}-N_{l}, \, \ldots, \, -M_{l} ],$ $R^s=[0,\infty) \backslash S^{+},$ $S^{+}=\bigcup\limits_{l=1}^{s}[ M_{l}, \, \ldots, \, M_{l}+N_{l}]$ is considered. Formulas for calculating the mean-square error and the spectral characteristic of the optimal linear estimate of the functional are proposed under the condition of spectral certainty, where spectral densities of the processes $\xi(t)$ and $\eta(t)$ are exactly known. The minimax (robust) method of estimation is applied in the case where spectral densities are not known exactly, but sets of admissible spectral densities are given. Formulas that determine the least favorable spectral densities and the minimax spectral characteristics are proposed for some special sets of admissible spectral densities.
Read the full abstract ▾
Statistics Theory
On Minimax Estimation Problems for Periodically Correlated Stochastic Processes
Iryna Dubovets'ka and Mykhailo Moklyachuk
The aim of this article is to overview the problem of mean square optimal estimation of linear functionals which depend on unknown values of periodically correlated
stochastic process. Estimates are based on observations of this process and noise. These problems are investigated under conditions of spectral certainty and spectral uncertainty. Formulas for calculating the main characteristics (spectral characteristic, mean square error) of the optimal linear estimates of the functionals are proposed. The least favorable spectral densities and the minimax-robust spectral characteristics of optimal estimates of the functionals are presented for given sets of admissible spectral densities.
Read the full abstract ▾
Statistics Theory
Robust extrapolation problem for stochastic sequences with stationary increments
Maksym Luz and Mykhailo Moklyachuk
The problem of optimal estimation of functionals $A\xi =\sum\nolimits_{k=0}^{\infty }{}a(k)\xi (k)$ and ${{A}_{N}}\xi =\sum\nolimits_{k=0}^{N}{}a(k)\xi (k)$ which depend on the unknown values of
stochastic sequence $\xi (k)$ with stationary $n$th increments is considered. Estimates are based on observations of the sequence $\xi (m)$ at points of time $m=-1,-2,\ldots$. Formulas for calculating the value of the mean square error and the spectral characteristic of the optimal linear estimates of the functionals are derived in the case where spectral density of the sequence is exactly known. Formulas that determine the least favorable spectral densities and minimax (robust) spectral characteristic of the optimal linear estimates of the functionals are proposed in the case where the spectral density of the sequence is not known but a set of admissible spectral densities is given.
Read the full abstract ▾
Statistics Theory
Batch learning equals online learning in Bayesian supervised learning
Hông Vân Lê
In this paper we study
Bayesian supervised learning models proposed by Lê in \cite{Le2025}. Using functoriality of probabilistic morphisms, we prove that sequential and batch
Bayesian inversions coincide in supervised learning models with conditionally independent (possibly non-i.i.d.) data \cite{Le2025}. This equivalence holds without domination or discreteness assumptions on sampling operators. We derive a recursive formula for posterior predictive distributions, which reduces to the Kalman filter in Gaussian process regression. For Souslin label spaces $\mathcal{Y}$ and arbitrary input sets $\mathcal{X}$, we characterize probability measures on $\mathcal{P}(\mathcal{Y})^{\mathcal{X}}$ via projective systems, generalizing Orbanz \cite{Orbanz2011}. We revisit MacEachern's Dependent Dirichlet Processes (DDP) \cite{MacEachern2000} using copula-based constructions \cite{BJQ2012} and show how to compute posterior predictive distributions in universal
Bayesian supervised models with DDP priors.
Read the full abstract ▾
Statistical Computation · Cross-listed from econ.EM
Equilibrium-Constrained Estimation of Recursive Logit Choice Models
Hung Tran, Tien Mai and Minh Hoang Ha
The recursive logit (RL) model provides a flexible framework for modeling sequential decision-making in transportation and choice networks, with important applications in route choice analysis, multiple discrete choice problems, and activity-based travel demand modeling. Despite its versatility, estimation of the RL model typically relies on nested fixed-point (NFXP)
algorithms that are computationally expensive and prone to numerical instability. We propose a new approach that reformulates the maximum likelihood estimation problem as an optimization problem with
equilibrium constraints, where both the structural parameters and the value functions are treated as decision variables. We further show that this formulation can be equivalently transformed into a conic optimization problem with exponential cones, enabling efficient solution using modern conic solvers such as MOSEK. Experiments on synthetic and real-world
datasets demonstrate that our convex reformulation achieves accuracy comparable to traditional methods while offering significant improvements in computational stability and efficiency, thereby providing a practical and scalable alternative for recursive logit model estimation.
Read the full abstract ▾
Statistical Methodology
Causal inference for calibrated scaling interventions on time-to-event processes
Helene Charlotte Wiese Rytgaard and Mark van der Laan
This work develops a flexible inferential framework for nonparametric causal inference in time-to-event settings, based on
stochastic interventions defined through multiplicative scaling of the intensity governing an intermediate event process. These interventions induce a family of estimands indexed by a scalar parameter {\alpha}, representing effects of modifying event rates while preserving the temporal and
covariate-dependent structure of the observed data generating mechanism. To enhance interpretability, we introduce calibrated interventions, where {\alpha} is chosen to achieve a pre-specified goal, such as a desired level of cumulative risk of the intermediate event, and define corresponding composite target parameters capturing the downstream effects on the outcome process. This yields clinically meaningful contrasts while avoiding unrealistic deterministic intervention regimes. Under a nonparametric model, we derive efficient influence curves for {\alpha}-indexed, calibrated, and composite target parameters and establish their double robustness properties. We further sketch a targeted maximum likelihood estimation (TMLE) strategy that accommodates flexible, machine learning based nuisance estimation. The proposed framework applies broadly to (causal) questions involving time-to-event treatments or mediators and is illustrated through different examples event-history settings. A simulation study demonstrates finite-sample inferential properties, and highlights the implications of practical positivity violations when interventions extend beyond observed data support.
Read the full abstract ▾
Statistics Theory
On Robust Hypothesis Testing with respect to the Hellinger Distance
Eeshan Modak, Sivaraman Balakrishnan and Ananda Theertha Suresh
We study a variant of the simple hypothesis testing problem where observed samples do not necessarily come from either of the specified distributions, but rather from a close variant of them. In this setting, we require a test that is robust to misspecification and identifies which distribution is closer in Hellinger distance. If the underlying distribution is nearly equidistant from both hypotheses, the problem becomes intractable. Our main result is a lower bound on the slack factor, which quantifies how much closer the underlying distribution must be to one hypothesis relative to the other for any test to remain robust. We also demonstrate the implications of this result for testing with respect to symmetric chi-squared distance. Finally, we study an alternative way to specify robustness, where each hypothesis is a Hellinger ball around a fixed distribution. We provide and analyze a test for this composite hypothesis testing problem.
Read the full abstract ▾
Machine Learning (Statistics)
Kernel-Based Nonparametric Tests For Shape Constraints
Rohan Sen
We propose a kernel-based nonparametric framework for mean-variance optimization that enables inference on economically motivated shape constraints in finance, including positivity, monotonicity, and convexity. Many central hypotheses in financial econometrics are naturally expressed as shape relations on latent functions (e.g., term premia, CAPM relations, and the pricing kernel), yet enforcing such constraints during estimation can mask economically meaningful violations; our approach therefore separates learning from validation by first estimating an unconstrained solution and then testing shape properties. We establish statistical properties of the regularized sample
estimator and derive rigorous guarantees, including asymptotic consistency, a functional central limit
theorem, and a finite-sample deviation bound achieving the Monte Carlo rate up to a regularization term. Building on these results, we construct a joint Wald-type statistic to test shape constraints on finite grids. An efficient
algorithm based on a pivoted Cholesky factorization yields scalability to large
datasets. Numerical studies, including an options-based asset-pricing application, illustrate the usefulness of the proposed method for evaluating monotonicity and convexity restrictions.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.AI
Surrogate Modeling and Explainable Artificial Intelligence for Complex Systems: A Workflow for Automated Simulation Exploration
Paul Saves, Pramudita Satria Palar, Muhammad Daffa Robani et al. (+7 more)
Complex systems are increasingly explored through simulation-driven workflows that combine physics-based and empirical models with optimization and analytics. Despite their power, these workflows face two central obstacles: high computational cost from repeated expensive simulator runs, and limited transparency when decisions rely on opaque black-box models. We propose a workflow that addresses both challenges by training lightweight emulators on compact designs of experiments that (i) provide fast, low-latency approximations of expensive simulators, (ii) enable rigorous uncertainty quantification, and (iii) support global and local eXplainable Artificial Intelligence (XAI) analyses. This work formalizes a methodological bridge accommodating continuous, categorical, and integer variables across diverse domains, from engineering design to socio-environmental
agent-based models. To validate surrogate explanations and examine how faithfully they reflect simulator behavior, we introduce a comparative methodology incorporating active validation filters, including explanation consistency and aleatoric uncertainty quantification. Furthermore, we provide practical guidelines for coupling experimental designs with emulator training to select architectures that balance computational efficiency with interpretability. We demonstrate the framework across two contrasting paradigms: a deterministic multidisciplinary design simulation of a hybrid-electric aircraft and a
stochastic agent-based simulation of urban segregation. Results show that coupling surrogates with XAI enables large-scale exploration in seconds, reveals critical nonlinear interactions, and highlights potential design and policy levers while establishing consistent explanations across architectures.
Read the full abstract ▾
All authors Paul Saves, Pramudita Satria Palar, Muhammad Daffa Robani, Nicolas Verstaevel, Moncef Garouani, Jordan Levy, Julien Aligon, Benoit Gaudou, Koji Shimoyama, Joseph Morlier
Statistics Applications
Bayesian reliability acceptance sampling plans for competing risks data under interval censoring
Biswabrata Pradhan and Rathin Das
We obtain a reliability acceptance sampling plan for independent competing risk data under interval censoring schemes using the
Bayesian approach. At first, the
Bayesian reliability acceptance sampling plan is obtained where the decision criteria of accepting a lot is pre-fixed. For large samples, computing Bayes risk is computationally intensive. Therefore, an approximate Bayes risk is obtained using the asymptotic properties of the maximum likelihood
estimators. Lastly, the
Bayesian reliability acceptance sampling plan is obtained, where the decision function is arbitrary. The manufacturer can derive an optimal decision function by minimizing the Bayes risk among all decision functions. This optimal decision function is known as Bayes decision function. The optimal sampling plan is obtained by minimizing the Bayes risk. The
algorithms are provided for the computation of optimum
Bayesian reliability acceptance sampling plan. Numerical results are provided and comparisons between the
Bayesian reliability acceptance sampling plans are carried out.
Read the full abstract ▾
Machine Learning (Statistics)
Local regression on path spaces with signature metrics
Christian Bayer, Davit Gogolashvili and Luca Pelizzari
We study nonparametric regression and classification for path-valued data. We introduce a functional Nadaraya-Watson
estimator that combines the signature transform from rough path theory with local kernel regression. The signature transform provides a principled way to encode sequential data through iterated integrals, enabling direct comparison of paths in a natural metric space. Our approach leverages signature-induced distances within the classical kernel regression framework, achieving computational efficiency while avoiding the scalability bottlenecks of large-scale kernel
matrix operations. We establish finite-sample convergence bounds demonstrating favorable statistical properties of signature-based distances compared to traditional metrics in infinite-dimensional settings. We propose robust signature variants that provide stability against outliers, enhancing practical performance. Applications to both synthetic and real-world data - including
stochastic differential equation learning and time series classification - demonstrate competitive accuracy while offering significant computational advantages over existing methods.
Read the full abstract ▾
Statistical Methodology
Correlation of divergency: c_delta. Being different in a similar way or not
Johan F. Hoorn and Johnny K. W. Ho
Matched sets may have unlike mean scores and spreads yet share the same way in which some scores stand apart. Mean gaps and raw-score correlations do not answer this question. Correlation of divergency, c_delta, is put forward to fill that gap. For every score in each of two equally sized, matched sets, it finds how far that score lies from every other score in its own set. It then draws these gaps into one divergence score through a power mean. The Pearson correlation between the two lists of divergence scores is c_delta, which lies between -1 and 1. A high positive value means that the same people or things stand out in both sets, no matter which way their raw scores point. A negative value means that a score far from the rest in one set tends to be matched with a score near the rest in the other, whilst a value near zero shows little linear link. As c_delta stems from within-set gaps, it is unchanged by shifts in overall level or units. It can show whether the same people feel sharply shut out in matched neighbourhood sets, whether the same bacterial strains grow oddly under two drugs, or whether the same
proteins stand out in two drug-binding fingerprints. It should sit alongside means and raw-score correlations rather than take their place: c_delta does not show whether an outlier is high or low, nor why it arose. The power p sets the weight given to wide gaps. With p=1 it uses Gini distance, and with p=2 root-mean-square distance. Higher p gives more weight to the furthest gaps but cuts Kish's effective sample size. The raw scores and the choice of p must therefore be read with care.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.LG
On the Granularity of Causal Effect Identifiability
Yizuo Chen and Adnan Darwiche
The classical notion of causal effect identifiability is defined in terms of treatment and outcome variables. In this paper, we consider the identifiability of state-based causal effects: how an intervention on a particular state of treatment variables affects a particular state of outcome variables. We demonstrate that state-based causal effects may be identifiable even when variable-based causal effects may not. Moreover, we show that this separation occurs only when additional knowledge – such as context-specific independencies – is available. We further examine knowledge that constrains the states of variables, and show that such knowledge can improve both variable-based and state-based identifiability when combined with other knowledge such as context-specific independencies. We finally propose an approach for identifying causal effects under these additional constraints, and conduct empirical studies to further illustrate the separations between the two levels of identifiability.
Read the full abstract ▾