Statistical Methodology
Estimand framework and intercurrent events handling for clinical trials with time-to-event outcomes
Yixin Fang and Man Jin
The ICH E9(R1) guideline presents a framework of estimand for clinical trials, proposes five strategies for handling intercurrent events (ICEs), and provides a comprehensive discussion and many real-life clinical examples for quantitative outcomes and categorical outcomes. However, in ICH E9(R1) the discussion is lacking for time-to-event (TTE) outcomes. In this paper, we discuss how to define estimands and how to handle ICEs for clinical trials with TTE outcomes. Specifically, we discuss six ICE handling strategies, including those five strategies proposed by ICH E9(R1) and a new strategy, the competing-risk strategy. Compared with ICH E9(R1), the novelty of this paper is three-fold: (1) the estimands are defined in terms of potential outcomes, (2) the methods can utilize time-dependent
covariates straightforwardly, and (3) the efficient
estimators are discussed accordingly.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Optimal Control Theoretic Neural Optimizer: From Backpropagation to Dynamic Programming
Guan-Horng Liu, Tianrong Chen and Evangelos A. Theodorou
Optimization of deep neural networks (DNNs) has been a driving force in the advancement of modern machine learning and artificial intelligence. With DNNs characterized by a prolonged sequence of nonlinear propagation, determining their optimal parameters given an objective naturally fits within the framework of Optimal Control Programming. Such an interpretation of DNNs as dynamical systems has proven crucial in offering a theoretical foundation for principled analysis from numerical equations to physics. In parallel to these theoretical pursuits, this paper focuses on an algorithmic perspective. Our motivated observation is the striking algorithmic resemblance between the Backpropagation
algorithm for computing gradients in DNNs and the optimality conditions for dynamical systems, expressed through another backward process known as dynamic programming. Consolidating this connection, where Backpropagation admits a variational structure, solving an approximate dynamic programming up to the first-order expansion leads to a new class of optimization methods exploring higher-order expansions of the Bellman equation. The resulting optimizer, termed Optimal Control Theoretic Neural Optimizer (OCNOpt), enables rich algorithmic opportunities, including layer-wise feedback policies, game-theoretic applications, and higher-order training of continuous-time models such as Neural ODEs. Extensive experiments demonstrate that OCNOpt improves upon existing methods in robustness and efficiency while maintaining manageable computational complexity, paving new avenues for principled algorithmic design grounded in dynamical systems and optimal control theory.
Read the full abstract ▾
Machine Learning (Statistics)
High-Dimensional BWDM: A Robust Nonparametric Clustering Validation Index for Large-Scale Data
Mohammed Baragilly and Hend Gabr
Determining the appropriate number of clusters in unsupervised learning is a central problem in statistics and data science. Traditional validity indices such as Calinski-Harabasz, Silhouette, and Davies-Bouldin-depend on centroid-based distances and therefore degrade in high-dimensional or contaminated data. This paper proposes a new robust, nonparametric clustering validation framework, the High-Dimensional Between-Within Distance Median (HD-BWDM), which extends the recently introduced BWDM criterion to high-dimensional spaces. HD-BWDM integrates random projection and principal component analysis to mitigate the curse of dimensionality and applies trimmed clustering and medoid-based distances to ensure robustness against outliers. We derive theoretical results showing consistency and convergence under Johnson-Lindenstrauss
embeddings. Extensive simulations demonstrate that HD-BWDM remains stable and interpretable under high-dimensional projections and contamination, providing a robust alternative to traditional centroid-based validation criteria. The proposed method provides a theoretically grounded, computationally efficient stopping rule for nonparametric clustering in modern high-dimensional applications.
Read the full abstract ▾
Statistical Methodology
Complier General Causal Effect in Randomized Controlled Trials with One-Sided Noncompliance
Yin Tang, Yanyuan Ma and Jiwei Zhao
A randomized controlled trial (RCT) is widely regarded as the gold standard for assessing the causal effect of a treatment or intervention, assuming perfect implementation. In practice, however, randomization can be compromised for various reasons, such as one-sided noncompliance. In this paper, we first systematically study the likelihood-based identifiability in an RCT with one-sided noncompliance. This foundational analysis naturally gives rise to the complier general causal effect (CGCE) as the primary estimand. We further develop two
estimators for the CGCE: a simple
estimator that requires no nonparametric procedures, and an efficient
estimator that achieves the semiparametric efficiency bound. Our theoretical analysis shows that, achieving semiparametric efficiency requires only the nuisance
estimators to converge in $L_2$-norm, with no restriction on their convergence rates. This rate-free property opens the door to employing many more modern machine learning methods while still guaranteeing efficiency. Comprehensive simulation studies and a real data application are conducted to illustrate the proposed methods and to compare them with existing approaches.
Read the full abstract ▾
Statistics Theory · Cross-listed from q-fin.MF
On Time-subordinated Brownian Motion Processes for Financial Markets
Rohan Shenoy and Peter Kempthorne
In the context of time-subordinated Brownian motion models,
Fourier theory and methodology are proposed to modelling the
stochastic distribution of time increments. Gaussian Variance-Mean mixtures and time-subordinated models are reviewed with a key example being the Variance-Gamma process. A non-parametric characteristic function decomposition of subordinated Brownian motion is presented. The theory requires an extension of the real domain of certain characteristic functions to the complex plane, the validity of which is proven here. This allows one to characterise and study the
stochastic time-change directly from the full process. An empirical decomposition of S&P log-returns is provided to illustrate the methodology.
Read the full abstract ▾
Machine Learning (Statistics)
deFOREST: Fusing Optical and Radar satellite data for Enhanced Sensing of Tree-loss
Julio Enrique Castrillon-Candas, Hanfeng Gu, Caleb Meredith et al. (+4 more)
In this paper we develop a deforestation detection pipeline that incorporates optical and Synthetic Aperture Radar (SAR) data. A crucial component of the pipeline is the construction of anomaly maps of the optical data, which is done using the residual space of a discrete Karhunen-Loéve (KL) expansion. Anomalies are quantified using a concentration bound on the distribution of the residual components for the nominal state of the forest. This bound does not require prior knowledge on the distribution of the data. This is in contrast to statistical parametric methods that assume knowledge of the data distribution, an impractical assumption that is especially infeasible for high dimensional data such as ours. Once the optical anomaly maps are computed they are combined with SAR data, and the state of the forest is classified by using a Hidden Markov Model (HMM). We test our approach with Sentinel-1 (SAR) and Sentinel-2 (Optical) data on a $92\,km \times 92\,km$ region in the Amazon forest. The results show that both the hybrid optical-radar and optical only methods achieve high accuracy that is superior to the recent state-of-the-art hybrid method. Moreover, the hybrid method is significantly more robust in the case of sparse optical data that are common in highly cloudy regions.
Read the full abstract ▾
All authors Julio Enrique Castrillon-Candas, Hanfeng Gu, Caleb Meredith, Yulin Li, Xiaojing Tang, Pontus Olofsson, Mark Kon
Statistical Computation · Cross-listed from eess.SY
DiffOPF: Diffusion Solver for Optimal Power Flow
Milad Hoseinpour and Vladimir Dvorkin
The optimal power flow (OPF) is a multi-valued, non-convex mapping from loads to dispatch setpoints. The variability of system parameters (e.g., admittances,
topology) further contributes to the multiplicity of dispatch setpoints for a given load. Existing deep learning OPF solvers are single-valued and thus fail to capture the variability of system parameters unless fully represented in the feature space, which is prohibitive. To solve this problem, we introduce a
diffusion-based OPF solver, termed DiffOPF, that treats OPF as a conditional sampling problem. The solver learns the joint distribution of loads and dispatch setpoints from operational history, and returns the marginal dispatch distributions conditioned on loads. Unlike single-valued solvers, DiffOPF enables sampling statistically credible warm starts with favorable cost and constraint satisfaction trade-offs. We explore the sample complexity of DiffOPF to ensure the OPF solution within a prescribed distance from the optimization-based solution, and verify this experimentally on power system
benchmarks.
Read the full abstract ▾
Machine Learning (Statistics)
Exact Dynamics of Multi-class Stochastic Gradient Descent
Elizabeth Collins-Woodfin and Inbar Seroussi
We develop a framework for analyzing the learning dynamics of high-dimensional problems trained using one-pass
stochastic gradient descent (SGD) with data from multiple anisotropic classes. Our main
theorem provides exact expressions for quantities of interest, including the risk and the overlap with the true signal, in terms of a deterministic system of ODEs, valid in the high-dimensional limit. The
theorem holds for a broad class of optimization problems and extends to settings where the number of classes grows with dimension. To illustrate its utility, we investigate in detail the effect of the data's anisotropic structure on the problems of binary logistic regression and least-squares (LS) loss. We study the LS in a linear multiclass setup and derive a learning-rate threshold that depends on the average
eigenvalue of the covariance
matrices. In the binary logistic regression, we study three cases: isotropic covariances, data covariance
matrices with a large fraction of zero
eigenvalues (denoted as the zero-one model), and covariance
matrices with power-law spectra. We show that a structural phase transition occurs. In particular, for the zero-one model and the power-law model with sufficiently large power, SGD aligns more closely with values of the class mean that are projected onto the “clean directions” (i.e., directions of smaller variance). This is supported by analytical studies and numerical simulations, which show the exact asymptotic behavior of the loss in the high-dimensional limit. The effects of data anisotropy that we demonstrate are likely to hold beyond these examples and illustrate one application of the broader
theorem that we prove.
Read the full abstract ▾
Statistics Theory
Minimum Hellinger Distance Estimators for Complex Survey Designs
David Kepplinger and Anand N. Vidyashankar
Reliable inference from complex survey samples can be derailed by outliers and high-leverage observations induced by unequal inclusion probabilities and calibration. We develop a minimum Hellinger distance
estimator (MHDE) for parametric superpopulation models under complex designs, including Poisson PPS and fixed-size SRS/PPS without replacement, with possibly
stochastic post-stratified or calibrated weights. Using a Horvitz-Thompson-adjusted kernel density plug-in, we show: (i) $L^1$-consistency of the KDE with explicit large-deviation tail bounds driven by a variance-adaptive effective sample size; (ii) uniform exponential bounds for the Hellinger affinity that yield MHDE consistency under mild identifiability; (iii) an asymptotic Normal distribution for the MHDE with covariance $\mathbf A^{-1}\boldsymbol\Sigma \mathbf A^{\intercal}$ (and a finite-population correction under without-replacement designs); and (iv) robustness via the influence function and $\alpha$-influence curves in the Hellinger
topology. Simulations under Gamma and lognormal superpopulation models quantify efficiency-robustness trade-offs relative to weighted MLE under independent and high-leverage contamination. An application to NHANES 2021-2023 total water consumption shows that the MHDE remains stable despite extreme responses that markedly bias the MLE. The
estimator is simple to implement via quadrature over a fixed grid and is extensible to other divergence families.
Read the full abstract ▾
Statistical Methodology
Joint modeling and inference of multiple-subject high-dimensional sparse vector autoregressive models
Younghoon Kim, Zachary F. Fisher and Vladas Pipiras
The multiple-subject vector autoregression (multi-VAR) model captures heterogeneous temporal dependence across subjects through commonly shared and subject-unique paths (entries) of individual (subject-specific) sparse VAR transition
matrices. Available joint penalized regression approaches suffer from statistical inefficiency, whereby the convergence rates of the individual
estimators are determined by a subject with the largest sparsity level. An enhanced multi-VAR estimation approach is proposed based on a communication-efficient data integration framework and a suitable identifiability condition in meta-analysis. It is shown that component-wise different convergence rates can be achieved. A hypothesis testing framework is also developed to assess the significance of common paths, as well as the homogeneity of subject-specific paths. Furthermore, a multiple hypothesis test to control the false discovery rate is suggested within a subregion of interest among the entire paths. A simulation study and a real data application compare the performance of the considered methods to other approaches.
Read the full abstract ▾
Statistics Theory
Minimax Estimation Problem for Periodically Correlated Stochastic Processes
Iryna Dubovets'ka and Mykhailo Moklyachuk
The problem of optimal linear estimation of linear functionals depending on the unknown values of a periodically correlated
stochastic process from observations of the process with additive noise is considered. Formulas for calculating the mean square error and the spectral characteristic of the optimal linear estimate of the functionals are proposed in the case where spectral densities are exactly known and in the case where the spectral densities are unknown while a class of admissible spectral densities is given. Formulas that determine the least favorable spectral densities and the minimax (robust) spectral characteristics are proposed for a given class of admissible spectral densities.
Read the full abstract ▾
Statistics Theory
Filtering Problem for Random Processes with Stationary Increments
Maksym Luz and Mykhailo Moklyachuk
This paper deals with the problem of optimal mean-square filtering of the linear functionals $A{\xi}=\int_{0}^{\infty}a(t)\xi(-t)dt$ and $A_T{\xi}=\int_{0}^Ta(t)\xi(-t)dt$ which depend on the unknown values of random process $\xi(t)$ with stationary $n$th increments from observations of process $\xi(t)+\eta(t)$ at points $t\leq0$, where $\eta(t)$ is a stationary process uncorrelated with $\xi(t)$. We propose the values of mean-square errors and spectral characteristics of optimal linear estimates of the functionals when spectral densities of the processes are known. In the case where we can operate only with a set of admissible spectral densities relations that determine the least favorable spectral densities and the minimax spectral characteristics are proposed.
Read the full abstract ▾
Statistics Applications
A Data-Parsimonious Model for Long-Term Risk Assessments of West Nile Virus Spillover
Saman Hosseini, Lee W. Cohnstaedt, Matin Marjani et al. (+1 more)
Many West Nile virus (WNV)
forecasting frameworks incorporate entomological or avian surveillance data, which may be unavailable in some regions. We introduce a novel data-parsimonious probabilistic model to predict both the timing of outbreak onset and the seasonal severity of WNV spillover. Our approach combines a temperature-driven compartmental model of WNV with nonparametric kernel density estimation methods to construct a joint probability density function and a Poisson rate surface as function of mosquito abundance and normalized cumulative temperature. Calibrated on human incidence records, the model produces reliable forecasts several months before the transmission season begins, supporting proactive mitigation efforts. We evaluated the framework across three counties in California (Orange, Los Angeles, and Riverside), two in Texas (Dallas and Harris), and one in Florida (Duval), representing completely different ecology and distinct climatic regimes, and observed strong agreement across multiple performance metrics.
Read the full abstract ▾
All authors Saman Hosseini, Lee W. Cohnstaedt, Matin Marjani, Caterina Scoglio
Statistics Theory
Robust extrapolation problem for random processes with stationary increments
Maksym Luz and Mikhail Moklyachuk
The problem of optimal estimation of linear functionals $A {\xi}=\int_{0}^{\infty} a(t)\xi(t)dt$ and $A_T{\xi}=\int_{0}^{T} a(t)\xi(t)dt$ depending on the unknown values of random process $\xi(t)$, $t\in R$, with stationary $n$th increments from observations of ttis process for $t<0$ is considered. Formulas for calculating mean square error and spectral characteristic of optimal linear estimation of the functionals are proposed in the case when spectral density is exactly known. Formulas that determine the least favorable spectral densities are proposed for given sets of admissible spectral densities.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from astro-ph.IM
Dynamic SBI: Round-free Sequential Simulation-Based Inference with Adaptive Datasets
Huifang Lyu, James Alvey, Noemi Anau Montel et al. (+2 more)
Simulation-based inference (SBI) is emerging as a new statistical paradigm for addressing complex scientific inference problems. By leveraging the representational power of deep neural networks, SBI can extract the most informative simulation features for the parameters of interest. Sequential SBI methods extend this approach by iteratively steering the simulation process towards the most relevant regions of parameter space. This is typically implemented through an algorithmic structure, in which simulation and network training alternate over multiple rounds. This strategy is particularly well suited for high-precision inference in high-dimensional settings, which are commonplace in physics applications with growing data volumes and increasing model fidelity. Here, we introduce dynamic SBI, which implements the core ideas of sequential methods in a round-free, asynchronous, and highly parallelisable manner. At its core is an adaptive
dataset that is iteratively transformed during inference to resemble the target observation. Simulation and training proceed in parallel: trained networks are used both to filter out simulations incompatible with the data and to propose new, more promising ones. Compared to round-based sequential methods, this asynchronous structure can significantly reduce simulation costs and training overhead. We demonstrate that dynamic SBI achieves significant improvements in simulation and training efficiency while maintaining inference performance. We further validate our framework on two challenging astrophysical inference tasks: characterising the
stochastic gravitational wave background and analysing strong gravitational
lensing systems. Overall, this work presents a flexible and efficient new paradigm for sequential SBI.
Read the full abstract ▾
All authors Huifang Lyu, James Alvey, Noemi Anau Montel, Mauro Pieroni, Christoph Weniger
Statistical Methodology
Bayesian Inference for Single-factor Graphical Models
David Marcano and Adrian Dobra
We introduce efficient MCMC
algorithms for
Bayesian inference for single-factor models with correlated residuals where the residuals' distribution is a Gaussian graphical model. We call this family of models single-factor graphical models. We extend single-factor graphical models to
datasets that also involve binary and ordinal categorical variables and to the modeling of multiple
datasets that are spatially or temporally related. Our models are able to capture multivariate associations through latent factors across time and space, as well as residual conditional dependence structures at each spatial location or time point through Gaussian graphical models. We illustrate the application of single-factor graphical models in simulated and real-world examples.
Read the full abstract ▾
Statistics Applications
Earthquake Forecasting with ETAS.inlabru
Ziwen Zhong
The ETAS models are currently the most popular in the field of earthquake
forecasting. The MCMC method is time-consuming and limited by parameter correlation while bringing parameter uncertainty. The INLA-based method "inlabru" solves these problems and performs better at
Bayesian inference. The report introduces the composition of the ETAS model, then provides the model's log-likelihood and approximates it using Taylor expansion and binning strategies. We also present the general procedure of
Bayesian inference in inlabru. The report follows three experiments. The first one explores the effect of fixing one parameter at its actual or wrong values on the posterior distribution of other parameters. We found that $\alpha$ and $K$ have an apparent mutual influence relationship. At the same time, fixing $\alpha$ or $K$ to its actual value can reduce the model fitting time by more than half. The second experiment compares normalised inter-event-time distribution on real data and synthetic catalogues. The distributions of normalised inter-event-time of real data and synthetic catalogues are consistent. Compared with Exp(1), they have more short and long inter-event-time, indicating the existence of clustering. Change on $\mu$ and $p$ will influence the inter-event-time distribution. In the last one, we use events before the mainshock to predict events ten weeks after the mainshock. We use the number test and Continuous Ranked Probability Score (CRPS) to measure the accuracy and precision of the predictions. We found that we need at least one mainshock and corresponding offspring to make reliable
forecasting. And when we have more mainshocks in our data, our
forecasting will be better. Besides, we also figure out what is needed to obtain a good posterior distribution for each parameter.
Read the full abstract ▾
Statistics Applications
Long-Term Spatio-Temporal Forecasting of Monthly Rainfall in West Bengal Using Ensemble Learning Approaches
Jishu Adhikary and Raju Maiti
Rainfall
forecasting plays a critical role in
climate adaptation, agriculture, and water resource management. This study develops long-term forecasts of monthly rainfall across 19 districts of West Bengal using a century-scale
dataset spanning 1900-2019. Daily rainfall records are aggregated into monthly series, resulting in 120 years of observations for each district. The
forecasting task involves predicting the next 108 months (9 years, 2011-2019) while accounting for temporal dependencies and spatial interactions among districts. To address the nonlinear and complex structure of rainfall dynamics, we propose a hierarchical modeling framework that combines regression-based
forecasting of yearly features with multi-layer perceptrons (MLPs) for monthly prediction. Yearly features, such as annual totals, quarterly proportions, variability measures, skewness, and extremes, are first forecasted using regression models that incorporate both own lags and neighboring-district lags. These forecasts are then integrated as auxiliary inputs into an MLP model, which captures nonlinear temporal patterns and spatial dependencies in the monthly series. The results demonstrate that the hierarchical regression-MLP architecture provides robust long-term spatio-temporal forecasts, offering valuable insights for agriculture, irrigation planning, and water conservation strategies.
Read the full abstract ▾
Statistics Applications · Cross-listed from q-fin.ST
Multifractality and its sources in the digital currency market
Stanisław Drożdż, Robert Kluszczyński, Jarosław Kwapień et al. (+1 more)
Multifractality in time series analysis characterizes the presence of multiple scaling exponents, indicating heterogeneous temporal structures and complex dynamical behaviors beyond simple monofractal models. In the context of digital currency
markets, multifractal properties arise due to the interplay of long-range temporal correlations and heavy-tailed distributions of returns, reflecting intricate
market microstructure and trader interactions. Incorporating multifractal analysis into the modeling of cryptocurrency price dynamics enhances the understanding of
market inefficiencies, may improve
volatility forecasting and facilitate the detection of critical transitions or regime shifts. Based on the multifractal cross-correlation analysis (MFCCA) whose spacial case is the multifractal detrended fluctuation analysis (MFDFA), as the most commonly used practical tools for quantifying multifractality, in the present contribution a recently proposed method of disentangling sources of multifractality in time series was applied to the most representative instruments from the digital
market. They include Bitcoin (BTC), Ethereum (ETH), decentralized exchanges (DEX) and non-fungible
tokens (NFT). The results indicate the significant role of heavy tails in generating a broad multifractal spectrum. However, they also clearly demonstrate that the primary source of multifractality are temporal correlations in the series, and without them, multifractality fades out. It appears characteristic that these temporal correlations, to a large extent, do not depend on the thickness of the tails of the fluctuation distribution. These observations, made here in the context of the digital currency
market, provide a further strong argument for the validity of the proposed methodology of disentangling sources of multifractality in time series.
Read the full abstract ▾
All authors Stanisław Drożdż, Robert Kluszczyński, Jarosław Kwapień, Marcin Wątorek
Statistics Applications
Macro-Level Correlational Analysis of Mental Disorders: Economy, Education, Society, and Technology Development
Yingzhi Tao and Chang Yang
This paper quantifies the age-stratified global burden of four mental disorders in 27 regions from 1990 to 2021 using GBD 2021. To put it in detail, it links the age-standardized years of disability adjustment with 18 world development indicators across economic, educational, social and information technology sectors. Then, by means of Pearson correlation, mutual information, Granger causality and maximum information coefficient and other methods, the linear, nonlinear and lagged dependency relationships were evaluated. After research, it was found that there is a very prominent spatio-temporal heterogeneity among young people aged 20 to 39, and the coupling relationship is stronger. From the overall situation, education corresponds to a low burden. Unemployment corresponds to a high burden. Through lag analysis, it can be known that the influence time of economic and technological factors is relatively short, while that of educational factors is relatively long. These results highlight the macro determinants that play a role at different time scales and also provide population-level references for verifying computational mental health models and for intervention measures in specific regions and for specific ages.
Read the full abstract ▾
Machine Learning (Statistics)
PriorGuide: Test-Time Prior Adaptation for Simulation-Based Inference
Yang Yang, Severi Rissanen, Paul E. Chang et al. (+5 more)
Amortized simulator-based inference offers a powerful framework for tackling
Bayesian inference in computational fields such as engineering or neuroscience, increasingly leveraging modern generative methods like
diffusion models to map observed data to model parameters or future predictions. These approaches yield posterior or posterior-predictive samples for new
datasets without requiring further simulator calls after training on simulated parameter-data pairs. However, their applicability is often limited by the prior distribution(s) used to generate model parameters during this training phase. To overcome this constraint, we introduce PriorGuide, a technique specifically designed for
diffusion-based amortized inference methods. PriorGuide leverages a novel guidance approximation that enables flexible adaptation of the trained
diffusion model to new priors at test time, crucially without costly retraining. This allows users to readily incorporate updated information or expert knowledge post-training, enhancing the versatility of pre-trained inference models.
Read the full abstract ▾
All authors Yang Yang, Severi Rissanen, Paul E. Chang, Nasrulloh Loka, Daolang Huang, Arno Solin, Markus Heinonen, Luigi Acerbi
Statistics Theory
Optimal Bounds for Tyler's M-Estimator for Elliptical Distributions
Lap Chi Lau and Akshay Ramachandran
A fundamental problem in statistics is estimating the shape
matrix of an Elliptical distribution. This generalizes the familiar problem of Gaussian covariance estimation, for which the sample covariance achieves optimal estimation error. For Elliptical distributions, Tyler proposed a natural M-
estimator and showed strong statistical properties in the asymptotic regime, independent of the underlying distribution. Numerical experiments show that this
estimator performs very well, and that Tyler's iterative procedure converges quickly to the
estimator. Franks and Moitra recently provided the first distribution-free error bounds in the finite sample setting, as well as the first rigorous convergence analysis of Tyler's iterative procedure. However, their results exceed the sample complexity of the Gaussian setting by a $\log^{2} d$ factor. We close this gap by proving optimal sample threshold and error bounds for Tyler's M-
estimator for all Elliptical distributions, fully matching the Gaussian result. Moreover, we recover the algorithmic convergence even at this lower sample threshold. Our approach builds on the operator scaling connection of Franks and Moitra by introducing a novel pseudorandom condition, which we call $\infty$-expansion. We show that Elliptical distributions satisfy $\infty$-expansion at the optimal sample threshold, and then prove a novel scaling result for inputs satisfying this condition.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Asymptotically optimal reinforcement learning in Block Markov Decision Processes
Thomas van Vuren, Fiona Sloothaak, Maarten G. Wolf et al. (+1 more)
The curse of dimensionality renders Reinforcement Learning (RL) impractical in many real-world settings with exponentially large state and action spaces. Yet, many environments exhibit exploitable structure that can accelerate learning. To formalize this idea, we study RL in Block Markov Decision Processes (BMDPs). BMDPs model problems with large observation spaces, but where transition dynamics are fully determined by latent states. Recent advances in clustering methods have enabled the efficient recovery of this latent structure. However, a regret analysis that exploits these techniques to determine their impact on learning performance remained open. We are now addressing this gap by providing a regret analysis that explicitly leverages clustering, demonstrating that accurate latent state estimation can indeed effectively speed up learning. Concretely, this paper analyzes a two-phase RL
algorithm for BMDPs that first learns the latent structure through random exploration and then switches to an optimism-guided strategy adapted to the uncovered structure. This
algorithm achieves a regret that is $O(\sqrt{T}+n)$ on a large class of BMDPs susceptible to clustering. Here, $T$ denotes the number of time steps, $n$ is the cardinality of the observation space, and the Landau notation $O(\cdot)$ holds up to constants and polylogarithmic factors. This improves the best prior bound, $O(\sqrt{T}+n^2)$, especially when $n$ is large. Moreover, we prove that no
algorithm can achieve lower regret uniformly on this same class of BMDPs. This establishes that, on this class, the
algorithm achieves asymptotic optimality.
Read the full abstract ▾
All authors Thomas van Vuren, Fiona Sloothaak, Maarten G. Wolf, Jaron Sanders
Statistical Methodology
Exact Coordinate Descent for High-Dimensional Regularized Huber Regression
Younghoon Kim, Po-Ling Loh and Sumanta Basu
This study develops a coordinate descent
algorithm for high-dimensional Huber regression with an elastic-net penalty. Unlike existing gradient descent
algorithms and coordinate descent methods based on second-order optimization, the proposed
algorithm performs each coordinate update exactly, without approximation or gradient and Hessian computations. Consequently, it remains stable and converges rapidly even in the presence of highly correlated
covariates or heavy-tailed noise. Building on these exact updates, adaptive variable screening rules and optimality-condition validation procedures are introduced to determine which coordinates to update at each iteration, substantially accelerating convergence in high-dimensional settings. Theoretical guarantees are established for both the convergence of the proposed
algorithm and the validity of validation procedures. Extensive simulation studies under heavy-tailed noise and highly correlated designs, together with a real-data application, demonstrate that the proposed method is both accurate and computationally efficient in these challenging settings.
Read the full abstract ▾
Statistics Theory
Toward an Asymptotic Efficiency Theory on Regular Parameter Manifolds
Lvfang Sun, Zhenhua Lin and Lin Liu
Asymptotic efficiency theory is one of the pillars in the foundations of modern mathematical statistics. Not only does it serve as a rigorous theoretical
benchmark for evaluating statistical methods, but it also sheds light on how to develop and unify novel statistical procedures. For example, the calculus of influence functions has led to many important statistical breakthroughs in the past decades. Responding to the pressing challenge of analyzing increasingly complex
datasets, particularly those with non-Euclidean/nonlinear structures, many novel statistical models and methods have been proposed in recent years. However, the existing efficiency theory is not always readily applicable to these cases, as the theory was developed, for the most part, under the often neglected premise that both the sample space and the parameter space are normed linear spaces. As a consequence, efficiency results outside normed linear spaces are quite rare and isolated, obtained on a case-by-case basis. This paper aims to develop a more unified asymptotic efficiency theory, allowing the sample space, the parameter space, or both to be Riemannian
manifolds satisfying certain regularity conditions. We build a vocabulary that helps translate essential concepts in efficiency theory from normed linear spaces to Riemannian
manifolds, such as (locally) regular
estimators, differentiable functionals, etc. Efficiency bounds are established under conditions parallel to those for normed linear spaces. We also demonstrate the conceptual advantage of the new framework by applying it to two concrete examples in statistics: the population Frechet mean and the regression coefficient vector of Single-Index Models.
Read the full abstract ▾
Statistics Applications
Hierarchical Bayesian Modeling of Dengue in Recife, Brazil (2015-2024): The Role of Spatial Granularity and Data Quality for Epidemiological Risk Mapping
Marcílio Ferreira dos Santos and Andreza dos Santos Rodrigues de Melo
Dengue remains one of Brazil's major epidemiological challenges, marked by strong intra-urban inequalities and the influence of climatic and socio-environmental factors. This study analyzed confirmed dengue cases in Recife from 2015 to 2024 using a
Bayesian hierarchical spatio-temporal model implemented in R-INLA, combining a BYM2 spatial structure with an RW1 temporal component.
Covariates included population density, household size, income, drainage channels, lagged precipitation, and mean temperature. Population density and household size had positive effects on dengue risk, while income and channel presence were protective. Lagged precipitation increased risk, and higher temperatures showed an inverse association, suggesting thermal thresholds for vector activity. The model achieved good fit (DIC=65817; WAIC=64506) and stable convergence, with moderate residual spatial autocorrelation (phi=0.06) and a smooth temporal trend between 2016 and 2019. Spatio-temporal estimates revealed persistent high-risk clusters in northern and western Recife, overlapping with areas of higher density and social vulnerability. Beyond reproducing historical patterns, the
Bayesian model supports probabilistic
forecasting and early warning systems. Compared with classical models (GLM, SAR, GWR, GTWR), INLA explicitly integrates uncertainty and spatial-temporal dependence, offering credible interval inference for decision-making in urban health management.
Read the full abstract ▾
Statistical Methodology
Non-asymptotic goodness-of-fit tests and model selection in valued stochastic blockmodels
Félix Almendra-Hernández, Miles Bakenhus, Vishesh Karwa et al. (+2 more)
A valued
stochastic blockmodel (SBM) is a general way to view networked data in which nodes are grouped into blocks and links between them are measured by counts or labels. This family allows for varying dyad sampling schemes, thereby including the classical, Poisson, and labeled SBMs, as well as those in which some edge observations are censored. This paper addresses the question of testing goodness-of-fit of such non-Bernoulli SBMs, focusing in particular on finite-sample tests. We derive explicit Markov bases moves necessary to generate samples from reference distributions and define goodness-of-fit statistics for determining model fit, comparable to those in the literature for related model families. For the labeled SBM, which includes in particular the censored-edge model, we study the asymptotic behavior of said statistics. One of the main purposes of testing goodness-of-fit of an SBM is to determine whether block membership of the nodes influences network formation. Power and Type 1 error rates are verified on simulated data. Additionally, we discuss the use of asymptotic results in selecting the number of blocks under the latent-block modeling assumption. The method derived for Poisson SBM is applied to ecological networks of host-parasite interactions. Our data analysis conclusions differ in selecting the number of blocks for the species from previous results in the literature.
Read the full abstract ▾
All authors Félix Almendra-Hernández, Miles Bakenhus, Vishesh Karwa, Mitsunori Ogawa, Sonja Petrović
Statistics Applications
Model-assisted estimation for MRV: How to boost the economics of SOC sequestration projects without compromising on scientific integrity
Ahmad Awad and Erik Scharwächter
Soil organic carbon (SOC) sequestration projects require unbiased, precise and cost-effective Monitoring, Reporting, and Verification (MRV) systems that balance sampling costs against uncertainty deductions imposed by regulatory frameworks. Design-based
estimators guarantee unbiasedness but cannot exploit auxiliary data. Model-based approaches (VCS Methodology VT0014 v1.0 (2025)) can improve precision but require independent validation for each project. Model-assisted estimation offers a robust compromise, combining model predictions with probability sampling to retain design-based guarantees while improving precision. We evaluate the scientific integrity and efficiency of the simple regression
estimator (SRE), a well-known model-assisted
estimator, via an extensive simulation study. Our simulations span diverse SOC stock variances, sample sizes, and model performances. We assess three core properties: empirical bias, empirical confidence interval coverage, and precision gain relative to the design-based Horvitz-Thompson
estimator (HTE). Results show negligible bias and valid coverage probabilities for n > 40, regardless of SOC stock variance. Below this threshold, variance approximations and normality assumptions yield unreliable uncertainty estimates. With correlated ancillary variables (r^2 = 0.3), SRE achieves 30% precision gains over HTE. With uncorrelated variables, no gains are observed, but performance converges to HTE for n >= 40. Model-assisted estimation can enhance project economics without compromising scientific rigor. Regulators should permit such
estimators while mandating minimum sample size thresholds. Project proponents should routinely employ such
estimators when correlated ancillary variables exist. The industry should prioritize the
retrieval of high-quality, project-specific
covariates to maximize precision gains and thereby the project economics.
Read the full abstract ▾
Statistics Theory
The generalized Marshall-Olkin Lomax distribution with applications to AIDS and COVID-19 data
Alexsandro A. Ferreira and Gauss M. Cordeiro
The generalized Marshall-Olkin Lomax distribution is introduced, and its properties are easily obtained from those of the Lomax distribution. A regression model for censored data is proposed. The parameters are estimated through maximum likelihood, and consistency is verified by simulations. Three real
datasets are selected to illustrate the superiority of the new models compared to those from two well-known classes.
Read the full abstract ▾
Machine Learning (Statistics)
On the Identifiability of Causal Graphs with the Invariance Principle
Francesco Montagna
Causal discovery from i.i.d. observational data is known to be generally ill-posed. We demonstrate that if we have access to the distribution {induced} by a structural causal model, and additional data from (in the best case) only two environments that sufficiently differ in the noise statistics, the unique causal graph is identifiable. Notably, this is the first result in the literature that guarantees the entire causal graph recovery with a constant number of environments and arbitrary nonlinear mechanisms. Our only constraint is the Gaussianity of the noise terms; however, we propose potential ways to relax this requirement. Of interest on its own, we expand on the well-known duality between independent component analysis (ICA) and causal discovery; recent advancements have shown that nonlinear ICA can be solved from multiple environments, at least as many as the number of sources: we show that the same can be achieved for causal discovery while having access to much less auxiliary information.
Read the full abstract ▾