Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Tuesday 11 November

Statistical Methodology

Deep neural expected shortfall regression with tail-robustness

Expected shortfall (ES), also known as conditional value-at-risk, is a widely recognized risk measure that complements value-at-risk by capturing tail-related risks more effectively. Compared with quantile regression, which has been extensively developed and applied across disciplines, ES regression remains in its early stage, partly because the traditional empirical risk minimization framework is not directly applicable. In this paper, we develop a nonparametric framework for expected shortfall regression based on a two-step approach that treats the conditional quantile function as a nuisance parameter. Leveraging the representational power of deep neural networks, we construct a two-step ES estimator using feedforward ReLU networks, which can alleviate the curse of dimensionality when the underlying functions possess hierarchical composition structures. However, ES estimation is inherently sensitive to heavy-tailed response or error distributions. To address this challenge, we integrate a properly tuned Huber loss into the neural network training, yielding a robust deep ES estimator that is provably resistant to heavy-tailedness in a non-asymptotic sense and first-order insensitive to quantile estimation errors in the first stage. Comprehensive simulation studies and an empirical analysis of the effect of El Niño on extreme precipitation illustrate the accuracy and robustness of the proposed method.
Read the full abstract

All authors Myeonghun Yu, Kean Ming Tan, Huixia Judy Wang, Wen-Xin Zhou

Statistics Applications · Cross-listed from cs.HC

Modeling multi-agent motion dynamics in immersive rooms

Immersive rooms are increasingly popular augmented reality systems that support multi-agent interactions within a virtual world. However, despite extensive content creation and technological developments, insights about perceptually-driven social dynamics, such as the complex movement patterns during virtual world navigation, remain largely underexplored. Computational models of motion dynamics can help us understand the underlying mechanism of human interaction in immersive rooms and develop applications that better support spatially distributed interaction. In this work, we propose a new agent-based model of emergent human motion dynamics. The model represents human agents as simple spatial geometries in the room that relocate and reorient themselves based on the salient virtual spatial objects they approach. Agent motion is modeled as an interactive process combining external diffusion-driven influences from the environment with internal self-propelling interactions among agents. Further, we leverage simulation-based inference (SBI) to show that the governing parameters of motion patterns can be estimated from simple observables. Our results indicate that the model successfully captures action-related agent properties but exposes local non-identifiability linked to environmental awareness. We argue that our simulation-based approach paves the way for creating adaptive, responsive immersive rooms – spaces that adjust their interfaces and interactions based on human collective movement patterns and spatial attention.
Read the full abstract

Statistical Methodology

Practical considerations when designing an online learning algorithm for an app-based mHealth intervention

The ubiquitous nature of mobile health (mHealth) technology has expanded opportunities for the integration of reinforcement learning into traditional clinical trial designs, allowing researchers to learn individualized treatment policies during the study. LowSalt4Life 2 (LS4L2) is a recent trial aimed at reducing sodium intake among hypertensive individuals through an app-based intervention. A reinforcement learning algorithm, which was deployed in one of the trial arms, was designed to send reminder notifications to promote app engagement in contexts where the notification would be effective, i.e., when a participant is likely to open the app in the next 30-minute and not when prior data suggested reduced effectiveness. Such an algorithm can improve app-based mHealth interventions by reducing participant burden and more effectively promoting behavior change. We encountered various challenges during the implementation of the learning algorithm, which we present as a template to solving challenges in future trials that deploy reinforcement learning algorithms. We provide template solutions based on LS4L2 for solving the key challenges of (i) defining a relevant reward, (ii) determining a meaningful timescale for optimization, (iii) specifying a robust statistical model that allows for automation, (iv) balancing model flexibility with computational cost, and (v) addressing missing values in gradually collected data.
Read the full abstract

All authors Rachel T Gonzalez, Madeline R Abbott, Brahmajee Nallamothu, Scott Hummel, Michael Dorsch, Walter Dempsey

Machine Learning (Statistics)

Optimal control of the future via prospective learning with control

Optimal control of the future is the next frontier for AI. Current approaches to this problem are typically rooted in reinforcement learning (RL). RL is mathematically distinct from supervised learning, which has been the main workhorse for the recent achievements in AI. Moreover, RL typically operates in a stationary environment with episodic resets, limiting its utility. Here, we extend supervised learning to address learning to control in non-stationary, reset-free environments. Using this framework, called ”Prospective Learning with Control” (PLuC), we prove that under certain fairly general assumptions, empirical risk minimization (ERM) asymptotically achieves the Bayes optimal policy. We then consider a specific instance of prospective learning with control: foraging, a canonical task relevant to both natural and artificial agents. We illustrate that modern RL algorithms, which assume stationarity, struggle in these non-stationary reset-free environments. Even with time-aware modifications, they converge orders of magnitude slower than our prospective foraging agents on a simple 1-D foraging benchmark. Code is available at: https://github.com/neurodata/procontrol.
Read the full abstract

All authors Yuxin Bai, Aranyak Acharyya, Ashwin De Silva, Zeyu Shen, James Hassett, Joshua T. Vogelstein

Machine Learning (Statistics) · Cross-listed from cs.LG

TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models

The first tabular foundation model, TabPFN, and its successor TabPFNv2 have impacted tabular AI substantially, with dozens of methods building on it and hundreds of applications across different use cases. This report introduces TabPFN-2.5, the next generation of our tabular foundation model, built for datasets with up to 50,000 data points and 2,000 features, a 20x increase in data cells compared to TabPFNv2. TabPFN-2.5 is now the leading method for the industry standard benchmark TabArena (which contains datasets with up to 100,000 training data points), substantially outperforming tuned tree-based models and matching the accuracy of AutoGluon 1.4, a complex four-hour tuned ensemble that even includes the previous TabPFNv2. Remarkably, default TabPFN-2.5 has a 100% win rate against default XGBoost on small to medium-sized classification datasets (<=10,000 data points, 500 features) and a 87% win rate on larger datasets up to 100K samples and 2K features (85% for regression). For production use cases, we introduce a new distillation engine that converts TabPFN-2.5 into a compact MLP or tree ensemble, preserving most of its accuracy while delivering orders-of-magnitude lower latency and plug-and-play deployment. This new release will immediately strengthen the performance of the many applications and methods already built on the TabPFN ecosystem.
Read the full abstract

All authors Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manium, Rosen Yu, Felix Jablonski, Shi Bin Hoo, Anurag Garg, Jake Robertson, Magnus Bühler, Vladyslav Moroshan, Lennart Purucker, Clara Cornu, Lilly Charlotte Wehrhahn, Alessandro Bonetto, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, Frank Hutter

Statistical Methodology

Reluctant Transfer Learning in Penalized Regressions for Individualized Treatment Rules under Effect Heterogeneity

Estimating individualized treatment rules (ITRs) is fundamental to precision medicine, where the goal is to tailor treatment decisions to individual patient characteristics. While numerous methods have been developed for ITR estimation, there is limited research on model updating that accounts for shifted treatment-covariate relationships in the ITR setting. In practice, models trained on source data must be updated for new (target) datasets that exhibit shifts in treatment effects. To address this challenge, we propose a Reluctant Transfer Learning (RTL) framework that enables efficient model adaptation by selectively transferring essential model components (e.g., regression coefficients) from source to target data, without requiring access to individual-level source data. Leveraging the principle of reluctant modeling, the RTL approach incorporates model adjustments only when they improve performance on the target dataset, thereby controlling complexity and enhancing generalizability. Our method supports multi-armed treatment settings, performs variable selection for interpretability, and provides a regret bound for the difference in value of the optimal ITR and that of the estimated ITR. Through simulation studies and an application to a real data example from the Best Apnea Interventions for Research (BestAIR) trial, we demonstrate that RTL outperforms existing alternatives. The proposed framework offers an efficient, practically feasible approach to adaptive treatment decision-making under evolving treatment effect conditions.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

Learning manipulable representations of the world and its dynamics is central to AI. Joint-Embedding Predictive Architectures (JEPAs) offer a promising blueprint, but lack of practical guidance and theory has led to ad-hoc R&D. We present a comprehensive theory of JEPAs and instantiate it in LeJEPA, a lean, scalable, and theoretically grounded training objective. First, we identify the isotropic Gaussian as the optimal distribution that JEPAs' embeddings should follow to minimize downstream prediction risk. Second, we introduce a novel objective–Sketched Isotropic Gaussian Regularization (SIGReg)–to constrain embeddings to reach that ideal distribution. Combining the JEPA predictive loss with SIGReg yields LeJEPA with numerous theoretical and practical benefits: (i) single trade-off hyperparameter, (ii) linear time and memory complexity, (iii) stability across hyper-parameters, architectures (ResNets, ViTs, ConvNets) and domains, (iv) heuristics-free, e.g., no stop-gradient, no teacher-student, no hyper-parameter schedulers, and (v) distributed training-friendly implementation requiring only $\approx$50 lines of code. Our empirical validation covers 10+ datasets, 60+ architectures, all with varying scales and domains. As an example, using imagenet-1k for pretraining and linear evaluation with frozen backbone, LeJEPA reaches 79% with a ViT-H/14. We hope that the simplicity and theory-friendly ecosystem offered by LeJEPA will reestablish self-supervised pre-training as a core pillar of AI research (\href{https://github.com/rbalestr-lab/lejepa}{GitHub repo}).
Read the full abstract

Statistics Theory

Neumann-series corrections for regression adjustment in randomized experiments

We study average treatment effect (ATE) estimation under complete randomization with many covariates in a design-based, finite-population framework. In randomized experiments, regression adjustment can improve precision of estimators using covariates, without requiring a correctly specified outcome model. However, existing design-based analyses establish asymptotic normality only up to $p = o(n^{1/2})$, extendable to $p = o(n^{2/3})$ with a single de-biasing. We introduce a novel theoretical perspective on the asymptotic properties of regression adjustment through a Neumann-series decomposition, yielding a systematic higher-degree corrections and a refined analysis of regression adjustment. Specifically, for ordinary least squares regression adjustment, the Neumann expansion sharpens analysis of the remainder term, relative to the residual difference-in-means. Under mild leverage regularity, we show that the degree-$d$ Neumann-corrected estimator is asymptotically normal whenever $p^{ d+3}(\log p)^{ d+1}=o(n^{ d+2})$, strictly enlarging the admissible growth of $p$. The analysis is purely randomization-based and does not impose any parametric outcome models or super-population assumptions.
Read the full abstract

Statistical Methodology

Multi-level Latent Variable Models for Coheritability Analysis in Electronic Health Records

Electronic health records (EHRs) linked with familial relationship data offer a unique opportunity to investigate the genetic architecture of complex phenotypes at scale. However, existing heritability and coheritability estimation methods often fail to account for the intricacies of familial correlation structures, heterogeneity across phenotype types, and computational scalability. We propose a robust and flexible statistical framework for jointly estimating heritability and genetic correlation among continuous and binary phenotypes in EHR-based family studies. Our approach builds on multi-level latent variable models to decompose phenotypic covariance into interpretable genetic and environmental components, incorporating both within- and between-family variations. We derive iteration algorithms based on generalized equation estimations (GEE) for estimation. Simulation studies under various parameter configurations demonstrate that our estimators are consistent and yield valid inference across a range of realistic settings. Applying our methods to real-world EHR data from a large, urban health system, we identify significant genetic correlations between mental health conditions and endocrine/metabolic phenotypes, supporting hypotheses of shared etiology. This work provides a scalable and rigorous framework for coheritability analysis in high-dimensional EHR data and facilitates the identification of shared genetic influences in complex disease networks.
Read the full abstract

Machine Learning (Statistics)

Source-Optimal Training is Transfer-Suboptimal

We prove that training a source model optimally for its own task is generically suboptimal when the objective is downstream transfer. We study the source-side optimization problem in L2-SP ridge regression and show a fundamental mismatch between the source-optimal and transfer-optimal source regularization: outside of a measure-zero set, $\tau_0^* \neq \tau_S^*$. We characterize the transfer-optimal source penalty $\tau_0^*$ as a function of task alignment and identify an alignment-dependent reversal: with imperfect alignment ($0<\rho<1$), transfer benefits from stronger source regularization, while in super-aligned regimes ($\rho>1$), transfer benefits from weaker regularization. Additionally, in isotropic settings, the decision of whether transfer helps is independent of the target sample size and noise, depending only on task alignment and source characteristics. We verify the linear predictions in a synthetic ridge regression experiment, and we present experiments on MNIST, CIFAR-10, and 20 Newsgroups as evidence that the source-optimal versus transfer-optimal mismatch persists in standard nonlinear transfer learning pipelines.
Read the full abstract

Statistical Methodology

Principal nested spheres for high-dimensional data

The method of Principal Nested Spheres (PNS) is a non-linear dimension reduction technique for spherical data. The method is a backwards fitting procedure, starting with fitting a high-dimensional sphere and then successively reducing dimension at each stage. After reviewing the PNS method in detail, we introduce some new methods for model selection at each stage between great and small subspheres, based on the Kolmogorov-Smirnov test, a variance test and a likelihood ratio test. The current PNS fitting method is slow for high-dimensional spherical data, and so we introduce a fast PNS method which involves an initial principal components analysis decomposition to select a basis for lower dimensional PNS. A new visual method called the PNS biplot is introduced for examining the effects of the original variables on the PNS, and this involves procedures for back-fitting from the PNS scores back to the original variables. The methodology is illustrated with two high-dimensional datasets from cancer research: Melanoma proteomics data with 500 variables and 205 patients, and a Pan Cancer dataset with 12,478 genes and 300 patients. In both applications the PNS biplot is used to select variables for effective classification.
Read the full abstract

Statistical Methodology

Clinicians' Interpretation and Preferences for Survival Data Visualisation: A Pre-Post Study Comparing Kaplan-Meier and Mean Residual Life Plots

Effective visualization of survival data is essential for clinician interpretation and patient communication. While Kaplan-Meier (KM) plots are widely used, Mean Residual Life (MRL) plots may offer a more intuitive display of prognosis over time. However, little is known about clinicians' knowledge and preferences regarding these alternatives. This pre-post pilot cross-sectional survey assessed 32 medical students and doctors who interpreted four survival plot types (KM, survival difference, MRL, and MRL difference) before and after a brief learning section. Interpretation accuracy, learning gain, and ranking preferences were analyzed. Overall accuracy improved from 50.0 percent pre-learning to 81.2 percent post-learning (p = 0.002), with the largest improvement for MRL plots (+37.5 percentage points). KM plots remained the most preferred for ease of clinical use (59 percent), while MRL plots were valued for patient communication (9 percent). Participants with lower self-rated survival knowledge showed the greatest learning gains. These findings suggest that with minimal instruction, clinicians can interpret MRL plots as effectively as KM plots. Incorporating MRL visualizations into clinical dashboards and medical education could improve understanding of survival outcomes and patient-centered communication.
Read the full abstract

Machine Learning (Statistics)

Concentration bounds on response-based vector embeddings of black-box generative models

Generative models, such as large language models or text-to-image diffusion models, can generate relevant responses to user-given queries. Response-based vector embeddings of generative models facilitate statistical analysis and inference on a given collection of black-box generative models. The Data Kernel Perspective Space embedding is one particular method of obtaining response-based vector embeddings for a given set of generative models, already discussed in the literature. In this paper, under appropriate regularity conditions, we establish high probability concentration bounds on the sample vector embeddings for a given set of generative models, obtained through the method of Data Kernel Perspective Space embedding. Our results tell us the required number of sample responses needed in order to approximate the population-level vector embeddings with a desired level of accuracy. The algebraic tools used to establish our results can be used further for establishing concentration bounds on Classical Multidimensional Scaling embeddings in general, when the dissimilarities are observed with noise.
Read the full abstract

All authors Aranyak Acharyya, Joshua Agterberg, Youngser Park, Carey E. Priebe

Machine Learning (Statistics)

Semi-Supervised Treatment Effect Estimation with Unlabeled Covariates for Prediction-Powered Causal Inference

This study investigates treatment effect estimation in the semi-supervised setting, also can be interpreted as prediction-powered inference. In our setting, we can use not only the standard triple of covariates, treatment indicator, and outcome, but also unlabeled auxiliary covariates. For this problem, we develop efficiency bounds and efficient estimators whose asymptotic variance aligns with the efficiency bound. In the analysis, we introduce two different data-generating processes: the one-sample setting and the two-sample setting. The one-sample setting considers the case where we can observe treatment indicators and outcomes for a part of the dataset, which is also called the censoring setting. In contrast, the two-sample setting considers two independent datasets with labeled and unlabeled data, which is also called the case-control setting or the stratified setting. In both settings, we find that by incorporating auxiliary covariates, we can lower the efficiency bound and obtain an estimator with an asymptotic variance smaller than that without such auxiliary covariates. We frame our framework as prediction-powered causal inference.
Read the full abstract

Statistics Applications

The impact of Women's empowerment on childhood vaccination coverage in Nigeria: a spatio-temporal analysis

Immunization remains one of the most effective public health interventions, substantially reducing childhood morbidity and mortality worldwide. Yet, gender disparity and women's disempowerment continue to hinder access to vaccination services in low- and middle-income countries. In Nigeria, variations in social norms and cultural values shape gender roles, limiting women's autonomy in healthcare decisions and household participation. These constraints contribute to spatial differences in immunization uptake. Using data from four waves of the Nigeria Demographic and Health Survey, we developed two empowerment indices capturing women's participation in household decision-making and their ability to decide on personal healthcare needs. A structured spatiotemporal statistical model was applied to assess how much of the observed vaccination disparities could be attributed to women's empowerment and to predict vaccination outcomes at the third administrative level. We examined five indicators: Bacillus Calmette-Guerin (BCG), zero-dose, complete DPT, MCV-1 (first dose of measles-containing vaccine), and all-basic vaccination coverage. Model validation involved comparing empirical estimates with projections at the second administrative level. Results indicate that empowerment related to household participation and healthcare autonomy generally increases vaccination uptake, though the magnitude of effects varies geographically, particularly among highly empowered women. Despite ongoing national efforts to close immunization gaps, the study highlights the need for context-specific strategies that enhance women's decision-making power and community engagement to reduce regional disparities and improve overall vaccination coverage.
Read the full abstract

All authors Ezra Gayawan, Osafu Augustine Egbon, Edson Utazi, Jamila Abubakar Umar, Caroline Trotter

Statistical Computation

High-Performance Variance-Covariance Matrix Construction Using an Uncentered Gram Formulation

Reichel (2025) defined the bariance as a pairwise-difference measure that can be rewritten in linear time using only scalar sums. We extend this idea to the covariance matrix by showing that the standard matrix expression involving the uncentered Gram matrix and a correction term is algebraically identical to the pairwise-difference definition while avoiding explicit centering. The computation then reduces to one outer product of dimension p-by-p and a single subtraction. Benchmarks in Python show clear runtime gains, especially when BLAS optimizations are absent. Optionally faster Gram-matrix routines such as RXTX (Rybin et al., 2025) further reduce overall cost.
Read the full abstract

Statistics Theory

Spatial Confidence Regions for Piecewise Continuous Processes

In scientific disciplines such as neuroimaging, climatology, and cosmology it is useful to study the uncertainty of excursion sets of imaging data. While the case of imaging data obtained from a single study condition has already been intensively studied, confidence statements about the intersection, or union, of the excursion sets derived from different subject conditions have only been introduced recently. Such methods aim to model the images from different study conditions as asymptotically Gaussian random processes with differentiable sample paths. In this work, we remove the restricting condition of differentiability and only require continuity of the sample paths. This allows for a wider range of applications including many settings which cannot be treated with the existing theory. To achieve this, we introduce a novel notion of convergence on piecewise continuous functions over finite partitions. This notion is of interest in its own right, as it implies convergence results for maxima of sequences of piecewise continuous functions over sequences of sets. Generalizing well-known results such as the extended continuous mapping theorem, this novel convergence notion also allows us to construct for the first time confidence regions for mathematically challenging examples such as symmetric differences of excursion sets.
Read the full abstract

Statistical Methodology

Geometric modelling of spatial extremes

Recent developments in extreme value statistics have established the so-called geometric approach as a powerful modelling tool for multivariate extremes. We tailor these methods to the case of spatial modelling and examine their efficacy at inferring extremal dependence and performing extrapolation. The geometric approach is based around a limit set described by a gauge function, which is a key target for inference. We consider a variety of spatially-parameterised gauge functions and perform inference on them by building on the framework of Wadsworth and Campbell (2024), where extreme radii are modelled via a truncated gamma distribution. We also consider spatial modelling of the angular distribution, for which we propose two candidate models. Estimation of extreme event probabilities is possible by combining draws from the radial and angular models respectively. We compare our method with two other established frameworks for spatial extreme value analysis and show that our approach generally allows for unbiased, albeit more uncertain, inference compared to the more classical models. We illustrate the methodology on a space weather dataset of daily geomagnetic field fluctuations.
Read the full abstract

Statistical Methodology

Reclustering: A New Method to Test the Appropriate Level of Clustering

When scholars suspect units are dependent on each other within clusters but independent of each other across clusters, they employ cluster-robust standard errors (CRSEs). Nevertheless, what to cluster over is sometimes unknown. For instance, in the case of cross-sectional survey samples, clusters may be households, municipalities, counties, or states. A few approaches have been proposed, although they are based on asymptotics. I propose a new method to address this issue that works in a finite sample: reclustering. That is, we randomly and repeatedly group fine clusters into new gross clusters and calculate a statistic such as CRSEs. Under the null hypothesis that fine clusters are independent of each other, how they are grouped into gross clusters should not matter for any cluster-sensitive statistic. Thus, if the statistic based on the original clustering is a significant outlier against the distributions of the statistics induced by reclustering, it is reasonable to reject the null hypothesis and employ gross clusters. I compare the performance of reclustering with that of a few previous tests using Monte Carlo simulation and application.
Read the full abstract

Statistical Methodology

Simulation-Based Fitting of Intractable Models via Sequential Sampling and Local Smoothing

This paper presents a comprehensive algorithm for fitting generative models whose likelihood, moments, and other quantities typically used for inference are not analytically or numerically tractable. The proposed method aims to provide a general solution that requires only limited prior information on the model parameters. The algorithm combines a global search phase, aimed at identifying the region of the solution, with a local search phase that mimics a trust region version of the Fisher scoring algorithm for computing a quasi-likelihood estimator. Comparisons with alternative methods demonstrate the strong performance of the proposed approach. An R package implementing the algorithm is available on CRAN.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories

In this work, we study the problem of offline safe imitation learning (IL). In many real-world settings, online interactions can be risky, and accurately specifying the reward and the safety cost information at each timestep can be difficult. However, it is often feasible to collect trajectories reflecting undesirable or risky behavior, implicitly conveying the behavior the agent should avoid. We refer to these trajectories as non-preferred trajectories. Unlike standard IL, which aims to mimic demonstrations, our agent must also learn to avoid risky behavior using non-preferred trajectories. In this paper, we propose a novel approach, SafeMIL, to learn a parameterized cost that predicts if the state-action pair is risky via Multiple Instance Learning. The learned cost is then used to avoid non-preferred behaviors, resulting in a policy that prioritizes safety. We empirically demonstrate that our approach can learn a safer policy that satisfies cost constraints without degrading the reward performance, thereby outperforming several baselines.
Read the full abstract

Statistical Computation

gemlib: Probabilistic programming for epidemic models

gemlib is a Python library for defining, simulating, and calibrating Markov state-transition models. Stochastic models are often computationally intensive, making them impractical to use in pandemic response efforts despite their favourable interpretations compared to their deterministic counterparts. gemlib decomposes state-transition models into three key ingredients which succinctly encapsulate the model and are sufficient for executing the subsequent computational routines. Simulation is performed using implementations of Gillespie's algorithm for continuous-time models and a generic Tau-leaping algorithm for discrete time models. gemlib models integrate seamlessly with Markov Chain Monte Carlo samplers as they provide a target distribution for the inference algorithm. Algorithms are implemented using the machine learning computational frameworks JAX and TensorFlow Probability, thus taking advantage of modern hardware to accelerate computation. This abstracts away computational concerns from modellers, allowing them to focus on developing and testing different model structures or assumptions. The gemlib library enables users to rapidly implement and calibrate stochastic epidemic models with the flexibility and robustness required to support decision during an emerging outbreak.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

On the Kantorovich contraction of Markov semigroups

We develop an operator-theoretic framework for the contraction properties of Markov semigroups with respect to a general class of Kantorovich semi-distances, which includes Wasserstein distances and weighted total variation norms. The framework is based on Kantorovich contraction coefficients, which extend Dobrushin's ergodic coefficient from the total variation distance to general transport costs. Our main result shows that a geometric drift condition, combined with a local contraction condition on the sub-level sets of the Lyapunov function, implies that the Markov operator is a strict contraction with respect to an explicit modification of the underlying Kantorovich cost. This yields, in a unified way, the existence and uniqueness of invariant measures and exponential decay estimates in Wasserstein distances, for both discrete time and continuous time Markov semigroups, as well as for time-varying semigroups. We illustrate the scope of the framework with (i) Markov transitions on models with boundary states, including bounded domains with entrance boundaries, (ii) operator products of a Markov kernel and its adjoint, including two-block Gibbs samplers, (iii) iterated random functions, including Euler-type schemes, (iv) diffusion models, including overdamped Langevin diffusions with potentials that are convex at infinity, jump-diffusions on compact manifolds, and Jacobi and Riccati matrix diffusions, and (v) time-varying and nonlinear semigroups, including Feynman-Kac distribution flows and Sinkhorn bridges. In several of these settings we obtain new contraction estimates, while in others we recover known results through short and elementary arguments.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from math.OC

Model Predictive Control is almost Optimal for Heterogeneous Restless Multi-armed Bandits

We consider a general infinite horizon Heterogeneous Restless multi-armed Bandit (RMAB). Heterogeneity is a fundamental problem for many real-world systems largely because it resists many concentration arguments. In this paper, we assume that each of the $N$ arms can have different model parameters. Model predictive control is a well-known control strategy that repeatedly solves a finite-horizon optimization problem of length $\tau$ to produce a policy that can be applied to an infinite-horizon setting. In this paper, we adopt this approach by repeatedly solving a finite linear program, yielding what we call the LP-update policy for the infinite-horizon problem. Under a mild assumption of uniform ergodicity, we show an $\mathcal{O}\left(\sqrt{1/N}\right)$ suboptimality gap on this well-known algorithm that works very well in practice. In addition to the LP-update policy we are able to derive a finite-horizon policy (LP-update with recomputation) that segments the infinite time horizon into finite horizon problems that allow us to explicitly connect the length of computation time to the acceptable error tolerance. Our simulations demonstrate that our algorithm works extremely well even when this finite-horizon, $\tau$, is very small (in our case $5$), which makes it computationally efficient. Our theoretical results draw on techniques from the model predictive control literature by invoking the concept of dissipativity and generalize quite easily to the more general weakly coupled heterogeneous Markov Decision Process setting. In addition, we draw a parallel between our own policy and the LP-index policy by showing that the LP-index policy corresponds to $\tau=1$.
Read the full abstract

Statistical Methodology

Who's Afraid of the Wallenius Distribution?

This paper is about the use of the Wallenius noncentral hypergeometric distribution for analysing contingency tables with two or more groups and two categories and with row margins and sample size, that is both margins, fixed. The parameters of the distribution are taken to be weights which are positive and sum to one and are thus defined on a regular simplex. The approach to analysis is presented for likelihood-based and Bayesian inference and is illustrated by example, with datasets taken from the literature and, in one case, used to generate semi-synthetic data. The analysis of two-by-two contingency tables using the univariate Wallenius distribution is shown to be straightforward, with the parameter a single weight which translates immediately to the requisite odds and the odds ratio. The analysis of contingency tables with more than two groups based on the multivariate Wallenius distribution was however more nuanced than that of the two-group tables. Specifically, some numerical subtleties were required in order to implement the necessary calculations. In particular, optimisation with respect to the weights was performed by transforming the weights to yield an unconstrained optimisation problem and likelihoods which are extremely small were scaled by an appropriate multiplying factor without compromising the elements of inference. Furthermore, a novel Markov chain Monte Carlo algorithm for Bayesian inference, termed the sphere walk Metropolis, was constructed. The proposal is implemented in Cartesian coordinates on the reference simplex and the Metropolis filter in barycentric coordinates on the regular simplex, with the transition between barycentric and Cartesian coordinates effected seamlessly.
Read the full abstract

Statistics Applications

Parameter Estimation and Seasonal Modification of the Fractional Poisson Process with Application to Vorticity Extremes over the North Atlantic

The fractional Poisson process (FPP) generalizes the standard Poisson process by replacing exponentially distributed return times with Mittag-Leffler distributed ones with an extra tail parameter, allowing for greater flexibility. The FPP has been applied in various fields, such as modeling occurrences of extratropical cyclones in meteorology and solar flares in physics. We propose a new estimation method for the parameters of the FPP, based on minimizing the distance between the empirical and the theoretical distribution at selected quantiles. We conduct an extensive simulation study to evaluate the advantages and limitations of the new estimation method and to compare it with several competing estimators, some of which have not yet been examined in the Mittag-Leffler setting. To enhance the applicability of the FPP in real-world scenarios, particularly in meteorology, we propose a method for incorporating seasonality into the FPP through distance-based weighting. We then analyze the return times of relative vorticity extremes in the North Atlantic-European region using our seasonal modeling approach.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Online Linear Regression with Paid Stochastic Features

We study an online linear regression setting in which the observed feature vectors are corrupted by noise and the learner can pay to reduce the noise level. In practice, this may happen for several reasons: for example, because features can be measured more accurately using more expensive equipment, or because data providers can be incentivized to release less private features. Assuming feature vectors are drawn i.i.d. from a fixed but unknown distribution, we measure the learner's regret against the linear predictor minimizing a notion of loss that combines the prediction error and payment. When the mapping between payments and noise covariance is known, we prove that the rate $\sqrt{T}$ is optimal for regret if logarithmic factors are ignored. When the noise covariance is unknown, we show that the optimal regret rate becomes of order $T^{2/3}$ (ignoring log factors). Our analysis leverages matrix martingale concentration, showing that the empirical loss uniformly converges to the expected one for all payments and linear predictors.
Read the full abstract

Statistical Methodology

ANOVATS: A subsampling-based test to detect differences among short time series in marine studies

Assessing marine ecosystems is important for understanding the impacts of climate change and human activity, as well as for maintaining healthy oceans and ecosystems. In marine science, it is common for biologists and geologists to identify regional differences based on expert knowledge, frequently through data visualization. However, time series data collected through surveys in marine studies typically span only a few decades, limiting the applicability of classical time series methods. Additionally, without expert knowledge, detecting significant differences becomes challenging. To address these issues, we introduce ANOVATS (ANOVA for small-sample time series data), a subsampling-based method to detect regional differences in small-sample time series data with a fixed number of groups. This method bypasses the need for spectral density estimation, which requires a large number of time points in the data. Furthermore, after detecting differences in homogeneity across all areas using the ANOVATS procedure, we devised a simple ANOVATS post hoc procedure to group the areas. Finally, we demonstrate the effectiveness of our method by analyzing zooplankton biomass data collected in different strata of the North Sea, showing its ability to quantify differences in species between geographical areas without relying on prior biological or geographical knowledge.
Read the full abstract

All authors Yuichi Goto, Hiroko Kato Solvang, Masanobu Taniguchi, Tone Falkenhaug

Statistical Methodology

Inference on multiple quantiles in regression models by a rank-score approach

This paper tackles the challenge of performing multiple quantile regressions across different quantile levels and the associated problem of controlling the familywise error rate, an issue that is generally overlooked in practice. We propose a multivariate extension of the rank-score test and embed it within a closed-testing procedure to efficiently account for multiple testing. Then we further generalize the multivariate test to enhance statistical power against alternatives in selected directions. Theoretical foundations and simulation studies demonstrate that our method effectively controls the familywise error rate while achieving higher power than traditional corrections, such as Bonferroni.
Read the full abstract

Machine Learning (Statistics)

PrAda-GAN: A Private Adaptive Generative Adversarial Network with Bayes Network Structure

We revisit the problem of generating synthetic data under differential privacy. To address the core limitations of marginal-based methods, we propose the Private Adaptive Generative Adversarial Network with Bayes Network Structure (PrAda-GAN), which integrates the strengths of both GAN-based and marginal-based approaches. Our method adopts a sequential generator architecture to capture complex dependencies among variables, while adaptively regularizing the learned structure to promote sparsity in the underlying Bayes network. Theoretically, we establish diminishing bounds on the parameter distance, variable selection error, and Wasserstein distance. Our analysis shows that leveraging dependency sparsity leads to significant improvements in convergence rates. Empirically, experiments on both synthetic and real-world datasets demonstrate that PrAda-GAN outperforms existing tabular data synthesis methods in terms of the privacy-utility trade-off.
Read the full abstract

All authors Ke Jia, Yuheng Ma, Yang Li, Feifei Wang