Peer Revenue Now accepting positions

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Wednesday 15 April

Statistical Methodology

Forecasting Multivariate Time Series under Predictive Heterogeneity: A Validation-Driven Clustering Framework

We study adaptive pooling under predictive heterogeneity in high-dimensional multivariate time series forecasting, where global models improve statistical efficiency but may fail to capture heterogeneous predictive structure, while naive specialization can induce negative transfer. We formulate adaptive pooling as a statistical decision problem and propose a validation-driven framework that determines when and how specialization should be applied. Rather than grouping series based on representation similarity, we define partitions through out-of-sample predictive performance, thereby aligning data organization with predictive risk, defined as expected out-of-sample loss and approximated via validation error. Cluster assignments are iteratively updated using validation losses for both point (Huber) and probabilistic (pinball) forecasting, improving robustness to heavy-tailed errors and local anomalies. To ensure reliability, we introduce a leakage-free fallback mechanism that reverts to a global model whenever specialization fails to improve validation performance, providing a safeguard against performance degradation under a strict training-validation-test protocol. Experiments on large-scale traffic datasets demonstrate consistent improvements over strong baselines while avoiding degradation when heterogeneity is weak. Overall, the proposed framework provides a principled and practically reliable approach to adaptive pooling in high-dimensional forecasting problems.
Read the full abstract

All authors Ziling Ma, Ángel López Oriona, Hernando Ombao, Ying Sun

Machine Learning (Statistics) · Cross-listed from cs.LG

Online learning with noisy side observations

We propose a new partial-observability model for online learning problems where the learner, besides its own loss, also observes some noisy feedback about the other actions, depending on the underlying structure of the problem. We represent this structure by a weighted directed graph, where the edge weights are related to the quality of the feedback shared by the connected nodes. Our main contribution is an efficient algorithm that guarantees a regret of $\widetilde{O}(\sqrt{\alpha^* T})$ after $T$ rounds, where $\alpha^*$ is a novel graph property that we call the effective independence number. Our algorithm is completely parameter-free and does not require knowledge (or even estimation) of $\alpha^*$. For the special case of binary edge weights, our setting reduces to the partial-observability models of Mannor and Shamir (2011) and Alon et al. (2013) and our algorithm recovers the near-optimal regret bounds.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Spectral Thompson sampling

Thompson Sampling (TS) has attracted a lot of interest due to its good empirical performance, in particular in the computational advertising. Though successful, the tools for its performance analysis appeared only recently. In this paper, we describe and analyze SpectralTS algorithm for a bandit problem, where the payoffs of the choices are smooth given an underlying graph. In this setting, each choice is a node of a graph and the expected payoffs of the neighboring nodes are assumed to be similar. Although the setting has application both in recommender systems and advertising, the traditional algorithms would scale poorly with the number of choices. For that purpose we consider an effective dimension d, which is small in real-world graphs. We deliver the analysis showing that the regret of SpectralTS scales as d*sqrt(T ln N) with high probability, where T is the time horizon and N is the number of choices. Since a d*sqrt(T ln N) regret is comparable to the known results, SpectralTS offers a computationally more efficient alternative. We also show that our algorithm is competitive on both synthetic and real-world data.
Read the full abstract

All authors Tomas Kocak, Michal Valko, Remi Munos, Shipra Agrawal

Machine Learning (Statistics)

Covariance-adapting algorithm for semi-bandits with application to sparse rewards

We investigate stochastic combinatorial semi-bandits, where the entire joint distribution of outcomes impacts the complexity of the problem instance (unlike in the standard bandits). Typical distributions considered depend on specific parameter values, whose prior knowledge is required in theory but quite difficult to estimate in practice; an example is the commonly assumed sub-Gaussian family. We alleviate this issue by instead considering a new general family of sub-exponential distributions, which contains bounded and Gaussian ones. We prove a new lower bound on the expected regret on this family, that is parameterized by the unknown covariance matrix of outcomes, a tighter quantity than the sub-Gaussian matrix. We then construct an algorithm that uses covariance estimates, and provide a tight asymptotic analysis of the regret. Finally, we apply and extend our results to the family of sparse outcomes, which has applications in many recommender systems.
Read the full abstract

Statistical Methodology

Adaptive Sample Size Simulations with R package adsasi

Planning empirical experiments such as clinical trials or A/B tests requires sample size determination, which in many interesting cases has no closed-form solution (e.g. factorial or adaptive designs). adsasi is a new R package that enables simulations-first sample size calculations for any trial that can be simulated in short compute time. First, the user specifies as a function that takes a sample size as argument, simulates the experiment, and returns a boolean for success/failure. Then, adsasi functions adsasi_0d and adsasi_1d iteratively call it on different sample sizes and progressively home in on the one with nominal success rate (power), assuming that increasing sample size increases power. adsasi_1d can also draw, purely empirically, the relationship between a design parameter and sample size. The implementation uses a modified probit regression (with success/failure as the dependent variable), informed by simulations conducted around the target size, and provides standard errors at each stage using the Cramér-Rao bound derived from a custom analytical Hessian matrix. Simple examples are first presented, yielding results within Monte Carlo variance of their closed-form expressions, then intractable ones (including bootstrapping from an existing medical cohort). adsasi will hopefully facilitate the funding and conduct of interesting, highly complex experimental designs by making their sizing straightforward.
Read the full abstract

Statistical Methodology

Fractional lower-order covariance-based measures for cyclostationary time series with heavy-tailed distributions: application to dependence testing and model order identification

This article introduces new methods for the analysis of cyclostationary time series with infinite variance. Traditional cyclostationary analysis, based on periodically correlated (PC) processes, relies on the autocovariance function (ACVF). However, the ACVF is not suitable for data exhibiting a heavy-tailed distribution, particularly with infinite variance. Thus, we propose a novel framework for the analysis of cyclostationary time series with heavy-tailed distribution, utilizing the fractional lower-order covariance (FLOC) as an alternative to covariance. This leads to the introduction of two new autodependence measures: the periodic fractional lower-order autocorrelation function (peFLOACF) and the periodic fractional lower-order partial autocorrelation function (peFLOPACF). These measures generalize the classical periodic autocorrelation function (peACF) and periodic partial autocorrelation function (pePACF), offering robust tools for analyzing infinite-variance processes. Two practical applications of the proposed measures are explored: a portmanteau test for testing dependence in cyclostationary series and a method for order identification in periodic autoregressive (PAR) and periodic moving average (PMA) models with infinite variance. Both applications demonstrate the potential of new tools, with simulations validating their efficiency. The methodology is further illustrated through the analysis of real-world air pollution data, which showcases its practical utility. The results indicate that the proposed measures based on FLOC provide reliable and efficient techniques for analyzing cyclostationary processes with heavy-tailed distributions.
Read the full abstract

Statistics Theory · Cross-listed from cs.LG

Ordinary Least Squares is a Special Case of Transformer

The statistical essence of the Transformer architecture has long remained elusive: Is it a universal approximator, or a neural network version of known computational algorithms? Through rigorous algebraic proof, we show that the latter better describes Transformer's basic nature: Ordinary Least Squares (OLS) is a special case of the single-layer Linear Transformer. Using the spectral decomposition of the empirical covariance matrix, we construct a specific parameter setting where the attention mechanism's forward pass becomes mathematically equivalent to the OLS closed-form projection. This means attention can solve the problem in one forward pass, not by iterating. Building upon this prototypical case, we further uncover a decoupled slow and fast memory mechanism within Transformers. Finally, the evolution from our established linear prototype to standard Transformers is discussed. This progression facilitates the transition of the Hopfield energy function from linear to exponential memory capacity, thereby establishing a clear continuity between modern deep architectures and classical statistical inference.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning

Recent reinforcement learning (RL) approaches have advanced radiology report generation (RRG), yet two core limitations persist: (1) report-level rewards offer limited evidence-grounded guidance for clinical faithfulness; and (2) current methods lack an explicit self-improving mechanism to align with clinical preference. We introduce clinically aligned Evidence-aware Self-Correcting Reinforcement Learning (ESC-RL), comprising two key components. First, a Group-wise Evidence-aware Alignment Reward (GEAR) delivers group-wise, evidence-aware feedback. GEAR reinforces consistent grounding for true positives, recovers missed findings for false negatives, and suppresses unsupported content for false positives. Second, a Self-correcting Preference Learning (SPL) strategy automatically constructs a reliable, disease-aware preference dataset from multiple noisy observations and leverages an LLM to synthesize refined reports without human supervision. ESC-RL promotes clinically faithful, disease-aligned reward and supports continual self-improvement during training. Extensive experiments on two public chest X-ray datasets demonstrate consistent gains and state-of-the-art performance.
Read the full abstract

All authors Qin Zhou, Guoyan Liang, Qianyi Yang, Jingyuan Chen, Sai Wu, Chang Yao, Zhe Wang

Statistics Theory · Cross-listed from math.NA

Covariance-Informed Subspace: an Adaptive Gradient-Free Input Dimension Reduction Method for Bayesian Inference

This paper addresses the challenge of dimension reduction (DR) in Bayesian inference of high-resolution two-or three-dimensional fields, where a priori parametrizations require a large number of terms. The underlying idea is common to state-of-the-art methods in which the parameter space is decomposed into two subspaces, one informed by the likelihood and one constrained by the prior. DR techniques generally use gradient information from the log-likelihood to derive the corresponding subspaces. However, the gradient may be unavailable or expensive to compute accurately, for instance in the case of simulation-based inference. Inspired by approaches based on likelihood-informed subspaces, we develop a new DR method tailored for settings where gradient computation is not feasible. More specifically, we propose a gradient-free indicator for determining whether a direction is informed by the data. This indicator is derived from the posterior-to-prior covariance ratio introduced in Spantini et al. (2015). We show that, in the linear Gaussian case, this indicator combined with an approximate likelihood leads to a better posterior approximation. The method is then extended to nonlinear cases, and strategies to approximate the posterior covariance are detailed. We demonstrate the effectiveness of this DR through two high-dimensional inference problems arising from groundwater and atmospheric applications.
Read the full abstract

All authors Nadège Polette, Olivier Le Maître, Pierre Sochala, Alexandrine Gesret

Statistics Applications

Relative plausibility versus probabilism: A level-of-analysis error in juridical proof

Debates about juridical proof are often framed as a conflict between probabilistic approaches and relative plausibility theory (RPT). This paper argues that this opposition rests on a level-of-analysis error. Drawing on Marr's distinction between levels of analysis, we show that RPT and probabilistic approaches operate at different conceptual levels and are therefore compatible rather than competing theories. RPT provides a computational-level description of juridical proof, characterizing the task of comparing explanations in light of the evidence and assessing whether a standard of proof has been met. Probabilistic approaches supply algorithmic-level accounts that specify how such comparative assessments can be represented and computed. When plausibility judgments satisfy minimal coherence conditions, relative plausibility corresponds to posterior odds. Recognizing this distinction clarifies longstanding disputes and highlights the complementary roles of explanation and probability in legal reasoning.
Read the full abstract

Machine Learning (Statistics)

Robust Low-Rank Tensor Completion based on M-product with Weighted Correlated Total Variation and Sparse Regularization

The robust low-rank tensor completion problem addresses the challenge of recovering corrupted high-dimensional tensor data with missing entries, outliers, and sparse noise commonly found in real-world applications. Existing methodologies have encountered fundamental limitations due to their reliance on uniform regularization schemes, particularly the tensor nuclear norm and $\ell_1$ norm regularization approaches, which indiscriminately apply equal shrinkage to all singular values and sparse components, thereby compromising the preservation of critical tensor structures. The proposed tensor weighted correlated total variation (TWCTV) regularizer addresses these shortcomings through an $M$-product framework that combines a weighted Schatten-$p$ norm on gradient tensors for low-rankness with smoothness enforcement and weighted sparse components for noise suppression. The proposed weighting scheme adaptively reduces the thresholding level to preserve both dominant singular values and sparse components, thus improving the reconstruction of critical structural elements and nuanced details in the recovered signal. Through a systematic algorithmic approach, we introduce an enhanced alternating direction method of multipliers (ADMM) that offers both computational efficiency and theoretical substantiation, with convergence properties comprehensively analyzed within the $M$-product framework.Comprehensive numerical evaluations across image completion, denoising, and background subtraction tasks validate the superior performance of this approach relative to established benchmark methods.
Read the full abstract

Machine Learning (Statistics)

Joint Representation Learning and Clustering via Gradient-Based Manifold Optimization

Clustering and dimensionality reduction have been crucial topics in machine learning and computer vision. Clustering high-dimensional data has been challenging for a long time due to the curse of dimensionality. For that reason, a more promising direction is the joint learning of dimension reduction and clustering. In this work, we propose a Manifold Learning Framework that learns dimensionality reduction and clustering simultaneously. The proposed framework is able to jointly learn the parameters of a dimension reduction technique (e.g. linear projection or a neural network) and cluster the data based on the resulting features (e.g. under a Gaussian Mixture Model framework). The framework searches for the dimension reduction parameters and the optimal clusters by traversing a manifold,using Gradient Manifold Optimization. The obtained The proposed framework is exemplified with a Gaussian Mixture Model as one simple but efficient example, in a process that is somehow similar to unsupervised Linear Discriminant Analysis (LDA). We apply the proposed method to the unsupervised training of simulated data as well as a benchmark image dataset (i.e. MNIST). The experimental results indicate that our algorithm has better performance than popular clustering algorithms from the literature.
Read the full abstract

Statistics Applications · Cross-listed from math.OC

Deepbullwhip: An Open-Source Simulation and Benchmarking for Multi-Echelon Bullwhip Analyses

The bullwhip effect remains operationally persistent despite decades of analytical research. Two computational deficiencies hinder progress: the absence of modular open-source simulation tools for multi-echelon inventory dynamics with asymmetric costs, and the lack of a standardized benchmarking protocol for comparing mitigation strategies across shared metrics and datasets. This paper introduces deepbullwhip, an open-source Python package that integrates a simulation engine for serial supply chains (with pluggable demand generators, ordering policies, and cost functions via abstract base classes, and a vectorized Monte Carlo engine achieving 50 to 90 times speedup) with a registry-based benchmarking framework shipping a curated catalog of ordering policies, forecasting methods, six bullwhip metrics, and demand datasets including WSTS semiconductor billings. Five sets of experiments on a four-echelon semiconductor chain demonstrate cumulative amplification of 427x (Monte Carlo mean across 1,000 paths), a stochastic filtering phenomenon at upstream tiers (CV = 0.01), super-exponential lead time sensitivity, and scalability to 20.8 million simulation cells in under 7 seconds. Benchmark experiments reveal a 155x disparity between synthetic AR(1) and real WSTS bullwhip severity under the Order-Up-To policy, and quantify the BWR-NSAmp tradeoff across ordering policies, demonstrating that no single metric captures policy quality.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Universality of Gaussian-Mixture Reverse Kernels in Conditional Diffusion

We prove that conditional diffusion models whose reverse kernels are finite Gaussian mixtures with ReLU-network logits can approximate suitably regular target distributions arbitrarily well in context-averaged conditional KL divergence, up to an irreducible terminal mismatch that typically vanishes with increasing diffusion horizon. A path-space decomposition reduces the output error to this mismatch plus per-step reverse-kernel errors; assuming each reverse kernel factors through a finite-dimensional feature map, each step becomes a static conditional density approximation problem, solved by composing Norets' Gaussian-mixture theory with quantitative ReLU bounds. Under exact terminal matching the resulting neural reverse-kernel class is dense in conditional KL.
Read the full abstract

All authors Nafiz Ishtiaque, Syed Arefinul Haque, Kazi Ashraful Alam, Fatima Jahara

Statistics Applications · Cross-listed from physics.ao-ph

Modeling the Sea-Level Change from U.S. Vehicle Emissions

Recent U.S. Environmental Protection Agency (EPA) analyses have argued that greenhouse gas emissions from U.S. on-road vehicles contribute negligibly to global mean sea-level rise (GMSLR). Here, I replicate and extend the EPA's modeling framework using the FaIR climate model coupled with the BRICK sea-level model, incorporating a probabilistic weighting approach and a longer model timescale to better represent joint climate-sea-level uncertainty. In addition to the baseline SSP2-4.5 scenario and an EPA-consistent emissions reduction case, I examine alternative scenarios reflecting stalled technological progress and a counterfactual pre-regulation vehicle fleet. Results reproduce EPA estimates of approximately 1-2 cm of GMSLR reduction by 2100 under vehicle emissions mitigation but show that these differences grow substantially over multi-century timescales, exceeding 6 cm by 2200. Downscaling to U.S. coastlines reveals larger local effects, particularly along the Gulf of Mexico Coast. These findings highlight the long-term and regionally amplified benefits of emissions reductions from the transportation sector.
Read the full abstract

Statistical Methodology

Estimating Continuous Treatment Effects with Two-Stage Kernel Ridge Regression

We study the problem of estimating the effect function for a continuous treatment, which maps each treatment value to a population-averaged outcome. A central challenge in this setting is confounding: treatment assignment often depends on covariates, creating selection bias that makes direct regression of the response on treatment unreliable. To address this issue, we propose a two-stage kernel ridge regression method. In the first stage, we learn a model for the response as a function of both treatment and covariates; in the second stage, we use this model to construct pseudo-outcomes that correct for distribution shift, and then fit a second model to estimate the treatment effect. Although the response varies with both treatment and covariates, the induced effect function obtained by averaging over covariates is typically much simpler, and our estimator adapts to this structure. Our optimal learning bounds are achieved without estimating the conditional treatment density, thereby bypassing a major bottleneck in existing methods. Furthermore, we introduce a fully data-driven model selection procedure that achieves provable adaptivity to both the unknown degree of overlap and the spectral decay of the underlying kernel.
Read the full abstract

Statistical Methodology

Leveraging machine learning to estimate individualized treatment effects in cluster-randomized trials

Cluster-randomized trials (CRTs) are widely used to evaluate interventions delivered at the clinic, practice, or community level. Although standard analyses typically target average treatment effects, such summaries mask potentially meaningful variation in treatment response across individuals and clusters. This work addresses the estimation of conditional average treatment effects (CATEs) for continuous outcomes in two-arm parallel CRTs by defining causal estimands that incorporate both individual- and cluster-level baseline covariates while marginalizing over unobserved cluster heterogeneity. To estimate these quantities, we develop a unified framework based on mixed-effects machine learning, integrating and extending a range of existing approaches, including Bayesian additive regression trees with random effects, multilevel Bayesian causal forests, mixed-effects random forests, several mixed-effects gradient boosting procedures, and generalized additive mixed models, while incorporating cluster-specific random intercepts to account for within-cluster dependence. We evaluate these methods across diverse simulation scenarios and demonstrate their use in the Task Shifting and Blood Pressure Control in Ghana CRT, which investigates strategies for improving hypertension management. Drawing on these investigations, we provide practical guidance for applying mixed-effects machine learning to quantify treatment-effect heterogeneity in CRTs, together with reproducible code that enables investigators to implement all methods within a coherent workflow.
Read the full abstract

All authors Changjun Li, Xi Fang, Michael O. Harhay, Andrew B. Forbes, F. Perry Wilson, Guangyu Tong, Fan Li

Machine Learning (Statistics) · Cross-listed from math.OC

A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity

Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically away from their minimizers. The argument is intricate, relying on monitoring the performance of the algorithm relative to a certain manifold of slow growth – called the ravine. In this work, we provide a direct Lyapunov-based argument that bypasses these difficulties when the objective is in addition convex and a has a unique minimizer. As a byproduct of the argument, we obtain a more adaptive variant than the original algorithm with encouraging numerical performance.
Read the full abstract

Tuesday 14 April

Statistics Applications

Cyclic fractional Gaussian noise: time and frequency domain properties

This article introduces cyclic fractional Gaussian noise (cfGn), a stochastic model that integrates second-order cyclostationarity with long-range dependence property. While classical cyclostationary processes are widely discussed in the literature, they often lack the capacity to account for the persistent, slow-decaying correlations found in complex empirical data. To bridge this gap, we extend the amplitude-modulated stationary framework by utilizing increments of two-dimensional fractional Brownian motion (2d fBm) as the underlying driving process. The proposed cfGn model is constructed by summing two components, which include periodic deterministic functions modulating the univariate coordinates of 2d fGn. We provide a rigorous derivation of the considered model's properties, specifically the autocovariance function (ACVF) and frequency-domain characteristics, including the cyclic spectrum. Through theoretical considerations of asymptotic properties and Monte Carlo simulations, we demonstrate that cfGn preserves periodic behavior of ACVF while inheriting long-memory traits which is manifested in time and frequency domains. This framework offers a robust foundation, for instance, in signal-based condition monitoring in systems where periodic fault components coexist with long-range dependent background noise.
Read the full abstract

Statistics Applications · Cross-listed from cs.CY

Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces

STEM dropout rates remain high at universities, particularly in computer science programs with theory-intensive courses. Digital learning environments now capture rich behavioral data that could help identify struggling students early, yet the generalizability of data-driven prediction models across courses and institutions remains uncertain. Guided by self-regulated learning (SRL) theory, this study analyzed multimodal digital-trace data from three undergraduate theoretical computer science courses (N1 = 137, N2 = 104, N3 = 148) at two universities. Weekly SRL-aligned digital-trace indicators were modeled using Elastic Net, Random Forest, and XGBoost to evaluate predictive performance over time and across settings, and model calibration both within and across courses. Early prediction of at-risk students was feasible, with SRL-related behaviors such as time management, effort regulation, and sustained engagement emerging as key predictors. While Random Forest achieved the highest in-sample accuracy, Elastic Net generalized more robustly across contexts. Out-of-sample accuracy and calibration declined between institutions with different base rates, underscoring the contextual nature of predictive analytics in higher education. These findings suggest that digital learning traces enable early identification of at-risk students within courses, but generalizing predictive models beyond their original context requires caution, particularly if the at-risk rates differ between contexts.
Read the full abstract

All authors Jakob Schwerter, Loreen Sabel, Judith Bose, Matthew L. Bernacki, Di Xu, Marko Schmellenkamp, Thomas Zeume, Philipp Doebler

Machine Learning (Statistics) · Cross-listed from math.OC

Sliced Wasserstein Steering between Gaussian Measures

Optimal transport with quadratic cost provides a geometric framework for steering an ensemble, modeled by a probability law, with minimal effort. Yet ambient-space formulations become unwieldy in high dimensions, and sensing or actuation in practice often reveals only linear views of the state – camera silhouettes, LiDAR beams, tomographic slices. We develop a sliced feedback controller for distribution steering: the evolving law is projected onto one-dimensional directions on the sphere, the optimal one-dimensional velocity is synthesized in each projection, and these velocities are averaged to produce a feedback control in the ambient space. The construction reduces to the Benamou–Brenier problem in one dimension. In addition, it is invariant under orthogonal transforms, nonexpansive under projections, and well posed on $\mathcal{P}_2(\mathbb{R}^n)$. Computation proceeds by sampling directions on the sphere and solving independent one-dimensional subproblems, yielding a scalable method aligned with partial observations. In the Gaussian setting, we show that the developed sliced controller steers the law to the prescribed target. Furthermore, we derive an identity relating the energy consumption incurred by the controller to the sliced Wasserstein distance.
Read the full abstract

Statistics Applications

A Statistical Market-Design Framework for Academic Job Markets

The academic job market for new statisticians is highly congested at the interview stage, where departments must rank and select candidates from large applicant pools without credible signals of candidate interest. As a result, interviews and offers are often misallocated, leading to unfilled positions and poor mutual fit. We frame interview allocation as a statistical ranking problem under uncertainty and propose a market-design framework that incorporates structured preference signaling into interview selection. Candidates submit a single standardized questionnaire describing preferences over interpretable job characteristics, which departments combine with traditional application materials and historical hiring data to estimate candidate-specific acceptance probabilities and expected utilities. To account for estimation uncertainty, we employ a confidence-calibrated ranking procedure based on pairwise utility comparisons that provides statistical guarantees for candidate ranking. We establish that truthful participation is optimal for candidates and that preference information improves departmental outcomes and matching stability. We use a dataset of U.S. statistics departments to show that the proposed framework substantially increases matching rates, improves match quality, and reduces hiring failures relative to the current practice.
Read the full abstract

Statistics Applications

A Hybrid Statistical Learning Framework for Capability-Based Decision Support in Manufacturing

Process capability indices are widely used in manufacturing quality control, but capability approval is often implemented by directly thresholding finite-sample estimates, which can produce unstable and poorly calibrated decisions near the approval boundary. This paper develops a hybrid statistical–learning framework for capability-based decision support in manufacturing. The proposed UC-Cap approach combines a statistically grounded capability baseline with a residual learning component that uses process, distributional, specification-related, and measurement-related features to refine capability-decision risk estimates under non-ideal manufacturing conditions. The statistical baseline preserves the interpretability of classical capability analysis, while the learning component provides data-driven correction for systematic deviations arising from non-normality, measurement effects, and finite-sample variability. A nested Monte Carlo evaluation is introduced to assess probabilistic calibration under controlled synthetic settings, and an empirical manufacturing study is used to examine decision behavior under realistic capability data. Results show that deterministic thresholding can lead to substantial instability and miscalibration in near-boundary regimes, whereas the proposed framework provides calibrated risk estimates, interpretable decision outputs, and improved support for capability approval decisions. The framework is compatible with existing capability-analysis workflows and can be integrated into manufacturing quality decision-support systems.
Read the full abstract

Statistical Methodology

Newton's Algorithm as a Gradient Flow: A Geometric Framework for Recursive Mixture Estimation

Bayesian nonparametric mixture models provide a flexible framework for data analysis but are often hindered by the computational expense of traditional inference methods like MCMC. A fast, recursive algorithm proposed by Newton (2002) offers a practical alternative, yet its formal connection to Bayesian inference and its theoretical properties remain only partially understood. This paper reveals a new geometric interpretation of this class of predictive recursions. We demonstrate that Newton's recursion is a discrete-time approximation of a gradient flow on the space of probability measures governed by the Fisher-Rao geometry, providing the first rigorous dynamical characterisation of this family of estimators. This geometric perspective provides a principled theoretical foundation for studying these recursions: it clarifies their convergence behaviour, situates them within the variational Bayes literature, and yields a systematic basis for generalisation by modifying the underlying geometry and discretisation. In contrast to approaches that construct gradient flows from a prescribed variational objective, this work proceeds in the reverse direction: beginning from an existing recursive estimator and uncovering the variational problem it implicitly solves, it opens a pathway for the systematic analysis and extension of a broad class of sequential Bayesian estimators.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Some Theoretical Limitations of t-SNE

t-SNE has gained popularity as a dimension reduction technique, especially for visualizing data. It is well-known that all dimension reduction techniques may lose important features of the data. We provide a mathematical framework for understanding this loss for t-SNE by establishing a number of results in different scenarios showing how important features of data are lost by using t-SNE.
Read the full abstract

Statistical Methodology

Addressing Confounding by Indication Through (Un)Measured Centre Characteristics in Learn-As-you-GO(LAGO) Trials

The Learn-As-you-Go (LAGO) design is an adaptive clinical trial design allowing modifications to multicomponent intervention packages across stages. Centers participate in more than one stage, as is common in large-scale implementation trials. In LAGO trials, center characteristics may act as confounders, predicting both the intervention package and the outcomes. We extend LAGO theory by introducing fixed center effects to control for confounding by indication through measured and unmeasured center characteristics. Conditioning on center characteristics by including fixed center effects ensures asymptotic results hold without requiring explicit characterization of unmeasured confounders. Our methods apply even with small numbers of centers. LAGO theory is established for continuous outcomes following a generalized linear model and binary outcomes following a logistic regression model, unifying theory across outcome types. Point- and interval estimators are derived, and consistency and asymptotic normality are established. Valid hypothesis tests for the overall intervention effect are provided, and the optimal intervention package minimizing cost subject to a target outcome mean is obtained via constrained optimization.
Read the full abstract

All authors Minh Thu Bui, Christopher T. Longenecker, Ante Bing, Donna Spiegelman, Allison R. Webel, Hayden B. Bosworth, Judith J. Lok

Statistics Theory

Sequential Change Detection for Multiple Data Streams with Differential Privacy

Sequential change-point detection seeks to rapidly identify distributional changes in streaming data while controlling false alarms. Existing multi-stream detection methods typically rely on non-private access to raw observations or intermediate statistics, limiting their usage in privacy-sensitive settings. We study sequential change-point detection for multiple data streams under differential privacy constraints. We consider multiple independent streams undergoing a synchronized change at an unknown time and in an unknown subset of streams, and propose DP-SUM-CUSUM, a differentially private detection procedure based on the summation of per-stream CUSUM statistics with calibrated Laplace noise injection. We show that DP-SUM-CUSUM satisfies sequential $\varepsilon$-differential privacy and derive bounds on the average run length to false alarm and the worst-case average detection delay, explicitly characterizing the privacy–efficiency tradeoff. A truncation-based extension is also presented to handle distributional shifts with unbounded log-likelihood ratios. Simulations and experiments on an Internet of Things (IoT) botnet dataset validate the proposed approach.
Read the full abstract

Statistical Methodology

Efficient estimation of cumulative incidence curves via data fusion with surrogates: application to integrated analysis of vaccine trial and immunobridging data

Refined vaccine regimens containing variant-matched inserts are often authorized based on historical phase 3 efficacy trials together with immunobridging studies. Phase 3 trials are essential for establishing immune biomarkers that reliably predict disease risk or vaccine efficacy against clinical endpoints. Once such immune correlates are identified, updated vaccine regimens can be approved through immunobridging designs that compare the immunogenicity of the updated regimen to that of an already-approved vaccine. We develop methods of inference for the counterfactual cumulative incidence curve using participant-level data from both a historical vaccine efficacy trial and an immunobridging study. We further extend these methods to pathogens with multiple serotypes – such as dengue virus and influenza – by estimating cause-specific cumulative incidence curves. We describe the identification assumptions, propose efficient and multiply robust estimators, and assess their finite-sample performance through simulation studies. We then apply the proposed methods to (1) estimating the hypothetical cumulative incidence curve for a bivalent mRNA booster and (2) testing a key assumption of no controlled direct effects, using data from the COVID-19 Variant Immunologic Landscape (COVAIL) Trial, a multistage randomized clinical study evaluating the safety and immunogenicity of a second COVID-19 booster dose.
Read the full abstract

All authors Pan Zhao, Peter B. Gilbert, Oliver Dukes, Bo Zhang

Statistical Methodology

Estimating effect thresholds and beyond: A flexible framework for multivariate alert detection

Evaluating the influence of continuous covariates, like exposure time or dose, on a response variable is a pivotal objective in the assessment of a compound's effect, particularly when determining toxicity in pre-clinical research or pharmacokinetics in clinical trials. The determination of an alert, such as the ED50 value, at which a pre-specified threshold of the response variable is crossed, is an important tool for the evaluation process. In practice, response data might be available for combinations of different covariates and the alert depending on both is of interest. In this case, it is crucial to use all available information and extrapolate between cases to ensure the optimal utilization of the data. In this paper, we introduce a parametric approach that allows alerts to be estimated in a multidimensional setting. For time-dose-response data, for instance, alert doses at a given time can be determined, even when there are no measurements available at that exact time. Likewise, it allows estimation of alert times for a given dose. More generally, the method makes it possible to characterize the complete alert relationship between covariates by leveraging all available data. This is achieved by fitting a parametric model and constructing either a confidence band for the two-dimensional curve given for example a fixed time or dose or by constructing a confidence plane for the three-dimensional model fit. The initial model fit is achieved by the flexible framework of Generalized Additive Models for Location, Scale and Shape (GAMLSS), which offers the possibility to account for a plethora of complex three-dimensional data structures. We demonstrate the validity of our approach through a simulation study and present an application to data from a study investigating the relevance of the exposure duration on cytotoxicity in primary human hepatocytes.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

Bias-Corrected Adaptive Conformal Inference for Multi-Horizon Time Series Forecasting

Adaptive Conformal Inference (ACI) provides distribution-free prediction intervals with asymptotic coverage guarantees for time series under distribution shift. However, ACI only adapts the quantile threshold – it cannot shift the interval center. When a base forecaster develops persistent bias after a regime change, ACI compensates by widening intervals symmetrically, producing unnecessarily conservative bands. We propose Bias-Corrected ACI (BC-ACI), which augments standard ACI with an online exponentially weighted moving average (EWM) estimate of forecast bias. BC-ACI corrects nonconformity scores before quantile computation and re-centers prediction intervals, addressing the root cause of miscalibration rather than its symptom. An adaptive dead-zone threshold suppresses corrections when estimated bias is indistinguishable from noise, ensuring no degradation on well-calibrated data. In controlled experiments across 688 runs spanning two base models, four synthetic regimes, and three real datasets, BC-ACI reduces Winkler interval scores by 13–17% under mean and compound distribution shifts (Wilcoxon p < 0.001) while maintaining equivalent performance on stationary data (ratio 1.002x). We provide finite-sample analysis showing that coverage guarantees degrade gracefully with bias estimation error.
Read the full abstract