Peer Revenue Now accepting positions

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Tuesday 14 April

Statistical Computation

Multi-Object Posterior Computation via Gibbs Sampling

This work presents a tractable approach to multi-object posterior computation under a generic measurement likelihood function. While filtering is a popular solution, valuable historical information is discarded. Posterior inference, which captures the full history of the multi-object states, provides a more comprehensive solution but is notoriously difficult and has received limited attention. Our proposed approach uses Gibbs Sampling (GS) to generate samples from the multi-object posterior. In particular, we establish that the conditional distributions of the multi-object posterior are Bernoulli random finite sets with explicit existence probabilities and attribute densities. These conditionals are straightforward to evaluate and sample from, enabling the construction of an efficient Gibbs sampler with standard convergence guarantees. To demonstrate its versatility, we develop the first multi-scan multi-object smoothing algorithm for superpositional measurements. Numerical experiments show that the proposed method delivers robust performance in challenging low-SNR scenarios where detection based smoothing deteriorates. Moreover, posterior samples obtained from our approach provide statistical characterizations of key variables and parameters, highlighting the advantages of posterior inference. This approach enriches multi-object estimation techniques, which historically lacked smoothing capabilities for non-standard measurements.
Read the full abstract

Machine Learning (Statistics)

A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learning

In-context learning enables transformers to adapt to new tasks from a few examples at inference time, while grokking highlights that this generalization can emerge abruptly only after prolonged training. We study task generalization and grokking in in-context learning using a Bayesian perspective, asking what enables the delayed transition from memorization to generalization. Concretely, we consider modular arithmetic tasks in which a transformer must infer a latent linear function solely from in-context examples and analyze how predictive uncertainty evolves during training. We combine approximate Bayesian techniques to estimate the posterior distribution and we study how uncertainty behaves across training and under changes in task diversity, context length, and context noise. We find that epistemic uncertainty collapses sharply when the model groks, making uncertainty a practical label-free diagnostic of generalization in transformers. Additionally, we provide theoretical support with a simplified Bayesian linear model, showing that asymptotically both delayed generalization and uncertainty peaks arise from the same underlying spectral mechanism, which links grokking time to uncertainty dynamics.
Read the full abstract

Statistical Methodology

A sub-asymptotic model for bivariate threshold exceedances

Extreme value theory offers a statistical framework for quantifying the risk of rare events, with the generalized Pareto (GP) distribution providing the canonical limit model for univariate threshold exceedances. In many applications, however, extremes are intrinsically multivariate, requiring models that capture both marginal tail behaviours and joint extremal dependencies. Under asymptotic dependence, the multivariate GP distribution represents a suitable modelling family, but when asymptotic independence arises, sub-asymptotic models are needed. In this work, we propose and study a flexible sub-asymptotic parametric class to model bivariate threshold exceedances. Our new model accommodates a broad range of tail dependence behaviours and contains the standardised multivariate GP distribution as a limiting case while retaining margins that converge to univariate GP tails. Our formulation allows extremal dependence to evolve naturally with the marginal parameters on the original data scale, facilitating direct computation and interpretation of failure probabilities. Model inference is done via a likelihood-free neural Bayes estimation approach, with tailored prior specifications. An extensive simulation study and an application to Belgian rainfall extremes illustrate the estimation framework and the flexibility of the model.
Read the full abstract

Machine Learning (Statistics)

Information-Geometric Decomposition of Generalization Error in Unsupervised Learning

We decompose the Kullback–Leibler generalization error (GE) – the expected KL divergence from the data distribution to the trained model – of unsupervised learning into three non-negative components: model error, data bias, and variance. The decomposition is exact for any e-flat model class and follows from two identities of information geometry: the generalized Pythagorean theorem and a dual e-mixture variance identity. As an analytically tractable demonstration, we apply the framework to $\epsilon$-PCA, a regularized principal component analysis in which the empirical covariance is truncated at rank $N_K$ and discarded directions are pinned at a fixed noise floor $\epsilon$. Although rank-constrained $\epsilon$-PCA is not itself e-flat, it admits a technical reformulation with the same total GE on isotropic Gaussian data, under which each component of the decomposition takes closed form. The optimal rank emerges as the cutoff $\lambda_{\mathrm{cut}}^{*} = \epsilon$ – the model retains exactly those empirical eigenvalues exceeding the noise floor – with the cutoff reflecting a marginal-rate balance between model-error gain and data-bias cost. A boundary comparison further yields a three-regime phase diagram – retain-all, interior, and collapse – separated by the lower Marchenko–Pastur edge and an analytically computable collapse threshold $\epsilon_{*}(\alpha)$, where $\alpha$ is the dimension-to-sample-size ratio. All claims are verified numerically.
Read the full abstract

Statistical Computation · Cross-listed from math.PR

On additive averaging kernels for finite Markov chains

We study additive mixtures of Markov kernels of the form $A_\alpha = \alpha P + (1-\alpha)G$, where $\alpha \in [0,1]$, $P$ is a baseline sampler and $G$ is a Gibbs kernel induced by a partition of the state space. We first motivate the study of $A_\alpha$, which can be interpreted as the projection of a lifted Markov chain. We then consider the minimisation of distance to stationarity under two objectives: the squared Frobenius norm and the Kullback-Leibler (KL) divergence. For the Frobenius objective, we derive explicit trace formulae and identify a Cheeger-type functional that characterises optimal two-block partitions. This yields a structured combinatorial optimisation problem admitting a difference-of-submodular decomposition, enabling efficient approximation via majorisation-minimisation. We also obtain geometric decay rates governed by the absolute spectral gap of $P$. For the KL divergence, we establish convexity-based bounds showing that the divergence of $A_\alpha$ is controlled by those of both $P$ and $G$, thereby reducing partition selection to the Gibbs component. Numerical experiments on the Curie-Weiss model demonstrate that suitable choice of both the partition and the parameter $\alpha$ can significantly accelerate convergence in total variation distance. We observe a consistent trade-off between local exploration and global averaging, with intermediate values of $\alpha$ achieving the best performance across regimes.
Read the full abstract

Statistical Methodology

An Empirical Comparison of Methods for Quantifying the Similarity of Numeric Datasets

Methods for quantifying the similarity of datasets are relevant in applications where two or more datasets, or their underlying distributions, need to be compared, ranging from two- and k-sample testing to applications in machine learning and synthetic data generation. Many methods for quantifying the similarity of datasets are available from the literature, but due to the lack of neutral comparison studies, it is unclear which method to choose when. Here, 36 methods applicable to continuous data are compared across various scenarios, including two or more datasets drawn from different distributions. Several deviations between datasets are considered, including shift and scale alternatives or differences in higher moments. An overall method ranking is established based on the methods' abilities to differentiate between datasets from different distributions, combined with computational aspects. Based on this, concrete decision rules for finding the best method based on characteristics of the datasets are determined. Moreover, combinations of four to six methods are proposed in the two-sample case such that in 90% to 95% of the considered scenarios, at least one of these methods is almost as good as the best method. In the multi-sample case, a combination of two to three methods is proposed analogously.
Read the full abstract

Statistics Applications

HICM: An approach towards Harmonizing Indian Census Migration data and its applications

Reliable analysis of migration is critically dependent on the quality and consistency of the underlying data. Indian migration data, primarily derived from decennial census records, are affected by systematic gaps arising from uneven coverage and measurement inconsistencies across states and time. This paper presents a data-centric framework, HICM, for harmonizing Indian census migration data recorded under the Indian census and correcting prominent sources of bias prior to downstream analyses. We explicitly identify two types of bias across three decades of migration data: measurement bias and representativeness bias. We propose to address these gaps through principled pre-processing, mitigation, and validation strategies grounded in statistical diagnostics. An empirical evaluation of harmonized Indian interstate migration data reveals that bias-aware data correction substantially improves the consistency in the structure of the data and enhances the reliability of subsequent temporal analysis results. By improving data quality through reproducible data imputation and smoothing, this work advances migration analytics and provides a robust foundation for policy-relevant longitudinal network analysis of Indian internal migration.
Read the full abstract

Statistics Applications

Racial Comparability in Authoritarianism Scales: Latent Beliefs or Biased Measurement?

Racial differences in authoritarianism are widely used to explain variation in political attitudes, yet it is unclear whether they reflect true latent differences or measurement artifacts. Using anchor-based multi-group confirmatory factor analysis across multiple nationally representative surveys, this paper examines measurement equivalence in the standard child-rearing authoritarianism battery. We find systematic differences in how respondents use response categories across groups. Accounting for this non-invariance alters but does not eliminate racial differences in authoritarianism; African Americans continue to exhibit higher latent authoritarianism under partial scalar invariance. However, conventional multi-item scales substantially attenuate the association between authoritarianism and policy attitudes. These results show that measurement non-invariance is not merely a technical concern but can meaningfully shape substantive inferences about racial differences and their political consequences, underscoring the importance of explicit measurement modeling in studies of public opinion and political behavior.
Read the full abstract

Statistics Applications · Cross-listed from q-bio.QM

The IQ-Motion Confound in Multi-Site Autism fMRI May Be Inflated by Site-Correlated Measurement Uncertainty

Multi-site autism neuroimaging studies routinely control for the confound between full-scale IQ and head motion by regressing framewise displacement against IQ scores and removing shared variance. This procedure assumes that ordinary least squares (OLS) provides an unbiased estimate of the confound magnitude. We tested this assumption on the ABIDE-I phenotypic dataset (n=935 subjects across 19 international scanning sites) using Probability Cloud Regression, an errors-in-variables (EIV) estimator that models per-observation measurement uncertainty in both variables. IQ measurement error was derived from published Wechsler test-retest reliability coefficients; response-side uncertainty was represented by a site-level proxy equal to the within-site standard deviation of mean framewise displacement. Three findings emerged. First, OLS overestimates the IQ-motion slope by a factor of 4.67 relative to the EIV-corrected estimate when the bias factor is computed from the full-precision fitted coefficients (OLS -0.00125, EIV -0.00027 mm per IQ point after rounding for display). Second, under leave-site-out cross-validation a single pooled predictor of raw FD produces negative out-of-sample R^2 at all 19 sites (overall R^2 = -0.074), indicating that the pooled predictor does not transport cleanly across sites once site information is removed. Third, the direction of the EIV-corrected slope is robust across all 64 configurations of an 8x8 sensitivity grid spanning 12-fold ranges of each noise parameter. These results suggest that pooled OLS may overstate the IQ-motion association in ABIDE-I, but direct downstream consequences for motion-correction pipelines remain to be quantified using raw motion traces and connectivity-level re-analysis. Formal EIV methods appear to remain uncommon in multi-site neuroimaging confound estimation.
Read the full abstract

Machine Learning (Statistics)

SMART Fine-tuning Factor Augmented Neural Lasso

Fine-tuning is a widely used strategy for adapting pre-trained models to new tasks, yet its methodology and theoretical properties in high-dimensional nonparametric settings with variable selection have not yet been developed. We propose a source-model-augmented residual tuning (SMART) framework, which incorporates the pre-trained source model as an augmented feature into the target learner and estimates only the residual target-specific component. The approach is widely applicable, from parametric and sparse models to neural networks and blackbox machine learning models. We focus on the development of fine-tuning factor-augmented neural Lasso, resulting in SMART-FAN-Lasso. This transfer-learning framework for high-dimensional nonparametric regression with variable selection simultaneously handles covariate and posterior shifts. We use a low-rank factor structure to manage high-dimensional dependent covariates and a residual tuning decomposition in which the target function is expressed as a function of source model and other target-specific variables, thereby reducing the effective complexity of the target task. We derive minimax-optimal excess risk bounds, characterizing the precise conditions, in terms of relative sample sizes and function complexities, under which fine-tuning yields statistical acceleration over single-task learning. Extensive numerical experiments across diverse covariate- and posterior-shift scenarios demonstrate that SMART-FAN-Lasso consistently outperforms standard baselines and achieves near-oracle performance even under severe target sample size constraints, empirically validating the derived rates.
Read the full abstract

All authors Jinhang Chai, Jianqing Fan, Cheng Gao, Qishuo Yin

Statistical Methodology

Partial Identification of Policy-Relevant Treatment Effects with Instrumental Variables via Optimal Transport

Policy-Relevant Treatment Effects (PRTEs) are generally not point-identified under standard Instrumental Variable (IV) assumptions when the instrument generates limited support in treatment propensity. We show that PRTE partial identification in the generalized Roy model can instead be formulated as a Constrained Conditional Optimal Transport (CCOT) problem over the joint conditional law of the potential outcome and the latent resistance. The resulting multidimensional CCOT problem reduces analytically to separable one-dimensional OT problems with product costs, yielding sharp closed-form bounds and avoiding direct solution of the original high-dimensional CCOT problem. We also develop estimation and inference procedures for these bounds: for discrete instruments, we use a Double Machine Learning (DML) approach based on Neyman-orthogonal scores that accommodates high-dimensional covariates while achieving the parametric $\sqrt{n}$ rate and asymptotic normality; for continuous instruments, we explicitly characterize the corresponding nonparametric convergence rates. The framework accommodates covariates, discrete and continuous instruments, and extensions to general treatment settings. In simulations and a bed-net subsidy application, the resulting bounds are substantially tighter than the moment-relaxation method.
Read the full abstract

Statistical Methodology

Robust Spatial-Sign-Based Testing of High-Dimensional Alpha in Conditional Factor Models

This paper develops a new framework for alpha testing in high-dimensional factor pricing models with time-varying coefficients. To detect sparse alternatives, we propose a spatial-sign-based max-type test and derive its limiting null distribution. A key theoretical result is that our statistic is asymptotically independent of the spatial-sign-based sum-type test proposed by Zhao (2023). Exploiting this independence, we construct an adaptive testing procedure via the Cauchy combination method. This approach integrates the complementary strengths of both max-type and sum-type statistics, ensuring robust power across diverse sparsity levels. Extensive simulations and an empirical application demonstrate that the proposed test is resilient to heavy-tailed distributions and maintains superior performance under various alternative specifications.
Read the full abstract

Monday 13 April

Statistics Applications · Cross-listed from cs.DL

NIH-MPINet: A Large-Scale Feature-Rich Network Dataset for Mapping the Frontiers of Team Science

This study presents a large-scale network dataset, NIH-MPINet, curated from NIH RePORTER and PubMed, characterizing collaboration among multiple Principal Investigators (multi-PIs) on NIH R01-equivalent grants from 2006 to 2023. The network characterizes 30,127 PIs as nodes and their collaborations on 86,743 NIH R01-equivalent grants as edges, spanning 888 recipient organizations and supported by 40 NIH Institutes and Centers. We also curated comprehensive metadata, including node-level features such as PI affiliation, alongside edge-level features comprising grant years, titles, and abstracts. Using these data, we constructed a PI collaboration network and identified 19 communities as well as 20 major research topics. Several collaboration communities showed distinct thematic profiles, such as cardiovascular health, cancer immunotherapy, neuroscience, and microbiome research, while genetics and genomics were broadly represented across communities. By incorporating temporal analysis, we observed shifts in research topics and collaboration patterns over time. Topics like healthcare and outcomes research, cognitive health, and Alzheimer's disease have become more prominent in recent years, whereas molecular and cellular biology has seen a relative decline. Overall, this work provides a high-fidelity, feature-rich resource for advancing statistical learning methods and network analysis-based discoveries in the study of long-term biomedical collaboration.
Read the full abstract

All authors Cuiran Shi, Shuying Han, Shreya Kusumanchi, Mia Zhou, Didong Li

Statistics Applications

Research Funding as a Decision Problem Under Heavy-Tailed Uncertainty

Heavy-tailed impact distributions, intrinsic uncertainty, and the high costs of proposal-based peer review increasingly challenge research funding decisions. Using large-scale bibliometric data, we show that past scientific performance provides statistically meaningful, though imperfect, information about future productivity and impact across multiple dimensions. An aggregated, percentile-normalised proxy signal captures this predictive structure robustly across research domains. We analyse deterministic and stochastic funding allocation mechanisms under impact-based objectives and find that both converge to highly concentrated allocations that favour a small number of top-performing researchers. To address the limitations of pure exploitation, we introduce a biased lottery framework based on a regularised decision-theoretic objective that explicitly balances exploration and exploitation while accounting for practical funding constraints. Our results suggest that biased lottery mechanisms offer a transparent, efficient, and scalable alternative to conventional peer review in environments characterised by heavy-tailed scientific returns. Additionally, we provide a web application, available at http://scilottery.biocomputingunit.es, that implements the deterministic allocation method presented in this work.
Read the full abstract

Statistical Computation

R Package iglm: Regression under Interference in Connected Populations

We introduce R package iglm, which implements a comprehensive framework for studying relationships among predictors and outcomes under interference. The implemented regression framework facilitates the study of spillover and other phenomena in connected populations and has important advantages over existing packages, among them scalability and provable theoretical guarantees. On the computational side, the regression framework relies on scalable methods that can be applied to small and large data sets, by solving a convex optimization program based on pseudo-likelihoods using Minorization-Maximization and Quasi-Newton algorithms. On the statistical side, the regression framework comes with provable theoretical guarantees. To increase the versatility of iglm, users can add custom-built model terms. We showcase iglm using two data sets, including hate speech on the social media platform X and communications among students.
Read the full abstract

Statistics Applications

The Broken Shield of European Palliative Care: Evidence from Synthetic Counterfactuals on Financial Toxicity and Informal Care

The transition of end-of-life care to palliative care (PC) sparks intense debate: does it provide economic relief or shift unremunerated labor costs onto families? Evaluating this is hindered by causal inference challenges and skewed healthcare costs. To overcome these limitations, we introduce a Synthetic Data Generation framework. Using pan-European SHARE data (2016-2021), we deploy Tabular Denoising Diffusion Probabilistic Models within a Two-Learner architecture to synthesize high-fidelity digital twins. By including the 2020-2021 lockdowns, we leverage the COVID-19 pandemic to isolate structural inequalities from transient market shocks. Our findings challenge the strict cost-shifting hypothesis: on average, PC acts as a "double shield", truncating out-of-pocket expenditures (liquid toxicity) and informal caregiving shadow values (time poverty). However, quantile treatment models expose a "broken shield" for vulnerable households and severe tail events. Non-cancer trajectories drive massive structural penalties that escalate at the distribution's tail, mechanically compounded by physical dependency. Socio-demographics heavily modulate this exposure: lacking a spousal net inflates the burden, rigid gender dynamics trigger labor market ejection, and financial distress acts as a profound multiplier. Institutionally, high-wage Nordic regimes paradoxically impose opportunity costs, while severe penalties in underfunded Eastern systems-mediated by financial distress-drive families toward resource exhaustion. We conclude that while PC is an ethical imperative, its expansion must be decoupled from the oncological paradigm and matched with state-funded long-term care to protect against clinical decline and financial shocks.
Read the full abstract

All authors Pietro Grassi, Edoardo Paperi, Chiara Seghieri, Daniele Vignoli

Machine Learning (Statistics) · Cross-listed from cs.LG

Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates

We study learning to learn through the lens of hyperparameter tuning. We propose the Langevin Gradient Descent Algorithm (LGD), which approximates the mean of the posterior distribution defined by the loss function and regularizer of a regression task with convex objective. For classification tasks, the LGD algorithm estimates the posterior probabilities of each class on the test set. We prove the existence of an optimal hyperparameter configuration for which the LGD algorithm achieves the Bayes' optimal solution for squared loss on regression tasks, and for which LGD closely approximates the posterior probabilities for well-specified classification tasks. Subsequently, we study generalization guarantees on meta learning optimal hyperparameters for the LGD algorithm from a given set of tasks in the data-driven setting. For a number of parameters $d$ and hyperparameter dimension $h$, we show a pseudo-dimension bound of $O(dh)$, up to logarithmic terms under mild assumptions on LGD. This matches the dependence of the bounds on number of parameters obtained in prior work for linear regression using the elastic net, which only allows for $h=2$ hyperparameters, and extends their bounds to regression on convex loss. Compared to bounds on regularized logistic regression that allow for only $h=1$ hyperparameter, our bounds improve greatly on the dependence on samples per task at the cost of worse dependence on the number of parameters by accounting for hardware-aware procedures. Finally, we show empirical evidence of the success of LGD and the meta learning procedure for few-shot learning on linear and logistic regression using synthetically created datasets.
Read the full abstract

All authors Saumya Goyal, Rohith Rongali, Ritabrata Ray, Barnabás Póczos

Statistics Applications

Observing the unobserved confounding through its effects: toward randomized trial-like estimates from real-world survival data

Background: Randomized controlled trials (RCTs) are costly, time-consuming, and often infeasible, while treatment-effect estimation from observational data is limited by unobserved confounding. Methods: We developed a three-step framework to address unobserved confounding in observational survival data. First, we infer a latent prognostic factor (U) from restricted mean survival time (RMST) discrepancies between patients with similar observed factors, the same treatment, and divergent outcomes, leveraging the idea that the aggregate effect of unmeasured factors can be inferred even if individual factors cannot. Second, we balance U with observed baseline covariates using prognostic matching, entropy balancing, or inverse probability of treatment weighting. Third, we apply multivariable survival analysis to estimate hazard ratios (HRs). We evaluated the framework in three observational cohorts with RCT benchmarks, two RCT cohorts, and six multicenter observational cohorts. Results: In three observational cohorts (nine comparisons), balancing U improved agreement with trial HRs in all cases; in the strongest settings, it reduced absolute log-HR error by approximately ten-fold versus using observed covariates alone (mean reduction 0.344; p=0.001). In two RCT cohorts, U was balanced across arms (most SMDs <0.1) and adjustment had minimal impact on log-HRs (mean absolute change 0.08). Across six multicenter cohorts, balancing U within centers reduced cross-center dispersion in chemotherapy log-HR estimates (mean reduction 0.147; p=0.016); when populations were directly balanced across centers to account for case-mix differences, cross-center survival differences were narrowed in 75%-100% of comparisons. Conclusions: Inferring and balancing a latent prognostic signal may reduce unobserved confounding and improve treatment-effect estimation from real-world data.
Read the full abstract

All authors Vasiliki Stoumpou, Dimitris Bertsimas, Samuel Singer, Georgios Antonios Margonis

Statistical Methodology

Reliability estimation in dependent stress-strength model with Clayton copula and modified Weibull margins

Stress-strength models are widely used to assess the reliability of systems under uncertain conditions. While most studies assume independence between stress and strength variables, such an assumption may be unrealistic in many practical situations where these components are inherently dependent. In this study, we investigate stress-strength reliability under a dependent framework, where both stress and strength variables follow modified Weibull distributions and their dependence is modeled via a Clayton copula. The proposed model allows distinct parameter sets, resulting in a flexible seven-parameter structure that extends Weibull-based models. We consider several estimation procedures for the model parameters and reliability, including two-step maximum likelihood, least squares, weighted least squares, and maximum product of spacings, with interval estimation obtained via asymptotic and bootstrap confidence intervals. The performance of the proposed estimators is evaluated through an extensive Monte Carlo simulation study under various parameter configurations and sample sizes. Finally, the applicability of the proposed model is illustrated using monthly occupancy data from Istanbul's two largest dams, with the Clayton copula capturing their dependence structure. This application demonstrates how stress-strength reliability can inform water management decisions and mitigate inter-regional operational risks.
Read the full abstract

Statistical Methodology

Causal inference for spatiotemporal point processes in the presence of outcome spillover and carryover

We develop a framework for causal inference with continuous spatiotemporal point-process outcomes under cell-level interventions and outcome spillover. Potential outcomes are indexed by full treatment allocations, and the observed post-treatment process is represented as an unlabelled superposition of latent control and treatment components. On the observed design support, expected post-treatment event counts in any spacetime region under a given treatment allocation are identified under consistency, exchangeability, and positivity; off-support contrasts are identified relative to a regime-stable structural point-process model. Estimation is likelihood-based and implemented with stochastic EM. To understand when this is feasible, we analyse a predictable blockwise hard-EM surrogate and show nonasymptotic contraction of estimation error to a statistical floor governed by locally ambiguous regions. This yields plug-in guarantees for cell-level and global causal functionals, and clarifies the additional array conditions needed for unnormalised growing-window contrasts. The framework covers history dependent spatiotemporal point processes including Poisson and Hawkes models, with applications to settings such as epidemiology, seismology, and finance. We provide an application assessing the causal effect of injecting wastewater into the ground on seismic activity in Oklahoma.
Read the full abstract

All authors Conor Kresin, Duncan A. Clark, Louis Davis, Martin Hazelton

Statistics Applications

Comparing Powerwise (PWR) and the NCAA Power Index (NPI): Advising the NCAA Men's Division I Lacrosse Committee

This memo compares two methods, Powerwise (PWR) and the NCAA Power Index (NPI), that aim to rank NCAA Division I, II, and III teams on the basis of deservedness of an invite to end-of-season championship tournaments. It find that while the NPI might be a fit for sports like hockey, it falls short of the PWR method for use in ranking team sports that regularly feature somewhat wider margins of victory, including football, basketball, and lacrosse. In comparing the methods, this memo highlights differences in i) accuracy; ii) procedural integrity; iii) objectivity; iv) reproducibility; v) simplicity; and vi) stability; before drawing conclusions.
Read the full abstract

Machine Learning (Statistics)

A Nonparametric Adaptive EWMA Control Chart for Binary Monitoring of Multiple Stream Processes

Monitoring binomial proportions across multiple independent streams is a critical challenge in Statistical Process Control (SPC), with applications from manufacturing to cybersecurity. While EWMA charts offer sensitivity to small shifts, existing implementations rely on asymptotic variance approximations that fail during early-phase monitoring. We introduce a Cumulative Standardized Binomial EWMA (CSB-EWMA) chart that overcomes this limitation by deriving the exact time-varying variance of the EWMA statistic for binary multiple-stream data, enabling adaptive control limits that ensure statistical rigor from the first sample. Through extensive simulations, we identify optimal smoothing ({\lambda}) and limit (L) parameters to achieve target in-control average run length (ARL0) of 370 and 500. The CSB-EWMA chart demonstrates rapid shift detection across both ARL0 targets, with out-of-control average run length (ARL1) dropping to 3-7 samples for moderate shifts ({\delta}=0.2), and exhibits exceptional robustness across different data distributions, with low ARL1 Coefficients of Variation (CV < 0.10 for small shifts) for both ARL0 = 370 and 500. This work provides practitioners with a distribution-free, sensitive, and theoretically sound tool for early change detection in binomial multiple-stream processes.
Read the full abstract

All authors Faruk Muritala, Austin Brown, Dhrubajyoti Ghosh, Sherry Ni

Statistics Theory

QBIC of SEM for jump-diffusion processes based on high-frequency data

Structural equation modeling (SEM) is a statistical method for analyzing relationships among latent variables. Since SEM is a confirmatory method, the model needs to be specified in advance. In practice, however, statisticians have several candidate models and aim to select the most appropriate one among them. In this paper, we consider model selection in SEM for jump-diffusion processes. We propose a quasi-Bayesian information criterion (QBIC) for the SEM and show that the proposed criterion has model-selection consistency.
Read the full abstract

Statistics Theory

Adaptivity of the NPMLE to finitely discrete mixing distributions in Gaussian/Poisson mixtures

We study the nonparametric maximum likelihood estimator (NPMLE) for Gaussian and Poisson mixture models, assuming the support of the true mixing distribution lies in a fixed bounded set. In this setting, we establish exact parametric rates for both, marginal density estimation and the posterior mean when the true mixing distribution is finitely discrete. Moreover, we show that the NPMLE attains the optimal demixing rate previously known for overparameterized finite mixture models. Finally, we identify a new adaptivity phenomenon for inference: the likelihood ratio test statistic is asymptotically tight if and only if the true mixing distribution is finitely discrete.
Read the full abstract

Statistical Methodology

Is There an AI Bubble? Robust Date-Stamping for Periods of Exuberance

The recent surge in valuations among AI related firms has renewed concerns that markets may be entering a new phase of speculative exuberance, especially in the technology and semiconductor sectors at the center of the AI investment wave. This paper develops a practical econometric framework for detecting, date-stamping, and drawing inference on the origination and collapse of bubble episodes when prices evolve under persistent, time-varying volatility. Standard bubble tests are typically derived under homoskedasticity or weak heteroskedasticity and may therefore yield misleading inference in more general settings. We extend right-tailed Dickey-Fuller unit root tests to autoregressive models with highly persistent mean and volatility dynamics, delivering a stochastic-volatility-robust ADF (SV-ADF) test that accommodates persistent variance without imposing strict parametric structure. Building on a moderate-deviation asymptotic theory, the SV-ADF yields nuisance-parameter-free procedures with distinct critical values for origination and collapse, producing more stable alarms and fewer transient false positives around volatility spikes. We establish consistency of the date-stamping estimator and show that it remains asymptotically tractable. Monte Carlo simulations document strong power and substantial gains over homoskedastic (PWY) procedures when volatility dynamics are pronounced. An empirical analysis of AI-exposed equities, including the "Magnificent Seven" and leading semiconductor firms, finds pervasive exuberance with substantial heterogeneity in timing, intensity, and duration. The evidence points to especially strong bubble dynamics for Alphabet and TSMC in the current cycle, while Tesla and Nvidia exhibited pronounced explosive episodes in earlier phases of the AI-driven market cycle.
Read the full abstract

Machine Learning (Statistics)

On the continuum limit of t-SNE for data visualization

This work is concerned with the continuum limit of a graph-based data visualization technique called the t-Distributed Stochastic Neighbor Embedding (t-SNE), which is widely used for visualizing data in a variety of applications, but is still poorly understood from a theoretical standpoint. The t-SNE algorithm produces visualizations by minimizing the Kullback-Leibler divergence between similarity matrices representing the high dimensional data and its low dimensional representation. We prove that as the number of data points $n \to \infty$, after a natural rescaling and in applicable parameter regimes, the Kullback-Leibler divergence is consistent as the number of data points $n \to \infty$ and the similarity graph remains sparse with a continuum variational problem that involves a non-convex gradient regularization term and a penalty on the magnitude of the probability density function in the visualization space. These two terms represent the continuum limits of the attraction and repulsion forces in the t-SNE algorithm. Due to the lack of convexity in the continuum variational problem, the question of well-posedeness is only partially resolved. We show that when both dimensions are $1$, the problem admits a unique smooth minimizer, along with an infinite number of discontinuous minimizers (interpreted in a relaxed sense). This aligns well with the empirically observed ability of t-SNE to separate data in seemingly arbitrary ways in the visualization. The energy is also very closely related to the famously ill-posed Perona-Malik equation, which is used for denoising and simplifying images. We present numerical results validating the continuum limit, provide some preliminary results about the delicate nature of the limiting energetic problem in higher dimensions, and highlight several problems for future work.
Read the full abstract

All authors Jeff Calder, Zhonggan Huang, Ryan Murray, Adam Pickarski

Statistical Methodology

Convolutional Maximum Mean Discrepancy for Inference in Noisy Data

Modern data analyses frequently encounter settings where samples of variables are contaminated by measurement error. Ignoring measurement noise can substantially degrade statistical inference, while existing correction techniques are often computationally costly and inefficient. Recent advances in kernel methods, particularly those based on Maximum Mean Discrepancy (MMD), have enabled flexible, distribution-free inference, yet typically assume precise data and overlook contamination by measurement error. In this work, we introduce a novel framework for inference with samples corrupted by potentially heteroscedastic noise from a known distribution. Central to our approach is the convolutional MMD (convMMD), which compares distributions after noise convolution and retains metric validity under standard kernel conditions. We establish finite-sample deviation bounds that are unaffected by measurement error and prove an equivalence between testing under noise and kernel smoothing. Leveraging these insights, we introduce a convMMD-based estimator for inference with noisy, heteroscedastic observations. We establish its consistency and asymptotic normality, and provide an efficient implementation using stochastic gradient descent. We demonstrate the practical effectiveness of our approach through simulations and applications in astronomy and social sciences.
Read the full abstract

All authors Ritwik Vashistha, Jeff M. Phillips, Abhra Sarkar, Arya Farahi

Statistical Methodology

Inference on Survival Reliability with Type-I Censored Weibull data

Reliability inference based on parametric distributions is an important problem in electrical and mechanical engineering. Most existing methods rely on approximations or bootstrap procedures, which may not perform satisfactorily when data are censored or sample sizes are small. Hence, there is an urgent need to develop exact inference approaches for these situations. This article introduces a new approach for deriving exact parametric tests and confidence intervals for distributions such as the lognormal, loglogistic, and Weibull. We revisit several issues in classical reliability analysis based on the survival function. Because lifetime data are often censored in practice, the proposed approach is designed for such settings. We illustrate the method using the Weibull distribution and expect it to be applicable to other widely used lifetime distributions such as the loglogistic distribution. Our Simulation study show that the new approach provides better performance than existing methods when handling complete data and type-I censored data. Two numerical examples are provided to demonstrate the application of the proposed method. The proposed method is expected to be widely applicable in reliability engineering and survival analysis, offering a robust alternative to existing methods, particularly in scenarios involving censored data and small sample sizes.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Offline-Online Reinforcement Learning for Linear Mixture MDPs

We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift. In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment, while in the online phase the learner interacts with the target environment. We propose an algorithm that adaptively leverages offline data. When the offline data are informative, either due to sufficient coverage or small environment shift, the algorithm provably improves over purely online learning. When the offline data are uninformative, it safely ignores them and matches the online-only performance. We establish regret upper bounds that explicitly characterize when offline data are beneficial, together with nearly matching lower bounds. Numerical experiments further corroborate our theoretical findings.
Read the full abstract

Statistical Computation

Sobolev-Regularized Objective Functions for Robust Pairwise Alignment of Functional Data

Functional data registration is a critical challenge in modern statistics, essential for separating phase variability from amplitude variability. While derivative-based frameworks offer mathematically elegant solutions, their dependence on signal velocities renders them susceptible to additive noise. This study proposes and evaluates a family of robust, Sobolev-regularized objective functions for the pairwise alignment of functional data, operating entirely within the original function space to avoid the need for numerical differentiation of the data. We define our optimization over a second-order Sobolev space and utilize the Centered Log-Ratio (CLR) transform to represent the warping functions. By penalizing both the velocity and acceleration of the centered log-derivative, this geometric approach preempts degenerate "pinching" artifacts and ensures the resulting warps are strictly monotonic, valid diffeomorphisms. In practice, this allows for highly efficient, unconstrained optimization within a finite-dimensional space. We systematically investigate four distinct pairwise data mismatch formulations: a Standard L2 baseline, a Symmetric L2 formulation, an Isometry (L2-preserving) mapping, and a Jacobian-weighted L2 functional. We establish robust theoretical foundations for these methods, proving the existence of optimal warps and the asymptotic consistency of the finite-dimensional estimators. Our results demonstrate that this CLR-regularized framework offers a powerful, computationally scalable, and noise-robust alternative to traditional derivative-based registration.
Read the full abstract