Statistics Applications · Cross-listed from cs.DL
NIH-MPINet: A Large-Scale Feature-Rich Network Dataset for Mapping the Frontiers of Team Science
Cuiran Shi, Shuying Han, Shreya Kusumanchi et al. (+2 more)
This study presents a large-scale network
dataset, NIH-MPINet, curated from NIH RePORTER and PubMed, characterizing collaboration among multiple Principal Investigators (multi-PIs) on NIH R01-equivalent grants from 2006 to 2023. The network characterizes 30,127 PIs as nodes and their collaborations on 86,743 NIH R01-equivalent grants as edges, spanning 888 recipient organizations and supported by 40 NIH Institutes and Centers. We also curated comprehensive metadata, including node-level features such as PI affiliation, alongside edge-level features comprising grant years, titles, and abstracts. Using these data, we constructed a PI collaboration network and identified 19 communities as well as 20 major research topics. Several collaboration communities showed distinct thematic profiles, such as cardiovascular health, cancer immunotherapy, neuroscience, and microbiome research, while genetics and genomics were broadly represented across communities. By incorporating temporal analysis, we observed shifts in research topics and collaboration patterns over time. Topics like healthcare and outcomes research, cognitive health, and Alzheimer's disease have become more prominent in recent years, whereas molecular and cellular biology has seen a relative decline. Overall, this work provides a high-fidelity, feature-rich resource for advancing statistical learning methods and network analysis-based discoveries in the study of long-term biomedical collaboration.
Read the full abstract ▾
All authors Cuiran Shi, Shuying Han, Shreya Kusumanchi, Mia Zhou, Didong Li
Statistics Applications
Research Funding as a Decision Problem Under Heavy-Tailed Uncertainty
Carlos Oscar S. Sorzano and B. Pueche-Granados
Heavy-tailed impact distributions, intrinsic uncertainty, and the high costs of proposal-based peer review increasingly challenge research funding decisions. Using large-scale bibliometric data, we show that past scientific performance provides statistically meaningful, though imperfect, information about future productivity and impact across multiple dimensions. An aggregated, percentile-normalised proxy signal captures this predictive structure robustly across research domains. We analyse deterministic and
stochastic funding allocation mechanisms under impact-based objectives and find that both converge to highly concentrated allocations that favour a small number of top-performing researchers. To address the limitations of pure exploitation, we introduce a biased lottery framework based on a regularised decision-theoretic objective that explicitly balances exploration and exploitation while accounting for practical funding constraints. Our results suggest that biased lottery mechanisms offer a transparent, efficient, and scalable alternative to conventional peer review in environments characterised by heavy-tailed scientific returns. Additionally, we provide a web application, available at http://scilottery.biocomputingunit.es, that implements the deterministic allocation method presented in this work.
Read the full abstract ▾
Statistical Computation
R Package iglm: Regression under Interference in Connected Populations
Cornelius Fritz and Michael Schweinberger
We introduce R package iglm, which implements a comprehensive framework for studying relationships among predictors and outcomes under interference. The implemented regression framework facilitates the study of spillover and other phenomena in connected populations and has important advantages over existing packages, among them scalability and provable theoretical guarantees. On the computational side, the regression framework relies on scalable methods that can be applied to small and large data sets, by solving a convex optimization program based on pseudo-likelihoods using Minorization-Maximization and Quasi-Newton
algorithms. On the statistical side, the regression framework comes with provable theoretical guarantees. To increase the versatility of iglm, users can add custom-built model terms. We showcase iglm using two data sets, including hate
speech on the social media platform X and communications among students.
Read the full abstract ▾
Statistics Applications
The Broken Shield of European Palliative Care: Evidence from Synthetic Counterfactuals on Financial Toxicity and Informal Care
Pietro Grassi, Edoardo Paperi, Chiara Seghieri et al. (+1 more)
The transition of end-of-life care to palliative care (PC) sparks intense debate: does it provide economic relief or shift unremunerated labor costs onto families? Evaluating this is hindered by causal inference challenges and skewed healthcare costs. To overcome these limitations, we introduce a Synthetic Data Generation framework. Using pan-European SHARE data (2016-2021), we deploy Tabular Denoising
Diffusion Probabilistic Models within a Two-Learner architecture to synthesize high-fidelity digital twins. By including the 2020-2021 lockdowns, we leverage the COVID-19 pandemic to isolate structural inequalities from transient
market shocks. Our findings challenge the strict cost-shifting hypothesis: on average, PC acts as a "double shield", truncating out-of-pocket expenditures (
liquid toxicity) and informal caregiving shadow values (time poverty). However,
quantile treatment models expose a "broken shield" for vulnerable households and severe tail events. Non-cancer trajectories drive massive structural penalties that escalate at the distribution's tail, mechanically compounded by physical dependency. Socio-demographics heavily modulate this exposure: lacking a spousal net inflates the burden, rigid gender dynamics trigger labor
market ejection, and financial distress acts as a profound multiplier. Institutionally, high-wage Nordic regimes paradoxically impose opportunity costs, while severe penalties in underfunded Eastern systems-mediated by financial distress-drive families toward resource exhaustion. We conclude that while PC is an ethical imperative, its expansion must be decoupled from the oncological paradigm and matched with state-funded long-term care to protect against clinical decline and financial shocks.
Read the full abstract ▾
All authors Pietro Grassi, Edoardo Paperi, Chiara Seghieri, Daniele Vignoli
Machine Learning (Statistics) · Cross-listed from cs.LG
Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates
Saumya Goyal, Rohith Rongali, Ritabrata Ray et al. (+1 more)
We study learning to learn through the lens of hyperparameter tuning. We propose the Langevin Gradient Descent
Algorithm (LGD), which approximates the mean of the posterior distribution defined by the loss function and regularizer of a regression task with convex objective. For classification tasks, the LGD
algorithm estimates the posterior probabilities of each class on the test set. We prove the existence of an optimal hyperparameter configuration for which the LGD
algorithm achieves the Bayes' optimal solution for squared loss on regression tasks, and for which LGD closely approximates the posterior probabilities for well-specified classification tasks. Subsequently, we study generalization guarantees on meta learning optimal hyperparameters for the LGD
algorithm from a given set of tasks in the data-driven setting. For a number of parameters $d$ and hyperparameter dimension $h$, we show a pseudo-dimension bound of $O(dh)$, up to logarithmic terms under mild assumptions on LGD. This matches the dependence of the bounds on number of parameters obtained in prior work for linear regression using the elastic net, which only allows for $h=2$ hyperparameters, and extends their bounds to regression on convex loss. Compared to bounds on regularized logistic regression that allow for only $h=1$ hyperparameter, our bounds improve greatly on the dependence on samples per task at the cost of worse dependence on the number of parameters by accounting for hardware-aware procedures. Finally, we show empirical evidence of the success of LGD and the meta learning procedure for few-shot learning on linear and logistic regression using synthetically created
datasets.
Read the full abstract ▾
All authors Saumya Goyal, Rohith Rongali, Ritabrata Ray, Barnabás Póczos
Statistics Applications
Observing the unobserved confounding through its effects: toward randomized trial-like estimates from real-world survival data
Vasiliki Stoumpou, Dimitris Bertsimas, Samuel Singer et al. (+1 more)
Background: Randomized controlled trials (RCTs) are costly, time-consuming, and often infeasible, while treatment-effect estimation from observational data is limited by unobserved confounding. Methods: We developed a three-step framework to address unobserved confounding in observational survival data. First, we infer a latent prognostic factor (U) from restricted mean survival time (RMST) discrepancies between patients with similar observed factors, the same treatment, and divergent outcomes, leveraging the idea that the aggregate effect of unmeasured factors can be inferred even if individual factors cannot. Second, we balance U with observed baseline
covariates using prognostic matching,
entropy balancing, or inverse probability of treatment weighting. Third, we apply multivariable survival analysis to estimate hazard ratios (HRs). We evaluated the framework in three observational cohorts with RCT
benchmarks, two RCT cohorts, and six multicenter observational cohorts. Results: In three observational cohorts (nine comparisons), balancing U improved agreement with trial HRs in all cases; in the strongest settings, it reduced absolute log-HR error by approximately ten-fold versus using observed
covariates alone (mean reduction 0.344; p=0.001). In two RCT cohorts, U was balanced across arms (most SMDs <0.1) and adjustment had minimal impact on log-HRs (mean absolute change 0.08). Across six multicenter cohorts, balancing U within centers reduced cross-center dispersion in chemotherapy log-HR estimates (mean reduction 0.147; p=0.016); when populations were directly balanced across centers to account for case-mix differences, cross-center survival differences were narrowed in 75%-100% of comparisons. Conclusions: Inferring and balancing a latent prognostic signal may reduce unobserved confounding and improve treatment-effect estimation from real-world data.
Read the full abstract ▾
All authors Vasiliki Stoumpou, Dimitris Bertsimas, Samuel Singer, Georgios Antonios Margonis
Statistical Methodology
Reliability estimation in dependent stress-strength model with Clayton copula and modified Weibull margins
Fatih K{\i}z{\i}laslan
Stress-strength models are widely used to assess the reliability of systems under uncertain conditions. While most studies assume independence between stress and strength variables, such an assumption may be unrealistic in many practical situations where these components are inherently dependent. In this study, we investigate stress-strength reliability under a dependent framework, where both stress and strength variables follow modified Weibull distributions and their dependence is modeled via a Clayton copula. The proposed model allows distinct parameter sets, resulting in a flexible seven-parameter structure that extends Weibull-based models. We consider several estimation procedures for the model parameters and reliability, including two-step maximum likelihood, least squares, weighted least squares, and maximum product of spacings, with interval estimation obtained via asymptotic and bootstrap confidence intervals. The performance of the proposed
estimators is evaluated through an extensive Monte Carlo simulation study under various parameter configurations and sample sizes. Finally, the applicability of the proposed model is illustrated using monthly occupancy data from Istanbul's two largest dams, with the Clayton copula capturing their dependence structure. This application demonstrates how stress-strength reliability can inform water management decisions and mitigate inter-regional operational risks.
Read the full abstract ▾
Statistical Methodology
Causal inference for spatiotemporal point processes in the presence of outcome spillover and carryover
Conor Kresin, Duncan A. Clark, Louis Davis et al. (+1 more)
We develop a framework for causal inference with continuous spatiotemporal point-process outcomes under
cell-level interventions and outcome spillover. Potential outcomes are indexed by full treatment allocations, and the observed post-treatment process is represented as an unlabelled superposition of latent control and treatment components. On the observed design support, expected post-treatment event counts in any spacetime region under a given treatment allocation are identified under consistency, exchangeability, and positivity; off-support contrasts are identified relative to a regime-stable structural point-process model. Estimation is likelihood-based and implemented with
stochastic EM. To understand when this is feasible, we analyse a predictable blockwise hard-EM surrogate and show nonasymptotic contraction of estimation error to a statistical floor governed by locally ambiguous regions. This yields plug-in guarantees for
cell-level and global causal functionals, and clarifies the additional array conditions needed for unnormalised growing-window contrasts. The framework covers history dependent spatiotemporal point processes including Poisson and Hawkes models, with applications to settings such as epidemiology, seismology, and finance. We provide an application assessing the causal effect of injecting wastewater into the ground on seismic activity in Oklahoma.
Read the full abstract ▾
All authors Conor Kresin, Duncan A. Clark, Louis Davis, Martin Hazelton
Statistics Applications
Comparing Powerwise (PWR) and the NCAA Power Index (NPI): Advising the NCAA Men's Division I Lacrosse Committee
Lawrence Feldman and Matthew Bomparola
This memo compares two methods, Powerwise (PWR) and the NCAA Power Index (NPI), that aim to rank NCAA Division I, II, and III teams on the basis of deservedness of an invite to end-of-season championship tournaments. It find that while the NPI might be a fit for sports like hockey, it falls short of the PWR method for use in ranking team sports that regularly feature somewhat wider margins of victory, including football, basketball, and lacrosse. In comparing the methods, this memo highlights differences in i) accuracy; ii) procedural integrity; iii) objectivity; iv) reproducibility; v) simplicity; and vi) stability; before drawing conclusions.
Read the full abstract ▾
Machine Learning (Statistics)
A Nonparametric Adaptive EWMA Control Chart for Binary Monitoring of Multiple Stream Processes
Faruk Muritala, Austin Brown, Dhrubajyoti Ghosh et al. (+1 more)
Monitoring binomial proportions across multiple independent streams is a critical challenge in Statistical Process Control (SPC), with applications from manufacturing to cybersecurity. While EWMA charts offer sensitivity to small shifts, existing implementations rely on asymptotic variance approximations that fail during early-phase monitoring. We introduce a Cumulative Standardized Binomial EWMA (CSB-EWMA) chart that overcomes this limitation by deriving the exact time-varying variance of the EWMA statistic for binary multiple-stream data, enabling adaptive control limits that ensure statistical rigor from the first sample. Through extensive simulations, we identify optimal smoothing ({\lambda}) and limit (L) parameters to achieve target in-control average run length (ARL0) of 370 and 500. The CSB-EWMA chart demonstrates rapid shift detection across both ARL0 targets, with out-of-control average run length (ARL1) dropping to 3-7 samples for moderate shifts ({\delta}=0.2), and exhibits exceptional robustness across different data distributions, with low ARL1 Coefficients of Variation (CV < 0.10 for small shifts) for both ARL0 = 370 and 500. This work provides practitioners with a distribution-free, sensitive, and theoretically sound tool for early change detection in binomial multiple-stream processes.
Read the full abstract ▾
All authors Faruk Muritala, Austin Brown, Dhrubajyoti Ghosh, Sherry Ni
Statistics Theory
QBIC of SEM for jump-diffusion processes based on high-frequency data
Shogo Kusano and Masayuki Uchida
Structural equation modeling (SEM) is a statistical method for analyzing relationships among latent variables. Since SEM is a confirmatory method, the model needs to be specified in advance. In practice, however, statisticians have several candidate models and aim to select the most appropriate one among them. In this paper, we consider model selection in SEM for jump-
diffusion processes. We propose a quasi-
Bayesian information criterion (QBIC) for the SEM and show that the proposed criterion has model-selection consistency.
Read the full abstract ▾
Statistics Theory
Adaptivity of the NPMLE to finitely discrete mixing distributions in Gaussian/Poisson mixtures
Yan Zhang and Stanislav Volgushev
We study the nonparametric maximum likelihood
estimator (NPMLE) for Gaussian and Poisson mixture models, assuming the support of the true mixing distribution lies in a fixed bounded set. In this setting, we establish exact parametric rates for both, marginal density estimation and the posterior mean when the true mixing distribution is finitely discrete. Moreover, we show that the NPMLE attains the optimal demixing rate previously known for overparameterized finite mixture models. Finally, we identify a new adaptivity phenomenon for inference: the likelihood ratio test statistic is asymptotically tight if and only if the true mixing distribution is finitely discrete.
Read the full abstract ▾
Statistical Methodology
Is There an AI Bubble? Robust Date-Stamping for Periods of Exuberance
Abir Sarkar and Martin T. Wells
The recent surge in valuations among AI related firms has renewed concerns that
markets may be entering a new phase of speculative exuberance, especially in the technology and
semiconductor sectors at the center of the AI investment wave. This paper develops a practical econometric framework for detecting, date-stamping, and drawing inference on the origination and collapse of bubble episodes when prices evolve under persistent, time-varying
volatility. Standard bubble tests are typically derived under homoskedasticity or weak heteroskedasticity and may therefore yield misleading inference in more general settings. We extend right-tailed Dickey-Fuller unit root tests to autoregressive models with highly persistent mean and
volatility dynamics, delivering a
stochastic-
volatility-robust ADF (SV-ADF) test that accommodates persistent variance without imposing strict parametric structure. Building on a moderate-deviation asymptotic theory, the SV-ADF yields nuisance-parameter-free procedures with distinct critical values for origination and collapse, producing more stable alarms and fewer transient false positives around
volatility spikes. We establish consistency of the date-stamping
estimator and show that it remains asymptotically tractable. Monte Carlo simulations document strong power and substantial gains over homoskedastic (PWY) procedures when
volatility dynamics are pronounced. An empirical analysis of AI-exposed equities, including the "Magnificent Seven" and leading
semiconductor firms, finds pervasive exuberance with substantial heterogeneity in timing, intensity, and duration. The evidence points to especially strong bubble dynamics for Alphabet and TSMC in the current cycle, while Tesla and Nvidia exhibited pronounced explosive episodes in earlier phases of the AI-driven
market cycle.
Read the full abstract ▾
Machine Learning (Statistics)
On the continuum limit of t-SNE for data visualization
Jeff Calder, Zhonggan Huang, Ryan Murray et al. (+1 more)
This work is concerned with the continuum limit of a graph-based data visualization technique called the t-Distributed
Stochastic Neighbor
Embedding (t-SNE), which is widely used for visualizing data in a variety of applications, but is still poorly understood from a theoretical standpoint. The t-SNE
algorithm produces visualizations by minimizing the Kullback-Leibler divergence between similarity
matrices representing the high dimensional data and its low dimensional representation. We prove that as the number of data points $n \to \infty$, after a natural rescaling and in applicable parameter regimes, the Kullback-Leibler divergence is consistent as the number of data points $n \to \infty$ and the similarity graph remains sparse with a continuum variational problem that involves a non-convex gradient regularization term and a penalty on the magnitude of the probability density function in the visualization space. These two terms represent the continuum limits of the attraction and repulsion forces in the t-SNE
algorithm. Due to the lack of convexity in the continuum variational problem, the question of well-posedeness is only partially resolved. We show that when both dimensions are $1$, the problem admits a unique smooth minimizer, along with an infinite number of discontinuous minimizers (interpreted in a relaxed sense). This aligns well with the empirically observed ability of t-SNE to separate data in seemingly arbitrary ways in the visualization. The energy is also very closely related to the famously ill-posed Perona-Malik equation, which is used for denoising and simplifying images. We present numerical results validating the continuum limit, provide some preliminary results about the delicate nature of the limiting energetic problem in higher dimensions, and highlight several problems for future work.
Read the full abstract ▾
All authors Jeff Calder, Zhonggan Huang, Ryan Murray, Adam Pickarski
Statistical Methodology
Convolutional Maximum Mean Discrepancy for Inference in Noisy Data
Ritwik Vashistha, Jeff M. Phillips, Abhra Sarkar et al. (+1 more)
Modern data analyses frequently encounter settings where samples of variables are contaminated by measurement error. Ignoring measurement noise can substantially degrade statistical inference, while existing correction techniques are often computationally costly and inefficient. Recent advances in kernel methods, particularly those based on Maximum Mean Discrepancy (MMD), have enabled flexible, distribution-free inference, yet typically assume precise data and overlook contamination by measurement error. In this work, we introduce a novel framework for inference with samples corrupted by potentially heteroscedastic noise from a known distribution. Central to our approach is the convolutional MMD (convMMD), which compares distributions after noise convolution and retains metric validity under standard kernel conditions. We establish finite-sample deviation bounds that are unaffected by measurement error and prove an equivalence between testing under noise and kernel smoothing. Leveraging these insights, we introduce a convMMD-based
estimator for inference with noisy, heteroscedastic observations. We establish its consistency and asymptotic normality, and provide an efficient implementation using
stochastic gradient descent. We demonstrate the practical effectiveness of our approach through simulations and applications in astronomy and social sciences.
Read the full abstract ▾
All authors Ritwik Vashistha, Jeff M. Phillips, Abhra Sarkar, Arya Farahi
Statistical Methodology
Inference on Survival Reliability with Type-I Censored Weibull data
Bowen Liu, Malwane M.A. Ananda and Sam Weerahandi
Reliability inference based on parametric distributions is an important problem in electrical and mechanical engineering. Most existing methods rely on approximations or bootstrap procedures, which may not perform satisfactorily when data are censored or sample sizes are small. Hence, there is an urgent need to develop exact inference approaches for these situations. This article introduces a new approach for deriving exact parametric tests and confidence intervals for distributions such as the lognormal, loglogistic, and Weibull. We revisit several issues in classical reliability analysis based on the survival function. Because lifetime data are often censored in practice, the proposed approach is designed for such settings. We illustrate the method using the Weibull distribution and expect it to be applicable to other widely used lifetime distributions such as the loglogistic distribution. Our Simulation study show that the new approach provides better performance than existing methods when handling complete data and type-I censored data. Two numerical examples are provided to demonstrate the application of the proposed method. The proposed method is expected to be widely applicable in reliability engineering and survival analysis, offering a robust alternative to existing methods, particularly in scenarios involving censored data and small sample sizes.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Offline-Online Reinforcement Learning for Linear Mixture MDPs
Zhongjun Zhang and Sean R. Sinclair
We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift. In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment, while in the online phase the learner interacts with the target environment. We propose an
algorithm that adaptively leverages offline data. When the offline data are informative, either due to sufficient coverage or small environment shift, the
algorithm provably improves over purely online learning. When the offline data are uninformative, it safely ignores them and matches the online-only performance. We establish regret upper bounds that explicitly characterize when offline data are beneficial, together with nearly matching lower bounds. Numerical experiments further corroborate our theoretical findings.
Read the full abstract ▾
Statistical Computation
Sobolev-Regularized Objective Functions for Robust Pairwise Alignment of Functional Data
Wei Wu
Functional data registration is a critical challenge in modern statistics, essential for separating phase variability from amplitude variability. While derivative-based frameworks offer mathematically elegant solutions, their dependence on signal velocities renders them susceptible to additive noise. This study proposes and evaluates a family of robust, Sobolev-regularized objective functions for the pairwise
alignment of functional data, operating entirely within the original function space to avoid the need for numerical differentiation of the data. We define our optimization over a second-order Sobolev space and utilize the Centered Log-Ratio (CLR) transform to represent the warping functions. By penalizing both the velocity and acceleration of the centered log-derivative, this
geometric approach preempts degenerate "pinching" artifacts and ensures the resulting warps are strictly monotonic, valid diffeomorphisms. In practice, this allows for highly efficient, unconstrained optimization within a finite-dimensional space. We systematically investigate four distinct pairwise data mismatch formulations: a Standard L2 baseline, a Symmetric L2 formulation, an Isometry (L2-preserving) mapping, and a Jacobian-weighted L2 functional. We establish robust theoretical foundations for these methods, proving the existence of optimal warps and the asymptotic consistency of the finite-dimensional
estimators. Our results demonstrate that this CLR-regularized framework offers a powerful, computationally scalable, and noise-robust alternative to traditional derivative-based registration.
Read the full abstract ▾