Statistical Methodology · Cross-listed from cs.CL
How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats
Ian Arawjo
Researchers across academia increasingly base significance claims on LLM judge scores and small-sample AI evaluations. Yet without well-calibrated confidence intervals (CIs), hypothesis tests, and judge-bias corrections, such claims are unreliable. We address these issues in several contributions. First, we find that running statistics over raw LLM judge scores leads to inflated false positives: counterintuitively, for many inter-rater agreement metrics, false positive risk peaks at "almost perfect" human-LLM agreement. To help researchers understand how to run statistics over LLM judges responsibly, we present guidance and tooling for the statistical analysis of mixed human-AI judge designs, and implement nine hypothesis tests via prediction-powered inference (PPI), including the first known PPI corrections for four rank-based tests (Wilcoxon signed-rank, Mann-Whitney U, and omnibus variants). To keep PPI++ stable with small human-labeled calibration sets, we introduce bootstrap-adaptive power tuning, which shrinks the estimated weight toward a target estimated from the labeled data, and accounts for that weight's own sampling variance. Second, through Monte Carlo simulations, we derive recommendations for what CI, p-value, and FWER correction methods to use for small-sample AI evaluations (N<100), and warn researchers against bootstrap CIs. We package these recommendations into evalstats, an open-source Python package that selects calibrated methods automatically, and demonstrate it in three scenarios, including one where a real LLM judge validated at "substantial agreement" would have led a researcher to publish a spurious finding. evalstats is publicly available at https://github.com/ianarawjo/evalstats.
Read the full abstract ▾
Statistical Methodology
A Decision-Theoretic Framework for Comparing Likelihood Ratio Methods for the Rare Type Match Problem
Giulia Cereda, Fabio Corradi and Cecilia Viscardi
The rare type match problem is a challenging situation faced by a forensic statistician who aims at providing the value of a match between some characteristic of a crime stain and the corresponding characteristic of a suspect's stain when this characteristic has not been observed before. Several methods have been designed in the literature to assess likelihood ratios for the rare type match case when evidence consists of a Y-STR profile found on the crime scene matching the Y-STR profile of a designated suspect. We develop a general
Bayesian decision-theoretic framework for quantifying the expected cost of alternative approaches using the logarithmic scoring rule. The framework provides a novel formalization of the posterior cross-
entropy, which explicitly enhances the contribution of the method-specific strategy used to extract information from the data. We revisit existing decompositions of posterior cross-
entropy and introduce a new complementary decomposition that provides a more interpretable characterization of this contribution. This work compares nine different methods by assessing their performance using empirical validation experiments. Its ultimate goal is to provide forensic experts and triers of fact with methodological insight and a pre-experimental guide to these alternative approaches.
Read the full abstract ▾
Statistical Methodology · Cross-listed from stat.ML
Sparse Regression Distilled from a Single Robust Fit
Wooyoung Shin and Seunghwan Park
Robust linear fits can resist response contamination yet remain too dense or unstable for useful global explanations. We propose penalized
distillation, which fits a smoothly clipped absolute deviation (SCAD)
estimator to a robust initial
estimator's empirical fitted surface along a safeguarded coordinate-descent path and evaluates candidate states separately for fidelity, parsimony, perturbation stability, and held-out prediction. The new results attach to the states the
algorithm actually computes. Conditional on a fixed uncontaminated design, deterministic bounds transfer response-replacement boundedness from the initial fit to every retained path state. Turning to fixed dimension, we characterize the oracle-support branch by its empirical-Gram projection and influence function, give conditions for covariance-weighted least-squares approximation equivalence, and establish a path-conditional generalized information criterion. By contrast, at large dimension-to-sample ratios the full-coordinate robust fit collapses without warning, and screening restores the construction. Under a sure-screening framework, the robustness bound and the support and selection guarantees transfer to the screened fit. Simulations separate robustness transfer from support recovery, efficiency, and computation across the dimension-to-sample ratio, with p up to 240, and the signal density, which isolates what the sparse stage adds once the screen over-selects. In a duplicate-grouped superconductivity study, the distilled
estimator remains predictively stable under prespecified training-response shifts but retains 66.8–68.8 of 81 slopes. Stronger sparsification reduces the model to 12.6–14.0 slopes only at visible fidelity and prediction cost.
Distillation therefore preserves predictive stability on these data without substantiating a compact coordinate-level explanation.
Read the full abstract ▾
Statistical Methodology
Proxy-Adjusted Causal Discovery from Targeted Interventions
Li Chen, Wei Pan and Xiaotong Shen
Randomized perturbations can reveal downstream responses without identifying which causal relationships are direct. Conditioning on intermediate responses may induce associations through unmeasured common causes, even when the source is randomized. We introduce Proxy-Adjusted Balanced Masking (PABM), a nonparametric framework for recovering directed acyclic graphs from targeted interventions, response data, and recorded proxies. PABM uses valid target-specific interventions to identify candidate ancestors. It then compares conditional target distributions with and without a candidate response and, when supported by the design, its intervention variable, adjusting for other ancestors and proxies. Each comparison uses matched observations, adjustment variables, and fitting procedures. Identification requires every included nonparent comparison to have zero conditional gain and each parent to have positive gain in at least one comparison. We establish population identification, finite-sample recovery conditions that accommodate incomplete adjustment and estimation error, and familywise error control under valid held-out p-values. Continuous-response simulations show favorable graph recovery relative to specified comparator pipelines; K562-calibrated count simulations reveal a selection–ranking tradeoff. Proxy ablations assess sensitivity to recorded information. An analysis of K562 Perturb-seq data illustrates descriptive candidate-network construction; proxy adequacy for unmeasured biological variation remains unresolved.
Read the full abstract ▾
Statistical Methodology · Cross-listed from stat.AP
Timescales in Time-to-Event Analyses in Pregnancy
Chase D Latour, Shalini S. Ramachandra, Daniel A. Xu et al. (+8 more)
The target trial framework is increasingly used to define the causal estimand for an observational analysis. However, designing time-to-event analyses that align with the causal estimand is challenging in pregnancy because the timescale on which outcomes are often defined, "gestational age", differs from the timescale on which outcomes are measured, "time since study entry". We argue that neither timescale is "best" but that each aligns with a different estimand. In this article, we use the target trial framework to define the causal estimands corresponding to each timescale. We consider a (hypothetical) target trial comparing randomization to 17-alpha hydroxyprogesterone caproate (17-OHPC) versus placebo between 16-20 weeks of gestation to prevent delivery before 37 weeks' gestation, whether by miscarriage, stillbirth, or live birth. We define the estimand of time-to-event analyses on the "time since study entry" timescale as the effect of randomization to 17-OHPC versus placebo between 16-20 weeks of gestation on delivery before 37 weeks'. In contrast, the causal estimand of time-to-event analyses using the "gestational age" timescale corresponds to the effect had, counter to fact, all participants been randomized at one gestational age (e.g., 16 weeks). We review specific challenges to estimation and possible solutions for each timescale.
Read the full abstract ▾
All authors Chase D Latour, Shalini S. Ramachandra, Daniel A. Xu, Elizabeth Diemer, Viktor H. Ahlqvist, Kristin Palmsten, Stefanie Hinkle, Sunni L. Mumford, Enrique Schisterman, Jessie K. Edwards, Ellen C. Caniglia
Statistical Methodology
Bayesian Posterior Learning of Mixed Graphical Models
Erdong Guo, Alexandros Beskos and Maria De Iorio
Mixed Graphical Models (MGMs) provide a flexible framework for structure learning from heterogeneous data by treating sets of both continuous and discrete.
Bayesian inference for MGMs remains challenging due to the combinatorial complexity of graph space exploration and posterior computation. In this paper, we model discrete components through latent Gaussian variables and consider two likelihood specifications: a copula-based ranked likelihood, yielding the copula-MGM, and a probit formulation based on cut-off points, yielding the probit-MGM. We then propose the Mixed Graph WWA, a class of MCMC methods for posterior simulation in
Bayesian MGMs. Building upon the WWA
algorithm, we develop two specialized
algorithms: copula-WWA for copula-MGMs and probit-WWA for probit-MGMs. Both methods exploit the latent Gaussian representations to perform posterior inference through a Gibbs sampling scheme that alternates between latent-variable augmentation and graph-structure updates. Through extensive simulation studies we demonstrate that the proposed methods achieve graph recovery accuracy comparable to or better than existing approaches, including copula-BD MCMC and probit-BD MCMC based on the Birth-Death MCMC methodology, while maintaining efficient posterior exploration and favorable effective sample size per unit computational time. We further illustrate the practical utility of our approach through an application to the PAM$50$ breast cancer gene expression
dataset, where the inferred MGMs reveal meaningful dependencies between gene expression profiles and cancer subtypes. These results highlight the effectiveness of the Mixed Graph WWA method as a scalable and principled tool for
Bayesian structure learning in MGMs.
Read the full abstract ▾
Statistical Methodology
Optimal Biological Dose Combination Finding: A Design Roadmap and Robust Cross-Indication Bayesian Borrowing
Ayon Mukherjee, Kentaro Takeda and James M.S. Wason
Early-phase combination oncology trials increasingly seek the optimal biological dose combination (OBDC) rather than a maximum tolerated dose combination, in line with the FDA's Project Optimus initiative. Existing
Bayesian OBDC designs span rule-based, model-assisted, and model-based paradigms, but no unified framework exists for choosing among them, and none supports borrowing information across indications sharing a combination regimen. We propose a corrected three-by-two taxonomy of OBDC designs by mechanism and objective, a rule-based design (Ji3+3-Comb) closing a documented gap in transparent, model-free combination dose-finding, and a
Bayesian Hierarchical Utility-based Cross-indication (BHUC) design that borrows information across indications via a robust mixture prior while discounting non-exchangeable information. We derive the shared utility function as the Bayes-optimal decision under a linear clinical loss, and prove that BHUC's mixture-prior posterior weight automatically vanishes as two indications' true rates become discordant, formalizing its robustness to non-exchangeable borrowing, and complement this asymptotic guarantee with an exact, sample-size-free ceiling on borrowed influence that holds even before any own-indication data accrue. A roadmap links trial features to design choice, and a 5000-replication simulation study, sensitivity analyses, and a case study built from a published phase Ib trial show that model-assisted designs offer the most robust safety-efficacy trade-off, while BHUC improves correct selection over independent per-indication designs even under discordance. We conclude with concrete, protocol-actionable recommendations for pharmaceutical and trial biostatisticians, and directions for future methodological and computational development.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.LG
Belted Engression: Sufficient Dimension Reduction for Generative Distributional Regression
Wenxi Tan, Bing Li and Lingzhou Xue
Modern conditional generative models face significant challenges when learning complex
covariate dependencies. While sufficient dimension reduction (SDR) provides a principled approach to compress these dependencies, traditional SDR frameworks were not formulated for conditional generation. To bridge this gap, we propose Belted Engression, a unified and architecturally parameter-efficient framework for generative distributional regression. Our approach establishes an end-to-end compress-then-generate paradigm driven by sufficient representation learning,
embedding a structural bottleneck into the generative architecture. Theoretically, we prove that the standard SDR condition is equivalent to a law-preserving generative factorization, which is achieved at the global optimum of the population Belted Engression objective. Furthermore, by uncovering a localized Bernstein-type control for the energy-score loss, we establish finite-sample convergence rates that are sharper than those of existing results. We also prove that this belted architecture is strictly smaller, operating with an asymptotically vanishing parameter count relative to the unstructured baseline. Extensive simulations and real-world applications demonstrate that Belted Engression achieves superior distributional prediction and SDR recovery with fewer trainable parameters.
Read the full abstract ▾
Statistical Methodology
A Stochastic EM Algorithm with Sampling-Importance Resampling for Missing Data in Regression with Nonlinear Predictors
Dale S. Kim
Estimating regression models with nonlinear predictor transformations is challenging when data are missing, because nonlinearity typically renders the conditional distribution of the missing values intractable. Previous methods require specific nonlinear forms, such as
polynomials or interactions, or rely on approximations that can induce bias. We propose a
stochastic EM
algorithm that uses sampling-importance resampling (SIR-StEM) to handle missing data under arbitrary nonlinear transformations of predictors, either substantively motivated or incidental, such as spline basis expansions. Unlike approaches that require linearity or closed-form conditionals, SIR-StEM only requires evaluating the complete-data likelihood up to proportionality, making it applicable across a broad class of nonlinear regression models. We construct an
algorithm that makes use of missing data pattern information for computational efficiency, and establish asymptotic normality of the
estimator. We demonstrate the method in two simulation studies, one with parametric nonlinear transformations and another with a spline basis expansion based on real behavioral measures. Results show that SIR-StEM yields low bias and near-nominal confidence interval coverage, outperforming other common approaches. We conclude with limitations and directions for future research.
Read the full abstract ▾
Statistical Methodology
PAR2COX: Survival-Informed Tensor Decomposition for Phenotyping and Risk Prediction from Irregular Longitudinal Data
Mariafrancesca Patalano, Elif Konyar and Kamran Paynabar
Accurate risk prediction is crucial for clinical-decision making, intervention planning, treatment and transplant allocation. However, longitudinal clinical data are often irregular and subject to censoring. We propose PAR2COX, a joint framework that integrates PARAFAC2 decomposition with Cox proportional hazards model, using patient-specific latent factors as
covariates in the likelihood. The proposed alternating optimization framework jointly estimates phenotypes and survival parameters, enabling survival-guided representation learning. PAR2COX accommodates both historical patients with observed outcomes and current patients whose outcomes remain unknown. Numerical experiments and a case study based on MIMIC-IV data demonstrate improved risk stratification compared with existing approaches, highlighting the value of survival-informed phenotype learning.
Read the full abstract ▾
Statistical Methodology
Model-free and Distributionally Robust Feature Screening with False Discovery Control for High-Dimensional Heterogeneous Data
Cong Cheng, Runze Li and Yuan Ke
In this paper, we propose a model-free feature screening framework tailored for high-dimensional and heterogeneous
datasets, based on a novel distributionally robust dependence measure termed Copula Divergence. The proposed screening method, named CD-Screen, addresses critical limitations of existing feature screening methods, such as restrictive modeling assumptions and sensitivity to heterogeneous feature distributions. CD-Screen ranks features according to their Copula Divergence without relying on a specific regression model or distributional assumptions. Additionally, we introduce CD-FDR, a data-driven procedure to control false discoveries, ensuring accurate and efficient feature selection. Theoretical analyses establish the sure screening and rank consistency properties of CD-Screen, along with asymptotic control of the false discovery rate by CD-FDR. Extensive simulation studies demonstrate the superior performance of our methods compared to traditional screening approaches across diverse scenarios. Furthermore, a real data analysis of the relationship between stock returns and inflation in the United States {illustrates the practical use of our method and provides descriptive evidence on} sector-specific responses to economic changes.
Read the full abstract ▾
Statistical Methodology · Cross-listed from stat.ML
PACE: Plug-and-Play Contextual Embedding for Feature Screening with Pretrained Tabular Foundation Models
Qi Qin, Erbo Li, Ting Wei et al. (+4 more)
In high-dimensional tabular learning, feature screening provides a lightweight, model-agnostic way to remove irrelevant features before model fitting. However, scoring raw values directly can miss nonlinear or distributional structure. We introduce PACE (Plug-and-Play Contextual
Embedding), which inserts a frozen tabular foundation model (TFM) column encoder before an existing feature-scoring rule, expanding each feature into a higher-dimensional contextual representation. Across controlled studies, PACE improves raw-space screening of complex nonlinear dependence with only modest additional encoding cost. These gains translate to downstream prediction on TALENT
datasets: PACE-DC improves binary AUC by 0.077 and multiclass macro-AUC by 0.064, with a median normalized RMSE improvement of 0.063 across ten learners. Matched random-weight and random-feature controls show that PACE gains from pretrained structure beyond generic dimensional expansion. PACE further achieves favorable performance–time trade-offs against task-fitted selectors and attribution-based methods, positioning pretrained column geometry as a reusable upstream primitive for tabular learning.
Read the full abstract ▾
All authors Qi Qin, Erbo Li, Ting Wei, Zizhou Huang, Zixuan Qin, Wu Wang, Yifan Sun
Statistical Methodology
Distributional Balancing with Machine Learning for Clinical Trial Augmentation Using Real-World Data
Zern Ke, Mingshi Cui, Gemma Moran et al. (+1 more)
In clinical trials, randomization of treatment and control groups is typically used to ensure the groups have similar
covariate distributions on average, resulting in unbiased causal effect estimation. Such balanced
covariate distributions are hard to achieve in practice, however, due to recruitment costs, patient dropouts, and more. One possible solution to this problem is to include control patients from external, real world databases. In this paper, we propose DBML, a method that selects control units from a real world database by matching the distribution between the treatment group and the potential control group, instead of matching units between two groups. DBML has three steps. First, we detect anomalous database units (with respect to the treatment distribution) using a variational autoencoder. Second, we re-weight the remaining database units to match the distribution of the treatments. Finally, we use these weights to sample units for our control group. The proposed method is compared to alternative matching-based
algorithms and weighted
algorithms, achieving superior performance in
covariate balance.
Read the full abstract ▾
All authors Zern Ke, Mingshi Cui, Gemma Moran, Javier Cabrera
Statistical Methodology
tteICE: An R Package for Estimating Treatment Effects on Time-to-Event Outcomes with Intercurrent Events in Two-Arm Trials
Yuhao Deng, Yi Zhou, Yi Wang et al. (+1 more)
Evaluating treatment effects in clinical trials with time-to-event outcomes is complicated by intercurrent events (ICEs), e.g., treatment discontinuation or competing events. While the ICH E9 (R1) addendum outlines five strategies to address these challenges, accessible software implementing these methods is limited. This article introduces the \pkg{tteICE} package for \proglang{R}, which implements five strategies to facilitate valid causal inference in the presence of ICEs. \pkg{tteICE} enables researchers to either nonparametrically or semiparametrically efficiently estimate potential cumulative incidence functions under both treatment and control conditions. The treatment effect is assessed by contrasting these functions, with statistical uncertainty quantified using pointwise confidence intervals and $p$-values. The package supports data from randomized controlled trials and observational studies, including settings with competing risks and semi-competing risks. To reduce barriers for applied researchers, \pkg{tteICE} features integrated plotting functions and an accompanying interactive Shiny application that provides a user-friendly graphical interface with step-by-step instructions.
Read the full abstract ▾
All authors Yuhao Deng, Yi Zhou, Yi Wang, Shasha Han
Statistical Methodology · Cross-listed from math.ST
Retrieving predictive densities from conformal predictive distributions
Dan Andrei Tudor, Dorota Kurowicka, Balint Negyesi et al. (+1 more)
Conformal prediction intervals are used to construct marginally well-calibrated regions around point predictions. Recent work has focused on extending these ideas to conformal predictive distributions, whose marginal probability integral transform (PIT) follows a Unif[0, 1] distribution. In this work, we investigate how to recover predictive densities from conformal predictive distributions. We extend the theory of conformal predictive distributions by proving asymptotic marginal validity for a tail-corrected version of conformal predictive distributions. We propose a method called
quantile matching, which preserves an upper bound on the deviation of the marginal PIT from uniformity. Furthermore, we show that the distribution induced by
quantile matching is equivalent to the crisp version of conformal predictive distributions when the number of
quantiles equals the size of the calibration set. For the recovery of conformal densities, we construct a fidelity-constrained bandwidth optimization for kernel smoothing that preserves asymptotic marginal validity and has a closed-form solution. We apply and compare our proposed methodology on a large simulated real estate transactions
dataset based on the Hierarchical Trend Model.
Read the full abstract ▾
All authors Dan Andrei Tudor, Dorota Kurowicka, Balint Negyesi, Valerii Zoller
Statistical Methodology
Streaming PCA: averaging from a geometric perspective
Tuan Pham, Alessandro Rinaldo and Purnamrita Sarkar
We study principal component analysis (PCA) under
memory constraints, a setting that is increasingly important in large-scale data analysis. Our focus is on Oja's
algorithm, which is a one-pass,
memory-efficient
algorithm requiring only $O(p)$ storage in the rank-one case and $O(pk)$ storage for $k$- PCA. The main goal is to develop a procedure based on Oja's
algorithm that is optimal without prior knowledge of the eigengap, which is otherwise needed to tune the learning rates effectively. We do this by first introducing a
geometric perspective on the convergence of $k$-PCA: by
embedding the Oja iterates into the exterior space, we establish an exact equivalence between $k$-PCA in the ambient space and $1$-PCA in the exterior space. This
geometric viewpoint enables us to establish several convergence guarantees, including one based only on a single learning-rate schedule. Building on this
geometric perspective, we develop an averaging theory for $k$-PCA and show that the resulting averaged
estimator is adaptive: it achieves the nearly optimal convergence rate without prior knowledge of the eigengap. As an application, we construct
memory-efficient
estimators for elliptical component analysis (ECA) \cite{han2014scale,han2018eca}. Simulation studies and real data analysis are conducted to demonstrate the benefits of our proposed
algorithms.
Read the full abstract ▾
Statistical Methodology
Unbiased Treatment Effect Estimation under Network Interference via Neighborhood-Excluded Cross-Fitting
Haoyang Yu, Anqi Zhao and Hanzhong Liu
Without interference, cross-fitting enables flexible
covariate adjustment while preserving finite-sample unbiasedness under independent unit-level randomization. Under network interference, out-of-sample prediction alone no longer guarantees unbiasedness: assignments entering evaluation-fold Horvitz–Thompson weights may also affect outcomes in the training sample, inducing dependence between fitted predictions and those weights. We develop neighborhood-excluded cross-fitting, which constructs estimand- and design-specific training samples to restore the conditional independence needed for finite-sample unbiasedness without a correctly specified outcome model. We establish asymptotically valid design-based Wald inference for direct and indirect effects under Bernoulli randomization and for the global average treatment effect under Bernoulli cluster randomization. Neighborhood exclusion creates a trade-off in choosing the number of folds: unit-level splitting may require the number of folds to grow with average exclusion-neighborhood size, while cluster-level splitting can substantially relaxes this requirement, permitting a fixed number of folds under partial interference. For linear adjustment under Bernoulli randomization, we derive variance-optimal and confidence-interval-length-optimal procedures, establish explicit rate conditions allowing the
covariate dimension to diverge, and show that the variance-optimal procedure is asymptotically no-harm. Simulations illustrate the bias from omitting neighborhood exclusion and the precision gains from adjustment. An application to a social network experiment yields confidence intervals substantially shorter than those from the unadjusted
estimator.
Read the full abstract ▾