Statistical Methodology · Cross-listed from cs.LG
Three Ways Classical Test Theory Can Mislead About LLM Judges
Louis Yiven Zhu
Evaluations that use a large
language model (LLM) as a judge have begun to borrow reliability statistics from classical test theory and its extensions. We examine three such statistics that need one administration and no gold labels. None of them can isolate the judge, because one judge under one
prompt supplies no variance component of its own. Claude Haiku 4.5 judged 210 constructed short answers against ten-element checklists. On the 180 with parsed verdicts, the Kuder-Richardson coefficient (KR-20) came out at 0.5223 on the judge's verdicts and 0.5231 on error-free gold verdicts. In simulation, bank design alone moves KR-20 from 0.01 to 0.68 at the judge's measured 4.72% error rate. The dependability index $\Phi(\lambda)$, a ratio of mean squared distances from the pass mark, sits 0.22 to 0.38 below the judge's accuracy against gold and returns 0.54 to 0.68 on error-free gold verdicts. Livingston-Lewis accuracy treats the rubric elements as a sample, and at a pass mark of five elements it credits error-free gold scores with 0.78, close to the judge's 0.81. A statement about the judge therefore needs gold labels or a varied scorer facet, and a reliability ratio needs the bank's spread beside it. One of the four closest judge-evaluation papers varies the
prompt and still reads a reliability below 0.7 as a sign that a model cannot serve as a judge, although that reliability moves with the spread of the samples scored. We derive a decision table and four reporting lines from these two rules.
Read the full abstract ▾
Statistical Methodology
A normal-inverse-Wishart (NIW) Bayesian synthesizer for multivariate normal data with application to polygenic risk scores
Rasmus Rask Kragh Jørgensen, Anne Krogh Nøhr, Jan Reiter Sørensen et al. (+2 more)
Bayesian synthesis, which generates synthetic data by sampling from the posterior predictive distribution, is a popular approach for privatizing sensitive personal data. However, how attribute disclosure risk is affected by feature dimensionality, the number of individuals in the original
dataset, and the amount of released synthetic information remains poorly understood. We propose a mathematically tractable
Bayesian synthesizer for multivariate normal data based on a conjugate normal-inverse-Wishart prior for the mean vector and covariance
matrix. The conjugate structure yields closed-form posteriors and enables direct investigation of an adversary's ability to infer records under different data and release settings. We then demonstrate several intuitive properties of synthetic data generation through several simulations. Specifically, we show that disclosure risk decreases with the size of the original
dataset, but increases with the dimensionality of the feature space and the amount of synthetic information released, whether through the release of larger synthetic
datasets or multiple generator realizations. Finally, the proposed synthesizer was used to generate synthetic versions of a polygenic risk score
dataset, with the synthetic data exhibiting distributional properties comparable to those of the original data.
Read the full abstract ▾
All authors Rasmus Rask Kragh Jørgensen, Anne Krogh Nøhr, Jan Reiter Sørensen, Martin Bøgsted, Heidi Søgaard Christensen
Statistical Methodology
Genetic association testing with multivariate survival phenotypes under interval censoring
Juhee Lee, Kun Xia, Jianrui Zhang et al. (+3 more)
Set-based genetic association tests provide a powerful framework for detecting genetic effects on complex traits by jointly analyzing multiple genetic variants. Although set-based methods have been developed for interval-censored survival outcomes, existing approaches primarily focus on a single survival phenotype and therefore do not fully use information from multiple correlated outcomes. In this paper, we develop two Weighted V Tests for Multivariate Interval-Censored Data (WV-M-IC), extending the weighted V-statistic framework (Wu et al., 2021) to the joint analysis of multiple correlated interval-censored survival outcomes. The performance of these methods is evaluated through simulation studies, showing that the proposed approaches can provide power gains compared with single-outcome analyses. We apply the proposed methods to the ZOE 2.0 study to investigate dental caries progression in children.
Read the full abstract ▾
All authors Juhee Lee, Kun Xia, Jianrui Zhang, Gongjun Xu, Qing Lu, Chenxi Li
Statistical Methodology · Cross-listed from stat.ML
A convolutional framework for detecting event-driven dynamics in energy price series
Caixia Xu and Piotr Fryzlewicz
This paper develops a general convolutional neural network (CNN) framework for detecting heterogeneous event-driven dynamics in univariate time series windows. We show that the induced CNN class exactly represents classifiers based on range, maximum drawup, maximum drawdown and slope change, and uniformly approximates realised
volatility and autoregressive explosiveness on compact domains. We further establish error bounds for representative rules in finite samples and an oracle inequality for learning across them. Simulations show that the proposed model can match or outperform classifiers based on individual statistics as the training sample grows. In an application to six daily energy price series, a hierarchical CNN distinguishes event windows and event families. Applied without retraining to observations withheld after 20 February 2026, the fitted model identifies predominantly geopolitical dynamics in several oil and refined product series around the outbreak of the 2026 Iran war, while distinguishing a contemporaneous natural gas spike associated with weather.
Read the full abstract ▾
Statistical Methodology
Generalized Bayesian Clustering with Regression for Unaligned Longitudinal Binary Data
Khai Nguyen, Elizabeth Juarez-Colunga and Peter Mueller
We propose a generalized
Bayesian clustering with regression model for unaligned longitudinal binary outcomes, motivated by seizure diary data from the Human Epilepsy Project. Seizure diaries are sparse, irregularly observed, and vary enormously across patients. A single fully-specified generative model tends to be either misspecified or computationally inefficient. We address the challenge by two strategies. We set up a regression by way of clustering as model-based clustering using a mixture model. For the latter, we take a generalized
Bayesian perspective which replaces the full likelihood with a loss-based update using a generalized likelihood. We combine a trajectory similarity loss and a regression loss, so that clustering is informed by both trajectory similarity and the prediction of outcomes The trajectory similarity loss is constructed by representing each trajectory as an (empirical) distribution of subsequences, called reads, and then is defined based on the sliced Wasserstein distance between these empirical distributions. This loss allows
alignment-free comparison of sequences that are irregularly observed or temporally misaligned, and it scales quasi-linearly in trajectory length. The regression loss is the negative log-likelihood of a probit regression. A prior on the cluster-specific parameters is defined by way of a Dirichlet process prior on the mixing measure.
Read the full abstract ▾
Statistical Methodology
Parameterising Gaussian Graphical Models
Jack Storror Carter
Gaussian graphical models (GGMs) describe the dependence structure among jointly Gaussian random variables. However, the most common parameterisation of GGMs, the precision
matrix, describes both the dependence and scale of the variables. This has been shown to lead to model selection methods that depend on the scale of the variables, despite graphical models being scale invariant. Even after standardising data to have unit sample variances, entries of the precision
matrix can be on different scales leading to poor model selection. This paper decomposes the precision
matrix into marginal variances and interpretable scale-invariant parameters - the partial correlations and variance inflation factors. This decomposition gives new insights into the precision
matrix entries and the dynamics of model selection methods such as penalised likelihoods. In particular, it explains the observed phenomenon that penalties on the precision
matrix perform poorly at selecting hub variables and motivates the necessity of data standardisation. It also shows why methods based on partial correlations have better hub detection properties. The effect of penalisation of different quantities on the estimation of marginal variances is then investigated and an interesting simplification of the log-likelihood is shown when using maximum likelihood estimation of the marginal variances.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.AI
Different representation learning objectives recover distinct latent structures from the same psychometric data
Cong Cao, Tassos C. Kyriakides and Pambos Vrasidas
Psychometric questionnaires contain rich item-level information, yet it remains unclear whether different representation learning objectives recover the same latent organization. We investigated this question using 757 matched teacher-child pairs from the baseline assessment of the Cyprus ProW preschool trial. Behavioral structure was characterized from child SDQ, ASBI, and CBRS item responses using principal component analysis and clustering, yielding four behavioral phenotypes. A contrastive objective substantially improved teacher-child
retrieval relative to PCA-based representations, increasing Top-1 accuracy from 0.13% to 7.27% and Top-10 accuracy from 1.98% to 56.14%. However, contrastive representations preserved behavioral phenotype structure less effectively than PCA-based representations. A multi-task objective jointly optimizing
alignment and behavioral prediction partially restored behavioral organization but reduced
retrieval performance. These findings indicate that teacher-child correspondence and behavioral phenotypes represent distinct forms of latent organization and demonstrate that the latent structure recovered from linked psychometric data depends on the representation learning objective.
Read the full abstract ▾
Statistical Methodology · Cross-listed from math.ST
Scale Analysis and Shape Selection for the Generalized Gaussian Mechanism under Approximate Differential Privacy
Xiang Zhang, Mohamedou Ould Haye and Yiqiang Q. Zhao
Differential
privacy provides a rigorous framework for protecting private information, typically achieved by adding random noise to query results. The generalized Gaussian family is a flexible class of additive noise distributions indexed by the shape parameter $p$ and includes the Laplace and Gaussian distributions as special cases $p=1$ and $p=2$, respectively. This paper studies the
privacy-feasible scale estimation and the shape parameter selection of the generalized Gaussian mechanism (GGM) under $(\varepsilon,\delta)$-differential
privacy. For a given sensitivity vector $\Delta$ and $p\in[1,\infty]$, let $b(p)$ denote the smallest value of the scale parameter for which the mechanism satisfies this
privacy requirement. In the one-dimensional case, $b(p)$ can be implicitly characterized by a system of equations. For vector-valued queries, we construct a computable upper approximation of $b(p)$ that preserves the
privacy guarantee. Shapes are compared under a scale-homogeneous utility criterion, with the $m$-th absolute moment as the main example. We develop an interval-wise shape search
algorithm with an approximation guarantee that can be made arbitrarily precise. We also establish the invariance of the optimal shape under rescaling of the sensitivity vector and characterize its limiting behaviour under high
privacy limits. Computational experiments show that optimizing shape parameters can improve utility by reducing the variance of each coordinate by 5% to 20% across a variety of cases, with some cases showing even greater reductions, while maintaining the same level of
privacy protection. Task-specific experiments further show that shape optimization can improve task-level utility, reduce attacker success, or achieve both.
Read the full abstract ▾
Statistical Methodology
Posterior Geometry and Identifiability in Multi-Response Bayesian Calibration
Anton van Beek, Adam. M Boyce, Will J. Dawson et al. (+1 more)
Calibration under model misspecification is inherently ill-posed because calibration parameters and structural discrepancy are statistically confounded without additional assumptions.
Bayesian formulations address this ambiguity through prior and covariance modeling choices, including multi-response observations and cross-source
alignment. However, how these choices, together with output-dependent uncertainty, shape posterior geometry and practical parameter identifiability remains poorly understood. We present a latent-variable multi-response calibration model that combines empirical Bayes estimation of nuisance hyperparameters with a Fisher-information-based local Gaussian approximation of the calibration-parameter posterior. Closed-form expressions are derived for squared-exponential and Matérn covariance functions, revealing how multi-response correlations, latent
alignment, and output-specific uncertainty influence local posterior geometry. Controlled numerical experiments show that additional responses provide complementary
geometric constraints on calibration parameters, while noise modeling can improve uncertainty quantification by reducing overconfidence without necessarily improving parameter localization. A lithium-ion battery study further illustrates how the model distinguishes strongly from weakly constrained parameters. Together, these results establish expected-Fisher posterior geometry as a scalable diagnostic of local parameter identifiability in multi-response calibration.
Read the full abstract ▾
All authors Anton van Beek, Adam. M Boyce, Will J. Dawson, James B. Robinson
Statistical Methodology
Aggregate Disambiguation Systems
José María Lago, Albert Castellana and Edgars Nemše
Natural-
language tasks can elicit different verdicts from protocol-following evaluators that receive the same declared information. We study aggregate disambiguation systems (ADSs). Given a task and a candidate solution, each evaluator casts a binary vote on whether the solution should be accepted, and the system aggregates the votes of a finite panel. The target is protocol reproducibility relative to an explicitly declared evaluator reference, not semantic truth. We separate fixed finite censuses, probabilistic evaluator populations, and growing-census limits, since their endpoint laws and guarantees are not interchangeable. In the population setting, we use finite samples to estimate how often a finite panel reaches the same decision as the declared evaluator population. We provide a lower confidence bound on the fraction of candidate solutions for which the disagreement probability is at most a chosen tolerance. The calculation accounts separately for sampling candidate solutions and sampling evaluators. The construction permits arbitrary dependence among columns induced by shared evaluator rows and uses exact binomial intervals at the evaluator layer and an exact one-sided binomial inversion at the generator layer. Simulations check the implementation against known population coverages and expose power limitations.
Read the full abstract ▾
Statistical Methodology
From Test Performance to Risk-Based Effect Sizes: A Unified Wald-Type Framework to Design Clinical Validation Studies for Binary and Survival Outcomes
Yongqi Zhong, Anne-Renee Hartman and Jing Zhang
Clinical validation studies of predictive tests are usually designed to focus on sensitivity ($Se$) and specificity ($Sp$), while statistical power is often calculated on regression-effect scales (e.g., risk ratio, hazard ratio). However, these quantities are statistically connected. Here, we provide closed-form links from sensitivity, specificity, and disease prevalence ($\pi$) to predictive risks, risk contrasts, and Wald-type variance, power, and sample-size formulas for binary and fixed-horizon survival outcomes. Analyses of statistical efficiency via C- and D-optimal principles demonstrate how prevalence and threshold choices affect study efficiency, supporting rapid decisions in preliminary studies and informing the design of subsequent, larger studies. Simulations show good calibration across most realistic scenarios; when events are rare and test effects are simultaneously very large, continuity and minimum-event corrections are needed to stabilize the approximation. We illustrate the framework with a case study describing use of the coronary artery calcium score for predicting incident cardiovascular disease in patients with type 2 diabetes mellitus. The formulas let investigators check power and required enrollment directly from $(Se,Sp,\pi)$, without running a separate simulation for each design candidate.
Read the full abstract ▾
Statistical Methodology
Marginal Coordinate Test for Fréchet Regression with Random Objects
Jiaye Chen, Rui Qiu, Roulin Wang et al. (+1 more)
We develop a marginal coordinate test for regression with Euclidean predictors and a random-object response in a separable metric space. The goal is to test whether a predictor provides additional information about the response conditional on the remaining predictors. In a semi-supervised design, an unlabeled sample is used to estimate predictor conditional means, while an independent labeled sample is reserved for inference. The resulting residuals are combined with a product-space kernel to form a kernel conditional mean dependence (KCMD) U-statistic without requiring a response residual. The primary identity-based test targets a necessary conditional mean restriction, while a multiple-transformation extension probes broader alternatives. We establish a weighted centered chi-square null limit, wild bootstrap validity, consistency against fixed detectable alternatives, and local power under mean-element alternatives. For simultaneous inference, truncated p-to-e calibration combined with e-BH provides asymptotic false discovery rate control under general dependence. Simulations with Euclidean and non-Euclidean responses, together with a New York City taxi-flow analysis, illustrate the method.
Read the full abstract ▾
All authors Jiaye Chen, Rui Qiu, Roulin Wang, Zhou Yu
Statistical Methodology · Cross-listed from cs.LG
When the Martingale Never Stops Firing: Anytime-Valid Gating on Real Forecast Streams
Weijia Han and Lisha Qu
Machine learning systems are increasingly corrected while they run, and the decision of when to intervene is increasingly delegated to statistical monitors. Anytime-valid inference promises evidence that can be acted on at any moment, exactly the guarantee this setting needs, and it is moving from theory into deployed monitoring. Conformal test martingales are the change-detection instrument, and Ville's inequality caps their false-alarm probability on exchangeable data. The guarantee is conditional. A deployment inherits it only if the stream it monitors behaves exchangeably. The premise is hardest to satisfy where these monitors are most useful, on dependent data and inside loops where the monitor modifies the learner whose scores it reads. It is also rarely measured. We measure it in a pre-specified case study, where such a monitor gates the online updates of a Kalman adapter correcting frozen time-series foundation models on five forecasting streams. On exchangeable synthetic streams, the same implementation fires in at most 1 of 60 runs. On the real streams, at alpha = 0.05, 135 of 135 clean-stream runs fired. The construction does not explain the firing; the failure comes from the deployed score stream itself. Repeated fires hold the gate's drift response active, and the gated filter amplifies the very transient it was designed to prevent. The component worth keeping makes no validity claim. Huber-style gating of the filter's own updates cuts isolated-spike degradation by an order of magnitude with no
dataset specific tuning. Anytime-valid methods proposed for dependent data should therefore be accompanied by null-calibration controls and mechanism traces.
Read the full abstract ▾
Statistical Methodology
Power and sample size calculations for causal mediation analysis with a binary mediator in randomized trials
Bosen Cui, Yuhong Yang and Fan Yang
Mediation analyses are increasingly conducted in randomized trials, but a sample size adequate for the total treatment effect may leave the natural indirect effect (NIE) or natural direct effect (NDE) substantially underpowered. Randomization does not extend to the mediator, so precision depends on the conditional mediator distribution and the mediator-outcome association, neither of which enters a total-effect calculation. Planning outside linear structural equation models is largely based on simulation under a fully specified data-generating mechanism rarely available at the design stage. This paper develops analytic power and sample size formulas for the NIE and NDE with a binary mediator and a continuous or binary outcome. Under standard identification assumptions, we focus on the ratio-of-mediator-probability weighting (RMPW)
estimator that does not require an outcome model for effect estimation. We decompose the oracle variances of the RMPW
estimators into components capturing mediator-probability-ratio variability, outcome variation, and their association, with an additional shared-arm covariance term for the NIE. Under a probit latent-index mediator and a working outcome model, these components are determined by a small number of interpretable design inputs rather than by the full joint distribution of
covariates, mediator, and outcome. Simulations show that the analytic sample sizes closely match simulation-based
benchmarks, attain the target power, and maintain type I error near the nominal level. An ACTG175 illustration shows how pilot data can calibrate the inputs.
Read the full abstract ▾
Statistical Methodology
Causal inference with staggered entries and effects that change over calendar time
Lorenzo Gasparollo and Mats J. Stensrud
Studies with staggered entry, in which individuals enroll at different calendar times, are ubiquitous in medicine and related disciplines. Because these studies usually have a fixed administrative end of follow-up, identification of the estimand of interest relies on assumptions about the right-censoring mechanism. The assumptions are often considered plausible only conditional on
covariates, including the time an individual entered the study (E). Yet, censoring assumptions formulated conditional on E are ill-posed due to positivity violations. Here, we study the consequences of this issue for common procedures in causal survival analysis, such as those based on marginal structural models. We further give conditions for valid identification and introduce sensitivity analyses to assess practical implications. We illustrate our methodology through two case studies. The first builds on the seminal article by Hernán et al. (2000) on the effect of zidovudine treatment. The second reanalyzes a recent study on the effect of the mRNA vaccine in patients receiving immune checkpoint inhibitors.
Read the full abstract ▾