Statistical Methodology · Cross-listed from cs.LG
LIDS: LLM Summary Inference Under the Layered Lens
Dylan Park, Yingying Fan and Jinchi Lv
Large
language models (LLMs) have gained significant
attention by many researchers and practitioners in natural
language processing (NLP) since the introduction of ChatGPT in 2022. One notable feature of ChatGPT is its ability to generate summaries based on
prompts. Yet evaluating the quality of these summaries remains challenging due to the complexity of
language. To this end, in this paper we suggest a new method of LLM summary inference with BERT-SVD-based direction metric and SOFARI (LIDS) that assesses the summary accuracy equipped with interpretable key words for layered themes. The LIDS uses a latent SVD-based direction metric to measure the similarity between the summaries and original text, leveraging the BERT
embeddings and repeated
prompts to quantify the statistical uncertainty. As a result, LIDS gives a natural
embedding of each summary for large text reduction. We further exploit SOFARI to uncover important key words associated with each latent theme in the summary with controlled false discovery rate (FDR). Comprehensive empirical studies demonstrate the practical utility and robustness of LIDS through human verification and comparisons to other similarity metrics, including a comparison of different LLMs.
Read the full abstract ▾
Statistical Methodology
M-estimation under Two-Phase Multiwave Sampling with Applications to Prediction-Powered Inference
Dan M. Kluger and Stephen Bates
In two-phase multiwave sampling, inexpensive measurements are collected on a large sample and expensive, more informative measurements are adaptively obtained on subsets of units across multiple waves. Adaptively collecting the expensive measurements can increase efficiency but complicates statistical inference. We give valid
estimators and confidence intervals for M-estimation under adaptive two-phase multiwave sampling. We focus on the case where proxies for the expensive variables – such as predictions from pretrained machine learning models – are available for all units and propose a Multiwave Predict-Then-Debias
estimator that combines proxy information with the expensive, higher-quality measurements to improve efficiency while removing bias. We establish asymptotic linearity and normality and propose asymptotically valid confidence intervals. We also develop an approximately greedy sampling strategy that improves efficiency relative to uniform sampling. Data-based simulation studies support the theoretical results and demonstrate efficiency gains.
Read the full abstract ▾
Machine Learning (Statistics)
Poisson-MNL Bandit: Nearly Optimal Dynamic Joint Assortment and Pricing with Decision-Dependent Customer Arrivals
Junhui Cai, Ran Chen, Qitao Huang et al. (+2 more)
We study dynamic joint assortment and pricing where a seller updates decisions at regular accounting/operating intervals to maximize the cumulative per-period revenue over a horizon $T$. In many settings, assortment and prices affect not only what an arriving customer buys but also how many customers arrive within the period, whereas classical multinomial logit (MNL) models assume arrivals as fixed, potentially leading to suboptimal decisions. We propose a Poisson-MNL model that couples a contextual MNL choice model with a Poisson arrival model whose rate depends on the offered assortment and prices. Building on this model, we develop an efficient
algorithm PMNL based on the idea of upper confidence bound (UCB). We establish its (near) optimality by proving a non-asymptotic regret bound of order $\sqrt{T\log{T}}$ and a matching lower bound (up to $\log T$). Simulation studies underscore the importance of accounting for the dependency of arrival rates on assortment and pricing: PMNL effectively learns customer choice and arrival models and provides joint assortment-pricing decisions that outperform others that assume fixed arrival rates.
Read the full abstract ▾
All authors Junhui Cai, Ran Chen, Qitao Huang, Linda Zhao, Wu Zhu
Statistical Methodology
A statistical perspective on transformers for small longitudinal cohort data
Kiana Farhadyar, Maren Hackenberg, Kira Ahrens et al. (+10 more)
Modeling of longitudinal cohort data typically involves complex temporal dependencies between multiple variables. There, the
transformer architecture, which has been highly successful in
language and
vision applications, allows us to account for the fact that the most recently observed time points in an individual's history may not always be the most important for the immediate future. This is achieved by assigning
attention weights to observations of an individual based on a transformation of their values. One reason why these ideas have not yet been fully leveraged for longitudinal cohort data is that typically, large
datasets are required. Therefore, we present a simplified
transformer architecture that retains the core
attention mechanism while reducing the number of parameters to be estimated, to be more suitable for small
datasets with few time points. Guided by a statistical perspective on
transformers, we use an autoregressive model as a starting point and incorporate
attention as a kernel-based operation with temporal decay, where aggregation of multiple
transformer heads, i.e. different candidate weighting schemes, is expressed as accumulating evidence on different types of underlying characteristics of individuals. This also enables a permutation-based statistical testing procedure for identifying contextual patterns. In a simulation study, the approach is shown to recover contextual dependencies even with a small number of individuals and time points. In an application to data from a resilience study, we identify temporal patterns in the dynamics of stress and mental health. This indicates that properly adapted
transformers can not only achieve competitive predictive performance, but also uncover complex context dependencies in small data settings.
Read the full abstract ▾
All authors Kiana Farhadyar, Maren Hackenberg, Kira Ahrens, Charlotte Schenk, Bianca Kollmann, Oliver Tüscher, Klaus Lieb, Michael M. Plichta, Andreas Reif, Raffael Kalisch, Martin Wolkewitz, Moritz Hess, Harald Binder
Machine Learning (Statistics) · Cross-listed from cs.LG
ML-driven detection and reduction of ballast information in multi-modal datasets
Yaroslav Solovko
Modern
datasets often contain ballast as redundant or low-utility information that increases dimensionality, storage requirements, and computational cost without contributing meaningful analytical value. This study introduces a generalized,
multimodal framework for ballast detection and reduction across structured, semi-structured, unstructured, and sparse data types. Using diverse
datasets,
entropy, mutual information, Lasso, SHAP, PCA, topic modelling, and
embedding analysis are applied to identify and eliminate ballast features. A novel Ballast Score is proposed to integrate these signals into a unified, cross-modal
pruning strategy. Experimental results demonstrate that significant portions of the feature space as often exceeding 70% in sparse or semi-structured data, can be pruned with minimal or even improved classification performance, along with substantial reductions in training time and
memory footprint. The framework reveals distinct ballast typologies (e.g. statistical, semantic, infrastructural), and offers practical guidance for leaner, more efficient machine learning pipelines.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
Jianliang He, Leda Wang, Siyu Chen et al. (+1 more)
We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual
neurons learn single-frequency
Fourier features and phase
alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase
symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual
neurons produce noisy signals, the phase
symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify the correct sum. Furthermore, we explain the emergence of these features under random initialization via a lottery ticket mechanism. Our gradient flow analysis proves that frequencies compete within each
neuron, with the "winner" determined by its initial spectral magnitude and phase
alignment. From a technical standpoint, we provide a rigorous characterization of the layer-wise phase coupling dynamics and formalize the competitive landscape using the ODE comparison lemma. Finally, we use these insights to demystify grokking, characterizing it as a three-stage process involving memorization followed by two generalization phases, driven by the competition between loss minimization and weight decay.
Read the full abstract ▾
All authors Jianliang He, Leda Wang, Siyu Chen, Zhuoran Yang
Statistics Applications
The Impact of Formations on Football Matches Using Double Machine Learning. Is it worth parking the bus?
Genís Ruiz-Menárguez and Llorenç Badiella
This study addresses a central tactical dilemma for football coaches: whether to employ a defensive strategy, colloquially known as "parking the bus", or a more offensive one. Using an advanced Double Machine Learning (DML) framework, this project provides a robust and interpretable tool to estimate the causal impact of different formations on key match outcomes such as goal difference, possession, corners, and disciplinary actions. Leveraging a
dataset of over 22,000 matches from top European leagues, formations were categorized into six representative types based on tactical structure and expert consultation. A major methodological contribution lies in the adaptation of DML to handle categorical treatments, specifically formation combinations, through a novel
matrix-based residualization process, allowing for a detailed estimation of formation-versus-formation effects that can inform a coach's tactical decision-making. Results show that while offensive formations like 4-3-3 and 4-2-3-1 offer modest statistical advantages in possession and corners, their impact on goals is limited. Furthermore, no evidence supports the idea that defensive formations, commonly associated with parking the bus, increase a team's winning potential. Additionally, red cards appear unaffected by formation choice, suggesting other behavioral factors dominate. Although this approach does not fully capture all aspects of playing style or team strength, it provides a valuable framework for coaches to analyze tactical efficiency and sets a precedent for future research in sports analytics.
Read the full abstract ▾
Machine Learning (Statistics)
Beyond Procedure: Substantive Fairness in Conformal Prediction
Pengqi Liu, Zijun Yu, Mouloud Belbahri et al. (+3 more)
Conformal prediction (CP) offers distribution-free uncertainty quantification for machine learning models, yet its interplay with
fairness in downstream decision-making remains underexplored. Moving beyond CP as a standalone operation (procedural
fairness), we analyze the holistic decision-making pipeline to evaluate substantive
fairness-the equity of downstream outcomes. Theoretically, we derive an upper bound that decomposes prediction-set size disparity into interpretable components, clarifying how label-clustered CP helps control method-driven contributions to unfairness. To facilitate scalable empirical analysis, we introduce an LLM-in-the-loop evaluator that approximates human assessment of substantive
fairness across diverse modalities. Our experiments show that label-clustered CP often provides a favorable balance between utility and substantive
fairness, while reducing set-size disparities in line with our theory. Finally, we empirically show that equalized set sizes, rather than coverage, strongly correlate with improved substantive
fairness, enabling practitioners to design more fair CP systems. Our code is available at https://github.com/layer6ai-labs/llm-in-the-loop-conformal-
fairness.
Read the full abstract ▾
All authors Pengqi Liu, Zijun Yu, Mouloud Belbahri, Arthur Charpentier, Masoud Asgharian, Jesse C. Cresswell
Statistical Methodology
First versus full or first versus last: U-statistic change-point tests under fixed and local alternatives
Herold Dehling, Daniel Vogel and Martin Wendler
The use of U-statistics in the change-point context has received considerable
attention in the literature. We compare two approaches of constructing CUSUM-type change-point tests, which we call the first-vs-full and first-vs-last approach. Both have been pursued by different authors. The question naturally arises if the two tests substantially differ and, if so, which of them is better in which data situation. In large samples, both tests are similar: they are asymptotically equivalent under the null hypothesis and under sequences of local alternatives. In small samples, there may be quite noticeable differences, which is in line with a different asymptotic behavior under fixed alternatives. We derive a simple criterion for deciding which test is more powerful. We examine the examples Gini's mean difference, the sample variance, and Kendall's tau in detail. Particularly, when testing for changes in scale by Gini's mean difference, we show that the first-vs-full approach has a higher power if and only if the scale changes from a smaller to a larger value – regardless of the population distribution or the location of the change. The asymptotic derivations are under weak dependence. The results are illustrated by numerical simulations and data examples.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.LG
Omitted Variable Bias in Language Models Under Distribution Shift
Victoria Lin, Louis-Philippe Morency and Eli Ben-Michael
Despite their impressive performance on a wide variety of tasks, modern
language models remain susceptible to distribution shifts, exhibiting brittle behavior when evaluated on data that differs in distribution from their training data. In this paper, we describe how distribution shifts in
language models can be separated into observable and unobservable components, and we discuss how established approaches for dealing with distribution shift address only the former. Importantly, we identify that the resulting omitted variable bias from unobserved variables can compromise both evaluation and optimization in
language models. To address this challenge, we introduce a framework that maps the strength of the omitted variables to bounds on the worst-case generalization performance of
language models under distribution shift. In empirical experiments, we show that using these bounds directly in
language model evaluation and optimization provides more principled measures of out-of-distribution performance, improves true out-of-distribution performance relative to standard distribution shift adjustment methods, and further enables inference about the strength of the omitted variables when target distribution labels are available.
Read the full abstract ▾
Statistics Theory · Cross-listed from cs.LG
Knowledge-Embedded Latent Projection for Robust Representation Learning
Weijing Tang, Ming Yuan, Zongqi Xia et al. (+1 more)
Latent space models are widely used for analyzing high-dimensional discrete data
matrices, such as patient-feature
matrices in electronic health records (EHRs), by capturing complex dependence structures through low-dimensional
embeddings. However, estimation becomes challenging in the imbalanced regime, where one
matrix dimension is much larger than the other. In EHR applications, cohort sizes are often limited by disease prevalence or data availability, whereas the feature space remains extremely large due to the breadth of medical coding system. Motivated by the increasing availability of external semantic
embeddings, such as pre-trained
embeddings of clinical concepts in EHRs, we propose a knowledge-embedded latent projection model that leverages semantic side information to regularize representation learning. Specifically, we model column
embeddings as smooth functions of semantic
embeddings via a mapping in a reproducing kernel
Hilbert space. We develop a computationally efficient two-step estimation procedure that combines semantically guided subspace construction via kernel principal component analysis with scalable projected gradient descent. We establish estimation error bounds that characterize the trade-off between statistical error and approximation error induced by the kernel projection. Furthermore, we provide local convergence guarantees for our non-convex optimization procedure. Extensive simulation studies and a real-world EHR application demonstrate the effectiveness of the proposed method.
Read the full abstract ▾
All authors Weijing Tang, Ming Yuan, Zongqi Xia, Tianxi Cai
Statistical Methodology
Synthetic-Powered Multiple Testing with FDR Control
Yonghoon Lee, Meshi Bashari, Edgar Dobriban et al. (+1 more)
Multiple hypothesis testing with false discovery rate (FDR) control is a fundamental problem in statistical inference, with broad applications in genomics, drug screening, and outlier detection. In many such settings, researchers may have access not only to real experimental observations but also to auxiliary or synthetic data – from past, related experiments or generated by generative models – that can provide additional evidence about the hypotheses of interest. We introduce SynthBH, a synthetic-powered multiple testing procedure that safely leverages such synthetic data. We prove that SynthBH guarantees finite-sample, distribution-free FDR control under a mild PRDS-type positive dependence condition, without requiring the pooled-data p-values to be valid under the null. The proposed method adapts to the (unknown) quality of the synthetic data: it enhances the sample efficiency and may boost the power when synthetic data are of high quality, while controlling the FDR at a user-specified level regardless of their quality. We demonstrate the empirical performance of SynthBH on tabular outlier detection
benchmarks and on genomic analyses of drug-cancer sensitivity associations, and further study its properties through controlled experiments on simulated data.
Read the full abstract ▾
All authors Yonghoon Lee, Meshi Bashari, Edgar Dobriban, Yaniv Romano
Machine Learning (Statistics)
Enhanced Diffusion Sampling: Efficient Rare Event Sampling and Free Energy Calculation with Diffusion Models
Yu Xie, Ludwig Winkler, Lixin Sun et al. (+10 more)
The rare-event sampling problem has long been the central limiting factor in molecular dynamics (MD), especially in biomolecular simulation. Recently,
diffusion models such as BioEmu have emerged as powerful
equilibrium samplers that generate independent samples from complex molecular distributions, eliminating the cost of sampling rare transition events. However, a sampling problem remains when computing observables that rely on states which are rare in
equilibrium, for example folding free energies. Here, we introduce enhanced
diffusion sampling, enabling efficient exploration of rare-event regions while preserving unbiased thermodynamic
estimators. The key idea is to perform quantitatively accurate steering protocols to generate biased ensembles and subsequently recover
equilibrium statistics via exact reweighting. We instantiate our framework in three
algorithms: UmbrellaDiff (umbrella sampling with
diffusion models), MetaDiff (a batchwise analogue for metadynamics), and $\Delta$G-Diff (free-energy differences via tilted ensembles). Across toy systems,
protein folding landscapes and folding free energies, our methods achieve fast, accurate, and scalable estimation of
equilibrium properties within GPU-minutes to hours per system-closing the rare-event sampling gap that remained after the advent of
diffusion-model
equilibrium samplers.
Read the full abstract ▾
All authors Yu Xie, Ludwig Winkler, Lixin Sun, Sarah Lewis, Adam E. Foster, José Jiménez Luna, Tim Hempel, Michael Gastegger, Yaoyi Chen, Iryna Zaporozhets, Cecilia Clementi, Christopher M. Bishop, Frank Noé
Statistics Applications
Design and Analysis Strategies for Pooling in High Throughput Screening: Application to the Search for a New Anti-Microbial
Byran Smucker, Benjamin Brennan, Emily Rego et al. (+4 more)
A major public health issue is the growing resistance of bacteria to antibiotics. An important part of the needed response is the discovery and development of new antimicrobial strategies. These require the screening of potential new drugs, typically accomplished using high-throughput screening (HTS). Traditionally, HTS is performed by examining one compound per well, but a more efficient strategy pools multiple compounds per well. In this work, we study several recently proposed pooling construction methods, as well as a variety of pooled high-throughput screening analysis methods, in order to provide guidance to practitioners on which methods to use. This is done in the context of an application of the methods to the search for new drugs to combat bacterial infection. We discuss both an extensive pilot study as well as a small screening campaign, and highlight both the successes and challenges of the pooling approach.
Read the full abstract ▾
All authors Byran Smucker, Benjamin Brennan, Emily Rego, Meng Wu, Zhihong Lin, Brian Ahmer, Blake Peterson
Statistics Theory
On Sharpened Convergence Rate of Generalized Sliced Inverse Regression for Nonlinear Sufficient Dimension Reduction
Chak Fung Choi, Yin Tang and Bing Li
Generalized Sliced Inverse Regression (GSIR) is one of the most important methods for nonlinear sufficient dimension reduction. As shown in Li and Song (2017), it enjoys a convergence rate that is independent of the dimension of the predictor, thus avoiding the curse of dimensionality. In this paper we establish an improved convergence rate of GSIR under additional mild
eigenvalue decay rate and smoothness conditions. Our convergence rate can be made arbitrarily close to $n^{-1/3}$ under appropriate decay rate and smoothness parameters. As a comparison, the rate of Li and Song (2017) is $n^{-1/4}$ under the best conditions. This improvement is significant because, for example, in a semiparametric estimation problem involving an infinite-dimensional nuisance parameter, the convergence rate of the
estimator of the nuisance parameter is often required to be faster than $n^{-1/4}$ to guarantee desired semiparametric properties such as asymptotic efficiency. This can be achieved by the improved convergence rate, but not by the original rate. The sharpened convergence rate can also be established for GSIR in more general settings, such as functional sufficient dimension reduction.
Read the full abstract ▾
Machine Learning (Statistics)
Quantifying Error Propagation and Model Collapse in Diffusion Models
Nail B. Khelifa, Richard E. Turner and Ramji Venkataramanan
Machine learning models are increasingly trained or fine-tuned on synthetic data. Recursively training on such data has been observed to significantly degrade performance in a wide range of tasks, often characterized by a progressive drift away from the target distribution. In this work, we theoretically analyze this phenomenon in the setting of score-based
diffusion models. For a realistic pipeline where each training round uses a combination of synthetic data and fresh samples from the target distribution, we obtain upper and lower bounds on the accumulated divergence between the generated and target distributions. Notably, to the best of our knowledge, this is the first lower bound on the divergence between the learned and target distributions, even for standard
diffusion models. Our results allow us to characterize different regimes of drift, depending on the score estimation error and the proportion of fresh data used in each generation. In a certain regime, the accumulated divergence after several retraining rounds can be expressed as a discounted sum of score estimation errors made at each generation. We also provide empirical results on synthetic data and images to illustrate the theory.
Read the full abstract ▾
Statistics Theory · Cross-listed from cs.LG
Sequential Membership Inference Attacks
Thomas Michel, Debabrota Basu and Emilie Kaufmann
Modern AI models are not static. They go through multiple updates in their lifecycles. We propose to design Sequential Membership Inference (SeMI) attacks leading to tighter
privacy audits by exploiting the sequence of models and injecting a target canary at a controlled insertion time. First, for empirical mean computation, we develop SeMI*, an {optimal SeMI attack to identify the presence of a target inserted at a specific insertion step}. We derive the power of SeMI* to show that accessing the model sequence yields more powerful MI attacks than scrutinising only the final model. SeMI* exhibits an isolation property – its power depends on the statistics obtained right before and after insertion of the target. Leveraging this insight, we develop practical white-box (accessing model gradients) and black-box (accessing loss) SeMI attacks against models trained with (DP-)SGD. Across
datasets and models trained with (DP-)SGD, our experiments show that SeMI attacks achieve higher powers than snapshot-independent baselines, and yield tighter
privacy audits thanks to (a) control over the insertion time and (b) observations across the model sequence.
Read the full abstract ▾
Statistics Applications
Physical Activity Trajectories Preceding Incident Major Depressive Disorder Diagnosis Using Consumer Wearable Devices in the All of Us Research Program: Case-Control Study
Yuezhou Zhang, Amos Folarin, Hugh Logan Ellis et al. (+7 more)
Low physical activity is a known risk factor for major depressive disorder (MDD), but changes in activity before a first clinical diagnosis remain unclear, especially using long-term objective measurements. This study characterized trajectories of wearable-measured physical activity during the year preceding incident MDD diagnosis. We conducted a retrospective nested case-control study using linked electronic health record and Fitbit data from the All of Us Research Program. Adults with at least 6 months of valid wearable data in the year before diagnosis were eligible. Incident MDD cases were matched to controls on age, sex, body mass index, and index time (up to four controls per case). Daily step counts and moderate-to-vigorous physical activity (MVPA) were aggregated into monthly averages. Linear mixed-effects models compared trajectories from 12 months before diagnosis to diagnosis. Within cases, contrasts identified when activity first significantly deviated from levels 12 months prior. The cohort included 4,104 participants (829 cases and 3,275 controls; 81.7% women; median age 48.4 years). Compared with controls, cases showed consistently lower activity and significant downward trajectories in both step counts and MVPA during the year before diagnosis (P < 0.001). Significant declines appeared about 4 months before diagnosis for step counts and 5 months for MVPA. Exploratory analyses suggested subgroup differences, including steeper declines in men, greater intensity reductions at older ages, and persistently low activity among individuals with obesity. Sustained within-person declines in physical activity emerged months before incident MDD diagnosis. Longitudinal wearable monitoring may provide early signals to support risk stratification and earlier intervention.
Read the full abstract ▾
All authors Yuezhou Zhang, Amos Folarin, Hugh Logan Ellis, Rongrong Zhong, Callum Stewart, Heet Sankesara, Hyunju Kim, Shaoxiong Sun, Abhishek Pratap, Richard JB Dobson
Statistical Computation · Cross-listed from math.NA
Whittle-Matérn Fields with Variable Smoothness
Hamza Ruzayqat, Wenyu Lei, David Bolin et al. (+2 more)
We introduce and analyze a nonlocal generalization of Whittle–Matérn Gaussian fields in which the smoothness parameter varies in space through the fractional order, $s=s(x)\in[\underline s,\overline s]\subset(0,1)$. The model is defined via an integral-form operator whose kernel is constructed from the modified Bessel function of the second kind and whose local singularity is governed by the symmetric exponent $\beta(x,y)=(s(x)+s(y))/2$. This variable-order nonlocal formulation departs from the classical constant-order pseudodifferential setting and raises new analytic and numerical challenges. We develop a novel variational framework adapted to the kernel, prove existence and uniqueness of weak solutions on truncated bounded domains, and derive Sobolev regularity of the Gaussian (spectral) solution controlled by the minimal local order, with realizations lying in $H^r(\mathcal G)$ for every $r<2\underline s-\tfrac{d}{2}$ with $d\ge2$ or $\underline s\le1/2$, with a slightly reduced range in the remaining case (i.e. $d=1$ and $\underline s > 1/2$), hence in $L_2(\mathcal G)$ when $\underline s>d/4$. Here $H^r(\mathcal G)$ denotes the Sobolev space on the bounded domain $\mathcal G$. We also present a finite-element sampling method for the integral model, derive error estimates, and provide numerical experiments in one dimension that illustrate the impact of spatially varying smoothness on the solution covariance. Computational aspects and directions for scalable implementations are discussed.
Read the full abstract ▾
All authors Hamza Ruzayqat, Wenyu Lei, David Bolin, George Turkiyyah, Omar Knio