Peer Revenue Past citations are no guarantee of future returns

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Thursday 19 February

Machine Learning (Statistics) · Cross-listed from cs.LG

Online Learning with Improving Agents: Multiclass, Budgeted Agents and Bandit Learners

We investigate the recently introduced model of learning with improvements, where agents are allowed to make small changes to their feature values to be warranted a more desirable label. We extensively extend previously published results by providing combinatorial dimensions that characterize online learnability in this model, by analyzing the multiclass setup, learnability in a bandit feedback setup, modeling agents' cost for making improvements and more.
Read the full abstract

Statistical Computation · Cross-listed from math.PR

Diffusive Scaling Limits of Forward Event-Chain Monte Carlo: Provably Efficient Exploration with Partial Refreshment

Piecewise deterministic Markov process samplers are attractive alternatives to Metropolis–Hastings algorithms. A central design question is how to incorporate partial velocity refreshment to ensure ergodicity without injecting excessive noise. Forward Event-Chain Monte Carlo (FECMC) is a generalization of the Bouncy Particle Sampler (BPS) that addresses this issue through a stochastic reflection mechanism, thereby reducing reliance on global refreshment moves. Despite promising empirical performance, its theoretical efficiency remains largely unexplored. We develop a high-dimensional scaling analysis for standard Gaussian targets and prove that the negative log-density (or potential) process of FECMC converges to an Ornstein–Uhlenbeck diffusion, under the same scaling as BPS. We derive closed-form expressions for the limiting diffusion coefficients of both methods by analyzing their associated radial momentum processes and solving the corresponding Poisson equations. These expressions yield a sharp efficiency comparison: the diffusion coefficient of FECMC is strictly larger than that of optimally tuned BPS, and the optimum for FECMC is attained at zero global refreshment. Specifically, they imply an approximately eightfold increase in effective sample size per event over optimal BPS. Numerical experiments confirm the predicted diffusion coefficients and show that the resulting efficiency gains remain substantial for a range of non-Gaussian targets. Finally, as an application of these results, we propose an asymptotic variance estimator for Piecewise deterministic Markov processes that becomes increasingly efficient in high dimensions by extracting information from the velocity variable.
Read the full abstract

Statistics Theory · Cross-listed from econ.TH

Dynamic Decision-Making under Model Misspecification: A Stochastic Stability Approach

Dynamic decision-making under model uncertainty is central to many economic environments, yet existing bandit and reinforcement learning algorithms rely on the assumption of correct model specification. This paper studies the behavior and performance of one of the most commonly used Bayesian reinforcement learning algorithms, Thompson Sampling (TS), when the model class is misspecified. We first provide a complete dynamic classification of posterior evolution in a misspecified two-armed Gaussian bandit, identifying distinct regimes: correct model concentration, incorrect model concentration, and persistent belief mixing, characterized by the direction of statistical evidence and the model-action mapping. These regimes yield sharp predictions for limiting beliefs, action frequencies, and asymptotic regret. We then extend the analysis to a general finite model class and develop a unified stochastic stability framework that represents posterior evolution as a Markov process on the belief simplex. This approach characterizes two sufficient conditions to classify the ergodic and transient behaviors and provides inductive dimensional reductions of the posterior dynamics. Our results offer the first qualitative and geometric classification of TS under misspecification, bridging Bayesian learning with evolutionary dynamics, and also build the foundations of robust decision-making in structured bandits.
Read the full abstract

Statistics Applications

Environmental policy in the context of complex systems: Statistical optimization and sensitivity analysis for ABMs

Coupled human-environment systems are increasingly being understood as complex adaptive systems (CAS), in which micro-level interactions between components lead to emergent behavior. Agent-based models (ABMs) hold great promise for environmental policy design by capturing such complex behavior, enabling a sophisticated understanding of potential interventions. One limitation, however, is that ABMs can be computationally costly to simulate, which hinders their use for policy optimization. To address this, we propose a new statistical framework that exploits machine learning techniques to accelerate policy optimization with costly ABMs. We first develop a statistical approach for sensitivity testing of the optimal policy, then leverage a reinforcement learning method for efficient policy optimization. We test this framework on the classic “Sugarscape” model, an ABM for resource harvesting. We show that our approach can quickly identify optimal and interpretable policies that improve upon baseline techniques, with insightful sensitivity and dynamic analyses that connect back to economic theory.
Read the full abstract

Statistical Methodology

General sample size analysis for probabilities of causation: a delta method approach

Probabilities of causation (PoCs), such as the probability of necessity and sufficiency (PNS), are important tools for decision making but are generally not point identifiable. Existing work has derived bounds for these quantities using combinations of experimental and observational data. However, there is very limited research on sample size analysis, namely, how many experimental and observational samples are required to achieve a desired margin of error. In this paper, we propose a general sample size framework based on the delta method. Our approach applies to settings in which the target bounds of PoCs can be expressed as finite minima or maxima of linear combinations of experimental and observational probabilities. Through simulation studies, we demonstrate that the proposed sample size calculations lead to stable estimation of these bounds.
Read the full abstract

All authors Tianyuan Cheng, Ruirui Mao, Judea Pearl, Ang Li

Statistical Methodology

Generative modeling for the bootstrap

Generative modeling builds on and substantially advances the classical idea of simulating synthetic data from observed samples. This paper shows that this principle is not only natural but also theoretically well-founded for bootstrap inference: it yields statistically valid confidence intervals that apply simultaneously to both regular and irregular estimators, including settings in which Efron's bootstrap fails. In this sense, the generative modeling-based bootstrap can be viewed as a modern version of the smoothed bootstrap: it could mitigate the curse of dimensionality and remain effective in challenging regimes where estimators may lack root-$n$ consistency or a Gaussian limit.
Read the full abstract

All authors Leon Tran, Ting Ye, Peng Ding, Fang Han

Statistics Applications

Quantifying the limits of human athletic performance: A Bayesian analysis of elite decathletes

Because the decathlon tests many facets of athleticism, including sprinting, throwing, jumping, and endurance, many consider it to be the ultimate test of athletic ability. On this view, estimating the maximal decathlon score and understanding what it would take to achieve that score provides insight into the upper limits of human athletic potential. To this end, we develop a Bayesian composition model for forecasting how individual decathletes perform in each of the 10 decathlon events of time. Besides capturing potential non-linear temporal trends in performance, our model carefully captures the dependence between performance in an event and all preceding events. Using our model, we can simulate and evaluate the distribution of the maximal possible scores and identify profiles of decathletes who could realistically attain scores approaching this limit.
Read the full abstract

All authors Paul-Hieu V. Nguyen, James M. Smoliga, Benton Lindaman, Sameer K. Deshpande

Statistical Methodology

Reframing Population-Adjusted Indirect Comparisons as a Transportability Problem: An Estimand-Based Perspective and Implications for Health Technology Assessment

Population-adjusted indirect comparisons (PAICs) are widely used to synthesize evidence when randomized controlled trials enroll different patient populations and head-to-head comparisons are unavailable. Although PAICs adjust for observed population differences across trials, adjustment alone does not ensure transportability of estimated effects to decision-relevant populations for health technology assessment (HTA). We examine and formalize transportability in PAICs from an estimand-based perspective. We distinguish conditional and marginal treatment effect estimands and show how transportability depends on effect modification, collapsibility, and alignment between the scale of effect modification and the effect measure. Using illustrative examples, we demonstrate that even when effect modifiers are shared across treatments, marginal effects are generally population-dependent for commonly used non-collapsible measures, including hazard ratios and odds ratios. Conversely, collapsible and conditional effects defined on the linear predictor scale exhibit more favorable transportability properties. We further show that pairwise PAIC approaches typically identify effects defined in the comparator population and that applying these estimates to other populations entails an additional, often implicit, transport step requiring further assumptions. This has direct implications for HTA, where PAIC-derived effects are routinely applied within cost-effectiveness and decision models defined for different target populations. Our results clarify when applying PAIC-derived treatment effects to desired target populations is justified, when doing so requires additional assumptions, and when results should instead be interpreted as population-specific rather than decision-relevant, supporting more transparent and principled use of indirect evidence in HTA and related decision-making contexts.
Read the full abstract

Statistics Applications

Using Time Series Measures to Explore Family Planning Survey Data and Model-based Estimates

Family planning is a global development priority and a key indicator of reproductive health. Monitoring progress is challenged by gaps in survey data across countries. The United Nations Population Division addresses this with the Family Planning Estimation Model (FPEM), a Bayesian hierarchical time series model producing annual estimates of modern contraceptive use while sharing information across countries and regions. This paper evaluates how well FPEM estimates align with survey data using time series diagnostic indices from the wdiexplorer R package, which account for countries nested within sub-regions. Visualisation of survey data, modelled trajectories, and diagnostics enables assessment of model performance, highlighting where trends align and where discrepancies occur.
Read the full abstract

Statistical Methodology

Modeling Multivariate Missingness with Tree Graphs and Conjugate Odds

In this paper, we analyze a specific class of missing not at random (MNAR) assumptions called tree graphs, extending upon the work of pattern graphs. We build off previous work by introducing the idea of a conjugate odds family in which certain parametric models on the selection odds can preserve the data distribution family across all missing data patterns. Under a conjugate odds family and a tree graph assumption, we are able to model the full data distribution elegantly in the sense that for the observed data, we obtain a model that is conjugate from the complete-data, and for the missing entries, we create a simple imputation model. In addition, we investigate the problem of graph selection, sensitivity analysis, and statistical inference. Using both simulations and real data, we illustrate the applicability of our method.
Read the full abstract

Statistics Applications

Temperature and Respiratory Emergency Department Visits: A Mediation Analysis with Ambient Ozone Exposure

High temperatures are associated with adverse respiratory health outcomes and increases in ambient air pollution. Limited research has quantified air pollution's mediating role in the relationship between temperature and respiratory morbidity, such as emergency department (ED) visits. In this study, we conducted a causal mediation analysis to decompose the total effect of daily temperature on respiratory ED visits in Los Angeles from 2005 to 2016. We focused on ambient ozone as a mediator because its precursors and formation are directly driven by sunlight and temperature. We estimated natural direct, indirect, and total effects on the relative risk scale across deciles of temperature exposure compared to the median. We utilized Bayesian additive regression trees (BART) to flexibly characterize the nonlinear relationship between temperature and ozone and quantified uncertainty via posterior prediction and the Bayesian bootstrap. Our results showed that ozone partially mediated the association between high temperatures and respiratory ED visits, particularly at moderately high temperatures. We also validated our modeling approach through simulation studies. This study extends the existing literature by considering acute respiratory morbidity and employing a flexible modeling approach, offering new insights into the mechanisms underlying temperature-related health risks.
Read the full abstract

All authors Chen Li, Thomas W. Hsiao, Stefanie Ebelt, Rebecca H. Zhang, Howard H. Chang

Wednesday 18 February

Statistical Methodology · Cross-listed from cs.LG

LIDS: LLM Summary Inference Under the Layered Lens

Large language models (LLMs) have gained significant attention by many researchers and practitioners in natural language processing (NLP) since the introduction of ChatGPT in 2022. One notable feature of ChatGPT is its ability to generate summaries based on prompts. Yet evaluating the quality of these summaries remains challenging due to the complexity of language. To this end, in this paper we suggest a new method of LLM summary inference with BERT-SVD-based direction metric and SOFARI (LIDS) that assesses the summary accuracy equipped with interpretable key words for layered themes. The LIDS uses a latent SVD-based direction metric to measure the similarity between the summaries and original text, leveraging the BERT embeddings and repeated prompts to quantify the statistical uncertainty. As a result, LIDS gives a natural embedding of each summary for large text reduction. We further exploit SOFARI to uncover important key words associated with each latent theme in the summary with controlled false discovery rate (FDR). Comprehensive empirical studies demonstrate the practical utility and robustness of LIDS through human verification and comparisons to other similarity metrics, including a comparison of different LLMs.
Read the full abstract

Statistical Methodology

M-estimation under Two-Phase Multiwave Sampling with Applications to Prediction-Powered Inference

In two-phase multiwave sampling, inexpensive measurements are collected on a large sample and expensive, more informative measurements are adaptively obtained on subsets of units across multiple waves. Adaptively collecting the expensive measurements can increase efficiency but complicates statistical inference. We give valid estimators and confidence intervals for M-estimation under adaptive two-phase multiwave sampling. We focus on the case where proxies for the expensive variables – such as predictions from pretrained machine learning models – are available for all units and propose a Multiwave Predict-Then-Debias estimator that combines proxy information with the expensive, higher-quality measurements to improve efficiency while removing bias. We establish asymptotic linearity and normality and propose asymptotically valid confidence intervals. We also develop an approximately greedy sampling strategy that improves efficiency relative to uniform sampling. Data-based simulation studies support the theoretical results and demonstrate efficiency gains.
Read the full abstract

Machine Learning (Statistics)

Poisson-MNL Bandit: Nearly Optimal Dynamic Joint Assortment and Pricing with Decision-Dependent Customer Arrivals

We study dynamic joint assortment and pricing where a seller updates decisions at regular accounting/operating intervals to maximize the cumulative per-period revenue over a horizon $T$. In many settings, assortment and prices affect not only what an arriving customer buys but also how many customers arrive within the period, whereas classical multinomial logit (MNL) models assume arrivals as fixed, potentially leading to suboptimal decisions. We propose a Poisson-MNL model that couples a contextual MNL choice model with a Poisson arrival model whose rate depends on the offered assortment and prices. Building on this model, we develop an efficient algorithm PMNL based on the idea of upper confidence bound (UCB). We establish its (near) optimality by proving a non-asymptotic regret bound of order $\sqrt{T\log{T}}$ and a matching lower bound (up to $\log T$). Simulation studies underscore the importance of accounting for the dependency of arrival rates on assortment and pricing: PMNL effectively learns customer choice and arrival models and provides joint assortment-pricing decisions that outperform others that assume fixed arrival rates.
Read the full abstract

All authors Junhui Cai, Ran Chen, Qitao Huang, Linda Zhao, Wu Zhu

Statistical Methodology

A statistical perspective on transformers for small longitudinal cohort data

Modeling of longitudinal cohort data typically involves complex temporal dependencies between multiple variables. There, the transformer architecture, which has been highly successful in language and vision applications, allows us to account for the fact that the most recently observed time points in an individual's history may not always be the most important for the immediate future. This is achieved by assigning attention weights to observations of an individual based on a transformation of their values. One reason why these ideas have not yet been fully leveraged for longitudinal cohort data is that typically, large datasets are required. Therefore, we present a simplified transformer architecture that retains the core attention mechanism while reducing the number of parameters to be estimated, to be more suitable for small datasets with few time points. Guided by a statistical perspective on transformers, we use an autoregressive model as a starting point and incorporate attention as a kernel-based operation with temporal decay, where aggregation of multiple transformer heads, i.e. different candidate weighting schemes, is expressed as accumulating evidence on different types of underlying characteristics of individuals. This also enables a permutation-based statistical testing procedure for identifying contextual patterns. In a simulation study, the approach is shown to recover contextual dependencies even with a small number of individuals and time points. In an application to data from a resilience study, we identify temporal patterns in the dynamics of stress and mental health. This indicates that properly adapted transformers can not only achieve competitive predictive performance, but also uncover complex context dependencies in small data settings.
Read the full abstract

All authors Kiana Farhadyar, Maren Hackenberg, Kira Ahrens, Charlotte Schenk, Bianca Kollmann, Oliver Tüscher, Klaus Lieb, Michael M. Plichta, Andreas Reif, Raffael Kalisch, Martin Wolkewitz, Moritz Hess, Harald Binder

Machine Learning (Statistics) · Cross-listed from cs.LG

ML-driven detection and reduction of ballast information in multi-modal datasets

Modern datasets often contain ballast as redundant or low-utility information that increases dimensionality, storage requirements, and computational cost without contributing meaningful analytical value. This study introduces a generalized, multimodal framework for ballast detection and reduction across structured, semi-structured, unstructured, and sparse data types. Using diverse datasets, entropy, mutual information, Lasso, SHAP, PCA, topic modelling, and embedding analysis are applied to identify and eliminate ballast features. A novel Ballast Score is proposed to integrate these signals into a unified, cross-modal pruning strategy. Experimental results demonstrate that significant portions of the feature space as often exceeding 70% in sparse or semi-structured data, can be pruned with minimal or even improved classification performance, along with substantial reductions in training time and memory footprint. The framework reveals distinct ballast typologies (e.g. statistical, semantic, infrastructural), and offers practical guidance for leaner, more efficient machine learning pipelines.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking

We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual neurons learn single-frequency Fourier features and phase alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual neurons produce noisy signals, the phase symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify the correct sum. Furthermore, we explain the emergence of these features under random initialization via a lottery ticket mechanism. Our gradient flow analysis proves that frequencies compete within each neuron, with the "winner" determined by its initial spectral magnitude and phase alignment. From a technical standpoint, we provide a rigorous characterization of the layer-wise phase coupling dynamics and formalize the competitive landscape using the ODE comparison lemma. Finally, we use these insights to demystify grokking, characterizing it as a three-stage process involving memorization followed by two generalization phases, driven by the competition between loss minimization and weight decay.
Read the full abstract

All authors Jianliang He, Leda Wang, Siyu Chen, Zhuoran Yang

Statistics Applications

The Impact of Formations on Football Matches Using Double Machine Learning. Is it worth parking the bus?

This study addresses a central tactical dilemma for football coaches: whether to employ a defensive strategy, colloquially known as "parking the bus", or a more offensive one. Using an advanced Double Machine Learning (DML) framework, this project provides a robust and interpretable tool to estimate the causal impact of different formations on key match outcomes such as goal difference, possession, corners, and disciplinary actions. Leveraging a dataset of over 22,000 matches from top European leagues, formations were categorized into six representative types based on tactical structure and expert consultation. A major methodological contribution lies in the adaptation of DML to handle categorical treatments, specifically formation combinations, through a novel matrix-based residualization process, allowing for a detailed estimation of formation-versus-formation effects that can inform a coach's tactical decision-making. Results show that while offensive formations like 4-3-3 and 4-2-3-1 offer modest statistical advantages in possession and corners, their impact on goals is limited. Furthermore, no evidence supports the idea that defensive formations, commonly associated with parking the bus, increase a team's winning potential. Additionally, red cards appear unaffected by formation choice, suggesting other behavioral factors dominate. Although this approach does not fully capture all aspects of playing style or team strength, it provides a valuable framework for coaches to analyze tactical efficiency and sets a precedent for future research in sports analytics.
Read the full abstract

Machine Learning (Statistics)

Beyond Procedure: Substantive Fairness in Conformal Prediction

Conformal prediction (CP) offers distribution-free uncertainty quantification for machine learning models, yet its interplay with fairness in downstream decision-making remains underexplored. Moving beyond CP as a standalone operation (procedural fairness), we analyze the holistic decision-making pipeline to evaluate substantive fairness-the equity of downstream outcomes. Theoretically, we derive an upper bound that decomposes prediction-set size disparity into interpretable components, clarifying how label-clustered CP helps control method-driven contributions to unfairness. To facilitate scalable empirical analysis, we introduce an LLM-in-the-loop evaluator that approximates human assessment of substantive fairness across diverse modalities. Our experiments show that label-clustered CP often provides a favorable balance between utility and substantive fairness, while reducing set-size disparities in line with our theory. Finally, we empirically show that equalized set sizes, rather than coverage, strongly correlate with improved substantive fairness, enabling practitioners to design more fair CP systems. Our code is available at https://github.com/layer6ai-labs/llm-in-the-loop-conformal-fairness.
Read the full abstract

All authors Pengqi Liu, Zijun Yu, Mouloud Belbahri, Arthur Charpentier, Masoud Asgharian, Jesse C. Cresswell

Statistical Methodology

First versus full or first versus last: U-statistic change-point tests under fixed and local alternatives

The use of U-statistics in the change-point context has received considerable attention in the literature. We compare two approaches of constructing CUSUM-type change-point tests, which we call the first-vs-full and first-vs-last approach. Both have been pursued by different authors. The question naturally arises if the two tests substantially differ and, if so, which of them is better in which data situation. In large samples, both tests are similar: they are asymptotically equivalent under the null hypothesis and under sequences of local alternatives. In small samples, there may be quite noticeable differences, which is in line with a different asymptotic behavior under fixed alternatives. We derive a simple criterion for deciding which test is more powerful. We examine the examples Gini's mean difference, the sample variance, and Kendall's tau in detail. Particularly, when testing for changes in scale by Gini's mean difference, we show that the first-vs-full approach has a higher power if and only if the scale changes from a smaller to a larger value – regardless of the population distribution or the location of the change. The asymptotic derivations are under weak dependence. The results are illustrated by numerical simulations and data examples.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

Omitted Variable Bias in Language Models Under Distribution Shift

Despite their impressive performance on a wide variety of tasks, modern language models remain susceptible to distribution shifts, exhibiting brittle behavior when evaluated on data that differs in distribution from their training data. In this paper, we describe how distribution shifts in language models can be separated into observable and unobservable components, and we discuss how established approaches for dealing with distribution shift address only the former. Importantly, we identify that the resulting omitted variable bias from unobserved variables can compromise both evaluation and optimization in language models. To address this challenge, we introduce a framework that maps the strength of the omitted variables to bounds on the worst-case generalization performance of language models under distribution shift. In empirical experiments, we show that using these bounds directly in language model evaluation and optimization provides more principled measures of out-of-distribution performance, improves true out-of-distribution performance relative to standard distribution shift adjustment methods, and further enables inference about the strength of the omitted variables when target distribution labels are available.
Read the full abstract

Statistics Theory · Cross-listed from cs.LG

Knowledge-Embedded Latent Projection for Robust Representation Learning

Latent space models are widely used for analyzing high-dimensional discrete data matrices, such as patient-feature matrices in electronic health records (EHRs), by capturing complex dependence structures through low-dimensional embeddings. However, estimation becomes challenging in the imbalanced regime, where one matrix dimension is much larger than the other. In EHR applications, cohort sizes are often limited by disease prevalence or data availability, whereas the feature space remains extremely large due to the breadth of medical coding system. Motivated by the increasing availability of external semantic embeddings, such as pre-trained embeddings of clinical concepts in EHRs, we propose a knowledge-embedded latent projection model that leverages semantic side information to regularize representation learning. Specifically, we model column embeddings as smooth functions of semantic embeddings via a mapping in a reproducing kernel Hilbert space. We develop a computationally efficient two-step estimation procedure that combines semantically guided subspace construction via kernel principal component analysis with scalable projected gradient descent. We establish estimation error bounds that characterize the trade-off between statistical error and approximation error induced by the kernel projection. Furthermore, we provide local convergence guarantees for our non-convex optimization procedure. Extensive simulation studies and a real-world EHR application demonstrate the effectiveness of the proposed method.
Read the full abstract

All authors Weijing Tang, Ming Yuan, Zongqi Xia, Tianxi Cai

Statistical Methodology

Synthetic-Powered Multiple Testing with FDR Control

Multiple hypothesis testing with false discovery rate (FDR) control is a fundamental problem in statistical inference, with broad applications in genomics, drug screening, and outlier detection. In many such settings, researchers may have access not only to real experimental observations but also to auxiliary or synthetic data – from past, related experiments or generated by generative models – that can provide additional evidence about the hypotheses of interest. We introduce SynthBH, a synthetic-powered multiple testing procedure that safely leverages such synthetic data. We prove that SynthBH guarantees finite-sample, distribution-free FDR control under a mild PRDS-type positive dependence condition, without requiring the pooled-data p-values to be valid under the null. The proposed method adapts to the (unknown) quality of the synthetic data: it enhances the sample efficiency and may boost the power when synthetic data are of high quality, while controlling the FDR at a user-specified level regardless of their quality. We demonstrate the empirical performance of SynthBH on tabular outlier detection benchmarks and on genomic analyses of drug-cancer sensitivity associations, and further study its properties through controlled experiments on simulated data.
Read the full abstract

All authors Yonghoon Lee, Meshi Bashari, Edgar Dobriban, Yaniv Romano

Machine Learning (Statistics)

Enhanced Diffusion Sampling: Efficient Rare Event Sampling and Free Energy Calculation with Diffusion Models

The rare-event sampling problem has long been the central limiting factor in molecular dynamics (MD), especially in biomolecular simulation. Recently, diffusion models such as BioEmu have emerged as powerful equilibrium samplers that generate independent samples from complex molecular distributions, eliminating the cost of sampling rare transition events. However, a sampling problem remains when computing observables that rely on states which are rare in equilibrium, for example folding free energies. Here, we introduce enhanced diffusion sampling, enabling efficient exploration of rare-event regions while preserving unbiased thermodynamic estimators. The key idea is to perform quantitatively accurate steering protocols to generate biased ensembles and subsequently recover equilibrium statistics via exact reweighting. We instantiate our framework in three algorithms: UmbrellaDiff (umbrella sampling with diffusion models), MetaDiff (a batchwise analogue for metadynamics), and $\Delta$G-Diff (free-energy differences via tilted ensembles). Across toy systems, protein folding landscapes and folding free energies, our methods achieve fast, accurate, and scalable estimation of equilibrium properties within GPU-minutes to hours per system-closing the rare-event sampling gap that remained after the advent of diffusion-model equilibrium samplers.
Read the full abstract

All authors Yu Xie, Ludwig Winkler, Lixin Sun, Sarah Lewis, Adam E. Foster, José Jiménez Luna, Tim Hempel, Michael Gastegger, Yaoyi Chen, Iryna Zaporozhets, Cecilia Clementi, Christopher M. Bishop, Frank Noé

Statistics Applications

Design and Analysis Strategies for Pooling in High Throughput Screening: Application to the Search for a New Anti-Microbial

A major public health issue is the growing resistance of bacteria to antibiotics. An important part of the needed response is the discovery and development of new antimicrobial strategies. These require the screening of potential new drugs, typically accomplished using high-throughput screening (HTS). Traditionally, HTS is performed by examining one compound per well, but a more efficient strategy pools multiple compounds per well. In this work, we study several recently proposed pooling construction methods, as well as a variety of pooled high-throughput screening analysis methods, in order to provide guidance to practitioners on which methods to use. This is done in the context of an application of the methods to the search for new drugs to combat bacterial infection. We discuss both an extensive pilot study as well as a small screening campaign, and highlight both the successes and challenges of the pooling approach.
Read the full abstract

All authors Byran Smucker, Benjamin Brennan, Emily Rego, Meng Wu, Zhihong Lin, Brian Ahmer, Blake Peterson

Statistics Theory

On Sharpened Convergence Rate of Generalized Sliced Inverse Regression for Nonlinear Sufficient Dimension Reduction

Generalized Sliced Inverse Regression (GSIR) is one of the most important methods for nonlinear sufficient dimension reduction. As shown in Li and Song (2017), it enjoys a convergence rate that is independent of the dimension of the predictor, thus avoiding the curse of dimensionality. In this paper we establish an improved convergence rate of GSIR under additional mild eigenvalue decay rate and smoothness conditions. Our convergence rate can be made arbitrarily close to $n^{-1/3}$ under appropriate decay rate and smoothness parameters. As a comparison, the rate of Li and Song (2017) is $n^{-1/4}$ under the best conditions. This improvement is significant because, for example, in a semiparametric estimation problem involving an infinite-dimensional nuisance parameter, the convergence rate of the estimator of the nuisance parameter is often required to be faster than $n^{-1/4}$ to guarantee desired semiparametric properties such as asymptotic efficiency. This can be achieved by the improved convergence rate, but not by the original rate. The sharpened convergence rate can also be established for GSIR in more general settings, such as functional sufficient dimension reduction.
Read the full abstract

Machine Learning (Statistics)

Quantifying Error Propagation and Model Collapse in Diffusion Models

Machine learning models are increasingly trained or fine-tuned on synthetic data. Recursively training on such data has been observed to significantly degrade performance in a wide range of tasks, often characterized by a progressive drift away from the target distribution. In this work, we theoretically analyze this phenomenon in the setting of score-based diffusion models. For a realistic pipeline where each training round uses a combination of synthetic data and fresh samples from the target distribution, we obtain upper and lower bounds on the accumulated divergence between the generated and target distributions. Notably, to the best of our knowledge, this is the first lower bound on the divergence between the learned and target distributions, even for standard diffusion models. Our results allow us to characterize different regimes of drift, depending on the score estimation error and the proportion of fresh data used in each generation. In a certain regime, the accumulated divergence after several retraining rounds can be expressed as a discounted sum of score estimation errors made at each generation. We also provide empirical results on synthetic data and images to illustrate the theory.
Read the full abstract

Statistics Theory · Cross-listed from cs.LG

Sequential Membership Inference Attacks

Modern AI models are not static. They go through multiple updates in their lifecycles. We propose to design Sequential Membership Inference (SeMI) attacks leading to tighter privacy audits by exploiting the sequence of models and injecting a target canary at a controlled insertion time. First, for empirical mean computation, we develop SeMI*, an {optimal SeMI attack to identify the presence of a target inserted at a specific insertion step}. We derive the power of SeMI* to show that accessing the model sequence yields more powerful MI attacks than scrutinising only the final model. SeMI* exhibits an isolation property – its power depends on the statistics obtained right before and after insertion of the target. Leveraging this insight, we develop practical white-box (accessing model gradients) and black-box (accessing loss) SeMI attacks against models trained with (DP-)SGD. Across datasets and models trained with (DP-)SGD, our experiments show that SeMI attacks achieve higher powers than snapshot-independent baselines, and yield tighter privacy audits thanks to (a) control over the insertion time and (b) observations across the model sequence.
Read the full abstract

Statistics Applications

Physical Activity Trajectories Preceding Incident Major Depressive Disorder Diagnosis Using Consumer Wearable Devices in the All of Us Research Program: Case-Control Study

Low physical activity is a known risk factor for major depressive disorder (MDD), but changes in activity before a first clinical diagnosis remain unclear, especially using long-term objective measurements. This study characterized trajectories of wearable-measured physical activity during the year preceding incident MDD diagnosis. We conducted a retrospective nested case-control study using linked electronic health record and Fitbit data from the All of Us Research Program. Adults with at least 6 months of valid wearable data in the year before diagnosis were eligible. Incident MDD cases were matched to controls on age, sex, body mass index, and index time (up to four controls per case). Daily step counts and moderate-to-vigorous physical activity (MVPA) were aggregated into monthly averages. Linear mixed-effects models compared trajectories from 12 months before diagnosis to diagnosis. Within cases, contrasts identified when activity first significantly deviated from levels 12 months prior. The cohort included 4,104 participants (829 cases and 3,275 controls; 81.7% women; median age 48.4 years). Compared with controls, cases showed consistently lower activity and significant downward trajectories in both step counts and MVPA during the year before diagnosis (P < 0.001). Significant declines appeared about 4 months before diagnosis for step counts and 5 months for MVPA. Exploratory analyses suggested subgroup differences, including steeper declines in men, greater intensity reductions at older ages, and persistently low activity among individuals with obesity. Sustained within-person declines in physical activity emerged months before incident MDD diagnosis. Longitudinal wearable monitoring may provide early signals to support risk stratification and earlier intervention.
Read the full abstract

All authors Yuezhou Zhang, Amos Folarin, Hugh Logan Ellis, Rongrong Zhong, Callum Stewart, Heet Sankesara, Hyunju Kim, Shaoxiong Sun, Abhishek Pratap, Richard JB Dobson

Statistical Computation · Cross-listed from math.NA

Whittle-Matérn Fields with Variable Smoothness

We introduce and analyze a nonlocal generalization of Whittle–Matérn Gaussian fields in which the smoothness parameter varies in space through the fractional order, $s=s(x)\in[\underline s,\overline s]\subset(0,1)$. The model is defined via an integral-form operator whose kernel is constructed from the modified Bessel function of the second kind and whose local singularity is governed by the symmetric exponent $\beta(x,y)=(s(x)+s(y))/2$. This variable-order nonlocal formulation departs from the classical constant-order pseudodifferential setting and raises new analytic and numerical challenges. We develop a novel variational framework adapted to the kernel, prove existence and uniqueness of weak solutions on truncated bounded domains, and derive Sobolev regularity of the Gaussian (spectral) solution controlled by the minimal local order, with realizations lying in $H^r(\mathcal G)$ for every $r<2\underline s-\tfrac{d}{2}$ with $d\ge2$ or $\underline s\le1/2$, with a slightly reduced range in the remaining case (i.e. $d=1$ and $\underline s > 1/2$), hence in $L_2(\mathcal G)$ when $\underline s>d/4$. Here $H^r(\mathcal G)$ denotes the Sobolev space on the bounded domain $\mathcal G$. We also present a finite-element sampling method for the integral model, derive error estimates, and provide numerical experiments in one dimension that illustrate the impact of spatially varying smoothness on the solution covariance. Computational aspects and directions for scalable implementations are discussed.
Read the full abstract

All authors Hamza Ruzayqat, Wenyu Lei, David Bolin, George Turkiyyah, Omar Knio