Peer Revenue Equity in science

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Thursday 23 July

Statistical Methodology

Globally aligned Principal Component Analysis for multi-group data

We propose a novel principal component analysis (PCA) for multi-group datasets, where the same numerical variables are measured across different groups of observations. Existing approaches either ignore group structure entirely by working with global (pooled) data, focus exclusively on local structure (group-wise PCA), or impose restrictive assumptions of common principal components. Our approach respects the multi-group nature of data while improving global comparability of components. We combine group-specific principal components with global ones through an explicit alignment mechanism based on regularized optimization. We introduce the notion of globally aligned covariance matrix, incorporating weighted contributions from global principal directions in the group-wise covariance matrix. The alignment strength is controlled by regularization parameters that can be tuned to achieve the desired trade-off. Through a comprehensive simulation study, we demonstrate that the proposed aligned PCA achieves a favorable compromise between capturing local variation within groups and maintaining interpretability and stability across groups. Furthermore, in an application to the 2021 Canadian Census socioeconomic data, the proposed aligned PCA yields more comparable and stable region-specific components than pooled or region-wise PCA.
Read the full abstract

Machine Learning (Statistics)

Priors learned from legacy reconstructions inherit undetectable overconfidence

Where truths are scarce (e.g., seismic and medical imaging), learned priors in ill-posed inverse problems are trained on archives of legacy reconstructions—i.e., an older method's outputs—and their reported uncertainty is taken as data-driven. We show that this prior is, in the population limit, exactly the regularizer that produced its archive of posterior samples, advanced one expectation–maximization step toward the truth. While the step improves the regularizer on the directions the measurements resolve, it leaves the regularizer's assumption on the operator's blind subspace unchanged. An archive of single-best reconstructions collapses the blind interval to zero width. Neither error is detectable in practice, as truths differing only on the blind subspace share the data law, and simulation-based calibration is neutral by construction. We identify from the operator alone which directions the measurements do not inform, and, given a handful of ground-truth models, build intervals there that contain the truth as often as they claim to. We validate these findings on a two-dimensional example with closed-form predictions and in controlled experiments on seismic-imaging and groundwater-flow operators, against priors trained on the truth.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

A Defense of the Quadratic Model

Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically tractable the model is, and how accurately it describes the true optimization dynamics. In this work, we stress test the simplest possible model of optimization – the quadratic model – and show that it can be surprisingly predictive in an LLM setting with 150M parameters and 3B training tokens. Specifically, we show that Taylor expanding the model and the loss function at intermediate checkpoints through training can accurately predict the optimization dynamics over windows that can last up to 10% of training. Having established this agreement, we then turn to analyzing the structure of these local quadratic optimization problems through two lenses: the Hessian spectrum and local stability. Using Lanczos quadrature with extremely deep probes, we are able to estimate the Hessian spectrum deep into the tail, and we find a surprising amount of structure in both the eigenvalues and eigenvectors, which depends on the batch size, preconditioner, and training time. We also empirically test local linear stability at intermediate checkpoints and compare it to theoretical predictions to demonstrate that optimization in LLMs typically occurs at a stochastic edge of stability, whose nature is also determined by batch size. Our results indicate the quadratic model may be a theoretically tractable proxy for pretraining optimization dynamics.
Read the full abstract

All authors Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade, Alex Damian

Machine Learning (Statistics) · Cross-listed from cs.LG

An Introduction to Bayesian and Frequentist Simulation-Based Inference with Machine Learning

Simulation-based inference (SBI) with machine learning is an increasingly important tool for solving inverse problems in science and engineering, including parameter inference and the inversion of detector effects. We provide an overview of the Bayesian and frequentist statistical frameworks, describe how machine-learning-based SBI methods, such as neural posterior estimation and neural likelihood estimation, can be used for parameter estimation within these frameworks, and show that the same methods can also be applied to Empirical Bayes or unfolding tasks. We also discuss how to validate inference results and the limitations of SBI with machine learning.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibration

Test-time adaptive out-of-distribution (OOD) detectors update a memory bank from the unlabelled stream. We show this adaptation obeys a provable dynamical law. Modelling bank impurity as a generalized Pólya urn, we prove almost-sure convergence to a mean-field equilibrium whose slope acts as a reproduction number. Below one, impurity stays benign. Above one, the bank is fully poisoned and the detector collapses. The measured admission kernel is affine ($R^2 \ge 0.996$) with slope just below one in every encoder family (a protocol signature), so this detector class is near-critical by design, and across 96 settings the predicted threshold matches the empirical collapse, where ungated dictionaries lose up to $0.163$ AUROC. We then prove that a certified admission gate, reading only a frozen reserve, severs the feedback loop and removes the transition at every contamination rate, even adversarially, while controlling false positives label-free. For the complementary static-calibration failure under drift we give CDC, which restores nominal FPR label-free on all tested drift-affected cells. Finally we prove a two-world impossibility theorem. Drift and contamination are indistinguishable without labels, forcing a closed-form power ceiling our procedure approaches. Together these give a complete possibility/impossibility characterization of label-free adaptive OOD detection.
Read the full abstract

Statistical Methodology

Equilibrium Causal Digital Twins: Validation, Transport, and Identification Limits

Digital twins are often used to predict how a system would respond to an intervention. In systems with feedback, a twin must reproduce an equilibrium counterfactual, and a twin developed in one domain may fail after mechanisms change. We study when these predictions can be validated and transported. For equilibrium causal games, we give conditions on the mechanisms, equilibrium selection, and intervention design under which agreement with experimental distributions identifies the counterfactual of interest. We show why agreement of means and covariances is insufficient for distributional queries. We then introduce cyclic selection diagrams and derive criteria for direct reuse and for hybrid models that combine invariant source mechanisms with target information. An impossibility result constructs systems that agree under every experiment in a finite design but disagree on the target counterfactual, showing that validation requires structural assumptions. For linear models, we derive intervention requirements that depend on the mechanisms that changed, the observation model, and graph support. When point identification fails, we characterize the remaining range of query values. We also provide statistical tests for reconstructed means and covariances and illustrate the theory in synthetic feedback systems.
Read the full abstract

Statistics Theory · Cross-listed from physics.soc-ph

Reconstructing the Probability Measure of a Multi-group Curie-Weiss Model with Interacting Groups

We study the problem of reconstructing the probability measure of a multi-group version of the Curie-Weiss or mean-field model of ferromagnetism from a sample of the voting behaviour the population. While originally used to study phase transitions in statistical mechanics, the Curie-Weiss or mean-field model has been applied to study phenomena where many agents interact with each other, in particular in case of a heterogeneous population with identifiable subpopulations. The degree of social cohesion within social groups manifests in the way the members of the group influence each others' decisions as well as how they behave under outside influence from voters belonging to another group. Contrary to single-group Curie-Weiss models, here we have a larger number of coupling parameters which have to be estimated. While the maximum likelihood estimator of the coupling parameters has desirable statistical properties in theory, computational challenges make applications to larger populations impractical. Therefore, we analyse an estimator based on asymptotic approximations to the behaviour of the Curie-Weiss model valid for large populations. Due to the wide applicability of models such as Curie-Weiss, the estimator is potentially useful in disciplines such as political science, sociology, automated voting, and preference aggregation.
Read the full abstract

All authors Miguel Ballesteros, Gerardo Franco Cordova, Fedro Guillen, Gabor Toth

Statistics Applications · Cross-listed from cs.AI

Unsupervised Consensus-Based Anomaly Detection for Spatiotemporal Malaria Incidence in Ghana

A consensus anomaly detection framework was applied to monthly malaria surveillance data from Ghana (2014-2023) to identify atypical transmission patterns. Anomalies were highly structured in space and time. Ashanti and Northern Regions accounted for most recurrent anomalies, with persistent hotspots at Tamale, Kumasi, and Accra. A key finding was the spatial distinction between anomaly burden (cumulative cases during anomalous periods) and anomaly frequency (persistence of unusual behaviour). Tamale had the highest burden during anomalies, whereas the highest anomaly rates clustered in Ashanti districts, showing that high-burden areas are not necessarily those with the most frequent anomalous transmission. Anomalous months formed a statistically distinct group, with much higher case counts (Cohen's $d = 3.252$) and large seasonal deviations ($d > 1.2$) compared with normal months. Malaria burden alone provides an incomplete picture of transmission dynamics. By distinguishing where malaria is most prevalent from where transmission behaves most unusually, this framework can strengthen surveillance, prioritise investigations, and support targeted control strategies.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Zero-Flow Two-Sample Tests

Motivated by the success of modern flow-based generative models in modeling complex data, we study two-sample testing through the lens of flow-based methods. We propose the Zero-Flow Two-Sample Test (ZF2ST), built on the zero-flow criterion, which characterizes distributional equality through a time-reversal antisymmetry of a learnable velocity field. We extend this criterion and further develop the Zero-Flow Discrepancy, an identifying discrepancy that controls the Wasserstein distance, and derive a variational representation in terms of a witness function. This representation naturally leads to a witness-based test whose power is governed by the signal-to-noise ratio (SNR), allowing direct power maximization for witness learning. ZF2ST learns the witness on one data split and performs testing on held-out samples, thereby maintaining Type-I error control and admitting a simple asymptotic null distribution. Experimentally, ZF2ST performs competitively across various synthetic and real-world benchmarks, while showing particularly strong performance in distinguishing image distributions from different sources.
Read the full abstract

All authors Yakun Wang, Leyang Wang, Song Liu, Taiji Suzuki

Statistics Theory

Optimal use of a black-box learner in semiparametric estimation

Consider the partial linear model $Y = \mu_0(X) + \beta_0 \cdot T + \varepsilon$ and $T = \pi_0(X) + u$ in the structure-agnostic setting, where we are blind to the structure $\mu_0$ and $\pi_0$ and estimate the nuisances by a black-box hypothesis class. The learnability of the class is characterized by the estimation error $\delta_s$ in the absence of model misspecification and its $L_2$ mis-specification error $\delta_{a, \mu}$ and $\delta_{a, \pi}$ for $\mu_0$ and $\pi_0$, respectively. We propose a novel estimator of the target linear coefficient $\theta_0 = \beta_0$ with error rate \[ \frac{1}{\sqrt{n}} + \delta_{a, \mu} \cdot \delta_{a, \pi} + [\delta_s]^2. \] A matching lower bound is also established, implying that this rate is unimprovable. Compared with the product rate yielded by double machine learning (DML), our estimator removes the suboptimal term $\max(\delta_{a, \mu}, \delta_{a, \pi})\cdot \delta_s$ at no extra cost or assumption. Building on the underlying insights, which are neither tailored to the one-learner setting nor the partial linear model, we propose Transductive Adversarial Moment-calibrated Editing (TAME), which locally edits debiasing weights induced by black-box regression estimates on the inference sample through adversarial conditional moment calibration. TAME can be combined with any initial black-box estimates and can strictly improve on DML guarantees when the nuisance difficulties are imbalanced. We discuss how to fully exploit the advantages introduced by TAME, including the gains from using two learners, the resulting under-smoothing principle for model selection, and extensions to other linear functional estimation problems.
Read the full abstract

Statistics Theory · Cross-listed from quant-ph

Unified theory of classical and quantum semiparametric efficiency

In classical and quantum statistics, high-dimensional unknown parameters are abundant and it is often prudent to make minimal assumptions about them using so-called semiparametric models. To attack a wide range of semiparametric problems in one broad stroke, we present a unified treatment of statistical efficiency for classical and quantum semiparametric models, generalizing the Cramér-Rao and Helstrom bounds beyond finite-dimensional parameters. We introduce the fundamental concepts in abstract and geometric terms before applying them to many examples, covering general classical and quantum models as well as the paradigmatic special cases of Gaussian and Poisson fields. We give an in-depth treatment of channels in the semiparametric efficiency theory and advocate the use of the singular value decomposition to elucidate the statistical effects of channels. To demonstrate the utility of the formalism, we apply it to coherent and incoherent optical imaging problems, assuming an arbitrary field or intensity on the object plane without parametric assumptions. Our formalism enables us to compute classical and quantum limits to coherent and incoherent imaging resolution in statistical terms. For subdiffraction incoherent imaging, we demonstrate that spatial-mode demultiplexing can be far superior to direct imaging in estimating generalized Fourier coefficients and come closer to the quantum limits. We envision our theory becoming an essential tool for both classical and quantum statistics with useful applications to sensing and imaging, whenever minimal assumptions about a high-dimensional parameter should be made.
Read the full abstract

Statistical Methodology

Identification and Robust Inference for Multiple Treatments with Possibly Invalid Instruments

The instrumental variable (IV) method is widely used to infer causal effects in observational studies with unmeasured confounding, but invalid instruments can compromise both population identification and finite-sample inference. This paper studies linear IV models with multiple endogenous treatments and possibly invalid instruments. Identification of multiple effects is more delicate than in the single-treatment setting because a single instrument no longer identifies a single candidate effect; instead, each relevant instrument defines a hyperplane in the multidimensional effect space. For identification of multiple treatment effects, we introduce generalized plurality and majority rules which require a sufficiently large number of IVs to be valid. For inference, data-dependent instrument selection may fail to separate certain invalid IVs from valid ones, leading to undercoverage of confidence intervals when these invalid instruments are mistakenly selected as valid. We propose a sampling confidence interval for each treatment effect, which is robust to IV selection errors. We establish asymptotic coverage and parametric-rate length of our sampling confidence interval under regularity conditions and illustrate this method in Monte Carlo simulations and a Mendelian randomization application.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Finite-Sample Coverage Audits for High-Recall Candidate Generation: Certification and Learning-Theoretic Design

An initial high-recall stage in an empirical pipeline decides which items pass to later review, labelling, or modelling, and relevant items it misses are lost to every subsequent stage. We study how many audit labels are needed to certify, with finite-sample validity, that this missed relevant mass is small, and our main results characterise the label complexity of this problem. We first show that no procedure using only labels from inside the candidate set can certify any non-trivial bound on the missed mass: the audit must sample the excluded pool, the only region where unrecovered relevant items can lie. We then prove a matching finite-corpus lower bound. Any valid audit that certifies fewer than $m$ missed relevant items with high probability when none are present, even if adaptive and permitted to label the entire included pool, must inspect on the order of $N_0/m$ excluded-pool labels. Excluded-pool auditing is therefore minimax rate-optimal, not merely convenient, for missed-mass certification in the zero-miss regime. Building on this characterisation, we develop an exact finite-sample toolkit, using binomial and hypergeometric inversion rather than asymptotic approximation, that certifies missed mass, converts it to recall through a two-pool design, certifies pre-specified families of nested candidate generators simultaneously, and produces stress-test certificates against declared perturbation mechanisms. These certificates can be paired with observable review burden to select the least burdensome pre-specified candidate generator meeting a missed-mass target. Every guarantee holds under one discipline: the candidate generator, or the pre-specified family from which it is selected, and the audit rule are fixed before the certification labels are examined.
Read the full abstract

Statistical Methodology · Cross-listed from cs.AI

Detecting LLM-Generated Tokens in Human–LLM Coauthored Text

The rise of human-AI collaborative writing has created a growing need for fine-grained detection methods that support localizing likely LLM-generated content in mixed-authorship documents. Existing methods for detecting LLM-generated text mainly focus on document-level classification and cannot identify which parts of the text are generated by LLMs. This paper introduces a new method to address this urgent need. Our method operates at the token level, the natural unit of modern language models, and builds on existing token-level detection scores. The key idea is to smooth adjacent token scores to reduce their variability, while using an adaptive Lepski-type rule to select the bandwidth according to the local authorship structure. Our method is simple to implement and does not require token-level labeled data for training. Theoretically, we characterize this trade-off and show that the proposed method achieves favorable mean square error performance in estimating the underlying signal. Empirically, we demonstrate strong performance of our method against a wide range of baselines in both synthetic datasets and a realistic dataset. We deploy a publicly accessible website that implements the methods as well.
Read the full abstract

All authors Yangjun Lu, Hongyi Zhou, Fabian Spill, Kai Ye, Chengchun Shi, Jin Zhu

Statistics Applications · Cross-listed from astro-ph.IM

Modeling Dependence Structures in Astronomical Multi-Band Time Series Data via Multi-Output Gaussian Processes

Modern astronomical time-domain surveys routinely collect multi-band light curves that provide complementary information about the physical processes governing source variability. Gaussian processes (GPs) provide a flexible probabilistic framework for modeling irregularly sampled and noisy time-series data. While considerable attention has been devoted to developing covariance kernels for individual time series, comparatively less attention has been paid to the statistical representation of dependence among multiple photometric bands. In this work, we present a unified statistical framework for modeling such dependence structures using multi-output GPs. Within this framework, we consider two complementary formulations. The covariance-based formulation specifies dependence directly through matrix-valued covariance functions and emphasizes the stochastic properties of the observed light curves, including covariance functions and power spectral densities. In contrast, the latent-process formulation represents the observed light curves as transformations of latent GPs and emphasizes the physical mechanisms generating the observed dependence. To illustrate these formulations, we develop covariance-based and latent-process multi-output damped random walk models and derive their corresponding spectral representations. We further demonstrate the practical implications of dependence-structure modeling through applications to multi-band active galactic nucleus variability and continuum reverberation mapping. Rather than advocating a universally preferred formulation, this work provides a principled basis for selecting dependence structures according to the scientific objectives and clarifies how this choice influences the statistical characterization and scientific interpretation of stochastic variability in astronomical sources.
Read the full abstract

All authors Samata Das, Lishan Shi, Yasaman Hamayouni, Hyungsuk Tak, Jong-Hak Woo

Machine Learning (Statistics) · Cross-listed from cs.LG

Semantic-Aware Task Clustering for Constructive and Cooperative Multi-Tasking

Cooperative multi-task semantic communication (CMT-SemCom) improves task execution performance by leveraging shared representations. However, as we demonstrated in [1], cooperative multi-tasking can be either constructive or destructive, depending on the semantic relationships among tasks. To ensure constructive cooperation, we propose a semantic-aware task clustering method for CMT-SemCom. We have formulated a sequential multi-stage optimization problem in which semantically aligned tasks are clustered once after a short initial training phase, and then end-to-end (E2E) joint training is conducted exclusively within the discovered groups. Specifically, the problem decomposes into two stages: (i) a semantic clustering problem leveraging hierarchical density-based spatial clustering, and (ii) an intra-cluster E2E CMT-SemCom learning problem. Simulation results demonstrate that the proposed framework effectively mitigates destructive cooperation and negative transfer, yielding accuracy gains compared to unclustered multi-tasking and individual training baselines.
Read the full abstract

All authors Ahmad Halimi Razlighi, Maximilian H. V. Tillmann, Edgar Beck, Bho Matthiesen, Armin Dekorsy

Machine Learning (Statistics) · Cross-listed from quant-ph

Cautious optimism for deep parameterized quantum circuits

A central challenge in quantum machine learning is understanding the scaling behavior of parameterized quantum circuits (PQCs). In particular, it remains unclear how their performance on unseen data changes as the number of trainable parameters increases. Prior works have derived formal generalization guarantees for quantum models, but it is well-known that many such results do not fully characterize generalization behavior in practice. In this work, we show that gradient-based PQCs can exhibit improved performance on unseen data as model size increases, displaying the phenomenon of double descent. This contrasts with the traditional view that larger models lead to degraded generalization. We provide analytical results rigorously underpinning this behavior by leveraging add-one-in perturbation techniques and spectral properties of random matrices. We support these results with numerical experiments on re-uploading PQCs across several data sets and training set sizes, consistently observing the predicted double descent behavior. While other obstacles on the path toward practical quantum machine learning remain, our finding that deeper parameterized quantum circuits do not necessarily exhibit degraded performance provides reasons for cautious optimism.
Read the full abstract

All authors Marie Kempkes, Elies Gil-Fuster, Carlos Bravo-Prieto, Aroosa Ijaz, Alissa Wilms, Jens Eisert, Evert van Nieuwenburg, Vedran Dunjko

Statistical Methodology · Cross-listed from cs.LG

A Diffusion-Model Subpopulation Digital Twin for Mobile Health Deployment: A Case Study on the HeartSteps Intervention

Mobile-health interventions increasingly use online learning and decision making algorithms to personalize when to nudge users toward healthier behavior, but a poorly designed algorithm can burden and disengage participants. New algorithm design decisions should therefore be vetted against realistic simulated users before each real-life deployment. We propose a method to develop “JITAI-Twins”: digital twins of a target subpopulation for comparing candidate online algorithms before a just-in-time adaptive intervention (JITAI) deployment. The method builds on a conditional time-series diffusion model that is temporally consistent (future actions do not affect the generated past), and it supports repeated updating from three sources of information, in three steps: pre-training on a large observational dataset, fine-tuning on small prior intervention deployments in related populations, and inference-time calibration to the next target population from domain-scientist expertise. We validate the twin at each pre-deployment stage of the long-running HeartSteps series (v2 through v4) of physical-activity suggestion intervention deployments, treating each successive deployment as an upcoming study. The proposed method reproduces the target subpopulation's temporal and between-participant structure better than simpler simulators. These results suggest that our twin can be used to simulate a target deployment before it runs, the prerequisite for testing and informing online algorithm design decisions.
Read the full abstract

All authors Ziping Xu, Yuyi Chang, Chenshun Ni, Nithin Sugavanam, Asim H. Gazi, Pedja Klasnja, Emre Ertin, Susan A. Murphy

Statistics Theory · Cross-listed from math.NA

3D Uncertainty Quantification for Photoacoustic Tomography

Photoacoustic tomography (PAT) is a promising modality for high-resolution biomedical imaging, motivating the need for reliable uncertainty quantification (UQ) of reconstructed images. Bayesian approaches provide a rigorous framework for UQ but remain computationally challenging for realistic three-dimensional PAT and are sensitive to numerical approximations in the governing wave equation. We develop a finite-element Bayesian UQ framework for PAT that accommodates complex computational domains and detector geometries while enabling large-scale three-dimensional inference. The proposed methodology reformulates the randomize-then-optimize (RTO) sampling strategy as a matrix-free algorithm that generates independent posterior samples using only forward and adjoint wave propagations. Particular attention is given to constructing an adjoint discretization that forms an exact transpose pair with the discrete forward operator while remaining consistent with the continuous PAT adjoint, enabling efficient least-squares solvers within the sampling procedure. We investigate the influence of temporal discretization, artificial boundary conditions, and adjoint consistency on posterior uncertainty and identify discretization strategies that avoid numerical artifacts. The framework is validated against exact posterior statistics, existing Bayesian PAT methods, and Hamiltonian Monte Carlo using the No-U-Turn Sampler (NUTS), and is demonstrated on a three-dimensional problem with approximately $2\times 10^5$ unknowns on a general finite-element domain. To the best of our knowledge, this is the first large-scale Bayesian PAT study on general three-dimensional finite-element geometries, and the methodology extends naturally to a broad class of linear PDE-constrained inverse problems.
Read the full abstract

All authors Babak Maboudi Afkham, Amal Mohammed A Alghami, Hassan Yazdanian, Tanja Tarvainen

Machine Learning (Statistics) · Cross-listed from cs.LG

Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks

Deep neural networks encode complex representations, but deconstructing this internal knowledge remains a challenge. Given the link between learning and compression, network compression offers a promising lens to analyze this knowledge. However, standard compression heuristics often suffer from scale symmetries and architectural biases. To resolve these, we introduce Hilbert Operator for Progressive Encoding (HOPE), a mathematical framework to gradually deconstruct the representations in trained network weights. HOPE shifts network compression from the discrete domain into a Hilbert space of continuous functions. By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection. Extending this formulation, HOPE introduces macro block eviction to encompass multi-layer structures like entire residual pathways under the same unified metric. This unified approach enables unbiased architectural decisions across layers with different types and sizes. HOPE is a data-free and hyperparameter-free framework. We present proof-of-concept experiments in model compression and fine-tuning to highlight the practical potential of our theory.
Read the full abstract

Statistical Methodology · Cross-listed from econ.EM

Uniformly Consistent Semi-nonparametric Demand Estimation with Micro-Data

This paper develops a profiled sieve minimum-distance estimator for a semi-nonparametric differentiated-products demand model with micro-level choice data. Building on Berry and Haile (2024), the estimator uses within-market variation in consumer covariates to recover a flexible consumer-heterogeneity function and market-specific composite intercepts. Excluded price instruments then separate these intercepts into a flexible price-side function and structural demand shocks. The main statistical challenge is that the number of profiled market intercepts grows with the number of markets. I show that, when both the number of markets and the minimum within-market sample size grow, this profiling step is asymptotically negligible and the common structural functions are uniformly consistently estimated. Monte Carlo evidence supports the consistency result and illustrates the value of flexible price-side estimation for counterfactual demand analysis.
Read the full abstract

Statistical Computation

The acceptance-complement method revisited

We revisit the acceptance-complement method in random variate generation and show how it can replace the rejection method in many examples. While the rejection method has geometrically distributed execution times, the acceptance-complement method has a constant (deterministic) run time and qualifies as a “one-liner”. We show how this method can be used to efficiently generate random variates from several distributions, such as the gamma and beta. In addition, we show that there is an acceptance-complement method that is valid for all log-concave densities with known location of the mode and black-box type access to the density.
Read the full abstract

Statistics Applications · Cross-listed from q-bio.MN

From Berg-Purcell precision bounds to clock-limited information capacity

Physical limits to chemical sensing are traditionally expressed as Berg-Purcell bounds on estimation accuracy. Whether these bounds also limit the total amount of information a molecular receptor can transmit has remained unclear, despite the fact that cellular signaling performance is naturally quantified in bits rather than precision alone. Here we derive an explicit link between Berg-Purcell-type sensing limits and the information capacity of a single two-state receptor, yielding a compact expression that separates contributions from concentration range, receptor copy number, and averaging time. We show that, in the ideal fixed-time occupancy model, diffusion-limited sampling alone does not define a finite global information bound: with a perfectly specified integration window and unbounded input range, information capacity grows without bound with dynamic range, albeit slowly. A finite saturation arises when time integration is treated as an explicit physical resource. Finite timing precision yields a clock-limited bound on information capacity, and in the high-occupancy regime information transmission crosses over from diffusion-limited to clock-limited behavior. Together, these results establish a receptor-level information bound in bits that is finite once constraints on timing precision are taken into account.
Read the full abstract

Machine Learning (Statistics)

Transformer-based Diffusion models for Hydrological Time Series Probabilistic Imputation and Forecasting

The modeling of hydrometeorological time series with limited observations is a key challenge in the monitoring of hydro-systems and water resources, as well as for flood or drought risk assessment. Due to the high variability of the underlying processes and the sparsity of available measurements, traditional statistical approaches often struggle to accurately represent their dynamics. In this context, recent advances in deep learning offer a promising direction for improving the representation and generation of complex temporal processes sampled at several observation sites. This study investigates the application of transformer-based diffusion models to the simulation and reconstruction of hydrological time series. The proposed framework is applied to the joint modeling of water quantity and quality at six sites spread across three adjacent headwater catchments located in North-East France on a limestone plateau covered by forests and field crops. The model is calibrated and validated using available observational data, which has been quality controlled and corrected for sensor drift and malfunction through collaborative efforts by LNE metrology expertise and Andra monthly quality control over more than 15 years. Its performance is compared with several established baseline approaches commonly used for time series modeling. Quantitative evaluation metrics are employed to assess the ability of the proposed method to reproduce key temporal characteristics of the observed signals in two settings: the imputation of incomplete time series and the forecasting of upcoming hydrological conditions. Results support the effectiveness of the transformer-based approach and highlight its capacity to capture and simulate the complex patterns present in hydrological data. In particular, the results indicate that diffusion models can efficiently sample realistic time series distributions under observation settings with variable missing data for both forecasting and imputation.
Read the full abstract

All authors Ferdinand Bhavsar, Lionel Benoit, Maxime Savatier, Edith Gabriel

Statistics Applications

Enhancing Gender Equality Assessment through Object-Oriented Bayesian networks: the European Gender Equality Index Case

A novel data-driven framework is introduced to assess gender equality by complementing and empowering a widely used European gender composite indicator, the Gender Equality Index (GEI). The GEI synthetizes the latent construct of gender equality into a single score and is extensively employed for cross-country comparison and monitoring. While effective for communication and benchmarking, this practice is affected by conceptual and methodological limitations, including marginal analysis that leaves interactions and conditional (in)dependencies unmeasured, and a lack of predictive capability. To address these limitations, this paper proposes the use of Object-Oriented Bayesian Networks (OOBNs) to model the GEI. By preserving the hierarchical structure of the index, OOBNs extend Bayesian Networks and enable a multivariate and probabilistic representation of interdependencies among the components of gender equality. This approach advances intersectional gender statistics by shifting the focus from computing a single composite score to modelling the underlying mechanisms that shape gender inequalities. The proposed methodology enhances the assessment and monitoring of gender equality and adds a predictive dimension through scenario-based evaluation, thereby supporting Gender Impact Assessment and policy decision-making. An application to Italian official statistics illustrates the practical relevance of the framework and its applicability to other national contexts and policy needs.
Read the full abstract

All authors Lorenzo Giammei, Fulvia Mecatti, Flaminia Musella, Paola Vicard

Machine Learning (Statistics) · Cross-listed from cs.LG

Smooth Neural Point Processes via B-Splines

Temporal point processes (TPPs) provide a general and flexible framework for modeling sequences of events in continuous time. Neural networks have been successfully employed to model TPPs in a highly expressive and data-driven way. Neural TPPs are typically trained via Maximum Likelihood Estimation (MLE) by minimizing the negative log-likelihood (NLL), which depends on both the conditional intensity function (CIF) and its integral over time, the compensator. Recent neural TPP approaches enable exact evaluation of the NLL without numerical integration. However, these methods typically model the compensator rather than the CIF directly, impose constraints on the neural network architecture, and are computationally expensive during training, as event contributions to the NLL are evaluated sequentially rather than in parallel. In this work, we propose a novel neural TPP model that directly parametrizes the CIF as a non-negative combination of B-spline basis functions, whose coefficients are predicted by a neural network. This formulation enables exact evaluation of the NLL, preserves full flexibility in the neural architecture, allows efficient parallelization during training, and naturally supports CIF smoothness regularization through the integrated squared second derivative. Experiments on both synthetic and real-world datasets show improved computational efficiency and predictive accuracy compared to the reference neural TPP baseline.
Read the full abstract

All authors Michele Bellomo, Riccardo Ramaschi, Alberto Dolara, Tomaso Aste

Machine Learning (Statistics)

Automatic knot selection in smooth additive models

B-spline regression constitutes a widely used framework for nonparametric modeling. The performance of this methodology depends on specifying the number and placement of changepoints, known as knots, prior to the estimation process. Such knot sequence determines the dimension of the B-spline basis used to represent the regression function and the number of coefficients to be estimated. Therefore, the knots' choice affects the model's flexibility, influencing its smoothness and goodness-of-fit. Traditionally, this problem has been addressed either by explicitly selecting knots, via knot-selection algorithms, or by regularization methods, such as P-splines, which automatically tune the regressor's smoothness. The latter have become the standard in generalized additive models (GAMs). In contrast, knot-selection techniques, frequently neglected because of computational or modeling limitations, provide certain advantages which can be valuable in some contexts. In this work, we introduce a novel explicit knot-selection technique for GAMs based on an extension of the adaptive splines (A-splines) knot selection methodology, combined with a customized Fellner-Schall scheme for tuning the associated parameters. Our approach is evaluated on various synthetic and real datasets and compared with P-splines and state-of-the-art knot-selection techniques. The results indicate comparable performance, while producing models built on a substantially smaller number of basis elements.
Read the full abstract

Statistics Applications

Risk-Limiting Audits for Parliamentary Majorities

Existing methods for risk-limiting audits typically focus on certifying individual contests. In parliamentary elections, however, the politically relevant outcome is often whether a party has won enough seats to form government, not whether every reported seat outcome is correct. Extending on the work of Mohanty et al. (2019), we formulate the certification of a parliamentary majority as a partial conjunction testing problem: it is enough to verify that the reported winning party truly won at least a majority of its reported seats. Building on the SHANGRLA auditing framework, we construct a sequential audit statistic for the majority outcome by combining seat-level statistics. We then propose adaptive sampling strategies that allocate auditing effort across seats, including variants that learn to avoid spending excessive effort on seats that appear unlikely to have been truly won. Using simulations based on synthetic and real data, from the 2014 Indian Lok Sabha election, we show that auditing the parliamentary majority can substantially reduce the number of ballots inspected (by almost a thousand-fold) compared to certifying every reported winning seat.
Read the full abstract

Statistical Methodology

Information criteria exploiting latent structure for model selection in Structural Equation Models

Structural equation models (SEM) are widely used to describe dependency structures between latent variables, making model selection a key issue in many applications. Existing information criteria are generally based on the integrated observed-data likelihood and therefore do not explicitly account for the latent structure of the model. In this paper, we propose two new information criteria derived from the integrated complete-data likelihood. The first adapts the Integrated Completed Likelihood criterion to Gaussian SEM, while the second proposes an alternative approach to approximating the integrated observed-data log-likelihood by incorporating latent structural information and using an importance sampling strategy. Their performance is assessed through an extensive simulation study covering null, direct, indirect and complete latent structures under different sample sizes and signal strengths. The results show that the proposed importance sampling strategy provides robust and competitive model selection across a wide range of scenarios, whereas the proposed ICL criterion is particularly effective for recovering latent dependency structures when the latent variables are accurately estimated. These findings demonstrate the potential benefits of explicitly exploiting the latent structure when developing information criteria for structural equation models.
Read the full abstract

All authors Marion Naveau, Magalie Houée-Bigot, Matthieu Marbac, Anouk Zancarini, Mathieu Emily

Statistics Applications

Trust by Context, Not by Design? A Quantitative Study of Data Donation Willingness for Open-Source Civic AI in Switzerland

Civic AI systems increasingly support democratic participation, yet interactions with them may reveal sensitive political views, creating tension between improving AI models and residents' expectations of privacy and consent. This study examines the conditions of transparency and user control under which Swiss residents are willing to donate their anonymized chatbot conversations to train an open-source AI model. A 2x2 between-subjects factorial design evaluated how a Data Nutrition Label and a granular consent dashboard influence donation decisions. The experiment was delivered via a multilingual online survey featuring a custom chatbot powered by the Apertus-70B model. Analysis of the 205 participants revealed that neither transparency nor control significantly affected donation behavior. Rates were uniformly high (91.7% overall), producing a ceiling effect, and Bayesian checks confirmed the absence of treatment effects. The dashboard raised perceived control but not donation, and high-control participants actively restricted their data-use settings. A qualitative analysis of 120 open-ended responses indicates that residents framed donation as a contribution to the public good, motivated by democratic participation, an open-source model, and research, while many regarded their anonymized queries as non-personal and therefore low in risk. Interpreted through the privacy calculus, a high perceived benefit coincided with a low perceived risk under high institutional trust, so both sides of the trade-off aligned and interface design had little leverage. Offering control served less to raise donation than to let residents define the terms of their contribution.
Read the full abstract