Peer Revenue Impact factor: volatile

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Sunday 9 August

Statistics Applications

Modeling Disease Transmission Dynamics with Population-Weighted Spatial Inference for Wastewater-Based Epidemiology

Wastewater-based epidemiology (WBE) is an effective, noninvasive tool for tracking community-level circulation of respiratory viruses, yet standard renewal models treat each wastewater treatment plant (WWTP) in isolation, even though respiratory syncytial virus (RSV) spreads through contact networks that cross service-area boundaries. Using weekly data from $m = 15$ Houston WWTPs, we extend a single-plant Bayesian renewal model by coupling neighboring WWTPs via a spatial weight matrix built using a novel Population Extended Hausdorff Distance, defined as a directional distance between WWTPs. This distance measures how far a WWTP must reach to access a meaningful share of a neighboring WWTP's residents. A single parameter $\rho$ controls spatial mixing in the plant-level growth rate, allowing us to recover the effective reproduction number $R_{it}$. We find that $\rho$ is estimated well above zero, indicating substantial cross-WWTP transmission that the independent-plant model cannot capture. Spatial coupling reshapes plant-level $R_{it}$ relative to the independent-plant baseline ($\rho = 0$) while leaving inferred infection trajectories essentially unchanged.
Read the full abstract

All authors Jose R. Palacio, Katherine B. Ensor, Julia Schedler, Rebecca Schneider, Kaavya Domakonda, Loren Hopkins, Lauren B. Stadler, Sharmila Bhandari

Statistics Theory · Cross-listed from physics.bio-ph

Physical limits to concentration and gradient sensing by perfect monitors

Cells often estimate chemical concentrations from only a few diffusing molecules, whose stochastic motion limits sensing precision. We consider a perfect monitoring instrument that records the instantaneous molecular density throughout a finite region without perturbing the concentration field or distinguishing molecular identities. The standard Berg-Purcell estimator weights all positions uniformly. Here we derive the optimal spatial weighting within a large class of unbiased estimators for both concentration and gradient sensing. Variance minimization maps to an electrostatic problem. Surprisingly, although the instrument monitors the entire volume, the optimal estimator assigns all weight to its boundary, reducing the uncertainty in both concentration and gradient sensing. We extend the results to arbitrary geometries and spatial dimensions.
Read the full abstract

Statistics Applications

Risk-Aware Pump Control in Water Supply Systems Using Probabilistic Water Demand and Electricity Price Forecasts

Drinking water utilities face uncertain water demand and electricity prices, as well as requirements for reliable and cost-efficient operation. This paper investigates a sequential multi-stage pump scheduling problem under uncertainty for a drinking water supplier participating in the day-ahead auction with subsequent imbalance market settlement. We integrate probabilistic forecasts of water demand and electricity prices into a risk-aware stochastic optimization framework. Reliability and economic risk preferences are represented through a lexicographic objective incorporating Conditional Value-at-Risk and Exceedance Risk measures, where the framework allows to explicitly account for the imbalance market settlement. A numerical study demonstrates that improvements in forecasting performance generally translate into improved policy performance. Furthermore, we found that the primary benefit of stochastic optimization lies in improved operational robustness rather than lower expected costs. The water demand is found to be a dominant driver of policy quality, the marginal value of further improvements in electricity price forecasts appears comparatively limited, indicating that the economically exploitable component of the price information is already captured to a large extent. Compared with a price-invariant benchmark, the best stochastic policy achieves cost savings of up to $9\%$, indicating that substantial economic benefits can be realized by accepting a carefully controlled increase in imbalance exposure and operational risk.
Read the full abstract

Statistical Methodology

A Least-Squares Approach to Sample-Based Prior Elicitation

An expert who supplies examples of a quantity often also signals how plausible each one is; when is that signal worth using? We study eliciting a Bayesian prior from an expert who provides example points together with their approximate likelihoods. We propose fitting the prior by least squares–minimizing the squared discrepancy between a parametric density and the elicited likelihoods–which defines an M-estimator that remains well posed even for families whose moments do not exist. We establish consistency and asymptotic normality, and prove–under explicit regularity conditions, comprising a well-separation and a uniform-concentration requirement that we verify for the families considered–a non-asymptotic O(1/sqrt(n)) Berry–Esseen bound on its sampling distribution, uniform and nonuniform, by extending a result of Pinelis for maximum-likelihood estimators to the M-estimation setting. We then relax the assumptions that most limit the method in practice. Experts need not report on the density's own scale: an unknown reporting scale can be profiled out in closed form and estimated jointly, at no asymptotic cost for location families. The theory extends to multivariate parameters, where a directional Berry–Esseen bound follows from the multivariate delta method applied to a smooth implicit proxy for the estimator. An additive error floor in the noise model removes a degeneracy in the optimal design, making optimal designs interior. Simulations for normal and beta families confirm the predicted n^-1 error rate and the accuracy of the normal approximation at moderate sample sizes. Finally, we compare the estimator with the sample-only maximum-likelihood baseline, derive an explicit threshold on the expert's reporting noise below which the elicited likelihoods provably reduce estimation error, and calibrate that threshold against eleven datasets of human frequency judgments.
Read the full abstract

Machine Learning (Statistics)

A Mean-Field Framework for Inference-Time Distributional Control of Diffusion Models

Diffusion models are increasingly used as controllable samplers, whose generations can be steered at inference time according to a chosen reward function. While such rewards are typically defined on individual samples, for many applications it is desirable to steer according to distribution-level rewards, for example to calibrate with population-level information or to encourage diversity. In both cases, simply incorporating the reward gradient into the dynamics, while often effective, comes with few theoretical guarantees on the sampled distribution. For pointwise rewards, recent work has therefore sought to develop a principled framework for targeting a prescribed tilted distribution using particle reweighting. However, an analogous theoretically-grounded approach for distributional rewards is currently lacking. In this work, we formulate inference-time distributional control as targeting a tilted measure under a mean-field framework, and derive a weighted interacting particle scheme to target it in a principled manner. Our framework recovers pointwise-reward steering as a special case, while providing a theoretical foundation for existing batch-level steering methods. Empirically, we verify that the procedure correctly targets the prescribed distribution in tractable low-dimensional settings, and investigate its behaviour in higher-dimensional protein conformation tasks.
Read the full abstract

Machine Learning (Statistics)

A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning

Reinforcement learning (RL) seeks to optimize sequential decisions to maximize population-level benefits over time. However, when deployed in high-stakes settings such as healthcare, RL decisions might systematically restrict some subpopulation's access to valuable services in a manner contrary to the values and goals of stakeholders. Counterfactual fairness (CF) offers a promising framework to address this problem based on causal reasoning. This paper develops a data preprocessing algorithm that, when used in tandem with policy learning, enables CF in RL. Our algorithm relies on a novel quantile distribution mapping method for sequentially estimating the counterfactual states and rewards in the data preprocessing step, subsuming common additivity assumptions used for counterfactual prediction as a special case. We theoretically prove that the per-step level of counterfactual unfairness and infinite-horizon suboptimality gap can be bounded under mild regularity conditions. We also empirically test our algorithm in numerical experiments as well as in application to a real-world interventional digital health dataset.
Read the full abstract

All authors Jianhan Zhang, Jitao Wang, John D. Piette, Donglin Zeng, Chengchun Shi, Zhenke Wu

Machine Learning (Statistics)

Multi-kernel spectral clustering: Entrywise eigenvector perturbation bounds and exact recovery

Kernel spectral clustering with a single bandwidth can be inadequate for data exhibiting multiple characteristic pairwise-distance scales, a problem particularly prevalent in the high-dimensional regime. We address this issue through a multi-kernel formulation that aggregates kernels with different bandwidths. The bandwidths are selected as prescribed empirical quantiles of the pairwise squared distances, thereby capturing the relevant distance scales without requiring prior population-scale information. We develop a rigorous theoretical analysis of the resulting method under a general high-dimensional, multi-scale mixture model with heterogeneous cluster centers and covariance geometries. We construct a blockwise constant, low-rank informative approximation to the empirical multi-kernel matrix and establish row-wise $\ell_{2,\infty}$ perturbation bounds for its leading spectral components, as well as for the associated normalized Laplacian matrix. These bounds yield observation-level control of the spectral embedding, which is more informative than conventional global eigenspace perturbation estimates. Under suitable eigen-gap and cluster-separation conditions, we show that approximate $K$-means applied to the multi-kernel spectral embedding achieves exact recovery with high probability.
Read the full abstract

All authors Zeqin Lin, Guangming Pan, Zhixiang Zhang, Yinbing Zhou

Statistics Theory

Conditionally Resampled Sliding-Window Count Kernels: Spectral-Gap Bounds and Poincaré Inequalities

We study the conditionally resampled sliding-window count kernel associated with the empirical counts of length-$n$ windows from a stationary finite-state reversible Markov chain. Although the resulting count process is generally not Markov, its stationary one-step conditional law defines a genuine Markov kernel. For every fixed strictly positive reversible kernel \(P\) on a finite state space, we present a Poincaré inequality for the induced count kernel $\tP_n$ of length $n$. In other words, we derive the lower bound of the spectral gap $\Gap(\tP_n)$ of $\tP_n$ as \[ \Gap(\tP_n)\ge \frac{c(P)}{n}, \] where \(c(P)>0\) depends only on \(P\). The proof combines a martingale oscillation inequality for the stationary path law with a direct comparison of coordinate oscillations to the Dirichlet form of the count kernel. A linear statistic of the count vector gives the matching \(O(1/n)\) upper bound, so for every fixed strictly positive reversible \(P\) one has \(\Gap(\tP_n)=\Theta_P(1/n)\). The resulting count-space Poincaré inequality yields a local-to-global variance bound for finite-window count statistics and, together with a general matrix-concentration principle, operator-norm concentration for matrix-valued empirical averages.
Read the full abstract

Statistics Theory · Cross-listed from math.NA

Mass Lumping and Numerical Quadrature for Approximation of Fractional Elliptic Differential Equations Driven by Gaussian White Noise

Fractional elliptic stochastic partial differential equations (SPDEs) are widely used in statistics and machine learning for computationally efficient and flexible modeling of Gaussian random fields. The computational efficiency of the SPDE approach relies on finite element approximations combined with numerical quadrature and mass lumping, which enable sparse matrix methods during inference. Although many works have studied finite element approximations of fractional SPDEs, the effect of the mass lumping and quadrature approximations used in practice has not been fully analyzed. To fill this gap, we derive convergence rates for numerical approximations of fractional SPDEs based on finite element discretizations combined with numerical quadrature and mass lumping. Specifically, we obtain explicit convergence rates for the mean-squared error of the covariance function in a general framework that covers the main settings where mass lumping is used in the SPDE approach. We also analyze non-stationary variance-control factors of the form $L^\beta(\tau u)=\mathcal{W}$, where $\tau$ is spatially varying, and derive covariance error estimates showing how the regularity of $\tau$ affects the convergence rate. As specific examples, we provide results for random fields on bounded Euclidean domains, Riemannian manifolds, and metric graphs. Numerical experiments are presented that confirm the theoretical results.
Read the full abstract

Statistics Theory

Partial Differential Equation Barriers to Identifiability in Infinite Mixture Models

We study identifiability of mixing measures in infinite mixture models. We show that, in many common cases, lack of identifiability can be characterized in terms of certain differential structures of the kernel family with respect to its parameters. In our main results, we prove that when the kernel is annihilated by a non-trivial differential or difference-differential operator over the parameter space, there exist infinitely many distinct mixing measures yielding the same mixture density. We give verifiable conditions for such operators to exist, covering many common cases, including the location-scale Gaussian, location-scale Student-t, Gamma, Beta, Dirichlet, negative binomial and non-central Chi-squared families. Furthermore, our conditions apply to any exponential family whose parameter dimension exceeds the dimension of its sufficient statistic and, more generally, to kernels with polynomially-growing score functions. We complement our results with a minimax lower bound on the estimation error for the mixing measure in the Wasserstein distance under non-identifiability. On the flips side, we describe three classes of kernels for which identifiability is preserved and nonparametric statistical inference remains possible.
Read the full abstract

All authors Dung Le, Nicola Bariletto, Alessandro Rinaldo, Nhat Ho

Machine Learning (Statistics)

LazyHMC: Hamiltonian Monte Carlo Simulation for Lazy, Infinite Dimensional Probabilistic Programs

Hamiltonian Monte Carlo (HMC) is a successful generic inference method in probabilistic programming, but in its ordinary formulation it needs gradients and finite-dimensional parameter spaces. In Haskell, lazy evaluation lets probabilistic programs express stochastic processes and other non-parametric Bayesian models over implicit infinite-dimensional spaces. This paper develops new formulations of gradient-based HMC for this infinite-dimensional setting, via lazy evaluation. For automatic differentiation, we provide an analysis based on a new notion of "piecewise analytic under cylindrical analytic partition" (PACAP), to show that even if a program is infinite-dimensional and defined lazily, the gradient of the likelihood function is finitely supported. For the Monte Carlo method itself, we develop several HMC variants and a No-U-Turn Sampler that operate over the infinite-dimensional parameter space but are still productive because of lazy evaluation. Experiments cover Gaussian mixture clustering, random walks, and piecewise-constant regression with Poisson-process changepoints.
Read the full abstract

All authors Maria-Nicoleta Crăciun, C.-H. Luke Ong, Tom Schrijvers, Sam Staton

Statistics Theory

Ultra-high-dimensional spot support screening in continuous-time regression

We study variable screening for high-frequency continuous-time regression, targeting the support of the spot regression coefficient. We establish uniform entrywise bounds for local estimates of the marginal correlation vector and covariate correlation matrix under coordinatewise truncation and finite-variation jumps. Under a local averaged $\alpha$-Hölder condition, the rate is $\{(\log p)/n\}^{\alpha/(2\alpha+1)}$, up to logarithmic factors. This yields sure screening for the marginal support. We also derive a list-size minimax lower bound that matches the localization and multiplicity order when the retained list size $d$ satisfies $\log(p/d)\asymp\log p$. For regression-support recovery under local dependence, we propose partial-residual spot iterative screening with pathwise Lepski adaptation. Under sparse eigenvalues and standardized beta-min, the procedure has the sure screening property and requires only correlation columns associated with the current active set. Simulation studies and a high-frequency factor application illustrate its performance.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

When Can Fraud Operations Authorize Automation? A Decision-Support Framework for Fresh Audit Evidence and Review Workload

Fraud operations must allocate events among automatic approval, analyst review, and automatic blocking even though the labels needed to evaluate these actions are selective and delayed. Predictive scores order cases, but they do not show whether the evidence is current and representative enough to delegate an action to the model. We develop freshness-constrained audit capacity (FCAC), a decision-support framework that treats automation as an authorization decision constrained by action risk, evidence freshness, and shared review capacity. It evaluates candidate action regions from mature randomized audits and a prespecified temporal allowance. Supported regions are automated; unsupported regions remain in review. The resulting decision record reports evidence age, audit demand, total review workload, value exposure, and compatible temporal change. We show that current action risk is unidentified without restricting unobserved label evolution. Under representative randomized audits, label-independent evidence windows, and a prespecified condition linking historical and current action risk, we derive simultaneous finite-sample control of unsafe authorization. Chronological evaluations with simulated audits on IEEE-CIS, ULB-Worldline, and Elliptic++ yield zero-drift automation rates of 84.4%, 67.4%, and 81.3%, with total review workloads of 24.1%, 46.0%, and 43.1%. The experiments reveal an audit-capacity trade-off: sparse auditing delays authorization, whereas intensive auditing eventually increases workload. A separately specified BAF stress test further indicates that fallback thresholds must reflect candidate-specific evidence rather than a common fraction of the risk limit. These findings identify audit freshness and analyst capacity as joint design considerations for fraud decision support.
Read the full abstract

Statistical Methodology

Exact Conditional Distributions of Chi-Square-Family Statistics for Two-Way Contingency Tables, by Cell-Separable Dynamic Programming

Every chi-square-family statistic for a two-way contingency table (Pearson's X^2, the power-divergence members, the variance-stabilized T_root) is referred to an approximate null distribution (chi-square, moment-matched chi-square, saddlepoint, or bootstrap) that miscalibrates on sparse or heterogeneous tables, where the exact distribution is a lattice of atoms rather than a continuous curve. The exact reference conditions on both margins (the multivariate Fisher noncentral hypergeometric law) but is usually treated as uncomputable, because the number of tables sharing a margin is astronomical. We show that the exact conditional moments and the exact conditional distribution of any chi-square-family statistic are computable without enumerating tables, because the statistic is additive over cells and the margin-conditional law factorizes cell by cell: a dynamic program walks the table one cell at a time, carrying per row-capacity state either a few moment accumulators, a map from statistic value to probability, or one complex number (the characteristic function), at a cost set by the number of margin states rather than the number of tables. The moment engine computes the exact conditional moments of a 5x5 table at three per cell (about 79 billion tables on the margin fibre) in three seconds; the distribution engine returns the exact tail, verified against enumeration to machine precision, where a three-moment chi-square fit misses it by up to 0.28. The engine furnishes Monte-Carlo-free ground truth against which any approximate reference can be measured, extends to the non-null alternative, and is the computational substrate beneath the exact conditional interval and reporting tools of the companion papers.
Read the full abstract

2608.08576 · 9 Aug · 25 pages, 5 figures, 4 tables. Reproducible code, data, and the derivation companion (DH1-DH8) are openly archived on Zenodo: https://doi.org/10.5281/zenodo.21831680 . Part of a contingency-table methodology series (companions on the honest effect size, the exact conditional interval for Cramer's V, and reporting)

Statistics Theory · Cross-listed from cs.IT

Memory AMP: Overflow Avoidance, Complexity Reduction, and Comparative Analysis

Approximate message passing (AMP)-type algorithms are widely used for signal recovery in high-dimensional noisy linear systems. Recently, a framework called memory AMP (MAMP) was introduced, offering a new approach to incorporating memory terms within AMP algorithms. Building on this, a low-complexity gradient descent MAMP (GD-MAMP) was proposed for right-unitarily invariant matrices. In this paper, we first address an overflow problem in GD-MAMP caused by intermediate variables exceeding the floating-point range, which typically occurs when the condition number is large. Second, we propose two low-complexity variants of GD-MAMP: one replaces full-length memory with partial memory, while the other reduces the number of matrix-vector products per iteration by $1/3$ (from three to two). Neither degrades the convergence speed notably. Third, we develop a general gradient-based formulation for designing MAMP algorithms. This formulation recovers warm-started conjugate gradient VAMP (WS-CG-VAMP) as a special case. Furthermore, we show that the computation of the orthogonalization parameters in this formulation can suffer from catastrophic cancellation, which explains the finite-precision instability of WS-CG-VAMP. Finally, we derive an equivalent reformulation, termed WS-CG-VAMP(r), which reduces the number of matrix-vector products by up to $50\%$. Measured by matrix-vector products, GD-MAMP converges faster for small condition numbers, whereas WS-CG-VAMP(r) converges faster for large ones under high-precision arithmetic but may diverge in IEEE double precision due to catastrophic cancellation.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

No Unique Minimizer, No Problem: On the Consistency of Robust Neural Classifiers

Neural network classifiers trained by cross-entropy minimization are highly sensitive to label noise and adversarial contamination. While robust alternatives offer bounded influence and resistance to corruption, their statistical foundations in the deep learning setting are insufficient due to a fundamental difficulty: neural parameterizations are non-identifiable, so the population loss minimizer is an equivalence class of parameters, not a unique point. We develop a consistency theory for robust neural classifiers based on the S-divergence family that requires no identifiability assumption. Casting training as stochastic optimization over a non-identifiable parameter space, we prove that empirical S-divergence minimizers converge to the population-optimal equivalence class under mild regularity conditions, and verify these conditions for three architecture choices. We further establish that limit points of the robust training algorithm are stationary points of the empirical objective. Experiments on vision and language benchmark datasets confirm that S-divergence training maintains clean-data accuracy while exhibiting performance competitive with existing robust methods.
Read the full abstract

All authors Subhabrata Majumdar, Anand Deo, Partha Pratim Saha, Abhik Ghosh

Statistical Methodology

Efficient Bayesian calibration of many-parameter system models

Computer models of complex engineering systems rely on proper tuning of their model parameters to ensure accurate predictions of the system behavior. The challenge of effectively calibrating many-parameter models is the difficulty of sampling in high-dimensional spaces and the computational expense of generating a large number of samples to characterize the calibrated parameter distributions. The method of active subspaces has been shown to be effective at constructing low-dimensional latent spaces for Bayesian inverse problems when the misfit function (i.e., negative log-likelihood) is treated as the function of interest. On the other hand, works that implement surrogate modeling for inference often focus on approximating the predictive model itself. In this work, an integrated dimension reduction and surrogate modeling framework for efficient and robust model calibration based on the Kennedy O'Hagan framework is proposed, with the following key components. First, an active subspace of the misfit function is identified. Then, a surrogate model for the misfit is constructed in this low-dimensional latent space. Care is taken to ensure that the assumed probabilistic structure of the misfit surrogate is compatible with the structure imposed on the misfit by the observation noise and computer model discrepancy. Further, a generalized likelihood function is defined that can account for the misfit surrogate uncertainty along with the other usual sources of uncertainty, e.g., experimental noise, model inadequacy, etc. This general formulation is shown to be valid for surrogates of any deterministic bijective function of the original likelihood, not just the misfit. Finally, a strategy for incorporating the uncertainty in identifying the active subspace is included.
Read the full abstract

Statistical Methodology

Factor-Adjusted Location Tests for High-Dimensional Time Series

We study high-dimensional one-sample mean testing for time series with strong common serial dependence driven by latent dynamic factors. After estimating the dynamic factor loading space from lagged autocovariance, we project the data onto its orthogonal complement and construct three factor-adjusted tests: a max test for sparse alternatives, a quadratic test for dense alternatives, and a Cauchy combination test for unknown sparsity. The idiosyncratic component is allowed to be non-Gaussian sub-Gaussian vector white noise. We establish the Gumbel limit of the max statistic, the normal limit and local power function of the quadratic statistic, their asymptotic independence, and the validity of the Cauchy combination. In the strong-factor case, the refined projection expansion shows that the quadratic statistic remains valid for dimensions as large as $p=o(n^2)$. A random-loading residual bootstrap is developed for finite-sample calibration. Simulation studies and a real data application demonstrate reliable size control and competitive power for high-dimensional observations with strong dependence.
Read the full abstract

Machine Learning (Statistics)

ARC: Augmented-Rank Conformalization for Changepoint Localization — Finite-Sample Validity and Distribution-Robust Efficiency

Conformal changepoint localization turns any score into a confidence set for the changepoint with finite-sample coverage. Coverage is universal; efficiency is not. The oracle score is a likelihood ratio, so practical scores estimate density ratios, and set length deteriorates under heavy tails, skewness, and distribution shift, where no length guarantee applies. We propose ARC (Augmented-Rank Conformalization), a family of scores depending on the data only through within-segment ranks: rank-CUSUM location and scale channels, their fixed combinations, and a lightweight neural score frozen after synthetic training. Every ARC score inherits finite-sample coverage for every frozen weight configuration, including random initialization and mistraining. The main result is an efficiency transfer theorem: the entire ARC confidence set is almost surely invariant under strictly increasing marginal transforms, so the set length distribution depends on the data pair only through its rank structure, and lengths certified once hold verbatim across its monotone orbit, whereas a plug-in score's length changes with every re-expression. Across different rank structures lengths do change, and are reported as such. Classical rank-test theory positions ARC as targeting the optimal invariant score at bounded cost. Simulations confirm nominal coverage for all scores, including sabotaged networks, identical sets under monotone transforms where plug-in scores inflate, and smooth degradation where plug-in sets become vacuous; on the well-log benchmark ARC localizes annotated shifts to three to five candidates and flags misfit by an empty set. Two boundaries are stated rather than hidden: serial dependence destroys exactness, and trend-type alternatives lie outside the piecewise-exchangeable model.
Read the full abstract

All authors Chenchen Peng, Mixia Wu, Qijing Yan, Zhiqi Shen, Jie Zhang

Statistical Methodology

Population-Level Generative Modeling for Ranking Data

Ranking data arise in scientific and machine learning applications, including recommendation systems, information retrieval, voting, marketing, and AI preference ranking from human feedback. Existing statistical work has primarily focused on inference tasks such as preference estimation, rank aggregation, and ranking prediction. However, generating realistic synthetic rankings from an observed population is important for privacy-preserving data sharing, benchmark construction, simulation, and uncertainty quantification. This task is challenging because rankings are high-dimensional combinatorial objects with non-Euclidean dependence structures, while ranking populations often exhibit substantial preference heterogeneity. We propose a framework for population-level generative modeling through a latent preference simplex embedding. It estimates a low-dimensional latent preference simplex through a likelihood-based ranking model, leverages flow matching to learn the population distribution of latent preferences, and generates new rankings through the fitted probabilistic ranking model. We show that ranking generation admits an oracle reduction to latent distribution learning and derive finite-sample generative guarantees that clarify how the number of items, ranking length, and latent dimension affect accuracy. Experiments on synthetic and real datasets demonstrate improved population-level fidelity and provide a statistically interpretable representation of preference heterogeneity.
Read the full abstract

Statistics Theory · Cross-listed from cs.LG

Constrained Learning with Universally Learnable Concept Classes

We study constrained statistical learning over infinite-dimensional hypothesis classes in the fully nonconvex setting, and establish universal PACC learnability of the solutions of dual algorithms: Probably Approximately Correct on Constraints, guaranteeing optimality and constraint satisfaction at once. This strengthens near-PACC results, whose feasibility residual no amount of data can remove. Optimality is caught between generalization, governed by Rademacher complexity and favoring small classes, and strong Lagrangian duality, which rests on Lyapunov convexity for vector measures and needs decomposability, a demand pulling the other way. We reconcile the two by posing the population problem over a universal RKHS $\mathcal{H}_K$, dense in a decomposable envelope, and learning over norm balls of growing radius. This yields the Tikhonov complexity $\mathfrak{T}^{\varepsilon}_{n}$, the least RKHS norm reaching an $\varepsilon$-optimal Lagrangian level set; we prove it finite, obtain exact learnability of the optimal value, and make the sample threshold explicit and polynomial in $1/\varepsilon$ under a source condition. Feasibility is harder: absent convexity the Lagrangian may not attain its infimum, and dual information pins down only an averaged constraint-risk vector, not the risks of any returned predictor. We introduce the closure-realization gap $\varepsilon^\star_\infty$, an index of how well $\mathcal{H}_K$ retrieves feasible solutions from dualization; it is a property of the problem, not of a modeling choice. Learnability is exact when $\varepsilon^\star_\infty=0$, in particular under dual differentiability, and near-PACC with residual exactly $\varepsilon^\star_\infty$ otherwise. Finally, no distribution-free threshold exists already in the unconstrained specialization, so universality is the canonical frame for dual algorithms over large hypothesis classes.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Exact Rank and Convex Calibration Dimension Lower Bounds for the Multi-Label F1 Loss

The instance-wise $F_1$ measure is a central performance measure for multi-label classification. For a problem with $s$ labels, it defines a $2^s\times 2^s$ loss matrix. Previous work exhibited $s^2+1$-coordinate affine and shifted low-rank representations and used them to construct quadratic-dimensional convex calibrated surrogates. We determine the exact rank. Under the convention $F_1(\varnothing,\varnothing)=1$, the $F_1$ score matrix, the shifted loss matrix, and the unshifted loss matrix all have rank $s^2-s+2$, while the column-affine dimension of the loss is $s^2-s+1$. The proof factors the nonempty score matrix through subset-incidence matrices and a positive-definite Cauchy matrix. Exact rank does not, by itself, lower-bound the dimension of an arbitrary convex calibrated surrogate. We therefore analyze the Bayes geometry of $F_1$ directly. We construct a distribution for which precisely all supersets of a fixed core label set are Bayes optimal, and show that the corresponding active loss columns, restricted to the witness support, have affine dimension $hn$, where $n=s-\lfloor s/3\rfloor$ and $h=\lceil(s\lfloor s/3\rfloor)^{1/2}\rceil-1$. Applying the feasible-subspace lower bound for convex calibration dimension gives \[ \operatorname{CCdim}(L^{F_1}) \ge \left(\frac{2}{3\sqrt{3}}-o(1)\right)s^2. \] Together with the quadratic upper bound, this establishes $\operatorname{CCdim}(L^{F_1})=\Theta(s^2)$.
Read the full abstract

Statistical Methodology

A systematic framework for the identification and statistical quantification of impulsivity in condition monitoring signals

This article proposes a comprehensive framework for the identification and statistical quantification of impulsive behavior in signals, with a primary focus on condition monitoring. We concentrate on evaluating impulsivity, where such behavior results from normal operation or additional disturbances. Such an evaluation is crucial in the context of local damage detection, as the presence of impulsive disturbances significantly complicates the machine condition monitoring process. To address problem of impulsivity assessment we introduce a two-stage methodology to make processing workflow effective. First, we propose an objective selection criterion for "best-performing" impulsivity measures based on the Mann-Whitney statistic, allowing for the systematic comparison of various classical and advanced metrics across diverse signal scenarios. Second, we establish a formal procedure for assessing statistical significance using bootstrap-driven resampling and define a magnitude index to quantify the intensity of detected impulsivity. The framework is validated through extensive Monte Carlo simulations for three reference signal scenarios and applied to real-world vibration data from an industrial compressor. By systematizing existing measures and providing a statistically grounded pipeline, this research extends prior works, offering a scalable tool for distinguishing between diagnostically useful signals and those corrupted by anomalous interference.
Read the full abstract

All authors Aleksandra Grzesiek, Justyna Witulska, Daniel Kuzio, Radosław Zimroz, Tomasz Barszcz, Agnieszka Wyłomańska

Statistics Theory

Lasso Universality Under Linearly Dependent Covariates in the Sparse Regime

Throughout the last decade, Gaussian universality has been widely studied for high-dimensional estimation problems. Most of the literature focuses on i.i.d. sensing matrices or accounts for special forms of dependence, such as block dependence or other specific row/column dependencies. More general simultaneous row and column mixing has not yet been fully studied. In this paper, we focus on that setting. We prove a Gaussian universality theorem for the lasso in the sparse regime, where the non- Gaussian covariates have linearly dependent rows and columns. To the best of our knowledge, our setting permits a broader simultaneous row and column dependence structure than those treated in much of the prior universality literature. Numerical illustrations for various sparse profiles support the universality claims of this paper.
Read the full abstract

Saturday 8 August

Statistical Methodology

Three Routes to One Answer: Reconciling AIPW, TMLE, and Double Machine Learning for Applied Researchers

Augmented inverse-probability weighting (AIPW), targeted maximum likelihood estimation (TMLE), and double/debiased machine learning (DML) are three routes to the same efficient influence function for the average treatment effect — settled theory we treat as background. This tutorial's contribution is its worked, shared-nuisance reconciliation on real data: what a practitioner must actually match for the routes, and the software packages, to agree. Working the effect of smoking cessation on weight change in the open NHEFS data (n=1566) with one shared Super Learner library and identical cross-fitting folds, we build all three estimators by hand from one influence function, in full-sample, cross-fit, and double-cross-fit variants; the six resulting doubly-robust estimates span only 3.32–3.42 kg, consistent with the established benchmark. We then reconcile the same estimand across our engine and the tmle, AIPW, DoubleML, and tmle3 packages. At their defaults the estimates span 3.32–3.49 kg. Once the library and folds are matched and single-split noise is averaged out, the three library-sharing implementations agree to within 0.01 kg — so the residual spread traces to the nuisance library, folds, and repetitions, not to the estimator label. Because the estimators share one influence function, they agree under good overlap; under a positivity violation the pooled ATE is not identified without additional extrapolation assumptions, and their finite-sample estimates can then diverge sharply. We therefore place a positivity diagnosis ahead of estimator choice, illustrate the failure on a no-overlap example, and close with a reporting checklist. Open-source R code reproduces every number.
Read the full abstract

Statistics Applications

Real-time anomaly detection in base station testbeds via scalable kernel density estimation framework

Large-scale testing infrastructures are critical for validating telecommunication systems, yet their growing complexity makes efficient resource utilization and anomaly detection increasingly challenging. In reservation-based testbed environments, errors in resource allocation or preparation often manifest as abrupt spikes or regime changes in time-based metrics. This paper proposes a scalable, unsupervised framework for real-time anomaly detection in such environments. We introduce CALM (Continuous Anomaly Localization for univariate and Multivariate data), a nonparametric method based on kernel density estimation and bootstrap-based thresholding, designed for anomaly detection at the individual testbed level. To address system-wide visibility, we further propose AggCALM, an aggregation framework that consolidates local anomaly signals across multiple testbeds to detect statistically significant global anomalies while mitigating alarm fatigue. The methodology is evaluated using simulated multivariate data and real-world data from a large-scale base station testing platform. Results demonstrate that the proposed framework enables timely, flexible, and accurate anomaly detection without requiring labeled data, supporting reliable operation of complex test environments. Although the proposed methodology is presented within the context of a telecommunication testing labs, it can be effectively used to other applications, such as condition monitoring, where anomaly detection serves as a pivotal pre-processing step for diagnostic signals.
Read the full abstract

All authors Justyna Witulska, Marcin Szczukiewicz, Artur Tabaka, Rafał Sarniak, Ireneusz Jabłoński, Agnieszka Wyłomańska, Dominik Dulas

Statistics Theory

Pointwise convergence of purely random partition estimators: from random trees to prototype rules

We study pointwise convergence rates of purely random partition estimators in nonparametric regression, where the partition – into hyper-rectangles by purely random trees, or into Voronoi cells by prototype rules – is built independently of the responses. Our analysis rests on a single geometric criterion, shape regularity, relating the diameter of a cell to its volume, which is shown by Bettinger, Portier and Saumard (2026) to be necessary and sufficient, up to logarithmic factors, for achieving the minimax rate $n^{-1/(d+2)}$. We show that centered and uniform trees are not shape-regular – their cells' aspect ratio grows exponentially with the number of splits with probability bounded away from zero – explaining the super-logarithmic corrections in their error bounds, whereas Mondrian trees, whose splits adapt to the current cell geometry, are shape-regular in probability and attain the minimax rate. The same analysis applied to Voronoi partitions yields the first pointwise concentration bounds for Proto-NN, resolving an open problem of Györfi and Weiss (2021), and shows that OptiNet achieves the minimax rate with markedly better success probability – even almost surely, for a suitable choice of parameters – thanks to its $\eta$-net construction.
Read the full abstract

Statistics Theory

Nonparametric quantile inference using Dirichlet processes

This chapter deals with nonparametric inference for quantiles from a Bayesian perspective, using the Dirichlet process. The posterior distribution for quantiles is characterised, enabling also explicit formulae for posterior mean and variance. Unlike the Bayes estimator for the distribution function, our Bayes estimator for the quantile function is a smooth curve. A Bernstein–von Mises type theorem is given, exhibiting the limiting posterior distribution of the quantile process. Links to kernel-smoothed quantile estimators are provided. As a side product we develop an automatic nonparametric density estimator, free of smoothing parameters, with support exactly matching that of the data range. Nonparametric Bayes estimators are also provided for other quantile-related quantities, including the Lorenz curve and the Gini index, for Doksum's shift curve and for Parzen's comparison distribution in two-sample situations, and finally for the quantile regression function in situations with covariates.
Read the full abstract

2608.08355 · 8 Aug · 30 pages, 4 figures. This is the Statistical Research Repoert version, Department of Mathematics, University of Oslo, later published in "Advances in Statistical Modeling and Inference", Festschrift for Kjell Doksum, 2007, IMS Lecture Notes Series (ed. V. Nair), Ch. 12. The paper is arXiv'd since it contains more material than in the published version and for easier accessability

Statistics Theory

On inferential equivalence classes of causal models

Causal models in the same Markov equivalence class are, in the absence of strong assumptions, statistically indistinguishable at any sample size. For modest sample sizes there is also the possibility that several such classes are compatible with the data, pointing to a confidence set of causal models as the appropriate presentation of evidence. Non-identifiability of Gaussian causal models from the same Markov equivalence class corresponds to a plurality of inverse-covariance representations in the unconstrained parametrisation of Cox and Wermuth (1993). This parametrisation, avoiding conic constraints that would otherwise complicate distributional approximations, facilitates construction of, and theoretical analysis for, a confidence set of causal models based on standard likelihood theory. By drawing on a geometric formulation of Evans (2020), we provide insight into which causal models are most likely to be included in the confidence set when the true parameter values of the generating model are at the borderline of detectability, delineating those models whose inclusion probabilities are stable at the nominal level, slowly decaying with sample size, and quickly decaying with sample size. We also study settings in which two or more causal models are in operation, exploring how the mixture weights, and the geometry of the models in the mixture relative to candidate models, interact with the inclusion probabilities. The purpose of the paper is to probe, from the perspective of structural properties of the true causal mechanism, the limits of what is achievable in causal inferential settings.
Read the full abstract

Statistics Applications

A Gaussian Process Model of 3D Udder Point Clouds for Teat Length Phenotyping in Dairy Cows

Phenotyping conformation traits is important for dairy cattle breeding and management. Although large-scale phenotyping is possible with 3D imaging technologies, manual annotation of anatomical landmarks and long run times prevent full pipeline automation. In particular, the morphological heterogeneity of cow udders makes automated detection of teat landmarks challenging. To address this limitation, we propose and evaluate a method for teat length estimation from udder point clouds with a Gaussian process. We model the vertical coordinates as the sum of a Gaussian process representing the udder floor and an unknown function representing the teat. Since the udder floor process is smooth and has a significantly wider dependence lengthscale than the teat function, this model allows separating the two terms needed for teat landmark definition. To ensure computational feasibility, we implement a low-rank approximation of the covariance matrix, reducing the computational complexity of the method from $\mathcal{O}(n^3)$ to $\mathcal{O}(n)$. This approach is both faster and more accurate than existing methods, reducing RMSE by a factor of two. It is also more robust to uncommon udder morphologies, making it better suited for automated phenotyping of large numbers of individuals.
Read the full abstract