Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Mathematics Statistics Theory

math.ST Watch Statistics Theory (sign in to use)

Tuesday 25 August

Statistics Theory · Cross-listed from math.PR

A Unified Exact Factorial-Moment Theory for Multi-set Allocation Occupancy (MAO) in Finite Populations

Let $A_1,\ldots,A_T$ be independent uniformly selected subsets of a finite population of size $n$, with prescribed cardinalities $m_1,\ldots,m_T$. For each population element, define its occupancy level as the number of selected subsets containing it. Let $x_t$ and $x_{\geq t}$ denote the numbers of elements with occupancy exactly $t$ and at least $t$, respectively. The 2025 work introduced a general multi-set allocation occupancy (MAO) representation for higher-order occupancy moments. The present paper establishes its joint-probabilistic interpretation and provides a rigorous unified derivation for arbitrary joint occupancy categories and moment orders. Specifically, for arbitrary $B_1,\ldots,B_\ell\subseteq\{0,1,\ldots,T\}$, we prove the exact representation $F_\ell(B_1,\ldots,B_\ell)=G_T(B_1,\ldots,B_\ell)/(n)_\ell^{T-1}$, where $G_T(B_1,\ldots,B_\ell)$ is the corresponding generalized MAO transversal sum. This identity unifies the joint factorial moments of arbitrary occupancy categories within a single exact finite-population framework. In particular, writing $B_{\geq t}=\{t,t+1,\ldots,T\}$, the factorial moments of exact- and threshold-occupancy counts are obtained as the specializations $\mathbb{E}[(x_t)_\ell]=F_\ell(\{t\},\ldots,\{t\})$ and $\mathbb{E}[(x_{\geq t})_\ell]=F_\ell(B_{\geq t},\ldots,B_{\geq t})$. Mixed factorial moments, raw moments, variances, and covariances follow from the same representation through standard transformations. The formulas are verified by exhaustive enumeration over feasible parameter ranges and by Monte Carlo simulation. The resulting theory provides a rigorous and unified finite-population foundation for exact and threshold multi-set occupancy statistics.
Read the full abstract

Statistics Theory

Learning from Uncertainty-dependent Missing Labels for Semi-supervised Classification

Missing labels are usually regarded as a source of information loss in classification. We study a semi-supervised setting in which the probability of label missingness depends on the observed features through posterior classification uncertainty. In this setting, the missingness indicator is not only a record of an unobserved label, but also an observable signal generated by a mechanism linked to the classifier. We develop a likelihood-based information theory for such uncertainty-dependent missing labels. Under correct specification, we derive a Fisher-information decomposition that separates a partial-labeling component from a nonnegative mechanism-curvature term. Under joint misspecification of the label model and the missingness mechanism, we obtain the corresponding Godambe–Eicker–Huber–White sensitivity and sandwich-covariance partitions. We also clarify the relevant complete-data benchmark: favorable missingness can increase information relative to ordinary fully labeled or budget-matched non-informative labeling baselines, but cannot exceed the information in the augmented experiment in which labels and mechanism indicators are both observed. For plug-in classifiers, we connect the information decomposition to margin-based excess-risk bounds. In regular two-component mixture settings this yields the parametric \(n^{-1}\) excess-risk rate, with constants determined by the nuisance-adjusted information in discriminant directions. Gaussian-mixture calculations and a medical diagnosis example illustrate how uncertainty-dependent labeling mechanisms can improve estimation and classification under a fixed labeling budget.
Read the full abstract

Statistics Theory · Cross-listed from stat.ML

Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime

Modern score-based generative models have achieved remarkable empirical success in high-dimensional tasks such as image, audio, and video synthesis. These models reduce distribution learning to a sequence of regression problems that, if solved exactly on finite data, would ultimately reproduce the training samples. Their ability to generalize must therefore arise from the implicit or explicit regularization during training. In this work, we develop a generative counterpart to the theory of benign overfitting and algorithmic regularization for overparameterized neural networks in the supervised lazy-training regime. We study denoising score matching in a vector-valued reproducing kernel Hilbert space with an inner-product kernel. In the proportional high-dimensional regime $n\asymp d$, we derive exact risk trajectories under gradient flow training. These trajectories exhibit three phases governed by qualitatively distinct estimators: a spectral estimator that generalizes, a pure-noise score with localized peaks that interpolate the training objective, and an empirical Bayes estimator that memorizes the data. We then analyze how these estimators combine along the reverse-time SDE and characterize the distribution of the resulting samples. The analysis reveals familiar mechanisms from supervised learning, including kernel linearization and self-induced regularization from the nonlinear part of the kernel, but also reveals a distinct phenomenology specific to generative modeling.
Read the full abstract

All authors Hugo Latourelle-Vigeant, Sinho Chewi, Aram-Alexandre Pooladian, John Sous, Theodor Misiakiewicz

Monday 24 August

Statistics Theory

Fourth-Moment Strong Universality for Finite-Type Transpose-Correlated Random Matrices

We prove a strong-universality theorem at the exact fourth-moment threshold for finite families of non-Hermitian random matrices assembled from independent unordered-pair vector atoms. Within one atom, the matrix colors and the two endpoint orientations may have arbitrary joint real covariance, subject to reversal consistency across ordered type pairs and endpoint exchangeability in same-type blocks; the law may also depend on finitely many endpoint types. The matrices may be adjoined to an arbitrary deterministic tuple that converges jointly strongly with the type projections. For every fixed matrix amplification and fixed noncommutative \(*\)-polynomial, the resulting tuple converges strongly to an explicit covariance-matched free Gaussian family. Cross-type blocks are described by masked circular variables, whereas same-type blocks split into independent endpoint-symmetric and endpoint-antisymmetric semicircular sectors. Only a finite radial fourth moment is assumed off the diagonal, and a finite second moment suffices on the diagonal. The mode of convergence depends on the coupling: corners of one infinite array converge almost surely on a common event, while fixed-law nonnested triangular arrays converge in probability for each fixed test. As an application, we obtain exact-fourth-moment strong limits for finite-separable continuous left/right profiles, including separately weighted literal-transpose terms. For the nested almost-sure formulation, the fourth-moment threshold is sharp already on the Wigner subfamily; at this threshold arbitrary fresh rows admit no coupling-invariant almost-sure upgrade of our in-probability conclusion.
Read the full abstract

Statistics Theory

Ranking by points and ordinal models

Sport leagues assign a fixed score-point to each match outcome and rank the players (or teams) by their accumulated scores. Since the score-points are mandated without a probabilistic model in mind, we investigate models that make the score the right summary of the data. We find the score to be a sufficient statistic if and only if the points rule is constant-sum, so that every match distributes a fixed number of points between its participants, and the outcomes follow the adjacent categories model whose slope parameters are the score-points themselves. The constant-sum condition fails in the National Hockey League and in football, which awards three points for a win instead of the two required by sufficiency. We also establish that when two players meet every other opponent equally often, and each plays as often at home as away, they are ordered identically by the score and by the estimated skills; a venue-balanced round-robin satisfies the condition for all pairs. The Elo ranking then produces the same order. In nine leagues from football, ice hockey and volleyball, the two orders differ only where the schedule condition fails, and the uniform rule awarding one point per outcome level agrees with the estimated skills more closely than each league's own rule, unless that rule is already the uniform one.
Read the full abstract

Statistics Theory · Cross-listed from cs.LG

Provably adaptive sampling with uniform and remasking discrete diffusion models

Discrete diffusion models offer a promising alternative to autoregressive generation by enabling parallel updates, but their sampling efficiency can depend strongly on the choice of the forward process and the sampler. For the uniform forward process, existing lower bounds for the standard $\tau$-leaping sampler scale linearly with the ambient dimension $d$, raising the question of whether this dependence is intrinsic to the forward process. We answer this question in the negative. We consider a first-order sampler based on the leave-one-out denoiser for uniform and remasking processes whose coordinate updates can be performed in parallel. In both cases, the sampler can correct denoising mistakes during the sampling process, which becomes necessary when many coordinates are updated together. Our main result establishes an adaptive sampling guarantee: up to logarithmic factors, $N = O(\mathrm{DTC}(X_0) / \varepsilon)$ discretization steps suffice to achieve sampling error $O(\varepsilon_{\mathrm{score}}+\varepsilon)$, where $\varepsilon_{\mathrm{score}}$ is the error in score estimation. Thus, the sampling complexity is governed by the intrinsic dependence structure of the target distribution, as measured by its dual total correlation $\mathrm{DTC}(X_0)$, rather than directly by the ambient dimension $d$. Our analysis proceeds through a Bayes-optimal auxiliary sampler that separates discretization error from score-estimation error. We also derive an exact information-theoretic representation of the discretization error in terms of the mutual information between different coordinates of the forward process at different times. This representation applies to general forward processes and, in the uniform and remasking cases, can be controlled by $\mathrm{DTC}(X_0)$. Numerical experiments on structured synthetic distributions illustrate the predicted dimension-adaptive behavior.
Read the full abstract

Statistics Theory

Asymptotics for Model Selection in Probabilistic Principal Component Analysis

The probabilistic formulation of principal component analysis promises statistically grounded solutions to the problem of selecting the number of principal components. However, developing tractable model selection methods is complicated by the fact that the probabilistic principal component analysis (PPCA) model exhibits non-standard large sample asymptotics at model singularities, where the Fisher information matrix does not have full rank. In this work, tools from singular learning theory are used to provide a complete description of the marginal likelihood asymptotics for PPCA. The singular Bayesian information criterion (sBIC) along with our asymptotic results provides an effective procedure for selecting the number of principal components. In particular, the sBIC corrects for the overpenalization of the standard dimension-based BIC, which leads to too few principal components being selected, while also selecting the smallest true model with probability converging to one. Our framework extends beyond PPCA, where we also provide expressions for the sBIC that can be used to select between PPCA and factor analysis models. In both simulations and on real data the effectiveness of the proposed sBIC methodology is demonstrated.
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

Transporting Randomized Trial Effects to Real-World Populations via Riesz-Calibrated Optimal Transport

Randomized trials support causal inference, but differences between trial and target populations can limit the transportability of treatment effects to real-world settings. Many existing approaches model the propensity of trial participation and can therefore be sensitive to model misspecification and weak overlap of the covariate distributions. Optimal Transport (OT) offers a different route by comparing the trial and target populations directly in covariate space. We develop RICOT, a Riesz-calibrated OT procedure transporting treatment effects to a treated target population. We consider a semi-unbalanced OT with entropic regularization where the source marginals are relaxed. We show that the uncalibrated OT introduces a bias which does not shrink with increasing sample size. RICOT removes this bias by imposing calibration equations directly within the transport problem. With a growing calibration sieve, the calibrated weight consistently estimates the target-to-trial density ratio, equivalently the Riesz representer of the target expectation functional, even when the entropic and source-relaxation parameters remain fixed and positive. Combined with outcome regression, the resulting estimator is doubly robust and attains the semiparametric efficiency bound under suitable rate conditions. Its variance is estimated directly from the influence function, without resampling or repeated OT optimization. Simulations show low bias and near-nominal coverage across a range of overlap and misspecification settings, including settings in which sampling-score methods perform poorly. We illustrate RICOT in a real-world application involving a rare progressive cardiomyopathy, comparing conventional IPW and AIPW estimators with our OT-based IPW and doubly robust estimators for transporting the randomized treatment effect to a real-world population receiving the same treatment.
Read the full abstract

All authors Anik Burman, Margaret Gamalo, Promit Ghosal, Prosenjit Kundu

Statistics Theory

Spectral stability of empirical metric-measure Laplacians

The variance of nonparametric estimators is typically insensitive to the regularity of the object being estimated. We establish such a property for the spectra of graph Laplacian matrices at a fixed bandwidth $h>0$. Specifically, given $n$ i.i.d. samples from a probability measure $\mu$ on a Polish metric space, we compare the eigenvalues of the empirical weighted Laplacian operator $\Delta_{\mu_n}^h$ to those of the population counterpart $\Delta_\mu^h$ under a spectral gap condition, bounding the relative error by $1/\sqrt{nv_\mu(h)}$ for eigenvalues of order smaller than $h^{-2}$, where $v_\mu(h)$ is the smallest mass of a ball of radius $h$. This bound requires very weak regularity conditions on $\mu$: it is satisfied if $\mu$ belongs to the class of coarse PI measures that we introduce. This class contains measures on metric graphs, spaces with sufficiently regular boundaries, corners, or branch points, together with discretizations or thickenings of these at scale $O(h)$. Even for measures having densities of regularity $s>2$ on manifolds (the only known case so far), our bound improves on the state-of-the-art by shaving off logarithmic factors.
Read the full abstract

Statistics Theory · Cross-listed from quant-ph

Batched and Complete U-Statistics for Trace-Polynomial Estimation from Classical Shadows

We study estimation of the trace polynomial $\operatorname{tr} p(P\rho P)$ from global classical shadows, where $\rho$ is an unknown quantum state and $P$ is a fixed projector. Disjoint batching and complete U-statistics yield unbiased estimators of the same trace moments, but assign different sample-size factors to the degenerate terms in their Hoeffding decompositions. Under the global Clifford protocol, exact degree-two variance formulas show that, on a null projected block of rank $s$, the quadratic degenerate term has order $s^2/N$ under batching and $s^2/N^2$ under complete symmetrization. For a logarithmic-degree polynomial used in entropy approximation, the quadratic coefficient raises the batched variance to at least order $s^2N\log^2N$ at the classical entropy cutoff. For complete U-statistics, we derive a cross-degree covariance identity and an exact variance decomposition for polynomial estimators. We also bound every Hoeffding order at a fixed degree and obtain a growing-dimensional risk bound for a small-spectrum entropy functional. The higher-order bounds retain a polynomial dependence on the ambient dimension and therefore do not cover logarithmically increasing degrees. Monte Carlo experiments confirm the degree-two formulas, and exact calculations illustrate the entropy risks.
Read the full abstract

Statistics Theory

Pre-Disclosure Experiment Menus: Oracle-Relative Risk and Joint Sample–Menu Asymptotics

We study a resolution problem in local asymptotic decision theory: individual risks may admit Gaussian approximations that do not determine their vanishing difference. A finite menu of experiments is installed before context disclosure, although observations may be routed adaptively afterward. A greatest-element Blackwell order collapses adaptive routing to the best installed experiment and reduces the fixed-menu excess to an inverse-information distortion with frontier $A_k$. We develop differentiated, all-prior posterior transfer along a one-dimensional degradation chain and establish $F_{n,k_n}(H_n)=A_{k_n}\{1+o(1)\}$ for every diverging menu sequence with positive frontier and every admissible localization radius, without an additional direct sample-menu restriction. The transfer is exact under Gaussian degradation. Prior-free likelihood-generator conditions imply it for jump generators and are verified for binary attenuation, Poisson thinning, and negative-binomial thinning. If the distortion is uniformly quadratic on an Ahlfors-regular oracle image of dimension $r$, then $A_k\asymp k^{-2/r}$, and the original-scale excess mean squared error is of order $n^{-1}k^{-2/r}$. Calibrated Poisson sensor and radial-qubit measurement menus illustrate the result. A triangular Gaussian counterexample shows why pointwise Gaussian convergence is insufficient.
Read the full abstract

Statistics Theory · Cross-listed from stat.CO

Favourable Missingness in Semi-Supervised Classification for Exponential Mixture Models

Semi-supervised classifiers are commonly trained from samples in which all features are observed but some class labels are missing. When label missingness is independent of the observed data, unavailable class memberships reduce Fisher information relative to a completely classified sample. We study a different regime in which the probability of label missingness depends on posterior classification uncertainty, so that the observed missing-label indicators can themselves carry information about the Bayes decision boundary. Building on the conditionally weighted information decomposition of Ahfock and McLachlan, we develop this phenomenon for a two-component exponential mixture. Although the exponential model is non-Gaussian, asymmetric, and supported on the positive half-line, its log-posterior odds remain linear in the feature. We derive Bayes' rule and its exact error rate, formulate entropy-logistic and squared-discriminant missingness mechanisms, and obtain the full partially classified likelihood. We then derive a decomposition of the Fisher information into the complete-data information, the conditionally weighted loss due to missing labels, and the information contributed by the missing labels. Numerical quadrature identifies regions in which the full likelihood classifier has asymptotic relative efficiency above or below one. Monte Carlo experiments with finite training samples broadly support the population calculations, with the largest departures from the asymptotic predictions occurring near the transition at which the relative efficiency crosses one.
Read the full abstract

All authors Huanchao Zhou, Jinran Wu, Fariborz Setoudehtazang, Geoffrey J. McLachlan

Sunday 23 August

Statistics Theory

Randomized Spectral Inference for Hyperuniformity

We test hyperuniformity from one large realization of a stationary point process. Hyperuniformity is equivalent to the average $A_r$ of the structure factor over $B_r$ vanishing as $r\downarrow0$, so the target is a low-frequency average rather than the value of the structure factor at the origin, which need not exist. We estimate $A_r$ from squared Fourier coefficients at frequencies drawn uniformly from $B_r$; if these coefficients are asymptotically Gaussian at almost every fixed frequency, the squared coefficients converge to independent variables with mean $A_r$. Assuming $A_r=s+c r^\alpha+\varepsilon_r$ with $|\varepsilon_r|\leq Lr^\beta$ and given exponents $0<\alpha<\beta$, a two-radius extrapolation removes the $r^\alpha$ term and estimates $s$ with deterministic error of order $r^\beta$. This gives confidence bounds and a one-sided test of $H_0:s=0$ with asymptotic level at most $\gamma$, consistent against every fixed $s>0$ as $R\to\infty$, then the number of sampled frequencies tends to infinity, and then $r\downarrow0$. Essentially free translation actions with completely positive entropy satisfy the Fourier assumption.
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

Estimating Pathway Treatment Effects in the Presence of Intermediate Events with Multi-State Data

During clinical trials evaluating a drug's effect on a survival endpoint, intermediate events often occur in addition to the primary event. The treatment can exert its effect on the primary endpoint along multiple pathways through intermediate events. Assumptions for identifying mediation effects, such as sequential ignorability in natural effects or the dismissible components condition in separable effects, fail because intermediate events act as treatment-induced confounding. To understand the effect along each pathway, we consider hypothetical interventions in transitions between event statuses to mimic the treatment mechanism. The hypothetical interventions adjust for effects through intermediate events and marginalize over unobserved treatment-induced confounding, if any. Based on the derived efficient influence functions for the counterfactual cumulative incidences under hypothetical interventions, we construct multiply robust and semiparametrically efficient estimators for pathway treatment effects. Our proposed framework enables the examination of treatment effects through each transition, on each event, and along each path. By analyzing data from the LEADER Trial, we find that liraglutide significantly reduces the risk of cardiovascular and microvascular events. The reduction in all-cause mortality is primarily mediated by its effects on expanded major adverse cardiovascular events.
Read the full abstract

All authors Yuhao Deng, Haoyu Wei, Donglin Zeng, Rui Song, Xiao-Hua Zhou

Statistics Theory · Cross-listed from stat.ML

Sparse Additive Off-Policy Evaluation for Reinforcement Learning with Potentially Limited Number of Trajectories

We develop a new framework for flexible, nonlinear, and interpretable off-policy evaluation for infinite-horizon reinforcement learning. To handle large state spaces and support transparent decision-making, we model the Q-function using a nonlinear function class with a sparse additive structure. We derive high-probability finite-sample error bounds for estimating the value function of a target policy and show that the bounds depend only logarithmically on the ambient dimension $d$, thereby alleviating the curse of dimensionality. In contrast to most existing theory for off-policy evaluation, which typically assumes access to many trajectories, our analysis guarantees accurate value estimation when either the number of trajectories or the time horizon is sufficiently large. In addition, we propose a group-sparsity-based feature screening procedure that identifies, with high probability, a reduced feature set containing all relevant covariates. Numerical experiments demonstrate the effectiveness of the proposed approach.
Read the full abstract

All authors Tuoyi Zhao, Chengchun Shi, Zhengling Qi, Lan Wang

Statistics Theory · Cross-listed from stat.ME

Detection of Cognitive Diagnostic Model Misspecification using New Lancaster-Chesher Information Matrix Tests

Model specification tests play a crucial role in evaluating the appropriateness of probability models for estimation and inference. Existing methods for the detection of model misspecification such as the chi-square goodness-of-fit (GOF) tests and more recently the M2 statistic \autocite{MaydeuOlivares2005,MaydeuJoe2014} tend to result in test statistics with excessive degrees of freedom for models with larger numbers of parameters. An alternative approach is based upon the Information Matrix (IM) equality. The IM equality asserts that if a probability model is correctly specified, the asymptotic covariance matrix of the maximum likelihood estimators can be asymptotically estimated using a methodology based upon either the first or second derivatives of the log-likelihood function. Using a contrapositive argument, White (1982) \nocite{Wh82} proposed a misspecification test methodology based upon comparing these two alternative covariance matrix estimators. Extending this work, Presnell and Boos (2004) \nocite{Presnell2004} showed how to develop a misspecification test which only requires one degree of freedom regardless of the complexity of the model or data. In this paper, we extend prior work and additionally apply methods of Golden et al. (2013, 2016) \nocite{golden2013}\nocite{Golden2016} to derive and evaluate misspecification tests for Cognitive Diagnostic Models (CDMs) which only require 1 or 2 degrees of freedom regardless of model or data complexity. Analytic formulas for the tests are derived so they can be applied without requiring computationally intensive bootstrap simulation methods. Our simulation studies show the asymptotic statistical tests have good level (type 1 error) and power performance for CDM models and data which might be encountered in practice.
Read the full abstract

Statistics Theory

Non-asymptotic Analysis of Matérn Regression: The Roles of Target and Kernel Lengthscales

Theoretical guarantees for kernel regression are typically formulated in terms of smoothness, but practical accuracy depends critically on how the design resolution compares with the target and kernel lengthscales. We develop a finite-sample theory for Matérn regression on periodic domains with quasi-uniform designs, covering noiseless interpolation and noisy kernel ridge regression. Our minimax result shows that accurate recovery requires a design dense enough to resolve the target lengthscale and sufficient information at that scale to overcome noise. We prove that Matérn interpolation additionally requires the design to resolve the kernel lengthscale: if the kernel is too short relative to the point spacing, an additional error remains even when the target is well resolved. For noisy Matérn regression, we derive a three-term fixed-ridge risk characterization consisting of target-scale bias, kernel-scale bias, and variance, and show that optimizing over the ridge parameter yields four distinct contributions. When the target is no more than twice as smooth as the kernel, choosing a kernel lengthscale longer than the target does not worsen the oracle risk, whereas choosing one too short can. Thus these resolution conditions determine when accurate recovery becomes possible, while smoothness determines how rapidly the error decreases thereafter.
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

Exact grid-free confidence-region computation from dependent p-value functions under arbitrary dependence

Repeated sample splitting, cross-fitting, conformal ensembling, and related randomized workflows often produce multiple dependent valid p-value functions for the same target. Existing p-merging theory guarantees pointwise validity under arbitrary dependence, but turning the merged output into a confidence region typically requires repeated evaluation on a grid over the parameter space. This inversion step can be computationally costly, approximation-dependent, and increasingly difficult to scale. We study when confidence regions can instead be computed exactly from split-wise regions without gridding the parameter space. Our main structural result shows that mergers induced by step calibrators form a broad exactly executable class at the region level, and a partial converse within the calibrator-induced family indicates that exact executability is closely tied to step structure. Within this class, we develop exact voting algorithms, including an adaptive multi-quantile contour aggregator that avoids pre-specifying a single order-statistic threshold while preserving finite-sample validity under arbitrary dependence. Simulation studies on repeated-split regression and conformal prediction, together with real-data regression examples, show that the proposed methods provide stable, robustness-oriented exact inference with substantial runtime gains over grid-inversion comparators. A grid-resolution benchmark shows that fixed-k voting and adaptive multi-quantile voting are essentially insensitive to inversion-grid refinement, while a multidimensional single-threshold stress test illustrates the dimensional blow-up faced by grid-inversion baselines in a simple box-geometry setting. Taken together, the results show that arbitrary-dependence contour merging can be turned into an exactly executable region-computation framework rather than merely a pointwise validity device.
Read the full abstract

Statistics Theory · Cross-listed from cs.DS

Near-Optimal Bounds for Sketching the Schatten Norms

Let $k_{1,\varepsilon}(n)$ be the smallest number of real linear measurements needed by a randomized oblivious sketch that estimates the nuclear norm of every fixed real $n\times n$ matrix within a factor $1\pm\varepsilon$, with probability at least $2/3$. For every fixed $0<\varepsilon<1$, we prove \[ \frac{n^2}{(\log n)^{A_\varepsilon}} \le k_{1,\varepsilon}(n) \le C_\varepsilon \frac{n^2\{\log\log(e^e n)\}^2}{\log(e n)}. \] Previously, the best unrestricted bounds for general linear sketches of the Schatten–1 norm were $\Omega(n)$ and the trivial $O(n^2)$ upper bound (Li, Nguyen, Woodruff'19), leaving a polynomial gap. Our bounds close that gap up to polylogarithmic factors and give a nontrivial logarithmic saving below the $n^2$-measurement storage bound. The result extends much further. Write $k_{p,\varepsilon}(n)$ for the analogous sketch dimension for the Schatten–$p$ norm. For every fixed finite $p>0$ that is not a positive even integer, there are positive constants $A_{p,\varepsilon},C_{p,\varepsilon},c_p$ such that \[ \frac{n^2}{(\log n)^{A_{p,\varepsilon}}} \le k_{p,\varepsilon}(n) \le C_{p,\varepsilon}\frac{n^2}{(\log n)^{c_p}}, \] so $k_{p,\varepsilon}(n)=n^{2-o(1)}$ throughout the non-even regime. Together with the known tight bounds $\Theta_{p,\varepsilon}(n^{2-4/p})$ for positive even $p$ and $\Theta_\varepsilon(n^2)$ for $p=\infty$ (Li, Woodruff'16), our results close the remaining polynomial gap across the Schatten family and complete, up to polylogarithmic factors, the polynomial-order classification of general linear sketches for all Schatten-$p$ norms.
Read the full abstract

Statistics Theory

Robust Scale Estimation in Additive Noise via Weighted Order Statistics

This manuscript develops a non-parametric and robust framework for estimating the scale of additive noise in weakly sparse systems. The method does not require independence, prescribed dependence, or temporal regularity of the noise sequence. We introduce a class of order-statistic estimators based on comparing the sorted observations with deterministic or random proxies generated from a reference noise distribution. This purely spatial approach avoids preliminary filtering or temporal decorrelation, and therefore preserves the sparsity structure of the latent signal. We establish non-asymptotic concentration inequalities for weighted loss functions, with bounds that separate the contribution of the signal from the discrepancy between the ordered noise and the proxy. We then control this proxy discrepancy in independent and correlated regimes, including heavy-tailed reference laws. Finally, we apply the method to high-frequency observations of continuous-time stochastic processes, obtaining scale estimators for fractional Brownian motion and stable Lévy noise in the presence of lower-variation additive perturbations.
Read the full abstract

Saturday 22 August

Statistics Theory

Comparing the reliability of coherent systems with heterogeneous, dependent and distribution-free components

In recent years, there has been a growing interest in comparing the reliability of systems with dependent components. Most of the recent works consider Archimedean copulas to define the dependency structure of systems and examine specific familiy of distributions. In this work, we provide distribution-free results to compare, in the usual stochastic order under some majorization conditions, coherent systems with heterogeneous and dependent components where the dependency structure can be defined by any copula. Applications of our main results for specific copulas and different families of marginal distributions are also given.
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

Dirichlet Process Mixtures of Trees with Gaussian Process Splits: A Bayesian Nonparametric Framework with Posterior Contraction Rate

We propose a Bayesian nonparametric mixture of regression trees with a Dirichlet process prior over tree-parameter pairs, enabling data-driven selection of ensemble size and unifying CART, BART, random forests, and boosting. A novel splitting rule driven by the posterior predictive of a Gaussian process within each terminal node generates flexible, smooth decision boundaries; remarkably, the GP density cancels exactly in the Metropolis–Hastings ratio for GROW/PRUNE moves, ensuring computational feasibility. An exact Gibbs sampler for posterior predictive inference propagates uncertainty through random tree traversal. A parallel MPI implementation distributes independent tree updates across processors, achieving adequate speedups. We prove posterior consistency at rate $n^{-1/4}$ in Hellinger distance under only continuity of the true regression function, allowing misspecification, via the identity $h(\Theta)=0$. Simulations on Friedman benchmark show near-nominal coverage (0.94 Gaussian, 0.92 Cauchy), robust to high-dimensional noise and heavy tails, outperforming BART and bagged CART. Applications to QSAR toxicity, crime, riboflavin, wheat genomics, and air quality confirm reliable credible intervals and automatic sparsity. The DP mixture offers a principled, robust, theoretically justified alternative for challenging regression with honest uncertainty quantification.
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

Separating Time-Varying Network Composition from Predictive Dependence under Noisy Network Measurement

A common question about networked time series is whether outcomes changed because shocks transmit more strongly or because the pattern of connections changed. Standard practice inserts a recorded network into an outcome regression and reads movements of the fitted coefficient as changes in transmission strength. When the network is latent, time varying, and measured with error, this reading fails: changes in strength and changes in composition can produce the same outcome distribution at a single date, and the population coefficient moves under composition changes alone. The question becomes answerable when outcomes are analyzed jointly with repeated noisy measurements of the network, such as paired reports of bilateral trade flows. For the joint model we establish necessary and sufficient conditions for local identification, estimators of the strength and composition paths, a simultaneous confidence band for the strength path, confidence sets that remain exact under weak identification, breakdown bounds under common reporting bias, and an exactly sized test that detects changes on the observed path and attributes them to strength or to composition. Simulations assess each procedure at its stated boundary. On a mirror-reported trade panel of eighteen economies over 1995 to 2020, the diagnostics flag exactly the crisis years and the composition coordinate attached to European Union membership declines by roughly two thirds. The estimand is predictive dependence, not a causal effect.
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

Optimal Variance Reduction in Randomized Experiments

This paper describes an approach to variance reduction in randomized experiments using side information (covariates of the response unaffected by treatment). As with Double ML[Chernozhukov 2018], models of arbitrary complexity may be employed without concern for bias due to overfitting or regularization. For additive treatment metrics, the approach minimizes variance optimally. For more complex treatment metrics (e.g., multiplicative and ratios) the Delta Method estimate of variance is minimized. Through a variety of examples, the paper also explores modeling considerations.
Read the full abstract

Statistics Theory · Cross-listed from econ.EM

Uniformly Valid Inference Under Interactive and High-Dimensional Constraints

Asymptotic normality approximations often fail to hold for extremum estimators when the true value of the parameter is at or close to the boundary of a parameter space. I analyze and develop tests using a quasi-unconstrained estimator, which is asymptotically normal even when the true parameter vector is near or at the boundary. These results generalize previous work with this estimator by allowing for more types of constraints and showing how the method can naturally be modified when a nuisance parameter is also high-dimensional. I show that variations of Wald, Likelihood Ratio, and Lagrange Multiplier tests can control size in a uniform sense, provided the initial constrained estimator is sufficiently accurate. Lastly, I apply the method to an application involving network estimation with panel data.
Read the full abstract

Statistics Theory

A new matrix-variate integer-valued autoregressive process with matrical negative binomial thinning

To address the overdispersion problem in matrix-variate integer-valued time series data arising in sociology, medicine, and related fields, this paper proposes a matrix integer-valued autoregressive model based on the negative binomial thinning operator. By introducing left and right matricial negative binomial thinning operators, the proposed model not only preserves the integer-valued nature of the data but also effectively handles overdispersion. The probabilistic and statistical properties of the proposed model are systematically investigated. Two estimation methods, namely projection estimation and iterative conditional least squares estimation are developed, and the corresponding asymptotic theories are established. Simulation studies provide concrete numerical results to evaluate the finite-sample performance of the estimators. Real data analysis demonstrates that the proposed model outperforms both the continuous matrix autoregressive model and the multivariate integer-valued autoregressive model in fitting matrix-variate integer-valued time series, while also proving effective in accommodating overdispersed count data.
Read the full abstract

Friday 21 August

Statistics Theory

Asymptotically Lossless Exact and Pivotal E-Values

Consider testing a finite composite null $\cP={P_1,\ldots,P_L}$ against a simple alternative $Q$ using $n$ i.i.d. observations. Let $\ell_n$ denote the supremum of the expected log e-value over e-variables that are exact under every $P_i$ and pivotal across the nulls. Zhang, Ramdas, and Wang (2024) showed that $\ell_n$ is superadditive and asked whether $\ell_n/n$ converges to the upper bound $\min_i D(Q|P_i)$ under their joint atomlessness and absolute continuity assumptions. We prove that it does. The proof draws on techniques developed by Zhang, Ramdas, and Wang (2024) and by Farooq, Fritz, Haapasalo, and Tomamichel (2024).
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

A Sparse-Group Pliable Lasso

The sparse-group pliable Lasso (SGPL) extends the pliable Lasso and group pliable Lasso by combining sparse-group regularization with a predictor-level coupling penalty, enabling simultaneous group-level selection, within-group sparsity, and hierarchical structure between main effects and interactions. We propose a blockwise coordinate descent algorithm for fitting the SGPL that exploits the convexity and structure of the objective function, establish convexity and Karush–Kuhn–Tucker optimality conditions, and prove that the algorithm converges to a global minimizer. Simulation studies demonstrate competitive predictive performance and smaller interaction estimation error than the pliable Lasso and group pliable Lasso, albeit with the expected precision–recall trade-off in support recovery. We further illustrate the proposed method using a Parkinson's disease gut microbiome study and an adrenocortical carcinoma (ACC) copy-number dataset from The Cancer Genome Atlas. The Parkinson's application identifies interpretable interactions between microbial abundances and dietary variables, while the ACC application illustrates that the effectiveness of group-structured regularization depends on how well the prespecified grouping reflects the underlying signal structure.
Read the full abstract

All authors Mohammad Javad Davoudabadi, Minh Long Nguyen, Amirhossein Ghatari, Mina Aminghafari, Kerrie Mengersen

Statistics Theory · Cross-listed from stat.ME

$K$-functions for point processes on complex surfaces

The $K$-function is a fundamental summary statistic for assessing clustering or regularity of point processes in two or three dimensional Euclidean space. In practice, however, many planar point patterns arise from projecting locations of objects on a surface in three dimensional space to two dimensional space. For example, when events or objects occur in a landscape their elevation is often ignored. This can lead to erroneous conclusions regarding properties of the point process generating the point pattern. There is not a unique way to extend the classical $K$-function to point patterns on a complex surface. In this paper we propose, explore, and discuss several approaches in terms of their theoretical and computational properties. The best performing approach, coined the surface area $K$-function, can be viewed is an analogue of the classical $K$-function replacing counts of points in Euclidean balls with counts of points in surface geodesic balls. However, an important distinction is that the argument of our surface area $K$-function is area instead of radius of geodesic balls. The performances of the various surface $K$-functions are compared in applications to simulated and real data.
Read the full abstract

All authors Francisco Cuevas-Pacheco, Scott Ward, Ed Cohen, Rasmus Waagepetersen

Statistics Theory · Cross-listed from cs.LG

Loss-Parameterized Fisher Width Along Learning Trajectories

Fisher width measures the Gaussian width of a probe set after deformation by the local Fisher geometry. We study its evolution along learning trajectories and ask when training loss can serve as an effective coordinate for this quantity. We first derive an exact trace–shape factorization and a deterministic stability bound for fixed compact probes. In a population Gaussian-teacher logistic model, the teacher-aligned state is extremal on every loss level below $\log 2$: it has minimal parameter norm and maximizes both Fisher trace and Euclidean-ball Fisher width. We then show that population gradient flow asymptotically selects this branch, with explicit rates for the aligned and orthogonal coordinates. This yields, for $d\geq2$, \[ \frac{w_F(B_2^d;\theta(t))} {\sqrt{L(\theta(t))}} \longrightarrow \frac{\sqrt6}{\pi}\mathbb E[\chi_{d-1}]. \] Controlled full-Fisher experiments support the matched-loss branch and the population predictions. In a nonlinear MLP with a diagonal model-Fisher approximation, GD and SGD remain close at matched loss, whereas Adam follows a substantially displaced branch; the fixed probes tested retain highly similar temporal shapes. These results support a branchwise, rather than universal, loss parametrization of Fisher width.
Read the full abstract