Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Mathematics Statistics Theory

math.ST Watch Statistics Theory (sign in to use)

Tuesday 8 September

Statistics Theory

Nonparametric Inference for Cumulative Residual Mathai–Haubold Entropy of order $\alpha$

In this paper, we study the properties of cumulative residual Mathai–Haubold entropy of order $\alpha$. A dynamic version of this entropy measure is then proposed, and its properties are examined within the framework of reliability modeling. We show that the dynamic cumulative residual Mathai–Haubold entropy of order $\alpha$ uniquely determines the survival function. Characterization results for the exponential and generalized Pareto distributions are derived using the proposed measure. Furthermore, we develop nonparametric estimators for the cumulative residual Mathai–Haubold entropy and its dynamic counterpart of order $\alpha$, based on the kernel estimation of the survival function. The performance of these estimators is evaluated through a Monte Carlo simulation study. Finally, the practical relevance of the proposed dynamic estimator is illustrated using two real data; failure-time data from aircraft windshields and failure-time data from 40 randomly selected mechanical switches.
Read the full abstract

All authors Anija C.R, Smitha, S., Sudheesh, K. Kattumannil

Statistics Theory

A note on a local entropy condition in the Wasserstein space

Entropy conditions are widely used in empirical-process analyses to establish asymptotic properties of M-estimators. In regression problems with responses taking values in a general metric space, a commonly imposed condition requires the entropy integral associated with a shrinking ball centered at the target object to remain uniformly bounded as the ball radius tends to zero. We show that this condition generally fails in the quadratic Wasserstein space of univariate probability distributions supported on a compact interval. Specifically, when the target object is a strictly increasing and absolutely continuous distribution function whose derivative is bounded away from zero and infinity, the corresponding entropy integral diverges as the ball radius tends to zero. The same divergence persists even when the ambient space is restricted to the class of distribution functions satisfying fixed uniform two-sided Lipschitz bounds. These results indicate that different asymptotic analyses are required for the quadratic Wasserstein space.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

Radial Marchenko-Pastur laws: projection characterizations and rigidity

Let \(R_p=\|x_p\|^2/p\Rightarrow \nu\) and \(p/N\to c\in(0,\infty)\). We prove that the radial quadratic-form condition (RQC), requiring energy in deterministic subspaces to follow the parent radius, is equivalent to the following spectral property: for one fixed \(\alpha\in(0,1)\), every deterministic rank-\(\lfloor \alpha p\rfloor\) projected covariance has empirical spectral distribution converging to \(\mu_{c\alpha,\nu}\). The radius law \(\nu\) may have arbitrary tails; no moment assumptions, conditional isotropy, or independence between radius and direction are required. We also show that the radius law can be recovered from the projected spectral limits when their common limit has finite second moment and subspaces of vanishing relative dimension carry vanishing normalized energy, again without moment assumptions on the original vectors. A finite-sample inverse estimate bounds quadratic-form defects by bounded tests of expected projected spectra. The converse argument combines angular symmetrization and fantope rigidity with a determinant comparison between radial column deletion and spectral trimming. A cancellation of the deleted radial factors makes the comparison error depend only on the deleted fraction, rather than on the magnitudes of the deleted radii.
Read the full abstract

Monday 7 September

Statistics Theory · Cross-listed from cs.LG

Sharp Structure-Agnostic Minimax Risk for Partial Linear Models

We characterize the sharp structure-agnostic minimax risk for coefficient estimation in the partial linear model when the outcome and treatment nuisances are learned by two distinct black-box learners, which resolves the open problem in double machine learning posed by Gu (2025). For each nuisance \(q\in\{\mu,\pi\}\), we characterize the available learner by an approximation-error budget \(a_q\) and a stochastic-error budget \(s_q\), with the latter controlled through localized Rademacher complexity. Writing \(\mathcal E_n\) for the minimax mean-squared error, we show that \[\mathcal E_n\asymp1\wedge\left\{\frac1n+\left(a_\mu a_\pi+\min\left\{a_\pi s_\mu+s_\pi^2,\,a_\mu s_\pi+s_\mu^2\right\}\right)^2\right\}.\] The main new ingredient is a novel lower bound for the general two-learner problem. Our proof constructs four finite-mixture testing experiments using orthogonal code functions. Across these experiments, the hidden perturbations are placed outside both learner classes, outside only the treatment learner class, outside only the outcome learner class, or inside both learner classes. These four configurations capture, respectively, the interaction between the two approximation errors, the two asymmetric interactions between one learner's approximation error and the other learner's learning error, and the joint estimation difficulty of learning both nuisances. Combining the four resulting lower bounds yields the displayed rate, which matches the latest upper bound in Gu (2026). Our result shows that standard double machine learning can overstate the intrinsic difficulty of target estimation and provides a target-specific principle for learner selection: approximation error and stochastic complexity must be jointly balanced across the two nuisance learners rather than optimized separately.
Read the full abstract

Statistics Theory

Theoretical Foundations of Ordinal Spherical Multidimensional Scaling

There has been general interest in spherically constrained embeddings as data with an inherently circular or spherical structure arise in a number of applications. While many methods have been proposed on the metric side, little work has been done on the ordinal side in terms of methodology or theory. Here, we focus on the fundamental question of uniqueness of ordinal spherical MDS: Given a realizable setting in which underlying objects lie on the unit sphere and all that is known are their ordinal comparisons of the form `object $i$ is more similar to object $j$ than object $k$', is it possible to uniquely recover the original objects, up to an orthogonal transformation, in the large-sample limit? We answer affirmatively, both in the setting just described, as well as in the settings of spherical external unfolding (aka lateration) and spherical internal unfolding (aka preference mapping) in their ordinal variants.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

Markov Chain CLTs: Resolving Open Problems

Markov chain central limit theorems (CLTs) and their associated variances are very important for implementing Markov chain Monte Carlo algorithms among other applications. Häggström and Rosenthal (2007) presented various results regarding the equality of different formulae for this variance and also posed seven open problems. We resolve all seven in this paper. For stationary, ergodic and reversible chains, we prove that whenever the normalized partial sums satisfy a $\sqrt{n}$-CLT, the variance limit is finite if the function is square-integrable, otherwise undefined. Moreover, failure of the $\sqrt{n}$-CLT forces the normalized partial sums to be non-tight. We also show that Roberts' holding-probability condition precludes a CLT even without assuming reversibility or square-integrability. Finally, we develop a general principle that expresses Fourier coefficients as the autocovariances of an ergodic nonreversible Markov chain.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

Non-asymptotic bounds for the average singular value of a complex Gaussian matrix

Let $G_{d}$ be a $d \times d$ matrix with independent standard complex Gaussian entries, let $\alpha_{\mathbb{C}}(d)$ be the expected average singular value of $G_{d}/\sqrt{d}$, and set $\Delta_d := \alpha_{\mathbb{C}}(d)-\alpha_{\mathbb{C}}(d+1)$. The statistic $\alpha_{\mathbb{C}}(d)$ admits a variational representation as an expected normalized maximum over the unitary group and governs approximation guarantees for the little Grothendieck problem over the unitary group and related unitary registration problems. We obtain a strictly positive lower bound and an upper bound for $\Delta_d$, both valid in every dimension, together with corresponding bounds for $\alpha_{\mathbb{C}}(d)$ around the Marchenko–Pastur limit. These bounds match the sharp leading behavior of complete asymptotic expansions for both quantities, whose coefficients are explicitly computable. The proof combines a three-term recurrence for $Y_d = d^{3/2}\alpha_{\mathbb{C}}(d)$, obtained from its continuous dual Hahn representation, along with singularity analysis of the underlying Laguerre moment generating function.
Read the full abstract

Statistics Theory

Geometric Fluctuations of the $\sin\Theta$ Distance in High-Dimensional Principal Subspace Estimation

We investigate the geometric fluctuations of principal subspaces for high-dimensional covariance matrices through the squared Frobenius $\sin\Theta$ distance between the sample and population eigenspaces associated with the $r_p$ largest eigenvalues. An explicit first-order expansion and a central limit theorem are established for this subspace distance. The theory allows the subspace dimension to diverge subject to $r_p=o(n)$, where $n$ is the sample size. It also permits a diverging spectral norm of the population covariance matrix, population spikes of different orders, and repeated or closely spaced spikes. This sharp characterisation captures features of the subspace estimation error that are not reflected in existing perturbation bounds. As applications, we derive an explicit asymptotic expansion for the expected PCA excess risk and a refined error bound for distributed PCA. In both cases, existing upper bounds can increase with the spiked-block condition number when some leading spikes become stronger, whereas our results show that the corresponding estimation errors need not increase and may instead decrease. Numerical experiments reproduce this contrasting behaviour and demonstrate the finite-sample accuracy of our theoretical findings.
Read the full abstract

Statistics Theory

Statistical Inference for Persistence Diagrams via Landmark Embeddings: Minimax Theory and Finite Approximation

Hilbert-space embeddings enable inference for populations of persistence diagrams, but separation between individual diagrams need not survive population averaging. We develop a framework for inference on population mean embeddings, with particular attention to the additive landmark representations PLACE and PALACE. Treating each diagram as one independent observation, we apply Hilbert-space limit theory to obtain covariance estimators, two-sample tests, and confidence balls under suitable moment conditions, without requiring a lower-distortion bound. For additive embeddings, we identify the population mean as an embedding of the mean counting measure and show that geometric separation of these measures alone cannot guarantee uniform testing power. We then introduce a model with latent template diagrams, missing features, and location perturbations. Under common or feature-specific prevalence conditions, a diagram-level lower-distortion certificate yields explicit lower bounds on population mean separation. These margins provide finite-sample uniform power guarantees, and an additional information-divergence comparison gives matching sample-complexity bounds over restricted scale ranges. Confidence sets yield lower bounds on transport separation of population mean measures and exclusion guarantees for specified structured alternatives. We also quantify how orthogonal truncation changes the certified signal and the approximation allowance needed for confidence sets targeting the full embedding, relating sample size, retained coordinates, and template separation. Simulations examine calibration, power, and coverage, and an analysis of resting-state connectivity from the Autism Brain Imaging Data Exchange illustrates the procedures.
Read the full abstract

All authors Pramita Bagchi, Sushovan Majhi, Atish Mitra, Žiga Virk

Statistics Theory · Cross-listed from stat.ME

Quasi-randomization-based inference for multivariate Mann-Whitney effects under random missingness

Marginal Mann-Whitney effects are widely used across various fields of research, and extensions of this estimand have been developed in many directions in statistical methodology. In this paper, we focus on an extensions for repeated measurements and factorial designs subject to randomly missing data. In a previous work by Rubarth et al. (2022a), asymptotically correct tests were developed under the assumption of deterministic missing indicators. In contrast, the approach in the present paper accounts for the stochastic nature of missing values under realistic mechanisms. Thus, the involved covariance matrix incorporates the true variability of missing data. The combination with a quasi-randomization procedure using random permutations within each data point yields asymptotically exact tests and a generally improved type-I error control. Additionally, the tests control the type-I error for finite sample sizes in the special case of exchangeable sampling distributions. Simulations across a wide range of settings demonstrate the benefits of the proposed method in small samples, also for different missingness mechanisms. A real data analysis about school children learning math illustrates several practical aspects of the tests' application.
Read the full abstract

All authors Dennis Dobler, Jörg-Tobias Kuhn, Lubna Amro, Paavo Sattler

Statistics Theory

Glivenko–Cantelli Theorems for Integrated Volatility Functionals in Pure-Jump Semimartingales with an Application to Cryptocurrency Markets

We develop a two-step procedure for estimating integrated volatility functionals, defined through the occupation measure of the latent spot volatility process, when the asset price is a pure-jump semimartingale. In the first step, block-based estimators formed from absolute powers of high-frequency increments uniformly approximate local averages of powers of volatility. In the second step, these estimates are aggregated into an empirical occupation measure. Since price increments have infinite variance in this setting, arguments based on local Gaussianity are unavailable, and the uniform theory instead rests on maximal inequalities tailored to the stable regime. We establish Glivenko–Cantelli-type uniform consistency over classes of bounded monotone, Lipschitz-in-parameter, and locally Hölder test functions. These results deliver consistent estimation of volatility occupation times and quantiles, together with an argmax-consistency theory for $M$-estimators built on nonparametrically recovered latent processes. We further propose a stability-based rule for selecting the power index of the volatility estimator, which tracks an infeasible ex ante optimal choice closely in Monte Carlo experiments. An application to high-frequency cryptocurrency markets illustrates the framework in a jump-dominated, heavy-tailed environment.
Read the full abstract

All authors Dachuan Chen, Jia Li, Tian Xie, Chengxin Yan

Statistics Theory

Quantiles for multivariate distribution: a short survey of population concepts

Quantiles are among the most fundamental concepts in Probability and Statistics, from descriptive to inferential. However, quantile functions are well-defined and well-understood in the context of one-dimensional probability distributions, where their definition as the inverse of distribution functions—a definition which is intimately related to the canonical ordering of the real line. Starting with dimension $d=2$, such a canonical ordering is no longer available in~${\mathbb R}^d$; nor is it available in nonlinear manifolds such as hyperspheres, tori, or polyspheres (vectors of directional variables). This results in the absence of an obvious and widely accepted definition of quantiles. The need to extend the concept of quantile beyond the classical univariate context nevertheless has sparked a large body of literature, giving rise to a variety of more or less satisfactory quantile concepts; the recent years have been particularly active in this respect. This short review is an attempt to complement and update Serfling's 25-year-old survey (Serfling 2008) by summarizing and unifying some of these concepts in light of recent contributions.
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

Adaptive Conformal Inference Under Delayed Feedback: Coverage Guarantees and a Delay-to-Memory Diagnostic

Adaptive Conformal Inference (ACI) extends conformal prediction to non-exchangeable settings by adjusting the nominal miscoverage level online in response to recent coverage errors. When forecasts are issued with horizon $\tau$, however, the outcome needed to evaluate a prediction is observed only $\tau$ steps later, so these adaptive updates must rely on delayed feedback. We study this setting and ask how the effect of delay depends on the persistence of the residual process. First, we show that the $\tau$-delayed ACI recursion can be decomposed into $\tau$ interleaved ACI-like sequences. This representation yields a finite-sample bound on long-run empirical coverage with explicit dependence on $\tau$. We also derive an approximate marginal coverage bound that relates coverage deviation to changes in the underlying environment across the forecast horizon and to the adaptation rate $\gamma$. We then introduce the delay-to-memory ratio $r=\tau/L$, where $L$ is the time scale over which the temporal signal driving non-exchangeability decays. Simulation results show that the usefulness of this ratio depends on the form of temporal dependence: it strongly organizes performance under AR(1) dependence, is less predictive of overall performance under GARCH(1,1) and Markov switching, but more clearly characterizes when scale normalization remains useful in those settings. Abrupt mean- and variance-shift experiments further show that the preferred adaptation rate depends on the residual dynamics. Overall, the results show that the effect of forecast delay is best understood relative to the time scale over which past residual information remains relevant.
Read the full abstract

Statistics Theory

Tensor network representations of discrete maximum entropy distributions via mean polytopes

We present tensor network representations for discrete maximum entropy distributions under expectation constraints. To this end, we introduce Computation-Activation Networks (CompActNets), a tensor network architecture that subsumes exponential families. By leveraging the geometry of the convex polytope of realizable expectation vectors, we represent any maximum entropy distribution in the same architecture. We exploit the fact that proper faces of this polytope correspond to the boundary closure of exponential families, which restricts the distribution's support. We then derive explicit representations for the support within the CompActNet architecture. The proposed framework suggests tensor network ranks as complexity measures for faces. Finally, a case study on Boolean statistics links the geometry of 0/1-polytopes directly to propositional formulas.
Read the full abstract

Statistics Theory

Optimal rates for aggregation of affine estimators

Aggregation of estimation procedures has found important applications in econometrics, statistics, and machine learning. Classical statistical aggregation theory has mainly focused on a pure aggregation setting, where the candidate estimators are either deterministic or constructed using a held-out sample independent of that used for aggregation. When the candidate estimators and the aggregation weights are estimated from the same dataset without sample splitting, Bellec [Ann. Statist. 46 (2018), 30–59] established the optimal rate for model-selection aggregation of affine estimators. Beyond this regime, however, fundamental questions regarding the optimal aggregation rates and the construction of aggregation rules attaining these rates remain largely unresolved. In this paper, we consider the problem of aggregating a finite collection of affine estimators to learn an optimal convex combination of them. This framework encompasses a rich class of estimators widely used in statistics and machine learning, including least squares estimators, kernel ridge estimators, random feature regression estimators, and among many others. We establish the minimax rate for convex aggregation of affine estimators. In particular, we show that estimating the weights by minimizing a Mallows' $C_p$ criterion attains the optimal rate. We further study the linear aggregation regime with unrestricted weights and establish matching minimax lower and upper bounds over suitable classes of affine estimators.
Read the full abstract

Statistics Theory

Network recovery from aggregated relational data: An impossibility theorem

We prove an impossibility theorem for uniformly consistent recovery of labeled edge probabilities from complete aggregated relational data, even when the population count law identifies every probability. For any known partition into two equal trait groups, we consider an independent-edge logistic network with a balanced rank-one signal and unknown activity effects with bounded total dyadic energy. The signal amplitude is known and fixed at a small positive value. Every node reports its counts to both groups. The minimax mean squared error for the probability matrix remains bounded away from zero as the network grows, whereas it tends to zero under full adjacency on the same parameter class. The lower bound accounts for the dependence across the entire count array through a two-node oracle from which all observed counts can be reconstructed. Conditional binomial smoothing bounds the information about a local sign orientation, and an anchored many-bit construction converts these ambiguities into a nonvanishing normalized matrix loss. A mixed-cumulant identity recovers every edge probability from the population degree law, locating the obstruction in estimation from a single aggregated network rather than in population identification.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

Optimal central limit theorem for bounded random variables in high dimensions

Let $W=n^{-1/2}\sum_{i=1}^n X_i$, where the $X_i$ are independent centered random vectors in ${\mathbb R}^p$ with $|X_{ij}|\le B$ almost surely. Suppose that $\text{Cov}(W)$ has unit diagonal and smallest eigenvalue at least $b^2>0$. We prove that the distance between $W$ and a Gaussian vector with the same covariance, uniformly over axis-aligned rectangles, is at most $C\min\{1,b^{-2}Bn^{-1/2}\log^{3/2}(ep)\}$. For fixed $b$, the dependence on summand size and dimension matches known lower bounds in growing-dimensional regimes. The proof combines a concentration estimate near rectangle boundaries with a carefully chosen Gaussian comparison.
Read the full abstract

Statistics Theory

The joint density of top k sample eigenvectors and principal subspace inference

In this paper, the exact joint density of the top $k$ sample eigenvectors is derived for any $p\times p$ population covariance matrix $\varSigma$, sample size $n > p -1$, and $1 \le k \le p$. Using symmetric functions such as zonal polynomials and a dual summation identity by I. G. Macdonald, the result is expressed as a series in terms of determinants of differential operators. A matrix Kummer transformation converts the resulting local alternating expansion into a globally absolutely convergent series over the entire positive definite matrix space. For $k=2$, the ordered eigenvalue integrals reduce to the Gauss hypergeometric function ${}_2F_1$ with a simple closed form when $n=p+1$. Previously, explicit formulas were only available for a scalar matrix $\varSigma = \sigma^2 I_p$ or a general matrix $\varSigma$ with $p=2$ or $k=1$, obtained by T. W. Anderson and T. Sugiyama, respectively. The frame law also induces an exact Grassmann density that provides a finite-sample benchmark for principal subspace inference.
Read the full abstract

Statistics Theory

Almost Sharp Equivalence between Approximate Message Passing and Low-Degree Polynomials

We prove a sharp lower bound for growing-degree polynomial estimation in the Gaussian planted submatrix model. The observation is $$ \boldsymbol{Y}= \frac{\lambda}{\sqrt{n}} \boldsymbol{\theta} \boldsymbol{\theta}^{\top}+\boldsymbol{W}, $$ where the coordinates of $\boldsymbol{\theta}$ are independent $\mathsf{Ber}(\rho)$ variables and $\boldsymbol{W}$ is symmetric with independent standard Gaussian upper-triangular entries. For every fixed $\lambda>0$ and $\rho\in(0,1)$, we give an explicit finite-dimensional bound implying that every sequence of polynomial estimators of degree $D(n)=o(n^{1/60})$ has normalized mean-square error with limit inferior at least $\rho-q_{\mathsf{amp}}/\lambda$, the limiting error of Bayes approximate message passing (AMP). This extends the constant-degree result of Montanari and Wein~\cite{montanari2025equivalence} for the Bernoulli prior. Combined with their polynomial approximation of fixed-iteration AMP, the bound identifies the exact limiting low-degree MMSE whenever $D(n)\to\infty$ within this range. It therefore resolves the Bernoulli rank-one case of the growing-degree AMP-equivalence question discussed in~\cite{wein2025computational, maleki2026high}. The proof constructs a low-degree certificate using conditional joint cumulants of the signal coordinates and their products. Specifically, we condition on an auxiliary Gaussian channel $\boldsymbol{R}$ calibrated to the AMP fixed point. This retains signal dependence that is lost in unconditional cumulant bounds and produces the cancellations needed for quantitative control as the degree grows. Most of the arguments in this paper were generated using GPT-6 Astra.
Read the full abstract

Statistics Theory · Cross-listed from cs.IT

Equivalence of Fixed-Rank and Rank-One Even-Order Symmetric Tensor Factorization

In the recent work of Barbier, Ko, and the second present author on sublinear-rank symmetric matrix factorization [Math. Stat. Learn. 9 (2026), 1-68], a key result is that, in the Bayes-optimal setting, the large-size limit of the free entropy of the finite-rank spiked Wigner model is the same as in the rank-one case when the signal has centered i.i.d. entries. In this paper, we show that this rank-one equivalence result extends to the case of finite-rank, even-order, symmetric tensor factorization. Moreover, we give a natural reformulation of a hypothesis that was stated in the aforementioned work to be necessary for this result. As in the matrix case, we use information-theoretic identities and replica symmetry to reduce a known multi-dimensional variational formula for the limiting free entropy to its one-dimensional analog. The novelty stems from the fact that said formula involves a replica symmetric potential containing Hadamard (entrywise) powers, rather than squares, of the matrix-valued variational parameter, so the eigenvalue-based approach used in the matrix case must be adjusted.
Read the full abstract

Statistics Theory

Smoothed Picard Hamiltonian Monte Carlo

We develop a new low-accuracy sampler, called smoothed Picard Hamiltonian Monte Carlo, which combines Gaussian smoothing, Picard iteration, and higher-order discretization. For a log-concave target $\pi \propto \exp(-V)$ in dimension $d$ satisfying $0 \prec \alpha I \preceq \nabla^2 V \preceq \beta I$, with condition number $\kappa := \beta/\alpha$, smoothed Picard HMC returns a sample with $\sqrt \alpha\,W_2(\cdot,\pi) \le \varepsilon$ using $\widetilde O(\kappa^2 + \kappa^{7/6} d^{1/6}/\varepsilon^{1/3})$ gradient queries. We also prove stronger $W_q$ bounds, and then develop an algorithmic framework, the recursive warm start generator, to upgrade these $W_q$ bounds to stronger divergence guarantees. This produces a warm start for the proximal bouncy particle sampler, introduced in a companion work, leading to a high-accuracy log-concave sampler with complexity $\widetilde O((\kappa^{7/6} d^{1/6} + \kappa^{1/2} d^{1/4})\mathrm{polylog}(1/\varepsilon))$.
Read the full abstract

All authors Fan Chen, Sinho Chewi, Jianfeng Lu, Matthew S Zhang

Statistics Theory

Accelerated High-Accuracy Sampling from a Warm Start via the Proximal Bouncy Particle Sampler

We study the problem of sampling from $\mu(\mathrm{d}x)\propto e^{-V(x)}\,\mathrm{d}x$ on $\mathbb{R}^d$, where $V$ is $\alpha$-strongly convex and $\beta$-smooth, and write $\kappa:=\beta/\alpha$. We design and analyze the Proximal Bouncy Particle Sampler (Proximal BPS), a new sampler that combines ideas from the proximal sampler and the bouncy particle sampler. From a warm start initialization with $ O(1) $ Rényi divergence w.r.t. $\mu$, Proximal BPS returns a sample whose law is $\varepsilon$-close to $\mu$ in total variation distance using $\widetilde O(\sqrt\kappa\,d^{1/4} \,\mathrm{polylog}(1/\varepsilon))$ gradient queries in expectation.
Read the full abstract

All authors Fan Chen, Sinho Chewi, Jianfeng Lu, Matthew S Zhang

Sunday 6 September

Statistics Theory · Cross-listed from stat.ME

Cluster-Based Dimensionality Reduction by Nonparametric Distributional Screening

We consider dimensionality reduction for high-dimensional observations accompanied by a supplied partition into two or more clusters. The objective is not to construct a low-rank projection, but to retain an interpretable subset of the original coordinates that preserves the distributional information distinguishing the clusters. For each coordinate, the proposed procedure compares the cluster-specific empirical distribution functions through a several-sample Kolmogorov-Smirnov separation statistic. We formalize the resulting marginal cluster support and establish simultaneous finite-sample concentration over all coordinates, explicit bounds for false inclusions and omissions, and exact support recovery when the minimum distributional separation dominates the high-dimensional stochastic error. We also quantify the dimension inflation induced by using an unadjusted testing level and give a familywise-error-controlled version. Under a conditional sufficiency condition, sure screening preserves the full-data posterior cluster probabilities, mutual information, and Bayes risk; an additional result characterizes robustness to imperfectly estimated cluster labels. The procedure is invariant to strictly increasing coordinate transformations and can retain low-variance cluster signals that principal components may discard. We further develop average dual information, a criterion combining partition agreement after transformation with structural coverage of cluster-relevant coordinates, and derive its basic properties and consistency. Simulations illustrate the theory, the interpretability of the selected coordinates, and the distinction between cluster-directed screening and variance-directed projection.
Read the full abstract

All authors Sanoja Jha, Rishikesh Muralimohan, Praveen Athauda Arachchi, Abhishek Bhattacharjee

Statistics Theory · Cross-listed from cond-mat.stat-mech

Improved mean squared displacement analysis for anomalous single particle trajectories

The mean squared displacement (MSD) is a cornerstone in the analysis of diffusion processes in complex media. When the system is heterogeneous and, in particular, when single-particle trajectories are short, it is essential to extract maximal information from each measured trajectory. This is typically done by time-averaging squared increments and examining the scaling of the time-averaged MSD in log-log space. However, classical regression methods perform poorly in this setting because time-averaging introduces correlations aggravated by those inherent to anomalous diffusion. We tackle these limitations by applying a generalized least squares framework, which substantially reduces variance and bias in diffusion parameter estimates, especially for short (around 100 points) and ultra-short (around 10 points) trajectories. The method is fully automated and requires no supervision. Furthermore, it enables prediction of estimation error probability density, which is asymptotically Gaussian, for both classical and enhanced approaches. Leveraging this prediction, we introduce a specialized deconvolution algorithm that reconstructs the underlying particle ensemble structure from experimental data.
Read the full abstract

All authors Jakub Ślęzak, Joanna Janczura, Diego Krapf, Ralf Metzler

Statistics Theory

Phase transitions and approximations of mean squared error for state-space models with fractional differencing

We study trend estimation in state-space models in which the trend has a fractional stochastic difference of order $d>0$ and the observation errors form a short-range-dependent stationary process. Using finite-sequence fractional summation and differencing operators, we analyze the penalized least-squares estimator obtained by shrinking the fractional differences of the trend. We derive asymptotic mean squared error (MSE) approximations for all $d>0$ and identify a sharp phase transition at $d=1/2$. When $d>1/2$, the estimator is consistent and its optimally balanced MSE has order $n^{-(2d-1)/(2d)}$. At the boundary $d=1/2$, we obtain a refined finite-sample approximation and show that the MSE decreases at the slower order $\log\log n/\log n$. When $0<d<1/2$, the MSE converges to an explicit positive limit, so consistent recovery of the trend is impossible under the considered scaling. We also describe a practical criterion for choosing the penalty parameter and differencing order, and numerical experiments illustrate the MSE approximations and the behavior of the selection procedure.
Read the full abstract

Statistics Theory

Global minimax risk and acquisition laws for heterogeneous information fusion

We prove an all-allocation global target-risk theorem for independent Gaussian sources that share a scalar nuisance and an unknown contact coordinate. For a fixed immersed nuisance curve with finitely many multiple fibres and pairwise nonparallel branch tangents, squared target risk is comparable to a primary estimation floor plus a target-gap-weighted Gaussian discrimination profile. Independent localization, finite branch selection and target-class refitting give an estimator attaining this comparison across every nonnegative integer allocation. A full-box specialization has positive-definite primary Fisher information everywhere and globally identifies its target, yet models with identical derivatives of every order along a critical hypersurface have different polynomial risk exponents. We construct a finite certified estimator and quantify the numerical accuracy needed to preserve rare branch decisions and acquisition windows. A polynomial tangency family both demonstrates the geometric boundary and quantifies its repair: known auxiliary gain, source contact order and target vanishing order determine a risk law uniform through zero gain. For a shared spherical direction, a separately proved composite-testing result transfers through an unknown contact coordinate using only counted observations. These theorems distinguish the resources needed for local estimation, discrimination between parameter regions and nuisance alignment, and yield acquisition thresholds under explicit scalar-observation costs.
Read the full abstract

Statistics Theory · Cross-listed from stat.ME

A Likelihood-Ratio Test for Verifying Weak Stochastic Transitivity

This paper proposes and studies a likelihood-ratio test of the null hypothesis that weak stochastic transitivity (WST) does not hold. This is the reverse of the formulation commonly used in the literature, where WST is set as the null hypothesis. We show that, even when the number of items grows with the number of comparisons per pair, the uniform size converges to the nominal level, with the critical value determined by a chi-bar-square distribution. We further establish that the Type II error converges uniformly to zero under a sufficient signal-strength condition. Simulation studies demonstrate good finite-sample Type I error control and show that power increases with the sample size and signal strength.
Read the full abstract

Statistics Theory

Bergsma–Dassios sign covariance with ties: a nonnegative decomposition and sharp bounds

We construct tie-symmetrised extensions $\widetilde D$ and $\widetilde R$ of the unnormalised Hoeffding and Blum–Kiefer–Rosenblatt functionals $D$ and $R$ and prove, for every real-valued bivariate law, the exact nonnegative decomposition $ \tau^{*}(X,Y)=12\widetilde D(X,Y)+24\widetilde R(X,Y), \qquad \widetilde D,\widetilde R\geq0, \qquad \widetilde R(X,Y)=0\quad\Longleftrightarrow\quad X\perp\!\!\!\perp Y. $ For atomless margins the components reduce to $D$ and $R$, whereas in the presence of ties the corresponding classical identity $\tau^{*}=12D+24R$ can fail. The components arise from the Hoeffding projections of a symmetrised pair kernel. For the four strict/non-strict boundary versions $D_{\epsilon,\eta}$ and $R_{\epsilon,\eta}$, we establish the sharp comparisons $ 9\widetilde D\geq\sum_{\epsilon,\eta}D_{\epsilon,\eta}, \qquad 9\widetilde R\geq\sum_{\epsilon,\eta}R_{\epsilon,\eta}. $ Consequently, $\tau^{*}\geq0$ for every bivariate law, and the Bergsma–Dassios conjecture is settled: $\tau^{*}=0$ if and only if $X\perp\!\!\!\perp Y$. The boundary comparisons also yield the universal bound $\tau^{*}\geq(8/3)R$. Finite-table arguments, including an exact sum-of-squares certificate, establish the boundary inequalities, and weak-order quantisation transfers them to arbitrary laws. The factor $9$ in each comparison and the coefficient $24$ in $\tau^{*}\geq24\widetilde R$ are sharp. The usual permutation test is consistent against every fixed dependent alternative, without assumptions on the margins.
Read the full abstract

Statistics Theory

Sharp Causal Bounds for Dynamic Treatment Regimes

We study dynamic treatment regimes under contemporaneous confounding: at each stage, an unmeasured factor may affect both treatment and the next observed state, but has no further direct effect on later stages. From the observational distribution, the causal graph, and specified structural restrictions on possible next states, we construct at each state–treatment pair the set of transition probabilities compatible with this information. These local sets require no sensitivity parameter and can be combined by backward induction to obtain lower and upper bounds on the expected outcome under any given treatment regime. Our main result shows that these bounds are sharp: their endpoints are exactly the smallest and largest expected outcomes generated by causal models compatible with the same observational distribution, causal graph, and structural restrictions. The same backward-induction method gives a maximin rule for choosing treatments by maximizing the worst-case expected outcome. Thus, for this class of models with contemporaneous confounding, the framework generalizes the classical g-formula of \citet{Robins1986} for evaluating treatment regimes and the Q-learning framework of \citet{Murphy2003} for selecting them. When the local transition probabilities are identified, the two recursions reduce to these classical methods.
Read the full abstract

Saturday 5 September

Statistics Theory · Cross-listed from quant-ph

Algorithmic Randomness and Physical Typicality

Appeals to typicality are common in physics, but it is often unclear what it means for a physical state to be typical relative to a probability measure, and correspondingly unclear what a law that appeals to typicality asserts. Here we consider how one might characterize physical typicality using ideas from the theory of algorithmic randomness. As a concrete example, we show how taking a physical state to be typical relative to a computable measure when it is Martin-Löf random allows one to formulate the distribution postulate in Bohmian mechanics as a statistical constraining law of the theory. Using a toy model, we show how this constraint guarantees the standard Born statistics for computable experimental protocols. Algorithmic Bohmian mechanics (aBM) thus illustrates how algorithmic randomness may be used to provide precise content to a statistical law.
Read the full abstract