Peer Revenue Impact factor: volatile

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Tuesday 11 August

Statistical Methodology

Finite-sample correction for the covariate-adjusted log-rank test

The covariate-adjusted log-rank test is a novel method for covariate adjustment in randomized trials with time-to-event endpoints, offering guaranteed efficiency gains compared to the standard log-rank test. However, it has been noted that, in small samples, this method may lead to type I error rate inflation. This issue is particularly pronounced in trials with imbalanced allocation and settings where the number of adjustment covariates is large relative to the sample size. We propose a finite-sample correction for the denominator of the covariate-adjusted log-rank test statistic that accounts for the loss of the residual degrees of freedom as well as the uncertainty in the unknown regression coefficients. In simulations, we show that applying this correction leads to a substantial reduction in the type I error rate inflation across multiple scenarios.
Read the full abstract

Statistical Methodology

Endogeneity-Aware Cognitive Diagnostic Model for Multidomain Ordinal Assessments

Multidomain assessment batteries generate ordinal item responses that are often summarized through latent attribute profiles. Conventional cognitive diagnostic models (CDMs) provide interpretable measurement models for such profiles, but they typically do not represent directed dependence among latent attributes from distinct domains. We propose an endogeneity-aware cognitive diagnostic model (EACDM) for multivariate ordinal assessments. The model combines a block-structured diagnostic measurement component, in which item groups are linked to domain-specific binary attributes through a block-diagonal Q-matrix, with a logistic structural component, in which one attribute block is regressed on another block and subject-level covariates while accounting for latent classification uncertainty. This formulation yields a parsimonious framework for studying endogenous relationships among diagnostic attributes without collapsing domain-specific measurement structure. We establish identifiability conditions for the Q-matrix, effective loadings, latent-profile probabilities, and structural coefficients, and develop a Markov chain Monte Carlo algorithm for joint estimation of the measurement and structural components. Simulation studies demonstrate accurate recovery of item parameters, latent structures, and structural coefficients for the proposed EACDM, whereas conventional CDMs can fail to recover the ground truth when endogeneity is present. We apply the proposed method to Parkinson's disease data to examine how non-motor latent traits relate to motor impairment profiles.
Read the full abstract

Machine Learning (Statistics)

Self-Normalized Inference for Constant-Stepsize Temporal-Difference Learning under Markovian Sampling

Constant-stepsize temporal-difference (TD) learning is attractive for policy evaluation, but inference from a single Markov trajectory must account for serial dependence and a stepsize-dependent stationary target. For fixed-stepsize linear TD, we establish a functional central limit theorem whose covariance retains the multiplicative component induced by the random TD matrix and the stationary iterate error. We then derive a joint functional limit for parallel Richardson–Romberg (RR) recursions driven by the same trajectory. A Brownian-bridge self-normalizer yields asymptotically pivotal confidence regions for prespecified state-value contrasts without estimating the long-run covariance or selecting a bandwidth or batch length. For such a contrast, the procedure admits a one-pass implementation whose memory does not grow with the trajectory length. At a fixed stepsize, the inferential center is the RR stationary target. We also study horizon-indexed designs in which the stepsize remains constant within each run and decreases across longer horizons. Under an explicit RR-dependent rate window, the residual RR target shift, multiplicative remainder, and initialization effect are negligible at the root-$n$ scale, yielding inference for the projected Bellman solution. Experiments on FrozenLake and Garnet illustrate stationary-target coverage, RR target correction, and the finite-sample behavior of the horizon-indexed design.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Optimistic Rates for Multiclass PAC Learning

Worst-case multiclass bounds do not become smaller when the best classifier is already nearly correct: what is missing is an optimistic rate, a guarantee whose fluctuation scales with the oracle risk itself. For a class of Natarajan dimension $d_N$ and Daniely-Shalev-Shwartz dimension $d_{DS}$, the optimal excess risk is known at the two endpoints ($d_{DS}/n$ realizable, $\sqrt{d_N/n}+d_{DS}/n$ agnostic [HMZ24, CEH+26, Pab26]) and open in between. We close the gap: at every fixed oracle risk $L^\star$, the optimal excess risk is $\widetilde{\Theta}(\sqrt{L^\star d_N/n}+d_{DS}/n)$, uniformly in the alphabet size, attained by a learner that knows neither $L^\star$ nor the confidence level. The upper bound composes the cover-menu-compression architecture of [CEH+26], at the realizable rate of [Pab26], with a new comparator-facing relative compression theorem: a size-$k$ compression rule that empirically dominates a comparator $h$ has population risk at most $L(h)+O(\sqrt{L(h)\Gamma}+\Gamma)$ with $\Gamma=(k\log n+\log(1/\delta))/n$, without stability; this transfers the comparison principle of the sharp binary theory [MQZ26] while discarding its Boolean-cube geometry, which does not lift to multiclass labels. The lower bound forces both terms using one class and one distribution at every fixed $L^\star$, by a pair-Assouad scheme calibrated to $L^\star$ and a fiber argument on the pseudo-cubes underlying the Natarajan-versus-DS separation of [BCD+22]. Both theorems extend to list learning: against the best $r$-tuple of hypotheses, the same architecture and the same two engines yield an optimistic rate and a lower bound of the same shape, forcing the fluctuation term that [Pab26] expected to be necessary against list comparators, and removing the factor $r$ from the known realizable list lower bound.
Read the full abstract

All authors Xiaoyu Li, Andi Han, Jiaojiao Jiang, Junbin Gao

Machine Learning (Statistics)

Spectral Embeddings of Degree-$\alpha$ Laplacians in Random Dot Product Graphs

Spectral clustering methods for network data are commonly based on a few matrix representations, such as the adjacency matrix and the symmetric Laplacian. We study a continuum of degree-normalized spectral embeddings that includes these commonly used choices as special cases. Under a random dot product graph model, we establish a row-wise central limit theorem for this family of embeddings. The result provides an explicit description of how degree normalization affects both population geometry and the local uncertainty of embedded nodes. We use the limiting distributions to compare different normalizations in two-community stochastic block models through a projected-Gaussian Bayes-error diagnostic. These comparisons show that no single normalization is uniformly preferred. Instead, the favored normalization depends on network density, community imbalance, and block-probability structure. Typically, stronger normalization is favored in lower-density or more imbalanced settings. These results provide a unified distributional understanding of when and why alternative normalizations may improve spectral clustering.
Read the full abstract

Statistical Methodology

Optimal Stratification of a Sampling Frame: A Comparative Study of Classical, Quantum, and Quantum-Inspired Approaches

Optimal stratification aggregates strata into a small number of final strata to minimise total sample size required to meet target precision constraints. This combinatorial objective, reformulated as a within-cluster dispersion surrogate, can be expressed as a quadratic unconstrained binary optimisation (QUBO) problem. This paper reports a comparative case study of four solvers for that surrogate, run under heterogeneous free-tier constraints on an identical twenty-stratum frame from the swissmunicipalities dataset: a D-Wave-formulated QUBO, solved here by simulated annealing (QPU access unavailable); a gate-based quantum processor (IBM Quantum, running QAOA); a photonic entropy-quantum-computing device (QCi Dirac-3); and a classical GPU-based Ising machine as control (Fixstars Amplify AE). Solvers differ in hardware, computational budget, iteration count, and, for IBM, the encoding itself, so the comparison is a case study rather than a controlled experiment. Against a genetic-algorithm benchmark of 129 sample units, the best-known QUBO solution (objective 501.0) maps, after Bethel-Chromy evaluation, to a sample size of 160, a gap of roughly 24%, tied to the best solution found rather than a certified optimum. IBM and Dirac-3 fall further short, at 262 and 286; the IBM result is consistent with SWAP-routing degradation on a sparse qubit lattice, though not isolated from other causes, while Dirac-3, despite genuine optimisation, settles short of what a free GPU reaches in seconds. The paper concludes that a classical genetic algorithm remains, at the scale tested, the method of choice for optimal stratification, reflecting (a) limits of the dispersion surrogate versus the true Bethel-Chromy objective, not encodable as a low-degree polynomial, and (b) solver- and platform-specific limits in the IBM and Dirac-3 runs. These findings do not establish global optimality for either result.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Path Integral Value Matching for Linear Quadratic Stochastic Optimal Control

Linear Quadratic Stochastic Optimal Control (LQ-SOC) establishes a fundamental framework for steering noisy dynamical systems and has recently gained renewed interest in the machine learning community. However, current state-of-the-art policy-based methods suffer from prohibitive computational costs and instability due to their heavy reliance on full-trajectory simulation. To overcome these limitations, we propose a paradigm shift toward a value-based approach by revisiting Path Integral Control (PIC). Although standard PIC suffers from the same high-variance bottleneck as policy-based methods, we discover that by truncating and marginalizing the original path integral formulation, we can derive a temporal recursive form of the value function. Building upon this theoretical foundation, we propose the Path Integral Value Matching (PI-VM) algorithm. Specifically, we employ temporal-difference learning to approximate the recursive value dynamics, and further integrate the Girsanov theorem with experience replay to enable off-policy training. We benchmark PI-VM against SOTA policy-based methods across various SOC benchmarks and sampling tasks. Empirical results demonstrate that PI-VM matches SOTA precision with an order-of-magnitude efficiency gain in low-dimensional settings, while effectively mitigating mode collapse in high-dimensional scenarios. Consequently, PI-VM offers a scalable solution for solving complex SOC problems.
Read the full abstract

All authors Bangyan Liao, Chenglei Yu, Yuchen Yang, Chuanrui Wang, Zhisheng Song, Peidong Liu, Tailin Wu

Statistics Applications · Cross-listed from cs.SI

A Graph Approach to the Academic Publishing Network: A Heterogeneous Model and Structural Screening over OpenAlex Open Data

The academic publishing ecosystem is a vast, heterogeneous network of works, authors, institutions, journals, and topics. Traditional scientometrics reduces it to isolated tabular indicators (h-index, Impact Factor) that ignore topological context and are not designed to capture coordinated illegitimate practices. Building on our companion review, which proposed graph analysis of publishing integrity, this paper implements that approach. We define a heterogeneous multivariate graph model over OpenAlex open data (seven node types, seven edge types) and a methodology based on projections (citation and co-authorship networks), interpretable structural metrics, community detection, and three screening detectors of anomalous publishing patterns. We deliberately avoid binary classification: detectors return ranked candidates with explicit structural evidence for human assessment. On the institutional corpus of VSB - Technical University of Ostrava (2020-2025) with its one-hop citation neighbourhood, community detection recovers real research groups, centralities identify cross-disciplinary bridges, and the screenings flag dense co-authorship cliques, locally closed citation loops, and thematically isolated venues. On a second, venue-centric corpus with external ground truth (journals delisted by Scopus and DOAJ) and size-matched controls, a naive case-control design yields seemingly strong but spurious detectors (a prominence confound), whereas after matching the only robust signal is the breadth of disciplinary scope (AUC 0.70); an open graph-based prestige measure (PageRank over the journal citation network) tracks a JIF proxy while being an order of magnitude more resistant to citation gaming than count-based indicators. We release the method as the open-source library apnet with a reproducible CLI workflow and a web interface; the analysis runs on commodity hardware in minutes.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.AI

Rule of Thumb: Explaining Artificial Intelligence Systems using Partial Information

Explainable Artificial Intelligence (XAI) seeks to explain how an Artificial Intelligence (AI) system arrived at a particular decision. We propose ”Rule of Thumb” (RoT) explanations, a new approach to XAI based upon a novel formulation that identifies the most relevant features for predicting the behaviour of an AI system, for a particular datapoint. We show how RoT is well-suited to enable XAI in: (a) zero-shot classification using large language models (LLMs), (b) auditing of opaque AI systems without model access, and (c) the use of AI in scientific discovery. Additionally, RoT meets specific requirements from leading AI regulations, provides a familiar interface and visualisations for XAI practitioners, is model-agnostic, and is substantially faster than alternatives. Code available at: https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information
Read the full abstract

All authors Kaivalya Rawal, Daria Onitiu, Brent Mittelstadt, Sandra Wachter, Chris Russell

Statistical Methodology

When Is a General Factor Distinguishable? Non-Proportionality, Stable Structure, and the Bifactor Decision

Whether an added general dimension is necessary beyond correlated first-order factors is a property of population covariance, not an estimator. A bifactor structure is covariance-equivalent to correlated factors when general and group loadings are proportional within every cluster. When every cluster violates proportionality, at least three indicators per cluster and mild conditions rule out exact reproduction by a K-factor model with diagonal uniquenesses. Mixed configurations remain only partly characterized, motivating graded distinguishability. We develop a two-step procedure that delivers a stable first-order structure only when it persists across direct wider-count comparisons, then compares oblique and bifactor representations conditionally on delivery. A preliminary simulation study provides tentative design evidence. A major study develops three non-nested persistence profiles, designates .80/r2 as the practical default, and evaluates the frozen family on held-out replications from the same generators. Within the studied scenarios, anchor-only persistence outperforms anchor-zero, minimum congruence modestly outperforms maximum RMSD, and depth four adds no precision at the fixed .70 cutoff. Held-out results also expose stable-yet-wrong delivery under a weak core and show that the conditional comparison is highly accurate after correct recovery. Four empirical applications illustrate agreement, adjacent-count uncertainty, and non-delivery. Persistence, not count alone, carries the structural decision.
Read the full abstract

Statistical Methodology

Win-Ratio Regression for Prioritized Composite Outcomes in Observational Studies: Doubly Robust and Efficient Estimation with Future-Score Correction

Prioritized pairwise outcomes are useful when clinical events follow a natural hierarchy, but censoring before pair resolution complicates estimation. We develop a win-ratio regression framework for this setting by defining a complete-data target over follow-up and deriving an estimating equation for the observed data. The central idea is future-score correction (FC): when censoring prevents later pairwise comparisons from being observed, the method replaces the remaining score with its conditional expectation given the observed history. This correction recovers pairwise information beyond that provided by inverse censoring weights alone. Additionally, we incorporate treatment weighting and baseline outcome augmentation to address baseline confounding. Together, these components yield double robustness for treatment assignment and censoring. Inference is obtained from U-statistic theory. Under standard regularity conditions, the AIPW-FC estimator is asymptotically normal and efficient when all nuisance functions are correctly specified. Simulations with 30%, 50%, and 65% censoring show that efficiency gains from future-score correction increase with the censoring rate, with relative efficiency reaching 1.50 under 65% censoring and near-nominal coverage for AIPW-FC. An application to OneFlorida electronic health record data illustrates the method for a composite outcome that prioritizes death over hospitalization.
Read the full abstract

All authors Zhuochao Huang, Lucy Shao, Yi Guo, Ying Lu, Xin M. Tu, Changyong Feng, Tuo Lin

Statistical Methodology

Survey-robust uncertainty quantification in generalised additive models for location, scale, and shape

Conventional mean-only regression models are often too restrictive for the analysis of complex survey data, where interest frequently extends beyond the conditional mean to other aspects of the response distribution. Generalised additive models for location, scale and shape (GAMLSS) provide a flexible framework by allowing all distributional parameters to depend on covariates. However, existing model-based and model-robust standard errors fail to account for complex survey designs and can substantially underestimate the variability of regression parameter estimates. We propose a linearisation-based sandwich variance estimator that incorporates the survey design while remaining computationally efficient. Using a simulation study based on synthetic survey data, we demonstrate that the proposed estimator provides accurate standard error estimates and reliable confidence interval coverage for all distributional parameters, while offering a computationally efficient alternative to replication-based methods. We further illustrate the practical utility of the approach through a re-analysis of data from the 2019/20 Rwanda Demographic and Health Survey (DHS).
Read the full abstract

Statistics Applications

Evaluating the Role of Blockage Deficit Models in Robust Wind Farm Design

Uncertainty in wind farm layout optimization regarding model choice and the impact of global blockage always exists. This paper evaluates these interactions by extending a multi-objective approach that maximizes mean Annual Energy Production (AEP) from a model ensemble while minimizing their variance. By incorporating the Self Similar blockage model into an ensemble of five wake models, we assess the impact of blockage physics on layout robustness. Results from a linear mixed-effects model indicate that including blockage leads to a non-significant average AEP reduction of 0.0624 GWh ($p$-value = 0.323). Conversely, it caused a highly statistically significant increase in uncertainty, with model disagreement rising by 0.177 GWh ($p$-value < 0.001). Additionally, computational runtime increased nearly nine times. These findings highlight an accuracy vs. certainty paradox, where theoretically necessary physics can compromise model consensus if implemented without re-calibration. Ultimately, this work suggests that simple blockage couplings act as a diagnostic for model incompatibility, emphasizing the necessity of careful model tuning for wind farm design.
Read the full abstract

2608.10630 · 11 Aug · This is an author-formatted, editorially revised version of the paper published in Journal of Physics: Conference Series, Vol. 3224, Wind farms and wakes (2026), https://doi.org/10.1088/1742-6596/3224/3/032072. The results and conclusions are unchanged. Changes include LaTeX reformatting and minor editorial/structural revisions. Please consider citing this refined preprint

Statistics Applications

Probabilistic forecasting via post-processing prediction errors: In- or out-of-sample?

Many forecasting systems produce point forecasts even when decisions require information about uncertainty. We investigate whether post-processing methods can systematically improve upon traditional Gaussian predictive distributions constructed from in-sample residuals. We propose a hybrid framework that combines forecast error post-processing with model-specific scaling of forecast uncertainty across horizons. For a comprehensive evaluation, we apply historical simulation, conformal prediction, quantile regression, and GARCH-based post-processing to point forecasts generated by Theta, exponential smoothing, and ARIMA models. Using 14,407 monthly series from the M4 competition and forecast horizons of 1 to 12 months, we evaluate performance using the continuous ranked probability score and rank-based statistical comparisons. Averaged across horizons, all post-processing variants improve upon the benchmark predictive distributions, with gains of up to 4.6%. In-sample calibration outperforms its out-of-sample counterpart in 11 of the 12 model-method combinations, although the preferred post-processing method depends on the base model and forecast horizon. The advantage of in-sample calibration generally increases at longer horizons. Our results show that organisations can extend existing point-forecasting systems to provide useful uncertainty quantification without computationally intensive repeated model re-estimation.
Read the full abstract

All authors Piotr Zaborowski, Arkadiusz Lipiecki, Fotios Petropoulos, Rafał Weron

Statistical Methodology

Duality and Error for Predictively Oriented Inference

Predictively oriented (PrO) inference quantifies uncertainty by selecting a distribution over model parameters to optimize a scoring rule applied to the induced predictive distribution, together with a divergence penalty from a reference distribution. By applying the scoring rule after averaging model densities, PrO inference targets predictive performance, accounting for model misspecification. We focus on the logarithmic score with general $\phi$-divergence regularization. Our contributions are twofold. First, we derive a finite-dimensional dual formulation of PrO inference. For $n$ observations, the dual problem has $n+1$ variables. We establish zero-duality-gap criteria and optimality conditions that relate the primal and dual solutions. When primal and dual solutions exist, these conditions yield a semi-analytical representation of the PrO posterior and certificates for assessing the accuracy of numerical solutions. For Kullback–Leibler regularization, the posterior has an exponential form. Second, we derive a finite-sample excess predictive-risk bound for approximate PrO posteriors that separates sampling fluctuation, approximation under a divergence budget, regularization, and numerical optimization error. The result applies even when the benchmark predictive risk is not attained by any probability distribution over the model parameters having finite divergence from the reference distribution. We use an exactly solvable categorical example to show that predictive-risk convergence can imply convergence to a unique predictive distribution even though the parameter distributions have no weak limit on the original parameter space. The example also shows that different $\phi$-divergences can require different regularization schedules. We conclude with a misspecified Gaussian location-mixture example that illustrates the dual computation, primal recovery, and numerical accuracy checks.
Read the full abstract

All authors Aurya Javeed, Drew P. Kouri, Teresa Portone, Rebekah White

Machine Learning (Statistics)

Iterative Erasure Count Is Not an Affine-Invariant Concept Dimension

How many directions does a neural representation use to encode a concept? A common answer repeatedly erases probe directions and reports the stopping count or cumulative removed rank. We show that both quantities can change under an information-preserving invertible reparameterization, so neither is intrinsically a concept dimension. We distinguish model-defined population quantities (generating dimension, sufficient linear dimension, and minimum guarding rank) from procedure-defined quantities such as stopping count and cumulative edit rank. In a population Gaussian construction, an invertible shear preserves the prediction problem and all three quantities, yet changes the cumulative Euclidean erasure count from one to two. The separation holds for Moore–Penrose ordinary least squares and every finite nonnegative ridge weight. For a two-output full-QR procedure matching our motivating video analysis, cumulative edit rank similarly changes from two to the ambient dimension four. Conversely, the complete cumulative metric-QR trajectory is affine-equivariant when its positive-definite metric, probe, regularizer, and tie-breaking are transported consistently; exact covariance is one corollary, not a canonical semantic metric. In a known-rank finite-sample Adam/QR calibration, identity mixing stops after one accepted update in all 20 large-sample runs, whereas each tested shear $a\in\{.5,.75,1,1.25,2\}$ accepts at least two updates in all 20 runs. Controlled reparameterizations of frozen V-JEPA2 features preserve rank-zero predictions yet alter later Euclidean trajectories under practical optimization. These visual contact experiments are stress tests, not estimates of contact dimension. Iterative erasure therefore returns a procedure-relative estimand jointly determined by representation geometry and the full measurement procedure, not a semantic dimension by itself.
Read the full abstract

All authors Tingan Jin, Shuhang Dong, Haosong Li, Chung-Hsien Chou

Machine Learning (Statistics) · Cross-listed from cs.DS

Improving TensorSketch Using Complex Random Variables

TensorSketch by~\cite{pham2013fast,kar2012random} provides efficient sketching algorithms for high-dimensional polynomial kernels $\vec{x}^{\otimes p} \in \R^{d^p}$. \cite{kar2012random} uses dense Johnson-Lindenstrauss (JL)-type projections with computational cost $O(pDd)$, where $D$ denotes the sketch dimension, whereas~\cite{pham2013fast} extends the sparse CountSketch~\citep{count_sketch} algorithm, yielding a faster algorithm for high-dimensional sparse inputs with running time $O\big(p(\nnz{\vec{x}} + D \log D)\big)$. However, the variance of both estimators grows exponentially with the polynomial degree $p$, scaling as $3^{p}/D$. Recent work by~\cite{pmlr-v206-wacker23a} showed that using complex-valued distribution reduces this dependence to $2^{p}/D$ for the approach of~\cite{kar2012random}. However, their method relies on dense JL-type projections with computational cost $O(pDd)$ and does not extend to the algorithm of~\cite{pham2013fast}. In this work, we introduce a simple variant of TensorSketch~\citep{pham2013fast} that achieves the same variance bound as~\cite{pmlr-v206-wacker23a}, while retaining its advantage of the input-sparsity running time. We validate our results with supporting experiments on synthetic and real-world datasets.
Read the full abstract

All authors Amit Sharma, Mohammad Azhar Khan, Rameshwar Pratap, Keegan Kang

Statistics Theory

Tensor-normal maximum likelihood estimation at the operator-norm sample threshold

Let $X_1,\ldots,X_n$ be independent Gaussian tensors in $\mathbb{R}^{d_1}\otimes\cdots\otimes\mathbb{R}^{d_k}$ with a common covariance matrix given by the Kronecker product of $k$ unknown positive-definite factors, and let $D=\prod_{a=1}^k d_a$ and $d_{\max}=\max_a d_a$. Franks et al. (2026) established condition-number-free guarantees for the tensor-normal maximum likelihood estimator under the sample-size condition $nD\gtrsim k^2 d_{\max}^3$ and asked whether the cubic dependence on $d_{\max}$ could be reduced to a quadratic one. We answer this question affirmatively. For $t\geq 1$, if $nD\geq C k^2 d_{\max}^2 t^2$, then with high probability the maximum likelihood estimator exists, is unique, and satisfies $d_{\rm FR}(\widehat\Theta,\Theta)\leq C t \sqrt{k} d_{\max}/\sqrt{n}$ and $d_{\rm FR}(\widehat\Theta_a,\Theta_a)\leq C t\sqrt{k d_a} d_{\max}/\sqrt{nD}$ for every mode $a$. For every mode $a$ with $d_a=d_{\max}$, we further establish the sharp Thompson-metric bound $d_{\rm op}(\widehat\Theta_a,\Theta_a)\leq C t d_{\max}/\sqrt{nD}$. These guarantees are uniform over the unknown covariance factors and require neither condition-number bounds nor sparsity assumptions. Gaussian submodel lower bounds match the full and largest-factor Fisher–Rao rates up to a factor of $\sqrt{k}$ and the largest-factor Thompson rate up to universal constants. Consequently, for fixed $k$, the quadratic dependence of the sample-size threshold on $d_{\max}$ is optimal. GPT-5.6 Sol and Claude Fable 5 were used to assist with proof development, verification, and manuscript preparation.
Read the full abstract

Statistics Theory

Predicting Diagonalizability of a Mean Matrix

Wu and Santhanam asked whether one can determine, from an increasing i.i.d. sample of binary random matrices, whether the unknown mean matrix is diagonalizable while making only finitely many errors almost surely. We answer this question affirmatively, for diagonalizability over either $\mathbb{R}$ or $\mathbb{C}$. The main observation is a general principle: every semialgebraic property of a fixed-dimensional bounded mean parameter is eventually almost surely predictable. We give a self-contained shrinking-confidence-set proof and an explicit predictor obtained from polynomial sign tests. Tarski–Seidenberg quantifier elimination shows that both the real- and complex-diagonalizable loci are semialgebraic, despite being neither closed nor open. We further extend the positive result to unbounded observations with any fixed finite moment of order $r>1$, using the Marcinkiewicz–Zygmund strong law. Combined with the Dembo–Peres topological criterion, this yields a sharp contrast: over the class of all merely integrable matrix laws, diagonalizability is not eventually almost surely predictable when the dimension is at least two. The construction is effective for fixed dimension, although no practical complexity bound is claimed.
Read the full abstract

Statistics Applications · Cross-listed from cs.LG

A Joint-Distribution Route to Fair Representations with Continuous Sensitive Attributes

Fair representation learning with a continuous sensitive attribute $S$ requires a representation $Z$ that is statistically independent of $S$. Existing criteria, including generalized demographic parity, the expectation of integral probability metrics (EIPM), and mutual information, enforce this independence by averaging a per-value discrepancy between the conditional law $P_{Z \mid S=s}$ and the marginal $P_Z$ over the law of $S$. This approach requires a nonparametric surrogate for the conditional law at each sensitive value. We propose evaluating independence through a single joint discrepancy $d\left(P_{Z, S}, P_Z \otimes P_S\right)$ between the joint law and the product of its marginals. We establish a disintegration identity; on decomposable witness classes it equals the conditional-integral functional that EIPM and generalized demographic parity instantiate. By reaching the same target without the conditional law, this discrepancy can be estimated directly from samples via a dependence statistic rather than conditional smoothing. We take the Hilbert-Schmidt independence criterion (HSIC) as an instance of the joint discrepancy $d$ to investigate the statistical efficiency of replacing the conditional formulation. The HSIC estimator is a closed-form $O\left(n^2\right)$ statistic that converges at the $O\left(n^{-1 / 2}\right)$ rate, in contrast to the nonparametric $O\left(n^{-2 / 5}\right)$ rate of the conditional-route estimators. We prove this instance is equivalent to the conditional maximum mean discrepancy (MMD) integral up to an explicit spectral tail. The corresponding algorithmic implementation, i.e., FRHSIC, attains fairness-accuracy tradeoffs comparable to conditional-route basel es while reducing per-epoch training time.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from math.OC

A lower bound for stepsize-based acceleration of gradient descent

Recent work has shown that, for smooth convex optimization, plain gradient descent can be accelerated from its textbook convergence rate of $O(T^{-1})$ (where $T$ denotes the number of iterations) to $O\big(T^{-\log_2(1+\sqrt{2})}\big)$ using carefully designed stepsize schedules alone, without resorting to momentum or other algorithmic modifications. Despite this progress, however, little was known about lower bounds for such methods beyond the classical $\Omega(T^{-2})$ benchmark for general first-order methods. In this work, we present a new lower bound of $\Omega(T^{-1.9319})$ for the last-iterate convergence rate of gradient descent with predetermined nonnegative stepsize schedules. This result provides rigorous evidence that stepsize schedules alone cannot accelerate plain GD to the optimal $O(T^{-2})$ convergence rate. The proof was developed by GPT-5.6 Sol Pro under the authors' guidance.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from q-bio.NC

Improved cross-validated distances for multivariate pattern analysis

Characterizing the dissimilarity of neural representations between experimental conditions, and tracking it across time, is a central goal of multivariate pattern analysis. Guggenmos et al. (2018) assessed the reliability of many of the measures that can be used for that purpose on MEG data and recommended the use of either the cross-validated Euclidean distance or the within-class-corrected Pearson distance. In this commentary, we show that we can improve upon these distances. First, we show that the cross-validated Euclidean distance is equivalent to a sum of between-partition distances and that this equivalence can be leveraged to obtain a generalized variant, with increased reliability and accuracy. Second, we use the relationship between Euclidean distance and Pearson correlation to define a cross-validated correlation distance in a similar way. The resulting distance is more accurate and interpretable than a formulation proposed by Guggenmos and colleagues. Finally, we discuss the relationship between our generalized cross-validation and within-class correction, another strategy often used to increase reliability, and we show that generalized cross-validation results in higher accuracy for the correlation distance.
Read the full abstract

Statistical Methodology

Conditional validity of prediction-powered inference for spatially dependent finite populations

Prediction-powered inference (PPI) combines full-frame predictions with a small labelled sample to estimate finite-population regression coefficients. We establish conditions for the conditional validity of existing calibrated residual-regression PPI estimators under nonuniform label acquisition and spatially dependent prediction errors. Approximate calibration can leave a population-specific displacement: the projection of prediction errors onto coefficient-specific calibration defects. We characterize its role in conditional inference through the noncentrality of a fixed-candidate Wald region. The relevant target-weight function is the coefficient loading divided by acquisition intensity. When mean-approximation error does not dominate, directional covariance control yields a wider sufficient recovery regime than a total residual-energy bound. A bounded spectral-basis theorem gives conditional coverage under directional covariance, tail and nondegeneracy conditions, with Gaussian and bounded non-Gaussian spatial examples. The analysis accounts for estimation of the one-step augmentation and permits same-label selection from a fixed finite library with a common first-order influence vector. A periodic Fourier construction attains the stochastic directional-gap order within a restricted dictionary class. Constructed and real-record-set experiments illustrate why mean calibration need not protect a slope. The results concern descriptive census projection coefficients under the stated sampling and field conditions.
Read the full abstract

Statistical Methodology

Expert-Guided g-computation with Large Language Models for Estimating Causal Effects on Timings: Applications to Hospital Quality Improvement

Hospital quality improvement (QI) programs routinely face multiple candidate interventions to optimize hospital flow, but existing methods struggle to estimate and rank the causal effects of such interventions. This work focuses on one of the most standard hospital metrics, the average length of stay (LOS), and its causal estimand, the average time saved. To characterize this causal effect, qualitative approaches rely on expert judgment to map patient trajectories, making them susceptible to cognitive biases; quantitative approaches rely on data-driven models, which fail when interventions are hypothetical with no historical data or have complex causal mechanisms that require clinical reasoning rather than data alone. We propose expert-guided g-computation, or egg-computation, which combines the complementary strengths of both approaches by connecting the Gantt charts commonly used to map patient trajectories with the causal DAG literature. We introduce a causal model over Gantt charts and establish identification using a variant of g-computation that seeks expert input only for components unidentifiable from data. To make egg-computation practical, we develop an LLM-assisted pipeline that reliably scales up expert reasoning. In simulations, egg-computation outperforms conventional causal inference methods when patients have diverse causal structures and intervention mechanisms. In a study of eleven candidate QI interventions at an urban safety-net hospital, the LLM pipeline generated graphs and time-saving estimates highly concordant with those of human experts. Beyond healthcare, egg-computation is a broadly applicable framework for estimating the average time saved for candidate interventions whose causal mechanisms can be represented using Gantt charts.
Read the full abstract

All authors Patrick Vossler, Jialin Ouyang, F. Richard Guo, Anran Huang, Ali Shojaie, Lucas Zier, Fan Xia, Jean Feng

Statistical Methodology · Cross-listed from econ.EM

A Multinomial Probit Model for Asymmetric Choice Responses

Standard multinomial probit (MNP) models specify symmetric latent utility distributions, implying that choice probabilities respond symmetrically to positive and negative covariate shifts of the same magnitude. This restriction is often implausible in empirical choice settings and can lead to misleading elasticity and substitution predictions. We propose a skewed multinomial probit (SMNP) model that captures asymmetric choice responses by specifying a multivariate skew-normal distribution for the latent utilities. The model preserves the flexible substitution patterns of the MNP framework, introduces alternative-specific skewness parameters, and nests the standard MNP model when skewness is zero. Introducing skewness creates identification and computational challenges because the skewness parameters interact with the MNP scale normalization and disrupt the conditional Gaussian updating structure used in Bayesian MNP estimation. We address these challenges through a covariance reparameterization that enforces identification and positive definiteness by construction, interpretable priors on the identified parameter space, and a double data-augmentation scheme that yields a Metropolis-Hastings within Gibbs sampler. Numerical experiments and applications to consumer choice data show that SMNP recovers asymmetric choice responses, improves probabilistic prediction, and produces economically meaningful differences in price elasticities and substitution patterns.
Read the full abstract

Monday 10 August

Statistics Theory · Cross-listed from math.MG

Random Width and Brightness: Polyhedral Density Theory, Reconstruction, and Gaussian Identifiability

Let U be uniformly distributed on the unit sphere. We develop a self-contained forward and inverse theory for the random width w_K(U) and brightness b_K(U) of three-dimensional convex bodies. For every full-dimensional polytope, a global spherical co-area formula expresses the width density as a finite sum of angular apertures determined by the normal fan of its difference body; in particular, the density is piecewise real analytic with a finite geometrically determined critical set. This theory yields exact densities for the width of the regular tetrahedron, resolving a question of Finch, and for the regular truncated octahedron, together with the tetrahedral brightness law and the equivalent rhombic-dodecahedral width law. On the inverse side, second- and third-order polarized cosine-transform moments reconstruct finite labelled direction systems whenever the observed triangles span the cycle space of the correlation graph; signed-graph switching describes the unavoidable ambiguity. In contrast, equal three-dimensional intrinsic volumes do not determine either the width law or the brightness law, even for centrally symmetric bodies. Removing the spatial rank constraint gives a dimension-free identifiability theorem for centered multivariate folded-normal vectors: pairwise absolute moments and an anchored family of triple absolute moments, comprising |m - 1|^2 labelled observations for a complete correlation graph, determine the correlation matrix up to diagonal sign conjugacy without fourth-order moments. A harmonic decomposition further identifies the degree-two variance contribution as a constant multiple of the squared Frobenius norm of the traceless part of the weighted frame operator and explains why this contribution vanishes under irreducible symmetry.
Read the full abstract

Statistical Methodology

A Plasmode Benchmark Can Refute but Cannot Certify: Projection and Regularity in Causal-Inference Simulation

Plasmode simulation - resampling covariates, and often treatment, from a real cohort and regenerating the outcome from a model with an injected effect - is a standard tool for comparing causal-inference estimators. We argue that its validity turns on two separable design choices, and that separating them clarifies a recent debate over resampling versus regenerating the treatment. Our central message limits what such a benchmark can show: a plasmode can refute a universal claim about an estimator but cannot certify one. Refuting needs only that the simulated data genuinely belong to the class over which a guarantee is quoted; certifying also needs the benchmark's difficulty to carry over to the target population, which the analyst cannot verify. Two choices set that difficulty. Regularity is whether the simulated law preserves the overlap that inverse-weighting and targeted-learning theory require. Resampling the real treatment with fine-grained covariates destroys it, and the resulting coverage failure falls on flexible, machine-learning-based estimators, while parametric-nuisance estimators - including inverse-probability weighting - stay near nominal; it disappears once the treatment is regenerated from a bounded propensity or the covariates are coarsened, and cross-fitting removes only part of it. Projection is that the injected outcome model hands a matching estimator the right answer by construction; this survives the regularity fix and can reverse estimator rankings even when overlap is intact. We give a regular-plasmode algorithm, a confirming real-data simulation, and practical recommendations. A plasmode is a filter against estimators, not a warrant for them.
Read the full abstract

Machine Learning (Statistics)

Black-Box Knowledge Transfer across Distinct Feature Sets

Pre-trained black-box predictive functions encode knowledge distilled from massive datasets and extensive computation. However, when the available input features differ from those the black box expects, direct use is infeasible. We introduce a method for transferring predictive knowledge from the black box to a new, heterogeneous input space. Our approach decomposes the target regression function into a transferable component, which the black box can inform, and a non-transferable component, which captures information unique to the new space. We propose a two-step neural network procedure, estimating the transferable component from abundant unlabeled feature pairs that bridge the two input spaces and the non-transferable component from limited labels. We derive prediction risk bounds that improve on those of a non-transfer alternative when the non-transferable component is small or smooth, and the procedure adapts to either case. Under additional conditions, the worst-case risk of our estimator is of strictly smaller polynomial order than the minimax risk of estimation from the labeled data alone. We extend the framework to multiple black boxes, each on its own input space, and show that aggregation can reduce prediction error relative to the best single black box. Simulated and real data demonstrate the practical value of the method.
Read the full abstract

Statistical Methodology

Statistical analysis of block structured latent variable models

This paper studies block structured latent variable models, in which observed variables are grouped into distinct blocks based on their relationships with the underlying latent variables. These block structures are prevalent in various fields such as psychology, education, economics, and genetics. Despite their widespread applications, the fundamental statistical properties of these models remain largely unexplored. In this work, we present a comprehensive statistical analysis of the block structured latent variable models. In particular, we first derive conditions for model identifiability across various block designs. Furthermore, we investigate the maximum likelihood estimation under these identifiability constraints. To accommodate these intricate constraints associated with various block configurations, we introduce a Lagrangian-type formulation for the constrained nonconvex optimization problem and show that its optimum coincides with that of the original problem. This formulation serves as a critical tool for understanding the behavior of the constrained estimator under various block structures. Building on that, we establish sharp non-asymptotic error bounds and asymptotic distributions of the constrained maximum likelihood estimator. We also propose a computational framework to obtain the estimator and establish theoretical properties for the algorithm output. Our theoretical findings are validated through simulation studies and empirical data analyses.
Read the full abstract

Statistics Theory

Limit theorems for a class of martingale arrays with applications in nonlinear cointegrating regression

This paper develops a new asymptotic theory for a broad class of martingales, establishing convergence to limiting distributions that involve a functional of stochastic integrals. The proposed limit theorem substantially extends existing martingale asymptotic theory by accommodating a wider class of dependence structures. As a primary application, the theory is applied to nonlinear regression models with nonstationary time series, yielding a rigorous framework for asymptotic inference on nonlinear least square estimators.
Read the full abstract