Peer Revenue Impact factor: volatile

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Monday 17 August

Statistics Theory

Gaussianization-Based Parameter Estimation for Gamma-Gamma and Lognormal-Rician Turbulence Channels

Accurate parameter estimation for atmospheric turbulence channels is challenging because the probability density functions of the Gamma-Gamma (GG) and Lognormal-Rician (LR) models involve special functions and numerical integrations. This paper proposes two Gaussianization parameter estimators for GG and LR turbulence channels, i.e., the quantile-transformation (QT) estimator and the Box-Cox estimator. The QT estimator employs bidirectional cross-transformation together with higher-order statistical matching, whereas the Box-Cox estimator constructs an approximate likelihood by incorporating the transformation Jacobian. Asymptotic analysis identifies skewness as the leading-order deviation from Gaussianity under weak-to-moderate turbulence and yields asymptotic expressions for the Box-Cox power parameters. In addition, a physics-informed regularizer based on the extended Rytov-theory mapping from the Rytov variance to the GG shape parameters is introduced to improve parameter identifiability. Simulation results demonstrate that both estimators provide robust performance under noiseless and noisy conditions, while the physics-informed regularization term can improve the parameter estimation performance by at least three orders of magnitude compared with the iterative moment-based estimator and the method-of-moments/convex-optimization estimator in specific turbulence scenarios.
Read the full abstract

All authors Maoke Miao, Bo Liu, Xinyu Zhang, Xiao-Yu Chen

Sunday 16 August

Statistical Methodology

Beyond the Mean: A Weighted k-Sample Omnibus Variance-Ratio Statistic for Covariate Balance Diagnostics

Rubin's variance ratio (VR) complements the standardized mean difference by detecting covariate imbalance in spread, but no purpose-built omnibus extension exists for more than two groups. We introduce FVR, a size-weighted quadratic combination of pairwise log-variance-ratios that generalizes VR to k groups, together with geometric-mean and maximum pairwise variance-ratio comparators. FVR has an exact relationship to Rubin's VR at k = 2 and a mathematical structure directly parallel to Cohen's f. In Monte Carlo simulations spanning four variance-driven bias mechanisms, k = 3, 4, 6, and sample sizes from 200 to 10,000, FVR and the geometric-mean statistic generally tracked downstream estimation bias at least as well as the maximum statistic. FVR retained substantially more signal when many groups shared a very small sample, while absolute-bias thresholds were unstable at N = 200. Across mechanisms, FVR < 0.10 was reasonably reassuring, values above approximately 0.30 generally indicated concern, and intermediate values were context-dependent. The statistic is implemented for arbitrary analysis weights in the Stata command varatio and illustrated using a multivalued-treatment application.
Read the full abstract

Statistics Applications

Where should the null live? Surrogate tests for clustering of threshold-derived air-quality indices, with application to PM2.5 exceedances in Chile

Air-quality and climate monitoring routinely convert native-resolution measurements into exceedance counts before testing whether high values cluster in time. A surrogate null can be imposed on the derived index or on the native series and then propagated through the same thresholding and aggregation. We show that the two placements can encode different scientific hypotheses and lead to opposite conclusions. In 31 Chilean PM2.5 monitoring stations (2018–2025), weekly clustering of days above the 50 ug m^-3 national 24-hour standard was significant at all stations under an index-level IAAFT null but at only two under a daily native-resolution null; a native generative null with seasonal marginals and short-memory Gaussian-copula dependence agreed with the native surrogate at 30 of 31 stations. Across thresholds spanning the WHO guideline (15 ug m^-3) to the Chilean episode-alert level (80 ug m^-3), index-only rejections increased from 8 of 31 to all eligible stations as monthly event probabilities became more seasonally concentrated, and equalizing monthly event probabilities at the 50 ug m^-3 level reduced index-only decisions from 29 to five. We formalize the issue as null transport through a lossy observation map: an exact aggregate-level equivalent of a native null exists if and only if the randomization kernel is lumpable with respect to that map. A within-fiber diagnostic, which perturbs native amplitudes while keeping the observed index exactly fixed, shows that this condition fails in the monitoring data: the native null distribution moved by a median of 18 times its Monte Carlo variability. Weekly clustering of PM2.5 exceedances in Chile is therefore fully compatible with seasonality plus short-memory daily persistence, and index-level surrogate rejections should not be read as evidence about daily concentration dynamics.
Read the full abstract

Statistical Computation · Cross-listed from cs.CE

Robust topology optimization with non-Gaussian material fields using polygonal finite elements

We present a computational framework for robust topology optimization that integrates polygonal finite-element discretizations, spatially correlated non-Gaussian material modeling, and non-intrusive polynomial-chaos surrogates. Spatial uncertainty in Young's modulus is represented as a homogeneous non-Gaussian random field obtained via a memoryless transformation of a truncated Karhunen-Loève expansion, ensuring physical admissibility through positivity of stiffness while preserving the prescribed autocovariance. Polygonal finite elements provide a stable discretization for density-based optimization on unstructured meshes and mitigate checkerboard artefacts and mesh bias, while the sparse polynomial-chaos expansion enables efficient estimation of low-order statistical moments required by the robust objective at a fraction of the cost of intrusive or Monte Carlo approaches. Numerical studies on a cantilever and a curved beam show that introducing non-Gaussian material variability leads to systematic load-path redistribution and a reallocation of 6-12% of the structural volume, together with a reduction in compliance scatter. The non-intrusive surrogate reproduces intrusive reference results within 3% using an order of magnitude fewer full finite-element analyses. These results demonstrate that the proposed framework offers a physically consistent and computationally efficient route to topology-optimized designs that remain reliable under realistic material uncertainty.
Read the full abstract

All authors Nilton Cuellar, Anderson Pereira, Ivan F. M. Menezes, Americo Cunha Jr

Statistical Methodology

A New Trained Supervised Method for Calculating Patient Similarity

Personalized predictive modelling has been growing rapidly with the increasing availability of Electronic Health Records. This approach aims to improve a model's predictive performance by fitting a unique model to each individual. We train the model on a subset of the training data consisting of individuals similar to the individual being predicted, identified through some similarity metric. Earlier studies show that using a personalized model trained on a customized subset of the data leads to better prediction than using a global model trained on the full dataset. In this work, we develop a new patient similarity metric to improve the prediction of a personalized model for binary response data. Specifically, we introduce a weighted cosine similarity metric that extends the standard cosine similarity metric by assigning predictor-specific weights when computing similarity between participants. These weights are estimated using a supervised approach with the relaxed adaptive group lasso. Results from simulation studies and an analysis of intensive care unit data show that although our proposed similarity metric leads to a slight deterioration in calibration, it produces substantial gains in discrimination. Overall predictive performance measured by the Brier Score improves because the increase in discrimination outweighs the loss in calibration; therefore, our proposed similarity metric more effectively identifies similar participants, resulting in improved predictive accuracy.
Read the full abstract

Statistical Methodology

Doubly robust estimation of while-alive estimands in individually-randomized and cluster-randomized trials

Randomized trials in chronic disease settings often measure treatment benefit through recurrent non-fatal events that are truncated by death, where conventional summaries either discard recurrences, conflate the treatment effect with survival, or treat death as censoring and forfeit a causal interpretation. While-alive estimands measure event burden per unit time alive, but doubly robust estimation for the exposure-weighted while-alive rate remains undeveloped, particularly in cluster-randomized trials (CRTs). We develop a doubly robust estimator based on a local Nelson-Aalen representation, with augmented estimating equations targeting the marginal hazard of the terminal event and the weighted recurrent event rate among those alive; the estimator accommodates multiple event types through prespecified clinical weights and remains consistent if either the censoring model or the outcome working models are correctly specified. For CRTs, we define a new pair of individual-average and cluster-average estimands under informative cluster size, with inference based on cluster-level influence functions. We establish component-wise double robustness and asymptotic normality, corroborate the theory in simulations, and illustrate the methods with reanalyses of data from two completed randomized trials.
Read the full abstract

Machine Learning (Statistics)

Generalized Linear Bandits with Memory

We study generalized linear bandits with memory, an endogenous non-stationary setting in which rewards depend on past actions through a finite memory matrix. Building on prior work for linear models (Clerici et al., 2024), we show that the previously known $\tilde{O}(T^{3/4})$ regret bound stems from a loose analysis, and we provide a sharpened analysis that recovers a $\tilde{O}(\sqrt{T})$ regret rate in the linear case. We then extend this improvement to generalized linear models and propose a block-wise algorithm based on shrunken confidence bounds. Our algorithm achieves a regret bound of $\tilde{O}\left(\sqrt{mT} + d\sqrt{T} + \sqrt{\kappa}\, d^{2} m^{1/4} T^{1/4} + \kappa d^{2} \right)$, where $d$ denotes the feature dimension, $m$ the memory length, and $\kappa$ a curvature parameter of the link function. This attains a $\sqrt{T}$-type rate despite nonlinear rewards and memory effects. To the best of our knowledge, this analysis provides a unified treatment of memory-induced non-stationarity and nonlinear link functions, while ensuring that the leading regret term is independent of the curvature of the link function. We conduct numerical experiments that are consistent with our theoretical findings.
Read the full abstract

All authors Heesang Ann, Hyunjun Choi, Taehyun Hwang, Younghoon Shin, Haeju Cheong, Min-hwan Oh

Machine Learning (Statistics) · Cross-listed from cs.LG

Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading

Reinforcement learning has gained increasing attention as a data-driven approach for stock trading. However, learning a policy that is both profitable and stable remains challenging due to non-stationary market behaviour and noisy reward signals. Auxiliary tasks are often used to improve representation learning and stabilize training, yet they are usually designed manually and depend heavily on prior assumptions about targets and prediction horizons. Such fixed designs may not remain suitable across changing market regimes. In this work, we propose a self-supervised framework that automatically discovers auxiliary tasks to support reinforcement learning for stock trading. The auxiliary tasks are formulated as General Value Functions so that their predictions enrich the learned state representation and assist policy optimization. The framework consists of two networks. The main network learns the trading policy along with the auxiliary predictions, while the secondary network generates the definitions of auxiliary tasks through learned cumulants and discount factors. These tasks are updated using a meta gradient mechanism that accounts for their long-term impact on trading performance and improves training stability. We evaluate the proposed approach across four major equity indices: DJI, FTSE, Sensex, and TAIEX. The empirical results demonstrate that automatically discovered auxiliary tasks lead to more robust learning and improved trading performance compared to existing baselines.
Read the full abstract

Statistics Theory

How Many Samples Are Needed to Determine Causal Direction? Sharp Minimax Bounds for Bivariate LiNGAM

We study how many observations are needed to determine the causal direction between two linearly related variables. Classical LiNGAM theory shows that independent non-Gaussian disturbances identify the direction, but does not quantify the difficulty when the causal effect is weak or the disturbances are nearly Gaussian. Let $\beta$ bound the absolute structural coefficient from below, let $\nu$ measure each standardized disturbance's distance from Gaussianity, and let the disturbance scales lie in $[\underline\sigma,\overline\sigma]$. We prove the sharp local minimax law \[ N_2^\star(\beta,\nu,\delta) \asymp \frac{\log(1/\delta)} {d_\beta^2+\beta^2\nu^2}, \qquad d_\beta= \left[\beta^2- \left(1-\frac{\underline\sigma^2}{\overline\sigma^2}\right)\right]_+. \] Previous theory established population identifiability or assumed a fixed separation between the two directions. By contrast, we establish the sharp sample complexity as a joint function of edge strength, distance from Gaussianity, and scale uncertainty, and characterize when identification comes from non-Gaussian dependence or from covariance alone. The proof was independently generated with GPT-5.6 Sol in Codex's Ultra mode during a two-hour session. The human author supplied the prompt and was responsible only forchecking the proof and revising and polishing the manuscript.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Cross-Entropy Risk Estimation for Language Models: Inconsistency Must Be Dense, and the Holdout Method Is No Exception

Language models are compared by their held-out per-token cross-entropy risk—the quantity scaling laws are fitted to. We show that it cannot be consistently estimated. Consistency, or convergence to the estimand, is defined relative to a possible state of the world: a pair consisting of a data-generating distribution and a model we turn out to train. Quantifying over models as well as data-generating mechanisms is essential, because what decides whether a model's risk is estimable is a tail property of the distribution its weights induce, which no sample reveals. The per-token cross-entropy risk is hard to estimate because of a topological fact: among the possible states, finite risk and infinite risk each lie arbitrarily close to every instance of the other. Consequently no estimator—not merely the holdout average—is consistent at every state at which the risk is defined. Worse, inconsistent estimation persists under both bounding the expected sequence length and restricting to full-support models; and in that restricted setting the states at which inconsistency occurs are even dense. Two interesting ways out are identified, and neither is free. Way out 1: using a bounded context window, we can floor a model's next-token probabilities, making its risk finite exactly when the data-generating distribution has finite expected sequence length—a new, statistical rationale for a choice that was made on computational grounds, though the assumption it substitutes is itself beyond the reach of any test. Way out 2: reporting the risk only when it falls below a threshold fixed in advance restores consistency, at no cost to what model selection actually requires—but we need to recognize that the goal of estimation is revised.
Read the full abstract

Machine Learning (Statistics)

Inferential Evaluation of Surrogate-Derived Models under Covariate Shift

In transfer-learning settings, a model derived from abundant surrogate labels may be deployed in a target population where gold-standard outcomes are unobserved. Evaluating its target performance is essential for determining whether decisions based on the model remain reliable, yet it is difficult when gold labels are scarce, and covariate distributions differ across data sources. We study a three-sample setting with a small gold-labeled source, a larger surrogate-labeled source, and an unlabeled target. Under conditional transportability, we evaluate the surrogate-derived model against the latent gold-standard outcome in the target population. We propose cross-fitted estimators that transport information from the two labeled sources through source-specific density ratios. We also combine outcome-regression augmentation with a kernel correction for estimating the model near a threshold, accounting for uncertainty from all three samples. We establish asymptotically linear inference for TPR and FPR, consistency and pointwise inference for the ROC curve, and asymptotically normal inference for AUC. Simulations assess bias, coverage, and sensitivity to bandwidth and relative sample sizes. A retrospective temporal validation on Chatbot Arena and a semi-synthetic ACS-Income study provide validation in real-world AI applications.
Read the full abstract

Statistical Methodology

Causal mediation analysis for zero-inflated longitudinal data in the presence of treatment non-compliance and multiple mediators

Understanding whether a digital marketing campaign is effective is central to designing effective customer engagement strategies. We analyze a large-scale, longitudinal promotional email campaign conducted by a U.S.\ retailer to evaluate how value-added incentives, such as free shipping, compare with traditional price discounts in influencing customer purchasing behavior. The analysis is complicated by non-compliance, due to not opening emails, multiple longitudinal mediators, and zero-inflated mediators and purchase outcomes. To address these challenges, we develop a Bayesian causal mediation framework based on enriched Dirichlet process mixture models and estimate the causal estimands using a scalable G-computation algorithm. We show that analyses ignoring email-opening behavior substantially attenuate estimated effects. Value-added incentives consistently outperform price discounts, yielding higher estimated potential purchase amounts, with benefits accumulating over time. We design an individualized sequential emailing strategy that optimizes expected purchase count in the observed data.
Read the full abstract

All authors Saurabh Bhandari, Wreetabrata Kar, Michael J. Daniels, Bikram Karmakar

Machine Learning (Statistics) · Cross-listed from cs.LG

Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning

Designing effective trading strategies using reinforcement learning remains challenging due to delayed and noisy rewards, poor exploration, and the difficulty of enforcing explicit risk constraints. In this work, we propose BRaG, a barycenter-based adversarial inverse reinforcement learning framework for stock trading that learns trading behavior from multiple heterogeneous expert strategies. BRaG aggregates expert demonstrations using a performance-weighted Wasserstein barycenter, yielding a stable pseudo-expert representation that captures shared structure across diverse trading styles. This representation is used to pretrain a trading policy via adversarial imitation learning, which alleviates unstable exploration during reinforcement learning. The pretrained policy is subsequently refined using reinforcement learning with true market rewards. To ensure risk-aware decision-making, BRaG incorporates control barrier functions that constrain action execution and regularize policy learning to satisfy drawdown limits. We evaluate the proposed approach on four major global equity markets, including the US, UK, Indian, and Taiwanese indices. Across all the markets, the proposed approach achieves stronger performance than both classical trading rules and recent deep reinforcement learning methods, while exhibiting more stable risk characteristics.
Read the full abstract

Statistics Theory

The Dual-Population Benchmark Model

Motivated by Fisher's and Hill's \cite{Fisher,Hill} ideas of ranking and pivotal quantities, we introduce a planar Poisson process (PPP) model in which the negative component \(\Pi_-\), acted upon by a choice operator C, supplies a set of past benchmarks that divide the future points of the positive component \(\Pi_+\) into ranked categories. The benchmarks act as separators in an ordered paintbox while simultaneously acquiring the dual role of past standards established by the choice operator. The exceptional homogeneity properties of the PPP offer an infinitude of possibilities which absorb many existing combinatorial structures and enrich the toolbox of Bayesian distribution-free inference. A particular benchmark generating mechanism considered here (the exponential race) amounts to the device of splitting into spacings of nonhomogeneous order statistics. On the methodological side, the paper aims to highlight the role of order as important characteristic of an exchangeable structure, complementary to the description in terms of the components size. Moreover, we advocate the viewpoint that the order induced by a latent strength parameter is intrinsically inherited from the distant-past temporal sampling order, hence the decoupled orders may coexist within the framework of population duality without disturbing size-biasedness of components and the full exchangeability within the sample. This implies that the indistinguishability of components in the nonlinear CRP, sometimes regarded as nonexchangeability, does not in fact destroy exchangeability and should be reconciled with the arrival ordering within the paradigm of ordered structures.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

FirstDiff: One-Step Diffusion-Based Anomaly Detection for Multivariate Time Series via Initial Noise Prediction

Diffusion models have recently shown strong potential for multivariate time-series anomaly detection by learning the distribution of normal data through iterative denoising. Existing diffusion-based approaches, however, typically perform anomaly detection after completing the reverse diffusion process, relying primarily on the final reconstructed signal and overlooking informative representations produced during denoising. This design incurs substantial computational cost and limits the use of intermediate diffusion information for anomaly detection. In this paper, we propose FirstDiff, a diffusion-based anomaly detection framework based on the observation that the predicted diffusion noise at the initial reverse-diffusion evaluation already contains sufficient information for accurate anomaly detection. FirstDiff models the statistical distribution of predicted diffusion noise under normal behavior using validation data, enabling anomaly inference from a single denoising-network evaluation rather than completing the reverse diffusion trajectory. To model complex temporal and inter-sensor dependencies, FirstDiff employs a Diffusion Transformer as the denoising backbone. Extensive experiments on five public benchmark datasets demonstrate that FirstDiff achieves state-of-the-art performance while reducing diffusion inference from the full reverse trajectory to a single denoising-network evaluation.
Read the full abstract

Machine Learning (Statistics)

On Stopping Rules and Spatial Adaptation for CART

The popular CART algorithm for regression trees combines a greedy splitting rule with a stopping rule, but while the splitting rule has been well studied, the statistical role of stopping rules is less well understood. Meanwhile, although regression trees fit using Bayesian methods or via empirical risk minimization (ERM) have been shown to be spatially adaptive to local smoothness and anisotropy, it is unknown whether CART can achieve the same adaptation. We address these gaps by proving that, under spatially heterogeneous and anisotropic smoothness and appropriate structural assumptions on the regression function and covariate distribution, CART with the minimum impurity decrease (MID) stopping rule and a suitable threshold achieves pointwise rates that are minimax up to logarithmic factors. These rates hold simultaneously over all points in the domain. Moreover, we prove that spatial adaptation cannot be achieved under the widely used minimum leaf size stopping rule. Together, these results establish a precise statistical role for the MID stopping rule and provide a theoretical basis for the empirical success of CART.
Read the full abstract

Statistics Theory

On the minimax-rate optimality of approximate Bayesian computation in nonparametric problems

Approximate Bayesian computation (ABC) replaces likelihood evaluation by simulation and comparison of observed and synthetic data. We establish minimax-rate guarantees for nonparametric ABC under random-series priors with simulable finite-dimensional coordinates. The contraction theorem uses local prior mass, bounds on ABC acceptance probabilities, and control of prior mass outside a sieve. In fixed-design orthogonal-series regression with centered $g$-and-$k$ errors, an infinite Gaussian series prior with a compact scale hyperprior yields minimax-rate contraction and a minimax-rate clipped posterior mean. In compound Poisson decompounding, only random sums are observed and the target is the underlying jump density. With an unknown count intensity in a fixed compact subinterval of $(0,\pi/2)$, we prove stability of the zero-count-augmented trigonometric population summaries and use a square-root Gaussian series prior on the space of probability density functions. Over bounded periodic Sobolev classes of smoothness $\alpha>d/2$, a polynomially enlarged synthetic sample yields ABC contraction at rate $n^{-\alpha/(2\alpha+d)}$ and posterior mean squared risk of order $n^{-2\alpha/(2\alpha+d)}$, matching a lower bound for the aggregate-observation model. Rejection-ABC Monte Carlo approximations inherit these rates under sufficient sampling budgets.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Sparse Prototype Code Underlies Classification and Prediction Across Modalities

Neural representations have become a central tool for studying the internal mechanisms of modern AI models, yet their complex high-dimensional structure makes them difficult to interpret. We show that classification tasks give rise to a universal representational geometry, shared across state-of-the-art models in vision, audio, and language processing. The key structure is that within-class variability is not random in representation space. Instead, its classifier-relevant component has strong and structured correlations with the class's own centroid and with the centroids of its competing classes. Building on this observation, we derive an analytical mean-field theory governed mainly by the variability along true-class and rival-class centroid coordinates, together with a global renormalization of the class radius that compensates for the non-Gaussian statistics of real representations. The theory accurately predicts classification accuracy across architectures and modalities. The relevant geometric quantities improve systematically with model scale, mirroring the observed gains in accuracy. A striking feature of the theory is its sparsity: accurate prediction requires only a small set of centroid coordinates associated with the true class and its strongest rivals - connecting our framework to sparse-feature extraction approaches such as sparse autoencoders. Together, these results provide a parsimonious predictive theory of neural representations and suggest that classification in deep networks is governed by a sparse, centroid-aligned structure embedded within the full high-dimensional representation space.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Benchmarking Quantum Machine Learning for Power-System Attack Detection: Evaluation Choices Decide the Outcome Before the Models Do

Machine-learning detectors for power-system cyberattacks are themselves attack surfaces, and quantum machine learning has been proposed for them. We benchmark fidelity-kernel SVMs and variational classifiers against six tuned classical models on public power-system attack data (Mississippi State/ORNL), across white-box, transfer, decision-based black-box, and poisoning attacks. Our headline finding is methodological: the benchmark's answers are set by the evaluator's choices before the models. Eight choices – six in the evaluation protocol, two in the tuning the benchmark itself runs – each reversed or moved a conclusion at fixed models. The largest is the split: the row-level protocol scores 0.905 macro-F1 where holding whole source files out leaves 0.594, and in the capped matched-dimensionality regime the quantum arm sits within noise of chance with the classical arm 0.024 above it. A fidelity kernel looks most robust until attacked directly (retention 0.886 to 0.064); a mis-fitted surrogate manufactures a 10x asymmetry; an unseeded black-box attack moves 75% between restarts. A positive control explains the accuracy null: the labels, not the pipeline. We give the control that catches each choice and release the seeded benchmark.
Read the full abstract

Statistics Applications

A Primer on Digital Health N-of-1 Studies and Single-Case Designs

Clinical studies generally assume that group-level averages are useful quantities for guiding individual-level decisions in the clinical care of each individual patient. Precision medicine has notably closed the gap towards truly individualized care through highly refined subgrouping. Today, digital health technologies and other modern sources of dense, personal "small data" enable a different approach to treatment individualization—one that seeks to characterize a single person's own recurring health patterns first and foremost, rather than identifying the best subgroup to which they might belong. In this chapter, we review the key concepts underlying n-of-1 studies, single-case designs, and other "multitudinal" approaches for digital health applications, and explore their relationships to other digital health methods. We also share some promising future directions for "esametry", the statistics of the digitized multitudes within each of us.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

PERO: Efficient Robust Post-Training Foundation Models for Encrypted Traffic Classification

Encrypted traffic classification is vital for network security, yet real-world deployments are inherently sensitive to rare but high-loss errors such as misclassification of malicious traffic. The encrypted traffic foundation model, as a promising general-purpose technique, can achieve impressive overall performance. However, employing standard objectives such as empirical risk minimization often overlooks high-risk tail events, and commonly used performance metrics hardly reflect robustness limitations in risk-sensitive scenarios. Directly applying robust optimization objectives, such as conditional value-at-risk, to post-training is computationally prohibitive for large models, as identifying high-loss samples exhausts substantial computation. To this end, we propose Pre-Evaluation Robust Optimization (PERO), an efficient robust post-training framework for encrypted traffic foundation models. PERO employs a lightweight proxy to estimate sample-wise risk and selects a subset of high-risk samples to update the foundation model, decoupling risk estimation from expensive large-model optimization. Extensive experiments on typical encrypted traffic datasets show that PERO achieves competitive or superior robustness and average performance compared to outstanding robust post-training methods, while significantly reducing computational and memory costs.
Read the full abstract

All authors Wumei Du, Jiarong Wen, Kaiyu Zhang, Zi Yang, Yiqin Lv, Longfei Zhang, Dong Liang, Zheng Xie

Statistical Methodology

Generalized Hierarchical Conformal Prediction

Many prediction problems arise with data collected in groups. In this setting, hierarchical conformal prediction (HCP) (Lee et al., 2026) provides distribution-free prediction sets for a new observation from a previously unseen group under hierarchical exchangeability. In many applications, however, prediction is conducted only after a few observations from the group of interest have already been collected. Standard HCP cannot leverage these observations, as its required symmetry conditions do not hold in this setting. At the same time, the initial sample may still be too small for standard conformal prediction applied within the test group to be informative. We develop predictive inference methods for this setting. Our proposed method, Generalized HCP (GHCP), restores the relevant symmetry needed for conformal inference by assigning the test group a randomly "donated" reference group size. GHCP further leverages the initial test group observations to improve the quality of the nonconformity scores for prediction within that group. To improve efficiency, we introduce a variant that restricts the set of eligible donors. We demonstrate the performance of the proposed method through simulations and an illustration on the American Community Survey dataset.
Read the full abstract

All authors Soham Mallick, Eric Tchetgen Tchetgen, Edgar Dobriban, Yonghoon Lee

Statistical Methodology

Energy Balancing Weights for Mediation Analysis

Causal mediation analysis requires reconstruction of counterfactual distributions to estimate natural direct and indirect effects. Inverse probability weighting estimators rely on models for treatment assignment and mediator density ratios, whereas moment balancing approaches require researchers to specify in advance which functions of the covariates and mediators should be balanced. We propose Energy Balancing Weights for Mediation Analysis (EBWMA), which targets the joint mediator-covariate distribution used to identify counterfactual means such as E[Y(1, M(0))]. Under standard identification conditions for natural effects, EBWMA constructs weights whose weighted empirical distribution approximates this target, without modeling treatment assignment, mediator density ratios, or the outcome regression. The weights minimize energy distance through two quadratic programming problems solved sequentially. In simulations with nonlinearly transformed, skewed, or binary covariates and nonlinear mediator and outcome models, EBWMA generally achieved favorable bias and root mean squared error, with uniformly lower Monte Carlo variability than gradient boosting-based inverse probability weighting and moment balancing weights. In an illustrative analysis of the National Health and Nutrition Examination Survey I Epidemiologic Follow-up Study, EBWMA gave the smallest standardized mean differences for most covariates and for the mediator.
Read the full abstract

Statistics Theory

Bootstrap Error Estimation and Sketch-Size Selection for Sketched Ridge Regression

Randomized sketching reduces the computational cost of large ridge-regression problems, but the coefficient error depends on the realized sketch. We extend the paired-row bootstrap from randomized least squares to ridge regression, enabling coefficient-error estimation using only compressed data. Under a fixed coefficient dimension and increasing data and sketch sizes, we derive asymptotic linear representations and Gaussian limits for the sketched estimator and the conditional bootstrap distribution. These results establish uniform consistency of the bootstrap error distribution and asymptotically exact coverage when the estimator and error bound are computed from the same sketch. For sketches with independent, mean-zero, variance-one entries, an explicit covariance formula separates the effects of the residual, regularization, and fourth moment of the sketch entries, and shows that Rademacher entries minimize the leading covariance matrix in the Loewner order. We also develop a fast linearized bootstrap, an order-statistic correction for finitely many bootstrap replicates, and a Bonferroni rule for selecting from a fixed set of sketch sizes. Experiments on two real and two synthetic data sets support the proposed methods. With a sketch size 15 times the number of coefficients, 199 bootstrap replicates, and nominal coverage of 95%, the bootstrap with refitting attains coverage between 92.0% and 95.3%; after the order-statistic correction, coverage ranges from 94.7% to 97.7%.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Optimal Lower Bounds for Networked Information Aggregation

The problem of networked information aggregation, studied in Kearns et al. (2026), involves a group of learners situated on the vertices of a directed acyclic graph $G$, each learning a linear predictor $\widehat Y$ for a fixed random variable $Y$ given access to a local feature, as well as the predictors learnt by its parents. Learning proceeds iteratively, with learners ordered according to a topological sort of $G$. The main quantity of interest is the error incurred by the current learner, constrained to this flow of information, with respect to the best linear predictor using all the features seen so far. When the studied error is the MSE, i.e., $\mathbb{E} (\widehat Y - Y)^2$, Kearns et al. (2026) show that the error is at most $O(1/\sqrt{D})$ along a path of length $D$. They also obtain a hard instance where the MSE is lower bounded by $\Omega(1/D)$, leaving the correct order open. In this work, we resolve this central open problem, and obtain a family of worst case problem instances with a MSE lower bound of $\Omega(1/\sqrt{D})$. By exploiting invariances in the structure of the learnt predictors, our analysis generalizes to all convex loss functions $\ell(\widehat Y, Y)$ satisfying regularity conditions which include strong convexity in a ball around the origin, and that the ideal predictor minimizing the population loss is positively correlated with the label. We show that networked information aggregation on a gaussian instance in our worst case family incurs an $\ell$-error lower bounded by $\Omega(1/\sqrt{D})$ with respect to this ideal predictor. We demonstrate that a variety of common losses satisfy these regularity conditions. In particular, the logistic loss satisfies them, and hence our analysis also closes the gap between the upper and lower bounds in Bateni et al. (2026).
Read the full abstract

Statistical Methodology

Competing-Risk Cure Models: A Comprehensive Systematic Review of Methodological Literature

Competing-risk cure models describe time-to-event populations with individuals immune to all event types or an event of interest, yet literature is fragmented across model families. We review 26 papers across five axes: cure definition/scope; decomposition/cure mechanism; latency; dependence, censoring, and masked causes; and estimation. We distinguish global from cause-specific cure and incidence–latency mixtures from vertical susceptibility factorizations, latent competing-causes/zero-count constructions, defective-survival models, and zero-inflated mixture or cumulative incidence function (CIF) formulations. We compare parametric, piecewise-constant, PH, AFT, transformation, CIF-based, nonparametric, and partially specified latency models for right/interval censoring, clustering, and masked causes. Mixture formulations dominate, but similar names can mask different estimands, cure mechanisms, latent-risk/censoring assumptions, and regression interpretations. Latent-failure dependence is modeled less often than cure or latency; failure–censoring dependence, within-cluster association, and masked causes occur in smaller subsets. Estimation spans likelihood and expectation-maximization (EM), including neural-network M-steps, estimating equations, inverse-probability-of-censoring weighting, Bayesian computation, and copula-graphic estimation. A reproducible defective-Gompertz analysis of public bone-marrow-transplant data shows that fitted tail probabilities require model- and endpoint-specific interpretation, not all-method comparison. Reproducibility remains limited: most implementations use custom code, few offer repository access, and no widely adopted, clearly licensed R/Python framework unifies the constructions. This taxonomy supports transparent model selection/reporting, estimation-method comparison, and needs for theory, software, benchmarking, and reproducible applications.
Read the full abstract

Saturday 15 August

Statistical Computation · Cross-listed from math.PR

Quadratic mixing for the Lee–Sidford Dikin walk

Let $K=\{x\in\R^d:Ax\ge b\}$ be a bounded full-dimensional polytope with $m$ inequalities. For targets on $K$ with density proportional to $\exp(-\vartheta^\transpose x)$, we prove a $\widetilde O(d^2)$ bound, measured in Markov chain iterations, for the chi-square mixing of the $1/2$-lazy Metropolis Dikin walk based on a polylogarithmically rescaled Lee–Sidford metric. The estimate holds for every initial law with finite chi-square divergence. The proof controls the reverse Gaussian proposal through a quantitative average self-concordance estimate at the required $O(1/d)$ scale. We establish this estimate from uniform analytic derivative bounds for the Lewis-weight subspace and an all-order projection-weighted contraction inequality for Gaussian chaoses. For order $j$, the latter bounds each base-point polynomial at scale $d^{j/2}$, up to $j$-dependent and logarithmic factors. Truncating at logarithmic order then absorbs the remaining pathwise loss. This resolves, in the affirmative, the open problem of whether the Gaussian Lee–Sidford Dikin walk mixes in $\widetilde O(d^2)$ iterations: the previously stated quadratic bounds \cite[Theorem~1.4]{LLV20}, \cite[Theorem~1.2]{GKM24} rest on reverse-proposal steps identified as unresolved in \cite[Section~1]{Kook26}, whose $\widetilde O(d^{9/4})$ bound was the best rigorously established rate prior to this work. The proof is found by ChatGPT 5.6 Pro.
Read the full abstract

Statistical Methodology

Toward Efficient Estimation of Regional Treatment Effects in Multi-Regional Clinical Trials

A multi-regional clinical trial (MRCT) is a single clinical trial conducted in multiple regions simultaneously under a common protocol, which may be used to support parallel submissions to multiple regulatory authorities. For a regional regulatory authority, treatment effects defined specifically for its own region are more relevant to consider than overall treatment effects based on all regions included in an MRCT. A regional treatment effect can be estimated consistently using local data from the region of interest; however, this approach is generally inefficient as it excludes data from other regions and ignores possible similarities between regions. On the other hand, simply pooling data across regions requires strong assumptions and may introduce bias when the required assumptions are not met. Here, we propose a simple and robust approach to estimating a regional treatment effect in a two-arm randomized MRCT. The proposed approach uses a working regression model to incorporate information from baseline covariates as well as data from other regions for improved efficiency. The model accounts for residual regional differences (after adjusting for measured covariates) using interaction terms that describe how the dependence of outcome on treatment and covariates may vary across regions. The adaptive lasso is used to identify null interactions and thus achieve selective borrowing of information from other regions. The resulting regional treatment effect estimator is consistent and asymptotically normal even when the working model is misspecified, and able to improve efficiency over local estimation when there are similarities between regions in the form of null interactions.
Read the full abstract

All authors Zhiwei Zhang, Yongwu Shao, Wei Zhang, Aiyi Liu

Statistical Methodology

Outcome Modeling in Design-Based Inference for Spatial Settings

In the face of spatial interference, researchers are often interested in estimating treatment effects at specific points located in space. Wang et al. (2025) and Pollmann (2023) provide design-based frameworks for estimating spillover effects on points located across a range of distances from interventions. Although these frameworks are design-based, we show their proposed estimands rely on outcomes that are directly unobservable and therefore, require outcome modeling. When using modeled outcomes in practice, even the typically design-unbiased Horvitz-Thompson estimator can accrue bias as a result of the modeling error. The performance of spatial outcome models depends on the density or resolution of observed outcomes. Through simulation, we find that the bias of the estimators decays with increasing outcome density, but not with increasing numbers of intervention units, and standard errors using modeled outcomes converge to the oracle standard errors. To demonstrate the role of outcome modeling with spatial interference, we reanalyze an experiment from Collazos et al. (2021) on the effect of hot spots policing on crime and provide several suggestions for practice.
Read the full abstract

Statistical Methodology

Design-Based Inference under Deep Domain Stratification: Language of Instruction and Private-Institution Choice in India's NSS 71st Round

Large household surveys support precise national estimates but can become statistically fragile after repeated disaggregation by geography, sector, sex, age, and outcome category. This paper develops an auditable design-based framework for deciding how far such disaggregation can be taken in a stratified multistage survey. The framework is built around a nested contribution ledger that reconstructs each domain total through the first stage probability proportional to size expansion, the certainty-plus-random hamlet-group selection, and the second-stage household expansion. Nonlinear domain parameters are expressed as ratios of these totals and analyzed by first-order linearization. The two independent National Sample Survey subsamples then provide a natural replication variance estimator. A granularity-stability profile combines the resulting relative standard error with replicate support and concentration diagnostics, so that a detailed estimate is accompanied by evidence about whether the design can sustain it. Finite population unbiasedness of the total estimator, asymptotic validity of the ratio linearization, and unbiasedness of the two-subsample variance estimator for linearized totals are established. The method is illustrated with the 71st-round Social Consumption: Education survey, focusing on home language versus medium of instruction and reported reasons for preferring private educational institutions in India and Himachal Pradesh. The application preserves the substantive analysis in the original project while replacing ad hoc calculation with a reproducible inferential workflow.
Read the full abstract