Statistical Methodology · Cross-listed from cs.LG
Measure-Theoretic Anti-Causal Representation Learning
Arman Behnam and Binghui Wang
Causal representation learning in the anti-causal setting (labels cause features rather than the reverse) presents unique challenges requiring specialized approaches. We propose Anti-Causal Invariant Abstractions (ACIA), a novel measure-theoretic framework for anti-causal representation learning. ACIA employs a two-level design, low-level representations capture how labels generate observations, while high-level representations learn stable causal patterns across environment-specific variations. ACIA addresses key limitations of existing approaches by accommodating prefect and imperfect interventions through interventional kernels, eliminating dependency on explicit causal structures, handling high-dimensional data effectively, and providing theoretical guarantees for out-of-distribution generalization. Experiments on synthetic and real-world medical
datasets demonstrate that ACIA consistently outperforms state-of-the-art methods in both accuracy and invariance metrics. Furthermore, our theoretical results establish tight bounds on performance gaps between training and unseen environments, confirming the efficacy of our approach for robust anti-causal learning.
Read the full abstract ▾
Machine Learning (Statistics)
Manifold Dimension Estimation via Local Graph Structure
Zelong Bi and Pierre Lafaye de Micheaux
Most existing
manifold dimension
estimators rely on the assumption that the underlying
manifold is locally flat within the neighborhoods under consideration. More recently,
curvature-adjusted principal component analysis (CA-PCA) has emerged as a powerful alternative by explicitly accounting for the
manifold's
curvature. Motivated by these ideas, we propose a
manifold dimension estimation framework that captures the local graph structure of the
manifold through regression on local PCA coordinates. Within this framework, we introduce two representative
estimators: quadratic
embedding (QE) and total least squares (TLS). Experiments on both synthetic and real-world
datasets demonstrate that these methods perform competitively with, and often outperform, state-of-the-art approaches.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
A Simple Method for PMF Estimation on Large Supports
Alex Shtoff
We study nonparametric estimation of a probability mass function (PMF) on a large discrete support, where the PMF is multi-modal and heavy-tailed. The core idea is to treat the empirical PMF as a signal on a line graph and apply a data-dependent low-pass filter. Concretely, we form a symmetric tri-diagonal operator, the path graph
Laplacian perturbed with a diagonal
matrix built from the empirical PMF, then compute the eigenvectors, corresponding to the smallest feq
eigenvalues. Projecting the empirical PMF onto this low dimensional subspace produces a smooth, multi-modal estimate that preserves coarse structure while suppressing noise. A light post-processing step of clipping and re-normalizing yields a valid PMF. Because we compute the eigenpairs of a symmetric tridiagonal
matrix, the computation is reliable and runs time and
memory proportional to the support times the dimension of the desired low-dimensional supspace. We also provide a practical, data-driven rule for selecting the dimension based on an orthogonal-series risk estimate, so the method "just works" with minimal tuning. On synthetic and real heavy-tailed examples, the approach preserves coarse structure while suppressing sampling noise, compares favorably to logspline and Gaussian-KDE baselines in the intended regimes. However, it has known failure modes (e.g., abrupt discontinuities). The method is short to implement, robust across sample sizes, and suitable for automated pipelines and exploratory analysis at scale because of its reliability and speed.
Read the full abstract ▾
Statistics Applications
Bayesian Additive Regression Trees (BART) in Food Authenticity: A Classification Approach to Food Fraud Detection
Mengxiang Zhu and Riccardo Rastelli
Feature engineering plays a critical role in handling hyperspectral data and is essential for identifying key wavelengths in food fraud detection. This study employs
Bayesian Additive Regression Trees (BART), a flexible machine learning approach, to discriminate and classify samples of olive oil based on their level of purity. Leveraging its built-in variable selection mechanism, we employ BART to effectively identify the most representative spectral features and to capture the complex interactions among variables. We use network representation to illustrate our findings, highlighting the competitiveness of our proposed methodology. Results demonstrate that when principal component analysis is used for dimensionality reduction, BART outperforms state-of-the-art models, achieving a classification accuracy of 96.8% under default settings, which further improves to 97.2% after hyperparameter tuning. If we leverage a variable selection procedure within BART, the model achieves perfect classification performance on this
dataset, improving upon previous optimal results both in terms of accuracy and interpretability. Our results demonstrate that three key wavelengths, 1160.71 nm, 1328.57 nm, and 1389.29 nm, play a central role in discriminating the olive oil samples, thus highlighting an application of our methodology in the context of food quality. Further analysis reveals that these variables do not function independently but rather interact synergistically to achieve accurate classification, and improved detection speed.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from math.NA
Fast spectral separation method for kinetic equation with anisotropic non-stationary collision operator retaining micro-model fidelity
Yue Zhao and Huan Lei
We present a generalized, data-driven collisional operator for one-component
plasmas, learned from molecular dynamics simulations, to extend the collisional kinetic model beyond the weakly coupled regime. The proposed operator features an anisotropic, non-stationary collision kernel that accounts for particle correlations typically neglected in classical Landau formulations. To enable efficient numerical evaluation, we develop a fast spectral separation method that represents the kernel as a low-rank tensor product of univariate basis functions. This formulation admits an $O(N \log N)$
algorithm via fast
Fourier transforms and preserves key physical properties, including discrete conservation laws and the H-
theorem, through a structure-preserving central difference discretization. Numerical experiments demonstrate that the proposed model accurately captures
plasma dynamics in the moderately coupled regime beyond the standard Landau model while maintaining high computational efficiency and structure-preserving properties.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Physics-informed data-driven machine health monitoring for two-photon lithography
Sixian Jia, Zhiqiao Dong and Chenhui Shao
Two-
photon lithography (TPL) is a sophisticated additive manufacturing technology for creating three-dimensional (3D) micro- and nano-structures. Maintaining the health of TPL systems is critical for ensuring consistent fabrication quality. Current maintenance practices often rely on experience rather than informed monitoring of machine health, resulting in either untimely maintenance that causes machine downtime and poor-quality fabrication, or unnecessary maintenance that leads to inefficiencies and avoidable downtime. To address this gap, this paper presents three methods for accurate and timely monitoring of TPL machine health. Through integrating physics-informed data-driven predictive models for structure dimensions with statistical approaches, the proposed methods are able to handle increasingly complex scenarios featuring different levels of generalizability. A comprehensive experimental
dataset that encompasses six process parameter combinations and six structure dimensions under two machine health conditions was collected to evaluate the effectiveness of the proposed approaches. Across all test scenarios, the approaches are shown to achieve high accuracies, demonstrating excellent effectiveness, robustness, and generalizability. These results represent a significant step toward condition-based maintenance for TPL systems.
Read the full abstract ▾
Machine Learning (Statistics)
The Minimax Lower Bound of Kernel Stein Discrepancy Estimation
Jose Cribeiro-Ramallo, Agnideep Aich, Florian Kalinke et al. (+2 more)
Kernel Stein discrepancies (KSDs) have emerged as a powerful tool for quantifying goodness-of-fit over the last decade, featuring numerous successful applications. To the best of our knowledge, all existing KSD
estimators with known rate achieve $\sqrt n$-convergence. In this work, we present two complementary results (with different proof strategies), establishing that the minimax lower bound of KSD estimation is $n^{-1/2}$ and settling the optimality of these
estimators. Our first result focuses on KSD estimation on $\mathbb R^d$ with the Langevin-Stein operator; our explicit constant for the Gaussian kernel indicates that the difficulty of KSD estimation may increase exponentially with the dimensionality $d$. Our second result settles the minimax lower bound for KSD estimation on general domains.
Read the full abstract ▾
All authors Jose Cribeiro-Ramallo, Agnideep Aich, Florian Kalinke, Ashit Baran Aich, Zoltán Szabó
Machine Learning (Statistics) · Cross-listed from cs.LG
AlignFlow: Improving Flow-based Generative Models with Semi-Discrete Optimal Transport
Lingkai Kong, Molei Tao, Yang Liu et al. (+4 more)
Flow-based Generative Models (FGMs) effectively transform noise into complex data distributions. Incorporating Optimal Transport (OT) to couple noise and data during FGM training has been shown to improve the straightness of flow trajectories, enabling more effective inference. However, existing OT-based methods estimate the OT plan using (mini-)batches of sampled noise and data points, which limits their scalability to large and high-dimensional
datasets in FGMs. This paper introduces AlignFlow, a novel approach that leverages Semi-Discrete Optimal Transport (SDOT) to enhance the training of FGMs by establishing an explicit, optimal
alignment between noise distribution and data points with guaranteed convergence. SDOT computes a transport map by partitioning the noise space into Laguerre
cells, each mapped to a corresponding data point. During FGM training, i.i.d. noise samples are paired with data points via the SDOT map. AlignFlow scales well to large
datasets and model architectures with negligible computational overhead. Experimental results show that AlignFlow improves the performance of a wide range of state-of-the-art FGM
algorithms and can be integrated as a plug-and-play component. Code is available at: https://github.com/konglk1203/AlignFlow.
Read the full abstract ▾
All authors Lingkai Kong, Molei Tao, Yang Liu, Bryan Wang, Jinmiao Fu, Chien-Chih Wang, Huidong Liu
Machine Learning (Statistics)
The Coverage Principle: How Pre-Training Enables Post-Training
Fan Chen, Audrey Huang, Noah Golowich et al. (+5 more)
Language models demonstrate remarkable abilities when pre-trained on large text corpora and fine-tuned for specific tasks, but how and why pre-training shapes the success of the final model remains poorly understood. Notably, although pre-training success is often quantified by cross-
entropy loss, cross-
entropy can be a poor predictor of downstream performance. Instead, we provide a theoretical perspective on this relationship through the lens of coverage, which quantifies the probability mass the pre-trained model places on high-quality responses and which is necessary and sufficient for post-training and test-time scaling methods such as Best-of-N to succeed. Our main results develop an understanding of the coverage principle, a phenomenon whereby next-
token prediction (more generally, maximum likelihood) implicitly optimizes toward a model with good coverage. In particular, we uncover a mechanism that explains the power of coverage in predicting downstream performance: coverage generalizes faster than cross-
entropy, avoiding spurious dependence on problem-dependent parameters such as the sequence length. We also study practical algorithmic interventions with provable benefits for improving coverage, including (i) model/checkpoint selection procedures, (ii) gradient normalization schemes, and (iii) test-time decoding strategies.
Read the full abstract ▾
All authors Fan Chen, Audrey Huang, Noah Golowich, Sadhika Malladi, Adam Block, Jordan T. Ash, Akshay Krishnamurthy, Dylan J. Foster
Machine Learning (Statistics)
The Tree-SNE Tree Exists
Jack Kendrick
The clustering and visualisation of high-dimensional data is a ubiquitous task in modern data science. Popular techniques include nonlinear dimensionality reduction methods like t-SNE or UMAP. These methods face the `scale-problem' of clustering: when dealing with the MNIST
dataset, do we want to distinguish different digits or do we want to distinguish different ways of writing the digits? The answer is task dependent and depends on scale. We revisit an idea of Robinson & Pierce-Hoffman that exploits an underlying scaling
symmetry in t-SNE to replace 2-dimensional with (2+1)-dimensional
embeddings where the additional parameter accounts for scale. This gives rise to the t-SNE tree (short: tree-SNE). We prove that the optimal
embedding depends continuously on the scaling parameter for all initial conditions outside a set of measure 0: the tree-SNE tree exists. This idea conceivably extends to other attraction-repulsion methods and is illustrated on several examples.
Read the full abstract ▾
Machine Learning (Statistics)
Reliable data clustering with Bayesian community detection
Magnus Neuman, Jelena Smiljanić and Martin Rosvall
From neuroscience and genomics to systems biology and ecology, researchers rely on clustering similarity data to uncover modular structure. Most methods work in two steps: first reduce noise, then cluster. The first step runs without regard to the second: it forces arbitrary choices, discards useful information, and can impose structure where none exists. Here we bring the two steps together. Using established
Bayesian community detection methods, we let one minimum description length criterion choose both how much to sparsify the data and how to cluster it. This one-step approach keeps all the samples for inference, removes the arbitrary threshold choice, and returns a single cluster when there is no real structure. On synthetic data, our approach recovers clusters under more noise and with fewer samples than standard methods. On gene co-expression data from the human
brain, it finds more coherent and more stable clusters than WGCNA, the standard tool in the field. Uniting sparsification and clustering in a single model-selection step provides a principled, noise-resistant way to uncover modular structure in similarity data across fields.
Read the full abstract ▾
Machine Learning (Statistics)
From Universal Approximation Theorem to Tropical Geometry of Multi-Layer Perceptrons
Yi-Shan Chu and Yueh-Cheng Kuo
We revisit the Universal Approximation
Theorem(UAT) through the lens of the tropical geometry of neural networks and introduce a constructive, geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs). Tropical geometry shows that Rectified Linear Unit (ReLU) networks admit decision functions with a combinatorial structure often described as a tropical rational, namely a difference of tropical
polynomials. Focusing on planar binary classification, we design purely sigmoidal MLPs that adhere to the finite-sum format of UAT: a finite linear combination of shifted and scaled sigmoids of affine functions. The resulting models yield decision boundaries that already align with prescribed shapes at initialization and can be refined by standard training if desired. This provides a practical bridge between the tropical perspective and smooth MLPs, enabling interpretable, shape-driven initialization without resorting to ReLU architectures. We focus on the construction and empirical demonstrations in two dimensions; theoretical analysis and higher-dimensional extensions are left for future work.
Read the full abstract ▾
Statistics Applications
Data-driven Calibration Sample Selection and Forecast Combination in Electricity Price Forecasting: An Application of the ARHNN Method
Tomasz Serafin and Weronika Nitka
Calibration sample selection and forecast combination are two simple yet powerful tools used in
forecasting. They can be combined with a variety of models to significantly improve prediction accuracy, at the same time offering easy implementation and low computational complexity. While their effectiveness has been repeatedly confirmed in prior scientific literature, the topic is still underexplored in the field of electricity price
forecasting. In this research article we apply the Autoregressive Hybrid Nearest Neighbors (ARHNN) method to three long-term time series describing the German, Spanish and New England electricity
markets. We show that it outperforms popular literature
benchmarks in terms of forecast accuracy by up to 10%. We also propose two simplified variants of the method, granting a vast decrease in computation time with only minor loss of prediction accuracy. Finally, we compare the forecasts' performance in a battery storage system trading case study. We find that using a forecast-driven strategy can achieve up to 80% of theoretical maximum profits while trading, demonstrating business value in practical applications.
Read the full abstract ▾
Statistical Methodology
Asymptotic distribution of the global clustering coefficient in a random annulus graph
Mingao Yuan
The global clustering coefficient is an effective measure for analyzing and comparing the structures of complex networks. The random annulus graph is a modified version of the well-known Erdős-Rényi random graph. It has been recently proposed in modeling network communities. This paper investigates the asymptotic distribution of the global clustering coefficient in a random annulus graph. It is demonstrated that the standardized global clustering coefficient converges in law to the standard normal distribution. The result is established using the asymptotic theory of degenerate U-statistics with a sample-size dependent kernel. As far as we know, this method is different from established approaches for deriving asymptotic distributions of network statistics. Moreover, we get the explicit expression of the limit of the global clustering coefficient.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores
Zachary Robertson
Pairwise comparisons of large
language models using total variation distance mutual information (TVD-MI) produce binary critic decisions per pair. We show that averaging TVD-MI's binary trials yields centered-probability scores with additive structure suitable for item-response theory (IRT) without nonlinear link functions. Maximum-likelihood approaches to IRT use logistic links, but we find empirically that these transformations introduce
curvature that breaks additivity: across three domains, the identity link yields median curl on raw data of 0.080-0.150 (P95 = [0.474, 0.580]), whereas probit/logit introduce substantially higher violations (median [0.245, 0.588], P95 [0.825, 2.252]). We derive this clipped-linear model from Gini
entropy maximization, yielding a box-constrained least-squares formulation that handles boundary saturation. At 33% coverage, we achieve holdout RMSE $0.117 \pm 0.008$ while preserving
agent rankings (Spearman $\rho = 0.972 \pm 0.015$), three times fewer evaluations than full dense. Judge robustness analysis (GPT-4o-mini vs. Llama3-70b) shows strong agreement in
agent rankings ($\rho = 0.872$) and consistent identity-link advantage. TVD-MI's geometry is best preserved by identity mapping for efficient LLM evaluation, applicable to other bounded-response domains.
Read the full abstract ▾
Statistical Methodology
A formative measurement validation methodology for survey questionnaires
Mark Dominique Dalipe Muñoz
Model misspecification of formative indicators remains a widely documented issue across academic literature, yet scholars lack a clear consensus on pragmatic, prescriptive approaches to manage this gap. This ambiguity forces researchers to rely on psychometric frameworks primarily intended for reflective models, and thus risks misleading findings. This article introduces a Multi-Step Validation Methodology Framework specifically designed for formative constructs in survey-based research. The proposed framework is grounded in an exhaustive literature review and integrates essential pilot diagnostics through descriptive statistics and multicollinearity checks. The methodology provides researchers with the necessary theoretical and structural clarity to finally justify and adhere to appropriate validation techniques that accurately account for the causal nature of the constructs while ensuring high psychometric and statistical integrity.
Read the full abstract ▾
Statistical Methodology
EM Approaches to Nonparametric Estimation for Mixture of Linear Regressions
Andrew Welbaum and Wanli Qiao
In a mixture of linear regression model, the regression coefficients are treated as random vectors that may follow either a continuous or discrete distribution. We propose two Expectation-Maximization (EM)
algorithms to estimate this prior distribution. The first
algorithm solves a kernelized version of the nonparametric maximum likelihood estimation (NPMLE). This method not only recovers continuous prior distributions but also accurately estimates the number of clusters when the prior is discrete. The second
algorithm, designed to approximate the NPMLE, targets prior distributions with a density. It also performs well for discrete priors when combined with a post-processing step. We study the convergence properties of both
algorithms and demonstrate their effectiveness through simulations and applications to real
datasets.
Read the full abstract ▾
Machine Learning (Statistics)
A Geometric Approach to Optimal Experimental Design
Gavin Kerrigan, Christian A. Naesseth and Tom Rainforth
We introduce a novel
geometric framework for optimal experimental design (OED). Traditional OED approaches, such as those based on mutual information, rely explicitly on probability densities, leading to restrictive invariance properties. To address these limitations, we propose the mutual transport dependence (MTD), a measure of statistical dependence grounded in optimal transport theory which provides a
geometric objective for optimizing designs. Unlike conventional approaches, the MTD can be tailored to specific downstream estimation problems by choosing appropriate geometries on the underlying spaces. We demonstrate that our framework produces high-quality designs while offering a flexible alternative to standard information-theoretic techniques.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Provable Unlearning with Gradient Ascent on Two-Layer ReLU Neural Networks
Odelia Melamed, Gilad Yehudai and Gal Vardi
Machine Unlearning aims to remove specific data from trained models, addressing growing
privacy and ethical concerns. We provide a theoretical analysis of a simple and widely used method - gradient ascent - used to reverse the influence of a specific data point without retraining from scratch. Leveraging the implicit bias of gradient descent towards solutions that satisfy the Karush-Kuhn-Tucker (KKT) conditions of a margin maximization problem, we quantify the quality of the unlearned model by evaluating how well it satisfies these conditions w.r.t. the retained data. To formalize this idea, we propose a new success criterion, termed \textbf{$(\epsilon, \delta, \tau)$-successful} unlearning, and show that, for both linear models and two-layer neural networks with high dimensional data, a properly scaled gradient-ascent step satisfies this criterion and yields a model that closely approximates the retrained solution on the retained data. We also show that gradient ascent performs successful unlearning while still preserving generalization in a synthetic Gaussian-mixture setting.
Read the full abstract ▾
Statistics Theory
Regression Model Selection Under General Conditions
Amaze Lusompa
Model selection criteria are one of the most important tools in statistics. Proofs showing a model selection criterion is asymptotically optimal are tailored to the type of model (linear regression,
quantile regression, penalized regression, etc.), the estimation method (linear smoothers, maximum likelihood, generalized method of moments, etc.), the type of data (i.i.d., dependent, high dimensional, etc.), and the type of model selection criterion. Moreover, assumptions are often restrictive and unrealistic making it a slow and winding process for researchers to determine if a model selection criterion is selecting an optimal model. This paper provides general proofs showing asymptotic optimality for a wide range of model selection criteria under general conditions. This paper not only asymptotically justifies model selection criteria for most situations, but it also unifies and extends a range of previously disparate results.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Causal Discovery for Linear DAGs with Dependent Latent Variables via Higher-order Cumulants
Ming Cai, Penggang Gao and Hisayuki Hara
This paper addresses the problem of estimating causal directed acyclic graphs in linear non-Gaussian acyclic models with latent confounders (LvLiNGAM). Existing methods assume mutually independent latent confounders or cannot properly handle models with causal relationships among observed variables. We propose a novel
algorithm that identifies causal DAGs in LvLiNGAM, allowing causal structures among latent variables, among observed variables, and between the two. The proposed method leverages higher-order cumulants of observed data to identify the causal structure. Extensive simulations and experiments with real-world data demonstrate the validity and practical utility of the proposed
algorithm.
Read the full abstract ▾
Statistics Applications
Bayes-ically fair: A Bayesian Ranking of the Olympic Medal Table
Cormac MacDermott, Carl J. Scarrott and John Ferguson
Evaluating a country's sporting success provides insight into its decision-making and infrastructure for developing athletic talent. The Olympic Games serve as a global
benchmark, yet conventional medal rankings can be unduly influenced by population size. We propose a
Bayesian ranking scheme to rank the performance of National Olympic Committees by their "long-run" medals-to-population ratio. The
algorithm aims to mitigate the influence of large populations and reduce the
stochastic fluctuations for smaller nations by applying shrinkage. These long-run rankings provide a more stable and interpretable ordering of national sporting performance across games compared to existing methods.
Read the full abstract ▾