Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Monday 17 November

Statistical Methodology

A Gentle Introduction to Conformal Time Series Forecasting

Conformal prediction is a powerful post-hoc framework for uncertainty quantification that provides distribution-free coverage guarantees. However, these guarantees crucially rely on the assumption of exchangeability. This assumption is fundamentally violated in time series data, where temporal dependence and distributional shifts are pervasive. As a result, classical split-conformal methods may yield prediction intervals that fail to maintain nominal validity. This review unifies recent advances in conformal forecasting methods specifically designed to address nonexchangeable data. We first present a theoretical foundation, deriving finite-sample guarantees for split-conformal prediction under mild weak-dependence conditions. We then survey and classify state-of-the-art approaches that mitigate serial dependence by reweighting calibration data, dynamically updating residual distributions, or adaptively tuning target coverage levels in real time. Finally, we present a comprehensive simulation study that compares these techniques in terms of empirical coverage, interval width, and computational cost, highlighting practical trade-offs and open research directions.
Read the full abstract

All authors M. Stocker, W. Małgorzewicz, M. Fontana, S. Ben Taieb

Statistics Applications

Variance Stabilizing Transformations for Electricity Price Forecasting in Periods of Increased Volatility

Accurate day-ahead electricity price forecasts are critical for power system operation and market participation, yet growing renewable penetration and recent crises have caused unprecedented volatility that challenges standard models. This paper revisits variance stabilizing transformations (VSTs) as a preprocessing tool by introducing a novel parametrization of the asinh transformation, systematically analyzing parameter sensitivity and calibration window size, and explicitly testing performance under volatile market regimes. Using data from Germany, Spain, and France over 2015-2024 with two model classes (NARX and LEAR), we show that VSTs substantially reduce forecast errors, with gains of up to 14.6% for LEAR and 8.7% for NARX relative to untransformed benchmarks. The new parametrized asinh consistently outperforms its standard form, while rolling averaging across transformations delivers the most robust improvements, reducing errors by up to 17.7%. Results demonstrate that VSTs are especially valuable in volatile regimes, making them a powerful tool for enhancing electricity price forecasting in today's power markets.
Read the full abstract

Statistics Theory

Nonparametric Estimation of Joint Entropy via Partitioned Sample-Spacing

We propose a nonparametric estimator of multivariate joint entropy based on partitioned sample spacing (PSS). The method extends univariate spacing ideas to $\mathbb{R}^{d}$ by partitioning into localized cells and aggregating within-cell statistics, with strong consistency guarantees under mild conditions. In benchmarks across diverse distributions, PSS consistently outperforms $k$-nearest neighbor estimators and achieves accuracy competitive with recent normalizing flow-based methods, while requiring no training or auxiliary density modeling. The estimator scales favorably in moderately high dimensions ($d = 10$–$40$) and shows particular robustness to correlated or skewed distributions. These properties position PSS as a practical and reliable alternative to both $k$NN and NF-based entropy estimators, with broad utility in information-theoretic machine learning tasks such as total-correlation estimation, representation learning, and feature selection.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

Asymptotic analysis of rare events in high dimensions

Understanding rare events is critical across domains ranging from signal processing to reliability and structural safety, extreme-weather forecasting, and insurance. The analysis of rare events is a computationally challenging problem, particularly in high dimensions $d$. In this work, we develop the first asymptotic high-dimensional theory of rare events. First, we exploit asymptotic integral methods recently developed by the first author to provide an asymptotic expansion of rare event probabilities. The expansion employs the geometry of the rare event boundary and the local behavior of the log probability density. Generically, the expansion is valid if $d^2\ll\lambda$, where $\lambda$ characterizes the extremity of the event. We prove this condition is necessary by constructing an example in which the first-order remainder is bounded above and below by $d^2/\lambda$. We also provide a nonasymptotic remainder bound which specifies the precise dependence of the remainder on $d$, $\lambda$, the density, and the boundary, and which shows that in certain cases, the condition $d^2\ll \lambda$ can be relaxed. As an application of the theory, we derive asymptotic approximations to rare probabilities under the standard Gaussian density in high dimensions. In the second part of our work, we provide an asymptotic approximation to densities conditional on rare events. This gives rise to simple procedure for approximately sampling conditionally on the rare event using independent Gaussian and exponential random variables.
Read the full abstract

Statistics Applications · Cross-listed from cs.CE

Making Evidence Actionable in Adaptive Learning Closing the Diagnostic Pedagogical Loop

Adaptive learning often diagnoses precisely yet intervenes weakly, producing help that is mistimed or misaligned. This study presents evidence supporting an instructor-governed feedback loop that converts concept-level assessment evidence into vetted microinterventions. The adaptive learning algorithm includes three safeguards: adequacy as a hard guarantee of gap closure, attention as a budgeted limit for time and redundancy, and diversity as protection against overfitting to a single resource. We formulate intervention assignment as a binary integer program with constraints for coverage, time, difficulty windows derived from ability estimates, prerequisites encoded by a concept matrix, and anti-redundancy with diversity. Greedy selection serves low-richness and tight-latency settings, gradient-based relaxation serves rich repositories, and a hybrid switches along a richness-latency frontier. In simulation and in an introductory physics deployment with 1204 students, both solvers achieved full skill coverage for nearly all learners within bounded watch time. The gradient-based method reduced redundant coverage by about 12 percentage points relative to greedy and produced more consistent difficulty alignment, while greedy delivered comparable adequacy at lower computational cost in resource-scarce environments. Slack variables localized missing content and guided targeted curation, sustaining sufficiency across student subgroups. The result is a tractable and auditable controller that closes the diagnostic pedagogical loop and enables equitable, load-aware personalization at the classroom scale.
Read the full abstract

All authors Amirreza Mehrabi, Jason Wade Morphew, Breejha Quezada, N. Sanjay Rebello

Machine Learning (Statistics)

The Shape of Data: Topology Meets Analytics. A Practical Introduction to Topological Analytics and the Stability Index (TSI) in Business

Modern business and economic datasets often exhibit nonlinear, multi-scale structures that traditional linear tools under-represent. Topological Data Analysis (TDA) offers a geometric lens for uncovering robust patterns, such as connected components, loops and voids, across scales. This paper provides an intuitive, figure-driven introduction to persistent homology and a practical, reproducible TDA pipeline for applied analysts. Through comparative case studies in consumer behavior, equity markets (SAX/eSAX vs.\ TDA) and foreign exchange dynamics, we demonstrate how topological features can reveal segmentation patterns and structural relationships beyond classical statistical methods. We discuss methodological choices regarding distance metrics, complex construction and interpretation, and we introduce the Topological Stability Index (TSI), a simple yet interpretable indicator of structural variability derived from persistence lifetimes. We conclude with practical guidelines for TDA implementation, visualization and communication in business and economic analytics.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimate

This paper studies the exponential decay mechanism of the second-moment estimate in Adam. We propose AdamNX and a time-varying second-moment decay rate that gradually weakens the correction applied to the update scale. Under the assumptions used in our analysis, this mechanism makes the updates approach momentum-SGD-like behavior during the training plateau phase. We report results on the image-classification, object-detection, and semantic-segmentation tasks, configurations, and comparators included in this paper. These results do not establish multi-seed statistical effects, flatness, or generalization beyond the reported tasks. Our code is open-sourced at https://github.com/mengzhu0308/AdamNX.
Read the full abstract

Statistics Theory

Random sets from the perspective of metric statistics

Since the seminal work by Beresteanu and Molinari(2008), the random set theory and related inference methods have been widely applied in partially identified econometric models. Meanwhile, there is an emerging field in statistics for studying random objects in metric spaces, called metric statistics. This paper clarifies a relationship between two fundamental concepts in these literatures, the Aumann and Fréchet means, and presents some applications of metric statistics to econometric problems involving random sets.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression

While data scaling laws of large language models (LLMs) have been widely examined in the one-pass regime with massive corpora, their form under limited data and repeated epochs remains largely unexplored. This paper presents a theoretical analysis of how a common workaround, training for multiple epochs on the same dataset, reshapes the data scaling laws in linear regression. Concretely, we ask: to match the performance of training on a dataset of size $N$ for $K$ epochs, how much larger must a dataset be if the model is trained for only one pass? We quantify this using the effective reuse rate of the data, $E(K, N)$, which we define as the multiplicative factor by which the dataset must grow under one-pass training to achieve the same test loss as $K$-epoch training. Our analysis precisely characterizes the scaling behavior of $E(K, N)$ for SGD in linear regression under either strong convexity or Zipf-distributed data: (1) When $K$ is small, we prove that $E(K, N) \approx K$, indicating that every new epoch yields a linear gain; (2) As $K$ increases, $E(K, N)$ plateaus at a problem-dependent value that grows with $N$ ($\Theta(\log N)$ for the strongly-convex case), implying that larger datasets can be repeated more times before the marginal benefit vanishes. These theoretical findings point out a neglected factor in a recent empirical study (Muennighoff et al. (2023)), which claimed that training LLMs for up to $4$ epochs results in negligible loss differences compared to using fresh data at each step, i.e., $E(K, N) \approx K$ for $K \le 4$ in our notation. Supported by further empirical validation with LLMs, our results reveal that the maximum $K$ value for which $E(K, N) \approx K$ in fact depends on the data size and distribution, and underscore the need to explicitly model both factors in future studies of scaling laws with data reuse.
Read the full abstract

All authors Tingkai Yan, Haodong Wen, Binghui Li, Kairong Luo, Wenguang Chen, Kaifeng Lyu

Machine Learning (Statistics) · Cross-listed from cs.LG

Fast and Robust Simulation-Based Inference With Optimization Monte Carlo

Bayesian parameter inference for complex stochastic simulators is challenging due to intractable likelihood functions. Existing simulation-based inference methods often require large number of simulations and become costly to use in high-dimensional parameter spaces or in problems with partially uninformative outputs. We propose a new method for differentiable simulators that delivers accurate posterior inference with substantially reduced runtimes. Building on the Optimization Monte Carlo framework, our approach reformulates inference for stochastic simulators in terms of deterministic optimization problems. Gradient-based methods are then applied to efficiently navigate toward high-density posterior regions and avoid wasteful simulations in low-probability areas. A JAX-based implementation further enhances the performance through vectorization of key method components. Extensive experiments, including high-dimensional parameter spaces, uninformative outputs, multiple observations and multimodal posteriors show that our method consistently matches, and often exceeds, the accuracy of state-of-the-art approaches, while reducing the runtime by a substantial margin.
Read the full abstract

Other Statistics · Cross-listed from q-fin.GN

CBDC Stress Test in a Dual-Currency Setting

This study explores the potential impact of introducing a Central Bank Digital Currency (CBDC) on financial stability in an emerging dual-currency economy (Romania), where the domestic currency (RON) coexists with the euro. It develops an integrated analytical framework combining econometrics, machine learning, and behavioural modelling. CBDC adoption probabilities are estimated using XGBoost and logistic regression models trained on behavioural and macro-financial indicators rather than survey data. Liquidity stress simulations assess how banks would respond to deposit withdrawals resulting from CBDC adoption, while VAR, MSVAR, and SVAR models capture the macro-financial transmission of liquidity shocks into credit contraction and changes in monetary conditions. The findings indicate that CBDC uptake (co-circulating Digital RON and Digital EUR) would be moderate at issuance, amounting to around EUR 1 billion, primarily driven by digital readiness and trust in the central bank. The study concludes that a non-remunerated, capped CBDC, designed primarily as a means of payment rather than a store of value, can be introduced without compromising financial stability. In dual currency economies, differentiated holding limits for domestic and foreign digital currencies (e.g., Digital RON versus Digital Euro) are crucial to prevent uncontrolled euroisation and preserve monetary sovereignty. A prudent design with moderate caps, non remuneration, and macroprudential coordination can transform CBDC into a digital liquidity buffer and a complementary monetary policy instrument that enhances resilience and inclusion rather than destabilising the financial system.
Read the full abstract

Statistics Theory

Local asymptotic normality for discretely observed McKean-Vlasov diffusions

We study the local asymptotic normality (LAN) property for the likelihood function associated with discretely observed $d$-dimensional McKean-Vlasov stochastic differential equations over a fixed time interval. The model involves a joint parameter in both the drift and diffusion coefficients, introducing challenges due to its dependence on the process distribution. We derive a stochastic expansion of the log-likelihood ratio using Malliavin calculus techniques and establish the LAN property under appropriate conditions. The main technical challenge arises from the implicit nature of the transition densities, which we address through integration by parts and Gaussian-type bounds. This work extends existing LAN results for interacting particle systems to the mean-field regime, contributing to statistical inference in non-linear stochastic models
Read the full abstract

Statistical Methodology

Novel Tau-Informed Initialization for Maximum Likelihood Estimation of Copulas with Discrete Margins

We study Gaussian-copula models with discrete margins, with primary emphasis on low-count (Poisson) data. Our goal is exact yet computationally efficient maximum likelihood (ML) estimation in regimes where many observations contain small counts, which imperils both identifiability and numerical stability. We develop three novel Kendall's tau-based approaches for initialization tailored to discrete margins in the low-count regime and embed it within an inference functions for margins (IFM) inspired start. We present three practical initializers (exact, low-intensity approximation, and a transformation-based approach) that substantially reduce the number of ML iterations and improve convergence. For the ML stage, we use an unconstrained reparameterization of the model's parameters using the log and spherical-Cholesky and compute exact rectangle probabilities. Analytical score functions are supplied throughout to stabilize Newton-type optimization. A simulation study across dimensions, dependence levels, and intensity regimes shows that the proposed initialization combined with exact ML achieves lower root-mean-squared error, lower bias and faster computation times than the alternative procedures. The methodology provides a pragmatic path to retain the statistical guarantees of ML (consistency, asymptotic normality, efficiency under correct specification) while remaining tractable for moderate- to high-dimensional discrete data. We conclude with guidance on initializer choice and discuss extensions to alternative correlation structures and different margins.
Read the full abstract

Statistics Applications

TacEleven: generative tactic discovery for football open play

Creating offensive advantages during open play is fundamental to football success. However, due to the highly dynamic and long-sequence nature of open play, the potential tactic space grows exponentially as the sequence progresses, making automated tactic discovery extremely challenging. To address this, we propose TacEleven, a generative framework for football open-play tactic discovery developed in close collaboration with domain experts from AJ Auxerre, designed to assist coaches and analysts in tactical decision-making. TacEleven consists of two core components: a language-controlled tactical generator that produces diverse tactical proposals, and a multimodal large language model-based tactical critic that selects the optimal proposal aligned with a high-level stylistic tactical instruction. The two components enables rapid exploration of tactical proposals and discovery of alternative open-play offensive tactics. We evaluate TacEleven across three tasks with progressive tactical complexity: counterfactual exploration, single-step discovery, and multi-step discovery, through both quantitative metrics and a questionnaire-based qualitative assessment. The results show that the TacEleven-discovered tactics exhibit strong realism and tactical creativity, with 52.50% of the multi-step tactical alternatives rated adoptable in real-world elite football scenarios, highlighting the framework's ability to rapidly generate numerous high-quality tactics for complex long-sequence open-play situations. TacEleven demonstrates the potential of creatively leveraging domain data and generative models to advance tactical analysis in sports.
Read the full abstract

All authors Siyao Zhao, Hao Ma, Zhiqiang Pu, Jingjing Huang, Yi Pan, Shijie Wang, Zhi Ming

Statistical Methodology

Scalable approximation of the transformation-free linear simplicial-simplicial regression via constrained iterative reweighted least squares

Simplicia-simplicial regression concerns statistical modeling scenarios in which both the predictors and the responses are vectors constrained to lie on the simplex. \cite{fiksel2022} introduced a transformation-free linear regression framework for this setting, wherein the regression coefficients are estimated by minimizing the Kullback-Leibler divergence between the observed and fitted compositions, using an expectation-maximization (EM) algorithm for optimization. In this work, we reformulate the problem as a constrained logistic regression model, in line with the methodological perspective of \cite{tsagris2025}, and we obtain parameter estimates via constrained iteratively reweighted least squares. Simulation results indicate that the proposed procedure substantially improves computational efficiency-yielding speed gains ranging from $6\times--326\times$-while providing estimates that closely approximate those obtained from the EM-based approach.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Counterfactual Explainable AI (XAI) Method for Deep Learning-Based Multivariate Time Series Classification

Recent advances in deep learning have improved multivariate time series (MTS) classification and regression by capturing complex patterns, but their lack of transparency hinders decision-making. Explainable AI (XAI) methods offer partial insights, yet often fall short of conveying the full decision space. Counterfactual Explanations (CE) provide a promising alternative, but current approaches typically prioritize either accuracy, proximity or sparsity – rarely all – limiting their practical value. To address this, we propose CONFETTI, a novel multi-objective CE method for MTS. CONFETTI identifies key MTS subsequences, locates a counterfactual target, and optimally modifies the time series to balance prediction confidence, proximity and sparsity. This method provides actionable insights with minimal changes, improving interpretability, and decision support. CONFETTI is evaluated on seven MTS datasets from the UEA archive, demonstrating its effectiveness in various domains. CONFETTI consistently outperforms state-of-the-art CE methods in its optimization objectives, and in six other metrics from the literature, achieving $\geq10\%$ higher confidence while improving sparsity in $\geq40\%$.
Read the full abstract

All authors Alan G. Paredes Cetina, Kaouther Benguessoum, Raoni Lourenço, Sylvain Kubler

Machine Learning (Statistics) · Cross-listed from cs.LG

Laplace Learning in Wasserstein Space

The manifold hypothesis posits that high-dimensional data typically resides on low-dimensional sub spaces. In this paper, we assume manifold hypothesis to investigate graph-based semi-supervised learning methods. In particular, we examine Laplace Learning in the Wasserstein space, extending the classical notion of graph-based semi-supervised learning algorithms from finite-dimensional Euclidean spaces to an infinite-dimensional setting. To achieve this, we prove variational convergence of a discrete graph p- Dirichlet energy to its continuum counterpart. In addition, we characterize the Laplace-Beltrami operator on asubmanifold of the Wasserstein space. Finally, we validate the proposed theoretical framework through numerical experiments conducted on benchmark datasets, demonstrating the consistency of our classification performance in high-dimensional settings.
Read the full abstract

All authors Mary Chriselda Antony Oliver, Michael Roberts, Carola-Bibiane Schönlieb, Matthew Thorpe

Machine Learning (Statistics)

Likelihood-guided Regularization in Attention Based Models

The transformer architecture has demonstrated strong performance in classification tasks involving structured and high-dimensional data. However, its success often hinges on large- scale training data and careful regularization to prevent overfitting. In this paper, we intro- duce a novel likelihood-guided variational Ising-based regularization framework for Vision Transformers (ViTs), which simultaneously enhances model generalization and dynamically prunes redundant parameters. The proposed variational Ising-based regularization approach leverages Bayesian sparsification techniques to impose structured sparsity on model weights, allowing for adaptive architecture search during training. Unlike traditional dropout-based methods, which enforce fixed sparsity patterns, the variational Ising-based regularization method learns task-adaptive regularization, improving both efficiency and interpretability. We evaluate our approach on benchmark vision datasets, including MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, demonstrating improved generalization under sparse, complex data and allowing for principled uncertainty quantification on both weights and selection parameters. Additionally, we show that the Ising regularizer leads to better-calibrated probability estimates and structured feature selection through uncertainty-aware attention mechanisms. Our results highlight the effectiveness of structured Bayesian sparsification in enhancing transformer-based architectures, offering a principled alternative to standard regularization techniques.
Read the full abstract

Statistical Methodology

Modeling group heterogeneity in spatio-temporal data via physics-informed semiparametric regression

In this work we propose a novel approach for modeling spatio-temporal data characterized by group structures. In particular, we extend classical mixed effect regression models by introducing a space-time nonparametric component, regularized through a partial differential equation, to embed the physical dynamics of the underlying process, while random effects capture latent variability associated with the group structure present in the data. We propose a two-step procedure to estimate the fixed and random components of the model, relying on a functional version of the Iterative Reweighted Least Squares algorithm. We investigate the asymptotic properties of both fixed and random components, and we assess the performance of the proposed model through a simulation study, comparing it with state-of-the-art alternatives from the literature. The proposed methodology is finally applied to the study of hourly nitrogen dioxide concentration data in Lombardy (Italy), using random effects to account for measurement heterogeneity across monitoring stations equipped with different sensor technologies.
Read the full abstract

All authors Marco F. De Sanctis, Eleonora Arnone, Francesca Ieva, Laura M. Sangalli

Statistics Theory

Asymptotic confidence bands for centered purely random forests

In a multivariate nonparametric regression setting we construct explicit asymptotic uniform confidence bands for centered purely random forests. Since the most popular example in this class of random forests, namely the uniformly centered purely random forests, is well known to suffer from suboptimal rates, we propose a new type of purely random forests, called the Ehrenfest centered purely random forests, which achieve minimax optimal rates. Our main confidence band theorem applies to both random forests. The proof is based on an interpretation of random forests as generalized U-Statistics together with a Gaussian approximation of the supremum of empirical processes. Our theoretical findings are illustrated in simulation examples.
Read the full abstract

Statistics Theory

Convergence rate of randomized midpoint Langevin Monte Carlo

The randomized midpoint Langevin Monte Carlo (RLMC), introduced by Shen and Lee (2019), is a variant of classical Unadjusted Langevin Algorithm. It was shown in the literature that the RLMC is an efficient algorithm for approximating high-dimensional probability distribution $\pi$. In this paper, we establish the exponential ergodicity of RLMC with constant step-size. Moreover, we design a dereasing-step size RLMC and provide its convergence rate in terms of a functional class distance.
Read the full abstract

Statistics Theory

Non-asymptotic Analysis of Poisson randomized midpoint Langevin Monte Carlo

The task of sampling from a high-dimensional distribution $\pi$ on $\R^d$ is a fundamental algorithmic problem with applications throughout statistics, engineering, and the sciences. Consider the Langevin diffusion on $\R^d$ \begin{align*} \dif X_t=-\nabla U(X_t)dt+\sqrt{2}dB_t, \end{align*} under mild conditions, it admits $\pi(\dif x)\propto \exp(-U(x))\dif x$ as its unique stationary distribution. Recently, Kandasamy and Nagaraj (2024) introduced a stochastic algorithm called Poisson Randomized Midpoint Langevin Monte Carlo (PRLMC) to enhance the rate of convergence towards the target distribution $\pi$. In this paper, we first show that under mild conditions, the PRLMC, as a Markov chain, admits a unique stationary distribution $\pi_\eta$ ($\eta$ is the step size) and obtain the convergence rate of PRLMC to $\pi_\eta$ in total variation distance. Then we establish a sharp error bound between $\pi_\eta$ and $\pi$ under the 2-Wasserstein distance. Finally, we propose a decreasing-step size version of PRLMC and provide its convergence rate to $\pi$ which is nearly optimal.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Generalization Bounds for Semi-supervised Matrix Completion with Distributional Side Information

We study a matrix completion problem where both the ground truth $R$ matrix and the unknown sampling distribution $P$ over observed entries are low-rank matrices, and share a common subspace. We assume that a large amount $M$ of unlabeled data drawn from the sampling distribution $P$ is available, together with a small amount $N$ of labeled data drawn from the same distribution and noisy estimates of the corresponding ground truth entries. This setting is inspired by recommender systems scenarios where the unlabeled data corresponds to `implicit feedback' (consisting in interactions such as purchase, click, etc. ) and the labeled data corresponds to the `explicit feedback', consisting of interactions where the user has given an explicit rating to the item. Leveraging powerful results from the theory of low-rank subspace recovery, together with classic generalization bounds for matrix completion models, we show error bounds consisting of a sum of two error terms scaling as $\widetilde{O}\left(\sqrt{\frac{nd}{M}}\right)$ and $\widetilde{O}\left(\sqrt{\frac{dr}{N}}\right)$ respectively, where $d$ is the rank of $P$ and $r$ is the rank of $M$. In synthetic experiments, we confirm that the true generalization error naturally splits into independent error terms corresponding to the estimations of $P$ and and the ground truth matrix $\ground$ respectively. In real-life experiments on Douban and MovieLens with most explicit ratings removed, we demonstrate that the method can outperform baselines relying only on the explicit ratings, demonstrating that our assumptions provide a valid toy theoretical setting to study the interaction between explicit and implicit feedbacks in recommender systems.
Read the full abstract

Statistics Theory

Reconciliation of Bayes and empirical Bayes interval estimation with application to small area estimation

Multi-level normal hierarchical models, also interpreted as mixed effects models, play an important role in developing statistical theory in multi-parameter estimation for a wide range of applications, including small area estimation. In this article, we propose a novel reconciliation framework of the empirical Bayes (EB) and hierarchical Bayes approaches for interval estimation of random effects under a two-level normal model. Our framework shows that a second-order efficient empirical Bayes confidence interval, with EB coverage error of order $O(m^{-3/2})$, $m$ being the number of areas in the area-level model, can also be viewed as a credible interval whose expected posterior coverage is close to the nominal level, provided a carefully chosen prior, referred to as a "matching prior", is placed on the hyperparameters. While existing literature has examined matching priors that reconcile frequentist and Bayesian inference in various settings, this paper is the first to study matching priors with the goal of interval estimation of random effects in a two-level model. We obtain a matching prior on the variance component that achieves a proper posterior under mild regularity conditions. The theoretical results in the paper are corroborated through a Monte Carlo simulation study and real data analysis.
Read the full abstract

Machine Learning (Statistics)

Recovering Geodesic Distances from Noisy Observations

Let $(M,d,\mu)$ be a nice geodesic probability space (such as a manifold) and $X=\{x_i\}_{i\in [N]} \subset M$ a finite $\mu$-i.i.d. sample. Suppose we observe noisy distance random variables $d'(x_i,x_j)$. When can the true distances be recovered to small additive error? This question has attracted interest in two disparate fields: latent-space graph models and boundary-value inverse problems. We first show that, taking $N \asymp \epsilon^{-2d-2}|\log\epsilon|$ and using runtime $O(N^{2.5+o_d (1)})$, we can recover all distances in $X$ to additive error $O(\epsilon|\log\epsilon|)$ with high probability, assuming $\mathbb{E} d'(x_i, x_j) = F(d(x_i,x_j))$ for an unknown, increasing bilipschitz $F$ and with the $d'(x_i,x_j)$ allowed to be arbitrarily heterogeneous in their distribution as long as they are bounded in subgaussian norm. This generalizes existing work, which required $F$ to be known and made strict assumptions on the distribution of $d'(x_i, x_j)$. Achieving this generalization required a novel “ruler-building” procedure to learn $F$. We extend our results to a missing data case where we randomly lose access to noisy distances. Our second result uses ideas from the first to tackle a partial observability case: $X = Y \cup Z$ with $Z$ sampled from $M$ and $Y$ sampled from an observer subset satisfying visibility conditions that model the assumptions made on the boundary in boundary-value inverse problems. In this setting, we recover all pairwise distances in $Z$ to additive error $O(\epsilon)$ with $|Z| \asymp \epsilon^{-2d-2}|\log\epsilon|$, $|Y| \asymp \epsilon^{-\max\{d_\nu,4\}}$ with $d_\nu$ determined by the volume-growth properties of the probability measure on the observer set, and runtime $O(|Z|^3)$. This is, to our knowledge, the first quantitative guarantee for this observer/observed split under noise, which complements existing stability results for boundary data.
Read the full abstract

Statistical Methodology

Bayesian Variable Selection on Small Sample Trial Data via Adaptive Posterior-Informed Shrinkage Prior

Identifying variables associated with clinical endpoints is of much interest in clinical trials. With the rapid growth of cell and gene therapy (CGT) and therapeutics for ultra-rare diseases, there is an urgent need for statistical methods that can detect meaningful associations under severe sample-size constraints. Motivated by data-borrowing strategies for historical controls, we propose the Adaptive Posterior-Informed Shrinkage Prior (APSP), a Bayesian approach that adaptively borrows information from external sources to improve variable-selection efficiency while preserving robustness across plausible scenarios. APSP builds upon existing Bayesian data borrowing frameworks, incorporating data-driven adaptive information selection, structure of mixture shrinkage informative priors and decision making with empirical null to enhance variable selection performances under small sample size. Extensive simulations show that APSP attains better efficiency relative to traditional and popular data-borrowing and Bayesian variable-selection methods while maintaining robustness under linear relationships. We further applied APSP to identify variables associated with peak C-peptide at Day 75 from the Clinical Islet Transplantation (CIT) Consortium study CIT06 by borrowing information from the study CIT07.
Read the full abstract

All authors Lingxuan Kong, Yumin Zhang, Chenkun Wang, Yaoyuan Vincent Tan

Statistics Applications

Scalable Vision-Guided Crop Yield Estimation

Precise estimation and uncertainty quantification for average crop yields are critical for agricultural monitoring and decision making. Existing data collection methods, such as crop cuts in randomly sampled fields at harvest time, are relatively time-consuming. Thus, we propose an approach based on prediction-powered inference (PPI) to supplement these crop cuts with less time-consuming field photos. After training a computer vision model to predict the ground truth crop cut yields from the photos, we learn a “control function" that recalibrates these predictions with the spatial coordinates of each field. This enables fields with photos but not crop cuts to be leveraged to improve the precision of zone-wide average yield estimates. Our control function is learned by training on a dataset of nearly 20,000 real crop cuts and photos of rice and maize fields in sub-Saharan Africa. To improve precision, we pool training observations across different zones within the same first-level subdivision of each country. Our final PPI-based point estimates of the average yield are provably asymptotically unbiased and cannot increase the asymptotic variance beyond that of the natural baseline estimator – the sample average of the crop cuts – as the number of fields grows. We also propose a novel bias-corrected and accelerated (BCa) bootstrap to construct accompanying confidence intervals. Even in zones with as few as 20 fields, the point estimates show significant empirical improvement over the baseline, increasing the effective sample size by as much as 73% for rice and by 12-23% for maize. The confidence intervals are accordingly shorter at minimal cost to empirical finite-sample coverage. This demonstrates the potential for relatively low-cost images to make area-based crop insurance more affordable and thus spur investment into sustainable agricultural practices.
Read the full abstract

All authors Harrison H. Li, Medhanie Irgau, Nabil Janmohamed, Karen Solveig Rieckmann, David B. Lobell

Statistics Theory

Transfer Learning and Locally Linear Regression for Locally Stationary Time Series

This paper investigates locally linear regression for locally stationary time series and develops theoretical results for locally linear smoothing and transfer learning. Existing analyses have focused on local constant estimators and given samples, leaving the principles of transferring knowledge from auxiliary sources across heterogeneous time-varying domains insufficiently established. We derive uniform convergence for multivariate locally linear estimators under strong mixing. The resulting error expansion decomposes stochastic variation, smoothing bias, and a term induced by local stationarity. This additional term, originating from the locally stationary structure, has smaller order than in the Nadaraya-Watson benchmark, explaining the improved local linear performance. Building on these results, we propose bias-corrected transfer learned estimators that connect a sparsely observed series with densely observed related sources through a smoothly varying bias function defined over rescaled time and covariates. An additional refinement shows how local temporal adjustment of this bias enhances stability and enables efficient information borrowing across domains. Simulation studies and an empirical analysis of international fuel prices support the theoretical predictions and demonstrate the practical advantages of transfer learning.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

On the Information Processing of One-Dimensional Wasserstein Distances with Finite Samples

Leveraging the Wasserstein distance – a summation of sample-wise transport distances in data space – is advantageous in many applications for measuring support differences between two underlying density functions. However, when supports significantly overlap while densities exhibit substantial pointwise differences, it remains unclear whether and how this transport information can accurately identify these differences, particularly their analytic characterization in finite-sample settings. We address this issue by conducting an analysis of the information processing capabilities of the one-dimensional Wasserstein distance with finite samples. By utilizing the Poisson process and isolating the rate factor, we demonstrate the capability of capturing the pointwise density difference with Wasserstein distances and how this information harmonizes with support differences. The analyzed properties are confirmed using neural spike train decoding and amino acid contact frequency data. The results reveal that the one-dimensional Wasserstein distance highlights meaningful density differences related to both rate and support.
Read the full abstract

All authors Cheongjae Jang, Jonghyun Won, Soyeon Jun, Chun Kee Chung, Keehyoung Joo, Yung-Kyun Noh

Statistical Methodology · Cross-listed from econ.EM

Identification-aware Markov chain Monte Carlo

Leaving posterior sensitivity concerns aside, non-identifiability of the parameters does not raise a difficulty for Bayesian inference as far as the posterior is proper, but multi-modality or flat regions of the posterior induced by the lack of identification leaves a challenge for modern Bayesian computation. Sampling methods often struggle with slow or non-convergence when dealing with multiple modes or flat regions of the target distributions. This paper develops a novel Markov chain Monte Carlo (MCMC) approach for non-identified models, leveraging the knowledge of observationally equivalent sets of parameters, and highlights an important role that identification plays in modern Bayesian analysis. We show that our identification-aware proposal eliminates mode entrapment, achieving a convergence rate uniformly bounded away from zero, in sharp contrast to the exponentially decaying rates characterizing standard Random Walk Metropolis and Hamiltonian Monte Carlo. Simulation studies show its superior performance compared to other popular computational methods including Hamiltonian Monte Carlo and sequential Monte Carlo. We also demonstrate that our method uncovers non-trivial modes in the target distribution in a structural vector moving-average (SVMA) application.
Read the full abstract