Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Monday 8 December

Statistics Applications · Cross-listed from eess.SY

Cabin Layout, Seat Density, and Passenger Segmentation in Air Transport: Implications for Prices, Ancillary Revenues, and Efficiency

This study investigates how the layout and density of seats in aircraft cabins influence the pricing of airline tickets on domestic flights. The analysis is based on microdata from boarding passes linked to face-to-face interviews with passengers, allowing us to relate the price paid to the location on the aircraft seat map, as well as market characteristics and flight operations. Econometric models were estimated using the Post-Double-Selection LASSO (PDS-LASSO) procedure, which selects numerous controls for unobservable factors linked to commercial and operational aspects, thus enabling better identification of the effect of variables such as advance purchase, reason for travel, fuel price, market structure, and load factor, among others. The results suggest that a higher density of seat rows is associated with lower prices, reflecting economies of scale with the increase in aircraft size and gains in operational efficiency. An unexpected result was also obtained: in situations where there was no seat selection fee, passengers with more expensive tickets were often allocated middle seats due to purchasing at short notice, when the side alternatives were no longer available. This behavior helps explain the economic logic behind one of the main ancillary revenues of airlines. In addition to quantitative analysis, the study incorporates an exploratory approach to innovative cabin concepts and their possible effects on density and comfort on board.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

LUNA: Linear Universal Neural Attention with Generalization Guarantees

Scaling attention faces a critical bottleneck: the $\mathcal{O}(n^2)$ quadratic computational cost of softmax attention, which limits its application in long-sequence domains. While linear attention mechanisms reduce this cost to $\mathcal{O}(n)$, they typically rely on fixed random feature maps, such as random Fourier features or hand-crafted functions. This reliance on static, data-agnostic kernels creates a fundamental trade-off, forcing practitioners to sacrifice significant model accuracy for computational efficiency. We introduce LUNA, a kernelized linear attention mechanism that eliminates this trade-off, retaining linear cost while matching and surpassing the accuracy of quadratic attention. LUNA is built on the key insight that the kernel feature map itself should be learned rather than fixed a priori. By parameterizing the kernel, LUNA learns a feature basis tailored to the specific data and task, overcoming the expressive limitations of fixed-feature methods. Luna implements this with a learnable feature map that induces a positive-definite kernel and admits a streaming form, yielding linear time and memory scaling in the sequence length. Empirical evaluations validate our approach across diverse settings. On the Long Range Arena (LRA), Luna achieves state-of-the-art average accuracy among efficient Transformers under compute parity, using the same parameter count, training steps, and approximate FLOPs. Luna also excels at post-hoc conversion: replacing softmax in fine-tuned BERT and ViT-B/16 checkpoints and briefly fine-tuning recovers most of the original performance, substantially outperforming fixed linearizations.
Read the full abstract

All authors Ashkan Shahbazi, Ping He, Ali Abbasi, Yikun Bai, Xinran Liu, Elaheh Akbari, Darian Salehi, Navid NaderiAlizadeh, Soheil Kolouri

Statistics Applications

Defining 3-dimensional marine provinces with phytoplankton compositions

Marine provinces rarely include fine-resolution biological data, and are often defined spatially across only latitude and longitude. Therefore, we aimed to determine how phytoplankton distributions define marine provinces across 3-dimensions (i.e., latitude, longitude, and depth). To do this, we developed a new algorithm called bioprovince which can be applied to compositional biological data. The algorithm first clusters compositional samples to identify spatially coherent groups of samples, then makes flexible province predictions in the broader 3d spatial grid based on environmental similarity. We applied bioprovince to phytoplankton Amplicon Sequencing Variants (ASVs) from five, depth-resolved ocean transects spanning north-south in the Pacific Ocean. In the surface layer of the ocean, our method agreed well with traditional Longhurst provinces. In some cases, the method revealed that with more granular taxonomic resolution afforded by ASVs, traditional Longhurst provinces were divided into smaller zones. Also, one of the major advances of this method is its ability to incorporate a third dimension, depth. Indeed, our analysis found significant depth-wise partitions throughout the Pacific with remarkable agreement in the equatorial region with the base of the euphotic zone. Our algorithm's ability to delineate 3-dimensional bioprovinces will enable scientists to discover new ecological interpretations of marine phytoplankton ecology and biogeography. Furthermore, as compositional biological data inherently exists in three spatial dimensions in nature, bioprovince is broadly applicable beyond marine plankton, offering a more holistic perspective on biological provinces across diverse environments.
Read the full abstract

All authors Rafael Catoia Pulgrossi, Nathan L R Williams, Yubin Raut, Jed Fuhrman, Sangwon Hyun

Statistical Computation · Cross-listed from cs.IT

Expectations in Expectation Propagation

Expectation Propagation (EP) is a widely used message-passing algorithm that decomposes a global inference problem into multiple local ones. It approximates marginal distributions (beliefs) using intermediate functions (messages). While beliefs must be proper probability distributions that integrate to one, messages may have infinite integral values. In Gaussian-projected EP, such messages take a Gaussian form and appear as if they have "negative" variances. Although allowed within the EP framework, these negative-variance messages can impede algorithmic progress. In this paper, we investigate EP in linear models and analyze the relationship between the corresponding beliefs. Based on the analysis, we propose both non-persistent and persistent approaches that prevent the algorithm from being blocked by messages with infinite integral values. Furthermore, by examining the relationship between the EP messages in linear models, we develop an additional approach that avoids the occurrence of messages with infinite integral values.
Read the full abstract

Machine Learning (Statistics)

Provable Diffusion Posterior Sampling for Bayesian Inversion

We propose a novel diffusion-based posterior sampling method within a plug-and-play framework. Our approach constructs a probability transport from an easy-to-sample distribution to the target posterior via a diffusion process. To initialize the sampler efficiently, we introduce a warm-start strategy for the particles. The posterior score is then approximated using a Monte Carlo estimator in which samples are generated via Langevin dynamics, avoiding the heuristic approximations prevalent in prior work. The score function driving the Langevin dynamics is learned from data, enabling the model to capture rich structural features of the underlying prior. We also establish non-asymptotic error bounds in Wasserstein-2 distance guaranteeing convergence of the proposed method even for complex, multimodal posterior distributions. We corroborate our theoretical findings with numerical experiments demonstrating the effectiveness of the method across a variety of inverse problems.
Read the full abstract

All authors Jinyuan Chang, Chenguang Duan, Yuling Jiao, Ruoxuan Li, Jerry Zhijian Yang, Cheng Yuan

Statistics Theory

The limit joint distributions of some statistics used in testing the quality of random number generators

The limit joint distribution of statistics that are generalizations of some statistics from the NIST STS, TestU01, and other packages is found under the following hypotheses $H_0$ and $H_1$. Hypothesis $H_0$ states that the tested sequence is a sequence of independent random vectors with a known distribution, and the simple alternative hypothesis $H_1$ converges in some sense to $H_0$ with increasing sample size. In addition, an analogue of the Berry-Esseen inequality is obtained for the statistics under consideration, and conditions for their asymptotic independence are found.
Read the full abstract

Statistical Methodology

Bayesian Semiparametric Joint Dynamic Model for Multitype Recurrent Events and a Terminal Event

In many biomedical research, recurrent events such as myocardial infraction, stroke, and heart failure often result in a terminal outcome such as death. Understanding the relationship among the multi-type recurrent events and terminal event is essential for developing interventions to prolong the terminal event such as death. This study introduces a Bayesian semiparametric joint dynamic model for type-specific hazards that quantifies how the type-specific event history dynamically changes the intensities of each recurrent event type and the terminal event over calendar time. The framework jointly captures unmeasured heterogeneity through a shared frailty term, cumulative effects of past recurrent events on themselves and terminal events, and the effects of covariates. Gamma process priors (GPP) are used as a nonparametric prior for the baseline cumulative hazard function (CHF) and parametric priors for covariates and frailty. For a more accurate risk assessment, this model provides an analytical closed-form estimator of cumulative hazard functions (CHF) and frailties. The Breslow-Aalen-type estimators of CHFs are special cases of our estimators when the precision parameters are set to zero. We evaluate the performance of the model through extensive simulations and apply the method to the Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT). The analysis offers a practical past event effect based risk assessment for acute and chronic cardiovascular recurrent events with a terminal end point death and provides new information to support the prevention and treatment of cardiovascular disease to clinicians.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Provable Long-Range Benefits of Next-Token Prediction

Why do modern language models, trained to do well on next-word prediction, appear to generate coherent documents and capture long-range structure? Here we show that next-token prediction is provably powerful for learning longer-range structure, even with common neural network architectures. Specifically, we prove that optimizing next-token prediction over a Recurrent Neural Network (RNN) yields a model that closely approximates the training distribution: for held-out documents sampled from the training distribution, no algorithm of bounded description length limited to examining the next $k$ tokens, for any $k$, can distinguish between $k$ consecutive tokens of such documents and $k$ tokens generated by the learned language model following the same prefix. We provide polynomial bounds (in $k$, independent of the document length) on the model size needed to achieve such $k$-token indistinguishability, offering a complexity-theoretic explanation for the long-range coherence observed in practice.
Read the full abstract

Machine Learning (Statistics)

Distribution-informed Online Conformal Prediction

Conformal prediction provides a pivotal and flexible technique for uncertainty quantification by constructing prediction sets with a predefined coverage rate. Many online conformal prediction methods have been developed to address data distribution shifts in fully adversarial environments, resulting in overly conservative prediction sets. We propose Conformal Optimistic Prediction (COP), an online conformal prediction algorithm incorporating underlying data pattern into the update rule. Through estimated cumulative distribution function of non-conformity scores, COP produces tighter prediction sets when predictable pattern exists, while retaining valid coverage guarantees even when estimates are inaccurate. We establish a joint bound on coverage and regret, which further confirms the validity of our approach. We also prove that COP achieves distribution-free, finite-sample coverage under arbitrary learning rates and can converge when scores are $i.i.d.$. The experimental results also show that COP can achieve valid coverage and construct shorter prediction intervals than other baselines.
Read the full abstract

All authors Dongjian Hu, Junxi Wu, Shu-Tao Xia, Changliang Zou

Machine Learning (Statistics)

Physics-Informed Neural Networks for Joint Source and Parameter Estimation in Advection-Diffusion Equations

Recent studies have demonstrated the success of deep learning in solving forward and inverse problems in engineering and scientific computing domains, such as physics-informed neural networks (PINNs). Source inversion problems under sparse measurements for parabolic partial differential equations (PDEs) are particularly challenging to solve using PINNs, due to their severe ill-posedness and the multiple unknowns involved including the source function and the PDE parameters. Although the neural tangent kernel (NTK) of PINNs has been widely used in forward problems involving a single neural network, its extension to inverse problems involving multiple neural networks remains less explored. In this work, we propose a weighted adaptive approach based on the NTK of PINNS including multiple separate networks representing the solution, the unknown source, and the PDE parameters. The key idea behind our methodology is to simultaneously solve the joint recovery of the solution, the source function along with the unknown parameters thereby using the underlying partial differential equation as a constraint that couples multiple unknown functional parameters, leading to more efficient use of the limited information in the measurements. We apply our method on the advection-diffusion equation and we present various 2D and 3D numerical experiments using different types of measurements data that reflect practical engineering systems. Our proposed method is successful in estimating the unknown source function, the velocity and diffusion parameters as well as recovering the solution of the equation, while remaining robust to additional noise in the measurements.
Read the full abstract

All authors Brenda Anague, Bamdad Hosseini, Issa Karambal, Jean Medard Ngnotchouye

Statistical Methodology

Symmetric Vaccine Efficacy

Traditional measures of vaccine efficacy (VE) are inherently asymmetric, constrained above by $1$ but unbounded below. As a result, VE estimates and corresponding confidence intervals can extend far below zero, making interpretation difficult and potentially obscuring whether the apparent effect reflects true harm or simply statistical uncertainty. The proposed symmetric vaccine efficacy (SVE) is a bounded and interpretable alternative to VE that maintains desirable statistical properties while resolving these asymmetries. SVE is defined as a symmetric transformation of infection risks, with possible values within $[-1, 1]$, providing a common scale for both beneficial and harmful vaccine effects. This paper describes the relationship between SVE and traditional VE, considers inference about SVE, and illustrates the utility of the proposed measure by reanalyzing data from a randomized trial of a candidate HIV vaccine. Open-source tools for computing estimates of SVE and corresponding confidence intervals are available in R through the sve package.
Read the full abstract

Statistical Computation · Cross-listed from econ.EM

Bounds on inequality with incomplete data

We study inequality measures when outcomes are observed only in intervals, as in historical tabulations, privacy-protected grouped data, and modern surveys. We develop a nonparametric framework for sharp identification and inference with grouped and interval-valued data, covering brackets and overlapping intervals. For a class of inequality indices, sharp bounds are attained by discrete distributions with finite support, reducing the problem to optimization; linear-fractional indices, including the Gini and quantile ratios, yield linear or quadratic programs. Plug-in bound endpoints have a $\sqrt{n}$ asymptotic distribution, using an $m$-out-of-$n$ bootstrap. Applications to wealth and historical income data compare identified sets with imputation-based estimates.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

A Bootstrap Perspective on Stochastic Gradient Descent

Machine learning models trained with stochastic gradient descent (SGD) can generalize better than those trained with deterministic gradient descent (GD). In this work, we study SGD's impact on generalization through the lens of the statistical bootstrap: SGD uses gradient variability under batch sampling as a proxy for solution variability under the randomness of the data collection process. We use empirical results and theoretical analysis to substantiate this claim. In idealized experiments on empirical risk minimization, we show that SGD is drawn to parameter choices that are robust under resampling and thus avoids spurious solutions even if they lie in wider and deeper minima of the training loss. We prove rigorously that by implicitly regularizing the trace of the gradient covariance matrix, SGD controls the algorithmic variability. This regularization leads to solutions that are less sensitive to sampling noise, thereby improving generalization. Numerical experiments on neural network training show that explicitly incorporating the estimate of the algorithmic variability as a regularizer improves test performance. This fact supports our claim that bootstrap estimation underpins SGD's generalization advantages.
Read the full abstract

Statistics Theory

Location and scatter halfspace median under {\alpha}-symmetric distributions

In a landmark result, Chen et al. (2018) showed that multivariate medians induced by halfspace depth attain the minimax optimal convergence rate under Huber contamination and elliptical symmetry, for both location and scatter estimation. We extend some of these findings to the broader family of {\alpha}-symmetric distributions, which includes both elliptically symmetric and multivariate heavy-tailed distributions. For location estimation, we establish an upper bound on the estimation error of the location halfspace median under the Huber contamination model. An analogous result for the standard scatter halfspace median matrix is feasible only under the assumption of elliptical symmetry, as ellipticity is deeply embedded in the definition of scatter halfspace depth. To address this limitation, we propose a modified scatter halfspace depth that better accommodates {\alpha}-symmetric distributions, and derive an upper bound for the corresponding {\alpha}-scatter median matrix. Additionally, we identify several key properties of scatter halfspace depth for {\alpha}-symmetric distributions.
Read the full abstract

Machine Learning (Statistics)

$\phi$-Table: A Statistical Explanation for Global SHAP

Global SHAP explanations are typically presented as feature-importance rankings, which identify variables that matter to a black-box model but do not indicate whether their effects admit clear directional summaries, how uncertain those summaries are, or how faithfully they represent the fitted response. This paper proposes the $\phi$-table, a SHAP-based statistical explanation table for tabular black-box regression models. The procedure selects features by SHAP importance and fits a standardized linear surrogate to the fitted model response $f(X)$, reporting SHAP importance together with model-response coefficients, uncertainty summaries, surrogate fidelity, and bootstrap coefficient stability. The resulting coefficients are interpreted as projections of the fitted model response onto the SHAP-selected feature set. Across synthetic, semi-synthetic, and real-data experiments, the $\phi$-table extends ranking-only SHAP into a statistical global explanation by exposing direction, uncertainty, fidelity, and stability as distinct components of fitted model behavior.
Read the full abstract

All authors Dongseok Kim, Hyoungsun Choi, Mohamed Jismy Aashik Rasool, Gisung Oh

Statistical Methodology

Surprisingly-early bias in forecasts for unscheduled events

When a dataset contains forecasts on unscheduled events, such as natural catastrophes, outcomes may be censored or “hidden” since some events have not yet occurred. This article finds that this can lead to a selection bias which affects the perceived accuracy and calibration of forecasts. This selection bias can be eliminated by excluding forecasts on outcomes which have been verified surprisingly early.
Read the full abstract

Machine Learning (Statistics)

On Conditional Independence Graph Learning From Multi-Attribute Gaussian Dependent Time Series

Estimation of the conditional independence graph (CIG) of high-dimensional multivariate Gaussian time series from multi-attribute data is considered. Existing methods for graph estimation for such data are based on single-attribute models where one associates a scalar time series with each node. In multi-attribute graphical models, each node represents a random vector or vector time series. In this paper we provide a unified theoretical analysis of multi-attribute graph learning for dependent time series using a penalized log-likelihood objective function formulated in the frequency domain using the discrete Fourier transform of the time-domain data. We consider both convex (sparse-group lasso) and non-convex (log-sum and SCAD group penalties) penalty/regularization functions. We establish sufficient conditions in a high-dimensional setting for consistency (convergence of the inverse power spectral density to true value in the Frobenius norm), local convexity when using non-convex penalties, and graph recovery. We do not impose any incoherence or irrepresentability condition for our convergence results. We also empirically investigate selection of the tuning parameters based on the Bayesian information criterion, and illustrate our approach using numerical examples utilizing both synthetic and real data.
Read the full abstract

Machine Learning (Statistics)

High-Dimensional Change Point Detection via Graph Spanning Ratio

Inspired by graph-based methodologies, we introduce a novel graph-spanning algorithm designed to identify changes in both offline and online data across low to high dimensions. This versatile approach is applicable to Euclidean and graph-structured data with unknown distributions, while maintaining control over error probabilities. Theoretically, we demonstrate that the algorithm achieves high detection power when the magnitude of the change surpasses the lower bound of the minimax separation rate, which scales on the order of $\sqrt{nd}$. Our method outperforms other techniques in terms of accuracy for both Gaussian and non-Gaussian data. Notably, it maintains strong detection power even with small observation windows, making it particularly effective for online environments where timely and precise change detection is critical.
Read the full abstract

Statistics Applications

Meta-analyses of dietary exposures must consider energy adjustment: recommendations from a meta-scientific review

In observational studies of dietary exposures, the energy adjustment strategy has a critical impact on the effect being estimated. Adjusting for total energy intake or expressing the exposure as a percentage of total energy, leads to a substitution effect being estimated. This impacts the interpretation of primary studies and meta-analyses. Unless energy adjustment strategies are considered, meta-analyses may end up pooling estimates for incomparable effects. This meta-scientific review aimed to investigate the extent to which meta-analyses of dietary exposures may be pooling incomparable effects by reviewing the energy adjustment strategies. We identified all meta-analyses examining the relationship between saturated fat and fish and cardiovascular disease. The two most recent and two most cited reviews for each exposure were examined, along with all primary studies. Information on the study aims, targeted effects, and interpretations were summarized. The eight meta-analyses summarised results from 82 primary studies including 144 unique models. Only one meta-analysis explicitly considered the energy adjustment strategy of the primary studies to determine eligibility for a substitution subgroup analysis. None of the meta-analyses acknowledged that they were pooling estimates for different effects. 82% of the models from the primary studies were implicitly estimating substitution effects but this was not explicitly stated in most study aims, interpretation or conclusions. Our meta-scientific review found little evidence that the energy adjustment strategies of the primary studies were being considered in the synthesis or interpretation of evidence. Consequently, the pooled estimates reflect ill-defined quantities with unclear interpretations. We offer recommendations to improve the conduct of future meta-analyses and the quality of evidence that informs nutritional recommendations.
Read the full abstract

All authors Natalia Ortega, Peter WG Tennant, Darren C Greenwood, Octavio Pano, Christina C Dahm, Russell J de Souza, Daniel B Ibsen, Conor J MacDonald, Deirdre K Tobias, Georgia D Tomova

Statistics Applications · Cross-listed from econ.GN

Rice Price Dynamics during the 1945–1947 Famine in Post-War Taiwan: A Quantitative Reassessment

We compiled the first high-frequency rice price panel for Taiwan from August 1945 to March 1947, during the transition from Japanese rule to China rule. Using regression models, we found that the pattern of rice price changes could be divided into four stages, each with distinct characteristics. Based on different stages, we combined the policies formulated by the Taiwan government at the time to demonstrate the correlation between rice prices and policies. The research results highlight the dominant role of policy systems in post-war food crises.
Read the full abstract

Statistics Applications

Permanent and transitory crime risk in variable-density hot spot analysis

Crime prevention measures, aiming for the effective and efficient spending of public resources, rely on the empirical analysis of spatial and temporal data for public safety outcomes. We perform a variable-density cluster analysis on crime incident reports in the City of Chicago for the years 2001–2022 to investigate changes in crime share composition for hot spots of different densities. Contributing to and going beyond the existing wealth of research on criminological applications in the operational research literature, we study the evolution of crime type shares in clusters over the course of two decades and demonstrate particularly notable impacts of the COVID-19 pandemic and its associated social contact avoidance measures, as well as a dependence of these effects on the primary function of city areas. Our results also indicate differences in the relative difficulty to address specific crime types, and an analysis of spatial autocorrelations further shows variations in incident uniformity between clusters and outlier areas at different distance radii. We discuss our findings in the context of the interplay between operational research and criminal justice, the practice of hot spot policing and public safety optimization, and the factors contributing to, and challenges and risks due to, data biases as an often neglected factor in criminological applications.
Read the full abstract

Statistics Applications · Cross-listed from cs.LG

Parallel Algorithms for Structured Sparse Support Vector Machines: Application in Music Genre Classification

Mathematical modelling, particularly through approaches such as structured sparse support vector machines (SS-SVM), plays a crucial role in processing data with complex feature structures, yet efficient algorithms for distributed large-scale data remain lacking. To address this gap, this paper proposes a unified optimization framework based on a consensus structure. This framework is not only applicable to various loss functions and combined regularization terms but can also be effectively extended to non-convex regularizers, demonstrating strong scalability. Building upon this framework, we develop a distributed parallel alternating direction method of multipliers (ADMM) algorithm to efficiently solve SS-SVMs under distributed data storage. To ensure convergence, we incorporate a Gaussian back-substitution technique. Additionally, for completeness, we introduce a family of sparse group Lasso support vector machine (SGL-SVM) and apply it to music information retrieval. Theoretical analysis confirms that the computational complexity of the proposed algorithm is independent of the choice of regularization terms and loss functions, underscoring the universality of the parallel approach. Experiments on both synthetic and real-world music archive datasets validate the reliability, stability, and efficiency of our algorithm.
Read the full abstract

All authors Rongmei Liang, Zizheng Liu, Xiaofei Wu, Jingwen Tu

Statistics Theory

A multivariate extension of Azadkia-Chatterjee's rank coefficient

The Azadkia-Chatterjee coefficient is a rank-based measure of dependence between a random variable $Y \in \mathbb{R}$ and a random vector ${\boldsymbol Z} \in \mathbb{R}^{d_Z}$. In this paper, we propose a multivariate extension that measures the dependence between random vectors ${\boldsymbol Y} \in \mathbb{R}^{d_Y}$ and ${\boldsymbol Z} \in \mathbb{R}^{d_Z}$, based on $n$ i.i.d. samples. The proposed coefficient converges almost surely to a limit with the following properties: i) it lies in $[0, 1]$; ii) it is equal to zero if and only if ${\boldsymbol Y}$ and ${\boldsymbol Z}$ are independent; and iii) it is equal to one if and only if ${\boldsymbol Y}$ is almost surely a function of ${\boldsymbol Z}$. Remarkably, the only assumption required by this convergence is that ${\boldsymbol Y}$ is not almost surely a constant vector. We further prove that under the same mild condition and after a proper scaling, this coefficient converges in distribution to a standard normal random variable when ${\boldsymbol Y}$ and ${\boldsymbol Z}$ are independent. This asymptotic normality result allows us to construct a Wald-type hypothesis test of independence based on this coefficient. To compute this coefficient, we propose a merge sort based algorithm that runs in $O(n (\log n)^{d_Y})$. Finally, we show that it can be used to measure the conditional dependence between ${\boldsymbol Y}$ and ${\boldsymbol Z}$ conditional on a third random vector ${\boldsymbol X}$, and prove that the measure is monotonic with respect to the deviation from an independence distribution under certain model restrictions.
Read the full abstract

Statistics Applications

Bridging CORDEX and CMIP6: Machine Learning Downscaling for Wind and Solar Energy Droughts in Central Europe

Reliable regional climate information is essential for assessing the impacts of climate change and for planning in sectors such as renewable energy; yet, producing high-resolution projections through coordinated initiatives like CORDEX that run multiple physical regional climate models is both computationally demanding and difficult to organize. Machine learning emulators that learn the mapping between global and regional climate fields offer a promising way to address these limitations. Here we introduce the application of such an emulator: trained on CMIP5 and CORDEX simulations, it reproduces regional climate model data with sufficient accuracy. When applied to CMIP6 simulations not seen during training, it also produces realistic results, indicating stable performance. Using CORDEX data, CMIP5 and CMIP6 simulations, as well as regional data generated by two machine learning models, we analyze the co-occurrence of low wind speed and low solar radiation and find indications that the number of such energy drought days is likely to decrease in the future. Our results highlight that downscaling with machine learning emulators provides an efficient complement to efforts such as CORDEX, supplying the higher-resolution information required for impact assessments.
Read the full abstract

All authors Nina Effenberger, Maxim Samarin, Maybritt Schillinger, Reto Knutti

Statistics Theory

A model-free Screening procedure

In this article, we propose a generic screening method for selecting explanatory variables correlated with the response variable Y . We make no assumptions about the existence of a model that could link Y with a subset of explanatory variables, nor about the distribution of the variables. Our procedure can therefore be described as ”model-free” and can be applied in a wide range of situations. In order to obtain precise theoretical guarantees (Sure Screening Property and control of the False Positive Rate), we establish a Berry-Esseen type inequality for the studentized statistic of the slope estimator. We illustrate our selection procedure using two simulated examples and a real data set.
Read the full abstract

Statistics Theory

Nonparametric optimal density estimation for censored circular data

We consider the problem of estimating the probability density function of a circular random variable observed under censoring. To this end, we introduce a projection estimator constructed via a regression approach on linear sieves. We first establish a lower bound for the mean integrated squared error in the case of Sobolev densities, thereby identifying the minimax rate of convergence for this estimation problem. We then derive a matching upper bound for the same risk, showing that the proposed estimator attains the minimax rate when the underlying density belongs to a Sobolev class. Finally, we develop a data-driven version of the procedure that preserves this optimal rate, thus yielding an adaptive estimator. The practical performance of the method is demonstrated through simulation studies.
Read the full abstract

Machine Learning (Statistics)

Machine learning in an expectation-maximisation framework for nowcasting

Decision making often occurs in the presence of incomplete information, leading to the under- or overestimation of risk. Leveraging the observable information to learn the complete information is called nowcasting. In practice, incomplete information is often a consequence of reporting or observation delays. In this paper, we propose an expectation-maximisation (EM) framework for nowcasting that uses machine learning techniques to model both the occurrence as well as the reporting process of events. We allow for the inclusion of covariate information specific to the occurrence and reporting periods as well as characteristics related to the entity for which events occurred. We demonstrate how the maximisation step and the information flow between EM iterations can be tailored to leverage the predictive power of neural networks and (extreme) gradient boosting machines (XGBoost). With simulation experiments, we show that we can effectively model both the occurrence and reporting of events when dealing with high-dimensional covariate information. In the presence of non-linear effects, we show that our methodology outperforms existing EM-based nowcasting frameworks that use generalised linear models in the maximisation step. Finally, we apply the framework to the reporting of Argentinian Covid-19 cases, where the XGBoost-based approach again is most performant.
Read the full abstract

Machine Learning (Statistics)

Exact Synthetic Populations for Scalable Societal and Market Modeling

We introduce a constraint-programming framework for generating synthetic populations that reproduce target statistics with high precision while enforcing full individual consistency. Unlike data-driven approaches that infer distributions from samples, our method directly encodes aggregated statistics and structural relations, enabling exact control of demographic profiles without requiring any microdata. We validate the approach on official demographic sources and study the impact of distributional deviations on downstream analyses. This work is conducted within the Pollitics project developed by Emotia, where synthetic populations can be queried through large language models to model societal behaviors, explore market and policy scenarios, and provide reproducible decision-grade insights without personal data.
Read the full abstract

Statistics Theory

Estimation of the elasticity for CKLS model from high-frequency observations

We investigate parametric estimation of the elasticity parameter in the CKLS diffusion based on high-frequency data. First, we transform the CKLS diffusion to a CIR-type one via a smooth state-space mapping and the general Girsanov change of measure. This transformation enables the applications of existing inference tools for CIR processes while ensuring possibilities of transferring the resulting limit theorems back to the original probability space. However, because Feller's condition fails, many existing high-frequency likelihood-based procedures cannot be applied directly, since their discretization schemes approximate likelihood terms involving the reciprocal of the process by Riemann sums that are no longer well-defined once the paths are allowed to hit zero. Instead, we estimate the drift coefficient of the transformed CIR-type model via a procedure based on its positive Harris recurrence, which is valid in the high-frequency regime. Exploiting the drift-elasticity relationship implied by the CKLS–CIR transformation, with the help of an initial estimation, we obtain an estimator of the CKLS elasticity from the CIR drift estimator in the transformed model. This yields a closed-form estimator of the elasticity parameter with an explicit asymptotic variance. We establish its $p$-consistency, stable convergence in law, and asymptotic normality. Finally, we show that stable convergence in law is invariant under equivalent changes of measure, thereby guaranteeing that the Gaussian limit remains invariant under the original measure.
Read the full abstract

Statistics Applications

Facilitating Conditions as an Enabler, Not a Direct Motivator: A Robustness and Mediation Analysis of E-Learning Adoption

Despite substantial institutional investment in e-learning infrastructure, student engagement often fails to meet expectations–a persistent paradox that challenges the established direct relationship between Facilitating Conditions (FC) and behavioral intention within the classic UTAUT framework. To resolve this theoretical puzzle, we reconceptualized the role of FC through an empirical study of 470 Indonesian university students. Our robust, multi-stage analytical approach first confirmed the significant influence of established drivers–Performance Expectancy (beta=0.190), Effort Expectancy (beta=0.198), Social Influence (beta=0.151), and Perceived Enjoyment (beta=0.472)–on Behavioral Intention (BI), which in turn strongly predicted Use Behavior (beta=0.666). Crucially, however, the direct effect of FC on BI proved non-significant (beta=-0.085). A subsequent mediation model revealed FC's true function as a foundational enabling construct that operates indirectly by powerfully enhancing both Performance Expectancy (beta=0.556) and Effort Expectancy (beta=0.419). Our findings demonstrate that the value of technological infrastructure lies not in its mere presence, but in its dynamic capacity to enable learning and optimize user experience. This research advances a refined "enabling pathway" theoretical framework, guiding administrators to shift the focus of technological investment from merely providing tools to strategically crafting learning experiences.
Read the full abstract

All authors Jaka Nugraha, Noyyn Sun, Xinlin Zhao, Vindi Kusuma Wardani, Inna Koblianska, Jiunn-Woei Lian