Statistics Applications · Cross-listed from q-fin.ST
The Evolution of Probabilistic Price Forecasting Techniques: A Review of the Day-Ahead, Intra-Day, and Balancing Markets
Ciaran O'Connor, Mohamed Bahloul, Steven Prestwich et al. (+1 more)
Electricity price
forecasting has become a critical tool for decision-making in energy
markets, particularly as the increasing penetration of renewable energy introduces greater
volatility and uncertainty. Historically, research in this field has been dominated by point
forecasting methods, which provide single-value predictions but fail to quantify uncertainty. However, as power
markets evolve due to renewable integration, smart grids, and regulatory changes, the need for probabilistic
forecasting has become more pronounced, offering a more comprehensive approach to risk assessment and
market participation. This paper presents a review of probabilistic
forecasting methods, tracing their
evolution from
Bayesian and distribution based approaches, through
quantile regression techniques, to recent developments in conformal prediction. Particular emphasis is placed on advancements in probabilistic
forecasting, including validity-focused methods which address key limitations in uncertainty estimation. Additionally, this review extends beyond the Day-Ahead
Market to include the Intra-Day and Balancing
Markets, where
forecasting challenges are intensified by higher temporal granularity and real-time operational constraints. We examine state of the art methodologies, key evaluation metrics, and ongoing challenges, such as forecast validity, model selection, and the absence of standardised
benchmarks, providing researchers and practitioners with a comprehensive and timely resource for navigating the complexities of modern electricity
markets.
Read the full abstract ▾
All authors Ciaran O'Connor, Mohamed Bahloul, Steven Prestwich, Andrea Visentin
Machine Learning (Statistics) · Cross-listed from cs.LG
Quadratic Direct Forecast for Training Multi-Step Time-Series Forecast Models
Hao Wang, Licheng Pan, Yuan Lu et al. (+7 more)
The design of learning objectives is central to training time-series
forecasting models. Existing learning objectives such as mean squared error mostly treat each future step as an independent, equally weighted task, which leads to the following two challenges: (1) they overlook the label autocorrelation effect among future steps, leading to biased learning objectives; (2) they fail to set heterogeneous task weights for different
forecasting tasks corresponding to varying future steps, limiting the
forecasting performance. To fill this gap, we propose a novel quadratic-form weighted learning objective, addressing both issues simultaneously. Specifically, the off-diagonal elements of the weighting
matrix account for the label autocorrelation effect, whereas the non-uniform diagonals are expected to match the preferred weights of the
forecasting tasks with varying future steps. On this basis, we propose a Quadratic Direct Forecast (QDF) learning
algorithm, which trains the forecast model using the adaptively updated quadratic-form weighting
matrix. Experiments show that our QDF effectively improves the performance of various forecast models, achieving state-of-the-art results. Code is available at https://github.com/Master-PLC/QDF.
Read the full abstract ▾
All authors Hao Wang, Licheng Pan, Yuan Lu, Zhichao Chen, Tianqiao Liu, Shuting He, Zhixuan Chu, Qingsong Wen, Haoxuan Li, Zhouchen Lin
Statistics Applications
Systematic Absence of Low-Confidence Nighttime Fire Detections in VIIRS Active Fire Product: Evidence of Undocumented Algorithmic Filtering
Rohit Rajendra Dhage
The Visible Infrared Imaging Radiometer Suite (VIIRS) active fire product is widely used for global fire monitoring, yet its confidence classification scheme exhibits an undocumented systematic pattern. Through analysis of 21,540,921 fire detections spanning one year (January 2023 - January 2024), I demonstrate a complete absence of low-confidence classifications during nighttime observations. Of 6,007,831 nighttime fires, zero were classified as low confidence, compared to an expected 696,908 under statistical independence (chi-squared = 1,474,795, p < 10^-15, Z = -833). This pattern persists globally across all months, latitude bands, and both NOAA-20 and Suomi-NPP satellites. Machine learning reverse-engineering (88.9% accuracy), bootstrap simulation (1,000 iterations), and spatial-temporal analysis confirm this is an algorithmic constraint rather than a geophysical phenomenon. Brightness temperature analysis reveals nighttime fires below approximately 295K are likely excluded entirely rather than flagged as low-confidence, while daytime fires show normal confidence distributions. This undocumented behavior affects 27.9% of all VIIRS fire detections and has significant implications for fire risk assessment, day-night detection comparisons, confidence-weighted analyses, and any research treating confidence levels as uncertainty metrics. I recommend explicit documentation of this algorithmic constraint in VIIRS user guides and reprocessing strategies for affected analyses.
Read the full abstract ▾
Statistics Applications
Toward precision soil health: A regional framework for site-specific management across Missouri
Dipal Shah, Jordon Wade, Timothy Haithcoat et al. (+2 more)
Effective soil health management is crucial for sustaining agriculture, adopting ecosystem resilience, and preserving water quality. However, Missouri's diverse landscapes limit the effectiveness of broad generalized management recommendations. The lack of resolution in existing soil grouping systems necessitates data driven, site specific insights to guide tailored interventions. To address these critical challenges, a regional soil clustering framework designed to support precision soil health management strategies across the state. The methodology leveraged high resolution SSURGO
dataset, explicitly processing soil properties aggregated across the 0 to 30 cm root zone. Multivariate analysis incorporating a variational autoencoder and KMeans clustering was used to group soils with similar properties. The derived clusters were validated using statistical metrics, including silhouette scores and checks against existing taxonomic units, to confirm their spatial coherence. This approach enabled us to delineate soil groups that capture textures, hydraulic properties, chemical fertility, and biological indicators unique to Missouri's diverse agroecological regions. The clustering map identified ten distinct soil health management zones. This
alignment of 10 clusters was selected as optimal because it was sufficiently large to capture inherited soil patterns while remaining manageable for practical statewide application. Rooting depth limitation and saturated hydraulic conductivity emerged as principal variables driving soil differentiation. Each management zone is defined by a unique combination of clay, organic matter, pH, and available water capacity. This framework bridges sophisticated data analysis with actionable, site targeted recommendations, enabling conservation planners, and agronomists to optimize management practices and enhance resource efficiency statewide.
Read the full abstract ▾
All authors Dipal Shah, Jordon Wade, Timothy Haithcoat, Robert Myers, Kelly Wilson
Statistical Methodology
Kellin N. Rumsey, Devin Francom, Graham C. Gibson et al. (+2 more)
Polynomial chaos expansions (PCE) are widely used for uncertainty quantification (UQ) tasks, particularly in the applied mathematics community. However, PCE has received comparatively less
attention in the statistics literature, and fully
Bayesian formulations remain rare, especially with implementations in R. Motivated by the success of adaptive
Bayesian machine learning models such as BART, BASS, and BPPR, we develop a new fully
Bayesian adaptive PCE method with an efficient and accessible R implementation: khaos. Our approach includes a novel proposal distribution that enables data-driven interaction selection, and supports a modified g-prior tailored to PCE structure. Through simulation studies and real-world UQ applications, we demonstrate that
Bayesian adaptive PCE provides competitive performance for surrogate modeling, global sensitivity analysis, and ordinal regression tasks.
Read the full abstract ▾
All authors Kellin N. Rumsey, Devin Francom, Graham C. Gibson, J. Derek Tucker, Gabriel Huerta
Statistics Theory · Cross-listed from cs.AI
Cyclic Counterfactuals under Shift-Scale Interventions
Saptarshi Saha, Dhruv Vansraj Rathore and Utpal Garain
Most counterfactual inference frameworks traditionally assume acyclic structural causal models (SCMs), i.e. directed acyclic graphs (DAGs). However, many real-world systems (e.g. biological systems) contain feedback loops or cyclic dependencies that violate acyclicity. In this work, we study counterfactual inference in cyclic SCMs under shift-scale interventions, i.e., soft, policy-style changes that rescale and/or shift a variable's mechanism.
Read the full abstract ▾
Statistical Computation
Bayesian Neural Networks vs. Mixture Density Networks: Theoretical and Empirical Insights for Uncertainty-Aware Nonlinear Modeling
Riddhi Pratim Ghosh and Ian Barnett
This paper investigates two prominent probabilistic neural modeling paradigms:
Bayesian Neural Networks (BNNs) and Mixture Density Networks (MDNs) for uncertainty-aware nonlinear regression. While BNNs incorporate epistemic uncertainty by placing prior distributions over network parameters, MDNs directly model the conditional output distribution, thereby capturing
multimodal and heteroscedastic data-generating mechanisms. We present a unified theoretical and empirical framework comparing these approaches. On the theoretical side, we derive convergence rates and error bounds under Hölder smoothness conditions, showing that MDNs achieve faster Kullback-Leibler (KL) divergence convergence due to their likelihood-based nature, whereas BNNs exhibit additional approximation bias induced by variational inference. Empirically, we evaluate both architectures on synthetic nonlinear
datasets and a radiographic
benchmark (RSNA Pediatric Bone Age Challenge). Quantitative and qualitative results demonstrate that MDNs more effectively capture
multimodal responses and adaptive uncertainty, whereas BNNs provide more interpretable epistemic uncertainty under limited data. Our findings clarify the complementary strengths of posterior-based and likelihood-based probabilistic learning, offering guidance for uncertainty-aware modeling in nonlinear systems.
Read the full abstract ▾
Statistical Methodology
Bayesian Geostatistical Modeling for Cluster Randomized Trials
Jooyeon Lee, M.S., Evan Kwiatkowski et al. (+1 more)
Cluster randomized trials (CRTs) offer a practical alternative for addressing logistical challenges and ensuring feasibility in community health, education, and prevention studies, even though individually-randomized controlled trials are considered the gold standard in evaluating therapeutic interventions. Despite their utility, CRTs are often criticized for limited precision and complex modeling requirements. Advances in robust
Bayesian methods and the incorporation of spatial correlation into CRT design and analysis remain relatively underdeveloped. This paper introduces a
Bayesian geostatistical framework that models individuals nested within geographic clusters while explicitly accounting for spatial dependence. We demonstrate that conventional non-spatial models are susceptible to underestimating uncertainty and lead to misleading inferences, whereas our spatial approach improves estimation stability, controls type I error, and enhances statistical power. Additionally, we explore design implications that are suggested through the exploration of spatial predictive uncertainty. Our results of simulation and real-world data application demonstrate the value and need for wider adoption of spatial methods in CRTs.
Read the full abstract ▾
All authors Jooyeon Lee, M.S., Evan Kwiatkowski, Ph.D
Machine Learning (Statistics) · Cross-listed from cs.LG
Sequences of Logits Reveal the Low Rank Structure of Language Models
Noah Golowich, Allen Liu and Abhishek Shetty
A major problem in the study of large
language models is to understand their inherent low-dimensional structure. We introduce an approach to study the low-dimensional structure of
language models at a model-agnostic level: as sequential probabilistic models. We first empirically demonstrate that a wide range of modern
language models exhibit low-rank structure: in particular,
matrices built from the model's logits for varying sets of
prompts and responses have low approximate rank. We then show that this low-rank structure can be leveraged for generation – in particular, we can generate a response to a target
prompt using a linear combination of the model's outputs on unrelated, or even nonsensical
prompts. On the theoretical front, we observe that studying the approximate rank of
language models in the sense discussed above yields a simple universal abstraction whose theoretical predictions parallel our experiments. We then analyze the representation power of the abstraction and give provable learning guarantees.
Read the full abstract ▾
Statistical Methodology
Differential Density Analysis in Single-Cell Genomics Using Specially Designed Exponential Families
Hanxuan Ye, Zachary Qian and Hongzhe Li
Recent advances in high-resolution sequencing have paved the way for population-scale analysis in single-
cell RNA-sequencing (scRNA-seq) data. scRNA-seq data, in particular, have proven to be extremely powerful in profiling a variety of outcomes such as disease and aging. The abundance of scRNA-seq data makes it possible to model each individual's gene expression as a probability density across
cells, offering a richer representation than summary statistics such as means or variances, and allowing for more nuanced group comparisons. To this end, we propose a model-agnostic framework for density estimation and inference based on specially designed exponential families~(SEF), which accommodates diverse underlying models without requiring prior specifications. The proposed method enables estimation and visualization for both individual-specific and group-level gene expression densities, as well as conducting formal hypothesis testing for expression density difference across groups of interest. It relies on relaxed assumptions with established asymptotic properties and a consistent covariance
estimator for valid inference. Through simulation under various scenarios, the SEF-based approach demonstrates good error control and improved statistical power over competing methods,including pseudo-bulk tests and moment
estimators. Application to a population-scale scRNA-seq
dataset from patients with systemic lupus erythematosus identified genes and gene sets that are missed from pseudo-bulk based tests.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.LG
Topic Analysis with Side Information: A Neural-Augmented LDA Approach
Biyi Fang, Truong Vo, Kripa Rajshekhar et al. (+1 more)
Traditional topic models such as Latent Dirichlet Allocation (LDA) have been widely used to uncover latent structures in text corpora, but they often struggle to integrate auxiliary information such as metadata, user attributes, or document labels. These limitations restrict their expressiveness, personalization, and interpretability. To address this, we propose nnLDA, a neural-augmented probabilistic topic model that dynamically incorporates side information through a neural prior mechanism. nnLDA models each document as a mixture of latent topics, where the prior over topic proportions is generated by a neural network conditioned on auxiliary features. This design allows the model to capture complex nonlinear interactions between side information and topic distributions that static Dirichlet priors cannot represent. We develop a
stochastic variational Expectation-Maximization
algorithm to jointly optimize the neural and probabilistic components. Across multiple
benchmark datasets, nnLDA consistently outperforms LDA and Dirichlet-Multinomial Regression in topic coherence, perplexity, and downstream classification. These results highlight the benefits of combining neural representation learning with probabilistic topic modeling in settings where side information is available.
Read the full abstract ▾
All authors Biyi Fang, Truong Vo, Kripa Rajshekhar, Diego Klabjan
Statistics Theory
Maximum Likelihood Estimation in the Multivariate and Matrix Variate Symmetric Laplace Distributions through Group Actions
Pooja Yadav and Tanuja Srivastava
In this paper, we study the maximum likelihood estimation of the parameters of the multivariate and
matrix variate symmetric Laplace distributions through group actions. The multivariate and
matrix variate symmetric Laplace distributions are not in the exponential family of distributions. We relate the maximum likelihood estimation problems of these distributions to norm minimization over a group and build a correspondence between stability of data with respect to the group action and the properties of the likelihood function.
Read the full abstract ▾
Machine Learning (Statistics)
Tree Ensemble Explainability through the Hoeffding Functional Decomposition and TreeHFD Algorithm
Clément Bénard
Tree ensembles have demonstrated state-of-the-art predictive performance across a wide range of problems involving tabular data. Nevertheless, the black-box nature of tree ensembles is a strong limitation, especially for applications with critical decisions at stake. The Hoeffding or ANOVA functional decomposition is a powerful explainability method, as it breaks down black-box models into a unique sum of lower-dimensional functions, provided that input variables are independent. In standard learning settings, input variables are often dependent, and the Hoeffding decomposition is generalized through hierarchical orthogonality constraints. Such generalization leads to unique and sparse decompositions with well-defined main effects and interactions. However, the practical estimation of this decomposition from a data sample is still an open problem. Therefore, we introduce the TreeHFD
algorithm to estimate the Hoeffding decomposition of a tree ensemble from a data sample. We show the convergence of TreeHFD, along with the main properties of orthogonality, sparsity, and causal variable selection. The high performance of TreeHFD is demonstrated through experiments on both simulated and real data, using our treehfd Python package (https://github.com/ThalesGroup/treehfd). Besides, we empirically show that the widely used TreeSHAP method, based on Shapley values, is strongly connected to the Hoeffding decomposition.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning
Junsoo Oh, Jerry Song and Chulhee Yun
Weak-to-strong generalization refers to the phenomenon where a stronger model trained under supervision from a weaker one can outperform its teacher. While prior studies aim to explain this effect, most theoretical insights are limited to abstract frameworks or linear/random feature models. In this paper, we provide a formal analysis of weak-to-strong generalization from a linear CNN (weak) to a two-layer ReLU CNN (strong). We consider structured data composed of label-dependent signals of varying difficulty and label-independent noise, and analyze gradient descent dynamics when the strong model is trained on data labeled by the pretrained weak model. Our analysis identifies two regimes – data-scarce and data-abundant – based on the signal-to-noise characteristics of the
dataset, and reveals distinct mechanisms of weak-to-strong generalization. In the data-scarce regime, generalization occurs via benign overfitting or fails via harmful overfitting, depending on the amount of data, and we characterize the transition boundary. In the data-abundant regime, generalization emerges in the early phase through label correction, but we observe that overtraining can subsequently degrade performance.
Read the full abstract ▾
Statistical Methodology
Machine-Learning-Assisted Comparison of Regression Functions
Jian Yan, Zhuoxi Li, Yang Ning et al. (+1 more)
We revisit the classical problem of comparing regression functions, a fundamental question in statistical inference with broad relevance to modern applications such as data integration, transfer learning, and causal inference. Existing approaches typically rely on smoothing techniques and are thus hindered by the curse of dimensionality. We propose a generalized notion of kernel-based conditional mean dependence that provides a new characterization of the null hypothesis of equal regression functions. Building on this reformulation, we develop two novel tests that leverage modern machine learning methods for flexible estimation. We establish the asymptotic properties of the test statistics, which hold under both fixed- and high-dimensional regimes. Unlike existing methods that often require restrictive distributional assumptions, our framework only imposes mild moment conditions. The efficacy of the proposed tests is demonstrated through extensive numerical studies.
Read the full abstract ▾
All authors Jian Yan, Zhuoxi Li, Yang Ning, Yong Chen
Machine Learning (Statistics) · Cross-listed from math.OC
A Single-Loop First-Order Algorithm for Linearly Constrained Bilevel Optimization
Wei Shen, Jiawei Zhang, Minhui Huang et al. (+1 more)
We study bilevel optimization problems where the lower-level problems are strongly convex and have coupled linear constraints. To overcome the potential non-smoothness of the hyper-objective and the computational challenges associated with the Hessian
matrix, we utilize penalty and augmented Lagrangian methods to reformulate the original problem as a single-level one. Especially, we establish a strong theoretical connection between the reformulated function and the original hyper-objective by characterizing the closeness of their values and derivatives. Based on this reformulation, we propose a single-loop, first-order
algorithm for linearly constrained bilevel optimization (SFLCB). We provide rigorous analyses of its non-asymptotic convergence rates, showing an improvement over prior double-loop
algorithms – form $O(\epsilon^{-3}\log(\epsilon^{-1}))$ to $O(\epsilon^{-3})$. The experiments corroborate our theoretical findings and demonstrate the practical efficiency of the proposed SFLCB
algorithm. Simulation code is provided at https://github.com/ShenGroup/SFLCB.
Read the full abstract ▾
All authors Wei Shen, Jiawei Zhang, Minhui Huang, Cong Shen
Machine Learning (Statistics) · Cross-listed from cs.LG
Greedy Sampling Is Provably Efficient for RLHF
Di Wu, Chengshuai Shi, Jing Yang et al. (+1 more)
Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique for post-training large
language models. Despite its empirical success, the theoretical understanding of RLHF is still limited, as learning the KL-regularized target with only preference feedback poses additional challenges compared with canonical RL. Existing works mostly study the reward-based Bradley-Terry (BT) preference model, and extend classical designs utilizing optimism or pessimism. This work, instead, considers the general preference model (whose practical relevance has been observed recently) and obtains performance guarantees with major, order-wise improvements over existing ones. Surprisingly, these results are derived from
algorithms that directly use the empirical estimates (i.e., greedy sampling), as opposed to constructing optimistic or pessimistic estimates in previous works. This insight has a deep root in the unique structural property of the optimal policy class under the KL-regularized target, and we further specialize it to the BT model, highlighting the surprising sufficiency of greedy sampling in RLHF.
Read the full abstract ▾
All authors Di Wu, Chengshuai Shi, Jing Yang, Cong Shen
Machine Learning (Statistics) · Cross-listed from cs.LG
Eigenfunction Extraction for Ordered Representation Learning
Burak Var{\i}c{\i}, Che-Ping Tsai, Ritabrata Ray et al. (+2 more)
Recent advances in representation learning reveal that widely used objectives, such as contrastive and non-contrastive, implicitly perform spectral decomposition of a contextual kernel, induced by the relationship between inputs and their contexts. Yet, these methods recover only the linear span of top eigenfunctions of the kernel, whereas exact spectral decomposition is essential for understanding feature ordering and importance. In this work, we propose a general framework to extract ordered and identifiable eigenfunctions, based on modular building blocks designed to satisfy key desiderata, including compatibility with the contextual kernel and scalability to modern settings. We then show how two main methodological paradigms, low-rank approximation and Rayleigh quotient optimization, align with this framework for eigenfunction extraction. Finally, we validate our approach on synthetic kernels and demonstrate on real-world image
datasets that the recovered
eigenvalues act as effective importance scores for feature selection, enabling principled efficiency-accuracy tradeoffs via adaptive-dimensional representations.
Read the full abstract ▾
All authors Burak Var{\i}c{\i}, Che-Ping Tsai, Ritabrata Ray, Nicholas M. Boffi, Pradeep Ravikumar
Machine Learning (Statistics) · Cross-listed from astro-ph.CO
Bridging Simulators with Conditional Optimal Transport
Justine Zeghal, Benjamin Remy, Yashar Hezaveh et al. (+2 more)
We propose a new field-level emulator that bridges two simulators using unpaired simulation
datasets. Our method leverages a flow-based approach to learn the likelihood transport from one simulator to the other. Since multiple transport maps exist, we employ Conditional Optimal Transport Flow Matching (COT-FM) to ensure that the transformation minimally distorts the underlying structure of the data. We demonstrate the effectiveness of this approach by bridging weak
lensing simulators: a Lagrangian Perturbation Theory (LPT) to a N-body Particle-Mesh (PM). We demonstrate that our emulator captures the full correction between the simulators by showing that it enables full-field inference to accurately recover the true posterior, validating its accuracy beyond traditional summary statistics.
Read the full abstract ▾
All authors Justine Zeghal, Benjamin Remy, Yashar Hezaveh, Francois Lanusse, Laurence Perreault Levasseur
Machine Learning (Statistics) · Cross-listed from cs.DS
Coreset for Robust Geometric Median: Eliminating Size Dependency on Outliers
Ziyi Fang, Lingxiao Huang and Runkai Yang
We study the robust
geometric median problem in Euclidean space $\mathbb{R}^d$, with a focus on coreset construction.A coreset is a compact summary of a
dataset $P$ of size $n$ that approximates the robust cost for all centers $c$ within a multiplicative error $\varepsilon$. Given an outlier count $m$, we construct a coreset of size $\tilde{O}(\varepsilon^{-2} \cdot \min\{\varepsilon^{-2}, d\})$ when $n \geq 4m$, eliminating the $O(m)$ dependency present in prior work [Huang et al., 2022 & 2023]. For the special case of $d = 1$, we achieve an optimal coreset size of $\tilde{\Theta}(\varepsilon^{-1/2} + \frac{m}{n} \varepsilon^{-1})$, revealing a clear separation from the vanilla case studied in [Huang et al., 2023; Afshani and Chris, 2024]. Our results further extend to robust $(k,z)$-clustering in various metric spaces, eliminating the $m$-dependence under mild data assumptions. The key technical contribution is a novel non-component-wise error analysis, enabling substantial reduction of outlier influence, unlike prior methods that retain them.Empirically, our
algorithms consistently outperform existing baselines in terms of size-accuracy tradeoffs and runtime, even when data assumptions are violated across a wide range of
datasets.
Read the full abstract ▾