Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Wednesday 29 October

Machine Learning (Statistics)

Generative Bayesian Optimization: Generative Models as Acquisition Functions

We present a general strategy for turning generative models into candidate solution samplers for batch Bayesian optimization (BO). The use of generative models for BO enables large batch scaling as generative sampling, optimization of non-continuous design spaces, and high-dimensional and combinatorial design. Inspired by the success of direct preference optimization (DPO), we show that one can train a generative model with noisy, simple utility values directly computed from observations to then form proposal distributions whose densities are proportional to the expected utility, i.e., BO's acquisition function values. Furthermore, this approach is generalizable beyond preference-based feedback to general types of reward signals and loss functions. This perspective avoids the construction of surrogate (regression or classification) models, common in previous methods that have used generative models for black-box optimization. Theoretically, we show that the generative models within the BO process follow a sequence of distributions which asymptotically approximate an optimal target under certain conditions. We also evaluate the performance through experiments on challenging optimization problems involving large batches in high dimensions.
Read the full abstract

Statistical Methodology

Improving time series estimation and prediction via transfer learning

There are many time series in the literature with high dimension yet limited sample sizes, such as macroeconomic variables, and it is almost impossible to obtain efficient estimation and accurate prediction by using the corresponding datasets themselves. This paper fills the gap by introducing a novel representation-based transfer learning framework for vector autoregressive models, and information from related source datasets with rich observations can be leveraged to enhance estimation efficiency through representation learning. A two-stage regularized estimation procedure is proposed with well established non-asymptotic properties, and algorithms with alternating updates are suggested to search for the estimates. Our transfer learning framework can handle time series with varying sample sizes and asynchronous starting and/or ending time points, thereby offering remarkable flexibility in integrating information from diverse datasets. Simulation experiments are conducted to evaluate the finite-sample performance of the proposed methodology, and its usefulness is demonstrated by an empirical analysis on 20 macroeconomic variables from Japan and another nine countries.
Read the full abstract

Statistics Applications · Cross-listed from cs.SI

Stable Emotional Co-occurrence Patterns Revealed by Network Analysis of Social Media

Examining emotion interactions as an emotion network in social media offers key insights into human psychology, yet few studies have explored how fluctuations in such emotion network evolve during crises and normal times. This study proposes a novel computational approach grounded in network theory, leveraging large-scale Japanese social media data spanning varied crisis events (earthquakes and COVID-19 vaccination) and non-crisis periods over the past decade. Our analysis identifies and evaluates links between emotions through the co-occurrence of emotion-related concepts (words), revealing a stable structure of emotion network across situations and over time at the population level. We find that some emotion links (represented as link strength) such as emotion links associated with Tension are significantly strengthened during earthquake and pre-vaccination periods. However, the rank of emotion links remains highly intact. These findings challenge the assumption that emotion co-occurrence is context-based and offer a deeper understanding of emotions' intrinsic structure. Moreover, our network-based framework offers a systematic, scalable method for analyzing emotion co-occurrence dynamics, opening new avenues for psychological research using large-scale textual data.
Read the full abstract

All authors Qianyun Wu, Orr Levy, Yoed N. Kenett, Yukie Sano, Hideki Takayasu, Shlomo Havlin, Misako Takayasu

Statistics Applications

Forecasting Australian Electricity Generation by Fuel Mix

Electricity demand and generation have become increasingly unpredictable with the growing share of variable renewable energy sources in the power system. Forecasting electricity supply by fuel mix is crucial for market operation, ensuring grid stability, optimizing costs, integrating renewable energy sources, and supporting sustainable energy planning. We introduce two statistical methods, centering on forecast reconciliation and compositional data analysis, to forecast short-term electricity supply by different types of fuel mix. Using data for five electricity markets in Australia, we study the forecast accuracy of these techniques. The bottom-up hierarchical forecasting method consistently outperforms the other approaches. Moreover, fuel mix forecasting is most accurate in power systems with a higher share of stable fossil fuel generation.
Read the full abstract

Statistical Methodology

TabMGP: Martingale Posterior with TabPFN

Bayesian inference provides principled uncertainty quantification but is often limited by the challenges of prior and likelihood elicitation. The martingale posterior (MGP) (Fong et al., 2023) offers an alternative by replacing these requirements with a predictive rule. In addition, the MGP focuses inference on parameters defined through a loss function. This framework is especially resonant in the era of foundation transformers; practitioners increasingly leverage models like TabPFN for their state-of-the-art capabilities, yet often require epistemic uncertainty for a scientific estimand $\theta$ that need not parameterise the implicit latent model. The MGP provides a mechanism to recover these posterior distributions. We introduce TabMGP, an MGP built on TabPFN for tabular data. TabMGP produces credible sets with near-nominal coverage and often outperforms both handcrafted MGP constructions and standard Bayesian baselines.
Read the full abstract

All authors Kenyon Ng, Edwin Fong, David T. Frazier, Jeremias Knoblauch, Susan Wei

Statistical Methodology

Bayesian probabilistic projections of proportions with limited data: An application to subnational contraceptive method supply shares

Engaging the private sector in contraceptive method supply is critical for creating equitable, sustainable, and accessible healthcare systems. To achieve this, it is essential to understand where women obtain their modern contraceptives. While national-level estimates provide valuable insights into overall trends in contraceptive supply, they often obscure variation within and across subnational regions. Addressing localized needs has become increasingly important as countries adopt decentralized models for family planning services. Decentralization has also underscored the need for reliable subnational estimates of key family planning indicators. The absence of regularly collected subnational data has hindered effective monitoring and decision-making. To bridge this gap, we propose a novel approach that leverages latent attributes in Demographic and Health Survey (DHS) data to produce Bayesian probabilistic projections of contraceptive method supply shares (the proportions of modern contraceptive methods supplied by public and private sectors) with limited data. Our modeling framework is built on Bayesian hierarchical models. Using penalized splines to track public and private supply shares over time, we leverage the spatial nature of the data and incorporate a correlation structure between recent supply share observations at national and subnational levels. This framework contributes to the domain of subnational estimation of proportions in data-sparse settings, outperforming comparable and previous approaches. As decentralization continues to reshape family planning services, producing reliable subnational estimates of key indicators is increasingly vital for researchers and policymakers.
Read the full abstract

All authors Hannah Comiskey, Niamh Cahill, Leontine Alkema, David Fraizer, Worapree Maneesoonthorn

Machine Learning (Statistics) · Cross-listed from cs.LG

An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentation

The technique of data augmentation (DA) is often used in machine learning for regularization purposes to better generalize under i.i.d. settings. In this work, we present a unifying framework with topics in causal inference to make a case for the use of DA beyond just the i.i.d. setting, but for generalization across interventions as well. Specifically, we argue that when the outcome generating mechanism is invariant to our choice of DA, then such augmentations can effectively be thought of as interventions on the treatment generating mechanism itself. This can potentially help to reduce bias in causal effect estimation arising from hidden confounders. In the presence of such unobserved confounding we typically make use of instrumental variables (IVs) – sources of treatment randomization that are conditionally independent of the outcome. However, IVs may not be as readily available as DA for many applications, which is the main motivation behind this work. By appropriately regularizing IV based estimators, we introduce the concept of IV-like (IVL) regression for mitigating confounding bias and improving predictive performance across interventions even when certain IV properties are relaxed. Finally, we cast parameterized DA as an IVL regression problem and show that when used in composition can simulate a worst-case application of such DA, further improving performance on causal estimation and generalization tasks beyond what simple DA may offer. This is shown both theoretically for the population case and via simulation experiments for the finite sample case using a simple linear example. We also present real data experiments to support our case.
Read the full abstract

All authors Uzair Akbar, Niki Kilbertus, Hao Shen, Krikamol Muandet, Bo Dai

Machine Learning (Statistics) · Cross-listed from math.NA

Energy Approach from $\varepsilon$-Graph to Continuum Diffusion Model with Connectivity Functional

We derive an energy-based continuum limit for $\varepsilon$-graphs endowed with a general connectivity functional. We prove that the discrete energy and its continuum counterpart differ by at most $O(\varepsilon)$; the prefactor involves only the $W^{1,1}$-norm of the connectivity density as $\varepsilon\to0$, so the error bound remains valid even when that density has strong local fluctuations. As an application, we introduce a neural-network procedure that reconstructs the connectivity density from edge-weight data and then embeds the resulting continuum model into a brain-dynamics framework. In this setting, the usual constant diffusion coefficient is replaced by the spatially varying coefficient produced by the learned density, yielding dynamics that differ significantly from those obtained with conventional constant-diffusion models.
Read the full abstract

All authors Yahong Yang, Sun Lee, Jeff Calder, Wenrui Hao

Machine Learning (Statistics) · Cross-listed from cs.LG

Shift is Good: Mismatched Data Mixing Improves Test Performance

We consider training and testing on mixture distributions with different training and test proportions. We show that in many settings, and in some sense generically, distribution shift can be beneficial, and test performance can improve due to mismatched training proportions, even if the components are unrelated and with no transfer between components. In a variety of scenarios, we identify the optimal training proportions and the extent to which such distribution shift can be beneficial. We show how the same analysis applies also to a compositional setting with differing distribution of component "skills” at training and test.
Read the full abstract

All authors Marko Medvedev, Kaifeng Lyu, Zhiyuan Li, Nathan Srebro

Statistical Methodology

Designing a quasi-experiment to study the clinical impact of adaptive risk prediction models

Clinical risk prediction is a valuable tool for guiding healthcare interventions toward those most likely to benefit. Yet, evaluating the pairing of a risk prediction model with an intervention using randomized controlled trials presents substantial challenges, making quasi-experimental designs an attractive alternatives. Existing designs, however, assume that both the model and the decision rules used to trigger interventions (typically a risk threshold) remain fixed. This limits their utility in modern healthcare, where both are routinely updated. We introduce a regression discontinuity framework that accommodates adaptation in both the model and the risk threshold. We precisely characterize the form of interference introduced by these adaptations and exploit this structure to establish conditions for identification and thus design estimation strategies. The key idea is to define counterfactual risks-the scores patients would have received under hypothetical reorderings-thereby restoring local exchangeability and enabling valid estimation of the local average treatment effect. Our estimator leverages the fact that, although counterfactual risk vectors grow with time, they typically lie in a low-dimensional space. In simulations of cardiovascular prevention programs, we show that our method accurately recovers treatment effects even as thresholds adapt to meet operational or clinical targets and models are updated to align predicted and observed outcomes or to exclude demographic predictors such as race.
Read the full abstract

All authors Valerie Odeh-Couvertier, Gabriel Zayas-Caban, Brian Patterson, Amy Cochran

Tuesday 28 October

Statistics Applications · Cross-listed from q-fin.ST

The Evolution of Probabilistic Price Forecasting Techniques: A Review of the Day-Ahead, Intra-Day, and Balancing Markets

Electricity price forecasting has become a critical tool for decision-making in energy markets, particularly as the increasing penetration of renewable energy introduces greater volatility and uncertainty. Historically, research in this field has been dominated by point forecasting methods, which provide single-value predictions but fail to quantify uncertainty. However, as power markets evolve due to renewable integration, smart grids, and regulatory changes, the need for probabilistic forecasting has become more pronounced, offering a more comprehensive approach to risk assessment and market participation. This paper presents a review of probabilistic forecasting methods, tracing their evolution from Bayesian and distribution based approaches, through quantile regression techniques, to recent developments in conformal prediction. Particular emphasis is placed on advancements in probabilistic forecasting, including validity-focused methods which address key limitations in uncertainty estimation. Additionally, this review extends beyond the Day-Ahead Market to include the Intra-Day and Balancing Markets, where forecasting challenges are intensified by higher temporal granularity and real-time operational constraints. We examine state of the art methodologies, key evaluation metrics, and ongoing challenges, such as forecast validity, model selection, and the absence of standardised benchmarks, providing researchers and practitioners with a comprehensive and timely resource for navigating the complexities of modern electricity markets.
Read the full abstract

All authors Ciaran O'Connor, Mohamed Bahloul, Steven Prestwich, Andrea Visentin

Machine Learning (Statistics) · Cross-listed from cs.LG

Quadratic Direct Forecast for Training Multi-Step Time-Series Forecast Models

The design of learning objectives is central to training time-series forecasting models. Existing learning objectives such as mean squared error mostly treat each future step as an independent, equally weighted task, which leads to the following two challenges: (1) they overlook the label autocorrelation effect among future steps, leading to biased learning objectives; (2) they fail to set heterogeneous task weights for different forecasting tasks corresponding to varying future steps, limiting the forecasting performance. To fill this gap, we propose a novel quadratic-form weighted learning objective, addressing both issues simultaneously. Specifically, the off-diagonal elements of the weighting matrix account for the label autocorrelation effect, whereas the non-uniform diagonals are expected to match the preferred weights of the forecasting tasks with varying future steps. On this basis, we propose a Quadratic Direct Forecast (QDF) learning algorithm, which trains the forecast model using the adaptively updated quadratic-form weighting matrix. Experiments show that our QDF effectively improves the performance of various forecast models, achieving state-of-the-art results. Code is available at https://github.com/Master-PLC/QDF.
Read the full abstract

All authors Hao Wang, Licheng Pan, Yuan Lu, Zhichao Chen, Tianqiao Liu, Shuting He, Zhixuan Chu, Qingsong Wen, Haoxuan Li, Zhouchen Lin

Statistics Applications

Systematic Absence of Low-Confidence Nighttime Fire Detections in VIIRS Active Fire Product: Evidence of Undocumented Algorithmic Filtering

The Visible Infrared Imaging Radiometer Suite (VIIRS) active fire product is widely used for global fire monitoring, yet its confidence classification scheme exhibits an undocumented systematic pattern. Through analysis of 21,540,921 fire detections spanning one year (January 2023 - January 2024), I demonstrate a complete absence of low-confidence classifications during nighttime observations. Of 6,007,831 nighttime fires, zero were classified as low confidence, compared to an expected 696,908 under statistical independence (chi-squared = 1,474,795, p < 10^-15, Z = -833). This pattern persists globally across all months, latitude bands, and both NOAA-20 and Suomi-NPP satellites. Machine learning reverse-engineering (88.9% accuracy), bootstrap simulation (1,000 iterations), and spatial-temporal analysis confirm this is an algorithmic constraint rather than a geophysical phenomenon. Brightness temperature analysis reveals nighttime fires below approximately 295K are likely excluded entirely rather than flagged as low-confidence, while daytime fires show normal confidence distributions. This undocumented behavior affects 27.9% of all VIIRS fire detections and has significant implications for fire risk assessment, day-night detection comparisons, confidence-weighted analyses, and any research treating confidence levels as uncertainty metrics. I recommend explicit documentation of this algorithmic constraint in VIIRS user guides and reprocessing strategies for affected analyses.
Read the full abstract

Statistics Applications

Toward precision soil health: A regional framework for site-specific management across Missouri

Effective soil health management is crucial for sustaining agriculture, adopting ecosystem resilience, and preserving water quality. However, Missouri's diverse landscapes limit the effectiveness of broad generalized management recommendations. The lack of resolution in existing soil grouping systems necessitates data driven, site specific insights to guide tailored interventions. To address these critical challenges, a regional soil clustering framework designed to support precision soil health management strategies across the state. The methodology leveraged high resolution SSURGO dataset, explicitly processing soil properties aggregated across the 0 to 30 cm root zone. Multivariate analysis incorporating a variational autoencoder and KMeans clustering was used to group soils with similar properties. The derived clusters were validated using statistical metrics, including silhouette scores and checks against existing taxonomic units, to confirm their spatial coherence. This approach enabled us to delineate soil groups that capture textures, hydraulic properties, chemical fertility, and biological indicators unique to Missouri's diverse agroecological regions. The clustering map identified ten distinct soil health management zones. This alignment of 10 clusters was selected as optimal because it was sufficiently large to capture inherited soil patterns while remaining manageable for practical statewide application. Rooting depth limitation and saturated hydraulic conductivity emerged as principal variables driving soil differentiation. Each management zone is defined by a unique combination of clay, organic matter, pH, and available water capacity. This framework bridges sophisticated data analysis with actionable, site targeted recommendations, enabling conservation planners, and agronomists to optimize management practices and enhance resource efficiency statewide.
Read the full abstract

All authors Dipal Shah, Jordon Wade, Timothy Haithcoat, Robert Myers, Kelly Wilson

2510.26815 · 28 Oct · The preprint is being withdrawn following a later change in my supervisor's position regarding its public release. The submission was made in good faith, based on prior documented discussions and mutual understanding, including verbal consent to proceed. Although the subsequent request for withdrawal was unexpected, I respect the decision

Statistical Methodology

Bayesian Adaptive Polynomial Chaos Expansions

Polynomial chaos expansions (PCE) are widely used for uncertainty quantification (UQ) tasks, particularly in the applied mathematics community. However, PCE has received comparatively less attention in the statistics literature, and fully Bayesian formulations remain rare, especially with implementations in R. Motivated by the success of adaptive Bayesian machine learning models such as BART, BASS, and BPPR, we develop a new fully Bayesian adaptive PCE method with an efficient and accessible R implementation: khaos. Our approach includes a novel proposal distribution that enables data-driven interaction selection, and supports a modified g-prior tailored to PCE structure. Through simulation studies and real-world UQ applications, we demonstrate that Bayesian adaptive PCE provides competitive performance for surrogate modeling, global sensitivity analysis, and ordinal regression tasks.
Read the full abstract

All authors Kellin N. Rumsey, Devin Francom, Graham C. Gibson, J. Derek Tucker, Gabriel Huerta

Statistics Theory · Cross-listed from cs.AI

Cyclic Counterfactuals under Shift-Scale Interventions

Most counterfactual inference frameworks traditionally assume acyclic structural causal models (SCMs), i.e. directed acyclic graphs (DAGs). However, many real-world systems (e.g. biological systems) contain feedback loops or cyclic dependencies that violate acyclicity. In this work, we study counterfactual inference in cyclic SCMs under shift-scale interventions, i.e., soft, policy-style changes that rescale and/or shift a variable's mechanism.
Read the full abstract

Statistical Computation

Bayesian Neural Networks vs. Mixture Density Networks: Theoretical and Empirical Insights for Uncertainty-Aware Nonlinear Modeling

This paper investigates two prominent probabilistic neural modeling paradigms: Bayesian Neural Networks (BNNs) and Mixture Density Networks (MDNs) for uncertainty-aware nonlinear regression. While BNNs incorporate epistemic uncertainty by placing prior distributions over network parameters, MDNs directly model the conditional output distribution, thereby capturing multimodal and heteroscedastic data-generating mechanisms. We present a unified theoretical and empirical framework comparing these approaches. On the theoretical side, we derive convergence rates and error bounds under Hölder smoothness conditions, showing that MDNs achieve faster Kullback-Leibler (KL) divergence convergence due to their likelihood-based nature, whereas BNNs exhibit additional approximation bias induced by variational inference. Empirically, we evaluate both architectures on synthetic nonlinear datasets and a radiographic benchmark (RSNA Pediatric Bone Age Challenge). Quantitative and qualitative results demonstrate that MDNs more effectively capture multimodal responses and adaptive uncertainty, whereas BNNs provide more interpretable epistemic uncertainty under limited data. Our findings clarify the complementary strengths of posterior-based and likelihood-based probabilistic learning, offering guidance for uncertainty-aware modeling in nonlinear systems.
Read the full abstract

Statistical Methodology

Bayesian Geostatistical Modeling for Cluster Randomized Trials

Cluster randomized trials (CRTs) offer a practical alternative for addressing logistical challenges and ensuring feasibility in community health, education, and prevention studies, even though individually-randomized controlled trials are considered the gold standard in evaluating therapeutic interventions. Despite their utility, CRTs are often criticized for limited precision and complex modeling requirements. Advances in robust Bayesian methods and the incorporation of spatial correlation into CRT design and analysis remain relatively underdeveloped. This paper introduces a Bayesian geostatistical framework that models individuals nested within geographic clusters while explicitly accounting for spatial dependence. We demonstrate that conventional non-spatial models are susceptible to underestimating uncertainty and lead to misleading inferences, whereas our spatial approach improves estimation stability, controls type I error, and enhances statistical power. Additionally, we explore design implications that are suggested through the exploration of spatial predictive uncertainty. Our results of simulation and real-world data application demonstrate the value and need for wider adoption of spatial methods in CRTs.
Read the full abstract

All authors Jooyeon Lee, M.S., Evan Kwiatkowski, Ph.D

Machine Learning (Statistics) · Cross-listed from cs.LG

Sequences of Logits Reveal the Low Rank Structure of Language Models

A major problem in the study of large language models is to understand their inherent low-dimensional structure. We introduce an approach to study the low-dimensional structure of language models at a model-agnostic level: as sequential probabilistic models. We first empirically demonstrate that a wide range of modern language models exhibit low-rank structure: in particular, matrices built from the model's logits for varying sets of prompts and responses have low approximate rank. We then show that this low-rank structure can be leveraged for generation – in particular, we can generate a response to a target prompt using a linear combination of the model's outputs on unrelated, or even nonsensical prompts. On the theoretical front, we observe that studying the approximate rank of language models in the sense discussed above yields a simple universal abstraction whose theoretical predictions parallel our experiments. We then analyze the representation power of the abstraction and give provable learning guarantees.
Read the full abstract

Statistical Methodology

Differential Density Analysis in Single-Cell Genomics Using Specially Designed Exponential Families

Recent advances in high-resolution sequencing have paved the way for population-scale analysis in single-cell RNA-sequencing (scRNA-seq) data. scRNA-seq data, in particular, have proven to be extremely powerful in profiling a variety of outcomes such as disease and aging. The abundance of scRNA-seq data makes it possible to model each individual's gene expression as a probability density across cells, offering a richer representation than summary statistics such as means or variances, and allowing for more nuanced group comparisons. To this end, we propose a model-agnostic framework for density estimation and inference based on specially designed exponential families~(SEF), which accommodates diverse underlying models without requiring prior specifications. The proposed method enables estimation and visualization for both individual-specific and group-level gene expression densities, as well as conducting formal hypothesis testing for expression density difference across groups of interest. It relies on relaxed assumptions with established asymptotic properties and a consistent covariance estimator for valid inference. Through simulation under various scenarios, the SEF-based approach demonstrates good error control and improved statistical power over competing methods,including pseudo-bulk tests and moment estimators. Application to a population-scale scRNA-seq dataset from patients with systemic lupus erythematosus identified genes and gene sets that are missed from pseudo-bulk based tests.
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

Topic Analysis with Side Information: A Neural-Augmented LDA Approach

Traditional topic models such as Latent Dirichlet Allocation (LDA) have been widely used to uncover latent structures in text corpora, but they often struggle to integrate auxiliary information such as metadata, user attributes, or document labels. These limitations restrict their expressiveness, personalization, and interpretability. To address this, we propose nnLDA, a neural-augmented probabilistic topic model that dynamically incorporates side information through a neural prior mechanism. nnLDA models each document as a mixture of latent topics, where the prior over topic proportions is generated by a neural network conditioned on auxiliary features. This design allows the model to capture complex nonlinear interactions between side information and topic distributions that static Dirichlet priors cannot represent. We develop a stochastic variational Expectation-Maximization algorithm to jointly optimize the neural and probabilistic components. Across multiple benchmark datasets, nnLDA consistently outperforms LDA and Dirichlet-Multinomial Regression in topic coherence, perplexity, and downstream classification. These results highlight the benefits of combining neural representation learning with probabilistic topic modeling in settings where side information is available.
Read the full abstract

All authors Biyi Fang, Truong Vo, Kripa Rajshekhar, Diego Klabjan

Statistics Theory

Maximum Likelihood Estimation in the Multivariate and Matrix Variate Symmetric Laplace Distributions through Group Actions

In this paper, we study the maximum likelihood estimation of the parameters of the multivariate and matrix variate symmetric Laplace distributions through group actions. The multivariate and matrix variate symmetric Laplace distributions are not in the exponential family of distributions. We relate the maximum likelihood estimation problems of these distributions to norm minimization over a group and build a correspondence between stability of data with respect to the group action and the properties of the likelihood function.
Read the full abstract

Machine Learning (Statistics)

Tree Ensemble Explainability through the Hoeffding Functional Decomposition and TreeHFD Algorithm

Tree ensembles have demonstrated state-of-the-art predictive performance across a wide range of problems involving tabular data. Nevertheless, the black-box nature of tree ensembles is a strong limitation, especially for applications with critical decisions at stake. The Hoeffding or ANOVA functional decomposition is a powerful explainability method, as it breaks down black-box models into a unique sum of lower-dimensional functions, provided that input variables are independent. In standard learning settings, input variables are often dependent, and the Hoeffding decomposition is generalized through hierarchical orthogonality constraints. Such generalization leads to unique and sparse decompositions with well-defined main effects and interactions. However, the practical estimation of this decomposition from a data sample is still an open problem. Therefore, we introduce the TreeHFD algorithm to estimate the Hoeffding decomposition of a tree ensemble from a data sample. We show the convergence of TreeHFD, along with the main properties of orthogonality, sparsity, and causal variable selection. The high performance of TreeHFD is demonstrated through experiments on both simulated and real data, using our treehfd Python package (https://github.com/ThalesGroup/treehfd). Besides, we empirically show that the widely used TreeSHAP method, based on Shapley values, is strongly connected to the Hoeffding decomposition.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning

Weak-to-strong generalization refers to the phenomenon where a stronger model trained under supervision from a weaker one can outperform its teacher. While prior studies aim to explain this effect, most theoretical insights are limited to abstract frameworks or linear/random feature models. In this paper, we provide a formal analysis of weak-to-strong generalization from a linear CNN (weak) to a two-layer ReLU CNN (strong). We consider structured data composed of label-dependent signals of varying difficulty and label-independent noise, and analyze gradient descent dynamics when the strong model is trained on data labeled by the pretrained weak model. Our analysis identifies two regimes – data-scarce and data-abundant – based on the signal-to-noise characteristics of the dataset, and reveals distinct mechanisms of weak-to-strong generalization. In the data-scarce regime, generalization occurs via benign overfitting or fails via harmful overfitting, depending on the amount of data, and we characterize the transition boundary. In the data-abundant regime, generalization emerges in the early phase through label correction, but we observe that overtraining can subsequently degrade performance.
Read the full abstract

Statistical Methodology

Machine-Learning-Assisted Comparison of Regression Functions

We revisit the classical problem of comparing regression functions, a fundamental question in statistical inference with broad relevance to modern applications such as data integration, transfer learning, and causal inference. Existing approaches typically rely on smoothing techniques and are thus hindered by the curse of dimensionality. We propose a generalized notion of kernel-based conditional mean dependence that provides a new characterization of the null hypothesis of equal regression functions. Building on this reformulation, we develop two novel tests that leverage modern machine learning methods for flexible estimation. We establish the asymptotic properties of the test statistics, which hold under both fixed- and high-dimensional regimes. Unlike existing methods that often require restrictive distributional assumptions, our framework only imposes mild moment conditions. The efficacy of the proposed tests is demonstrated through extensive numerical studies.
Read the full abstract

All authors Jian Yan, Zhuoxi Li, Yang Ning, Yong Chen

Machine Learning (Statistics) · Cross-listed from math.OC

A Single-Loop First-Order Algorithm for Linearly Constrained Bilevel Optimization

We study bilevel optimization problems where the lower-level problems are strongly convex and have coupled linear constraints. To overcome the potential non-smoothness of the hyper-objective and the computational challenges associated with the Hessian matrix, we utilize penalty and augmented Lagrangian methods to reformulate the original problem as a single-level one. Especially, we establish a strong theoretical connection between the reformulated function and the original hyper-objective by characterizing the closeness of their values and derivatives. Based on this reformulation, we propose a single-loop, first-order algorithm for linearly constrained bilevel optimization (SFLCB). We provide rigorous analyses of its non-asymptotic convergence rates, showing an improvement over prior double-loop algorithms – form $O(\epsilon^{-3}\log(\epsilon^{-1}))$ to $O(\epsilon^{-3})$. The experiments corroborate our theoretical findings and demonstrate the practical efficiency of the proposed SFLCB algorithm. Simulation code is provided at https://github.com/ShenGroup/SFLCB.
Read the full abstract

All authors Wei Shen, Jiawei Zhang, Minhui Huang, Cong Shen

Machine Learning (Statistics) · Cross-listed from cs.LG

Greedy Sampling Is Provably Efficient for RLHF

Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique for post-training large language models. Despite its empirical success, the theoretical understanding of RLHF is still limited, as learning the KL-regularized target with only preference feedback poses additional challenges compared with canonical RL. Existing works mostly study the reward-based Bradley-Terry (BT) preference model, and extend classical designs utilizing optimism or pessimism. This work, instead, considers the general preference model (whose practical relevance has been observed recently) and obtains performance guarantees with major, order-wise improvements over existing ones. Surprisingly, these results are derived from algorithms that directly use the empirical estimates (i.e., greedy sampling), as opposed to constructing optimistic or pessimistic estimates in previous works. This insight has a deep root in the unique structural property of the optimal policy class under the KL-regularized target, and we further specialize it to the BT model, highlighting the surprising sufficiency of greedy sampling in RLHF.
Read the full abstract

All authors Di Wu, Chengshuai Shi, Jing Yang, Cong Shen

Machine Learning (Statistics) · Cross-listed from cs.LG

Eigenfunction Extraction for Ordered Representation Learning

Recent advances in representation learning reveal that widely used objectives, such as contrastive and non-contrastive, implicitly perform spectral decomposition of a contextual kernel, induced by the relationship between inputs and their contexts. Yet, these methods recover only the linear span of top eigenfunctions of the kernel, whereas exact spectral decomposition is essential for understanding feature ordering and importance. In this work, we propose a general framework to extract ordered and identifiable eigenfunctions, based on modular building blocks designed to satisfy key desiderata, including compatibility with the contextual kernel and scalability to modern settings. We then show how two main methodological paradigms, low-rank approximation and Rayleigh quotient optimization, align with this framework for eigenfunction extraction. Finally, we validate our approach on synthetic kernels and demonstrate on real-world image datasets that the recovered eigenvalues act as effective importance scores for feature selection, enabling principled efficiency-accuracy tradeoffs via adaptive-dimensional representations.
Read the full abstract

All authors Burak Var{\i}c{\i}, Che-Ping Tsai, Ritabrata Ray, Nicholas M. Boffi, Pradeep Ravikumar

Machine Learning (Statistics) · Cross-listed from astro-ph.CO

Bridging Simulators with Conditional Optimal Transport

We propose a new field-level emulator that bridges two simulators using unpaired simulation datasets. Our method leverages a flow-based approach to learn the likelihood transport from one simulator to the other. Since multiple transport maps exist, we employ Conditional Optimal Transport Flow Matching (COT-FM) to ensure that the transformation minimally distorts the underlying structure of the data. We demonstrate the effectiveness of this approach by bridging weak lensing simulators: a Lagrangian Perturbation Theory (LPT) to a N-body Particle-Mesh (PM). We demonstrate that our emulator captures the full correction between the simulators by showing that it enables full-field inference to accurately recover the true posterior, validating its accuracy beyond traditional summary statistics.
Read the full abstract

All authors Justine Zeghal, Benjamin Remy, Yashar Hezaveh, Francois Lanusse, Laurence Perreault Levasseur

Machine Learning (Statistics) · Cross-listed from cs.DS

Coreset for Robust Geometric Median: Eliminating Size Dependency on Outliers

We study the robust geometric median problem in Euclidean space $\mathbb{R}^d$, with a focus on coreset construction.A coreset is a compact summary of a dataset $P$ of size $n$ that approximates the robust cost for all centers $c$ within a multiplicative error $\varepsilon$. Given an outlier count $m$, we construct a coreset of size $\tilde{O}(\varepsilon^{-2} \cdot \min\{\varepsilon^{-2}, d\})$ when $n \geq 4m$, eliminating the $O(m)$ dependency present in prior work [Huang et al., 2022 & 2023]. For the special case of $d = 1$, we achieve an optimal coreset size of $\tilde{\Theta}(\varepsilon^{-1/2} + \frac{m}{n} \varepsilon^{-1})$, revealing a clear separation from the vanilla case studied in [Huang et al., 2023; Afshani and Chris, 2024]. Our results further extend to robust $(k,z)$-clustering in various metric spaces, eliminating the $m$-dependence under mild data assumptions. The key technical contribution is a novel non-component-wise error analysis, enabling substantial reduction of outlier influence, unlike prior methods that retain them.Empirically, our algorithms consistently outperform existing baselines in terms of size-accuracy tradeoffs and runtime, even when data assumptions are violated across a wide range of datasets.
Read the full abstract