Machine Learning (Statistics) · Cross-listed from cs.LG
Chronos-2: From Univariate to Universal Forecasting
Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken et al. (+20 more)
Pretrained time series models have enabled inference-only
forecasting systems that produce accurate predictions without task-specific training. However, existing approaches largely focus on univariate
forecasting, limiting their applicability in real-world scenarios where multivariate data and
covariates play a crucial role. We present Chronos-2, a pretrained model capable of handling univariate, multivariate, and
covariate-informed
forecasting tasks in a zero-shot manner. Chronos-2 employs a group
attention mechanism that facilitates in-context learning (ICL) through efficient information sharing across multiple time series within a group, which may represent sets of related series, variates of a multivariate series, or targets and
covariates in a
forecasting task. These general capabilities are achieved through training on synthetic
datasets that impose diverse multivariate structures on univariate series. Chronos-2 delivers state-of-the-art performance across three comprehensive
benchmarks: fev-bench, GIFT-Eval, and Chronos
Benchmark II. On fev-bench, which emphasizes multivariate and
covariate-informed
forecasting, Chronos-2's universal ICL capabilities lead to substantial improvements over existing models. On tasks involving
covariates, it consistently outperforms baselines by a wide margin. Case studies in the energy and retail domains further highlight its practical advantages. The in-context learning capabilities of Chronos-2 establish it as a general-purpose
forecasting model that can be used "as is" in real-world
forecasting pipelines.
Read the full abstract ▾
All authors Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Pablo Guerron, Tony Hu, Junming Yin, Nick Erickson, Prateek Mutalik Desai, Hao Wang, Huzefa Rangwala, George Karypis, Yuyang Wang, Michael Bohlke-Schneider
Machine Learning (Statistics)
Error analysis of a compositional score-based algorithm for simulation-based inference
Camille Touron, Gabriel V. Cardoso, Julyan Arbel et al. (+1 more)
Simulation-based inference (SBI) has become a widely used framework in applied sciences for estimating the parameters of
stochastic models that best explain experimental observations. A central question in this setting is how to effectively combine multiple observations in order to improve parameter inference and obtain sharper posterior distributions. Recent advances in score-based
diffusion methods address this problem by constructing a compositional score, obtained by aggregating individual posterior scores within the
diffusion process. While it is natural to suspect that the accumulation of individual errors may significantly degrade sampling quality as the number of observations grows, this important theoretical issue has so far remained unexplored. In this paper, we study the compositional score produced by the GAUSS
algorithm of Linhart et al. (2024) and establish an upper bound on its mean squared error in terms of both the individual score errors and the number of observations. We illustrate our theoretical findings on a Gaussian example, where all analytical expressions can be derived in a closed form.
Read the full abstract ▾
All authors Camille Touron, Gabriel V. Cardoso, Julyan Arbel, Pedro L. C. Rodrigues
Machine Learning (Statistics)
On Universality of Deep Equivariant Networks
Marco Pacini, Mircea Petrache, Bruno Lepri et al. (+2 more)
Universality results for equivariant neural networks remain rare. Those that do exist typically hold only in restrictive settings: either they rely on regular or higher-order tensor representations, leading to impractically high-dimensional hidden spaces, or they target specialized architectures, often confined to the invariant setting. This work develops a more general account. For invariant networks, we establish a universality
theorem under separation constraints, showing that the addition of a fully connected readout layer secures approximation within the class of separation-constrained continuous functions. For equivariant networks, where results are even scarcer, we demonstrate that standard separability notions are inadequate and introduce the sharper criterion of $\textit{entry-wise separability}$. We show that with sufficient depth or with the addition of appropriate readout layers, equivariant networks attain universality within the entry-wise separable regime. Together with prior results showing the failure of universality for shallow models, our findings identify depth and readout layers as a decisive mechanism for universality, additionally offering a unified perspective that subsumes and extends earlier specialized results.
Read the full abstract ▾
All authors Marco Pacini, Mircea Petrache, Bruno Lepri, Shubhendu Trivedi, Robin Walters
Statistics Theory
A Simple Geometric Proof of the Optimality of the Sequential Probability Ratio Test for Symmetric Bernoulli Hypotheses
Chirag Pabbaraju, Gregory Valiant and Rishi Verma
This paper revisits the classical problem of determining the bias of a weighted coin, where the bias is known to be either $p = 1/2 + \varepsilon$ or $p = 1/2 - \varepsilon$, while minimizing the expected number of coin tosses and the error probability. The optimal strategy for this problem is given by Wald's Sequential Probability Ratio Test (SPRT), which compares the log-likelihood ratio against fixed thresholds to determine a stopping time. Classical proofs of this result typically rely on analytical, continuous, and non-constructive arguments. In this paper, we present a discrete, self-contained proof of the optimality of the SPRT for this problem. We model the problem as a biased random walk on the two-dimensional (heads, tails) integer
lattice, and model strategies as marked stopping times on this
lattice. Our proof takes a straightforward greedy approach, showing how any arbitrary strategy may be transformed into the optimal, parallel-line "difference policy" corresponding to the SPRT, via a sequence of local perturbations that improve a Bayes risk objective.
Read the full abstract ▾
Statistics Applications
Enhanced Renewable Energy Forecasting using Context-Aware Conformal Prediction
Alireza Moradi, Mathieu Tanneau, Reza Zandehshahvar et al. (+1 more)
Artificial intelligence (AI) is increasingly used to support renewable energy
forecasting and grid operations. As renewable penetration grows, reliable probabilistic
forecasting is becoming essential for managing uncertainty and supporting risk-aware operational decision-making. However, these forecasts often suffer from miscalibration due to temporal variability, changing weather conditions, and heterogeneous operating regimes. In many real-world settings, renewable energy forecasts are provided by external sources, vendors, or independently trained systems, making retraining infeasible because of limited model access or computational constraints. This creates a need for efficient and model-agnostic methods that can improve forecast reliability after they are produced. This paper presents Context-Aware Conformal Prediction (CACP), a framework for calibrating renewable energy forecasts. The proposed method relies on a weighting mechanism during the calibration procedure which assigns higher weights to historical observations that are more similar to the target
forecasting condition. This enables adaptive prediction intervals that reflect local uncertainty regimes without requiring access to, or retraining of, the underlying
forecasting model. Experiments are performed on a large-scale
dataset from National Renewable Energy Laboratory (NREL) day-ahead
solar forecasting, covering multiple systems including MISO, ERCTO, and SPP. The results show that CACP improves the reliability-efficiency tradeoff at both site and system levels compared to NREL's base
forecasting model and the other conformal prediction baselines. These results suggest that CACP can serve as a practical reliability-enhancement layer for trustworthy AI-enabled renewable energy
forecasting and operational decision support.
Read the full abstract ▾
All authors Alireza Moradi, Mathieu Tanneau, Reza Zandehshahvar, Pascal Van Hentenryck
Statistics Applications
Incorporating estimands into meta-analyses of clinical trials
Antonio Remiro-Azócar, Pepa Polavieja, Emmanuelle Boutmy et al. (+9 more)
The estimand framework is increasingly established to pose research questions in confirmatory clinical trials. In evidence synthesis, the uptake of estimands has been modest, and the PICO (Population, Intervention, Comparator, Outcome) framework is more often applied. While PICOs and estimands have overlapping elements, the estimand framework explicitly considers different strategies for intercurrent events. We propose a pragmatic framework for the use of estimands in meta-analyses of clinical trials, highlighting the value of estimands to systematically identify and mitigate key sources of quantitative heterogeneity, and to enhance the applicability or external validity of pooled estimates. Focus is placed on the role of strategies for intercurrent events, within the specific context of meta-analyses for health technology assessment. We apply the estimand framework to a network meta-analysis of clinical trials, comparing the efficacy of semaglutide versus dulaglutide in type 2 diabetes. We explore the impact of a treatment policy strategy for treatment discontinuation or initiation of rescue medication versus a hypothetical strategy for the corresponding intercurrent events. The specification of different target estimands at the meta-analytical level allows us to be explicit about the source of heterogeneity, the intercurrent event strategy, driving any potential differences in results. We advocate for the integration of estimands into the planning of meta-analyses, while acknowledging that potential challenges exist in the absence of subject-level data. Estimands can complement PICOs to strengthen communication between stakeholders about what evidence syntheses seek to demonstrate, and to ensure that the generated evidence is maximally relevant to healthcare decision-makers.
Read the full abstract ▾
All authors Antonio Remiro-Azócar, Pepa Polavieja, Emmanuelle Boutmy, Alessandro Ghiretti, Lise Lotte Nystrup Husemoen, Khadija Rerhou Rantell, Tatsiana Vaitsiakhovich, David M. Phillippo, Jay J. H. Park, Helle Lynggaard, Robert Bauer, Antonia Morga
Statistical Methodology
A Multiclass ROC Curve
Paolo Giudici, Rosa C. Rosciano, Johanna Schrader et al. (+1 more)
This paper introduces a novel methodology for constructing multiclass ROC curves using the multidimensional Gini index. The proposed methodology leverages the established relationship between the Gini coefficient and the ROC Curve and extends it to multiclass settings through the multidimensional Gini index. The framework is validated by means of two comprehensive case studies in health care and finance. The paper provides a theoretically grounded solution to multiclass performance evaluation, particularly valuable for imbalanced
datasets, for which a prudential assessment should take precedence over class frequency considerations.
Read the full abstract ▾
All authors Paolo Giudici, Rosa C. Rosciano, Johanna Schrader, Delf-Magnus Kummerfeld
Machine Learning (Statistics)
Disentanglement of Sources in a Multi-Stream Variational Autoencoder
Veranika Boukun and Jörg Lücke
Variational autoencoders (VAEs) are among leading approaches to address the problem of learning disentangled representations. Typically a single VAE is used and disentangled representations are sought within its single continuous latent space. In this paper, we propose and provide a proof of concept for a novel Multi-Stream Variational Autoencoder (MS-VAE) that achieves disentanglement of sources by combining discrete and continuous latents. The discrete latents are used in an explicit source combination model, that superimposes a set of sources as part of the MS-VAE decoder. We formally define the MS-VAE approach, derive its inference and learning equations, and numerically investigate its principled functionality. The MS-VAE model is very flexible and can be trained using little supervision (we use fully unsupervised learning after pretraining with some labels). In our numerical experiments, we explored the ability of the MS-VAE approach in separating both superimposed hand-written digits as well as sound sources. For the former task we used superimposed MNIST digits (an increasingly common
benchmark). For sound separation, our experiments focused on the task of speaker diarization in a recording conversation between two speakers. In all cases, we observe a clear separation of sources and competitive performance after training. For digit superpositions, performance is particularly competitive in complex mixtures (e.g., three and four digits). For the speaker diarization task, we observe an especially low rate of missed speakers and a more precise speaker attribution. Numerical experiments confirm the flexibility of the approach across varying amounts of supervision, and we observed high performance, e.g., when using just 10% of the labels for pretraining.
Read the full abstract ▾
Statistics Applications
A nonstationary seasonal Dynamic Factor Model: an application to temperature time series from the state of Minas Gerais
Davi Oliveira Chaves, Chang Chiann and Pedro Alberto Morettin
In many scientific fields, such as agriculture, temperature time series are of interest both as explanatory variables and as objects of study in their own right. However, at the state level, incorporating information from all possible locations in an analysis can be overwhelming, while using a summary measure, such as the state-wide average temperature, can result in significant information loss. In this context, using Dynamic Factor Models (DFMs) provides a compelling alternative for analyzing such multivariate time series, as they allow for the extraction of a small number of common factors that capture the majority of the variability in the data. Given that temperature series are typically seasonal, this study applies a nonstationary seasonal DFM to analyze a multivariate temperature time series from the state of Minas Gerais. The results show that the data can be effectively represented by two seasonal factors: the first captures the general seasonal pattern of the state, while the second contrasts the months of highest annual temperatures between two distinct regions.
Read the full abstract ▾
Statistical Methodology
Bayesian Inference for PDE-based Inverse Problems using the Optimization of a Discrete Loss
Lucas Amoudruz, Sergey Litvinov, Costas Papadimitriou et al. (+1 more)
Inverse problems are crucial for many applications in science, engineering and medicine that involve data assimilation, design, and imaging. Their solution infers the parameters or latent states of a complex system from noisy data and partially observable processes. When measurements are an incomplete or indirect view of the system, additional knowledge is required to accurately solve the inverse problem. Adopting a physical model of the system in the form of partial differential equations (PDEs) is a potent method to close this gap. In particular, the method of optimizing a discrete loss (ODIL) has shown great potential in terms of robustness and computational cost. In this work, we introduce B-ODIL, a
Bayesian extension of ODIL, that integrates the PDE loss of ODIL as prior knowledge and combines it with a likelihood describing the data. B-ODIL employs a
Bayesian formulation of PDE-based inverse problems to infer solutions with quantified uncertainties. We demonstrate the capabilities of B-ODIL in a series of synthetic
benchmarks involving PDEs in one, two, and three dimensions. We showcase the application of B-ODIL in estimating tumor concentration and its uncertainty in a patient's
brain from MRI scans using a three-dimensional tumor growth model.
Read the full abstract ▾
All authors Lucas Amoudruz, Sergey Litvinov, Costas Papadimitriou, Petros Koumoutsakos
Statistical Methodology
Robust estimation of polyserial correlation coefficients: A density power divergence approach
Max Welz
The association between a continuous and an ordinal variable is commonly modeled through the polyserial correlation model. However, this model, which is based on a partially-latent normality assumption, may be misspecified in practice, due to, for example (but not limited to), outliers or careless responses. The typically used maximum likelihood (ML)
estimator is highly susceptible to such misspecification: One single observation not generated by partially-latent normality can suffice to produce arbitrarily poor estimates. As a remedy, we propose a novel
estimator of the polyserial correlation model designed to be robust against the adverse effects of observations discrepant to that model. The
estimator leverages density power divergence estimation to achieve robustness by implicitly downweighting such observations; the ensuing weights constitute a useful tool for pinpointing potential sources of model misspecification. The proposed
estimator generalizes ML and is consistent as well as asymptotically Gaussian. As price for robustness, some efficiency must be sacrificed, but substantial robustness can be gained while maintaining more than 98% of ML efficiency. We demonstrate our
estimator's robustness and practical usefulness in simulation experiments and an empirical application in personality psychology where our
estimator helps identify outliers. Finally, the proposed methodology is implemented in free open-source software.
Read the full abstract ▾
Statistical Methodology
Adaptive Influence Diagnostics in High-Dimensional Regression
Abdul-Nasah Soale and Adewale Lukman
An adaptive Cook's distance (ACD) for diagnosing influential observations in high-dimensional single-index models with multicollinearity and outlier contamination is proposed. ACD is a model-free technique built on sparse local linear gradients to temper leverage effects. In simulations spanning low- and high-dimensional design settings with strong correlation, ACD based on LASSO (ACD-LASSO) and SCAD (ACD-SCAD) penalties reduced masking and swamping relative to classical Cook's distance and local influence as well as the DF-Model and Case-Weight adjusted solution for LASSO. Trimming points flagged by ACD stabilizes variable selection while preserving core signals. Applications to two
datasets–the 1960 US cities pollution study and a high-dimensional riboflavin genomics experiment show consistent gains in selection stability and interpretability.
Read the full abstract ▾
Machine Learning (Statistics)
Kernel-Based Evaluation of Conditional Biological Sequence Models
Pierre Glaser, Steffanie Paul, Alissa M. Hummer et al. (+3 more)
We propose a set of kernel-based tools to evaluate the designs and tune the hyperparameters of conditional sequence models, with a focus on problems in computational biology. The backbone of our tools is a new measure of discrepancy between the true conditional distribution and the model's estimate, called the Augmented Conditional Maximum Mean Discrepancy (ACMMD). Provided that the model can be sampled from, the ACMMD can be estimated unbiasedly from data to quantify absolute model fit, integrated within hypothesis tests, and used to evaluate model reliability. We demonstrate the utility of our approach by analyzing a popular
protein design model, ProteinMPNN. We are able to reject the hypothesis that ProteinMPNN fits its data for various
protein families, and tune the model's temperature hyperparameter to achieve a better fit.
Read the full abstract ▾
All authors Pierre Glaser, Steffanie Paul, Alissa M. Hummer, Charlotte M. Deane, Debora S. Marks, Alan N. Amin
Statistical Computation · Cross-listed from cs.CV
Standardization for improved Spatio-Temporal Image Fusion
Harkaitz Goyena, Peter M. Atkinson, Unai Pérez-Goya et al. (+1 more)
Spatio-Temporal Image Fusion (STIF) methods usually require sets of images with matching spatial and spectral resolutions captured by different sensors. To facilitate the application of STIF methods, we propose and compare two different standardization approaches. The first method is based on traditional upscaling of the fine-resolution images. The second method is a sharpening approach called Anomaly Based Satellite Image Standardization (ABSIS) that blends the overall features found in the fine-resolution image series with the distinctive attributes of a specific coarse-resolution image to produce images that more closely resemble the outcome of aggregating the fine-resolution images. Both methods produce a significant increase in accuracy of the Unpaired Spatio Temporal Fusion of Image Patches (USTFIP) STIF method, with the sharpening approach increasing the spectral and spatial accuracies of the fused images by up to 49.46% and 78.40%, respectively.
Read the full abstract ▾
All authors Harkaitz Goyena, Peter M. Atkinson, Unai Pérez-Goya, M. Dolores Ugarte
Statistics Applications
Temporal Functional Factor Analysis of Brain Connectivity
Kyle Stanley, Nicole Lazar and Matthew Reimherr
Many analyses of functional
magnetic resonance imaging (fMRI) examine functional connectivity (FC), or the statistical dependencies among distant
brain regions. These analyses are typically exploratory, guiding future confirmatory research. In this work, we present an approach based on factor analysis (FA) that is well-suited to studying FC. FA is appealing in this context because its flexible model assumptions permit a guided investigation of its target subspace consistent with the exploratory role of connectivity analyses. However, applying FA to fMRI data poses three problems: (1) its target subspace captures short-range spatial dependencies that should be treated as noise, (2) it requires factorization of a massive spatial covariance, and (3) it overlooks temporal dependencies in the data. To address these limitations, we develop a factor model within the framework of functional data analysis–a field which views certain data as arising from smooth underlying curves. The proposed approach (1) uses
matrix completion techniques to filter short-range spatial dependencies out of its target subspace, (2) employs a distributed
algorithm for factorizing large-scale covariance
matrices, and (3) leverages functional regression to exploit temporal dynamics. Together, these innovations yield a comprehensive and scalable method for studying FC.
Read the full abstract ▾
Statistics Applications
Residual Kriging for Regional-Scale Canopy Height Mapping: Insights into GEDI-Induced Anisotropies and Sparse Sampling
Kamel Lahssini, Guerric le Maire, Nicolas Baghdadi et al. (+1 more)
Quantifying aboveground biomass (AGB) is essential in the context of global
climate change. Canopy height, which is related to AGB, can be mapped using machine learning models trained with multi-source spatial data and GEDI measurements. In this study, a comparative analysis of canopy height estimates derived from two models is presented: a U-Net deep learning model (CHNET) and a Random Forest
algorithm (RFH). Both models were trained using GEDI
lidar data and utilized multi-source inputs, including optical, radar, and environmental data. While CHNET can leverage its convolutional architecture to account for spatial correlations, we observed that it does not fully incorporate all the spatial autocorrelation present in GEDI canopy height measurements. By conducting a spatial analysis of the models' residuals, we also identified that GEDI data acquisition parameters, particularly the variability in
laser beam energy combined with the azimuthal directions of the observation tracks, introduce spatial inconsistencies in the measurements in the form of periodic patterns. To address these anisotropies, we considered exclusively GEDI power beams, and we conducted our spatial autocorrelation analysis in the GEDI track azimuthal direction. Next, we employed the residual kriging (RK) spatial interpolation technique to account for the spatial autocorrelation of canopy heights and improve the accuracies of CHNET and RFH estimates. Adding RK corrections improved the performance of both CHNET and RFH, with more substantial gains observed for RFH. The corrections appeared to be localized around the GEDI sample points and the density of usable GEDI information is therefore an important factor in the effectiveness of spatial interpolation. Furthermore, our findings reveal that a Random Forest model combined with spatial interpolation can deliver performance comparable to that of a U-Net model alone.
Read the full abstract ▾
All authors Kamel Lahssini, Guerric le Maire, Nicolas Baghdadi, Ibrahim Fayad
Machine Learning (Statistics)
Geometric Convergence Analysis of Variational Inference via Bregman Divergences
Sushil Bohara and Amedeo Roberto Esposito
Variational Inference (VI) provides a scalable framework for
Bayesian inference by optimizing the Evidence Lower Bound (ELBO), but convergence analysis remains challenging due to the objective's non-convexity and non-smoothness in Euclidean space. We establish a novel theoretical framework for analyzing VI convergence by exploiting the exponential family structure of distributions. We express negative ELBO as a Bregman divergence with respect to the log-partition function, enabling a
geometric analysis of the optimization landscape. We show that this Bregman representation admits a weak monotonicity property that, while weaker than convexity, provides sufficient structure for rigorous convergence analysis. By deriving bounds on the objective function along rays in parameter space, we establish properties governed by the spectral characteristics of the Fisher information
matrix. Under this
geometric framework, we prove non-asymptotic convergence rates for gradient descent
algorithms with both constant and diminishing step sizes.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity
Naoki Yoshida, Satoshi Hayakawa, Yuhta Takida et al. (+3 more)
In this study, we propose an enhancement to the similarity computation mechanism in
multimodal contrastive pretraining frameworks such as CLIP. Prior theoretical research has demonstrated that the optimal similarity metrics between paired modalities should correspond to the pointwise mutual information (PMI) between the two modalities. However, the current implementations of CLIP and its variants fail to fully utilize the underlying linear structure of PMI. We therefore propose KME-CLIP, which leverages this structure through the inner product in a reproducing kernel
Hilbert space (RKHS). We theoretically prove that, under our assumptions, the KME-CLIP similarity can bring the contrastive loss arbitrarily close to its optimal value, which is attained by PMI, as the size of the point set grows, and we empirically evaluate KME-CLIP against CLIP and its kernel-based variants across several
retrieval and classification tasks.
Read the full abstract ▾
All authors Naoki Yoshida, Satoshi Hayakawa, Yuhta Takida, Toshimitsu Uesaka, Hiromi Wakaki, Yuki Mitsufuji
Statistics Applications
AI and analytics in sports: Leveraging BERTopic to map the past and chart the future
Manit Mishra
Purpose: The purpose of this study is to map the body of scholarly literature at the intersection of artificial intelligence (AI), analytics and sports and thereafter, leverage the insights generated to chart guideposts for future research. Design/methodology/approach: The study carries out systematic literature review (SLR). Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) protocol is leveraged to identify 204 journal articles pertaining to utilization of AI and analytics in sports published during 2002 to 2024. We follow it up with extraction of the latent topics from sampled articles by leveraging the topic modelling technique of BERTopic. Findings: The study identifies the following as predominant areas of extant research on usage of AI and analytics in sports: performance modelling, physical and mental health, social media sentiment analysis, and tactical tracking. Each extracted topic is further examined in terms of its relative prominence, representative studies, and key term associations. Drawing on these insights, the study delineates promising avenues for future inquiry. Research limitations/implications: The study offers insights to academicians and sports administrators on transformational impact of AI and analytics in sports. Originality/value: The study introduces BERTopic as a novel approach for extracting latent structures in sports research, thereby advancing both scholarly understanding and the methodological toolkit of the field.
Read the full abstract ▾
Machine Learning (Statistics)
Fast Best-in-Class Regret for Contextual Bandits
Samuel Girard, Aurelien Bibaut, Arthur Gretton et al. (+2 more)
We study the problem of
stochastic contextual bandits in the agnostic setting, where the goal is to compete with the best policy in a given class without assuming realizability or imposing model restrictions on losses or rewards. In this work, we establish the first fast rate for regret relative to the best-in-class policy. Our proposed
algorithm updates the policy at every round by minimizing a pessimistic objective, defined as a clipped inverse-propensity estimate of the policy value plus a variance penalty. By leveraging
entropy assumptions on the policy class and a Hölderian error-bound condition (a generalization of the margin condition), we achieve fast best-in-class regret rates, including polylogarithmic rates in the parametric case. The analysis is driven by a sequential self-normalized maximal inequality for bounded martingale empirical processes, which yields uniform variance-adaptive confidence bounds and guarantees pessimism under adaptive data collection.
Read the full abstract ▾
All authors Samuel Girard, Aurelien Bibaut, Arthur Gretton, Nathan Kallus, Houssam Zenati
Machine Learning (Statistics) · Cross-listed from cs.LG
Adversary-Free Counterfactual Prediction via Information-Regularized Representations
Shiqin Tang, Rong Feng, Shuxin Zhuang et al. (+2 more)
We study counterfactual prediction under assignment bias and propose a mathematically grounded, information-theoretic approach that removes treatment-
covariate dependence without
adversarial training. Starting from a bound that links the counterfactual-factual risk gap to mutual information, we learn a
stochastic representation Z that is predictive of outcomes while minimizing I(Z; T). We derive a tractable variational objective that upper-bounds the information term and couples it with a supervised decoder, yielding a stable, provably motivated training criterion. The framework extends naturally to dynamic settings by applying the information penalty to sequential representations at each decision time. We evaluate the method on controlled numerical simulations and a real-world clinical
dataset, comparing against recent state-of-the-art balancing, reweighting, and
adversarial baselines. Across metrics of likelihood, counterfactual error, and policy evaluation, our approach performs favorably while avoiding the training instabilities and tuning burden of
adversarial schemes.
Read the full abstract ▾
All authors Shiqin Tang, Rong Feng, Shuxin Zhuang, Youzhi Zhang, Hongzong Li
Machine Learning (Statistics) · Cross-listed from cs.LG
Learning to Answer from Correct Demonstrations
Nirmit Joshi, Gene Li, Siddharth Bhandari et al. (+3 more)
We study the problem of learning to generate an answer (or completion) to a question (or
prompt), where there could be multiple correct answers, any one of which is acceptable at test time. Learning is based on demonstrations of some correct answer to each training question, as in Supervised Fine Tuning (SFT). We formalize the problem as imitation learning (i.e., apprenticeship learning) in contextual bandits, with offline demonstrations from some expert (optimal, or very good) policy, without explicitly observed rewards. In contrast to prior work, which assumes the demonstrator belongs to a bounded-complexity policy class, we propose relying only on the underlying reward model (i.e., specifying which answers are correct) being in a bounded-complexity class, which we argue is a strictly weaker assumption. We show that likelihood-maximization methods can fail in this setting, and instead present an approach that learns to answer nearly as well as the demonstrator, with sample complexity logarithmic in the cardinality of the reward class. Our method is similar to Syed and Schapire 2007, when adapted to a contextual bandit (i.e., single step) setup, but is a simple one-pass online approach that enjoys an "optimistic rate" (i.e., $1/\varepsilon$ when the demonstrator is optimal, versus $1/\varepsilon^2$ in Syed and Schapire), and works even with arbitrarily adaptive demonstrations.
Read the full abstract ▾
All authors Nirmit Joshi, Gene Li, Siddharth Bhandari, Shiva Prasad Kasiviswanathan, Cong Ma, Nathan Srebro
Machine Learning (Statistics)
Robust Optimization in Causal Models and G-Causal Normalizing Flows
Gabriele Visentin and Patrick Cheridito
In this paper, we show that interventionally robust optimization problems in causal models are continuous under the $G$-causal Wasserstein distance, but may be discontinuous under the standard Wasserstein distance. This highlights the importance of using generative models that respect the causal structure when augmenting data for such tasks. To this end, we propose a new normalizing flow architecture that satisfies a universal approximation property for causal structural models and can be efficiently trained to minimize the $G$-causal Wasserstein distance. Empirically, we demonstrate that our model outperforms standard (non-causal) generative models in data augmentation for causal regression and mean-variance portfolio optimization in causal factor models.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Particle Dynamics for Latent-Variable Energy-Based Models
Shiqin Tang, Shuxin Zhuang, Rong Feng et al. (+3 more)
Latent-variable energy-based models (LVEBMs) assign a single normalized energy to joint pairs of observed data and latent variables, offering expressive generative modeling while capturing hidden structure. We recast maximum-likelihood training as a saddle problem over distributions on the latent and joint
manifolds and view the inner updates as coupled Wasserstein gradient flows. The resulting
algorithm alternates overdamped Langevin updates for a joint negative pool and for conditional latent particles with
stochastic parameter ascent, requiring no discriminator or auxiliary networks. We prove existence and convergence under standard smoothness and dissipativity assumptions, with decay rates in KL divergence and Wasserstein-2 distance. The saddle-point view further yields an ELBO strictly tighter than bounds obtained with restricted amortized posteriors. Our method is evaluated on numerical approximations of physical systems and performs competitively against comparable approaches.
Read the full abstract ▾
All authors Shiqin Tang, Shuxin Zhuang, Rong Feng, Runsheng Yu, Hongzong Li, Youzhi Zhang
Machine Learning (Statistics)
Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction
Lin Wang
Open world Machine Learning (OWML) aims to develop intelligent systems capable of recognizing known categories, rejecting unknown samples, and continually learning from novel information. Despite significant progress in open set recognition, novelty detection, and continual learning, the field still lacks a unified theoretical foundation that can quantify uncertainty, characterize information transfer, and explain learning adaptability in dynamic, nonstationary environments. This paper presents a comprehensive review of information theoretic approaches in open world machine learning, emphasizing how core concepts such as
entropy, mutual information, and Kullback Leibler divergence provide a mathematical
language for describing knowledge acquisition, uncertainty suppression, and risk control under open world conditions. We synthesize recent studies into three major research axes: information theoretic open set recognition enabling safe rejection of unknowns, information driven novelty discovery guiding new concept formation, and information retentive continual learning ensuring stable long term adaptation. Furthermore, we discuss theoretical connections between information theory and provable learning frameworks, including PAC Bayes bounds, open-space risk theory, and causal information flow, to establish a pathway toward provable and trustworthy open world intelligence. Finally, the review identifies key open problems and future research directions, such as the quantification of information risk, development of dynamic mutual information bounds,
multimodal information fusion, and integration of information theory with causal
reasoning and world model learning.
Read the full abstract ▾
Machine Learning (Statistics)
Recursive Inference for Heterogeneous Multi-Output GP State-Space Models with Arbitrary Moment Matching
Tengjie Zheng, Jilan Mei, Di Wu et al. (+2 more)
Accurate learning of system dynamics is becoming increasingly crucial for advanced control and decision-making in engineering. However, real-world systems often exhibit multiple channels and highly nonlinear transition dynamics, challenging traditional modeling methods. To enable online learning for these systems, this paper formulates the system as Gaussian process state-space models (GPSSMs) and develops a recursive learning method. The main contributions are threefold. First, a heterogeneous multi-output kernel is designed, allowing each output dimension to adopt distinct kernel types, hyperparameters, and input variables, improving expressiveness in multi-dimensional dynamics learning. Second, an inducing-point management
algorithm enhances computational efficiency through independent selection and
pruning for each output dimension. Third, a unified recursive inference framework for GPSSMs is derived, supporting general moment matching approaches, including the extended Kalman filter (EKF), unscented Kalman filter (UKF), and assumed density filtering (ADF), enabling accurate learning under strong nonlinearity and significant noise. Experiments on synthetic and real-world
datasets show that the proposed method matches the accuracy of SOTA offline GPSSMs with only 1/100 of the runtime, and surpasses SOTA online GPSSMs by around 70% in accuracy under heavy noise while using only 1/20 of the runtime.
Read the full abstract ▾
All authors Tengjie Zheng, Jilan Mei, Di Wu, Lin Cheng, Shengping Gong
Statistical Methodology
Nonparametric Testing of Spatial Dependence in 2D and 3D Random Fields
Christian H. Weiß and Philipp Adämmer
We propose a flexible and robust nonparametric framework for testing spatial dependence in two- and three-dimensional random fields. Our approach involves converting spatial data into one-dimensional time series using space-filling
Hilbert curves. We then apply ordinal pattern-based tests for serial dependence to this series. Because
Hilbert curves preserve spatial locality, spatial dependence in the original field manifests as serial dependence in the transformed sequence. The approach is easy to implement, accommodates arbitrary grid sizes through generalized
Hilbert (“gilbert”) curves, and naturally extends beyond three dimensions. This provides a practical and general alternative to existing methods based on spatial ordinal patterns, which are typically limited to two-dimensional settings.
Read the full abstract ▾
Machine Learning (Statistics)
Kernel Regression in Structured Non-IID Settings: Theory and Implications for Denoising Score Learning
Dechen Zhang, Zhenmei Shi, Yi Zhang et al. (+2 more)
Kernel ridge regression (KRR) is a foundational tool in machine learning, with recent work emphasizing its connections to neural networks. However, existing theory primarily addresses the i.i.d. setting, while real-world data often exhibits structured dependencies - particularly in applications like denoising score learning where multiple noisy observations derive from shared underlying signals. We present the first systematic study of KRR generalization for non-i.i.d. data with signal-noise causal structure, where observations represent different noisy views of common signals. By developing a novel blockwise decomposition method that enables precise concentration analysis for dependent data, we derive excess risk bounds for KRR that explicitly depend on: (1) the kernel spectrum, (2) causal structure parameters, and (3) sampling mechanisms (including relative sample sizes for signals and noises). We further apply our results to denoising score learning, establishing generalization guarantees and providing principled guidance for sampling noisy data points. This work advances KRR theory while providing practical tools for analyzing dependent data in modern machine learning applications.
Read the full abstract ▾
All authors Dechen Zhang, Zhenmei Shi, Yi Zhang, Yingyu Liang, Difan Zou
Machine Learning (Statistics)
RankSEG-RMA: An Efficient Segmentation Algorithm via Reciprocal Moment Approximation
Zixun Wang and Ben Dai
Semantic
segmentation labels each pixel in an image with its corresponding class, and is typically evaluated using the Intersection over Union (IoU) and Dice metrics to quantify the overlap between predicted and ground-truth
segmentation masks. In the literature, most existing methods estimate pixel-wise class probabilities, then apply argmax or thresholding to obtain the final prediction. These methods have been shown to generally lead to inconsistent or suboptimal results, as they do not directly maximize
segmentation metrics. To address this issue, a novel consistent
segmentation framework, RankSEG, has been proposed, which includes RankDice and RankIoU specifically designed to optimize the Dice and IoU metrics, respectively. Although RankSEG almost guarantees improved performance, it suffers from two major drawbacks. First, it is its computational expense-RankDice has a complexity of O(d log d) with a substantial constant factor (where d represents the number of pixels), while RankIoU exhibits even higher complexity O(d^2), thus limiting its practical application. For instance, in LiTS, prediction with RankSEG takes 16.33 seconds compared to just 0.01 seconds with the argmax rule. Second, RankSEG is only applicable to overlapping
segmentation settings, where multiple classes can occupy the same pixel, which contrasts with standard
benchmarks that typically assume non-overlapping
segmentation. In this paper, we overcome these two drawbacks via a reciprocal moment approximation (RMA) of RankSEG with the following contributions: (i) we improve RankSEG using RMA, namely RankSEG-RMA, reduces the complexity of both
algorithms to O(d) while maintaining comparable performance; (ii) inspired by RMA, we develop a pixel-wise score function that allows efficient implementation for non-overlapping
segmentation settings.
Read the full abstract ▾
Machine Learning (Statistics)
Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression
Yeichan Kim, Ilmun Kim and Seyoung Park
Transfer learning is a key component of modern machine learning, enhancing the performance of target tasks by leveraging diverse data sources. Simultaneously, overparameterized models such as the minimum-$\ell_2$-norm interpolator (MNI) in high-dimensional linear regression have garnered significant
attention for their remarkable generalization capabilities, a property known as benign overfitting. Despite their individual importance, the intersection of transfer learning and MNI remains largely unexplored. Our research bridges this gap by proposing a novel two-step Transfer MNI approach and analyzing its trade-offs. We characterize its non-asymptotic excess risk and identify conditions under which it outperforms the target-only MNI. Our analysis reveals free-lunch
covariate shift regimes, where leveraging heterogeneous data yields the benefit of knowledge transfer at limited cost. To operationalize our findings, we develop a data-driven procedure to detect informative sources and introduce an ensemble method incorporating multiple informative Transfer MNIs. Finite-sample experiments demonstrate the robustness of our methods to model and data heterogeneity, confirming their advantage.
Read the full abstract ▾