Machine Learning (Statistics) · Cross-listed from cs.LG
Active Inference as a Convex Markov Decision Process
Nikola Milosevic, Nicolás Hinrichs and Nico Scherf
Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). This perspective reveals that policy-dependent reward prediction errors transmit natural gradients of the expected free energy backwards in time rather than up a hierarchy. Finally, we show that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning. Together this places EFE minimization within modern reinforcement learning and optimization theory and opens a route toward principled
algorithms for active inference.
Read the full abstract ▾
Machine Learning (Statistics)
Directional Kernel Mean Difference: A Fast Signed Statistic for Univariate Distribution Comparison
Shijie Zhong and Jiangfeng Fu
We introduce the Directional Kernel Mean Difference (DKMD), a signed statistic for univariate distribution comparison that preserves the direction of distributional shifts. Unlike the squared Maximum Mean Discrepancy (MMD), which discards directional information by squaring the RKHS distance, DKMD integrates the difference of kernel mean
embeddings against a fixed odd weighting function. This construction yields three structural properties: antisymmetry, immunity to symmetric distributional differences, and directional monotonicity under
stochastic dominance. We derive a data-driven Riemann
estimator that ensures asymptotic consistency with the continuous formulation, strictly preserving the theoretical guarantees of the signed statistic in empirical evaluations. To overcome the quadratic computational cost of kernel methods, we develop an $O(N \log N)$ prefix–suffix scanning
algorithm that exploits the total order of the real line while requiring only $O(N)$
memory. Experiments on synthetic
benchmarks demonstrate that DKMD correctly isolates directional shifts from symmetric perturbations, remains robust to heavy-tailed outliers that can flip the sign of the mean difference, and scales to millions of samples in seconds.
Read the full abstract ▾
Machine Learning (Statistics)
Non–negative matrix factorization using the R package nnmf
Volkan Sevinç, Nikolas Kontemeniotis, Theodoros Perdikis et al. (+1 more)
Non–negative
matrix factorization (NMF) has become an established dimensionality reduction technique for extracting latent structures from non–negative data and has found widespread applications in fields such as bioinformatics, text mining, image analysis, and recommender systems. As the popularity of NMF has increased, numerous R packages implementing different optimization strategies and computational frameworks have been developed. Despite their widespread availability, comprehensive evaluations of these implementations under real–world data conditions remain limited. Consequently, researchers often lack objective guidance when selecting an appropriate package for practical applications. This study introduces a new R package for NMF and offers asystematic performance comparison with two widely available R packages for NMF analysis. Rather than relying on simulated
datasets, the evaluation is conducted using real–world data to better reflect the complexity, heterogeneity, and noise characteristics encountered in practical analytical settings. The packages are assessed using a consistent experimental framework, with emphasis on computational efficiency, convergence behavior, reconstruction accuracy,
memory utilization, and the stability of the resulting
matrix factorization.
Read the full abstract ▾
All authors Volkan Sevinç, Nikolas Kontemeniotis, Theodoros Perdikis, Michail Tsagris
Statistical Methodology
focus and focus-cpt: Fast Online Changepoint Detection in R and Python
Gaetano Romano, Kes Ward, Yuntang Fan et al. (+4 more)
We present an R and Python package for fast online changepoint detection in univariate and multivariate data streams for a variety of models. The package implements the focus family of
algorithms, which compute the Generalised Likelihood Ratio test for a single changepoint exactly and efficiently, with a per-iteration cost of approximately $\log(n)^d$ for a d-dimensional sequence, without introducing approximations. This is achieved by exploiting a connection between the location of the changepoint candidates and the geometry of the data. The package supports a broad range of models from the natural exponential family, including Gaussian, Poisson, Binomial, Exponential and Gamma distributions, as well as a non-parametric detector based on the empirical cumulative distribution function and a detector for autoregressive data.
Read the full abstract ▾
All authors Gaetano Romano, Kes Ward, Yuntang Fan, Guillem Rigaill, Vincent Runge, Idris A. Eckley, Paul Fearnhead
Machine Learning (Statistics) · Cross-listed from cs.LG
Asymptotically Optimal Regret for Reinforcement Learning without Horizon Dependence
Runlong Zhou, Zihan Zhang, Maryam Fazel et al. (+1 more)
We study horizon-free regret minimization for finite-horizon time-homogeneous tabular Markov decision processes with $S$ states, $A$ actions, horizon $H$, and per-trajectory total reward bounded by $1$. We propose a new
algorithm and prove a regret upper bound \[\tilde O(\sqrt{SAK}+S^8A^3)\] with failure probability $\delta$, where $K$ is the number of episodes and $\tilde O(\cdot)$ hides $\mathsf{poly}\log(S,A,K,1/\delta)$. Thus, the regret is $H$-free and asymptotically optimal, matching the contextual-bandit lower bound $\Omega(\sqrt{SAK})$ up to logarithmic factors. This completely removes the $\log H$ dependence from the previous $\tilde O(\sqrt{SAK\log H}+S^2A\log H)$ guarantee of Zhang et al. (2021), and drastically improves the prior best horizon-free regret $\tilde O(\sqrt{S^9A^3K})$ of Zhang et al. (2022) asymptotically. The main technical difficulty is that the optimal value functions $\{V_h^*\}_{h=1}^H$ are time-inhomogeneous even though the transition kernel is time-homogeneous. A direct union bound over all value functions typically incurs an additional $\min\{\log H,S\}$ factor. We avoid this factor by (i) exploiting the monotonicity of $V_h^*$ in $h$ and (ii) non-trivially projecting the value functions onto an $S$-dimensional grid. Our analysis relies on three additional ingredients. First, we introduce a horizon-truncation argument that enables reward-based exploration and removes the cost of a separate reward-free exploration phase. Second, we design a cutting bonus that preserves both optimism and the monotonicity needed for planning. Third, we prove a new bound on total deviation for time-homogeneous MDPs, which controls the clipped variance terms in the cutting bonus with adjustable
polynomial dependence on $S$ and without any dependence on $H$. Together, these tools yield an asymptotically optimal horizon-free regret guarantee.
Read the full abstract ▾
All authors Runlong Zhou, Zihan Zhang, Maryam Fazel, Simon S. Du
Statistics Theory
Finite-horizon quantile martingale posteriors: raw-urn laws and matrix-gain regression
Nam Anh Le
Martingale posteriors quantify uncertainty by forward-imputing observations from one-step-ahead predictive distributions, but implementations stop after finitely many imputations. For the empirical Pólya-urn posterior of a
quantile the law of the stopped state is derived. The
quantile of the stopped urn measure keeps the familiar martingale tail-sum variance fraction; the deployed
stochastic-approximation tracker with frozen gain $c$ does not. Its variance carries an explicit factor $G_a$ with $a=cf_0(q_\tau)$, which may fall below or exceed the tail fraction, and a density-adapted gain restores calibration through a density-free inflation. Shared urn innovations yield the joint law of finitely many
quantile levels. For conditional
quantile regression, a smoothed martingale posterior started at the ordinary
quantile-regression
estimator with a full inverse-Jacobian
matrix gain satisfies a process Bernstein–von Mises
theorem with calibrated finite-horizon bands; scalar or diagonal gains cannot match the sandwich covariance process.
Read the full abstract ▾
Statistical Methodology
Mens: Nonlinear shrinkage estimation in nonparanormal models for financial applications
Hamid Karamikabir and Mohammad Arashi
We develop a theory of nonlinear shrinkage covariance estimation for nonparanormal (Gaussian-copula) models, in which each observed coordinate is an unknown strictly increasing transformation of a latent Gaussian vector. This model accommodates arbitrary marginal skewness and heavy marginal tails while retaining a Gaussian dependence structure, and it is the natural semiparametric setting for heavy-tailed, asymmetric financial returns. Our
estimator, marginal-free nonlinear shrinkage (MENS), applies an oracle nonlinear shrinkage function to the
eigenvalues of the normal-scores rank-covariance
matrix. We give the almost-sure convergence of the empirical spectral distribution of the normal-scores covariance to the generalized Marchenko-Pastur law of Sigma, and asymptotic optimality of MENS among rotation-equivariant
estimators under Frobenius loss. We establish a Baik-Ben Arous-Peche phase transition for spiked latent correlations. The MENS attains the robustness of rank-based estimation and the efficiency of nonlinear shrinkage at once within this class. We corroborate the theory with a simulation study that isolates the marginal-invariance property and the spiked transition. In an out-of-sample minimum-variance backtest on S&P 500 stocks, MENS delivers a better-conditioned covariance estimate, lower realized portfolio
volatility, and lower turnover than linear shrinkage, illustrating its practical value for high-dimensional allocation and decision-making.
Read the full abstract ▾
Statistics Applications
Do In-Match Hydration Breaks Alter Match Momentum? A Within-Match Case-Crossover Analysis of the 2026 FIFA World Cup
Debangan Dey
The 2026 FIFA World Cup was the first to mandate hydration breaks, brief stoppages near the twenty-second minute of each half, in every match regardless of weather. A widespread claim holds that these breaks blunt the momentum of whichever side is dominant. We test the claim with a within-match case-crossover design, using non-break minutes of the same match as self-matched controls. This differences out every match-level characteristic and adjusts only for game time, scoreline, and pre-break momentum state. The apparent post-break fade is not caused by the break: the dominant side's momentum decays at essentially the same rate whether or not a break occurs, reverting through the stoppage as if play had continued. What the break removes instead is the continuation of that side's pressure during the stopped minutes, and this leaves no trace in net expected goals. The average effect is only weakly identified, resting on a trend extrapolation into a window with no untreated minutes, whereas break-by-
covariate interactions are stable; that average is small and near zero. With only 99 matches the design lacks power to reject a small effect, so we read the results by effect size and confidence interval. The estimates are not uniform across match states. The clearest modifier is team strength: the break effect shifts by about +1.6 momentum points per 100 Elo points of rating advantage, favouring a stronger in-form side over a weaker side that is momentarily dominant, and it is the only statistically significant interaction. A weaker pattern appears by scoreline, where a comfortably-leading side of average strength shows an implied loss near -4 points on a pre-break level of about +20. We report these state-dependent signals as exploratory and in need of replication.
Read the full abstract ▾
Other Statistics
A Conditional Quantile Approach to Vector-Valued Bivariate Lorenz Surfaces: Properties and Applications
Shifna P R and S.M. Sunoj
The Lorenz curve is a fundamental tool for measuring inequality, but its extension to multivariate settings remains challenging due to the complex dependence structure among variables and the need to capture directional aspects of inequality. In this paper, we introduce a novel vector-valued bivariate Lorenz surface (VBLS) based on conditional distributions and conditional
quantile functions. Unlike existing symmetric bivariate Lorenz surfaces, the proposed VBLS effectively captures the directional inequality arising from the conditional dependence between two variables. We establish several fundamental properties of the proposed surface and investigate its mathematical properties. The corresponding egalitarian surface is defined, leading to the development of associated vector-valued bivariate Gini measures for quantifying inequality. We further derive characterization results that demonstrate the uniqueness of the proposed VBLS within the underlying distributional framework. Nonparametric
estimators of the VBLS are developed and their finite-sample performance is evaluated through a simulation study. The usefulness of the proposed methodology is also illustrated with applications to income inequality and actuarial data.
Read the full abstract ▾
Statistics Applications
Domain-Adapted Power Curve for Cross-Farm Applications
Ahmadreza Chokhachian, V. Roshan Joseph and Yu Ding
The wind energy industry relies on accurate power curve models to make power forecast, evaluate turbine performance, quantify upgrade, or support site-planning decisions. In this paper, we focus on site-planning power curves, i.e., we investigate how power curve models trained using turbine data on an operating wind farm can be transferred to a new, undeveloped farm. The traditional wisdom in the wind energy literature relies on distance, layout, or terrain characteristics for making cross-farm power curve transfer. Through the lens of domain adaptation, we propose a more reliable transfer learning approach for cross-farm power curve modeling. In the cross-farm applications, a domain is specified by the temporal environmental variates and spatial terrain variables. Domain adaptation is to find a capable similarity metric to adapt the domain on the new farm to that on the existing farm. Empirical results show that our domain adapted power curve consistently outperforms competing approaches by an appreciable margin for site-planning power predictions.
Read the full abstract ▾
Statistical Methodology
Two-Stage Estimation of Population Abundance with Robust Inference under Interacting Survey Protocols
Yusaku Ohkubo, Tatsuki Shimamoto, Yuya Eguchi et al. (+1 more)
Estimating population abundance from field surveys is often complicated by interference between multiple survey protocols. In this paper, we propose a two-stage estimation framework for abundance models in which detection processes interact, leading to both missed detections and sample loss caused by survey procedures. Our approach separates the calibration of sample loss from the estimation of detection probability, thereby avoiding the feedback and weak identifiability that can arise in fully joint hierarchical models. We further derive a sandwich-type robust variance
estimator that propagates first-stage uncertainty into the second stage and remains valid under certain forms of model misspecification. Simulation studies demonstrated that the proposed method provides more reliable uncertainty quantification than a
Bayesian hierarchical joint model, which tends to underestimate uncertainty even under correct specification. We illustrate the practical utility of the method using ectoparasite abundance data from the invasive Pallas's squirrel Callosciurus erythraeus.
Read the full abstract ▾
All authors Yusaku Ohkubo, Tatsuki Shimamoto, Yuya Eguchi, Hirotaka Katahira
Machine Learning (Statistics) · Cross-listed from cs.LG
Efficient Clustering with Quality Guardrails for LLM-based Recommender Systems at Industry Scale
Longshaokan Wang, Wai Tsang Keung, Punit Ghodasara et al. (+3 more)
LLMs can be prohibitively expensive and slow to run at scale, especially for applications that invoke an LLM per sample over millions of inputs. A natural way to scale is to cluster the inputs, run the LLM only on cluster representatives, and propagate the outputs to other cluster members. However, the outputs a member receives are only as good as its match to the representative. Off-the-shelf clustering methods optimize an aggregate objective, targeting average-case quality without per-sample guardrails. As a result, members can be assigned to poorly-matched representatives, and the inherited outputs – though appropriate for the representative – may be irrelevant or even unsafe for the member. For example, a parent of a toddler grouped with parents of older children could receive age-inappropriate recommendations. Most clustering methods also scale poorly to millions of inputs in runtime and
memory, limiting their use at industry scale. We propose a scalable two-stage clustering
algorithm with provable per-sample guardrails: every sample is guaranteed to share a user-specified minimal
embedding similarity and exact attribute match with its representative. The
algorithm first generates initial clusters with Mini-batch K-Means, then greedily selects representatives within each to satisfy the guardrails. We provide theoretical guarantees, complexity analysis, and
benchmarks against common methods on internal and public
datasets. Our method delivers per-sample guardrails while running substantially faster and scaling to data sizes where most standard methods become intractable. We demonstrate its impact in a real-world deployment clustering 38 million customers, reducing downstream LLM cost and runtime by 50-fold while preserving personalization. This unblocked the launch of a persona-based recommender system that delivers significant gains in revenue and engagement in an A/B test.
Read the full abstract ▾
All authors Longshaokan Wang, Wai Tsang Keung, Punit Ghodasara, Roman Wang, Ali Dashti, Francesc Moreno-Noguer
Machine Learning (Statistics)
Data-Poisoning Audits for Causal Effect Estimation
Kwangho Kim
Observational causal analyses increasingly pool records across sites, vendors, and collection systems, creating vulnerability to append-only attacks in which plausible records are strategically selected to alter a reported treatment effect. We develop a data-poisoning audit for augmented inverse-probability-weighted estimation. The analyst specifies a finite catalog of feasible records, an append budget, and nested source capacities, and the adversary selects a feasible subset to maximize movement in a prespecified direction. With preprocessing and nuisance fits held fixed, we propose a greedy scan that computes the exact finite-sample worst-case movement at every append budget. To account for nuisance refitting, we go on to derive a total-influence score combining each record's direct contribution with its effect through the propensity and outcome models. We further obtain a conservative finite-budget bound for the fully refitted estimate. Extensive simulations validate the exact result and show that total influence improves local refit prediction, while multisite and public-data analyses demonstrate material sensitivity at small append budgets. By translating
adversarial data-composition risk into movement curves and critical budgets, the framework supports more reliable causal reporting and the design of source-level safeguards.
Read the full abstract ▾
Machine Learning (Statistics)
Optimal Recalibration of an Online Predictor
Lunjia Hu, Kevin Tian and Chutong Yang
We study the problem of recalibrating an online predictor [KE17, OKS24]: given an arbitrary "hint" sequence of forecasts, the learner must output new predictions that are calibrated while incurring small excess error relative to the original forecasts, under a proper loss. We give an online
algorithm that achieves $(\varepsilon, \varepsilon^2)$-recalibration for Lipschitz proper losses in $T \approx \varepsilon^{-3}$ rounds, using an imbalanced extension of the recent simultaneous Blackwell approachability reduction framework of [HTY26]. We show that this tradeoff is optimal by proving a matching lower bound for recalibrating against the squared loss. We also prove a companion $\mathcal{K}_2$-recalibration
theorem that obtains the same tradeoffs up to a logarithmic factor. As our main application, we show how our recalibration
algorithms can be combined with the online refinement method of [FH23] to obtain simultaneous $\varepsilon$-calibration and $\varepsilon^2$-calibeating for smooth proper losses at the same asymptotic rate, improving upon prior works that achieved these properties separately or with a worse $\varepsilon$ dependence. In particular, the $\mathcal{K}_2$ variant answers a question of [CHJL26] on simultaneously achieving near-optimal calibeating and calibration rates. We also derive extensions to settings with multiple hint sequences. Finally, we empirically evaluate our
algorithms on a classification
dataset undergoing distribution shift.
Read the full abstract ▾
Statistics Applications · Cross-listed from physics.soc-ph
From risk to flourishing: Seeds for responding to a complex world
Ryan M. McGranaghan, Elisabeth H. Krueger, Alexandre Dunant et al. (+6 more)
Complex risk arises when interacting hazards, infrastructures, ecological systems, institutions, and human behaviors produce nonlinear impacts that cannot be inferred from any component in isolation. We call the study of these dynamics complex risk science. Flourishing is the actualization of all beings toward their good–individually and collectively–answering the question of `how do we live well with and for each other?'. In science as in society, the tendency to reduce, segregate, specialize, and settle has precluded embracing the concept of flourishing as a guiding principle. We describe how complex risk science reframes risk science research so that it can promote the flourishing of socio-ecological-technological systems, and explore how its practice may point society toward a more flourishing future. Our exploration mirrors the unsettled and searching nature of flourishing: examine ideologies traditionally believed to be rigidly in opposition and consider what emerges in the space(s) and conversation between them. We call this a dialectical approach: not a means of resolving contradiction, but a practice of sustaining it, so that systems, and our understanding of them, remain capable of reframing as conditions change. Within this dialectical paradigm, we revisit longstanding tensions in risk science: physical|social, determinism|uncertainty, and contextuality|generality. Examining the conversation among these tensions reveals five seeds that are dispositions for practicing complex risk science: interdependence, responsivity, participation, pluralism, and ongoingness. They can support any scientific endeavor to embrace complexity and promote flourishing. Practicing these seeds at scale will depend on institutional reform: creating collectives toward complex risk science, supporting them with skillfully governed commons, and centering care in our research and evaluation.
Read the full abstract ▾
All authors Ryan M. McGranaghan, Elisabeth H. Krueger, Alexandre Dunant, Guillaume Falmagne, Lynee L. Turek-Hankins, Sara M. Constantino, Raissa D'Souza, Elke U. Weber, Graydon W. Manzke
Statistics Applications · Cross-listed from q-bio.NC
Perceived vertical and eye level as one orientation process: a closed-form account of the Li-Matin combination rules for egocentric space
A. Y. Shavit
Li and Matin (2005a, 2005b) reported three regularities of the induced visually-perceived vertical (VPV), the direction judged upright. For small tilts, a peripheral line shifts it nearly linearly with tilt, two lines combine nearly linearly, and mirror-image tilts cancel. Each line votes at twice its angle, weighted by its strength. The votes' vector sum $c_1$ is the first circular moment of the doubled angles; half its angle, $\phi=\tfrac12\arg c_1$, is the induced vertical, a stimulus axis, not the judged upright. This is the orientation order-parameter model (PLUMB). We prove that $c_1$ is, up to a constant factor, the only additive summary of axes (lines without arrowheads) that scales with line strength and turns twice as far as the field rotates. With Li and Matin's own length gain and no per-configuration fit, the model matches their multiline coefficients in-sample. Their data rule out complete summation but not simple averaging for long lines. But a line's effect is fourfold (period $90^\circ$): it reverses past $45^\circ$ (Li and Matin, 2005a; Vingerhoets et al., 2009), and near-horizontal lines shift VPV as near-vertical ones do, which $c_1$ forbids. In a two-harmonic fit to Li and Matin's data, $c_1$ carries an estimated 2–4% of a near-vertical line's slope and a fourfold term the rest. To first order in tilt, for near-vertical segments, the readout equals the committee rule Shavit, Li, and Matin (2026) fit to eight line-array experiments, with its drives read as concentrations. If the half-fields' signals are read as sum and difference, perceived eye level (VPEL) is the difference, recovering Shavit, Li, and Matin's (2013) reversed integration rules; a fourfold term gives the same signals, so this does not single out $c_1$. Re-analysed data (30 observers) show the predicted signs and nulls, an internal check only; the cross-field gain is fitted.
Read the full abstract ▾
Statistics Theory
Estimating the Hüsler–Reiss variogram matrix by clipped moments
Shuang Hu and Johan Segers
In multivariate extreme value analysis, the tail dependence between some of the risk variables at hand may be weak, even when other variables do tend to become large simultaneously. Weak tail dependence may induce a substantial bias in estimation procedures based on the limiting multivariate (generalized) Pareto distribution of excesses over high thresholds. We consider a Hüsler–Reiss multivariate generalized Pareto model and, motivated by this issue, propose first- and second-order moment
estimators of its variogram
matrix constructed from a lower-tail-clipped version of the underlying random vector. The asymptotic normality of the proposed
estimators is established. We demonstrate by simulation studies that they have lower bias than the empirical variogram
estimator in certain cases, particularly when the dependence between components is weak. The
estimators are applied to flood discharge data from the Danube river basin and the US flight delay data, showing that the tail dependence structure implied by the fitted model based on the first-order clipped moment
estimator aligns more closely with the empirical tail dependence of the data than that based on the empirical variogram
estimator.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from econ.EM
A Fixed-Effects Causal Forest for Staggered Adoption, with an Application to Medicaid Expansion
Harry Aytug
Difference-in-differences with staggered adoption identifies group-time average treatment effects ATT(g,t) by comparing each cohort to units not yet treated, which avoids the "forbidden comparisons" that bias two-way fixed-effects
estimators when effects are heterogeneous. This paper studies the
covariate-conditional version of that object, tau_{g,t}(x), and estimates it with a fixed-effects causal forest. Within each (g,t) comparison block, the outcome and treatment are residualized on unit and period fixed effects inside each tree node, and honest causal trees split on treatment-effect heterogeneity in the
covariates. The estimand is not new: Hatamyar, Kreif, Rocha and Huber (2023) introduced it using a doubly-robust R-learner, and Imai, Qin and Yanagi (2023) study it for a single continuous
covariate. What we add is a different way to estimate it. Where those methods remove confounding by modeling nuisance functions, we remove it by differencing out unit and period effects within each tree node, following the fixed-effects residualization of Kattenberg, Scheer and Thiel (2023) and Gavrilova, Langorgen and Zoutman (2025) and carrying it into the Callaway-Sant'Anna group-time structure. In Monte Carlo experiments the
estimator is the only forest-based method that stays unbiased and correctly covered for the overall effect under staggered timing with cohort-varying effects; two-way fixed effects and a pooled causal forest inherit large forbidden-comparison bias. We apply the method to the Callaway-Sant'Anna minimum-wage panel as a validation and to the staggered county-level rollout of the ACA Medicaid expansion, where it recovers an average 2.25 percentage-point fall in the uninsured rate and a conditional surface on which poorer and lower-income counties gained substantially more coverage – heterogeneity measured along socioeconomic
covariates that are not lags of the outcome.
Read the full abstract ▾