Machine Learning (Statistics)
A Bayesian Vertical Federated Learning Framework for Multivariate Reduced-Rank High-Dimensional Regression
Brigham Halverson, Sharmistha Guha, Jessica Bernard et al. (+1 more)
Federated learning (FL) has emerged as a leading
privacy-preserving framework for collaborative machine learning across decentralized environments. While considerable progress has been made in horizontal
federated learning (HFL), where data with common features is distributed across sites, vertical
federated learning (VFL), where sites share observations across distinct feature sets, remains less explored. Advancing
Bayesian high-dimensional multivariate reduced-rank regression methods for VFL poses unique challenges: (a) stringent
privacy regulations preventing local site data sharing, and (b) fitting local regressions overlooks essential modeling aspects like inter-variable correlations. In contrast HFL allows each site to fit a comparable model independently. We present a novel
Bayesian VFL framework for multivariate high-dimensional reduced-rank regression, termed BayesVFLReg, which enables precise coefficient estimation while safeguarding both feature and response
privacy. Participating sites use a shared random sketching
matrix to compress local variables into
privacy-preserving sketches. A central server collects these sketches where
Bayesian multivariate reduced-rank regression uses Gaussian scale mixture priors. For feature selection, we introduce a single-step post-processing strategy based on mixture-model clustering of the absolute posterior coefficient means to distinguish signal from noise per response variable. BayesVFLReg is computationally scalable for large, high-dimensional
datasets and facilitates efficient variable selection. Theoretically, we establish sharp non-asymptotic bounds on the posterior probability that the fitted density falls within a Hellinger ball centered at the true data-generating density. Comparative simulation studies and real-world data analyses show that BayesVFLReg reliably identifies sparse feature effects, even under feature correlation.
Read the full abstract ▾
All authors Brigham Halverson, Sharmistha Guha, Jessica Bernard, Rajarshi Guhaniyogi
Machine Learning (Statistics) · Cross-listed from cs.LG
Monotone-Constrained Diffusion Models for Long-Horizon Production Forecasting
Temesgen Mikael Abraha and Yves Lucet
Forecasting a long horizon from only the first observations of a sequence is ill-posed: many trajectories are consistent with the same short history. We study this problem in oil and gas production forecasting, where forecasts made after roughly the first fifth of a well's producing life drive development and abandonment decisions, and where a usable forecast must describe a monotone decline. We present Physics-SIMS-TS, a conditional
diffusion forecaster that combines negative guidance against synthetic artifacts, decline-curve constraints and an isotonic projection applied during sampling, spatial training augmentation, and an ensembled
stochastic sampler yielding a full predictive distribution. Across three jurisdictions and more than 35,000 wells, under a shared-space, validation-frozen protocol, Physics-SIMS-TS is the most accurate
diffusion forecaster in the comparison and is competitive with, but not superior to, ensembled
transformer forecasters. Its forecasts are monotone by construction at a cost of at most 0.5% in mean squared error, and its trajectory ensemble yields calibrated intervals after one dispersion factor is fitted per jurisdiction. On six standard
benchmarks a reversible-instance-normalization variant of the backbone is the leading
diffusion baseline. We also quantify four protocol choices on which the measured ranking depends. Code and evaluation artifacts are released.
Read the full abstract ▾
Machine Learning (Statistics)
Locally Private Inference for Riemannian Stochastic Optimization
Xiaotian Chang, Yangdi Jiang and Qirui Hu
We develop inference for
manifold-valued population minimizers when each observation belongs to a different participant and only locally private messages reach the analyst. The method releases randomized tangent gradients and combines them through Riemannian
stochastic approximation and Polyak-Ruppert averaging. Directly inserting a private data surrogate into a nonlinear loss can shift its population target, whereas conditional centring of the released gradient preserves the first-order equation. We introduce symmetric-pair regression (SPR) to estimate the asymptotic variance from the same private messages used for point estimation, without holding out participants or requesting a second release. We prove the central limit
theorem and consistency of the fully transcript-based sandwich covariance and intrinsic Wald region under local differential
privacy. Simulations across various statistical problems and
manifolds support the predicted decrease in estimation error and near-nominal coverage under moderate
privacy. An application to NHANES anthropometric data illustrates private estimation of a leading body-size direction and its uncertainty.
Read the full abstract ▾
Statistics Applications
Adapting Pairs Trading to Gambling Markets A Case Study of the U.S. Presidential Election
Haoyu Liu, Len Thomas, Benjamin Baer et al. (+1 more)
Pairs trading exploits mean reversion in the relationship between related assets. We adapt this idea to political betting
markets by modelling the combined implied probability of the two major-party nominees with a latent Ornstein-Uhlenbeck process whose mean-reversion level varies over time and whose observations contain additive noise. Model parameters are estimated from regularly sampled odds data using a state-space likelihood, with consecutive repeated values represented by a single retained observation and the elapsed number of sampling intervals preserved in the continuous-time transition. Parametric-bootstrap upper prediction bounds identify signal times at which the combined implied probability is likely to decline, and a no-intercept Bradley-Terry-type model selects the candidate-specific odds quote. The candidate-selection model is trained on 2020 U.S. presidential-election data and evaluated out of sample on 2024 data. The 2024 analysis produced 130 signals, empirical one-step coverage of 95.1%, a mean synthetic odds-price return of 1.86%, and an unannualized per-trade Sharpe-type ratio of 1.12. These returns are frictionless descriptive quantities rather than executable betting-exchange profits. The results support the integrated framework as a proof of concept for two-candidate electoral
markets.
Read the full abstract ▾
All authors Haoyu Liu, Len Thomas, Benjamin Baer, Carl Donovan
Statistical Methodology
Orthogonal Policy Learning with Ordinal Outcomes
Yue Zhang, Shanshan Luo and Yangbo He
Policy learning methods based on conditional average treatment effects can obscure subpopulation heterogeneity when applied to ordinal outcomes. We develop a policy learning framework for ordinal outcomes with heterogeneous utilities for individuals who strictly benefit from treatment and those who do not. Since the probability of strict benefit is only partially identified without further conditions, we adopt a minimax strategy which minimizes the worst-case regret over the identification region. To estimate the resulting nonsmooth objective, we combine smooth approximations with Neyman orthogonalization to remove first-order bias from nuisance estimation. We also derive an excess worst-case regret bound over a restricted policy class. The proposed method is validated through extensive simulations and an application to the 2022 Survey of Income and Program Participation (SIPP) data.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Classification with Abstention Under Class-Conditional Error Constraints
Mohammadreza M. Kalan, Yuyang Deng and Sanaz Hamidi
We study binary classification with abstention under separate class-conditional error constraints, with the objective of minimizing abstention while keeping both errors below prescribed thresholds. We characterize the distribution-free minimax rate of excess abstention risk, up to logarithmic factors, in terms of the complexity of the hypothesis class and the sample size. To make the framework amenable to computation with models such as neural networks, we introduce surrogate-loss formulations and derive finite-sample guarantees for excess surrogate ambiguity risk. We formulate the resulting learning task as a constrained optimization problem and characterize its computational complexity in the convex setting. Finally, we evaluate our approach on various
datasets and compare its performance with a competing method for this problem.
Read the full abstract ▾
Statistics Theory
Exact maximum likelihood inference for drifted multi-sub-fractional Brownian motion at discrete observation
Afrah Al-Harby, Ezzedine Mliki and Manal Al-Ohali
Sub-fractional Brownian motion is self-similar and long-range dependent but has no stationary increments, so the increment covariance is not Toeplitz and no spectral density is available. We show that a complete finite-sample likelihood theory survives nonetheless. The model is a linear trend observed at $N$ equidistant times through a superposition of $m$ independent sub-fractional Brownian motions with known Hurst indices and a common scale. Nondegeneracy follows from realising the process as the even part of a two-sided fractional Brownian motion, and the maximum likelihood
estimators of the trend and of the scale are explicit. The statistics on which inference rests are pivotal, their laws depending on the sample size alone, so intervals and tests of exact level are available at every $N\ge2$, together with complete sufficiency, minimum variance unbiasedness and attainment of the Cramér–Rao bound. An explicit variance bound gives strong consistency and asymptotic normality, and simulations confirm the exact coverage and the predicted effect of a misspecified Hurst vector.
Read the full abstract ▾
Statistical Methodology
Stochastic Epidemic Model Criticism with Neural Evidence Estimation
Prayag Chatha, Fan Bu and Jon Zelner
Neural techniques for simulation-based inference (SBI) have become popular in scientific domains featuring complex simulation models. These methods train neural networks on simulated data to infer unknown parameters; their reliability depends on the simulator faithfully representing reality. In infectious disease modeling, multiple models encoding different hypotheses often provide plausible explanations of observed data. Model selection is a difficult problem when likelihoods are unavailable, as is the case with many common
stochastic transmission models. We propose a model criticism methodology designed for SBI, Neural Evidence Estimation (NEE), that integrates model selection and model misspecification detection. NEE learns a model's marginal likelihood (a.k.a. evidence) directly from simulations using normalizing flows, thereby avoiding noisy or complex Monte Carlo
estimators. Through synthetic experiments, we demonstrate that NEE reliably selects the correct data generating process among multiple models and flags poorly specified models. We apply NEE to a real-world modeling problem concerning competing mechanisms for influenza reinfection in a multiple wave epidemic. This work argues for the importance of principled model criticism in SBI and infectious disease modeling.
Read the full abstract ▾
Machine Learning (Statistics)
RLVR is a Kernel, Not a Function: Statistical Inference for pass@$k$ Crossovers
Chen Yang and Jun Chen
Reinforcement learning with verifiable rewards (RLVR) often improves pass@1 while falling behind its base model at larger sampling budgets $k$, a crossover read as evidence that RLVR only sharpens existing capability. We identify two limits to this reading. First, a visible crossing need not be statistically established: comparing models on the same
prompts, we build confidence bands across sampling budgets $k$ that require evidence of both an early gain and a later loss. Across five public RLVR pairs no crossing is statistically established in the initial evaluations, while a 32k-
token evaluation on fresh
prompts locates a reversal with first loss between 11 and 61 samples; power analysis shows why failure to detect a crossing need not mean no crossing, and why more
prompts can help more than more answers per
prompt. Second, base success alone does not determine what RLVR does to a
prompt:
prompts with the same base success rate have different post-RL success rates, and these differences repeat across independent generation halves. The relationship is a conditional distribution—a Markov kernel—rather than a single curve, and fitting it predicts crossings in independent generations for the same
prompts and corrects the simple model's power estimates. Theory further shows how losses on a minority of the hardest
prompts can overturn an early lead even when training improves other
prompts, separating evidence that a crossover exists from claims about what it means for capability.
Read the full abstract ▾
Statistical Methodology
Adaptive online kernel changepoint detection
Qianqian Jiang and Dean Bodenham
We propose an adaptive online kernel-based changepoint detection method for streaming data that is capable of detecting a broad range of changes in the underlying data distribution. The method maintains a recursively-weighted reproducing kernel
Hilbert space representation of observations and adaptively updates the forgetting factor through a gradient-based procedure driven by a maximum mean discrepancy-type statistic between the current observation and the weighted empirical distribution of the past. This self-tuning mechanism allows the detector to adapt its effective
memory and responsiveness to changes in the underlying process. Simulation results demonstrate that the proposed method achieves strong detection performance across a wide range of distributional changes. Further, our proposed approach maintains constant computational and storage cost through recursive updates, and is very computationally efficient in comparison to competing methods. Experiments on both simulated data and
benchmark real-world
datasets show improved performance over several other leading kernel-based methods.
Read the full abstract ▾
Statistical Computation · Cross-listed from physics.comp-ph
Stochastic consensus dynamics for decentralized decision systems
André L. M. Vilela, Caio B. L. Silva, Kenric P. Nelson et al. (+2 more)
A fundamental mechanism underlying collective phenomena in social, technological, and economic systems is decentralized decision-making, in which the behavior of individual
agents follows from local interactions in the absence of central coordination. Consensus formation is a central feature of such systems, with relevance to blockchain networks, distributed artificial intelligence, autonomous multi-
agent systems, distributed control, and collective decision networks. We investigate consensus formation using a
stochastic consensus model on random networks and examine how initial conditions, network connectivity, and system size shape the emergence of unanimous states. We show that the
stochastic dynamics strongly amplify small initial majorities, progressively suppress the competing state, and drive the system toward a predictable collective outcome. Network connectivity primarily controls the efficiency of this process: increasing connectivity accelerates the propagation of local agreement and reduces the likelihood that fluctuations reverse the initially dominant state, although these gains gradually saturate in highly connected networks. System size produces a complementary effect. Larger networks require more individual updates to reach unanimity, but they are also increasingly reliable in selecting the state favored by the initial majority. Finite-size analysis shows that the range of initial conditions associated with uncertain outcomes becomes progressively narrower as the network grows, decreasing approximately with the inverse square root of the system size. These results reveal a collective amplification mechanism by which weak initial asymmetries become increasingly decisive in large decentralized networks, and they provide a simple framework for understanding the efficiency, predictability, and reliability of consensus formation in distributed decision systems.
Read the full abstract ▾
All authors André L. M. Vilela, Caio B. L. Silva, Kenric P. Nelson, Emilio Cobanera, Gaogao Dong
Statistics Theory
Asymptotics and finite sample bounds for prediction and smoothing in Wright-Fisher hidden Markov models
Luigi M. Malgieri, Filippo Ascolani, Matteo Giordano et al. (+1 more)
We study signal prediction and smoothing in hidden Markov models whose latent signal is a multi-type Wright-Fisher
diffusion and whose discrete-time observations are categorical. We consider a repeated-sampling regime with finitely many inspection times and large within-time samples. Under parent-independent mutation, the
Bayesian predictive and smoothing laws admit exact representations as mixtures of Dirichlet distributions. We show that the one-step predictive and local smoothing laws converge in total variation to the corresponding Wright-Fisher transition and bridge laws determined by the neighboring population frequencies, thereby justifying plug-in approximations based on empirical proportions. For prediction, we derive explicit finite-sample bounds for this approximation and for bounded predictive functionals such as diversity, together with a finite-sample concentration bound for the marginal smoother. The joint posterior at the inspection times concentrates at the target frequencies and satisfies a Bernstein-von Mises limit with distinct interior and boundary regimes, with Gaussian and Gamma limits on different scales. The results allow arbitrary dependence across inspection times. A key technical ingredient is a fixed-interval tail bound for the coalescent-based dual death process.
Read the full abstract ▾
All authors Luigi M. Malgieri, Filippo Ascolani, Matteo Giordano, Matteo Ruggiero
Statistics Applications
County-Level Heterogeneity in Opioid Harm Reduction and Treatment Effects: A Simulation Modeling Analysis
Abdulrahman A. Ahmed, M. Amin Rahimian, Qiushi Chen et al. (+1 more)
Opioid overdose deaths remain a severe public health crisis in the US, with heterogeneous burden across counties that differ in epidemic trajectory, baseline resources, and local context. While harm reduction through naloxone distribution and buprenorphine treatment are both evidence-based strategies, limited information on county-level effects hinders the ability of policymakers to prioritize resources across counties. We developed a simulation model of opioid use disorder (OUD), calibrated separately to six Pennsylvania counties spanning large urban (Allegheny, Philadelphia), intermediate-sized (Erie, Dauphin), and rural (Clearfield, Columbia) settings. We projected county-specific overdose mortality trajectories under three levels of increase in buprenorphine dispensing and naloxone distribution (10%, 20%, and 30% above each county's baseline), over a five-year horizon from 2025 to 2029. A 30% increase in naloxone distribution above observed county baseline levels was projected to reduce 2029 overdose deaths by approximately 70% (95% uncertainty interval, UI: 55-81%) in Allegheny County, 11% (95% UI:4-17%) in Erie County, and 28% (95% UI:2-64%) in Clearfield County. Projected reductions in overdose deaths from increasing buprenorphine were consistently smaller (10%-23%), except that in Erie buprenorphine produced larger projected reduction by 20% vs 11% for naloxone. Heterogeneity in naloxone responsiveness was strongly associated with each county's historical naloxone dispensing variability. The same proportional increase in naloxone distribution yields substantially different projected mortality reductions across counties depending on each county's baseline distribution history, a pattern invisible from mortality statistics alone. County-level context is important for informing harm reduction and treatment prioritization at the county level.
Read the full abstract ▾
All authors Abdulrahman A. Ahmed, M. Amin Rahimian, Qiushi Chen, Praveen Kumar
Statistical Methodology
Bridging Local and Population Causal Effects: A Proximal Instrumental Variable Approach
Zixuan Yao and Guosheng Yin
Instrumental variable (IV) methods address treatment endogeneity, but with non-compliance and heterogeneous treatment effects a binary instrument generally identifies the local average treatment effect (LATE) among compliers rather than the population average treatment effect (ATE). When treatment effects and compliance probabilities are heterogeneous and dependent through latent factors, the ATE need not be identified by IV variation alone. We develop a proximal instrumental variable framework that uses proxies for these factors to adjust for the compliance weighting in LATE and recover the ATE. We show that the ATE is identified through either an outcome bridge capturing latent treatment effect heterogeneity or a compliance bridge representing inverse compliance probabilities, and combining the two yields a doubly robust representation. We derive the efficient influence function under known and unknown instrument propensities. For estimation, we propose regularized kernel minimax bridge
estimators, with orthogonalized conditional moments for estimated propensity, and a three-way sequential cross-fitting scheme. We establish projected bridge-risk bounds, and give conditions for asymptotic normality and semiparametric efficiency. The framework provides a proxy-based route from local IV effects to population causal effects under latent treatment effect and compliance heterogeneity.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.SI
Auditing bipartite motif interpretations: a worked example with conservation checks and open-path decomposition
Tengfei Shao
Motif profiles of bipartite
agent-object networks, such as tourist-site visits and customer-item transactions, are read as evidence about structural roles and about differences between networks, often without asking what the two degree sequences already fix. In a simple bipartite graph the induced k-fan count on one node type is a sum of degree combinations, so it has zero variance under a null that preserves both degree sequences. We apply this known result to a reconstructed tourism rating network of 17 tourists, 80 sites and 637 edges, the sole inferential worked example, and, as a provenance-limited illustration, to published motif-instance aggregates over 36 monthly luxury customer-item networks. The four fan classes are exact functions of the degree sequences: in the tourism network the raw fan counts and the size-3 two-fan ratio (84.7% fan-out) restate those sequences. The published luxury counts require at least 89,502 customer-item edges against 26,451 reported transactions, so their 99.8% fan-in is reported as a descriptive value only. Against a hard bipartite configuration null, the four-cycle count is degree-consistent (z about +1.0) and the open path is deficient (z about -5.8) by 2,243 instances, 2.4% of the null mean; the deficit survives every leave-one-tourist-out re-run (z -4.5 to -8.1). An exact identity splits it at the point estimate into 64.4% mixing and 35.6% four-cycle, but that split is not an attribution: the observed mixing term lies below all 500 null samples, the two components are almost collinear under the null (r = 0.968), and the mixing share ranges from 36.6% to 116.5% under leave-one-tourist-out deletion. The deficit is extreme relative to the sampled null, while its class-level interpretation is undetermined and unstable. We give a four-step pre-interpretation check and a reference implementation.
Read the full abstract ▾
Statistical Methodology
RKHS-Based Inference for Nonlinear Granger Causality via Conditional Centering
Yuhan Tian, Adam Waterbury and Marie-Christine Düker
Granger causality is commonly formulated through linear prediction in vector autoregressive models, limiting its ability to detect nonlinear predictive relationships. We propose a reproducing kernel
Hilbert space (RKHS)-based test for nonlinear Granger non-causality in conditional mean for nonlinear autoregressive processes. The key idea is a conditional-centering decomposition of the target regression function into an own-history component and an orthogonal component capturing the additional predictive contribution of the potential source history. Non-causality is characterized by the vanishing of the latter component. The own-history component is estimated by kernel ridge regression, and the residuals are embedded in a second RKHS using a conditionally centered kernel. This yields an RKHS-valued residual moment whose squared norm forms the test statistic. We establish a weighted chi-square null limit and consistency against fixed alternatives for the population-centered statistic. An empirically centered version is shown to retain the null limit and enables spectral calibration of critical values and $p$-values without resampling. Simulation studies demonstrate accurate size control and power against nonlinear alternatives, and real-data applications illustrate the usefulness of the method for detecting nonlinear predictive relationships in time series.
Read the full abstract ▾
Statistical Methodology
A Novel Convolution-Based Stratified Attribute Estimator for QRE Determination in R&D Tax Credit Studies
Deborah Lynn Goldwasser
The One Big, Beautiful Bill Act reinstates immediate expensing of domestic research expenses, reducing the tax burden and incentivizing reinvestment in U.S.-based research and development including software development across a wide range of industries. In this context, accurate and defensible estimation of qualified research expenses (QREs) is of high importance. Statistical sampling provides a practical framework for estimating QREs for a well-defined population of business components (sampling frame) documented in accordance with Internal Revenue Code Section 41 (Form 6765). IRS Revenue Procedure 2011-42 permits both attribute and variable statistical methods, although the latter (stratified mean and difference
estimators) are often regarded as the standard approach to QRE estimation. In this paper, we compare attribute and variable statistical methods within a simulation study and demonstrate that attribute methods offer several theoretical and practical advantages. A key concern with the simple attribute
estimator is the possibility of upward bias in QRE determination arising from a preponderance of high potential QRE (pQRE), non-qualified projects in the sampling frame. We address this concern by introducing a stratified sampling design that ensures adequate representation of high pQRE projects in the sample. We demonstrate that a convolution-based stratified attribute
estimator produces a valid one-sided 95% lower confidence bound on total QREs across a range of sampling frame structures.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.CL
Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment
Maciej Skorski
Most work in computational ethics treats annotator disagreement on moral content as noise to be voted away, collapsed into majority vote or the more permissive any-annotator rule the moment a single annotator flags an item. We argue this uncertainty should instead be modeled and learned from. We introduce Moral
Entropy, a
Bayesian framework that keeps a full posterior over the true label and decomposes its
entropy into aleatoric uncertainty (irreducible disagreement about the moral content) and epistemic uncertainty (from insufficient or noisy annotation) – and lets any heuristic consensus rule be audited against a calibrated ground truth via
entropy methods such as cross-
entropy/KL, Brier score, and expected calibration error. Across three corpora and fifteen discourse domains, auditing the standard aggregation rules against this posterior reveals bias that no current pipeline reports: the any-annotator rule disagrees with the calibrated posterior on roughly 30% of items – pooled, almost entirely false positives, though the errors invert at the foundation level (19.9%/38.9% mean FPR/FNR on MFTC) – while the stricter majority and two-vote rules miss 63-83% of true positives.
Read the full abstract ▾
Statistical Methodology
Integrating Multivariate Adaptive Regression Splines into Small Area Estimation for Nonlinear Poverty Modeling in Java Island
Ajriansyah, Bambang Widjanarko Otok and Sutikno
Small area estimation (SAE) modeling is often used to address the unreliability of estimates in areas with limited survey samples. However, standard SAE models (Fay-Herriot) can only accommodate linear patterns, whereas in real cases, including poverty, many patterns are nonlinear. This study proposes the implementation of a supervised learning model, multivariate adaptive regression splines (MARS), integrated within the SAE framework (MARS-SAE) to address the challenge of unreliable direct estimation, particularly in nonlinear cases. The hyperparameter tuning results yielded the best model with the following hyperparameter combinations: maximum basis function (Max BF) = 10, maximum interaction (MI) = 2, minimum observation (MO) = 1, penalty = 3, with general cross validation (GCV) = 6.128. MARS-SAE demonstrated very strong performance; empirically, the model proved more efficient at reducing estimation error due to its higher relative efficiency (RE) compared to the SAE Fay-Herriot model, as well as better reduction of relative standard error (RSE) than the comparison model, and it yielded a lower relative root mean squared error (RRMSE) compared to the SAE Fay-Herriot model (13.64% vs. 13.75%). MARS-SAE has been proven to capture nonlinear patterns very well while still maintaining interpretability, making it an effective tool for the predictive and diagnostic analyses of phenomena. Thus, this research can serve as an important reference for readers regionally and in Southeast Asia, given that Java Island is a strategic region for Indonesia, the largest economic power in the ASEAN region.
Read the full abstract ▾
Statistical Methodology
Consistency of Optimal Matching-based Clustering for Mixtures of Markov Chains
Ottavio Khalifa
We study clustering of categorical sequences using the Optimal Matching (OM) distance under finite mixtures of finite-state Markov chains. We show that the normalized OM distance between two independent chains converges almost surely to a deterministic population quantity, concentrates exponentially around its finite-horizon mean, and admits an $O(\sqrt{\log n/n})$ convergence rate when the two chains have the same transition kernel. These population quantities yield a natural separation condition: the largest within-component limit must be smaller than the smallest between-component limit. Under this condition, hierarchical clustering with any bracketed linkage and Partitioning Around Medoids consistently recover the latent mixture partition. We also propose a consistent
estimator of the number of components based on empirical OM distance profiles. The results extend to finite-state hidden Markov models and multichannel categorical observations. Overall, they provide a statistical justification for standard OM-based clustering methods for categorical time series.
Read the full abstract ▾