Peer Revenue Impact factor: volatile

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Saturday 19 September

Statistical Methodology

Max-Test-Calibrated Stein Shrinkage with Honest Submodel Selection in Ultra-High-Dimensional Regression

Classical preliminary-test and Stein-type estimators interpolate between restricted and full regression fits, but OLS and chi-squared calibration fail when $p\gg n$ and the restriction is data-adaptive. We propose an honest sample-separated framework built around a common selected-null law. Independent selection data extend a mandatory core; separate estimation data pair a complete-design regularized full-model (FM) fit with a fresh exact-null submodel refit. A maximum residual-association statistic assesses all excluded coordinates. Projected Gaussian draws reproduce its conditional null distribution under homoskedastic Gaussian errors, yielding a finite-sample-valid rank test. The inverse moment of the same design-specific law replaces $q-2$ and calibrates preliminary-test (PT), Stein-type (S), and positive-part Stein-type (PS) rules. We derive exact endpoint-risk formulas, honest conditional validity, null-law concentration, inverse-moment consistency, an explicit selection-failure remainder, and endpoint adaptivity without uniform-dominance claims. Gaussian experiments with 2,000 replications and up to 30,000 predictors compare Ridge and LASSO FMs, tuned by frozen-fold cross-validation, under a common submodel, test, and calibration. PS provides large near-null coefficient-risk gains and approaches the relevant full-model risk under strong departures. A separate CPSS-LASSO/CPSS-MCP audit evaluates data-adaptive selection, while a split-sample DepMap study with 19,152 predictors illustrates prediction gains and sensitivity to error assumptions. The HDMaxShrink R package implements the procedure.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

On attention heads and bilinear forms

We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from real bilinear forms to a three-dimensional simplex and observe that profiles of trained bilinear forms accumulate near profiles of rank-one bilinear forms. We prove that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.
Read the full abstract

Statistical Methodology

Lorentz Hyperbolic Weighted Regression: Theory for Fixed and Estimated Representations

Many applications provide each observation with a meaningful representation in addition to an ordinary response and covariates. When that representation is hierarchical, hub-periphery structured, or network derived, Euclidean or geographic proximity may define the wrong peer groups. We present Lorentz hyperbolic weighted regression (LHWR) as a practical local regression method for this setting. Responses and predictors remain real valued; only locality is defined by distances between observations represented on the Lorentz model of hyperbolic space. We describe coordinate construction, adaptive bandwidth selection, prediction, local coefficient summaries, collinearity checks, and residual autocorrelation diagnostics. Theoretical results explain consistency, bias-variance tradeoffs, curvature effects, and the extra uncertainty caused by estimated representations. Simulations show that the Lorentz geometry is most useful for sharply localized coefficient surfaces, whereas tangent-plane approximations can be competitive for smooth surfaces. In a 141-country World Development Indicators illustration, economic similarity based on income and trade openness defines local peer groups. Representation-based locality predicts GDP growth better than global least squares and geographically weighted regression, although differences among Lorentz, Poincare, and tangent metrics are modest. The main practical lesson is that the representation should be chosen scientifically and the distance geometry should be checked rather than assumed.
Read the full abstract

Statistics Theory

Statistical models as natural transformations: meaningfulness, coherence and priors as states in Markov categories

We show that a statistical model in the sense of McCullagh, in the form given by Brøns, is a natural transformation between two functors from the category of designs to the Kleisli category Stoch of the Giry monad, provided that its components are measurable in the parameter. The condition is empty for finite models. A design-indexed quantity is a family of morphisms of Stoch defined on the parameter objects, called meaningful if it is natural. We prove that Tjur's criterion, imposed on parameter functions indexed by finite samples with multiplicities, forces the indexing by the support and then coincides with naturality over the insertions. For finite designs we show that a quantity can be corrected to a natural one within a given class of corrections if and only if a class vanishes in the first cohomology group of a Baues-Wirsching complex relative to that class, while its image in the absolute group is always zero. In the one-way layout, marginal dispersion is not meaningful, and within-group dispersion is the unique correction that leaves the merged design unchanged. A prior is a family of states on the parameter objects, called coherent over a class of design morphisms if it is natural over that class. We show that coherence at a merge confines the prior to the image of the corresponding parameter map, that coherence over the insertions is Kolmogorov consistency, and that coherence over the injections adds the exchangeability assumed by the categorical de Finetti theorem. In the finite one-way scheme, the coherent priors form polytopes of known dimension. The analogue of Jeffreys' general rule is not coherent, while the analogue for location-scale families is. Finally, we show that ridge regression is the Bayesian inversion of the Gaussian linear model with respect to a Gaussian prior, which is coherent over the insertions and never over the injections.
Read the full abstract

Statistics Applications · Cross-listed from cs.CV

Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning

Video foundation models now reach human-level accuracy on physical-reasoning benchmarks, yet such tasks require predicting unobserved physical outcomes. Do these models perform human-like forward simulation, or do they exploit statistical regularities in visible scenes? Accuracy alone cannot distinguish these strategies. We introduce a distributional evaluation framework that treats model seeds and human raters as populations, enabling comparison of consensus, uncertainty, and strategy. On the Physion benchmark, we evaluate three ViT-L architectures (V-JEPA2, VideoMAEv2, DINOv2). V-JEPA2 narrows the accuracy gap to ~1 percentage point (73.2% vs. 74.2%), yet model-human disagreement reaches 26.4%, far exceeding human-human disagreement (4.8%), with substantially lower agreement (kappa ~ 0.48 vs. 0.91). The divergence follows forward-simulation demands: models outperform humans on geometric reasoning (linking, +11.8 pp) but underperform on gravitational dynamics (rolling, -11.8 pp) and causal chains (dominoes, -10.5 pp). Strategy fingerprinting confirms all three architectures share non-human strategies while none aligns with humans. Attribution analysis suggests that unobservable outcome features, rather than visible scene properties, predict this divergence, consistent with models relying more on scene-level statistical regularities than on explicit forward simulation, a systematic divergence that accuracy alone cannot reveal. Code is available at https://github.com/fanhong-li/model-human-divergence.
Read the full abstract

All authors Fanhong Li, Shurui Zheng, Zi Yin, Junbo Cui, Lei Ji, Jia Liu

Statistics Applications

Modeling Transition Dynamics and Network Structure in Cross-National Process Data: A Hierarchical Multi-State Survival Framework

Process data from computer-based assessments record the sequence and timing of actions through which respondents solve a task, providing information about both the pace and structure of problem-solving behavior. Modeling such processes across countries is challenging because country-by-response-group cells are often small and unbalanced and the observed transition supports can differ substantially across countries. We propose a hierarchical framework that integrates a Bayesian multi-state survival model with a network-based representation of transition structure. Partial pooling across countries yields country-specific covariate and key-action effects, transition speed, and estimates of between-country heterogeneity. Posterior transition probability networks are embedded in a common latent space using a directed graph auto-encoder adapted to heterogeneous supports, and 1-Wasserstein distances between node-role distributions are evaluated across posterior draws to characterize global network structure while propagating estimation uncertainty. We apply the framework to two problem-solving items from the Programme for the International Assessment of Adult Competencies across 14 countries. The results reveal cross-country heterogeneity in transition speed, systematic response-group differences in global network organization, item-dependent variation in within-group dispersion across countries, and local differences in routing around shared intermediate actions.
Read the full abstract

All authors Doungjun Kim, Daeun Hwangbo, Minjeong Jeon, Ick Hoon Jin

Statistics Applications

Criminal Justice Risk Assessments with Unreliable Class Assignments

Quantitative risk assessments have been used for decades to help inform criminal justice decisions. A set of predictors and a categorical response variable are used to train a statistical classifier. The trained classifier can be employed to compute risk scores for a new, unlabeled case for which a forecast is needed, one risk score for each response variable class. In the spirit of Bayes classifiers, the class with the largest risk score conventionally becomes the forecasted outcome label for that case. The largest and next largest risk scores can be very similar or very different. However, when they are much the same, the outcome class is essentially being assigned by a coin flip; the classification is unreliable. Forecasting error becomes more likely, and concerns about fairness can arise. This paper shows how Mondrian conformal prediction sets can increase classification reliability when the classifier is indecisive and provide valid estimates of forecast uncertainty. Risk forecasts for a large dataset of offenders on probation are used to illustrate the issues.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Multi-Armed Bernoulli Bandits via Minimax Single-Arm Stopping

We develop an index policy for finite-horizon Bernoulli multi-armed bandits from minimax solutions to single-arm bandit (SAB) problems. Each SAB problem involves choosing between an unknown Bernoulli arm and a known reward. We show that minimizing worst-case regret of SAB problems over all non-anticipative policies admits an exact semi-infinite linear programming formulation. The resulting stopping policies offer a natural way to compare arms: the higher the known reward against which a policy continues sampling, the more promising the unknown arm. We turn this intuition into indices based on cumulative continuation probabilities, with a monotone adjustment and a reward-shortfall cap. By relating index errors to the regret of single-arm stopping policies, we establish a distribution-free regret bound of $4.45\sqrt{KT}+10.75K$ for $K$ arms and horizon $T$. This bound matches the minimax-optimal regret order established in the literature. The guarantee extends to rewards supported on $[0,1]$ through Bernoulli randomization. We also provide a finite-grid implementation with quantified approximation loss. In numerical experiments, the SAB-based index policy achieves lower worst-case regret than every tested benchmark policy across all evaluated numbers of arms and horizons, while closely matching the grid-based MAB minimax policy in the two-arm setting.
Read the full abstract

All authors Huikang Liu, Zhengchao Wang, Daniel Kuhn, Wolfram Wiesemann

Statistical Methodology

Validity of MMRM-based hypothesis testing under missing-not-at-random mechanisms

In randomized clinical trials with longitudinal continuous outcomes, missing-not-at-random (MNAR) missingness often motivates conservative alternatives to mixed models for repeated measures (MMRM). Such caution is important for estimation, but estimation and testing need not require identical assumptions. Moreover, overly conservative primary analyses may reduce power, increase required sample size, and raise trial costs. We investigated the validity of MMRM-based testing under the global null of identical longitudinal outcome distributions across groups. Because valid testing minimally requires treatment-effect estimators to converge to the null under the null hypothesis, we investigated sufficient conditions for this property. We introduced a proportional observation condition requiring ratios of observation probabilities relative to a reference group, conditional on the full outcome vector, to be outcome-independent, and showed that, with arbitrary post-baseline visits and monotone missingness, this condition is sufficient for convergence to the null value. The condition allows observation to depend on unobserved outcomes and permits between-group differences in overall observation probabilities through outcome-independent dropout, making it clinically interpretable while accommodating outcome-dependent MNAR missingness. Synthetic and data-based bootstrap simulations showed negligible bias and empirical test sizes near 0.05, including nonmonotone missingness. Thus, MNAR missingness does not by itself imply that a more conservative primary testing procedure is required. This result does not justify treatment-effect estimation under alternatives, which still requires estimand-based interpretation and sensitivity analyses.
Read the full abstract

All authors Kazushi Maruo, Ryota Ishii, Yusuke Yamaguchi, Keisuke Hanada, Naoki Isogawa, Masahiko Gosho

Machine Learning (Statistics) · Cross-listed from eess.SP

Subspace Learning with Interval-Censored Likelihoods for Dequantizing Percept PC LFP Snapshots

Implanted neurostimulators that sense local field potentials now enable chronic electrophysiology based biomarker tracking in patients at home. The Medtronic Percept PC, the only commercially available sensing-enabled deep brain stimulation (DBS) device, stores spectral amplitudes as 16-bit integers at approximately 0.1 $\mu$V per bit (quantum $q \approx 0.11$ $\mu$Vp). At frequencies where the true amplitude spans only a few quantization levels, consecutive bins round to the same stored value. Standard spectral parameterization (FOOOF, fitting oscillations and one over f), which separates periodic peaks from the aperiodic 1/f activity, treats every value as exact and fits oscillatory peaks to these plateaus. Because these spectra feed clinical biomarker pipelines and spectral foundation models for symptom decoding, spurious peaks can corrupt downstream inference. Across 9,438 spectra from 14 hemispheres in 7 subcallosal cingulate DBS patients, 20.6% of peaks detected at [2, 45] Hz have no match in ground truth synthesized by quantizing clean in-clinic BrainSense recordings, while aggregate beta band power and the aperiodic exponent are preserved. We formalize dequantization as interval-censored subspace estimation and compare five classes of correction methods. Quantized probabilistic PCA is the only tested method that reduces the spurious rate (20.6% to 18.3%) while preserving true peak detection and keeping noise floor RMSE below $q/\sqrt{12}$.
Read the full abstract

All authors Shreesh Karjagi, Elif Ceren Fitoz, Maryam Khalid, Tanya Nauvel, Parisa Sarikhani, Helen S. Mayberg, Christopher J. Rozell, Sankaraleengam Alagapan

Friday 18 September

Machine Learning (Statistics)

A Bayesian Vertical Federated Learning Framework for Multivariate Reduced-Rank High-Dimensional Regression

Federated learning (FL) has emerged as a leading privacy-preserving framework for collaborative machine learning across decentralized environments. While considerable progress has been made in horizontal federated learning (HFL), where data with common features is distributed across sites, vertical federated learning (VFL), where sites share observations across distinct feature sets, remains less explored. Advancing Bayesian high-dimensional multivariate reduced-rank regression methods for VFL poses unique challenges: (a) stringent privacy regulations preventing local site data sharing, and (b) fitting local regressions overlooks essential modeling aspects like inter-variable correlations. In contrast HFL allows each site to fit a comparable model independently. We present a novel Bayesian VFL framework for multivariate high-dimensional reduced-rank regression, termed BayesVFLReg, which enables precise coefficient estimation while safeguarding both feature and response privacy. Participating sites use a shared random sketching matrix to compress local variables into privacy-preserving sketches. A central server collects these sketches where Bayesian multivariate reduced-rank regression uses Gaussian scale mixture priors. For feature selection, we introduce a single-step post-processing strategy based on mixture-model clustering of the absolute posterior coefficient means to distinguish signal from noise per response variable. BayesVFLReg is computationally scalable for large, high-dimensional datasets and facilitates efficient variable selection. Theoretically, we establish sharp non-asymptotic bounds on the posterior probability that the fitted density falls within a Hellinger ball centered at the true data-generating density. Comparative simulation studies and real-world data analyses show that BayesVFLReg reliably identifies sparse feature effects, even under feature correlation.
Read the full abstract

All authors Brigham Halverson, Sharmistha Guha, Jessica Bernard, Rajarshi Guhaniyogi

Machine Learning (Statistics) · Cross-listed from cs.LG

Monotone-Constrained Diffusion Models for Long-Horizon Production Forecasting

Forecasting a long horizon from only the first observations of a sequence is ill-posed: many trajectories are consistent with the same short history. We study this problem in oil and gas production forecasting, where forecasts made after roughly the first fifth of a well's producing life drive development and abandonment decisions, and where a usable forecast must describe a monotone decline. We present Physics-SIMS-TS, a conditional diffusion forecaster that combines negative guidance against synthetic artifacts, decline-curve constraints and an isotonic projection applied during sampling, spatial training augmentation, and an ensembled stochastic sampler yielding a full predictive distribution. Across three jurisdictions and more than 35,000 wells, under a shared-space, validation-frozen protocol, Physics-SIMS-TS is the most accurate diffusion forecaster in the comparison and is competitive with, but not superior to, ensembled transformer forecasters. Its forecasts are monotone by construction at a cost of at most 0.5% in mean squared error, and its trajectory ensemble yields calibrated intervals after one dispersion factor is fitted per jurisdiction. On six standard benchmarks a reversible-instance-normalization variant of the backbone is the leading diffusion baseline. We also quantify four protocol choices on which the measured ranking depends. Code and evaluation artifacts are released.
Read the full abstract

Machine Learning (Statistics)

Locally Private Inference for Riemannian Stochastic Optimization

We develop inference for manifold-valued population minimizers when each observation belongs to a different participant and only locally private messages reach the analyst. The method releases randomized tangent gradients and combines them through Riemannian stochastic approximation and Polyak-Ruppert averaging. Directly inserting a private data surrogate into a nonlinear loss can shift its population target, whereas conditional centring of the released gradient preserves the first-order equation. We introduce symmetric-pair regression (SPR) to estimate the asymptotic variance from the same private messages used for point estimation, without holding out participants or requesting a second release. We prove the central limit theorem and consistency of the fully transcript-based sandwich covariance and intrinsic Wald region under local differential privacy. Simulations across various statistical problems and manifolds support the predicted decrease in estimation error and near-nominal coverage under moderate privacy. An application to NHANES anthropometric data illustrates private estimation of a leading body-size direction and its uncertainty.
Read the full abstract

Statistics Applications

Adapting Pairs Trading to Gambling Markets A Case Study of the U.S. Presidential Election

Pairs trading exploits mean reversion in the relationship between related assets. We adapt this idea to political betting markets by modelling the combined implied probability of the two major-party nominees with a latent Ornstein-Uhlenbeck process whose mean-reversion level varies over time and whose observations contain additive noise. Model parameters are estimated from regularly sampled odds data using a state-space likelihood, with consecutive repeated values represented by a single retained observation and the elapsed number of sampling intervals preserved in the continuous-time transition. Parametric-bootstrap upper prediction bounds identify signal times at which the combined implied probability is likely to decline, and a no-intercept Bradley-Terry-type model selects the candidate-specific odds quote. The candidate-selection model is trained on 2020 U.S. presidential-election data and evaluated out of sample on 2024 data. The 2024 analysis produced 130 signals, empirical one-step coverage of 95.1%, a mean synthetic odds-price return of 1.86%, and an unannualized per-trade Sharpe-type ratio of 1.12. These returns are frictionless descriptive quantities rather than executable betting-exchange profits. The results support the integrated framework as a proof of concept for two-candidate electoral markets.
Read the full abstract

All authors Haoyu Liu, Len Thomas, Benjamin Baer, Carl Donovan

Statistical Methodology

Orthogonal Policy Learning with Ordinal Outcomes

Policy learning methods based on conditional average treatment effects can obscure subpopulation heterogeneity when applied to ordinal outcomes. We develop a policy learning framework for ordinal outcomes with heterogeneous utilities for individuals who strictly benefit from treatment and those who do not. Since the probability of strict benefit is only partially identified without further conditions, we adopt a minimax strategy which minimizes the worst-case regret over the identification region. To estimate the resulting nonsmooth objective, we combine smooth approximations with Neyman orthogonalization to remove first-order bias from nuisance estimation. We also derive an excess worst-case regret bound over a restricted policy class. The proposed method is validated through extensive simulations and an application to the 2022 Survey of Income and Program Participation (SIPP) data.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Classification with Abstention Under Class-Conditional Error Constraints

We study binary classification with abstention under separate class-conditional error constraints, with the objective of minimizing abstention while keeping both errors below prescribed thresholds. We characterize the distribution-free minimax rate of excess abstention risk, up to logarithmic factors, in terms of the complexity of the hypothesis class and the sample size. To make the framework amenable to computation with models such as neural networks, we introduce surrogate-loss formulations and derive finite-sample guarantees for excess surrogate ambiguity risk. We formulate the resulting learning task as a constrained optimization problem and characterize its computational complexity in the convex setting. Finally, we evaluate our approach on various datasets and compare its performance with a competing method for this problem.
Read the full abstract

Statistics Theory

Exact maximum likelihood inference for drifted multi-sub-fractional Brownian motion at discrete observation

Sub-fractional Brownian motion is self-similar and long-range dependent but has no stationary increments, so the increment covariance is not Toeplitz and no spectral density is available. We show that a complete finite-sample likelihood theory survives nonetheless. The model is a linear trend observed at $N$ equidistant times through a superposition of $m$ independent sub-fractional Brownian motions with known Hurst indices and a common scale. Nondegeneracy follows from realising the process as the even part of a two-sided fractional Brownian motion, and the maximum likelihood estimators of the trend and of the scale are explicit. The statistics on which inference rests are pivotal, their laws depending on the sample size alone, so intervals and tests of exact level are available at every $N\ge2$, together with complete sufficiency, minimum variance unbiasedness and attainment of the Cramér–Rao bound. An explicit variance bound gives strong consistency and asymptotic normality, and simulations confirm the exact coverage and the predicted effect of a misspecified Hurst vector.
Read the full abstract

Statistical Methodology

Stochastic Epidemic Model Criticism with Neural Evidence Estimation

Neural techniques for simulation-based inference (SBI) have become popular in scientific domains featuring complex simulation models. These methods train neural networks on simulated data to infer unknown parameters; their reliability depends on the simulator faithfully representing reality. In infectious disease modeling, multiple models encoding different hypotheses often provide plausible explanations of observed data. Model selection is a difficult problem when likelihoods are unavailable, as is the case with many common stochastic transmission models. We propose a model criticism methodology designed for SBI, Neural Evidence Estimation (NEE), that integrates model selection and model misspecification detection. NEE learns a model's marginal likelihood (a.k.a. evidence) directly from simulations using normalizing flows, thereby avoiding noisy or complex Monte Carlo estimators. Through synthetic experiments, we demonstrate that NEE reliably selects the correct data generating process among multiple models and flags poorly specified models. We apply NEE to a real-world modeling problem concerning competing mechanisms for influenza reinfection in a multiple wave epidemic. This work argues for the importance of principled model criticism in SBI and infectious disease modeling.
Read the full abstract

Machine Learning (Statistics)

RLVR is a Kernel, Not a Function: Statistical Inference for pass@$k$ Crossovers

Reinforcement learning with verifiable rewards (RLVR) often improves pass@1 while falling behind its base model at larger sampling budgets $k$, a crossover read as evidence that RLVR only sharpens existing capability. We identify two limits to this reading. First, a visible crossing need not be statistically established: comparing models on the same prompts, we build confidence bands across sampling budgets $k$ that require evidence of both an early gain and a later loss. Across five public RLVR pairs no crossing is statistically established in the initial evaluations, while a 32k-token evaluation on fresh prompts locates a reversal with first loss between 11 and 61 samples; power analysis shows why failure to detect a crossing need not mean no crossing, and why more prompts can help more than more answers per prompt. Second, base success alone does not determine what RLVR does to a prompt: prompts with the same base success rate have different post-RL success rates, and these differences repeat across independent generation halves. The relationship is a conditional distribution—a Markov kernel—rather than a single curve, and fitting it predicts crossings in independent generations for the same prompts and corrects the simple model's power estimates. Theory further shows how losses on a minority of the hardest prompts can overturn an early lead even when training improves other prompts, separating evidence that a crossover exists from claims about what it means for capability.
Read the full abstract

Statistical Methodology

Adaptive online kernel changepoint detection

We propose an adaptive online kernel-based changepoint detection method for streaming data that is capable of detecting a broad range of changes in the underlying data distribution. The method maintains a recursively-weighted reproducing kernel Hilbert space representation of observations and adaptively updates the forgetting factor through a gradient-based procedure driven by a maximum mean discrepancy-type statistic between the current observation and the weighted empirical distribution of the past. This self-tuning mechanism allows the detector to adapt its effective memory and responsiveness to changes in the underlying process. Simulation results demonstrate that the proposed method achieves strong detection performance across a wide range of distributional changes. Further, our proposed approach maintains constant computational and storage cost through recursive updates, and is very computationally efficient in comparison to competing methods. Experiments on both simulated data and benchmark real-world datasets show improved performance over several other leading kernel-based methods.
Read the full abstract

Statistical Computation · Cross-listed from physics.comp-ph

Stochastic consensus dynamics for decentralized decision systems

A fundamental mechanism underlying collective phenomena in social, technological, and economic systems is decentralized decision-making, in which the behavior of individual agents follows from local interactions in the absence of central coordination. Consensus formation is a central feature of such systems, with relevance to blockchain networks, distributed artificial intelligence, autonomous multi-agent systems, distributed control, and collective decision networks. We investigate consensus formation using a stochastic consensus model on random networks and examine how initial conditions, network connectivity, and system size shape the emergence of unanimous states. We show that the stochastic dynamics strongly amplify small initial majorities, progressively suppress the competing state, and drive the system toward a predictable collective outcome. Network connectivity primarily controls the efficiency of this process: increasing connectivity accelerates the propagation of local agreement and reduces the likelihood that fluctuations reverse the initially dominant state, although these gains gradually saturate in highly connected networks. System size produces a complementary effect. Larger networks require more individual updates to reach unanimity, but they are also increasingly reliable in selecting the state favored by the initial majority. Finite-size analysis shows that the range of initial conditions associated with uncertain outcomes becomes progressively narrower as the network grows, decreasing approximately with the inverse square root of the system size. These results reveal a collective amplification mechanism by which weak initial asymmetries become increasingly decisive in large decentralized networks, and they provide a simple framework for understanding the efficiency, predictability, and reliability of consensus formation in distributed decision systems.
Read the full abstract

All authors André L. M. Vilela, Caio B. L. Silva, Kenric P. Nelson, Emilio Cobanera, Gaogao Dong

Statistics Theory

Asymptotics and finite sample bounds for prediction and smoothing in Wright-Fisher hidden Markov models

We study signal prediction and smoothing in hidden Markov models whose latent signal is a multi-type Wright-Fisher diffusion and whose discrete-time observations are categorical. We consider a repeated-sampling regime with finitely many inspection times and large within-time samples. Under parent-independent mutation, the Bayesian predictive and smoothing laws admit exact representations as mixtures of Dirichlet distributions. We show that the one-step predictive and local smoothing laws converge in total variation to the corresponding Wright-Fisher transition and bridge laws determined by the neighboring population frequencies, thereby justifying plug-in approximations based on empirical proportions. For prediction, we derive explicit finite-sample bounds for this approximation and for bounded predictive functionals such as diversity, together with a finite-sample concentration bound for the marginal smoother. The joint posterior at the inspection times concentrates at the target frequencies and satisfies a Bernstein-von Mises limit with distinct interior and boundary regimes, with Gaussian and Gamma limits on different scales. The results allow arbitrary dependence across inspection times. A key technical ingredient is a fixed-interval tail bound for the coalescent-based dual death process.
Read the full abstract

All authors Luigi M. Malgieri, Filippo Ascolani, Matteo Giordano, Matteo Ruggiero

Statistics Applications

County-Level Heterogeneity in Opioid Harm Reduction and Treatment Effects: A Simulation Modeling Analysis

Opioid overdose deaths remain a severe public health crisis in the US, with heterogeneous burden across counties that differ in epidemic trajectory, baseline resources, and local context. While harm reduction through naloxone distribution and buprenorphine treatment are both evidence-based strategies, limited information on county-level effects hinders the ability of policymakers to prioritize resources across counties. We developed a simulation model of opioid use disorder (OUD), calibrated separately to six Pennsylvania counties spanning large urban (Allegheny, Philadelphia), intermediate-sized (Erie, Dauphin), and rural (Clearfield, Columbia) settings. We projected county-specific overdose mortality trajectories under three levels of increase in buprenorphine dispensing and naloxone distribution (10%, 20%, and 30% above each county's baseline), over a five-year horizon from 2025 to 2029. A 30% increase in naloxone distribution above observed county baseline levels was projected to reduce 2029 overdose deaths by approximately 70% (95% uncertainty interval, UI: 55-81%) in Allegheny County, 11% (95% UI:4-17%) in Erie County, and 28% (95% UI:2-64%) in Clearfield County. Projected reductions in overdose deaths from increasing buprenorphine were consistently smaller (10%-23%), except that in Erie buprenorphine produced larger projected reduction by 20% vs 11% for naloxone. Heterogeneity in naloxone responsiveness was strongly associated with each county's historical naloxone dispensing variability. The same proportional increase in naloxone distribution yields substantially different projected mortality reductions across counties depending on each county's baseline distribution history, a pattern invisible from mortality statistics alone. County-level context is important for informing harm reduction and treatment prioritization at the county level.
Read the full abstract

All authors Abdulrahman A. Ahmed, M. Amin Rahimian, Qiushi Chen, Praveen Kumar

Statistical Methodology

Bridging Local and Population Causal Effects: A Proximal Instrumental Variable Approach

Instrumental variable (IV) methods address treatment endogeneity, but with non-compliance and heterogeneous treatment effects a binary instrument generally identifies the local average treatment effect (LATE) among compliers rather than the population average treatment effect (ATE). When treatment effects and compliance probabilities are heterogeneous and dependent through latent factors, the ATE need not be identified by IV variation alone. We develop a proximal instrumental variable framework that uses proxies for these factors to adjust for the compliance weighting in LATE and recover the ATE. We show that the ATE is identified through either an outcome bridge capturing latent treatment effect heterogeneity or a compliance bridge representing inverse compliance probabilities, and combining the two yields a doubly robust representation. We derive the efficient influence function under known and unknown instrument propensities. For estimation, we propose regularized kernel minimax bridge estimators, with orthogonalized conditional moments for estimated propensity, and a three-way sequential cross-fitting scheme. We establish projected bridge-risk bounds, and give conditions for asymptotic normality and semiparametric efficiency. The framework provides a proxy-based route from local IV effects to population causal effects under latent treatment effect and compliance heterogeneity.
Read the full abstract

Statistical Methodology · Cross-listed from cs.SI

Auditing bipartite motif interpretations: a worked example with conservation checks and open-path decomposition

Motif profiles of bipartite agent-object networks, such as tourist-site visits and customer-item transactions, are read as evidence about structural roles and about differences between networks, often without asking what the two degree sequences already fix. In a simple bipartite graph the induced k-fan count on one node type is a sum of degree combinations, so it has zero variance under a null that preserves both degree sequences. We apply this known result to a reconstructed tourism rating network of 17 tourists, 80 sites and 637 edges, the sole inferential worked example, and, as a provenance-limited illustration, to published motif-instance aggregates over 36 monthly luxury customer-item networks. The four fan classes are exact functions of the degree sequences: in the tourism network the raw fan counts and the size-3 two-fan ratio (84.7% fan-out) restate those sequences. The published luxury counts require at least 89,502 customer-item edges against 26,451 reported transactions, so their 99.8% fan-in is reported as a descriptive value only. Against a hard bipartite configuration null, the four-cycle count is degree-consistent (z about +1.0) and the open path is deficient (z about -5.8) by 2,243 instances, 2.4% of the null mean; the deficit survives every leave-one-tourist-out re-run (z -4.5 to -8.1). An exact identity splits it at the point estimate into 64.4% mixing and 35.6% four-cycle, but that split is not an attribution: the observed mixing term lies below all 500 null samples, the two components are almost collinear under the null (r = 0.968), and the mixing share ranges from 36.6% to 116.5% under leave-one-tourist-out deletion. The deficit is extreme relative to the sampled null, while its class-level interpretation is undetermined and unstable. We give a four-step pre-interpretation check and a reference implementation.
Read the full abstract

Statistical Methodology

RKHS-Based Inference for Nonlinear Granger Causality via Conditional Centering

Granger causality is commonly formulated through linear prediction in vector autoregressive models, limiting its ability to detect nonlinear predictive relationships. We propose a reproducing kernel Hilbert space (RKHS)-based test for nonlinear Granger non-causality in conditional mean for nonlinear autoregressive processes. The key idea is a conditional-centering decomposition of the target regression function into an own-history component and an orthogonal component capturing the additional predictive contribution of the potential source history. Non-causality is characterized by the vanishing of the latter component. The own-history component is estimated by kernel ridge regression, and the residuals are embedded in a second RKHS using a conditionally centered kernel. This yields an RKHS-valued residual moment whose squared norm forms the test statistic. We establish a weighted chi-square null limit and consistency against fixed alternatives for the population-centered statistic. An empirically centered version is shown to retain the null limit and enables spectral calibration of critical values and $p$-values without resampling. Simulation studies demonstrate accurate size control and power against nonlinear alternatives, and real-data applications illustrate the usefulness of the method for detecting nonlinear predictive relationships in time series.
Read the full abstract

Statistical Methodology

A Novel Convolution-Based Stratified Attribute Estimator for QRE Determination in R&D Tax Credit Studies

The One Big, Beautiful Bill Act reinstates immediate expensing of domestic research expenses, reducing the tax burden and incentivizing reinvestment in U.S.-based research and development including software development across a wide range of industries. In this context, accurate and defensible estimation of qualified research expenses (QREs) is of high importance. Statistical sampling provides a practical framework for estimating QREs for a well-defined population of business components (sampling frame) documented in accordance with Internal Revenue Code Section 41 (Form 6765). IRS Revenue Procedure 2011-42 permits both attribute and variable statistical methods, although the latter (stratified mean and difference estimators) are often regarded as the standard approach to QRE estimation. In this paper, we compare attribute and variable statistical methods within a simulation study and demonstrate that attribute methods offer several theoretical and practical advantages. A key concern with the simple attribute estimator is the possibility of upward bias in QRE determination arising from a preponderance of high potential QRE (pQRE), non-qualified projects in the sampling frame. We address this concern by introducing a stratified sampling design that ensures adequate representation of high pQRE projects in the sample. We demonstrate that a convolution-based stratified attribute estimator produces a valid one-sided 95% lower confidence bound on total QREs across a range of sampling frame structures.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.CL

Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment

Most work in computational ethics treats annotator disagreement on moral content as noise to be voted away, collapsed into majority vote or the more permissive any-annotator rule the moment a single annotator flags an item. We argue this uncertainty should instead be modeled and learned from. We introduce Moral Entropy, a Bayesian framework that keeps a full posterior over the true label and decomposes its entropy into aleatoric uncertainty (irreducible disagreement about the moral content) and epistemic uncertainty (from insufficient or noisy annotation) – and lets any heuristic consensus rule be audited against a calibrated ground truth via entropy methods such as cross-entropy/KL, Brier score, and expected calibration error. Across three corpora and fifteen discourse domains, auditing the standard aggregation rules against this posterior reveals bias that no current pipeline reports: the any-annotator rule disagrees with the calibrated posterior on roughly 30% of items – pooled, almost entirely false positives, though the errors invert at the foundation level (19.9%/38.9% mean FPR/FNR on MFTC) – while the stricter majority and two-vote rules miss 63-83% of true positives.
Read the full abstract

Statistical Methodology

Integrating Multivariate Adaptive Regression Splines into Small Area Estimation for Nonlinear Poverty Modeling in Java Island

Small area estimation (SAE) modeling is often used to address the unreliability of estimates in areas with limited survey samples. However, standard SAE models (Fay-Herriot) can only accommodate linear patterns, whereas in real cases, including poverty, many patterns are nonlinear. This study proposes the implementation of a supervised learning model, multivariate adaptive regression splines (MARS), integrated within the SAE framework (MARS-SAE) to address the challenge of unreliable direct estimation, particularly in nonlinear cases. The hyperparameter tuning results yielded the best model with the following hyperparameter combinations: maximum basis function (Max BF) = 10, maximum interaction (MI) = 2, minimum observation (MO) = 1, penalty = 3, with general cross validation (GCV) = 6.128. MARS-SAE demonstrated very strong performance; empirically, the model proved more efficient at reducing estimation error due to its higher relative efficiency (RE) compared to the SAE Fay-Herriot model, as well as better reduction of relative standard error (RSE) than the comparison model, and it yielded a lower relative root mean squared error (RRMSE) compared to the SAE Fay-Herriot model (13.64% vs. 13.75%). MARS-SAE has been proven to capture nonlinear patterns very well while still maintaining interpretability, making it an effective tool for the predictive and diagnostic analyses of phenomena. Thus, this research can serve as an important reference for readers regionally and in Southeast Asia, given that Java Island is a strategic region for Indonesia, the largest economic power in the ASEAN region.
Read the full abstract

Statistical Methodology

Consistency of Optimal Matching-based Clustering for Mixtures of Markov Chains

We study clustering of categorical sequences using the Optimal Matching (OM) distance under finite mixtures of finite-state Markov chains. We show that the normalized OM distance between two independent chains converges almost surely to a deterministic population quantity, concentrates exponentially around its finite-horizon mean, and admits an $O(\sqrt{\log n/n})$ convergence rate when the two chains have the same transition kernel. These population quantities yield a natural separation condition: the largest within-component limit must be smaller than the smallest between-component limit. Under this condition, hierarchical clustering with any bracketed linkage and Partitioning Around Medoids consistently recover the latent mixture partition. We also propose a consistent estimator of the number of components based on empirical OM distance profiles. The results extend to finite-state hidden Markov models and multichannel categorical observations. Overall, they provide a statistical justification for standard OM-based clustering methods for categorical time series.
Read the full abstract