Statistical Methodology
Hierarchical Clustering of Networks via Hierarchical Distance Matrices
Li Chen, Nathaniel Josephs, Eric D. Kolaczyk et al. (+1 more)
Clustering populations of networks while recovering their latent hierarchical organization is a fundamental yet largely unexplored problem in network analysis. To formalize this, we introduce the Hierarchical Distance
Matrix, a specific class of population-level distance
matrices that encodes latent hierarchical organization through recursively nested distance separation, accommodating unbalanced tree depths. Building on this framework, we propose a fully data-driven top-down procedure: network hierarchical clustering based on two-sample testing (NHC-TST). The
algorithm recursively splits networks via spectral clustering and uses a graph-based two-sample stopping rule. The procedure adaptively determines the branching structure without requiring prior knowledge of the number of clusters or tree depth. Theoretically, we establish exact recovery of the population-level hierarchical structure and statistical consistency in the empirical procedure. Simulation studies demonstrate highly accurate recovery of both cluster memberships and hierarchical relationships across a wide range of settings. Applied to a global migration
dataset, NHC-TST uncovers interpretable multi-resolution temporal structures that are not revealed by conventional flat clustering approaches.
Read the full abstract ▾
All authors Li Chen, Nathaniel Josephs, Eric D. Kolaczyk, Lizhen Lin
Statistical Methodology
Bayesian fusion forests for heterogeneous treatment effects on survival from randomised and real-world data
Tijn Jacobs, Stéphanie L. van der Pas and Wessel N. van Wieringen
We develop the
Bayesian fusion forest, a nonparametric framework to estimate heterogeneous treatment effects on survival outcomes by combining a randomised controlled trial and real-world data. The framework relaxes the unconfoundedness assumption on the real-world data by assuming instead that the treatment effect transports across the two sources. Our method opens up right- and interval-censored outcomes to data fusion. We model the survival time with an accelerated failure time decomposition into a shared baseline prognosis, a source-specific deviation, a treatment effect, and a confounding function. The confounding function absorbs the confounding bias in the real-world data. Each component receives a
Bayesian tree ensemble prior. The shared baseline prognosis borrows strength across sources, while the deviation captures between-source heterogeneity. A hierarchical Dirichlet process mixture models the error distribution nonparametrically. A simulation study shows efficiency gains over a trial-only analysis across varying levels of confounding and between-source heterogeneity. We combine the ACTG 175 trial with the Multicenter AIDS Cohort Study to estimate the effect of combination antiretroviral therapy for HIV. The fusion identifies a benefit for nearly every patient whereas the trial alone is inconclusive.
Read the full abstract ▾
Statistics Theory
Möbius transport on spheres
Eduargo García-Portugués and Shogo Kato
The Möbius transformation that generates the spherical Cauchy distribution from the uniform is the tangent-normal lift of a one-dimensional optimal transport: the monotone rearrangement of the cosine about the location axis. This identifies the probabilistic nature of the Möbius transformation and suggests a generalization: replacing the Cauchy target by any rotationally symmetric law, for instance the Poisson kernel or spherical cardioid, yields a generalized Möbius transformation. Möbius transport of a von Mises-Fisher base gives tractable anisotropic distributions on the sphere, with closed-form densities that inherit the base normalizing constant and allow immediate simulation. The Möbius-von Mises-Fisher and isotropic scaled von Mises-Fisher distributions, the latter also arising from a Möbius transport, are illustrated on paleomagnetic directions and short-period comet orbits, where they outperform classical and recently proposed alternatives.
Read the full abstract ▾
Machine Learning (Statistics)
Simple-regret rates and minimax optimality of fixed-prior expected improvement in Matérn and squared-exponential RKHSs
Emmanuel Vazquez and Sébastien Petit
We study expected improvement (EI) for minimizing a deterministic function $f$ in the RKHS $\mathcal H_k$ of a continuous positive-semidefinite kernel $k$ on a nonempty compact set $\mathcal X\subset\mathbb R^d$. Function values are observed exactly, and EI is computed from a fixed zero-mean Gaussian-process model with covariance $\sigma^2k$, $\sigma>0$. A weak-EI policy queries a point whose EI is at least a fixed positive fraction of its maximum. We introduce a notion of sequential separation radius relating ranked selected-point innovation norms to Kolmogorov widths, drawing on greedy approximation. Standard power-function estimates from scattered-data approximation and a finite-budget regret argument yield the rates. After $N$ post-initial queries, every weak-EI policy has simple regret $O(N^{-\nu/d})$ for isotropic Matérn kernels of smoothness $\nu>0$ and $O(\exp[-c_1\min\{N,N^{1/d}\log(eN)\}])$ for the isotropic squared-exponential kernel, with $c_1>0$. For $d=1$, the sharper bound $O(\exp[-c_2N\log(eN)])$ holds for exact EI, with $c_2>0$. These bounds are uniform over each fixed RKHS ball. If $\mathcal X$ has nonempty interior and $B>0$, the exact EI policy is minimax-rate optimal over the RKHS ball of radius $B$ for Matérn kernels, even among randomized strategies whose final recommendation need not be a query point. For the squared-exponential kernel, it is minimax-rate optimal up to constants in the exponent among deterministic methods whose final recommendation may be any point of $\mathcal X$.
Read the full abstract ▾
Statistical Computation · Cross-listed from econ.EM
Conditional projection methods for large-scale Bayesian VARs
Niko Hauzenberger and Michael Pfarrhofer
We develop fast methods for conditional
forecasting and structural scenario analysis with high-dimensional
Bayesian vector autoregressions (VARs). Our general framework features a factor structure on the reduced-form errors, which enables fast and order-invariant equation-by-equation estimation; suitably identified factors admit a structural interpretation. The scenarios are defined through separate distributional restrictions on observables, structural shocks and idiosyncratic components. The computational cost of our proposed
algorithm is cubic only in the number of restrictions, while the dimension of the forecasted system enters linearly. In our application with $33$ macroeconomic and financial variables and ten set-identified structural shocks for the US, we compute counterfactual predictions for oil price scenarios in the context of the 2026 closure of the Strait of Hormuz. The same oil price path is consistent with outcomes ranging from a mostly benign episode to pronounced stagflation, depending on which structural and idiosyncratic shocks are allowed to deliver it.
Read the full abstract ▾
Statistical Methodology
Mixed Frequency Stochastic Frontier Model: with application to the linkage of weather extremes and firm efficiency
Erniel B. Barrios, Nur Syazwani Mazlan, Lim Foo Weng et al. (+3 more)
A strategy for incorporating higher-frequency determinants of inefficiency into the
stochastic frontier models with production function indicators compiled at lower frequencies is proposed. Furthermore, the problem with positive skewness in
stochastic frontier models is resolved by postulating a logistic function for the inefficiency equation and refraining from assuming a more complicated distribution of the inefficiency term. High frequency determinants are incorporated into the inefficiency equation in a nonparametric function. The model is estimated with hybrid methods in a backfitting
algorithm instead of maximum likelihood estimation (MLE), which typically struggles with convergence issues and is vulnerable to distributional assumptions. The model is then used to characterize the linkage between firm efficiency and extreme weather events using data from electric cooperatives in the Philippines. This provides an approach to assess micro level sustainability issues from extreme weather arising events from
climate change.
Read the full abstract ▾
All authors Erniel B. Barrios, Nur Syazwani Mazlan, Lim Foo Weng, Paolo Victor T. Redondo, Gian Karlo M. Torreno, Lee How Chinh
Statistical Methodology
Local spectral clustering for heterogeneous clustering structures
Yuanxing Chen, Qingzhao Zhang and Yuhong Yang
Classical clustering methods typically assume that all informative features support a single latent partition of the observations. This assumption can be overly restrictive for modern high-dimensional data, where different subsets of features may encode distinct notions of similarity and induce heterogeneous sample partitions, while some features may contain no meaningful clustering information. We develop a frequentist framework for local clustering that simultaneously identifies feature groups and estimates the sample clustering structure associated with each group. Our approach represents each sample partition by a label-invariant clustering
matrix and groups features according to their shared clustering structures, thereby reformulating local clustering as a feature-grouping, or clustering-of-clusterings, problem. Under a heterogeneous sub-Gaussian mixture model, we construct feature-specific Gaussian-kernel similarity
matrices and propose a local spectral clustering procedure based on a clustering-
matrix optimization criterion. The proposed method avoids explicit likelihood specification and
Bayesian posterior computation, accommodates heterogeneous feature distributions, and permits the presence of non-informative features. Extensive simulations and applications further demonstrate the practical utility and superiority of the proposed approach.
Read the full abstract ▾
Statistical Methodology
Frequentist-calibrated Bayesian group sequential design with dynamic borrowing
Francesco Mariani, Shirin Golchi, Stefania Gubbiotti et al. (+1 more)
Bayesian analysis is increasingly used in clinical trials. However, assessment of the design with respect to the frequentist operating characteristics, such as type I error and power, remains a regulatory requirement in many cases. It is well established that, when information is borrowed from external sources to the trial, imposing strict frequentist type I error rate control is equivalent to offsetting the borrowing, which results in no power gains. We propose a
Bayesian group sequential design with dynamic borrowing that exploits an explicit correspondence between
Bayesian decision criteria based on posterior odds and frequentist uniformly most powerful (UMP) tests. At each interim analysis, two evidential thresholds are made available: the one that exactly retrieves the frequentist UMP decision; the other, that allows the investigator to incorporate historical information when appropriate. We assess the performance of the proposed approach in numerical studies, and apply the framework to the design of a phase III tuberculosis prevention trial, incorporating historical adult and pediatric trial data.
Read the full abstract ▾
All authors Francesco Mariani, Shirin Golchi, Stefania Gubbiotti, Fulvio De Santis
Statistics Applications · Cross-listed from q-bio.NC
metasignal: A Python Package for Comprehensive Metacognitive Analysis and Decision-Making
Saurabh Ranjan, Mukesh Makwana, Konstantina Sokratous et al. (+1 more)
Metasignal is an open-source Python package for signal detection theory (SDT) and metacognitive measurement. It implements the 17 metacognitive measures evaluated by Rahnev (2025), together with the reference variables d' (perceptual sensitivity), response criterion c (response bias), and mean confidence. The 17 measures comprise three meta-d' family estimates, meta-d', M-ratio, and M-difference; four nonparametric Type-2 measures, the Type-2 area under the receiver-operating-characteristic curve (AUC2), Gamma, Phi, and delta confidence, together with their eight SDT-normalized ratio and difference forms; and two model-based measures, meta-noise and meta-uncertainty. A single function computes the complete set from trial-level stimulus, response, and confidence arrays. `metasignal` currently supports binary (two-alternative) discrimination tasks, in which each trial's stimulus and response are coded with exactly two categories. The package also provides a command-line interface, group summaries, bootstrap confidence intervals, permutation tests, optional hierarchical
Bayesian models, and information-theoretic measures. `metasignal` unifies these measures in a single platform to encourage broader metacognition research and adoption in decision-making studies.
Read the full abstract ▾
All authors Saurabh Ranjan, Mukesh Makwana, Konstantina Sokratous, Brian Odegaard
Machine Learning (Statistics) · Cross-listed from cs.AI
A Generalized-Bayes Perspective on Counterfactual Explanations: Posterior-Based Decision-Making and Evaluation
Keita Kinjo
Counterfactual explanations (CEs) enhance the interpretability of machine learning models by identifying the smallest change to an input required to obtain a desired output. Although CEs are conventionally formulated as a distance-minimization problem, the theoretical basis of this formulation has received limited
attention. We show that a distance-minimization-based CE is mathematically equivalent to the maximum a posteriori (MAP) estimate of a Gibbs posterior within the generalized Bayes framework, specifically when a distance-based prior is used. We call this formulation the Distance-Prior Generalized Bayes CE (DP-GBCE). Building on this posterior perspective, we introduce two decision rules beyond MAP within a unified framework: a Bayes decision that minimizes expected decision loss and CVaR-CE, a risk-averse decision rule. We also propose an extension that uses
Bayesian model weights to mix the posterior distributions of multiple models, thereby accounting for model multiplicity, where several models have comparable predictive performance. Finally, we define metrics for evaluating both individual CEs and the posterior distribution as a whole, and use experiments on simulated data and Google Trends data to quantify the trade-offs among the decision rules.
Read the full abstract ▾
Machine Learning (Statistics)
Persistent Convolution: A Topological Framework for AI Alignment Testing and Semantic Space Characterization
Tyler Ashoff and Jordan Rodu
Modern opaque AI models prize performance over interpretability, which makes testing difficult. However, formal statistical tests conducted on a model's
embedding space can provide robust characterizations of semantic structure, concept separation, and knowledge graph
alignment. Model developers would benefit from a model comparison technique that leverages human-curated knowledge structures to test
alignment. The scale of the input space for even relatively simple tasks motivates the need for
alignment checks that augment standard outcome
reasoning. This work develops and demonstrates a
topology-based multi-modal
alignment test to make deployment, selection, and comparison of opaque models more interpretable. These methods also offer an intuitive connection to possibility theory and a unified decision theoretic framework from data to deployment.
Read the full abstract ▾
Statistics Theory
Asymptotic Efficiency of the Global Maximum-Likelihood Estimator in Multi-Emitter Localization Microscopy
Yi Sun
We realize the GML
estimator by an expectation-maximization
algorithm initialized near the true positions and verify the theory by simulation.
Statistical Methodology
Distance Profile Embedding for Independence and Conditional Independence Testing of Random Objects
Wenxi Tan, Bing Li and Lingzhou Xue
Testing independence or conditional independence is fundamental to statistical inference, yet existing methods for non-Euclidean random objects often face a difficult trade-off between
geometric flexibility and theoretical tractability. We introduce the Distance Profile
Embedding (DPE), a novel representation that maps random objects from general metric spaces into a
Hilbert space of square-integrable functions. We prove that this mapping is injective and preserves full distributional information without requiring isometric
Hilbert embeddings or one-to-one correspondence conditions. Leveraging the DPE, we develop a unified framework for marginal and conditional independence testing of random objects that enjoys a rigorous asymptotic theory for both size and power. Notably, our framework is the first in the literature to accommodate object-valued conditioning variables when testing conditional independence, overcoming the Euclidean or Hilbertian constraints of existing methodologies. We facilitate the calculation of analytic $p$-values using closed-form asymptotic null distributions, which avoids the computational burden of permutation tests common in existing metric-based methods. The numerical properties of our methods are demonstrated through both simulations and two real-world applications involving gut microbiome compositions and global human mortality distributions, respectively.
Read the full abstract ▾
Statistics Theory
A Simple Example of Bayesian Nonparametric Inconsistency
Antonio R. Linero
I present a very simple example in which a full-support prior over distribution functions has an inconsistent posterior, which I believe has instructive value. The example is $Y_i \stackrel{\text{iid}}{\sim} P_0$ under the
Bayesian hierarchical model $[Y_i \mid P] \stackrel{\text{iid}}{\sim} P$ and $P \sim \int \mbox{DP}(\alpha, H) \, \pi_\alpha(\alpha) \ d\alpha$ when $\pi_\alpha(\cdot)$ has exponential (or heavier) tails, where $\mbox{DP}(\alpha, H)$ denotes a Dirichlet process with concentration parameter $\alpha$ and mean $H(\cdot)$; on the other hand, consistency is obtained under a light-tailed prior. This unifies and generalizes a consistency result of Freedman and Diaconis (1983) with an inconsistency result described by Ferguson et al. (1992).
Read the full abstract ▾
Statistical Methodology
Comparing Two Survival Functions at a Fixed Timepoint
Michael P. Fay, Allyson Mateja and Megan C. Grieco
For comparing two survival curves at a fixed timepoint with right censoring, a standard method uses asymptotic methods on two Kaplan-Meier
estimators with Greenwood variance
estimators and transformations using the delta method. Confidence intervals on associated estimands can either breakdown or undercover due to Kaplan-Meier estimates of zero or one, small sample sizes, or heavy censoring. Although others have proposed adjustments to the Greenwood variance for a single sample, these adjustments have not been incorporated into the two-sample application of the delta method and we do that here as well as provide modifications when Kaplan-Meier estimates are zero or one. An alternative non-asymptotic existing method called melding on the beta product confidence procedure (BPCP) appears to have at least nominal coverage regardless of the sample size. Without censoring the problem reduces to comparing two independent binomials and melding on the BPCP gives p-values equivalent to Fisher's exact test p-values and further gives compatible confidence intervals for treatment comparison estimands; however, the melding can be very conservative. In this paper, we expand the melding on the BPCP intervals to include mid-p versions, which are designed to achieve coverage closer to nominal without guaranteeing coverage for all cases. We study these existing methods and our minor modifications of them primarily by numerical methods or simulation. Results suggest only the melding on the BPCP can guarantee coverage in all studied scenarios, and the mid-p version most often has closest to nominal coverage. We provide R functions in the bpcp R package.
Read the full abstract ▾
Statistical Methodology · Cross-listed from astro-ph.CO
A Sequentially-Valid Reanalysis of DESI's Dynamical Dark Energy Signal
Jinyoung Kim, David F. Mota and Andrius Tamosiunas
Surveys such as DESI release data in stages, and each release invites a fresh assessment of whether dark energy is consistent with a cosmological constant. Standard Wilks-based $\sigma$ values are interpreted as if the test were performed only once, yet revisiting the same question at DR1, DR2, DR3, and beyond inflates the chance of an apparently significant fluctuation. We reanalyse the DESI BAO evidence for dynamical dark energy using an e-process, a likelihood-ratio-based measure of evidence that remains valid under repeated looks at accumulating data: it controls the false-detection probability under continued testing across releases with no trials-factor penalty. Applied to the DESI DR1$\to$DR2 BAO sequence, the result depends strongly on which departures from $\Lambda$CDM the test is designed to detect. For a pre-specified alternative aligned with the DR2-preferred direction, the running evidence reaches $M_{DR2} = 33.97$, crossing an illustrative 5% threshold of 20. The evidence is concentrated almost entirely in the LRG2
redshift bin, and removing that bin reduces the running e-value to $M = 0.49$, mildly favouring $\Lambda$CDM. When allowing any of the seven bins to have produced the excess, the signal does not survive a look-elsewhere correction. Other physically agnostic alternatives also fail to reject $\Lambda$CDM, while a physically motivated thawing-quintessence alternative rejects more strongly than any agnostic choice. The concentration of evidence in LRG2 specifically is difficult to reconcile with a smoothly evolving equation of state. The data therefore points to a fragile, single-bin, specification-dependent signal rather than robust evidence for dynamical dark energy. We recommend that future DESI, Euclid, Roman, and LSST data releases report an anytime-valid e-process value with its test specification stated alongside conventional $\sigma$ significances.
Read the full abstract ▾
Machine Learning (Statistics)
Structured Neural Chaos: An Adaptive Surrogate Modeling Framework for Functional Uncertainty Quantification and Global Sensitivity Analysis
Isabel Corona Guevara and Yeping Hu
Variance-based global sensitivity analysis (GSA) plays a key role in uncertainty quantification by identifying the contributions of uncertain inputs to the variability of the model response. The repeated model evaluations required for these tasks are often prohibitively expensive; surrogate models provide an efficient alternative by constructing inexpensive approximations of the underlying system response. Constructing surrogate models that combine scalability and interpretability for systems with high-dimensional
stochastic inputs and functional responses remains challenging, particularly when sensitivity estimates are required across spatial or temporal domains.
Polynomial chaos expansion (PCE) provides an effective framework for uncertainty propagation and sensitivity analysis due to its orthogonal structure and direct relationship with variance-based sensitivity measures. However, PCE suffers from the curse of dimensionality, whose computational burden is amplified for problems with functional responses. In this work, we introduce the Structured Neural Chaos (sNC) expansion as a surrogate modeling framework for variance-based GSA, inspired by the interpretability and orthogonal structure of PCE. The proposed framework retains the interpretability of structured decompositions while leveraging the expressive power of neural networks. The sNC expansion mirrors a truncated functional ANOVA decomposition, where each interaction component admits a separable low-rank approximation whose basis functions and coefficients are parameterized by neural networks. The expansion is constructed sequentially, adaptively identifying the dominant modes within each ANOVA subspace and determining the effective complexity of the representation. The resulting structure enables the extraction of statistical and sensitivity quantities directly from the coefficients of the sNC expansion at negligible cost.
Read the full abstract ▾