Peer Revenue Fully funded

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics Statistical Methodology

stat.ME Watch Statistical Methodology (sign in to use)

Sunday 23 August

Statistical Methodology · Cross-listed from stat.OT

Data Quality Assessments: A Theoretically Structured Overview of Approaches and Methods

The quality of data is crucial for both practice and academia, and this holds for descriptive statistics, AI and advanced analytics alike. This study addresses an apparent gap in the literature and as such presents a full and theoretically grounded overview of data quality assessment approaches and methods, as well as their defining characteristics and inter-relationships. For this purpose a broad typology is introduced that employs two theoretical dimensions, namely the evaluation logic (formal versus informal) and the assessment driver (norms versus data). This yields four high-level approaches and 15 methods for assessing data quality. The research results are relevant for academia, as they provide a theory-based overview and definition of the ways that data quality can be evaluated. The study is also relevant for practice because it allows professionals to make informed decisions on using these methods, e.g. as part of an audit or broad data quality assessment strategy.
Read the full abstract

Statistical Methodology · Cross-listed from nlin.CD

Noise Effects on Ordinal Pattern Statistics via Majorization

The Bandt-Pompe permutation entropy framework, alongside the complexity-entropy causality plane, has become a standard tool for characterizing the dynamical properties of time series. However, observational noise distorts ordinal pattern probability distributions in ways that can systematically misplace time series within the causality plane, compromising dynamical classification. This effect is particularly relevant for geophysical signals, which are typically poorly and irregularly sampled, and have a low signal-to-noise level. In this work, we characterize the distortions on ordinal pattern statistics using the formalism of majorization. We provide theoretical results and propose corrective strategies that restore discriminability under realistic measurement conditions. To achieve this, we introduce methodology that allows the characterization of noisy dynamical series and further allows the quantification of observational noise without the need of a fitting procedure. Finally, we illustrate our methodology by analyzing paleomagnetic records to determine if the geological evolution of the Earth dipole is better described by a stochastic or chaotic system.
Read the full abstract

Statistical Methodology

Nearest-Neighbor Non-Gaussian Processes for Geostatistical Modeling

We develop a general class of nearest-neighbor non-Gaussian processes (NNnGP) for modeling geostatistical data. By introducing non-linear conditional mean functions into the Vecchia approximation, NNnGP extends the popular nearest-neighbor Gaussian process (NNGP) to effectively capture complex, non-Gaussian spatial characteristics. To ensure coherent spatial predictions from a single realization, we propose a regularized homogeneity condition across the univariate conditionals. We then formulate a Bayesian non-parametric construction of the conditional mean using Gaussian processes and embed the resulting NNnGP as a non-Gaussian spatial prior within a hierarchical regression model. For posterior computation and prediction, we adopt a normalizing flow-based variational inference approach. Numerical experiments on synthetic data and real-world PRISM precipitation anomalies demonstrate that NNnGP captures local non-linear dependencies and significantly mitigates the underestimation of extreme spatial events, all the while maintaining global predictive accuracy.
Read the full abstract

Statistical Methodology

Exploring complex dependence structures using Bayesian bi-clustering and log-linear graphical modelling

Bayesian partitioning is utilised simultaneously on subjects and categorical variables to reveal complex dependence structures. Clusters of variables are referred to as views. Variable selection highlights the variables that drive the clustering of the subjects within each view. We derive theoretical results on the relation between the variables' dependence structure and the inferences derived from bi-clustering. The results relate to marginal independence and conditional independence. Using simulated and real data, we demonstrate the applicability of bi-clustering results in assisting log-linear graphical model determination, leading to the efficient exploration of typically vast model spaces. This work sheds light on the relation between two very different but equally popular Bayesian approaches; mixture modelling, that benefits from a large number of variables, and graphical log-linear modelling, which describes explicitly the variables' dependence structure and allows to evaluate the uncertainty associated with model determination.
Read the full abstract

Statistical Methodology

Exact grid-free confidence-region computation from dependent p-value functions under arbitrary dependence

Repeated sample splitting, cross-fitting, conformal ensembling, and related randomized workflows often produce multiple dependent valid p-value functions for the same target. Existing p-merging theory guarantees pointwise validity under arbitrary dependence, but turning the merged output into a confidence region typically requires repeated evaluation on a grid over the parameter space. This inversion step can be computationally costly, approximation-dependent, and increasingly difficult to scale. We study when confidence regions can instead be computed exactly from split-wise regions without gridding the parameter space. Our main structural result shows that mergers induced by step calibrators form a broad exactly executable class at the region level, and a partial converse within the calibrator-induced family indicates that exact executability is closely tied to step structure. Within this class, we develop exact voting algorithms, including an adaptive multi-quantile contour aggregator that avoids pre-specifying a single order-statistic threshold while preserving finite-sample validity under arbitrary dependence. Simulation studies on repeated-split regression and conformal prediction, together with real-data regression examples, show that the proposed methods provide stable, robustness-oriented exact inference with substantial runtime gains over grid-inversion comparators. A grid-resolution benchmark shows that fixed-k voting and adaptive multi-quantile voting are essentially insensitive to inversion-grid refinement, while a multidimensional single-threshold stress test illustrates the dimensional blow-up faced by grid-inversion baselines in a simple box-geometry setting. Taken together, the results show that arbitrary-dependence contour merging can be turned into an exactly executable region-computation framework rather than merely a pointwise validity device.
Read the full abstract

Statistical Methodology

Analysis of Nonnegative Observations using Gamma Model with 2 Factors (ANOGaM-2): Theory, Method and Applications with Real-life Data (including R code)

Two-factor ANOVA is widely used in experimental studies but relies on additivity, normality, independence, and homoscedasticity. These assumptions are often violated for nonnegative, positively skewed observations. Although Box–Cox-type transformations are commonly used, they may reduce interpretability and require a subjective choice of transformation. We propose an alternative framework in which nonnegative observations affected by two factors are modeled by gamma distributions with unknown shape and scale parameters that may depend on factor levels. We develop likelihood ratio tests (LRTs) for main and interaction effects. The asymptotic LRT (ALRT) uses the asymptotic chi-square distribution, which may be inaccurate for small to moderate samples. We therefore propose a parametric bootstrap LRT (PBLRT) that determines critical values by simulation. Extensive simulations show that the PBLRT maintains the nominal significance level well. Real-data examples demonstrate its applicability and show that its inferences can differ from those of traditional ANOVA.
Read the full abstract

Statistical Methodology

Efficient Regression Models for Scan Statistics

We introduce a new class of regression models for scan statistics on real-valued signals. These allow for improved fitting of non-stationary signals to contrast with the interval anomalies identified by the scan statistics. Our models can represent generalized likelihood ratio statistics. While these methods naively require $O(n^4)$ for a length $n$ signal, we provide algorithmic improvements which lead to linear time algorithms (with assumptions on max interval width). Our methods, especially ones based on Nadaraya-Watson kernel regression, are demonstrated as especially effective in detecting both synthetically planted anomalies, and for identifying a real “platforming” issue in interferometric astronomy.
Read the full abstract

All authors Gazi Abdur Rakib, Tristan Ashton, Ryan A. Loomis, Brian S. Mason, Eric J. Murphy, Ci Xue, Jeff M. Phillips

Statistical Methodology · Cross-listed from stat.ML

Token-Level Likelihood-Array Regression for Membership Inference and AI-Generated Text Detection

Membership inference asks whether a text was used to train a language model, whereas AI-generated text detection asks whether it was generated by a language model rather than written by a human. Existing likelihood-based methods typically compress token-level probabilities into a few prespecified scores, most often using only probabilities conditioned on the full preceding context. We propose likelihood-array regression (LAR), which evaluates each target token under nested left-context windows and organizes the resulting likelihood-derived features into a structured array. After aligning arrays across texts of different lengths, LAR learns how detection information varies with context scale, token position, and likelihood features. LAR-1 aggregates learned contributions from individual aligned cells, while LAR-2 adds second-order features formed from pairs of evaluations of the same target token across context lengths. For within-path quadratic model, we establish matching minimax lower and upper bounds, characterize errors from finite-dimensional approximation and random squared projections, and derive conditions under which an oracle spectral sieve attains the minimax rate. Across multiple scoring language models, LAR substantially improves membership inference and AI-generated text detection over likelihood-based baselines. The analyses further show that shorter-context likelihoods contain information beyond conventional full-context probabilities, while second-order features provide additional gains for membership inference.
Read the full abstract

Statistical Methodology

Functional Multiple-Set Canonical Correlation Analysis Revisited: From Finite-Dimensional Samples to Infinite-Dimensional Populations

We develop a population-level formulation of functional multiple-set canonical correlation analysis (P-FMCCA) for multivariate functional data in an infinite-dimensional Hilbert space. Since covariance operators for functional data are typically compact, the inverse covariance operators that appear in the formal extension of multiple-set canonical correlation analysis (MCCA) are generally unbounded and are not defined on the whole Hilbert space. We therefore provide sufficient conditions under which the proposed population formulation is well-defined and show that the resulting constrained maximization problem is characterized by an eigenvalue problem for a Hilbert-Schmidt extension of the relevant correlation operator. We further establish a canonical decomposition induced by the P-FMCCA components and introduce the associated truncated canonical representation. In addition, we formulate functional homogeneity analysis at the population level and show that the finite-dimensional equivalence between homogeneity analysis and MCCA does not generally carry over to the infinite-dimensional setting. Finally, we prove that, for the finite-rank truncated canonical representation, functional homogeneity analysis admits an explicit characterization in terms of the P-FMCCA components, thereby providing a population-level counterpart of the classical finite-dimensional correspondence.
Read the full abstract

Saturday 22 August

Statistical Methodology · Cross-listed from econ.EM

Distributional Difference-in-Differences: Aggregation Before or After Quantile Inversion?

Staggered distributional difference-in-differences produces cohort-specific potential-outcome distributions, but applied work typically wants one overall quantile treatment effect. Two natural summaries—averaging cohort quantile treatment effects (QTTs) and mixing cohort distributions before inversion—use the same policy weights yet answer different target-population questions and can disagree even in sign. We derive the exact sharp interval for their gap conditional on cohort quantiles and weights, a globally sharp range-only envelope, and a locally sharp density-tilt representation. We develop joint smooth and mass-point-safe inference and show that neither estimator is uniformly more precise, even when the estimands coincide. In a same-object reconstruction of a public staggered-QTT application, holding data, identification, distributions, and weights fixed while changing only aggregation order reverses reported signs at several quantiles. Aggregation order is therefore part of the estimand and must be chosen before inversion.
Read the full abstract

Statistical Methodology

Dirichlet Process Mixtures of Trees with Gaussian Process Splits: A Bayesian Nonparametric Framework with Posterior Contraction Rate

We propose a Bayesian nonparametric mixture of regression trees with a Dirichlet process prior over tree-parameter pairs, enabling data-driven selection of ensemble size and unifying CART, BART, random forests, and boosting. A novel splitting rule driven by the posterior predictive of a Gaussian process within each terminal node generates flexible, smooth decision boundaries; remarkably, the GP density cancels exactly in the Metropolis–Hastings ratio for GROW/PRUNE moves, ensuring computational feasibility. An exact Gibbs sampler for posterior predictive inference propagates uncertainty through random tree traversal. A parallel MPI implementation distributes independent tree updates across processors, achieving adequate speedups. We prove posterior consistency at rate $n^{-1/4}$ in Hellinger distance under only continuity of the true regression function, allowing misspecification, via the identity $h(\Theta)=0$. Simulations on Friedman benchmark show near-nominal coverage (0.94 Gaussian, 0.92 Cauchy), robust to high-dimensional noise and heavy tails, outperforming BART and bagged CART. Applications to QSAR toxicity, crime, riboflavin, wheat genomics, and air quality confirm reliable credible intervals and automatic sparsity. The DP mixture offers a principled, robust, theoretically justified alternative for challenging regression with honest uncertainty quantification.
Read the full abstract

Statistical Methodology

Separating Time-Varying Network Composition from Predictive Dependence under Noisy Network Measurement

A common question about networked time series is whether outcomes changed because shocks transmit more strongly or because the pattern of connections changed. Standard practice inserts a recorded network into an outcome regression and reads movements of the fitted coefficient as changes in transmission strength. When the network is latent, time varying, and measured with error, this reading fails: changes in strength and changes in composition can produce the same outcome distribution at a single date, and the population coefficient moves under composition changes alone. The question becomes answerable when outcomes are analyzed jointly with repeated noisy measurements of the network, such as paired reports of bilateral trade flows. For the joint model we establish necessary and sufficient conditions for local identification, estimators of the strength and composition paths, a simultaneous confidence band for the strength path, confidence sets that remain exact under weak identification, breakdown bounds under common reporting bias, and an exactly sized test that detects changes on the observed path and attributes them to strength or to composition. Simulations assess each procedure at its stated boundary. On a mirror-reported trade panel of eighteen economies over 1995 to 2020, the diagnostics flag exactly the crisis years and the composition coordinate attached to European Union membership declines by roughly two thirds. The estimand is predictive dependence, not a causal effect.
Read the full abstract

Statistical Methodology

Optimal Variance Reduction in Randomized Experiments

This paper describes an approach to variance reduction in randomized experiments using side information (covariates of the response unaffected by treatment). As with Double ML[Chernozhukov 2018], models of arbitrary complexity may be employed without concern for bias due to overfitting or regularization. For additive treatment metrics, the approach minimizes variance optimally. For more complex treatment metrics (e.g., multiplicative and ratios) the Delta Method estimate of variance is minimized. Through a variety of examples, the paper also explores modeling considerations.
Read the full abstract

Statistical Methodology

Barycentric Fused Gromov-Wasserstein Balancing for Causal Inference under Multiple Treatments

Estimating heterogeneous single and interaction treatment effects from observational data under multiple simultaneous treatments is crucial for decision-making. To mitigate estimation variance, previous studies balance representation distributions between every pair of treatment patterns. However, such pairwise balancing scales quadratically with the number of treatment patterns and fails to preserve consistent local proximity structures across patterns, which degrades counterfactual estimation. To address these challenges, we propose the Causal Inference for Heterogeneous Single and Interaction Treatment Effects Network (CIHSI-Net), a deep learning framework built on a novel Barycentric Fused Gromov-Wasserstein Balancing (BFG-WB) objective. BFG-WB aligns the representation distribution of each treatment pattern with a shared Wasserstein barycenter, achieving global alignment while reducing the computational complexity from quadratic to linear, and its Fused Gromov-Wasserstein discrepancy preserves the local proximity structures essential for reliable heterogeneous effect estimation. Simulation studies show that CIHSI-Net consistently outperforms state-of-the-art baselines, and an application to real-world marketing data demonstrates its practical utility in complex multi-treatment scenarios.
Read the full abstract

Statistical Methodology · Cross-listed from stat.ML

Guidance for Prior Change via Density Ratio Estimation

Simulation-Based Inference (SBI) serves as a vital framework for parameter inference in scientific fields where simulators involve intractable likelihoods, yet while amortized generative models offer rapid posterior estimation, they are often restricted by the specific priors used during training, thereby limiting their flexibility as prior knowledge evolves. To address this prior dependency, PriorGuide was introduced as an inference-time guidance method, but due to its intractable formulation, it relies on Gaussian approximations of the reverse transition kernel and Gaussian mixture model fitting for the prior ratio, both of which introduce systematic bias. Motivated by these limitations, we propose an unbiased test-time guidance framework that leverages Density Ratio Estimation (DRE) to learn a score guidance term, effectively decoupling the inference process from the prior training. Moreover, our framework remains agnostic to the specific density ratio estimators, making it a general and flexible framework for handling prior changes. Experimental results across multiple tasks demonstrate that our method matches or outperforms PriorGuide on C2ST and MMD in most tasks while maintaining robustness even under limited overlap between the training and target priors. Furthermore, we apply our method to Bayesian updating for parameter inference from planetary light-curve data, where it also demonstrates strong effectiveness and robustness. Code is available at https://github.com/a-chenchen/dre-based-prior-guidance .
Read the full abstract

Statistical Methodology

Gaussian-efficient testing by betting on the mean of bounded data

Given $[0,1]$-valued random variables $X_1,\dots,X_n$ such that $\mathbb{E}[X_i | X_1,\dots,X_{i-1}]= \mu$ for all $i$, we propose a new nonasymptotic confidence interval for $\mu$ that is obtained by inverting terminal e-values generated by a novel betting strategy. When the data are iid, its limiting width matches that of the central limit theorem (“Gaussian-efficient”), finally surpassing the inefficient limits of previous betting intervals. Our main conceptual advance involves designing betting fractions that track the conditional rejection probability of the most powerful terminal test in a limiting Gaussian experiment. When one predictable variance estimator is shared across candidate means, the deterministic inversion is an interval for every data sequence and its two endpoints can be found easily. The width can be improved further with external randomization. In simulations, our method yields the tightest intervals to date; for every distribution tested and all sufficiently large $n$, our deterministic version beats STaR-Bets and is competitive with Gaffke, while the randomized improvement beats both. It thus combines finite-sample validity under martingale dependence, easy endpoint computation, Gaussian-efficient inference for iid data, and excellent empirical performance. We also extend the construction and its efficiency theory to sampling without replacement, where it again achieves state-of-the-art empirical performance.
Read the full abstract

Friday 21 August

Statistical Methodology · Cross-listed from stat.ML

On the Tightness and Computational Tractability of Higher-Dimensional Confidence Sequences

Modern sequential monitoring problems often involve multiple metrics, where we monitor several data streams simultaneously and may act once the evidence is strong enough. Confidence sequences (CSs) are a natural tool for such continuous monitoring. However, for bounded vector means, existing multivariate CSs are either tight but computationally intractable, or fast to compute but conservative. To address this, we study three lifts of one-dimensional betting-based CSs to higher dimensions: a weighted Bonferroni region, an equivalent max-wealth form, and a portfolio region. The portfolio is typically much tighter, especially in higher dimensions, but its boundary and properties such as volume are not available in closed form. To make this tighter construction usable, we propose tractable outer approximations of the portfolio region that preserve statistical validity: a bounding box, an $\ell_p$-ellipsoid, and their intersection. We prove set relations among all constructions and show empirically that these approximations (i) achieve regions close to the intractable portfolio, (ii) substantially outperform existing tractable multivariate CSs, and (iii) enable practical use cases such as multi-metric A/B testing and model comparison.
Read the full abstract

All authors Fabian Denoodt, Sibylle Hess, Joaquin Vanschoren, Christian A. Naesseth

Statistical Methodology

Regression to the Mean-Adjusted Sample-Size Determination for Cutoff-Selected Single-Arm Pre-Post Studies

Studies enrolling participants on the basis of an extreme baseline value are susceptible to regression to the mean (RTM), such that some observed pre-post change is expected even without treatment. Existing methods estimate and decompose RTM retrospectively. We extend this framework to prospective study design by deriving a closed-form sample-size method for continuous, cutoff-selected, single-arm pre-post studies. The method partitions anticipated total change into that expected from RTM and the residual treatment effect, and powers the study to detect the latter. It uses the general bivariate-normal RTM expression, allowing baseline and follow-up variances to differ, and incorporates the corresponding conditional variance of the pre-post change. To our knowledge, no published method or software implements this cutoff-based RTM framework for prospective closed-form sample-size determination in this setting. The Stata command power onemean_rtm provides solutions for sample size and minimum detectable effect, normal-theory power evaluation, attrition adjustment, and sensitivity analysis. Monte Carlo simulation across 58 scenarios demonstrated accurate Type I error control and showed that achieved power rapidly approached nominal power as sample size increased, with the closed-form calculation generally conservative at very small sample sizes.
Read the full abstract

Statistical Methodology

A Hybrid Spatial Statistical Learning Framework for Individualized Probability Estimation: Application to Multisite Autism Neuroimaging

Autism spectrum disorder is a heterogeneous neurodevelopmental condition whose functional brain organization varies across individuals and imaging centers. Resting-state functional connectivity provides an opportunity to study this variation, but analysis is complicated by high dimensionality, strong dependence among connections, and substantial site heterogeneity. We introduce a Hybrid Spatial Statistical Learning Framework for individualized probability estimation in multisite autism neuroimaging. The framework combines edge-level connectivity, graph-based network organization, and participant characteristics as complementary representations of a common probability target. Fine-scale connectivity retains discriminatory information, while broader representations stabilize probability estimates under site shift. All preprocessing and model development are performed within complete site-held-out validation to assess transportability to unseen acquisition environments. Simulations show that the hybrid preserves the discrimination of the strongest edge model while improving probability accuracy as between-site heterogeneity increases. In 860 participants from 20 ABIDE sites, the hybrid retained ROC AUC while reducing Brier score and log loss. These results show that multiscale integration can improve the reliability and transportability of individualized probabilities from complex, spatially dependent biomedical data.
Read the full abstract

Statistical Methodology

Bayesian Tensor Regression for Neuroimaging Data

Multidimensional array data, or tensors, arise naturally in neuroimaging and other high-dimensional applications. We propose a parsimonious Bayesian tensor regression model for studies in which a brain image is the response and predictors are vector-valued covariates. The method extends Bayesian envelope dimension reduction to tensor responses, identifying material subspaces that contain regression information while removing variation that is immaterial to the predictors. This formulation leads naturally to a Tucker tensor decomposition and allows spatial dependence and multiple sources of uncertainty to be modeled jointly. We develop a computationally feasible Markov chain Monte Carlo algorithm based on Gibbs sampling and establish posterior consistency for the proposed model. Simulation studies demonstrate substantial gains in estimation accuracy and uncertainty quantification when meaningful dimension reduction is present. We apply the method to Human Connectome Project neuroimaging data to investigate associations between alcohol use and brain activity. The results illustrate the value of Bayesian tensor envelope regression for inference with high-dimensional, spatially dependent imaging responses.
Read the full abstract

All authors Zahra Naji, Montserrat Fuentes, Liangsuo Ma, Hossein Moradi Rekabdarkolaee

Statistical Methodology

Estimating the True Effect Size Distribution with SIMEX

Large-scale online experimentation produces noisy effect estimates, which can overstate gains and complicate decisions about launches and testing policies. We propose a nonparametric method based on SIMulation-EXtrapolation (SIMEX) to estimate the latent distribution of true effects from estimated average treatment effects with known variances. The method evaluates quantiles after adding progressively more simulated measurement noise and extrapolates the resulting inverse cumulative distribution function to the zero-noise setting while enforcing monotonicity of the quantiles. In a synthetic example with normally distributed true effects and measurement error, the method recovers the underlying effect distribution and performs nearly as well as a parametric empirical Bayes normal means approach. This provides a flexible way to characterize effect-size distributions without fully specifying a parametric model.
Read the full abstract

All authors Zhaoqi Li, Daniel Ting, Ilya Gorbachev, Ehsan Emamjomeh-Zadeh, Houssam Nassif

Statistical Methodology

A Sparse-Group Pliable Lasso

The sparse-group pliable Lasso (SGPL) extends the pliable Lasso and group pliable Lasso by combining sparse-group regularization with a predictor-level coupling penalty, enabling simultaneous group-level selection, within-group sparsity, and hierarchical structure between main effects and interactions. We propose a blockwise coordinate descent algorithm for fitting the SGPL that exploits the convexity and structure of the objective function, establish convexity and Karush–Kuhn–Tucker optimality conditions, and prove that the algorithm converges to a global minimizer. Simulation studies demonstrate competitive predictive performance and smaller interaction estimation error than the pliable Lasso and group pliable Lasso, albeit with the expected precision–recall trade-off in support recovery. We further illustrate the proposed method using a Parkinson's disease gut microbiome study and an adrenocortical carcinoma (ACC) copy-number dataset from The Cancer Genome Atlas. The Parkinson's application identifies interpretable interactions between microbial abundances and dietary variables, while the ACC application illustrates that the effectiveness of group-structured regularization depends on how well the prespecified grouping reflects the underlying signal structure.
Read the full abstract

All authors Mohammad Javad Davoudabadi, Minh Long Nguyen, Amirhossein Ghatari, Mina Aminghafari, Kerrie Mengersen

Statistical Methodology

$K$-functions for point processes on complex surfaces

The $K$-function is a fundamental summary statistic for assessing clustering or regularity of point processes in two or three dimensional Euclidean space. In practice, however, many planar point patterns arise from projecting locations of objects on a surface in three dimensional space to two dimensional space. For example, when events or objects occur in a landscape their elevation is often ignored. This can lead to erroneous conclusions regarding properties of the point process generating the point pattern. There is not a unique way to extend the classical $K$-function to point patterns on a complex surface. In this paper we propose, explore, and discuss several approaches in terms of their theoretical and computational properties. The best performing approach, coined the surface area $K$-function, can be viewed is an analogue of the classical $K$-function replacing counts of points in Euclidean balls with counts of points in surface geodesic balls. However, an important distinction is that the argument of our surface area $K$-function is area instead of radius of geodesic balls. The performances of the various surface $K$-functions are compared in applications to simulated and real data.
Read the full abstract

All authors Francisco Cuevas-Pacheco, Scott Ward, Ed Cohen, Rasmus Waagepetersen

Statistical Methodology · Cross-listed from stat.ML

Random Hazard Forests

Clinical data sources such as electronic health records and wearable sensors record patient status repeatedly over follow-up, often at irregular times and on different schedules for different measurements. These data create opportunities for continuously updated, individualized risk prediction. Existing approaches, however, often simplify the temporal structure for model fitting. We introduce Random Hazard Forests (RHF), a survival tree ensemble that estimates how a patient's hazard changes in continuous time as new measurements become available. The method formulates the estimation problem directly through a nonparametric hazard likelihood for predictable covariate processes. An efficient working model guides tree construction, after which flexible time-varying hazards are estimated for each terminal node. Given any predictable covariate path, each tree follows the path through its terminal nodes over time and assembles the corresponding node-level hazards into a trajectory. Averaging these trajectories across trees yields the pathwise hazard estimate. Because routing at each time uses only the covariate state available immediately beforehand, the construction accommodates internal longitudinal covariates without lookahead. Simulations and an intensive care application show that the forest accurately estimates changing risk under irregular and asynchronous covariate updates.
Read the full abstract

All authors Hemant Ishwaran, Eileen M. Hsich, Udaya B. Kogalur, Donald K.K. Lee

Statistical Methodology

Outlier Impact: Detection by Consequences

We introduce a novel outlier detection method that utilizes a concrete measure of the statistical impact of an outlier rather than a vague notion of "unusualness". This allows practitioners to only flag points that materially affect the inferences made from the data while also ensuring these points are anomalous. This is of particular interest in experimentation, where outliers affect the validity of the causal inferences. In particular, we consider a potential outlier's impact on the mean and the False Positive Rate (FPR) of a hypothesis test.
Read the full abstract

Statistical Methodology · Cross-listed from stat.ML

Sparse Separable Factor Analysis in the Complex Domain with an Application to Local Field Potential Data

Complex-valued arrays arise in signal processing, where scientific interpretation depends on retaining amplitude and phase information. Existing covariance estimation methods either ignore the multiway organization of such data or rely on real-domain embeddings that do not directly exploit their complex structure. We develop sparse separable factor analysis (SSFA), a latent factor model for complex-valued arrays with a separable covariance structure across modes. Each mode-specific covariance matrix is modeled through a low-rank Hermitian factor structure and a diagonal residual covariance matrix. To obtain interpretable estimates, we impose elementwise lasso penalties on the complex loading matrices and estimate the SSFA parameters using a mode-wise parameter-expanded expectation-maximization procedure. The resulting loading updates admit closed-form complex soft-thresholding solutions, which shrink the modulus of each loading while preserving its phase. A separate balancing step resolves the scale nonidentifiability of the separable covariance structure. Simulation studies show that SSFA improves covariance estimation relative to vectorization-based methods, including complex principal component analysis. We apply SSFA to local field potential recordings from mice, where we compare separability structures induced by different groupings of brain region, frequency, and time and perform model-based imputation of recordings missing because of electrode misplacement.
Read the full abstract

All authors Ian Hultman, Kirtikanth Kalapatapu, Yassine Filali, Rainbo Hultman, Sanvesh Srivastava

Statistical Methodology · Cross-listed from stat.ML

A Data-Driven Approach to State Construction in Markov Models

A Markov chain is a widely used stochastic process modelling random events over time. These models are built on subsets of the entire dataset, referred to as states, which are considered to be homogeneous regarding transition probabilities. However, the creation of these states is often disregarded or based on prior assumption, potentially violating the homogeneity requirement and thus decreasing the validity and predictive power of the model. In order to fill this gap, this paper combines supervised feature selection with unsupervised learning techniques for data-driven state construction. Density-based clustering, spectral clustering, and Kohonen self-organizing maps are examined for their ability to identify latent groups without prior assumptions. The contribution of this study is twofold. First, the paper presents a methodological framework for state construction incorporating suitable unsupervised learning techniques, with appropriate measures both for classification performance and Markov model accuracy. Secondly, the framework is tested on an application, resulting in a comparative analysis showing that spectral clustering and Kohonen self-organizing maps are best at capturing inherent structure. These results serve as a cornerstone in providing theoretical and methodological guidance for improving state definition in applied Markov modelling.
Read the full abstract

Statistical Methodology · Cross-listed from cs.HC

Event-Time Confounding Under Bursty Human Dynamics

Studies of digital behavior often align users at moments they choose, such as opening an AI assistant, clicking a recommendation, or visiting a product page, and interpret higher activity afterward as an event effect. We show how this creates an endogenous time zero: the event occurs during an ongoing task episode, so the aligned curve can trace episode continuation rather than a response to the event. In same-user, cross-surface web logs, AI, shopping, news, coding, and reference events are all preceded by broad activity increases that peak before time zero. Our strongest test uses known-null timestamps that cause nothing. Among the 5.8% of AI responses meeting strict pre-event activity and washout criteria, these timestamps show 3.42 times the post-event search activity of a within-user placebo, compared with 4.32 times for real events. The fraction of excess reproduced by the known null falls from 0.56 at detectably active moments to -0.04 at quiet moments, where the design detects none. We formalize this episode-selection bias, prove that a single-surface event window cannot separate it from a genuine effect without additional assumptions, and show in zero-effect simulations why user fixed effects and coarse activity matching can fail: the confound is within-user and time-varying. We provide a diagnostic protocol, public-data benchmarks, and burstcheck, a lightweight audit tool. User-timed events may have real effects, but post-event volume does not identify them by default; studies should compare similar episodes with and without the event.
Read the full abstract

Statistical Methodology

Further Results on Controlling the False Discovery Rate in Two-Sided Gaussian Mean Testing

The recent work of Sarkar and Zhang (2025) introduced Positive Tail Dependence Under the Null (PTDN) and developed Generalized Shifted Benjamini-Hochberg (BH) procedures for two-sided Gaussian $z$- and $t$-testing under known covariance structures. This paper develops further consequences of that framework. First, we derive explicit dependence-adaptive lower and upper bounds for the FDR of the original BH procedure in terms of the conditional variance parameters $\tau_i=1-R_i^2$, where $R_i^2$ is the squared multiple correlation between the $i$th statistic and the remaining coordinates. These bounds recover the exact BH FDR under independence and provide finite-sample, covariance-specific information complementary to generic bounds. We also identify conditions under which the coordinate-specific calibration of shifted BH can provide a rejection advantage over the original BH procedure. Second, we consider the practically important setting in which the covariance matrix is unknown but an independent Wishart estimator is available. Using simultaneous lower confidence bounds for the $\tau_i$'s, we construct a confidence-bound shifted BH procedure and establish finite-sample FDR control. To our knowledge, this is the first shifted-BH-type procedure with a finite-sample guarantee for two-sided Gaussian mean testing under a completely unknown covariance matrix estimated independently. Numerical studies illustrate the behavior of the covariance-adaptive bounds, the potential advantage of shifted BH over BH, and the performance of confidence-bound shifting under unknown covariance.
Read the full abstract

Statistical Methodology

LABS: Extending the scope of binary segmentation via a look-ahead device

Binary segmentation is widely used for multiple change-point detection because it is fast, simple to describe, and simple to implement. Its validity rests on the requirement that, at each recursive stage, the procedure identifies one of the true change-points when several are present in the current interval. This holds for detecting changes in mean using the CUSUM statistic, but fails in some other settings, in particular in slope change detection for continuous piecewise-linear signals. We propose Look-Ahead Binary Segmentation (LABS), a modification in which the change-points returned by the two child recursions define a narrower interval on which the parent estimate is re-evaluated. LABS inherits the computational speed of standard binary segmentation, but achieves the near-optimal consistency rate of $O\{(n\log n)^{1/2}\}$ in the slope-change signal setting when the LABS model is chosen via either thresholding or a Schwarz-like information criterion. Simulations show that LABS is fast and achieves state-of-the-art performance.
Read the full abstract