Statistics Applications · Cross-listed from cs.LG
Global tree forecasters collapse at the hierarchical aggregate: a five-panel failure characterization
Md Rezwanul Islam and Wael Mohammed
Global forecasting models pool many series and learn one shared function. Gradient-boosted trees are their most common form. We measure a failure of this design that has not, to our knowledge, been documented. Train a global tree on the individual series of a hierarchy, then ask it for the hierarchical aggregate. The aggregate sits far outside the model's training range, and the forecast collapses. The model under-predicts the total by 30-50x in our production deployment, and by up to 496x in a public M5 reconstruction. The mechanism is known: beyond its training range, a tree predicts a constant. It surfaces at the aggregate because the total dwarfs every training series. The cure is not new. Per-series scaling, the preprocessing step that Montero-Manso and Hyndman (2021) recommend, prevents the collapse. So do a weighted aggregate-level training row and seasonal differencing. Our contribution is the characterization. The collapse reproduces on five panels: a production business-to-business marketplace, a synthetic hierarchy, M5, Australian Tourism, and a public business-buyer panel. It holds on three tree libraries, is invariant across training seeds, and is statistically significant. Its onset is immediate and tracks a simple support bound: a scale gap of only 1.15x already costs a third of the total. No standard configuration change prevents it: pooling every hierarchy level into training fails at scale, and the one knob that fits linear models in the leaves softens it without curing it. Rolling the forecasts forward recursively separates the cures: the aggregate-row cure re-collapses, per-series scaling degrades but stays low, and only seasonal differencing keeps its one-step accuracy unchanged. We close with a three-step procedure for diagnosing and preventing the failure in deployed systems.
Read the full abstract ▾
Statistical Computation
A Uniformly Efficient Rejection Sampler for the Multivariate Expectile-Based Distribution
Sam Power
The multivariate expectile-based distribution is a `spiked' perturbation of a multivariate Gaussian distribution, proposed by Arbel et al. in 2023. The rejection sampler proposed for this distribution in the initial work is valid, but its acceptance probability can degenerate badly both in high dimension and in the strongly asymmetric limit. We give a simple alternative. After affine whitening and a polar decomposition, the sampling problem reduces exactly to a univariate distribution, and an additional
hyperbolic change of variables exhibits this distribution as log-concave, so that a universal and uniformly efficient construction of Devroye applies. The resulting exact sampler therefore enjoys an acceptance probability of at least $1 - \exp \left( -1 \right) = 0.632120 \ldots$, uniformly over the dimension and all admissible asymmetry parameters.
Read the full abstract ▾
Statistical Methodology
A Hybrid Spatial Statistical Learning Framework for Individualized Probability Estimation: Application to Multisite Autism Neuroimaging
Montserrat Fuentes and Veronica B. Patterson
Autism spectrum disorder is a heterogeneous neurodevelopmental condition whose functional
brain organization varies across individuals and imaging centers. Resting-state functional connectivity provides an opportunity to study this variation, but analysis is complicated by high dimensionality, strong dependence among connections, and substantial site heterogeneity. We introduce a Hybrid Spatial Statistical Learning Framework for individualized probability estimation in multisite autism neuroimaging. The framework combines edge-level connectivity, graph-based network organization, and participant characteristics as complementary representations of a common probability target. Fine-scale connectivity retains discriminatory information, while broader representations stabilize probability estimates under site shift. All preprocessing and model development are performed within complete site-held-out validation to assess transportability to unseen acquisition environments. Simulations show that the hybrid preserves the discrimination of the strongest edge model while improving probability accuracy as between-site heterogeneity increases. In 860 participants from 20 ABIDE sites, the hybrid retained ROC AUC while reducing Brier score and log loss. These results show that multiscale integration can improve the reliability and transportability of individualized probabilities from complex, spatially dependent biomedical data.
Read the full abstract ▾
Statistical Methodology
Bayesian Tensor Regression for Neuroimaging Data
Zahra Naji, Montserrat Fuentes, Liangsuo Ma et al. (+1 more)
Multidimensional array data, or tensors, arise naturally in neuroimaging and other high-dimensional applications. We propose a parsimonious
Bayesian tensor regression model for studies in which a
brain image is the response and predictors are vector-valued
covariates. The method extends
Bayesian envelope dimension reduction to tensor responses, identifying material subspaces that contain regression information while removing variation that is immaterial to the predictors. This formulation leads naturally to a Tucker tensor decomposition and allows spatial dependence and multiple sources of uncertainty to be modeled jointly. We develop a computationally feasible Markov chain Monte Carlo
algorithm based on Gibbs sampling and establish posterior consistency for the proposed model. Simulation studies demonstrate substantial gains in estimation accuracy and uncertainty quantification when meaningful dimension reduction is present. We apply the method to Human Connectome Project neuroimaging data to investigate associations between alcohol use and
brain activity. The results illustrate the value of
Bayesian tensor envelope regression for inference with high-dimensional, spatially dependent imaging responses.
Read the full abstract ▾
All authors Zahra Naji, Montserrat Fuentes, Liangsuo Ma, Hossein Moradi Rekabdarkolaee
Statistical Methodology
Estimating the True Effect Size Distribution with SIMEX
Zhaoqi Li, Daniel Ting, Ilya Gorbachev et al. (+2 more)
Large-scale online experimentation produces noisy effect estimates, which can overstate gains and complicate decisions about launches and testing policies. We propose a nonparametric method based on SIMulation-EXtrapolation (SIMEX) to estimate the latent distribution of true effects from estimated average treatment effects with known variances. The method evaluates
quantiles after adding progressively more simulated measurement noise and extrapolates the resulting inverse cumulative distribution function to the zero-noise setting while enforcing monotonicity of the
quantiles. In a synthetic example with normally distributed true effects and measurement error, the method recovers the underlying effect distribution and performs nearly as well as a parametric empirical Bayes normal means approach. This provides a flexible way to characterize effect-size distributions without fully specifying a parametric model.
Read the full abstract ▾
All authors Zhaoqi Li, Daniel Ting, Ilya Gorbachev, Ehsan Emamjomeh-Zadeh, Houssam Nassif
Statistical Methodology
A Sparse-Group Pliable Lasso
Mohammad Javad Davoudabadi, Minh Long Nguyen, Amirhossein Ghatari et al. (+2 more)
The sparse-group pliable Lasso (SGPL) extends the pliable Lasso and group pliable Lasso by combining sparse-group regularization with a predictor-level coupling penalty, enabling simultaneous group-level selection, within-group sparsity, and hierarchical structure between main effects and interactions. We propose a blockwise coordinate descent
algorithm for fitting the SGPL that exploits the convexity and structure of the objective function, establish convexity and Karush–Kuhn–Tucker optimality conditions, and prove that the
algorithm converges to a global minimizer. Simulation studies demonstrate competitive predictive performance and smaller interaction estimation error than the pliable Lasso and group pliable Lasso, albeit with the expected precision–recall trade-off in support recovery. We further illustrate the proposed method using a Parkinson's disease gut microbiome study and an adrenocortical carcinoma (ACC) copy-number
dataset from The Cancer Genome Atlas. The Parkinson's application identifies interpretable interactions between microbial abundances and dietary variables, while the ACC application illustrates that the effectiveness of group-structured regularization depends on how well the prespecified grouping reflects the underlying signal structure.
Read the full abstract ▾
All authors Mohammad Javad Davoudabadi, Minh Long Nguyen, Amirhossein Ghatari, Mina Aminghafari, Kerrie Mengersen
Statistical Methodology
$K$-functions for point processes on complex surfaces
Francisco Cuevas-Pacheco, Scott Ward, Ed Cohen et al. (+1 more)
The $K$-function is a fundamental summary statistic for assessing clustering or regularity of point processes in two or three dimensional Euclidean space. In practice, however, many planar point patterns arise from projecting locations of objects on a surface in three dimensional space to two dimensional space. For example, when events or objects occur in a landscape their elevation is often ignored. This can lead to erroneous conclusions regarding properties of the point process generating the point pattern. There is not a unique way to extend the classical $K$-function to point patterns on a complex surface. In this paper we propose, explore, and discuss several approaches in terms of their theoretical and computational properties. The best performing approach, coined the surface area $K$-function, can be viewed is an analogue of the classical $K$-function replacing counts of points in Euclidean balls with counts of points in surface geodesic balls. However, an important distinction is that the argument of our surface area $K$-function is area instead of radius of geodesic balls. The performances of the various surface $K$-functions are compared in applications to simulated and real data.
Read the full abstract ▾
All authors Francisco Cuevas-Pacheco, Scott Ward, Ed Cohen, Rasmus Waagepetersen
Machine Learning (Statistics) · Cross-listed from cs.LG
GeoQ: Geometry-Aware Conditional Quantile Error Estimation for Scientific Surrogate Models
Khoa Nguyen, Daniel Serino, Aviral Prakash et al. (+1 more)
Neural-network surrogate models are increasingly used to accelerate scientific simulations, but their deployment in extrapolative and autoregressive settings requires input-dependent estimates of prediction error. In this work, we introduce GeoQ (Geometry-Aware Conditional
Quantile Error Estimation), a non-intrusive calibration framework for estimating surrogate error at individual query points. GeoQ represents the error at a query point as an anchor-averaged calibration error plus a learned nonnegative correction. This correction is modeled as an upper conditional
quantile of the anchor-relative error increment, using geometry-based features that encode representation-space displacement and local support density. A cross-fitting procedure generates approximately out-of-sample calibration tuples, while a feature-space k-nearest-neighbor support score identifies regions \textcolor{black}{where the learned error model is supported by calibration data}. We evaluate GeoQ on scalar regression, chaotic dynamics, medium-range weather forecasting, and Richtmyer-Meshkov instability prediction. The results demonstrate that geometry-aware conditional
quantile modeling provides a practical and non-intrusive approach for validity-aware error estimation in scientific surrogate models.
Read the full abstract ▾
All authors Khoa Nguyen, Daniel Serino, Aviral Prakash, Marc Klasky
Machine Learning (Statistics) · Cross-listed from cs.LG
Subzero matrix completion for sparse data analysis: large-scale learning of latent low-rank structure
Lawrence K. Saul, Ningyuan Huang, Dennis Bollweg et al. (+2 more)
We investigate when a sparse nonnegative
matrix can be recovered from a real-valued
matrix of much lower rank by zeroing out its negative elements. The potential for such decompositions suggests a mathematical connection between sparsity and rank; we analyze a number of sparse
matrices with this latent low-rank structure and use them to illustrate the
geometric origins of this connection. Previous
algorithms have discovered these decompositions via an alternating minimization over the factors of a low-rank
matrix, but to do so, they have also needed to compute and store another
matrix, neither sparse nor low-rank, that is the size of their product. We develop a
stochastic, alternating least-squares
algorithm that operates on smaller blocks of this dense
matrix and scales as a result to much larger problems. We also show how to further accelerate this
algorithm with sparse optimizations and customized CUDA kernels. As one example, we use the
algorithm to analyze the sparse
matrix of synaptic weights for the recently published $\textit{Drosphilia}$ connectome. The nonzero elements of this
matrix, with 139,255 rows and columns, record the number of synapses between
cells in the nervous system of a female fruit fly. Despite a slowly decaying spectrum of singular values, this
matrix exhibits a latent low-rank structure that is predictive of
cell categories across multiple levels of specificity.
Read the full abstract ▾
All authors Lawrence K. Saul, Ningyuan Huang, Dennis Bollweg, Jeff Soules, Diana C. Halikias
Machine Learning (Statistics)
Random Hazard Forests
Hemant Ishwaran, Eileen M. Hsich, Udaya B. Kogalur et al. (+1 more)
Clinical data sources such as electronic health records and wearable sensors record patient status repeatedly over follow-up, often at irregular times and on different schedules for different measurements. These data create opportunities for continuously updated, individualized risk prediction. Existing approaches, however, often simplify the temporal structure for model fitting. We introduce Random Hazard Forests (RHF), a survival tree ensemble that estimates how a patient's hazard changes in continuous time as new measurements become available. The method formulates the estimation problem directly through a nonparametric hazard likelihood for predictable
covariate processes. An efficient working model guides tree construction, after which flexible time-varying hazards are estimated for each terminal node. Given any predictable
covariate path, each tree follows the path through its terminal nodes over time and assembles the corresponding node-level hazards into a trajectory. Averaging these trajectories across trees yields the pathwise hazard estimate. Because routing at each time uses only the
covariate state available immediately beforehand, the construction accommodates internal longitudinal
covariates without lookahead. Simulations and an intensive care application show that the forest accurately estimates changing risk under irregular and asynchronous
covariate updates.
Read the full abstract ▾
All authors Hemant Ishwaran, Eileen M. Hsich, Udaya B. Kogalur, Donald K.K. Lee
Statistics Applications · Cross-listed from cs.LG
Predicting Early Functional Decline from Longitudinal Laboratory and Vital Sign Trajectories: A Large-Scale Study Using the All of Us Research Program
Rashmita Kudamala, Aravind V. Kuruvikkattil, Lalitha Pranathi Pulavarthy et al. (+1 more)
Functional decline in older adults is typically recognized only after falls or observable gait impairment, closing the window for prevention. We investigated whether temporal trajectories of routine biomarkers, already recorded but rarely analyzed longitudinally, can identify patients in the pre-clinical phase of mobility decline. Using the All of Us Research Program (N = 297,861; 11.1% cases), we derived trajectory features (slope, variability, delta, mean) for twelve biomarkers over a three-year pre-index window. LightGBM models incorporating trajectories significantly outperformed static laboratory summaries (AUROC 0.797 vs. 0.755; DeLong p < 0.001; AUPRC 0.380 vs. 0.304). A 1:1 age- and sex-matched analysis confirmed an independent trajectory signal (AUROC 0.727 vs. demographics-only 0.680). A horizon analysis demonstrated sustained prediction 3-12 months before decline onset (AUROC 0.768-0.740). Because the model uses only measurements already ordered in routine care, it supports passive, zero-burden EHR integration for early detection of pre-clinical functional decline.
Read the full abstract ▾
All authors Rashmita Kudamala, Aravind V. Kuruvikkattil, Lalitha Pranathi Pulavarthy, Saptarshi Purkayastha
Statistics Theory · Cross-listed from cs.LG
Loss-Parameterized Fisher Width Along Learning Trajectories
Vu Khac Ky
Fisher width measures the Gaussian width of a probe set after deformation by the local Fisher geometry. We study its
evolution along learning trajectories and ask when training loss can serve as an effective coordinate for this quantity. We first derive an exact trace–shape factorization and a deterministic stability bound for fixed compact probes. In a population Gaussian-teacher logistic model, the teacher-aligned state is extremal on every loss level below $\log 2$: it has minimal parameter norm and maximizes both Fisher trace and Euclidean-ball Fisher width. We then show that population gradient flow asymptotically selects this branch, with explicit rates for the aligned and orthogonal coordinates. This yields, for $d\geq2$, \[ \frac{w_F(B_2^d;\theta(t))} {\sqrt{L(\theta(t))}} \longrightarrow \frac{\sqrt6}{\pi}\mathbb E[\chi_{d-1}]. \] Controlled full-Fisher experiments support the matched-loss branch and the population predictions. In a nonlinear MLP with a diagonal model-Fisher approximation, GD and SGD remain close at matched loss, whereas Adam follows a substantially displaced branch; the fixed probes tested retain highly similar temporal shapes. These results support a branchwise, rather than universal, loss parametrization of Fisher width.
Read the full abstract ▾
Statistical Methodology
Outlier Impact: Detection by Consequences
Daniel Ting, Ilya Gorbachev and Sammy Shen
We introduce a novel outlier detection method that utilizes a concrete measure of the statistical impact of an outlier rather than a vague notion of "unusualness". This allows practitioners to only flag points that materially affect the inferences made from the data while also ensuring these points are anomalous. This is of particular interest in experimentation, where outliers affect the validity of the causal inferences. In particular, we consider a potential outlier's impact on the mean and the False Positive Rate (FPR) of a hypothesis test.
Read the full abstract ▾
Machine Learning (Statistics)
Sparse Separable Factor Analysis in the Complex Domain with an Application to Local Field Potential Data
Ian Hultman, Kirtikanth Kalapatapu, Yassine Filali et al. (+2 more)
Complex-valued arrays arise in signal processing, where scientific interpretation depends on retaining amplitude and phase information. Existing covariance estimation methods either ignore the multiway organization of such data or rely on real-domain
embeddings that do not directly exploit their complex structure. We develop sparse separable factor analysis (SSFA), a latent factor model for complex-valued arrays with a separable covariance structure across modes. Each mode-specific covariance
matrix is modeled through a low-rank Hermitian factor structure and a diagonal residual covariance
matrix. To obtain interpretable estimates, we impose elementwise lasso penalties on the complex loading
matrices and estimate the SSFA parameters using a mode-wise parameter-expanded expectation-maximization procedure. The resulting loading updates admit closed-form complex soft-thresholding solutions, which shrink the modulus of each loading while preserving its phase. A separate balancing step resolves the scale nonidentifiability of the separable covariance structure. Simulation studies show that SSFA improves covariance estimation relative to vectorization-based methods, including complex principal component analysis. We apply SSFA to local field potential recordings from mice, where we compare separability structures induced by different groupings of
brain region, frequency, and time and perform model-based imputation of recordings missing because of electrode misplacement.
Read the full abstract ▾
All authors Ian Hultman, Kirtikanth Kalapatapu, Yassine Filali, Rainbo Hultman, Sanvesh Srivastava
Statistics Applications · Cross-listed from q-fin.RM
What Quantitative Risk Modellers Can Learn from Durkheim's Study of Suicide
Mahmood Alaghmandan
Emile Durkheim's Suicide: A Study in Sociology (1897) predates much of the statistical machinery that quantitative modellers now take for granted. Yet, working with sparse and imperfect observational data, Durkheim repeatedly arrives at practices that remain remarkably relevant to modern modelling. This paper revisits Suicide from the perspective of quantitative risk modelling, not for its substantive conclusions, but for the
reasoning by which Durkheim reached them. His approach illustrates how careful definition, common sense, logical investigation, scepticism toward convenient explanations, and close
attention to what the data can and cannot support—all of which must precede, and can often substitute for, statistical sophistication. The broader lesson is simple: good modelling begins not with technique, but with understanding the problem, interrogating the evidence, and
reasoning carefully about what it can actually tell us.
Read the full abstract ▾
Statistics Theory · Cross-listed from cs.LG
The geometry of AI validation: From structural blindness to reusable audits
Ricardo Fitas
AI systems increasingly search among candidate answers and deploy the highest-scoring one. Increasing search changes which errors matter, so a precise evaluation at one computation budget can leave another budget unresolved. We connect this information gap to the cost of closing it. For independent best-of-n search, aggregate reliability measurements identify deployment only through the directions they observe; we derive an exact ambiguity frontier when only small search widths are audited. Retaining candidate ranks and truth labels enables a constructive alternative: one audit can estimate reliability across all widths up to N. With known score percentiles, the minimax worst-coordinate mean squared error scales as (1 + log N)/T + N/M, capped at a constant, for expected budgets of T truth labels and M candidate observations. Matching lower bounds allow adaptive label acquisition, establishing that the distinct label and candidate costs are intrinsic to this experiment. An explicit design attains this order; a complementary record-based procedure supplies simultaneous guarantees without a known score distribution. Retrospective mathematical-
reasoning and code-generation analyses show why search-dependent validation matters. In held-out CodeRM pools, a shared audit reduces the 95th-percentile maximum error across 100 widths by 58% and 40% relative to uniform labeling. These results turn structural ambiguity into a quantitative prescription for reusable validation.
Read the full abstract ▾
Machine Learning (Statistics)
A Data-Driven Approach to State Construction in Markov Models
Linde Van Gestel, Marie-Anne Guerry and Evy Rombaut
A Markov chain is a widely used
stochastic process modelling random events over time. These models are built on subsets of the entire
dataset, referred to as states, which are considered to be homogeneous regarding transition probabilities. However, the creation of these states is often disregarded or based on prior assumption, potentially violating the homogeneity requirement and thus decreasing the validity and predictive power of the model. In order to fill this gap, this paper combines supervised feature selection with unsupervised learning techniques for data-driven state construction. Density-based clustering, spectral clustering, and Kohonen self-organizing maps are examined for their ability to identify latent groups without prior assumptions. The contribution of this study is twofold. First, the paper presents a methodological framework for state construction incorporating suitable unsupervised learning techniques, with appropriate measures both for classification performance and Markov model accuracy. Secondly, the framework is tested on an application, resulting in a comparative analysis showing that spectral clustering and Kohonen self-organizing maps are best at capturing inherent structure. These results serve as a cornerstone in providing theoretical and methodological guidance for improving state definition in applied Markov modelling.
Read the full abstract ▾
Statistics Applications
Stochastics of News Article Propagation: Joint Modeling of Counts and Reliability Composition in Event-Driven News Cascades
Harry Wang, Ruthesh Thavamani and Sriyan Madugula
We study the news cascade triggered by a single high-impact event as a pair of coupled
stochastic processes: the daily article count $N(t)$ and the reliable-vs-unreliable composition $\pi_{\mathrm{pos}}(t)$. Using the 2017 Las Vegas shooting as a primary case study (CC-News + NewsGuard, 91 days, 2,702 articles), we fit and compare four counting-process models for $N(t)$ and four distributional models for $\pi_{\mathrm{pos}}(t)$ on a common likelihood scale. For counts, a hybrid Inhomogeneous-Poisson plus Hawkes model wins decisively on AIC, BIC, and log-likelihood, as the IHP accounts for the initial exogenous shock, while the Hawkes accounts for the subsequent self-excitation. A profile-likelihood sweep of the Hawkes decay rate collapses the optimal Hawkes kernel to an AR(1) one-step lag $\lambda(t)=\mu+nN(t-1)$. This Markovian structure on the counts directly motivates the Part-B reliability models: we treat the reliability label as a first-order Markov chain on a binary state space and test homogeneous, regime-switching, and mean-field parameterizations against a bivariate Hawkes baseline model using a shared conditional binomial likelihood. The results show that the bivariate Hawkes model and the non-homogeneous Markov models are nearly indistinguishable on fit, as they exhaust the signal and start fitting to the noise. A brief replication on the 2017 Hurricane Harvey data shows that the same methodology applies but produces different results due to Hurricane Harvey's gradual endogenous build-up.
Read the full abstract ▾
Machine Learning (Statistics)
Gauss–Hermite Quadrature for Gaussian-Mixture Entropy with an Action-Space Hermite Surrogate
Jae Wan Shim
Gaussian distributions are used to model uncertainty in signals and states, and Gaussian mixtures are often used when the underlying distribution is
multimodal. Unlike a single Gaussian, a Gaussian mixture generally has no closed-form expression for differential
entropy and therefore requires numerical approximation. We propose a Gauss–Hermite quadrature method for evaluating Gaussian mixture differential
entropy. The quadrature order controls the numerical resolution of the approximation. The method is evaluated on one- and two-dimensional Gaussian mixture
benchmarks against Taylor approximations, analytic
entropy bounds, and numerical integration references. For repeated optimization over continuous actions, we also propose a Hermite
polynomial surrogate in action space. In a radar pointing
benchmark, its second-order form achieves substantially lower surrogate error and optimizer regret than a second-order Taylor surrogate based on local derivatives at the nominal action, while both methods use nine direct objective evaluations per replanning step. The Hermite surrogate also improves pointing performance in the tested
benchmark.
Read the full abstract ▾
Machine Learning (Statistics)
Spectral partitioning for $k$-block averaging kernels of finite Markov chains
Michael C.H. Choi and Youjia Wang
We develop spectral
algorithms for selecting state-space partitions that define averaging kernels for finite, ergodic and reversible Markov chains. For a partition $\mathcal O$, the Gibbs kernel $G_{\mathcal O}$ resamples within the current block from the stationary conditional distribution; when this update is tractable, composing or mixing it with a baseline kernel $P$ can accelerate convergence. We select $\mathcal O$ by rounding the bottom nonconstant eigenfunctions of $P^2$, or the algebraically smallest eigenfunctions of $P$ for additive mixtures, using weighted $k$-means. For $F(\mathcal O)=\|G_{\mathcal O}P-\Pi\|_{F,\pi}^2$, we derive exact trace and normalized-cut representations and show that $F$ equals the Pearson $\chi^2$-mutual information between the initial block label and the state after one transition, giving this
matrix objective a natural probabilistic interpretation. In the two-block case, a threshold sweep exactly solves the associated one-dimensional weighted two-means rounding problem. For general $k \geq 2$, weighted $k$-means rounds the bottom $(k-1)$-dimensional
embedding, after which candidates are rescored by $F$; the rounding distortion is a distance between subspaces that yields spectral approximation bounds. We extend the framework to additive mixtures, finite-horizon objectives, and discounted infinite-horizon objectives. In contrast to classical normalized spectral clustering, which uses top nonconstant modes to find low-flow persistent clusters, our method uses bottom modes to favor large normalized cross-block flow and rapid loss of block-label information. Experiments on a controlled-spectrum graph, a mean-field Ising model, and
Bayesian variable selection show notable per-iteration improvements in convergence and statistical estimation.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.HC
Event-Time Confounding Under Bursty Human Dynamics
Michael Iannelli and Alan Ai
Studies of digital behavior often align users at moments they choose, such as opening an AI assistant, clicking a recommendation, or visiting a product page, and interpret higher activity afterward as an event effect. We show how this creates an endogenous time zero: the event occurs during an ongoing task episode, so the aligned curve can trace episode continuation rather than a response to the event. In same-user, cross-surface web logs, AI, shopping, news, coding, and reference events are all preceded by broad activity increases that peak before time zero. Our strongest test uses known-null timestamps that cause nothing. Among the 5.8% of AI responses meeting strict pre-event activity and washout criteria, these timestamps show 3.42 times the post-event search activity of a within-user placebo, compared with 4.32 times for real events. The fraction of excess reproduced by the known null falls from 0.56 at detectably active moments to -0.04 at quiet moments, where the design detects none. We formalize this episode-selection bias, prove that a single-surface event window cannot separate it from a genuine effect without additional assumptions, and show in zero-effect simulations why user fixed effects and coarse activity matching can fail: the confound is within-user and time-varying. We provide a diagnostic protocol, public-data
benchmarks, and burstcheck, a lightweight audit tool. User-timed events may have real effects, but post-event volume does not identify them by default; studies should compare similar episodes with and without the event.
Read the full abstract ▾
Statistical Computation
Comprehensive Regression and Diagnostics for Non-Negative Data Using the BCSreg Package
Francisco F. Queiroz and Rodrigo M. R. de Medeiros
Continuous positive data characterized by high skewness and heavy tails frequently arise in applied statistics. In other applications, these characteristics are accompanied by a point mass at zero, resulting in a non-negative response with a mixed discrete-continuous distribution. Standard regression models often fail to capture these complex features adequately, requiring more flexible approaches. In this paper, we introduce the BCSreg package for R, which provides a comprehensive and unified computational framework for fitting Box-Cox symmetric and log-symmetric regression models for positive continuous data and their zero-adjusted extensions for mixed non-negative data. These broad classes of models accommodate varying degrees of skewness and tail-heaviness while allowing the parameters to be interpreted directly on the original scale of the data. Through a user-friendly multi-part formula interface, the BCSreg package allows practitioners to simultaneously specify regression structures for the scale parameter (which is proportional to the
quantiles of the response), the relative dispersion, and, when appropriate, the probability of zero occurrences. Furthermore, the package provides a complete suite of diagnostic tools specifically tailored to these classes of models, including randomized
quantile residuals, simulated envelopes, and influence diagnostics. The package's features and capabilities are illustrated through applications to real data.
Read the full abstract ▾
Statistical Methodology
Further Results on Controlling the False Discovery Rate in Two-Sided Gaussian Mean Testing
Deepra Ghosh and Sanat K. Sarkar
The recent work of Sarkar and Zhang (2025) introduced Positive Tail Dependence Under the Null (PTDN) and developed Generalized Shifted Benjamini-Hochberg (BH) procedures for two-sided Gaussian $z$- and $t$-testing under known covariance structures. This paper develops further consequences of that framework. First, we derive explicit dependence-adaptive lower and upper bounds for the FDR of the original BH procedure in terms of the conditional variance parameters $\tau_i=1-R_i^2$, where $R_i^2$ is the squared multiple correlation between the $i$th statistic and the remaining coordinates. These bounds recover the exact BH FDR under independence and provide finite-sample, covariance-specific information complementary to generic bounds. We also identify conditions under which the coordinate-specific calibration of shifted BH can provide a rejection advantage over the original BH procedure. Second, we consider the practically important setting in which the covariance
matrix is unknown but an independent Wishart
estimator is available. Using simultaneous lower confidence bounds for the $\tau_i$'s, we construct a confidence-bound shifted BH procedure and establish finite-sample FDR control. To our knowledge, this is the first shifted-BH-type procedure with a finite-sample guarantee for two-sided Gaussian mean testing under a completely unknown covariance
matrix estimated independently. Numerical studies illustrate the behavior of the covariance-adaptive bounds, the potential advantage of shifted BH over BH, and the performance of confidence-bound shifting under unknown covariance.
Read the full abstract ▾
Machine Learning (Statistics)
The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
Adam Noonan
Suppose we want a cutoff that 90% of a population falls below. We estimate it from a sample, and another sample would give a different cutoff and a different fraction below it. We ask how much that fraction varies when observations come in independent groups, such as pupils in classrooms or sentences in news articles. We prove that grouping multiplies its large-sample variance by $1+(m-1)\rho_I(p)$, where $m$ is the group size, $p$ is the target fraction, and $\rho_I(p)$ measures whether two members of a group fall on the same side of the cutoff. That correlation can differ from the correlation between the scores themselves, and it changes with the target. We give a direct proof, a counterexample to using score correlation, and an extension to unequal group sizes. A
dataset therefore does not have one effective sample size. How much information it contains depends on the question you ask. In our document experiment, the same 1,000 rows carried about 217 independent observations' worth of information at the median. At the 95th percentile, they carried about 621. Nothing about the
dataset changed. We asked it a different question. The number of rows is a property of the
dataset. The effective sample size belongs to the analysis.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
TRACE-C: Rank-Calibrated Relational Anomaly Detection for Multi-Stream Operational Telemetry
Matthew Faucher
Operational telemetry can be jointly anomalous while every individual stream stays inside its familiar range. TRACE-C is an auditable strictly-prior rank-calibrated detector for aligned multi-stream telemetry: same-regime rolling median/MAD residuals feed three window channels – a maximum normalized local sum, a Gaussian copula-form dependence contrast on robust-z residuals, and a worst standardized AR(1) innovation – whose channel ranks are Fisher-aggregated and ranked against earlier aggregates. We evaluate six Great Britain grid streams with a January-April 2019 fit, July-December 2019 development evidence, and a 2020 hold-out frozen before inspection. TRACE-C ranks Storm Atiyah first among 2019 test windows, but a disclosed channel ablation attributes that rank to the local channel, not the copula-form channel: copula-only ranks Atiyah 59th. The short 9 August frequency event is ranked far lower by the fused detector (143) than by the temporal channel alone (40), and reconstruction baselines rank it first. In 2020 no window is selected, which is consistent with record-rule saturation rather than an uneventful year; the highest-ranked frozen window was later interpreted as Storm Ellen. Three interpretive limits carry throughout. The resulting p-values are selection quantities, not event probabilities. The copula-form channel is not a literal copula density: the method applies no probability-integral or normal-score transform. Empirical rank counts are diagnostics, not coverage or false-discovery proofs. Every table and figure in this paper is generated from committed machine-readable reports.
Read the full abstract ▾
Statistics Applications
Matching Urban Flood Sensor Placement to Monitoring Objectives Using Bayesian Optimal Experimental Design
Chen Cheng, Vinh Ngoc Tran, Jiayuan Dong et al. (+5 more)
Flood-monitoring sensors are often placed according to coverage, access, or expected inundation. However, the value of a measurement depends on the prediction or decision it is intended to inform. Using tRIBS-Urban simulations and a neural-network surrogate of the August 2014 metropolitan Detroit flood, we examine how this learning target changes single-sensor placement. Across 2,576 candidate locations, we compare parameter-oriented optimal experimental design (PO-OED), which values expected information gain (EIG) about model parameters, with goal-oriented optimal experimental design (GO-OED), which values EIG about specified flood predictions. We also examine how parameter EIG evolves during the event, and illustrate that parameter learning translates unevenly into reductions in predictive uncertainty across locations and lead times. Under GO-OED, point-depth targets favor nearby locations, whereas regional-average and regional maximum-depth targets can favor nonlocal locations. Weighted multi-point objectives retain similar broad spatial patterns, although their computed max-EIG locations differ. Public geospatial data further provide illustrative feasibility and contextual classifications for deployment screening. These results show how monitoring objectives shape sensor placement in optimal experimental design, and motivates an objective-first workflow that defines the intended prediction and priorities, applies field-verified restrictions, and ranks locations by EIG.
Read the full abstract ▾
All authors Chen Cheng, Vinh Ngoc Tran, Jiayuan Dong, Sarah Whitaker, Shannon Bergt, John Ziker, Valeriy Y. Ivanov, Xun Huan
Statistics Applications
Statistical and Deep Learning Approaches for Predicting Degradation of Polymeric Materials in Photovoltaics
Yili Hong and Xiaohong Gu
Polymeric materials are widely used in photovoltaic (PV) systems, making it essential to understand their service life to ensure reliable PV performance. The primary failure mechanism of polymeric materials in PV systems is photodegradation caused by ultraviolet (UV) radiation. Degradation modeling provides a framework for predicting service life, with a key step being the development of predictive models for degradation paths. This paper presents statistical and machine learning approaches for predicting the outdoor degradation of polymeric components in PV systems. We describe the study design and data collection process for developing predictive models based on indoor laboratory testing data, which are then extended to outdoor field conditions with time-varying environmental variables, with prediction uncertainty quantified through simulation. Deep learning (DL) methods are also explored, and results are compared across modeling approaches. The parametric statistical model demonstrates good fit and predictive performance across
datasets and shows greater robustness by incorporating physical and chemical knowledge. The DL model provides flexibility in capturing complex
covariate relationships and often yields accurate predictions, though it is less robust across
datasets. The paper concludes with remarks on key findings and their implications for PV reliability.
Read the full abstract ▾
Statistics Applications
A Synthetic Benchmark Dataset with Endogenous Marketing Spend for Validating Marketing Mix Models
Niklas Heusch
Marketing Mix Models (MMMs) estimate the incremental sales effect of advertising from observational time series, yet they are rarely validated against ground truth, because ground truth is unobservable in real data. Synthetic data closes that gap in principle, but existing generators produce marketing spend exogenously - omitting the central difficulty of the estimation problem, since real budgets are planned around promotional calendars, seasons, and recent performance. This paper presents a parameterized generator, and a fixed reference instance, of a synthetic weekly retail
dataset (156 weeks, three media channels) in which spend arises from four documented coordination mechanisms - quarterly budget feedback, anticipatory spending ahead of a promotional calendar, scheduled TV bursts, and algorithmic performance chasing - on a demand baseline with seasonal, quality, price, and unobserved sentiment components. Spend translates into incremental sales through two transformations, carryover and diminishing returns, instantiated here as
geometric adstock and logistic saturation with known parameters; the true causal decomposition of every week's sales is recorded alongside the error-contaminated variables a practitioner would observe. Every mechanism is a parameter that can be varied or switched off, and a companion procedure simulates go-dark geo-experiments with exact treatment effects. The seeded generator and reference instance are publicly released with notebooks that reproduce every number in this paper.
Read the full abstract ▾
Statistics Applications
Structural Estimation of Marketing Mix Model Parameters from Geo-Experiments
Niklas Heusch
Marketing Mix Models (MMMs) are widely used for marketing measurement and budget allocation, but face fundamental identification challenges: due to endogenous marketing spend decisions, MMM estimation on observational time-series data cannot recover the true causal effects of marketing. On the other hand, geo-experiments provide causal identification through randomization, but it is not clear how to use them efficiently to calibrate marketing mix models. We propose a novel structural estimation approach that recovers the complete set of MMM parameters - adstock decay ($\alpha$), saturation ($\lambda$), and effectiveness ($\beta$) - directly from geo-experimental time-series. By differencing outcomes between treatment and control regions, our method eliminates observed and unobserved confounding factors while preserving the temporal variation that identifies each parameter. We demonstrate on synthetic data that this approach recovers the true ROAS and the response curve over the range of spending covered by the experiments, together with credible estimates of the underlying parameters. Our framework enables efficient pooling across multiple experiments and provides a principled foundation for MMM calibration that fully utilizes the information experiments contain.
Read the full abstract ▾
Statistical Methodology
LABS: Extending the scope of binary segmentation via a look-ahead device
Piotr Fryzlewicz
Binary segmentation is widely used for multiple change-point detection because it is fast, simple to describe, and simple to implement. Its validity rests on the requirement that, at each recursive stage, the procedure identifies one of the true change-points when several are present in the current interval. This holds for detecting changes in mean using the CUSUM statistic, but fails in some other settings, in particular in slope change detection for continuous piecewise-linear signals. We propose Look-Ahead Binary Segmentation (LABS), a modification in which the change-points returned by the two child recursions define a narrower interval on which the parent estimate is re-evaluated. LABS inherits the computational speed of standard binary segmentation, but achieves the near-optimal consistency rate of $O\{(n\log n)^{1/2}\}$ in the slope-change signal setting when the LABS model is chosen via either thresholding or a Schwarz-like information criterion. Simulations show that LABS is fast and achieves state-of-the-art performance.
Read the full abstract ▾