Peer Revenue Fully funded

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Friday 16 January

Statistics Applications

A Note on Harmonic Underspecification in Log-Normal Trigonometric Regression

Analysis of biological rhythm data often involves performing least squares trigonometric regression, which models the oscillations of a response over time as a sum of sinusoidal components. When the response is not normally distributed, an investigator will either transform the response before applying least squares trigonometric regression or extend trigonometric regression to a generalized linear model (GLM) framework. In this note, we compare these two approaches when the number of oscillation harmonics is underspecified. We assume data are sampled under an equispaced experimental design and that a log link function would be appropriate for a GLM. We show that when the response follows a generalized gamma distribution, least squares trigonometric regression with a log-transformed response, or log-normal trigonometric regression, produces unbiased parameter estimates for the oscillation harmonics, even when the number of oscillation harmonics is underspecified. In contrast, GLMs require correct specification to produce unbiased parameter estimates. We apply both methods to cortisol level data and find that only log-normal trigonometric regression produces parameter estimates that are invariant to the number of specified oscillation harmonics. Additionally, when a sufficiently large number of oscillation harmonics is specified, both methods produce identical parameter estimates for the oscillation harmonics.
Read the full abstract

Thursday 15 January

Statistics Applications · Cross-listed from cond-mat.soft

Is gelation a singularity or a flow induced instability?

Gelation in the Smoluchowski coagulation equation is commonly interpreted as a finite-time singularity marked by mass loss or moment divergence. We instead characterize gelation as a loss of dynamical stability of the Smoluchowski flow, quantified through the time-dependent spectrum of the Jacobian along the evolving aggregation dynamics. Studying homogeneous kernels $K(i,j)=(ij)^{\alpha}$ together with the classical Smoluchowski, we show that gelation is consistently preceded by the appearance of positive real eigenvalues, indicating a loss of local dynamical stability. While non-gelling kernels exhibit only transient finite-size effects, gelling kernels display persistent spectral destabilization associated with macroscopic gel formation. Our results identify gelation as a genuine dynamical instability of the Smoluchowski flow.
Read the full abstract

Statistics Applications

Climate Vulnerability and Community Health: Identifying Greensboro Neighborhoods at Intersectional Risk

This study develops an integrated, intersectional climate vulnerability assessment for Greensboro, North Carolina, a midsize city in the rapidly changing American Southeast. Moving beyond generalized mapping, we combine demographic, socioeconomic, health, and environmental data at the census tract level to identify neighborhoods where flood exposure, chronic health burdens, and social disadvantage spatially converge. Through k-means and hierarchical clustering, we identify four distinct neighborhood typologies, including a critically high-risk cluster characterized by high flood exposure, extreme poverty, poor respiratory health, and aging housing. The findings demonstrate that climate-related risks are not randomly distributed but systematically cluster in historically marginalized communities, revealing a clear environmental justice disparity. This place-based typology approach provides a targeted framework for policymakers to design integrated interventions that bridge flood management, public health, housing, and social services to build equitable urban resilience
Read the full abstract

Statistical Methodology · Cross-listed from cs.LG

Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning

Numerous offline and model-based reinforcement learning systems incorporate world models to emulate the inherent environments. A world model is particularly important in scenarios where direct interactions with the real environment is costly, dangerous, or impractical. The efficacy and interpretability of such world models are notably contingent upon the quality of the underlying training data. In this context, we introduce Action Shapley as an agnostic metric for the judicious and unbiased selection of training data. To facilitate the computation of Action Shapley, we present a randomized dynamic algorithm specifically designed to mitigate the exponential complexity inherent in traditional Shapley value computations. Through empirical validation across five data-constrained real-world case studies, the algorithm demonstrates a computational efficiency improvement exceeding 80% in comparison to conventional exponential time computations. Furthermore, our Action Shapley-based training data selection policy consistently outperforms ad-hoc training data selection.
Read the full abstract

Statistical Methodology

On the use of cross-fitting in causal machine learning with correlated units

In causal machine learning, the fitting and evaluation of nuisance models are often performed on separate partitions, or folds, of the observed data. This technique, called cross-fitting, eliminates bias introduced by the use of black-box predictive algorithms. When study units may be correlated, such as in spatial, clustered, or time-series data, investigators often design bespoke forms of cross-fitting to minimize correlation between folds. We prove that, perhaps contrary to popular belief, this is typically unnecessary: performing cross fitting as if study units were independent still eliminates key bias terms even when units may be correlated. In simulation experiments with various correlation structures, we show that causal machine learning estimators achieve the same or improved bias and precision under cross-fitting that ignores correlation compared to techniques striving to eliminate correlation between folds.
Read the full abstract

Statistics Applications · Cross-listed from astro-ph.IM

Optimal and Unbiased Fluxes from Up-the-Ramp Detectors under Variable Illumination

Near-infrared (NIR) detectors – which use non-destructive readouts to measure time-series counts-per-pixel – play a crucial role in modern astrophysics. Standard NIR flux extraction techniques were developed for space-based observations and assume that source fluxes are constant over an observation. However, ground-based telescopes often see short-timescale atmospheric variations that can dramatically change the number of photons arriving at a pixel. This work presents a new statistical model that shares information between neighboring spectral pixels to characterize time-variable observations and extract unbiased fluxes with optimal uncertainties. We generate realistic synthetic data using a variety of flux and amplitude-of-time-variability conditions to confirm that our model recovers unbiased and optimal estimates of both the true flux and the time-variable signal. We find that the time-variable model should be favored over a constant-flux model when the observed count rates change by more than 3.5%. Ignoring time variability in the data can result in flux-dependent, unknown-sign biases that are as large as ~120% of the flux uncertainty. Using real APOGEE spectra, we find empirical evidence for approximately wavelength-independent, time-dependent variations in count rates with amplitudes much greater than the 3.5% threshold. Our model can robustly measure and remove the time-dependence in real data, improving the quality of data-model comparison. We show several examples where the observed time-dependence quantitatively agrees with independent measurements of observing conditions, such as variable cloud cover and seeing.
Read the full abstract

All authors Bowen Li, Kevin A. McKinnon, Andrew K. Saydjari, Conor Sayres, Gwendolyn M. Eadie, Andrew R. Casey, Jon A. Holtzman, Timothy D. Brandt, Jose G. Fernandez-Trincado

Statistical Methodology

Locally sparse varying coefficient mixed model with application to longitudinal microbiome differential abundance

Differential abundance (DA) analysis in microbiome studies has recently been used to uncover a plethora of associations between microbial composition and various health conditions. While current approaches to DA typically apply only to cross-sectional data, many studies feature a longitudinal design to better understand the underlying microbial dynamics. To study DA in longitudinal microbial studies, we introduce a novel varying coefficient mixed-effects model with local sparsity. The proposed method can identify time intervals of significant group differences while accounting for temporal dependence. Specifically, we exploit a penalized kernel smoothing approach for parameter estimation and include a random effect to account for serial correlation. In particular, our method operates effectively regardless of whether sampling times are shared across subjects, accommodating irregular sampling and missing observations. Simulation studies demonstrate the necessity of modeling dependence for precise estimation and support recovery. The application of our method to a longitudinal study of mice oral microbiome during cancer development revealed significant scientific insights that were otherwise not discernible through cross-sectional analyses. An R implementation is available at https://github.com/fontaine618/LSVCMM.
Read the full abstract

All authors Simon Fontaine, Nisha J. D'Silva, Marcell Costa de Medeiros, Grace Y. Chen, Ji Zhu, Gen Li

Statistics Applications · Cross-listed from econ.GN

Beyond Unidimensionality: General Factors and Residual Heterogeneity in Performance Evaluation

How do evaluation systems compress multidimensional performance information into summary ratings? Using expert assessments of 9,669 professional soccer players on 28 attributes, we characterize the dimensional structure of evaluation outputs. The first principal component explains 40.6% of attribute variance, indicating a strong general factor, but formal noise discrimination procedures retain four components and bootstrap resampling confirms that this structure is highly stable. Internal consistency is high without evidence of redundancy. In out of sample prediction of expert overall ratings, a comprehensive model using the full attribute set substantially outperforms a single-factor summary (cross-validated R squared = 0.814). Overall, performance evaluations exhibit moderate information compression; they combine shared variance with stable residual dimensions that are economically meaningful for evaluation outcomes, with direct implications for the design of measurement systems.
Read the full abstract

Statistics Theory · Cross-listed from cs.LG

High-accuracy and dimension-free sampling with diffusions

Diffusion models have shown remarkable empirical success in sampling from rich multi-modal distributions. Their inference relies on numerically solving a certain differential equation. This differential equation cannot be solved in closed form, and its resolution via discretization typically requires many small iterations to produce high-quality samples. More precisely, prior works have shown that the iteration complexity of discretization methods for diffusion models scales polynomially in the ambient dimension and the inverse accuracy $1/\varepsilon$. In this work, we propose a new solver for diffusion models relying on a subtle interplay between low-degree approximation and the collocation method (Lee, Song, Vempala 2018), and we prove that its iteration complexity scales polylogarithmically in $1/\varepsilon$, yielding the first “high-accuracy” guarantee for a diffusion-based sampler that only uses (approximate) access to the scores of the data distribution. In addition, our bound does not depend explicitly on the ambient dimension; more precisely, the dimension affects the complexity of our solver through the effective radius of the support of the target distribution only.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

On the origin of neural scaling laws: from random graphs to natural language

Scaling laws have played a major role in the modern AI revolution, providing practitioners predictive power over how the model performance will improve with increasing data, compute, and number of model parameters. This has spurred an intense interest in the origin of neural scaling laws, with a common suggestion being that they arise from power law structure already present in the data. In this paper we study scaling laws for transformers trained to predict random walks (bigrams) on graphs with tunable complexity. We demonstrate that this simplified setting already gives rise to neural scaling laws even in the absence of power law structure in the data correlations. We further consider dialing down the complexity of natural language systematically, by training on sequences sampled from increasingly simplified generative language models, from 4,2,1-layer transformer language models down to language bigrams, revealing a monotonic evolution of the scaling exponents. Our results also include scaling laws obtained from training on random walks on random graphs drawn from Erdös-Renyi and scale-free Barabási-Albert ensembles. Finally, we revisit conventional scaling laws for language modeling, demonstrating that several essential results can be reproduced using 2 layer transformers with context length of 50, provide a critical analysis of various fits used in prior literature, demonstrate an alternative method for obtaining compute optimal curves as compared with current practice in published literature, and provide preliminary evidence that maximal update parameterization may be more parameter efficient than standard parameterization.
Read the full abstract

Statistics Theory · Cross-listed from physics.soc-ph

Transforming Crises into Opportunities: From Chaos to Urban Antifragility

Urban crises - floods, pandemics, economic shocks, and conflicts - function as accelerators of urban change, exposing structural vulnerabilities while creating windows for reinvention. Building on a prior theoretical contribution that identified fifteen principles of urban antifragility, this paper tests and operationalizes the framework through an empirical assessment of 26 cities selected for their post-crisis adaptation trajectories. Using a tailored diagnostic methodology, we benchmark cities' Stress Response Strategies (SRS) and then evaluate Urban Development Trajectories (UDT) across four weighted dimensions, positioning each case along a fragility-robustness-resilience-antifragility continuum and applying a balanced-threshold rule to confirm antifragile status. Results show that "resilience enhanced by innovation and technology" is the most effective response typology (86.9/100), and that six cities meet the antifragile trajectory criteria. By mapping best practices to activated principles and analysing co-activations, the study identifies a robust "hard core" of principles - Sustainable Resilience (O), Strategic Diversity (F), Proactive Innovation (I), and Active Prevention (N) - supplemented by operational enablers (e.g., anticipation, mobilization, shock absorption). The paper concludes by proposing an evidence-based, SDG-aligned operational model that links high-impact principle pairings to measurable indicators, offering a practical roadmap for cities seeking to convert crises into sustained transformation. Keywords: Post-crisis strategies, Urban antifragility, Sustainable cities and communities, Disaster resilience and urban regeneration, Risk governance and Black Swan adaptation.
Read the full abstract

Statistical Methodology

Adjusted Similarity Measures and a Violation of Expectations

Adjusted similarity measures, such as Cohen's kappa for inter-rater reliability and the adjusted Rand index used to compare clustering algorithms, are a vital tool for comparing discrete labellings. These measures are intended to have the property of 0 expectation under a null distribution and maximum value 1 under maximal similarity to aid in interpretation. Measures are frequently adjusted with respect to the permutation distribution for historic and analytic reasons. There is currently renewed interest in considering other null models more appropriate for context, such as clustering ensembles permitting a random number of identified clusters. The purpose of this work is two – fold: (1) to generalize the study of the adjustment operator to general null models and to a more general procedure which includes statistical standardization as a special case and (2) to identify sufficient conditions for the adjustment operator to produce the intended properties, where sufficient conditions are related to whether and how observed data are incorporated into null distributions. We demonstrate how violations of the sufficient conditions may lead to substantial breakdown, such as by producing a non-positive measure under traditional adjustment rather than one with mean 0, or by producing a measure which is deterministically 0 under statistical standardization.
Read the full abstract

Machine Learning (Statistics)

Classification Imbalance as Transfer Learning

Classification imbalance arises when one class is much rarer than the other. We frame this setting as transfer learning under label (prior) shift between an imbalanced source distribution induced by the observed data and a balanced target distribution under which performance is evaluated. Within this framework, we study a family of oversampling procedures that augment the training data by generating synthetic samples from an estimated minority-class distribution to roughly balance the classes, among which the celebrated SMOTE algorithm is a canonical example. We show that the excess risk decomposes into the rate achievable under balanced training (as if the data had been drawn from the balanced target distribution) and an additional term, the cost of transfer, which quantifies the discrepancy between the estimated and true minority-class distributions. In particular, we show that the cost of transfer for SMOTE dominates that of bootstrapping (random oversampling) in moderately high dimensions, suggesting that we should expect bootstrapping to have better performance than SMOTE in general. We corroborate these findings with experimental evidence. More broadly, our results provide guidance for choosing among augmentation strategies for imbalanced classification.
Read the full abstract

Machine Learning (Statistics)

Parametric RDT approach to computational gap of symmetric binary perceptron

We study potential presence of statistical-computational gaps (SCG) in symmetric binary perceptrons (SBP) via a parametric utilization of fully lifted random duality theory (fl-RDT) [96]. A structural change from decreasingly to arbitrarily ordered $c$-sequence (a key fl-RDT parametric component) is observed on the second lifting level and associated with satisfiability ($\alpha_c$) – algorithmic ($\alpha_a$) constraints density threshold change thereby suggesting a potential existence of a nonzero computational gap $SCG=\alpha_c-\alpha_a$. The second level estimate is shown to match the theoretical $\alpha_c$ whereas the $r\rightarrow \infty$ level one is proposed to correspond to $\alpha_a$. For example, for the canonical SBP ($\kappa=1$ margin) we obtain $\alpha_c\approx 1.8159$ on the second and $\alpha_a\approx 1.6021$ (with converging tendency towards $\sim 1.59$ range) on the seventh level. Our propositions remarkably well concur with recent literature: (i) in [20] local entropy replica approach predicts $\alpha_{LE}\approx 1.58$ as the onset of clustering defragmentation (presumed driving force behind locally improving algorithms failures); (ii) in $\alpha\rightarrow 0$ regime we obtain on the third lifting level $\kappa\approx 1.2385\sqrt{\frac{\alpha_a}{-\log\left ( \alpha_a \right ) }}$ which qualitatively matches overlap gap property (OGP) based predictions of [43] and identically matches local entropy based predictions of [24]; (iii) $c$-sequence ordering change phenomenology mirrors the one observed in asymmetric binary perceptron (ABP) in [98] and the negative Hopfield model in [100]; and (iv) as in [98,100], we here design a CLuP based algorithm whose practical performance closely matches proposed theoretical predictions.
Read the full abstract

Statistics Theory

Differentially Private Inference for Longitudinal Linear Regression

Differential Privacy (DP) provides a rigorous framework for releasing statistics while protecting individual information present in a dataset. Although substantial progress has been made on differentially private linear regression, existing methods almost exclusively address the item-level DP setting, where each user contributes a single observation. Many scientific and economic applications instead involve longitudinal or panel data, in which each user contributes multiple dependent observations. In these settings, item-level DP offers inadequate protection, and user-level DP - shielding an individual's entire trajectory - is the appropriate privacy notion. We develop a comprehensive framework for estimation and inference in longitudinal linear regression under user-level DP. We propose a user-level private regression estimator based on aggregating local regressions, and we establish finite-sample guarantees and asymptotic normality under short-range dependence. For inference, we develop a privatized, bias-corrected covariance estimator that is automatically heteroskedasticity- and autocorrelation-consistent. These results provide the first unified framework for practical user-level DP estimation and inference in longitudinal linear regression under dependence, with strong theoretical guarantees and promising empirical performance.
Read the full abstract

Statistical Methodology

Fair Regression under Demographic Parity: A Unified Framework

We propose a unified framework for fair regression tasks formulated as risk minimization problems subject to a demographic parity constraint. Unlike many existing approaches that are limited to specific loss functions or rely on challenging non-convex optimization, our framework is applicable to a broad spectrum of regression tasks. Examples include linear regression with squared loss, binary classification with cross-entropy loss, quantile regression with pinball loss, and robust regression with Huber loss. We derive a novel characterization of the fair risk minimizer, which yields a computationally efficient estimation procedure for general loss functions. Theoretically, we establish the asymptotic consistency of the proposed estimator and derive its convergence rates under mild assumptions. We illustrate the method's versatility through detailed discussions of several common loss functions. Numerical results demonstrate that our approach effectively minimizes risk while satisfying fairness constraints across various regression settings.
Read the full abstract

All authors Yongzhen Feng, Weiwei Wang, Raymond K. W. Wong, Xianyang Zhang

Statistical Methodology

A Bayesian Discrete Framework for Enhancing Decision-Making Processes in Clinical Trial Designs and Evaluations

This study examines the application of Bayesian approach in the context of clinical trials, emphasizing their increasing importance in contemporary biomedical research. While conventional frequentist approach provides a foundational basis for analysis, it often lacks the flexibility to integrate prior knowledge, which can constrain its effectiveness in adaptive settings. In contrast, Bayesian methods enable continual refinement of statistical inferences through the assimilation of accumulating evidence, thereby supporting more informed decision-making and improving the reliability of trial findings. This paper also considers persistent challenges in clinical investigations, including replication difficulties and the misinterpretation of statistical results, suggesting that Bayesian strategies may offer a path toward enhanced analytical robustness. Moreover, discrete probability models, specifically the Binomial, Poisson, and Negative Binomial distributions are explored for their suitability in modeling clinical endpoints, particularly in trials involving binary responses or data with overdispersion. The discussion further incorporates Bayesian networks and Bayesian estimation techniques, with a comparative evaluation against maximum likelihood estimation to elucidate differences in inferential behavior and practical implementation.
Read the full abstract

All authors Paramahansa Pramanik, Arnab Kumar Maity, Anjan Mandal, Haley Kate Robinson

Statistical Methodology

From aggressive to conservative early stopping in Bayesian group sequential designs

Group sequential designs (GSDs) are widely used in confirmatory trials to allow interim monitoring while preserving control of the type I error rate. In the frequentist framework, O'Brien-Fleming-type stopping boundaries dominate practice because they impose highly conservative early stopping while allowing more liberal decisions as information accumulates. Bayesian GSDs, in contrast, are most often implemented using fixed posterior probability thresholds applied uniformly at all analyses. While such designs can be calibrated to control the overall type I error rate, they do not penalise early analyses and can therefore lead to substantially more aggressive early stopping. Such behaviour can risk premature conclusions and inflation of treatment effect estimates, raising concerns for confirmatory trials. We introduce two practically implementable refinements that restore conservative early stopping in Bayesian GSDs. The first introduces a two-phase structure for posterior probability thresholds, applying more stringent criteria in the early phase of the trial and relaxing them later to preserve power. The second replaces posterior probability monitoring at interim looks with predictive probability criteria, which naturally account for uncertainty in future data and therefore suppress premature stopping. Both strategies require only one additional tuning parameter and can be efficiently calibrated. In the HYPRESS setting, both approaches achieve higher power than the conventional Bayesian design while producing alpha-spending profiles closely aligned with O'Brien-Fleming-type behaviour at early looks. These refinements provide a principled and tractable way to align Bayesian GSDs with accepted frequentist practice and regulatory expectations, supporting their robust application in confirmatory trials.
Read the full abstract

All authors Zhangyi He, Feng Yu, Suzie Cro, Laurent Billot

Machine Learning (Statistics) · Cross-listed from econ.EM

causalfe: Causal Forests with Fixed Effects in Python

The causalfe package provides a Python implementation of Causal Forests with Fixed Effects (CFFE) for estimating heterogeneous treatment effects in panel data settings. Standard causal forest methods struggle with panel data because unit and time fixed effects induce spurious heterogeneity in treatment effect estimates. The CFFE approach addresses this by performing node-level residualization during tree construction, removing fixed effects within each candidate split rather than globally. This paper describes the methodology, documents the software interface, and demonstrates the package through simulation studies that validate the estimator's performance under various data generating processes.
Read the full abstract

Statistical Methodology

A Propagation Framework for Network Regression

We introduce a unified and computationally efficient framework for regression on network data, addressing limitations of existing models that require specialized estimation procedures or impose restrictive decay assumptions. Our Network Propagation Regression (NPR) models outcomes as functions of covariates propagated through network connections, capturing both direct and indirect effects. NPR is estimable via ordinary least squares for continuous outcomes and standard routines for binary, categorical, and time-to-event data, all within a single interpretable framework. We establish consistency and asymptotic normality under weak conditions and develop valid hypothesis tests for the order of network influence. Simulation studies demonstrate that NPR consistently outperforms established approaches, such as the linear-in-means model and regression with network cohesion, especially under model misspecification. An application to social media sentiment analysis highlights the practical utility and robustness of NPR in real-world settings.
Read the full abstract

Machine Learning (Statistics)

Coarsening Causal DAG Models

Directed acyclic graphical (DAG) models are a powerful tool for representing causal relationships among jointly distributed random variables, especially concerning data from across different experimental settings. However, it is not always practical or desirable to estimate a causal model at the granularity of given features in a particular dataset. There is a growing body of research on causal abstraction to address such problems. We contribute to this line of research by (i) providing novel graphical identifiability results for practically-relevant interventional settings, (ii) proposing an efficient, provably consistent algorithm for directly learning abstract causal graphs from interventional data with unknown intervention targets, and (iii) uncovering theoretical insights about the lattice structure of the underlying search space, with connections to the field of causal discovery more generally. As proof of concept, we apply our algorithm on synthetic and real datasets with known ground truths, including measurements from a controlled physical system with interacting light intensity and polarization.
Read the full abstract

Statistics Theory · Cross-listed from cs.IT

On the suboptimality of linear codes for binary distributed hypothesis testing

We study a binary distributed hypothesis testing problem where two agents observe correlated binary vectors and communicate compressed information at the same rate to a central decision maker. In particular, we study linear compression schemes and show that simple truncation is the best linear scheme in two cases: (1) testing opposite signs of the same magnitude of correlation, and (2) testing for or against independence. We conjecture, supported by numerical evidence, that truncation is the best linear code for testing any correlations of opposite signs. Further, for testing against independence, we also compute classical random coding exponents and show that truncation, and consequently any linear code, is strictly suboptimal.
Read the full abstract

Statistical Methodology · Cross-listed from econ.EM

Semiparametric inference for inequality measures under nonignorable nonresponse using callback data

This paper develops semiparametric methods for estimation and inference of widely used inequality measures when survey data are subject to nonignorable nonresponse, a challenging setting in which response probabilities depend on the unobserved outcomes. Such nonresponse mechanisms are common in household surveys and invalidate standard inference procedures due to selection bias and lack of population representativeness. We address this problem by exploiting callback data from repeated contact attempts and adopting a semiparametric model that leaves the outcome distribution unspecified. We construct semiparametric full-likelihood estimators for the underlying distribution and the associated inequality measures, and establish their large-sample properties for a broad class of functionals, including quantiles, the Theil index, and the Gini index. Explicit asymptotic variance expressions are derived, enabling valid Wald-type inference under nonignorable nonresponse. To facilitate implementation, we propose a stable and computationally convenient expectation-maximization algorithm, whose steps either admit closed-form expressions or reduce to fitting a standard logistic regression model. Simulation studies demonstrate that the proposed procedures effectively correct nonresponse bias and achieve near-benchmark efficiency. An application to Consumer Expenditure Survey data illustrates the practical gains from incorporating callback information when making inference on inequality measures.
Read the full abstract

All authors Xinyu Wang, Chunlin Wang, Tao Yu, Pengfei Li

Machine Learning (Statistics)

CROCS: A Two-Stage Clustering Framework for Behaviour-Centric Consumer Segmentation with Smart Meter Data

With grid operators confronting rising uncertainty from renewable integration and a broader push toward electrification, Demand-Side Management (DSM) – particularly Demand Response (DR) – has attracted significant attention as a cost-effective mechanism for balancing modern electricity systems. Unprecedented volumes of consumption data from a continuing global deployment of smart meters enable consumer segmentation based on real usage behaviours, promising to inform the design of more effective DSM and DR programs. However, existing clustering-based segmentation methods insufficiently reflect the behavioural diversity of consumers, often relying on rigid temporal alignment, and faltering in the presence of anomalies, missing data, or large-scale deployments. To address these challenges, we propose a novel two-stage clustering framework – Clustered Representations Optimising Consumer Segmentation (CROCS). In the first stage, each consumer's daily load profiles are clustered independently to form a Representative Load Set (RLS), providing a compact summary of their typical diurnal consumption behaviours. In the second stage, consumers are clustered using the Weighted Sum of Minimum Distances (WSMD), a novel set-to-set measure that compares RLSs by accounting for both the prevalence and similarity of those behaviours. Finally, community detection on the WSMD-induced graph reveals higher-order prototypes that embody the shared diurnal behaviours defining consumer groups, enhancing the interpretability of the resulting clusters. Extensive experiments on both synthetic and real Australian smart meter datasets demonstrate that CROCS captures intra-consumer variability, uncovers both synchronous and asynchronous behavioural similarities, and remains robust to anomalies and missing data, while scaling efficiently through natural parallelisation. These results...
Read the full abstract

All authors Luke W. Yerbury, Ricardo J. G. B. Campello, G. C. Livingston Jr, Mark Goldsworthy, Lachlan O'Neil

Statistical Computation

Mesh Denoising

In this paper, we study four mesh denoising methods: linear filtering, a heat diffusion method, Sobolev regularization, and, to a lesser extent, a barycentric approach based on the Sinkhorn algorithm. We illustrate that, for a simple image denoising task, a naive choice of a Gibbs kernel can lead to unsatisfactory results. We demonstrate that while Sobolev regularization is the fastest method in our implementation, it produces slightly less faithful denoised meshes than the best results obtained with iterative filtering or heat diffusion. We empirically show that, for the large mesh considered, the heat diffusion method is slower and not more effective than filtering, whereas on a small mesh an appropriate choice of diffusion parameters can improve the quality. Finally, we observe that all three mesh-based methods perform markedly better on the large mesh than on the small one.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from math.OC

High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks

We study the high-dimensional training dynamics of a shallow neural network with quadratic activation in a teacher-student setup. We focus on the extensive-width regime, where the teacher and student network widths scale proportionally with the input dimension, and the sample size grows quadratically. This scaling aims to describe overparameterized neural networks in which feature learning still plays a central role. In the high-dimensional limit, we derive a dynamical characterization of the gradient flow, in the spirit of dynamical mean-field theory (DMFT). Under l2-regularization, we analyze these equations at long times and characterize the performance and spectral properties of the resulting estimator. This result provides a quantitative understanding of the effect of overparameterization on learning and generalization, and reveals a double descent phenomenon in the presence of label noise, where generalization improves beyond interpolation. In the small regularization limit, we obtain an exact expression for the perfect recovery threshold as a function of the network widths, providing a precise characterization of how overparameterization influences recovery.
Read the full abstract

Statistics Applications

MitoFREQ: A Novel Approach for Mitogenome Frequency Estimation from Top-level Haplogroups and Single Nucleotide Variants

Lineage marker population frequencies can serve as one way to express evidential value in forensic genetics. However, for high-quality whole mitochondrial DNA genome sequences (mitogenomes), population data remain limited. In this paper, we offer a new method, MitoFREQ, for estimating the population frequencies of mitogenomes. MitoFREQ uses the mitogenome resources HelixMTdb and gnomAD, harbouring information from 195,983 and 56,406 mitogenomes, respectively. Neither HelixMTdb nor gnomAD can be queried directly for individual mitogenome frequencies, but offers single nucleotide variant (SNV) allele frequencies for each of 30 "top-level" haplogroups (TLHG). We propose using the HelixMTdb and gnomAD resources by classifying a given mitogenome within the TLHG scheme and subsequently using the frequency of its rarest SNV within that TLHG weighted by the TLHG frequency. We show that this method is guaranteed to provide a higher population frequency estimate than if a refined haplogroup and its SNV frequencies were used. Further, we show that top-level haplogrouping can be achieved by using only 227 specific positions for 99.9% of the tested mitogenomes, potentially making the method available for low-quality samples. The method was tested on two types of datasets: high-quality forensic reference datasets and a diverse collection of scrutinised mitogenomes from GenBank. This dual evaluation demonstrated that the approach is robust across both curated forensic data and broader population-level sequences. This method produced likelihood ratios in the range of 100-100,000, demonstrating its potential to strengthen the statistical evaluation of forensic mtDNA evidence. We have developed an open-source R package `mitofreq` that implements our method, including a Shiny app where custom TLHG frequencies can be supplied.
Read the full abstract

All authors Mikkel Meyer Andersen, Nicole Huber, Kimberly S Andreaggi, Tóra Oluffa Stenberg Olsen, Walther Parson, Charla Marshall

Statistical Computation · Cross-listed from cs.HC

LangLasso: Interactive Cluster Descriptions through LLM Explanation

Dimensionality reduction is a powerful technique for revealing structure and potential clusters in data. However, as the axes are complex, non-linear combinations of features, they often lack semantic interpretability. Existing visual analytics (VA) methods support cluster interpretation through feature comparison and interactive exploration, but they require technical expertise and intense human effort. We present LangLasso, a novel method that complements VA approaches through interactive, natural language descriptions of clusters using large language models (LLMs). It produces human-readable descriptions that make cluster interpretation accessible to non-experts and allow integration of external contextual knowledge beyond the dataset. We systematically evaluate the reliability of these explanations and demonstrate that \langlasso provides an effective first step for engaging broader audiences in cluster interpretation. The tool is available at https://langlasso.vercel.app
Read the full abstract

All authors Raphael Buchmüller, Dennis Collaris, Linhao Meng, Angelos Chatzimparmpas

Statistics Applications

Modeling mental health trajectories during the COVID-19 pandemic using UK-wide data in the presence of sociodemographic variables

Background: The negative effects of the COVID-19 pandemic on the mental health and well-being of populations are an important public health issue. Our study aims to determine the underlying factors shaping mental health trajectories during the COVID-19 pandemic in the UK. Methods: Data from the Understanding Society COVID-19 Study were utilized and the core analysis focussed on GHQ36 scores as the outcome variable. We used GAMs to evaluate trends over time and the role of sociodemographic variables, i.e., age, sex, ethnicity, country of residence (in UK), job status (employment), household income, living with a partner, living with children under age 16, and living with a long-term illness, on the variation of mental health during the study period. Results: Statistically significant differences in mental health were observed for age, sex,ethnicity, country of residence (in UK), job status (employment), household income, living with a partner, living with children under age 16, and living with a long-term illness. Women experienced higher GHQ36 scores relative to men with the GHQ36 score expected to increase by 1.260 (95%CI: 1.176, 1.345). Individuals living without a partner were expected to have higher GHQ36 scores, of 1.050 (95%CI: 0.949, 1.148) more than those living with a partner, and age groups 16-34, 35-44, 45-54, 55-64 experienced higher GHQ36 scores relative to those who were 65+. Individuals with relatively lower household income were likely to have poorer mental health relative to those who were more well off. Conclusion: This study identifies key demographic determinants shaping mental health trajectories during the COVID-19 pandemic in the UK. Policies aiming to reduce mental health inequalities should target women, youth, individuals living without a partner, individuals living with children under 16, individuals with a long-term illness, and lower income families.
Read the full abstract

All authors Glenna Nightingale, Karthik Mohan, Eloi Ribe, Valentin Popov, Shakes Wang, Clara Calia, Luciana Brondi, Sohan Seth