Statistics Applications · Cross-listed from cond-mat.soft
Is gelation a singularity or a flow induced instability?
Manuel Dedola and Ludovico Cademartiri
Gelation in the Smoluchowski coagulation equation is commonly interpreted as a finite-time singularity marked by mass loss or moment divergence. We instead characterize gelation as a loss of dynamical stability of the Smoluchowski flow, quantified through the time-dependent spectrum of the Jacobian along the evolving aggregation dynamics. Studying homogeneous kernels $K(i,j)=(ij)^{\alpha}$ together with the classical Smoluchowski, we show that gelation is consistently preceded by the appearance of positive real
eigenvalues, indicating a loss of local dynamical stability. While non-gelling kernels exhibit only transient finite-size effects, gelling kernels display persistent spectral destabilization associated with macroscopic gel formation. Our results identify gelation as a genuine dynamical instability of the Smoluchowski flow.
Read the full abstract ▾
Statistics Applications
Climate Vulnerability and Community Health: Identifying Greensboro Neighborhoods at Intersectional Risk
Rehinatu Usman and Onyedikachi J. Okeke
This study develops an integrated, intersectional
climate vulnerability assessment for Greensboro, North Carolina, a midsize city in the rapidly changing American Southeast. Moving beyond generalized mapping, we combine demographic, socioeconomic, health, and environmental data at the census tract level to identify neighborhoods where flood exposure, chronic health burdens, and social disadvantage spatially converge. Through k-means and hierarchical clustering, we identify four distinct neighborhood typologies, including a critically high-risk cluster characterized by high flood exposure, extreme poverty, poor respiratory health, and aging housing. The findings demonstrate that
climate-related risks are not randomly distributed but systematically cluster in historically marginalized communities, revealing a clear environmental justice disparity. This place-based typology approach provides a targeted framework for policymakers to design integrated interventions that bridge flood management, public health, housing, and social services to build equitable urban resilience
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.LG
Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning
Rajat Ghosh and Debojyoti Dutta
Numerous offline and model-based reinforcement learning systems incorporate world models to emulate the inherent environments. A world model is particularly important in scenarios where direct interactions with the real environment is costly, dangerous, or impractical. The efficacy and interpretability of such world models are notably contingent upon the quality of the underlying training data. In this context, we introduce Action Shapley as an agnostic metric for the judicious and unbiased selection of training data. To facilitate the computation of Action Shapley, we present a randomized dynamic
algorithm specifically designed to mitigate the exponential complexity inherent in traditional Shapley value computations. Through empirical validation across five data-constrained real-world case studies, the
algorithm demonstrates a computational efficiency improvement exceeding 80% in comparison to conventional exponential time computations. Furthermore, our Action Shapley-based training data selection policy consistently outperforms ad-hoc training data selection.
Read the full abstract ▾
Statistical Methodology
On the use of cross-fitting in causal machine learning with correlated units
Salvador V. Balkus, Hasan Laith and Nima S. Hejazi
In causal machine learning, the fitting and evaluation of nuisance models are often performed on separate partitions, or folds, of the observed data. This technique, called cross-fitting, eliminates bias introduced by the use of black-box predictive
algorithms. When study units may be correlated, such as in spatial, clustered, or time-series data, investigators often design bespoke forms of cross-fitting to minimize correlation between folds. We prove that, perhaps contrary to popular belief, this is typically unnecessary: performing cross fitting as if study units were independent still eliminates key bias terms even when units may be correlated. In simulation experiments with various correlation structures, we show that causal machine learning
estimators achieve the same or improved bias and precision under cross-fitting that ignores correlation compared to techniques striving to eliminate correlation between folds.
Read the full abstract ▾
Statistics Applications · Cross-listed from astro-ph.IM
Optimal and Unbiased Fluxes from Up-the-Ramp Detectors under Variable Illumination
Bowen Li, Kevin A. McKinnon, Andrew K. Saydjari et al. (+6 more)
Near-infrared (NIR) detectors – which use non-destructive readouts to measure time-series counts-per-pixel – play a crucial role in modern astrophysics. Standard NIR flux extraction techniques were developed for space-based observations and assume that source fluxes are constant over an observation. However, ground-based
telescopes often see short-timescale atmospheric variations that can dramatically change the number of
photons arriving at a pixel. This work presents a new statistical model that shares information between neighboring spectral pixels to characterize time-variable observations and extract unbiased fluxes with optimal uncertainties. We generate realistic synthetic data using a variety of flux and amplitude-of-time-variability conditions to confirm that our model recovers unbiased and optimal estimates of both the true flux and the time-variable signal. We find that the time-variable model should be favored over a constant-flux model when the observed count rates change by more than 3.5%. Ignoring time variability in the data can result in flux-dependent, unknown-sign biases that are as large as ~120% of the flux uncertainty. Using real APOGEE spectra, we find empirical evidence for approximately wavelength-independent, time-dependent variations in count rates with amplitudes much greater than the 3.5% threshold. Our model can robustly measure and remove the time-dependence in real data, improving the quality of data-model comparison. We show several examples where the observed time-dependence quantitatively agrees with independent measurements of observing conditions, such as variable cloud cover and seeing.
Read the full abstract ▾
All authors Bowen Li, Kevin A. McKinnon, Andrew K. Saydjari, Conor Sayres, Gwendolyn M. Eadie, Andrew R. Casey, Jon A. Holtzman, Timothy D. Brandt, Jose G. Fernandez-Trincado
Statistical Methodology
Locally sparse varying coefficient mixed model with application to longitudinal microbiome differential abundance
Simon Fontaine, Nisha J. D'Silva, Marcell Costa de Medeiros et al. (+3 more)
Differential abundance (DA) analysis in microbiome studies has recently been used to uncover a plethora of associations between microbial composition and various health conditions. While current approaches to DA typically apply only to cross-sectional data, many studies feature a longitudinal design to better understand the underlying microbial dynamics. To study DA in longitudinal microbial studies, we introduce a novel varying coefficient mixed-effects model with local sparsity. The proposed method can identify time intervals of significant group differences while accounting for temporal dependence. Specifically, we exploit a penalized kernel smoothing approach for parameter estimation and include a random effect to account for serial correlation. In particular, our method operates effectively regardless of whether sampling times are shared across subjects, accommodating irregular sampling and missing observations. Simulation studies demonstrate the necessity of modeling dependence for precise estimation and support recovery. The application of our method to a longitudinal study of mice oral microbiome during cancer development revealed significant scientific insights that were otherwise not discernible through cross-sectional analyses. An R implementation is available at https://github.com/fontaine618/LSVCMM.
Read the full abstract ▾
All authors Simon Fontaine, Nisha J. D'Silva, Marcell Costa de Medeiros, Grace Y. Chen, Ji Zhu, Gen Li
Statistics Theory
Rigidity theory in statistical inference
Daniel Irving Bernstein
In this expository article, we summarize what is known about maximum likelihood thresholds of Gaussian models, paying special
attention to connections with
rigidity theory.
Statistics Applications · Cross-listed from econ.GN
Beyond Unidimensionality: General Factors and Residual Heterogeneity in Performance Evaluation
Krishna Sharma and Pritam Basnet
How do evaluation systems compress multidimensional performance information into summary ratings? Using expert assessments of 9,669 professional soccer players on 28 attributes, we characterize the dimensional structure of evaluation outputs. The first principal component explains 40.6% of attribute variance, indicating a strong general factor, but formal noise discrimination procedures retain four components and bootstrap resampling confirms that this structure is highly stable. Internal consistency is high without evidence of redundancy. In out of sample prediction of expert overall ratings, a comprehensive model using the full attribute set substantially outperforms a single-factor summary (cross-validated R squared = 0.814). Overall, performance evaluations exhibit moderate information compression; they combine shared variance with stable residual dimensions that are economically meaningful for evaluation outcomes, with direct implications for the design of measurement systems.
Read the full abstract ▾
Statistics Theory · Cross-listed from cs.LG
High-accuracy and dimension-free sampling with diffusions
Khashayar Gatmiry, Sitan Chen and Adil Salim
Diffusion models have shown remarkable empirical success in sampling from rich multi-modal distributions. Their inference relies on numerically solving a certain differential equation. This differential equation cannot be solved in closed form, and its resolution via discretization typically requires many small iterations to produce high-quality samples. More precisely, prior works have shown that the iteration complexity of discretization methods for
diffusion models scales polynomially in the ambient dimension and the inverse accuracy $1/\varepsilon$. In this work, we propose a new solver for
diffusion models relying on a subtle interplay between low-degree approximation and the collocation method (Lee, Song, Vempala 2018), and we prove that its iteration complexity scales polylogarithmically in $1/\varepsilon$, yielding the first “high-accuracy” guarantee for a
diffusion-based sampler that only uses (approximate) access to the scores of the data distribution. In addition, our bound does not depend explicitly on the ambient dimension; more precisely, the dimension affects the complexity of our solver through the effective radius of the support of the target distribution only.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
On the origin of neural scaling laws: from random graphs to natural language
Maissam Barkeshli, Alberto Alfarano and Andrey Gromov
Scaling laws have played a major role in the modern AI revolution, providing practitioners predictive power over how the model performance will improve with increasing data, compute, and number of model parameters. This has spurred an intense interest in the origin of neural scaling laws, with a common suggestion being that they arise from power law structure already present in the data. In this paper we study scaling laws for
transformers trained to predict random walks (bigrams) on graphs with tunable complexity. We demonstrate that this simplified setting already gives rise to neural scaling laws even in the absence of power law structure in the data correlations. We further consider dialing down the complexity of natural
language systematically, by training on sequences sampled from increasingly simplified generative
language models, from 4,2,1-layer
transformer language models down to
language bigrams, revealing a monotonic
evolution of the scaling exponents. Our results also include scaling laws obtained from training on random walks on random graphs drawn from Erdös-Renyi and scale-free Barabási-Albert ensembles. Finally, we revisit conventional scaling laws for
language modeling, demonstrating that several essential results can be reproduced using 2 layer
transformers with context length of 50, provide a critical analysis of various fits used in prior literature, demonstrate an alternative method for obtaining compute optimal curves as compared with current practice in published literature, and provide preliminary evidence that maximal update parameterization may be more parameter efficient than standard parameterization.
Read the full abstract ▾
Statistics Theory · Cross-listed from physics.soc-ph
Transforming Crises into Opportunities: From Chaos to Urban Antifragility
Joseph Uguet, Nicola Tollin and Jordi Morato
Urban crises - floods, pandemics, economic shocks, and conflicts - function as accelerators of urban change, exposing structural vulnerabilities while creating windows for reinvention. Building on a prior theoretical contribution that identified fifteen principles of urban antifragility, this paper tests and operationalizes the framework through an empirical assessment of 26 cities selected for their post-crisis adaptation trajectories. Using a tailored diagnostic methodology, we
benchmark cities' Stress Response Strategies (SRS) and then evaluate Urban Development Trajectories (UDT) across four weighted dimensions, positioning each case along a fragility-robustness-resilience-antifragility continuum and applying a balanced-threshold rule to confirm antifragile status. Results show that "resilience enhanced by innovation and technology" is the most effective response typology (86.9/100), and that six cities meet the antifragile trajectory criteria. By mapping best practices to activated principles and analysing co-activations, the study identifies a robust "hard core" of principles - Sustainable Resilience (O), Strategic Diversity (F), Proactive Innovation (I), and Active Prevention (N) - supplemented by operational enablers (e.g., anticipation, mobilization, shock absorption). The paper concludes by proposing an evidence-based, SDG-aligned operational model that links high-impact principle pairings to measurable indicators, offering a practical roadmap for cities seeking to convert crises into sustained transformation. Keywords: Post-crisis strategies, Urban antifragility, Sustainable cities and communities, Disaster resilience and urban regeneration, Risk governance and Black Swan adaptation.
Read the full abstract ▾
Statistical Methodology
Adjusted Similarity Measures and a Violation of Expectations
William L. Lippitt, Edward J. Bedrick and Nichole E. Carlson
Adjusted similarity measures, such as Cohen's kappa for inter-rater reliability and the adjusted Rand index used to compare clustering
algorithms, are a vital tool for comparing discrete labellings. These measures are intended to have the property of 0 expectation under a null distribution and maximum value 1 under maximal similarity to aid in interpretation. Measures are frequently adjusted with respect to the permutation distribution for historic and analytic reasons. There is currently renewed interest in considering other null models more appropriate for context, such as clustering ensembles permitting a random number of identified clusters. The purpose of this work is two – fold: (1) to generalize the study of the adjustment operator to general null models and to a more general procedure which includes statistical standardization as a special case and (2) to identify sufficient conditions for the adjustment operator to produce the intended properties, where sufficient conditions are related to whether and how observed data are incorporated into null distributions. We demonstrate how violations of the sufficient conditions may lead to substantial breakdown, such as by producing a non-positive measure under traditional adjustment rather than one with mean 0, or by producing a measure which is deterministically 0 under statistical standardization.
Read the full abstract ▾
Machine Learning (Statistics)
Classification Imbalance as Transfer Learning
Eric Xia and Jason M. Klusowski
Classification imbalance arises when one class is much rarer than the other. We frame this setting as transfer learning under label (prior) shift between an imbalanced source distribution induced by the observed data and a balanced target distribution under which performance is evaluated. Within this framework, we study a family of oversampling procedures that augment the training data by generating synthetic samples from an estimated minority-class distribution to roughly balance the classes, among which the celebrated SMOTE
algorithm is a canonical example. We show that the excess risk decomposes into the rate achievable under balanced training (as if the data had been drawn from the balanced target distribution) and an additional term, the cost of transfer, which quantifies the discrepancy between the estimated and true minority-class distributions. In particular, we show that the cost of transfer for SMOTE dominates that of bootstrapping (random oversampling) in moderately high dimensions, suggesting that we should expect bootstrapping to have better performance than SMOTE in general. We corroborate these findings with experimental evidence. More broadly, our results provide guidance for choosing among augmentation strategies for imbalanced classification.
Read the full abstract ▾
Machine Learning (Statistics)
Parametric RDT approach to computational gap of symmetric binary perceptron
Mihailo Stojnic
We study potential presence of statistical-computational gaps (SCG) in symmetric binary perceptrons (SBP) via a parametric utilization of fully lifted random duality theory (fl-RDT) [96]. A structural change from decreasingly to arbitrarily ordered $c$-sequence (a key fl-RDT parametric component) is observed on the second lifting level and associated with satisfiability ($\alpha_c$) – algorithmic ($\alpha_a$) constraints density threshold change thereby suggesting a potential existence of a nonzero computational gap $SCG=\alpha_c-\alpha_a$. The second level estimate is shown to match the theoretical $\alpha_c$ whereas the $r\rightarrow \infty$ level one is proposed to correspond to $\alpha_a$. For example, for the canonical SBP ($\kappa=1$ margin) we obtain $\alpha_c\approx 1.8159$ on the second and $\alpha_a\approx 1.6021$ (with converging tendency towards $\sim 1.59$ range) on the seventh level. Our propositions remarkably well concur with recent literature: (i) in [20] local
entropy replica approach predicts $\alpha_{LE}\approx 1.58$ as the onset of clustering defragmentation (presumed driving force behind locally improving
algorithms failures); (ii) in $\alpha\rightarrow 0$ regime we obtain on the third lifting level $\kappa\approx 1.2385\sqrt{\frac{\alpha_a}{-\log\left ( \alpha_a \right ) }}$ which qualitatively matches overlap gap property (OGP) based predictions of [43] and identically matches local
entropy based predictions of [24]; (iii) $c$-sequence ordering change phenomenology mirrors the one observed in asymmetric binary perceptron (ABP) in [98] and the negative Hopfield model in [100]; and (iv) as in [98,100], we here design a CLuP based
algorithm whose practical performance closely matches proposed theoretical predictions.
Read the full abstract ▾
Statistics Theory
Differentially Private Inference for Longitudinal Linear Regression
Getoar Sopa, Marco Avella Medina and Cynthia Rush
Differential
Privacy (DP) provides a rigorous framework for releasing statistics while protecting individual information present in a
dataset. Although substantial progress has been made on differentially private linear regression, existing methods almost exclusively address the item-level DP setting, where each user contributes a single observation. Many scientific and economic applications instead involve longitudinal or panel data, in which each user contributes multiple dependent observations. In these settings, item-level DP offers inadequate protection, and user-level DP - shielding an individual's entire trajectory - is the appropriate
privacy notion. We develop a comprehensive framework for estimation and inference in longitudinal linear regression under user-level DP. We propose a user-level private regression
estimator based on aggregating local regressions, and we establish finite-sample guarantees and asymptotic normality under short-range dependence. For inference, we develop a privatized, bias-corrected covariance
estimator that is automatically heteroskedasticity- and autocorrelation-consistent. These results provide the first unified framework for practical user-level DP estimation and inference in longitudinal linear regression under dependence, with strong theoretical guarantees and promising empirical performance.
Read the full abstract ▾
Statistical Methodology
Fair Regression under Demographic Parity: A Unified Framework
Yongzhen Feng, Weiwei Wang, Raymond K. W. Wong et al. (+1 more)
We propose a unified framework for fair regression tasks formulated as risk minimization problems subject to a demographic parity constraint. Unlike many existing approaches that are limited to specific loss functions or rely on challenging non-convex optimization, our framework is applicable to a broad spectrum of regression tasks. Examples include linear regression with squared loss, binary classification with cross-
entropy loss,
quantile regression with pinball loss, and robust regression with Huber loss. We derive a novel characterization of the fair risk minimizer, which yields a computationally efficient estimation procedure for general loss functions. Theoretically, we establish the asymptotic consistency of the proposed
estimator and derive its convergence rates under mild assumptions. We illustrate the method's versatility through detailed discussions of several common loss functions. Numerical results demonstrate that our approach effectively minimizes risk while satisfying
fairness constraints across various regression settings.
Read the full abstract ▾
All authors Yongzhen Feng, Weiwei Wang, Raymond K. W. Wong, Xianyang Zhang
Statistical Methodology
A Bayesian Discrete Framework for Enhancing Decision-Making Processes in Clinical Trial Designs and Evaluations
Paramahansa Pramanik, Arnab Kumar Maity, Anjan Mandal et al. (+1 more)
This study examines the application of
Bayesian approach in the context of clinical trials, emphasizing their increasing importance in contemporary biomedical research. While conventional frequentist approach provides a foundational basis for analysis, it often lacks the flexibility to integrate prior knowledge, which can constrain its effectiveness in adaptive settings. In contrast,
Bayesian methods enable continual refinement of statistical inferences through the assimilation of accumulating evidence, thereby supporting more informed decision-making and improving the reliability of trial findings. This paper also considers persistent challenges in clinical investigations, including replication difficulties and the misinterpretation of statistical results, suggesting that
Bayesian strategies may offer a path toward enhanced analytical robustness. Moreover, discrete probability models, specifically the Binomial, Poisson, and Negative Binomial distributions are explored for their suitability in modeling clinical endpoints, particularly in trials involving binary responses or data with overdispersion. The discussion further incorporates
Bayesian networks and
Bayesian estimation techniques, with a comparative evaluation against maximum likelihood estimation to elucidate differences in inferential behavior and practical implementation.
Read the full abstract ▾
All authors Paramahansa Pramanik, Arnab Kumar Maity, Anjan Mandal, Haley Kate Robinson
Statistical Methodology
From aggressive to conservative early stopping in Bayesian group sequential designs
Zhangyi He, Feng Yu, Suzie Cro et al. (+1 more)
Group sequential designs (GSDs) are widely used in confirmatory trials to allow interim monitoring while preserving control of the type I error rate. In the frequentist framework, O'Brien-Fleming-type stopping boundaries dominate practice because they impose highly conservative early stopping while allowing more liberal decisions as information accumulates.
Bayesian GSDs, in contrast, are most often implemented using fixed posterior probability thresholds applied uniformly at all analyses. While such designs can be calibrated to control the overall type I error rate, they do not penalise early analyses and can therefore lead to substantially more aggressive early stopping. Such behaviour can risk premature conclusions and inflation of treatment effect estimates, raising concerns for confirmatory trials. We introduce two practically implementable refinements that restore conservative early stopping in
Bayesian GSDs. The first introduces a two-phase structure for posterior probability thresholds, applying more stringent criteria in the early phase of the trial and relaxing them later to preserve power. The second replaces posterior probability monitoring at interim looks with predictive probability criteria, which naturally account for uncertainty in future data and therefore suppress premature stopping. Both strategies require only one additional tuning parameter and can be efficiently calibrated. In the HYPRESS setting, both approaches achieve higher power than the conventional
Bayesian design while producing alpha-spending profiles closely aligned with O'Brien-Fleming-type behaviour at early looks. These refinements provide a principled and tractable way to align
Bayesian GSDs with accepted frequentist practice and regulatory expectations, supporting their robust application in confirmatory trials.
Read the full abstract ▾
All authors Zhangyi He, Feng Yu, Suzie Cro, Laurent Billot
Machine Learning (Statistics) · Cross-listed from econ.EM
causalfe: Causal Forests with Fixed Effects in Python
Harry Aytug
The causalfe package provides a Python implementation of Causal Forests with Fixed Effects (CFFE) for estimating heterogeneous treatment effects in panel data settings. Standard causal forest methods struggle with panel data because unit and time fixed effects induce spurious heterogeneity in treatment effect estimates. The CFFE approach addresses this by performing node-level residualization during tree construction, removing fixed effects within each candidate split rather than globally. This paper describes the methodology, documents the software interface, and demonstrates the package through simulation studies that validate the
estimator's performance under various data generating processes.
Read the full abstract ▾
Statistical Methodology
A Propagation Framework for Network Regression
Yingying Ma and Chenlei Leng
We introduce a unified and computationally efficient framework for regression on network data, addressing limitations of existing models that require specialized estimation procedures or impose restrictive decay assumptions. Our Network Propagation Regression (NPR) models outcomes as functions of
covariates propagated through network connections, capturing both direct and indirect effects. NPR is estimable via ordinary least squares for continuous outcomes and standard routines for binary, categorical, and time-to-event data, all within a single interpretable framework. We establish consistency and asymptotic normality under weak conditions and develop valid hypothesis tests for the order of network influence. Simulation studies demonstrate that NPR consistently outperforms established approaches, such as the linear-in-means model and regression with network cohesion, especially under model misspecification. An application to social media sentiment analysis highlights the practical utility and robustness of NPR in real-world settings.
Read the full abstract ▾
Machine Learning (Statistics)
Coarsening Causal DAG Models
Francisco Madaleno, Pratik Misra and Alex Markham
Directed acyclic graphical (DAG) models are a powerful tool for representing causal relationships among jointly distributed random variables, especially concerning data from across different experimental settings. However, it is not always practical or desirable to estimate a causal model at the granularity of given features in a particular
dataset. There is a growing body of research on causal abstraction to address such problems. We contribute to this line of research by (i) providing novel graphical identifiability results for practically-relevant interventional settings, (ii) proposing an efficient, provably consistent
algorithm for directly learning abstract causal graphs from interventional data with unknown intervention targets, and (iii) uncovering theoretical insights about the
lattice structure of the underlying search space, with connections to the field of causal discovery more generally. As proof of concept, we apply our
algorithm on synthetic and real
datasets with known ground truths, including measurements from a controlled physical system with interacting light intensity and polarization.
Read the full abstract ▾
Statistics Theory · Cross-listed from cs.IT
On the suboptimality of linear codes for binary distributed hypothesis testing
Adway Girish, Robinson D. H. Cung and Emre Telatar
We study a binary distributed hypothesis testing problem where two
agents observe correlated binary vectors and communicate compressed information at the same rate to a central decision maker. In particular, we study linear compression schemes and show that simple truncation is the best linear scheme in two cases: (1) testing opposite signs of the same magnitude of correlation, and (2) testing for or against independence. We
conjecture, supported by numerical evidence, that truncation is the best linear code for testing any correlations of opposite signs. Further, for testing against independence, we also compute classical random coding exponents and show that truncation, and consequently any linear code, is strictly suboptimal.
Read the full abstract ▾
Statistical Methodology · Cross-listed from econ.EM
Semiparametric inference for inequality measures under nonignorable nonresponse using callback data
Xinyu Wang, Chunlin Wang, Tao Yu et al. (+1 more)
This paper develops semiparametric methods for estimation and inference of widely used inequality measures when survey data are subject to nonignorable nonresponse, a challenging setting in which response probabilities depend on the unobserved outcomes. Such nonresponse mechanisms are common in household surveys and invalidate standard inference procedures due to selection bias and lack of population representativeness. We address this problem by exploiting callback data from repeated contact attempts and adopting a semiparametric model that leaves the outcome distribution unspecified. We construct semiparametric full-likelihood
estimators for the underlying distribution and the associated inequality measures, and establish their large-sample properties for a broad class of functionals, including
quantiles, the Theil index, and the Gini index. Explicit asymptotic variance expressions are derived, enabling valid Wald-type inference under nonignorable nonresponse. To facilitate implementation, we propose a stable and computationally convenient expectation-maximization
algorithm, whose steps either admit closed-form expressions or reduce to fitting a standard logistic regression model. Simulation studies demonstrate that the proposed procedures effectively correct nonresponse bias and achieve near-
benchmark efficiency. An application to Consumer Expenditure Survey data illustrates the practical gains from incorporating callback information when making inference on inequality measures.
Read the full abstract ▾
All authors Xinyu Wang, Chunlin Wang, Tao Yu, Pengfei Li
Machine Learning (Statistics)
CROCS: A Two-Stage Clustering Framework for Behaviour-Centric Consumer Segmentation with Smart Meter Data
Luke W. Yerbury, Ricardo J. G. B. Campello, G. C. Livingston Jr et al. (+2 more)
With grid operators confronting rising uncertainty from renewable integration and a broader push toward electrification, Demand-Side Management (DSM) – particularly Demand Response (DR) – has attracted significant
attention as a cost-effective mechanism for balancing modern electricity systems. Unprecedented volumes of consumption data from a continuing global deployment of smart meters enable consumer
segmentation based on real usage behaviours, promising to inform the design of more effective DSM and DR programs. However, existing clustering-based
segmentation methods insufficiently reflect the behavioural diversity of consumers, often relying on rigid temporal
alignment, and faltering in the presence of anomalies, missing data, or large-scale deployments. To address these challenges, we propose a novel two-stage clustering framework – Clustered Representations Optimising Consumer
Segmentation (CROCS). In the first stage, each consumer's daily load profiles are clustered independently to form a Representative Load Set (RLS), providing a compact summary of their typical diurnal consumption behaviours. In the second stage, consumers are clustered using the Weighted Sum of Minimum Distances (WSMD), a novel set-to-set measure that compares RLSs by accounting for both the prevalence and similarity of those behaviours. Finally, community detection on the WSMD-induced graph reveals higher-order prototypes that embody the shared diurnal behaviours defining consumer groups, enhancing the interpretability of the resulting clusters. Extensive experiments on both synthetic and real Australian smart meter
datasets demonstrate that CROCS captures intra-consumer variability, uncovers both synchronous and asynchronous behavioural similarities, and remains robust to anomalies and missing data, while scaling efficiently through natural parallelisation. These results...
Read the full abstract ▾
All authors Luke W. Yerbury, Ricardo J. G. B. Campello, G. C. Livingston Jr, Mark Goldsworthy, Lachlan O'Neil
Statistical Computation
Mesh Denoising
Constantin Vaillant Tenzer
In this paper, we study four mesh denoising methods: linear filtering, a heat
diffusion method, Sobolev regularization, and, to a lesser extent, a barycentric approach based on the Sinkhorn
algorithm. We illustrate that, for a simple image denoising task, a naive choice of a Gibbs kernel can lead to unsatisfactory results. We demonstrate that while Sobolev regularization is the fastest method in our implementation, it produces slightly less faithful denoised meshes than the best results obtained with iterative filtering or heat
diffusion. We empirically show that, for the large mesh considered, the heat
diffusion method is slower and not more effective than filtering, whereas on a small mesh an appropriate choice of
diffusion parameters can improve the quality. Finally, we observe that all three mesh-based methods perform markedly better on the large mesh than on the small one.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from math.OC
High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks
Simon Martin, Giulio Biroli and Francis Bach
We study the high-dimensional training dynamics of a shallow neural network with quadratic activation in a teacher-student setup. We focus on the extensive-width regime, where the teacher and student network widths scale proportionally with the input dimension, and the sample size grows quadratically. This scaling aims to describe overparameterized neural networks in which feature learning still plays a central role. In the high-dimensional limit, we derive a dynamical characterization of the gradient flow, in the spirit of dynamical mean-field theory (DMFT). Under l2-regularization, we analyze these equations at long times and characterize the performance and spectral properties of the resulting
estimator. This result provides a quantitative understanding of the effect of overparameterization on learning and generalization, and reveals a double descent phenomenon in the presence of label noise, where generalization improves beyond interpolation. In the small regularization limit, we obtain an exact expression for the perfect recovery threshold as a function of the network widths, providing a precise characterization of how overparameterization influences recovery.
Read the full abstract ▾
Statistics Applications
MitoFREQ: A Novel Approach for Mitogenome Frequency Estimation from Top-level Haplogroups and Single Nucleotide Variants
Mikkel Meyer Andersen, Nicole Huber, Kimberly S Andreaggi et al. (+3 more)
Lineage marker population frequencies can serve as one way to express evidential value in forensic genetics. However, for high-quality whole mitochondrial DNA genome sequences (mitogenomes), population data remain limited. In this paper, we offer a new method, MitoFREQ, for estimating the population frequencies of mitogenomes. MitoFREQ uses the mitogenome resources HelixMTdb and gnomAD, harbouring information from 195,983 and 56,406 mitogenomes, respectively. Neither HelixMTdb nor gnomAD can be queried directly for individual mitogenome frequencies, but offers single nucleotide variant (SNV) allele frequencies for each of 30 "top-level" haplogroups (TLHG). We propose using the HelixMTdb and gnomAD resources by classifying a given mitogenome within the TLHG scheme and subsequently using the frequency of its rarest SNV within that TLHG weighted by the TLHG frequency. We show that this method is guaranteed to provide a higher population frequency estimate than if a refined haplogroup and its SNV frequencies were used. Further, we show that top-level haplogrouping can be achieved by using only 227 specific positions for 99.9% of the tested mitogenomes, potentially making the method available for low-quality samples. The method was tested on two types of
datasets: high-quality forensic reference
datasets and a diverse collection of scrutinised mitogenomes from GenBank. This dual evaluation demonstrated that the approach is robust across both curated forensic data and broader population-level sequences. This method produced likelihood ratios in the range of 100-100,000, demonstrating its potential to strengthen the statistical evaluation of forensic mtDNA evidence. We have developed an open-source R package `mitofreq` that implements our method, including a Shiny app where custom TLHG frequencies can be supplied.
Read the full abstract ▾
All authors Mikkel Meyer Andersen, Nicole Huber, Kimberly S Andreaggi, Tóra Oluffa Stenberg Olsen, Walther Parson, Charla Marshall
Statistical Computation · Cross-listed from cs.HC
LangLasso: Interactive Cluster Descriptions through LLM Explanation
Raphael Buchmüller, Dennis Collaris, Linhao Meng et al. (+1 more)
Dimensionality reduction is a powerful technique for revealing structure and potential clusters in data. However, as the axes are complex, non-linear combinations of features, they often lack semantic interpretability. Existing visual analytics (VA) methods support cluster interpretation through feature comparison and interactive exploration, but they require technical expertise and intense human effort. We present LangLasso, a novel method that complements VA approaches through interactive, natural
language descriptions of clusters using large
language models (LLMs). It produces human-readable descriptions that make cluster interpretation accessible to non-experts and allow integration of external contextual knowledge beyond the
dataset. We systematically evaluate the reliability of these explanations and demonstrate that \langlasso provides an effective first step for engaging broader audiences in cluster interpretation. The tool is available at https://langlasso.vercel.app
Read the full abstract ▾
All authors Raphael Buchmüller, Dennis Collaris, Linhao Meng, Angelos Chatzimparmpas
Statistics Applications
Modeling mental health trajectories during the COVID-19 pandemic using UK-wide data in the presence of sociodemographic variables
Glenna Nightingale, Karthik Mohan, Eloi Ribe et al. (+5 more)
Background: The negative effects of the COVID-19 pandemic on the mental health and well-being of populations are an important public health issue. Our study aims to determine the underlying factors shaping mental health trajectories during the COVID-19 pandemic in the UK. Methods: Data from the Understanding Society COVID-19 Study were utilized and the core analysis focussed on GHQ36 scores as the outcome variable. We used GAMs to evaluate trends over time and the role of sociodemographic variables, i.e., age, sex, ethnicity, country of residence (in UK), job status (employment), household income, living with a partner, living with children under age 16, and living with a long-term illness, on the variation of mental health during the study period. Results: Statistically significant differences in mental health were observed for age, sex,ethnicity, country of residence (in UK), job status (employment), household income, living with a partner, living with children under age 16, and living with a long-term illness. Women experienced higher GHQ36 scores relative to men with the GHQ36 score expected to increase by 1.260 (95%CI: 1.176, 1.345). Individuals living without a partner were expected to have higher GHQ36 scores, of 1.050 (95%CI: 0.949, 1.148) more than those living with a partner, and age groups 16-34, 35-44, 45-54, 55-64 experienced higher GHQ36 scores relative to those who were 65+. Individuals with relatively lower household income were likely to have poorer mental health relative to those who were more well off. Conclusion: This study identifies key demographic determinants shaping mental health trajectories during the COVID-19 pandemic in the UK. Policies aiming to reduce mental health inequalities should target women, youth, individuals living without a partner, individuals living with children under 16, individuals with a long-term illness, and lower income families.
Read the full abstract ▾
All authors Glenna Nightingale, Karthik Mohan, Eloi Ribe, Valentin Popov, Shakes Wang, Clara Calia, Luciana Brondi, Sohan Seth