Statistics Applications · Cross-listed from stat.ME
Robust K-means Clustering using the Density Power Divergence Measure
Anirban Mondal, Paromita Banerjee and Abhijit Mandal
We introduce a robust clustering method, MK-means DPD, that estimates cluster centers and covariance
matrices using density power divergence (DPD) measures combined with Mahalanobis distance, making it resistant to outliers and adaptable to heterogeneous, elliptical clusters, unlike the classical K-means
algorithm. Since Mahalanobis distance-based K-means lacks a general convergence guarantee, we further introduce a convergent variant, Density-Consistent MK-means DPD (DC-MK-means DPD), which redefines the cluster assignment step in terms of a pointwise DPD loss. We prove a formal
theorem establishing that the resulting
algorithm converges in a finite number of steps. We also propose two new robust internal evaluation indices, a Median Davies-Bouldin Index and a Trimmed Calinski-Harabasz Index, to ensure that performance comparisons are not themselves distorted by outliers. The efficacy of the proposed methods is demonstrated on simulated data, showing superiority over existing methods, and on two real
datasets: Iris data, to identify similar species, and COVID-19 case fatality rate and infection rate data for countries worldwide, examining the resulting clusters' geographic and socio-economic patterns.
Read the full abstract ▾
Statistics Applications
Nonlinear trajectories of lung function recovery in patients with pulmonary disease: empirical evaluation of longitudinal modeling approaches
Zongyue Teng, Ningkun Zhou, Xinyu Zhang et al. (+2 more)
Introduction: Longitudinal lung function recovery after pulmonary disease commonly follows nonlinear trajectories, and failure to adequately model these trajectories can lead to biased or misleading estimates of treatment effects. However, an important methodological gap remains as there is limited assessment of statistical methods for modeling nonlinear lung function trajectories. Methods: We compared several longitudinal modeling approaches for characterizing recovery in percent predicted forced expiratory volume in one second (FEV1p) using data from a phase 2 randomized trial for pulmonary tuberculosis (TB). We estimate the differences in repeated mean FEV1p between each treatment arm and control arm over a 180-days follow-up period. We compared 6 different statistical models: (1) linear mixed-effects model, (2) a piecewise linear mixed-effects model, (3) quadratic and (4) natural cubic spline mixed-effects models, (5) a nonlinear mixed-effects model with exponential recovery function, and (6) a generalized additive mixed model. We discussed the assumptions, clinical interpretations, and resulting treatment-effect estimates across these approaches. Results: The results from the TB trial analyses showed that the conventional linear mixed-effects model provided limited evidence of treatment differences over follow-up, whereas several flexible models identified significant differences during specific periods of recovery. Conclusion: Flexible longitudinal models can complement conventional linear mixed-effects models by revealing treatment differences at certain periods of follow-up that may be obscured by assuming a single linear trend. The choice of nonlinear modeling strategy should be guided by the scientific objective, available data, and the desired balance between clinical interpretability and flexibility.
Read the full abstract ▾
All authors Zongyue Teng, Ningkun Zhou, Xinyu Zhang, Robert Wallis, Qingyan Xiang
Statistics Applications · Cross-listed from stat.ME
Design of Experiment in Complex Systems based on Computational Taxonomy
Eric Goldman and Fushing Hsieh
Via Computational Taxonomy (CT), we develop Design of Experiment(DoE) based on rigorously redefined constituting ingredients of complex system dynamics: randomness, nonlinearity and even class, through a data-driven constructed Taxonomic Hierarchy. As an opposite quest of Classification without man-made assumptions and structures, we illustrate this new perspective of DoE through a civil engineering complex system: Concrete Compressive Strength (CCS). CT begins by building a Taxonomic Hierarchy as a heterogeneity-vs-homogeneity map framed with a tree geometry to represent CCS-system dynamics. At each internal node of this hierarchy, a heatmap is computed via Scientific Data Analysis (SDA) to reveal locality-embraced heterogeneity through block structured
covariate homogeneity annotated with response's locality-split. Only arriving at each ending-node of this hierarchy, coherence of homogeneity is achieved on both response and
covariate sides. As such a class of finite sample nature is computationally recognized and confirmed. In contrast, nonlinearity is evidently observed as incoherence of response-vs-
covariate homogeneity when comparing two classes located on two distinct branches. This hierarchy explicitly maps out system's randomness and nonlinearity to serve as a scientific basis for any DoE quest. Design of Experiment (DoE) for any designated class is redefined as a search for a
covariate subspace that embraces the class-representative randomness and at the same time avoids potential nonlinearities with respect to the rest of classes. This is a brand-new theme of DoE.
Read the full abstract ▾
Statistics Applications
Decarbonising price formation: unit-level evidence on battery storage and the imbalance price in the GB Balancing Mechanism
Robert Dalton and Aidan O'Sullivan
Renewables now dominate Great Britain's generation mix but rarely occupy the marginal price-setting position, which raises the question of which flexible technologies translate a renewable-rich system into real-time price formation. This study reconstructs the price-ranked edge of the eligible bid or offer stack in the GB Balancing Mechanism across 50,684 half-hourly Settlement Periods from 2023 to 2025 and attributes it to individual Balancing Mechanism Units, separating long-system bid-active from short-system offer-active conditions and retaining co-marginal ties. Batteries rose from 0.8% to 36.2% of bid-active and from 3.2% to 26.9% of offer-active marginality, displacing combined-cycle gas on the bid side and pumped storage on the offer side. Capacity-normalised marginal capture rose on both sides, and unit-quarter fixed effects show that an additional 100 MW of Balancing Mechanism active capacity was associated with a 0.55 and 0.48 percentage-point increase in quarterly bid- and offer-side marginal share. By 2025, battery actions were around {\pounds}9 to 10/MWh more favourable than volume-matched non-battery alternatives drawn from the same ladder, yet batteries were under-represented by 12.9 percentage points in the highest-priced 5% of short-system periods. Individual batteries remain consistent with price-taking, while the fleet has become endogenous to routine balancing price formation.
Read the full abstract ▾
Statistics Applications · Cross-listed from stat.ME
A Unified Approach to Interpretable Causal Root Cause Attribution
Jing Zhou, Dominik Janzing, Sepp Tsang et al. (+2 more)
Understanding why a target metric changes is a fundamental problem in data-driven decision making, beyond anomaly detection alone. We study root cause attribution for metric changes in complex e-commerce systems, focusing on trade-offs between interpretability, efficiency, and causal validity. As a starting point, we extend a metric-decomposition method into a recursive metric-tree framework for multi-level root cause analysis, but this relies on independence and decomposability assumptions that miss complex causal dependencies. In contrast, graphical causal models (GCMs) relax these assumptions and improve causal validity, at the cost of interpretability, higher computational and data demands, and potential attribution target misalignment. Through real-world applications, mathematical proofs, and simulations, we characterize the fundamental sources of these trade-offs. Guided by these insights, we propose a unified, causally informed attribution approach that integrates structural causal information into the metric-tree decomposition framework and corrects key sources of misalignment in GCM-based causal attributions, substantially improving causal validity while preserving interpretability and fast computation. Analytical proofs and simulations demonstrate that the proposed approach produces more accurate root cause attributions, and we also present a real-world application.
Read the full abstract ▾
All authors Jing Zhou, Dominik Janzing, Sepp Tsang, Patrick Blöbaum, Marco Visentini Scarzanella
Statistics Applications
Text Data Analysis and Classification Methods - Insights from Customer Letters in Life Insurance
Andreas Groll, Marie Punsmann and Leonid Zeldin
The business of life insurance companies is characterized by long-term contracts. For this reason, data describing customers is of immense value. A portion of the data provided to the customer is rarely or not at all analyzed. This includes customer letters of any kind. This work focuses on classifying customer letters as cancellations and identifying the respective reason, if available. The outlined approach can also be applied to other business transactions and reasons. We discuss data acquisition and preparation, present alternatives, and explain the reasons for the chosen approach. A successful implementation of such a tool can lead to a better understanding of customer cancellation behavior by the insurer, enabling more targeted actions in certain situations.
Read the full abstract ▾
Statistics Applications
Scalable, Likelihood-Free Calibration of Ice-Sheet Models with Deep Diffusion Emulators and Feature Matching
Kanghyun Wi, Jaewoo Park, Saumya Bhatnagar et al. (+1 more)
The Antarctic ice sheet is a major source of uncertainty in future sea-level projections, and physical simulators such as the PSU3D-ICE model are essential for studying its
evolution. Calibrating them against observations is challenging: the simulator outputs and observed ice-thickness fields are high-dimensional, spatially dependent, and semi-continuous, with a large point mass at zero denoting ice-free regions. These features make conventional Gaussian-process emulation and likelihood-based calibration ill-suited and computationally infeasible at full resolution. We propose the sequential calibration method guided by a
diffusion model and a Siamese network (SC-DS), a fully neural framework. For emulation, we develop a single conditional
diffusion model that jointly generates the binary ice presence–absence pattern and the continuous thickness field, using global and local conditioning to represent how the input parameters shape the output. Because the emulator induces an intractable likelihood, our likelihood-free calibration replaces the hand-chosen distance and tolerance of approximate
Bayesian computation with a probabilistic acceptance rule learned by an iteratively retrained Siamese network, together with a data–model discrepancy adjustment. Applied to the West Antarctic Ice Sheet, SC-DS matches the accuracy of state-of-the-art Gaussian-process calibration at a fraction of its computational cost and scales to the full-resolution domain, where existing methods become intractable.
Read the full abstract ▾
All authors Kanghyun Wi, Jaewoo Park, Saumya Bhatnagar, Won Chang