Peer Revenue Now accepting positions

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Sunday 12 April

Statistics Applications · Cross-listed from econ.GN

Unveiling contrasting impacts of heat mitigation and adaptation policies on U.S. internal migration

While climate-induced population migration has received rising attention, the role played by human climate endeavors remains underexplored. Here, we combine machine learning with attribution mapping to analyze the impacts of 4,713 heat-related policies (HPs) on 11,177 migration flows between U.S. counties. We find that heat adaptation policies (APs) and heat mitigation policies (MPs) have significant and opposing impacts on internal migration: APs reduce out-migration, while MPs increase it. These policies have heterogeneous effects on migration among policy types. Behavioral and cultural MPs at origins lead to a 0.24%-0.68% (95% confidence interval) increase in annual outflows per policy, whereas behavioral and cultural APs at destinations elevate outflows of origins by 0.11%-1.55% (95% confidence interval). Migration patterns are nonlinearly moderated by income, ageing, education, and racial diversity of both origin and destination counties. Ageing rates have the most noticeable U-shaped relationship in shaping migration responses to behavioral and cultural MPs at origins, and inverted U-shapes for institutional MPs at origins and nature-based MPs at destinations. These findings offer critical insights for policymakers on how HPs influence migration as global warming and policy interventions persist.
Read the full abstract

All authors Chao Li, Xing Su, Chao Fan, Yang Li, Luping Li, Chunmo Zheng, Wenglong Chao, Leena Jarvi, Han Lin, Juan Tu

Other Statistics

On Some Multivariate Extensions to Zenga Curve: Properties and Applications

Measures of inequality are often limited in their ability to capture multidimensional aspects that arise from the joint distribution of multiple socio-economic variables. In this paper, we develop bivariate extensions of the Zenga inequality measure using bivariate quantile functions. We propose new bivariate Zenga surfaces and study their theoretical properties. A vector-valued bivariate Zenga curve is also introduced to provide a more detailed characterization of inequality. A non-parametric estimator is proposed and methods are evaluated through simulation studies and applied to the analysis of digital inequality across countries using indicators such as broadband penetration and digital literacy. The results highlight the effectiveness of the proposed framework in capturing multidimensional inequality.
Read the full abstract

Statistical Methodology

The Fréchet correlation coefficient for heterogeneous random objects

Modern regression analysis often involves responses and predictors taking values in the same or distinct metric spaces. To rank non-Euclidean heterogeneous predictors in regression by explanatory strength, analogous to the classical $R^2$, we introduce the Fréchet correlation coefficient (FCC), defined as the relative reduction in the Fréchet variance of the response after conditioning on a specific predictor. FCC is directional, model-free, and interpretable on a unit-scale, attaining one under almost sure functional dependence and zero when the Fréchet mean is invariant to conditioning. We propose a novel partition-based estimator that avoids explicit nonparametric estimation of the conditional Fréchet mean function, thereby improving both computational efficiency and flexibility. A tailored wild bootstrap algorithm is further developed for testing the Fréchet conditional mean dependence. We establish asymptotic theory and evaluate power through extensive simulations.
Read the full abstract

Machine Learning (Statistics)

Orthogonal machine learning for conditional odds and risk ratios

Conditional effects are commonly used measures for understanding how treatment effects vary across different groups, and are often used to target treatments/interventions to groups who benefit most. In this work we review existing methods and propose novel ones, focusing on the odds ratio (OR) and the risk ratio (RR). While estimation of the conditional average treatment effect (ATE) has been widely studied, estimators for the OR and RR lag behind, and cutting edge estimators such as those based on doubly robust transformations or orthogonal risk functions have not been generalized to these parameters. We propose such a generalization here, focusing on the DR-learner and the R-learner. We derive orthogonal risk functions for the OR and RR and show that the associated pseudo-outcomes satisfy second-order conditional-mean remainder properties analogous to the ATE case. We also evaluate estimators for the conditional ATE, OR, and RR in a comprehensive nonparametric Monte Carlo simulation study to compare them with common alternatives under hundreds of different data-generating distributions. Our numerical studies provide empirical guidance for choosing an estimator. For instance, they show that while parametric models are useful in very simple settings, the proposed nonparametric estimators significantly reduce bias and mean squared error in the more complex settings expected in the real world. We illustrate the methods in the analysis of physical activity and sleep trouble in U.S. adults using data from the National Health and Nutrition Examination Survey (NHANES). The results demonstrate that our estimators uncover substantial treatment effect heterogeneity that is obscured by traditional regression approaches and lead to improved treatment decision rules, highlighting the importance of data-adaptive methods for advancing precision health research.
Read the full abstract

Statistical Methodology

Estimating heterogeneous treatment effects with survival outcomes via a deep survival learner

Estimating heterogeneous treatment effects in survival settings is complicated by right censoring as well as the time-varying nature of the estimand. While the conditional average treatment effect (CATE) provides a natural target, most existing approaches focus on a single prespecified time point and do not account for the temporal trajectory, leading to instability in estimation. We propose a deep survival learner (DSL) for estimating heterogeneous treatment effects with right-censored outcomes. The method is based on a doubly robust pseudo-outcome whose conditional expectation identifies time-specific CATEs under standard assumptions. This construction remains unbiased if either the outcome model or the treatment assignment model is correctly specified, when properly accounting for censoring. To estimate CATEs over a clinically relevant time spectrum, DSL employs a multi-output deep neural network with shared representations, enabling joint estimation of treatment effect trajectories. From a theoretical perspective, we derive error bounds for both pointwise and joint estimation over time. We show that joint estimation can leverage temporal structure to control estimation error without incurring much additional approximation cost under smoothness conditions, leading to improved stability relative to separate estimation. Cross-fitting is incorporated to reduce overfitting and mitigate bias arising from flexible nuisance estimation. Simulation studies demonstrate favorable finite-sample performance, particularly under nuisance model misspecification. Applied to the Boston Lung Cancer Study, DSL reveals heterogeneity in the effects of perioperative chemotherapy across patient characteristics and over time.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

A remark on the comparison of the sum and the maximum of positive random variables

We disprove a conjecture stated in a recent paper by Arnold and Villasenor concerning the sum and the maximum of independent and identically distributed half-normal random variables. Our method is applicable to generalized gamma distributions.

Saturday 11 April

Statistics Applications · Cross-listed from q-fin.RM

When AAA Satisfies Nothing: Impossibility Theorems for Structured Credit Ratings

A credit rating of AAA asserts near-certainty of repayment. This paper asks whether the pre-crisis information environment could have supported that assertion for structured products. Bayes' theorem implies that any reliability target requires a minimum level of statistical discrimination between instruments that will repay and those that will not. At structured-finance base rates, a four-nines reliability target demands discrimination on the order of 10,000 to 1. A three-nines target demands 1,000 to 1. Nothing in the published credit-prediction literature provides an affirmative basis for believing that discrimination of this magnitude was achievable with the data available at rating time. Retrospectively, the realized system fell short of the four-nines benchmark by roughly 90,000-fold. The framework accommodates the historical feasibility of corporate AAA ratings, where high base rates and rich information produce low required discrimination. Illustrative calibrations for contemporary collateralized loan obligations suggest that material tension between the precision target and the information environment persists. The central implication is that the AAA precision claim itself likely exceeded what the available information could support.
Read the full abstract

Statistics Theory

Spectral analysis of multivariate stationary Hawkes processes

We establish the asymptotic validity of frequency-domain inference for stationary multivariate Hawkes processes under mild conditions, bridging the gap between theory and application. By developing upper-bounds on the reduced cumulant measures from the cluster representation of the Hawkes processes, we prove a functional central limit theorem and, as a consequence, consistency of the Whittle estimator under stationarity alone (i.e., the spectral radius of the interactions matrix $\rho(\boldsymbol\nu)<1$), applicable to Hawkes processes with heavy-tailed mutual-excitation kernels. Under mild extra moment conditions, we further obtain asymptotic normality with an explicit limiting covariance in terms of second- and fourth-order cumulant spectral densities. We also propose a simple frequency-domain method to detect joint independence of subprocesses of a multivariate Hawkes process. The performance of the Whittle estimator and the test of independence are demonstrated via simulation studies.
Read the full abstract

All authors Yifu Tang, Conor Kresin, Boris Baeumer, Ting Wang

Statistics Applications · Cross-listed from q-fin.RM

On the Structure of Risk Contribution: A Leave-One-Out Decomposition into Inherent and Correlation Risk

This paper develops a decomposition of standard Risk Contribution (RC) into two economically interpretable components: inherent risk and correlation risk. Using a leave-one-out representation, each position's RC separates into a term reflecting its own volatility contribution independent of the portfolio and a term capturing its covariance with the remainder of the portfolio. The inherent component is always positive, arising from the intrinsic volatility of the position, while the correlation component may amplify or mitigate total portfolio risk depending on how the position moves relative to other holdings. Because the decomposition operates within standard RC, it preserves the property of strict additivity. This separation provides diagnostic insight not visible from aggregate risk contributions alone. It distinguishes whether a position contributes risk because it is volatile in isolation or because it is highly correlated with the rest of the portfolio, and it clarifies when a negatively correlated position functions as an effective hedge. Two approaches to time-series analysis are presented to track how inherent and correlation risk evolve across market regimes, revealing whether changes in portfolio risk during stress periods are driven by volatility shocks, correlation shifts, or both. Empirical illustrations suggest that the decomposition provides stable, transparent, and easily implementable risk diagnostics that can support portfolio risk reporting, stress testing, and performance attribution.
Read the full abstract

Statistical Methodology

Uncertainty Quantification for Noisy Low-tubal-rank Tensor Completion

High-dimensional tensor data often exhibit strong temporal correlations that appear as low-dimensional structures in the frequency domain. While the low-tubal-rank tensor model effectively captures these spectral features, making it potentially suitable for geophysical data, existing methods primarily focus on point estimation. Uncertainty quantification (UQ) of imputed values and rigorous statistical inference for these models remain largely unexplored. In this work, we propose a flexible inference framework for linear forms of high-dimensional tensors. Employing a double-sample debiasing technique followed by a low-rank projection, we construct asymptotically Gaussian estimators that yield valid statistical inference under mild assumptions. More precisely, we can perform hypothesis testing and construct confidence intervals with this result. We validate the theoretical results through extensive simulations and demonstrate the practical effectiveness of our method in completing the global total electron content data. We demonstrate, using those numerical results, that our entrywise confidence intervals are robust and reliable, yielding informative uncertainty quantification that captures underlying variability.
Read the full abstract

Statistics Theory · Cross-listed from math.PR

Weak convergence from projected laws on a positive-measure set of directions

The Cramér-Wold device characterises weak convergence of probability measures on $\mathbb{R}^d$ through convergence of all one-dimensional projected laws. We prove that, if the target projected laws are moment-determinate for surface-almost every direction, then weak convergence already follows from projected convergence on a positive-measure set of directions. This yields a simple probabilistic interpretation: if one samples a direction at random from any distribution on the sphere that is absolutely continuous with respect to surface measure, then, with probability one, convergence of the projected law along the sampled direction already forces global weak convergence under the same moment-determinacy assumption.
Read the full abstract

Statistical Methodology

Considerations for the Integration of Randomized Controlled Trials and Real-World Data

As clinical decision-making increasingly moves toward individualized and context-specific treatment recommendations, reliance on any single evidence source, randomized or observational, may be insufficient. Principled integration of randomized controlled trials and real-world data, grounded in explicit causal frameworks, offers a path toward evidence that is both internally credible and externally relevant. In this article, we describe distinct objectives for the integration of randomized controlled trials and real-world data and discuss how these objectives shape key design and analytic considerations, illustrating the resulting choices through example estimands. We highlight practical issues that commonly arise in applied settings, including data relevance and curation, cross-source comparability, estimand specification, and sensitivity analysis. We aim for this article to help readers evaluate and implement principled approaches to integrating randomized controlled trials and real-world data in ways that can support more reliable treatment recommendations while maintaining regulatory-grade evidentiary standards.
Read the full abstract

All authors Sky Qiu, Charles Barr, Lauren Dang, Larry Han, Kajsa Kvist, Hana Lee, Andrew Mertens, Nerissa Nance, Lei Nie, Kara Rudolph, Xu Shi, Jens Tarp, Salina P. Waddy, Kenneth Wiley, Andy Wilson, Margot Lisa Jing Yann, Zhiwei Zhang, Tianyue Zhou, Maya Petersen, Mark van der Laan

Statistical Methodology

Gaussian Graphical Models for Functional Connectivity Analysis: A Statistical Review with Applications to Alzheimer's Disease

Functional connectivity analysis is an important tool for characterizing interactions among brain regions, particularly in studies of neurodegenerative disorders such as Alzheimer's disease (AD). Gaussian graphical models (GGMs) provide a promising statistical framework for estimating functional connectivity by capturing conditional dependence relationships among brain regions. Although a variety of regularized precision matrix estimators have been proposed to estimate sparse conditional dependency structures for GGMs, their comparative performance and practical implications for neuroimaging studies are not well understood. In this work, we present a comprehensive statistical review and empirical evaluation of widely used GGM estimation methods, including the graphical lasso (glasso), ridge-based glasso, graphical elastic net, adaptive glasso, smoothly clipped absolute deviation (SCAD), minimax concave penalty (MCP), constrained $\ell_1$ minimization for inverse matrix estimation (CLIME), and tuning-insensitive graph estimation and regression (TIGER). Their performance is evaluated through extensive data-driven simulations designed to reflect realistic neuroimaging settings, along with an application to an AD cohort study to illustrate methodological differences and their impact on downstream network analysis. In addition, a user-friendly R package, spice, is provided to facilitate implementation and enhance the reproducibility of empirical studies.
Read the full abstract

All authors Panpan Zhang, Shiying Xiao, W. Hudson Robb, Dandan Liu, Angela L. Jefferson, Jun Yan

Statistics Theory · Cross-listed from econ.EM

Maximum score and non-smooth M-estimators with multiway dependence

Under i.i.d. sampling, the maximum score estimator of Manski (1975) converges at the cube-root rate to a non-Gaussian limit. We show that, under multiway dependence and low-level conditions, it is asymptotically normal at a parametric rate in the smallest number of clusters and, when the cluster-level variance is non-degenerate, as fast as regular estimators such as probit. The result rests on a general theory for non-smooth M-estimators, under which a modified bootstrap is first-order valid given a consistent Hessian estimate. With two-way clustering, a cube-root term can remain; under high-level conditions, symmetric bootstrap intervals for suitable scalar parameters still control size uniformly.
Read the full abstract

Statistics Theory

Normalized Likelihood Criteria for Model Selection in the Stochastic Block Model

Estimating the number of communities is a fundamental problem in network analysis under the stochastic block model (SBM). In this paper, we study penalized estimators for this task based on normalized likelihood criteria. We show that a penalized estimator derived from the Normalized Maximum Likelihood (NML) is strongly consistent with a logarithmic penalty term, although its computation is intractable. To overcome this limitation, we consider the Normalized Maximum Complete Likelihood (NMCL) and the Decomposed Normalized Maximum Likelihood (DNML). The DNML admits an explicit formulation with cubic computational complexity in the number of nodes. We prove that the NMCL- and DNML- based estimators are strongly consistent for sparse networks in which the average node degree diverges with the network size. Empirical results show that the DNML estimator performs competitively with existing methods, particularly in unbalanced networks.
Read the full abstract

Statistical Methodology

Bayesian Distance-to-Set Models: from Latent Variable to Latent Projection

Statistical models often assume that data are generated near a structured, smooth, or low-dimensional set. A common approach is to use Bayesian latent variable models, in which each observation is associated with a latent coordinate on the set, and the observed data are modeled as noisy deviations from these coordinates. The deviation is typically characterized by a location-scale distribution, such as Gaussian. Despite their intuitive appeal and popularity, latent variable models often present practical challenges in posterior computation. In particular, Markov chain Monte Carlo samplers may suffer from slow mixing, especially when the sample size is large and there is no closed form for integrating out the latent coordinates. In this article, we propose an alternative approach that replaces the deviation-from-coordinate with a distance-to-set. Specifically, the distance-to-set is defined as the distance between a data point and its projection onto the set, where the projection can be rapidly computed by optimization and replaces the latent coordinate in the likelihood. This change substantially reduces the dimensionality of the parameter X latent variable space, leading to efficient posterior computation. We establish several important statistical properties for the distance-to-set models, such as the independence between the normal-cone noise and fixed-effect parameters, posterior consistency, and an Occam's razor effect that automatically penalizes overfitting. We demonstrate the effectiveness of our approach through simulation studies, applications to multi-environment study and Bayesian transfer learning.
Read the full abstract

Statistical Methodology

Cox Model Predicting Covariate Subject to Right Censoring

Time-to-event endpoints are frequently used as outcomes in oncology and other disease areas where the outcome of interest may not be observed within a predetermined period. Although many analytical methods address the challenges of censoring in outcomes, limited research has focused on censored covariates. Conventional methods such as the complete case (CC) analysis, where data from patients with censored covariates are discarded, suffer from efficiency loss and potential bias due to reduced sample size. Alternatively, imputing censored covariates with a constant value can underestimate variability. Recognizing these limitations, novel estimation procedures within the generalized linear model framework have been proposed, with some research emerging in time-to-event outcomes. In this paper, we investigate the association between progression-free survival and overall survival using a semi-parametric Cox model framework. We modify the Cox model's partial likelihood function to account for censored covariates by replacing the relative risk associated with censored covariates with a weighted average of patients with observed covariates. The performance of the proposed method is demonstrated through simulations and applications to two oncology clinical trials. Results indicate that the proposed method offers improved estimation efficiency and better utilization of available data compared to other approaches.
Read the full abstract

Statistical Methodology

Inference from multivariate differential recruitment in respondent-driven sampling data

Respondent-Driven Sampling (RDS) is a chain-referral design used for collecting data from hidden or hard-to-reach populations through their social networks. In RDS, respondents recruit their peers from the population of interest. As such, inference with RDS data commonly relies on estimated sampling probabilities derived from specific recruitment assumptions. Early literature assumes random recruitment, which is often unrealistic because individuals may recruit based on their personal preferences. This behavior is known as Differential Recruitment (DR). Recent works have incorporated univariate categorical DR in the estimation procedures. The main objective of this paper is to introduce Multivariate Differential Recruitment (MDR), a framework that incorporates multiple simultaneous covariates, both categorical and continuous, into the sampling representation. We model RDS as a Markov process with transition probabilities that depend on continuous or categorical variables associated with nodes or their ties. We then extend various prevalence estimators to this multivariate framework and implement a slightly modified neighborhood bootstrap for variance estimation. The proposed methodology is assessed through simulation studies for a range of network and sampling features. It is applied to an RDS study conducted among the adult Venezuelan population living in the Metropolitan Region of Santiago, Chile.
Read the full abstract

All authors Vanesa Reinoso, Danilo Alvares, Jonathan Acosta, Isabelle S. Beaudry

Statistical Methodology

Toward Exact Convergence in Byzantine-Robust Decentralized Learning: A Statistical Identification Approach

To defend against Byzantine attacks in decentralized learning, most existing methods rely on robust aggregation rules to mitigate the influence of malicious machines. However, these strategies inherently introduce bias, leading to inexact convergence with non-vanishing steady-state errors. In this paper, we propose a strategic shift from passive aggregation to active identification by introducing the Decentralized Rescaled Stochastic Gradient Descent with Byzantine Machine Identification (DRSGD-ByMI) framework. The core of our approach is an identification-based “detect-then-optimize” pipeline, where a p-value-free detection procedure is developed to accurately prune malicious nodes from the network. By leveraging sample-splitting score statistics, this identification mechanism achieves false discovery rate control without requiring restrictive distributional assumptions. We theoretically demonstrate that this precise identification allows the decentralized network to recover sufficient connectivity among the normal nodes, thereby enabling DRSGD-ByMI to match, even in the presence of Byzantine machines, the same order-optimal convergence rate as standard decentralized stochastic first-order methods. Numerical experiments validate our theoretical results and demonstrate the effectiveness of DRSGD-ByMI for decentralized robust learning problems.
Read the full abstract

All authors Siyuan Zhang, Chengde Qian, Xin Liu, Changliang Zou

Friday 10 April

Statistics Applications · Cross-listed from q-fin.GN

Exploring Drivers of Extreme Housing Prices in Australia

In recent years Australia has observed a growing, unexplained resilience of increasing house price trends. Here, we seek to understand what is driving Australia's indestructible asset using insights from market experts. We construct a differential equation model of house price to develop intuition for its historical behaviour and responsiveness to changes in mortgage rates. Using this model, we identify a point of 'decoupling' between house price and mortgage rate in the system with supply limitations found to be the main driver for this change. From there, modern extreme value techniques are implemented on real-world data to investigate how the effectiveness of mortgage rate in moderating extreme house price has changed before and after this historical decoupling. We find that without an increase in the housing supply chain, through either deregulation or reduced competition with government building, an 11% increase in mortgage rate will be needed to slow extreme housing costs.
Read the full abstract

Statistics Theory

Bayesian bivariate survival estimation

There is no easy extension of Kaplan-Meier and Nelson-Aalen estimators to the bivariate case, and estimating bivariate survival distributions nonparametrically is associated with various non-trivial problems. The Dabrowska estimator will for example associate negative mass to some subsets. Bayesian methods hold some promise as they will avoid the negative mass problem, butare also prone to difficulties. We simplify and extend an example by Pruitt to show that the posterior distribution from a Dirichlet process prior is inconsistent. We construct a different nonparametric prior via Beta processes and provide an updating scheme that utilizes only the most relevant parts of the likelihood, and show that this leads to a consistent estimator.
Read the full abstract

All authors J.K. Ghosh, Nils Lid Hjort, C. Messan, R.V. Ramamoorthi

2604.11819 · 10 Apr · 15 pages, 0 figures. This is a 2005 technical report, with some more material than in the published version (Journal of Statistical Planning and Inference, vol. 136, 2006, pages 2297-2308). NLH honours JK Ghosh (1937-2017) and RV Ramamoorthi (1950-2023) by this arXiv version, for better visibility and easier access

Statistics Theory

Confidence envelopes for the false discoveries with heterogeneous data

In the context of selective inference, confidence envelopes for the false discoveries allow the user to select any subset of null hypotheses while having a statistical guarantee on the number of false discoveries in the selected set. Many constructions of such envelopes have been proposed recently, using local test families (Genovese and Wasserman, 2006; Goeman and Solari, 2011), paths (Katsevich and Ramdas, 2020) or interpolation (Blanchard et al., 2020a). All those methods have in common that they have been well-studied for the homogeneous case where all p-values under the null have a uniform distribution over [0, 1]. However, in many applications the data are heterogeneous and discrete, hence the p-values have heterogeneous, discrete distributions, and the previous constructions may incur a loss of power, in the sense that they over-estimate the number of false discoveries. In this paper, we bridge the previous constructions under the homogeneous case with new tools. We also apply these tools to propose several confidence envelopes based on tools tailored for heterogeneous data, like the Bretagnolle inequality, or a new variant of the Simes inequality. We compare these new envelopes to their homogeneous counterparts on simulated data.
Read the full abstract

All authors Romain Périer, Gilles Blanchard, Sebastian Döhler, Guillermo Durand, Etienne Roquain

Statistical Methodology

Partial correlation networks of Gaussian processes

In Gaussian graphical models, conditional independence and partial correlations are natural inferential targets for understanding direct relationships in multivariate data. No comparable framework exists for spatial processes, where multivariate analysis defaults to modeling unconditional cross-covariance structure, even when direct relationships remain of scientific interest. We address this gap by establishing a novel characterization of process-level partial correlation for multivariate Gaussian processes that recovers a direct link with Gaussian graphical models. Our analysis proceeds through a class of stationary multivariate processes, termed spectrally inside-out, in which a precision matrix modulates the strength of conditional dependence and yields necessary and sufficient conditions for conditional independence. Within this class, partial cross-correlation functions factorize into a process-level partial correlation coefficient and an attenuation term independent of cross-process parameters. The spectrally inside-out class includes the separable coregionalization model, a process convolution construction, and the parsimonious multivariate Matérn, for which such a characterization was previously thought unavailable. We further show that a nonstationary inside-out model satisfies the same factorization and admits the same necessary and sufficient conditions. Our results clarify the limitations of existing approaches: linear coregionalization models encode conditional independence through the zero pattern of the inverse factor loading matrix and do not result in interpretable partial cross-correlation functions. Low-rank spatial factor models lack a meaningful graphical characterization. Methods that enforce network structure through auxiliary graphical layers only characterize presence or absence of graph edges. We illustrate our results through synthetic and real data.
Read the full abstract

Statistics Theory

On a copula product characterizing independence and perfect functional dependence

Recently studied dependence measures take values in [0,1], where 0 characterizes independence of X and Y, and 1 characterizes perfect functional (not necessarily monotone) dependence of Y on X. The most prominent example is Chatterjee's rank correlation, which is based on the concept of conditional independence. In contrast, we show that dependence measures such as Wasserstein correlations and rearranged dependence measures are naturally linked to the concept of conditional comonotonicity. This connection is captured by a new copula product T(C)=C\vee\Pi which models conditional comonotonicity and characterizes independence and perfect functional dependence. As a main contribution, we prove that the mapping T acts as a reflection on the class of stochastically increasing copulas. Further, T^2=T \circ T projects a copula onto its increasing rearranged copula. To better understand the behavior of such dependence measures, we also study fixed points, ordering results, and continuity properties of T. Our results demonstrate that conditional comonotonicity is a rather intrinsic feature of dependence measures, whereas conditional independence underlying Chatterjee's rank correlation is a quite exceptional property.
Read the full abstract

Statistical Methodology

Performance of weakly-supervised electronic health record-based phenotyping methods in rare-outcome settings

Accurately identifying patients with specific medical conditions is a key challenge when using clinical data from electronic health records. Our objective was to comprehensively assess when weakly-supervised prediction methods, which use silver-standard labels (proxy measures of the true outcome) rather than gold-standard true labels, perform well in rare-outcome settings like vaccine safety studies. We compared three methods (PheNorm, MAP, and sureLDA) that combine structured features and features derived from clinical text using natural language processing, through an extensive simulation study with data-generating mechanisms ranging from simple to complex, varying outcome rates, and varying degrees of informative silver labels. We also considered using predicted probabilities to design a chart review validation study. No single method dominated the other across all prediction performance metrics. Probability-guided sampling selected a cohort enriched for patients with more mentions of important concepts in chart notes. SureLDA, the most complex of the three algorithms we considered, often performed well in simulations. Performance depended greatly on selected tuning parameters. Care should be taken when using weakly-supervised prediction methods in rare-outcome settings, particularly if the probabilities will be used in downstream analysis, but these methods can work well when silver labels are strong predictors of true outcomes.
Read the full abstract

Statistical Methodology

Mixed Membership Models for Multilevel Functional Data

Mixed membership models extend classical clustering by substituting the notion of uncertain membership with the notion of mixed membership. In particular, these models allow each observation to partially belong to multiple pure membership classes. We discuss mixed membership models for functional data by extending the framework to multilevel functional observations. We show how the classical multivariate Karhunen-Loeve decomposition can be translated into a simple hierarchical model for scalable and flexible expressivity of the underlying stochastic processes. The identifiability of partial membership structures is aided by the definition of a hierarchical repulsive prior on the unitary simplex. Our work is motivated and illustrated by applications to a study on functional brain imaging through electroencephalography (EEG) of children with autism spectrum disorder (ASD).
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Last-Iterate Convergence of Randomized Kaczmarz and SGD with Greedy Step Size

We study last-iterate convergence of SGD with greedy step size over smooth quadratics in the interpolation regime, a setting which captures the classical Randomized Kaczmarz algorithm as well as other popular iterative linear system solvers. For these methods, we show that the $t$-th iterate attains an $O(1/t^{3/4})$ convergence rate, addressing a question posed by Attia, Schliserman, Sherman, and Koren, who gave an $O(1/t^{1/2})$ guarantee for this setting. In the proof, we introduce the family of stochastic contraction processes, whose behavior can be described by the evolution of a certain deterministic eigenvalue equation, which we analyze via a careful discrete-to-continuous reduction.
Read the full abstract

Statistical Methodology

crumble: A comprehensive framework for modern causal mediation analysis with intermediate confounding

Causal mediation analysis is widely used to investigate how causal effects operate through specific pathways linking treatments or exposures to outcomes. Recently, crumble was developed to enable nonparametric estimation of several mediation parameters, even when mediators are continuous and/or multi-dimensional or when treatments are non-binary. But a practical and accessible guide to using crumble – one that does not require deep familiarity with mediation analysis or semiparametric theory – is currently lacking. This tutorial aims to an accessible introduction to crumble while minimizing technical complexity. We first review the mediation parameters implemented in crumble – natural direct and indirect effects, randomized interventional effects, and recanting-twin effects. For each, we give the definition, interpretation, identification assumptions, and suitability in the presence or absence of intermediate confounding. Then, we demonstrate the usage of crumble by examining an example configuration. Next, we describe how crumble accommodates non-binary treatments through modified treatment policies. Finally, we illustrate the practical use of crumble through two case studies – one with a binary treatment and one with a non-binary treatment – based on the Job Search Intervention Study data.
Read the full abstract

All authors Richard Liu, Nicholas T. Williams, Kara E. Rudolph, Ivan Diaz

Statistics Applications

Evaluating the impact of longitudinal treatment strategies in the presence of informative monitoring and time-dependent confounding

Routinely collected data from electronic health records (EHR) provide opportunities to study effects of longitudinal treatment strategies in real-world clinical settings. A challenge presented by EHR data is that frequency of covariate monitoring differs by patient, covariate type and over time, and may be informative about a patient's health status. Many causal inference methods assume measurements of covariates are observed at a common set of regular time points. In this paper we describe and evaluate methods for estimating causal effects of longitudinal treatments on time-to-event outcomes in the presence of informative monitoring of time-dependent confounders. We show how methods based on inverse probability weighting, G-computation and longitudinal targeted maximum likelihood estimation (TMLE) can be adapted to allow for informative monitoring by incorporating monitoring indicator variables as additional time-dependent confounders. We evaluate these methods using a simulation study, comparing against more simple approaches that ignore monitoring variables. We demonstrate that ignoring monitoring can result in biased estimates of treatment effects. The methods are illustrated through an investigation into the effect of early versus delayed initiation of invasive mechanical ventilation on mortality of intensive care patients using routinely-collected data from an intensive care unit. We consider static treatment strategies such as `always treat' and `never treat' but also generalise to treatment strategies that allow for flexibility in the exact initiation time and duration of treatment.
Read the full abstract

All authors Leah Pirondini, Karla Diaz-Ordaz, Edward Palmer, Ruth H. Keogh

Statistics Applications

Blume-Capel model: Estimation of a three stable state network for $-\bf 1$, $\bf 0$ and $\bf +1$ data

An extension of the Ising model is proposed as a viable alternative for data with values $-1$, $0$ and $+1$ in the inverse problem, i.e., estimation of the parameters. This model is called the Blume-Capel (BC) model, adapted from physics for small networks. The advantage of the BC model is not only the fact that it is possible to have a neutral (centrist) position on the response scale, but also that this model allows for three stable states. We illustrate magnetisation properties of the BC model using simulations and mean field results. For estimation of the BC parameters, we show that the BC model is part of the exponential family of distributions and show that the model is identified, except for the (inverse) temperature. We then show that combining pseudo-likelihood with lasso yields accurate parameter recovery for the BC model, even in small networks. Moreover, confidence intervals with good coverage properties can be obtained using the desparsified lasso together with sandwich and shrinkage techniques. We apply the methods to data obtained from the online platform Stemwijzer, intended to aid people in deciding for whom to vote.
Read the full abstract

All authors Lourens Waldorp, Jonas Dalege, Maarten Marsman, Adam Finnemann, Irene Ferri, Han L. J. van der Maas