Statistical Methodology
A Causal Inference Approach for Evaluating Diagnostic Tests and AI-Enabled Medical Devices: From Effect Modification to Information-Augmented Decision-Making
Wenxin Zhang, Rachael Phillips and Mark van der Laan
Diagnostic medical tests and devices provide useful information for evaluating the potential benefits and risks of therapeutic treatments. However, unlike treatments, their impact on health outcomes is generally indirect because measuring diagnostic information typically does not itself affect patient outcomes, which complicates evaluation of their effectiveness. In this work, we develop a causal inference approach for evaluating diagnostic tests by distinguishing explanatory and pragmatic effectiveness. Explanatory effectiveness evaluates whether a diagnostic test result provides treatment-relevant information beyond baseline
covariates by explaining additional treatment-effect heterogeneity, which we characterize using a variance-based treatment-effect variable importance measure. Pragmatic effectiveness evaluates whether incorporating this information into personalized treatment decisions improves expected outcomes, a question central to decision-making for patients, clinicians, and other health care stakeholders. We formalize pragmatic effectiveness as a contrast between expected outcomes under optimal personalized treatment rules defined with and without access to the diagnostic test result. We establish identification of the proposed estimand and provide Targeted Maximum Likelihood Estimation (TMLE) and cross-validated TMLE procedures for nonparametric estimation and inference. Simulation studies and a synthetic colorectal cancer application illustrate the proposed estimands and estimation performance. More broadly, this approach provides a causal inference perspective for evaluating AI-enabled devices by distinguishing their dual roles as information-enrichment and personalized decision-optimization tools, clarifying whether AI improves outcomes by expanding information for downstream decisions, improving the decision rule used to act on that information, or through both pathways.
Read the full abstract ▾
Statistical Methodology
Shape-Preserving Covariate Adjustment via Empirical Likelihood in Randomized Experiment
Zhilan Lou, Jun Shao, Yuhan Qian et al. (+4 more)
Covariate adjustment improves estimation efficiency in randomized experiments, but standard calibration and augmentation methods, when applied to distribution or survival functions, do not preserve monotonicity—a fundamental property of the estimand. We propose using empirical likelihood with
covariate-balancing constraints to construct a
covariate-adjusted empirical measure for each treatment arm.
Estimators of a broad class of distributional functionals, including cumulative distribution functions, survival functions,
quantiles, and restricted mean survival times, are then derived as plug-in functionals of this measure, automatically inheriting proper shape constraints. We establish asymptotic normality with an explicit, guaranteed efficiency gain over unadjusted
estimators. The asymptotic distributions are invariant to the randomization scheme, providing a unified inference procedure under simple randomization and all commonly used
covariate-adaptive designs satisfying a mild balancing condition. This unified construction, adjusting the empirical measure once and deriving all
estimators from it, offers a principled reconciliation of
covariate adjustment with shape preservation. Simulations and an application to the SURPASS-4 trial confirm the theoretical gains.
Read the full abstract ▾
All authors Zhilan Lou, Jun Shao, Yuhan Qian, Tuo Wang, Yanyao Yi, Yu Du, Ting Ye
Statistical Methodology
Causal Generalization of Continuous Treatment Effects under Covariate Shift
Jay Jojo Cheng and Guanhua Chen
Average dose-response functions are widely used to summarize causal effects of continuous treatments, but most existing methods assume that the observed sample represents the target population. We study a
covariate-shift setting in which
covariates, treatment, and outcome are observed in a labelled source sample, while only
covariates are observed in the target sample. We develop a two-sample local
polynomial regression framework based on pseudo-outcomes that use source outcomes to address confounding and target
covariates to define the population of interest. We further propose a source-to-target extension of distance covariance optimal weighting (DCOW), designed to remove treatment-
covariate dependence in the source sample while aligning the weighted source
covariate distribution with the target population. A central theoretical contribution is a weight-level analysis of this optimization-based procedure: we show that the population criterion identifies the oracle source-to-target weights and that approximate empirical minimizers, including exact minimizers as a special case, converge uniformly to these weights under regularity conditions. We also establish consistency and asymptotic normality of the resulting
estimator. Simulations show that the proposed method improves target dose-response estimation relative to DCOW, generalized-propensity-score weighting,
entropy balancing, and unweighted alternatives. We illustrate the method in a county-level analysis of PM2.5 exposure and subsequent heart-disease mortality using a source-target validation design.
Read the full abstract ▾
Statistical Methodology
Recovering Nonlinear Functions of Latent Variables: A Plausible-Value Neural Network Framework
Eunjeong Song and Sehee Hong
When factor scores replace true latent scores in nonlinear prediction, measurement error attenuates the recoverable variance of any $k$th-order component of the regression function by $\rho^k$ – the $k$th power of the score's coefficient of determination – for any linear score type. This study derives the bound via Hermite
polynomial expansion and proposes PV-ANN – plausible values (posterior draws preserving latent variance) combined with artificial neural networks (learning functional form without prespecification). The bound governs recovery of the latent-scale function, not prediction of the outcome from observed indicators, for which factor scores are already sufficient; the two metrics are therefore predicted to dissociate. An 18-condition simulation supports both predictions: in the nonlinear low-reliability conditions PV-ANN closes about four fifths of the function-shape recovery gap between a factor-score learner and one given the true latent values, and the margin widens as reliability falls, while predictive accuracy is not improved, as the theory requires. A Big Five application illustrates the intended exploratory workflow and delineates boundary conditions under weak signal and measurement model misspecification.
Read the full abstract ▾
Statistical Methodology · Cross-listed from stat.AP
Scalable Amortized Variational Inference for Non-Poisson Buy-'Til-You-Die Models
Sulagna Ghosh and Aaron Schein
Despite the wide variety of existing Buy-`Til-You-Die (BTYD) models, nearly all rely upon the convenient assumption of transactions following a Poisson process. As modern customer bases grow larger and more diverse, a major gap in the marketing literature is BTYD models that can account for heterogeneity in timing patterns across millions of customers. This paper addresses that gap, introducing a family of models that assume transactions follow a Weibull renewal process and developing a highly scalable scheme for parameter estimation based on an amortized variational inference procedure. The proposed model fits to a proprietary
dataset of 5 million online retail customers in 8 minutes which would take the current state-of-the-art an estimated 3-4 days. We show both theoretically and empirically that this dramatic improvement in computational performance comes with no appreciable change to either model interpretation or predictive performance. Beyond scalability, gradient-based variational inference also makes it easy to extend the model to
covariates, which we illustrate on a public
dataset of 4 million political donors during the 2020 US General election cycle. More generally, this paper demonstrates how to blend recent advances in approximate
Bayesian inference and the tools of modern machine learning to dramatically improve the efficiency and expressivity of probabilistic models for customer base analysis.
Read the full abstract ▾
Statistical Methodology · Cross-listed from econ.EM
Don't Drop the Singletons: Efficient Inference for Pairwise Experiments with Independent Attrition
Simon Heß and Patrick W. Schmidt
Pairwise randomization can yield substantial efficiency gains in experiments. Yet methodological guidance cautions against pairwise randomization, especially in settings with attrition, partly because common practices for estimation (i.e., pair fixed effects) imply discarding data from incomplete pairs thus exacerbating data loss from attrition. This practice of dropping incomplete pairs reduces statistical power of tests as well as precision of estimates, in paired experiments, compared to designs with less finely stratified treatment assignment. We argue that this concern is misplaced if attrition is independent of treatment status and potential outcomes, and that these issues follow from an inefficient use of the data that remains post-attrition. First, we show how, by using a specific permutation test, it is possible to use all observed units for inference (complete pairs and incomplete pairs where one unit attrits) while still exploiting the pairwise randomization design structure. The test procedure we suggest provides exact size control under the sharp null. Second, we study an optimally weighted
estimator that efficiently combines within-pair and across-pair comparisons. Finally, we show that combining these two insights yields a test procedure that dominates the two commonly used inference methods (a paired $t$-test and the two-sample $t$-test) in power, for any level of attrition. Usefully for applied researchers, we show that the efficient procedure can be implemented via a weighted fixed effects regression, straightforward in standard software. In sum, our results provide researchers with practical tools for conducting experiments with pairwise randomization without sacrificing observations or statistical power when facing independent attrition.
Read the full abstract ▾
Statistical Methodology
Mixed Membership Model of Low-rank Matrices with Multimodal Extension
David Snider, Zhongyuan Lyu, Jian Kang et al. (+1 more)
Matrix-valued observations arise in multiplex networks, neuroimaging, and other domains where population-level patterns are often low-rank and subjects may express several latent patterns simultaneously. Existing tensor PCA methods provide continuous subject scores but their loading
matrices can be difficult to interpret as population prototypes, while low-rank clustering yields interpretable prototypes with hard labels. We introduce a low-rank mixed membership model for
matrix-valued data in which the expected value of each subject's
matrix is a convex combination of latent low-rank basis
matrices. The model yields both interpretable population-level extreme profiles and continuous subject-level memberships. Our
multimodal extension shares memberships across modalities with modality-specific basis
matrices and can restore identifiability when one modality is insufficient. We establish identifiability under a pure-subject condition, propose a constrained least-squares
estimator and scalable
algorithm with spectral initialization and low-rank refinement, and derive nonasymptotic error bounds. The
estimator achieves a minimax-optimal reconstruction rate up to a logarithmic factor, with separate basis and membership convergence rates under a
geometric condition. Simulations corroborate the theoretical rates and show strong performance. In an analysis of Human Connectome Project functional connectivity data, the proposed method identifies interpretable
brain connectivity profiles whose estimated memberships are strongly associated with cognitive phenotypes.
Read the full abstract ▾
All authors David Snider, Zhongyuan Lyu, Jian Kang, Yuqi Gu
Statistical Methodology · Cross-listed from math.ST
A Composite Divergence Approach to Robust Multivariate Estimation under Cellwise and Casewise Contamination
Abhik Ghosh, Claudio Agostinelli and Ayanendranath Basu
Composite likelihood (CL) methods provide a computationally efficient alternative to full likelihood inference for complex multivariate models by replacing the joint likelihood with a product of lower-dimensional marginal or conditional components. Like the MLE, however, the maximum CL
estimator (MCLE) is highly sensitive to data contamination. On the other hand, robust divergence-based procedures such as the minimum density power divergence (DPD)
estimator require the full joint density and so scale poorly to complex multivariate models. We introduce the composite DPD (CDPD), a genuine statistical divergence built entirely from the low-dimensional component densities defining a CL, combining the computational scalability of CL with the robustness of the DPD. The resulting minimum CDPD
estimator (MCDPDE) robustifies the MCLE without requiring integration over the full multivariate sample space. We establish consistency, asymptotic normality, and the influence function of the MCDPDE under regularity conditions on the component models alone, without requiring correct specification of the full joint distribution. We show that it is qualitatively robust for every positive value of its tuning parameter, unlike the MCLE recovered as the limit. Because its components can be chosen at the pairwise or
cell level, the framework guards simultaneously against casewise and cellwise contamination. Operating directly on component densities rather than elliptical distance structures, it extends robust inference beyond the elliptical models to which most existing cellwise-robust procedures are confined. We develop computational
algorithms implemented in the accompanying R package mvdpd. Simulation studies and real-data applications show that the MCDPDE achieves substantial robustness gains over the MCLE while retaining competitive efficiency under the assumed model.
Read the full abstract ▾
Statistical Methodology
Model-based bootstrap inference for Cox models after Lasso selection
Lena Schemet, Andreas Groll and Sarah Friedrich-Welz
Inference after variable selection in Cox regression is difficult because simple Wald-type intervals after selection can have poor finite-sample conditional coverage. We study a model-based bootstrap for inference after Cox-Lasso variable selection. The Cox-Lasso is fitted once to the original data to select a set of variables, after which an unpenalized Cox model is fitted using only those variables. Bootstrap samples are generated from a semiparametric plug-in Cox model specified by the coefficient estimate from this unpenalized Cox refit, the Breslow baseline cumulative hazard
estimator, and a plug-in censoring distribution. In every bootstrap sample, the selected variable set is kept fixed and only the unpenalized Cox model is refitted. Under oracle-type sparse-model assumptions and standard Cox model regularity conditions, we prove first-order bootstrap validity for this procedure. In the simulation scenarios considered, percentile and studentized bootstrap intervals showed improved conditional coverage relative to the bootstrap-Wald interval in several small- and moderate-sample settings. Their performance was broadly competitive with debiased intervals, although the comparison depended on signal strength, tuning, and selection stability. A SEER breast cancer example illustrates that the procedure can be implemented in a realistic survival analysis and provides interpretable uncertainty quantification for effects reported after variable selection.
Read the full abstract ▾
Statistical Methodology
Functional forms in joint models for longitudinal and time-to-event data: A practical guide with application and interpretation
Felix Boakye Oppong, Dimitris Rizopoulos, Thierry Gorlia et al. (+1 more)
Background: Joint models for longitudinal and time-to-event data are widely used in clinical research. However, the choice of functional form linking the biomarker trajectory to event risk is often treated as a technical detail, despite its importance for model assumptions and interpretation. Default specifications may fail to capture clinically relevant features of biomarker trajectories. Methods: We provide a structured overview of functional forms linking longitudinal and survival processes in joint models. We compare association structures including instantaneous effects (current value, slope, and acceleration), cumulative and change-based formulations, shared random effects, and variability-based associations. Using longitudinal white blood
cell measurements and overall survival data from the MIRAGE glioblastoma trial, we illustrate how different functional forms capture distinct features of biomarker trajectories and define different biomarker-risk relationships. Results: Instantaneous forms capture the biomarker's current level or short-term dynamics, whereas cumulative and change-based forms reflect longer-term exposure or trends. Variability-based structures quantify instability in the biomarker trajectory as an alternative prognostic signal. Association parameters depend on the functional form, biomarker scale, and time scale, and effect sizes are therefore not directly comparable. In the MIRAGE application, alternative functional forms produced different effect interpretations and, in some cases, different conclusions regarding the biomarker-risk relationship. Conclusions: The choice of functional form is a key modelling decision in joint models and determines the interpretation of the biomarker-risk association. Aligning the functional form with the scientific question is essential for valid interpretation and transparent reporting.
Read the full abstract ▾
All authors Felix Boakye Oppong, Dimitris Rizopoulos, Thierry Gorlia, Nicole Erler
Statistical Methodology · Cross-listed from cs.LG
GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models
Qi Qin, Jiajie Zhu, Dali Chen et al. (+6 more)
Tabular foundation models (TFMs) achieve strong performance through in-context learning, but context-dependent inference imposes substantial latency and
memory costs, hindering large-scale deployment. We propose GEAR (Generative Expansion and Real Anchoring), a modular two-stage framework that distills TFMs into lightweight MLP or tree-based predictors that can be deployed on commodity CPUs. Stage 1 uses synthetic
covariates solely as teacher-query locations and trains the student on soft TFM targets, expanding coverage beyond observed rows. Stage 2 re-anchors the student to the target distribution using real labels and out-of-fold teacher predictions, whitch avoids self-labeling leakage. We further derive a risk certificate characterizing the trade-off between generated-query volume and generator fidelity. Experiments on TALENT and TabArena demonstrate the broad applicability of GEAR. Two-stage MLPs outperform supervised MLPs by 1.81–2.00 AUC points on binary tasks and 1.19–1.35 points on multiclass tasks, with additional gains over real-data-only
distillation of 1.76–2.19 and 2.09–2.40 points, respectively. On binary tasks, the gains also transfer to LightGBM and XGBoost, and all three student families outperform CatBoost, the strongest non-TFM baseline, in mean AUC. Ablations show gains beyond longer training or alternative warm starts, greater stability from staged than mixed optimization, and generator-dependent diminishing returns as query volume increases. Finally, GEAR reduces median inference time by 57–2866 times and peak prediction
memory by 1.9–3.3 times, while retaining higher AUC than matched supervised baselines.
Read the full abstract ▾
All authors Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Peng Zhang, Ying Yan, Yifan Sun, Yu Su
Statistical Methodology
Regularised Iterative Generalised Least Squares with Optimal Selection of the Hyper-Parameter for Identifying Nonlinear Phenomenological Models
Mark Cary and Charles Bokor
In some fields currently dominated by empirical approaches, such as state of health (SoH) prediction for lithium-ion batteries, phenomenological models motivated by quasi-physical thinking contain parameters to be estimated from experimental data. Often the structure of such models yields fully or partially confounded parameters, which are difficult or even impossible to estimate reliably. To preserve the desired model formulation and simultaneously improve the numerical conditioning for the problem we introduce a ridge regression scheme. An automated method is provided, based on information theoretic measures of model performance, which optimises the ridge regression hyper-parameter at each iteration. The formulae presented require fixed point iteration to solve for the hyper-parameter. Given a suitable starting value, analysis demonstrates convergence is very rapid. The optimal hyper-parameter selection mechanism is incorporated within an efficient regularised iterative generalised least squares mechanism, capable of fitting both heteroscedastic and serially correlated data as required. Simulation confirms the efficacy of the overall method.
Read the full abstract ▾
Statistical Methodology
Robust Modeling of Extremes in the Presence of Inliers with Enhanced Tail Estimation
Shivshankar Nila, Ishapathik Das and N. Balakrishna
Extreme value theory provides a fundamental framework for modeling rare and extreme events; however, threshold selection remains a persistent challenge, particularly in the presence of inliers such as instantaneous or early failures. Such observations commonly arise in applications including reliability studies and environmental data, where clusters of observations near the origin or at the origin can substantially distort classical threshold selection procedures and tail inference. In this paper, we propose a robust modeling framework that accounts for inliers, extremes, and the tail proportion. Parameter estimation is carried out using maximum likelihood. The proposed methodology is compared with classical numerical and graphical diagnostic tools, including the mean excess plot, parameter stability plot, Hill plot, and Pickands plot, as well as existing extreme value mixture models. The theoretical properties of the proposed model are established, and its performance is evaluated through extensive Monte Carlo simulations and real-data applications. The results demonstrate that the proposed methodology provides more accurate threshold estimation and more reliable extreme-value inference in the presence of inliers compared with existing classical approaches. Overall, the proposed methodology provides more accurate threshold estimation and tail inference in the presence of inliers, addressing key limitations of existing methods.
Read the full abstract ▾
Statistical Methodology
gridcp: Fast Online Changepoint Detection in Python
Per August Jarval Moen, Sebastian Grau Nielsen, Espen Bjørge Urheim et al. (+2 more)
Online changepoint detection is the problem of detecting distributional changes in a data stream in real-time. A large body of methodology exists for the offline (fixed-size) setting, but applying these methods online quickly becomes infeasible since the per-observation computational cost and
memory consumption typically grow at least linearly with the sample size. A recently proposed grid-based methodology (Moen, 2026) overcomes this by evaluating an offline test statistic over a sparse
geometric grid of split points, with grid points spaced increasingly far apart further in the past. For a wide class of test statistics, this approach keeps update time and
memory consumption growing logarithmic in the length of the data stream, while admitting finite-sample guarantees on the detection delay. Building on this methodology, we present gridcp, an open-source Python package that turns offline changepoint tests into efficient online detectors through a single, uniform interface. Users can choose from nine Numba-accelerated built-in tests, spanning changes in the mean, variance, covariance, and regression coefficients, as well as nonparametric tests and generalized likelihood-ratio tests for exponential-family models. Users can also supply their own test, which the package handles identically. For any test, gridcp provides Monte Carlo routines that calibrate the detection threshold to a target false alarm probability or average run length, including data-driven variants when no parametric null model is available. Through simulations and three real-data case studies, we show that calibration is accurate, that runtime scales favorably with both stream length and dimension, and that the complete pipeline, from calibration to deployment, runs efficiently on long real-world streams with only a short detection delay.
Read the full abstract ▾
All authors Per August Jarval Moen, Sebastian Grau Nielsen, Espen Bjørge Urheim, Martin Tveten, Ingrid Kristine Glad
Statistical Methodology · Cross-listed from cs.LG
ProxyGuard: Direct Reliability Inference for Randomized Data Release Mechanisms with Shared Targets
Dipesh Tharu Mahato and Pramod Dhungana
Researchers often choose a proxy
dataset from many releases, transformations, or seeds. Search can make an invalid release appear adequate, while one adequate release does not establish that its generator is reliable. ProxyGuard controls both errors using prespecified bounded risks and a sealed target set. Named-release mode corrects for multiplicity and certifies specific releases. Direct shared-target mode evaluates independent mechanism draws on a common target, lower-bounds their favorable-score rate, and subtracts a bound on favorable scores contributed by invalid releases. Conditional on the target, release scores are independent, yielding a finite-sample mechanism-reliability guarantee without independent target batches or assumptions on release-level $p$-value dependence. We show that the mean-only penalty is sharp and derive a smooth-score certificate with additive target concentration. In a registered three-requirement study, direct mode raises power from 5.6% to 64.2% at reliability 0.95, while named mode remains stronger under high-signal evidence. Prospective audits span full-pipeline Rice–TVAE, which retrains on every draw, and a non-tabular text mechanism.
Read the full abstract ▾
Statistical Methodology · Cross-listed from stat.CO
Convex Reparameterization and Self-Concordant Algorithms for Multivariate Regression with Covariance Estimation
Hongru Zhao and Huiqian Feng
Building on a reparameterization for multivariate linear regression that yields a jointly convex penalized likelihood in the reparameterized regression coefficient
matrix and the precision
matrix, we show that the resulting scaled Gaussian loss is standard self-concordant. This places the joint estimation problem within composite self-concordant optimization and leads to two
algorithms: a proximal gradient method and a damped proximal Newton method. In simulations, we evaluate algorithmic robustness, iterations to convergence, and elapsed time. In a
protein expression application, compared with the classical-parameterization formulation, the proposed convex formulation attains similar mean squared prediction error and can be substantially faster when the fitted precision
matrix is dense.
Read the full abstract ▾
Statistical Methodology
A seamless dose-optimization design for monotherapy and combination therapy
Kentaro Takeda and Masahiro Kojima
The emergence of molecular-targeted
agents and immune-oncology therapies has fundamentally transformed oncology drug development, necessitating
evolution beyond traditional dose-finding approaches designed for cytotoxic
agents. While conventional
agents exhibit predictable monotonic dose-response relationships, novel anticancer
agents often demonstrate plateau-effect patterns where higher doses may compromise therapeutic benefit, requiring identification of optimal biological doses that balance efficacy and tolerability. The FDA's Project Optimus initiative emphasizes comprehensive dose optimization through parallel randomized cohorts and patient backfilling to better understand pharmacological profiles across multiple dose levels. Contemporary drug development increasingly prioritizes combination therapy alongside monotherapy evaluation, yet existing designs typically assume equivalent roles for both
agents, diverging from clinical practice where novel
agents combine with established treatments having limited dose options. This paper proposes a seamless dose-optimization design that adaptively evaluates both monotherapy and combination therapy based on efficacy and toxicity outcomes through adaptive subtrials with patient backfilling capabilities. The model-assisted framework employs predetermined
Bayesian optimal boundaries, eliminating real-time model fitting while accommodating evaluation of both monotherapy and combination therapy and enabling sequential enrollment with strategic backfilling. Simulation studies demonstrate robust performance across diverse dose-response patterns relevant to contemporary oncology. The design addresses critical gaps between methodological assumptions and clinical reality, offering a practical approach that integrates monotherapy and combination therapy evaluation with efficacy-toxicity-based backfilling.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.CE
Multi-Level Bayesian Calibration of a Multi-Component Dynamic System Model
Berkcan Kapusuzoglu, Sankaran Mahadevan, Shunsaku Matsumoto et al. (+2 more)
This paper proposes a multi-level
Bayesian calibration approach that fuses information from heterogeneous sources and accounts for uncertainties in modeling and measurements for time-dependent multi-component systems. The developed methodology has two elements: quantifying the uncertainty at component and system levels, by fusing all available information, and corrected model prediction. A multi-level
Bayesian calibration approach is developed to estimate component-level and system-level parameters using measurement data that are obtained at different time instances for different system components. Such heterogeneous data are consumed in a sequential manner, and an iterative strategy is developed to calibrate the parameters at the two levels. This calibration strategy is implemented for two scenarios: offline and online. The offline calibration uses data that is collected over all the time-steps, whereas online calibration is performed in real-time as new measurements are obtained at each time-step. Analysis models and observation data for the thermo-mechanical behavior of gas turbine engine rotor blades are used to analyze the effectiveness of the proposed approach.
Read the full abstract ▾
All authors Berkcan Kapusuzoglu, Sankaran Mahadevan, Shunsaku Matsumoto, Yoshitomo Miyagi, Daigo Watanabe
Statistical Methodology
Centroid-Referenced Mahalanobis Matching (CRM): A Scalable, Representation-Based Framework for Causal Inference in Large Observational Studies
Keming Hu and Yingpei He
Matching for causal inference can be computationally expensive at scale and can silently change the target population when overlap is limited. We propose Centroid-Referenced Mahalanobis Matching (CRM), which replaces global pairwise search with stratified sampling in two reference coordinates: each unit's Mahalanobis distance from the treated centroid and its Fisher coordinate along the treated-control mean shift. All
covariates enter through the treated covariance geometry; CRM is therefore not principal-component preprocessing followed by nearest-neighbor matching. For $n$ units and $p$ pretreatment
covariates, its implemented cost is $O(np^2+p^3+n\log n)$, simplifying to $O(np^2+n\log n)$ when $n \ge p$. We derive an error decomposition separating representation, support, discretization, and
stochastic components. A pre-matching shortage fraction $\hat{\pi}$ estimates the population support restriction $\pi$, which enters a gap bound under bounded treatment-effect heterogeneity. Final retention is reported separately for capacity-driven exclusions. Under representation sufficiency, smoothness, and adequate
cell capacity, CRM has a conservative two-dimensional histogram mean-squared-error bound $O(n_T^{-1/2})$; representation sufficiency is an additional assumption, not a consequence of ignorability given the original
covariates. On Criteo, CRM retains at least 99.4% of treated units, has lower MaxSMD than corrected propensity-score matching in 31 of 36 large-scale configurations, and is roughly an order of magnitude faster. Moderate-size simulations favor some pairwise and weighting baselines on balance, locating CRM's contribution in scalability and explicit support diagnostics rather than universal finite-sample dominance.
Read the full abstract ▾