Statistical Methodology
EARL: Exposure- and Allocation-Reweighted Linear Estimator for Bipartite Experiments with Partial Assignment
Alexey Kurennoy
Bipartite A/B tests are experiments in which treatment is randomised over one set of units, while outcomes are measured on another. For example, an online marketplace may test a new pricing
algorithm on a random subset of items, while the outcome of interest, say, purchase satisfaction, is measured on customers, each of whom interacts with many items. Existing methods for analysing bipartite experiments assume that every randomisation unit is assigned to treatment or control. In practice, often only a subset participates: platforms cap rollout risk, reserve holdout groups, and split their population across concurrent tests. Ignoring the unassigned units biases estimation, while including them requires care. We construct an unbiased linear
estimator for bipartite experiments with partial assignment. Observations must be reweighted not only by the (centred) share of treated connections among participating ones (the exposure) but also by the number of participating connections, each inverse-weighted by its participation propensity, so that units whose connections are well covered carry proportionally more weight. The resulting
estimator, EARL (Exposure- and Allocation-Reweighted Linear), is unbiased, consistent, and asymptotically normal; we devise two asymptotic variance
estimators and show that it has minimal variance in a natural class of linear
estimators. EARL remains unbiased regardless of the experience the unassigned units receive, as long as they contribute to expected outcomes additively; in particular, they can be allocated to other, non-overlapping tests. Our theory is complemented with a simulation study on two public
datasets, in which EARL attains up to six times lower error than the strongest existing baseline and, in some configurations, over an order of magnitude lower than Horvitz-Thompson-style alternatives.
Read the full abstract ▾
Statistical Computation
Group recovery after trimming, and level-free flagging, in robust clusterwise regression
Samir Orujov
Trimming methods for robust clusterwise regression discard a fixed fraction of the data. Too low a level breaks the fit; too generous a level can trim away a small group. We first propose a group-recovery step that can follow any trimming or flagging method: it searches the discarded units for a line, tests whether those near it form a peak rather than a band, and restores the line as a group when the likelihood of Gaussian groups plus uniform noise improves by a margin like that of the
Bayesian information criterion. In simulations it repaired the failures of a generous TCLUST-REG level with unequal groups, but not with three or four groups. The second proposal, ESF (exact-subsample flagging), is a flagging procedure without a trimming level: it solves the clusterwise least-squares problem exactly on small subsamples, flags units far from the best fit, and draws later subsamples from the rest. Two constants stand in for the level: a subsample size, set from a lower bound on the smallest group proportion, and a cap on the flagged set. It is meant for data about whose contamination nothing is known: TCLUST-REG at the fixed level 0.30, followed by reweighting and the recovery step, was as accurate as ESF on average up to a fifth of outliers, and higher levels were more accurate beyond. The flagged fraction estimates the contamination only when the errors are close to Gaussian. On taxi fares with known tariffs, ESF found both in every sample.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.CL
Coupled Usage-Sense Processes: Temporal and Attributable Lexical Semantic Change
Haruka Ezoe and Ryohei Hisano
Lexical semantic change is usually summarized by a scalar distance between independently sampled period distributions. This measures how much a word changed, but does not reveal when it changed, which mechanisms and component movements carried the change, or which usages support the attribution. We introduce Coupled Usage–Sense Processes (CUSP), which derives these answers from a single marginal preserving temporal process. A hierarchical coupling relates contextual distributions through latent usage components, while Markov composition makes adjacent and longer span correspondences compatible. Displacement operators quantify change magnitude and timing, split variation exactly between movement of component centers and reorganization within components, and attribute it to transported component pairs. Word-local modes resolve distinct directions of change and their activity over time, while representative passages from attributed components ground the analysis in text. Under a Gaussian mixture specialization, we prove parametric recovery of the operators and squared distances. Synthetic experiments support the predicted rate. CUSP remains competitive on English and German DWUG and recovers controlled Janus profiles while maintaining compositionally coherent transport. A large corpus of US court opinions demonstrates transition, mode, and passage attribution in unlabeled natural text. CUSP thus makes magnitude, timing, mechanism, movement, modes, and textual evidence compatible views of one lexical history.
Read the full abstract ▾
Statistical Methodology
Summary-powered prediction under distribution shift
Ivy Zhang and Dominik Rothenhaeusler
Prediction models can perform poorly when the deployment population differs from the training population. Data from the target population would help, but individual-level target data may be inaccessible because of access restrictions or reporting conventions. We consider a multi-resolution setting in which individual-level data are available from a source population, while the target population is observed only through subgroup summaries. We propose SAGE, a one-step
estimator that updates a source-trained predictor using a gradient estimated from these summaries. Motivated by diagnostics consistent with the random distribution shift model, we choose SAGE's step-size to account for both sampling and distributional uncertainty. Under this model, SAGE reduces mean asymptotic target excess risk relative to the source-trained predictor. We also show that, under the model,
entropy-balancing weighted empirical risk minimization (EB), which reweights source observations to match the target summaries, is asymptotically equivalent to a full-step SAGE update. SAGE with the optimal step-size has asymptotic mean squared error no larger than that of EB. Across real-world
datasets, SAGE generally improves on the source-trained predictor and one-step updates that ignore distribution shift, including when the random shift model only partially captures the observed shifts. Compared to EB, SAGE improves prediction more consistently across the sample size and shift settings studied.
Read the full abstract ▾
Other Statistics · Cross-listed from cs.SI
Network Analysis in Communication Research: Research Topics, Knowledge Organization, and Research Practices
Pengjia Cui
Communication network research explains access to information, patterns of participation, and the organization of public meaning through different relationships and observations. This integrative review connects research topics, knowledge organization, and research practices, using bibliographic analysis of a Web of Science candidate pool to guide selective reading. Three judgments emerge from the comparisons. First, the relevance of a connection depends on the task and the criterion of value: team members anticipate consulting colleagues whose expertise they recognize, while journalists distinguish monitoring sources, using their information, and citing them. Second, commonality at one level can coexist with differentiation at another: audiences share outlets while selecting different articles, and shared issue agendas accommodate different evaluations. Third, some differences remain unresolved: contrasting media-use influence findings cannot be explained simply by whether models include selection and content co-nomination. Reading the uses of homophily, transactive
memory, sourcing, agenda-setting, and framing resources clarifies which expectations and observations support these judgments. Selected citation contexts also show how conceptual and measurement resources enter the same argument. The resulting agenda calls for comparisons of relationship types across group stages, source use across reporting tasks, and encountered content with recipients' interpretations. These purposive comparisons establish specific connections among literatures without estimating their prevalence or demonstrating a common causal mechanism.
Read the full abstract ▾
Machine Learning (Statistics)
Conformal Prediction under Exponential-Tilt Joint Shift
Seungjin Choi
Conformal prediction can lose coverage when the data distribution changes after deployment. We study adaptation using labeled source data and unlabeled target inputs, allowing both the input distribution and its relationship with outcomes to change. We use Exponential Tilt Reweighting
Alignment (ExTRA), introduced for classification by Maity et al. (2023), to estimate structured distribution shifts. We compare using its estimated weights in conformal calibration with additionally tilting the source predictive distribution. Shared learned predictors, estimated weights, calibration samples, and test observations isolate the effect of tilting. Existing theory gives both procedures target coverage with true weights and a common coverage bound with estimated weights. Identification calculations and an analysis of how scoring interacts with weight estimation error help explain why their performance can nevertheless differ. In a synthetic regression setting where the assumed models match the data-generating process and target inputs are informative about the shift, tilting reduces mean set length by about $30\%$ relative to weighting alone, with both methods attaining coverage near nominal. Tilting can instead cause substantial coverage losses in synthetic classification and in regression when target inputs provide little information about the response shift. Real-data experiments also show no consistent benefit. Good coverage from weighted calibration alone does not ensure that adding predictive tilting will preserve coverage. Deciding when to apply this additional adjustment using only source labels and target inputs remains an open problem.
Read the full abstract ▾
Statistical Methodology
Joint-Sparse Transfer Learning for High-Dimensional Multi-Output Regression
Sunwoo Lim and Mladen Kolar
Multitask linear models can improve estimation and prediction by exploiting structure shared across responses, bridging taskwise fitting and complete pooling. In many applications, however, the objective is estimation in a data-limited target domain, while data-rich but heterogeneous source domains are available. Borrowing from these sources can improve efficiency but introduce bias. We develop a joint-sparse transfer-learning framework for high-dimensional multi-output regression that combines shared predictor structure across responses with source-target similarity. The framework yields two complementary
estimators: a fused
estimator that aggregates jointly fitted domain-specific coefficients and a target-based debiased
estimator that adjusts for source-induced shifts. Our error bounds show how transfer increases the available information and sharing predictors across responses reduces selection costs. They also reveal a tradeoff: the fused
estimator benefits from larger sources but may retain bias if source shifts point in similar directions, whereas debiasing trades this bias for additional estimation error governed by the smaller target sample. Comparison with a minimax lower bound identifies regimes where the bounds match up to logarithmic factors, where matching remains unresolved, and where projection onto a target-based convex set closes the gap. Simulations and an analysis of single-
cell RNA and surface-
protein profiles across
cell types support the theory.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.LG
Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift
Liang You, Dongwen Ou, Hengyu Shi et al. (+1 more)
Clinical measurements are recorded for some patients but not others, at rates that differ across hospitals, and marginal conformal coverage does not ensure coverage within groups defined by missingness. We propose a missingness-aware conformal calibration procedure for mortality prediction under cross-hospital distribution shift. It selects a measurement on an independent sample, groups patients by whether that measurement is recorded, and applies Mondrian calibration within each group, so no calibration outcome is reused. We evaluate the procedure across hospitals in eICU and across care units within one MIMIC-IV hospital, using three predictors. Relative to pooled calibration, it reduces the average worst-group coverage gap on its selected groups in all six settings, with a median reduction of 1.9 percentage points; paired site-bootstrap intervals exclude zero in five. These gains do not extend uniformly. Calibration by predicted risk achieves smaller gaps on a broader panel of missingness groups, and when eICU hospitals are evaluated separately, the gain shrinks for all three predictors and reverses in sign for one. We explain this discrepancy with a hospital-level decomposition. Pooling reweights hospitals through a covariance between group shares and coverage errors, and lets errors of opposite sign cancel: weighting explains the reversal, and cancellation accounts for most of the attenuation for the other two predictors. Constructed population distributions show that pooled and within-hospital evaluations can rank calibration methods oppositely even without sampling noise. Pooled improvement alone therefore cannot establish better coverage within hospitals, even when the calibration groups are fixed.
Read the full abstract ▾
All authors Liang You, Dongwen Ou, Hengyu Shi, Siyuan Dai
Statistics Theory
Asymptotic Theory for Combining Dependent $p$-Values for Global Hypothesis Testing
Haoyi Yang and Lingzhou Xue
Combining $p$-values is a fundamental procedure in global hypothesis testing. In modern high-dimensional settings, however, component $p$-values often exhibit complex dependence and rely on asymptotic approximations rather than exact finite-sample uniform distributions. This paper establishes a unified asymptotic theory for weighted transformation statistics that decouples marginal finite-sample approximation error from the joint dependence structure. We also provide sufficient conditions based on conditional probability bounds to verify the joint-tail conditions. Utilizing this framework, we derive explicit dimension-growth and correlation rates for test statistics operating under asymptotic Gaussian and chi-square calibrations. For non-exact finite-sample statistics, we analyze standardized weighted sums, demonstrating how Cramér moderate deviations control relative tail error. Analytical examples demonstrate why both marginal and joint conditions are mathematically indispensable for valid global inference under dependence, and numerical experiments confirm that our asymptotic framework maintains accurate finite-sample size control at extreme significance levels.
Read the full abstract ▾
Statistical Computation · Cross-listed from math.OC
TR-SSQP: A Trust-Region Method for Constrained Stochastic Optimization under Heavy-Tailed Noise
Haoxuan Wang, Yuchen Fang and Sen Na
We consider
stochastic nonlinear optimization problems with deterministic equality constraints. While unconstrained
stochastic optimization is well understood, the interplay between optimality and feasibility in the constrained setting poses significant challenges. Moreover, existing theoretical guarantees for constrained
stochastic methods predominantly rely on bounded-variance assumptions, leaving the heavy-tailed noise regime largely unexplored. To address this gap, we propose a novel trust-region method within the
stochastic sequential quadratic programming framework, termed TR-SSQP. Our method employs a normal-tangential decomposition in the step computation to balance optimality and feasibility. In addition, we incorporate a normalization mechanism in the design of the trust-region radius, together with Polyak momentum for gradient estimation, ensuring stable updates without gradient clipping. When the trust-region radius and the momentum parameter decay at appropriate rates, we establish global almost-sure convergence of the method. To the best of our knowledge, this is the first asymptotic convergence result for constrained
stochastic optimization under heavy-tailed noise. We demonstrate the promising performance of the proposed method through extensive numerical experiments, including comparisons among its variants and with existing constrained
stochastic optimization methods.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.NE
Structured Bayesian Modeling of Dynamic Receptive1 Fields in Salamander Retinal Ganglion Cells
Alokesh Manna
Neurons in the visual system are selective for specific spatial and temporal stimulus features, described by their receptive field. Estimating one means a coefficient per pixel per time bin from few trials – a high-dimensional problem requiring regularization. Sparse regularizers such as the LASSO handle the dimension but select pixels independently at each time point, with nothing to keep the region coherent in space or smooth in time; it can fragment or reorganize discontinuously even when the true response evolves smoothly, a failure since this evolving pattern is what a receptive-field estimate should capture. We formulate dynamic receptive-field estimation as a high-dimensional
Bayesian problem: a Poisson model combining a Gaussian Markov random field in space with an autoregressive process in time, so the estimated field is smooth and coherent across space and time. On recordings from $155$ salamander retinal ganglion
cells, fitting this model independently per
neuron recovers a coherent surface, where a pixel-level Poisson-LASSO comparison instead returns a fragmented one. Summarizing each
neuron's surface by its space-averaged temporal response and clustering these curves with a model-based functional-clustering procedure, BIC selects three balanced temporal-response phenotypes ($85$, $32$, $38$
neurons), against a degenerate grouping from clustering the raw surfaces. A simulation study with known ground truth confirms the same pattern, with the model beating an unregularized Poisson GLM, LASSO, and the elastic net on recovery and estimation accuracy, though LASSO controls false positives better. The per-
neuron field identification, its contrast with LASSO, and the functional-clustering population typing constitute this paper's contribution.
Read the full abstract ▾
Statistical Methodology
Optimal Personalized Subspace Learning for Multi-view Tensor Observations
Kangxiang Qin, Zeyu Li, Xinbing Kong et al. (+1 more)
In this work, we model the observed multi-view tensors by decomposing the underlying signal in each view into two components: (i) the shared component that captures common dynamics across all views, and (ii) the private component that accounts for view-wise unique variations. To decouple the shared and private components, we introduce a novel Tucker personalized subspace principal component analysis (TPS-PCA) approach for tensors, which admits a one-step closed-form solution and serves as an ideal surrogate for our extended tensor-version personalized PCA (TP-PCA), adapted from the seminal work by \cite{shi2024personalized}. The theoretical analysis reveals that the proposed TPS-PCA
estimators reach the minimax lower bound in terms of view-wise tensor decoupling, whereas the TP-PCA
estimators only achieve a rate of average decoupling error across views, which is still slower than that of the TPS-PCA
estimators. Extensive numerical experiments are conducted on synthetic and real
datasets, demonstrating the wide applicability of the proposed method in fields including power management, financial analysis, and activity recognition.
Read the full abstract ▾
All authors Kangxiang Qin, Zeyu Li, Xinbing Kong, Wang Zhou
Machine Learning (Statistics)
Parameter Estimation for Unnormalized Discrete Models via Empirically Localized Deformed Bregman Divergence
Takashi Takenouchi
Estimation of parameter of probabilistic models is an important task in the field of machine learning.For models of discrete variables, calculation of the normalization constant of model is sometimes difficult and a lot of researches have been done to avoid the calculation of the normalization constant. In this paper, we tackle with the difficulty by combining a technique of empirical localization and a deformed Bregman divergence.The technique of empirical localization makes it possible to drastically reduce computational cost of the calculation of the normalization constant, and in addition, appropriate choice of the deformation for the Bregman divergence can invest the proposed
estimator with various kinds of favorable statistical properties, such as efficiency or robustness against outlier noise.
Read the full abstract ▾
Statistical Methodology
Design-Ignoring versus Design-Respecting World Models for Epidemiology
Xiangyu Yu and Weiyu Liu
World models for epidemiology learn from records shaped by study designs, including assignment, sampling, measurement, and related processes. A model may therefore reconstruct observed trajectories while learning an intervention contrast that depends on how records were collected. We formalize study design as constraints on a world model latent-world interface, action mechanism, observation likelihood, and target readout. This distinguishes design-respecting models, which encode these constraints, from design-ignoring models, which fit selected records without relating assignment and observation to the intended intervention question. Using a large scale cluster-randomized test-negative trial, we hold the latent structure and fitting settings fixed and compare a design-ignoring case-count model with a design-respecting test-negative observation model. Both models achieve comparable factual reconstruction. Yet across 500 paired resampling experiments at different relative sampling intensity, median contrast changes are substantial for the design-ignoring model, versus merely marginal for the design-respecting model. The same qualitative separation holds when sampling also varies across clusters. Thus, factual reconstruction alone does not establish design
alignment; testing whether fitted contrasts preserve design-implied observation-process invariances provides a sharper evaluation.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Causal Retention in Interactive Agents: Interface Factorization and Selective Adaptation
Shengjun Zhang, Tingyi Liu, Dong Xie et al. (+3 more)
Task performance need not determine which intervention mechanism an
agent retains. We study causal retention: whether a frozen learned state answers a mechanism-probe map fixed independently of training, including action, context, direct target, value, and delay. For finite structural causal model classes, the optimal probe error is a Bayes decision risk. It vanishes exactly when every learning-interface fiber lies within one probe-answer fiber; any state obtained by post-processing that interface inherits the same lower bound. A posterior-coverage
theorem characterizes budgeted retesting, while an exact edit decomposition shows that the shifted set is the unique support of an error-free target update. Causal Core implements these conditions through evidence-gated writing, readout filtering, temporal credit, hidden-context setup, and local diagnostic updates. Experiments cover finite causal systems, continuous simulators, an official TD-MPC2 world model, and Qwen2.5-7B-Instruct. A frozen Qwen last-layer probe reaches 0.958 balanced accuracy on source mechanisms but 0.583 on changed delays; the gated mechanism state reaches 1.000 and accepts only 0.056 of synchronized-readout candidates. In TD-MPC2, five target states per actuator recover effect-sign accuracy from 0.057 to 0.948 without degrading stable responses. Causal retention is therefore distinct from task sufficiency and source-domain decodability.
Read the full abstract ▾
All authors Shengjun Zhang, Tingyi Liu, Dong Xie, Yunlong Dong, Xiang Wang, Cheng Zeng
Statistical Methodology
Amortized Bayesian Disease Mapping and Boundary Detection on Heterogeneous Spatial Graphs
Luca Aiello and Sudipto Banerjee
Spatial disease maps help public-health researchers identify geographic inequalities, but standard
Bayesian smoothing can obscure localized disparities when neighboring communities have sharply different socioeconomic or behavioral profiles. Analysts therefore need to determine where smoothing should be interrupted and repeat that analysis as maps, adjacency structures, and outcomes change. We develop a
covariate-informed
Bayesian boundary model and an amortized posterior approximation trained across heterogeneous areal graphs. The model distinguishes local interruptions in smoothing from broader residual spatial dependence; the trained approximation handles maps with different numbers of regions. Simulations examine posterior calibration, boundary-probability recovery, replicated-data behavior, and MCSE-controlled agreement with prior-matched MCMC. In contrast to traditional approaches that analyze these data separately, we demonstrate the effectiveness of using a single trained deep learning network to analyze respiratory hospitalizations in Greater Glasgow; lung cancer incidence in California; and tracheal, bronchial, and lung cancer mortality in South Korea, comprising 58 to 241 regions. Selected boundary density is greatest in Glasgow and lowest in South Korea despite substantial residual spatial dependence in both, showing that local interruption and broader spatial persistence need not vary together. Across all three applications, edge-level boundary probabilities agree substantially with
dataset-specific analyses, although posterior spread and thresholded boundary sets differ. These results support reusable
Bayesian boundary analysis across the evaluated disease-map class and identify the validation needed before deployment to new applications.
Read the full abstract ▾
Machine Learning (Statistics)
MARCEDES: Score-based causal discovery under non-Gaussianity with continuous optimization
Anamitra Chaudhuri, Anirban Bhattacharya and Yang Ni
We consider the problem of learning the underlying causal directed acyclic graph (DAG) structure corresponding to a structural equation model (SEM) with non-Gaussian errors. Motivated by an intentionally misspecified non-Gaussian SEM with all Laplace errors, we first introduce the mean absolute residual risk, defined over the space of all real
matrices, and show that, asymptotically, the risk of the true weighted causal DAG
matrix is strictly smaller than that of any other
matrix. Nevertheless, to enhance generality and account for high-dimensional and finite-sample settings, we further incorporate row-specific sparsity penalties along with a soft DAG constraint to derive a continuous score function over the space of real
matrices. Accordingly, we propose a score-based DAG learning method, named MARCEDES, formulated as an unconstrained score minimization problem, which can be efficiently solved using gradient-based optimization techniques, thereby circumventing the challenges associated with constrained optimization. Furthermore, we develop a computational
algorithm to handle the non-smoothness of the score objective and to enable optimal tuning of row-specific sparsity penalties under a generalized Bayes framework. Finally, we demonstrate the efficiency and improved performance of the proposed method over existing approaches through an extensive simulation study.
Read the full abstract ▾
Statistical Methodology
Towards more plausible point-identifying assumptions in two-sample Mendelian randomization
Fernando Pires Hartwig, Neil Martin Davies and George Davey Smith
Two-sample Mendelian randomization (MR) is a widely applied methodology in epidemiology. In two-sample MR, summary data (typically, regression coefficients and standard errors) quantifying the association between multiple genetic variants and the exposure and the outcome are used in an instrumental variable framework aimed at estimating the causal effect of the exposure on the outcome. Most two-sample MR methods were developed under data-generating models where the association of for each candidate genetic instrument with the exposure, as well as the causal effect of the exposure on the outcome, are constant in the additive scale. These assumptions are useful because they imply that, had all genetic variants been valid IVs, they would all estimate the same causal parameter - namely, the constant causal effect. We refer to this condition as summary-level homogeneity. However, these are rather strong homogeneity conditions which may raise concerns about the plausibility of these methods in practice. In this paper, we show that summary-level homogeneity is implied by the following conditions: the causal effect is additive linear, but not necessarily constant across, all strata of the population; and uncorrelatedness between heterogeneity in the causal effect and in the association between each genetic variant and the exposure. Under these conditions, typical two-sample MR methods can be interpreted as
estimators of the average causal effect. These results clarify that point-identifying assumptions required for two-sample MR methods are weaker than previously anticipated, which contributes to their plausibility and interpretation in at least some practical applications.
Read the full abstract ▾