Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Statistics

stat Watch Statistics (sign in to use)

Monday 24 November

Machine Learning (Statistics) · Cross-listed from cs.LG

Terminal Velocity Matching

We propose Terminal Velocity Matching (TVM), a generalization of flow matching that enables high-fidelity one- and few-step generative modeling. TVM models the transition between any two diffusion timesteps and regularizes its behavior at its terminal time rather than at the initial time. We prove that TVM provides an upper bound on the $2$-Wasserstein distance between data and model distributions when the model is Lipschitz continuous. However, since Diffusion Transformers lack this property, we introduce minimal architectural changes that achieve stable, single-stage training. To make TVM efficient in practice, we develop a fused attention kernel that supports backward passes on Jacobian-Vector Products, which scale well with transformer architectures. On ImageNet-256x256, TVM achieves 3.29 FID with a single function evaluation (NFE) and 1.99 FID with 4 NFEs. It similarly achieves 4.32 1-NFE FID and 2.94 4-NFE FID on ImageNet-512x512, representing state-of-the-art performance for one/few-step models from scratch.
Read the full abstract

All authors Linqi Zhou, Mathias Parger, Ayaan Haque, Jiaming Song

Statistical Methodology

Threshold Tensor Factor Model in CP Form

This paper proposes a new Threshold Tensor Factor Model in Canonical Polyadic (CP) form for tensor time series. By integrating a thresholding autoregressive structure for the latent factor process into the tensor factor model in CP form, the model captures regime-switching dynamics in the latent factor processes while retaining the parsimony and interpretability of low-rank tensor representations. We develop estimation procedures for the model and establish the theoretical properties of the resulting estimators. Numerical experiments and a real-data application illustrate the practical performance and usefulness of the proposed framework.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

When +1% Is Not Enough: A Paired Bootstrap Protocol for Evaluating Small Improvements

Recent machine learning papers often report 1-2 percentage point improvements from a single run on a benchmark. These gains are highly sensitive to random seeds, data ordering, and implementation details, yet are rarely accompanied by uncertainty estimates or significance tests. It is therefore unclear when a reported +1-2% reflects a real algorithmic advance versus noise. We revisit this problem under realistic compute budgets, where only a few runs are affordable. We propose a simple, PC-friendly evaluation protocol based on paired multi-seed runs, bias-corrected and accelerated (BCa) bootstrap confidence intervals, and a sign-flip permutation test on per-seed deltas. The protocol is intentionally conservative and is meant as a guardrail against over-claiming. We instantiate it on CIFAR-10, CIFAR-10N, and AG News using synthetic no-improvement, small-gain, and medium-gain scenarios. Single runs and unpaired t-tests often suggest significant gains for 0.6-2.0 point improvements, especially on text. With only three seeds, our paired protocol never declares significance in these settings. We argue that such conservative evaluation is a safer default for small gains under tight budgets.
Read the full abstract

Statistical Methodology

Differentially Private Computation of the Gini Index for Income Inequality

The Gini index is a widely reported measure of income inequality. In some settings, the underlying data used to compute the Gini index are confidential. The organization charged with reporting the Gini index may be concerned that its release could leak information about the underlying data. We present an approach for bounding this information leakage by releasing a differentially private version of the Gini index. In doing so, we analyze how adding, deleting, or altering a single observation in any specific dataset can affect the computation of the Gini index; this is known as the local sensitivity. We then derive a smooth upper bound on the local sensitivity. Using this bound, we define a mechanism that adds noise to the Gini index, thereby satisfying differential privacy. Using simulated and genuine income data, we show that the mechanism can reduce the errors from noise injection substantially relative to differentially private algorithms that rely on the global sensitivity, that is, the maximum of the local sensitivities over all possible datasets. We characterize settings where using smooth sensitivity can provide highly accurate estimates, as well as settings where the noise variance is simply too large to provide reliably useful results. We also present a Bayesian post-processing step that provides interval estimates about the value of the Gini index computed with the confidential data.
Read the full abstract

Statistical Methodology

Order Selection in Vector Autoregression by Mean Square Information Criterion

Vector autoregressive (VAR) processes are ubiquitously used in economics, finance, and biology. Order selection is an essential step in fitting VAR models. While many order selection methods exist, all come with weaknesses. Order selection by minimizing AIC is a popular approach but is known to consistently overestimate the true order for processes of small dimension. On the other hand, methods based on BIC or the Hannan-Quinn (HQ) criteria are shown to require large sample sizes in order to accurately estimate the order for larger-dimensional processes. We propose the mean square information criterion (MIC) based on the observation that the expected squared error loss is flat once the fitted order reaches or exceeds the true order. MIC is shown to consistently estimate the order of the process under relatively mild conditions. Our simulation results show that MIC offers better performance relative to AIC, BIC, and HQ under misspecification. This advantage is corroborated when forecasting COVID-19 outcomes in New York City. Order selection by MIC is implemented in the micvar R package available on CRAN.
Read the full abstract

Machine Learning (Statistics)

Clustering Approaches for Mixed-Type Data: A Comparative Study

Clustering is widely used in unsupervised learning to find homogeneous groups of observations within a dataset. However, clustering mixed-type data remains a challenge, as few existing approaches are suited for this task. This study presents the state-of-the-art of these approaches and compares them using various simulation models. The compared methods include the distance-based approaches k-prototypes, PDQ, and convex k-means, and the probabilistic methods KAy-means for MIxed LArge data (KAMILA), the mixture of Bayesian networks (MBNs), and latent class model (LCM). The aim is to provide insights into the behavior of different methods across a wide range of scenarios by varying some experimental factors such as the number of clusters, cluster overlap, sample size, dimension, proportion of continuous variables in the dataset, and clusters' distribution. The degree of cluster overlap and the proportion of continuous variables in the dataset and the sample size have a significant impact on the observed performances. When strong interactions exist between variables alongside an explicit dependence on cluster membership, none of the evaluated methods demonstrated satisfactory performance. In our experiments KAMILA, LCM, and k-prototypes exhibited the best performance, with respect to the adjusted rand index (ARI). All the methods are available in R.
Read the full abstract

Statistics Applications

Anchoring Convenience Survey Samples to a Baseline Census for Vaccine Coverage Monitoring in Global Health

While conducting probabilistic surveys is the gold standard for assessing vaccine coverage, implementing these surveys poses challenges for global health. There is a need for more convenient option that is more affordable and practical. Motivated by childhood vaccine monitoring programs in rural areas of Chad and Niger, we conducted a simulation study to evaluate calibration-weighted design-based and logistic regression-based imputation estimators of the finite-population proportion of MCV1 coverage. These estimators use a hybrid approach that anchors non-probabilistic follow-up survey to probabilistic baseline census to account for selection bias. We explored varying degrees of non-ignorable selection bias (odds ratios from 1.0-1.5), percentage of villages sampled (25-75%), and village-level survey response rate to the follow-up survey (50-80%). Our performance metrics included bias, coverage, and proportion of simulated 95% confidence intervals falling within equivalence margins of 5% and 7.5% (equivalence tolerance). For both adjustment methods, the performance worsened with higher selection bias and lower response rate and generally improved as a larger proportion of villages was sampled. Under the worst scenario with 1.5 OR, 25% village sampled, and 50% survey response rate, both methods showed empirical biases of 2% or less, below 95% coverage, and low equivalence tolerances. In more realistic scenarios, the performance of our estimators showed lower biases and close to 95% coverage. For example, at OR$\leq$1.2, both methods showed high performance, except at the lowest village sampling and participation rates. Our simulations show that a hybrid anchoring survey approach is a feasible survey option for vaccine monitoring.
Read the full abstract

All authors Nathaniel Dyrkton, Shomoita Alam, Susan Shepherd, Ibrahim Sana, Kevin Phelan, Jay JH Park

Statistical Methodology

Integrating RCTs, RWD, AI/ML and Statistics: Next-Generation Evidence Synthesis

Randomized controlled trials (RCTs)have been the cornerstone of clinical evidence; however, their cost, duration, and restrictive eligibility criteria limit power and external validity. Studies using real-world data (RWD), historically considered less reliable for establishing causality, are now recognized as an important source of real-world evidence (RWE). In parallel, artificial intelligence and machine learning (AI/ML) are increasingly used throughout the drug development process, providing scalability and flexibility but also presenting challenges in interpretability and statistical rigor. This Perspective argues that the future of evidence generation will not depend on RCTs versus RWD, or statistics versus AI/ML, but on their principled integration under a statistical evidence framework that clarifies estimands, evaluates data fitness, controls bias, quantifies uncertainty, and determines when evidence is strong enough to support decisions. Building on the Causal Roadmap for high-quality real-world evidence, we present a six-step statistical roadmap for integrative evidence synthesis and organize the discussion around five core questions that statisticians must confront: when RWD is fit for causal use and when it is not; what AI genuinely contributes across the evidence lifecycle; what remains distinctly statistical and indispensable; how to build trustworthy end-to-end evidence systems in pharmaceutical, regulatory, and industry settings; and how statistical training should evolve.
Read the full abstract

Statistical Methodology

Clarifying identification and estimation of treatment effects in the Sequential Parallel Comparison Design

Sequential parallel comparison design (SPCD) clinical trials aim to adjust active treatment effect estimates for placebo response to minimize the impact of placebo responders on the estimates. This is potentially accomplished using a two stage design by measuring treatment effects among all participants during the first stage, then classifying some placebo arm participants as placebo non-responders who will be re-randomized in the second stage. In this paper, we use causal inference tools to clarify under what assumptions treatment effects can be identified in SPCD trials and what effects the conventional estimators target at each stage of the SPCD trial. We further illustrate the highly influential impact of placebo response misclassification on the second stage estimate. We conclude that the conventional SPCD estimators do not target meaningful treatment effects.
Read the full abstract

All authors Benjamin Stockton, Michele Santacatterina, Soutrik Mandal, Charles M. Cleland, Erinn M. Hade, Nicholas Illenberger, Sharon Meropol, Andrea B. Troxel, Eva Petkova, Chang Yu, Thaddeus Tarpey

Statistics Applications

Introducing Discipline Score Based on League Overall Swinging Probability

Plate discipline is an important feature of a hitter's success. Hitter who are able to recognize good pitches to swing at and balls to take are generally recognized as disciplined hitters. Although there are some metrics that can provide insight into the patience of a hitter, most do not capture the ability of a batter to take balls. In this research, we introduce two new metrics, Discipline Score (DS) and Adjusted Discipline Score (ADS), which evaluate batters' discipline when the pitch is a ball compared with the predicted tendencies of all batters in the league.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds

We derive a new theoretical interpretation of the reweighted losses that are widely used for training diffusion models. Our method is based on constructing a cascade of time-dependent variational lower bounds on the data log-likelihood, that provably improves upon the standard evidence lower bound and results in reduced data-model KL-divergences. Combining such bounds gives rise to reweighted objectives that can be applied to any generative diffusion model including both continuous Gaussian diffusion and masked (discrete) diffusion models. Then, we showcase this framework in masked diffusion and report significant improvements over previous training losses in pixel-space image modeling, approaching sample quality comparable to continuous diffusion models. Our results also provide a theoretical justification for the simple weighting scheme widely used in masked image models.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Lower Complexity Bounds for Nonconvex-Strongly-Convex Bilevel Optimization with First-Order Oracles

Although upper bound guarantees for bilevel optimization have been widely studied, progress on lower bounds has been limited due to the complexity of the bilevel structure. In this work, we focus on the smooth nonconvex-strongly-convex setting and develop new hard instances that yield nontrivial lower bounds under deterministic and stochastic first-order oracle models. In the deterministic case, we prove that any first-order zero-respecting algorithm requires at least $\Omega(\kappa^{3/2}\epsilon^{-2})$ oracle calls to find an $\epsilon$-accurate stationary point, improving the optimal lower bounds known for single-level nonconvex optimization and for nonconvex-strongly-convex min-max problems. In the stochastic case, we show that at least $\Omega(\kappa^{5/2}\epsilon^{-4})$ stochastic oracle calls are necessary, again strengthening the best known bounds in related settings. Our results expose substantial gaps between current upper and lower bounds for bilevel optimization and suggest that even simplified regimes, such as those with quadratic lower-level objectives, warrant further investigation toward understanding the optimal complexity of bilevel optimization under standard first-order oracles.
Read the full abstract

Statistics Applications

A Win-Expectancy Framework for Contextualizing Runs Batted In: Introducing ARBI and CRBI

Runs Batted IN (RBI) records the number of runs a hitter directly drives in during their plate appearances and reflects a batter's ability to convert opportunities into scoring. Because producing runs determines game outcomes, RBI has long served as a central statistic in evaluating offensive performance. However, traditional RBI treats all batted-in runs equally and ignores th game context in which they occur, such as leverage, score state, and the actual impact of a run on a team's chance of winning. In this paper, we introduce two new context-aware metrics-Adjusted RBI (ARBI) and Contextual RBI (CRBI)-that address the fundamental limitations of RBI by incorporating Win Expectancy (WE). ARBI rescales each RBI according to the change in WE before and after the scoring event, assigning more value to runs that meaningfully shift the likelihood of winning and less to runs scored in low-leverage situations. We then extend this framework to CRBI, which further differentiates RBIs with the same WE change by accounting for the terminal WE at the end of the event. This refinement captures the idea that an RBI increasing WE from, for example, 0.45 to 0.65 has a larger competitive impact than one increasing WE from 0.05 to 0.25, even though both represent a 20% increase. Together, ARBI and CRBI provide calibrated, context-sensitive measures of offensive contribution that more accurately reflect the true value of run production. These metrics modernize the interpretation of RBI and have broad applications in player evaluation, forecasting, contract evaluation, and decision-making in baseball analytics.
Read the full abstract

Statistics Applications · Cross-listed from physics.ao-ph

Concept drift of simple forecast models as a diagnostic of low-frequency, regime-dependent atmospheric reorganisation

Data-driven weather prediction models implicitly assume that the statistical relationship between predictors and targets is stationary. Under anthropogenic climate change, this assumption is violated, yet the structure of the resulting concept drift remains poorly understood. Here we introduce concept drift of simple forecast models as a diagnostic of atmospheric reorganisation. Using ERA5 reanalysis, we quantify drift in spatially explicit linear models of daily mean sea-level pressure and 2\,m temperature. Models are trained on the 1950s and 2000s and evaluated on 2020 tp 2024; their performance difference defines a local, interpretable drift metric. By decomposing errors by frequency band, circulation regime and region, and by mapping drift globally, we show that drift is dominated by low-frequency variability and is strongly regime-dependent. Over the North Atlantic-European sector, low-frequency drift peaks in positive NAO despite a stable large-scale NAO pattern, while Western European summer temperature drift is tightly linked to changes in land-atmosphere coupling rather than mean warming alone. In winter, extreme high-pressure frequencies increase mainly in neutral and negative NAO, whereas structural drift is concentrated in positive NAO and Alpine hotspots. Benchmarking against variance-based diagnostics shows that drift aligns much more with changes in temporal persistence than with changes in volatility or extremes. These findings demonstrate that concept drift can serve as a physically meaningful diagnostic of evolving predictability, revealing aspects of atmospheric reorganisation that are invisible to standard deviation and storm-track metrics.
Read the full abstract

Machine Learning (Statistics)

Optimization and Regularization Under Arbitrary Objectives

This study investigates the limitations of applying Markov Chain Monte Carlo (MCMC) methods to arbitrary objective functions, focusing on a two-block MCMC framework which alternates between Metropolis-Hastings and Gibbs sampling. While such approaches are often considered advantageous for enabling data-driven regularization, we show that their performance critically depends on the sharpness of the employed likelihood form. By introducing a sharpness parameter and exploring alternative likelihood formulations proportional to the target objective function, we demonstrate how likelihood curvature governs both in-sample performance and the degree of regularization inferred by the training data. Empirical applications are conducted on reinforcement learning tasks: including a navigation problem and the game of tic-tac-toe. The study concludes with a separate analysis examining the implications of extreme likelihood sharpness on arbitrary objective functions stemming from the classic game of blackjack, where the first block of the two-block MCMC framework is replaced with an iterative optimization step. The resulting hybrid approach achieves performance nearly identical to the original MCMC framework, indicating that excessive likelihood sharpness effectively collapses posterior mass onto a single dominant mode.
Read the full abstract

Statistics Applications

Beyond the ACE Score: Replicable Combinations of Adverse Childhood Experiences That Worsen Depression Risk

Adverse childhood experiences (ACEs) are categories of childhood abuse, neglect, and household dysfunction. Screening by a single additive ACE score (e.g., a $\ge 4$ cutoff) has poor individual-level discrimination. We instead identify replicable combinations of ACEs that elevate adult depression risk. Our data turnover framework enables a single research team to explore, confirm, and replicate within one observational dataset while controlling the family-wise error rate. We integrate isotonic subgroup selection (ISS) to estimate a higher-risk subgroup under a monotonicity assumption – additional ACE exposure or higher intensity cannot reduce depression risk. We pre-specify a risk threshold $\tau$ corresponding to roughly a two-fold increase in the odds of depression relative to the no-ACE baseline. Within data turnover, the prespecified component improves power while maintaining FWER control, as demonstrated in simulations. Guided by EDA, we adopt frequency coding for ACE items, retaining intensity information that reduces false positives relative to binary or score codings. The result is a replicable, pattern-based higher-risk subgroup. On held-out BRFSS 2022, we show that, at the same level of specificity (0.95), using our replicable subgroup as the screening rule increases sensitivity by 26% compared with an ACE-score cutoff, yielding concrete triggers that are straightforward to implement and help target scarce clinical screening resources toward truly higher-risk profiles.
Read the full abstract

All authors Ruizhe Zhang, Jooyoung Kong, Dylan S. Small, William Bekerman

Machine Learning (Statistics) · Cross-listed from cs.LG

Neural Tractability via Structure: Learning-Augmented Algorithms for Graph Combinatorial Optimization

Neural solvers provide fast solutions to graph combinatorial optimization problems, but training alone does not guarantee solution quality. Exact methods guarantee optimality but can be prohibitively expensive. We propose Neural Fixed-Parameter Tractable (N-FPT), a neural-model-agnostic framework that uses parameterized algorithms to improve neural solutions without retraining and guide learning. It restricts neural advice to a treewidth modulator and completes the bounded-treewidth remainder exactly. It returns and certifies the best achievable solution under any valid advice, improving the neural solution whenever a better compatible completion exists. We prove this guarantee and global optimality under perfect advice. Its incremental-confidence N-FPT (NIC-FPT) and randomized-deferral N-FPT (NRD-FPT) variants consolidate neural advice. Beyond inference, N-FPT feedback becomes available as neural decisions make exact completion tractable. From that state onward, it supplies what final solution rewards lack: the first decision that rules out every optimal completion and alternatives that preserve one. One exact computation supplies reusable guidance for continuations from that state, even when neural samples miss its optimum. Experiments across graph combinatorial optimization problems demonstrate solution-quality gains with different neural models and robustness to graph-size and distribution shifts. The learned completion model reaches optimal completions more often than reinforcement learning (RL) alone, without an exact solver at inference. N-FPT thus uses graph structure to improve neural solutions and guide learning toward optimal completion.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

ModHiFi: Identifying High Fidelity predictive components for Model Modification

Open weight models, which are ubiquitous, rarely provide access to their training data or loss function. This makes modifying such models for tasks such as pruning or unlearning, which are constrained by this unavailability, an active area of research. Existing techniques typically require gradients or ground-truth labels, rendering them infeasible in settings with limited computational resources. In this work, we investigate the fundamental question of identifying components that are critical to the model's predictive performance, without access to either gradients or the loss function, and with only distributional access such as synthetic data. We theoretically demonstrate that the global error is linearly bounded by local reconstruction errors for Lipschitz-continuous networks such as CNNs and well-trained Transformers (which, contrary to existing literature, we find exhibit Lipschitz continuity). This motivates using the locally reconstructive behavior of component subsets to quantify their global importance, via a metric that we term Subset Fidelity. In the uncorrelated features setting, selecting individual components based on their Subset Fidelity scores is optimal, which we utilize to propose ModHiFi, an algorithm for model modification that requires neither training data nor access to a loss function. ModHiFi-P, for structured pruning, achieves an 11% speedup over the current state of the art on ImageNet models and competitive performance on language models. ModHiFi-U, for classwise unlearning, achieves complete unlearning on CIFAR-10 without fine-tuning and demonstrates competitive performance on Swin Transformers.
Read the full abstract

All authors Dhruva Kashyap, Chaitanya Murti, Pranav K Nayak, Tanay Narshana, Chiranjib Bhattacharyya

Statistics Applications · Cross-listed from q-bio.QM

Parallelism in Neurodegenerative Biomarker Tests: Hidden Errors and the Risk of Misconduct

Biomarkers are critical tools in the diagnosis and monitoring of neurodegenerative diseases. Reliable quantification depends on assay validity, especially the demonstration of parallelism between diluted biological samples and the assay's standard curve. Inadequate parallelism can lead to biased concentration estimates, jeopardizing both clinical and research applications. Here we systematically review the evidence of analytical parallelism in body fluid (serum, plasma, cerebrospinal fluid) biomarker assays for neurodegeneration and evaluate the extent, reproducibility, and reporting quality of partial parallelism. This systematic review was registered on PROSPERO (CRD42024568766) and conducted in accordance with PRISMA guidelines. We included studies published between December 2010 to July 2024 without language restrictions. ... In conclusion, partial parallelism was infrequently observed and inconsistently reported in most biomarker assays for neurodegeneration. Narrow dilution ranges and variable methodologies limit generalizability. Transparent reporting of dilution protocols and adherence to established analytical validation guidelines is needed. This systematic review has practical implications for clinical trial design, regulatory approval processes, and the reliability of biomarker-based diagnostics.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Shortcut Invariance: Targeted Jacobian Regularization in Disentangled Latent Space

Deep neural networks are prone to learning shortcuts, spurious correlations present in the training data that undermine out-of-distribution (OOD) generalization. Most prior work mitigates shortcut learning through input-space reweighting, either relying on explicit shortcut labels or inferring shortcut structure from heuristics such as per-sample loss. Moreover, these approaches typically assume the presence of some shortcut-conflicting examples in the training set, an assumption that is often violated in practice, particularly in medical imaging where data is aggregated across institutions with different acquisition protocols. We propose a latent-space method that views shortcut learning as over-reliance on shortcut-aligned axes. In a disentangled latent space, we identify candidate shortcut-aligned axes via their strong correlation with labels and reduce classifier reliance on them by injecting targeted anisotropic noise during training. Unlike prior latent-space based approaches that remove, project out, or adversarially suppress shortcut features, our method preserves the full representation and instead impose functional invariance by regularizing the classifier's sensitivity along those axes. We show that injecting anisotropic noise induces targeted Jacobian and curvature regularization, effectively flattening the decision boundary along shortcut axes while leaving core feature dimensions largely unaffected. Our method achieves state-of-the-art OOD performance across standard shortcut-learning benchmarks without requiring shortcut labels or shortcut-conflicting samples.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.CV

Breaking the Likelihood-Quality Trade-off in Diffusion Models by Merging Pretrained Experts

Diffusion models for image generation often exhibit a trade-off between perceptual sample quality and data likelihood: training objectives emphasizing high-noise denoising steps yield realistic images but poor likelihoods, whereas likelihood-oriented training overweights low-noise steps and harms visual fidelity. We introduce a simple plug-and-play sampling method that combines two pretrained diffusion experts by switching between them along the denoising trajectory. Specifically, we apply an image-quality expert at high noise levels to shape global structure, then switch to a likelihood expert at low noise levels to refine pixel statistics. The approach requires no retraining or fine-tuning – only the choice of an intermediate switching step. On CIFAR-10 and ImageNet32, the merged model consistently matches or outperforms its base components, improving or preserving both likelihood and sample quality relative to each expert alone. These results demonstrate that expert switching across noise levels is an effective way to break the likelihood-quality trade-off in image diffusion models.
Read the full abstract

All authors Yasin Esfandiari, Stefan Bauer, Sebastian U. Stich, Andrea Dittadi

Statistical Methodology

Hierarchical Bayesian spectral analysis of multiple stationary time series

The power spectrum of biomedical time series provides important indirect measurements of physiological processes underlying health and biological functions. However, simultaneously characterizing power spectra for multiple time series remains challenging due to extra spectral variability and varying time series lengths. We propose a method for hierarchical Bayesian estimation of stationary time series (HBEST) that provides an interpretable framework for efficiently modeling multiple power spectra. HBEST models log power spectra using a truncated cosine basis expansion with a novel global-local coefficient decomposition, enabling simultaneous estimation of population-level and individual-level power spectra and accommodating time series of varying lengths. The fully Bayesian framework provides shrinkage priors for regularized estimation and efficient information sharing. Simulations demonstrate HBEST's advantages over competing methods in computational efficiency and estimation accuracy. An application to heart rate variability time series demonstrates HBEST's ability to accurately characterize power spectra and capture associations with traditional cardiovascular risk factors.
Read the full abstract

All authors Rebecca Lee, Alexander Coulter, Greg J. Siegle, Scott A. Bruce, Anirban Bhattacharya

Machine Learning (Statistics)

Nonparametric Instrumental Variable Regression with Observed Covariates

We study the problem of nonparametric instrumental variable regression with observed covariates, which we refer to as NPIV-O. Compared with standard nonparametric instrumental variable regression (NPIV), the additional observed covariates facilitate causal identification and enables heterogeneous causal effect estimation. However, the presence of observed covariates introduces two challenges for its theoretical analysis. First, it induces a partial identity structure, which renders previous NPIV analyses - based on measures of ill-posedness, stability conditions, or link conditions - inapplicable. Second, it imposes anisotropic smoothness on the structural function. To address the first challenge, we introduce a novel Fourier measure of partial smoothing; for the second challenge, we extend the existing kernel 2SLS instrumental variable algorithm with observed covariates, termed KIV-O, to incorporate Gaussian kernel lengthscales adaptive to the anisotropic smoothness. We prove upper $L^2$-learning rates for KIV-O and the first $L^2$-minimax lower learning rates for NPIV-O. Both rates interpolate between known optimal rates of NPIV and nonparametric regression (NPR). Interestingly, we identify a gap between our upper and lower bounds, which arises from the choice of kernel lengthscales tuned to minimize a projected risk. Our theoretical analysis also applies to proximal causal inference, an emerging framework for causal effect estimation that shares the same conditional moment restriction as NPIV-O.
Read the full abstract

All authors Zikai Shen, Zonghao Chen, Dimitri Meunier, Ingo Steinwart, Arthur Gretton, Zhu Li

Statistics Theory · Cross-listed from cs.DS

PTF Testing Lower Bounds for Non-Gaussian Component Analysis

This work studies information-computation gaps for statistical problems. A common approach for providing evidence of such gaps is to show sample complexity lower bounds (that are stronger than the information-theoretic optimum) against natural models of computation. A popular such model in the literature is the family of low-degree polynomial tests. While these tests are defined in such a way that make them easy to analyze, the class of algorithms that they rule out is somewhat restricted. An important goal in this context has been to obtain lower bounds against the stronger and more natural class of low-degree Polynomial Threshold Function (PTF) tests, i.e., any test that can be expressed as comparing some low-degree polynomial of the data to a threshold. Proving lower bounds against PTF tests has turned out to be challenging. Indeed, we are not aware of any non-trivial PTF testing lower bounds in the literature. In this paper, we establish the first non-trivial PTF testing lower bounds for a range of statistical tasks. Specifically, we prove a near-optimal PTF testing lower bound for Non-Gaussian Component Analysis (NGCA). Our NGCA lower bound implies similar lower bounds for a number of other statistical problems. Our proof leverages a connection to recent work on pseudorandom generators for PTFs and recent techniques developed in that context. At the technical level, we develop several tools of independent interest, including novel structural results for analyzing the behavior of low-degree polynomials restricted to random directions.
Read the full abstract

All authors Ilias Diakonikolas, Daniel M. Kane, Sihan Liu, Thanasis Pittas

Statistical Computation

Metric/Nonmetric Elastic MDS

We present R and C implementations for metric (ratio) and non-metric (ordinal) versions of Elastic MDS, the multidimensional scaling technique proposed by McGee (1966). The R and C versions are compared for speed, with the C version anywhere from 15 to 100 times as fast as the R version.
Read the full abstract

Machine Learning (Statistics) · Cross-listed from cs.LG

Predicting partially observable dynamical systems via diffusion models with a multiscale inference scheme

Conditional diffusion models provide a natural framework for probabilistic prediction of dynamical systems and have been successfully applied to fluid dynamics and weather prediction. However, in many settings, the available information at a given time represents only a small fraction of what is needed to predict future states, either due to measurement uncertainty or because only a small fraction of the state can be observed. This is true for example in solar physics, where we can observe the Sun's surface and atmosphere, but its evolution is driven by internal processes for which we lack direct measurements. In this paper, we tackle the probabilistic prediction of partially observable, long-memory dynamical systems, with applications to solar dynamics and the evolution of active regions. We show that standard inference schemes, such as autoregressive rollouts, fail to capture long-range dependencies in the data, largely because they do not integrate past information effectively. To overcome this, we propose a multiscale inference scheme for diffusion models, tailored to physical processes. Our method generates trajectories that are temporally fine-grained near the present and coarser as we move farther away, which enables capturing long-range temporal dependencies without increasing computational cost. When integrated into a diffusion model, we show that our inference scheme significantly reduces the bias of the predicted distributions and improves rollout stability.
Read the full abstract

All authors Rudy Morel, Francesco Pio Ramunno, Jeff Shen, Alberto Bietti, Kyunghyun Cho, Miles Cranmer, Siavash Golkar, Olexandr Gugnin, Geraud Krawezik, Tanya Marwah, Michael McCabe, Lucas Meyer, Payel Mukhopadhyay, Ruben Ohana, Liam Parker, Helen Qu, François Rozet, K.D. Leka, François Lanusse, David Fouhey, Shirley Ho

Statistical Methodology

Asymptotic linear dependence and ellipse statistics for multivariate two-sample homogeneity test

Statistical depth, which measures the center-outward rank of a given sample with respect to its underlying distribution, has become a popular and powerful tool in nonparametric inference. In this paper, we investigate the use of statistical depth in multivariate two-sample problems. We propose a new depth-based nonparametric two-sample test, which has the Chi-square(1) asymptotic distribution under the null hypothesis. Simulations and real-data applications highlight the efficacy and practical value of the proposed test.
Read the full abstract

All authors Chifeng Shen, Yuejiao Fu, Michael Chen, Xiaoping Shi

Statistical Methodology

Product Depth for Temporal Point Processes Observed Only Up to the First k Events

Temporal point processes (TPPs) model the timing of discrete events along a timeline and are widely used in fields such as neuroscience and fi- nance. Statistical depth functions are powerful tools for analyzing centrality and ranking in multivariate and functional data, yet existing depth notions for TPPs remain limited. In this paper, we propose a novel product depth specifically designed for TPPs observed only up to the first k events. Our depth function comprises two key components: a normalized marginal depth, which captures the temporal distribution of the final event, and a conditional depth, which characterizes the joint distribution of the preceding events. We establish its key theoretical properties and demonstrate its practical utility through simulation studies and real data applications.
Read the full abstract

All authors Chifeng Shen, Yuejiao Fu, Xiaoping Shi, Michael Chen

Statistical Methodology

Community-level core-periphery structures in collaboration networks

Uncovering structural patterns in collaboration networks is key for understanding how knowledge flows and innovation emerges. These networks often exhibit a rich interplay of meso-scale structures, such as communities, core-periphery organization, and influential hubs, which shape the complexity of scientific collaboration. The coexistence of such structures challenges traditional approaches, which typically isolate specific network patterns at the node level. We introduce a novel framework for detecting core-periphery structures at the community level. Given a reference grouping of the nodes, the method optimizes an objective function that assigns core or peripheral roles to communities by accounting for the density and strength of their inter-community connections. The node-level partition may correspond to either inferred communities or to a node-attribute classification, such as discipline or location, enabling direct interpretation of how different social or organizational groups occupy central positions in the network. The method is motivated by an application to a co-authorship network of Italian academics in three different disciplines, where it reveals a hierarchical core-periphery structure associated with institutional role, regional location, and research topics.
Read the full abstract

Machine Learning (Statistics)

The Unified Non-Convex Framework for Robust Causal Inference: Overcoming the Gaussian Barrier and Optimization Fragility

This document proposes a Unified Robust Framework that re-engineers the estimation of the Average Treatment Effect on the Overlap (ATO). It synthesizes gamma-Divergence for outlier robustness, Graduated Non-Convexity (GNC) for global optimization, and a "Gatekeeper" mechanism to address the impossibility of higher-order orthogonality in Gaussian regimes.
Read the full abstract