Machine Learning (Statistics) · Cross-listed from cs.LG
Network Recovery from Cascade Data: A Debiased Jacobian-Based Machine Learning Approach
Lei Huang
Many important outcomes unfold as dynamic cascades, including product adoption, disease spread, financial distress, and information
diffusion. A central challenge is to recover the hidden influence network behind these cascades. Existing methods typically assume a specific
diffusion model, and their performance degrades substantially when that assumption is misspecified. We propose CascadeNet, a Jacobian-based machine learning framework for network recovery that does not require specifying a
diffusion mechanism. The key idea is that the underlying influence structure can be characterized by the Jacobian of the one-step transition function. CascadeNet first constructs a flexible
estimator of the transition function, and further applies Neyman-orthogonal debiasing via the Riesz representer, so that the debiased Jacobian is $\sqrt{n}$-consistent and asymptotically normal, enabling formal inference on the network structure. We validate CascadeNet in both a simulation exercise and a real-world empirical application. In simulations, where the data-generating process is known, CascadeNet achieves the highest network recovery accuracy across nine common data-generating processes. In an empirical application to COVID-19 transmission across Spain's 52 provinces, CascadeNet recovers transmission networks that are significantly correlated with the true inter-province mobility network, whereas networks recovered by baseline methods show no significant
alignment with the ground truth.
Read the full abstract ▾
Statistical Methodology
Covariance-Adaptive Residualization and Stagewise Calibration for Dependent Multiple Testing
Prasenjit Ghosh and Arijit Chakrabarti
In this paper, we study simultaneous hypothesis testing for multivariate Gaussian means under arbitrary covariance dependence. Building upon the Maximum Residual Down (MRD) procedure of Cohen et al. (2009), we investigate a systematic stagewise calibration strategy based on the generalized step-down critical constants of Gavrilov et al. (2009). The proposed methodology retains the covariance-adaptive residualization mechanism of MRD while replacing the original model-dependent threshold specification with a simple and principled calibration rule. Since the resulting procedure belongs to the class of monotone residual-based step-down procedures studied by Ghosh and Chakrabarti (2026), its admissibility follows directly from their general theory. We also derive alternative representations of the MRD residual statistics that express all active residuals through a single active precision
matrix, substantially reducing computational complexity while revealing a direct connection between covariance-adaptive residualization and active precision-
matrix geometry. Extensive simulation studies under a broad range of dependence structures demonstrate that the proposed methodology frequently achieves substantially lower normalized misclassification risk than several widely used marginal testing procedures. Under several structured dependence models, it also exhibits remarkably strong signal-recovery behavior, simultaneously attaining false discovery rates close to the nominal level, extremely small false non-discovery rates, powers approaching one, and average numbers of rejections close to the expected number of true signals. These findings suggest that covariance-adaptive residualization and stagewise calibration play complementary roles in exploiting dependence information for large-scale multiple testing under arbitrary covariance structures.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Time series Foundation Models based on Physics-Informed Synthetic Histories for Cold-Start Photovoltaic Forecasting
Lorenzo Longarini, Alessandro Rongoni, Simone Silenzi et al. (+2 more)
At commissioning time, Photovoltaic (PV) operators must forecast production before target-site observations are available, limiting the direct use of standard supervised forecasters. This cold-start setting is addressed with a zero-shot pipeline that generates a synthetic production history from plant metadata and meteorological
covariates, enabling time-series foundation models (TSFMs) to forecast through inference-time conditioning. Five TSFMs are benchmarked against classical baselines under strict Cold-Start Baseline, Real Feedback, and Self-Forecast Feedback strategies. The evaluation spans $440$ PV sites across four
datasets and diverse
climate regimes.
Covariate-aware foundation models outperform baselines by approximately $1.7-2\times$: TabPFN-TS achieves the lowest error under Real Feedback (MAE $0.514$, RMSE $0.721$ $kWh$ ${kWp}^{-1}$ ${d}^{-1}$), while Chronos-2 is most robust under Self-Forecast Feedback. Performance is largely insensitive to the synthetic-history source, indicating that accuracy is driven more by the availability of plausible temporal context than by the specific generator.
Read the full abstract ▾
All authors Lorenzo Longarini, Alessandro Rongoni, Simone Silenzi, Emanuele Frontoni, Riccardo Rosati
Statistical Methodology
Community Detection on a Randomly Growing Network
Jianxiang Wang and Min Xu
We study community detection on Markovian random networks outside of the
Stochastic Block Model (SBM) framework. Specifically, we consider a random network growth process which generates $K$ separate preferential attachment trees and connects them with Erdős–Rényi edges, so that each tree represents a community and each node inherits the label of the tree to which it belongs. This model is able to produce many features of real world networks that are improbable under SBM, such as power law degree distribution and the existence of chains and hubs. Given only the final graph, without any knowledge of the growth process, we seek to recover the unobserved community membership of the nodes. We first prove that it is impossible for any
algorithm to consistently recover the community label of all the nodes. However, we design
algorithms which are provably able to recover the community labels of subsets of central nodes, for several different notions of node centrality such as arrival time or degree. Our procedure consists of two stages where, in the first stage, we classify high degree nodes and then, in the second stage, extend the community assignments to the remaining vertices. Numerical experiments and a real data application on a coauthorship network demonstrate the effectiveness of our proposed approach.
Read the full abstract ▾
Statistical Methodology
Deriving the Variance-Minimizing Design for Standard Addition via c-Optimality
Gerhard Gössler, Vera Hofer and Walter Goessler
Knowledge about optimal designs for standard addition seems to be scattered among literature and is also, at least partially, only available in mathematical literature that is not quickly accessible for readers not skilled in the field of design optimality theory. Therefore, the idea for this work was to summarize what is already available in analytical literature and to apply the respective results from optimality theory, where needed, to the special case of standard addition. It is shown, for measurement errors that are non-decreasing, e.g., are constant or increase linearly or quadratically with increasing analyte concentration, that the optimal design in the case of a linear response is a two-point design irrespective of the particular behavior of measurement error variance. In addition, it is demonstrated that the optimal allocation of measurements depends on the concrete setting, which means that the optimal distribution of measurements may deviate significantly from a 50:50 ratio. It is also investigated how the range, i.e., the largest added concentration influences the result. Last but not least, also the question of applying weighted regression is discussed and it is shown, that, in contrast to designs using more than two spiked concentrations, no weighting is necessary to achieve optimal results, when a two-point design is used. While the focus lies on the precision of the concentration estimate also the implications for the bias are investigated.
Read the full abstract ▾
Machine Learning (Statistics)
Are Good Generators Good Decision-Makers? Policy Learning for General Interventions via Retargeted Counterfactual Generation
Raphael C Kim, Jingsen Zhu, Ramin Zabih et al. (+1 more)
Generative models are increasingly used to support decision-making in complex systems, where interventions may be joint and high-dimensional, and outcomes are high-dimensional. However, using generators for these decision-making settings are challenged by three problems. First, they are often trained on noisy logs with limited intervention data. Second, generator learning can be noisy across different environments. Third, generators are not optimized for the decisions they support. We introduce policy learning via retargeted counterfactual generation, which trains a generator for the decisions it supports in three steps. We (1) learn a doubly-robust, invariant counterfactual generator for high-dimensional interventions and outcomes, (2) conduct policy-learning based on its rollouts, and (3) retarget the generator toward the learned policy's interventions and relearn the policy, so the generator is accurate where decisions are made. Theoretically, our generator's excess counterfactual risk has a doubly robust product remainder, and retargeting removes the worst-case density ratio between the logged and learned policies from the regret. We demonstrate the efficacy of our approach across synthetic data,
Cell Painting images of SARS-CoV-2-infected
cells, and physical
video simulations.
Read the full abstract ▾
All authors Raphael C Kim, Jingsen Zhu, Ramin Zabih, Michele Santacatterina
Machine Learning (Statistics) · Cross-listed from cs.AI
Online Pandora's Box for Contextual LLM Cascading
Alexandre Belloni, Yan Chen and Yehua Wei
Motivated by Large
Language Model (LLM) cascading, we propose an online contextual Pandora's Box model for adaptively querying and selecting LLM APIs. In each period, a decision-maker observes a request context and faces a two-phase decision problem. In the query phase, the decision-maker sequentially queries APIs, where each query reveals a generated output and the decision-maker incurs an (output-dependent) cost. In the selection phase, the decision-maker selects one of the generated outputs to deploy and observes only the downstream reward of the deployed output. This output-mediated feedback structure differs from classical online contextual Pandora's Box models, in which opening a box directly reveals its reward. Rather than estimating the full conditional output and cost distributions of each API, we directly model the reservation index and develop a learning approach for the query phase. Specifically, we impose a parametric structure on the contextual reservation index functions induced by the classical Weitzman's policy. Our policy combines generalized method of moments (GMM) type estimation of these reservation indices with UCB-style confidence bounds for both these indices and the shared output-level reward evaluator. Under regularity conditions, we prove that the resulting policy achieves dimension-dependent $\widetilde O(\sqrt T)$ cumulative regret over a horizon of $T$ periods.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Covariance Shrinkage via Stochastic Interpolation
Mathieu Chalvidal, Florentin Coeurdoux and Eric Vanden-Eijnden
We recast classical shrinkage of high-dimensional covariance
estimators as empirical risk minimization over a parametric
stochastic interpolant between a source and a target distribution. This formalism recovers known shrinkage
estimators as special cases and reveals three distinct mechanisms for reducing statistical risk: (i) Scheduling: the interpolant schedule determines the class of admissible covariances, and hence the achievable risk. (ii) Flow maps and couplings: whereas naive constructions amount to assuming independence between the distributions, specific coupling structures (e.g., solutions of optimal transport problems) can lower the empirical risk. Moreover, non-linear flow maps realizing such couplings free the interpolant covariance from the eigenbasis of the empirical estimate, enabling eigenvector regularization. (iii) Early stopping:
estimators defined by integrating a regressed vector field afford an additional bias-variance trade-off through approximation of the true interpolant distribution. We then propose a neural
estimator of the interpolant, together with an upper bound on its quadratic risk in terms of the interpolant approximation error, and validate both on synthetic experiments. Finally, we apply the
estimator to real neuroimaging data, demonstrating the additional regularization power this approach offers in practice.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.LG
Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests
Thanawat Lodkaew, Johannes Ackermann, Soichiro Nishimori et al. (+3 more)
A growing failure mode in
agent evaluation and training is that models can achieve high evaluation scores by exploiting shortcuts instead of solving the intended task, producing deceptive performance. This makes evaluation scores unreliable as measures of true task-solving ability. We propose CapCode, a framework for constructing coding
datasets with randomized tests whose best achievable non-cheating performance is deliberately capped below one. This capped-performance design gives evaluation scores a clearer interpretation: scores substantially above the cap are implausible and therefore provide evidence of cheating. To prevent cheating, we propose CapReward, a reward design based on the CapCode principle to discourage optimization beyond the cap. Experiments across multiple
datasets show that CapCode detects cheating while preserving performance ranking of models, and CapReward reduces cheating behavior, yielding models that better follow the intended task specification.
Read the full abstract ▾
All authors Thanawat Lodkaew, Johannes Ackermann, Soichiro Nishimori, Nontawat Charoenphakdee, Masashi Sugiyama, Takashi Ishida
Statistical Methodology
Learning Collapsed Patterns in Compositional Data: A Bayesian Heterogeneous Relative-Shift Approach
Maoran Xu and Guanyu Hu
Relative-shift regression provides a principled framework for modeling compositional
covariates by quantifying how the response changes when mass is reallocated from one component to another. Yet many emerging compositional data problems extend beyond this classical setting, involving high-dimensional predictors and regression effects that vary across latent subpopulations. This complexity poses a dual challenge unmet by existing methods: recovering latent cluster structure while simultaneously achieving dimension reduction within each cluster. We propose a
Bayesian heterogeneous relative-shift regression model that jointly learns latent clusters and parsimonious effect structures. Methodologically, we combine a projection-based shrinkage prior on identifiable contrasts, which induces exact coefficient ties within mixture components, with a mixture of finite mixtures prior that infers the number of clusters. Computationally, we develop a scalable hybrid MCMC
algorithm that embeds a deterministic surrogate collapse operator within NUTS. Theoretically, we establish posterior consistency for both the latent partition and cluster-specific effect structures. Simulations confirm accurate recovery and strong predictive performance, and applications to cross-country macroeconomic data and spatial transcriptomics demonstrate the method's interpretability and practical utility.
Read the full abstract ▾
Statistics Applications
S2A3: Thompson Sampling and Stochastic Exposure Control for High-Stakes CATs
James Sharpnack, Alexander Tsigler, J. R. Lockwood et al. (+2 more)
High-stakes computerized adaptive tests (CATs) require a continuous supply of calibrated items, yet traditional item piloting is slow, expensive, and operationally hazardous. We introduce the S2A3 framework – Soft Scoring (S2) and Adaptive Adaptive Administration (A3) – which unifies item calibration and test administration into a single online process. Thompson sampling enhances item selection by drawing provisional parameters from each item's posterior distribution and selecting the item maximizing expected Fisher information, naturally routing uncertain items to informative test-takers while maintaining measurement precision. Soft scoring integrates over parameter uncertainty so that incompletely calibrated items exert appropriately attenuated influence on ability estimates. A
stochastic variant of Sympson-Hetter exposure control balances measurement efficiency against bank security via a tunable temperature parameter and item-specific weights. We validate S2A3 on Yes/No Vocabulary and Vocabulary-in-Context tasks from the Duolingo English Test, demonstrating rapid item calibration and preserved scoring reliability even when cold-start items constitute a significant fraction of the active pool.
Read the full abstract ▾
All authors James Sharpnack, Alexander Tsigler, J. R. Lockwood, Steven Nydick, Alina A. von Davier
Statistics Theory
Dependence Measures via Adapted Optimal Transport: Stability and Rates of Convergence
Jonathan Ansari and Johannes Wiesel
Recently studied dependence measures, such as Chatterjee's rank correlation, that characterize both independence and perfect functional dependence, provide a powerful framework for detecting nonlinear dependencies. However, these measures cannot be weakly continuous, which limits the applicability of classical plug-in
estimators based on empirical distributions. This obstruction is natural, as such measures are defined via conditional distributions and not through their joint law alone. In this paper, we introduce an optimal transport-based mode of convergence that captures weak convergence of conditional distributions and restores continuity for a broad class of dependence measures. We relate this mode of convergence to the adapted Wasserstein distance, the Knothe-Rosenblatt distance and the d1-metric on copulas. Building on this perspective, we propose a copula
estimator based on the adapted empirical measure and compare it with the classical rank-based checkerboard
estimator. For both
estimators, we derive O(N^{-1/3})-rates of convergence with respect to metrics that capture conditional weak continuity. As a consequence, we obtain the same rates for plug-in
estimators of several classes of dependence measures, including rank-based and rearranged dependence measures.
Read the full abstract ▾
Statistics Theory
A Temporal Spatial Minimax Rate for Smoothly-Varying Distributions in Wasserstein Space
Munsik Kim
We study the minimax rate of estimating a future value $\mu_{t_n+h}$ of a curve $t\mapsto\mu_t$ in the $2$-Wasserstein space $\mathcal{P}_2(\mathbb{R}^d)$ from finitely many noisy snapshots of its past, under an adiabatic bound $\|\nabla_t^k v\|\le\varepsilon$ on the $k$-th covariant derivative of the velocity field. Our central result is a unified temporal-spatial minimax lower bound: over regular, locally transport-rich subclasses, every
estimator incurs $W_2$-risk with $M$-exponent $\gamma_d(k+1)/(k+1+\gamma_d)$, $\gamma_d=\min(1/d,1/2)$ ($M$ the total sample size). It follows from a temporal-to-spatial reduction: the smoothness budget defines a reachable $W_2$-ball into which a transport packing is embedded along the time axis, and the information of the entire snapshot experiment is controlled by a Fano argument – the spatial packing is classical, but its smoothness-admissible temporal
embedding and the full-window analysis are new. The bound interpolates a dimension-free extrapolation floor of order $\varepsilon h^{k+1}$ – the irreducible cost of an unobserved future, present even with the exact past – and the spatial estimation curse $M^{-\gamma_d}$, recovering the static distribution-estimation rate as $k\to\infty$. We state the lower bound in a design-dependent form – with a design-weighted effective sample size – valid for arbitrary observation times, and obtain the closed-form exponent in the dense (equispaced) regime. The matching upper bound is established at $k=0$ (rate $M^{-1/(d+1)}$, $d\ge3$) and, in a translation submodel, for all $k$; for $k\ge1$ a covariant
estimator attains the rate conditionally on two estimates (a comparison-geometry bias bound and an optimal-transport map-estimation rate), leaving the unconditional general-$k$ upper bound as an open problem. Numerical experiments on synthetic curved and flat families corroborate the predicted exponents.
Read the full abstract ▾
Statistics Theory
The Balance Property: The Constrained Case, with a View on Risk Sharing
Mario V. Wüthrich
The balance property is an important property of fitted statistical models deployed for insurance pricing. It guarantees that the total actuarial price in the fitted model is equal to the totally observed loss used to fit the model. This can be seen as an in-sample global unbiasedness property. Maximum likelihood fitted generalized linear models (GLMs) with canonical links automatically fulfill the balance property. Lindholm-Wüthrich (Scandinavian Actuarial Journal, 2026) discussed two popular balance correction methods in case the balance property fails to hold. This note extends this discussion with a third method, constrained GLM fitting, that turns out to be superior over the two previously discussed ones. Moreover, we highlight the connection between the balance property and ex-post risk sharing rules.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cond-mat.dis-nn
Theory of learning of high-dimensional controlled non-linear dynamical systems (I): models and methods
Pierfrancesco Urbani
Neural ordinary differential equations (neural ODEs) have rapidly gained prominence as a powerful and unifying framework for conceptualizing artificial neural networks, elegantly connecting the continuous-time modeling of dynamical systems with the discrete, data-driven paradigm of modern deep learning. Beyond their practical advantages they offer fresh theoretical insights into the training and generalization properties of neural networks. The distinctive feature of this framework is its dual dynamical nature: inference dynamics, which govern the ODE
evolution during forward computation, and training dynamics, which control the optimization of model parameters. This makes neural ODEs a particularly well-suited theoretical framework for studying a large variety of settings such as multi-layer neural networks (ResNets for example), autoregressive models (with next-
token generation dynamics), generative models, and recurrent neural networks in theoretical neuroscience. In this work, we introduce a theoretically grounded class of models for studying neural ODEs trained via online
stochastic gradient descent. We solve the training dynamics of these models via dynamical mean field theory and derive learning curves in the high-dimensional limit.
Read the full abstract ▾
Statistical Methodology
Principal Component Analysis for Multivariate Extremes
Dan Cooley, Anne Sabourin and Troy Wixson
This chapter explores ways to reduce the dimensionality of the data while preserving key information relevant to the analysis of multivariate extreme values.
Statistical Methodology
One-step Outcome Imputation: An Alternative to Multiple Imputation
Andreas Nordland, Klaus K. Holst, David Redek et al. (+2 more)
Missing outcomes in randomized controlled trials are often handled by multiple imputation (MI). Rubin's rules are routinely used to estimate standard errors but can fail to provide valid standard error estimates for some commonly used procedures, such as reference-based imputation. We propose a one-step alternative by explicitly targeting the treatment effect implied by a given imputation model and constructing an efficient one-step
estimator for that treatment effect via its influence function. Unlike Rubin's rules, this approach yields asymptotically valid inference. Moreover, the proposed method circumvents the
stochastic component and computational burden of MI. We illustrate the approach with examples spanning a range of imputation models, including reference-based imputation and intercurrent-event-dependent imputation. The one-step imputation estimation procedure is implemented in the `targeted` R package.
Read the full abstract ▾
All authors Andreas Nordland, Klaus K. Holst, David Redek, Christian B. Pipper, Aske T. Iversen
Statistical Methodology
When can a posterior predictive check identify the learning rate? Exact degeneracy in Gaussian models and implications for Generalised Bayesian Inerence
Nam Anh Le
Generalised
Bayesian inference tempers the likelihood by a learning rate $\eta$ to mitigate model misspecification, and the choice of $\eta$ is consequential. Zafar and Nicholls (2024) proposed selecting $\eta$ by a posterior predictive check (PPC): one chooses the smallest $\eta$ at which a log-likelihood PPC $p$-value is not rejected. An exact, finite-sample analysis of this selector on the Gaussian linear model is given. With known variance and a flat prior, the PPC $p$-value equals $P(\chi^2_n > \mathrm{RSS}/\sigma_0^2)$ for every $\eta$, so the selector is $\eta$-invariant; under variance misspecification it is two-sided non-identifying. With unknown variance and the reference prior, the $p$-value depends only on $(n,d,\eta)$ and not on the realised data or the data-generating process. Consequently the selector's output is fixed before any data are seen, typically collapsing to the smallest grid value, which over-tempers and inflates predictive intervals relative to held-out selection. The phenomenon is a pivotality property specific to the Gaussian scale–location family and the reference prior; it disappears under informative priors. These results delineate the selector's scope, identify a canonical class on which it cannot identify the learning rate, and motivate a cheap, data-free pre-screening diagnostic.
Read the full abstract ▾
Statistics Applications
Collaborative estimation and evaluation of SARS-CoV-2 variant nowcasting in the United States
Isaac MacArthur, Thomas Robacker, Bren Case et al. (+27 more)
The ability to estimate and predict pathogen variant dynamics can inform public health responses, including planning for increased transmission or severity, shifts in population immunity, or changes to vaccine or therapeutic effectiveness. The COVID-19 pandemic demonstrated the importance of monitoring SARS-CoV-2 variant
evolution through viral genome sequencing, enabling predictive models to estimate variant frequencies in the recent past, present, and short-term future. Collaborative
forecasting Hubs provided a valuable way to centralize predictive modeling of epidemiological indicators such as cases, hospitalizations, and deaths during the pandemic; however, none existed for variant dynamics. Here, we discuss the creation of the United States SARS-CoV-2 Variant Nowcast Hub, designed to solicit estimates of the relative abundance of a specified set of SARS-CoV-2 variants at the U.S. state level. We discuss the design decisions and challenges in building the Hub and its scoring procedures. Using submissions from the Hub's first respiratory virus season (nowcast dates October 9th, 2024 to June 4th, 2025), we evaluate five individual models and a baseline model. We found that the baseline model, which pools sequences across the U.S., performs well overall, with most individual models performing similarly or slightly worse. Locations with lower sequencing volumes exhibited greater variability in model performance. Models submitted for a single location outperformed those submitted for all locations, potentially due to greater timeliness and magnitude of local data. Much remains to be investigated regarding relative model performance across different phases of variant emergence, and we conclude by proposing future directions within and beyond this Hub.
Read the full abstract ▾
All authors Isaac MacArthur, Thomas Robacker, Bren Case, Spencer J. Fox, Dylan H. Morris, Evan L. Ray, Benjamin Rogers, Becky Sweger, Natalie M. Linton, John Huddleston, Andrew Magee, Zachary Susswein, Jover Lee, Trevor Bedford, Marlin D. Figgins, Ehsan Suez, Rajath Prabhakar, Tomas Leon, Brent Siegel, Mugdha Thakur, Christopher M. Hoover, Rahil Ryder, Jesse Elder, Michael Kupperman, Ruian Ke, Emma Goldberg, Sebastian Funk, Maryclare Griffin, Nicholas G. Reich, Kaitlyn E. Johnson
Machine Learning (Statistics) · Cross-listed from cs.IT
Information-Theoretic Bounds for Sparse Covariance Estimation in the Vertical-Split Distributed Model
Jing Yee Tan and Guangyue Han
We study the minimax estimation error for distributed covariance
matrix estimation in the vertical-split (feature-split) setting, where two
agents each observe different coordinates of~$m$ i.i.d.\ sub-Gaussian samples and communicate a limited number of bits to a central server. While \cite{rahmani2025fundamental} established nearly tight bounds for dense (unstructured) cross-covariance
matrices, we investigate whether imposing elementwise $s$-sparsity on the cross-covariance $C_{21}$ can reduce the required communication and sample complexity. We show that sparsity can reduce the communication required for cross-covariance estimation in the vertical split at a fixed target distortion. Specifically, for sufficiently large $d_1d_2/s'$ and $0<\varepsilon<\sigma^2\sqrt{s'}/32$, any scheme achieving expected Frobenius distortion at most $\varepsilon$ must satisfy $B_k = \Omega(\sigma^4 d_k\, s' \log(d_1 d_2/s')/\varepsilon^2)$ and $m = \Omega(\sigma^4\, s' \log(d_1 d_2/s')/\varepsilon^2)$ for cross-covariance estimation, where $s' = s \wedge d_{\min}$. For the $1$-sparse case, our achievable scheme reduces the $d_1d_2$ factor in the dense communication rate to $\log(d_1d_2)$, up to polylogarithmic factors, for the cross-covariance communication component in the matching regime. Our lower bounds are established via Fano's method with an explicit sparse packing using a Varshamov–Gilbert-type argument for signed partial permutation
matrices combined with the Conditional Strong Data Processing Inequality of \cite{rahmani2025fundamental}. We show that the communication lower bound is tight up to polylogarithmic factors under the conditions of Remark 4.9, using an achievable scheme based on covering-net quantization and entry-wise hard thresholding.
Read the full abstract ▾
Statistics Theory
Ising Models on Inhomogeneous Random Graphs: Inference, Local Asymptotic Minimaxity, and Limit of Experiments
Somabha Mukherjee, Sanchayan Bhowal, Anirban Chatterjee et al. (+1 more)
In this paper, we develop an inferential framework with sharp asymptotic optimality guarantees for Ising models on inhomogeneous random graphs in the subcritical parameter regime. We begin by characterizing the asymptotic distribution of the maximum likelihood (ML) estimate of the natural parameter, based on a single sample from the underlying model, covering both sparse and dense network regimes. Next, to overcome the computational intractability of the ML method, we propose a simple closed-form estimate obtained from a one-step approximation to the likelihood equation. We show that this estimate attains the same asymptotic distribution and variance as the ML estimate, thereby yielding a computationally efficient and asymptotically valid confidence interval for the natural parameter. We complement these inferential results by establishing a Hájek–Le Cam-type local asymptotic minimax
theorem, showing that the proposed estimate achieves the smallest possible asymptotic maximum risk, both in rate and in leading constant, over shrinking neighborhoods of the true parameter. We also derive the corresponding limit of experiments. To the best of our knowledge, these are among the first sharp asymptotic optimality results for network-dependent data. Finally, we study goodness-of-fit testing for the natural parameter, deriving the local power of the likelihood ratio test and minimax detection rates. Our analysis relies on new fluctuation results for the sufficient statistic (
Hamiltonian) and for the random partition function of Ising models on inhomogeneous random graphs, which are of independent interest.
Read the full abstract ▾
All authors Somabha Mukherjee, Sanchayan Bhowal, Anirban Chatterjee, Bhaswar B. Bhattacharya
Statistical Computation
CATEKAPPA: An R Shiny Application for Design and Analysis of Consistency Tests Based on the Kappa Statistic for Categorical Responses
Zheng Gai, Li Xincheng, Jiang Wangyingjie et al. (+1 more)
The kappa statistic is the most widely used measure of inter-rater agreement for categorical data. Despite its popularity, applied researchers often encounter two major hurdles: (i) determining the sample size required to achieve a desired level of agreement with given power, and (ii) computing appropriate kappa coefficients with proper interpretation. Existing R packages such as irr and kappaSize provide these functionalities but require programming skills and lack an integrated, user-friendly interface. We present CATEKAPPA, an R package that bridges this gap by combining sample size planning (via kappaSize) and agreement analysis (via irr) into a single Shiny-based web application. The package supports Cohen's kappa for two raters, Fleiss' kappa for three or more raters, and Light's kappa, and provides automatic interpretation using the Landis & Koch scale. Users can either launch an interactive graphical interface or use command-line functions for scripting. The package is freely available on CRAN.
Read the full abstract ▾
All authors Zheng Gai, Li Xincheng, Jiang Wangyingjie, Zhao Panwei
Machine Learning (Statistics) · Cross-listed from cs.LG
Constructing VAE Latent Spaces with Prescribed Topology
Jilles S. van Hulst, Jakub M. Tomczak, W.P.M.H. Heemels et al. (+1 more)
Variational autoencoders (VAEs) learn low-dimensional latent representations of high-dimensional data. When the data lies on a
manifold with non-Euclidean
topology, the standard Gaussian prior introduces a topological mismatch that degrades reconstruction quality and prevents faithful representation. We present a constructive mathematical framework that resolves this mismatch for all
manifolds that admit a product covering space. These are
manifolds expressible as products of elementary factors (circles, intervals, or lines) or as quotients of such products by a finite
symmetry group. The class includes cylinders,
tori, Möbius strips, Klein bottles, and real projective spaces. Factorized distributions over the elementary factors yield product
topologies with closed-form, decoupled KL divergences, so that each latent factor can be shaped independently while keeping training tractable. We catalogue reparametrizable encoder-prior pairs for periodic, bounded, and unbounded supports, and provide coordinate transformations that allow standard neural networks to output non-Euclidean parameters with smooth gradients. For quotient
manifolds, the decoder receives group-invariant features of the covering-space coordinates, so that identified points produce identical outputs. Anchor constraints fix the coordinate system relative to the data or create soft topological holes. Experiments on synthetic
manifolds and real-image
datasets (rotated and cyclically shifted MNIST) confirm that a
topology-matched prior aligns KL regularization with the data
manifold. The resulting
topology-aware models outperform the Gaussian baseline at all practically relevant regularization strengths. The code is available at https://github.com/JvHulst/VAE-
Topology.
Read the full abstract ▾
All authors Jilles S. van Hulst, Jakub M. Tomczak, W.P.M.H. Heemels, Duarte J. Antunes
Statistical Methodology
Influence of continuous predictor modelling methods on prediction stability in clinical prediction model development: an empirical comparison using real clinical data
Phichayut Phinyo, Pakpoom Wongyikul, Noraworn Jirattikanwong et al. (+5 more)
Background and objective: Prediction stability is increasingly recognised as important for reliable clinical prediction model development, but the effect of continuous predictor modelling choices is unclear. This study examined how approaches to modelling continuous predictors influence prediction stability. Methods: We used a real clinical
dataset of 19,418 emergency department patients to create five sample size scenarios ranging from 437 to 8,739 patients. Six methods were compared: dichotomisation at the median (DIC), tertile categorisation (TER), linear terms (LIN), quadratic terms (QUA), multivariable fractional
polynomials (MFP), and extreme gradient boosting (XGB). Prediction stability was evaluated using a bootstrap-based framework. Optimism-corrected AUC and calibration were estimated through internal validation. A method was considered stable when at least 90% of individual predictions had a mean absolute prediction error (MAPE) <=5%. Results: Stability increased with sample size and varied by method. At n = 437, no method met the stability criterion; LIN was the most stable, followed by DIC. At n = 874, DIC and LIN achieved stable predictions with similar calibration, although DIC had lower AUC. At n = 1,748, QUA achieved stability, whereas MFP and XGB did not. At n = 3,496 and n = 8,739, all methods achieved stability. LIN, QUA, MFP, and XGB generally had higher AUCs than DIC and TER, while XGB showed the highest AUC but persistent miscalibration. Conclusion: Continuous predictor modelling methods appeared to influence prediction stability. LIN achieved stable predictions from the base sample size onwards, whereas QUA, MFP, and XGB required larger samples. Although XGB showed high discrimination, calibration concerns persisted. These findings suggest that, in smaller
datasets, simpler approaches, particularly LIN, may provide more stable predictions.
Read the full abstract ▾
All authors Phichayut Phinyo, Pakpoom Wongyikul, Noraworn Jirattikanwong, Natthanaphop Isaradech, Wuttipat Kiratipaisarl, Suppachai Lawanaskol, Noppadon Seesuwan, Wachiranun Sirikul
Statistics Theory
Integral stochastic orders of $m$-generalized order statistics from transform-ordered nonparametric families
Idir Arab, Tommaso Lando, Paulo Eduardo Oliveira et al. (+1 more)
We provide sufficient conditions for comparing $m$-generalized order statistics with respect to the increasing concave, increasing convex, and
star-shaped
stochastic orders. These conditions allow us to rank classical order statistics, selected censored type-II order statistics, and records. They depend on both the parameters of the generalized order statistics and the underlying distribution. Rather than assuming a specific parametric form, we adopt a nonparametric approach and assume some
stochastic transform-ordered property, that is, some suitable shape condition. This framework encompasses many relevant classes of distributions that are related, via transform order, to the generalized and the negative generalized Pareto distribution.
Read the full abstract ▾
All authors Idir Arab, Tommaso Lando, Paulo Eduardo Oliveira, Tomasz Rychlik
Statistics Applications
An Integrated Roadside Sensing and Communication Framework for Vulnerable Road User Safety at Signalized Intersections
Parvez Anowar
Vulnerable road users (VRUs) account for approximately half of urban traffic deaths globally, with intersections concentrating a disproportionate share of these casualties. Recent reviews of sensing technology for VRU protection have cataloged dozens of single-sensor and dual-sensor deployments, yet none of the surveyed systems couples multi-modal sensing with edge-side near-miss analytics and bidirectional vehicle-to-everything (V2X) and pedestrian-to-everything (P2X) messaging in a single intersection cabinet. This paper presents an integrated framework for VRU protection at signalized intersections, combining
LiDAR, radar, RGB camera, and thermal camera at the perception layer, edge-based prediction and surrogate-safety analytics at the computation layer, V2X and P2X messaging at the communication layer, and adaptive signal control at the actuation layer. The framework is grounded in an empirical case study using R-LiViT, the first publicly released roadside
LiDAR-Visual-Thermal
dataset, which provides 200 multi-modal sequences and 2,400 annotated RGB-T frames at three German intersections. Analysis of 53,319 detection annotations reveals that VRUs comprise approximately 49% of all road-user observations, that day-to-night density drops by 38% for pedestrians and 45% for vehicles while the night distribution shows a higher close-proximity share, that per-frame close-proximity event counts vary approximately 10-fold across the eight unique locations at three intersections, and that 83% of pedestrian bounding boxes are small in image space, indicating that VRUs are typically far from any single sensor. These findings support multi-modal sensing, edge-side analytics, and adaptive context-sensitive deployment rather than uniform single-sensor solutions.
Read the full abstract ▾
Statistics Applications
Networked Spatial Effects in European Electricity Price Forecasting
Sultan Mahmud Chomon and Florian Ziel
As European bidding zones are highly interconnected by physical transmission lines, spatial influences propagate across neighboring nodes through a network. It is reflected in the day-ahead electricity prices across European bidding zones, as the auction
algorithm also uses information beyond each bidding zone's geographic boundary. To capture how this interconnection affects the electricity prices in neighboring bidding zones, we have used a metric graph to map the spatial coverage of information using a well-defined neighborhood measure. We propose the Networked Spatio-Temporal Model (NSTM), which maps irregular spatial nodes into an ordered network, enabling the systematic incorporation of neighborhood information. We implement the NSTM across 39 bidding zones covering the majority of European electricity
markets in a high-resolution, streaming-
forecasting setup. The model uses autoregressive, cross-hour, and seasonal effects, along with fuel and emission prices and day-ahead forecasts of fundamentals, as interconnected information to predict the day-ahead prices for each bidding zone. A Europe-wide study presented in this paper shows that the NSTM consistently outperforms traditional island-based pure local models. This paper provides a framework that demonstrates the critical role the networked structure plays in propagating information across interconnected
markets and its vast implications for day-ahead electricity price
forecasting.
Read the full abstract ▾
Statistical Methodology
Causal inference of Plackett-Burman designs in applications
Shuchen Chang and Zhi-ming Li
Driven by four applications of Plackett-Burman (PB) designs, this paper proposes a causal inference framework based on potential outcomes. First, we define the causal effects of the PB designs under finite populations. The Neymanian
estimator of causal effects is then obtained, including the estimated variance and covariance. Furthermore, we conduct a sharp null-hypothesis test and construct the Fisherian interval using an
algorithm. Finally, the proposed methods are illustrated through these applications.
Read the full abstract ▾
Machine Learning (Statistics)
Deep Single-Index Fréchet Regression
Muqing Cui, Yidong Zhou, Su I Iao et al. (+1 more)
Predicting outputs that are located in non-Euclidean spaces, such as probability distributions, networks, and symmetric positive-definite
matrices, is becoming increasingly important in modern data analysis, particularly when inputs are high-dimensional. We propose DeSI (Deep Single-Index Fréchet Regression), a semiparametric framework for regression with metric space-valued outputs and multivariate inputs that assumes a single-index structure for the conditional Fréchet mean. DeSI estimates an interpretable index direction, which quantifies the relative importance of inputs, using a deep neural network, and performs Fréchet regression along the resulting one-dimensional index in the target metric space. This structure mitigates the curse of dimensionality while retaining interpretability, which stands in contrast to standard deep neural networks. We establish theoretical guarantees for DeSI, including uniform approximation and convergence rates, and demonstrate its strong predictive performance through simulations on distributions, networks, and symmetric positive-definite
matrices, as well as an application to compositional mood data from New Jersey.
Read the full abstract ▾
All authors Muqing Cui, Yidong Zhou, Su I Iao, Hans-Georg Müller
Statistical Methodology
Testing Equality of Conditional Distributions via Generative Models
Hanjia Gao, Linjun Huang, Yun Yang et al. (+1 more)
We study the problem of testing whether two conditional distributions are equal using generative models. The proposed method learns a conditional generator from each sample and uses it to create responses at
covariate values observed in the other sample, allowing generated and observed responses to be compared directly. By aligning
covariates through cross-generation, the approach avoids conditional density-ratio estimation and local smoothing over high-dimensional
covariates. The population version of this construction yields a conditional discrepancy that characterizes equality of the two conditional distributions under suitable overlap conditions, while the sample version leads to a test statistic defined as the supremum of an RKHS-indexed empirical process with multiplier bootstrap calibration. A computationally efficient
algorithm for evaluating the statistic and its bootstrap analogue is developed based on alternating maximization and the kernel trick. Theoretically, we derive the limiting distribution of the test statistic under both the null and alternative hypotheses, prove bootstrap validity and consistency of the resulting test, and show that the proposed procedure attains a double-robustness property with respect to conditional generator estimation errors. Simulations and real data applications suggest that the proposed method performs well for multivariate responses and high-dimensional
covariates.
Read the full abstract ▾
All authors Hanjia Gao, Linjun Huang, Yun Yang, Xiaofeng Shao