Statistical Methodology
[DeepGOF] Where Does a Logistic Risk Model Fail? An Audited Neural Goodness-of-Fit Test for Model Development and External Validation
Ebrahim Khaled Ebrahim, Osama Abd El-Aziz Hussein and Ahmed El-Kotory
Goodness-of-fit tests for logistic regression are routine in clinical risk modelling, yet the classical tests say whether a model misfits, not where. A pretrained network is not a test until its level, power and blind spots are established. We audit DeepGOF-1, which renders the residuals of a fitted logistic model as a map over the ranks of two
covariates, scores the map with a frozen convolutional network, and calibrates the score by the analyst's parametric bootstrap. The level is first-order valid under standard bootstrap regularity and two conditions, consistency against a named alternative is decided by one forward pass, and local power has an explicit limit. In simulation the level holds from 20 patients to 8,873; the network adds power over a chi-square statistic of the same map when misfit is sparse among many
covariates; the map finds a missed interaction that one-
covariate diagnostics cannot see; and, applied to a published risk model on new patients, it is an exactly valid test of calibration within patient subgroups, where the calibration belt has little power. In the SUPPORT study of 8,873 inpatients, DeepGOF-1 rejects a linear in-hospital mortality model, its map shows the U-shaped risks of blood pressure and respiratory rate that a calibration curve hides, and it guides the repairs. It ran in 13 seconds; the projection test, the strongest omnibus rival, took 9 hours 53 minutes and 4.5 GB on the same data, too slow for routine use. The test is deepgof1() in the R package ebrahim.gof.
Read the full abstract ▾
Machine Learning (Statistics)
What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Content
Marios Papamichalis and Regina Ruane
Entropy over chain-of-thought
tokens decides which
tokens receive the policy gradient, which get pruned, and whether a run has collapsed, yet each such statistic reads a next-
token distribution mixing three choices: whether to emit connective scaffolding, which connective, and what the substantive continuation should be. Designating a scaffold vocabulary subset separates the three, exactly, for
entropy, Kullback–Leibler divergence, and the first-order
entropy velocity of a softmax policy. We prove the raw and content conventions disagree about which position is the larger fork on an explicit open region, and bound answer diversity by the content channel plus a leakage term a measured witness certifies. Across twenty-three configurations the scaffold side carries up to 41% of the raw high-
entropy set; on a matched-tokenizer ladder, coupling changes only at the math-corpus step while the scaffold's
entropy share keeps growing through
distillation; a closed-form forecast from one channel correlation tracks selection retention over a 54-point range to five points, unfitted. On compression, the content convention beats raw surprisal in every
cell; an answer-leakage audit then corrects our own headline control: re-fed chains earn a quarter to a half of their accuracy from restated answers, and once stripped, no
token scorer beats a random contiguous block.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
Aidar Amankulov and Denis Mamatin
Speculative decoding in Mixture-of-Experts (MoE) models faces the problem of unstable verification cost caused by input-dependent expert loading. To study the physics of this process, we formulate speculation-budget selection as an offline
Stochastic Shortest Path (SSP) problem over reference sequences and build a diagnostic Oracle that uses counterfactual simulation to account for MoE verification cost. A detailed analysis of the Oracle's decisions on the Qwen3-Coder and EAGLE-3 pairing, in the space of marginal deltas (Delta Space), shows that rejected candidates form a strict linear boundary. This result demonstrates that a complex global optimization is locally governed by a necessary condition balancing marginal cost against expected progress ($\frac{\Delta \mathbb{E}[Cost]}{\Delta \mathbb{E}[a]}$), providing a rigorous mathematical reference point for designing future adaptive online heuristics.
Read the full abstract ▾
Statistics Applications · Cross-listed from physics.med-ph
Impact of Multiple Non-Invasive Biosignals on Cardiovascular Biomarker Estimation via Simulation-Based Inference
Shusaku Maeda, Masahiro Nakano, Tomoharu Iwata et al. (+3 more)
As the population ages, the number of patients with cardiovascular diseases continues to increase, highlighting the need for early detection before progression to severe and irreversible functional decline. Consequently, estimating cardiovascular biomarkers from non-invasive biosignals, such as photoplethysmography (PPG) and arterial pressure wave (APW) signals, has attracted increasing
attention. These signals can be measured using wearable and cuff-type devices. Previous studies have used PPG and APW signals to estimate cardiovascular biomarkers. However, these signals exhibit strong similarities in both the temporal and frequency domains and primarily reflect peripheral and arterial pulse waveforms. Therefore, they may provide limited information about cardiac mechanical function. In contrast, the quantitative impact of additional biosignals, such as ballistocardiography (BCG), which reflect the body's minute mechanical responses to cardiac ejection, remains unclear. In this study, we generated synthetic PPG, APW, and BCG signals from a unified whole-body cardiovascular circulation model and evaluated the complementary contribution of BCG to probabilistic cardiovascular biomarker estimation. We estimated posterior distributions of cardiovascular biomarkers using neural posterior estimation and simulation-based inference. The results showed that adding BCG signals significantly improved estimation performance. Furthermore, even in ill-posed cases where PPG and APW alone produced
multimodal posterior distributions, adding BCG yielded unimodal posterior distributions. These findings provide fundamental insights into signal selection for estimating cardiovascular dynamics.
Read the full abstract ▾
All authors Shusaku Maeda, Masahiro Nakano, Tomoharu Iwata, Kenji Komiya, Ryo Nishikimi, Kunio Kashino
Machine Learning (Statistics) · Cross-listed from cs.MA
Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations
Aditya Makkar, Benjamin Unger, Jeongyeol Kwon et al. (+3 more)
Modern multi-
agent systems are increasingly deployed at scale over large populations of
agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each
agent's policy independently, treating the other
agents as part of a fixed single-
agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each
agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the
equilibrium quality of the resulting policies.
Read the full abstract ▾
All authors Aditya Makkar, Benjamin Unger, Jeongyeol Kwon, Mathieu Laurière, Eugene Vinitsky, Yonathan Efroni
Statistical Methodology
Simple Covariate Adjustment for Many Estimands Using Stable Balancing Weights
Kayla Irish, José Zubizarreta and Alex Luedtke
Covariate adjustment can improve precision in estimating treatment effects in clinical trials, and regulatory guidance increasingly encourages its implementation. Despite its benefits,
covariate adjustment can require advanced statistical techniques or tailored approaches for different estimands, which can deter its use in practice. To address this issue, we introduce a simplified approach to
covariate adjustment using stable balancing weights that applies directly across many clinical trial estimands, including the average treatment effect, relative risk, Mann-Whitney estimand, and survival ratio. More precisely, our results cover any estimand that is a Hadamard differentiable functional of the arm-specific distributions. Once the weights are obtained, our adjusted
estimator can be implemented with the same software used for an unadjusted analysis, provided that software can take observation weights. This construction improves asymptotic efficiency relative to unadjusted estimation and preserves the plug-in relationship between arm-specific marginal and subgroup-specific summaries.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.CL
Compositional Generalization via Structural Identification in a Category-Theoretic Framework
Akihiro Maeda, Thomas Seiller and Yohei Oseki
Compositional generalization is usually evaluated through model accuracy. We instead ask which structural or lexical identifications make held-out COGS examples admissible from the structures observed in training. Sentences are represented as functors from syntactic addresses to lexical
tokens, and selective collapses induce Kan extensions that propagate observed associations. Across 21 COGS generalization types, admissibility follows distinct identification profiles, while residual failures separate unsupported structural templates. These data-side diagnoses characterize what the training corpus licenses under specified identifications, without training a predictive model.
Read the full abstract ▾
Statistical Computation
Kernelized Stein Discrepancy for Goodness-of-Fit Tests and Stein Sampling in R
Junhao Gao and Ery Arias-Castro
Stein's method constructs computable discrepancies between a target distribution and a candidate distribution without requiring the target distribution's normalizing constant. These discrepancies support goodness-of-fit tests for model assessment as well as sampling tools for empirical approximation. The R package steinsampling provides the first unified R workflow for applying score-based Stein methods to kernel goodness-of-fit testing of independent or serially dependent observations, point transport, greedy point construction, and sample compression. High-level functions carry out each task in a single call, while the kernel, calibration, optimization, and transition components are provided separately so that users can replace any one of them. A single score and kernel setup can therefore be reused across sampling and testing, making these methods easier to reproduce, compare, and extend.
Read the full abstract ▾
Statistical Methodology
Online detection of distributional changes for time series in metric spaces
B. Cooper Boniece, Lajos Horváth and Lorenzo Trapani
We propose an online testing framework for detecting distributional changes in serially dependent data with values in a separable metric space. Based on two-sample $U$-statistics, the framework encompasses sequential analogs of energy distance and maximum mean discrepancy (MMD) procedures while accommodating temporal dependence. We establish asymptotic theory for finite and open-ended monitoring horizons that characterizes the full asymptotic run-length distribution under $H_0$ and yields asymptotic false-alarm control. We further establish new spectral approximation results for kernel
matrices formed from serially dependent observations, and use them to construct a feasible Monte Carlo calibration procedure. Our flexible window construction encompasses classical, Page-type, and full-scan historical-baseline monitoring and can achieve short detection delays for both early and late changepoints, without requiring sub-Gaussianity or high-order moments of the raw observations. Simulations show reliable false-alarm control across linear, nonlinear, high-dimensional, and functional time-series models and further demonstrate that, over a broad range of alternatives and changepoint locations, the proposed method can achieve substantially shorter delays than recent procedures designed specifically for rapid detection. Applications to foreign exchange rates, electricity-
market curves, and daily air transportation networks illustrate the methodology across scalar, functional, and network-valued time series.
Read the full abstract ▾
Statistics Theory
Dimension comparison for Student's statistic under symmetric unimodality
Jacopo Lenzi
Let $q_n(r)$ denote the tail probability at $r$ of the self-normalized sum of $n$ independent centered uniform variables. At $r=3$, the first distribution-sensitive term in the two-sided Edgeworth expansion of Student's statistic vanishes. We evaluate the expansion at the common moving boundary $r_n=3+\lambda/n$ in dimensions $n$ and $n-k$. Uniformly over deletion ranks retaining a fixed positive fraction of observations, the first nonzero difference converges to an explicit phase surface $H_\delta(\lambda)$; its zero curve unifies fixed, sublinear and fixed-fraction deletions, with tangent crossing $12/35$. Through the Khintchine scale-mixture representation, this comparison yields a single compactly supported $C^\infty$ symmetric unimodal parent, independent of $n$, whose Student tail exceeds the equal-scale uniform tail for all sufficiently large $n$ along nominal levels tending to $2\{1-\Phi(\sqrt3)\}$ from below. In contrast, a
quantile-ratio order shows that the uniform parent maximizes every even moment and every convergent even power series with nonnegative coefficients. We also derive the fixed-confidence dimension expansion and an exact reversal at $n=6$.
Read the full abstract ▾
Statistical Methodology
Marginal Structural Models for Electricity Demand under Treatment-Confounder Feedback: A Continuous-Treatment Outcome-Adaptive and Fused LASSO Approach
Shalini Jayanetti and Sumeet Kalia
Temperature is the most important meteorological driver of electricity demand, but its causal effect has not been estimated under the time-varying confounding that characterizes weather processes. Current temperature is associated with weather conditions, including precipitation, snow cover, and cloud cover, that also influence electricity demand, and past temperature shapes future weather, producing treatment-confounder feedback under which standard regression adjustment is biased, due to conditioning on a time-varying confounder. We estimate the causal effect of temperature on daily Ontario electricity demand from 2018 to 2019 using a marginal structural model with inverse-probability weighting, formulated for a single observed time series rather than a panel of independent subjects, and we extend the longitudinal outcome-adaptive LASSO and adaptive fused LASSO, previously developed for binary treatments, to a continuous treatment using density-ratio weights and a weighted-covariance balance criterion. In a Monte Carlo study, unadjusted regression is biased toward the null with coverage of $0.12$ to $0.15$, whereas the stabilized outcome-adaptive
estimators are nearly unbiased with coverage near $0.92$, and the cumulative three-day
estimators are approximately unbiased but less efficient. In the Ontario data all
estimators identify a positive and highly significant quadratic temperature effect; the unadjusted estimate is $9.34$, the stabilized and outcome-adaptive single-lag
estimators give $9.8$ to $10.1$, and the cumulative
estimators give smaller values that coincide with a sharp fall in effective sample size and near-unit air-density collinearity. The temperature effect on demand is therefore large and robust to single-lag confounding adjustment, while the cumulative estimates are compromised by positivity limitation that outcome-adaptive selection cannot remove.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.DL
Gender and the Production of Research Impact
Sanger Wagner, Charles Rahal and Melinda C. Mills
Evaluating the impact of scientific research beyond academia — on policy, health, the economy, and cultural life — has become a cornerstone of science policy and research-funding allocation worldwide. Yet which researchers produce the research underpinning this impact, and how this production is shaped by gender, remains poorly understood. We combine structured and unstructured records from the United Kingdom's latest Research Excellence Framework, the largest national research assessment currently in operation, with large-scale bibliometric data to quantify gender differences among the researchers underpinning documented impact. Women account for 38.16% of these contributors: underrepresented overall, but with a consistently higher share than in research-output authorships (33.63%), both overall and across all four REF panels. Impact production is also strongly gendered across domains: women are better represented in case studies concerning education, health, cultural, and civil-society impact, whereas those concerning commercialisation pathways such as patenting and manufacturing remain dominated by men. These findings reveal critical inequalities across the pathways that connect research to impact beyond academia, offering crucial evidence for policymakers and academic institutions aiming to build more equitable and representative systems for evaluating scientific contributions.
Read the full abstract ▾
Statistical Methodology
Multilevel modelling of double-sampled clustered social networks with individual-level data on between-cluster ties
Fiona Steele, Justin Weltz, Eleanor A. Power et al. (+1 more)
We consider the analysis of dyadic network data on ties between individuals in different clusters where the presence of a directed tie is reported by each individual in a dyad and the cluster-level network is of interest. A generalisation of the Social Relations Model (SRM) is proposed which includes actor, partner and dyad effects at the individual and cluster levels. The model additionally uses “double-sampling” of ties to estimate a measurement model which adjusts for and quantifies the extent of reporter effects. The model can be viewed as a type of multilevel structural equation model, with multiple cross-classified random effects, which can be estimated using Markov chain Monte Carlo (MCMC) methods in
Bayesian software. Using parameter estimates from this multilevel SRM, we then propose two alternative ways of deriving the between-cluster network that are based on predictions of the strength of between-cluster ties. Our approach is illustrated using data on social support networks in a rural community in Nicaragua where individual reports of bidirectional exchanges of support with individuals from other households are used to derive the between-household network.
Read the full abstract ▾
All authors Fiona Steele, Justin Weltz, Eleanor A. Power, Jeremy Koster
Statistical Methodology
Regularized High-Dimensional Additive Tensor Autoregressive Model
Debika Ghosh, Nilanjana Chakraborty and Samrat Roy
High-dimensional time series has diverse applications in econometrics and finance. Recent models for capturing temporal dependence have employed a bilinear representation for
matrix time series, or the Tucker-decomposition based representation in case of tensor time series. A Tucker-decomposition based temporal effect is difficult to interpret on many occasions, along with its computational complexity due to the non-convex nature of the underlying optimization problem. Moreover, the existing tensor models have not sufficiently explored the possibilities of imposing any lower-dimensional pattern on the transition
matrices. In this work, we propose a regularized additive tensor autoregressive model with additive interaction of row-wise, column-wise and tube-wise temporal dependence, that offers more interpretability, less computational burden due to its convex nature and estimation of the underlying low rank plus sparse pattern of its transition
matrices. We address the issue of identifiability of the various components in our model and subsequently develop a scalable alternating block minimization
algorithm for estimating the parameters. We provide a finite sample error bound under high-dimensional scaling for the model parameters. Finally, the efficacy of the proposed model is demonstrated on synthetic and real data.
Read the full abstract ▾
Statistics Applications
FLEX-CP-DT: A Flexible Conditional Power Framework for Interim Futility Analysis in Clinical Trials with Count Endpoints and Temporal Trends
Yanzhao Wang, Dateng Li, Ningya Wang et al. (+2 more)
Many large-scale phase III trials with recurrent event endpoints include a pre-planned interim analysis to evaluate early futility. Conditional power (CP), which quantifies the probability of achieving statistical significance at the final analysis given the interim data, is a commonly used tool to support such decisions. The standard negative binomial model with an offset term, widely adopted for analyzing recurrent events, implicitly assumes that event rates and treatment effects remain constant over the study period. At the interim analysis, however, a substantial proportion of patients have incomplete follow-up, and when the treatment effect is delayed in onset or diminishes over time, the constant-rate assumption introduces systematic bias into the interim estimate and can lead to incorrect futility decisions. In this paper, we propose FLEX-CP-DT, a piecewise negative binomial framework that captures temporal trends in both event rates and treatment effects without imposing the constant-rate assumption. The framework yields a formula-based conditional power calculation that does not require resampling or trial simulation at the interim stage. Through extensive simulations spanning constant-effect and delayed-onset scenarios, we demonstrate that FLEX-CP-DT performs comparably to the standard approach when the constant-rate assumption holds and improves interim futility decision-making when it is violated. A case study calibrated to a published phase 3 bronchiectasis trial further illustrates the practical advantage of the proposed method in reducing the probability of falsely terminating an efficacious drug with delayed treatment onset.
Read the full abstract ▾
All authors Yanzhao Wang, Dateng Li, Ningya Wang, Haitao Gao, Chenguang Wang
Statistics Applications · Cross-listed from physics.soc-ph
District-Level Food Environment Indicators and Social Vulnerability in São Paulo
Pedro Lemes Sixel Lobo, Eric Tokuda, Kuruvilla Joseph Abraham et al. (+4 more)
Urban food environments may reflect broader socioeconomic inequalities, but district-level evidence remains limited in Brazilian cities. This study examined whether indicators of food retail and street-
market availability discriminate between levels of social vulnerability across the 96 districts of São Paulo. We conducted an exploratory cross-sectional ecological analysis integrating the São Paulo Social Vulnerability Index (IPVS), establishment records from the Relação Anual de Informações Sociais (RAIS), and street-
market data from CAISAN. Census-sector information was aggregated at the district level. Twenty districts without an IPVS classification were excluded, resulting in 76 observations. The outcome distinguished districts classified as IPVS level 1 from those classified as levels 2–7. Predictors described the densities of healthy and unhealthy food establishments, the number of street
markets, and the availability of establishments selling fresh or in natura food. Eight conventional machine-learning classifiers were evaluated using leave-one-out cross-validation. Reported mean F-scores ranged from 0.62 to 0.75, with XGBoost obtaining the highest value. In the Random Forest model, the densities of healthy and unhealthy food establishments jointly accounted for approximately 60% of the total impurity-based feature importance. These findings indicate that publicly available food-environment indicators contain information associated with the district-level distribution of social vulnerability. However, the small ecological sample, class imbalance, outcome binarization, and cross-sectional design limit predictive generalization and preclude causal or household-level interpretations.
Read the full abstract ▾
All authors Pedro Lemes Sixel Lobo, Eric Tokuda, Kuruvilla Joseph Abraham, Roberto Fray, Dirce Maria Marchioni, Alexandre Cláudio Botazzo Delbem, Rogerio Salvini
Machine Learning (Statistics) · Cross-listed from cs.LG
Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization
Mingyi Li and Taira Tsuchiya
Muon has emerged as a strong optimizer for the
matrix-valued parameters in large
language model pretraining, approximately orthogonalizing its momentum with a few Newton-Schulz iterations. Existing theory either replaces this iteration with the exact polar factor it approximates, or treats its finite depth as an approximation error, and thus the iteration
Muon actually runs can only hurt the guarantees. We show that finite Newton-Schulz can instead be beneficial for nonsmooth nonconvex optimization. To this end, we analyze
Muon through the online-to-nonconvex conversion, which views the update rule as an online learner and converts its regret bound into a stationarity guarantee. The finite Newton-Schulz iteration smooths the discontinuous polar map into a Lipschitz map of the singular values, and
Muon with finite Newton-Schulz can be regarded as an online learner with a smoothed spectral potential. This smoothing is exactly what the conversion needs: we prove that a Newton-Schulz depth growing only logarithmically in the target accuracy suffices for convergence to stationary points in nonsmooth nonconvex optimization, whereas
Muon with the exact-polar update may fail to converge. The resulting sample complexity bounds match the best-known guarantees for nonsmooth nonconvex optimization and are optimal for smooth nonconvex optimization up to problem-dependent factors. The argument extends beyond Newton-Schulz to general spectral maps with the same smoothing property.
Read the full abstract ▾
Statistical Methodology
A Latent-Space Statistical Learning Framework for Semicontinuous Outcomes: Regularized Deviance and Boundary-Stabilized Optimization
Jianping Philip Wang
Semicontinuous outcomes characterized by an exact zero mass and heavy-tailed continuous distributions challenge empirical risk modeling and machine learning. Compound Poisson-Gamma Tweedie processes impose a rigid mean-variance coupling that distorts zero-mass probabilities under extreme tail dispersion, flattening continuous density and reducing extreme-risk discrimination. Furthermore, heuristic post-hoc target capping introduces systematic bias while leaving loss gradients vulnerable to tail leverage. We present a unified, statistically coherent likelihood framework for semicontinuous modeling under arbitrary zero-inflation and explicit upper boundary constraints. Optimization and likelihood evaluations operate strictly in a latent space defined by the power-scaling mapping $y_i^* = \min\{(y_i/s)^\lambda, U^*\}$, where $s \in (0, U)$ anchors scale and $\lambda \in (0,1)$ regularizes tail
curvature. The architecture partitions the response into three regimes: an unconstrained Bernoulli hurdle, an analytical continuous Gamma body, and an upper boundary point mass governed by incomplete Gamma survival ratios. Rather than an arbitrary truncation, the boundary functions as an intrinsic tail accumulator bounding score and Hessian dynamics. We derive the closed-form deviance objective, score vectors, and block-diagonal Hessian
matrix, proving that parameter spaces are information-orthogonal to enable concurrent classification and regression updates. Asymptotic boundary analysis demonstrates that score and
curvature scale linearly in the deep tail, establishing a quadratic restoring force that prevents gradient collapse. The framework breaks the Tweedie parameter rigidity paradox while maintaining structural fidelity across both zero occurrence and extreme severity differentiation.
Read the full abstract ▾
Statistics Theory · Cross-listed from eess.SP
(Sequential) Joint Detection and Estimation: Classic Results and New Directions
Dominik Reinhard and Abdelhak M. Zoubir
We provide an overview of the problem of jointly testing two hypotheses and estimating a parameter of the selected model. Such problems arise in a variety of applications. First, we present a conceptual introduction to suboptimal and optimal procedures for joint detection and estimation. A numerical example illustrates the advantages of the optimal procedure over suboptimal ones. Next, we discuss how more advanced problem formulations affect the presented results. The second part covers joint detection and estimation in a sequential framework. First, we provide an introduction to sequential analysis through sequential hypothesis testing. Then, suboptimal and optimal sequential procedures for joint detection and estimation are discussed. A numerical example shows the advantages of optimal sequential procedures over suboptimal and optimal sequential procedures with a fixed number of samples. The third part discusses open problems and future research directions in joint detection and estimation.
Read the full abstract ▾
Statistics Applications
A causal graph-informed temporal convolution architecture for interpretable retail electricity price forecasting
Yufan Ji, Abdollah Shafieezadeh and Noah Dormady
Retail electricity
markets in deregulated systems face significant price
volatility and complex interactions with forward and futures products, posing challenges for effective operational decision-making. This study introduces a Causal Graph-Informed Temporal Convolutional Network (CG-TCN), a forecasting architecture that integrates a learned causal graph into a temporal convolutional network via a graph-neural
embedding to enhance both forecasting accuracy and interpretability of retail electricity price dynamics. It first applies a multi-resolution decomposition to isolate semiannual, quarterly, and monthly trends from high-frequency fluctuations. A causal graph is then discovered over these components and key
covariates, including wholesale forward prices and retail contract attributes such as early termination fees, with domain constraints that preserve causal directionality and exogeneity. The learned causal structure is encoded as an adjacency
embedding that conditions the TCN's convolutions and
attention, aligning representation learning with causal pathways. Using ten years of daily 12-month fixed-price residential contracts from Ohio's deregulated
market, we find that wholesale forward prices primarily determine long-term retail price trends, whereas contract attributes influence short-term fluctuations. CG-TCN consistently outperforms
benchmark models, achieving mean absolute percentage errors of 3.08%, 3.82%, and 5.43% for one-, ten-, and fifteen-step-ahead forecasts of daily retail electricity median prices, respectively. By combining predictive performance with interpretability, CG-TCN provides transparent, policy-relevant insight to support
market analytics, consumer protection, regulatory oversight, risk assessment and procurement planning in competitive electricity
markets.
Read the full abstract ▾
Machine Learning (Statistics)
TRACE: Retrospective Streaming Generation of Physical Fields under Sparse Structured Sensing
Xinyu Zhang, Lihao Chen, Panqi Chen et al. (+4 more)
Reconstructing continuous physical fields from sparse measurements is central to scientific monitoring, inverse modeling, and digital-twin construction. Generative reconstruction has recently emerged as a promising paradigm for this task by learning data-driven physical priors that complete plausible full fields from limited observations. However, existing methods largely assume fixed, batch conditioning, whereas real sensing systems often produce structured streams: probes scan local regions, instruments observe moving fields of view, and communication constraints may leave entire frames missing. We propose TRACE, a retrospective streaming generative reconstruction framework for physical fields under structured sensing. TRACE performs approximate
Bayesian inference in a learned continuous-coordinate latent space, converting sparse off-grid measurements into generative latent evidence, fusing it with a state-space temporal prior through Kalman-style filtering, and refining under-observed past frames via retrospective smoothing. Experiments on active matter, ocean sound-speed fields, and
supernova simulations show that TRACE matches or surpasses frame-wise generative reconstructors, offline spatiotemporal methods, and streaming data-assimilation baselines in reconstruction quality under temporally sparse and spatially localized sensing protocols.
Read the full abstract ▾
All authors Xinyu Zhang, Lihao Chen, Panqi Chen, Lei Cheng, Ting Zhang, Jianlong Li, Shikai Fang
Statistical Methodology
Copula based dependent censoring in cure models with covariates
Morine Delhelle, Anouar El Ghouch and Ingrid Van Keilegom
In survival analysis, the time-to-event variable T is frequently subject to right censoring. Individuals may withdraw from the study for various reasons, or may not experience the event of interest before the end of follow-up. In this paper, we distinguish between two types of censoring: a potentially dependent censoring time C, which may be stochastically related to T, and an independent administrative censoring time A. In addition, the data may exhibit a cure fraction, meaning that some individuals will never experience the event. We build upon a recent work about a fully parametric mixture cure model, which accounts for dependent censoring through copulas. The proposed extension incorporates administrative censoring and allows
covariates to affect all model parameters. This framework enables a more accurate modelling of the dependence between survival and censoring times while providing greater flexibility through
covariate effects, leading to more individualised estimation of the cure fraction, the dependence structure, and other clinically relevant quantities. Moreover, the presence of
covariates allows for weaker identification conditions.
Read the full abstract ▾
Statistics Applications
Beyond Participant-Level Cross-Validation: Reliable Inference for Longitudinal Machine Learning
Shahran Rahman Alve
Longitudinal sensing studies routinely collect thousands of windows from a few dozen participants. The records are numerous; the independent scientific units are not. When the outcome is defined per participant, this mismatch makes apparently precise findings vulnerable to pseudo-replication, to partition choice, and to the ordinary analytic flexibility of comparing several pipelines before reporting one. Splitting on participants prevents a person's records from straddling a split, but it does not calibrate the label-dependent workflow fold construction, preprocessing, tuning, calibration, and candidate selection that produced the reported number. We define a participant-level estimand and obtain an analysis-matched null by permuting participant labels and rerunning that entire workflow. In controlled simulation, window-level inference rejects in 70-80% of replicates when no effect exists and a window bootstrap rejects at the same rate; a participant bootstrap still rejects at 10-17%; the analysis-matched test holds 0.025-0.100 across cohorts of 20 to 80 participants. Freezing the selected pipeline instead of repeating the search inflates Type-I error to 0.240 with eight candidates, where repeating it holds 0.040. Applied to two public cohorts, wrist actigraphy (n=55) yields participant AUROC 0.928 with p=0.0050, a conclusion that persists under a scale-robust rank-pooled statistic and under a matched permutation null computed after excluding hospitalized participants (p=0.0089). Smartphone sensing (n=38, 7 positives) yields 0.636 and does not reject (p=0.1724) despite sufficient resolution, with sensitivity 0.143. The practical rule is narrow: every step that reads labels belongs inside the permuted analysis, and repeated records do not create additional independent participants.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
ICON decomposition: Multivariate concept auditing reveals shortcuts that neural networks rely on
Roshan Prakash Rane, Marco Simnacher, Manuel Pfeuffer et al. (+7 more)
Deep neural networks often exploit spurious associations, a failure known as shortcut learning. Before deployment, models should therefore be audited for reliance on concepts such as acquisition artifacts and patient demographics. Current methods, including linear probes and concept activation vectors, measure concept reliance by asking whether each concept is decodable from a layer. We show that this framing is misleading: decodability reflects not only concept reliance but also correlations among concepts and the outcome in the auditing
dataset. We reframe shortcut auditing as a decomposition of the variance of intermediary representations with Independent Canonical cONcepts (ICON). ICON quantifies the share of a layer's variance explained by each concept conditional on all other concepts and the outcome, and also reports the variance left unexplained. Because ICON explanations are comparable across layers, their trajectories distinguish concepts that drive predictions from those that are merely encoded in the representations. In simulated data with ground truth, ICON recovers concept importance more accurately than seven baselines. In skin-cancer and neuroimaging applications, it separates learned shortcuts from merely correlated concepts, which we confirm with independent validation checks.
Read the full abstract ▾
All authors Roshan Prakash Rane, Marco Simnacher, Manuel Pfeuffer, Marc-Andre Schulz, Nys Tjade Siegel, Maximilian Dreyer, Frederik Pahde, Wojciech Samek, Sonja Greven, Kerstin Ritter
Machine Learning (Statistics) · Cross-listed from cs.CL
Fine-Tuning Whisper for Automatic Speech Recognition in Baniwa: A Preliminary Study
Leonardo Duart, Tiago Fonseca and Thiago Chacón
Automatic
Speech Recognition (ASR) technologies have achieved remarkable performance in recent years through the use of large multilingual foundation models. However, most advances remain concentrated on high-resource
languages, while indigenous
languages continue to suffer from a lack of
speech resources and
language technologies. This work presents a preliminary study on the adaptation of Whisper for Automatic
Speech Recognition in Baniwa, an indigenous Arawakan
language spoken in Brazil, Colombia, and Venezuela. The experiments were conducted using a corpus of 1,373 manually transcribed recordings obtained from a linguistic documentation project. The corpus contains approximately 0.54 hours of
speech and consists primarily of isolated words and short elicited utterances. The Whisper Small model was fine-tuned using supervised learning and evaluated using Word Error Rate (WER) and Character Error Rate (CER). The best model achieved a WER of 37.5% and a CER of 7.45%, demonstrating that multilingual foundation models can be successfully adapted to extremely low-resource indigenous
languages. The results establish an initial baseline for Baniwa Automatic
Speech Recognition and provide a foundation for future research involving larger
datasets,
language-specific adaptation strategies, and post-processing techniques.
Read the full abstract ▾
Statistics Theory
Statistical Properties of Nonparametric MLE under Laplace Noise
Yifei Xiong, Nianqiao Phyllis Ju and Vinayak Rao
Local differential
privacy (LDP) protects individuals in a
dataset by perturbing each measurement before release. For real-valued data, a widely used mechanism is additive Laplace noise. We study the problem of estimating the distribution of the latent confidential data from the privatized observations via the nonparametric maximum likelihood
estimator (NPMLE) under an i.i.d. sampling model. We first show that under the Laplace convolution model, the NPMLE admits a finite-dimensional reformulation in which the support is restricted to the observation set. This reduces the original infinite-dimensional optimization over all mixing distributions to an $n$-dimensional convex optimization over mixture weights, where $n$ is the sample size. We then study the statistical convergence of the NPMLE under the 1-Wasserstein distance, and explicitly connect its convergence rate with the
privacy noise scale. Allowing the
privacy noise level to change with the sample size, our analysis shows that the NPMLE remains consistent when the Laplace noise grows at a rate slower than $n^{3/16}$. Conversely, when the Laplace noise is of the order $\sqrt n$ or larger, no
estimator can achieve uniformly consistent recovery of the latent distribution.
Read the full abstract ▾
Statistical Methodology
The Symmetric Pair Matching Design: A Self-Controlled Method with Automatic Adjustment for Time Effects
Robin Denz, Filippo Saatkamp, Katharina Meiszl et al. (+1 more)
Self-controlled study designs eliminate confounding by individual-level characteristics that remain constant during the observation time and are thus widely used in pharmacoepidemiology and vaccine safety research. However, existing methods remain vulnerable to time effects, including temporal trends and seasonality in the exposure or outcome, unless these are explicitly modeled or controlled through the study design. We introduce the symmetric pair matching design (SPM), a novel self-controlled method that combines design-based adjustment for time effects with automatic control of time-invariant confounding. Unlike previous design-based approaches that account for time effects, SPM uses observation time both before and after event occurrence, thereby retaining a larger proportion of the available information. We derive the theoretical properties of the method and evaluate its finite-sample performance through simulations. In the simulations, SPM produced unbiased estimates in the presence of temporal trends in both the outcome and exposure, while maintaining greater or comparable statistical efficiency than existing approaches. SPM extends the family of self-controlled methods by providing design-based control of time effects without requiring explicit modeling thereof. By combining robustness to temporal confounding with efficient use of observation time, SPM offers a practical alternative for observational studies with transient exposures. An accompanying R package is provided to facilitate its usage in applied research.
Read the full abstract ▾
All authors Robin Denz, Filippo Saatkamp, Katharina Meiszl, Nina Timmesfeld
Statistical Methodology
Controlling for Omitted Variable Bias in Deep Neural Networks
Manuel Pfeuffer, Roshan Prakash Rane, Kerstin Ritter et al. (+1 more)
Control variables are widely used in statistical modelling to account for omitted variable bias of known confounders. However, they have largely been underexplored in deep learning. This is surprising, given that deep learning models encode image-inferable
covariates, such as demographic variables, into their predictions when these
covariates are correlated with the outcome—a form of omitted variable bias referred to as 'shortcut learning'. While many existing confound-control or
fairness methods try to restrict the correlation of such
covariates with model predictions, we show that this fails to correct for omitted variable bias. We therefore propose a control variable approach for deep learning models, based on generalised additive modelling of the effects of model inputs and
covariates. As flexible additive models can suffer from concurvity, we introduce an estimation procedure that refits the final layer of a pre-trained network to include
covariate effects, using cross-fitting with ridge penalisation. We show how these effects can be orthogonalised with respect to
covariates to exclude their mediated effects and that model predictions can be marginalised over the
covariate distribution to control for their effect. This yields unbiased, interpretable predictions and offers flexibility to model the desired effects depending on the scientific or
fairness objective. We verify our approach using simulated images, and demonstrate consistent estimation of true effects. Existing methods either require more data or fail to recover the true effects. We apply our method to real neuroimaging data with experimentally induced confounding, where it recovers prediction performance to near the level of a model trained on unconfounded data. Code is available at https://github.com/mpff/cocodeel.
Read the full abstract ▾
All authors Manuel Pfeuffer, Roshan Prakash Rane, Kerstin Ritter, Sonja Greven
Statistical Methodology
Random Invariance Testing on Quadratic Form Statistics with Application to Autocorrelation
Amitakshar Biswas and Adam B Kashlak
Randomization testing with permutations is a very common nonparametric approach to hypothesis testing. However, randomization testing can be done with other group transformations including random rotations. In this work, we consider the problem of invariance in quadratic form statistics under a unified framework with closed form p-values. In particular, we propose a nonparametric variant of the classic Durbin-Watson test for testing for autocorrelation in time series data at arbitrary lags. Our test is performed by integrating over a group of invariances of the test statistic, and easy-to-compute analytic formulae for the p-value are derived from concentration inequalities on compact groups on a per-lag basis. Thus, the usual necessity of large-scale Monte Carlo simulations is rendered unnecessary. Our tests outperform the classic Breusch-Godfrey and Ljung-Box tests on simulated data with respect to statistical power to identify significant autocorrelation. They also can be used to identify the presence of autocorrelation at large lags such as in monthly solar intensity data, which follows an approximate 11 year (132 month) cycle. The general formulation of this approach can be easily adapted to other quadratic form statistics.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Quantum-Inspired Modeling of Driving Behavior
Mohammad Elayan, Omid Armantalab and Wissam Kontar
Driver behavior is heterogeneous, context-dependent, and changes over time, and these properties shape the traffic phenomena we observe. Most models, however, fix in advance which behavioral variables interact and how. Behavior outside that form is absorbed as noise, while models flexible enough to capture it tend to lose interpretability. We introduce a
quantum-inspired representation of driver behavior that combines properties usually treated separately or in part: it is continuous, probabilistic, context-dependent, history-dependent, and represents interactions among behavioral variables as learned from data. Each driver is encoded as an evolving density
matrix, providing a unified representation of behavioral uncertainty, temporal
evolution, and context-dependent behavioral variation. Trained without supervision on the I-24 MOTION
dataset, the framework recovers three interpretable driving profiles representing three regimes: free flow, transition, and congestion. The profiles capture the behavioral range of the data and the smooth transitions drivers make between regimes as conditions change. The same representation also reproduces known macroscopic phenomena, aligning with the fundamental diagram and reproducing hysteresis loops. We also show how the representation supports practical use: it supplies context-dependent parameters to classical car-following models, and gives an autonomous vehicle a live behavioral read of the surrounding drivers with a short-horizon forecast of their motion. The framework points toward models of traffic that are interpretable and trustworthy by construction. We release an open-source toolkit on GitHub (https://github.com/mselayan/
quantum-driver-representation) spanning data processing, training, inference, and analysis.
Read the full abstract ▾