Statistics Applications · Cross-listed from stat.ML
Learning to Price Electricity for Optimal Demand Response
Jing Shang, Mohammad Mehrabi, Xinyang Zhou et al. (+3 more)
There is considerable interest in using time-varying electricity prices to shape consumer demand response, and better align energy demand with renewable production. However, optimal prices generally vary over time in response to complex signals such as weather forecasts, sunrise/sunset times, and day-of-week patterns; and existing methods are not able to make efficient use of such rich contextual information. Here, we propose a neural-network-based
algorithm for contextual energy pricing, modeling pricing as a Stackelberg game and leveraging a mean-field solution representation from Mehrabi et al.(2024). The approach learns constrained mappings from contextual features to feasible price signals. We validate our approach by simulating the energy grid in several US cities, and show that incorporating contextual information can considerably increase the value of the demand response programs.
Read the full abstract ▾
All authors Jing Shang, Mohammad Mehrabi, Xinyang Zhou, Mahmoud Saleh, Andrey Bernstein, Stefan Wager
Statistics Applications · Cross-listed from cs.AI
Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard
Michael Hardy, Ruhana Azam, Anka Reuel et al. (+2 more)
Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current
agent evaluations reliably support and what additional evaluation would improve them. We develop a
Bayesian variance-decomposition framework for sparse, imbalanced
agent leaderboards and apply it to 22
benchmarks from the Holistic
Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a
benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse
benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling
benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.
Read the full abstract ▾
All authors Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer, Sanmi Koyejo
Statistics Applications · Cross-listed from stat.ME
Foundation or Formula? A Simulation-Based Comparison of Google's TimesFM-3 and Classical Time-Series Models
Ebrahim Khaled Ebrahim, Somaia Mohamed Ali and Ahmed El-Kotory
Time-series foundation models such as Google's TimesFM-3 forecast series they were never trained on, but whether they should replace classical methods such as ARIMA and exponential smoothing is hard to settle on public
benchmarks, because few
benchmark datasets are absent from every model's pre-training corpus. This exploratory study compares TimesFM-3 with classical methods on data generated from nine known processes, which the model cannot have seen. Across 7200 series and a pre-registered protocol with a 12-step horizon, neither family dominates. TimesFM-3's error is never more than 1.31 times that of the best method in a scenario, whereas every classical method's is at least 2.2 times somewhere; the largest classical failures occur with only two seasonal cycles, where the automatic methods fall back to non-seasonal models. With four or more cycles automatic ARIMA is 21-24% more accurate than TimesFM-3 on seasonal ARIMA data. Among five foundation models this robustness is specific to TimesFM-3, and it holds only at short horizons: at steps 25 to 48 the three foundation models tested there forecast a decline on a saturating process whose level stays flat, and automatic ARIMA has the smaller worst case elsewhere. On intermittent demand TimesFM-3 shows no advantage over simple
benchmarks built from the history. Randomised parameters, heavy-tailed noise and outliers leave these patterns qualitatively unchanged; on the official test period of 1000 M4 monthly series TimesFM-3 is level with the fifth- to seventh-ranked M4 entries, behind the four best, and on 101 macroeconomic series observed after the models' documented training data it is statistically level with the classical methods. The study characterises the behaviour of a black-box model, not the reasons for it, and its findings are conditional on the processes and horizons examined.
Read the full abstract ▾
Statistics Applications · Cross-listed from stat.ME
Nonparametric Identification of Latent Dimension under Heavy-Tailed and Mixed-Signals
Chetkar Jha
Principal component analysis and factor analysis are foundational to the study of high-dimensional data, yet their efficacy depends entirely on correctly identifying the latent dimension $r$. While parallel analysis (PA) is widely regarded as the gold standard for this task, its performance suffers in high-dimensional regimes consisting of mixed signals and/or heavy-tailed error distributions. This arises because under the preceding conditions, column-wise permutation mechanism (or any modern variations) applied to the data
matrix $\textbf{X}$ fails to destroy the signal, leading to an inflated null spectrum. To address this, we propose a nonparametric solution. We show that the latent dimension is uniquely preserved in the degree of order within the rank-transformed data
matrix, which restores spectral exchangeability under permutation. We prove the consistency of Rank PA under relaxed conditions, requiring neither the pervasiveness of factors nor restrictive fourth moment assumptions. We establish that Rank PA can consistently estimate the number of factors up to the maximum of $O(n^{1/3 - \tau})$ where $\tau >0$. The robust performance of our remedy is demonstrated through numerical simulations and an analysis of the FRED-MD
dataset.
Read the full abstract ▾
Statistics Applications · Cross-listed from stat.ME
Three-group variance-ratio tests for heteroscedasticity in linear regression: an exact null distribution and an outlier-resistant version
Ahmed El-Kotory and Ebrahim Khaled Ebrahim
Tests for heteroscedasticity in linear regression lose their level or power in three situations: when the data contain outliers, when the variance is not monotone, and when it changes along a variable outside the regressors, such as time. The proposed tests sort the observations by any chosen variable, split them into three equal parts, fit the regression in each part and compare the largest with the smallest error scale. With least squares fits and normal errors, the ratio follows Hartley's maximum F distribution with three groups exactly. With least trimmed squares fits, the ratio resists outliers spread along the ordering. Its square is approximately a maximum F ratio with effective degrees of freedom, whose limit we derive in closed form from the influence function of the trimmed variance. A factorial simulation covered 72 settings of variance shape, ordering variable, number of regressors, contamination and sample size. The robust test had the highest mean size-adjusted power, 67%, against 45% for the next test; when the variance changed along a regressor, which all tests were given, it led White's test after an outlier screen, 67% against 56%. With heavy-tailed t3 errors its lead was similar, 64% against 35%. Unlike the tests built on the regressors, it can follow a variance changing along time, and it identifies the part of the data where the variance changes. Two published robust versions of the Goldfeld-Quandt test, as implemented from their descriptions, did not hold their nominal level. The R package KOTORY implements the methods.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.LG
SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions
Liang You, Hengyu Shi and Dongwen Ou
Conformal prediction guarantees marginal coverage, but a single calibration threshold can still spread that coverage unevenly, over-covering easy regions and under-covering hard ones. SimplexUQ is, to our knowledge, the first
benchmark and reproducible protocol for measuring this allocation problem on simplex-valued predictions; it compares existing conformal wrappers rather than proposing a new one. Its task suite, SimplexTasks-12, combines six controlled synthetic regimes with six frozen-predictor real tasks spanning class probabilities, topic mixtures, spectral abundances,
cell-type fractions, age distributions, and emotion mixtures. Each comparison fixes the predictor, score, and response-free stratification map, varies only the wrapper, and reports marginal coverage, worst-stratum coverage, max disparity, and within-task radius and compute. Global calibration can look valid while failing badly: on CIFAR-10 it attains 0.900 marginal coverage but only 0.542 in the worst
entropy stratum, and Mondrian calibration raises that stratum to 0.886 while reducing max disparity from 0.358 to 0.022. No wrapper dominates, however. Under smooth synthetic heterogeneity, several repairs are competitive; fixed-map analyses show that rankings depend on the evaluation groups and protocol; and in a 12-task comparison, Mondrian has lower disparity on its single target partition for all 12 tasks, whereas BatchMVP has lower disparity over overlapping groups on five. These are empirical comparisons, not new coverage guarantees. A controlled predictor-bias sweep shows that removing predictor bias only partly reduces global-threshold disparity. We release task cards, result provenance, permitted derived arrays, and rebuild instructions, and treat wrapper selection as a diagnostic comparison rather than a universal ranking.
Read the full abstract ▾
Statistics Applications · Cross-listed from physics.ao-ph
reGRAF: a global MPAS reforecast data set with convection-allowing refinements over the US and Europe
Thomas M Hamill, Akshay Subramaniam, Montgomery Flora et al. (+8 more)
NVIDIA and The Weather Company (TWC) have generated a data set of reforecasts from TWC's GRAF (Global high-Resolution Atmospheric Forecasting) model, a version of the National Center for Atmospheric Research (NCAR) Model for Predictions Across Scales (\href{https://ncar.ucar.edu/what-we-offer/models/model-prediction-across-scales-mpas}{\underline{MPAS}}). GRAF is global, but the configuration for this reforecast had a mesh refinement to \textasciitilde4 km over the US, Caribbean Basin, and Europe, and 15 km elsewhere. This model was designed to run much of the computation on graphical processing units (GPUs), with this development assisted by NVIDIA. The 1836 reforecast cases (\textasciitilde5 years) were generated from ECMWF reanalyses (ERA5) for selected initial condition dates spanning more than 20 years, 2004–2024. These dates of the chosen initial conditions were mostly selected based on high-impact weather in the contiguous US (CONUS) and Caribbean. Sampling in this way, the reforecast spanned a wider range of interesting, high-impact weather scenarios than had we performed five contiguous years of once-daily reforecasts. The reforecast still provides many samples in non-precipitating regions with more ordinary weather. GRAF reforecasts were mostly run to $+$27 h lead time, assuming a 3-h
spin-up followed by a full diurnal cycle. Data were saved in zarr format on the native model vertical coordinate. Most fields were archived at 15-min intervals, though several precipitation variables were saved at 5-min cadence. Data are made publicly available to all through Amazon Web Services' Open-Data Initiative at https://registry.opendata.aws/graf-reforecast.
Read the full abstract ▾
All authors Thomas M Hamill, Akshay Subramaniam, Montgomery Flora, Raghu Raj Prasanna Kumar, Karthik Kashinath, Carl Ponder, Tao Ge, Sepideh Khajehie, John Wong, Brett Wilt, Peter Neilley
Statistics Applications · Cross-listed from cs.LG
Efficient Active Auditing of Multi-Group Fairness with Bias Probes
Ayoub Ajarra and Debabrota Basu
Over the past decade, Machine Learning (ML) has been trained under dual objectives: minimizing prediction error via Empirical Risk Minimization (ERM) while controlling unfairness bias. In practice, however,
fairness-aware training often yields limited improvements over standard ERM, making reliable post hoc auditing essential. Existing auditing approaches for black-box models either rely on model reconstruction –exposing systems to extraction attacks– or directly estimate
fairness metrics, offering limited insight into which regions of the data distribution drive bias. More fundamentally, property-specific auditing –aimed at extracting only targeted
fairness information without reconstructing the model– remains poorly understood. In this work, we introduce the bias probe framework, which enables targeted and adaptive querying to reveal bias structure while preserving model confidentiality. Building on this framework, we propose ALeBi, an active auditor that learns such probes to efficiently estimate multi-group
fairness metrics. We establish novel sample complexity guarantees governed by a property-specific complexity measure, resolving a previously posed open question, and extend our analysis to
adversarial settings where the model owner may strategically obscure bias. Our results uncover a fundamental trade-off between model confidentiality and reliable auditing, and show that property-specific probing enables both accurate estimation and interpretable identification of high and low-bias regions. Extensive experiments support our theoretical findings and demonstrate the practical effectiveness of our approach.
Read the full abstract ▾
Statistics Applications
Evidence-to-Decision (E2D): Structuring Action-Specific Evidence for Early-Phase Oncology Development
Haitao Pan
Early-phase oncology evidence is often summarized by asking whether a treatment is promising, but an encouraging signal can support several different next studies. We propose Evidence-to-Decision (E2D), a framework that organizes evidence around specific development actions. For each candidate action, E2D defines the contemplated future study, what would count as success, what evidence is needed before taking that step, what evidence is available now, and what assumptions connect the current and future settings. Statistical methods can then quantify action-specific support and sensitivity to those assumptions, while the final development choice remains multidisciplinary. We illustrate E2D in a rare pediatric oncology setting using predictive probabilities for same-setting single-arm expansion, same-setting randomized Phase II, movement to an earlier-line or frontline randomized study, and a separate de-escalation example. Calibrated support rules produced clinically distinct evidence states: intermediate evidence could support same-setting development without supporting frontline movement, safety concerns could narrow support despite strong activity, and stronger evidence could support several paths simultaneously. E2D therefore reframes the question from whether a treatment is generally promising to which next studies are supported by the evidence available now, what remains unsupported, and what additional evidence could change that assessment.
Read the full abstract ▾
Statistics Applications
Days Alive at Home vs Out of Hospital: Why the Difference Matters for Trial Endpoints
Letao Yuan, Sarah N. Dawson, David S. Robertson et al. (+5 more)
Days Alive and Out of Hospital (DAOH) and Days Alive at Home (DAH) are increasingly used as patient-centred primary outcomes in perioperative trials, and a recent editorial has advocated DAOH for its simplicity and reliance on routinely collected data. We argue that the choice between these endpoints affects power, treatment effect estimation and missing data, and should be made at the design stage with these consequences in mind. By treating days in nursing homes or rehabilitation facilities as equivalent to days at home, DAOH assumes that being out of hospital is a valid surrogate for good recovery, an assumption that patient perspectives, including those from the NOTACS trial, call into question. Choosing DAH over DAOH therefore involves a trade-off between measuring what matters to patients more accurately and the added burden of tracking discharge destination, including a greater risk of missing data. If an intervention affects only length of stay, readmission or mortality, the two endpoints yield the same expected treatment effect, and the extra burden of tracking discharge destination brings no benefit. If it instead enables more patients to return directly home rather than to a care facility, DAOH may conceal this benefit and lose statistical power. We propose a baseline-adjusted DAH, which counts only days spent in a setting representing an escalation of care relative to the patient's baseline, as a better proxy for recovery that also accommodates hospital-at-home and virtual ward services.
Read the full abstract ▾
All authors Letao Yuan, Sarah N. Dawson, David S. Robertson, Yue Lan, Andrew A. Klein, Dominique-Laurent Couturier, Mia S. Tackney, Sofía S. Villar
Statistics Applications
Plug-in Optimization Method for Penalty Parameters in Nonparametric GMANOVA model
Isamu Nagai
In order to estimate the longitudinal trend, we often use the generalized multivariate analysis of variance (GMANOVA) model (Potthoff & Roy, 1964), when the longitudinal data is balanced data. Usually, we use this GMANOVA model with some
polynomial at the time of measurement. However, when the longitudinal trend has flexible curve, we cannot derive good fitting estimated curve when we use some
polynomial curves. Then, Nagai (2011) proposed the nonparametric GMANOVA model which uses on several known basis functions instead of using the
polynomial curves. If we use several basis functions, then overfitting problem is occurred. Nagai (2011) also proposed the estimation method for avoiding overfitting and unstable problems, and reducing computational iterative
algorithm by extending the generalized ridge regression model (Yanagihara, Nagai & Satoh, 2009). In the present paper, we extend one of the optimization methods in Nagai, Yanagihara and Satoh (2012) into the estimation method in Nagai (2011). Through numerical studies, we show some properties of each optimization method.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.LG
A Rank Graduation metric for Algorithmic fairness
Dalia Atif and Paolo Giudici
Fairness assessment in algorithmic decisions that affect individuals, such as credit scoring, often relies on parity measures calculated at the aggregate group level. Such measures may not reveal which individuals experience unfairness or which explanatory factors contribute to it. In this paper, we propose a rank-based framework that evaluates
fairness through the distribution of model prediction errors, thereby linking
fairness assessment with predictive accuracy and explainability. The framework combines Rank Graduation
Fairness (RGF), its integrated measure AURGF, a centered Cramer–von Mises permutation test, and a feature removal procedure for
fairness explainability. We evaluate the methodology using logistic regression, random forest, gradient boosting, and a multilayer perceptron. The simulation study shows that protected-group imbalance can reverse descriptive
fairness comparisons, whereas the proposed inferential procedure correctly distinguishes fair from unfair mechanisms. Its application to HMDA mortgage data produces model rankings that differ from those obtained with classical
fairness criteria. Tree-based models, rather than logistic regression, provide the strongest combination of predictive accuracy and rank-based
fairness, while the
fairness null hypothesis is rejected for all four models. The persistence of disparity across statistical, bagging, boosting, and neural network specifications, together with the feature removal results, indicates that the observed unfairness is not specific to a single
algorithm or predictor, but is associated with group differences embedded in the characteristics of the lending data. These findings support a broader approach to trustworthy artificial intelligence that combines predictive accuracy,
fairness measurement, statistical inference, and explainability.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.AI
Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting
Liang Twist Shan, Tianyu Hu, Hao Yan et al. (+1 more)
Large
language models (LLMs) have been rapidly improving in long-context tasks, powered by Chain-of-Thought (CoT)
reasoning. However, the internal mechanisms underlying this improvement remain unclear. We investigate these mechanisms through a needle-in-a-haystack (NIAH) counting task, where an LLM is asked to count the number of records dispersed in a long text. Across twelve model comparison groups, Thinking (or
reasoning) improves counting accuracy over Non-thinking, with pronounced gains at larger counts. This motivates our mechanistic analysis, which identifies two contrasting mechanisms: (i) broad
retrieval, where Non-thinking models broadly attend to multiple needles; (ii) targeted
retrieval, where Thinking models use enumeration in CoT traces to successively retrieve needles. Targeted
retrieval concentrates
attention on individual needles and is accompanied by more compact internal representations. Moreover, causal intervention analysis suggests that Thinking models use the CoT trace to maintain and update an internal counter as needles are successively retrieved, even without explicit numbering. In small controlled experiments, both
retrieval mechanisms and counter states emerge under standard autoregressive training. Together, our results connect long-context
retrieval with representation geometry of counting, supporting a state-tracking account of CoT
reasoning.
Read the full abstract ▾
All authors Liang Twist Shan, Tianyu Hu, Hao Yan, Yiqiao Zhong
Statistics Applications
Regional-scale nonlinear structural response simulation using building-specific Bouc-Wen models
Sebin Oh and Taeyong Kim
Regional seismic assessments often assign structural properties by building archetype, limiting their representation of building-specific responses. This study develops a modular physics-regularized framework for identifying equivalent single-degree-of-freedom Bouc–Wen models from seismic input–response histories. A differentiable forward solver enables joint estimation of hysteretic parameters and viscous damping through an objective combining response matching with independently weighted dynamic-
equilibrium and hysteretic state-
evolution residuals. Before regional application, synthetic systems and four-story steel-frame hybrid simulation data are used to evaluate identification and prediction under unseen inputs, showing improved parameter estimation relative to a genetic
algorithm and improved predictive performance with physics regularization, respectively. The framework is then applied to a numerical
benchmark of 14,280 buildings in Milpitas, California, using models identified from simulated responses to a smaller earthquake to predict responses under larger scenarios. With the inventory, ground-motion inputs, and damage-and-loss models held fixed, archetypal and response-informed portfolios yield distinct damage-fraction distributions and substantially different structural repair costs. For the earthquake scenario with moment magnitude $M_w=6.8$, the response-informed portfolio yields a mean structural repair cost nearly twice that of the archetypal portfolio, despite nearly identical mean damage fractions. These results highlight the influence of structural model assignment on regional damage and loss estimates and the importance of incorporating building-specific response information into regional assessment.
Read the full abstract ▾
Statistics Applications
Mental Health Support Hotlines as Capacity-Limited Services: Joint Modeling of Demand, Assessment, and Unassessed Calls
Siliang Zhang and Jing Ouyang
Mental health support hotlines log help-seeking call attempts in real time, yet limited service capacity allows only a small proportion of calls to be answered and assessed. The time of every attempt is recorded, whereas issue types and marks such as caller demographics and crisis severity are observed only for assessed calls. Caller anonymity also prevents linking repeated attempts to the same individual. Nevertheless, service planning and crisis monitoring require estimating unassessed demand, recovering temporal patterns in call content, and quantifying how much the partially observed marks improve these estimates. We develop a Joint Marked Dynamic Factor Model (JM-DFM) in which a shared low-dimensional latent state drives attempt intensity, issue composition, and mixed-type mark distributions. We establish identifiability, prove consistency of the proposed sieve
estimator, and show that incorporating marks improves the asymptotic estimation precision. The proposed framework is then applied to records of 68,371 call attempts to China's national 12356 mental health support hotline. We estimate that about 35 unassessed attempts per day involve suicidal ideation, representing a higher proportion than among assessed calls. Incorporating the marks narrows the latent-state uncertainty bands by 18% to 49%, most during periods with few issue-labeled calls.
Read the full abstract ▾