Machine Learning (Statistics)
Automatic Statistical Test for Rationally Expressible Algorithms by Selective Inference, with Applications to Feature Selection
Teruyuki Katsuoka, Tomohiro Shiraishi, Shuichi Nishino et al. (+1 more)
Selective inference (SI) provides statistically valid $p$-values for hypotheses selected by applying an
algorithm to the data, correcting for the bias that arises when the same data are used both to select and to test a hypothesis. Developing an SI procedure for a new
algorithm, however, has required an expert to derive, and then implement, the selection event, i.e., the conditions under which the hypothesis is selected. Repeating this specialized effort for every new
algorithm is why exact SI has so far been available for only a narrow class. We propose AutoSI, a framework that removes this barrier in two ways. First, AutoSI constructs the selection event automatically from the
algorithm's individual operations, so the user only writes the
algorithm as ordinary NumPy-like code and derives nothing by hand. Second, AutoSI broadens the class of selection events SI can handle: existing exact methods are limited to selection events characterized by linear or quadratic inequalities in the data, whereas AutoSI covers any
algorithm expressible through rational functions of the data (ratios of
polynomials). We prove that the $p$-values computed by AutoSI are exactly valid in finite samples. We demonstrate AutoSI on three feature-selection methods, each written in a few dozen lines of code. One of these methods, the lasso with its tuning parameter selected by cross-validated $R^2$, cannot be handled within existing exact SI frameworks and is made possible by AutoSI. Experiments on synthetic and real
datasets show that the resulting $p$-values control the type I error rate (i.e., the false positive rate) at the nominal level while retaining high power.
Read the full abstract ▾
All authors Teruyuki Katsuoka, Tomohiro Shiraishi, Shuichi Nishino, Ichiro Takeuchi
Statistical Methodology · Cross-listed from econ.EM
Clustered Local Projections for Short and Ultra-Short Time Series – A Hierarchical Bayesian Framework
Todd Clark and Florian Huber
Estimating the dynamic effects of economic shocks in short and very short samples is impeded by a lack of degrees of freedom. We offer a solution based on a
Bayesian hierarchical framework for estimating local projection (LP) impulse response functions across a panel of related time series. The framework explicitly accommodates unbalanced panels in which some series are substantially shorter than others, allowing the short series to borrow information from longer ones at horizons where the short series carry little or no own data. Since series might exhibit heterogeneous dynamics, we develop a sparse finite mixture pool that clusters units by similarity of their impulse response profiles. We show in simulations that our approach substantially improves LP estimation accuracy relative to the standard approach if the time series are short while producing similar LPs for longer time series. Using a US price
dataset, augmented with survey responses, we find that supply-chain and oil shocks trigger heterogeneous reactions of different price measures, with headline price indices responding more sharply than their core counterparts and goods prices changing more than services prices.
Read the full abstract ▾
Statistics Applications · Cross-listed from econ.EM
Drivers of Success: A Bayesian State-Space Model to Disentangling Latent Driver and Constructor Abilities in Formula One
Tim Lindner, Rui Jorge Almeida, Nalan Baştürk et al. (+1 more)
Formula One outcomes reflect the joint contributions of drivers and constructors, but these contributions are unobserved and vary over time. We propose a
Bayesian state-space model that disentangles dynamic driver and constructor abilities using two observed outcomes: fastest qualifying lap times and race rankings. Both outcomes depend jointly on latent driver and constructor states that evolve at the Grand Prix level, while the race equation additionally accounts for starting-grid position. The decomposition is supported by constraints that center the driver and constructor abilities at zero, together with variation in driver-constructor assignments over time.
Bayesian inference is performed using the No-U-Turn sampler under weakly informative priors that treat driver and constructor abilities symmetrically. Applying the model to the Formula One hybrid era from 2014 to 2021, we find substantial heterogeneity in both driver and constructor abilities. Driver abilities are generally more stable over time, whereas constructor abilities exhibit greater variation and, for many driver–constructor combinations, contribute more strongly to observed performance.
Read the full abstract ▾
All authors Tim Lindner, Rui Jorge Almeida, Nalan Baştürk, Stephan Smeekes
Machine Learning (Statistics) · Cross-listed from cs.LG
An entropic explanation of insistence on sameness in autism
Przemysław Śliwiński
An information theory-based framework is proposed in attempt to explain insistence on sameness in autism as an instance of a general behavior pattern in which an individual tries to reduce surprise and uncertainty. It offers a new definition of autism as an impairment in which cognitive functions are restricted to discrimination, memorization and prediction of tangible properties of the environment. An analogy between insistence on sameness and constrained minimization of the
entropy metric is observed and examined for a set of assumptions that describe cognitive limitations of a person with autism. The metric is given by the formula $D_H(R, M) = H(R|M) + H(M|R)$, where $R$ represents sequences of random stimuli, $M$ is a
memory that stores and retrieves them, and where $H(.|.)$ denotes their conditional
entropies interpreted as surprise and uncertainty, respectively. It is first inferred that to minimize the metric an individual can learn about $R$ (and store that knowledge in $M$) or can restrict $R$ to the already known $M$. Then, it is concluded that insistence on sameness is a manifestation of the latter. Moreover, it is shown that the proposed framework: (1) Helps to quantify the concepts of surprise, uncertainty, sensory overload and deprivation, anxiety, comfort zone, disappointment, disorientation, pedantry, rigidness, observance or aberrant precision. (2) Leads to a list of guidelines for learning therapies and daily care routines, and allows them to be defined as optimization
algorithms and implemented as programs for robotic live-in caregivers. (3) Can be validated with the help of a Turing test-like approach that requires no experiments involving individuals with autism. The framework-if positively validated-will provide formal foundations and design guidelines for therapies aimed at improving self-reliance of individuals with autism in basic activities of daily living.
Read the full abstract ▾
Statistics Theory
Optimal minimization of an unknown function in a nonparametric multivariate regression model thanks to a dimension reduction approach
Cédric Adam, Ilaria Giulini and Céline Lévy-Leduc
In this paper, we propose a novel approach for estimating the minimum of a smooth function and its location from observations corresponding to a multivariate regression function depending a priori on d variables but actually only on r < d active variables and corrupted by some additional noise. Our method consists of two steps: The rst one is a variable selection approach which is used for identifying the r active variables on which f depends and the second one consists in estimating the minimum of the function and its location. The estimation of the minimizers is obtained by using a projected gradient descent where the gradient is estimated using a local
polynomial approximation of the regression function limited to its active variables obtained in the rst step. The estimation of the minimum is obtained by evaluating the
estimator of the regression function using a local
polynomial approach at the
estimator of one of the minimizers previously obtained. We establish non asymptotic upper bounds for the quadratic risk of the
estimators of the minimizers and of the minimum and prove that they reach the optimal rate that could be expected as if the active variables were known beforehand up to a factor smaller than a power of a logarithmic term.
Read the full abstract ▾
Statistical Methodology
Semiparametric robust mixture of experts based on nonparametric maximum likelihood
Sangkon Oh, Victor H. Lachos and Byungtae Seo
The mixture of experts (MoE) model provides a flexible approach for modeling heterogeneous regression relationships by allowing
covariate-dependent mixing through a gating network, but most existing MoE models rely on parametric assumptions for expert error distributions, typically Gaussian, which can lead to inefficiency and sensitivity to outliers or heavy-tailed behavior when misspecified. We propose a semiparametric MoE model in which each expert error distribution is represented as a nonparametric Gaussian scale mixture estimated via nonparametric maximum likelihood, relaxing parametric assumptions within the Gaussian scale-mixture class while preserving the interpretability and structure of the MoE framework. The resulting model adapts to complex error structures, improves robustness under contamination and heavy tails, and remains competitive under well-specified Gaussian settings, providing a practical and theoretically grounded alternative to parametric MoE formulations.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Discretization and Statistical Consistency of Functional Flow Matching
Lennon J. Shikhman
Functional flow matching is posed on distributions of functions but implemented from finitely many coefficients or point values. Under scattered or adaptive refinement, the resulting conditioning sigma-
algebras need not be nested, so martingale convergence does not justify the sensor limit. We prove strong $L^2$ convergence of finite conditional velocity targets for every strongly consistent sequence of finite-rank reconstructions, with quantitative bounds for orthogonal projections and a point-sensor extension through a regularity space. For learned flows, coupling directly to a population superposition path yields an end-to-end Wasserstein bound without assuming uniqueness of the population finite-dimensional ODE. We verify sensor-independent constants for a normalized quadrature neural operator, including globally Lipschitz activations through an explicit magnitude recurrence. A noncommuting trace-class Gaussian example gives boundary multiplier $0$ under projected restriction and $0.72$ under exact conditioning. A spatial regularity–cubature certificate closes the operator-realization term, a Bernstein argument gives a $\widetilde{O}(n^{-1})$ excess-risk term for fixed model dimension and envelopes, and an exactly realizable clipped Gaussian scaling specialization yields an explicit end-to-end rate.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.IR
The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
Yuanyuan Shen, Yiren Yan, Wenjie Li et al. (+1 more)
On two-sided content platforms, symmetric two-sided isolation (assigning matched fractions of creators and viewers to isolated treatment and control submarkets) is widely used for creator-side and cold-start experiments because it removes cross-arm marketplace interference. Isolation, however, thins each viewer's candidate catalog, and intuition suggests the resulting engagement cost should fade as the platform grows: a small fraction of a vast catalog is still vast. We show that, in an order-statistics model of engagement, whether this intuition holds depends on the upper tail of match quality. Extreme-value theory yields tail-class loss laws with a sharp dichotomy: for light or bounded tails the loss vanishes as the candidate pool grows, whereas under heavy tails it converges to a size-independent constant, so expanding the candidate pool, even by orders of magnitude, does not asymptotically eliminate the cost. Evidence from two production experiments on a platform with millions of active creators is consistent with this picture: a pure A/A traffic sweep reveals a measurable, depth-graded engagement cost; a one-sided catalog ablation independently shows that per-viewer thinning contributes to the loss; and a tail index calibrated on the small exploration pool predicts an effect consistent with the one observed in the far larger full-catalog ablation. Isolation thus carries a price that experimenters should budget for, like any other cost. We give practitioners a preflight procedure that estimates it before launch, sizes traffic accordingly, and recommends a fallback design when the predicted cost exceeds a chosen tolerance.
Read the full abstract ▾
All authors Yuanyuan Shen, Yiren Yan, Wenjie Li, Chunhui Zhu
Machine Learning (Statistics) · Cross-listed from math.NA
An adaptive split-combine Gaussian mixture filter for nonlinear and multimodal state estimation
San Kim, Won Chang, Daniel B. Forger et al. (+1 more)
Filtering combines model predictions with measurements to estimate the probability density function (PDF) of a system state over time. The PDF often becomes highly asymmetric and even
multimodal in nonlinear systems with oscillatory or chaotic dynamics. Such non-Gaussian features violate the single-Gaussian assumption underlying Kalman-type filters. To address this problem, Gaussian mixture filtering has been proposed. However, accurately propagating mixture components and adaptively adjusting their number and weights over time remain open challenges. Here, we develop an adaptive split-combine Gaussian mixture filter (AMF) that estimates the time
evolution of asymmetric and
multimodal PDFs by adaptively splitting and combining Gaussian particles without auxiliary online numerical optimization. Notably, the proposed splitting method guarantees a reduction in variance along a target level-set-point direction of a Gaussian particle. This enables accurate and efficient propagation of particles. We show that AMF consistently outperforms various baseline filters across diverse
benchmarks, including single and coupled slow-fast Van der Pol oscillators and the Lorenz attractor. We also propose a parallel implementation of AMF, allowing high-fidelity PDF estimation with practical computational cost.
Read the full abstract ▾
All authors San Kim, Won Chang, Daniel B. Forger, Dae Wook Kim
Statistical Methodology
Best for which estimand? A known-truth benchmark of longitudinal-matching and target-trial-emulation methods for time-varying treatments
M. Ehsan Karim
On a non-collapsible survival mechanism, longitudinal-matching and target-trial-emulation methods are not competing
estimators of one truth but answers to different causal questions, so a
benchmark that scores them against a single "true hazard ratio" fabricates bias. We provide the direct comparison of relative efficiency, variance estimation, and model sensitivity that reviews find lacking. On a deliberately non-collapsible continuous-time Cox mechanism with known truth, the dominant families (sequential Cox, sequential stratification, risk-set matching, and inverse-probability-of-treatment-weighted (IPTW) marginal structural models) target numerically distinct causal estimands (marginal, conditional, two average-treatment-effect-on-the-treated, and intention-to-treat versus per-protocol). First, we quantify the phantom bias a shared marginal truth fabricates: 0.32-0.33 log-cumulative-hazard-ratio units for the matching
estimators and 0.15 for the conditional method; the associational naive time-dependent Cox sits 0.76 away, a total discrepancy compounding the estimand gap with confounding. Second, a rank reversal: the recommended method flips with the target estimand, and a low-variance off-target
estimator can still win on mean-squared error. Third, a cross-family variance result: the cluster-robust sandwich is closer to nominal for the trial-stacking
estimator (0.90) but under-covers the matching
estimators (0.77-0.82), which a prespecified n=500 bootstrap sub-study brings to 0.95-0.96. Fourth, model sensitivity: omitting a confounder induces 0.45-0.50 log-hazard-ratio bias and undercoverage, and intention-to-treat and per-protocol effects diverge as switching increases; a heart-transplant analysis illustrates these. On a second mechanism three of four findings replicate, the rank reversal attenuating and model sensitivity proving calibration-dependent.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from math.OC
Incremental Aggregation on the Grassmannian for Asynchronous Eigenspace Computation
Xiaolu Wang, Jiang Hu and Hoi-To Wai
We study asynchronous optimization for finite-sum eigenspace computation in heterogeneous distributed systems. The theoretical foundations for asynchronous eigenspace computation remain scarce, with existing approaches offering limited coverage of dynamics directly on the Grassmannian under stale information. In this paper, we propose a Grassmannian incremental aggregation method that refreshes only arriving components and reuses cached gradients, retaining low per-update cost without global synchronization. The method employs an extrinsic polar update that preserves the intrinsic subspace geometry without requiring parallel transport of stale tangent vectors. Our analysis establishes a tight angle-dependent gradient-dominance characterization of the objective and a basin-invariance property for stale aggregated updates. These yield two-phase linear convergence, comprising an explicit broad-basin regime and a sharper local regime, with constants controlled by component spectral spreads. Experiments on serial and distributed PCA demonstrate improved sample efficiency and wall-clock convergence over representative baselines.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics
Han Bao
Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization. Implicit bias emerging from optimization, though not being encoded by the learning objective, often prevents from overfitting to spurious patterns. A typical instance is the max-margin implicit bias of a linear classifier, widely established for exponentially tailed loss functions. Even after having a given
dataset separated, the parameter vector continues to evolve towards the max-margin direction asymptotically along the gradient descent dynamics. This phenomenon corroborates a frequent empirical observation of "train longer, generalize better." However, the max-margin convergence is an asymptotic phenomenon, and what is worse, this asymptotic convergence rate is significantly slower than pure convex optimization. Even so, the parameter vector along gradient descent dynamics commonly correlates with the max-margin direction positively (though not exactly) within considerably fewer iterations than the asymptotic rate. By shedding another light on this classical problem, this work aims to understand the mechanism of this early-stage
alignment phenomenon. Our theoretical results demonstrate that the parameter vector weakly aligns with the max-margin direction within $O(\exp(\exp(-\delta)))$ iterations, where $\delta>0$ is the permissible
alignment error, which is shown to be tight. By tracking the radial and tangential flows, our proof operates on the
alignment dynamics directly with
dataset geometry and gets rid of the asymptotic expansion, which is a key insight to establishing faster weak
alignment.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.CV
iStructTab: Structured Feature Sequencing for Multimodal Learning of Image and Tabular Data
Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Gyawali et al. (+2 more)
Multimodal learning of images and tabular data is often impaired by ineffective representations, resulting in redundancy, dispersion, and generalization problems. To tackle this challenge, we introduce Graph-Enhanced Descriptor Sequencing (GEDS), a structured feature sequencing
algorithm grounded in principles from the Column Permutation Problem (CPP). GEDS refines statistical descriptors of the features through similarity graph-based computations, systematically determining an effective feature sequencing. We incorporate GEDS within an order-aware efficient
transformer framework, utilizing order-aware
memory tokens that explicitly adhere to the derived feature sequencing via a dedicated loss function. Experimental results across
multimodal benchmarks demonstrate that iStructTab effectively minimizes feature dispersion, improving predictive performance and robustness, and highlighting the significance of structured feature sequencing in
multimodal learning.
Read the full abstract ▾
All authors Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Gyawali, Gianfranco Doretto, Donald A. Adjeroh
Machine Learning (Statistics) · Cross-listed from cs.CL
Equitable System-Prompt Selection via Constrained Mixed-Strategy GroupDRO
Mengyu Xu, Qiaoxin Yang, Zhihan Liu et al. (+4 more)
Large
language models are increasingly used for information seeking, yet semantically equivalent questions phrased in different ways can receive answers of considerably different quality. System
prompts are widely employed to steer response behavior, but they are typically optimized for average-case quality, so some question phrasings may still receive incomplete or low-quality answers. To address this, we formulate a constrained mixed-strategy GroupDRO framework for system-
prompt selection. Instead of optimizing the system-
prompt text, the framework assigns weights to system
prompts in an existing pool to minimize the worst-case information-quality loss across evaluation metrics and groups, while constraining the mean loss to stay close to that of average-based selection. Because pool generation and selection are decoupled, the method applies to any system-
prompt pool and can leverage an ensemble of complementary system
prompts rather than a single one. Across five LLMs on two bilingual medical and consumer-finance
benchmarks, the constrained method reduces the Overall Mean, Worst 25% Mean, and Worst by 13.1%, 13.2%, and 13.7% on average relative to no mitigation while keeping overall quality close to Average selection. Its multi-
prompt weights reveal complementarity across metric-group pairs. Code and data are available at https://github.com/Rainxu09/equitable-system-
prompt-selection.
Read the full abstract ▾
All authors Mengyu Xu, Qiaoxin Yang, Zhihan Liu, Ruiyao Xu, Zachary Liu, Kezhen Chen, Chongyang Gao
Statistical Methodology
Bivariate Prior Specification for Bayesian Decision Making in Early Phase Clinical Trials
Chengyuan Yang and Evan Kwiatkowski
Bayesian Go/No-Go decisions with co-primary endpoints require specifying prior distributions under the Normal-Inverse-Wishart framework; however guidance on how prior hyperparameters influence trial decisions remains limited. We propose a calibrated prior specification framework for bivariate Go/No-Go decisions. Skeptical and enthusiastic priors are calibrated so that each assigns a target probability to a clinically relevant decision region. We prove that for any prior precision $\kappa > 0$, a unique scale parameter $\lambda_0$ achieves the target calibration. Operating characteristics are evaluated across different $\kappa $ via simulation and applied to a phase~3 telitacicept lupus trial.The simulation result indicates $\kappa$ is the primary driver of prior discrimination. At $\kappa = 1$, the go rate difference between priors was 0.07; at $\kappa = 10$ it reached 0.56, with false positive rates below 0.01. Operating characteristics were robust to the degrees of freedom parameter $\nu_0$ and prior correlation $\rho_0$, supporting a default of $\nu_0 = 2$. In the lupus application, prior sensitivity was negligible at $\kappa = 1$ but at $\kappa = 10$ the enthusiastic go rate was three times the skeptical rate at small sample sizes. The framework reduces prior specification to two choices: the prior center and the prior precision $\kappa$. The identification of $\kappa$ as the dominant parameter, together with the cautious choice of $\kappa$ before the trial, motivates adaptive approaches to prior precision.
Read the full abstract ▾
Statistics Theory
Fixed-Point Characterisations of Extremal Distributions under Partial Distributional Constraints
Kizito Salako and Rabiu Tsoho Muhammad
We present a methodological framework for solving robust inference problems with partially specified distributions over measurable subsets of a parameter space. Partial specifications define a set of admissible distributions. The goal is to determine extremal values (over these admissible distributions) for statistical quantities, where these quantities—these objective functions—are ratios of expectations of analytic functions, continuous functions, piecewise continuous functions, and uniform limits of piecewise continuous functions. We show that extremal values are approached by sequences of admissible distributions, whose limiting extremal distributions are characterised by fixed-point conditions on their support locations. This characterises where extremal distributions place probability mass and yields a practical computational framework for solving the corresponding optimisation problems. We establish convergence and asymptotic properties of the resulting extremal distributions and extremal objective function values. This work extends robust inference methods (e.g. robust
Bayesian inference) by combining extremal-distribution reduction, fixed-point characterisation, and approximation-based analysis within a unified framework.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from math.OC
Achieving First-Order Statistical Improvements in Data-Driven Optimization: From No-Free-Lunch to Amplified Decision Perturbation
Henry Lam and Tianyu Wang
Recent proliferation of data-optimization integration has led to a range of methods that aim to improve the statistical performance of data-driven optimization decisions. However, while many of these methods are motivated intuitively from a robustness or regularization perspective, their resulting statistical benefits are often unclear and, even if available, are established on a case-by-case basis. We provide a systematic dissection of data-driven optimization formulations using the view of "directionally perturbed" empirical optimization (EO). Specifically, this umbrella of formulations, which we call "EO+", covers many existing data-driven optimization methods, including regularization, distributionally robust optimization, transfer learning, and analogous methods for contextual optimization. On the one hand, we argue that without additional, correctly specified, side information, any EO+ method can result in at most second-order improvements. This provides a negative conclusion, namely “no free lunch is possible", on the statistical power of EO+. On the other hand, we show that when leveraging side information that is geometrically effective, achieving first-order improvements is possible by choosing hyperparameters that are significantly larger than what is typically suggested in the literature. Moreover, we construct a principled methodology based on excess risk estimation, via either system knowledge or bootstrap resampling, to maximize the first-order gain. We demonstrate how this gain connects to the control-variate principle, a variance reduction technique in the Monte Carlo simulation literature, which helps explain why geometrically effective side information is necessary.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
ArborEnum: Decision Tree Rashomon Sets over Continuous Features
Zakk Heile, Hayden McTavish, Margo Seltzer et al. (+1 more)
The Rashomon effect describes the phenomenon that many models can achieve nearly equivalent performance on the same learning task, with significant ramifications for robustness, feature importance, and customizability. These use cases motivate the computation of Rashomon sets: the set of all models whose regularized loss is near-optimal. Decision trees are one of the few model classes for which Rashomon sets can be fully enumerated, but this computation has always been conditional on a binarization of the original data, either restricting which splits each tree is allowed to make or substantially increasing the complexity of an already difficult combinatorial problem. We introduce the first
algorithm that exactly enumerates decision-tree Rashomon sets while exploiting the ordered structure of continuous features. We further develop a relaxation for approximate enumeration and an anytime
algorithm that progressively refines the set of candidate thresholds, producing increasingly detailed approximations that converge to the continuous-feature Rashomon set. Experiments show that coarse binarization can miss many trees, important features, and predictive multiplicity; our
algorithms achieve orders-of-magnitude speedups over existing enumeration methods, with approximations providing further speedups while maintaining near-perfect recall.
Read the full abstract ▾
All authors Zakk Heile, Hayden McTavish, Margo Seltzer, Cynthia Rudin