Machine Learning (Statistics) · Cross-listed from cs.LG
Fubini Study geometry of representation drift in high dimensional data
Arturo Tozzi
High dimensional representation drift is commonly quantified using Euclidean or cosine distances, which presuppose fixed coordinates when comparing representations across time, training or preprocessing stages. While effective in many settings, these measures entangle intrinsic changes in the data with variations induced by arbitrary parametrizations. We introduce a projective
geometric view of representation drift grounded in the Fubini Study metric, which identifies representations that differ only by gauge transformations such as global rescalings or sign flips. Applying this framework to empirical high dimensional
datasets, we explicitly construct representation trajectories and track their
evolution through cumulative
geometric drift. Comparing Euclidean, cosine and Fubini Study distances along these trajectories reveals that conventional metrics systematically overestimate change whenever representations carry genuine projective ambiguity. By contrast, the Fubini Study metric isolates intrinsic
evolution by remaining invariant under gauge-induced fluctuations. We further show that the difference between cosine and Fubini Study drift defines a computable, monotone quantity that directly captures representation churn attributable to gauge freedom. This separation provides a diagnostic for distinguishing meaningful structural
evolution from parametrization artifacts, without introducing model-specific assumptions. Overall, we establish a
geometric criterion for assessing representation stability in high-dimensional systems and clarify the limits of angular distances.
Embedding representation dynamics in projective space connects data analysis with established
geometric programs and yields observables that are directly testable in empirical workflows.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Predicting and improving test-time scaling laws via reward tail-guided search
Muheng Li, Jian Qian and Wenlong Mou
Test-time scaling has emerged as a critical avenue for enhancing the
reasoning capabilities of Large
Language Models (LLMs). Though the straight-forward ”best-of-$N$” (BoN) strategy has already demonstrated significant improvements in performance, it lacks principled guidance on the choice of $N$, budget allocation, and multi-stage decision-making, thereby leaving substantial room for optimization. While many works have explored such optimization, rigorous theoretical guarantees remain limited. In this work, we propose new methodologies to predict and improve scaling properties via tail-guided search. By estimating the tail distribution of rewards, our method predicts the scaling law of LLMs without the need for exhaustive evaluations. Leveraging this prediction tool, we introduce Scaling-Law Guided (SLG) Search, a new test-time
algorithm that dynamically allocates compute to identify and exploit intermediate states with the highest predicted potential. We theoretically prove that SLG achieves vanishing regret compared to perfect-information oracles, and achieves expected rewards that would otherwise require a polynomially larger compute budget required when using BoN. Empirically, we validate our framework across different LLMs and reward models, confirming that tail-guided allocation consistently achieves higher reward yields than Best-of-$N$ under identical compute budgets. Our code is available at https://github.com/PotatoJnny/Scaling-Law-Guided-search.
Read the full abstract ▾
Statistical Methodology · Cross-listed from cs.LG
Causal Preference Elicitation
Edwin V. Bonilla, He Zhao and Daniel M. Steinberg
We propose causal preference elicitation, a
Bayesian framework for expert-in-the-loop causal discovery that actively queries local edge relations to concentrate a posterior over directed acyclic graphs (DAGs). From any black-box observational posterior, we model noisy expert judgments with a three-way likelihood over edge existence and direction. Posterior inference uses a flexible particle approximation, and queries are selected by an efficient expected information gain criterion on the expert's categorical response. Experiments on synthetic graphs,
protein signaling data, and a human gene perturbation
benchmark show faster posterior concentration and improved recovery of directed effects under tight query budgets.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability
Eric Regis and Sinho Chewi
How can we understand gradient-based training over non-convex landscapes? The edge of stability phenomenon, introduced in Cohen et al. (2021), indicates that the answer is not so simple: namely, gradient descent (GD) with large step sizes often diverges away from the gradient flow. In this regime, the "Central Flow", recently proposed in Cohen et al. (2025), provides an accurate ODE approximation to the GD dynamics over many architectures. In this work, we propose Rod Flow, an alternative ODE approximation, which carries the following advantages: (1) it rests on a principled derivation stemming from a physical picture of GD iterates as an extended one-dimensional object – a "rod"; (2) it better captures GD dynamics for simple toy examples and matches the accuracy of Central Flow for representative neural network architectures, and (3) is explicit and cheap to compute. Theoretically, we prove that Rod Flow correctly predicts the critical sharpness threshold and explains self-stabilization in quartic potentials. We validate our theory with a range of numerical experiments.
Read the full abstract ▾
Machine Learning (Statistics)
Density-Informed Pseudo-Counts for Calibrated Evidential Deep Learning
Pietro Carlotti, Nevena Gligić and Arya Farahi
Evidential Deep Learning (EDL) is a popular framework for uncertainty-aware classification that models predictive uncertainty via Dirichlet distributions parameterized by neural networks. Despite its popularity, its theoretical foundations and behavior under distributional shift remain poorly understood. In this work, we provide a principled statistical interpretation by proving that EDL training corresponds to amortized variational inference in a hierarchical
Bayesian model with a tempered pseudo-likelihood. This perspective reveals a major drawback: standard EDL conflates epistemic and aleatoric uncertainty, leading to systematic overconfidence on out-of-distribution (OOD) inputs. To address this, we introduce Density-Informed Pseudo-count EDL (DIP-EDL), a new parametrization that decouples class prediction from the magnitude of uncertainty by separately estimating the conditional label distribution and the marginal
covariate density. This separation preserves evidence in high-density regions while shrinking predictions toward a uniform prior for OOD data. Theoretically, we prove that DIP-EDL achieves asymptotic concentration. Empirically, we show that our method enhances interpretability and improves robustness and uncertainty calibration under distributional shift.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
Viet Nguyen, Tuan Minh Pham, Thinh Cao et al. (+4 more)
Self-
attention has greatly contributed to the success of the widely used
Transformer architecture by enabling learning from data with long-range dependencies. In an effort to improve performance, a gated
attention model that leverages a gating mechanism within the multi-head self-
attention has recently been proposed as a promising alternative. Gated
attention has been empirically demonstrated to increase the expressiveness of low-rank mapping in standard
attention and even to eliminate the
attention sink phenomenon. Despite its efficacy, a clear theoretical understanding of gated
attention's benefits remains lacking in the literature. To close this gap, we rigorously show that each entry in a gated
attention matrix or a multi-head self-
attention matrix can be written as a hierarchical mixture of experts. By recasting learning as an expert estimation problem, we demonstrate that gated
attention is more sample-efficient than multi-head self-
attention. In particular, while the former needs only a
polynomial number of data points to estimate an expert, the latter requires exponentially many data points to achieve the same estimation error. Furthermore, our analysis also provides a theoretical justification for why gated
attention yields higher performance when a gate is placed at the output of the scaled dot product
attention or the value map rather than at other positions in the multi-head self-
attention architecture.
Read the full abstract ▾
All authors Viet Nguyen, Tuan Minh Pham, Thinh Cao, Tan Dinh, Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Machine Learning (Statistics)
Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
Tuan Minh Pham, Thinh Cao, Viet Nguyen et al. (+3 more)
The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to
language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a
polynomial order.
Read the full abstract ▾
All authors Tuan Minh Pham, Thinh Cao, Viet Nguyen, Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Machine Learning (Statistics) · Cross-listed from math.NA
Dimension-Free Multimodal Sampling via Preconditioned Annealed Langevin Dynamics
Lorenzo Baldassari, Josselin Garnier, Knut Solna et al. (+1 more)
Designing sampling
algorithms for
multimodal targets that remain stable under refinement of the finite-dimensional approximation of an underlying function-space problem is a central challenge. Annealed Langevin dynamics (ALD) is a natural alternative to classical Langevin in this context, since it is often observed to improve exploration across modes. Yet a gap remains between its empirical success and existing theory: under which conditions can ALD be guaranteed to remain stable across dimensions? In this paper, we bridge this gap by providing a uniform-in-dimension analysis of continuous-time ALD for Gaussian-mixture targets. Along an explicit annealing path obtained by gradually removing Gaussian smoothing from the target, we identify spectral conditions linking the smoothing covariance to the component covariances under which ALD achieves a prescribed accuracy in Kullback-Leibler divergence within a dimension-uniform time horizon. We then establish stability in a perturbative regime with imperfect initialization and approximate scores. Under a misspecified-mixture score model, we show that preconditioning ALD with an operator whose spectrum decays sufficiently fast prevents error terms from accumulating across coordinates and thereby preserves dimension-uniform control.
Read the full abstract ▾
All authors Lorenzo Baldassari, Josselin Garnier, Knut Solna, Maarten V. de Hoop
Machine Learning (Statistics) · Cross-listed from cs.LG
Theoretical Analysis of Measure Consistency Regularization for Partially Observed Data
Yinsong Wang and Shahin Shahrampour
The problem of corrupted data, missing features, or missing modalities continues to plague the modern machine learning landscape. To address this issue, a class of regularization methods that enforce consistency between imputed and fully observed data has emerged as a promising approach for improving model generalization, particularly in partially observed settings. We refer to this class of methods as Measure Consistency Regularization (MCR). Despite its empirical success in various applications, such as image inpainting, data imputation and semi-supervised learning, a fundamental understanding of the theoretical underpinnings of MCR remains limited. This paper bridges this gap by offering theoretical insights into when MCR yields a more favorable finite-sample estimation-error upper bound, viewed through the lens of neural network distance. Under ideal interpolation and compatibility conditions, we show that the MCR estimation-error upper bound is no larger than vanilla supervised training and becomes strictly smaller when the mixed-sample alternative is favorable. We then extend the analysis to the non-ideal regime, where optimization and compatibility residuals can potentially offset this finite-sample advantage. Guided by these insights, we propose a novel practical diagnostic that leverages the duality gap and a calibrated drift
estimator to infer the potential benefit of MCR training. We present detailed empirical evidence to support our theoretical claims and to show the effectiveness and accuracy of our practical diagnostic. We further provide simulations on real-world
datasets to show the versatility of MCR under different model architectures designed for different data sources.
Read the full abstract ▾
Statistics Theory · Cross-listed from cs.LG
Phase Transitions for Feature Learning in Neural Networks
Andrea Montanari and Zihao Wang
According to a popular viewpoint, neural networks learn from data by first identifying low-dimensional representations, and subsequently fitting the best model in this space. Recent works provide a formalization of this phenomenon when learning multi-index models. In this setting, we are given $n$ i.i.d. pairs $({\boldsymbol x}_i,y_i)$, where the
covariate vectors ${\boldsymbol x}_i\in\mathbb{R}^d$ are isotropic, and responses $y_i$ only depend on ${\boldsymbol x}_i$ through a $k$-dimensional projection ${\boldsymbol \Theta}_*^{{\sf T}}{\boldsymbol x}_i$. Feature learning amounts to learning the latent space spanned by ${\boldsymbol \Theta}_*$. In this context, we study the gradient descent dynamics of two-layer neural networks under the proportional asymptotics $n,d\to\infty$, $n/d\to\delta$, while the dimension of the latent space $k$ and the number of hidden
neurons $m$ are kept fixed. Earlier work establishes that feature learning via
polynomial-time
algorithms is possible if $\delta> \delta_{\text{alg}}$, for $\delta_{\text{alg}}$ a threshold depending on the data distribution, and is impossible (within a certain class of
algorithms) below $\delta_{\text{alg}}$. Here we derive an analogous threshold $\delta_{\text{NN}}$ for two-layer networks. Our characterization of $\delta_{\text{NN}}$ opens the way to study the dependence of learning dynamics on the network architecture and training
algorithm. The threshold $\delta_{\text{NN}}$ is determined by the following scenario. Training first visits points for which the gradient of the empirical risk is large and learns the directions spanned by these gradients. Then the gradient becomes smaller and the dynamics becomes dominated by negative directions of the Hessian. The threshold $\delta_{\text{NN}}$ corresponds to a phase transition in the spectrum of the Hessian in this second phase.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
DCD: Decomposition-based Causal Discovery from Autocorrelated and Non-Stationary Temporal Data
Muhammad Hasan Ferdous, Ian Stockwell and Md Osman Gani
Multivariate time series in domains such as finance,
climate science, and healthcare often exhibit long-term trends, seasonal patterns, and short-term fluctuations, complicating causal inference under non-stationarity and autocorrelation. Existing causal discovery methods typically operate on raw observations, making them vulnerable to spurious edges and misattributed temporal dependencies. We introduce a decomposition-based causal discovery framework that separates each time series into trend, seasonal, and residual components and performs component-specific causal analysis. Trend components are assessed using stationarity tests, seasonal components using kernel-based dependence measures, and residual components using constraint-based causal discovery. The resulting component-level graphs are integrated into a unified multi-scale causal structure. This approach isolates long- and short-range causal effects, reduces spurious associations, and improves interpretability. Across extensive synthetic
benchmarks and real-world
climate data, our framework more accurately recovers ground-truth causal structure than state-of-the-art baselines, particularly under strong non-stationarity and temporal autocorrelation.
Read the full abstract ▾
Machine Learning (Statistics)
Robust Generalization with Adaptive Optimal Transport Priors for Decision-Focused Learning
Haixiang Sun and Andrew L. Liu
Few-shot learning requires models to generalize under limited supervision while remaining robust to distribution shifts. Existing Sinkhorn Distributionally Robust Optimization (DRO) methods provide theoretical guarantees but rely on a fixed reference distribution, which limits their adaptability. We propose a Prototype-Guided Distributionally Robust Optimization (PG-DRO) framework that learns class-adaptive priors from abundant base data via hierarchical optimal transport and embeds them into the Sinkhorn DRO formulation. This design enables few-shot information to be organically integrated into producing class-specific robust decisions that are both theoretically grounded and efficient, and further aligns the uncertainty set with transferable structural knowledge. Experiments show that PG-DRO achieves stronger robust generalization in few-shot scenarios, outperforming both standard learners and DRO baselines.
Read the full abstract ▾
Machine Learning (Statistics)
Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference
Kamélia Daudel, Minh-Ngoc Tran and Cheng Zhang
Importance-weighted variational inference approximates intractable densities by optimizing multi-sample Monte Carlo objectives whose tightness increases with the number of importance samples $N$. When reparameterization is unavailable, optimization relies on score-function (REINFORCE) gradient
estimators. In this paper, we provide a comprehensive analysis of score-function gradient
estimators for importance-weighted objectives by characterizing their asymptotic expectation and variance. We establish that standard variational inference for Monte Carlo objectives (VIMCO) gradient
estimators suffer from a vanishing signal-to-noise ratio (SNR) as $N$ increases, which arises from a suboptimal control variate choice and can impair optimization. To address this issue, we derive a general mechanism for constructing VIMCO gradient
estimators that avoid SNR collapse. We identify the High-Effective Sample Size optimal variance importance sampling
estimator as belonging to this category of
estimators and establish its $\sqrt{N}$ SNR scaling. This
estimator does not, however, have zero variance at optimality and achieving its $\sqrt{N}$ SNR rate may require prohibitively large $N$ when the importance weights are degenerate. We overcome these limitations by introducing VIMCO gradient
estimators that attain zero variance at optimality and support importance weight tempering. Experiments with discrete latent-variable and state-space models validate the theoretical predictions and illustrate the benefits of our
estimators.
Read the full abstract ▾
Machine Learning (Statistics)
Online Social Welfare Function-based Resource Allocation
Kanad Pardeshi, Samsara Foubert and Aarti Singh
In many real-world settings, a centralized decision-maker must repeatedly allocate finite resources to a population over multiple time steps. Individuals who receive a resource derive some
stochastic utility; to characterize the population-level effects of an allocation, the expected individual utilities are then aggregated using a social welfare function (SWF). We formalize this setting and present a general confidence sequence framework for SWF-based online learning and inference, valid for any monotonic, concave, and Lipschitz-continuous SWF. Our key insight is that monotonicity alone suffices to lift confidence sequences from individual utilities to anytime-valid bounds on optimal welfare. Building on this foundation, we propose SWF-UCB, a SWF-agnostic online learning
algorithm that achieves near-optimal $\tilde{O}(n+\sqrt{nkT})$ regret (for $k$ resources distributed among $n$ individuals at each of $T$ time steps). We instantiate our framework on three normatively distinct SWF families: Weighted Power Mean, Kolm, and Gini, providing bespoke oracle
algorithms for each. Experiments confirm $\sqrt{T}$ scaling and reveal rich interactions between $k$ and SWF parameters. This framework naturally supports inference applications such as sequential hypothesis testing, optimal stopping, and policy evaluation.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.LG
An Odd Estimator for Shapley Values
Fabian Fumagalli, Landon Butler, Justin Singh Kang et al. (+2 more)
The Shapley value is a ubiquitous framework for attribution in machine learning, encompassing feature importance, data valuation, and causal inference. However, its exact computation is generally intractable, necessitating efficient approximation methods. While the most effective and popular
estimators leverage the paired sampling heuristic to reduce estimation error, the theoretical mechanism driving this improvement has remained opaque. In this work, we provide an elegant and fundamental justification for paired sampling: we prove that the Shapley value depends exclusively on the odd component of the set function, and that paired sampling orthogonalizes the regression objective to filter out the irrelevant even component. Leveraging this insight, we propose OddSHAP, a novel consistent
estimator that performs
polynomial regression solely on the odd subspace. By utilizing the
Fourier basis to isolate this subspace and employing a proxy model to identify high-impact interactions, OddSHAP overcomes the combinatorial explosion of higher-order approximations. Through an extensive
benchmark, we find that OddSHAP achieves state-of-the-art estimation accuracy at larger sampling budgets.
Read the full abstract ▾
All authors Fabian Fumagalli, Landon Butler, Justin Singh Kang, Kannan Ramchandran, R. Teal Witter
Machine Learning (Statistics) · Cross-listed from cs.CL
On the Power of (Approximate) Reward Models for Inference-Time Scaling
Youheng Zhu and Yiping Lu
Inference-time scaling has recently emerged as a powerful paradigm for improving the
reasoning capability of large
language models. Among various approaches, Sequential Monte Carlo (SMC) has become a particularly important framework, enabling iterative generation, evaluation, rejection, and resampling of intermediate
reasoning trajectories. A central component in this process is the reward model, which evaluates partial solutions and guides the allocation of computation during inference. However, in practice, true reward models are never available. All deployed systems rely on approximate reward models, raising a fundamental question: Why and when do approximate reward models suffice for effective inference-time scaling? In this work, we provide a theoretical answer. We identify the Bellman error of the approximate reward model as the key quantity governing the effectiveness of SMC-based inference-time scaling. For a
reasoning process of length $T$, we show that if the Bellman error of the approximate reward model is bounded by $O(1/T)$, then combining this reward model with SMC reduces the computational complexity of
reasoning from exponential in $T$ to
polynomial in $T$. This yields an exponential improvement in inference efficiency despite using only approximate rewards.
Read the full abstract ▾
Machine Learning (Statistics) · Cross-listed from cs.CL
Context Dependence and Reliability in Autoregressive Language Models
Poushali Sengupta, Shashi Raj Pandey, Sabita Maharjan et al. (+1 more)
Large
language models (LLMs) generate outputs by utilizing extensive context, which often includes redundant information from
prompts, retrieved passages, and interaction history. In critical applications, it is vital to identify which context elements actually influence the output, as standard explanation methods struggle with redundancy and overlapping context. Minor changes in input can lead to unpredictable shifts in attribution scores, undermining interpretability and raising concerns about risks like
prompt injection. This work addresses the challenge of distinguishing essential context elements from correlated ones. We introduce RISE (Redundancy-Insensitive Scoring of Explanation), a method that quantifies the unique influence of each input relative to others, minimizing the impact of redundancies and providing clearer, stable attributions. Experiments demonstrate that RISE offers more robust explanations than traditional methods, emphasizing the importance of conditional information for trustworthy LLM explanations and monitoring.
Read the full abstract ▾
All authors Poushali Sengupta, Shashi Raj Pandey, Sabita Maharjan, Frank Eliassen
Statistical Methodology
A Fractional M/M/1 Queue Governed by Stretched Non-Local Time Operators
Mehmet S{\i}dd{\i}k Çad{\i}rc{\i}
We introduce a non-Markovian generalization of the classical M/M/1 queue by incorporating extended nonlocal time dynamics into Kolmogorov forward equations. We obtain the model by replacing the standard time derivative with an extended Caputo-type operator. It preserves the birth-death transition structure of the standard queue while introducing
memory effects into the temporal
evolution. We derive explicit representations for transient state probabilities in terms of the Kilbas-Saigo function, which naturally emerges as the relaxation kernel associated with the stretched operator, using Laplace transform techniques. We construct a time-varying interpretation and show that the fractional queue can be viewed as a distribution of a classical M/M/1 process evaluated at a non-decreasing random time. It is observed that the fractional queue can be viewed as a distribution of a classical M/M/1 process evaluated at a non-decreasing random time. We prove that under the standard stability condition $\rho<1$, the steady-state distribution remains
geometric and coincides with the distribution of the classical queue, whilst we prove that the stretched fractional parameters significantly affect the convergence rate in the transient regime. Numerical examples based on Monte Carlo simulations highlight the effect of the parameters $(\alpha,\gamma)$ on the distribution of empty states, tail length distributions, and the average tail
evolution, and validate the flexibility of the proposed framework in capturing long-
memory tail dynamics.
Read the full abstract ▾
Statistics Applications · Cross-listed from cs.CR
Adaptive Quantum-Safe Cryptography for 6G Vehicular Networks via Context-Aware Optimization
Poushali Sengupta, Mayank Raikwar, Sabita Maharjan et al. (+2 more)
Powerful
quantum computers in the future may be able to break the security used for communication between vehicles and other devices (Vehicle-to-Everything, or V2X). New security methods called post-
quantum cryptography can help protect these systems, but they often require more computing power and can slow down communication, posing a challenge for fast 6G vehicle networks. In this paper, we propose an adaptive post-
quantum cryptography (PQC) framework that predicts short-term mobility and channel variations and dynamically selects suitable
lattice-, code-, or hash-based PQC configurations using a predictive multi-objective evolutionary
algorithm (APMOEA) to meet vehicular latency and security constraints.However, frequent cryptographic reconfiguration in dynamic vehicular environments introduces new attack surfaces during
algorithm transitions. A secure monotonic-upgrade protocol prevents downgrade, replay, and desynchronization attacks during transitions. Theoretical results show decision stability under bounded prediction error, latency boundedness under mobility drift, and correctness under small forecast noise. These results demonstrate a practical path toward
quantum-safe cryptography in future 6G vehicular networks. Through extensive experiments based on realistic mobility (LuST), weather (ERA5), and NR-V2X channel traces, we show that the proposed framework reduces end-to-end latency by up to 27%, lowers communication overhead by up to 65%, and effectively stabilizes cryptographic switching behavior using reinforcement learning. Moreover, under the evaluated
adversarial scenarios, the monotonic-upgrade protocol successfully prevents downgrade, replay, and desynchronization attacks.
Read the full abstract ▾
All authors Poushali Sengupta, Mayank Raikwar, Sabita Maharjan, Frank Eliassen, Yan Zhang
Statistics Theory · Cross-listed from cs.LG
High-accuracy sampling for diffusion models and log-concave distributions
Fan Chen, Sinho Chewi, Constantinos Daskalakis et al. (+1 more)
We present
algorithms for
diffusion model sampling which obtain $\delta$-error in $\mathrm{polylog}(1/\delta)$ steps, given access to $\widetilde O(\delta)$-accurate score estimates in $L^2$. This is an exponential improvement over all previous results. Specifically, under minimal data assumptions, the complexity is $\widetilde O(d_\star \mathrm{polylog}(1/\delta))$ where $d_\star$ is the intrinsic dimension of the data. Further, under a non-uniform $L$-Lipschitz condition, the complexity reduces to $\widetilde O(L \mathrm{polylog}(1/\delta))$. Our approach also yields the first $\mathrm{polylog}(1/\delta)$ complexity sampler for general log-concave distributions using only gradient evaluations.
Read the full abstract ▾
All authors Fan Chen, Sinho Chewi, Constantinos Daskalakis, Alexander Rakhlin
Machine Learning (Statistics) · Cross-listed from cs.LG
Multi-LLM Adaptive Conformal Inference for Reliable LLM Responses
Kangjun Noh, Seongchan Lee, Ilmun Kim et al. (+1 more)
Ensuring factuality is essential for the safe use of Large
Language Models (LLMs) in high-stakes domains such as medicine and law. Conformal inference provides distribution-free guarantees, but existing approaches are either overly conservative, discarding many true-claims, or rely on adaptive error rates and simple linear models that fail to capture complex group structures. To address these challenges, we reformulate conformal inference in a multiplicative filtering setting, modeling factuality as a product of claim-level scores. Our method, Multi-LLM Adaptive Conformal Inference (MACI), leverages ensembles to produce more accurate factuality-scores, which in our experiments led to higher retention, while validity is preserved through group-conditional calibration. Experiments show that MACI consistently achieves user-specified coverage with substantially higher retention and lower time cost than baselines. Our repository is available at https://github.com/MLAI-Yonsei/MACI
Read the full abstract ▾
All authors Kangjun Noh, Seongchan Lee, Ilmun Kim, Kyungwoo Song
Statistical Methodology
Explicit Expressions for Multidimensional Value-at-Risk under Archimedean Copulas
Dotamana Yéo, Saralees Nadarajah and Amadou Sawadogo
This paper studies multivariate Value-at-Risk (VaR) for financial portfolios with a focus on modeling dependence structures through Archimedean copulas. Using the generator representation of Archimedean copulas, we derive explicit analytical expressions for the marginal lower-tail multivariate VaR in arbitrary dimensions. Closed-form formulas are obtained for several commonly used copula families, including Clayton, Frank, Gumbel-Hougaard, Joe and Ali–Mikhail–Haq copulas, allowing a direct assessment of the impact of dependence on multivariate risk. These results complement existing approaches, which largely rely on numerical or simulation-based methods, by providing tractable alternatives for theoretical and applied risk analysis. Monte Carlo simulations are conducted to evaluate the finite-sample performance of the proposed VaR
estimator and to illustrate the role of different dependence structures. The proposed analytical setting offers transparent tools for multivariate risk measurement and systemic risk assessment.
Read the full abstract ▾
Statistical Methodology
Estimation of Tsallis entropy and its applications to goodness-of-fit tests
Siddhartha Chakraborty, Asok K. Nanda and Narayanaswamy Balakrishnan
In this paper, we consider the problem of estimating Tsallis
entropy from a given data set. We propose four different
estimators for Tsallis
entropy measure based on higher-order sample spacings, and then discuss estimation of Tsallis divergence measure. We compare the performance of the proposed
estimators by means of bias and mean squared error and also examine their robustness to outliers. Next, we propose a spacings-based
estimator for Tsallis
entropy under progressive type-II censoring and study its performance using Monte Carlo simulations. Another
estimator for Tsallis
entropy is proposed using
quantile function and its consistency and asymptotic normality are studied, and its performance is evaluated through Monte Carlo simulations. Goodness-of-fit tests for normal and exponential distributions as applications are developed using Tsallis divergence measure. The performance of the proposed tests are then compared with some known tests using simulations and it is shown that the proposed tests perform very well. Also, an exponentiality test under progressive type-II censoring is proposed, its performance is compared with an existing
entropy-based test using simulation. It is observed that the proposed test performs well. Finally, some real data sets are analysed for illustrative purposes.
Read the full abstract ▾
Statistics Theory
Estimating Conditional Distributions via Sklar's Theorem and Empirical Checkerboard Approximations, with Consequences to Nonparametric Regression
Kai Schärer and Wolfgang Trutschnig
We tackle the natural question of whether it is possible to estimate conditional distributions via Sklar's
theorem by separately estimating the conditional distributions of the underlying copula and the marginals. Working with so-called empirical checkerboard/Bernstein approximations with suitably chosen resolution/degree, we first show that uniform weak convergence to the true underlying copula can be established under very mild regularity assumptions. Building upon these results and plugging in the univariate empirical marginal distribution functions we then provide an affirmative answer to the afore-mentioned question and prove strong consistency of the resulting
estimators for the conditional distributions. Moreover, we show that aggregating our
estimators allows to construct consistent nonparametric
estimators for the mean, the
quantile, and the expectile regression function, and beyond. Some simulations illustrating the performance of the
estimators and a real data example complement the established theoretical results.
Read the full abstract ▾
Statistics Applications
Simultaneous Estimation of Seabed and Its Roughness With Longitudinal Waves
Babak Maboudi Afkham and Ana Carpio
This paper introduces an infinite-dimensional
Bayesian framework for acoustic seabed
tomography, leveraging wave scattering to simultaneously estimate the seabed and its roughness.
Tomography is considered an ill-posed problem where multiple seabed configurations can result in similar measurement patterns. We propose a novel approach focusing on the statistical isotropy of the seabed. Utilizing fractional differentiability to identify seabed roughness, the paper presents a robust numerical
algorithm to estimate the seabed and quantify uncertainties. Extensive numerical experiments validate the effectiveness of this method, offering a promising avenue for large-scale seabed exploration.
Read the full abstract ▾