Papers

Differentially private (DP) pipelines often combine heterogeneous mechanisms, yet their guarantees are best analyzed using different notions of privacy (e.g., bounded range, zCDP, ADP). When composing them, we have to analyze the guarantees through a reduced common representation that loses accounting precision. This issue is particularly relevant in DP synthetic data generation, where data preprocessing, generation algorithms, and DP-SGD could appear within the same pipeline. We develop novel tools for tight end-to-end accounting of such compositions using privacy loss distributions (PLD), $f$-DP, and Gaussian DP (GDP). In particular, we derive PLD and GDP analyses of bounded-range mechanisms such as the Exponential mechanism. Building on these theoretical results, we introduce AIM-GDP, a GDP-native variant of AIM (a state-of-the-art DP synthetic data mechanism) that enables easy composability with complex pipelines via PLD, attack-aware risk calibration, and tight end-to-end auditing while consistently achieving better utility; and MixCal, an algorithm for calibrating adaptive mixed compositions using PLD accounting. Overall, our tools enable tighter end-to-end accounting and thus less noise at the same privacy level in DP synthetic data pipelines.

Large language models can answer the same question in different styles, but users' preferred styles are hidden and vary across users and questions. We study inference-time personalization through online selection from a fixed set of prompts, each representing a persona. Users arrive sequentially, each asks a short sequence of questions, and the system displays two responses per question and observes one pairwise choice. We formulate this interaction as a variant of the contextual dueling bandit problem in which the learner starts without preference data, adapts within each session, and exploits information shared across users. The objective is to minimize cumulative strong regret, summing the utility gaps of both displayed responses relative to the user's best response among the available options. We make two contributions. First, we introduce the problem and release a reproducible preference benchmark and dataset built around a tutoring case study, containing educational questions, LLM-generated answers in six teaching styles, and pairwise judgments from LLM-simulated students with hidden preference profiles. We fit Bradley-Terry models to these judgments to obtain reference utilities for regret evaluation. Second, we propose a baseline that learns a mixture prior over users from completed sessions and adapts to each new user within the session. On students and topics never used in development, it removes 44-48% of random pairs' excess regret and achieves over half the performance of a per-user linear oracle fitted in hindsight. We release code, prompts, and the benchmark.

The Kullback–Leibler (KL) divergence is the expectation value of the log-likelihood ratio between two specified models, evaluated with respect to one of the two models. It provides an average measure of the distinguishability of the models. We examine its connection to discovery potential and the accuracy of parameter estimates when one model represents a null hypothesis, and the alternative is a family of models characterized by one or more continuous parameters. For data drawn from a multivariate Gaussian distribution, we derive the log-likelihood-ratio distribution and its cumulants under two such hypotheses. We then search over the alternative model's parameter space and use peak prominence, the height of a likelihood-profile peak above its surrounding baseline, as an operational detection statistic. Repeating this search on simulated data drawn from the null model allows us to calibrate the prominence threshold for a specified false-positive rate. We study a toy Gaussian model as well as three cosmological examples involving cosmic topology, the tensor-to-scalar ratio, and dark energy. In all these cases, detectability of a particular model depends not only on KL divergence, but also on the size of the model parameter space, the likelihood-ratio distributions, and the distance of the model from any parameter space boundaries. The relationship between KL divergence, discovery potential, and parameter estimation must therefore be calibrated for each family of models and detection statistic under consideration.

We develop a novel generalization of McDiarmid's concentration inequality and use it to obtain a uniform law of large numbers result and error bounds on stochastic optimization problems which simultaneously incorporate two key features: 1) They capture the benefit of training sample reuse in algorithms that also employ easily-sampled auxiliary random variables. 2) They allow for one-component mean-difference bounding functions which are both unbounded in the component of interest and have non-uniform dependence on another component. We demonstrate the utility of our method by deriving concentration inequalities for score-based generative modeling. This application naturally leverages both key features of our new tools: 1) It utilizes auxiliary Gaussian random variables in conjunction with the training samples; accounting for training sample reuse results in tighter error bounds by effectively integrating out the dependence on the auxiliary random variables. 2) The ability to handle non-uniform mean-difference bounding functions allows us to treat unbounded score models and data distributions with unbounded support.

Variational inference is a flexible tool for modeling experimental data and has proven useful in crystallography. Pathwise gradient estimators provide low-variance gradients for reparameterized variational inference, but are not readily available for some distributions central to crystallography. This work derives and implements implicit and explicit reparameterized gradient estimators for the folded Normal and Rice distributions in \prog{PyTorch}, and compares their variance analytically and empirically. In each case the implicit estimator achieves lower variance for the location derivative, particularly at small location-to-scale ratios, while the scale-derivative variances are nearly equal. For both distributions, implicit estimator is proved to be the conditional expectation of the explicit estimator given the sampled magnitude, so its variance is never larger. As a practical application, French-Wilson scaling is formulated as a variational inference problem, using the folded Normal and Rice distributions as variational families for centric and acentric reflections, respectively. Both estimators recover the exact posterior to better than $1\%$ on the amplitude, and converge at the same rate. The implementations are available in the open-source package \prog{rs-distributions}.

Bayesian predictive regions account for parameter uncertainty but can be poorly calibrated under model misspecification, even with expensive full MCMC. Cheaper approximations such as variational inference reduce the cost but often yield less accurate predictive distributions. We propose SCOPE (Score-guided Conformal Posterior Exploration), which retains the predictive quality of full MCMC while achieving calibrated coverage at a fraction of its cost. The key observation is that a prediction region can depend on the posterior mainly through a few parameter directions that move the region's boundary, and these are generally not the directions of largest posterior variance. SCOPE gives nonconformity scores a new role in guiding posterior computation: score gradients on a held-out guide set, weighted toward the target boundary, identify the relevant directions. MCMC explores this subspace, while an inexpensive projected variational approximation carries the remaining uncertainty. The same guide set calibrates the region's final density level. We establish asymptotic coverage for SCOPE and provide a split variant with finite-sample validity. Controlled simulations show that score-guided exploration yields more accurate calibrated regions than non-targeted direction choices at low active rank. Across simulated and real benchmarks, SCOPE has a median test-NLL gap of only 0.002 nats relative to full Hamiltonian Monte Carlo (HMC), and on the real benchmarks it runs 12-15 times faster.

Recent frameworks in differential privacy (DP), such as privacy loss distributions and $f$-DP, provide lossless representations of the privacy leakage of mechanisms. These approaches are often considered to yield tighter $(\varepsilon, \delta)$-DP guarantees than those obtained from relaxed notions such as Rényi DP. But is that always the case? We study this question for differentially private stochastic gradient descent (DP-SGD) with fixed-size batches under replace adjacency. Our key observation is that in this setting, privacy accounting based on Rényi DP can use a pair of worst-case distributions (i.e., a dominating pair) that differs from the one used in the lossless frameworks. In particular, we characterize the worst-case distributions as two-dimensional Gaussian mixtures and derive an improved Rényi DP bound for DP-SGD. Numerical and asymptotic comparisons show that across various hyperparameter settings, a Rényi DP accountant based on the new bound can yield better privacy guarantees than those obtained from the lossless frameworks.

Local partner markets differ not only in population size but in relative partner availability, the scale of opportunity, marriage timing, and population turnover. This paper develops a county-level framework for comparing these conditions across local partner markets. The empirical application focuses on unmarried women ages 25-34 and uses observed partner-age pairings to construct age-specific weights for the relevant unmarried male population. We model absolute opportunity separately, fitting a Michaelis-Menten diminishing-returns transformation to 192 scenarios that vary dating intensity, search horizon, fit, and first-date conversion. The resulting index also incorporates the share of women ages 35-39 who have never married and the inflow rate of adults ages 25-34. Among 320 counties with complete data, the District of Columbia ranks first and New York County second under the baseline specification, and rankings remain highly stable under alternative component weights. An endogenous mobility extension shows that favorable markets attract additional male participants, producing substantial adjustment while leaving the top composite rankings relatively stable over the parameter values considered. The framework is intended as a structural comparison of local partner-market conditions rather than a predictor of individual relationship outcomes.

Understanding the predictability of life outcomes is important for public policy and science, but existing studies focus on a limited number of outcomes and therefore struggle to describe the broad patterns of predictability. We developed a high-throughput approach that uses a single dataset to study the predictability of many future outcomes using past data. We deployed this approach with hundreds of outcomes and thousands of machine learning models per outcome in the Future of Families and Child Wellbeing Study. Three findings emerged. First, most outcomes were not very predictable, but some were much more predictable than others. Second, even for the most predictable outcomes, our complex prediction procedure typically exhibited little skill relative to a simple procedure. Third, we found modest patterns in how predictability varied across outcomes. We conclude by discussing scope conditions and how future research might address them.

We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its direction from Bernoulli-labeled context examples. We derive an asymptotic prediction for the test logit in a high-dimensional regime with context length proportional to token dimension, and total training prompts proportional to dimension squared. Our formula cleanly separates calibration, test-time context noise, and non-isotropic parameter-estimation noise as interpretable effects on in-context performance. Our analysis also identifies the ridgeless separability regime and addresses the effects of context length, pretraining-set size, regularisation, and sampling temperature on in-context classification performance. The theory's predictions closely agree with finite-dimensional simulations and fully trained linear-attention models, while experiments with standard softmax attention show qualitatively related behaviour beyond the regime assumed by our analysis.

Shapley-based measures are widely used for measuring variable importance in statistics and machine learning. However, these measures have a serious issue–a nonactive variable can get a positive importance measure when it is correlated with an active variable. Herin et al. recently proposed a measure called proportional marginal effects (PME) that mitigates this spurious effect. However, PME is developed for deterministic models and has high computational cost. In this article, we propose a new measure called proportional inclusive marginal effects (PRIME) that overcomes these two limitations of PME. The key idea in PRIME is to use a novel model-free variable selection technique to identify the active variables and include only them in the computation of the importance measure. Analysis of test functions and real datasets show that PRIME can accurately and reliably quantify variable importance from noisy input-output data.

For the counts $(X_1,\dots,X_n)\sim\mathrm{Mult}(N;1/n,\dots,1/n)$ of $N$ observations in $n$ ordered bins we study the variance of the discrete total variation $d=\sum_{i=1}^{n-1}|X_{i+1}-X_i|$. An exact decomposition splits $\mathrm{Var}(d)$ into per-edge, adjacent-pair and disjoint-pair terms and yields, for fixed $n$ as $N\to\infty$, $\mathrm{Var}(d)/N=\alpha(n)-\beta(n)/N+\gamma(n)/N^2+o(N^{-2})$. For every $n\ge 2$, $N\alpha(n)$ is the classical normal-theory variance (Hoel, Kamat) of the sum of absolute successive differences of $n$ independent $N(0,N/n)$ variables; the correction $\beta(n)$ is obtained in closed form for $n\ge 3$ (the case $n=2$ is treated separately). Two disjoint edge differences $D_1=X_2-X_1$ and $D_3=X_4-X_3$ are exactly uncorrelated and asymptotically independent, yet $\mathrm{Cov}(|D_1|,|D_3|)=-1/(\pi n)+O(1/N)$: a correlation of relative order $1/N$, invisible in the Gaussian limit and absent under Poissonized sampling, whose constant is proportional to the exact joint fourth cumulant $\kappa_{2,2}(D_1,D_3)=-4N/n^2$ and is proved here on the lattice. The adjacent pair contributes the lattice-Edgeworth constant $a_*=-2\sqrt{3}/(9\pi)$, of which one quarter is a smooth Edgeworth term and three quarters a lattice-corner term; the disjoint skeleton $h(m)$ is closed through a Bessel generating function; and $\gamma(n)$ ($n\ge 5$) is expressed through a single triangular-lattice constant whose closed form we conjecture. Together with the mean of Part 1, the variance supplies the two moments used by moment-matched approximations to the null law of $d$. This is Part 2 of a series; the cyclic statistic $d^{c}$ will be treated in Part 5.

We study whether an unnormalized graph Laplace operator can determine both the geometry of a manifold and its sampling density. On a fixed compact smooth manifold without boundary, we use a known positive, continuous, strictly decreasing radial kernel at a fixed positive bandwidth. For positive sampling densities, the operator built from geodesic distances determines the Riemannian metric and identifies the density up to equality almost everywhere. When the metric is induced by a Euclidean embedding, the operator built from ambient distances also determines the embedding up to rotations, reflections and translations, and the density. On compact parameter classes with positive continuous densities, minimizing the distance between empirical and popula- tion operators gives estimates that converge to the true parameters under independent, identically distributed sampling. This recovery result assumes that the empirical operator can be evaluated at arbitrary points of the manifold.

We present CLASSO, a path-following method for the LASSO subject to general linear equality and inequality constraints. The local direction is obtained from a quadratic programme on an augmented active set, which determines both the coefficients that leave zero and the weakly active inequalities that remain binding. For a positive-definite Hessian and a nonempty feasible set, this programme has a unique primal minimiser, even when the binding constraint rows are dependent, and gives the local derivative of the solution path. The construction recovers the LASSO-modified LARS direction without constraints, and over the full cone gives the segment leaving the origin. We prove that the primal path is piecewise affine with finitely many kinks, without a constraint qualification or unique multipliers. A finite path does not imply a finite traversal: re-certifying at each kink can fail to terminate. CLASSO instead follows the minimum-norm certificate along a lexicographic rate; we prove that this rule reaches $\lambda=0$ in exact arithmetic within an explicit bound and traverses the whole path. These results are machine-checked in Lean~4, and the floating-point implementation agrees with an exact rational one on all $210$ small test problems. We assess the implementation on a $39$-case synthetic suite and six constrained versions of the diabetes benchmark. On the $31$ nontrivial synthetic cases with $n\ge p$, sampled relative objective differences from numerical reference solves are below $10^{-9}$ in magnitude; unnormalised endpoint differences reach $1.5\times10^{-7}$ in magnitude. For $n<p$, matched differences range from $-3.6\times10^{-7}$ to $2\times10^{-9}$ across five ridge strengths. Across the synthetic and diabetes benchmarks, checks of all $580$ recorded segments found no missed interior event and all checked KKT residuals within the stated tolerances.

Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.

This article develops optimal order (up to logarithmic factors) weighted cubature rules for Monte Carlo numerical integration in Sobolev reproducing kernel Hilbert spaces over common matrix manifolds. Specifically, we consider matrix models of the Stiefel, Grassmann, flag, and Lagrangian Grassmann manifolds each over either real numbers $\mathbb{R}$, complex numbers $\mathbb{C}$, or quaternions $\mathbb{H}$. The proposed cubature rules use evaluation points which are independent and identically distributed uniform on the manifold. The weights are chosen to be optimal with respect to the reproducing kernel Hilbert space worst-case error, under the constraint that the weights sum to one. Crucially, this constraint ensures optimal weights may be found without having to evaluate the kernel mean embedding (the integral of the kernel), which is often difficult to compute. We consider computable Matérn kernels of Frobenius distances between matrices which reproduce the Sobolev reproducing kernel Hilbert spaces with equivalent norms. Open-source numerical experiments validate our theoretical development, including an application to simultaneous approximation of intrinsic volumes in the Cauchy–Kubota formula. We also present preliminary empirical results beyond independent points including transformations of low-discrepancy quasi-Monte Carlo points from the unit cube and jointly optimized points and weights.

This paper develops a mathematical framework connecting structural metadata, logical admissibility, and correction in statistical data editing. A normalized routing graph generates a canonical indexed representation and an associated Horn theory, incorporating explicit semantic edits through deterministically evaluated auxiliary variables. Under specified record semantics, the theory characterizes admissibility, distinguishes structural minimality from logical irredundancy, and permits validation without materializing the deductive closure. Canonical indices preserve the provenance and dependency scopes of violations. With complete diagnostics and strictly positive weights, a nonnegative violation Hamiltonian vanishes exactly on admissible corrections within the permitted candidate domain. Verified zero-energy outputs therefore certify feasibility and global minimization of violation energy. Minimum modification cost is formulated separately, with sufficient conditions for an equivalent penalized objective and optimality certificates obtained by matching attained costs with valid lower bounds. A household survey application verifies admissible corrections and certifies minimum cost for a subset of returned records through a retrospective audit. The framework connects structural requirements to correction certificates independently of the numerical search procedure, while statistical accuracy of replacement values requires separate assessment.

We develop a comprehensive theoretical analysis of Matrix Shampoo for tracking time-varying optima in nonstationary stochastic optimization. We derive finite-horizon high-probability bounds for average tracking error that disentangle initialization, optimizer drift, momentum-induced drift penalty, stochastic noise, and adaptive preconditioning, with explicit dependence on momentum parameter \(\beta_1\) and stabilization parameter \(\varepsilon\). These bounds reveal an explicit drift–noise and stability–geometry tradeoffs: noise-dominated regimes favor larger $\beta_1$, which provides stronger first-moment averaging and greater suppression of gradient noise, whereas drift-dominated regimes favor smaller $\beta_1$, allowing faster adaptation to moving optimizer. Larger $\varepsilon$ reduces the preconditioner-perturbation term while weakening Shampoo's ability to exploit matrix geometry, thereby reducing adaptivity. We establish a matching minimax lower bound under squared path-variation constraint, proving the information-theoretic optimality of Shampoo over the class of causal optimization procedures. Under a constant stepsize, we establish high-probability bounds for the average projected stationarity gap along general nonstationary nonconvex objectives. We validate our theoretical findings on both synthetic and multiple real-world problems.

We study a stochastic optimization problem in which the random vector follows a semiparametric elliptical distribution with unknown location and shape parameters and an unspecified radial distribution. Assuming that the decision maker only has access to samples of the random vector, we develop a framework to jointly address parameter uncertainty and distributional ambiguity arising from this composite structure. To update the posterior distribution of the parameters, a key challenge is that the density of an elliptical distribution depends on the unknown radial distribution, and hence conventional likelihood-based Bayesian inference is infeasible. To overcome this challenge, we propose a novel estimating-equation-based Gibbs inference framework that updates posterior distributions of model parameters solely through estimating equations. For elliptical distributions, we construct a pair of estimating equations for the location and shape parameters that are invariant to the radial component, which enables posterior learning without specifying the density. For the distributional uncertainty, we construct an ambiguity set for the radial distribution by transforming observed samples into pseudo-samples of the radial variable. Combining the above components, we formulate a composite Bayesian risk–distributionally robust optimization model, in which the inner problem captures worst-case radial distributions within the ambiguity set, and the outer problem evaluates the decision risk induced by the parameter uncertainty through a user-specified risk measure. We further develop a cutting-plane algorithm to solve the resulting optimization problem. Numerical experiments demonstrate that the proposed approach provides substantial performance improvements over benchmark methods, including sample average approximation and distributionally robust optimization.

Diffusion and flow-matching models may struggle to accurately represent distributional tails, especially for heavy-tailed data. Existing approaches often use heavy-tailed noise or modify the generative dynamics, requiring dedicated retraining and more difficult estimation. We take a simpler approach: rather than relearning the full distribution, we correct only the model's behavior in the extreme regime. Our method builds on the radial-angular decomposition of multivariate heavy tails, where radius and direction become asymptotically independent. This allows us to preserve the directional structure learned by the pretrained model and correct only its radial tail. We estimate the tail onset from reference and generated samples, then rescale generated extremes so that their radial survival matches the target distribution. The resulting method is a lightweight post-training correction applied to a frozen diffusion or flow-matching model, requiring neither retraining nor changes to its sampling dynamics. Experiments on challenging synthetic heavy-tailed distributions and real weather and wildfire datasets show that this simple correction substantially improves extreme-tail coverage using only a small number of fitting samples.

Standardizing a rare feature can help stochastic gradient descent learn its effect within a finite budget, but it also changes the noise cost of learning. We characterize the lowest rare-subgroup risks attainable by standardized and unscaled training when each chooses a nonnegative constant learning rate and a deterministic stopping time under the same observation cap. For two independent binary features, we derive both optimized risks as prevalence vanishes while the expected number of rare observations remains finite. Uniform control over rates and stopping times reduces each limiting oracle to a scalar minimization. With positive noise, the preferred scaling changes exactly once as we move signal to the rare feature while the limiting risk at initialization stays fixed and positive. The formulas also explain how noise and budget affect the comparison. Separate finite-prevalence bounds favor standardization on a rare-dominant signal family and unscaled training on a common-dominant family. The results concern final iterates with population preprocessing and fresh observations. Exact-moment searches assess the approximation at finite prevalence. In a controlled study, both orderings persist under observation reuse, estimated preprocessing, and intercept fitting when tuning choices are fixed before evaluation. Selecting a predictor after observing training can reverse an ordering, even when selection uses exact population risk.

In simulation-based inference (SBI), neural posterior estimators trained on a bounded range of sample sizes can become inaccurate on larger datasets because their parameterizations do not specify how posterior location, uncertainty and shape must change as observations accumulate. We propose center-estimated NPE (CE-NPE), which combines a statistical center with an explicit contraction rate and learns a local center correction and residual densities in posterior-scale coordinates. The center is a statistical estimate, such as a closed-form statistic or a simulator-based moment match, which does not require an evaluable likelihood. Our analysis explains how contraction amplifies center error and decomposes Gaussian posterior error into shape, mean and covariance contributions. On six regular benchmarks, CE-NPE gives the best reference-posterior agreement of all tested amortized estimators at every out-of-range size, up to five times the largest training size, without retraining; it also fits better within the training range on most of them. Matched controls and nonregular examples characterize the scope of these gains and identify the conditions on which they depend: an appropriate contraction rate, an accurate center, and a stable residual representation.

Calibration has been extensively studied for classifiers, but remains largely unexplored for crowdsourcing aggregation methods, which are increasingly used for making probabilistic decisions. To the best of our knowledge, we provide the first theoretical and empirical study of calibration in label aggregation. To our great surprise, we find that widely used aggregation methods can be severely miscalibrated, and in opposite directions: Majority Vote is underconfident while model-based aggregators are overconfident. We prove that the Bayesian aggregator is perfectly calibrated under its data-generating model. We also show that exact calibration of weighted majority votes requires highly restrictive conditions, which helps explain why Majority Vote is generally miscalibrated in realistic crowds. Empirically, we study calibration across $39$ human crowdsourcing datasets and LLM-based crowds. In both cases, we observe systematic miscalibration for aggregation models such as Majority Vote, Dawid–Skene, GLAD, and CrowdFM. Finally, we propose a post-hoc calibration procedure that aggregates the aggregators: for any aggregation function $\phi$, we introduce a transformation that first combines its predictive distribution with those of complementary aggregators, before applying temperature scaling. This simple and interpretable transformation makes temperature scaling more effective than when applied directly to $\phi$. It improves calibration, especially in the high-redundancy regime where there are many annotations per item, while generally preserving or even improving accuracy.

Tabular foundation models (TFMs) achieve state-of-the-art performance on IID benchmarks, but how do they respond to repeated measurements at the same inputs? Across six TFMs, synthetic and semi-synthetic experiments show that additional measurements can improve predictions at measured inputs while degrading them elsewhere. Harm is most frequent at low noise, and irrelevant features amplify it in several models; real repeated-measurement datasets show similar patterns. Statistically, the IID rule targeted by PFN pretraining treats each repeated input as fresh evidence about the input distribution. When the prior links this distribution to the response function, the resulting protocol mismatch can outweigh the benefit of additional measurements, even under exact inference. Computationally, interventions identify row-count dependence in TabPFN-v3's attention scaling as a contributor to harm. Resetting the count largely reverses distortion in a noiseless linear diagnostic and attenuates amplification by irrelevant features. Finally, we evaluate ways to retain the benefits of repeated measurements while limiting their harm. Averaging measurements usually outperforms supplying them separately at larger repetition counts, whereas count reset can suppress both harm and useful gains. Code, saved results and reproduction instructions are available at https://github.com/drizmiz/PFN-Repeated-Measurement-Supp

Chain-of-Thought (CoT) enables transformers to solve complex reasoning tasks by generating intermediate reasoning steps. Despite strong empirical success, the mechanisms by which such reasoning abilities emerge during training remain poorly understood, particularly from a theoretical perspective. In this work, we study this question in a stylized yet fully analyzable setting: in-context $\mathrm{K}$-nearest-neighbor ($\mathrm{K}$-NN) prediction with a one-layer transformer. Earlier work has demonstrated that one-layer transformers can be trained to perform $1$-NN without CoT (Li et al., 2024a) by having the softmax attention attend to the nearest neighbor in context. However, this result on $1$-NN cannot be extended to $\mathrm{K}$-NN for $K>1$, as softmax attention is much less naturally suited to attending to the 2nd through $K$-th nearest neighbors. We first give a rigorous negative result showing that this limitation already appears on a well-separated class of binary classification tasks for odd $K>1$. We then give an explicit construction of a one-layer transformer, and show that it can solve $\mathrm{K}$-NN via CoT reasoning. We further prove that, under a stylized training setup, gradient descent can recover this construction, thereby showing that transformers can acquire the capability to solve $\mathrm{K}$-NN through training. In addition, we show that the resulting trained model applies to a broader test class than that assumed in the training analysis. Our results shed light on how CoT expands the ability of transformers to learn and solve tasks that are otherwise hard to solve.

We study the information-theoretic limits of inference in rectangular rank-one spiked matrix models with additive rotationally invariant noise, thereby going beyond the standard assumption of independent noise entries. Using the replica method from statistical physics, we derive a variational formula for the mutual information between the signal and the data, together with a system of saddle point equations predicting the minimum mean squared error in estimating the rank-one spike. We further show that, in the Bayes-optimal setting, these saddle point equations coincide with the fixed point equations of the state evolution of a recently proposed orthogonal approximate message passing algorithm with Bayes-optimal denoisers. This equivalence provides an algorithmic interpretation of the replica prediction and supports the Bayes-optimality of orthogonal approximate message passing whenever the relevant fixed point is unique. Numerical evaluations of the saddle point equations for several rotationally invariant noise ensembles and signal priors, together with finite-size simulations of the algorithm, confirm the asymptotic predictions. They illustrate both regimes with a unique informative fixed point, where orthogonal approximate message passing is predicted to attain the minimum mean squared error, and regimes with multiple fixed points, where a statistical-to-computational gap exists.

Selecting weights and projection directions from the observations used for testing can invalidate a reference distribution. We develop a framework for exact conditional inference under rotation symmetry. A maximal invariant permits adaptation while leaving a uniform common phase for calibration. This yields explicit Cauchy references for invariantly selected weights and projections. Under separate block rotations, the same principle gives exact normal-score combinations with coefficients selected from all blockwise invariants. The calibration results require neither densities nor moments. To illustrate the framework, we study direction learning and power in spherical Gaussian location models, identifying a regime in which consistent detection is possible without consistent direction recovery. For heterogeneous functional data, a matched mean–residual construction produces the required null symmetry under a conditional Gaussian model. Simulations compare Radius and Blend, two representative procedures, with sample splitting, and an electrocardiogram analysis illustrates their application.

Monte Carlo sampling algorithms aim to produce samples from a target distribution with a given unnormalised probability density function. The high cost of running Monte Carlo algorithms, with many simulation steps or particles needed to obtain a good empirical approximation to the target, can be reduced by introducing learnable components into the sampler – a form of amortisation. Once these components are trained, amortised samplers can produce high-quality samples with less computation at sampling time than training-free algorithms would require. We propose an amortised sampler built upon the framework of adaptive importance sampling – a hierarchical importance sampling procedure where the proposal is iteratively adapted. Our amortised adaptive importance sampling algorithm fits a lightweight neural network that adapts the proposal using information obtained over the course of sampling. By training our sampler over many different targets taken from the same family of distributions, we obtain a sampler that can adapt to a specific target at sampling time without a priori access to the target's parameters. We show that our procedure can efficiently sample from targets within the family the sampler was trained on, giving improvements over target-specific amortised samplers and training-free adaptive importance sampling algorithms. We also explore the potential for generalisation, where the trained sampler is used to sample from targets outwith its training family.

Current performance criteria used in Applied Behavior Analysis (ABA) are often based on whether a learner meets a predetermined benchmark e.g., 8 out of 10 correct responses across several sessions. Although these criteria help practitioners make instructional decisions, they provide limited information about how likely it is that the learner has truly acquired the skill. In this paper, we introduce a Bayesian approach that combines practitioners' prior expectations with the learner's ongoing performance data to estimate the learner's future performance level. Rather than providing only a pass or fail decision based on the criteria, we add an element of precision around that criteria to make decisions more nuanced to the testing conditions used. We also developed an R Shiny application for Measurement of Individualized, Evidence-Based Learning (MIEBL) that allows practitioners to apply this method in practice, supporting more informed instructional decisions and providing insight into the strengths and limitations of commonly used mastery criteria.

Multilevel regression with post-stratification (MrP) is the dominant method for estimating public opinion in subnational units. The method requires the full joint distribution of demographic variables in a subnational unit, which is often unavailable from public census tabulations. Existing approaches fill this gap by imposing assumptions to combine marginal distributions of census variables into a synthetic joint distribution. We propose sharp MrP bounds instead: the smallest interval containing all estimates of subnational public opinion consistent with the marginal distributions alone. The bounds solve a pair of linear programs and are computationally feasible for typical applications in political science. Applied to two prominent examples in the literature, the bounds are tight enough to be informative and often more accurate than MrP restricted to a coarser, observed joint distribution of census variables. The bounds serve as margin-consistent interval estimates and a diagnostic when researchers use a synthetic joint distribution to construct MrP estimates.