Machine Learning
Fubini Study geometry of representation drift in high dimensional data
Arturo Tozzi
High dimensional representation drift is commonly quantified using Euclidean or cosine distances, which presuppose fixed coordinates when comparing representations across time, training or preprocessing stages. While effective in many settings, these measures entangle intrinsic changes in the data with variations induced by arbitrary parametrizations. We introduce a projective
geometric view of representation drift grounded in the Fubini Study metric, which identifies representations that differ only by gauge transformations such as global rescalings or sign flips. Applying this framework to empirical high dimensional
datasets, we explicitly construct representation trajectories and track their
evolution through cumulative
geometric drift. Comparing Euclidean, cosine and Fubini Study distances along these trajectories reveals that conventional metrics systematically overestimate change whenever representations carry genuine projective ambiguity. By contrast, the Fubini Study metric isolates intrinsic
evolution by remaining invariant under gauge-induced fluctuations. We further show that the difference between cosine and Fubini Study drift defines a computable, monotone quantity that directly captures representation churn attributable to gauge freedom. This separation provides a diagnostic for distinguishing meaningful structural
evolution from parametrization artifacts, without introducing model-specific assumptions. Overall, we establish a
geometric criterion for assessing representation stability in high-dimensional systems and clarify the limits of angular distances.
Embedding representation dynamics in projective space connects data analysis with established
geometric programs and yields observables that are directly testable in empirical workflows.
Read the full abstract ▾
Machine Learning
Effective Frontiers: A Unification of Neural Scaling Laws
Jiaxuan Zou, Zixuan Gong, Ye Su et al. (+2 more)
Neural scaling laws govern the prediction power-law improvement of test loss with respect to model capacity ($N$), datasize ($D$), and compute ($C$). However, existing theoretical explanations often rely on specific architectures or complex kernel methods, lacking intuitive universality. In this paper, we propose a unified framework that abstracts general learning tasks as the progressive coverage of patterns from a long-tail (Zipfian) distribution. We introduce the Effective Frontier ($k_\star$), a threshold in the pattern rank space that separates learned knowledge from the unlearned tail. We prove that reducible loss is asymptotically determined by the probability mass of the tail a resource-dependent frontier truncation. Based on our framework, we derive the precise scaling laws for $N$, $D$, and $C$, attributing them to capacity, coverage, and optimization bottlenecks, respectively. Furthermore, we unify these mechanisms via a Max-Bottleneck principle, demonstrating that the Kaplan and Chinchilla scaling laws are not contradictory, but
equilibrium solutions to the same constrained optimization problem under different active bottlenecks.
Read the full abstract ▾
All authors Jiaxuan Zou, Zixuan Gong, Ye Su, Huayi Tang, Yong Liu
Machine Learning
Learnable Koopman-Enhanced Transformer-Based Time Series Forecasting with Spectral Control
Ali Forootani and Raffaele Iervolino
This paper proposes a unified family of learnable Koopman operator parameterizations that integrate linear dynamical systems theory with modern deep learning
forecasting architectures. We introduce four learnable Koopman variants-scalar-gated, per-mode gated, MLP-shaped spectral mapping, and low-rank Koopman operators which generalize and interpolate between strictly stable Koopman operators and unconstrained linear latent dynamics. Our formulation enables explicit control over the spectrum, stability, and rank of the linear transition operator while retaining compatibility with expressive nonlinear backbones such as Patchtst, Autoformer, and Informer. We evaluate the proposed operators in a large-scale
benchmark that also includes LSTM, DLinear, and simple diagonal State-Space Models (SSMs), as well as lightweight
transformer variants. Experiments across multiple horizons and patch lengths show that learnable Koopman models provide a favorable bias-variance trade-off, improved conditioning, and more interpretable latent dynamics. We provide a full spectral analysis, including
eigenvalue trajectories, stability envelopes, and learned spectral distributions. Our results demonstrate that learnable Koopman operators are effective, stable, and theoretically principled components for deep
forecasting.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
Yanki Margalit, Nurit Cohen-Inger, Erni Avram et al. (+3 more)
Large
language model (LLM) evaluation commonly relies on fixed human-authored
benchmarks or centralized human/model judges. Fixed
benchmarks can age and enter model training corpora, while centralized judges introduce their own scoring preferences. We cast LLM evaluation as a multi-
agent peer-assessment problem and introduce
Agents-as-Peers, a distributed evaluation protocol in which every
agent generates tasks, answers the shared task set, and judges the answers of its peers, with no privileged evaluator. Self-ratings are excluded from the ranking, and judging is repeated under controlled identity and presentation regimes so that self bias, identity effects, presentation-position sensitivity, and judge severity are measured rather than left implicit. In an instantiation with 12 LLMs and 420
agent-generated questions,
Agents-as-Peers aggregates 55,440 non-self peer judgments per regime. Most
agents assign their own answers higher scores than those answers receive from peer judges; visible identity changes scores for some
agents; the first fixed answer slot is associated with scores 0.39 points higher than the shuffled baseline; and judges differ markedly in severity. On labeled
benchmarks, peer scores align strongly with accuracy on TruthfulQA (Pearson r=0.904) and GSM8K (r=0.873). On TruthfulQA, the peer-score correlation is substantially higher than the self-score correlation (r=0.538). A
benchmark-overlap screen and item-level human audit further characterize the endogenous task set. These results support
Agents-as-Peers as a cohort-relative measurement protocol whose ranking is reported together with the biases and rater effects that shape it. Code and
dataset: https://github.com/caura-ai/caura-PeerRank
Read the full abstract ▾
All authors Yanki Margalit, Nurit Cohen-Inger, Erni Avram, Ran Taig, Seffi Cohen, Oded Margalit
Machine Learning
Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems
Xuesong Wang, Michael Groom, Rafael Oliveira et al. (+3 more)
Recent years have seen a surge in data-driven surrogates for dynamical systems that can be orders of magnitude faster than numerical solvers. However, many machine learning-based models such as neural operators exhibit spectral bias, attenuating high-frequency components that often encode small-scale structure. This limitation is particularly damaging in applications such as weather
forecasting, where misrepresented high frequencies can induce long-horizon instability. To address this issue, we propose multi-scale wavelet
transformers (MSWTs), which learn system dynamics in a tokenized wavelet domain. The wavelet transform explicitly separates low- and high-frequency content across scales. MSWTs leverage a wavelet-preserving downsampling scheme that retains high-frequency features and employ wavelet-based
attention to capture dependencies across scales and frequency bands. Experiments on chaotic dynamical systems show substantial error reductions and improved long horizon spectral fidelity. On the ERA5
climate reanalysis, MSWTs further reduce climatological bias, demonstrating their effectiveness in a real-world
forecasting setting.
Read the full abstract ▾
All authors Xuesong Wang, Michael Groom, Rafael Oliveira, He Zhao, Terence O'Kane, Edwin V. Bonilla
Machine Learning
Predicting and improving test-time scaling laws via reward tail-guided search
Muheng Li, Jian Qian and Wenlong Mou
Test-time scaling has emerged as a critical avenue for enhancing the
reasoning capabilities of Large
Language Models (LLMs). Though the straight-forward ”best-of-$N$” (BoN) strategy has already demonstrated significant improvements in performance, it lacks principled guidance on the choice of $N$, budget allocation, and multi-stage decision-making, thereby leaving substantial room for optimization. While many works have explored such optimization, rigorous theoretical guarantees remain limited. In this work, we propose new methodologies to predict and improve scaling properties via tail-guided search. By estimating the tail distribution of rewards, our method predicts the scaling law of LLMs without the need for exhaustive evaluations. Leveraging this prediction tool, we introduce Scaling-Law Guided (SLG) Search, a new test-time
algorithm that dynamically allocates compute to identify and exploit intermediate states with the highest predicted potential. We theoretically prove that SLG achieves vanishing regret compared to perfect-information oracles, and achieves expected rewards that would otherwise require a polynomially larger compute budget required when using BoN. Empirically, we validate our framework across different LLMs and reward models, confirming that tail-guided allocation consistently achieves higher reward yields than Best-of-$N$ under identical compute budgets. Our code is available at https://github.com/PotatoJnny/Scaling-Law-Guided-search.
Read the full abstract ▾
Machine Learning
Causal Preference Elicitation
Edwin V. Bonilla, He Zhao and Daniel M. Steinberg
We propose causal preference elicitation, a
Bayesian framework for expert-in-the-loop causal discovery that actively queries local edge relations to concentrate a posterior over directed acyclic graphs (DAGs). From any black-box observational posterior, we model noisy expert judgments with a three-way likelihood over edge existence and direction. Posterior inference uses a flexible particle approximation, and queries are selected by an efficient expected information gain criterion on the expert's categorical response. Experiments on synthetic graphs,
protein signaling data, and a human gene perturbation
benchmark show faster posterior concentration and improved recovery of directed effects under tight query budgets.
Read the full abstract ▾
Machine Learning
Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability
Eric Regis and Sinho Chewi
How can we understand gradient-based training over non-convex landscapes? The edge of stability phenomenon, introduced in Cohen et al. (2021), indicates that the answer is not so simple: namely, gradient descent (GD) with large step sizes often diverges away from the gradient flow. In this regime, the "Central Flow", recently proposed in Cohen et al. (2025), provides an accurate ODE approximation to the GD dynamics over many architectures. In this work, we propose Rod Flow, an alternative ODE approximation, which carries the following advantages: (1) it rests on a principled derivation stemming from a physical picture of GD iterates as an extended one-dimensional object – a "rod"; (2) it better captures GD dynamics for simple toy examples and matches the accuracy of Central Flow for representative neural network architectures, and (3) is explicit and cheap to compute. Theoretically, we prove that Rod Flow correctly predicts the critical sharpness threshold and explains self-stabilization in quartic potentials. We validate our theory with a range of numerical experiments.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Density-Informed Pseudo-Counts for Calibrated Evidential Deep Learning
Pietro Carlotti, Nevena Gligić and Arya Farahi
Evidential Deep Learning (EDL) is a popular framework for uncertainty-aware classification that models predictive uncertainty via Dirichlet distributions parameterized by neural networks. Despite its popularity, its theoretical foundations and behavior under distributional shift remain poorly understood. In this work, we provide a principled statistical interpretation by proving that EDL training corresponds to amortized variational inference in a hierarchical
Bayesian model with a tempered pseudo-likelihood. This perspective reveals a major drawback: standard EDL conflates epistemic and aleatoric uncertainty, leading to systematic overconfidence on out-of-distribution (OOD) inputs. To address this, we introduce Density-Informed Pseudo-count EDL (DIP-EDL), a new parametrization that decouples class prediction from the magnitude of uncertainty by separately estimating the conditional label distribution and the marginal
covariate density. This separation preserves evidence in high-density regions while shrinking predictions toward a uniform prior for OOD data. Theoretically, we prove that DIP-EDL achieves asymptotic concentration. Empirically, we show that our method enhances interpretability and improves robustness and uncertainty calibration under distributional shift.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Learning to Guide Local Search for MPE Inference in Probabilistic Graphical Models
Brij Malhotra, Shivvrat Arya, Tahrima Rahman et al. (+1 more)
Most Probable Explanation (MPE) inference in Probabilistic Graphical Models (PGMs) is a fundamental yet computationally challenging problem arising in domains such as diagnosis, planning, and structured prediction. In many practical settings, the graphical model remains fixed while inference must be performed repeatedly for varying evidence patterns.
Stochastic Local Search (SLS)
algorithms scale to large models but rely on myopic best-improvement rule that prioritizes immediate likelihood gains and often stagnate in poor local optima. Heuristics such as Guided Local Search (GLS+) partially alleviate this limitation by modifying the search landscape, but their guidance cannot be reused effectively across multiple inference queries on the same model. We propose a neural amortization framework for improving local search in this repeated-query regime. Exploiting the fixed graph structure, we train an
attention-based network to score local moves by predicting their ability to reduce Hamming distance to a near-optimal solution. Our approach integrates seamlessly with existing local search procedures, using this signal to balance short-term likelihood gains with long-term promise during neighbor selection. We provide theoretical intuition linking distance-reducing move selection to improved convergence behavior, and empirically demonstrate consistent improvements over SLS and GLS+ on challenging high-treewidth
benchmarks in the amortized inference setting.
Read the full abstract ▾
All authors Brij Malhotra, Shivvrat Arya, Tahrima Rahman, Vibhav Giridhar Gogate
Machine Learning
P-EAGLE: Parallel-Drafting EAGLE with Scalable Training
Mude Hui, Xin Huang, Jaime Campos Salas et al. (+5 more)
Reasoning LLMs produce longer outputs, requiring speculative decoding drafters trained on extended sequences. Parallel drafting - predicting multiple
tokens per forward pass - offers latency benefits over sequential generation, but training complexity scales quadratically with the product of sequence length and parallel positions, rendering long-context training impractical. We present P(arallel)-EAGLE, which transforms EAGLE from autoregressive to parallel multi-
token prediction via a learnable shared hidden state. To scale training to long contexts, we develop a framework featuring
attention mask pre-computation and sequence partitioning techniques, enabling gradient accumulation within individual sequences for parallel-prediction training. We implement P-EAGLE in vLLM and demonstrate speedups of 1.10-1.36x over autoregressive EAGLE-3 across GPT-OSS 120B, 20B, and Qwen3-Coder 30B.
Read the full abstract ▾
All authors Mude Hui, Xin Huang, Jaime Campos Salas, Yue Sun, Nathan Pemberton, Xiang Song, Ashish Khetan, George Karypis
Machine Learning
A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
Viet Nguyen, Tuan Minh Pham, Thinh Cao et al. (+4 more)
Self-
attention has greatly contributed to the success of the widely used
Transformer architecture by enabling learning from data with long-range dependencies. In an effort to improve performance, a gated
attention model that leverages a gating mechanism within the multi-head self-
attention has recently been proposed as a promising alternative. Gated
attention has been empirically demonstrated to increase the expressiveness of low-rank mapping in standard
attention and even to eliminate the
attention sink phenomenon. Despite its efficacy, a clear theoretical understanding of gated
attention's benefits remains lacking in the literature. To close this gap, we rigorously show that each entry in a gated
attention matrix or a multi-head self-
attention matrix can be written as a hierarchical mixture of experts. By recasting learning as an expert estimation problem, we demonstrate that gated
attention is more sample-efficient than multi-head self-
attention. In particular, while the former needs only a
polynomial number of data points to recover the model parameters, the latter requires exponentially many data points to achieve the same estimation error. Furthermore, our analysis also provides a theoretical justification for why gated
attention is more effective at mitigating the
attention sink phenomenon and the low-rank bottleneck when a gate is placed at the output of the scaled dot product
attention or the value map rather than at other positions in the multi-head self-
attention architecture.
Read the full abstract ▾
All authors Viet Nguyen, Tuan Minh Pham, Thinh Cao, Tan Dinh, Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Machine Learning · Cross-listed from stat.ML
Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
Tuan Minh Pham, Thinh Cao, Viet Nguyen et al. (+3 more)
The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to
language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a
polynomial order.
Read the full abstract ▾
All authors Tuan Minh Pham, Thinh Cao, Viet Nguyen, Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Machine Learning · Cross-listed from math.OC
Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimator
Haoyu Han and Heng Yang
Policy-gradient methods are widely used in reinforcement learning, yet training often becomes unstable or slows down as learning progresses. We study this phenomenon through the noise-to-signal ratio (NSR) of a policy-gradient
estimator, defined as the
estimator variance (noise) normalized by the squared norm of the true gradient (signal). Our main result is that, for (i) finite-horizon linear systems with Gaussian policies and linear state-feedback, and (ii) finite-horizon
polynomial systems with Gaussian policies and
polynomial feedback, the NSR of the REINFORCE
estimator can be characterized exactly-either in closed form or via numerical moment-evaluation
algorithms-without approximation. For general nonlinear dynamics and expressive policies (including neural policies), we further derive a general upper bound on the variance. These characterizations enable a direct examination of how NSR varies across policy parameters and how it evolves along optimization trajectories (e.g. SGD and Adam). Across a range of examples, we find that the NSR landscape is highly non-uniform and typically increases as the policy approaches an optimum; in some regimes it blows up, which can trigger training instability and policy collapse.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Understanding vision transformer robustness through the lens of out-of-distribution detection
Joey Kuang and Alexander Wong
Vision transformers have shown remarkable performance in
vision tasks, but enabling them for accessible and real-time use is still challenging. Quantization reduces
memory and inference costs at the risk of performance loss. Strides have been made to mitigate low precision issues mainly by understanding in-distribution (ID) task behaviour, but the
attention mechanism may provide insight on quantization attributes by exploring out-of-distribution (OOD) situations. We investigate the behaviour of quantized small-variant popular
vision transformers (DeiT, DeiT3, and ViT) on common OOD
datasets. ID analyses show the initial instabilities of 4-bit models, particularly of those trained on the larger ImageNet-22k, as the strongest FP32 model, DeiT3, sharply drop 17% from quantization error to be one of the weakest 4-bit models. While ViT shows reasonable quantization robustness for ID calibration, OOD detection reveals more: ViT and DeiT3 pretrained on ImageNet-22k respectively experienced a 15.0% and 19.2% average quantization delta in AUPR-out between full precision to 4-bit while their ImageNet-1k-only counterparts experienced a 9.5% and 12.0% delta. Overall, our results suggest pretraining on large scale
datasets may hinder low-bit quantization robustness in OOD detection and that data augmentation may be a more beneficial option.
Read the full abstract ▾
Machine Learning
Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representations
Yilun Kuang, Yash Dagade, Tim G. J. Rudner et al. (+2 more)
Joint-
Embedding Predictive Architectures (JEPA) learn view-invariant representations and admit projection-based distribution matching for collapse prevention. Existing approaches regularize representations towards isotropic Gaussian distributions, but inherently favor dense representations and fail to capture the key property of sparsity observed in efficient representations. We introduce Rectified Distribution Matching Regularization (RDMReg), a sliced two-sample distribution-matching loss that aligns representations to a Rectified Generalized Gaussian (RGG) distribution. RGG enables explicit control over expected $\ell_0$ norm through rectification, while its continuous truncated component admits a maximum-
entropy characterization under expected $\ell_p$ norm and support constraints. Equipping JEPAs with RDMReg yields Rectified LpJEPA, which strictly generalizes prior Gaussian-based JEPAs. Empirically, Rectified LpJEPA learns sparse, non-negative representations with favorable sparsity–performance trade-offs and competitive downstream performance on image classification
benchmarks, showing that RDMReg can enforce sparsity while preserving task-relevant information.
Read the full abstract ▾
All authors Yilun Kuang, Yash Dagade, Tim G. J. Rudner, Randall Balestriero, Yann LeCun
Machine Learning
Modeling Topological Impact on Node Attribute Distributions in Attributed Graphs
Amirreza Shiralinasab Langari, Leila Yeganeh and Kim Khoa Nguyen
We investigate how the
topology of attributed graphs influences the distribution of node attributes. This work offers a novel perspective by treating
topology and attributes as structurally distinct but interacting components. We introduce an algebraic approach that combines a graph's
topology with the probability distribution of node attributes, resulting in
topology-influenced distributions. First, we develop a categorical framework to formalize how a node perceives the graph's
topology. We then quantify this point of view and integrate it with the distribution of node attributes to capture topological effects. We interpret these
topology-conditioned distributions as approximations of the posteriors $P(\cdot \mid v)$ and $P(\cdot \mid \mathcal{G})$. We further establish a principled sufficiency condition by showing that, on complete graphs, where
topology carries no informative structure, our construction recovers the original attribute distribution. To evaluate our approach, we introduce an intentionally simple testbed model, $\textbf{ID}$, and use unsupervised graph anomaly detection as a probing task.
Read the full abstract ▾
Machine Learning
The Horizon Threshold in Cooperative Multi-Agent Reward-Free Exploration
Idan Barnea, Orin Levy and Yishay Mansour
We study cooperative multi-
agent reinforcement learning in the setting of reward-free exploration, where multiple
agents jointly explore an unknown MDP in order to learn its dynamics (without observing rewards). We focus on a tabular finite-horizon MDP and adopt a phased learning framework. In each learning phase, multiple
agents independently interact with the environment. More specifically, in each learning phase, each
agent is assigned a policy, executes it, and observes the resulting trajectory. Our primary goal is to characterize the tradeoff between the number of learning phases and the number of
agents, especially when the number of learning phases is small. Our results identify a regime change governed by the horizon $H$. When the number of learning phases equals $H$, we present a computationally efficient
algorithm that uses only $\tilde{O}(S^6 H^6 A / \epsilon^2)$
agents to obtain an $\epsilon$ approximation of the dynamics (i.e., yields an $\epsilon$-optimal policy for any reward function). We complement our
algorithm with a lower bound showing that any
algorithm restricted to $\rho < H$ phases requires at least $A^{H/\rho}$
agents to achieve constant accuracy. Thus, we show that having $\Theta(H)$ learning phases is both necessary and sufficient when restricting the number of
agents to be
polynomial.
Read the full abstract ▾
Machine Learning
A Meta-Knowledge-Augmented LLM Framework for Hyperparameter Optimization in Time-Series Forecasting
Ons Saadallah, Mátyás andó and Tamás Gábor Orosz
Hyperparameter optimization (HPO) plays a central role in the performance of deep learning models, yet remains computationally expensive and difficult to interpret, particularly for time-series
forecasting. While
Bayesian Optimization (BO) is a standard approach, it typically treats tuning tasks independently and provides limited insight into its decisions. Recent advances in large
language models (LLMs) offer new opportunities to incorporate structured prior knowledge and
reasoning into optimization pipelines. We introduce LLM-AutoOpt, a hybrid HPO framework that combines BO with LLM-based contextual
reasoning. The framework encodes
dataset meta-features, model descriptions, historical optimization outcomes, and target objectives as structured meta-knowledge within LLM
prompts, using BO to initialize the search and mitigate cold-start effects. This design enables context-aware and stable hyperparameter refinement while exposing the
reasoning behind optimization decisions. Experiments on a multivariate time series
forecasting benchmark demonstrate that LLM-AutoOpt achieves improved predictive performance and more interpretable optimization behavior compared to BO and LLM baselines without meta-knowledge.
Read the full abstract ▾
Machine Learning
Hidden Heroes and Gradient Bloats: Layer-Wise Redundancy Inverts Attribution in Transformers
Donald Ye
Gradient-based attribution is the workhorse of mechanistic interpretability, yet whether it reliably tracks causal importance at the component level remains largely untested. We causally evaluate this assumption across two algorithmic tasks and up to 10 random seeds, uncovering a systematic, layer-wise failure: gradient attribution consistently overvalues early-layer Gradient Bloats and undervalues late-layer Hidden Heroes. Rank correlation collapses from $\rho = 0.72$ on sequence reversal to $0.27$ on sequence sorting, reaching $\rho = -0.18$ in individual seeds. This failure stems from first-order gradient attribution's inability to detect collective redundancy: joint Bloat ablation causes $14\times$ greater damage than individual results predict. Consequently, Bloats dominate gradient rankings despite negligible functional impact, while ablating Hidden Heroes destroys OOD accuracy ($-36.4\% \pm 22.8\%$). This systematic inversion of early-layer feature extraction and late-layer computation motivates causal validation as a prerequisite for
circuit-level claims.
Read the full abstract ▾
Machine Learning
TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
Perry Dong, Kuo-Han Hung, Alexander Swerdlow et al. (+2 more)
Despite scale driving substantial recent advancements in machine learning, reinforcement learning (RL) methods still primarily use small value functions. Naively scaling value functions – including with a
transformer architecture, which is known to be highly scalable – often results in learning instability and worse performance. In this work, we ask what prevents
transformers from scaling effectively for value functions? Through empirical analysis, we identify the critical failure mode in this scaling:
attention scores collapse as capacity increases. Our key insight is that we can effectively prevent this collapse and stabilize training by controlling the
entropy of the
attention scores, thereby enabling the use of larger models. To this end, we propose
Transformer Q-Learning (TQL), a method that unlocks the scaling potential of
transformers in learning value functions in RL. Our approach yields up to a 43% improvement in performance when scaling from the smallest to the largest network sizes, while prior methods suffer from performance degradation.
Read the full abstract ▾
All authors Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn
Machine Learning
Theoretical Analysis of Measure Consistency Regularization for Partially Observed Data
Yinsong Wang and Shahin Shahrampour
The problem of corrupted data, missing features, or missing modalities continues to plague the modern machine learning landscape. To address this issue, a class of regularization methods that enforce consistency between imputed and fully observed data has emerged as a promising approach for improving model generalization, particularly in partially observed settings. We refer to this class of methods as Measure Consistency Regularization (MCR). Despite its empirical success in various applications, such as image inpainting, data imputation and semi-supervised learning, a fundamental understanding of the theoretical underpinnings of MCR remains limited. This paper bridges this gap by offering theoretical insights into when MCR yields a more favorable finite-sample estimation-error upper bound, viewed through the lens of neural network distance. Under ideal interpolation and compatibility conditions, we show that the MCR estimation-error upper bound is no larger than vanilla supervised training and becomes strictly smaller when the mixed-sample alternative is favorable. We then extend the analysis to the non-ideal regime, where optimization and compatibility residuals can potentially offset this finite-sample advantage. Guided by these insights, we propose a novel practical diagnostic that leverages the duality gap and a calibrated drift
estimator to infer the potential benefit of MCR training. We present detailed empirical evidence to support our theoretical claims and to show the effectiveness and accuracy of our practical diagnostic. We further provide simulations on real-world
datasets to show the versatility of MCR under different model architectures designed for different data sources.
Read the full abstract ▾
Machine Learning
Phase Transitions for Feature Learning in Neural Networks
Andrea Montanari and Zihao Wang
According to a popular viewpoint, neural networks learn from data by first identifying low-dimensional representations, and subsequently fitting the best model in this space. Recent works provide a formalization of this phenomenon when learning multi-index models. In this setting, we are given $n$ i.i.d. pairs $({\boldsymbol x}_i,y_i)$, where the
covariate vectors ${\boldsymbol x}_i\in\mathbb{R}^d$ are isotropic, and responses $y_i$ only depend on ${\boldsymbol x}_i$ through a $k$-dimensional projection ${\boldsymbol \Theta}_*^{{\sf T}}{\boldsymbol x}_i$. Feature learning amounts to learning the latent space spanned by ${\boldsymbol \Theta}_*$. In this context, we study the gradient descent dynamics of two-layer neural networks under the proportional asymptotics $n,d\to\infty$, $n/d\to\delta$, while the dimension of the latent space $k$ and the number of hidden
neurons $m$ are kept fixed. Earlier work establishes that feature learning via
polynomial-time
algorithms is possible if $\delta> \delta_{\text{alg}}$, for $\delta_{\text{alg}}$ a threshold depending on the data distribution, and is impossible (within a certain class of
algorithms) below $\delta_{\text{alg}}$. Here we derive an analogous threshold $\delta_{\text{NN}}$ for two-layer networks. Our characterization of $\delta_{\text{NN}}$ opens the way to study the dependence of learning dynamics on the network architecture and training
algorithm. The threshold $\delta_{\text{NN}}$ is determined by the following scenario. Training first visits points for which the gradient of the empirical risk is large and learns the directions spanned by these gradients. Then the gradient becomes smaller and the dynamics becomes dominated by negative directions of the Hessian. The threshold $\delta_{\text{NN}}$ corresponds to a phase transition in the spectrum of the Hessian in this second phase.
Read the full abstract ▾
Machine Learning
DCD: Decomposition-based Causal Discovery from Autocorrelated and Non-Stationary Temporal Data
Muhammad Hasan Ferdous, Ian Stockwell and Md Osman Gani
Multivariate time series in domains such as finance,
climate science, and healthcare often exhibit long-term trends, seasonal patterns, and short-term fluctuations, complicating causal inference under non-stationarity and autocorrelation. Existing causal discovery methods typically operate on raw observations, making them vulnerable to spurious edges and misattributed temporal dependencies. We introduce a decomposition-based causal discovery framework that separates each time series into trend, seasonal, and residual components and performs component-specific causal analysis. Trend components are assessed using stationarity tests, seasonal components using kernel-based dependence measures, and residual components using constraint-based causal discovery. The resulting component-level graphs are integrated into a unified multi-scale causal structure. This approach isolates long- and short-range causal effects, reduces spurious associations, and improves interpretability. Across extensive synthetic
benchmarks and real-world
climate data, our framework more accurately recovers ground-truth causal structure than state-of-the-art baselines, particularly under strong non-stationarity and temporal autocorrelation.
Read the full abstract ▾
Machine Learning
Improving the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Models
Weiqing He, Xiang Li, Li Shen et al. (+2 more)
Watermarking is a principled approach for tracing the provenance of large
language model (LLM) outputs, but its deployment in practice is hindered by inference inefficiency. Speculative sampling accelerates inference, with efficiency improving as the acceptance rate between draft and target models increases. Yet recent work reveals a fundamental trade-off: higher watermark strength reduces acceptance, preventing their simultaneous achievement. We revisit this trade-off and show it is not absolute. We introduce a quantitative measure of watermark strength that governs statistical detectability and is maximized when
tokens are deterministic functions of pseudorandom numbers. Using this measure, we fully characterize the trade-off as a constrained optimization problem and derive explicit Pareto curves for two existing watermarking schemes. Finally, we introduce a principled mechanism that injects pseudorandomness into draft-
token acceptance, ensuring maximal watermark strength while maintaining speculative sampling efficiency. Experiments further show that this approach improves detectability without sacrificing efficiency. Our findings uncover a principle that unites speculative sampling and watermarking, paving the way for their efficient and practical deployment.
Read the full abstract ▾
All authors Weiqing He, Xiang Li, Li Shen, Weijie Su, Qi Long
Machine Learning · Cross-listed from stat.ML
Robust Generalization with Adaptive Optimal Transport Priors for Decision-Focused Learning
Haixiang Sun and Andrew L. Liu
Few-shot learning requires models to generalize under limited supervision while remaining robust to distribution shifts. Existing Sinkhorn Distributionally Robust Optimization (DRO) methods provide theoretical guarantees but rely on a fixed reference distribution, which limits their adaptability. We propose a Prototype-Guided Distributionally Robust Optimization (PG-DRO) framework that learns class-adaptive priors from abundant base data via hierarchical optimal transport and embeds them into the Sinkhorn DRO formulation. This design enables few-shot information to be organically integrated into producing class-specific robust decisions that are both theoretically grounded and efficient, and further aligns the uncertainty set with transferable structural knowledge. Experiments show that PG-DRO achieves stronger robust generalization in few-shot scenarios, outperforming both standard learners and DRO baselines.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
One Probe Won't Catch Them All: Towards Targeted Deception Detection
Vikram Natarajan, Devina Jain, Shivam Arora et al. (+2 more)
Linear probes are a promising approach for monitoring AI systems for deceptive behaviour. Previous work has shown that a linear classifier trained on a contrastive instruction pair and a simple
dataset can achieve good performance. However, these probes exhibit notable failures even in straightforward scenarios, including spurious correlations and false positives on non-deceptive responses. In this paper, we demonstrate that deception detection is inherently heterogeneous: while a single universal probe achieves modest improvements (+0.032 AUC), post-hoc oracle analysis reveals substantially higher potential (+0.108 AUC) when probes are matched to specific deception types, and synthetic validation experiments suggest this ceiling is achievable a priori when the deception type is known in advance. Our findings reveal that instruction pairs capture deceptive intent rather than content-specific patterns, explaining why
prompt choice dominates probe performance (70.6% of variance). Given this heterogeneity, we conclude that organizations should define their specific threat models and deploy appropriately matched probes rather than seeking a universal deception detector.
Read the full abstract ▾
All authors Vikram Natarajan, Devina Jain, Shivam Arora, Satvik Golechha, Joseph Bloom
Machine Learning
Semi-supervised CAPP Transformer Learning via Pseudo-labeling
Dennis Gross, Helge Spieker, Arnaud Gotlieb et al. (+3 more)
High-level Computer-Aided Process Planning (CAPP) generates manufacturing process plans from part specifications. It suffers from limited
dataset availability in industry, reducing model generalization. We propose a semi-supervised learning approach to improve
transformer-based CAPP
transformer models without manual labeling. An oracle, trained on available
transformer behaviour data, filters correct predictions from unseen parts, which are then used for one-shot retraining. Experiments on small-scale
datasets with simulated ground truth across the full data distribution show consistent accuracy gains over baselines, demonstrating the method's effectiveness in data-scarce manufacturing environments.
Read the full abstract ▾
All authors Dennis Gross, Helge Spieker, Arnaud Gotlieb, Emmanuel Stathatos, Panorios Benardos, George-Christopher Vosniakos
Machine Learning · Cross-listed from cs.CV
Parabolic Positional Encoding: A Vision-Centric, Principled, Extrapolatable, and General Approach
Christoffer Koo Øhrstrøm, Rafael I. Cabral Muchacho, Yifei Dong et al. (+4 more)
We propose
Parabolic Positional Encoding (PaPE), a parabola-based positional encoding for
vision modalities in
attention-based architectures. Given a set of
vision tokens-such as from
videos, event camera streams, images, or point clouds-our objective is to encode their positions while accounting for the characteristics of
vision modalities. Prior works have largely extended positional encodings from 1D-sequences in
language to nD-structures in
vision, but only with partial account of
vision characteristics. We address this gap by designing PaPE from principles distilled from prior work: translation invariance, rotation invariance (PaPE-RI), distance decay, directionality, and context awareness. Extrapolation experiments on ImageNet-1K show how PaPE extrapolates remarkably well, improving in absolute terms by up to 10.5 over the next-best encoding. Generality experiments on 8
datasets across 4 modalities show that PaPE is a general
vision positional encoding, as PaPE matches the best baseline on 5
datasets and exceeds all on 2
datasets. The code is available at https://github.com/DTU-PAS/
parabolic-position-encoding.
Read the full abstract ▾
All authors Christoffer Koo Øhrstrøm, Rafael I. Cabral Muchacho, Yifei Dong, Filippos Moumtzidellis, Ronja Güldenring, Florian T. Pokorny, Lazaros Nalpantidis
Machine Learning · Cross-listed from stat.ML
Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference
Kamélia Daudel, Minh-Ngoc Tran and Cheng Zhang
Importance-weighted variational inference approximates intractable densities by optimizing multi-sample Monte Carlo objectives whose tightness increases with the number of importance samples $N$. When reparameterization is unavailable, optimization relies on score-function (REINFORCE) gradient
estimators. In this paper, we provide a comprehensive analysis of score-function gradient
estimators for importance-weighted objectives by characterizing their asymptotic expectation and variance. We establish that standard variational inference for Monte Carlo objectives (VIMCO) gradient
estimators suffer from a vanishing signal-to-noise ratio (SNR) as $N$ increases, which arises from a suboptimal control variate choice and can impair optimization. To address this issue, we derive a general mechanism for constructing VIMCO gradient
estimators that avoid SNR collapse. We identify the High-Effective Sample Size optimal variance importance sampling
estimator as belonging to this category of
estimators and establish its $\sqrt{N}$ SNR scaling. This
estimator does not, however, have zero variance at optimality and achieving its $\sqrt{N}$ SNR rate may require prohibitively large $N$ when the importance weights are degenerate. We overcome these limitations by introducing VIMCO gradient
estimators that attain zero variance at optimality and support importance weight tempering. Experiments with discrete latent-variable and state-space models validate the theoretical predictions and illustrate the benefits of our
estimators.
Read the full abstract ▾