Machine Learning
Curvature-Aligned Probing for Local Loss-Landscape Stabilization
Nikita Kiselev and Andrey Grabovoy
Local loss-landscape stabilization under sample growth is typically measured either pointwise or through isotropic averaging in the full parameter space. Despite practical value, both choices probe directions that contribute little to the dominant local deformation of strongly anisotropic neural landscapes. We recast stabilization as an observational problem and introduce a unified family of criteria parameterized by an aggregation order and a probing distribution; within this family we propose a
curvature-aligned criterion $\Delta_2^{(D)}$ that probes the loss increment field in the top-$D$ eigenspace of the empirical Hessian near a trained solution. Solely from a local quadratic model, we prove that $\Delta_2^{(D)}$ preserves the $O(k^{-2})$ mean-squared rate of the full-space criterion while replacing ambient-dimension
curvature dependence with dependence on the subspace dimension $D$; a corollary gives a closed-form spectral expression and a proposition identifies the top-$D$ eigenspace as extremal within the eigenspace-aligned family. We also derive scalable
estimators based on Hessian-vector products, subspace Monte Carlo, and a closed-form Gaussian-moment proxy. On a decoder-only
transformer, a
curvature-aligned probe occupying a tiny fraction of parameter space already reproduces the full-space mean-squared signal to within numerical noise throughout the validated local regime, and the closed-form
estimator is orders of magnitude faster than direct Monte Carlo after subspace construction.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Best of both worlds: Stochastic & adversarial best-arm identification
Yasin Abbasi-Yadkori, Peter L. Bartlett, Victor Gabillon et al. (+2 more)
We study bandit best-arm identification with arbitrary and potentially
adversarial rewards. A simple random uniform learner obtains the optimal rate of error in the
adversarial scenario. However, this type of strategy is suboptimal when the rewards are sampled stochastically. Therefore, we ask: Can we design a learner that performs optimally in both the
stochastic and
adversarial problems while not being aware of the nature of the rewards? First, we show that designing such a learner is impossible in general. In particular, to be robust to
adversarial rewards, we can only guarantee optimal rates of error on a subset of the
stochastic problems. We give a lower bound that characterizes the optimal rate in
stochastic problems if the strategy is constrained to be robust to
adversarial rewards. Finally, we design a simple parameter-free
algorithm and show that its probability of error matches (up to log factors) the lower bound in
stochastic problems, and it is also robust to
adversarial ones.
Read the full abstract ▾
All authors Yasin Abbasi-Yadkori, Peter L. Bartlett, Victor Gabillon, Alan Malek, Michal Valko
Machine Learning
Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization
Zhiyuan Zhai, Bingcong Li, Bingnan Xiao et al. (+2 more)
Test-time compute scaling, the practice of spending extra computation during inference via repeated sampling, search, or extended
reasoning, has become a powerful lever for improving large
language model performance. Yet deploying these techniques under finite inference budgets requires a decision that current systems largely ignore: which inputs deserve more compute, and which can be answered cheaply? We formalize this as a constrained optimization problem (maximize expected accuracy subject to an average compute budget) and solve it with a two-stage Solve-then-Learn pipeline. In the solve stage, Lagrangian relaxation decomposes the global constraint into per-instance sub-problems, each admitting a closed-form oracle action that optimally prices accuracy against cost. We prove that the induced cost is monotone in the dual variable, enabling exact budget targeting via binary search. In the learn stage, a lightweight classifier is trained to predict oracle actions from cheap input features, amortizing the allocation rule for real-time deployment. We establish that the task-level regret of the learned policy is bounded by its imitation error times the worst-case per-instance gap, yielding a clean reduction from constrained inference to supervised classification. Experiments on MATH and GSM8K with three LLMs (DeepSeek-V3, GPT-4o-mini, Qwen2.5-7B) show that our method consistently outperforms uniform and heuristic allocation baselines, achieving up to 12.8% relative accuracy improvement on MATH under matched budget constraints, while closely tracking the Lagrangian oracle upper bound with over 91% imitation accuracy.
Read the full abstract ▾
All authors Zhiyuan Zhai, Bingcong Li, Bingnan Xiao, Ming Li, Xin Wang
Machine Learning · Cross-listed from cs.PL
Nautilus: An Auto-Scheduling Tensor Compiler for Efficient Tiled GPU Kernels
Yifan Zhao, Yuchen Yang, Matei Budiu et al. (+1 more)
We present Nautilus, a novel tensor compiler that moves toward fully automated math-to-kernel optimization. Nautilus compiles a high-level algebraic specification of tensor operators into efficient tiled GPU kernels. Nautilus's successive lowering design allows high-level optimizations, expression rewrites, and tile optimizations to be jointly applied in a single end-to-end system. Nautilus presents a novel auto-scheduler that discovers sequences of high-level optimizations, while preserving the regular program structure needed by tile optimizers. Nautilus's auto-scheduler captures complex interactions and trade-offs in the high-level optimizations, including aggressive global transformations like advanced reduction fusion. Nautilus is the first end-to-end tensor compiler capable of starting from a math-like description of
attention and automatically discovering FlashAttention-3-like kernels, offloading the entire burden of optimization from the programmer to the compiler. Across five
transformer-based models and 150 evaluation configurations on NVIDIA GH200 and RTX 5090 GPUs, Nautilus achieves up to 23% higher throughput than state-of-the-art compilers on GH200 and up to 42% on RTX 5090, while matching or exceeding manually written cuDNN kernels on many long-sequence configurations.
Read the full abstract ▾
All authors Yifan Zhao, Yuchen Yang, Matei Budiu, Sasa Misailovic
Machine Learning
Learning Ad Hoc Network Dynamics via Graph-Structured World Models
Can Karacelebi, Yusuf Talha Sahin, Elif Surer et al. (+1 more)
Ad hoc wireless networks exhibit complex, innate and coupled dynamics: node mobility, energy depletion and
topology change that are difficult to model analytically. Model-free deep reinforcement learning requires sustained online interaction whereas existing model based approaches use flat state representations that lose per node structure. Therefore we propose G-RSSM, a graph structured recurrent state space model that maintains per node latent states with cross node multi head
attention to learn the dynamics jointly from offline trajectories. We apply the proposed method to the downstream task clustering where a cluster head selection policy trains entirely through imagined rollouts in the learned world model. Across 27 evaluation scenarios spanning MANET, VANET, FANET, WSN and tactical networks with N=30 to 1000 nodes, the learned policy maintains high connectivity with only trained for N=50. Herein, we propose the first multi physics graph structured world model applied to combinatorial per node decision making in size agnostic wireless ad hoc networks.
Read the full abstract ▾
All authors Can Karacelebi, Yusuf Talha Sahin, Elif Surer, Ertan Onur
Machine Learning · Cross-listed from stat.ML
Scalable Model-Based Clustering with Sequential Monte Carlo
Connie Trojan, Pavel Myshkov, Paul Fearnhead et al. (+3 more)
In online clustering problems, there is often a large amount of uncertainty over possible cluster assignments that cannot be resolved until more data are observed. This difficulty is compounded when clusters follow complex distributions, as is the case with text data. Sequential Monte Carlo (SMC) methods give a natural way of representing and updating this uncertainty over time, but have prohibitive
memory requirements for large-scale problems. We propose a novel SMC
algorithm that decomposes clustering problems into approximately independent subproblems, allowing a more compact representation of the
algorithm state. Our approach is motivated by the knowledge base construction problem, and we show that our method is able to accurately and efficiently solve clustering problems in this setting and others where traditional SMC struggles.
Read the full abstract ▾
All authors Connie Trojan, Pavel Myshkov, Paul Fearnhead, James Hensman, Tom Minka, Christopher Nemeth
Machine Learning · Cross-listed from stat.ML
Expert-Guided Class-Conditional Goodness-of-Fit Scores for Interpretable Classification with Informative Missingness
Shahar Cohen, David M. Steinberg, Yael Radzyner et al. (+1 more)
We study a classification problem with three key challenges: pervasive informative missingness, the integration of partial prior expert knowledge into the learning process, and the need for interpretable decision rules. We propose a framework that encodes prior knowledge through an expert-guided class-conditional model for one or more classes, and use this model to construct a small set of interpretable goodness-of-fit features. The features quantify how well the observed data agree with the expert model, isolating the contributions of different aspects of the data, including both observed and missing components. These features are combined with a few transparent auxiliary summaries in a simple discriminative classifier, resulting in a decision rule that is easy to inspect and justify. We develop and apply the framework in the context of seismic monitoring used to assess compliance with the Comprehensive
Nuclear-Test-Ban Treaty. We show that the method has strong potential as a transparent screening tool, reducing workload for expert analysts. A simulation designed to isolate the contribution of the proposed framework shows that this interpretable expert-guided method can even outperform strong standard machine-learning classifiers, particularly when training samples are small.
Read the full abstract ▾
All authors Shahar Cohen, David M. Steinberg, Yael Radzyner, Yochai Ben Horin
Machine Learning · Cross-listed from q-bio.BM
PUFFIN: Protein Unit Discovery with Functional Supervision
Gökçe Uludoğan, Buse Giledereli, Elif Ozkirimli et al. (+1 more)
Proteins carry out biological functions through the coordinated action of groups of residues organized into structural arrangements. These arrangements, which we refer to as
protein units, exist at an intermediate scale, being larger than individual residues yet smaller than entire
proteins. A deeper understanding of
protein function can be achieved by identifying these units and their associations with function. However, existing approaches either focus on residue-level signals, rely on curated annotations, or segment
protein structures without incorporating functional information, thereby limiting interpretable analysis of structure-function relationships. We introduce PUFFIN, a data-driven framework for discovering
protein units by jointly learning structural partitioning and functional supervision. PUFFIN represents
proteins as residue-level structure graphs and applies a graph neural network with a structure-aware pooling mechanism that partitions each
protein into multi-residue units, with functional supervision that shapes the partition. We show that the learned units are structurally coherent, exhibit organized associations with molecular function, and show meaningful correspondence with curated InterPro annotations. Together, these results demonstrate that PUFFIN provides an interpretable framework for analyzing structure-function relationships using learned
protein units and their statistical function associations. We made our source code available at https://github.com/boun-tabi-lifelu/puffin.
Read the full abstract ▾
All authors Gökçe Uludoğan, Buse Giledereli, Elif Ozkirimli, Arzucan Özgür
Machine Learning · Cross-listed from eess.SY
Towards Trustworthy 6G Network Digital Twins: A Framework for Validating Counterfactual What-If Analysis in Edge Computing Resources
Julian Jimenez Agudelo, Paola Soto, Ayat Zaki-Hindi et al. (+5 more)
Network Digital Twins (NDTs) enable safe what-if analysis for 6G cloud-edge infrastructures, but adoption is often limited by fragmented workflows from telemetry to validation. We present a data-driven NDT framework that extends 6G-TWIN with a scalable pipeline for cloud-edge telemetry aggregation and semantic
alignment into unified data models. Our contributions include: (i) scalable cloud-edge telemetry collection, (ii) regime-aware feature engineering capturing the network's scaling behavior, and (iii) a validation methodology based on Sign Agreement and Directional Sensitivity. Evaluated on a Kubernetes-managed cluster, the framework extrapolates performance to unseen high-load regimes. Results show both Deep Neural Network (DNN) and XGBoost achieve high regression accuracy (R2 > 0.99), while the XGBoost model delivers superior directional reliability (Sa > 0.90), making the NDT a trustworthy tool for proactive resource scaling in out-of-distribution scenarios.
Read the full abstract ▾
All authors Julian Jimenez Agudelo, Paola Soto, Ayat Zaki-Hindi, Jean-Sébastien Sottet, Sébastien Faye, Nina Slamnik-Kriještorac, Johann Marquez-Barja, Miguel Camelo Botero
Machine Learning
Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization
Fu Feng, Yucheng Xie, Ruixiao Shi et al. (+2 more)
The pre-training and fine-tuning paradigm has become the dominant approach for model adaptation. However, conventional pre-training typically yields models at a fixed scale, whereas practical deployment often requires models of varying sizes, exposing its limitations when target model scales differ from those used during pre-training. To address this, we propose an innovative constraint-based pre-training paradigm that imposes structured constraints during pre-training to disentangle size-agnostic knowledge into reusable weight templates, while assigning size-specific adaptation to lightweight weight scalers, thereby reformulating variable-sized model initialization as a multi-task adaptation problem. Within this paradigm, we further introduce WeiT, which employs Kronecker-based constraints to regularize the pre-training process. Specifically, model parameters are represented as compositions of weight templates via concatenation and weighted aggregation, with adaptive connections governed by lightweight weight scalers whose parameters are learned from limited data. This design enables flexible and efficient construction of model weights across diverse downstream scales. Extensive experiments demonstrate the efficiency and effectiveness of WeiT, achieving state-of-the-art performance in initializing models with varying depths and widths across a broad range of perception and
embodied learning tasks, including Image Classification, Image Generation, and
Embodied Control. Moreover, its effectiveness generalizes to both
Transformer-based and Convolution-based architectures, consistently enabling faster convergence and improved performance even under full training.
Read the full abstract ▾
All authors Fu Feng, Yucheng Xie, Ruixiao Shi, Jing Wang, Xin Geng
Machine Learning
Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
Mathias Dus
We present a
geometric framework for Reinforcement Learning (RL) that views policies as maps into the Wasserstein space of action probabilities. First, we define a Riemannian structure induced by stationary distributions, providing rigorous guarantees for their existence in a general context. We then define the tangent space of policies and characterize the geodesics, specifically addressing the measurability of vector fields mapping from the state space to the tangent space of probability measures over the action space. Next, we formulate a general RL optimization problem and construct a gradient flow using Otto's calculus. We compute the gradient and the Hessian of the energy (i.e., the expected cumulative cost), providing a formal second-order analysis. Finally, we illustrate the method with numerical examples: we compute the gradient directly from our theoretical formalism for low-dimensional problems, and we illustrate the numerical applicability of our formal framework to high-dimensional continuous control by parameterizing the policy with a neural network optimized via an ergodic approximation of the cost.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IT
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
Adrian Edin, Michel Kieffer, Mikael Johansson et al. (+1 more)
The communication bottleneck in
federated learning (FL) has spurred extensive research into techniques to reduce the volume of data exchanged between client devices and the central parameter server. In this paper, we systematically classify gradient and model compression schemes into three categories based on the type of correlations they exploit: structural, temporal, and spatial. We examine the sources of such correlations, propose quantitative metrics for measuring their magnitude, and reinterpret existing compression methods through this unified correlation-based framework. Our experimental studies demonstrate that the degrees of structural, temporal, and spatial correlations vary significantly depending on task complexity, model architecture, and algorithmic configurations. These findings suggest that
algorithm designers should carefully evaluate correlation assumptions under specific deployment scenarios rather than assuming that they are always present. Motivated by these findings, we propose two adaptive compression designs that actively switch between different compression modes based on the measured correlation strength, and we evaluate their performance gains relative to conventional non-adaptive approaches. In summary, our unified taxonomy provides a clean and principled foundation for developing more effective and application-specific compression techniques for FL systems.
Read the full abstract ▾
All authors Adrian Edin, Michel Kieffer, Mikael Johansson, Zheng Chen
Machine Learning
Assessing the Performance-Efficiency Trade-off of Foundation Models in Probabilistic Electricity Price Forecasting
Jan Niklas Lettner, Hadeer El Ashhab, Veit Hagenmeyer et al. (+1 more)
Large-scale renewable energy deployment introduces pronounced
volatility into the electricity system, turning grid operation into a complex
stochastic optimization problem. Accurate electricity price
forecasting (EPF) is essential not only to support operational decisions, such as optimal bidding strategies and balancing power preparation, but also to reduce economic risk and improve
market efficiency. Probabilistic forecasts are particularly valuable because they quantify uncertainty stemming from renewable intermittency,
market coupling, and regulatory changes, enabling
market participants to make informed decisions that minimize losses and optimize expected revenues. However, it remains an open question which models to employ to produce accurate forecasts. Should these be task-specific machine learning (ML) models or Time Series Foundation Models (TSFMs)? In this work, we compare four models for day-ahead probabilistic EPF (PEPF) in European bidding zones: a deterministic NHITS backbone with
Quantile-Regression Averaging (NHITS+QRA) and a conditional Normalizing-Flow forecaster (NF) are compared with two TSFMs, namely Moirai and ChronosX. On the one hand, we find that TSFMs outperform task-specific deep learning models trained from scratch in terms of CRPS, Energy Score, and predictive interval calibration across
market conditions. On the other hand, we find that well-configured task-specific models, particularly NHITS combined with QRA, achieve performance very close to TSFMs, and in some scenarios, such as when supplied with additional informative feature groups or adapted via few-shot learning from other European
markets, they can even surpass TSFMs. Overall, our findings show that while TSFMs offer expressive modeling capabilities, conventional models remain highly competitive, emphasizing the need to weigh computational expense against marginal performance improvements in PEPF.
Read the full abstract ▾
All authors Jan Niklas Lettner, Hadeer El Ashhab, Veit Hagenmeyer, Benjamin Schäfer
Machine Learning · Cross-listed from cs.RO
World-Value-Action Model: Implicit Planning for Vision-Language-Action Systems
Runze Li, Hongyin Zhang, Junxi Jin et al. (+5 more)
Vision-
Language-Action (VLA) models have emerged as a promising paradigm for building
embodied agents that ground perception and
language into action. However, most existing approaches rely on direct action prediction, lacking the ability to reason over long-horizon trajectories and evaluate their consequences, which limits performance in complex decision-making tasks. In this work, we introduce World-Value-Action (WAV) model, a unified framework that enables implicit planning in VLA systems. Rather than performing explicit trajectory optimization, WAV model learn a structured latent representation of future trajectories conditioned on visual observations and
language instructions. A learned world model predicts future states, while a trajectory value function evaluates their long-horizon utility. Action generation is then formulated as inference in this latent space, where the model progressively concentrates probability mass on high-value and dynamically feasible trajectories. We provide a theoretical perspective showing that planning directly in action space suffers from an exponential decay in the probability of feasible trajectories as the horizon increases. In contrast, latent-space inference reshapes the search distribution toward feasible regions, enabling efficient long-horizon decision making. Extensive simulations and real-world experiments demonstrate that the WAV model consistently outperforms state-of-the-art methods, achieving significant improvements in task success rate, generalization ability, and robustness, especially in long-horizon and compositional scenarios. Code is available at https://github.com/Win-commit/WAV.
Read the full abstract ▾
All authors Runze Li, Hongyin Zhang, Junxi Jin, Qixin Zeng, Zifeng Zhuang, Yiqi Tang, Shangke Lyu, Donglin Wang
Machine Learning
Geometric Capacity of Transformers: A Tropical Geometry Perspective
Ye Su and Yong Liu
To quantify the
geometric capacity of
transformers, we develop a tropical-
geometric framework for analyzing the spatial partitions induced by conditioned self-
attention. In the zero-temperature limit, we show that fixed-key top-$1$ routing is exactly represented by a power diagram in query space, while an auxiliary log-lifted value parameterization yields a vector-valued tropical rational representation. For Multi-Head Self-
Attention (MHSA) with sequence length $N$ and $H$
attention heads, the joint routing geometry is encoded by Minkowski sums of headwise Newton polytopes, giving an $\mathcal{O}(N^H)$ universal bound that sharpens to $\mathcal{O}((HN)^{d_{\mathrm{model}}-1})$ once the number of heads reaches the intrinsic dimension $d_{\mathrm{model}}$. Extending this analysis across depth $L$, we derive the first tight asymptotic bounds on the number of linear regions in
transformers ($\Theta\!\left(N^{\min\{H,d_{\mathrm{model}}-1\}L}\right)$). We further show that finite-temperature softmax preserves the top-$1$ routing structure and admits exponentially decaying local approximation and differential bounds away from routing boundaries.
Read the full abstract ▾
Machine Learning
Catching Every Ripple: Enhanced Anomaly Awareness via Dynamic Concept Adaptation
Jiaqi Zhu, Shaofeng Cai, Jie Chen et al. (+3 more)
Online anomaly detection (OAD) plays a pivotal role in real-time analytics and decision-making for evolving data streams. However, existing methods often rely on costly retraining and rigid decision boundaries, limiting their ability to adapt both effectively and efficiently to concept drift in dynamic environments. To address these challenges, we propose DyMETER, a dynamic concept adaptation framework for OAD that unifies on-the-fly parameter shifting and dynamic thresholding within a single online paradigm. DyMETER first learns a static detector on historical data to capture recurring central concepts, and then transitions to a dynamic mode to adapt to new concepts as drift occurs. Specifically, DyMETER employs a novel dynamic concept adaptation mechanism that leverages a hypernetwork to generate instance-aware parameter shifts for the static detector, thereby enabling efficient and effective adaptation without retraining or fine-tuning. To achieve robust and interpretable adaptation, DyMETER introduces a lightweight
evolution controller to estimate instance-level concept uncertainty for adaptive updates. Further, DyMETER employs a dynamic threshold optimization module to adaptively recalibrates the decision boundary by maintaining a candidate window of uncertain samples, which ensures continuous
alignment with evolving concepts. Extensive experiments demonstrate that DyMETER significantly outperforms existing OAD approaches across a wide spectrum of application scenarios.
Read the full abstract ▾
All authors Jiaqi Zhu, Shaofeng Cai, Jie Chen, Fang Deng, Beng Chin Ooi, Wenqiao Zhang
Machine Learning · Cross-listed from cs.DB
RELOAD: A Robust and Efficient Learned Query Optimizer for Database Systems
Seokwon Lee, Jaeyoung Sim, Sihyun Kim et al. (+3 more)
Recent advances in query optimization have shifted from traditional rule-based and cost-based techniques towards machine learning-driven approaches. Among these, reinforcement learning (RL) has attracted significant
attention due to its ability to optimize long-term performance by learning policies over query planning. However, existing RL-based query optimizers often exhibit unstable performance at the level of individual queries, including severe performance regressions, and require prolonged training to reach the plan quality of expert, cost-based optimizers. These shortcomings make learned query optimizers difficult to deploy in practice and remain a major barrier to their adoption in production database systems. To address these challenges, we present RELOAD, a robust and efficient learned query optimizer for database systems. RELOAD focuses on (i) robustness, by minimizing query-level performance regressions and ensuring consistent optimization behavior across executions, and (ii) efficiency, by accelerating convergence to expert-level plan quality. Through extensive experiments on standard
benchmarks, including Join Order
Benchmark, TPC-DS, and
Star Schema
Benchmark, RELOAD demonstrates up to 2.4x higher robustness and 3.1x greater efficiency compared to state-of-the-art RL-based query optimization techniques.
Read the full abstract ▾
All authors Seokwon Lee, Jaeyoung Sim, Sihyun Kim, Yuhsing Li, Yiwen Zhu, Kwanghyun Park
Machine Learning · Cross-listed from cs.CV
HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet
Badri N. Patro and Vijay S. Agneeswaran
Vision State Space Models (SSMs) like Vim, VMamba, and SiMBA rely on complex scanning strategies to adapt sequential SSMs to process 2D images, introducing computational overhead and architectural complexity. We propose HAMSA, a scanning-free SSM operating directly in the spectral domain. HAMSA introduces three key innovations: (1) simplified kernel parameterization-a single Gaussian-initialized complex kernel replacing traditional (A, B, C)
matrices, eliminating discretization instabilities; (2) SpectralPulseNet (SPN)-an input-dependent frequency gating mechanism enabling adaptive spectral modulation; and (3) Spectral Adaptive Gating Unit (SAGU)-magnitude-based gating for stable gradient flow in the frequency domain. By leveraging FFT-based convolution, HAMSA eliminates sequential scanning while achieving O(L log L) complexity with superior simplicity and efficiency. On ImageNet-1K, HAMSA reaches 85.7% top-1 accuracy (state-of-the-art among SSMs), with 2.2 X faster inference than
transformers (4.2ms vs 9.2ms for DeiT-S) and 1.4-1.9X speedup over scanning-based SSMs, while using less
memory (2.1GB vs 3.2-4.5GB) and energy (12.5J vs 18-25J). HAMSA demonstrates strong generalization across transfer learning and dense prediction tasks.
Read the full abstract ▾
Machine Learning
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
Yuval Ran-Milo, Hila Ofek and Shahar Mendel
Transformers commonly exhibit an
attention sink: disproportionately high
attention to the first position. We study this behavior in GPT-2-style models with learned query biases and absolute positional
embeddings. Combining structural analysis with causal interventions, validated across natural-
language, mathematical, and code inputs, we find that the sink arises from the interaction among (i) a learned query bias, (ii) the first-layer MLP transformation of the positional encoding, and (iii) structure in the key projection. Crucially, each component we identify is individually dispensable: architectures omitting each of them robustly exhibit sinks. This indicates that
attention sinks may arise through distinct
circuits across architectures. These findings inform mitigation of sinks, and motivate broader investigation into why sinks emerge.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Layered Mutability: Continuity and Governance in Persistent Self-Modifying Agents
Krti Tallam
Persistent
language-model
agents increasingly combine tool use, tiered
memory, reflective prompting, and runtime adaptation. In such systems, behavior is shaped not only by current
prompts but by mutable internal conditions that influence future action. This paper introduces layered mutability, a framework for
reasoning about that process across five layers: pretraining, post-training
alignment, self-narrative,
memory, and weight-level adaptation. The central claim is that governance difficulty rises when mutation is rapid, downstream coupling is strong, reversibility is weak, and observability is low, creating a systematic mismatch between the layers that most affect behavior and the layers humans can most easily inspect. I formalize this intuition with simple drift, governance-load, and hysteresis quantities, connect the framework to recent work on temporal identity in
language-model
agents, and report a preliminary ratchet experiment in which reverting an
agent's visible self-description after
memory accumulation fails to restore baseline behavior. In that experiment, the estimated identity hysteresis ratio is 0.68. The main implication is that the salient failure mode for persistent self-modifying
agents is not abrupt misalignment but compositional drift: locally reasonable updates that accumulate into a behavioral trajectory that was never explicitly authorized.
Read the full abstract ▾
Machine Learning
Satwik Bathula and Anand A. Joshi
Multiplicative gating is widely used in neural architectures and has recently been applied to
attention layers to improve performance and training stability in large
language models. Despite the success of gated
attention, the mathematical implications of gated
attention mechanisms remain poorly understood. We study
attention through the geometry of its representations by modeling outputs as mean parameters of Gaussian distributions and analyzing the induced Fisher–Rao geometry. We show that ungated
attention operator is restricted to intrinsically flat statistical
manifolds due to its affine structure, while multiplicative gating enables non-flat geometries, including positively curved
manifolds that are unattainable in the ungated setting. These results establish a
geometric expressivity gap between ungated and gated
attention. Empirically, we show that gated models exhibit higher representation
curvature and improved performance on tasks requiring nonlinear decision boundaries whereas they provide no consistent advantage on tasks with linear decision boundaries. Furthermore, we identify a structured regime in which
curvature accumulates under composition, yielding a systematic depth amplification effect.
Read the full abstract ▾
Machine Learning
Mean Flow Policy Optimization
Xiaoyi Dong, Xi Sheryl Zhang and Jian Cheng
Diffusion models have recently emerged as expressive policy representations for online reinforcement learning (RL). However, their iterative generative processes introduce substantial training and inference overhead. To overcome this limitation, we propose to represent policies using MeanFlow models, a class of few-step flow-based generative models, to improve training and inference efficiency over
diffusion-based RL approaches. To promote exploration, we optimize MeanFlow policies under the maximum
entropy RL framework via soft policy iteration, and address two key challenges specific to MeanFlow policies: action likelihood evaluation and soft policy improvement. Experiments on MuJoCo, DeepMind Control Suite and HumanoidBench
benchmarks demonstrate that our method, Mean Flow Policy Optimization (MFPO), achieves performance comparable to or exceeding current
diffusion-based baselines while considerably reducing training and inference time. Our code is available at https://github.com/dongxiaoyi-xyz/MFPO.
Read the full abstract ▾
Machine Learning
Zeroth-Order Optimization at the Edge of Stability
Minhak Song, Liang Zhang, Bingcong Li et al. (+3 more)
Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and
memory-efficient fine-tuning of large models, yet their optimization dynamics in deep learning remain underexplored. In this work, we provide an explicit step size condition that exactly captures the (mean-square) linear stability of a family of ZO methods based on the standard two-point
estimator. Our characterization reveals a sharp contrast with first-order (FO) methods: whereas FO stability is governed solely by the largest Hessian
eigenvalue, mean-square stability of ZO methods depends on the entire Hessian spectrum. Since computing the full Hessian spectrum is infeasible in practical neural network training, we further derive tractable stability bounds that depend only on the largest
eigenvalue and the Hessian trace. Empirically, we find that full-batch ZO methods operate at the edge of stability: ZO-GD, ZO-GDM, and ZO-Adam consistently stabilize near the predicted stability boundary across a range of deep learning training problems. Our results highlight an implicit regularization effect specific to ZO methods, where large step sizes primarily regularize the Hessian trace, whereas in FO methods they regularize the top
eigenvalue.
Read the full abstract ▾
All authors Minhak Song, Liang Zhang, Bingcong Li, Niao He, Michael Muehlebach, Sewoong Oh
Machine Learning · Cross-listed from cs.SE
AIPC: Agent-Based Automation for AI Model Deployment with Qualcomm AI Runtime
Jianhao Su, Zhanwei Wu, ShengTing Huang et al. (+1 more)
Edge AI model deployment is a multi-stage engineering process involving model conversion, operator compatibility handling, quantization calibration, runtime integration, and accuracy validation. In practice, this workflow is long, failure-prone, and heavily dependent on deployment expertise, particularly when targeting hardware-specific inference runtimes. This technical report presents AIPC (AI Porting Conversion), an AI
agent-driven approach for constrained automation of AI model deployment. AIPC decomposes deployment into standardized, verifiable stages and injects deployment-domain knowledge into
agent execution through
Agent Skills, helper scripts, and a stage-wise validation loop. This design reduces both the expertise barrier and the engineering time required for hardware deployment. Using Qualcomm AI Runtime (QAIRT) as the primary scenario, this report examines automated deployment across representative
vision,
multimodal, and
speech models. In the cases covered here, AIPC can complete deployment from PyTorch to runnable QNN/SNPE inference within 7-20 minutes for structurally regular
vision models, with indicative API costs roughly in the range of USD 0.7-10. For more complex models involving less-supported operators, dynamic shapes, or autoregressive decoding structures, fully automated deployment may still require further advances, but AIPC already provides practical support for execution, failure localization, and bounded repair.
Read the full abstract ▾
All authors Jianhao Su, Zhanwei Wu, ShengTing Huang, Weidong Feng
Machine Learning · Cross-listed from cs.AI
AgentGA: Evolving Code Solutions in Agent-Seed Space
David Y.Y. Tan, Kellie Chin and Jingxian Zhang
We present AgentGA, a framework that evolves autonomous code-generation runs by optimizing the
agent seed: the task
prompt plus optional parent archives that initialize a fresh workspace. The outer loop searches over these reusable starting conditions rather than editing code directly. Each generation launches a fresh autonomous run in an isolated workspace, while selected parent archives provide inherited artifacts that descendants can inspect and reuse. AgentGA couples a population-level genetic
algorithm with long-horizon
agents; selection uses deterministic 1:1 elite tournaments and operator allocation is adapted online with a modified Hedge controller. We instantiate the approach for tabular AutoML on the 16-competition Weco-Kaggle Lite
benchmark. Across the full
benchmark, AgentGA averages 71.90% Exceeds % of Human versus 51.38% for the AIDE reference, winning 15/16 competitions. Within AgentGA runs, descendants conditioned on inherited parent archives win 51.9% of 1,680 parent-child tournaments versus 8.6% for de novo proposals. These results support
agent-seed optimization as a practical design choice for autonomous code-search systems.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Seen-to-Scene: Keep the Seen, Generate the Unseen for Video Outpainting
Inseok Jeon, Minhyeok Lee, Seunghoon Lee et al. (+3 more)
Video outpainting aims to expand the visible content of a
video beyond the original frame boundaries while preserving spatial fidelity and temporal coherence across frames. Existing methods primarily rely on large-scale generative models, such as
diffusion models. However, generationbased approaches suffer from implicit temporal modeling and limited spatial context. These limitations lead to intraframe and inter-frame inconsistencies, which become particularly pronounced in dynamic scenes and large outpainting scenarios. To overcome these challenges, we propose Seen-to-Scene, a novel framework that unifies propagationbased and generation-based paradigms for
video outpainting. Specifically, Seen-to-Scene leverages flow-based propagation with a flow completion network pre-trained for
video inpainting, which is fine-tuned in an end-to-end manner to bridge the domain gap and reconstruct coherent motion fields. To further improve the efficiency and reliability of propagation, we introduce a reference-guided latent propagation that effectively propagates source content across frames. Extensive experiments demonstrate that our method achieves superior temporal coherence and visual realism with efficient inference, surpassing even prior state-of-the-art methods that require input-specific adaptation.
Read the full abstract ▾
All authors Inseok Jeon, Minhyeok Lee, Seunghoon Lee, Minseok Kang, Suhwan Cho, Sangyoun Lee
Machine Learning · Cross-listed from cs.CL
CURaTE: Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge
Seyun Bae, Seokhan Lee and Eunho Yang
The inability to filter out in advance all potentially problematic data from the pre-training of large
language models has given rise to the need for methods for unlearning specific pieces of knowledge after training. Existing techniques overlook the need for continuous and immediate action, causing them to suffer from degraded utility as updates accumulate and protracted exposure of sensitive information. To address these issues, we propose Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge (CURaTE). Our method begins by training a sentence
embedding model on a
dataset designed to enable the formation of sharp decision boundaries for determining whether a given input
prompt corresponds to any stored forget requests. The similarity of a given input to the forget requests is then used to determine whether to answer or return a refusal response. We show that even with such a simple approach, not only does CURaTE achieve more effective forgetting than existing methods, but by avoiding modification of the
language model parameters, it also maintains near perfect knowledge preservation over any number of updates and is the only method capable of continual unlearning in real-time.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Physically-Induced Atmospheric Adversarial Perturbations: Enhancing Transferability and Robustness in Remote Sensing Image Classification
Weiwei Zhuang, Wangze Xie, Qi Zhang et al. (+9 more)
Adversarial attacks pose a severe threat to the reliability of deep learning models in remote sensing (RS) image classification. Most existing methods rely on direct pixel-wise perturbations, failing to exploit the inherent atmospheric characteristics of RS imagery or survive real-world image degradations. In this paper, we propose FogFool, a physically plausible
adversarial framework that generates fog-based perturbations by iteratively optimizing atmospheric patterns based on Perlin noise. By modeling fog formations with natural, irregular structures, FogFool generates
adversarial examples that are not only visually consistent with authentic RS scenes but also deceptive. By leveraging the spatial coherence and mid-to-low-frequency nature of atmospheric phenomena, FogFool embeds
adversarial information into structural features shared across diverse architectures. Extensive experiments on two
benchmark RS
datasets demonstrate that FogFool achieves superior performance: not only does it exceed in white-box settings, but also exhibits exceptional black-box transferability (reaching 83.74% TASR) and robustness against common preprocessing-based defenses such as JPEG compression and filtering. Detailed analyses, including confusion
matrices and Class Activation Map (CAM) visualizations, reveal that our atmospheric-driven perturbations induce a universal shift in model
attention. These results indicate that FogFool represents a practical, stealthy, and highly persistent threat to RS classification systems, providing a robust
benchmark for evaluating model reliability in complex environments.
Read the full abstract ▾
All authors Weiwei Zhuang, Wangze Xie, Qi Zhang, Xia Du, Zihan Lin, Zheng Lin, Hanlin Cai, Jizhe Zhou, Zihan Fang, Chi-man Pun, Wei Ni, Jun Luo
Machine Learning · Cross-listed from cs.CV
CMTM: Cross-Modal Token Modulation for Unsupervised Video Object Segmentation
Inseok Jeon, Suhwan Cho, Minhyeok Lee et al. (+6 more)
Recent advances in unsupervised
video object
segmentation have highlighted the potential of two-stream architectures that integrate appearance and motion cues. However, fully leveraging these complementary sources of information requires effectively modeling their interdependencies. In this paper, we introduce cross-modality
token modulation, a novel approach designed to strengthen the interaction between appearance and motion cues. Our method establishes dense connections between
tokens from each modality, enabling efficient intra-modal and inter-modal information propagation through relation
transformer blocks. To improve learning efficiency, we incorporate a
token masking strategy that addresses the limitations of relying solely on increased model complexity. Our approach achieves state-of-the-art performance across all public
benchmarks, outperforming existing methods.
Read the full abstract ▾
All authors Inseok Jeon, Suhwan Cho, Minhyeok Lee, Seunghoon Lee, Minseok Kang, Jungho Lee, Chaewon Park, Donghyeong Kim, Sangyoun Lee
Machine Learning
ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving
Yuseon Choi, Jingu Lee, Jungjun Oh et al. (+6 more)
Mixture-of-Experts (MoE) models have become the dominant architecture for large-scale
language models, yet on-premises serving remains fundamentally
memory-bound as batching turns sparse per-
token compute into dense
memory activation.
Memory-centric architectures (PIM, NMP) improve bandwidth but leave compute underutilized under MoE's low arithmetic intensity on high batch sizes. Speculative decoding (SD) trades idle compute for fewer target invocations, yet verification must load experts even for rejected
tokens, severely limiting its benefit in MoE especially on low batch sizes. We propose ELMoE-3D, a hybrid-bonding (HB)-based HW-SW co-designed framework that unifies cache-based acceleration and speculative decoding to offer overall speedup across batch sizes. We identify two intrinsic elasticity axes of MoE—expert and bit—and jointly scale them to construct Elastic Self-Speculative Decoding (Elastic-SD), which serves as both an expert cache and a strongly aligned self-draft model accelerated by high HB bandwidth. Our LSB-augmented bit-sliced architecture exploits inherent redundancy in bit-slice representations to natively support bit-nested execution. On our 3D-stacked hardware, ELMoE-3D achieves an average $6.6\times$ speedup and $4.4\times$ energy efficiency gain over naive MoE serving on xPU across batch sizes 1–16, and delivers $2.2\times$ speedup and $1.4\times$ energy efficiency gain over prior accelerator baselines.
Read the full abstract ▾
All authors Yuseon Choi, Jingu Lee, Jungjun Oh, Sunjoo Whang, Byeongcheol Kim, Minsung Kim, Junhyuk Lee, Hoi-Jun Yoo, Sangjin Kim