Machine Learning
Concept Heterogeneity-aware Representation Steering
Laziz U. Abdullaev, Noelle Y. L. Wong, Ryan T. Z. Lee et al. (+3 more)
Representation steering offers a lightweight mechanism for controlling the behavior of large
language models (LLMs) by intervening on internal activations at inference time. Most existing methods rely on a single global steering direction, typically obtained via difference-in-means over contrastive
datasets. This approach implicitly assumes that the target concept is homogeneously represented across the
embedding space. In practice, however, LLM representations can be highly non-homogeneous, exhibiting clustered, context-dependent structure, which renders global steering directions brittle. In this work, we view representation steering through the lens of optimal transport (OT), noting that standard difference-in-means steering implicitly corresponds to the OT map between two identical distributions with differing first moments, yielding a global translation. To relax this restrictive assumption, we theoretically model source and target representations as Gaussian mixture models and formulate steering as a discrete OT problem between semantic latent clusters. From the resulting transport plan, we derive an explicit, input-dependent steering map via barycentric projection, producing a smooth, kernel-weighted combination of cluster-level shifts. We term this method Concept Heterogeneity-aware Representation Steering (CHaRS). Through numerous experimental settings, we show that CHaRS yields more effective behavioral control than global steering.
Read the full abstract ▾
All authors Laziz U. Abdullaev, Noelle Y. L. Wong, Ryan T. Z. Lee, Shiqi Jiang, Khoi N. M. Nguyen, Tan M. Nguyen
Machine Learning
CUDABench: Benchmarking LLMs for Text-to-CUDA Generation
Jiace Zhu, Wentao Chen, Qi Fan et al. (+6 more)
Recent studies have demonstrated the potential of Large
Language Models (LLMs) in generating GPU Kernels. Current
benchmarks focus on the translation of high-level
languages into CUDA, overlooking the more general and challenging task of text-to-CUDA generation. Furthermore, given the hardware-specific and performance-critical features of GPU programming, accurately assessing the performance of LLM-generated GPU programs is nontrivial. In this work, we introduce CUDABench, a comprehensive
benchmark designed to evaluate the text-to-CUDA capabilities of LLMs. First, we construct CUDABench-Set, which covers Breadth-Depth-Difficulty evaluation space in diverse application domains, including artificial intelligence, scientific computing, and data analytics, etc. Furthermore, we propose CUDABench-Score and Generative Verification Pipeline that assess (1) compilation correctness, (2) functional consistency through execution-based verification, and (3) a novel roofline-based metric, Performance-Score. Benchmarking state-of-the-art LLMs reveals insightful findings and challenges of text-to-CUDA, such as a notable mismatch between high compilation success rates and low functional correctness, a lack of domain-specific algorithmic knowledge, and suboptimal utilization of GPU hardware resources. Our
benchmark is available at https://github.com/CUDA-Bench/CUDABench.
Read the full abstract ▾
All authors Jiace Zhu, Wentao Chen, Qi Fan, Zhixing Ren, Junying Wu, Xing Zhe Chai, Chotiwit Rungrueangwutthinon, Yehan Ma, An Zou
Machine Learning
Talking with Verifiers: Automatic Specification Generation for Neural Network Verification
Yizhak Y. Elboher, Reuven Peleg, Zhouxing Shi et al. (+2 more)
Neural network verification tools currently support only a narrow class of specifications, typically expressed as low-level constraints over raw inputs and outputs. This limitation significantly hinders their adoption and practical applicability across diverse application domains where correctness requirements are naturally expressed at a higher semantic level. This challenge is rooted in the inherent nature of deep neural networks, which learn internal representations that lack an explicit mapping to human-understandable features. To address this, we bridge this gap by introducing a novel component to the verification pipeline, making existing verification tools applicable to a broader range of domains and specification styles. Our framework enables users to formulate specifications in natural
language, which are then automatically analyzed and translated into formal verification queries compatible with state-of-the-art neural network verifiers. We evaluate our approach on both structured and unstructured
datasets, demonstrating that it successfully verifies complex semantic specifications that were previously inaccessible. Our results show that this translation process maintains high fidelity to user intent while incurring low computational overhead, thereby substantially extending the applicability of formal DNN verification to real-world, high-level requirements.
Read the full abstract ▾
All authors Yizhak Y. Elboher, Reuven Peleg, Zhouxing Shi, Guy Katz, Jan Křetínský
Machine Learning
Structured vs. Unstructured Pruning: An Exponential Gap
Davide Ferre', Frédéric Giroire, Frederik Mallmann-Trenn et al. (+1 more)
The Strong Lottery Ticket Hypothesis (SLTH) states that large, randomly initialized neural networks contain sparse subnetworks capable of approximating a target function at initialization without training, suggesting that
pruning alone is sufficient.
Pruning methods are typically classified as unstructured, where individual weights can be removed from the network, and structured, where parameters are removed according to specific patterns, as in
neuron pruning. Existing theoretical results supporting the SLTH rely almost exclusively on unstructured
pruning, showing that logarithmic overparameterization suffices to approximate simple target networks. In contrast,
neuron pruning has received limited theoretical
attention, despite its practical appeal for direct hardware speedups. In this work, we consider the problem of approximating a single bias-free ReLU
neuron by
pruning hidden units of a randomly initialized two-layer ReLU network, effectively isolating the intrinsic limitations of
neuron pruning. We show that achieving an $\varepsilon$-approximation requires a starting network size of $\Omega(1/\varepsilon)$ for
neuron pruning, whereas weight
pruning succeeds with only $O(\log(1/\varepsilon))$ hidden units, revealing an exponential separation between the two approaches.
Read the full abstract ▾
All authors Davide Ferre', Frédéric Giroire, Frederik Mallmann-Trenn, Emanuele Natale
Machine Learning
Adaptive Personalized Federated Learning via Multi-task Averaging of Kernel Mean Embeddings
Jean-Baptiste Fermanian, Batiste Le Bars and Aurélien Bellet
Personalized
Federated Learning (PFL) enables a collection of
agents to collaboratively learn individual models without sharing raw data. We propose a new PFL approach in which each
agent optimizes a weighted combination of all
agents' empirical risks, with the weights learned from data rather than specified a priori. The novelty of our method lies in formulating the estimation of these collaborative weights as a kernel mean
embedding estimation problem with multiple data sources, leveraging tools from multi-task averaging to capture statistical relationships between
agents. This perspective yields a fully adaptive procedure that requires no prior knowledge of data heterogeneity and can automatically transition between global and local learning regimes. By recasting the objective as a high-dimensional mean estimation problem, we derive finite-sample guarantees on local excess risks for a broad class of distributions, explicitly quantifying the statistical gains of collaboration. To address communication constraints inherent to
federated settings, we also propose a practical implementation based on random
Fourier features, which allows one to trade communication cost for statistical efficiency. Numerical experiments validate our theoretical results.
Read the full abstract ▾
Machine Learning
Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
Amirhossein Afsharrad, Ruida Zhou, Luca Viano et al. (+2 more)
Reward modeling is crucial for aligning large
language models with human preferences, yet current approaches lack a principled mathematical framework for leveraging ordinal preference data. When human annotators provide graded preferences on a Likert scale (e.g., significantly better, better, slightly better, negligibly better), existing methods typically apply ad-hoc heuristics, such as margin terms or scaling factors, to loss functions derived from binary preference models like Bradley-Terry. These approaches lack an underlying mathematical model for how ordinal preference data is generated. We present a theoretically grounded framework that formulates reward modeling with Likert scale preferences as a discrete ordinal regression problem. We derive two loss functions from this formulation: a negative log-likelihood loss and an all-threshold loss, both of which learn threshold parameters that naturally capture the ordinal structure of preferences. Unlike existing heuristic methods that manually specify fixed margins or scaling weights, our approach learns these parameters directly from data within a coherent probabilistic framework. Experimental results on multiple
benchmarks demonstrate that our ordinal regression approach consistently achieves competitive or superior performance compared to existing heuristic methods across diverse evaluation categories including chat,
reasoning, and safety tasks. Our work provides the first principled mathematical framework for incorporating Likert scale preferences into reward model training, moving beyond ad-hoc modifications of binary preference models to enable more effective utilization of fine-grained human feedback.
Read the full abstract ▾
All authors Amirhossein Afsharrad, Ruida Zhou, Luca Viano, Sanjay Lall, Mohammad Ghavamzadeh
Machine Learning
Physics-Informed Neural Networks with Architectural Physics Embedding for Large-Scale Wave Field Reconstruction
Huiwen Zhang, Feng Ye and Chu Ma
Large-scale wave field reconstruction requires precise solutions but faces challenges with computational efficiency and accuracy. The physics-based numerical methods like Finite Element Method (FEM) provide high accuracy but struggle with large-scale or high-frequency problems due to prohibitive computational costs. Pure data-driven approaches excel in speed but often lack sufficient labeled data for complex scenarios. Physics-informed neural networks (PINNs) integrate physical principles into machine learning models, offering a promising solution by bridging these gaps. However, standard PINNs embed physical principles only in loss functions, leading to slow convergence, optimization instability, and spectral bias, limiting their ability for large-scale wave field reconstruction. This work introduces architectural physics embedded (PE)-PINN, which integrates additional physical guidance directly into the neural network architecture beyond Helmholtz equations and boundary conditions in loss functions. Specifically, a new envelope transformation layer is designed to mitigate spectral bias with kernels parameterized by source properties, material interfaces, and wave physics. Experiments demonstrate that PE-PINN achieves more than 10 times speedup in convergence compared to standard PINNs and several orders of magnitude reduction in
memory usage compared to FEM. This breakthrough enables high-fidelity modeling for large-scale 2D/3D electromagnetic wave reconstruction involving reflections, refractions, and diffractions in room-scale domains, readily applicable to wireless communications, sensing, room acoustics, and other fields requiring large-scale wave field analysis.
Read the full abstract ▾
Machine Learning
Generalized Discrete Diffusion with Self-Correction
Linxuan Wang, Ziyi Wang, Yikun Bai et al. (+3 more)
Self-correction is an effective technique for maintaining parallel sampling in discrete
diffusion models with minimal performance degradation. Prior work has explored self-correction at inference time or during post-training; however, such approaches often suffer from limited generalization and may impair
reasoning performance. GIDD pioneers pretraining-based self-correction via a multi-step BERT-style uniform-absorbing objective. However, GIDD relies on a continuous interpolation-based pipeline with opaque interactions between uniform transitions and absorbing masks, which complicates hyperparameter tuning and hinders practical performance. In this work, we propose a Self-Correcting Discrete
Diffusion (SCDD) model to reformulate pretrained self-correction with explicit state transitions and learn directly in discrete time. Our framework also simplifies the training noise schedule, eliminates a redundant remasking step, and relies exclusively on uniform transitions to learn self-correction. Experiments at the GPT-2 scale demonstrate that our method enables more efficient parallel decoding while preserving generation quality.
Read the full abstract ▾
All authors Linxuan Wang, Ziyi Wang, Yikun Bai, Wei Deng, Guang Lin, Qifan Song
Machine Learning
Safety Training May Persist Through Helpfulness Optimization in LLM Agents
Benjamin Plaut
Safety post-training has been studied extensively in single-step "chat" settings where safety typically refers to refusing harmful requests. We study an "agentic" (i.e., multi-step, tool-use) setting where safety refers to harmful actions directly taken by the LLM. We investigate the effects of using direct preference optimization (DPO) to optimize safety and/or helpfulness on the ToolEmu agentic
benchmark. First, we find that safety training largely persists through subsequent helpfulness training. Second, we find a consistent negative linear correlation ($R^2 = 0.77$) between safety and helpfulness when considering all training configurations together. Even post-training on both metrics simultaneously simply results in another point on the same trend line rather than yielding a "best of both worlds" strategy, despite the presence of such strategies in our
dataset. Overall, our findings underscore the need for a better understanding of post-training.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
Data-Driven Deep MIMO Detection:Network Architectures and Generalization Analysis
Yongwei Yi, Xinping Yi, Wenjin Wang et al. (+2 more)
In practical Multiuser Multiple-Input Multiple-Output (MU-
MIMO) systems, symbol detection remains challenging due to severe inter-user interference and sensitivity to Channel State Information (CSI) uncertainty. In contrast to the mostly studied belief propagation-type model-driven methods, which incur high computational complexity, Soft Interference Cancellation (SIC) strikes a good balance between performance and complexity. To further address CSI mismatch and nonlinear effects, the recently proposed data-driven deep neural receivers, such as DeepSIC, leverage the advantages of deep neural networks for interference cancellation and symbol detection, demonstrating strong empirical performance. However, there is still a lack of theoretical underpinning for why and to what extent DeepSIC could generalize with the number of training samples. This paper proposes inspecting the fully data-driven DeepSIC detection within a Network-of-MLPs architecture, which is composed of multiple interconnected MLPs via outer and inner Directed Acyclic Graphs (DAGs). Within such an architecture, DeepSIC can be upgraded as a graph-based message-passing process using Graph Neural Networks (GNNs), termed GNNSIC, with shared model parameters across users and iterations. Notably, GNNSIC achieves excellent expressivity comparable to DeepSIC with substantially fewer trainable parameters, resulting in improved sample efficiency and enhanced user generalization. By conducting a norm-based generalization analysis using Rademacher complexity, we reveal that an exponential dependence on the number of iterations for DeepSIC can be eliminated in GNNSIC due to parameter sharing. Simulation results demonstrate that GNNSIC attains comparable or improved Symbol Error Rate (SER) performance to DeepSIC with significantly fewer parameters and training samples.
Read the full abstract ▾
All authors Yongwei Yi, Xinping Yi, Wenjin Wang, Xiao Li, Shi Jin
Machine Learning · Cross-listed from cs.NE
Enhancing Heat Sink Efficiency in MOSFETs using Physics Informed Neural Networks: A Systematic Study on Coolant Velocity Estimation
Aniruddha Bora, Isabel K. Alvarez, Julie Chalfant et al. (+1 more)
In this work, we present a methodology using Physics Informed Neural Networks (PINNs) to determine the required velocity of a coolant, given inlet and outlet temperatures for a given heat flux in a multilayered metal-
oxide-
semiconductor field-effect transistor (MOSFET). MOSFETs are integral components of Power Electronic Building Blocks (PEBBs) and experiences the majority of the thermal load. Effective cooling of MOSFETs is therefore essential to prevent overheating and potential burnout. Determining the required velocity for the purpose of effective cooling is of importance but is an ill-posed inverse problem and difficult to solve using traditional methods. MOSFET consists of multiple layers with different thermal conductivities, including aluminum, pyrolytic graphite sheets (PGS), and stainless steel pipes containing flowing water. We propose an
algorithm that employs sequential training of the MOSFET layers in PINNs. Mathematically, the sequential training method decouples the optimization of each layer by treating the parameters of other layers as constants during its training phase. This reduces the dimensionality of the optimization landscape, making it easier to find the global minimum for each layer's parameters and avoid poor local minima. Convergence of the PINNs solution to the analytical solution is theoretically analyzed. Finally we show the prediction of our proposed methodology to be in good agreement with experimental results.
Read the full abstract ▾
All authors Aniruddha Bora, Isabel K. Alvarez, Julie Chalfant, Chryssostomos Chryssostomidis
Machine Learning · Cross-listed from q-bio.BM
Cross-Chirality Generalization by Axial Vectors for Hetero-Chiral Protein-Peptide Interaction Design
Ziyi Yang, Zitong Tian, Yinjun Jia et al. (+7 more)
D-peptide binders targeting L-
proteins have promising therapeutic potential. Despite rapid advances in machine learning-based target-conditioned peptide design, generating D-peptide binders remains largely unexplored. In this work, we show that by injecting axial features to $E(3)$-equivariant (polar) vector features, it is feasible to achieve cross-chirality generalization from homo-
chiral (L–L) training data to hetero-
chiral (D–L) design tasks. By implementing this method within a latent
diffusion model, we achieved D-peptide binder design that not only outperforms existing tools in in silico
benchmarks, but also demonstrates efficacy in wet-lab validation. To our knowledge, our approach represents the first wet-lab validated generative AI for the de novo design of D-peptide binders, offering new perspectives on handling chirality in
protein design. Codes are available at https://github.com/YZY010418/PepMirror
Read the full abstract ▾
All authors Ziyi Yang, Zitong Tian, Yinjun Jia, Tianyi Zhang, Jiqing Zheng, Hao Wang, Yubu Su, Juncai He, Lei Liu, Yanyan Lan
Machine Learning
Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning
Kehao Zhang, Shangtong Gui, Sheng Yang et al. (+2 more)
Long-context LLMs and
Retrieval-Augmented Generation defer state tracking and evidence consolidation to query time, which is brittle when facts evolve and answers depend on latent states. We introduce Unified
Memory Agent (UMA) for a one-to-many setting: query-agnostic external
memory is constructed once from a stream and reused across multiple future QA sessions. A single policy maintains a structured
Memory Bank through CRUD operations and answers using both the
Memory Bank and raw context. Task-Stratified GRPO uses the mean reward of QA trajectories branching from each sampled
memory state to supervise
memory maintenance, while normalizing
memory and per-question QA groups separately. We also introduce Ledger-QA, a diagnostic
benchmark for long-horizon state tracking over accumulated updates. At the 16k budget, UMA-Generalist achieves the highest average score among compared methods across the test-time-learning and accurate-
retrieval benchmarks and transfers to Ledger-QA without task-specific training; UMA-Specialist further improves long-horizon tracking after task adaptation. These results support learned proactive
memory management for long-context
reasoning.
Read the full abstract ▾
All authors Kehao Zhang, Shangtong Gui, Sheng Yang, Wei Chen, Yang Feng
Machine Learning · Cross-listed from cs.SD
Speech to Speech Synthesis for Voice Impersonation
Bjorn Johnson and Jared Levy
Numerous models have shown great success in the fields of
speech recognition as well as
speech synthesis, but models for
speech to
speech processing have not been heavily explored. We propose
Speech to
Speech Synthesis Network (STSSN), a model based on current state of the art systems that fuses the two disciplines in order to perform effective
speech to
speech style transfer for the purpose of voice impersonation. We show that our proposed model is quite powerful, and succeeds in generating realistic audio samples despite a number of drawbacks in its capacity. We
benchmark our proposed model by comparing it with a generative
adversarial model which accomplishes a similar task, and show that ours produces more convincing results.
Read the full abstract ▾
Machine Learning
Singular Vectors of Attention Heads Align with Features
Gabriel Franco, Carson Loughridge and Mark Crovella
Identifying feature representations in
language models is a central task in mechanistic interpretability. Several recent studies have made the observation that feature representations can be inferred in some cases from singular vectors of
attention matrices. However, sound justification for this phenomenon is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we demonstrate that singular vectors robustly align with features in a model where features can be directly observed. We then show theoretically that such
alignment is expected under a range of conditions. We close by asking how, operationally,
alignment may be recognized in real models where feature representations are not directly observable. We identify sparse
attention decomposition as a testable prediction of
alignment, and show evidence that it emerges in real models in a manner consistent with predictions. Together these results suggest that
alignment of singular vectors with features can be a sound and theoretically justified basis for feature identification in
language models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning
Jintao Zhang, Kai Jiang, Chendong Xiang et al. (+5 more)
Many training-free sparse
attention methods are effective for accelerating
diffusion models. Recently, several works suggest that making sparse
attention trainable can further increase sparsity while preserving generation quality. We study three key questions: (1) when do the two common masking rules, i.e., Top-k and Top-p, fail, and how can we avoid these failures? (2) why can trainable sparse
attention reach higher sparsity than training-free methods? (3) what are the limitations of fine-tuning sparse
attention using the
diffusion loss, and how can we address them? Based on this analysis, we propose SpargeAttention2, a trainable sparse
attention method that achieves high sparsity without degrading generation quality. SpargeAttention2 includes (i) a hybrid masking rule that combines Top-k and Top-p for more robust masking at high sparsity, (ii) an efficient trainable sparse
attention implementation, and (iii) a
distillation-inspired fine-tuning objective to better preserve generation quality during fine-tuning using sparse
attention. Experiments on
video diffusion models show that SpargeAttention2 reaches 95%
attention sparsity and a 16.2x
attention speedup while maintaining generation quality, consistently outperforming prior sparse
attention methods.
Read the full abstract ▾
All authors Jintao Zhang, Kai Jiang, Chendong Xiang, Weiqi Feng, Yuezhou Hu, Haocheng Xi, Jianfei Chen, Jun Zhu
Machine Learning · Cross-listed from math.OC
Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization
Grant Norman, Conor Rowan, Kurt Maute et al. (+1 more)
In this work, we investigate the use of data-driven equation discovery for dynamical systems to model and forecast continuous-time dynamics of unconstrained optimization problems. To avoid expensive evaluations of the objective function and its gradient, we leverage trajectory data on the optimization variables to learn the continuous-time dynamics associated with gradient descent, Newton's method, and ADAM optimization. The discovered gradient flows are then solved as a surrogate for the original optimization problem. To this end, we introduce the Learned Gradient Flow (LGF) optimizer, which is equipped to build surrogate models of variable
polynomial order in full- or reduced-dimensional spaces at user-defined intervals in the optimization process. We demonstrate the efficacy of this approach on several standard problems from engineering mechanics and scientific machine learning, including two inverse problems, structural
topology optimization, and two forward solves with different discretizations. Our results suggest that the learned gradient flows can significantly expedite convergence by capturing critical features of the optimization trajectory while avoiding expensive evaluations of the objective and its gradient.
Read the full abstract ▾
All authors Grant Norman, Conor Rowan, Kurt Maute, Alireza Doostan
Machine Learning · Cross-listed from math.OC
Stochastic variance reduced extragradient methods for solving hierarchical variational inequalities
Pavel Dvurechensky, Andrea Ebner, Johannes Carl Schnebel et al. (+2 more)
We are concerned with optimization in a broad sense through the lens of solving variational inequalities (VIs) – a class of problems that are so general that they cover as particular cases minimization of functions, saddle-point (minimax) problems, Nash
equilibrium problems, and many others. The key challenges in our problem formulation are the two-level hierarchical structure and finite-sum representation of the smooth operators in each level. For this setting, we are the first to prove convergence rates and complexity statements for variance-reduced
stochastic algorithms approaching the solution of hierarchical VIs in Euclidean and Bregman setups.
Read the full abstract ▾
All authors Pavel Dvurechensky, Andrea Ebner, Johannes Carl Schnebel, Shimrit Shtern, Mathias Staudigl
Machine Learning
$\gamma$-weakly $\theta$-up-concavity: A Unified Framework for Non-Convex Optimization Beyond DR-Submodular and OSS Functions
Mohammad Pedramfar and Vaneet Aggarwal
Many objectives in machine learning and combinatorial optimization are non-concave. We introduce $\gamma$-weak $\theta$-up-concavity, a first-order condition for nonnegative monotone functions in which a scaling function $\theta$ lets the
curvature change with the size of the solution. The resulting class strictly contains monotone DR-submodular functions and One-Sided Smooth (OSS) functions. It also contains functions whose marginal gains first increase and then decrease, and functions that vanish faster than any power of the norm at the origin. We construct a surrogate gradient from gradients along the segment from the origin to the current point and prove an upper-linearization inequality with a point-dependent approximation coefficient. Restricting actions to the closed convex hull of the coordinate-wise maximal feasible points makes this coefficient uniformly positive, with a value determined by the
curvature parameters and the geometry of the feasible set. Standard reductions to linear optimization then give offline approximation
algorithms and online
algorithms with regret guarantees. The coefficient matches the optimal one for DR-submodular maximization and improves the best known coefficients for OSS maximization over matroid constraints whenever the OSS parameter is positive.
Read the full abstract ▾
Machine Learning
TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers
Peng Cheng, Jiucheng Zang, Qingnan Li et al. (+6 more)
Muon-style optimizers leverage Newton-Schulz (NS) iterations to orthogonalize updates, yielding update geometries that often outperform Adam-series methods. However, this orthogonalization discards magnitude information, rendering training sensitive to step-size hyperparameters and vulnerable to high-energy bursts. To mitigate this, we introduce TrasMuon (Trust Region Adaptive Scaling
Muon). TrasMuon preserves the near-isometric geometry of
Muon while stabilizing magnitudes through (i) global RMS calibration and (ii) energy-based trust-region clipping. We demonstrate that while reintroducing adaptive scaling improves optimization efficiency, it typically exacerbates instability due to high-energy outliers. TrasMuon addresses this by defining a trust region based on relative energy ratios, confining updates to a stable zone. Empirical experiments on
vision and
language models demonstrate that TrasMuon converges faster than baselines. Furthermore, experiments without warmup stages confirm TrasMuon's superior stability and robustness.
Read the full abstract ▾
All authors Peng Cheng, Jiucheng Zang, Qingnan Li, Liheng Ma, Yufei Cui, Yingxue Zhang, Boxing Chen, Ming Jian, Wen Tong
Machine Learning
Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity
Fei Wu, Jia Hu, Geyong Min et al. (+1 more)
Federated low-rank adaptation (FedLoRA) has facilitated communication-efficient and
privacy-preserving fine-tuning of foundation models for downstream tasks. In practical
federated learning scenarios, client heterogeneity in system resources and data distributions motivates the use of heterogeneous LoRA ranks across clients. However, we identify a previously overlooked phenomenon in heterogeneous FedLoRA with SVD-based allocation, termed rank collapse, where the energy of the global update becomes concentrated in the minimum shared rank, resulting in suboptimal performance and high sensitivity to rank configurations. Through theoretical analysis, we reveal the underlying mechanism of rank collapse: a mismatch between rank-agnostic aggregation weights and rank-dependent client contributions, which systematically suppresses higher-rank updates at a
geometric rate over rounds. Motivated by this insight, we propose raFLoRA, a rank-partitioned aggregation method that decomposes local updates into rank partitions and then aggregates each partition weighted by its effective client contributions. Extensive experiments across
vision,
language, and
reasoning tasks show that raFLoRA prevents rank collapse, improves model performance, and enhances robustness across diverse heterogeneous configurations compared with strong FedLoRA baselines.
Read the full abstract ▾
All authors Fei Wu, Jia Hu, Geyong Min, Shiqiang Wang
Machine Learning
Federated Learning of Nonlinear Temporal Dynamics with Graph Attention-based Cross-Client Interpretability
Ayse Tursucular, Ayush Mohanty, Nazal Mohamed et al. (+1 more)
Networks of modern industrial systems are increasingly monitored by distributed sensors, where each system comprises multiple subsystems that generate high-dimensional time-series. These subsystems are often interdependent, making it important to understand how temporal patterns observed at one subsystem relate to those observed across others. This problem is particularly challenging in decentralized settings where raw measurements cannot be shared, and client observations are heterogeneous. Moreover, in practice, each subsystem (client) operates a fixed, proprietary model that cannot be modified or retrained, limiting the applicability of existing approaches. Separately, nonlinear dynamics make cross-client temporal interdependencies difficult to characterize, as they are embedded implicitly within nonlinear state-transition functions. This paper presents a vertical
federated learning framework for learning one-step-ahead cross-client temporal interdependencies under these constraints. Each client maps high-dimensional local observations to low-dimensional latent states using a nonlinear state-space model, while a Graph
Attention Network (GAT)-based central server learns a state-transition model over the communicated states. Given a candidate graph skeleton, the server learns state-dependent
attention coefficients that capture cross-client interactions. We analytically relate the GAT
attention coefficients to the Jacobian of the learned state-transition model, yielding a first-order characterization of cross-client interdependencies. We further establish bounded closeness of the learned dynamics and Jacobians to a corresponding centralized oracle. Synthetic experiments validate the theoretical results on controlled systems with known graph skeletons. Experiments on real-world
datasets with unknown graph skeletons demonstrate superior
federated root cause analysis over baselines.
Read the full abstract ▾
All authors Ayse Tursucular, Ayush Mohanty, Nazal Mohamed, Nagi Gebraeel
Machine Learning
Gabriel Franco, Lucas M. Tassis, Azalea Rohr et al. (+1 more)
Understanding the internal
circuits that
language models use to solve tasks remains a central challenge in mechanistic interpretability. A crucial part of finding
circuits is understanding why each
attention head attends where it does. To this end, we introduce ACC++, an improved
circuit-tracing method based on the principle of
attention-causal communication (ACC) [1], which identifies signals, i.e., contents of low dimensional subspaces that cause
attention on a
token pair. ACC++ extracts
circuits from a single forward pass, without replacement models or patching.
Circuits identified by ACC++ consist of components that are causal for the model's
attention decisions, together with the low-dimensional signals used to communicate between them. Here, we first detail the conceptual advances that ACC++ makes over previous work. We then show that across multiple models, a substantial portion of ACC++ signals are interpretable: many signals admit a short natural-
language description. We next present a number of new insights into model behavior obtained via ACC++. First, we use ACC++'s interpretable
circuits to characterize the sensitivity of indirect object identification (IOI)
circuits to
prompt structure. We find that
prompt-specific
circuits form well-defined clusters, and across clusters, heads receive systematically different signals corresponding to distinct mechanisms for identifying the IO name. Next, in
multilingual IOI, ACC++
circuits show that while model components are reused across
languages, signals are often
language-specific. In a four-
language IOI case study, cross-
language circuit distances are consistent with linguistic relatedness. Together, these results show that ACC++ can shed light on a broad spectrum of model behaviors.
Read the full abstract ▾
All authors Gabriel Franco, Lucas M. Tassis, Azalea Rohr, Mark Crovella
Machine Learning
Comparing Classifiers: A Case Study Using PyCM
Sadra Sabouri, Alireza Zolanvari and Sepand Haghighi
Selecting an optimal classification model requires a robust and comprehensive understanding of the performance of the model. This paper provides a tutorial on the PyCM library, demonstrating its utility in conducting deep-dive evaluations of multi-class classifiers. By examining two different case scenarios, we illustrate how the choice of evaluation metrics can fundamentally shift the interpretation of a model's efficacy. Our findings emphasize that a multi-dimensional evaluation framework is essential for uncovering small but important differences in model performance. However, standard metrics may miss these subtle performance trade-offs.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CR
MELT: A Behavioral Trace Dataset for High-Risk Memecoin Launch Detection
Sihao Hu, Selim Furkan Tekin, Yichang Xu et al. (+1 more)
Launchpads have become the dominant mechanism for issuing memecoins, exposing investors to a new class of high-risk launches that existing rug-pull detection methods cannot capture. We argue that detecting these threats requires structured behavioral traces that underlie raw heterogeneous blockchain data, i.e., how insiders accumulate, coordinate, and unwind positions. To enable such analysis, we introduce MELT (MEmecoin Launch Trace, the first behavioral trace
dataset for analyzing and detecting high-risk memecoin launches on Solana. MELT covers 41k+ memecoin launches with 200M+ transactions parsed into typed behavioral records that distinguish swaps, wash trades, transfers, and mints. Beyond per-account behaviors, MELT contributes bundle-trace data that links accounts controlled by the same entity, revealing that, on average, 36.5% of
token supply is held by coordinated accounts, a concealment strategy that disguises the true ownership concentration from unsuspecting buyers. On top of these traces, MELT provides 122 behavioral features and risk-level annotations, enabling supervised learning at a population scale. We
benchmark representative ML models on the high-risk launch detection task. Integrating their predictions into a simple memecoin selection strategy reduces investment loss significantly, demonstrating that behavioral traces can be translated into risk mitigation. Our
dataset and code is available at https://github.com/git-disl/MELT.
Read the full abstract ▾
All authors Sihao Hu, Selim Furkan Tekin, Yichang Xu, Ling Liu
Machine Learning · Cross-listed from cs.RO
AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edge
Noriaki Hirose, Catherine Glossop, Dhruv Shah et al. (+1 more)
Robotic foundation models achieve strong generalization by leveraging internet-scale
vision-
language representations, but their massive computational cost creates a fundamental bottleneck: high inference latency. In dynamic environments, this latency breaks the control loop, rendering powerful models unsafe for real-time deployment. We propose AsyncVLA, an asynchronous control framework that decouples semantic
reasoning from reactive execution. Inspired by hierarchical control, AsyncVLA runs a large foundation model on a remote workstation to provide high-level guidance, while a lightweight, onboard Edge Adapter continuously refines actions at high frequency. To bridge the domain gap between these asynchronous streams, we introduce an end-to-end finetuning protocol and a trajectory re-weighting strategy that prioritizes dynamic interactions. We evaluate our approach on real-world
vision-based
navigation tasks with communication delays up to 6 seconds. AsyncVLA achieves a 40% higher success rate than state-of-the-art baselines, effectively bridging the gap between the semantic intelligence of large models and the reactivity required for edge robotics.
Read the full abstract ▾
All authors Noriaki Hirose, Catherine Glossop, Dhruv Shah, Sergey Levine
Machine Learning · Cross-listed from cs.CL
Benjamin L. Badger
The ability of machine learning models to store input information in hidden layer vector
embeddings, analogous to the concept of `
memory', is widely employed but not well characterized. We find that
language model
embeddings typically contain relatively little input information regardless of data and compute scale during training. In contrast,
embeddings from autoencoders trained for input regeneration are capable of nearly perfect
memory formation. The substitution of
memory embeddings for
token sequences leads to substantial computational efficiencies, motivating the introduction of a parallelizable encoder-decoder
memory model architecture. Upon causal training these models contain information-poor
embeddings incapable of arbitrary information access, but by combining causal and information retention objective functions they learn to form and decode information-rich
memories. Training can be further streamlined by freezing a high fidelity encoder followed by a curriculum training approach where decoders first learn to process
memories and then learn to additionally predict next
tokens. We introduce the perspective that next
token prediction training alone is poorly suited for accurate
memory formation as the objective itself is non-invertible, motivating the use of combined objective functions for models where the entire input is not exposed.
Read the full abstract ▾
Machine Learning · Cross-listed from q-bio.QM
Protect$^*$: Steerable Retrosynthesis through Neuro-Symbolic State Encoding
Shreyas Vinaya Sathyanarayana, Shah Rahil Kirankumar, Sharanabasava D. Hiremath et al. (+1 more)
Large
Language Models (LLMs) have shown remarkable potential in scientific domains like retrosynthesis; yet, they often lack the fine-grained control necessary to navigate complex problem spaces without error. A critical challenge is directing an LLM to avoid specific, chemically sensitive sites on a molecule - a task where unconstrained generation can lead to invalid or undesirable synthetic pathways. In this work, we introduce Protect$^*$, a neuro-symbolic framework that grounds the generative capabilities of Large
Language Models (LLMs) in rigorous chemical logic. Our approach combines automated rule-based
reasoning - using a comprehensive database of 55+ SMARTS patterns and 40+ characterized protecting groups - with the generative intuition of neural models. The system operates via a hybrid architecture: an “automatic mode” where symbolic logic deterministically identifies and guards reactive sites, and a “human-in-the-loop mode” that integrates expert strategic constraints. Through “active state tracking,” we inject hard symbolic constraints into the neural inference process via a dedicated protection state linked to canonical atom maps. We demonstrate this neuro-symbolic approach through case studies on complex natural products, including the discovery of a novel synthetic pathway for Erythromycin B, showing that grounding neural generation in symbolic logic enables reliable, expert-level autonomy.
Read the full abstract ▾
All authors Shreyas Vinaya Sathyanarayana, Shah Rahil Kirankumar, Sharanabasava D. Hiremath, Bharath Ramsundar
Machine Learning
Karish Grover, Hanqing Zeng, Yinglong Xia et al. (+2 more)
Does natural
language text have an intrinsic
curvature?
Language is increasingly modeled in curved geometries -
hyperbolic spaces for hierarchy, mixed-
curvature manifolds for compositional structure - yet a basic scientific question remains unresolved: what does
curvature mean for text itself, in a way that is native to
language rather than an artifact of the
embedding space we choose? We argue that text does indeed have
curvature, and show how to detect it, define it, and use it. To this end, we propose Texture, a text-native, word-level discrete
curvature signal, and make three contributions. (a) Existence: We provide empirical and theoretical certificates that semantic inference in natural corpora is non-flat. (b) Definition: We define Texture as a signed two-axis
curvature of the word-in-context belief field - the differential of reconciliation between prefix and suffix - measuring, via a debiased Schrödinger transport divergence, whether adding context from one side contracts the semantic effect of context from the other side (focus, positive) or expands it into competing continuations (fan-out, negative). (c) Utility: Texture is actionable: it serves as a general-purpose measurement and control primitive enabling geometry without
geometric training; we instantiate it on two representative tasks, improving long-context inference through
curvature-guided compression and
retrieval-augmented generation through
curvature-guided routing. Together, our results establish a text-native
curvature paradigm, making Texture practically useful.
Read the full abstract ▾
All authors Karish Grover, Hanqing Zeng, Yinglong Xia, Christos Faloutsos, Geoffrey J. Gordon
Machine Learning
High-Resolution Climate Projections Using Diffusion-Based Downscaling of a Lightweight Climate Emulator
Haiwen Guan, Dibyajyoti Chakraborty, Moein Darman et al. (+3 more)
The proliferation of data-driven models in weather and
climate sciences has marked a significant paradigm shift, with advanced models demonstrating exceptional skill in medium-range
forecasting. However, these models are often limited by long-term instabilities, climatological drift, and substantial computational costs during training and inference, restricting their broader application for
climate studies. Addressing these limitations, Guan et al. (2024) introduced LUCIE, a lightweight, physically consistent
climate emulator utilizing a Spherical
Fourier Neural Operator (SFNO) architecture. This model is able to reproduce accurate long-term statistics including climatological mean and seasonal variability. However, LUCIE's native resolution (~300 km) is inadequate for detailed regional impact assessments. To overcome this limitation, we introduce a deep learning-based downscaling framework, leveraging probabilistic
diffusion-based generative models with conditional and posterior sampling frameworks. These models downscale coarse LUCIE outputs to 25 km resolution. They are trained on approximately 14,000 ERA5 timesteps spanning 2000-2009 and evaluated on LUCIE predictions from 2010 to 2020. Model performance is assessed through diverse metrics, including latitude-averaged RMSE, power spectrum, probability density functions and First Empirical Orthogonal Function of the zonal wind. We observe that the proposed approach is able to preserve the coarse-grained dynamics from LUCIE while generating fine-scaled climatological statistics at ~28km resolution.
Read the full abstract ▾
All authors Haiwen Guan, Dibyajyoti Chakraborty, Moein Darman, Troy Arcomano, Ashesh Chattopadhyay, Romit Maulik