Machine Learning
Mechanistic Interpretability of Brain-to-Speech Models Across Speech Modes
Maryam Maghsoudi and Ayushi Mishra
Brain-to-
speech decoding models demonstrate robust performance in vocalized, mimed, and imagined
speech; yet, the fundamental mechanisms via which these models capture and transmit information across different
speech modalities are less explored. In this work, we use mechanistic interpretability to causally investigate the internal representations of a neural
speech decoder. We perform cross-mode activation patching of internal activations across
speech modes, and use tri-modal interpolation to examine whether
speech representations vary discretely or continuously. We use coarse-to-fine causal tracing and causal scrubbing to find localized causal structure, allowing us to find internal subspaces that are sufficient for cross-mode transfer. In order to determine how finely distributed these effects are within layers, we perform
neuron-level activation patching. We discover that small but not distributed subsets of
neurons, rather than isolated units, affect the cross-mode transfer. Our results show that
speech modes lie on a shared continuous causal
manifold, and cross-mode transfer is mediated by compact, layer-specific subspaces rather than diffuse activity. Together, our findings give a causal explanation for how
speech modality information is organized and used in
brain-to-
speech decoding models, revealing hierarchical and direction-dependent representational structure across
speech modes.
Read the full abstract ▾
Machine Learning
Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching
Tianhao Miao, Zhongyuan Bao and Lejun Zhang
Training efficiency in large-scale models is typically assessed through
memory consumption, training time, and model performance. Current methods often exhibit trade-offs among these metrics, as optimizing one generally degrades at least one of the others. Addressing this trade-off remains a central challenge in
algorithm design. While GaLore enables
memory-efficient training by updating gradients in a low-rank subspace, it incurs a comparable extra training time cost due to the Singular Value Decomposition(SVD) process on gradients. In this paper, we propose Lotus, a method that resolves this trade-off by simply modifying the projection process. We propose a criterion that quantifies the displacement of the unit gradient to enable efficient transitions between low-rank gradient subspaces. Experimental results indicate that Lotus is the most efficient method, achieving a 30% reduction in training time and a 40% decrease in
memory consumption for gradient and optimizer states. Additionally, it outperforms the baseline method in both pre-training and fine-tuning tasks.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority
Zhanming Shen, Zeyu Qin, Jiaqi Hu et al. (+7 more)
The transition from fitting empirical data to achieving true human utility is fundamentally constrained by a granularity mismatch, where fine-grained autoregressive generation is often supervised by coarse or uniform signals. This position paper advocates
Token Priority as the essential bridge, formalizing Supervised Fine-Tuning (SFT) not as simple optimization but as a precise distribution reshaping process that aligns raw data with the ideal
alignment manifold. We analyze recent breakthroughs through this unified lens, categorizing them into two distinct regimes: Positive Priority for noise filtration and Signed Priority for toxic modes unlearning. We revisit existing progress and limitations, identify key challenges, and suggest directions for future research.
Read the full abstract ▾
All authors Zhanming Shen, Zeyu Qin, Jiaqi Hu, Wentao Ye, Hao Chen, Xiaomeng Hu, Haokai Xu, Gang Chen, Yi R. Fung, Haobo Wang
Machine Learning
Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weights
Qishuai Wen, Zhiyuan Huang, Xianghan Meng et al. (+2 more)
The vanilla self-
attention mechanism in
Transformers can be viewed as a two-layer fast-weight MLP, whose weights are dynamically induced by inputs and whose hidden dimension is equal to the sequence length $N$. As the context extends, the expressive capacity of such an $N$-width MLP increases, but it becomes unscalable for extremely long sequences. Recently, this fast-weight perspective has motivated the Mixture-of-Experts (MoE)
attention mechanism, which partitions the sequence into rigid blocks, treats them as fast-weight experts, and sparsely routes the
tokens to them. In this paper, we elevate this perspective to a unifying framework for efficient
attention mechanisms, interpreting them as making fast weights scalable through either routing or compression, and organizing them into a five-dimensional taxonomy. Then, we propose Mixture-of-Top-$k$
Attention (MiTA), which employs a small set of landmark queries to gather top-$k$ attended key-value pairs as query-aware and deformable routed experts, while compressing the $N$-width MLP into a narrower shared expert. Consequently, our MiTA improves the flexibility of prior MoE
attention from rigid to deformable fast-weight experts, as well as the scalability of prior top-$k$
attention from query-specific set to reusable top-$k$ set. We conduct extensive experiments on
vision tasks showing the superior effectiveness and efficiency of our MiTA, and also uncovering intriguing properties such as an emergent
token-
pruning effect and easy generalization from standard
attention. Code is available at https://github.com/QishuaiWen/MiTA.
Read the full abstract ▾
All authors Qishuai Wen, Zhiyuan Huang, Xianghan Meng, Wei He, Chun-Guang Li
Machine Learning
Learning from Anonymized and Incomplete Tabular Data
Lucas Lange, Adrian Böttinger, Victor Christen et al. (+3 more)
User-driven
privacy allows individuals to control whether and at what granularity their data is shared, leading to
datasets that mix original, generalized, and missing values within the same records and attributes. While such representations are intuitive for
privacy, they pose challenges for machine learning, which typically treats non-original values as new categories or as missing, thereby discarding generalization semantics. For learning from such tabular data, we propose novel data transformation strategies that account for heterogeneous anonymization and evaluate them alongside standard imputation and LLM-based approaches. We employ multiple
datasets,
privacy configurations, and deployment scenarios, demonstrating that our method reliably regains utility. Our results show that generalized values are preferable to pure suppression, that the best data preparation strategy depends on the scenario, and that consistent data representations are crucial for maintaining downstream utility. Overall, our findings highlight that effective learning is tied to the appropriate handling of anonymized values.
Read the full abstract ▾
All authors Lucas Lange, Adrian Böttinger, Victor Christen, Anushka Vidanage, Peter Christen, Erhard Rahm
Machine Learning · Cross-listed from cond-mat.mtrl-sci
AI Meets Plasticity: A Comprehensive Survey
Hadi Bakhshan, Sima Farshbaf, Junior Ramirez Machado et al. (+2 more)
Artificial intelligence (AI) is rapidly emerging as a new paradigm of scientific discovery, namely data-driven science, across nearly all scientific disciplines. In materials science and engineering, AI has already begun to exert a transformative influence, making it both timely and necessary to examine its interaction with materials plasticity. In this study, we present a holistic survey of the convergence between AI and plasticity, highlighting state-of-the-art AI methodologies employed to discover, construct surrogate models for, and emulate the plastic behavior of materials. From a materials science perspective, we examine cause-and-effect relationships governing plastic deformation, including microstructural characterization and macroscopic responses described through plasticity constitutive models. From the perspective of AI methodology, we review a broad spectrum of applied approaches, ranging from frequentist techniques such as classical machine learning (ML), deep learning (DL), and physics-informed models to probabilistic frameworks that incorporate uncertainty quantification and generative AI methods. These data-driven approaches are discussed in the context of materials characterization and plasticity-related applications. The primary objective of this survey is to develop a comprehensive and well-organized taxonomy grounded in AI methodologies, with particular emphasis on distinguishing critical aspects of these techniques, including model architectures, data requirements, and predictive performance within the specific domain of materials plasticity. By doing so, this work aims to provide a clear road map for researchers and practitioners in the materials community, while offering deeper physical insight and intuition into the role of AI in advancing materials plasticity and characterization, an area of growing importance in the emerging AI-driven era.
Read the full abstract ▾
All authors Hadi Bakhshan, Sima Farshbaf, Junior Ramirez Machado, Fernando Rastellini Canela, Josep Maria Carbonell
Machine Learning
SimpleGPT: Improving GPT via A Simple Normalization Strategy
Marco Chen, Xianbiao Qi, Yelin He et al. (+2 more)
In this work, we revisit
Transformer optimization through the lens of second-order geometry and establish a direct connection between architectural design, activation scale, the Hessian
matrix, and the maximum tolerable learning rate. We introduce a simple normalization strategy, termed SimpleNorm, which stabilizes intermediate activation scales by construction. Then, by analyzing the Hessian of the loss with respect to network activations, we theoretically show that SimpleNorm significantly reduces the spectral norm of the Hessian, thereby permitting larger stable learning rates. We validate our theoretical findings through extensive experiments on large GPT models at parameter scales 1B, 1.4B, 7B and 8B. Empirically, SimpleGPT, our SimpleNorm-based network, tolerates learning rates 3$\times$-10$\times$ larger than standard convention, consistently demonstrates strong optimization stability, and achieves substantially better performance than well-established baselines. Specifically, when training 7B-scale models for 60K steps, SimpleGPT achieves a training loss that is 0.08 lower than that of LLaMA2 with QKNorm, reducing the loss from 2.290 to 2.208. Our source code will be released at https://github.com/Ocram7/SimpleGPT.
Read the full abstract ▾
All authors Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao
Machine Learning · Cross-listed from cs.CL
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Zizhuo Fu, Wenxuan Zeng, Runsheng Wang et al. (+1 more)
Large
Language Models (LLMs) often assign disproportionate
attention to the first
token, a phenomenon known as the
attention sink. Several recent approaches aim to address this issue, including Sink
Attention in GPT-OSS and Gated
Attention in Qwen3-Next. However, a comprehensive analysis of the relationship among these
attention mechanisms is lacking. In this work, we provide both theoretical and empirical evidence demonstrating that the sink in Vanilla
Attention and Sink
Attention naturally construct a Mixture-of-Experts (MoE) mechanism within
attention layers. This insight explains the head collapse phenomenon observed in prior work, where only a fixed subset of
attention heads contributes to generation. To mitigate head collapse, we propose a sink-aware training
algorithm with an auxiliary load balancing loss designed for
attention layers. Extensive experiments show that our method achieves effective head load balancing and improves model performance across Vanilla
Attention, Sink
Attention, and Gated
Attention. We hope this study offers a new perspective on
attention mechanisms and encourages further exploration of the inherent MoE structure within
attention layers.
Read the full abstract ▾
All authors Zizhuo Fu, Wenxuan Zeng, Runsheng Wang, Meng Li
Machine Learning
Unraveling the Hidden Dynamical Structure in Recurrent Neural Policies
Jin Li, Yue Wu, Mengsha Huang et al. (+3 more)
Recurrent neural policies are widely used in partially observable control and meta-RL tasks. Their abilities to maintain internal
memory and adapt quickly to unseen scenarios have offered them unparalleled performance when compared to non-recurrent counterparts. However, until today, the underlying mechanisms for their superior generalization and robustness performance remain poorly understood. In this study, by analyzing the hidden state domain of recurrent policies learned over a diverse set of training methods, model architectures, and tasks, we find that stable cyclic structures consistently emerge during interaction with the environment. Such cyclic structures share a remarkable similarity with limit cycles in dynamical system analysis, if we consider the policy and the environment as a joint hybrid dynamical system. Moreover, we uncover that the geometry of such limit cycles also has a structured correspondence with the policies' behaviors. These findings offer new perspectives to explain many nice properties of recurrent policies: the emergence of limit cycles stabilizes both the policies' internal
memory and the task-relevant environmental states, while suppressing nuisance variability arising from environmental uncertainty; the geometry of limit cycles also encodes relational structures of behaviors, facilitating easier skill adaptation when facing non-stationary environments.
Read the full abstract ▾
All authors Jin Li, Yue Wu, Mengsha Huang, Yuhao Sun, Hao He, Xianyuan Zhan
Machine Learning
The Gaussian-Head OFL Family: One-Shot Federated Learning from Client Global Statistics
Fabio Turazza, Marco Picone and Marco Mamei
Classical
Federated Learning relies on a multi-round iterative process of model exchange and aggregation between server and clients, with high communication costs and
privacy risks from repeated model transmissions. In contrast, one-shot
federated learning (OFL) alleviates these limitations by reducing communication to a single round, thereby lowering overhead and enhancing practical deployability. Nevertheless, most existing one-shot approaches remain either impractical or constrained, for example, they often depend on the availability of a public
dataset, assume homogeneous client models, or require uploading additional data or model information. To overcome these issues, we introduce the Gaussian-Head OFL (GH-OFL) family, a suite of one-shot
federated methods that assume class-conditional Gaussianity of pretrained
embeddings. Clients transmit only sufficient statistics (per-class counts and first/second-order moments) and the server builds heads via three components: (i) Closed-form Gaussian heads (NB/LDA/QDA) computed directly from the received statistics; (ii) FisherMix, a linear head with cosine margin trained on synthetic samples drawn in an estimated Fisher subspace; and (iii) Proto-Hyper, a lightweight low-rank residual head that refines Gaussian logits via knowledge
distillation on those synthetic samples. In our experiments, GH-OFL methods deliver state-of-the-art robustness and accuracy under strong non-IID skew while remaining strictly data-free.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CR
FedBGS: A Blockchain Approach to Segment Gossip Learning in Decentralized Systems
Fabio Turazza, Marcello Pietri, Marco Picone et al. (+1 more)
Privacy-Preserving
Federated Learning (PPFL) is a Decentralized machine learning paradigm that enables multiple participants to collaboratively train a global model without sharing their data with the integration of cryptographic and
privacy-based techniques to enhance the security of the global system. This
privacy-oriented approach makes PPFL a highly suitable solution for training shared models in sectors where data
privacy is a critical concern. In traditional FL, local models are trained on edge devices, and only model updates are shared with a central server, which aggregates them to improve the global model. However, despite the presence of the aforementioned
privacy techniques, in the classical
Federated structure, the issue of the server as a single-point-of-failure remains, leading to limitations both in terms of security and scalability. This paper introduces FedBGS, a fully Decentralized Blockchain-based framework that leverages Segmented Gossip Learning through
Federated Analytics. The proposed system aims to optimize blockchain usage while providing comprehensive protection against all types of attacks, ensuring both
privacy, security and non-IID data handling in
Federated environments.
Read the full abstract ▾
All authors Fabio Turazza, Marcello Pietri, Marco Picone, Marco Mamei
Machine Learning · Cross-listed from cs.CV
Refining Context-Entangled Content Segmentation via Curriculum Selection and Anti-Curriculum Promotion
Chunming He, Rihan Zhang, Fengyang Xiao et al. (+3 more)
Biological learning proceeds from easy to difficult tasks, gradually reinforcing perception and robustness. Inspired by this principle, we address Context-Entangled Content
Segmentation (CECS), a challenging setting where objects share intrinsic visual patterns with their surroundings, as in camouflaged object detection. Conventional
segmentation networks predominantly rely on architectural enhancements but often ignore the learning dynamics that govern robustness under entangled data distributions. We introduce CurriSeg, a dual-phase learning framework that unifies curriculum and anti-curriculum principles to improve representation reliability. In the Curriculum Selection phase, CurriSeg dynamically selects training data based on the temporal statistics of sample losses, distinguishing hard-but-informative samples from noisy or ambiguous ones, thus enabling stable capability enhancement. In the Anti-Curriculum Promotion phase, we design Spectral-Blindness Fine-Tuning, which suppresses high-frequency components to enforce dependence on low-frequency structural and contextual cues and thus strengthens generalization. Extensive experiments demonstrate that CurriSeg achieves consistent improvements across diverse CECS
benchmarks without adding parameters or increasing total training time, offering a principled view of how progression and challenge interplay to foster robust and context-aware
segmentation. Code will be released.
Read the full abstract ▾
All authors Chunming He, Rihan Zhang, Fengyang Xiao, Dingming Zhang, Zhiwen Cao, Sina Farsiu
Machine Learning
Analyzing and Improving Diffusion Models for Time-Series Data Imputation: A Proximal Recursion Perspective
Zhichao Chen, Hao Wang, Fangyikang Wang et al. (+5 more)
Diffusion models (DMs) have shown promise for Time-Series Data Imputation (TSDI); however, their performance remains inconsistent in complex scenarios. We attribute this to two primary obstacles: (1) non-stationary temporal dynamics, which can bias the inference trajectory and lead to outlier-sensitive imputations; and (2) objective inconsistency, since imputation favors accurate pointwise recovery whereas DMs are inherently trained to generate diverse samples. To better understand these issues, we analyze DM-based TSDI process through a proximal-operator perspective and uncover that an implicit Wasserstein distance regularization inherent in the process hinders the model's ability to counteract non-stationarity and dissipative regularizer, thereby amplifying diversity at the expense of fidelity. Building on this insight, we propose a novel framework called SPIRIT (Semi-Proximal Transport Regularized time-series Imputation). Specifically, we introduce
entropy-induced Bregman divergence to relax the mass preserving constraint in the Wasserstein distance, formulate the semi-proximal transport (SPT) discrepancy, and theoretically prove the robustness of SPT against non-stationarity. Subsequently, we remove the dissipative structure and derive the complete SPIRIT workflow, with SPT serving as the proximal operator. Extensive experiments demonstrate the effectiveness of the proposed SPIRIT approach.
Read the full abstract ▾
All authors Zhichao Chen, Hao Wang, Fangyikang Wang, Licheng Pan, Zhengnan Li, Yunfei Teng, Haoxuan Li, Zhouchen Lin
Machine Learning
Rethinking the Flow-Based Gradual Domain Adaptation: A Semi-Dual Optimal Transport Perspective
Zhichao Chen, Zhan Zhuang, Yunfei Teng et al. (+6 more)
Gradual domain adaptation (GDA) aims to mitigate domain shift by progressively adapting models from the source domain to the target domain via intermediate domains. However, real intermediate domains are often unavailable or ineffective, necessitating the synthesis of intermediate samples. Flow-based models have recently been used for this purpose by interpolating between source and target distributions. Notably, their training typically relies on sample-based log-likelihood estimation, which can discard useful information and thus degrade GDA performance. The key to addressing this limitation is constructing the intermediate domains via samples directly. To this end, we propose an
Entropy-regularized Semi-dual Unbalanced Optimal Transport (E-SUOT) framework to construct intermediate domains. Specifically, we reformulate flow-based GDA as a Lagrangian dual problem and derive an equivalent semi-dual objective that circumvents the need for likelihood estimation. However, the dual problem leads to an unstable min-max training procedure. To alleviate this issue, we further introduce the
entropy regularization to convert it into a more stable sequential optimization procedure. Based on this, we propose a novel GDA training framework and provide theoretical analysis in terms of stability and generalization. Finally, extensive experiments are conducted to demonstrate the efficacy of the E-SUOT framework.
Read the full abstract ▾
All authors Zhichao Chen, Zhan Zhuang, Yunfei Teng, Hao Wang, Fangyikang Wang, Zhengnan Li, Tianqiao Liu, Haoxuan Li, Zhouchen Lin
Machine Learning · Cross-listed from quant-ph
Privacy Implies Stability: Information-Theoretic Generalization Bounds for Quantum Learning
Ayanava Dasgupta, Naqueeb Ahmad Warsi and Masahito Hayashi
We develop an information-theoretic framework connecting stability,
privacy, and generalization for
quantum learning
algorithms. Learning procedures are modeled as
quantum instruments with classical-
quantum outputs, and losses are represented by observables. We prove that under a classical-
quantum sub-Gaussian condition, an information-theoretic stability measure controls the expected generalization error. Furthermore, we establish a high-probability generalization bound using
quantum Rényi divergences to manage higher-order dependencies under non-commutativity. In the trusted Data Processor setting,
quantum differential
privacy (QDP) provides a mechanism for stability. We show that one-neighbor QDP strictly bounds the information leaked by the classical-
quantum output. Combining this with our stability
theorem yields a direct
privacy-to-generalization guarantee. We also explore an untrusted Data Processor setting. Here, output
privacy alone is insufficient since an
adversarial processor could perform a highly informative procedure before applying noisy post-processing. To combat this, we introduce Information-Theoretic Admissibility (ITA), a certification condition ensuring the prescribed procedure is not just a degraded version of a strictly more informative, physically allowed operation on the encoded ensemble. We prove a fundamental separation: while admissibility and
privacy are in strong tension in classical models,
quantum non-orthogonality makes them compatible. A
quantum measurement can be ITA - exhausting all relevant accessible information - without perfectly recovering the classical
dataset. We illustrate this separation through a concrete
quantum ITA example.
Read the full abstract ▾
Machine Learning
When Does Cheap Physics Help? Gated Multi-Fidelity Physics-Informed Networks with Provable Sample Savings and Conformal Guarantees
Olaf Yunus Laitinen Imanov, Rana Irem Turhan and Isgandar Jafarli
High-fidelity (HF) simulations of parametric partial differential equations are expensive, whereas low-fidelity (LF) models such as coarse grids or simplified physics are cheap but biased. Multi-fidelity physics-informed neural networks combine the two, yet it remains unclear when cheap physics actually reduces the number of HF solves, and existing methods silently fail when the LF model is misleading and offer no guaranteed uncertainty. We answer the first question with a sharp sample-complexity bound: learning the HF field through its discrepancy from the LF model reduces the number of HF samples needed for accuracy $\varepsilon$ by the factor $\kappa^{2\gamma}$, where $\kappa$ is the ratio of the complexity of the discrepancy to that of the HF field and $\gamma$ is the effective-dimension exponent, and we confirm the predicted scaling with exact risk computations in a kernel model. Guided by the analysis, we propose GaMF-PINN, a gated multi-fidelity PINN that learns where to trust the LF model and falls back to physics-only learning where it does not, together with a function-level conformal procedure that returns uncertainty bands containing the entire HF solution with probability at least $1-\alpha$ in finite samples. We release MFPDE-Bench, an open
benchmark of paired LF/HF solvers for four parametric PDEs with coarse-grid and simplified-physics LF models. Across MFPDE-Bench, GaMF-PINN reaches the accuracy of a standard HF-PINN with up to 8x fewer HF solves, remains robust when the LF model is deliberately corrupted, and attains the nominal coverage where
Bayesian PINNs and deep ensembles under-cover.
Read the full abstract ▾
Machine Learning
Deep Time-Series Models Meet Volatility: Multi-Horizon Electricity Price Forecasting in the Australian National Electricity Market
Mohammed Osman Gani, Zhipeng He, Chun Ouyang et al. (+1 more)
Accurate electricity price
forecasting (EPF) is increasingly difficult in
markets characterised by extreme
volatility, frequent price spikes, and rapid structural shifts. Deep learning (DL) has been increasingly adopted in EPF due to its ability to achieve high
forecasting accuracy. Recently, state-of-the-art (SOTA) deep time-series models have demonstrated promising performance across general
forecasting tasks. Yet, their effectiveness in highly volatile electricity
markets remains underexplored. Moreover, existing EPF studies rarely assess how model accuracy varies across intraday periods, leaving model sensitivity to
market conditions unexplored. To address these gaps, this paper proposes an EPF framework that systematically evaluates SOTA deep time-series models using a direct multi-horizon
forecasting approach across day-ahead and two-day-ahead settings. We conduct a comprehensive empirical study across all five regions of the Australian National Electricity
Market using contemporary, high-
volatility data. The results reveal a clear gap between time-series
benchmark expectations and observed performance under real-world price
volatility: recent deep time-series models often fail to surpass standard DL baselines. All models experience substantial degradation under extreme and negative prices, yet DL baselines often remain competitive. Intraday performance analysis further reveals that all evaluated models are consistently vulnerable to prevailing
market conditions, where absolute errors peak during evening ramps, relative errors escalate during midday negative-price periods, and directional accuracy deteriorates sharply during abrupt shifts in price direction. These findings emphasise the need for
volatility-aware modelling strategies and richer feature representations to advance EPF.
Read the full abstract ▾
All authors Mohammed Osman Gani, Zhipeng He, Chun Ouyang, Sara Khalifa
Machine Learning
PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning
Shunpeng Yang, Ben Liu and Hua Chen
Among on-policy reinforcement learning
algorithms, Proximal Policy Optimization (PPO) demonstrates is widely favored for its simplicity, numerical stability, and strong empirical performance. Standard PPO relies on surrogate objectives defined via importance ratios, which require evaluating policy likelihood that is typically straightforward when the policy is modeled as a Gaussian distribution. However, extending PPO to more expressive, high-capacity policy models such as continuous normalizing flows (CNFs), also known as flow-matching models, is challenging because likelihood evaluation along the full flow trajectory is computationally expensive and often numerically unstable. To resolve this issue, we propose PolicyFlow, a novel on-policy CNF-based reinforcement learning
algorithm that integrates expressive CNF policies with PPO-style objectives without requiring likelihood evaluation along the full flow path. PolicyFlow approximates importance ratios using velocity field variations along a simple interpolation path, reducing computational overhead without compromising training stability. To further prevent mode collapse and further encourage diverse behaviors, we propose the Brownian Regularizer, an implicit policy
entropy regularizer inspired by Brownian motion, which is conceptually elegant and computationally lightweight. Experiments on diverse tasks across various environments including MultiGoal, PointMaze, IsaacLab and MuJoCo Playground show that PolicyFlow achieves competitive or superior performance compared to PPO using Gaussian policies and flow-based baselines including FPO and DPPO. Notably, results on MultiGoal highlight PolicyFlow's ability to capture richer
multimodal action distributions.
Read the full abstract ▾
Machine Learning
SMI: Statistical Membership Inference for Reliable Unlearned Model Auditing
Jialong Sun, Zeming Wei, Jiaxuan Zou et al. (+6 more)
Machine unlearning (MU) is essential for enforcing the right to be forgotten in machine learning systems. A key challenge of MU is how to reliably audit whether a model has truly forgotten specified training data. Membership Inference Attacks (MIAs) are widely used for unlearned model
auditing, where samples that evade membership detection are regarded as successfully forgotten. We show this assumption is fundamentally flawed: failed membership inference does not imply true forgetting. We prove that unlearned samples occupy fundamentally different positions in the feature space than non-member samples, making this
alignment bias unavoidable and unobservable, which leads to systematically optimistic evaluations of unlearning performance. Meanwhile, training shadow models for MIA incurs substantial computational overhead. To address both limitations, we propose Statistical Membership Inference (SMI), a training-free
auditing framework that reformulates
auditing as estimating the non-member mixture proportion in the unlearned feature distribution. Beyond estimating the forgetting rate, SMI also provides bootstrap reference ranges for quantified
auditing reliability. Extensive experiments show that SMI consistently outperforms all MIA-based baselines, with no shadow model training required. Overall, SMI establishes a principled and efficient alternative to MIA-based
auditing methods, with both theoretical guarantees and strong empirical performance.
Read the full abstract ▾
All authors Jialong Sun, Zeming Wei, Jiaxuan Zou, Jiacheng Gong, Jie Fu, Chengyang Dong, Heng Xu, Jialong Li, Bo Liu
Machine Learning · Cross-listed from cond-mat.mtrl-sci
A Generalizable Machine Learning Framework for Half-Heusler Thermoelectrics
Shoeb Athar, Adrien Mecibah and Philippe Jund
Machine learning (ML) has emerged as a powerful approach for accelerating the discovery of thermoelectric (TE) materials, yet many models suffer from limited generalizability. Here we present a robust ML framework for predicting the thermoelectric figure of merit (zT) in half-Heusler (hH) compounds, emphasizing prediction of previously unseen compositions. A principal component analysis (PCA)-based train/test splitting strategy is introduced to generate representative partitions of chemical space.
Bayesian hyperparameter optimization is combined with correlation-based k-best feature selection for Random Forest, XGBoost, and Neural Network models, together with the interpretable SISSO symbolic-regression method. Relative to the full set of 118 input features, the optimized feature subsets reduce the test RMSE by 16-27%. SHAP and SISSO analyses identify temperature, A-site dopant concentration, and A-site heat of vaporization as the most important descriptors of zT. The framework is validated through prediction of previously unexplored compositions, recovery of literature-reported optimal zT values for established hH systems, and first-principles calculations for the new compounds CeCoBi and CeIrBi, which exhibit favorable electronic transport properties. The framework is further applied to high-throughput screening of the doped hH chemical space, covering about 2.46 million elemental substitution combinations across 631 thermodynamically screened parent
matrices under stability constraints. This screening yields a concise set of promising candidates for future study. Overall, the work establishes a generalizable workflow for reliable data-driven thermoelectric materials discovery.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Capabilities and Fundamental Limits of Latent Chain-of-Thought
Jiaxuan Zou, Yaozhong Xiong and Yong Liu
Latent Chain-of-Thought (Latent CoT) models promise efficient
reasoning via continuous representations, yet exhibit puzzling performance inconsistencies: excelling at exploration (ProsQA: 97.0%) but failing at computation (GSM8K: 34.1%). We reveal that this trade-off is governed by decisional certainty. Our contributions are threefold: (1) We theoretically characterize the fundamental Exploration-Execution Trade-off, proving that high certainty enables precise execution but inhibits exploration, while low certainty facilitates search but causes error accumulation. (2) We introduce the Symbolic Index–quantifying decisional commitment–as the core mechanism governing this trade-off and establish its causal relationship with both execution stability and exploration capability. (3) We prove that curriculum learning is theoretically necessary, as direct training provably fails due to distributional mismatch. Our framework shifts the design paradigm from binary architectural choices toward adaptive systems that dynamically regulate decisional certainty based on task demands.
Read the full abstract ▾
Machine Learning
Generalized Radius and Integrated Codebook Transforms for Differentiable Vector Quantization
Haochen You, Heng Zhang, Hongyang He et al. (+2 more)
Vector quantization (VQ) underpins modern generative and representation models by turning continuous latents into discrete
tokens. Yet hard nearest-neighbor assignments are non-differentiable and are typically optimized with heuristic straight-through
estimators, which couple the update step size to the quantization gap and train each code in isolation, leading to unstable gradients and severe codebook under-utilization at scale. In this paper, we introduce GRIT-VQ (Generalized Radius and Integrated Transform-Vector Quantization), a unified surrogate framework that keeps hard assignments in the forward pass while making VQ fully differentiable. GRIT-VQ replaces the straight-through
estimator with a radius-based update that moves latents along the quantization direction with a controllable, geometry-aware step, and applies a data-agnostic integrated transform to the codebook so that all codes are updated through shared parameters instead of independently. Our theoretical analysis clarifies the fundamental optimization dynamics introduced by GRIT-VQ, establishing conditions for stable gradient flow, coordinated codebook
evolution, and reliable avoidance of collapse across a broad family of quantizers. Across image reconstruction, image generation, and recommendation tokenization
benchmarks, GRIT-VQ consistently improves reconstruction error, generative quality, and recommendation accuracy while substantially increasing codebook utilization compared to existing VQ variants.
Read the full abstract ▾
All authors Haochen You, Heng Zhang, Hongyang He, Yuqi Li, Baojing Liu
Machine Learning
Key Principles of Graph Machine Learning: Representation, Robustness, and Generalization
Yassine Abbahaddou
Graph Neural Networks (GNNs) have emerged as powerful tools for learning representations from structured data. Despite their growing popularity and success across various applications, GNNs encounter several challenges that limit their performance. in their generalization, robustness to
adversarial perturbations, and the effectiveness of their representation learning capabilities. In this dissertation, I investigate these core aspects through three main contributions: (1) developing new representation learning techniques based on Graph Shift Operators (GSOs, aiming for enhanced performance across various contexts and applications, (2) introducing generalization-enhancing methods through graph data augmentation, and (3) developing more robust GNNs by leveraging orthonormalization techniques and noise-based defenses against
adversarial attacks. By addressing these challenges, my work provides a more principled understanding of the limitations and potential of GNNs.
Read the full abstract ▾
Machine Learning
Self-Generative Adversarial Fine-Tuning for Large Language Models
Shiguang Wu, Yaqing Wang and Quanming Yao
Fine-tuning large
language models (LLMs) for
alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent self-play and synthetic data approaches reduce this dependence but often rely on heuristic assumptions or ungrounded self-evaluation, which can cause bias accumulation and performance drift. In this paper, we propose Self-Generative
Adversarial LLM (SGALM), a unified fine-tuning framework that formulates
alignment as a generative
adversarial game within a single LLM. SGALM jointly evolves generation and discrimination capabilities without external reward models. Theoretical and empirical results demonstrate that SGALM achieves state-of-the-art performance, serves as an effective
alignment algorithm and a robust synthetic data engine.
Read the full abstract ▾
Machine Learning
A Unified Matrix-Spectral Framework for Stability and Interpretability in Deep Learning
Ronald Katende
We develop a unified
matrix-spectral framework for analyzing stability and interpretability in deep neural networks. Representing networks as data-dependent products of linear operators reveals spectral quantities governing sensitivity to input perturbations, label noise, and training dynamics. We introduce a Global
Matrix Stability Index that aggregates spectral information from Jacobians, parameter gradients, Neural Tangent Kernel operators, and loss Hessians into a single stability scale controlling forward sensitivity, attribution robustness, and optimization conditioning. We further show that spectral
entropy refines classical operator-norm bounds by capturing typical, rather than purely worst-case, sensitivity. These quantities yield computable diagnostics and stability-oriented regularization principles. Synthetic experiments and controlled studies on MNIST, CIFAR-10, and CIFAR-100 confirm that modest spectral regularization substantially improves attribution stability even when global spectral summaries change little. The results establish a precise connection between spectral concentration and analytic stability, providing practical guidance for robustness-aware model design and training.
Read the full abstract ▾
Machine Learning
Your Autoregressive Model Already Reveals the Causal Graph
Hugo Math and Rainer Lienhart
Autoregressive models trained via next-
token prediction implicitly learn the conditional independence structure of their data-generating process. We exploit this observation to perform scalable causal discovery from a single observed sequence of discrete events – without any task-specific retraining. Such single-stream settings arise naturally in vehicle diagnostics, manufacturing systems, and patient trajectories, yet they remain largely unsolved: the absence of repeated samples, massive event vocabularies, and long-range temporal dependencies render existing methods either inaccurate or computationally intractable. We introduce TRACE, a framework that repurposes any pretrained autoregressive model as a density
estimator for conditional mutual information, the fundamental primitive for conditional independence testing. By constructing parallelized CI tests on GPUs, TRACE recovers both the sample-level time causal graph and its summary projection, scaling linearly with the vocabulary size while naturally handling delayed causal effects. Crucially, we prove that minimizing the standard cross-
entropy pretraining loss directly minimizes an upper bound on the causal identification error, establishing a duality between sequence prediction and causal discovery. On nonlinear SCMs (|X| = 8000) and real-world vehicle diagnostic logs (|X| = 29100), TRACE is the first applicable method at this scale, outperforming the strongest baseline by over 20 F1 points.
Read the full abstract ▾
Machine Learning
Tangent Space Fine-Tuning for Directional Preference Alignment in Large Language Models
Mete Erdogan
Our goal is to enable large
language models (LLMs) to balance multiple human preference dimensions; such as helpfulness, safety, and verbosity, through principled and controllable
alignment. Existing preference optimization methods, including Direct Preference Optimization (DPO), collapse feedback into a single scalar reward, fixing one balance among objectives and preventing traversal of the Pareto front. Recent work by Ortiz-Jimenez et al. (2023) showed that fine-tuning can be viewed in a model's tangent space, where linearized updates act as additive vectors that can be composed to jointly perform well on multiple tasks. Building on this formulation, we extend this idea to preference
alignment and propose Tangent-Space Direct Preference Optimization (TS-DPO), which performs DPO within this locally linear regime to learn per-objective update directions. These directions can be linearly combined at inference to generate user-specified behaviors without additional optimization. Evaluated on the helpfulness-verbosity trade-off using the HelpSteer and UltraFeedback
datasets, TS-DPO achieves broader Pareto-optimal coverage and smoother preference control than scalarized DPO. Canonical Correlation Analysis (CCA) further shows that tangent-space training amplifies canonical directions aligned with distinct preferences, improving disentanglement.
Read the full abstract ▾
Machine Learning
WinFLoRA: Incentivizing Client-Adaptive Aggregation in Federated LoRA under Privacy Heterogeneity
Mengsha Kou, Xiaoyu Xia, Ziqi Wang et al. (+4 more)
Large
Language Models (LLMs) increasingly underpin intelligent web applications, from chatbots to search and recommendation, where efficient specialization is essential. Low-Rank Adaptation (LoRA) enables such adaptation with minimal overhead, while
federated LoRA allows web service providers to fine-tune shared models without data sharing. However, in
privacy-sensitive deployments, clients inject varying levels of differential
privacy (DP) noise, creating
privacy heterogeneity that misaligns individual incentives and global performance. In this paper, we propose WinFLoRA, a
privacy-heterogeneous
federated LoRA that utilizes aggregation weights as incentives with noise awareness. Specifically, the noises from clients are estimated based on the uploaded LoRA adapters. A larger weight indicates greater influence on the global model and better downstream task performance, rewarding lower-noise contributions. By up-weighting low-noise updates, WinFLoRA improves global accuracy while accommodating clients' heterogeneous
privacy requirements. Consequently, WinFLoRA aligns heterogeneous client utility in terms of
privacy and downstream performance with global model objectives without third-party involvement. Extensive evaluations demonstrate that across multiple LLMs and
datasets, WinFLoRA achieves up to 52.58% higher global accuracy and up to 2.56x client utility than state-of-the-art
benchmarks. Source code is publicly available at https://github.com/koums24/WinFLoRA.git.
Read the full abstract ▾
All authors Mengsha Kou, Xiaoyu Xia, Ziqi Wang, Ibrahim Khalil, Runkun Luo, Jingwen Zhou, Minhui Xue
Machine Learning · Cross-listed from cs.CL
Long-range Modeling and Processing of Multimodal Event Sequences
Jichu Li, Yilun Zhong, Zhiting Li et al. (+2 more)
Temporal point processes (TPPs) have emerged as powerful tools for modeling asynchronous event sequences. While recent advances have extended TPPs to handle textual information, existing approaches are limited in their ability to generate rich,
multimodal content and reason about event dynamics. A key challenge is that incorporating
multimodal data dramatically increases sequence length, hindering the ability of
attention-based models to generate coherent, long-form textual descriptions that require long-range understanding. In this paper, we propose a novel framework that extends LLM-based TPPs to the visual modality, positioning text generation as a core capability alongside time and type prediction. Our approach addresses the long-context problem through an adaptive sequence compression mechanism based on temporal similarity, which reduces sequence length while preserving essential patterns. We employ a two-stage paradigm of pre-training on compressed sequences followed by supervised fine-tuning for downstream tasks. Extensive experiments, including on the challenging DanmakuTPP-QA
benchmark, demonstrate that our method outperforms state-of-the-art baselines in both predictive accuracy and the quality of its generated textual analyses.
Read the full abstract ▾
All authors Jichu Li, Yilun Zhong, Zhiting Li, Feng Zhou, Quyu Kong
Machine Learning
ChronoSpike: An Adaptive Spiking Graph Neural Network for Dynamic Graphs
Md Abrar Jahin, Taufikur Rahman Fuad, Jay Pujara et al. (+1 more)
Dynamic graph representation learning requires capturing both structural relations and temporal
evolution, yet existing approaches face a core trade-off:
attention-based methods offer expressiveness at O(T^2) complexity, while recurrent architectures suffer from gradient pathologies and dense state storage. Spiking neural networks provide event-driven efficiency but are constrained by sequential propagation, binary information loss, and local aggregation that lacks global context. We propose ChronoSpike, an adaptive spiking graph neural network that integrates learnable LIF
neurons with per-channel membrane dynamics, multi-head spatially-attentive aggregation over continuous features, and a lightweight
Transformer temporal encoder. This design enables fine-grained local modeling and long-range dependency capture with O(T.d) activation/state
memory and an additional O(T^2) per-node
attention term that remains small for the horizons evaluated here. ChronoSpike outperforms twelve state-of-the-art baselines on three large
benchmarks by 2.0%~Macro-F1 and 2.4%~Micro-F1 on average while achieving 3-10 times faster training than recurrent methods with a constant 105K-parameter budget independent of graph size. We provide theoretical guarantees for membrane potential boundedness, gradient flow stability under contraction factor h̊o}<1, and BIBO stability; interpretability analyses reveal heterogeneous temporal receptive fields and a learned primacy effect with 83-88% sparsity. Our code is publicly available at: https://github.com/Abrar2652/ChronoSpike
Read the full abstract ▾
All authors Md Abrar Jahin, Taufikur Rahman Fuad, Jay Pujara, Craig Knoblock