Machine Learning · Cross-listed from cs.DB
Jeffrey Tao, Yimeng Zeng, Haydn Thomas Jones et al. (+4 more)
Benchmark workloads are extremely important to the database management research community, especially as more machine learning components are integrated into database systems. Here, we propose a
Bayesian optimization technique to automatically search for difficult
benchmark queries, significantly reducing the amount of manual effort usually required. In preliminary experiments, we show that our approach can generate queries with more than double the optimization headroom compared to existing
benchmarks.
Read the full abstract ▾
All authors Jeffrey Tao, Yimeng Zeng, Haydn Thomas Jones, Natalie Maus, Osbert Bastani, Jacob R. Gardner, Ryan Marcus
Machine Learning
Same Compression Principle, Different Geometry: Rate-Distortion Signatures Dissociate Biological and Artificial Visual Systems
Leyla Roksan Caglar, Pedro A.M. Mediano and Baihan Lin
Efficient coding theory predicts that biological perceptual systems compress sensory input optimally under resource constraints, with the systematic structure of errors reflecting the geometry of that compression. Here we operationalize this principle using rate-distortion theory (RDT) to characterize how any system - biological or artificial - trades representational fidelity for informational efficiency. Treating stimulus-response behavior as an effective communication channel, we infer rate-distortion (RD) frontiers directly from confusion
matrices and summarize each system with three
geometric signatures: slope (beta),
curvature (kappa), and area under the RD curve (AUC), capturing the marginal cost, abruptness, and overall efficiency of the accuracy-compression trade-off respectively. Applying this framework to human psychophysical data and 18 deep
vision models across 12 families of controlled image perturbations at graded severities, we find that both biological and artificial systems follow a common lossy-compression principle but occupy systematically different regions of RD space. Humans exhibit smooth, flexible trade-offs characteristic of near-optimal efficient coding, while deep networks operate in steeper, more brittle regimes even at matched accuracy, with geometry dissociable from performance across training regimes. Critically, behavioral RD signatures track internal representational geometry, evidenced by the behaviorally inferred compression structure correlating with internal representational dissimilarity across all models. These results establish RD geometry as a compact diagnostic of perceptual compression strategy that recovers mechanistically interpretable structure in internal representations from behavioral input alone and extends naturally to the direct characterization of compression geometry in neural population activity.
Read the full abstract ▾
Machine Learning
Efficient Token-Level Policy Optimization for Masked Diffusion Models via Velocity Alignment
Chenxing Wei, Jiazhen Kang, Hong Wang et al. (+8 more)
Reinforcement Learning with Verifiable Rewards (RLVR) successfully aligns autoregressive models in rigorous domains like mathematical
reasoning and code generation. However, applying RLVR to
Diffusion Large
Language Models (dLLMs) is hindered by intractable sequence-level likelihoods, forcing existing methods to rely on unstable, high-variance approximations. We propose Sequence Likelihood-Free Policy Optimization (LFPO), an efficient framework that circumvents this bottleneck by formulating policy
alignment as discrete velocity
alignment. Inspired by continuous flow matching, LFPO eschews trajectory-level estimation and directly optimizes denoising logits via contrastive
token-level updates, providing a stable, low-variance surrogate. Furthermore, predicting terminal solutions from intermediate states implicitly straightens the generation trajectory, enabling high-quality sampling with significantly fewer denoising iterations. Extensive experiments on
reasoning and coding
benchmarks demonstrate that LFPO matches or exceeds the accuracy of likelihood-based baselines while substantially accelerating inference.
Read the full abstract ▾
All authors Chenxing Wei, Jiazhen Kang, Hong Wang, Jianqing Zhang, Hao Jiang, Xiaolong Xu, Ningyuan Sun, Ying He, F. Richard Yu, Yao Shu, Bo Jiang
Machine Learning · Cross-listed from cs.AI
State-Action Inpainting Diffuser for Continuous Control with Delay
Dongqi Han, Wei Wang, Enze Zhang et al. (+1 more)
Signal delay poses a fundamental challenge in continuous control and reinforcement learning (RL) by introducing a temporal gap between interaction and perception. Current solutions have largely evolved along two distinct paradigms: model-free approaches which utilize state augmentation to preserve Markovian properties, and model-based methods which focus on inferring latent beliefs via dynamics modeling. In this paper, we bridge these perspectives by introducing State-Action Inpainting Diffuser (SAID), a framework that integrates the inductive bias of dynamics learning with the direct decision-making capability of policy optimization. By formulating the problem as a joint sequence inpainting task, SAID implicitly captures environmental dynamics while directly generating consistent plans, effectively operating at the intersection of model-based and model-free paradigms. Crucially, this generative formulation allows SAID to be seamlessly applied to both online and offline RL. Extensive experiments on delayed continuous control
benchmarks demonstrate that SAID achieves state-of-the-art and robust performance. Our study suggests a new methodology to advance the field of RL with delay.
Read the full abstract ▾
All authors Dongqi Han, Wei Wang, Enze Zhang, Dongsheng Li
Machine Learning
Scalable Multi-Task Low-Rank Model Adaptation
Zichen Tian, Antoine Ledent and Qianru Sun
Scaling multi-task low-rank adaptation (LoRA) to a large number of tasks induces catastrophic performance degradation, such as an accuracy drop from 88.2% to 2.0% on DOTA when scaling from 5 to 15 tasks. This failure is due to parameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by a fundamental trade-off: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowledge concentrates in high-SV components (89%
alignment on Flanv2->BBH). Uniform regularization forces high-SV components to update in orthogonal directions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (e.g., W_q, W_v) amplifies gradient conflicts; we show block-level adaptation reduces this conflict by 76% with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable solution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve parameter efficiency, and 3) Fine-Grained Routing using dimension-specific weights for superior expressive power. On four large-scale (15-25 tasks)
vision (DOTA and iNat2018) and NLP (Dolly-15k and BBH)
benchmarks, mtLoRA achieves 91.7%, 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively, outperforming the state-of-the-art by 2.3% on average while using 47% fewer parameters and 24% less training time.
Read the full abstract ▾
Machine Learning
Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning
Gautam Goel, Mahdi Soltanolkotabi and Peter Bartlett
We study the training dynamics of gradient descent in a multilayer softmax self-
attention network trained to perform linear regression and show that in the infinite-data limit this regression problem is equivalent to a
matrix factorization problem. This connection motivates the design of Hera, a new first-order optimization
algorithm for training self-
attention networks. Key features of Hera include a reparameterization of the model arising from the
matrix factorization geometry and a rescaling of the parameters which ensures that the empirical loss gradients remain close to the auxiliary
matrix factorization gradients throughout training. Hera updates in the reparameterized model are equivalent to a certain preconditioned update in the original parameterization. We also propose a variant of Hera called Hera-PSD; our main theoretical result is that under small Gaussian initialization the excess risk of the model trained by Hera-PSD decreases exponentially fast in the number of gradient descent iterations, up to an additional error term that decreases as $1/n$, up to logarithmic factors, in the size of the training set. A particularly appealing aspect of our excess risk bound is that it does not degrade as the number of self-
attention layers increases. In a stylized time-series prediction experiment in a nonlinear dynamical system, we find that a model trained by Hera achieves $98.1\%$ lower mean-squared test error than a model trained by AdamW and $97.7\%$ lower than a model trained by
Muon at the same iteration budget.
Read the full abstract ▾
Machine Learning
GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
Haofeng Xu, Junwei Su, Yukun Tian et al. (+3 more)
Asynchronous execution is essential for scaling reinforcement learning (RL) to modern large model workloads, including large
language models and AI
agents, but it can fundamentally alter RL optimization behavior. While prior work on asynchronous RL focuses on training throughput and distributional correction, we show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability. Through systematic empirical and theoretical analysis, we identify a key signature of this instability: asynchronous training exhibits persistently high cosine similarity between consecutive policy gradients, in contrast to the near-orthogonal updates observed under synchronized training. This stale-aligned gradient effect amplifies correlated updates and increases the risk of overshooting and divergence. Motivated by this observation, we propose GRADIENT
ALIGNMENT CONTROL(GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection. We establish convergence guarantees under bounded staleness and demonstrate empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.
Read the full abstract ▾
All authors Haofeng Xu, Junwei Su, Yukun Tian, Lansong Diao, Zhengping Qian, Chuan Wu
Machine Learning · Cross-listed from cs.SE
Inference-Time Safety For Code LLMs Via Retrieval-Augmented Revision
Manisha Mukherjee and Vincent J. Hellendoorn
Large
Language Models (LLMs) are increasingly deployed for code generation in high-stakes software development, yet their limited transparency in security
reasoning and brittleness to evolving vulnerability patterns raise critical trustworthiness concerns. Models trained on static
datasets cannot readily adapt to newly discovered vulnerabilities or changing security standards without retraining, leading to the repeated generation of unsafe code. We present a principled approach to trustworthy code generation by design that operates as an inference-time safety mechanism. Our approach employs
retrieval-augmented generation to surface relevant security risks in generated code and retrieve related security discussions from a curated Stack Overflow knowledge base, which are then used to guide an LLM during code revision. This design emphasizes three aspects relevant to trustworthiness: (1) interpretability, through transparent safety interventions grounded in expert community explanations; (2) robustness, by allowing adaptation to evolving security practices without model retraining; and (3) safety
alignment, through real-time intervention before unsafe code reaches deployment. Across real-world and
benchmark datasets, our approach improves the security of LLM-generated code compared to prompting alone, while introducing no new vulnerabilities as measured by static analysis. These results suggest that principled,
retrieval-augmented inference-time interventions can serve as a complementary mechanism for improving the safety of LLM-based code generation, and highlight the ongoing value of community knowledge in supporting trustworthy AI deployment.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.AS
A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection
Hashim Ali, Nithin Sai Adupa, Surya Subramani et al. (+1 more)
Self-supervised learning (SSL) has transformed
speech processing, with
benchmarks such as SUPERB establishing fair comparisons across diverse downstream tasks. Despite it's security-critical importance, Audio deepfake detection has remained outside these efforts. In this work, we introduce Spoof-SUPERB, a
benchmark for audio deepfake detection that systematically evaluates 20 SSL models spanning generative, discriminative, and spectrogram-based architectures. We evaluated these models on multiple in-domain and out-of-domain
datasets. Our results reveal that large-scale discriminative models such as XLS-R, UniSpeech-SAT, and WavLM Large consistently outperform other models, benefiting from
multilingual pretraining, speaker-aware objectives, and model scale. We further analyze the robustness of these models under acoustic degradations, showing that generative approaches degrade sharply, while discriminative models remain resilient. This
benchmark establishes a reproducible baseline and provides practical insights into which SSL representations are most reliable for securing
speech systems against audio deepfakes.
Read the full abstract ▾
All authors Hashim Ali, Nithin Sai Adupa, Surya Subramani, Hafiz Malik
Machine Learning · Cross-listed from cs.IR
Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning
Jiahan Chen, Da Li, Hengran Zhang et al. (+6 more)
Multimodal embedding models, rooted in
multimodal large
language models (MLLMs), have yielded significant performance improvements across diverse tasks such as
retrieval and classification. However, most existing approaches rely heavily on large-scale contrastive learning, with limited exploration of how the architectural and training paradigms of MLLMs affect
embedding quality. While effective for generation, the causal
attention and next-
token prediction paradigm of MLLMs does not explicitly encourage the formation of globally compact representations, limiting their effectiveness as
multimodal embedding backbones. To address this, we propose CoCoA, a Content reconstruction pre-training paradigm based on Collaborative
Attention for
multimodal embedding optimization. Specifically, we restructure the
attention flow and introduce an EOS-based reconstruction task, encouraging the model to reconstruct input from the corresponding <EOS>
embeddings. This drives the
multimodal model to compress the semantic information of the input into the <EOS>
token, laying the foundations for subsequent contrastive learning. Extensive experiments on MMEB-V1 demonstrate that CoCoA built upon Qwen2-VL and Qwen2.5-VL significantly improves
embedding quality. Results validate that content reconstruction serves as an effective strategy to maximize the value of existing data, enabling
multimodal embedding models generate compact and informative representations, raising their performance ceiling.
Read the full abstract ▾
All authors Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi
Machine Learning
Randomized Kriging Believer for Parallel Bayesian Optimization with Regret Bounds
Shuhei Sugiura, Ichiro Takeuchi and Shion Takeno
We consider the optimization problem of an expensive-to-evaluate black-box function, in which we can obtain noisy function values in parallel. For this problem, parallel
Bayesian optimization (PBO) is a promising approach, which aims to optimize with fewer function evaluations by selecting a diverse input set for parallel evaluation. However, existing PBO methods suffer from poor practical performance or lack theoretical guarantees. In this study, we propose a PBO method, called randomized kriging believer (KB), based on a well-known KB heuristic and inheriting the advantages of the original KB: low computational complexity, a simple implementation, versatility across various BO methods, and applicability to asynchronous parallelization. Furthermore, we show that our randomized KB achieves
Bayesian expected regret guarantees. We demonstrate the effectiveness of the proposed method through experiments, including those on real-data emulators.
Read the full abstract ▾
Machine Learning · Cross-listed from astro-ph.IM
PhysFormer: A Physics-Embedded Generative Model for Physically Self-Consistent Spectral Synthesis
Siqi Wang, Mengmeng Zhang, Yude Bu et al. (+1 more)
In scientific and engineering domains, modeling high-dimensional complex systems governed by partial differential equations (PDEs) remains challenging in terms of physical consistency and numerical stability. However, existing approaches, such as physics-informed neural networks (PINNs), typically rely on known physical fields or coefficients and enforce physical constraints via external loss functions, which can lead to training instability and make it difficult to handle high-dimensional or unobservable scenarios. To this end, we propose PhysFormer, a generative modeling framework that is self-consistent at both the data and physical levels. PhysFormer leverages a low-dimensional, physically interpretable latent space to learn key physical quantities directly from data without requiring known high-dimensional physical field parameters, and embeds the physical process of radiative flux generation within the network to ensure the physical consistency of the generated spectra. In high-dimensional, degenerate inversion tasks, PhysFormer constrains generation within physical limits and enhances spectral fidelity and inversion stability under varying signal-to-noise ratios (SNRs). More broadly, this approach shifts the physical processes from external loss functions into the generative mechanism itself, providing a physically consistent generative modeling paradigm for complex systems involving unknown or unobservable physical quantities.
Read the full abstract ▾
All authors Siqi Wang, Mengmeng Zhang, Yude Bu, Chaozhou Mou
Machine Learning · Cross-listed from cs.DB
SEAnet: A Deep Learning Architecture for Data Series Similarity Search
Qitong Wang and Themis Palpanas
A key operation for massive data series collection analysis is similarity search. According to recent studies, SAX-based indexes offer state-of-the-art performance for similarity search tasks. However, their performance lags under high-frequency, weakly correlated, excessively noisy, or other
dataset-specific properties. In this work, we propose Deep
Embedding Approximation (DEA), a novel family of data series summarization techniques based on deep neural networks. Moreover, we describe SEAnet, a novel architecture especially designed for learning DEA, that introduces the Sum of Squares preservation property into the deep network design. We further enhance SEAnet with SEAtrans encoder. Finally, we propose novel sampling strategies, SEAsam and SEAsamE, that allow SEAnet to effectively train on massive
datasets. Comprehensive experiments on 7 diverse synthetic and real
datasets verify the advantages of DEA learned using SEAnet in providing high-quality data series summarizations and similarity search results.
Read the full abstract ▾
Machine Learning
Autoregressive Synthesis of Sparse and Semi-Structured Mixed-Type Data
Thomas Rückstieß and Robin Vujanic
Synthetic data generation is an important capability for
privacy-preserving data sharing, system benchmarking and test data provisioning. For mixed-type data, existing synthesizers largely target dense, fixed-schema tables, but many modern data systems store and exchange sparse, semi-structured JSON with nested objects, variable-length arrays and optional keys. Applying tabular synthesizers to such data requires flattening records into wide, sparse tables, turning nested structure and arrays into column-layout artifacts. We present ORiGAMi, an autoregressive
transformer architecture for modeling and synthesizing semi-structured records without flattening. ORiGAMi serializes JSON records into key, value, and structural
tokens, and encodes
token positions by their path in the document tree. Grammar and schema constraints enforce syntactically valid JSON and
dataset-consistent structure. We evaluate ORiGAMi against VAE, GAN,
diffusion, and autoregressive baselines that operate on flattened representations across six
datasets ranging from dense tabular
benchmarks to large-scale semi-structured collections. Across fidelity, detection, and utility metrics, ORiGAMi achieves the best score in 17 of 18
benchmark comparisons, while maintaining high
privacy scores above 96% across all settings. These results establish native record modeling as a strong alternative to tabular synthesis pipelines, preserving structure while achieving state-of-the-art
benchmark performance.
Read the full abstract ▾
Machine Learning · Cross-listed from math.OC
On the Stability Connection Between Discrete-Time Algorithms and Their Resolution ODEs: Applications to Min-Max Optimisation
Amir Ali Farzin, Yuen-Man Pun, Philipp Braun et al. (+1 more)
This work establishes a rigorous connection between stability properties of discrete-time
algorithms (DTAs) and corresponding continuous-time dynamical systems derived through $ O(s^r) $-resolution ordinary differential equations (ODEs). We show that for discrete- and continuous-time dynamical systems satisfying a mild error assumption, exponential stability of a common
equilibrium with respect to the continuous time dynamics implies exponential stability of the corresponding
equilibrium for the discrete-time dynamics, provided that the step size is chosen sufficiently small. We extend this result to common compact invariant sets. We prove that if an
equilibrium is exponentially stable for the $ O(s^r) $-resolution ODE, then it is also exponentially stable for the associated DTA. We apply this framework to analyse the limit point properties of several prominent optimisation
algorithms, including Two-Timescale Gradient Descent–Ascent (TT-GDA), Generalised Extragradient (GEG), Two-Timescale Proximal Point (TT-PPM), Damped Newton (DN), Regularised Damped Newton (RDN), and the Jacobian method (JM), by studying their $ O(1) $- and $ O(s) $-resolution ODEs. We show that under a proper choice of hyperparameters, the set of saddle points of the objective function is a subset of the set of exponentially stable
equilibria of GEG, TT-PPM, DN, and RDN. We relax the common Hessian invariance assumption through direct analysis of the resolution ODEs, broadening the applicability of our results. Numerical examples illustrate the theoretical findings.
Read the full abstract ▾
All authors Amir Ali Farzin, Yuen-Man Pun, Philipp Braun, Iman Shames
Machine Learning
Tackling multiphysics problems via finite element-guided physics-informed operator learning
Yusuke Yamazaki, Reza Najian Asl, Markus Apel et al. (+2 more)
This work presents a finite element-guided physics-informed operator learning framework for multiphysics problems with coupled partial differential equations (PDEs) on arbitrary domains. The proposed framework learns an operator from the input space to the solution space with a weighted residual formulation based on the finite element method, enabling discretization-independent prediction beyond the training resolution without relying on labeled simulation data. The present framework for multiphysics problems is implemented in Folax, a JAX-based operator learning platform, and is verified on nonlinear coupled thermo-mechanical problems. Two- and three-dimensional representative volume elements with varying heterogeneous microstructures, and a close-to-reality industrial casting example under varying boundary conditions are investigated as the example problems. We investigate the potential of several neural operators combined with the proposed finite element-guided approach, including
Fourier neural operators (FNOs), deep operator networks (DeepONets), and a newly proposed implicit finite operator learning (iFOL) approach based on conditional neural fields. The results demonstrate that FNOs yield highly accurate solution operators on regular domains, where the global features can be efficiently learned in the spectral domain, and iFOL offers efficient parametric operator learning capabilities for complex and irregular geometries. Furthermore, studies on training strategies, network decomposition, and training sample quality reveal that a monolithic training strategy using a single network is sufficient for accurate predictions, while training sample quality strongly influences performance. Overall, the present approach highlights the potential of physics-informed operator learning with a finite element-based loss as a unified and scalable approach for coupled multiphysics simulations.
Read the full abstract ▾
All authors Yusuke Yamazaki, Reza Najian Asl, Markus Apel, Mayu Muramatsu, Shahed Rezaei
Machine Learning · Cross-listed from cs.AI
MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning
Sicheng Zhu, Jiajun Wang, Jiawei Ai et al. (+1 more)
Large
Language Models (LLMs) often fail to generate correct code on the first attempt, which requires using generated unit tests as verifiers to validate the solutions. Despite the success of recent verification methods, they remain constrained by a "scaling-by-quantity" paradigm. This brute-force approach suffers from a critical limitation: it yields diminishing returns in fault detection while causing severe test redundancy. To address this, we propose MIST-RL (Mutation-based Incremental Suite Testing via Reinforcement Learning), a framework that shifts the focus to "scaling-by-utility". We formulate test generation as a sequential decision process optimized via Group Relative Policy Optimization (GRPO). Specifically, we introduce a novel incremental mutation reward combined with dynamic penalties, which incentivizes the model to discover new faults while it suppresses functionally equivalent assertions. Experiments on HumanEval+ and MBPP+ demonstrate that MIST-RL outperforms state-of-the-art baselines. It achieves a +28.5% higher mutation score while reducing the number of test cases by 19.3%. Furthermore, we show that these compact, high-utility tests serve as superior verifiers, which improves downstream code reranking accuracy on HumanEval+ by 3.05% over the SOTA baseline with 10 candidate samples. The source code and data are provided in the supplementary material.
Read the full abstract ▾
All authors Sicheng Zhu, Jiajun Wang, Jiawei Ai, Xin Li
Machine Learning
One Operator to Rule Them All? On Boundary-Indexed Operator Families in Neural PDE Solvers
Lennon J. Shikhman
Neural PDE solvers are often described as learning solution operators that map problem data to PDE solutions. In this work, we argue that this interpretation is generally incorrect when boundary conditions vary. We show that standard neural operator training implicitly learns a boundary-indexed family of operators, rather than a single boundary-agnostic operator, with the learned mapping fundamentally conditioned on the boundary-condition distribution seen during training. We formalize this perspective by framing operator learning as conditional risk minimization over boundary conditions, which leads to a non-identifiability result outside the support of the training boundary distribution. As a consequence, generalization in forcing terms or resolution does not imply generalization across boundary conditions. We support our theoretical analysis with controlled experiments on the Poisson equation, demonstrating sharp degradation under boundary-condition shifts, cross-distribution failures between distinct boundary ensembles, and convergence to conditional expectations when boundary information is removed. Our results clarify a core limitation of current neural PDE solvers and highlight the need for explicit boundary-aware modeling in the pursuit of foundation models for PDEs.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.DC
Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification
Guang Huang and Zeyi Wen
Speculative Decoding (SD) has emerged as a premier technique for accelerating Large
Language Model (LLM) inference by decoupling
token generation into rapid drafting and parallel verification. While recent advancements in self-speculation and lookahead decoding have successfully minimized drafting overhead, they have shifted the primary performance bottleneck to the verification phase. Since verification requires a full forward pass of the target model, it remains strictly
memory-bandwidth bound, fundamentally limiting the maximum achievable speedup.In this paper, we introduce
Quasar (Quantized Self-speculative Acceleration for Rapid Inference), a novel, training-free framework designed to overcome this "
memory wall" by employing low-bit quantization specifically for the verification stage. Our empirical analysis reveals that while aggressive structural
pruning significantly degrades verification accuracy, quantization-based verification preserves the logit distribution with high fidelity while effectively halving
memory traffic. Extensive experiments on state-of-the-art models (e.g., OpenPangu and Qwen3) demonstrate that
Quasar maintains a speculative acceptance length comparable to full-precision methods while achieving a $1.28\times$ improvement in end-to-end throughput. Being orthogonal to existing drafting strategies,
Quasar offers a generic and efficient pathway to accelerate the verification leg of speculative execution. Code is available at https://github.com/Tom-HG/
Quasar.
Read the full abstract ▾
Machine Learning
Invariant-Stratified Propagation for Expressive Graph Neural Networks
Asela Hevapathige, Ahad N. Zehmakan, Asiri Wijesinghe et al. (+1 more)
Graph Neural Networks (GNNs) face fundamental limitations in expressivity and capturing structural heterogeneity. Standard message-passing architectures are constrained by the 1-dimensional Weisfeiler-Leman (1-WL) test, unable to distinguish graphs beyond degree sequences, and aggregate information uniformly from neighbors, failing to capture how nodes occupy different structural positions within higher-order patterns. While methods exist to achieve higher expressivity, they incur prohibitive computational costs and lack unified frameworks for flexibly encoding diverse structural properties. To address these limitations, we introduce Invariant-Stratified Propagation (ISP), a framework comprising both a novel WL variant (ISP-WL) and its efficient neural network implementation (ISPGNN). ISP stratifies nodes according to graph invariants, processing them in hierarchical strata that reveal structural distinctions invisible to 1-WL. Through hierarchical structural heterogeneity encoding, ISP quantifies differences in nodes' structural positions within higher-order patterns, distinguishing interactions where participants occupy different roles from those with uniform participation. We provide formal theoretical analysis establishing enhanced expressivity beyond 1-WL, convergence guarantees, and inherent resistance to oversmoothing. Extensive experiments across graph classification, node classification, and influence estimation demonstrate consistent improvements over standard architectures and state-of-the-art expressive baselines.
Read the full abstract ▾
All authors Asela Hevapathige, Ahad N. Zehmakan, Asiri Wijesinghe, Saman Halgamuge
Machine Learning
3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
Mehdi Makni, Xiang Meng and Rahul Mazumder
Sparse plus Low-Rank $(\mathbf{S} + \mathbf{LR})$ decomposition of Large
Language Models (LLMs) has emerged as a promising direction in model compression, aiming to decompose pre-trained model weights into a sum of sparse and low-rank
matrices $(\mathbf{W} \approx \mathbf{S} + \mathbf{LR})$. Despite recent progress, existing methods often suffer from substantial performance degradation compared to dense models. In this work, we introduce 3BASiL-TM, an efficient one-shot post-training method for $(\mathbf{S} + \mathbf{LR})$ decomposition of LLMs that addresses this gap. Our approach first introduces a novel 3-Block Alternating Direction Method of Multipliers (ADMM) method, termed 3BASiL, to minimize the layer-wise reconstruction error with convergence guarantees. We then design an efficient
transformer-matching (TM) refinement step that jointly optimizes the sparse and low-rank components across
transformer layers. This step minimizes a novel
memory-efficient loss that aligns outputs at the
transformer level. Notably, the TM procedure is universal as it can enhance any $(\mathbf{S} + \mathbf{LR})$ decomposition, including pure sparsity. Our numerical experiments show that 3BASiL-TM reduces the WikiText2 perplexity gap relative to dense LLaMA-8B model by over 30% under a (2:4 Sparse + 64 LR) configuration, compared to prior methods. Moreover, our method achieves over 2.5x faster compression runtime on an A100 GPU compared to SOTA $(\mathbf{S} + \mathbf{LR})$ method. Our code is available at https://github.com/mazumder-lab/3BASiL.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation
Chenxing Wei, Hong Wang, Ying He et al. (+4 more)
Test-time policy adaptation for multi-turn interactions (T2PAM) is essential for aligning Large
Language Models (LLMs) with dynamic user needs during inference time. However, existing paradigms commonly treat test-time adaptation as a single-axis problem, either purely refining instructions (
Prompt Engineering) or only adjusting weights (Test-Time Training), ignoring that interaction failures stem from a coupled mix of ambiguity and incapacity. We argue that these two optimization paths are not merely additive but synergistic: semantic clarity acts as a pre-conditioner for effective parameter updates. To this end, we propose ROSA2, a framework that reformulates interaction as a joint optimization problem over the heterogeneous space of Words and Weights. By mathematically decomposing the error signal, ROSA2 utilizes textual gradients to rectify intent ambiguity and parameter updates to bridge capability gaps. Theoretically, we prove that this co-adaptation strictly reduces the required parameter shift for convergence. Empirically, ROSA2 outperforms state-of-the-art baselines by 30% on MATH while reducing interaction turns by 40%, demonstrating that refining the context unlocks the true potential of parameter updates.
Read the full abstract ▾
All authors Chenxing Wei, Hong Wang, Ying He, Zhongxiang Dai, Bo Jiang, F. Richard Yu, Yao Shu
Machine Learning
Causal Neural Probabilistic Circuits
Weixin Chen and Han Zhao
Concept Bottleneck Models (CBMs) enhance the interpretability of end-to-end neural networks by introducing a layer of concepts and predicting the class label from the concept predictions. A key property of CBMs is that they support interventions, i.e., domain experts can correct mispredicted concept values at test time to improve the final accuracy. However, typical CBMs apply interventions by overwriting only the corrected concept while leaving other concept predictions unchanged, which ignores causal dependencies among concepts. To address this, we propose the Causal Neural Probabilistic
Circuit (CNPC), which combines a neural attribute predictor with a causal probabilistic
circuit compiled from a causal graph. This
circuit supports exact, tractable causal inference that inherently respects causal dependencies. Under interventions, CNPC models the class distribution based on a Product of Experts (PoE) that fuses the attribute predictor's predictive distribution with the interventional marginals computed by the
circuit. We theoretically characterize the compositional interventional error of CNPC w.r.t. its modules and identify conditions under which CNPC closely matches the ground-truth interventional class distribution. Experiments on five
benchmark datasets in both in-distribution and out-of-distribution settings show that, compared with five baseline models, CNPC achieves higher task accuracy across different numbers of intervened attributes.
Read the full abstract ▾
Machine Learning
DUEL: Exact Likelihood Reshapes Masked Diffusion Scaling Laws and Beyond
Gilad Turok, Yair Schiff, Chris De Sa et al. (+1 more)
Masked
diffusion models (MDMs) are trained to predict
tokens in every order but generate in one. As such, MDM likelihood is an intractable sum over all orders. The evidence lower bound (ELBO) is used as a proxy for likelihood, but it is a loose bound and assumes positions are unmasked uniformly at random. Frontier MDMs, however, select positions with deterministic rules such as confidence. To resolve this mismatch we introduce DUEL, a framework that unifies common deterministic MDM samplers. DUEL computes the likelihood exactly, under the sampler that is actually used during generation. DUEL is a simple plug-and-play
algorithm applicable to any pretrained MDM with no retraining. To compute the likelihood of a text sequence, simply replay generation, reveal the true
tokens instead of sampled ones, and sum their log-probabilities. Evaluating with DUEL instead of the ELBO changes what we conclude about MDMs. The perplexity gap between MDMs and autoregressive (AR) models shrinks by up to 32% in-domain and 82% zero-shot. In scaling laws, the reported 16x MDM-AR compute gap falls to 9x under DUEL, showing over 1/3 of it is an evaluation artifact. Scoring multiple-choice answers with likelihood via DUEL instead of the ELBO improves LLaDA-8B accuracy on all six tasks, by up to 9.7 points. DUEL also selects samplers correctly: it matches a gold-standard LLM judge on every decided comparison, which generative perplexity and MAUVE fail to do. DUEL reveals that MDMs are more competitive with AR models than the ELBO suggests.
Read the full abstract ▾
All authors Gilad Turok, Yair Schiff, Chris De Sa, Volodymyr Kuleshov
Machine Learning
Align and Filter: Improving Performance in Asynchronous On-Policy RL
Homayoun Honari, Roger Creus Castanyer, Michael Przystupa et al. (+3 more)
Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: policy lag, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning
algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym) as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math
reasoning task.
Read the full abstract ▾
All authors Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, Glen Berseth
Machine Learning
Fed-GAME: Personalized Federated Learning with Graph Attention Mixture-of-Experts For Time-Series Forecasting
Yi Li, Han Liu, Mingfeng Fan et al. (+3 more)
Federated learning (FL) on graphs shows promise for distributed time-series
forecasting. Yet, existing methods rely on static
topologies and struggle with client heterogeneity. We propose Fed-GAME, a framework that models personalized aggregation as message passing over a learnable dynamic implicit graph. The core is a decoupled parameter difference-based update protocol, where clients transmit parameter differences between their fine-tuned private model and a shared global model. On the server, these differences are decomposed into two streams: (1) averaged difference used to updating the global model for consensus (2) the selective difference fed into a novel Graph
Attention Mixture-of-Experts (GAME) aggregator for fine-grained personalization. In this aggregator, shared experts provide scoring signals while personalized gates adaptively weight selective updates to support personalized aggregation. Experiments on two real-world electric vehicle charging
datasets demonstrate that Fed-GAME outperforms state-of-the-art personalized FL baselines.
Read the full abstract ▾
All authors Yi Li, Han Liu, Mingfeng Fan, Guo Chen, Chaojie Li, Biplab Sikdar
Machine Learning
Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain
Yuma Okochi, Fabio Milentiansen Sim and Tomoyasu Okada
In adapting LLMs to specific domains, achieving both domain expertise and
reasoning ability remains an urgent challenge. This study proposes a general method for constructing high-quality synthetic instruction data for any domain, starting from domain-specific vocabulary. As a demonstration, we applied this method to the financial domain and constructed a large-scale instruction
dataset totaling approximately 9.5 billion
tokens with Chain-of-Thought
reasoning traces. Evaluation results confirmed performance improvements over baseline models on financial
benchmarks, demonstrating the effectiveness of our approach. We also report findings on the impact of
reasoning trace length on performance and its limitations. Lastly, we open-source our models and
datasets on https://huggingface.co/nri-ai .
Read the full abstract ▾
Machine Learning
UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification
Yessin Moakher, Youssef Attia El Hili and Vasilii Feofanov
Self-supervised foundation models have achieved remarkable success across domains, including time series. However, the potential of non-contrastive methods, a paradigm that has driven significant advances in computer
vision, remains underexplored for time series. In this work, we adapt DINOv2-style self-
distillation to pretrain a time series foundation model, building on the Mantis tokenizer and
transformer encoder architecture as our backbone. Through a student-teacher framework, our method Utica learns representations that capture both temporal invariance via augmented crops and fine-grained local structure via patch masking. Our approach achieves state-of-the-art classification performance on both UCR and UEA
benchmarks. These results suggest that non-contrastive methods are a promising and complementary pretraining strategy for time series foundation models.
Read the full abstract ▾
Machine Learning
Relatively Smart: A New Approach for Instance-Optimal Learning
Shaddin Dughmi and Alireza F. Pour
We revisit the framework of Smart PAC learning, which seeks supervised learners which compete with semi-supervised learners that are provided full knowledge of the marginal distribution on unlabeled data. Prior work has shown that such marginal-by-marginal guarantees are possible for "most" marginals, with respect to an arbitrary fixed and known measure, but not more generally. We discover that this failure can be attributed to an "indistinguishability" phenomenon: There are marginals which cannot be statistically distinguished from other marginals that require different learning approaches. In such settings, semi-supervised learning cannot certify its guarantees from unlabeled data, rendering them arguably non-actionable. We propose relatively smart learning, a new framework which demands that a supervised learner compete only with the best "certifiable" semi-supervised guarantee. We show that such modest relaxation suffices to bypass the impossibility results from prior work. In the distribution-free setting, we show that the One-Inclusion Graph learner is relatively smart up to squaring the sample complexity, and show that no supervised learning
algorithm can do better. For distribution-family settings, we show that relatively smart learning can be impossible or can require idiosyncratic learning approaches, and its difficulty can be non-monotone in the inclusion order on distribution families.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CR
SubstratumGraphEnv: Reinforcement Learning Environment (RLE) for Modeling System Attack Paths
Bahirah Adewunmi, Edward Raff and Sanjay Purushotham
Automating network security analysis, particularly the identification of potential attack paths, presents significant challenges. Due in part to the sequential, interconnected, and evolutionary nature of system events which most artificial intelligence (AI) techniques struggle to model effectively. This paper proposes a Reinforcement Learning (RL) environment generation framework that simulates the sequence of processes executed on a Windows operating system, enabling dynamic modeling of malicious processes on a system. This methodology models operating system state and transitions using a graph representation. This graph is derived from open-source System Monitor (Sysmon) logs. To address the variety in system event types, fields, and log formats, a mechanism was developed to capture and model parent-child processes from Sysmon logs. A Gymnasium environment (SubstratumGraphEnv) was constructed to establish the perceptible basis for an RL environment, and a customized PyTorch interface was also built (SubstratumBridge) to translate Gymnasium graphs into Deep Reinforcement Learning (DRL) observations and discrete actions. Graph Convolutional Networks (GCNs) concretize the graph's local and global state, which feed the distinct policy and critic heads of an Advantage Actor-Critic (A2C) model. This work's central contribution lies in the design of a novel deep graphical RL environment that automates translation of sequential user and system events, furnishing crucial context for cybersecurity analysis. This work provides a foundation for future research into shaping training parameters and advanced reward shaping, while also offering insight into which system events attributes are critical to training autonomous RL
agents.
Read the full abstract ▾