Machine Learning
Online Packet Scheduling with Deadlines and Learning
Gianmarco Genalti, Achraf Azize and Vianney Perchet
Network routers that enforce Quality-of-Service (QoS) guarantees must decide, at every clock cycle, which expiring packet of information to transmit, even when the value of the packet is unknown until it is processed. We frame this problem as the Online Packet Scheduling with Deadlines (OPSD) problem under Partial Feedback: packets arrive at every clock cycle, with different deadlines, but the weights are only observed after execution. Under a
stochastic assumption on the unknown weights, we explore different variants of the OPSD problem with bandit feedback. We establish a connection between our setting and the sleeping bandits problem, and set our learning goal to $\alpha$-regret minimization. We provide
algorithms with provable $\alpha$-regret guarantees under different spans of slackness, distinguishing systems allowing for randomization and systems that do not. In every scenario, our
algorithms achieve an $\alpha$-regret upper bound of $\widetilde{\mathcal{O}}\left(\sqrt{KT}\right)$, matching the lower bound for the standard bandit setting. In the practically relevant case of $2$-bounded deadline instances, where the deadline is set at most one clock cycle away from the arrival, our deterministic
algorithm achieves the provably tightest possible competitive ratio. Remarkably, when the number of distinct packet types $K\ge 2$ is finite, it is possible to break the well-established $\Phi = \frac{1+\sqrt{5}}{2}$ competitive ratio barrier and attain a tighter competitive ratio $\theta_K$ ranging in $[\sqrt{2}, \Phi)$.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.AP
Hybrid Probabilistic Forecasting of Under-Five Malaria Admissions in Ghana: A Gaussian Process Regression with Holt-Winters Smoothing
T. Ansah-Narh, Y. Asare Afrane and J. Bremang Tandoh
Accurate malaria
forecasting remains a major challenge in sub-Saharan Africa, where strong seasonality, reporting uncertainty, and non-stationary transmission dynamics reduce the reliability of conventional models. In Ghana, district-level malaria surveillance requires
forecasting frameworks that are probabilistically rigorous and robust under limited data. This study proposes a hybrid framework integrating Gaussian Process Regression (GPR) with Holt-Winters exponential smoothing for modelling monthly under-five malaria admissions. GPR captures non-linear behaviour and predictive uncertainty, while Holt-Winters stabilises long-horizon forecasts and preserves seasonal structure. Using ten years of district-level data (2014-2023), performance was evaluated via rolling-origin expanding-window validation. The hybrid model achieved $R^2 = 0.9906$ versus $0.8213$ for Holt-Winters alone, with $94.2\%$ of residuals within $\pm 2\sigma$ bounds. Forecasts for 2024-2028 project average monthly admissions from approximately 8{,}000 to 12{,}200 cases. Spatio-temporal analysis revealed pronounced ecological heterogeneity: northern high-burden districts exhibited stable relative patterns despite large absolute fluctuations. The framework provides a scalable probabilistic approach for malaria early warning and operational planning in endemic settings, supporting Ghana's national malaria control strategy.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Subliminal Learning is a LoRA Artifact
Todd Nief, Harvey Yiyun Fu, Mark Muchane et al. (+1 more)
Subliminal learning is a phenomenon where
language models can transmit behavioral traits to other models through seemingly innocuous data (Cloud et al., 2025). In subliminal learning, a teacher model with a behavioral trait (e.g. obsession with cats) can transmit this cat obsession to a student model finetuned only on numerical sequences generated by the teacher. In this paper, we ask: how does this unexpected behavioral transmission occur? We show that subliminal learning is a LoRA artifact. When subliminal learning occurs, transmission has an inverted U-shaped relationship with LoRA rank; it also disappears with full finetuning. We show that subliminal learning is highly dependent on the context seen during finetuning and evaluation. For example, a Qwen model with the default system
prompt during finetuning ("You are Qwen, created by Alibaba Cloud. You are a helpful assistant.") does not show subliminal learning during generation when no system
prompt is included. We further demonstrate that subliminal behavior is localized to computation at
tokens seen during both finetuning and evaluation (e.g. the model's default system
prompt, the standard chat template
tokens, etc.). Overall, subliminal learning seems to be a fragile artifact of LoRA hyperparameters and finetuning context, making it an unstable channel for behavioral transmission.
Read the full abstract ▾
All authors Todd Nief, Harvey Yiyun Fu, Mark Muchane, Ari Holtzman
Machine Learning
Beyond Independent Manipulation: Individual Fairness-aware Strategic Classification with Peer Imitation
Xinpeng Lv, Chunyuan Zheng, Yunxin Mao et al. (+9 more)
Strategic classification (SC) investigates scenarios where
agents manipulate their features to obtain favorable decisions from predictive models. Existing
fairness-aware SC approaches primarily focus on group
fairness and typically assume that
agents respond independently. However, when individual
fairness is required, ensuring similar individuals receive similar outcomes,
agents' manipulation becomes interdependent: an
agent's preferred manipulation depends on the neighborhoods' outcomes. This induces a mismatch between classical SC formulations and
fairness-aware decision settings, where independent models no longer accurately characterize strategic manipulations. To address this issue, we introduce individual
fairness-aware strategic classification (IFSC), a framework that models peer-driven manipulation arising from individual
fairness, where
agents imitate nearby positively decided peers to obtain favorable outcomes. IFSC characterizes strategic manipulation as similarity-based imitation toward visible accepted peers and learns classifiers under the resulting post-manipulation distributions. To account for uncertainty in peer observability, IFSC employs a robust learning process that introduces
stochastic perturbations during manipulation simulation. Experiments on synthetic and real-world
datasets demonstrate that IFSC improves individual-
fairness consistency and mitigates imitation-induced distortions.
Read the full abstract ▾
All authors Xinpeng Lv, Chunyuan Zheng, Yunxin Mao, Renzhe Xu, Jinxuan Yang, Yuanlong Chen, Wangrong Huang, Shaowu Yang, Wenjing Yang, Xinwang Liu, Peng Cui, Haotian Wang
Machine Learning
Partial Fairness Awareness: Belief-Guided Strategic Mechanism for Strategic Agents
Xinpeng Lv, Chunyuan Zheng, Yunxin Mao et al. (+8 more)
Strategic machine learning investigates scenarios where
agents manipulate their features to receive favorable decisions from predictive models. To address
fairness concerns intrinsic to strategic classification, recent work has introduced group-specific
fairness constraints. However, current
fairness-aware approaches face a fundamental dilemma in the issue of
fairness exposure: making these constraints public enables strategic manipulation and can lead to
fairness reversal, while keeping them hidden may reduce social welfare and discourage genuine improvement. To fill this gap, we subsequently propose the problem of partial
fairness awareness (PFA), as our theoretical analysis informs that such a dilemma can be mitigated by releasing the candidate set of
fairness constraints and concealing the grounding constraint. To be specific, we introduce a belief-guided strategic mechanism, wherein
agents iteratively interact with the decision system and maintain a belief distribution over the candidate set of
fairness constraints. This belief-guided process enables
agents, through iterative interaction and feedback, to update their belief distribution over the candidate set, thereby gradually aligning their belief with the grounding
fairness constraint employed by the system. Extensive experiments on real-world and synthetic
datasets demonstrate that PFA achieves lower group
fairness gaps, higher acceptance of truly qualified individuals, and more stable outcomes compared to fully public or private
fairness regimes.
Read the full abstract ▾
All authors Xinpeng Lv, Chunyuan Zheng, Yunxin Mao, Renzhe Xu, Hao Zou, Shanzhi Gu, Liyang Xu, Huan Chen, Yuanlong Chen, Wenjing Yang, Haotian Wang
Machine Learning
A Comparative Analysis of Machine Learning Algorithms for Multi-Task Prediction of the Parameters of the Pectin Hydrolysis–Extraction Process
Mullosharaf K. Arabov, Shavkat Yo. Kholov and Zainiddin K. Muhiddin
This study addresses the challenge of controlling a complex, multi-parameter technological process – pectin hydrolysis–extraction – using machine learning methods. The experimental foundation is a unique database comprising 1,000 laboratory experiments conducted under controlled conditions on seven types of plant raw material with four variable process factors (temperature 85–130 C, pressure 0.9–2.2 atm, holding time 3–10 min, pH 1.5–2.0). Four output characteristics were recorded: pectin yield, galacturonic acid content, molecular weight, and degree of esterification. To solve the multi-task regression problem, 11
algorithms were trained and compared: regularised linear models, ensemble methods (Random Forest, Gradient Boosting, XGBoost, CatBoost, Extra Trees), k-nearest neighbours, support vector regression, and a multilayer perceptron. The best results were demonstrated by CatBoost (average R-squared approximately 0.946 after hyperparameter optimisation). Feature importance analysis revealed the dominant role of the raw material type (63.6% of total importance), followed by temperature and holding time. The developed pipeline was exported in a production-ready format and deployed as an interactive web interface. The findings demonstrate that ensemble methods combined with rigorous statistical analysis and interpretable AI significantly reduce the need for physical experiments and form the basis for intelligent pectin production control.
Read the full abstract ▾
Machine Learning
OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models
Ziling Lu, Zongsheng Li, Xinke Shen et al. (+11 more)
Electroencephalography (
EEG) supports a variety of
brain-computer interface (BCI) tasks ranging from
brain-state monitoring to human-LLM interactions.
EEG foundation models are emerging, but evaluation remains fragmented due to heterogeneous
datasets and nconsistent task protocols. Here, we introduce OmniEEG-Bench, a unified
benchmark and downstream task roadmap for
EEG foundation models (FMs). It organizes evaluation of
EEG FMs into six task families spanning (i) signal reliability, (ii) biometrics and disease, (iii) consciousness and state, (iv) cognition and
emotion, (v) naturalistic stimulus decoding, and (vi) motor and interaction, introducing a new generation of tasks not systematically benchmarked in prior
EEG FM work. OmniEEG-Bench standardizes model deployment, task definitions, and metrics through a task-card specification, and unifies 54
EEG datasets with consistent evaluation protocols. We
benchmark 10 representative
EEG foundation models and report a leaderboard that covers diverse evaluation settings. Both pretraining
dataset diversity and model size are significantly associated with better average ranks across
datasets, revealing scaling-law behavior in
EEG foundation models (Figure 1). These results suggest that scaling
EEG foundation models requires not only larger architectures but also broader and more diverse pretraining data. The
benchmark code is available at https://github.com/ncclab-sustech/omni-eegbench.git.
Read the full abstract ▾
All authors Ziling Lu, Zongsheng Li, Xinke Shen, Kexin Lou, Yingyue Xin, Xiaoqi Chen, Shinan Wang, Xiang Chen, Jiahao Fan, Chenyu Huang, Xin Xu, Zhoujie Hou, Chen Wei, Quanying Liu
Machine Learning · Cross-listed from cs.CR
Cross-Generational Transfer of Adversarial Attacks Reveals Non-Monotonic Safety Alignment in LLMs
Subhadip Mitra
Safety
alignment in LLMs does not improve monotonically across model generations. Studying four generations of Google's Gemma family (7B-31B) with quality-diversity
evolution (MAP-Elites) as an automated red-teaming probe, we find that Gemma 3 (12B) exhibits 68.7% +/- 5.7% attack success rate (ASR; mean +/- std, 3 seeds), significantly higher than its predecessor Gemma 2 (45.5% +/- 7.2%; p = 0.030, paired bootstrap) and its successor Gemma 4 (33.9% +/- 1.8%). Replaying evolved attack archives across generations reveals that attacks from other generations transfer to Gemma 3 at 44-46% but only 14-18% to Gemma 4, indicating that Gemma 4's safety gains generalize beyond the attack distributions evolved against earlier generations. Under our 8B judge, copyright and cybercrime vulnerabilities register at near-100% across all generations, though a second-judge audit (Section 6) suggests the copyright result is sensitive to judge choice. Misinformation ASR jumps from 29% to 99% between Gemma 2 and Gemma 3 and remains elevated at 77% in Gemma 4, indicating the regression was not fully addressed. These patterns are invisible to static
benchmarks and emerge only through adaptive, longitudinal probing. All experiments use 3 random seeds with a unified self-hosted judge; code and artifacts are available at https://github.com/bassrehab/red-queen.
Read the full abstract ▾
Machine Learning
Safe-Subspace Pseudo-Label Refinement for Source-Free Graph Domain Adaptation
Yingxu Wang, Xinwang Liu, Siyang Gao et al. (+1 more)
Source-free graph domain adaptation (SF-GDA) aims to adapt source-trained graph models to unlabeled target graphs when source graphs are no longer accessible. A central obstacle is pseudo-label reliability: under feature and topological shifts, source-induced predictions may become confidently wrong, and indiscriminate self-training can amplify systematic errors through graph message passing. This paper studies SF-GDA from a selective pseudo-labeling perspective. Instead of assuming globally bounded pseudo-label noise over the entire target domain, we identify a confidence-consistent safe subspace on which pseudo-label noise can be controlled under restricted posterior discrepancy, and derive a target-risk decomposition that separates safe-subspace fitting error, selected-label noise, and uncertain-set risk. Guided by this analysis, we propose SafeSubspace Pseudo-Label Refinement (S$^2$PLR), a source-free graph adaptation framework that applies hard pseudo-label supervision only to target graphs supported by both semantic and structural evidence. Specifically, S$^2$PLR estimates semantic reliability using source-committee confidence and disagreement, learns a targetintrinsic structural representation via graph contrastive learning, verifies pseudo-labels through neighborhood consistency, and exploits the remaining uncertain samples with noise-tolerant soft regularization rather than unreliable hard labels. Experiments on image and real-world graph
benchmarks under different domain shifts demonstrate that S$^2$PLR achieves robust and competitive performance across diverse source-free transfer settings.
Read the full abstract ▾
All authors Yingxu Wang, Xinwang Liu, Siyang Gao, Nan Yin
Machine Learning · Cross-listed from astro-ph.CO
Generative Diffusion Priors for 3D Mapping of the Dark Universe
Brandon Zhao, Diana Scognamiglio, Olivier Doré et al. (+1 more)
Reconstructing the three-dimensional distribution of dark matter from weak-
lensing observations is a central but highly ill-posed inverse problem in cosmology. Unlike standard 3D reconstruction with multiple viewpoints, we observe the universe from a single line of sight, through noisy shape distortions of
galaxies with uncertain distances, so meaningful recovery of the 3D matter field requires strong prior assumptions. Existing methods either produce point estimates with handcrafted priors or use neural ensembles for approximate
Bayesian uncertainty, and struggle to capture the non-Gaussian, filamentary structure of the
cosmic web. With the advent of new high-resolution cosmological simulations, we now have an alternative source of prior knowledge that captures the nonlinear statistics of structure formation with far greater fidelity than analytic prescriptions. We leverage these simulations to build a new
dataset $\texttt{Conicus3D}$, which enables us to learn a data-driven
diffusion-model prior capturing the full 3D distribution of dark matter structure across
cosmic time. Building on recent plug-and-play approaches, we modify a
diffusion-based posterior sampling scheme to the 3D weak-
lensing setting, combining the learned prior with a differentiable physical forward model. On realistic simulations targeting a modern weak
lensing survey, our approach yields substantially improved 2D and 3D reconstruction accuracy over baseline methods. Moreover, it produces posterior samples whose statistics closely track the underlying simulations, while remaining robust to moderate shifts in cosmology.
Read the full abstract ▾
All authors Brandon Zhao, Diana Scognamiglio, Olivier Doré, Katherine L. Bouman
Machine Learning · Cross-listed from cs.CR
Quality-Diversity Evolution for Discovering Diverse Vulnerabilities in LLM Safety
Subhadip Mitra
Current approaches to LLM
adversarial testing suffer from coverage gaps: manual red-teaming does not scale, LLM-as-attacker methods exhibit mode collapse, and gradient-based approaches produce uninterpretable gibberish. We introduce a quality-diversity evolutionary framework that operates at the semantic level, evolving interpretable attack strategies rather than
token sequences. Using MAP-Elites, we maintain a diverse archive of attacks across behavioral dimensions (strategy type, encoding method,
prompt length). In experiments across GPT-4o-mini, Claude 3.5 Sonnet, Gemini 2.0 Flash, and an open-weight coding model (Devstral-small-2), we discover distinct vulnerability profiles: GPT-4o-mini is vulnerable to hypothetical and multi-turn framing combined with ROT13 encoding (fitness 0.8), Gemini to direct attacks with ROT13 and multi-turn with Leetspeak (0.8), while Claude shows uniformly ambiguous responses across all strategies (max 0.4). The semantic representation produces interpretable attacks that reveal systematic, model-specific weaknesses, providing actionable insights for improving LLM safety and a reproducible baseline for evaluating future frontier models. Code and experiment artifacts are released at https://github.com/bassrehab/red-queen.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models
Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain et al. (+1 more)
Parameter compression of class-conditional
diffusion models exposes a structural limitation in output-level
distillation: supervising only the guided output leaves the two score branches non-identifiable, so the classifier-free guidance gap is not determined in the student. The zero-loss set admits degenerate solutions in which both branches collapse toward identical predictions, and guidance loses its effect at inference despite low training loss. For each fixed input the objective is exactly flat in a branch-output direction at every residual value, so the ambiguity is a property of the output-level objective rather than a poor local minimum. This paper introduces DASH, which supervises the conditional and unconditional branches independently. An anchor term regularises the conditional prediction toward ground-truth noise, and the teacher's final learned per-timestep curriculum transfers into the student as a frozen prior. Across CIFAR-10, CIFAR-100, and class-conditional ImageNet-64, a more than 5x compressed student stays within four FID points of its teacher and recovers at least 89% of the teacher's guidance-gap magnitude, where no other two-branch baseline with defined calibration exceeds 82%. Ablation isolates unconditional supervision as the term that separates this formulation from output-level
distillation. The account is quantitative: the null space fixes the ratio between a composite student's two branch errors, and the measured ratios approximately match that prediction on all three
datasets.
Read the full abstract ▾
All authors Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain, Engelbert Mephu Nguifo
Machine Learning
Extending Causal Metamodeling to a non-Markovian Queue
Pracheta Amaranath, Anant Bhide, David Jensen et al. (+1 more)
Metamodels for discrete-event simulations approximate the behavior of simulation models without running expensive simulations. Prior work introduced modular dynamic
Bayesian networks (MDBNs) – a class of metamodels that can estimate a range of probabilistic and causal queries (PCQs) using a single, trained model – but the method was limited to Markovian systems. In this paper, we initiate an extension of MDBNs to non-Markovian queues by approximating non-exponential distributions using phase-type distributions. This approach raises novel challenges, including balancing metamodeling accuracy and tractability when choosing the number of phases, efficiently learning metamodel parameters, and choosing the sampling interval that is used to approximate a continuous-time simulation by a discrete-time MDBN. We provide preliminary solutions to these challenges, yielding the first causal metamodeling technique for non-Markovian systems. Experiments on a G/M/1 queue demonstrate that the MDBN can produce accurate answers to PCQs with orders-of-magnitude speedup of inference times relative to direct simulation.
Read the full abstract ▾
All authors Pracheta Amaranath, Anant Bhide, David Jensen, Peter Haas
Machine Learning · Cross-listed from cond-mat.mtrl-sci
Pavlo Melnyk, Anmar Karmush, Mårten Wadenbäck et al. (+4 more)
Merging first-principles calculations with machine learning (ML), we aim to accelerate the exploration of catalytic behaviour in novel materials. We focus on two-dimensional (2D) Ti$_2$CT$_y$ MXenes, whose versatile surface chemistry makes them particularly compelling candidates for catalysis. Resolving their composition and structure under realistic conditions exceeds the reach of standard density functional theory (DFT) due to computational cost. To address this challenge, we generate a comprehensive
dataset of 50,000 DFT calculations for training and 10,000 for testing, encompassing both Ti$_2$CT$_y$ MXene configurations and molecular systems, along with an additional test
dataset with 1000 genuinely new, larger systems to investigate how well models generalise. We train and validate widely used and competitive machine learning interatomic potential (MLIP) models, including EquiformerV2, MACE, MatRIS, and UPET, that accurately predict atomic forces and formation energies – quantities that DFT must repeatedly compute for structural and catalytic investigations – for these 2D materials. This combined DFT-ML framework achieves computational acceleration on the order of approximately $1-4 \cdot 10^3$ (on a CPU) while maintaining desired-level accuracy (approximately +/- $10$ meV/A for forces and approximately +/- $1$ meV for per-atom energies), paving the way for more efficient investigations of MXene catalytic behaviour. Moreover, we perform an extensive qualitative evaluation of the trained models, showcasing the importance of comprehensive simulation-based comparison beyond
benchmark metrics. The
dataset and the trained models with the code are available at https://huggingface.co/
datasets/CatalystAnonymous/catalyst_mxenes.
Read the full abstract ▾
All authors Pavlo Melnyk, Anmar Karmush, Mårten Wadenbäck, Ania Beatriz Rodríguez-Barrera, Johanna Rosen, Michael Felsberg, Jonas Björk
Machine Learning
Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-Reinforcement Learning
Fuyuan Qian, Menglong Zhang, Song Wang et al. (+1 more)
Offline meta-reinforcement learning leverages static
datasets to enable
agents to generalize to unseen environments by combining offline efficiency with meta-learning adaptability, yet it faces key challenges from context and policy distribution shifts. These issues hinder
agents from adapting to online environments, and are further exacerbated under sparse-reward settings. As a result,
agents often become trapped in an inherent pattern dilemma, failing to achieve robust generalization. In this work, we propose a novel framework that integrates information-theoretic task representation learning with a
Transformer-based
stochastic world model. Our approach extracts task-defining latent variables that are invariant to behavior policy, thereby effectively mitigating the context distribution shift. To further handle policy shift and model exploitation, we apply a conservative value penalty to imagination-based rollouts, preventing the policy from exploiting model inaccuracies while maintaining robust adaptation. Extensive evaluations demonstrate that our method outperforms state-of-the-art approaches, with superior stability and generalization under out-of-distribution and sparse-reward settings.
Read the full abstract ▾
All authors Fuyuan Qian, Menglong Zhang, Song Wang, Quanying Liu
Machine Learning
Latent Diffusion Pretraining for Crystal Property Prediction
Shrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri et al. (+2 more)
Fast and accurate prediction of
crystal properties is a central challenge in new materials design. Graph neural networks and
Transformer-based models have emerged as powerful tools for this task due to their ability to encode the local structural environment of atoms within a
crystal. However, these models are data-hungry, and in practice, labeled data for
crystal properties are scarce. Pretraining-finetuning strategies, particularly those based on
diffusion models, have shown promise in addressing these limitations. In this work, we introduce a novel latent
diffusion based pretraining framework, CrysLDNet, designed to mitigate data scarcity. Our approach integrates a Variational Autoencoder (VAE) with a
diffusion model during the pretraining stage. The VAE encoder maps 3D
crystal structures into a smooth latent space within which the
diffusion process is applied. This latent
diffusion pretraining enables the graph encoder to effectively capture structural and chemical semantics from large-scale unlabeled data, which can then be finetuned for specific property prediction tasks. Comprehensive experiments on popular DFT
datasets for property prediction reveal that CrysLDNet significantly outperforms both training-from-scratch and pretrained baselines, with improvements of 4.26% and 4.90% on the JARVIS and MP
datasets, respectively. Additionally, the learned representations remain robust in sparse-data conditions and are expressive enough to correct DFT errors when finetuned with limited experimental data. Code is available at: https://github.com/shrimonmuke0202/CrysLDNet.git.
Read the full abstract ▾
All authors Shrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri, Pawan Goyal, Niloy Ganguly
Machine Learning
Yiru Yang, Junling Wang, Nishant Kumar Singh et al. (+2 more)
A simple way to improve the performance of almost any machine learning model is not to train a single but several models with diverse
algorithms which will make slightly distinct kinds of predictions and errors on the same data, and thus improve the average predictions and robustness. However, making predictions using a whole ensemble of models is cumbersome and computationally too expensive to allow deployment to a large number of users, especially if the models are large neural nets. In response to this, we introduce a layer and point wise projection mapping, which maps student and teacher representations into an aligned high-dimensional
embedding space during training process. The proposed approach combined with LoRA injection reduces the student model trainable parameters to less than 1% of the teacher model, while significantly improving word error rate (WER) compared to other
distillation methods, as demonstrated in ablation studies. Unlike a mixture of experts, our method can be trained rapidly and in parallel.
Read the full abstract ▾
All authors Yiru Yang, Junling Wang, Nishant Kumar Singh, Luohong Wu, Haoran Yan
Machine Learning
Confidence-Adaptive SwiGLU for Mixture-of-Experts
Shaohua Li, Xiuchao Sui, Xiaobing Sun et al. (+4 more)
SwiGLU has become a standard gated activation in modern
Transformer MLPs, yet its gate sharpness – the smoothness and selectivity of the gating function – is typically fixed throughout training. In this work, we propose Confidence-Aware SwiGLU ($\kappa$-SwiGLU), a variant of SwiGLU for Mixture-of-Experts (MoE) models that adjusts expert gate sharpness according to
token-level routing confidence. Specifically, $\kappa$-SwiGLU parameterizes the SiLU gate sharpness coefficient as a learnable function of the router logit, enabling each expert gate unit to interpolate between smooth, broadly active gating and sharp, selective gating. We evaluate $\kappa$-SwiGLU on the FineWeb-Edu
dataset across MoE
Transformer models ranging from 8 to 28 layers. Across these settings, $\kappa$-SwiGLU improves mean CORE performance while adding negligible parameters and incurring only a small computational overhead, demonstrating that confidence-aware gate sharpness is a promising mechanism for improving MoE MLPs. The code is available at https://github.com/askerlee/kappa-swiglu.
Read the full abstract ▾
All authors Shaohua Li, Xiuchao Sui, Xiaobing Sun, Yuhang Wu, Liangli Zhen, Yong Liu, Rick Siow Mong Goh
Machine Learning
Distributed GNEP Algorithms without Multiplier Sharing and Applications to Multi-Robot Coordination and Contextual Bandit-Based Active Learning
Shao-An Yin
Recent advances in artificial intelligence have expanded the focus from classical optimization to include
equilibrium analysis in noncooperative games. Many such games involve shared constraints, leading to Generalized Nash
Equilibrium Problems (GNEPs). Existing distributed
algorithms typically require
agents to exchange Lagrange multipliers to enforce consensus and compute variational-GNEs (v-GNEs). This work introduces fully distributed continuous-time
algorithms and establishes convergence without requiring multiplier exchange, thereby reducing information exchange per iteration while improving
privacy preservation. The analysis focuses on strongly monotone games with convex individual constraints and linear shared constraints. I also propose several discretization schemes for the continuous-time
algorithms. The proposed approach converges to general GNEs, rather than being restricted to v-GNEs, with the attained
equilibrium depending on the initialization. The effectiveness of the proposed method is demonstrated through applications in multi-
robot coordination and placement. In the second part, this work includes research conducted in collaboration with Amazon scientists. One of the most challenging problems in real-world machine learning is labeled data collection, which typically requires substantial human effort and cost. Active learning aims to reduce this labeling requirement. Existing handcrafted active learning strategies, however, generally perform well only on specific types of
datasets, which are often unknown in advance. In this work, I propose using contextual bandits to adaptively select the most suitable active learning strategy. The effectiveness of the proposed approach is demonstrated on publicly available external
datasets.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Statistical Testing on Directed Graphs by Surrogate Data Generation
Chun Hei Michael Chan, Alexandre Cionca and Dimitri Van De Ville
In recent years, graph signal processing has emerged as a powerful framework at the intersection of signal processing and graph theory, providing tools for the analysis of signals defined on nodes while accounting for their relationships represented by edges. These tools have been successfully applied to various settings, including statistical hypothesis testing. In particular, non-parametric approaches based on surrogate generation have been proposed for signals on undirected graphs. However, they are yet to be extended to directed graphs. In this work, we first revisit the notion of stationary graph signals on directed graphs. Specifically, and through the eigendecomposition of the graph shift operator, we define directed graph wide-sense stationary signals. Then, we propose a new framework to generate surrogate graph signals that preserve covariance structure under stationarity assumptions. Null distributions of the test metric can then be constructed from these surrogates and serve as a reference for the empirical data. Finally, we provide guiding examples and an application on real data, in which we compare the performance of our framework with existing techniques for undirected graphs or based on naive permutation, demonstrating feasibility and superiority of the proposed approach.
Read the full abstract ▾
Machine Learning
RADE: Random Add-Drop Edge as a Regularizer
Danial Saber and Amirali Salehi-Abari
Graph Neural Networks (GNNs) suffer from overfitting and over-squashing of long-range information.
Stochastic graph augmentations (e.g., edge deletion) regularize training against overfitting but can introduce train-inference misalignment and do not improve over-squashing. In contrast, rewiring methods improve connectivity to mitigate over-squashing, but are not designed to regularize training. We propose Random Add-Drop Edge (RADE), a
stochastic graph augmentation method that jointly drops and adds edges to address both overfitting and over-squashing simultaneously. RADE is provably designed to align training and inference so that random augmentations regularize training without distribution shift, while supporting long-range communication at inference. We further propose and study a mini-batch gradient-norm balancing
algorithm that adapts deletion and addition rates during training, rendering RADE hyperparameter-free in practice. Experiments on node- and graph-classification
benchmarks show that RADE is a strong regularizer and mitigates over-squashing. Ablations support the roles of train-inference
alignment, adaptive rate selection, and the complementary effects of random edge deletion and edge addition.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning
Xuewei Yang, Jiachen Yu, Jie Wu et al. (+3 more)
Reinforcement learning from verifiable rewards improves the
reasoning ability of large
language models, but often suffers from
entropy collapse, in which increasingly concentrated policies reduce rollout diversity and useful learning signals. Existing remedies either constrain the RL objective (e.g.,
entropy regularization) or adjust sampling temperature during rollout collection, but these interventions remain external to the model parameters. We propose Temperature-Scaled On-Policy Self-
Distillation (TS-OPSD), a lightweight policy reheating method that internalizes the exploratory effect of temperature into model parameters. Starting from an
entropy-collapsed RL checkpoint, TS-OPSD constructs a self-teacher by applying high-temperature scaling to the model's own logits, then distills the resulting smoother distribution back into the student. This policy reheating requires no external teacher, privileged data, or additional inference cost. Experiments on Qwen3-4B-Base and Qwen3-8B-Base show that policy reheating yields a stronger initialization for continued RL than both standard continued RL and rollout-level temperature reheating. Further analyses show that TS-OPSD mainly reduces output sharpness while preserving intermediate representations, top candidate sets, and
reasoning capability. These results suggest that
entropy restoration can serve as a simple post-collapse intervention for extending
reasoning-oriented RL.
Read the full abstract ▾
All authors Xuewei Yang, Jiachen Yu, Jie Wu, Shaoning Sun, Junjie Wang, Yujiu Yang
Machine Learning · Cross-listed from stat.ME
Causal Density Functions
Sridhar Mahadevan
We study the full density ratio between a specified intervention regime $P_a$ and an observational regime $P_0$, $\rho_a=dP_a/dP_0$, under the prerequisite $P_a\ll P_0$. We call the regime-indexed ratio a causal density function when $P_a$ is an identified or directly observed interventional law. The underlying Radon-Nikodym derivative and the identity $ \mathbb{E}_{a}[f(Z)] = \mathbb{E}_{0}\!\left[f(Z)\rho_a(Z)\right] $ are classical importance weighting, not new identification results. Our narrower question is whether retaining the entire pointwise ratio is useful as a reusable diagnostic across several downstream functionals. We evaluate a two-density plug-in baseline through held-out moment transport and overlap stress tests on synthetic and perturbation data. We also report a pairwise graph-scoring heuristic as a negative result: its F1 is \(0.10\) on a synthetic DAG, \(0.12\) on Sachs, and \(0.33\) on a multi-regime chain. These experiments do not establish an estimation advantage over direct density-ratio, inverse-probability, Riesz, or doubly robust methods; they instead delimit what the pointwise ratio target and the present plug-in
estimator do and do not provide.
Read the full abstract ▾
Machine Learning
A multimodal dataset of photoplethysmography and continuous behavioral responses to ASMR and nature videos
Tushar Das, Daigo Hozaki, Koushlendra Kumar Singh et al. (+1 more)
Autonomous Sensory Meridian Response (ASMR) is a somatosensory phenomenon characterized by pleasant tingling sensations and cardiovascular slowing. However, ASMR research has been hindered by a dearth of standardized, open-access
multimodal datasets. To address this limitation, we present REST-ASMR (Response to Environmental & Sensory Triggers), a synchronized
multimodal dataset designed to capture behavioral reports and physiological dynamics during ASMR, with nature-relaxation
videos as control stimuli. The
dataset includes high-resolution photoplethysmography (PPG), time-aligned audiovisual stimuli, and continuous subjective annotations from 34 participants. Technical validation showed high stimulus efficacy (97% responder rate), significant stimulus-specific inter-subject agreement (p < 0.05), and a robust PPG-derived ASMR-specific cardiovascular deceleration. Additionally, a Bidirectional Long-Short Term
Memory model successfully predicted subjective ASMR tingle states, achieving
video-level ASMR vs. Nature classification with perfect accuracy and a frame-level global mean accuracy of 75.51%, macro F1-score of 71.86%, and 100% Nature-baseline specificity, under a strict, leakage-free subject-
video double-independent 4-fold cross-validation. REST-ASMR constitutes a dense temporal foundation for affective computing,
multimodal research, and the development of personalized models of relaxation-related responses.
Read the full abstract ▾
All authors Tushar Das, Daigo Hozaki, Koushlendra Kumar Singh, Hirohito M. Kondo
Machine Learning · Cross-listed from cs.CV
Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders
Yitong Jiang, Hongjun Wang, Collin McCarthy et al. (+15 more)
Vision foundation models are bottlenecked by the quadratic cost of self-
attention, which limits usable resolution and increases the cost of large-scale pretraining. Subquadratic alternatives such as linear
attention and state-space models reduce this cost, but often serialize images into 1D
token streams and weaken the 2D spatial structure important for
vision. Generalized Spatial Propagation Networks (GSPN) instead propagate context directly on the 2D grid through line-scan recurrences, achieving near-linear complexity without positional
embeddings, but have seen little use as foundation-scale encoders. We present C-GSPN, a foundation-scale
vision encoder based on 2D spatial propagation. C-GSPN makes the operator practical through three improvements: (1) a fast GSPN CUDA kernel that fuses per-step launches into a single warp-specialized implementation with shared-
memory tiling, coalesced access, and a compact multi-channel propagation, reaching over 90% of peak
memory bandwidth and running up to 40–52x faster than the original GSPN implementation; (2) a compressed latent-space propagation block with fused normalization, which turns kernel-level speed into block- and model-level efficiency; and (3) a two-stage cross-operator
distillation recipe that trains the new architecture from an
attention teacher without the cost of from-scratch foundation-scale training. Distilled with 600M image-text pairs, C-GSPN matches an isomorphic ViT baseline with 15% fewer parameters, improves ADE20K
segmentation by +2.1%, transfers to high resolution with a fraction of the data needed from scratch, and delivers a 4x end-to-end block speedup at 2K with single-pass, tiling-free inference.
Read the full abstract ▾
All authors Yitong Jiang, Hongjun Wang, Collin McCarthy, Hanrong Ye, David Wehr, Xinhao Li, Qi Dou, Tianfan Xue, Ka Chun Cheung, Simon See, Wonmin Byeon, Ke Chen, Kai Han, Jinwei Gu, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Sifei Liu
Machine Learning
Quantum Tunneling-Aware Machine Learning: Physics-Derived Noise Models for Robust Deployment
Uiwon Hwang and Jaeho Hwang
Transistor scaling is approaching a
quantum-mechanical limit, as thin gate
oxides induce electron leakage through
quantum tunneling. Unlike conventional digital systems, AI inference can tolerate such errors provided their structure is modeled correctly. In this paper, we introduce
quantum tunneling-aware machine learning (QTAML). We derive the deployment-time weight-error distribution from first principles using the Wentzel-Kramers-Brillouin (WKB) approximation and show that it has structure that generic Gaussian noise models miss: an exact affine mean drift, a per-bit variance hierarchy dominated by the most-significant bit, and a per-layer dependence on $\|W_\ell\|_\infty$ and the trained-network Jacobian. We package these three structural properties into a single deployment-time
algorithm, Tunneling-Aware Compensation (TAC), that combines closed-form mean correction with an optimal layer-adaptive bit-budget allocation derived from the WKB variance decomposition. Across four convolutional architectures at $p_\mathrm{flip}$=0.10 and a
transformer encoder at $p_\mathrm{flip}$=0.05, TAC reaches $95\%$ of clean accuracy with 3.4$\times$ to 33.6$\times$ less ECC overhead than Uniform-MSP, the natural baseline derived from the same physics. The closed-form saturation ratio $\rho^*$ predicts these gains in advance, and on heterogeneous architectures WKB-derived scoring outperforms magnitude-based allocation by up to 24 percentage points at small budgets. The
algorithm requires no retraining, no labels, and no inference-time overhead. We also verify the WKB-derived distributional
theorems to Monte Carlo precision. These results connect WKB tunneling physics with noise-aware deep learning and suggest a principled path toward hardware–software co-design beyond conventional scaling limits.
Read the full abstract ▾
Machine Learning
Score $\times$ Decoder: A Unified View of Unsupervised Inference-Time Scaling for Hallucination Mitigation
Yun-Chen Cheng, Che-Yu Lin and Cheng-Lin Yang
Large
language models hallucinate even when the answer lies within their parameters. While inference-time scaling can surface this latent knowledge, the most effective methods require supervision: a trained verifier or reward model. We ask what can be done with only a base
language model: which intrinsic signal best identifies correct outputs, and how should it be decoded? We cast this as a score~$\times$~decoder grid pairing four scores (perplexity, contrastive, power-distribution likelihood, and self-verification) with three decoding families (optimization, sampling, consensus), and evaluate every
cell on MATH500 with the base and instruction-tuned Qwen3-1.7B. While self-verification, which
prompts the model to judge its own answer and is sharpened by a training-free virtual-thinking prefix, works well in most settings, no score has a fixed quality: its value depends on the decoder that consumes it and on model capability. When no supervision is available, the score and the decoding family must be chosen together.
Read the full abstract ▾
Machine Learning
SORA: Free Second-Order Attacks in Fast Adversarial Training
Mazdak Teymourian, Ramtin Moslemi, Farzan Rahmani et al. (+1 more)
Adversarial Training (AT) is a leading defense against
adversarial examples but often suffers from Catastrophic Overfitting (CO) in efficient single-step variants, where robustness to multi-step attacks collapses despite high single-step performance. We address this failure mode with two contributions. First, we formalize Epsilon Overfitting (EO), a perspective in which fixed perturbation magnitudes and directions exacerbate CO, and show that introducing perturbation variability significantly improves robust generalization across different architectures and
datasets. Second, we propose PertAlign (Perturbation
Alignment), a theoretically grounded, computationally negligible metric that predicts CO onset by measuring gradient
alignment across attack stages. Leveraging these insights, we introduce SORA, an adaptive step-size AT method that dynamically adjusts perturbations based on loss surface geometry. SORA consistently prevents CO, achieves state-of-the-art robustness and clean accuracy, and generalizes across
datasets and architectures using a single fixed set of hyperparameters, which is essential for applicability in fast AT. Extensive experiments on diverse
datasets and architectures show that SORA matches or surpasses the robustness of prior methods while delivering higher clean accuracy and superior efficiency. Code is available at https://github.com/SecondOrderAT/SORA.
Read the full abstract ▾
All authors Mazdak Teymourian, Ramtin Moslemi, Farzan Rahmani, Mohammad Hossein Rohban
Machine Learning · Cross-listed from cs.DC
ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
Seokjin Go, Marko Scrbak, Ephrem Wu et al. (+2 more)
In distributed Mixture-of-Experts (MoE) inference, input-dependent
token routing interacts with GPU performance variability to create persistent stragglers under synchronized execution, where the slowest GPU determines layer latency. This performance variability is inherent to modern accelerators: manufacturing variation, power limits, and thermal conditions introduce measurable execution-time differences across nominally identical GPUs. The core challenge is that MoE execution-time imbalance arises from the interaction of workload skew and hardware asymmetry.
Token routing produces uneven and layer-varying expert loads, while GPU throughput depends on device-specific operating characteristics and workload intensity. Prior work mitigates routing skew but assumes homogeneous hardware, optimizing
token balance rather than execution latency. As a result, even balanced
token assignments can leave hardware-induced stragglers unaddressed. Thus, we propose Variability-Informed Binning of Experts (ViBE), a hardware-aware expert placement framework that minimizes execution-time imbalance across GPUs. ViBE combines per-GPU performance modeling with expert activation profiling to assign high-load experts to faster devices and low-load experts to slower ones, reducing layer-level stragglers without modifying model semantics or hardware. Because both workload characteristics and effective GPU throughput can shift across serving conditions, ViBE supports lightweight recalibration under workload/performance drift to refresh its routing and performance estimates when needed. Results show that ViBE consistently reduces execution-time imbalance and improves SLO attainment by 14%, while lowering P90 TTFT by up to 45%. We further show that the impact of hardware variability increases at scale, making variability-aware placement important for efficient, high-utilization LLM serving.
Read the full abstract ▾
All authors Seokjin Go, Marko Scrbak, Ephrem Wu, Srilatha Manne, Divya Mahajan
Machine Learning · Cross-listed from cs.AI
SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition
Jayanta Dey, Shikhar Srivastava, Itamar Lerner et al. (+2 more)
Learning long-range non-stationary temporal patterns remains a core challenge for modern sequence models, particularly in strict streaming settings. In these settings, data arrive sequentially and must be processed in a single pass without simultaneously revisiting past observations. Standard architectures, including recurrent neural networks and
transformers, are constrained by either truncated backpropagation through time horizon or explicit input window length for long range credit assignment. To address these limitations, we propose SHARP (Sleep-based Hierarchical Accelerated Replay), a framework that decomposes temporal learning into two complementary components: a
memory module that accumulates a structured history of past inputs, and a pattern-recognition module that operates over this
memory. This separation enables resource- and compute-efficient adaptation to non-stationary dynamics by eliminating the need for backpropagation through time across many steps for long-range credit assignment. Inspired by the accelerated replay observed in rodents during slow-wave sleep, SHARP incorporates offline (sleep) phases in which temporally structured
memory traces are replayed in an accelerated form and integrated into higher-level
memory representations, improving long-range context retention. Through controlled simulations and ablation studies, we characterize the key properties of the proposed framework. In
benchmark datasets such as text8 and PG-19, we demonstrate that SHARP improves over recurrent baselines by retaining next-
token predictive performance on previously seen data while continuing to learn from the current stream and generalizing to future unseen data. These gains are enabled by its hierarchical structure, which yields an exponentially increasing effective temporal context with only linear-time computational cost.
Read the full abstract ▾
All authors Jayanta Dey, Shikhar Srivastava, Itamar Lerner, Christopher Kanan, Dhireesha Kudithipudi