Machine Learning · Cross-listed from cs.CL
Selective Augmentation: Improving Universal Automatic Phonetic Transcription via G2P Bootstrapping
Tobias Bystrich, Julia M. Pritzen, Christoph A. Schmidt et al. (+1 more)
In the field of universal automatic phonetic transcription (APT), clean and diverse training transcriptions are required. However, such high-quality data is limited. We propose the bootstrapping approach Selective Augmentation to improve the available training transcriptions by selectively transferring distinctions between
languages. Based on the model MultIPA, we exemplarily show that we could increase the accuracy of an existing feature (plosive voicing) and add a new feature (plosive aspiration) by augmenting the existing training data using information from a separate helper
language (Hindi). We describe intrinsic challenges of the evaluation and develop objective metrics to determine the success: Voicing accuracy was increased by 17.6% by reducing the number of false positives. Additionally, aspiration recognition was introduced: While the baseline transcribed 0% of German /p, t, k/ as aspirated, our approach transcribed them as aspirated in 61.2% of the cases. Introducing aspiration recognition to APT models allowed for the tenuis class to be successfully reduced by 32.2%, which also reduces the conflations between the test
language's plosives.
Read the full abstract ▾
All authors Tobias Bystrich, Julia M. Pritzen, Christoph A. Schmidt, Claudia Wich-Reif
Machine Learning · Cross-listed from cs.CL
Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation
Shouren Wang, Wang Yang, Chuang Ma et al. (+7 more)
Hybrid-thinking
language models expose explicit /think and /no_think modes, but current designs do not separate them cleanly. Even in /no_think mode, models often emit long and self-reflective responses, causing
reasoning leakage. Existing work reduces this issue through better data curation and multi-stage training, yet leakage remains because both modes are still encoded in the same feed-forward parameters. We propose Path-Lock Expert (PLE), an architecture-level solution that replaces the single MLP in each decoder layer with two semantically locked experts, one for /think and one for /no_think, while keeping
attention,
embeddings, normalization, and the
language-model head shared. A deterministic control-
token router selects exactly one expert path for the entire sequence, so inference preserves the dense model's per-
token computation pattern and each expert receives mode-pure updates during supervised fine-tuning. Across math and science
reasoning benchmarks, PLE maintains strong /think performance while producing a substantially stronger mode separation, a /no_think mode with higher accuracy and far less
reasoning leakage. On Qwen3-4B, for example, compared to the SFT-only baseline on AIME24, PLE generates 17x fewer reflective
tokens (6.01 vs. 0.35 per answer) and 2x shorter outputs (8665 vs. 4101
tokens), and improves /no_think accuracy from 35.33% to 44.67%, while maintaining /think-mode performance (61.33% vs. 60.00%). These results suggest that controllable hybrid thinking is fundamentally an architectural problem, and separating mode-specific feed-forward pathways is a simple and effective solution.
Read the full abstract ▾
All authors Shouren Wang, Wang Yang, Chuang Ma, Debargha Ganguly, Vikash Singh, Chaoda Song, Xinpeng Li, Xianxuan Long, Vipin Chaudhary, Xiaotian Han
Machine Learning · Cross-listed from stat.ME
Linear Models, Variable Selection, Artificial Intelligence
By Riyadh Alrawkan, Edward Boone, Ryad Ghanam et al. (+1 more)
Variable selection in linear regression models has been a problem since hypothesis testing began. Which variables to include or exclude from a model is not an easy task. Techniques such as Forward, Back ward, Stepwise Regression sequentially add or delete variables from a model. Penalized likelihood methods such as AIC, BIC, etc. seek to choose variables that have a significant contribution to the likelihood. Penalized sum of square methods such as LASSO and Elastic Net have been used to penalize small coefficients to only allow variables with large coefficients in the model. This work introduces an Artificial Intelligence approach to model selection where an ANN is trained to determine the significance of the variables based on OLS estimates. A simulation study shows the accuracy across various sample sizes and variances. Furthermore, a simulation study is conducted to compare the performance of the approach against Forward, Backward, AIC, BIC and LASSO. The approach is illustrated using a
dataset from the World Health Organization regarding Life Expectancy. A github link is provided to the pretrained ANN that can handle up to 100 predictor variables, the original WHO
dataset and the subset used in this work.
Read the full abstract ▾
All authors By Riyadh Alrawkan, Edward Boone, Ryad Ghanam, Anton Westveld
Machine Learning · Cross-listed from eess.SY
Learning to Spend: Model Predictive Control for Budgeting under Non-Stationary Returns
Nilavra Pathak, Smriti Shyamal, Prasant Mhasker et al. (+1 more)
We study finite-horizon budget allocation as a closed-loop economic control problem and evaluate receding-horizon Model Predictive Control (MPC) relative to reactive budgeting policies. Budgets are allocated periodically under execution noise and operational constraints, while return efficiency may evolve over time. Using a controlled simulation framework motivated by digital marketing, we compare reactive pacing to MPC across environments with increasing degrees of non-stationarity. Our results show that non-stationarity alone does not justify predictive control. When return dynamics are stationary or evolve through unpredictable
stochastic drift, MPC offers no systematic advantage over reactive baselines. By contrast, when return efficiency exhibits predictable structure over the planning horizon, that is captured through an underlying model, MPC consistently outperforms reactive budgeting by exploiting intertemporal trade-offs.
Read the full abstract ▾
All authors Nilavra Pathak, Smriti Shyamal, Prasant Mhasker, Christopher Swartz
Machine Learning
Preserving Temporal Dynamics in Time Series Generation
Ci Lin, Futong Li, Tet Yeap et al. (+1 more)
Time-series data augmentation plays a crucial role in regression-oriented
forecasting tasks, where limited data restricts the performance of deep learning models. While Generative
Adversarial Networks (GANs) have shown promise in synthetic time-series generation, existing approaches primarily focus on matching marginal data distributions and often overlook the temporal dynamics that naturally exist in the original multivariate time series. When generating multivariate time series, this mismatch leads to distribution shift and temporal drift, thereby degrading the fidelity of the synthetic sequences. In this work, we propose a model-agnostic Markov Chain Monte Carlo (MCMC)-based framework to mitigate distribution shift and preserve temporal dynamics in synthetic time series. We provide a theoretical analysis of how conditional generative models accumulate deviations under sequential generation and demonstrate that the MCMC
algorithm can correct these discrepancies by enforcing consistency with empirical transition statistics between neighboring time points. Extensive experiments on the Lorenz, Licor, ETTh, and ILI
datasets using RCGAN, GCWGAN, TimeGAN, SigCWGAN, and AECGAN demonstrate that the proposed MCMC framework consistently improves autocorrelation
alignment, skewness error, kurtosis error, R$^2$, discriminative score, and predictive score. These results suggest that synthetic time series consistent with the original data require explicit preservation of transition laws rather than solely relying on
adversarial distribution matching, thereby offering a principled direction for improving generative modeling of time-series data.
Read the full abstract ▾
All authors Ci Lin, Futong Li, Tet Yeap, Iluju Kiringa
Machine Learning
Context-Aware Graph Attention for Unsupervised Telco Anomaly Detection
Sara Malacarne, Eirik Hoel-Høiseth, Erlend Aune et al. (+2 more)
We propose C-MTAD-GAT, an unsupervised, context-aware graph-
attention model for anomaly detection in multivariate time series from mobile networks. C-MTAD-GAT combines graph
attention with lightweight context
embeddings, and uses a deterministic reconstruction head and multi-step forecaster to produce anomaly scores. Detection thresholds are calibrated without labels from validation residuals, keeping the pipeline fully unsupervised. On the public TELCO
dataset, C-MTAD-GAT consistently outperforms MTAD-GAT and the Telco-specific DC-VAE, two state-of-the-art baselines, in both event-level and pointwise F1, while triggering substantially fewer alarms. C-MTAD-GAT is also deployed in the Core network of a national mobile operator, demonstrating its resilience in real industrial settings.
Read the full abstract ▾
All authors Sara Malacarne, Eirik Hoel-Høiseth, Erlend Aune, David Zsolt Biro, Massimiliano Ruocco
Machine Learning · Cross-listed from cs.CL
Semantic Structure of Feature Space in Large Language Models
Austin C. Kozlowski and Andrei Boutyline
We show that the
geometric relations between semantic features in large
language models' hidden states closely mirror human psychological associations. We construct feature vectors corresponding to 360 words and project them on 32 semantic axes (e.g. beautiful-ugly, soft-hard), and find that these projections correlate highly with human ratings of those words on the respective semantic scales. Second, we find that the cosine similarities between the semantic axes themselves are highly predictive of the correlations between these scales in the survey. Third, we show that substantial variance across the 32 semantic axes lies on a low-dimensional subspace, reproducing patterns typical of human semantic associations. Finally, we demonstrate that steering a word on one semantic axis causes spillover effects on the model's rating of that word on other semantic scales proportionate to the cosine similarity between those semantic axes. These findings suggest that features should be understood not only in isolation but through their
geometric relations and the meaningful subspaces they form.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.GT
How a Cooperative-Override Circuit Suppresses Nash Play in Large Language Models
Paraskevas V. Lekeas and Giorgos Stamatopoulos
On the named Prisoner's Dilemma under direct prompting, three larger instruction-tuned models, Llama-3-70B, Qwen2.5-32B, and Qwen2.5-72B, lock at full cooperation, the metric's maximum distance from Nash with zero variance across replicates, while Llama-3-8B plays near-Nash. Opening the models, a logit-lens analysis finds a distributed cooperative override. Intermediate readouts lean toward the Nash action through roughly three quarters of network depth before a late surge toward cooperation, and the final layer settles the contest. The size of that final correction, not the surge, rank-matches chain-of-thought behavior across scale and two architectures. In the 8B the override is a single causally controllable direction in the residual stream; steering it dials the decision, and clamping its component at one position of one layer moves the choice strictly monotonically, Spearman rho = 1.000, with generation fluent. The
circuit is lexical. It survives name removal and payoff rescaling but disengages when Cooperate and Defect are replaced with neutral labels, and on 48 payoff-random games with neutral surfaces no model locks cooperative on any dilemma or shows general
equilibrium competence. In mixed-model populations a single Nash-playing
agent collapses cooperation contagiously. What suppresses Nash play in large
language models is a word-triggered
circuit rather than missing competence, and it can be measured, bounded, and controlled.
Read the full abstract ▾
Machine Learning
Distributional Alignment Games for Answer-Level Fine-Tuning
Mehryar Mohri, Jon Schneider and Yifan Wu
We focus on the problem of Answer-Level Fine-Tuning (ALFT), where the goal is to optimize a
language model based on the correctness or properties of its final answers, rather than the specific
reasoning traces used to produce them. Directly optimizing answer-level objectives is computationally intractable due to the need to marginalize over the vast space of latent
reasoning paths. To overcome this, we propose a general game-theoretical framework that lifts the problem to a Distributional
Alignment Game. We formulate ALFT as a two-player game between a Policy (the generator) and a Target (an auxiliary distribution). We prove that the Nash
Equilibrium of this game corresponds exactly to the solution of the original answer-level optimization problem. This variational perspective transforms the intractable marginalization problem into a tractable projection problem. We demonstrate that this framework unifies recent approaches to diversity and self-improvement (coherence) and provide efficient
algorithms compatible with Group Relative Policy Optimization (GRPO), such as Coherence-GRPO, yielding significant complexity gains in mathematical
reasoning tasks.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.MA
A High-Throughput Compute-Efficient POMDP Hide-And-Seek-Engine (HASE) for Multi-Agent Operations
Timothy Flavin and Sandip Sen
Reinforcement Learning (RL)
algorithms exhibit high sample complexity, particularly when applied to Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). As a response, projects such as SampleFactory, EnvPool, Brax, and IsaacLab migrate parallel execution of classic environments such as MuJoCo and Atari into C++ thread pools or the GPU to decrease the computational cost of environment steps. We are interested in optimizing the decision-level of human-AI joint operations, so we introduce a compute-efficient Dec-POMDP engine natively architected in C++ called Hide-And-Seek-Engine. By employing Data-Oriented Design (DOD) principles, explicit 64-byte cache-line
alignment to remove false sharing, and a zero-copy PyTorch
memory bridge using pinned
memory and Direct
Memory Access (DMA), our engine sustains throughput of up to 33,000,000 steps per second (SPS) in a single-
agent, 1024-environment, decentralized observations on an AMD Ryzen 9950X (16 cores). Ten
agents reduces FPS to 7M SPS with generating random actions contributing 1/3rd the total runtime for reference. The engine achieves a throughput increase of approximately 3,500$\times$ over the baseline single threaded vectorized NumPy implementation and successfully trains cooperative multi-
agent policies via PPO, DQN, and SAC in minutes, validating both its performance and generality.
Read the full abstract ▾
Machine Learning
Generalizing the Geometry of Model Merging Through Frechet Averages
Marvin F. da Silva, Mohammed Adnan, Felix Dangel et al. (+1 more)
Model merging aims to combine multiple models into one without additional training. Naïve parameter-space averaging can be fragile under architectural
symmetries, as their geometry does not take them into account. In this work we show that not only the geometry, but also the averaging procedure itself, must be
symmetry-invariant to achieve
symmetry-aware merges. Consequently, we propose a general solution: merging as Fréchet averaging, i.e., selecting parameters that minimize a sum of geodesic distances on an appropriate
manifold. In this view, the key design choice is the overall geometry, i.e., the choice of metric,
manifold, and distance approximation, that determines what it means for two models to be "close". We show that Fréchet averaging, combined with simplifying assumptions, contains Fisher merging. Building on this, we examine the particular case of low-rank adapters (LoRA), whose
symmetries induce a distinct geometry: that of a quotient
manifold. We outline the limitations of current LoRA merging methods, propose a practical
algorithm for this setting, and show how they compare with other commonly used approaches.
Read the full abstract ▾
All authors Marvin F. da Silva, Mohammed Adnan, Felix Dangel, Sageev Oore
Machine Learning
ConformaDecompose: Explaining Uncertainty via Calibration Localization
Fatima Rabia Yapicioglu, Meltem Aksoy, Alberto Rigenti et al. (+4 more)
Conformal Prediction provides distribution-free prediction intervals with guaranteed coverage, but its reliance on a single global calibration threshold obscures the sources of uncertainty at the instance level. In particular, it conflates irreducible noise with uncertainty induced by heterogeneous training data (aleatoric), model limitations, or calibration mismatch (epistemic), offering little insight into why an interval is wide or whether it could be reduced. We introduce an uncertainty-aware explainability framework that analyses the reducibility of calibration-induced epistemic conformal uncertainty via progressive calibration localisation for regression tasks. The approach is diagnostic rather than causal: it does not estimate true aleatoric or epistemic uncertainty, but explains how conformal intervals contract and stabilise as calibration support is localised around a test instance. Across
benchmarks and real-world data, absolute reducible uncertainty aligns with epistemic proxies, while its relative contribution varies by task, revealing regimes hidden by interval width. This instance-level view complements conformal uncertainty, enhancing interpretability without altering the predictor or coverage.
Read the full abstract ▾
All authors Fatima Rabia Yapicioglu, Meltem Aksoy, Alberto Rigenti, Tuwe Löfström-Cavallin, Helena Löfström-Cavallin, Seyda Yoncaci, Luca Longo
Machine Learning
How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance
Jerry Y. Huang, Justin Lin, Sheel Shah et al. (+2 more)
In generative modeling, we often wish to produce samples that maximize a user-specified reward such as aesthetic quality or
alignment with human preferences, a problem known as guidance. Despite their widespread use, existing guidance methods either require expensive multi-particle, many-step schemes or rely on poorly understood approximations. We reformulate guidance as a deterministic optimal control problem, yielding a hierarchy of
algorithms that subsumes existing approaches at the coarsest level. We show that the flow map, an object of significant recent interest for its role in fast inference, arises naturally in the optimal solution. Based on this observation, we propose Flow Map Reward Guidance (FMRG): a training-free, single-trajectory framework that uses the flow map to both integrate and guide the flow. At text-to-image scale, FMRG matches or surpasses baselines across inverse problems and reward-guided generation with as few as 3 NFEs, giving at least an order-of-magnitude speedup in comparison to prior state of the art. Code is available at https://github.com/jrrhuang/fmrg.
Read the full abstract ▾
All authors Jerry Y. Huang, Justin Lin, Sheel Shah, Kartik Nair, Nicholas M. Boffi
Machine Learning · Cross-listed from cs.AI
Unsupervised Electrofacies Classification and Porosity Characterization in the Offshore Keta Basin Using Wireline Logs
Hamdiya Adams, Theophilus Ansah-Narh, Daniel Kwadwo Asiedu et al. (+6 more)
This study presents an unsupervised machine learning workflow for electrofacies analysis in the offshore Keta Basin, Ghana, where core data are scarce. Six standard wireline logs from Well~C were analysed over a depth interval comprising approximately $11{,}195$ samples. K-means clustering was applied in multivariate log space, with the clustering structure evaluated using inertia and silhouette diagnostics. Four clusters were identified, supported by an average silhouette coefficient of approximately $0.50$, indicating moderate but meaningful separation. The resulting electrofacies exhibit systematic, depth-continuous patterns associated with variations in clay content, porosity, and rock framework properties, forming a geological continuum from shale-dominated to cleaner sandstone-dominated units. The results demonstrate that log-only, unsupervised clustering supported by quantitative metrics provides a robust and reproducible framework for subsurface characterisation. The proposed workflow offers a practical tool for early-stage formation evaluation in frontier offshore basins and a foundation for future integrated studies.
Read the full abstract ▾
All authors Hamdiya Adams, Theophilus Ansah-Narh, Daniel Kwadwo Asiedu, Bruce Kofi Banoeng-Yakubo, Marcellin Atemkeng, Thomas Armah, Richmond Opoku-Sarkodie, Rebecca Davis, Ezekiel Nii Noye Nortey
Machine Learning
Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models
Vijay Sadashivaiah, Georgios Dasoulas, Judith Mueller et al. (+1 more)
Training stable biological foundation models requires rethinking
attention mechanisms: we find that using sigmoid
attention as a drop in replacement for softmax
attention a) produces better learned representations: on six diverse single-
cell datasets, sigmoid achieves 25% higher
cell-type separation, better
cell-type cohesion metrics, and lower validation loss, b) faster training, models with sigmoid
attention train up to 10% faster than their softmax counterparts, and c) more stable training by eliminating inherent sources of instability in softmax
attention. We establish that sigmoid
attention has globally bounded derivatives ($\leq 0.25$) as opposed to softmax, and a diagonal Jacobian structure in contrast with softmax's dense coupling, which together help alleviate training instabilities. In stress tests on 160M-parameter bidirectional
attention models trained without gradient clipping on 8K-
token sequences, softmax diverges catastrophically, with gradients exploding by four orders of magnitude, while sigmoid remains stable. Finally, we implement and open-source TritonSigmoid, an efficient GPU kernel that achieves 515 TFLOPS on H100 GPUs, outperforming both FlashAttention-2 and FlashSigmoid, with native padding support, which is essential for biological sequences. Our results establish sigmoid
attention as both theoretically grounded and empirically superior for biological foundation models. Code is available at https://github.com/MSDLLCpapers/triton-sigmoid
Read the full abstract ▾
All authors Vijay Sadashivaiah, Georgios Dasoulas, Judith Mueller, Soumya Ghosh
Machine Learning · Cross-listed from cs.CV
Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations
Andrii Zadaianchuk, Leonardo Barcellona, Lennard Schuenemann et al. (+7 more)
Accurately reconstructing complex full multi-object scenes from sparse observations remains a core challenge in computer
vision and a key step toward scalable and reliable simulation for robotics. In this work, we introduce RecGen, a generative framework for probabilistic joint estimation of object and part shapes, as well as their pose under occlusion and partial visibility from one or multiple RGB-D images. By leveraging compositional synthetic scene generation and strong 3D shape priors, RecGen generalizes across diverse object types and real-world environments. RecGen achieves state-of-the-art performance on complex, heavily occluded
datasets, robustly handling severe occlusions, symmetric objects, object parts, and intricate geometry and texture. Despite using nearly 80% fewer training meshes than the previous state of the art SAM3D, RecGen outperforms it by 30.1% in
geometric shape quality, 9.1% in texture reconstruction, and 33.9% in pose estimation.
Read the full abstract ▾
All authors Andrii Zadaianchuk, Leonardo Barcellona, Lennard Schuenemann, Christian Gumbsch, Zehao Wang, Muhammad Zubair Irshad, Fabien Despinoy, Rahaf Aljundi, Stratis Gavves, Sergey Zakharov
Machine Learning
Anomaly Detection in Soil Heavy Metal Contamination Using Unsupervised Learning for Environmental Risk Assessment
Isaac Tettey Adjokatse, Samuel Senyo Koranteng, George Yamoah Afrifa et al. (+6 more)
Soil contamination by heavy metals poses a persistent environmental and public health concern in rapidly urbanising regions of Ghana, particularly at unregulated waste disposal sites. This study applies an unsupervised machine learning framework to detect and characterise anomalous heavy metal contamination patterns in soils from twelve waste sites and residential controls in the Central Region, of Ghana. Concentrations of eight metals (As, Cd, Cr, Cu, Hg, Ni, Pb, Zn) were analysed alongside standard health risk indices, including the Hazard Index (HI) and Incremental Lifetime Cancer Risk (ILCR). Isolation Forest and PCA reconstruction error each identified $12$ anomalous samples ($15.4\%$ of $78$ samples), while DBSCAN detected no density-isolated noise points. A consensus approach isolated six robust anomalies ($7.7\%)$, all spatially concentrated at a single site (S3). Anomalies exhibited approximately $70$–$80\%$ higher mean HI values than normal samples, with all consensus anomalies exceeding the HI$=1$ threshold. PCA reconstruction error showed a strong positive association with HI ($r \approx 0.8$), indicating consistency between multivariate deviation and health risk. Three distinct anomaly types were identified: extreme Cu enrichment at S3, anomalously low Ni at S4/S5, and moderate multi-metal (Pb–Zn) co-elevation at S9–S12. The results demonstrate that unsupervised machine learning provides granular, objective insight beyond aggregate indices, enabling targeted site prioritisation and risk-informed environmental management.
Read the full abstract ▾
All authors Isaac Tettey Adjokatse, Samuel Senyo Koranteng, George Yamoah Afrifa, Theophilus Ansah-Narh, Marcellin Atemkeng, Joseph Bremang Tandoh, Kow Ahor Essel-Yorke, Richmond Opoku-Sarkodie, Rebecca Davis
Machine Learning
AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
Ahan Gupta, Zhihao Wang, Neel Dani et al. (+3 more)
Large-
language-models (LLMs) demonstrate enormous utility in long-context tasks which require processing
prompts that consist of tens to hundreds of thousands of
tokens. However, existing LLM training libraries do not provide easy to use abstractions to optimize for long-context training, instead focusing on optimizations for models with large parameter counts through ZeRO-3/FSDP, Tensor and Pipeline parallelism. This forces users to rewrite LLM training libraries to incorporate compositions of various complex long-context optimizations, such as sequence-parallelism, to training pipelines; a process that requires in-depth expertise, reducing developer productivity. To tackle these challenges, we introduce AutoSP: the first automated solution to automatically optimize LLM training for longer-contexts. AutoSP compiles models and applies a targeted set of optimizations: automated sequence parallelism, and long-context aware activation-checkpointing, to drastically enhance LLM trainability at negligible cost to throughput. Our evaluation demonstrates AutoSP's capability on both NVIDIA and AMD hardware, increasing training contexts by upto 2.7$\times$ and 2.5$\times$ respectively over competitive hand-written baseline at negligible cost to runtime performance.
Read the full abstract ▾
All authors Ahan Gupta, Zhihao Wang, Neel Dani, Masahiro Tanaka, Olatunji Ruwase, Minjia Zhang
Machine Learning · Cross-listed from cs.DC
Efficient Training on Multiple Consumer GPUs with RoundPipe
Yibin Luo, Shiwei Gao, Huichuan Zheng et al. (+2 more)
Fine-tuning Large
Language Models (LLMs) on consumer-grade GPUs is highly cost-effective, yet constrained by limited GPU
memory and slow PCIe interconnects. Pipeline parallelism combined with CPU offloading mitigates these hardware bottlenecks by reducing communication overhead. However, existing PP schedules suffer from an inherent limitation termed the weight binding issue. Binding uneven model stages (e.g., the LM head is large) to GPUs limits the pipeline's throughput to that of the GPU with the heaviest load, leading to severe pipeline bubbles. In this paper, we propose RoundPipe, a novel pipeline schedule that breaks the weight binding constraint on consumer GPU servers. RoundPipe treats GPUs as a pool of stateless execution workers and dynamically dispatches computation stages across devices in a round-robin manner, achieving a near-zero-bubble pipeline. To ensure training correctness and system efficiency, RoundPipe integrates a priority-aware transfer scheduling engine, a Layerwise Model Consistency Protocol, and an automated layer partitioning
algorithm. Evaluations on an 8$\times$ RTX 4090 server demonstrate that RoundPipe achieves 1.48–2.16$\times$ speedups over state-of-the-art baselines when fine-tuning 1.7B to 32B models. Remarkably, RoundPipe enables LoRA fine-tuning of the Qwen3-235B model with 31K sequence length on a single server. RoundPipe has been open-sourced as a Python library.
Read the full abstract ▾
All authors Yibin Luo, Shiwei Gao, Huichuan Zheng, Youyou Lu, Jiwu Shu
Machine Learning
Naibin Gu, Chenxu Yang, Qingyi Si et al. (+7 more)
RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy
Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and
video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.
Read the full abstract ▾
All authors Naibin Gu, Chenxu Yang, Qingyi Si, Chuanyu Qin, Dingyu Yao, Peng Fu, Zheng Lin, Weiping Wang, Nan Duan, Jiaqi Wang
Machine Learning · Cross-listed from cs.AI
When Your LLM Reaches End-of-Life: A Framework for Confident Model Migration in Production Systems
Emma Casey, David Roberts, David Sim et al. (+1 more)
We present a framework for migrating production Large
Language Model (LLM) based systems when the underlying model reaches end-of-life or requires replacement. The key contribution is a
Bayesian statistical approach that calibrates automated evaluation metrics against human judgments, enabling confident model comparison even with limited manual evaluation data. We demonstrate this framework on a commercial question-answering system serving 5.3M monthly interactions across six global regions; evaluating correctness, refusal behavior, and stylistic adherence to successfully identify suitable replacement models. The framework is broadly applicable to any enterprise deploying LLM-based products, providing a principled, reproducible methodology for model migration that balances quality assurance with evaluation efficiency. This is a capability increasingly essential as the LLM ecosystem continues to evolve rapidly and organizations manage portfolios of AI-powered services across multiple models, regions, and use cases.
Read the full abstract ▾
All authors Emma Casey, David Roberts, David Sim, Ian Beaver
Machine Learning
Learning Rate Transfer in Normalized Transformers
Boris Shigida, Boris Hanin and Andrey Gromov
The Normalized
Transformer, or nGPT (arXiv:2410.01131) achieves impressive training speedups and does not require weight decay or learning rate warmup. However, despite having hyperparameters that explicitly scale with model size, we observe that nGPT does not exhibit learning rate transfer across model dimension and
token horizon. To rectify this, we combine numerical experiments with a principled use of
alignment exponents (arXiv:2407.05872) to revisit and modify the $\mu$P approach to hyperparameter transfer (arXiv:2011.14522). The result is a novel nGPT parameterization we call $\nu$GPT. Through extensive empirical validation, we find $\nu$GPT exhibits learning rate transfer across width, depth, and
token horizon.
Read the full abstract ▾
Machine Learning
Learning to Forget: Continual Learning with Adaptive Weight Decay
Aditya A. Ramesh, Alex Lewandowski and Jürgen Schmidhuber
Continual learning
agents with finite capacity must balance acquiring new knowledge with retaining the old. This requires controlled forgetting of knowledge that is no longer needed, freeing up capacity to learn. Weight decay, viewed as a mechanism for forgetting, can serve this role by gradually discarding information stored in the weights. However, a fixed scalar weight decay drives this forgetting uniformly over time and uniformly across all parameters, even when some encode stable knowledge while others track rapidly changing targets. We introduce Forgetting through Adaptive Decay (FADE), which adapts per-parameter weight decay rates online via approximate meta-gradient descent. We derive FADE for the online linear setting and apply it to the final layer of neural networks. Our empirical analysis shows that FADE automatically discovers distinct decay rates for different parameters, complements step-size adaptation, and consistently improves over fixed weight decay across online tracking and streaming classification problems.
Read the full abstract ▾
Machine Learning · Cross-listed from math.OC
Man, Machine, and Mathematics
Akshunna S. Dogra
Nonlinear models and optimization methods have successfully tackled a rapidly growing set of problems in recent years. Indeed, a relatively small toolbox of such models and methods can provide sufficient performance across a large landscape of tasks: deep learning alone has made significant recent contributions in scientific modelling, natural
language processing, visual analysis, etc. A similar relationship exists between physical theories and phenomena, where many applications and observations emerge neatly from remarkably minimal foundations. It is natural to wonder if sparse unified frameworks could be built to steer discussion and discovery in the fields concerned with learning, optimization, and modelling. In this work, we posit and examine a possible outline for such a unified theory, interpreting the notion of ”learning” in a broad sense. In particular, we pursue our goals by viewing learning as an inter-connected process on multiple levels: problem setup, choosing methods, and the analysis of their interplay via imposed optimisation dynamics. We begin by proposing a precise yet versatile definition for ”solvable” problems. We then define the ”parametrised methods” by which their solution(s) may be ”learned”. Our goal is to sketch a ”universal convergence
theorem”, specifying how and when solvable problems become amenable to the methods chosen for them. We find these constructions reduce the study of learning down to remarkably few ideas and tools - many of which are simply adapted from existing ones in dynamical systems theory, geometry, and fundamental physics.
Read the full abstract ▾
Machine Learning
Detecting Clinical Discrepancies in Health Coaching Agents: A Dual-Stream Memory and Reconciliation Architecture
Samuel L Pugh, Eric Yang, Alexander Muir Sutherland et al. (+1 more)
As Large
Language Model (LLM)
agents transition from single-session tools to persistent systems managing longitudinal healthcare journeys, their
memory architectures face a critical challenge: reconciling two imperfect sources of truth. The patient's evolving self-report is current but prone to recall bias, while the Electronic Health Record (EHR) is medically validated but frequently stale. General-purpose
agent memory systems optimize for coherence by overwriting older facts with the user's latest statement, a pattern that risks safety failures when applied to clinical data. We introduce a Dual-Stream
Memory Architecture that strictly separates the patient narrative from the structured clinical record (FHIR), governed by a dedicated Reconciliation Engine that evaluates every extracted
memory against the patient's FHIR profile and classifies discrepancies by type, severity, and the specific FHIR resources involved. We evaluate this architecture on 26 patients across 675 longitudinal wellness coaching sessions, using a hybrid
dataset that interleaves real provider-patient transcripts with synthetic, FHIR-grounded clinical scenarios. In isolated testing, the engine detects 84.4% of designed clinical discrepancies with 86.7% safety-critical recall. By coupling extraction and reconciliation evaluation on the same data, we directly quantify a 13.6% error cascade, tracing the degradation to clinical details lost during
memory extraction from unstructured conversation rather than to downstream classification errors. These findings establish that validating patient-reported
memories against clinical records is both feasible and necessary for safe deployment of longitudinal health
agents.
Read the full abstract ▾
All authors Samuel L Pugh, Eric Yang, Alexander Muir Sutherland, Alessandra Breschi
Machine Learning
Cross-Subject Generalization for EEG Decoding: A Survey of Deep Learning Methods
Taida Li, Yujun Yan, Fei Dou et al. (+2 more)
Deep learning for cross-subject
EEG decoding is hindered by high inter-subject variability, which introduces a severe domain shift between training and unseen test subjects. This survey presents a comprehensive review of deep learning methodologies specifically engineered to address this cross-subject generalization challenge. To ground this analysis, we formalize the cross-subject setting as a multi-source domain problem and delineate the rigorous, subject-independent evaluation protocols required for valid assessment. Central to this survey is a systematic taxonomy of the current literature into discrete methodological families, including feature
alignment,
adversarial learning, feature disentanglement, and contrastive learning. We conclude by examining three critical elements for advancing robust, real-world decoding: the theoretical limitations of current methodologies, the structural value of subject identity, and the emergence of
EEG foundation models.
Read the full abstract ▾
All authors Taida Li, Yujun Yan, Fei Dou, Wenzhan Song, Xiang Zhang
Machine Learning · Cross-listed from cs.SE
LLM-Guided Runtime Parameter Optimization for Energy-Efficient Model Inference
Katelyn Crumpacker and Dimitrios Nikolopoulos
Large
Language Models (LLMs) have become an integral part of many real-world workflows. However, LLMs consume a lot of energy, which becomes a large concern in the scale of the demand for these tools. As LLMs become integrated into different workflows, different applications have arisen to deal with the challenge of running inference for these tools. This raises another issue of choosing the runtime parameter values for these services in order to minimize the energy consumption. Oftentimes this requires deep knowledge of the application or traditional optimization methods that can take days to find optimal values. In this work, we created a human-in-the-loop flow with LLM-assisted runtime parameter optimization in order to solve this issue. With human-created, specific feedback prompting methods, chat-based LLMs can iteratively find energy-efficient inference parameters faster than traditional search methods. LLMs can also tailor their solutions to different hardware setups and easily take into account other system constraints. The enhanced
prompt template was able to converge below the threshold at an average of 3.4
prompts compared to the baseline, which converged in an average of 5.2
prompts, and consistently achieved lower final energy per
token. The enhanced
prompt template also outperformed Sobol sampling in convergence speed.
Read the full abstract ▾
Machine Learning
NORACL: Neurogenesis for Oracle-free Resource-Adaptive Continual Learning
Karthik Charan Raghunathan, Christian Metzner, Laura Kriener et al. (+1 more)
In a continual learning setting, we require a model to be plastic enough to learn a new task and stable enough to not disturb previously learned capabilities. We argue that this dilemma has an architectural root. A finite network has limited representational and plastic resources, yet the required capacity depends on properties of the future task stream that are unknown: how many tasks will be encountered, and how much they overlap in feature space. Regularization-based methods preserve past knowledge within fixed-capacity architectures and therefore implicitly rely on an oracle architecture sized for this unknown future. When tasks are only weakly related, fixed architectures progressively run out of plastic resources; when tasks are few or strongly overlapping, models are often over-provisioned. Inspired by neurogenesis in biology, we propose NORACL to address the stability-plasticity dilemma by tackling the oracle architecture problem through neuronal growth. Starting from a compact network, NORACL grows only when needed by monitoring two complementary signals for representational and plasticity saturation. We evaluate NORACL against oracle-sized static baselines across varying task counts and geometries. Across all settings, NORACL achieves final average accuracies that are better than or on par with oracle-provisioned static baselines while using fewer parameters. Additionally, NORACL yields architectures with interpretable growth, i.e. dissimilar tasks predominantly expand feature-extraction layers, whereas tasks which rely on common features shift growth toward later feature-combination layers. Our analysis further explains why fixed-capacity networks lose plasticity as tasks accumulate, whereas NORACL creates fresh capacity for new tasks through growth. Together, these results show that adaptive neurogenesis pushes the stability-plasticity Pareto frontier of continual learning.
Read the full abstract ▾
All authors Karthik Charan Raghunathan, Christian Metzner, Laura Kriener, Melika Payvand
Machine Learning · Cross-listed from stat.ML
SCOPE-FE: Structured Control of Operator and Pairwise Exploration for Feature Engineering via Quality-Aware Candidate-Space Reduction
Minhee Park, Seongyeon Son, Yonghyun Lee et al. (+1 more)
Automatic feature engineering can improve predictive performance on tabular data by generating diverse feature transformations. However, the candidate space induced by combinations of input features and operators grows rapidly with dimensionality, resulting in substantial computational cost. We propose SCOPE-FE, a framework that controls the search space before candidate generation. SCOPE-FE combines FeatureClustering, a structural pair gate based on mixed-type feature association, with OperatorProbing, a
dataset-specific utility control over operators. Unlike conventional expand-and-reduce approaches that generate a large candidate set and prune it afterward, SCOPE-FE focuses computation on a smaller, data-dependent candidate pool. Across ten OpenFE
benchmark datasets, SCOPE-FE achieves a median candidate-space reduction of 82.9% and lowers component-summed feature-engineering time-including separately measured FeatureClustering overhead-on all ten
datasets, yielding a
geometric-mean speedup of 2.66x and a maximum speedup of 5.48x. Despite this reduction, SCOPE-FE is within the stated practical-equivalence margin of OpenFE on 8 of 10
datasets. An exhaustive candidate audit shows enrichment above uniform-random expectation on 8 of 10
datasets, with a median enrichment of 1.35x. Against Random-Pair, SCOPE-FE has higher enrichment on 6 of 10
datasets, with 5 of 10 significant; against Random-Operator and Random-Joint, it has higher enrichment on 8 of 10
datasets, with 8 of 10 significant for each. These results demonstrate that pre-generation search-space control can substantially reduce feature-engineering time while retaining a utility-enriched candidate pool under the OpenFE-compatible evaluation protocol.
Read the full abstract ▾
All authors Minhee Park, Seongyeon Son, Yonghyun Lee, Eunchan Kim
Machine Learning
Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometry
Wenhao Lan, Shan Li, Xinhua Lai et al. (+4 more)
Safety-aligned
language models must refuse harmful requests without broad over-refusal, but it remains unclear how dynamic
adversarial fine-tuning changes refusal-control carriers: Kullback–Leibler (KL)-constrained directions or small subspaces that causally modulate refusal without large safe-
prompt distribution shifts. We study a 7B backbone under supervised fine-tuning (SFT) and Robust Refusal Dynamic Defense (R2D2), aligning HarmBench, StrongREJECT, and XSTest evaluations with five-anchor geometry measurements, causal interventions, and sparse adaptive stress tests. R2D2 drives fixed-source HarmBench attack success to zero at early checkpoints; however, these checkpoints also exhibit maximal XSTest refusal and fail a benign-utility audit. Later checkpoints partially recover utility-facing behavior while reopening attack success, with adaptive GCG attack success rate rising to 0.415 at step 250 and 0.613 at step 500. Internally, R2D2 preserves a late-layer admissible refusal-control carrier through step 100 and then relocates the best admissible carrier to an early layer; SFT relocates earlier yet remains less robust. Effective rank stays near 1.24, and SFT shows larger principal-angle drift, arguing against both dimensional expansion and drift magnitude as sufficient explanations. Causal interventions support a low-dimensional but utility-coupled carrier. These results support a geometry-reorganization account of R2D2 along a robustness–utility frontier, without establishing adaptive robustness.
Read the full abstract ▾
All authors Wenhao Lan, Shan Li, Xinhua Lai, Meiqi Wu, Junbin Yang, Haihua Shen, Yijun Yang