Machine Learning
Generalization Bounds for Spectral GNNs via Fourier Domain Analysis
Vahan A. Martirosyan, Daniele Malitesta, Hugues Talbot et al. (+2 more)
Spectral graph neural networks learn graph filters, but their behavior with increasing depth and
polynomial order is not well understood. We analyze these models in the graph
Fourier domain, where each layer becomes an element-wise frequency update, separating the fixed spectrum from trainable parameters and making depth and order explicit. In this setting, we show that Gaussian complexity is invariant under the Graph
Fourier Transform, which allows us to derive data-dependent, depth, and order-aware generalization bounds together with stability estimates. In the linear case, our bounds are tighter, and on real graphs, the data-dependent term correlates with the generalization gap across
polynomial bases, highlighting practical choices that avoid frequency amplification across layers.
Read the full abstract ▾
All authors Vahan A. Martirosyan, Daniele Malitesta, Hugues Talbot, Jhony H. Giraldo, Fragkiskos D. Malliaros
Machine Learning · Cross-listed from cs.SE
Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time
Razvan Mihai Popescu, David Gros, Andrei Botocan et al. (+3 more)
The rise of large
language models for code has reshaped software development. Autonomous coding
agents, able to create branches, open pull requests, and perform code reviews, now actively contribute to real-world projects. Their growing role offers a unique and timely opportunity to investigate AI-driven contributions and their effects on code quality, team dynamics, and software maintainability. In this work, we construct a novel
dataset of approximately $110,000$ open-source pull requests, including associated commits, comments, reviews, issues, and file changes, collectively representing millions of lines of source code. We compare five popular coding
agents, including OpenAI Codex, Claude Code, GitHub Copilot, Google Jules, and Devin, examining how their usage differs in various development aspects such as merge frequency, edited file types, and developer interaction signals, including comments and reviews. Furthermore, we emphasize that code authoring and review are only a small part of the larger software engineering process, as the resulting code must also be maintained and updated over time. Hence, we offer several longitudinal estimates of survival and churn rates for
agent-generated versus human-authored code. Ultimately, our findings indicate an increasing
agent activity in open-source projects, although their contributions are associated with more churn over time compared to human-authored code.
Read the full abstract ▾
All authors Razvan Mihai Popescu, David Gros, Andrei Botocan, Rahul Pandita, Prem Devanbu, Maliheh Izadi
Machine Learning
Orthogonal Learner for Estimating Heterogeneous Long-Term Treatment Effects
Haorui Ma, Dennis Frauen, Valentyn Melnychuk et al. (+1 more)
Estimation of heterogeneous long-term treatment effects (HLTEs) is relevant for personalized decision-making in marketing, economics, and medicine, where short-term observational
datasets are often combined with long-term observational
datasets. However, HLTE estimation is challenging due to limited overlap in treatment assignments or in long-term outcomes for certain subpopulations, which can lead to unstable HLTE estimates with large finite-sample variance. To address this challenge, we introduce the LT-O-learners (Long-Term Orthogonal Learners), a set of novel orthogonal learners for HLTE estimation in the canonical HLTE setting with surrogacy. The key idea of our LT-O-learners is to retarget the loss via custom overlap weights that downweight low-overlap samples. We show that the retargeted loss recovers the true HLTE pointwise and satisfies Neyman-orthogonality. We further prove two key theoretical results: (i) The nuisance error enters the error bound only through higher-order terms, which means our learners are robust to nuisance estimation error. (ii) Under a linear function class, the retargeting effectively controls the asymptotic variance of the HLTE
estimator via the overlap weights in low-overlap regimes. We conduct experiments on synthetic and real-world
datasets to confirm the theoretical properties of our LT-O-learners, particularly robustness in low-overlap regimes. To our knowledge, ours are the first orthogonal learners for HLTE estimation robust to low overlap in long-term settings.
Read the full abstract ▾
All authors Haorui Ma, Dennis Frauen, Valentyn Melnychuk, Stefan Feuerriegel
Machine Learning
Event Embedding of Protein Networks : Compositional Learning of Biological Function
Antonin Sulc
In this work, we study whether enforcing strict compositional structure in sequence
embeddings yields meaningful
geometric organization when applied to
protein-
protein interaction networks. Using Event2Vec, an additive sequence
embedding model, we train 64-dimensional representations on random walks from the human STRING interactome, and compare against a DeepWalk baseline based on Word2Vec, trained on the same walks. We find that compositional structure substantially improves pathway coherence (30.2$\times$ vs 2.9$\times$ above random), functional analogy accuracy (mean similarity 0.966 vs 0.650), and hierarchical pathway organization, while
geometric properties such as norm–degree anticorrelation are shared with or exceeded by the non-compositional baseline. These results indicate that enforced compositionality specifically benefits relational and compositional
reasoning tasks in biological networks.
Read the full abstract ▾
Machine Learning
Fatigue-Aware Learning to Defer via Constrained Optimisation
Zheng Zhang, Cuong C. Nguyen, David Rosewarne et al. (+2 more)
Learning to defer (L2D) enables human-AI cooperation by deciding when an AI system should act autonomously or defer to a human expert. Existing L2D methods, however, assume static human performance, contradicting well-established findings on fatigue-induced degradation. We propose Fatigue-Aware Learning to Defer via Constrained Optimisation (FALCON), which explicitly models workload-varying human performance using psychologically grounded fatigue curves. FALCON formulates L2D as a Constrained Markov Decision Process (CMDP) whose state includes both task features and cumulative human workload, and optimises accuracy under human-AI cooperation budgets via PPO-Lagrangian training. We further introduce FA-L2D, a
benchmark that systematically varies fatigue dynamics from near-static to rapidly degrading regimes. Experiments across multiple
datasets show that FALCON consistently outperforms state-of-the-art L2D methods across coverage levels, generalises zero-shot to unseen experts with different fatigue patterns, and demonstrates the advantage of adaptive human-AI collaboration over AI-only or human-only decision-making when coverage lies strictly between 0 and 1.
Read the full abstract ▾
All authors Zheng Zhang, Cuong C. Nguyen, David Rosewarne, Kevin Wells, Gustavo Carneiro
Machine Learning
Super-Resolving Coarse-Resolution Weather Forecasts With Flow Matching
Aymeric Delefosse, Anastase Charantonis and Dominique Béréziat
Machine learning-based weather
forecasting models now surpass state-of-the-art numerical weather prediction systems, but training and operating these models at high spatial resolution remains computationally expensive. We present a modular framework that decouples
forecasting from spatial resolution by applying learned generative super-resolution as a post-processing step to coarse-resolution forecast trajectories. We formulate super-resolution as a
stochastic inverse problem, using a residual formulation to preserve large-scale structure while reconstructing unresolved variability. The model is trained with flow matching exclusively on reanalysis data and is applied to global medium-range forecasts. We evaluate (i) design consistency by re-coarsening super-resolved forecasts and comparing them to the original coarse trajectories, and (ii) high-resolution forecast quality using standard ensemble verification metrics and spectral diagnostics. Results show that super-resolution preserves large-scale structure and variance after re-coarsening, introduces physically consistent small-scale variability, and achieves competitive probabilistic forecast skill at 0.25{\deg} resolution relative to an operational ensemble baseline, while requiring only a modest additional training cost compared with end-to-end high-resolution
forecasting.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
StanceMoE: Mixture-of-Experts Architecture for Stance Detection
Abdullah Al Shafi, Md. Milon Islam, Sk. Imran Hossain et al. (+1 more)
Actor-level stance detection aims to determine an author expressed position toward specific geopolitical actors mentioned or implicated in a text. Although
transformer-based models have achieved relatively good performance in stance classification, they typically rely on unified representations that may not sufficiently capture heterogeneous linguistic signals, such as contrastive discourse structures, framing cues, and salient lexical indicators. This motivates the need for adaptive architectures that explicitly model diverse stance-expressive patterns. In this paper, we propose StanceMoE, a context-enhanced Mixture-of-Experts (MoE) architecture built upon a fine-tuned BERT encoder for actor-level stance detection. Our model integrates six expert modules designed to capture complementary linguistic signals, including global semantic orientation, salient lexical cues, clause-level focus, phrase-level patterns, framing indicators, and contrast-driven discourse shifts. A context-aware gating mechanism dynamically weights expert contributions, enabling adaptive routing based on input characteristics. Experiments are conducted on the StanceNakba 2026 Subtask A
dataset, comprising 1,401 annotated English texts where the target actor is implicit in the text. StanceMoE achieves a macro-F1 score of 94.26%, outperforming traditional baselines, and alternative BERT-based variants.
Read the full abstract ▾
All authors Abdullah Al Shafi, Md. Milon Islam, Sk. Imran Hossain, K. M. Azharul Hasan
Machine Learning · Cross-listed from cs.DB
Accurate and Scalable Matrix Mechanisms via Divide and Conquer
Guanlin He, Yingtai Xiao, Jiamu Bai et al. (+4 more)
Matrix mechanisms are often used to provide unbiased differentially private query answers when publishing statistics or creating synthetic data. Recent work has developed
matrix mechanisms, such as ResidualPlanner and Weighted
Fourier Factorizations, that scale to high dimensional
datasets while providing optimality guarantees for workloads such as marginals and circular product queries. They operate by adding noise to a linearly independent set of queries that can compactly represent the desired workloads. In this paper, we present QuerySmasher, an alternative scalable approach based on a divide-and-conquer strategy. Given a workload that can be answered from various data marginals, QuerySmasher splits each query into sub-queries and re-assembles the pieces into mutually orthogonal sub-workloads. These sub-workloads represent small, low-dimensional problems that can be independently and optimally answered by existing low-dimensional
matrix mechanisms. QuerySmasher then stitches these solutions together to answer queries in the original workload. We show that QuerySmasher subsumes prior work, like ResidualPlanner (RP), ResidualPlanner+ (RP+), and Weighted
Fourier Factorizations (WFF). We prove that it can dominate those approaches, under sum squared error, for all workloads. We also experimentally demonstrate the scalability and accuracy of QuerySmasher.
Read the full abstract ▾
All authors Guanlin He, Yingtai Xiao, Jiamu Bai, Xin Gu, Zeyu Ding, Wenpeng Yin, Daniel Kifer
Machine Learning
Policy Improvement Reinforcement Learning
Huaiyang Wang, Xiaojie Li, Xiaohan Wang et al. (+10 more)
Reinforcement learning has become a central post-training paradigm for improving LLM and
agent capabilities. Yet existing RL post-training methods share a common blind spot: they construct local learning signals from sampled trajectories, rewards, or feedback-conditioned targets, then update the policy without explicitly verifying whether the resulting policy outperforms its predecessor. Optimizing these local signals does not necessarily produce a better policy, while finite sampling, generation stochasticity and feedback noise can further widen this gap. We argue that the missing ingredient is policy improvement feedback: the ability to measure progress across policy iterations. We introduce Policy Improvement Reinforcement Learning (PIRL), which formulates inter-iteration performance gain as an explicit objective structurally aligned with final task performance. Building on PIRL, we propose Policy Improvement Policy Optimization (PIPO), a plug-in closed-loop framework that verifies the previous update against a sliding-window historical performance anchor. PIPO uses this improvement feedback to modulate the local learning signal of the base policy optimization
algorithm, reinforcing updates associated with measured progress and suppressing those associated with performance drops. We provide theoretical evidence that PIPO locally aligns policy updates with the PIRL improvement objective. Experiments on mathematical
reasoning, code, tool-use, and self-
distillation settings show that PIPO yields consistent gains across PPO, group-relative, and self-
distillation policy optimization families.
Read the full abstract ▾
All authors Huaiyang Wang, Xiaojie Li, Xiaohan Wang, Zhixia Zhang, Xiaodong Lu, Zixuan Huang, Jiajun Chai, Guojun Yin, Deqing Wang, Haoyi Zhou, Yaodong Yang, Jianxin Li, Yikun Ban
Machine Learning · Cross-listed from cs.AI
Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants
Deepak Nathani, Cheng Zhang, Chang Huan et al. (+7 more)
Proactive
agents that anticipate user needs and autonomously execute tasks hold great promise as digital assistants, yet the lack of realistic user simulation frameworks hinders their development. Existing approaches model apps as flat tool-calling APIs, failing to capture the stateful and sequential nature of user interaction in digital environments and making realistic user simulation infeasible. We introduce Proactive
Agent Research Environment (Pare), a framework for building and evaluating proactive
agents in digital environments. Pare models applications as finite state machines with stateful
navigation and state-dependent action space for the user simulator, enabling active user simulation. Building on this foundation, we present Pare-Bench, a
benchmark of 143 diverse tasks spanning communication, productivity, scheduling, and lifestyle apps, designed to test context observation, goal inference, intervention timing, and multi-app orchestration.
Read the full abstract ▾
All authors Deepak Nathani, Cheng Zhang, Chang Huan, Jiaming Shan, Yinfei Yang, Alkesh Patel, Zhe Gan, William Yang Wang, Michael Saxon, Xin Eric Wang
Machine Learning
Meta-TTL: Meta-Learning Self-Improvement Policies for Language Agents
Zhanzhi Lou, Hui Chen, Yibo Li et al. (+2 more)
Test-Time Learning (TTL) enables
language agents to iteratively refine their performance through repeated interactions with the environment at inference time. At the core of TTL is a self-improvement policy that updates the actor policy based on experience from previous episodes, thereby improving future behavior. Existing methods rely on hand-crafted self-improvement rather than optimizing them for downstream improvement. We argue that optimal self-improvement policies should be learned from task environments, not hand-engineered based on human intuition. To achieve this, we introduce Meta-TTL, a framework that formulates the discovery of effective self-improvement policies as a bi-level optimization problem. Within this framework, the inner loop executes the standard TTL process, measuring how effectively a candidate self-improvement policy helps an
agent correct errors across sequential episodes. Guided by the
agent's performance, the outer loop performs reflective meta-training across diverse training tasks, using a balanced improvement score (BIS) to balance task contributions during candidate selection. We evaluate Meta-TTL on Jericho, WebArena-Lite, and -bench across both in-distribution (ID) and out-of-distribution (OOD) settings. Meta-TTL consistently outperforms existing baselines, improving TTL over the strongest baseline by up to 23% on ID tasks and 27% on OOD tasks. These results suggest that the optimized self-improvement policy encodes transferable meta-strategies that generalize beyond the training task distribution.
Read the full abstract ▾
All authors Zhanzhi Lou, Hui Chen, Yibo Li, Qian Wang, Bryan Hooi
Machine Learning
Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation
Yuhang Li, Donghyun Lee, Ruokai Yin et al. (+1 more)
Low-rank decomposition has emerged as an important problem in Large
Language Model (LLM) fine-tuning and inference. Through Singular Value Decomposition (SVD), the weight
matrix can be factorized into low-rank spaces optimally. Previously, a common practice was to decompose the weight in the activation-whitened space, and then achieve satisfying results. In this work, we propose Optimal
Brain Decomposition LLM (OBD-LLM), which studies the decomposition problem in the model space by utilizing second-order Hessian information. Through a rigorous Kronecker-factorization of the Hessian, we show that the decomposition needs to consider both input and output information of the layer, and achieves much better decomposition results compared to input only method. Our loss-aware decomposition method involves a bi-directional whitening on the weight
matrix. As a result, OBD-LLM is a closed-form solution for the optimal decomposition of weights in the
language model. Remarkably, we achieve ~20-40% better results than previous state-of-the-art decomposition methods, the SVD-LLM.
Read the full abstract ▾
All authors Yuhang Li, Donghyun Lee, Ruokai Yin, Priyadarshini Panda
Machine Learning
Cost-Penalized Fitness in FMA-Orchestrated Mixture of Experts: Experimental Evidence for Molecular Memory in Domain Adaptation
Martin Jaraiz
We present experimental results from seven controlled runs of nanoFMT, a Free-
Market Algorithm (FMA) orchestrated
transformer with dynamic Mixture-of-Experts (MoE) management. The experiments address a fundamental question for advanced LLM development: how should an MoE system manage its expert pool when operating at full capacity under changing data distributions? We demonstrate that cost-penalized fitness metrics, combined with a linear grace period for newborn experts, produce a system that accumulates domain expertise through diversification rather than replacement. The central result is a round-trip domain shift experiment showing 9-11x faster recovery when returning to a previously learned domain, with zero expert births or replacements required. This "molecular
memory" effect – where dormant experts survive and reactivate when their domain returns – has no analogue in current MoE management approaches. A preliminary cost analysis estimates annual savings of $39.1M and 27.1 GWh energy reduction for an OpenAI-scale provider under a moderate scenario.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Deconfounding Scores and Representation Learning for Causal Effect Estimation with Weak Overlap
Oscar Clivio, Alexander D'Amour, Alexander Franks et al. (+3 more)
Overlap, also known as positivity, is a key condition for causal treatment effect estimation. Many popular
estimators suffer from high variance and become brittle when features differ strongly across treatment groups. This is especially challenging in high dimensions: the curse of dimensionality can make overlap implausible. To address this, we propose a class of feature representations called deconfounding scores, which preserve both identification and the target of estimation; the classical propensity and prognostic scores are two special cases. We characterize the problem of finding a representation with better overlap as minimizing an overlap divergence under a deconfounding score constraint. We then derive closed-form expressions for a class of deconfounding scores under a broad family of generalized linear models with Gaussian features and show that prognostic scores are overlap-optimal within this class. We conduct extensive experiments to assess this behavior empirically.
Read the full abstract ▾
All authors Oscar Clivio, Alexander D'Amour, Alexander Franks, David Bruns-Smith, Chris Holmes, Avi Feller
Machine Learning
Routing-Free Mixture-of-Experts
Yilun Liu, Jinru Han, Sikuan Yan et al. (+2 more)
Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and
token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.
Read the full abstract ▾
All authors Yilun Liu, Jinru Han, Sikuan Yan, Volker Tresp, Yunpu Ma
Machine Learning
MIRANDA: MId-feature RANk-adversarial Domain Adaptation toward climate change-robust ecological forecasting with deep learning
Yuchang Jiang, Jan Dirk Wegner and Vivien Sainte Fare Garnot
Plant phenology modelling aims to predict the timing of seasonal phases, such as leaf-out or flowering, from meteorological time series. Reliable predictions are crucial for anticipating ecosystem responses to
climate change. While phenology modelling has traditionally relied on mechanistic approaches, deep learning methods have recently been proposed as flexible, data-driven alternatives with often superior performance. However, mechanistic models tend to outperform deep networks when data distribution shifts are induced by
climate change. Domain Adaptation (DA) techniques could help address this limitation. Yet, unlike standard DA settings,
climate change induces a temporal continuum of domains and involves both a
covariate and label shift, with warmer records and earlier start of spring. To tackle this challenge, we introduce Mid-feature Rank-
adversarial Domain Adaptation (MIRANDA). Whereas conventional
adversarial methods enforce domain invariance on final latent representations, an approach that does not explicitly address label shift, we apply
adversarial regularization to intermediate features. Moreover, instead of a binary domain-classification objective, we employ a rank-based objective that enforces year-invariance in the learned meteorological representations. On a country-scale
dataset spanning 70 years and comprising 67,800 phenological observations of 5 tree species, we demonstrate that, unlike conventional DA approaches, MIRANDA improves robustness to climatic distribution shifts and narrows the performance gap with mechanistic models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Multimodal Language Models Cannot Spot Spatial Inconsistencies
Om Khangaonkar, Hadi J. Rad and Hamed Pirsiavash
Spatial consistency is a fundamental property of the visual world and a key requirement for models that aim to understand physical reality. Despite recent advances,
multimodal large
language models (MLLMs) often struggle to reason about 3D geometry across multiple views. Rather than asking models to describe scene attributes, we introduce a more challenging task: given two views of the same scene, identify the object that violates 3D motion consistency. We propose a simple and scalable method for generating realistic, spatially inconsistent image pairs from multi-view scenes, enabling systematic evaluation of this capability. Our results show that state-of-the-art MLLMs significantly underperform human observers and exhibit substantial variability across different scene attributes, revealing a fragile and incomplete understanding of 3D structure. We hope our findings underscore the need for approaches that develop a more deeply grounded understanding of the physical world.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Preference Guided Iterated Pareto Referent Optimisation for Accessible Route Planning
Paolo Speziali, Arno De Greef, Mehrdad Asadi et al. (+3 more)
We propose the Preference Guided Iterated Pareto Referent Optimisation (PG-IPRO) for urban route planning for people with different accessibility requirements and preferences. With this
algorithm the user can interact with the system by giving feedback on a route, i.e., the user can say which objective should be further minimized, or conversely can be relaxed. This leads to intuitive user interaction, that is especially effective during early iterations compared to information-gain-based interaction. Furthermore, due to PG-IPRO's iterative nature, the full set of alternative, possibly optimal policies (the Pareto front), is never computed, leading to higher computational efficiency and shorter waiting times for users.
Read the full abstract ▾
All authors Paolo Speziali, Arno De Greef, Mehrdad Asadi, Willem Röpke, Ann Nowé, Diederik M. Roijers
Machine Learning
Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das et al. (+1 more)
Pretraining Large
Language Models (LLMs) from scratch requires massive amount of compute. Aurora super computer is an ExaScale machine with 127,488 Intel PVC (Ponte Vechio) GPU tiles. In this work, we showcase LLM pretraining on Aurora at the scale of 1000s of GPU tiles. Towards this effort, we developed Optimus, an inhouse training library with support for standard large model training techniques. Using Optimus, we first pretrained Mula-1B, a 1 Billion dense model and Mula-7B-A1B, a 7 Billion Mixture of Experts (MoE) model from scratch on 3072 GPU tiles for the full 4 trillion
tokens of the OLMoE-mix-0924
dataset. We then demonstrated model scaling by pretraining three large MoE models Mula-20B-A2B, Mula-100B-A7B, and Mula-220B-A10B till 100 Billion
tokens on the same
dataset. On our largest model Mula-220B-A10B, we pushed the compute scaling from 384 to 12288 GPU tiles and observed scaling efficiency of around 90% at 12288 GPU tiles. We significantly improved the runtime performance of MoE models using custom GPU kernels for expert computation, and a novel EP-Aware sharded optimizer resulting in training speedups up to 1.71x. As part of the Optimus library, we also developed a robust set of reliability and fault tolerant features to improve training stability and continuity at scale.
Read the full abstract ▾
All authors Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das, Bharat Kaul
Machine Learning
Using predefined vector systems to speed up neural network multimillion class classification
Nikita Gabdullin and Ilya Androsov
Label prediction in neural networks (NNs) has O(n) complexity proportional to the number of classes. This holds true for classification using fully connected layers and cosine similarity with some set of class prototypes. In this paper we show that if NN latent space (LS) geometry is known and possesses specific properties, label prediction complexity can be significantly reduced. This is achieved by associating label prediction with the O(1) complexity closest cluster center search in a vector system used as target for latent space configuration (LSC). The proposed method only requires finding indexes of several largest and lowest values in the
embedding vector making it extremely computationally efficient. We show that the proposed method does not change NN training accuracy computational results. We also measure the time required by different computational stages of NN inference and label prediction on multiple
datasets. The experiments show that the proposed method allows to achieve up to 11.6 times overall acceleration over conventional methods. Furthermore, the proposed method has unique properties which allow to predict the existence of new classes.
Read the full abstract ▾
Machine Learning
Thinking Wrong in Silence: Backdoor Attacks on Continuous Latent Reasoning
Swapnil Parekh
A new generation of
language models reasons entirely in continuous hidden states, producing no
tokens and leaving no audit trail. We show that this silence creates a fundamentally new attack surface. ThoughtSteer perturbs a single
embedding vector at the input layer; the model's own multi-pass
reasoning amplifies this perturbation into a hijacked latent trajectory that reliably produces the attacker's chosen answer, while remaining structurally invisible to every
token-level defense. Across two architectures (Coconut and SimCoT), three
reasoning benchmarks, and model scales from 124M to 3B parameters, ThoughtSteer achieves >=99% attack success rate with near-baseline clean accuracy, transfers to held-out
benchmarks without retraining (94-100%), evades all five evaluated active defenses, and survives 25 epochs of clean fine-tuning. We trace these results to a unifying mechanism: Neural Collapse in the latent space pulls triggered representations onto a tight
geometric attractor, explaining both why defenses fail and why any effective
backdoor must leave a linearly separable signature (probe AUC>=0.999). Yet a striking paradox emerges: individual latent vectors still encode the correct answer even as the model outputs the wrong one. The
adversarial information is not in any single vector but in the collective trajectory, establishing
backdoor perturbations as a new lens for mechanistic interpretability of continuous
reasoning. Code and checkpoints are available.
Read the full abstract ▾
Machine Learning
ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding
Lala Shakti Swarup Ray, Mengxi Liu, Alcina Pinto et al. (+4 more)
Wearable human activity recognition (HAR) has made steady progress, yet much of this progress remains grounded in fixed-window, closed-set classification
benchmarks. This formulation is poorly matched to everyday behavior, where activities are open-ended, unscripted, personalized, variable in duration, and often compositional. To address this mismatch, we introduce ActivityNarrated, an open-ended narrative paradigm for
language-grounded wearable activity understanding. We formulate this setting as dense sensor signal captioning with a comprehensive
benchmark protocol that measures temporal localization, caption quality, sensor-
language alignment, conventional closed-set classification as a downstream diagnostic, and additional robustness measures. We further present ActNarrator, a 3-stage architecture that discretizes continuous IMU signals into reusable motion
tokens and uses an external frozen small
language model to generate open-vocabulary activity captions. Experiments show that our method provides high quality dense sensor captioning with superior adaptivity and robustness, enabling various downstream tasks by turning sensor-based human activity understanding into sensor-grounded text-level
reasoning. This includes downstream classification where ActNarrator outperforms state-of-the-art HAR models by 3.8 - 31.6 % in Macro-F1. This paradigm also enables novel activity understanding capabilities such as complex question-answering over long time horizons.
Read the full abstract ▾
All authors Lala Shakti Swarup Ray, Mengxi Liu, Alcina Pinto, Deepika Gurung, Daniel Geissler, Paul Lukowoicz, Bo Zhou
Machine Learning · Cross-listed from cs.CL
Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
Zehao Jin and Yanan Sui
The whole-
brain connectome of a fruit fly comprises over 130K
neurons connected with a probability of merely 0.02%, yet achieves an average shortest path of only 4.4 hops. Despite being highly structured at the
circuit level, the network's long-range connections are broadly distributed across
brain regions, functioning as
stochastic shortcuts that enable efficient global communication. Inspired by this observation, we propose
Stochastic Attention (SA), a drop-in enhancement for sliding-window
attention (SWA) that applies a random permutation to the
token sequence before windowed
attention and restores the original order afterward. This transforms the fixed local window into a
stochastic global one within the same $O(nw)$ per-layer budget. Through depth, independently sampled permutations yield exponentially growing receptive fields, achieving full sequence coverage in $O(\log_w n)$ layers versus $O(n/w)$ for SWA. We validate SA in two settings: pre-training
language models from scratch, where a gated SA + SWA combination achieves the best average zero-shot accuracy, and training-free inference on Qwen3-8B and Qwen3-30B-A3B, where SA consistently outperforms SWA and matches or exceeds Mixture of Block
Attention at comparable compute budgets. These results suggest that connectome-inspired
stochastic routing is a practical primitive for improving the expressivity of efficient
attention, complementary to existing linear and sparse approaches.
Read the full abstract ▾
Machine Learning
BioCOMPASS: Integrating Biomarkers into Transformer-Based Immunotherapy Response Prediction
Sayed Hashim, Frank Soboczenski and Paul Cairns
Datasets used in immunotherapy response prediction are typically small in size, as well as diverse in cancer type, drug administered, and sequencer used. Models often drop in performance when tested on patient cohorts that are not included in the training process. Recent work has shown that
transformer-based models along with self-supervised learning show better generalisation performance than threshold-based biomarkers, but is still suboptimal. We present BioCOMPASS, an extension of a
transformer-based model called COMPASS, that integrates biomarkers and treatment information to further improve its generalisability. Instead of feeding biomarker data as input, we built loss components to align them with the model's intermediate representations. We found that components such as treatment gating and pathway consistency loss improved generalisability when evaluated with Leave-one-cohort-out, Leave-one-cancer-type-out and Leave-one-treatment-out strategies. Results show that building components that exploit biomarker and treatment information can help in generalisability of immunotherapy response prediction. Careful curation of additional components that leverage complementary clinical information and domain knowledge represents a promising direction for future research.
Read the full abstract ▾
Machine Learning
Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction
Björn Roman Kohlberger
The
memory wall remains the primary bottleneck for training large
language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight
matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense
matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel
manifold via QR decomposition after each optimizer step. SCT achieves up to 199x
memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak
memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule – not MLP rank – as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU
memory drops 46% at rank 32 while training throughput doubles.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CY
A CEFR-Inspired Classification Framework with Fuzzy C-Means To Automate Assessment of Programming Skills in Scratch
Ricardo Hidalgo-Aragón, Jesús M. González-Barahona and Gregorio Robles
Context: Schools, training platforms, and technology firms increasingly need to assess programming proficiency at scale with transparent, reproducible methods that support personalized learning pathways. Objective: This study introduces a pedagogical framework for Scratch project assessment, aligned with the Common European Framework of Reference (CEFR), providing universal competency levels for students and teachers alongside actionable insights for curriculum design. Method: We apply Fuzzy C-Means clustering to 2008246 Scratch projects evaluated via Dr.Scratch, implementing an ordinal criterion to map clusters to CEFR levels (A1-C2), and introducing enhanced classification metrics that identify transitional learners, enable continuous progress tracking, and quantify classification certainty to balance automated feedback with instructor review. Impact: The framework enables diagnosis of systemic curriculum gaps-notably a "B2 bottleneck" where only 13.3% of learners reside due to the cognitive load of integrating Logic Synchronization, and Data Representation–while providing certainty–based triggers for human intervention.
Read the full abstract ▾
Machine Learning
Exploring Silent Data Corruption as a Reliability Challenge in LLM Training
Anton Altenbernd, Philipp Wiesner and Odej Kao
As Large
Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU
matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and
attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
A Benchmark of State-Space Models vs. Transformers and BiLSTM-based Models for Historical Newspaper OCR
Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas et al. (+2 more)
End-to-end OCR for historical newspapers remains challenging, as models must handle long text sequences, degraded print quality, and complex layouts. While
Transformer-based recognizers dominate current research, their quadratic complexity limits efficient paragraph-level transcription and large-scale deployment. We investigate linear-time State-Space Models (SSMs), specifically Mamba, as a scalable alternative to
Transformer-based sequence modeling for OCR. We present to our knowledge, the first OCR architecture based on SSMs, combining a CNN visual encoder with bi-directional and autoregressive Mamba sequence modeling, and conduct a large-scale
benchmark comparing SSMs with
Transformer- and BiLSTM-based recognizers. Multiple decoding strategies (CTC, autoregressive, and non-autoregressive) are evaluated under identical training conditions alongside strong neural baselines (VAN, DAN, DANIEL) and widely used off-the-shelf OCR engines (PERO-OCR, Tesseract OCR, TrOCR, Gemini). Experiments on historical newspapers from the Bibliotheque nationale du Luxembourg, with newly released >99% verified gold-standard annotations, and cross-
dataset tests on Fraktur and Antiqua lines, show that all neural models achieve low error rates (~2% CER), making computational efficiency the main differentiator. Mamba-based models maintain competitive accuracy while halving inference time and exhibiting superior
memory scaling (1.26x vs 2.30x growth at 1000 chars), reaching 6.07% CER at the severely degraded paragraph level compared to 5.24% for DAN, while remaining 2.05x faster. We release code, trained models, and standardized evaluation protocols to enable reproducible research and guide practitioners in large-scale cultural heritage OCR available at https://github.com/MarcoPerson/ssm-ocr-
benchmark.
Read the full abstract ▾
All authors Merveilles Agbeti-Messan, Pierrick Tranouez, Stéphane Nicolas, Clément Chatelain, Thierry Paquet
Machine Learning · Cross-listed from cs.AI
CircuitProbe: Predicting Reasoning Circuits in Transformers via Stability Zone Detection
Rajkiran Panuganti
Transformer language models contain localized
reasoning circuits, contiguous layer blocks that improve
reasoning when duplicated at inference time. Finding these
circuits currently requires brute-force sweeps costing 25 GPU hours per model. We propose CircuitProbe, which predicts
circuit locations from activation statistics in under 5 minutes on CPU, providing a speedup of three to four orders of magnitude. We find that
reasoning circuits come in two types: stability
circuits in early layers, detected through the derivative of representation change, and magnitude
circuits in late layers, detected through anomaly scoring. We validate across 9 models spanning 6 architectures, including 2025 models, confirming that CircuitProbe top predictions match or are within 2 layers of the optimal
circuit in all validated cases. A scaling experiment across the Qwen 2.5 family reveals that layer duplication consistently benefits models under 3B parameters but degrades performance in 7B+ models, making this a practical scaling technique for small
language models. CircuitProbe requires as few as 10 calibration examples and its predictions are stable across English, Hindi, Chinese, and French.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
Karan Singh, Michael Yu, Varun Gangal et al. (+4 more)
Retrieval-augmented generation (RAG) improves
language model (LM) performance by providing relevant context at test time for knowledge-intensive situations. In this work, we systematically study the trade-off between pretraining and
retrieval by training OLMo-2-based LMs ranging from 30M to 3B parameters on up to 100B DCLM
tokens, while varying pretraining data scale,
retrieval store size, and
retrieval store source (pretraining vs. new data) across
reasoning, scientific QA, and open-domain QA
benchmarks. We find that
retrieval gains depend on model capacity and pretraining exposure and are strongly front-loaded, with a median 91% of the largest observed improvement realized by one
retrieval token per model parameter. However, the interaction is objective-dependent: smaller models gain more in gold-answer perplexity, whereas larger, more-pretrained models gain more in accuracy.
Retrieval from previously seen data also preserves most of the held-out
retrieval gain.
Retrieval is therefore a task-, regime-, and metric-dependent complement to parametric learning whose value also depends on datastore size and information novelty. Overall, this motivates the explicit partitioning of data between internalization and external access for LM design.
Read the full abstract ▾
All authors Karan Singh, Michael Yu, Varun Gangal, Zhuofu Tao, Sachin Kumar, Emmy Liu, Steven Y. Feng