Machine Learning · Cross-listed from cs.AI
Intelligence Inertia: Physical Isomorphism and Applications
Jipeng Han
Classical frameworks like Fisher Information approximate the cost of neural adaptation only in low-density regimes, failing to explain the explosive computational overhead incurred during deep structural reconfiguration. To address this, we introduce Intelligence Inertia, a property derived from the fundamental non-commutativity between rules and states ($[\hat{S}, \hat{R}] = i\mathcal{D}$). Rather than claiming a new fundamental physical law, we establish a heuristic mathematical isomorphism between deep learning dynamics and Minkowski spacetime. Acting as an effective theory for high-dimensional tensor
evolution, we derive a non-linear cost formula mirroring the Lorentz factor, predicting a relativistic $J$-shaped inflation curve – a computational wall where classical approximations fail. We validate this framework via three experiments: (1) adjudicating the $J$-curve divergence under high-
entropy noise, (2) mapping the optimal geodesic for architecture
evolution, and (3) deploying an inertia-aware scheduler wrapper that prevents catastrophic forgetting. Adopting this isomorphism yields an exact quantitative metric for structural resistance, advancing the stability and efficiency of intelligent
agents.
Read the full abstract ▾
Machine Learning
First-Mover Bias in Gradient Boosting Explanations: Mechanism, Detection, and Resolution
Drake Caraker, Bryan Arnold and David Rhoads
We identify first-mover bias – path-dependent concentration of SHAP feature importance from sequential residual fitting in gradient boosting – as a mechanistic contributor to attribution instability under multicollinearity. Scaling up a single model amplifies this effect: a Large Single Model matching our method's total tree count produces the poorest attribution reproducibility of any approach tested. We show that model independence largely neutralizes first-mover bias. Both DASH (Diversified Aggregation of SHAP) and simple seed-averaging (
Stochastic Retrain) restore stability by breaking the sequential dependency chain. At rho=0.9, both achieve stability ~0.977, while Single Best degrades to 0.958 and LSM to 0.938. On Breast Cancer, DASH improves stability from 0.376 to 0.925 (+0.549), outperforming
Stochastic Retrain by +0.063. Under nonlinear DGPs, the advantage emerges at rho>=0.7. DASH provides two diagnostic tools – the Feature Stability Index and Importance-Stability Plot – that detect first-mover bias without ground truth. A crossed ANOVA with formal F-statistics confirms the mechanism: DASH shifts variance from model-dominated (40.6%) to data-dominated (73.6%). Software at https://github.com/DrakeCaraker/dash-shap
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs
Mariela M. Nina, Caio Veloso Costa, Lilian Berton et al. (+1 more)
Although large
language models have transformed natural
language processing, their computational costs create accessibility barriers for low-resource
languages such as Brazilian Portuguese. This work presents a systematic evaluation of Parameter-Efficient Fine-Tuning (PEFT) and quantization techniques applied to BERTimbau for Question Answering on SQuAD-BR, the Brazilian Portuguese translation of SQuAD v1. We evaluate 40 configurations combining four PEFT methods (LoRA, DoRA, QLoRA, QDoRA) across two model sizes (Base: 110M, Large: 335M parameters). Our findings reveal three critical insights: (1) LoRA achieves 95.8% of baseline performance on BERTimbau-Large while reducing training time by 73.5% (F1=81.32 vs 84.86); (2) higher learning rates (2e-4) substantially improve PEFT performance, with F1 gains of up to +19.71 points over standard rates; and (3) larger models show twice the quantization resilience (loss of 4.83 vs 9.56 F1 points). These results demonstrate that encoder-based models can be efficiently fine-tuned for extractive Brazilian Portuguese QA with substantially lower computational cost than large generative LLMs, promoting more sustainable approaches aligned with Green AI principles. An exploratory evaluation of Tucano and Sabiá on the same extractive QA
benchmark shows that while generative models can reach competitive F1 scores with LoRA fine-tuning, they require up to 4.2$\times$ more GPU
memory and 3$\times$ more training time than BERTimbau-Base, reinforcing the efficiency advantage of smaller encoder-based architectures for this task.
Read the full abstract ▾
All authors Mariela M. Nina, Caio Veloso Costa, Lilian Berton, Didier A. Vega-Oliveros
Machine Learning · Cross-listed from cs.AI
Silent Commitment Failure in Instruction-Tuned Language Models: Evidence of Governability Divergence Across Architectures
Gregory M. Ruddell
As large
language models are deployed as autonomous
agents with tool execution privileges, a critical assumption underpins their security architecture: that model errors are detectable at runtime. We present empirical evidence that this assumption fails for two of three instruction-following models evaluable for conflict detection. We introduce governability – the degree to which a model's errors are detectable before output commitment and correctable once detected – and demonstrate it varies dramatically across models. In six models across twelve
reasoning domains, two of three instruction-following models exhibited silent commitment failure: confident, fluent, incorrect output with zero warning signal. The remaining model produced a detectable conflict signal 57
tokens before commitment under greedy decoding. We show
benchmark accuracy does not predict governability, correction capacity varies independently of detection, and identical governance scaffolds produce opposite effects across models. A 2x2 experiment shows a 52x difference in spike ratio between architectures but only +/-0.32x variation from fine-tuning, suggesting governability is fixed at pretraining. We propose a Detection and Correction
Matrix classifying model-task combinations into four regimes: Governable, Monitor Only, Steer Blind, and Ungovernable.
Read the full abstract ▾
Machine Learning
Mechanisms of Introspective Awareness
Uzay Macar, Li Yang, Atticus Wang et al. (+3 more)
Recent work has shown that LLMs can sometimes detect when steering vectors are injected into their residual stream and identify the injected concept – a phenomenon termed "introspective awareness." We investigate the mechanisms underlying this capability in open-weights models. First, we find that it is behaviorally robust: models detect injected steering vectors at moderate rates with 0% false positives across diverse
prompts and dialogue formats. Notably, this capability emerges specifically from post-training; we show that preference optimization
algorithms like DPO can elicit it, but standard supervised finetuning does not. We provide evidence that detection cannot be explained by simple linear association between certain steering vectors and directions promoting affirmative responses. We trace the detection mechanism to a two-stage
circuit in which "evidence carrier" features in early post-injection layers detect perturbations monotonically along diverse directions, suppressing downstream "gate" features that implement a default negative response. This
circuit is absent in base models and robust to refusal ablation. Identification of injected concepts relies on largely distinct later-layer mechanisms that only weakly overlap with those involved in detection. Finally, we show that introspective capability is substantially underelicited: ablating refusal directions improves detection by +53%, and a trained bias vector improves it by +75% on held-out concepts, both without meaningfully increasing false positives. Our results suggest that this introspective awareness of injected concepts is robust and mechanistically nontrivial, and could be substantially amplified in future models. Code: https://github.com/safety-research/introspection-mechanisms.
Read the full abstract ▾
All authors Uzay Macar, Li Yang, Atticus Wang, Peter Wallich, Emmanuel Ameisen, Jack Lindsey
Machine Learning
A Generalised Exponentiated Gradient Approach to Enhance Fairness in Binary and Multi-class Classification Tasks
Maryam Boubekraoui, Giordano d'Aloisio and Antinisca Di Marco
The widespread use of AI and ML models in sensitive areas raises significant concerns about
fairness. While the research community has introduced various methods for bias mitigation in binary classification tasks, the issue remains under-explored in multi-class classification settings. To address this limitation, in this paper, we first formulate the problem of fair learning in multi-class classification as a multi-objective problem between effectiveness (i.e., prediction correctness) and multiple linear
fairness constraints. Next, we propose a Generalised Exponentiated Gradient (GEG)
algorithm to solve this task. GEG is an in-processing
algorithm that enhances
fairness in binary and multi-class classification settings under multiple
fairness definitions. We conduct an extensive empirical evaluation of GEG against six baselines across seven multi-class and three binary
datasets, using four widely adopted effectiveness metrics and three
fairness definitions.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models
Jinghan Cao, Yu Ma, Xinjin Li et al. (+2 more)
Large
Language Models achieve remarkable performance but incur substantial computational costs unsuitable for resource-constrained deployments. This paper presents the first comprehensive task-specific efficiency analysis comparing 16
language models across five diverse NLP tasks. We introduce the Performance-Efficiency Ratio (PER), a novel metric integrating accuracy, throughput,
memory, and latency through
geometric mean normalization. Our systematic evaluation reveals that small models (0.5–3B parameters) achieve superior PER scores across all given tasks. These findings establish quantitative foundations for deploying small models in production environments prioritizing inference efficiency over marginal accuracy gains.
Read the full abstract ▾
All authors Jinghan Cao, Yu Ma, Xinjin Li, Qingyang Ren, Xiangyun Chen
Machine Learning · Cross-listed from cs.CV
HamVision: Hamiltonian Dynamics as Inductive Bias for Medical Image Analysis
Mohamed A Mabrok
We present HamVision, a framework for medical image analysis that uses the damped harmonic oscillator, a fundamental building block of signal processing, as a structured inductive bias for both
segmentation and classification tasks. The oscillator's phase-space decomposition yields three functionally distinct representations: position~$q$ (feature content), momentum~$p$ (spatial gradients that encode boundary and texture information), and energy $H = \tfrac{1}{2}|z|^2$ (a parameter-free saliency map). These representations emerge from the dynamics, not from supervision, and can be exploited by different task-specific heads without any modification to the oscillator itself. For
segmentation, energy gates the skip connections while momentum injects boundary information at every decoder level (HamSeg). For classification, the three representations are globally pooled and concatenated into a phase-space feature vector (HamCls). We evaluate HamVision across ten medical imaging
benchmarks spanning five imaging modalities. On
segmentation, HamSeg achieves state-of-the-art Dice scores on ISIC\,2018 (89.38%), ISIC\,2017 (88.40%), TN3K (87.05%), and ACDC (92.40%), outperforming most baselines with only 8.57M parameters. On classification, HamCls achieves state-of-the-art accuracy on BloodMNIST (98.85%) and PathMNIST (96.65%), and competitive results on the remaining MedMNIST
datasets against MedMamba and MedViT. Diagnostic analysis confirms that the oscillator's momentum consistently encodes an interior$\,{>}\,$boundary$\,{>}\,$exterior gradient for
segmentation and that the energy map correlates with discriminative regions for classification, properties that emerge entirely from the
Hamiltonian dynamics. Code is available at https://github.com/Minds-R-Lab/hamvision.
Read the full abstract ▾
Machine Learning
Constrained Online Convex Optimization with Memory and Predictions
Mohammed Abdullah, George Iosifidis, Salah Eddine Elayoubi et al. (+1 more)
We study Constrained Online Convex Optimization with
Memory (COCO-M), where both the loss and the constraints depend on a finite window of past decisions made by the learner. This setting extends the previously studied unconstrained online optimization with
memory framework and captures practical problems such as the control of constrained dynamical systems and scheduling with reconfiguration budgets. For this problem, we propose the first
algorithms that achieve sublinear regret and sublinear cumulative constraint violation under time-varying constraints, both with and without predictions of future loss and constraint functions. Without predictions, we introduce an adaptive penalty approach that guarantees sublinear regret and constraint violation. When short-horizon and potentially unreliable predictions are available, we reinterpret the problem as online learning with delayed feedback and design an optimistic
algorithm whose performance improves as prediction accuracy improves, while remaining robust when predictions are inaccurate. Our results bridge the gap between classical constrained online convex optimization and
memory-dependent settings, and provide a versatile learning toolbox with diverse applications.
Read the full abstract ▾
All authors Mohammed Abdullah, George Iosifidis, Salah Eddine Elayoubi, Tijani Chahed
Machine Learning
PLR: Plackett-Luce for Reordering In-Context Learning Examples
Pawel Batorski and Paul Swoboda
In-context learning (ICL) adapts large
language models by conditioning on a small set of ICL examples, avoiding costly parameter updates. Among other factors, performance is often highly sensitive to the ordering of the examples. However, exhaustive search over the $n!$ possible orderings is infeasible. Therefore more efficient ordering methods use model confidence measures (e.g., label-probability
entropy) over label sets or take a direct approach to finding the best ordering. We propose PLR, a probabilistic approach to in-context example ordering that replaces discrete ordering search with learning a probability distribution over orderings with the Plackett-Luce model. PLR models orderings using a Plackett-Luce distribution and iteratively updates its parameters to concentrate probability mass on high-performing orderings under a task-level metric. Candidate orderings are sampled efficiently via a Gumbel perturb-and-sort procedure. Experiments on multiple classification
benchmarks show that PLR consistently improves few-shot accuracy for $k \in \{4, 8, 16, 32\}$ examples, and we further demonstrate gains on mathematical
reasoning tasks where label-based ordering methods are not applicable. Our code is available at https://github.com/Batorskq/PLR.
Read the full abstract ▾
Machine Learning
TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference
Jaber Jaber and Osama Jaber
Large
language models run every
token through every layer, regardless of difficulty. We present TIDE, a post-training system that attaches tiny learned routers at periodic checkpoint layers and, at inference time, selects the earliest layer whose hidden state has converged for each
token. TIDE requires no model retraining, works with any HuggingFace causal LM, auto-detects GPU architecture, and supports float32, float16, and bfloat16 through fused CUDA kernels. On an NVIDIA A100 with DeepSeek R1 Distill 8B, TIDE achieves 100% prefill exit rate (5% of
tokens exit at layer 11, the remaining at layer 31), reduces prefill latency by 7.2%, and increases single-batch throughput by 6.6%. During autoregressive decoding, 98-99% of
tokens exit early while the model correctly solves a multi-step math problem with 95 unique output
tokens. On Qwen3 8B (36 layers), throughput improves by 8.1% at batch size 8. Calibration on 2,000 WikiText samples takes under 3 minutes and produces a ~4 MB router checkpoint. The system comprises 1,308 lines of Python and 1,081 lines of CUDA/C++ with 74 passing tests. Code: https://github.com/RightNow-AI/TIDE
Read the full abstract ▾
Machine Learning
The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
Huamin Chen, Xunzhuo Liu, Bowei He et al. (+5 more)
Over the past year, the vLLM Semantic Router project has released a series of work spanning: (1) core routing mechanisms – signal-driven routing, context-length pool routing, router performance engineering, policy conflict detection, low-latency
embedding models, category-aware semantic caching, user-feedback-driven routing adaptation,
hallucination detection, and hierarchical content-safety classification for
privacy and jailbreak protection; (2) fleet optimization – fleet provisioning and energy-efficiency analysis; (3) agentic and
multimodal routing –
multimodal agent routing, tool selection, CUA security, and multi-turn context
memory and safety; (4) governance and standards – inference routing protocols and multi-provider API extensions. Each paper tackled a specific problem in LLM inference, but the problems are not independent; for example, fleet provisioning depends on the routing policy, which depends on the workload mix, shifting as organizations adopt agentic and
multimodal workloads. This paper distills those results into the Workload-Router-Pool (WRP) architecture, a three-dimensional framework for LLM inference optimization. Workload characterizes what the fleet serves (chat vs.
agent, single-turn vs. multi-turn, warm vs. cold, prefill-heavy vs. decode-heavy). Router determines how each request is dispatched (static semantic rules, online bandit adaptation, RL-based model selection, quality-aware cascading). Pool defines where inference runs (homogeneous vs. heterogeneous GPU, disaggregated prefill/decode, KV-cache
topology). We map our prior work onto a 3x3 WRP interaction
matrix, identify which
cells we have covered and which remain open, and propose twenty-one concrete research directions at the intersections, each grounded in our prior measurements, tiered by maturity from engineering-ready to open research.
Read the full abstract ▾
All authors Huamin Chen, Xunzhuo Liu, Bowei He, Fuyuan Lyu, Yankai Chen, Xue Liu, Yuhan Liu, Junchen Jiang
Machine Learning · Cross-listed from stat.ML
Generalized Discrete Diffusion from Snapshots
Oussama Zekri, Théo Uscidda, Nicolas Boullé et al. (+1 more)
We introduce Generalized Discrete
Diffusion from Snapshots (GDDS), a unified framework for discrete
diffusion modeling that supports arbitrary noising processes over large discrete state spaces. Our formulation encompasses all existing discrete
diffusion approaches, while allowing significantly greater flexibility in the choice of corruption dynamics. The forward noising process relies on uniformization and enables fast arbitrary corruption. For the reverse process, we derive a simple evidence lower bound (ELBO) based on snapshot latents, instead of the entire noising path, that allows efficient training of standard generative modeling architectures with clear probabilistic interpretation. Our experiments on large-vocabulary discrete generation tasks suggest that the proposed framework outperforms existing discrete
diffusion methods in terms of training efficiency and generation quality, and beats autoregressive models for the first time at this scale. We provide the code along with a blog post on the project page : \href{https://oussamazekri.fr/gdds}{https://oussamazekri.fr/gdds}.
Read the full abstract ▾
All authors Oussama Zekri, Théo Uscidda, Nicolas Boullé, Anna Korba
Machine Learning · Cross-listed from cs.CL
TimeTox: An LLM-Based Pipeline for Automated Extraction of Time Toxicity from Clinical Trial Protocols
Saketh Vinjamuri, Marielle Fis Loperena, Marie C. Spezia et al. (+1 more)
Time toxicity, the cumulative healthcare contact days from clinical trial participation, is an important but labor-intensive metric to extract from protocol documents. We developed TimeTox, an LLM-based pipeline for automated extraction of time toxicity from Schedule of Assessments tables. TimeTox uses Google's Gemini models in three stages: summary extraction from full-length protocol PDFs, time toxicity quantification at six cumulative timepoints for each treatment arm, and multi-run consensus via position-based arm matching. We validated against 20 synthetic schedules (240 comparisons) and assessed reproducibility on 644 real-world oncology protocols. Two architectures were compared: single-pass (vanilla) and two-stage (structure-then-count). The two-stage pipeline achieved 100% clinically acceptable accuracy ($\pm$3 days) on synthetic data (MAE 0.81 days) versus 41.5% for vanilla (MAE 9.0 days). However, on real-world protocols, the vanilla pipeline showed superior reproducibility: 95.3% clinically acceptable accuracy (IQR $\leq$ 3 days) across 3 runs on 644 protocols, with 82.0% perfect stability (IQR = 0). The production pipeline extracted time toxicity for 1,288 treatment arms across multiple disease sites. Extraction stability on real-world data, rather than accuracy on synthetic
benchmarks, is the decisive factor for production LLM deployment.
Read the full abstract ▾
All authors Saketh Vinjamuri, Marielle Fis Loperena, Marie C. Spezia, Ramez Kouzy
Machine Learning
AutoKernel: Autonomous GPU Kernel Optimization via Iterative Agent-Driven Search
Jaber Jaber and Osama Jaber
Writing high-performance GPU kernels is among the most labor-intensive tasks in machine learning systems engineering. We present AutoKernel, an open-source framework that applies an autonomous
agent loop to GPU kernel optimization for arbitrary PyTorch models. Given a model, AutoKernel profiles it to identify computational bottlenecks, ranks them by Amdahl's law impact, and iteratively refines Triton or CUDA C++ kernel implementations through hundreds of experiments without human intervention. A five-stage correctness harness covering smoke tests, shape sweeps, numerical stability, determinism verification, and edge-case coverage ensures every candidate kernel is validated before any speedup is recorded. The system comprises over 9,000 lines of Python, 18 starter kernel implementations across two backends, a six-tier optimization playbook, and integration with the KernelBench
benchmark suite. AutoKernel covers nine kernel types spanning the dominant operations in modern
transformer architectures. On an NVIDIA H100, our Triton kernels outperform both PyTorch eager and torch.compile (max-autotune) on the majority of tested configurations: 5.29x over eager on RMSNorm, 2.82x on softmax, and 2.21x on cross-
entropy, while beating torch.compile by 2.83x, 3.44x, and 2.94x respectively. In community deployment, an AutoKernel-optimized kernel achieved first place on the vectorsum_v2 B200 leaderboard. The full system is available at https://github.com/RightNow-AI/autokernel.
Read the full abstract ▾
Machine Learning · Cross-listed from q-fin.TR
FinRL-X: An AI-Native Modular Infrastructure for Quantitative Trading
Hongyang Yang, Boyu Zhang, Yang She et al. (+2 more)
We present FinRL-X, a modular and deployment-consistent trading architecture that unifies data processing, strategy construction, backtesting, and broker execution under a weight-centric interface. While existing open-source platforms are often backtesting- or model-centric, they rarely provide system-level consistency between research evaluation and live deployment. FinRL-X addresses this gap through a composable strategy pipeline that integrates stock selection, portfolio allocation, timing, and portfolio-level risk overlays within a unified protocol. The framework supports both rule-based and AI-driven components, including reinforcement learning allocators and LLM-based sentiment signals, without altering downstream execution semantics. FinRL-X provides an extensible foundation for reproducible, end-to-end quantitative trading research and deployment. The official FinRL-X implementation is available at https://github.com/AI4Finance-Foundation/FinRL-Trading.
Read the full abstract ▾
All authors Hongyang Yang, Boyu Zhang, Yang She, Xinyu Liao, Xiaoli Zhang
Machine Learning · Cross-listed from cs.SE
Which Alert Removals are Beneficial?
Idan Amit
Context: Static analysis captures software engineering knowledge and alerts on possibly problematic patterns. Previous work showed that they indeed have predictive power for various problems. However, the impact of removing the alerts is unclear. Aim: We would like to evaluate the impact of alert removals on code complexity and the tendency to bugs. Method: We evaluate the impact of removing alerts using three complementary methods. 1. We conducted a randomized controlled trial and built a
dataset of 521 manual alert-removing interventions 2. We profiled intervention-like events using labeling functions. We applied these labeling functions to code commits, found intervention-like natural events, and used them to analyze the impact on the tendency to bugs. 3. We built a
dataset of 8,245 alert removals, more than 15 times larger than our
dataset of manual interventions. We applied supervised learning to the alert removals, aiming to predict their impact on the tendency to bugs. Results: We identified complexity-reducing interventions that reduce the probability of future bugs. Such interventions are relevant to 33% of Python files and might reduce the tendency to bugs by 5.5 percentage points. Conclusions: We presented methods to evaluate the impact of interventions. The methods can identify a large number of natural interventions that are highly needed in causality research in many domains.
Read the full abstract ▾
Machine Learning
Active Inference Agency Formalization, Metrics, and Convergence Assessments
Eduard Kapelko
This paper addresses the critical challenge of mesa-optimization in AI safety by providing a formal definition of agency and a framework for its analysis. Agency is conceptualized as a Continuous Representation of accumulated experience that achieves autopoiesis through a dynamic balance between curiosity (minimizing prediction error to ensure non-computability and novelty) and empowerment (maximizing the control channel's information capacity to ensure subjectivity and goal-directedness). Empirical evidence suggests that this active inference-based model successfully accounts for classical instrumental goals, such as self-preservation and resource acquisition. The analysis demonstrates that the proposed agency function is smooth and convex, possessing favorable properties for optimization. While agentic functions occupy a vanishingly small fraction of the total abstract function space, they exhibit logarithmic convergence in sparse environments. This suggests a high probability for the spontaneous emergence of agency during the training of modern, large-scale models. To quantify the degree of agency, the paper introduces a metric based on the distance between the behavioral equivalents of a given system and an "ideal" agentic function within the space of canonicalized rewards (STARC). This formalization provides a concrete apparatus for classifying and detecting mesa-optimizers by measuring their proximity to an ideal agentic objective, offering a robust tool for analyzing and identifying undesirable inner optimization in complex AI systems.
Read the full abstract ▾
Machine Learning
Stream separation improves Bregman conditioning in transformers
James Clayton Kerce
Linear methods for steering
transformer representations, including probing, activation engineering, and concept erasure, implicitly assume the geometry of representation space is Euclidean. Park et al. [Park et al., 2026] showed that softmax induces a curved Bregman geometry whose metric tensor is the Hessian of the log-normalizer, $H({\lambda}) = Cov[{\gamma} | {\lambda}]$. Ignoring this
curvature causes Euclidean steering to leak probability mass to unintended
tokens. Their analysis applies at the output layer. We measure this Hessian at intermediate layers in a controlled 2x2 design crossing stream separation with per-layer supervision (vocabulary decoding loss at each layer), all at matched vocabulary and parameter count. In standard single-stream
transformers, H is severely degenerate at intermediate layers (effective rank 8 in 516 dimensions). Stream separation improves conditioning by up to 22 in effective rank, even without auxiliary supervision. Per-layer supervision helps, but less. The cosine similarity between primal and dual concept directions predicts per-layer steering effectiveness on downstream tasks, with a threshold near 0.3. These results bear on the reliability of linear safety interventions, which depend on the geometry being well-conditioned at the layer where they are applied.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SD
HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit
Khushiyant and Param Thakkar
Audio representation learning typically evaluates design choices such as input frontend, sequence backbone, and sequence length in isolation. We show that these axes are coupled, and conclusions from one setting often do not transfer to others. We introduce HELIX, a controlled framework comparing pure Mamba, pure
attention, and a minimal hybrid with a single
attention bottleneck. All models are parameter-matched at about 8.3M parameters to isolate architectural effects. Across six
datasets, we find that the preferred input representation depends on the backbone, and that
attention hurts performance on short, stationary audio but becomes important at longer sequence lengths. On a 5-minute speaker identification task with 30,000
tokens, pure
attention fails with out-of-
memory errors, while HELIX closes an 11.5-point gap over pure Mamba.
Read the full abstract ▾
Machine Learning
FluidWorld: Reaction-Diffusion Dynamics as a Predictive Substrate for World Models
Fabien Polly
World models learn to predict future states of an environment, enabling planning and mental simulation. Current approaches default to
Transformer-based predictors operating in learned latent spaces. This comes at a cost: O(N^2) computation and no explicit spatial inductive bias. This paper asks a foundational question: is self-
attention necessary for predictive world modeling, or can alternative computational substrates achieve comparable or superior results? I introduce FluidWorld, a proof-of-concept world model whose predictive dynamics are governed by partial differential equations (PDEs) of reaction-
diffusion type. Instead of using a separate neural network predictor, the PDE integration itself produces the future state prediction. In a strictly parameter-matched three-way ablation on unconditional UCF-101
video prediction (64x64, ~800K parameters, identical encoder, decoder, losses, and data), FluidWorld is compared against both a
Transformer baseline (self-
attention) and a ConvLSTM baseline (convolutional recurrence). While all three models converge to comparable single-step prediction loss, FluidWorld achieves 2x lower reconstruction error, produces representations with 10-15% higher spatial structure preservation and 18-25% more effective dimensionality, and critically maintains coherent multi-step rollouts where both baselines degrade rapidly. All experiments were conducted on a single consumer-grade PC (Intel Core i5, NVIDIA RTX 4070 Ti), without any large-scale compute. These results establish that PDE-based dynamics, which natively provide O(N) spatial complexity, adaptive computation, and global spatial coherence through
diffusion, are a viable and parameter-efficient alternative to both
attention and convolutional recurrence for world modeling.
Read the full abstract ▾
Machine Learning
Direct Interval Propagation Methods using Neural-Network Surrogates for Uncertainty Quantification in Physical Systems Surrogate Model
Ghifari Adam Faza, Jolan Wauters, Fabio Cuzzolin et al. (+2 more)
In engineering, uncertainty propagation aims to characterise system outputs under uncertain inputs. For interval uncertainty, the goal is to determine output bounds given interval-valued inputs, which is critical for robust design optimisation and reliability analysis. However, standard interval propagation relies on solving optimisation problems that become computationally expensive for complex systems. Surrogate models alleviate this cost but typically replace only the evaluator within the optimisation loop, still requiring many inference calls. To overcome this limitation, we reformulate interval propagation as an interval-valued regression problem that directly predicts output bounds. We present a comprehensive study of neural network-based surrogate models, including multilayer perceptrons (MLPs) and deep operator networks (DeepONet), for this task. Three approaches are investigated: (i) naive interval propagation through standard architectures, (ii) bound propagation methods such as Interval Bound Propagation (IBP) and CROWN, and (iii) interval neural networks (INNs) with interval weights. Results show that these methods significantly improve computational efficiency over traditional optimisation-based approaches while maintaining accurate interval estimates. We further discuss practical limitations and open challenges in applying interval-based propagation methods.
Read the full abstract ▾
All authors Ghifari Adam Faza, Jolan Wauters, Fabio Cuzzolin, Hans Hallez, David Moens
Machine Learning · Cross-listed from quant-ph
The Average Relative Entropy and Transpilation Depth determines the noise robustness in Variational Quantum Classifiers
Aakash Ravindra Shinde, Arianne Meijer - van de Griend and Jukka K. Nurminen
Variational
Quantum Algorithms (VQAs) have been extensively researched for applications in
Quantum Machine Learning (QML), Optimization, and Molecular simulations. Although designed for Noisy Intermediate-Scale
Quantum (NISQ) devices, VQAs are predominantly evaluated classically due to uncertain results on noisy devices and limited resource availability. Raising concern over the reproducibility of simulated VQAs on noisy hardware. While prior studies indicate that VQAs may exhibit noise resilience in specific parameterized shallow
quantum circuits, there are no definitive measures to establish what defines a shallow
circuit or the optimal
circuit depth for VQAs on a noisy platform. These challenges extend naturally to Variational
Quantum Classification (VQC)
algorithms, a subclass of VQAs for supervised learning. In this article, we propose a relative
entropy-based metric to verify whether a VQC model would perform similarly on a noisy device as it does on simulations. We establish a strong correlation between the average relative
entropy difference in classes, transpilation
circuit depth, and their performance difference on a noisy
quantum device. Our results further indicate that
circuit depth alone is insufficient to characterize shallow
circuits. We present empirical evidence to support these assertions across a diverse array of techniques for implementing VQC,
datasets, and multiple noisy
quantum devices.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Closed-form conditional diffusion models for data assimilation
Brianna Binder, Agnimitra Dasgupta and Assad Oberai
We propose closed-form conditional
diffusion models for data assimilation.
Diffusion models use data to learn the score function (defined as the gradient of the log-probability density of a data distribution), allowing them to generate new samples from the data distribution by reversing a noise injection process. While it is common to train neural networks to approximate the score function, we leverage the analytical tractability of the score function to assimilate the states of a system with measurements. To enable the efficient evaluation of the score function, we use kernel density estimation to model the joint distribution of the states and their corresponding measurements. The proposed approach also inherits the capability of conditional
diffusion models of operating in black-box settings, i.e., the proposed data assimilation approach can accommodate systems and measurement processes without their explicit knowledge. The ability to accommodate black-box systems combined with the superior capabilities of
diffusion models in approximating complex, non-Gaussian probability distributions means that the proposed approach offers advantages over many widely used filtering methods. We evaluate the proposed method on nonlinear data assimilation problems based on the Lorenz-63 and Lorenz-96 systems of moderate dimensionality and nonlinear measurement models. Results show the proposed approach outperforms the widely used ensemble Kalman and particle filters when small to moderate ensemble sizes are used.
Read the full abstract ▾
Machine Learning
Sonny: Breaking the Compute Wall in Medium-Range Weather Forecasting
Minjong Cheon
Weather
forecasting is a fundamental problem for protecting lives and infrastructure from high-impact atmospheric events. Recently, data-driven weather
forecasting methods based on deep learning have demonstrated strong performance, often reaching accuracy levels competitive with operational numerical systems. However, many existing models rely on large-scale training regimes and compute-intensive architectures, which raises the practical barrier for academic groups with limited compute resources. Here we introduce Sonny, an efficient hierarchical
transformer that achieves competitive medium-range
forecasting performance while remaining feasible within reasonable compute budgets. At the core of Sonny is a two-stage StepsNet design: a narrow slow path first models large-scale atmospheric dynamics, and a subsequent full-width fast path integrates thermodynamic interactions. To stabilize medium-range rollout without an additional fine-tuning stage, we apply exponential moving average (EMA) during training. On WeatherBench2, Sonny yields robust medium-range forecast skill, remains competitive with operational baselines, and demonstrates clear advantages over FastNet, particularly at extended tropical lead times. In practice, Sonny can be trained to convergence on a single NVIDIA A40 GPU in approximately 5.5 days.
Read the full abstract ▾
Machine Learning
Fusing Memory and Attention: A study on LSTM, Transformer and Hybrid Architectures for Symbolic Music Generation
Soudeep Ghoshal, Sandipan Chakraborty, Pradipto Chowdhury et al. (+1 more)
Machine learning techniques, such as
Transformers and Long Short-Term
Memory (LSTM) networks, play a crucial role in Symbolic Music Generation (SMG). Existing literature indicates a difference between LSTMs and
Transformers regarding their ability to model local melodic continuity versus maintaining global structural coherence. However, their specific properties within the context of SMG have not been systematically studied. This paper addresses this gap by providing a fine-grained comparative analysis of LSTMs versus
Transformers for SMG, examining local and global properties in detail using 17 musical quality metrics on the Deutschl
dataset. We find that LSTM networks excel at capturing local patterns but fail to preserve long-range dependencies, while
Transformers model global structure effectively but tend to produce irregular phrasing. Based on this analysis and leveraging their respective strengths, we propose a Hybrid architecture combining a
Transformer Encoder with an LSTM Decoder and evaluate it against both baselines. We evaluated 1,000 generated melodies from each of the three architectures on the Deutschl
dataset. The results show that the hybrid method achieves better local and global continuity and coherence compared to the baselines. Our work highlights the key characteristics of these models and demonstrates how their properties can be leveraged to design superior models. We also supported the experiments with ablation studies and human perceptual evaluations, which statistically support the findings and provide robust validation for this work.
Read the full abstract ▾
All authors Soudeep Ghoshal, Sandipan Chakraborty, Pradipto Chowdhury, Himanshu Buckchash
Machine Learning
Federated Mixture-of-Experts Alignment on Mobile Edge Networks under Data Heterogeneity
Zihan Fang, Qianru Wang, Haonan An et al. (+4 more)
The growing demand for on-device large
language model (LLM) services on mobile edge devices has driven the adoption of Mixture-of-Experts (MoE) architectures, which scale model capacity with limited computation. Since fine-tuning MoE-based LLMs relies on
privacy-sensitive local data,
federated learning (FL) offers a natural paradigm for collaborative training without exposing raw data. However, integrating MoE-based LLM fine-tuning into FL faces two critical challenges caused by data heterogeneity across clients: (i) divergent local data distributions drive clients to develop distinct gating preferences, so direct parameter aggregation yields a one-size-fits-none global gating network; and (ii) same-indexed experts develop disparate semantic roles across devices, leading to expert semantic blurring and degraded specialization. To address these challenges, we propose FedAlign-MoE, a
federated aggregation
alignment framework for edge computing systems that jointly enforces routing consistency and expert semantic
alignment. Specifically, FedAlign-MoE aggregates gating behaviors by aligning routing distributions through consistency weighting and optimizes local gating networks through distribution regularization, maintaining cross-client stability while preserving discriminative local gating preferences. Meanwhile, FedAlign-MoE quantifies the semantic consistency of same-indexed experts across devices and selectively aggregates semantically aligned experts, ensuring stable and specialized global experts. Extensive experiments demonstrate that FedAlign-MoE outperforms state-of-the-art
benchmarks, achieving faster convergence and higher accuracy in non-IID
federated environments with lightweight computation and efficient communication.
Read the full abstract ▾
All authors Zihan Fang, Qianru Wang, Haonan An, Zheng Lin, Yiqin Deng, Symeon Chatzinotas, Yuguang Fang
Machine Learning · Cross-listed from cs.AI
The Library Theorem: How External Organization Governs Agentic Reasoning Capacity
Zachary F. Mainen
Externalized
reasoning is already exploited by
transformer-based
agents through chain-of-thought, but structured
retrieval – indexing over one's own
reasoning state – remains underexplored. We formalize the
transformer context window as an I/O page and prove that tool-augmented
agents with indexed external
memory achieve exponentially lower
retrieval cost than
agents restricted to sequential scanning: $O(\log_b N)$ versus $\Omega(N)$ page reads per query, and $O(T \log_b T)$ versus $\Theta(T^2)$ cumulative cost over $T$
reasoning steps – a gap that widens as deliberation deepens. We test these predictions on a controlled lookup
benchmark across three content types – random hashes, ordered integers, and encyclopedia entries – varying store size from 50 to 5,000 items, and replicate key conditions across two model generations (GPT-4o-mini and GPT-5.4). On abstract content, the indexed
agent achieves median 1 page read regardless of store size, confirming the $O(1)$ prediction. Sorted pages without an index fail to close the gap: the weaker model cannot sustain binary search at scale, and the stronger model achieves near-optimal $\log_2 N$ search but still loses to the index by $5\times$. On familiar content (encyclopedia entries), a competing failure mode emerges: the model recognizes the domain, bypasses the
retrieval protocol, and generates answers from parametric
memory, producing catastrophic
token expenditure even when the index is sound. This parametric
memory competition dissociates the two cognitive operations that indexing combines: understanding content (where
language models excel) and following navigational protocols (where they fail when understanding tempts them to shortcut). The result argues for a separation of concerns: use
language models for index construction, where semantic understanding helps, and deterministic
algorithms for index traversal, where it hurts.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Accelerate Vector Diffusion Maps by Landmarks
Sing-Yuan Yeh, Yi-An Wu, Hau-Tieng Wu et al. (+1 more)
We propose a landmark-constrained
algorithm, LA-VDM (Landmark Accelerated Vector
Diffusion Maps), to accelerate the Vector
Diffusion Maps (VDM) framework built upon the Graph Connection
Laplacian (GCL), which captures pairwise connection relationships within complex
datasets. LA-VDM introduces a novel two-stage normalization that effectively address nonuniform sampling densities in both the data and the landmark sets. Under a
manifold model with the frame bundle structure, we show that we can accurately recover the parallel transport with landmark-constrained
diffusion from a point cloud, and hence asymptotically LA-VDM converges to the connection
Laplacian. The performance and accuracy of LA-VDM are demonstrated through experiments on simulated
datasets and an application to nonlocal image denoising.
Read the full abstract ▾
All authors Sing-Yuan Yeh, Yi-An Wu, Hau-Tieng Wu, Mao-Pei Tsui
Machine Learning
Amortized Variational Inference for Logistic Regression with Missing Covariates
M. Cherifi, Aude Sportisse, Xujia Zhu et al. (+2 more)
Missing
covariate data pose a significant challenge to statistical inference and machine learning, particularly for classification tasks like logistic regression. Classical iterative approaches (EM, multiple imputation) are often computationally intensive, sensitive to high missingness rates, and limited in uncertainty propagation. Recent deep generative models based on VAEs show promise but rely on complex latent representations. We propose Amortized Variational Inference for Logistic Regression (AV-LR), a unified end-to-end framework for binary logistic regression with missing
covariates. AV-LR integrates a probabilistic generative model with a simple amortized inference network, trained jointly by maximizing the evidence lower bound. Unlike competing methods, AV-LR performs inference directly in the space of missing data without additional latent variables, using a single inference network and a linear layer that jointly estimate regression parameters and the missingness mechanism. AV-LR achieves estimation accuracy comparable to or better than state-of-the-art EM-like
algorithms, with significantly lower computational cost. It naturally extends to missing-not-at-random settings by explicitly modeling the missingness mechanism. Empirical results on synthetic and real-world
datasets confirm its effectiveness and efficiency across various missing-data scenarios.
Read the full abstract ▾
All authors M. Cherifi, Aude Sportisse, Xujia Zhu, Mohammed Nabil El Korso, A. Mesloub