Machine Learning · Cross-listed from cs.CL
An Empirical Recipe for Universal Phone Recognition
Shikhar Bharadwaj, Chin-Jou Li, Kwanghee Choi et al. (+4 more)
Phone recognition (PR) is a key enabler of
multilingual and low-resource
speech processing tasks, yet robust performance remains elusive. Highly performant English-focused models do not generalize across
languages, while
multilingual models underutilize pretrained representations. It also remains unclear how data scale, architecture, and training objective contribute to
multilingual PR. We present PhoneticXEUS – trained on large-scale
multilingual data and achieving state-of-the-art performance on both
multilingual (17.7% PFER) and accented English
speech (10.6% PFER). Through controlled ablations with evaluations across 100+
languages under a unified scheme, we empirically establish our training recipe and quantify the impact of SSL representations, data scale, and loss objectives. In addition, we analyze error patterns across
language families, accented
speech, and articulatory features. All data and code are released openly at https://github.com/changelinglab/PhoneticXeus
Read the full abstract ▾
All authors Shikhar Bharadwaj, Chin-Jou Li, Kwanghee Choi, Eunjung Yeo, William Chen, Shinji Watanabe, David R. Mortensen
Machine Learning
A Latent Risk-Aware Machine Learning Approach for Predicting Operational Success in Clinical Trials based on TrialsBank
Iness Halimi, Emmanuel Piffo, Oumnia Boudersa et al. (+11 more)
Clinical trials are characterized by high costs, extended timelines, and substantial operational risk, yet reliable prospective methods for predicting trial success before initiation remain limited. Existing artificial intelligence approaches often focus on isolated metrics or specific development stages and frequently rely on variables unavailable at the trial design phase, limiting real-world applicability. We present a hierarchical latent risk-aware machine learning framework for prospective prediction of clinical trial operational success using a curated subset of TrialsBank, a proprietary AI-ready database developed by Sorintellis, comprising 13,700 trials. Operational success was defined as the ability to initiate, conduct, and complete a clinical trial according to planned timelines, recruitment targets, and protocol specifications through database lock. This approach decomposes operational success prediction into two modeling stages. First, intermediate latent operational risk factors are predicted using more than 180 drug- and trial-level features available before trial initiation. These predicted latent risks are then integrated into a downstream model to estimate the probability of operational success. A staged data-splitting strategy was employed to prevent information leakage, and models were benchmarked using XGBoost, CatBoost, and Explainable Boosting Machines. Across Phase I-III, the framework achieves strong out-of-sample performance, with F1-scores of 0.93, 0.92, and 0.91, respectively. Incorporating latent risk drivers improves discrimination of operational failures, and performance remains robust under independent inference evaluation. These results demonstrate that clinical trial operational success can be prospectively forecasted using a latent risk-aware AI framework, enabling early risk assessment and supporting data-driven clinical development decision-making.
Read the full abstract ▾
All authors Iness Halimi, Emmanuel Piffo, Oumnia Boudersa, Yvan Marcel Carre Vilmorin, Melissa Ait-ikhlef, Karima Kone, Andy Tan, Augustin Medina, Juliette Hernando, Sheila Ernest, Vatche Bartekian, Karine Lalonde, Mireille E Schnitzer, Gianolli Dorcelus
Machine Learning
From Astronomy to Astrology: Testing the Illusion of Zodiac-Based Personality Prediction with Machine Learning
Abhinna Sundar Samantaray, Finnja Annika Fluhrer, Dhruv Saini et al. (+3 more)
Astrology has long been used to interpret human personality, estimate compatibility, and guide social decision-making. Zodiac-based systems in particular remain culturally influential across much of the world, including in South Asian societies where astrological
reasoning can shape marriage matching, naming conventions, ritual timing, and broader life planning. Despite this persistence, astrology has never established either a physically plausible mechanism or a statistically reliable predictive foundation. In this work, we examine zodiac-based personality prediction using a controlled machine-learning framework. We construct a synthetic
dataset in which individuals are assigned zodiac signs and personality labels drawn from a shared pool of 100 broadly human traits. Each sign is associated with a subset of 10 common descriptors, intentionally overlapping with those assigned to other signs, thereby reproducing the ambiguity characteristic of practical astrological systems. We then train Logistic Regression, Random Forest, and neural-network classifiers to infer personality labels from zodiac-based features and nuisance
covariates. Across all experiments, predictive performance remains at or near random expectation, while shuffled-label controls yield comparable accuracies. We argue that the apparent success of astrology arises not from measurable predictive structure, but from trait universality, category overlap, cognitive biases such as the Barnum effect and confirmation bias, and the interpretive flexibility of astrologers and pundits. We conclude that zodiac-based systems do not provide reliable information for predicting human behavior and instead function as culturally durable narrative frameworks. This paper is intended as a humorous academic exercise.
Read the full abstract ▾
All authors Abhinna Sundar Samantaray, Finnja Annika Fluhrer, Dhruv Saini, Omkar Charaple, Anish Kumar Singh, Dhruv Vansraj Rathore
Machine Learning
Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidance
Siva Kumar Sastry Hari, Vignesh Balaji, Sana Damani et al. (+2 more)
Optimizing GPU kernels with LLM
agents is an iterative process over a large design space. Every candidate must be generated, compiled, validated, and profiled, so fewer trials will save both runtime and cost. We make two key observations. First, the abstraction level that
agents operate at is important. If it is too low, the LLM wastes
reasoning on low-impact details. If it is too high, it may miss important optimization choices. Second,
agents cannot easily tell when they reach the point of diminishing returns, wasting resources as they continue searching. These observations motivate two design principles to improve efficiency: (1) a compact domain-specific
language (DSL) that can be learned in context and lets the model reason at a higher level while preserving important optimization levers, and (2) Speed-of-Light (SOL) guidance that uses first-principles performance bounds to steer and budget search. We implement these principles in $\mu$CUTLASS, a DSL with a compiler for CUTLASS-backed GPU kernels that covers kernel configuration, epilogue fusion, and multi-stage pipelines. We use SOL guidance to estimate headroom and guide optimization trials, deprioritize problems that are near SOL, and flag kernels that game the
benchmark. On 59 KernelBench problems with the same iteration budgets, switching from generating low-level code to DSL code using GPT-5-mini turns a 0.40x geomean regression into a 1.27x speedup over PyTorch. Adding SOL-guided steering raises this to 1.56x. Across model tiers, $\mu$CUTLASS + SOL-guidance lets weaker models outperform stronger baseline
agents at lower
token cost. SOL-guided budgeting saves 19-43% of
tokens while retaining at least 95% of geomean speedup, with the best policy reaching a 1.68x efficiency gain. Lastly, SOL analysis helps detect
benchmark-gaming cases, where kernels may appear fast while failing to perform the intended computation.
Read the full abstract ▾
All authors Siva Kumar Sastry Hari, Vignesh Balaji, Sana Damani, Qijing Huang, Christos Kozyrakis
Machine Learning · Cross-listed from math.OC
Transfer Learning in Bayesian Optimization for Aircraft Design
Ali Tfaily, Youssef Diouane, Nathalie Bartoli et al. (+1 more)
The use of transfer learning within
Bayesian optimization addresses the disadvantages of the so-called cold start problem by using source data to aid in the optimization of a target problem. We present a method that leverages an ensemble of surrogate models using transfer learning and integrates it in a constrained
Bayesian optimization framework. We identify challenges particular to aircraft design optimization related to heterogeneous design variables and constraints. We propose the use of a partial-least-squares dimension reduction
algorithm to address design space heterogeneity, and a meta data surrogate selection method to address constraint heterogeneity. Numerical
benchmark problems and an aircraft conceptual design optimization problem are used to demonstrate the proposed methods. Results show significant improvement in convergence in early optimization iterations compared to standard
Bayesian optimization, with improved prediction accuracy for both objective and constraint surrogate models.
Read the full abstract ▾
All authors Ali Tfaily, Youssef Diouane, Nathalie Bartoli, Michael Kokkolaras
Machine Learning
An Explicit Surrogate for Gaussian Mixture Flow Matching with Wasserstein Gap Bounds
Elham Rostami, Taous-Meriem Laleg-Kirati and Hamidou Tembine
We study training-free flow matching between two Gaussian mixture models (GMMs) using explicit velocity fields that transport one mixture into the other over time. Our baseline approach constructs component-wise Gaussian paths with affine velocity fields satisfying the continuity equation, which yields to a closed-form surrogate for the pairwise kinetic transport cost. In contrast to the exact Gaussian Wasserstein cost, which relies on
matrix square-root computations, the surrogate admits a simple analytic expression derived from the kinetic energy of the induced flow. We then analyze how closely this surrogate approximates the exact cost. We prove second-order agreement in a local commuting regime and derive an explicit cubic error bound in the local commuting regime. To handle nonlocal regimes, we introduce a path-splitting strategy that localizes the covariance
evolution and enables piecewise application of the bound. We finally compare the surrogate with an exact construction based on the Gaussian Wasserstein geodesic and summarize the results in a practical regime map showing when the surrogate is accurate and the exact method is preferable.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Mimosa Framework: Toward Evolving Multi-Agent Systems for Scientific Research
Martin Legrand, Tao Jiang, Matthieu Feraud et al. (+5 more)
Current Autonomous Scientific Research (ASR) systems, despite leveraging large
language models (LLMs) and agentic architectures, remain constrained by fixed workflows and toolsets that prevent adaptation to evolving tasks and environments. We introduce Mimosa, an evolving multi-
agent framework that automatically synthesizes task-specific multi-
agent workflows and iteratively refines them through experimental feedback. Mimosa leverages the Model Context Protocol (MCP) for dynamic tool discovery, generates workflow
topologies via a meta-orchestrator, executes subtasks through code-generating
agents that invoke available tools and scientific software libraries, and scores executions with an LLM-based judge whose feedback drives workflow refinement. On ScienceAgentBench, Mimosa achieves a success rate of 43.1% with DeepSeek-V3.2, surpassing both single-
agent baselines and static multi-
agent configurations. Our results further reveal that models respond heterogeneously to multi-
agent decomposition and iterative learning, indicating that the benefits of workflow
evolution depend on the capabilities of the underlying execution model. Beyond these
benchmarks, Mimosa modular architecture and tool-agnostic design make it readily extensible, and its fully logged execution traces and archived workflows support auditability by preserving every analytical step for inspection and potential replication. Combined with domain-expert guidance, the framework has the potential to automate a broad range of computationally accessible scientific tasks across disciplines. Released as a fully open-source platform, Mimosa aims to provide an open foundation for community-driven ASR.
Read the full abstract ▾
All authors Martin Legrand, Tao Jiang, Matthieu Feraud, Benjamin Navet, Yousouf Taghzouti, Fabien Gandon, Elise Dumont, Louis-Félix Nothias
Machine Learning · Cross-listed from eess.SY
A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic
Chengyang Gu, Yuxin Pan, Hui Xiong et al. (+1 more)
Model-based reinforcement learning (MBRL) improves sample efficiency by leveraging learned dynamics models for policy optimization. However, the effectiveness of methods such as actor-critic is often limited by compounding model errors, which degrade long-horizon value estimation. Existing approaches, such as Model-Based Value Expansion (MVE), partially mitigate this issue through multi-step rollouts, but remain sensitive to rollout horizon selection and residual model bias. Motivated by the Pontryagin Maximum Principle (PMP), we propose
Hamiltonian Actor-Critic (HAC), a model-based approach that eliminates explicit value function learning by directly optimizing a
Hamiltonian defined over the learned dynamics and reward for deterministic systems. By avoiding value approximation, HAC reduces sensitivity to model errors while admitting convergence guarantees. Extensive experiments on continuous control
benchmarks, in both online and offline RL settings, demonstrate that HAC outperforms model-free and MVE-based baselines in control performance, convergence speed, and robustness to distributional shift, including out-of-distribution (OOD) scenarios. In offline settings with limited data, HAC matches or exceeds state-of-the-art methods, highlighting its strong sample efficiency.
Read the full abstract ▾
All authors Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen
Machine Learning
Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training
Yongzhong Xu
We develop the spectral edge analysis: phase transitions in neural network training – grokking, capability gains, loss plateaus – are controlled by the spectral gap of the rolling-window Gram
matrix of parameter updates. In the extreme aspect ratio regime (parameters $P \sim 10^8$, window $W \sim 10$), the classical BBP detection threshold is vacuous; the operative structure is the intra-signal gap separating dominant from subdominant modes at position $k^* = \mathrm{argmax}\, \sigma_j/\sigma_{j+1}$. From three assumptions we derive: (i) gap dynamics governed by a Dyson-type ODE with
curvature asymmetry, damping, and gradient driving; (ii) a spectral loss decomposition linking each mode's learning contribution to its Davis–Kahan stability coefficient; (iii) the Gap Maximality Principle, showing that $k^*$ is the unique dynamically privileged position – its collapse is the only one that disrupts learning, and it sustains itself through an $\alpha$-feedback loop requiring no assumption on the optimizer. The adiabatic parameter $\mathcal{A} = \|\Delta G\|_F / (\eta\, g^2)$ controls
circuit stability: $\mathcal{A} \ll 1$ (plateau), $\mathcal{A} \sim 1$ (phase transition), $\mathcal{A} \gg 1$ (forgetting). Tested across six model families (150K–124M parameters): gap dynamics precede every grokking event (24/24 with weight decay, 1/24 without), the gap position is optimizer-dependent (
Muon: $k^*=1$, AdamW: $k^*=2$ on the same model), and 19/20 quantitative predictions are confirmed. The framework is consistent with the edge of stability, Tensor Programs, Dyson Brownian motion, the Lottery Ticket Hypothesis, and neural scaling laws.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Models
Mozhgan Pourkeshavarz, Tianran Liu and Nicholas Rhinehart
Simulation with realistic traffic
agents is essential for validating autonomous driving systems. Existing data-driven simulators learn
agent behavior from higher-level abstractions such as 3D bounding boxes and polylines, inferred by upstream perception pipelines. These lossy abstractions discard sensory context that directly shapes
agent behavior, limiting the distributional realism that simulation aims to reproduce. To address this limitation, we propose AutoWorld, a traffic simulation framework that grounds
agent behavior in raw sensor observations through a self-supervised world model trained on
LiDAR occupancy data. Given world model samples, AutoWorld constructs a coarse-to-fine predictive scene context as input to a multi-
agent motion generation model. Furthermore, we designed a motion-aware latent supervision objective that enriches AutoWorld's latent representation of scene dynamics. To better exploit this latent space during inference, AutoWorld employs a cascaded Determinantal Point Process framework to guide diversity-aware sampling across both the world model and motion model. Experiments on nuPlan and Waymo demonstrate that AutoWorld improves generation realism, with larger gains in partially observed scenarios where trajectory abstractions are most limited, reducing collision rates by 36%. AutoWorld also remains physically plausible under planner-controlled ego behavior and supports controllable, diverse safety-critical scenario generation.
Read the full abstract ▾
Machine Learning
Multi-Agent LLMs for Adaptive Acquisition in Bayesian Optimization
Mohammadsina Almasi, Andrea Carbonati and Hadis Anahideh
The exploration-exploitation trade-off is central to sequential decision-making and black-box optimization, yet how Large
Language Models (LLMs) reason about and manage this trade-off remains poorly understood. Unlike
Bayesian Optimization, where exploration and exploitation are explicitly encoded through acquisition functions, LLM-based optimization relies on implicit,
prompt-based
reasoning over historical evaluations, making search behavior difficult to analyze or control. In this work, we present a metric-level study of LLM-mediated search policy learning, studying how LLMs construct and adapt exploration-exploitation strategies under multiple operational definitions of exploration, including informativeness, diversity, and representativeness. We show that single-
agent LLM approaches, which jointly perform strategy selection and candidate generation within a single
prompt, suffer from cognitive overload, leading to unstable search dynamics and premature convergence. To address this limitation, we propose a multi-
agent framework that decomposes exploration-exploitation control into strategic policy mediation and tactical candidate generation. A strategy
agent assigns interpretable weights to multiple search criteria, while a generation
agent produces candidates conditioned on the resulting search policy defined as weights. This decomposition renders exploration-exploitation decisions explicit, observable, and adjustable. Empirical results across various continuous optimization
benchmarks indicate that separating strategic control from candidate generation substantially improves the effectiveness of LLM-mediated search.
Read the full abstract ▾
Machine Learning · Cross-listed from math.FA
Minimum Norm Interpolation via the Local Theory of Banach Spaces: The Role of $2$-Uniform Convexity
Gil Kur and Pierre Bizeul
The minimum-norm interpolator (MNI) framework has recently attracted considerable
attention as a tool for understanding generalization in overparameterized models, such as neural networks. In this work, we study the MNI under a $2$-uniform convexity assumption, which is weaker than requiring the norm to be induced by an inner product; in this setting, the MNI typically does not admit a closed-form solution. At a high level, we show that this condition yields an upper bound on the MNI bias in both linear and nonlinear models. We further show that this bound is sharp for overparameterized linear regression when the norm's unit ball is in isotropic or John's position and the
covariates are i.i.d.\ sub-Gaussian, for example, when each
covariate vector has i.i.d. Rademacher entries. Finally, under the same assumption on the
covariates, we prove sharp generalization bounds for the $\ell_p$-MNI when $p \in \bigl(1 + C/\log d, 2\bigr]$. To the best of our knowledge, this is the first work to establish sharp bounds for non-Gaussian
covariates in linear models when the norm is not induced by an inner product. This work is deeply inspired by classical work on $K$-convexity and recent work on the geometry of $2$-uniformly convex and isotropic convex bodies.
Read the full abstract ▾
Machine Learning
Differentiable Initialization-Accelerated CPU-GPU Hybrid Combinatorial Scheduling
Mingju Liu, Jiaqi Yin, Alvaro Velasquez et al. (+1 more)
This paper presents a hybrid CPU-GPU framework for solving combinatorial scheduling problems formulated as Integer Linear Programming (ILP). While scheduling underpins many optimization tasks in computing systems, solving these problems optimally at scale remains a long-standing challenge due to their NP-hard nature. We introduce a novel approach that combines differentiable optimization with classical ILP solving. Specifically, we utilize differentiable presolving to rapidly generate high-quality partial solutions, which serve as warm-starts for commercial ILP solvers (CPLEX, Gurobi) and rising open-source solver HiGHS. This method enables significantly improved early
pruning compared to state-of-the-art standalone solvers. Empirical results across industry-scale
benchmarks demonstrate up to a $10\times$ performance gain over baselines, narrowing the optimality gap to $<0.1\%$. This work represents the first demonstration of utilizing differentiable optimization to initialize exact ILP solvers for combinatorial scheduling, opening new opportunities to integrate machine learning infrastructure with classical exact optimization methods across broader domains.
Read the full abstract ▾
All authors Mingju Liu, Jiaqi Yin, Alvaro Velasquez, Cunxi Yu
Machine Learning
ReproMIA: A Comprehensive Analysis of Model Reprogramming for Proactive Membership Inference Attacks
Chihan Huang, Huaijin Wang and Shuai Wang
The pervasive deployment of deep learning models across critical domains has concurrently intensified
privacy concerns due to their inherent propensity for data memorization. While Membership Inference Attacks (MIAs) serve as the gold standard for
auditing these
privacy vulnerabilities, conventional MIA paradigms are increasingly constrained by the prohibitive computational costs of shadow model training and a precipitous performance degradation under low False Positive Rate constraints. To overcome these challenges, we introduce a novel perspective by leveraging the principles of model reprogramming as an active signal amplifier for
privacy leakage. Building upon this insight, we present ReproMIA, a unified and efficient proactive framework for membership inference. We rigorously substantiate, both theoretically and empirically, how our methodology proactively induces and magnifies latent
privacy footprints embedded within the model's representations. We provide specialized instantiations of ReproMIA across diverse architectural paradigms, including LLMs,
Diffusion Models, and Classification Models. Comprehensive experimental evaluations across more than ten
benchmarks and a variety of model architectures demonstrate that ReproMIA consistently and substantially outperforms existing state-of-the-art baselines, achieving a transformative leap in performance specifically within low-FPR regimes, such as an average of 5.25% AUC and 10.68% TPR@1%FPR increase over the runner-up for LLMs, as well as 3.70% and 12.40% respectively for
Diffusion Models.
Read the full abstract ▾
Machine Learning
Foundations of Polar Linear Algebra
Giovanni Guasti
This work revisits operator learning from a spectral perspective by introducing Polar Linear
Algebra, a structured framework based on polar geometry that combines a linear radial component with a periodic angular component. Starting from this formulation, we define the associated operators and analyze their spectral properties. As a proof of feasibility, the framework is evaluated on a canonical
benchmark (MNIST). Despite the simplicity of the task, the results demonstrate that polar and fully spectral operators can be trained reliably, and that imposing self-adjoint-inspired spectral constraints improves stability and convergence. Beyond accuracy, the proposed formulation leads to a reduction in parameter count and computational complexity, while providing a more interpretable representation in terms of decoupled spectral modes. By moving from a spatial to a spectral domain, the problem decomposes into orthogonal eigenmodes that can be treated as independent computational pipelines. This structure naturally exposes an additional dimension of model parallelization, complementing existing parallel strategies without relying on ad-hoc partitioning. Overall, the work offers a different conceptual lens for operator learning, particularly suited to problems where spectral structure and parallel execution are central.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SY
Optimistic Online LQR via Intrinsic Rewards
Marcell Bartos, Bruce D. Lee, Lenart Treven et al. (+3 more)
Optimism in the face of uncertainty is a popular approach to balance exploration and exploitation in reinforcement learning. Here, we consider the online linear quadratic regulator (LQR) problem, i.e., to learn the LQR corresponding to an unknown linear dynamical system by adapting the control policy online based on closed-loop data collected during operation. In this work, we propose Intrinsic Rewards LQR (IR-LQR), an optimistic online LQR
algorithm that applies the idea of intrinsic rewards originating from reinforcement learning and the concept of variance regularization to promote uncertainty-driven exploration. IR-LQR typically retains the structure of a standard LQR synthesis problem by only modifying the cost function, resulting in an intuitively pleasing, simple, computationally cheap, and efficient
algorithm. This is in contrast to existing optimistic online LQR formulations that rely on more complicated iterative search
algorithms or solve computationally demanding optimization problems. We show that IR-LQR achieves the optimal worst-case regret rate of $\sqrt{T}$, and compare it to various state-of-the-art online LQR
algorithms via numerical experiments carried out on an aircraft pitch angle control and an unmanned aerial vehicle example.
Read the full abstract ▾
All authors Marcell Bartos, Bruce D. Lee, Lenart Treven, Andreas Krause, Florian Dörfler, Melanie N. Zeilinger
Machine Learning
A Neural Tension Operator for Curve Subdivision across Constant Curvature Geometries
Hassan Ugail and Newton Howard
Interpolatory subdivision schemes generate smooth curves from piecewise-linear control polygons by repeatedly inserting new vertices. Classical schemes rely on a single global tension parameter and typically require separate formulations in Euclidean, spherical, and
hyperbolic geometries. We introduce a shared learned tension predictor that replaces the global parameter with per-edge insertion angles predicted by a single 140K-parameter network. The network takes local intrinsic features and a trainable geometry
embedding as input, and the predicted angles drive geometry-specific insertion operators across all three spaces without architectural modification. A constrained sigmoid output head enforces a structural safety bound, guaranteeing that every inserted vertex lies within a valid angular range for any finite weight configuration. Three theoretical results accompany the method: a structural guarantee of tangent-safe insertions; a heuristic motivation for per-edge adaptivity; and a conditional convergence certificate for continuously differentiable limit curves, subject to an explicit Lipschitz constraint verified post hoc. On 240 held-out validation curves, the learned predictor occupies a distinct position on the fidelity–smoothness Pareto frontier, achieving markedly lower bending energy and angular roughness than all fixed-tension and
manifold-lift baselines. Riemannian
manifold lifts retain a pointwise-fidelity advantage, which this study quantifies directly. On the out-of-distribution ISS orbital ground-track example, bending energy falls by 41% and angular roughness by 68% with only a modest increase in Hausdorff distance, suggesting that the predictor generalises beyond its synthetic training distribution.
Read the full abstract ▾
Machine Learning
Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection
Ivan Pasichnyk
The critical damping condition of the damped harmonic oscillator model of SGD with momentum (Qian, 1999) yields a momentum schedule with no tuned hyperparameters: mu(t) = 1 - 2*sqrt(alpha(t)). Across five seeds on ResNet-18/CIFAR-10 (200-epoch cosine schedule) it reaches 90% test accuracy 2.34x faster than constant mu=0.9 (range 1.71-2.86x, 5/5 seeds, one-sided paired t-test p=4e-4), at the cost of a real final-accuracy deficit of 0.46 pp (5/5 seeds, p=0.009). A short-schedule control rules out a schedule-length artifact: compressed baselines either pay 0.5-0.9 pp of accuracy or stay slower to 90% at equal accuracy. A hybrid recipe – critical-damping momentum until 90%, then constant mu=0.9 – removes the deficit and keeps the speedup: 95.45 +/- 0.05% final accuracy at 2.4x faster progress to 90% (n=5). The speedup generalizes across architectures (VGG-16 without skip connections: 1.72x, n=3); on CIFAR-100 early gains persist (2-4x to mid-training thresholds) but the accuracy cost grows (-1.7 pp), compressing the accuracy-matched gain to 1.14x. We also report an exhaustive negative result on
surgical layer selection. Version 2 of this paper claimed that gradient attribution on misclassified images selects which layers to retrain; running the identical correction protocol on all 35 combinations of 3-of-7 layer groups ranks the selected triple 11th of 35 (exact p=0.31) – no better than random. What survives is weaker but real: combinations containing the top-ranked layer outperform the rest (+6.2 vs -1.4 mean net error reduction), and the bottom of the gradient-norm ranking reliably predicts the most harmful interventions (down to -20 net errors). Gradient attribution on errors is a harm-avoidance signal, not a selector of repair targets. We release the full 35-combination landscape as a baseline for layer-selection claims.
Read the full abstract ▾
Machine Learning · Cross-listed from math.OC
Symmetrizing Bregman Divergence on the Cone of Positive Definite Matrices: Which Mean to Use and Why
Tushar Sial and Abhishek Halder
This work uncovers variational principles behind symmetrizing the Bregman divergences induced by generic mirror maps over the cone of positive definite
matrices. We show that computing the canonical means for this symmetrization can be posed as minimizing the desired symmetrized divergences over a set of mean functionals defined axiomatically to satisfy certain properties. For the forward symmetrization, we prove that the arithmetic mean over the primal space is canonical for any mirror map over the positive definite cone. For the reverse symmetrization, we show that the canonical mean is the arithmetic mean over the dual space, pulled back to the primal space. Applying this result to three common mirror maps used in practice, we show that the canonical means for reverse symmetrization, in those cases, turn out to be the arithmetic, log-Euclidean and harmonic means. Our results improve understanding of existing symmetrization practices in the literature, and can be seen as a navigational chart to help decide which mean to use when.
Read the full abstract ▾
Machine Learning
Structural Pass Analysis in Football: Learning Pass Archetypes and Tactical Impact from Spatio-Temporal Tracking Data
Oktay Karakuş, Hasan Arkadaş and Can Eyüpoğlu
Passing actions are commonly evaluated through ball progression and attacking outcomes, yet their relationship with the spatial organisation of the opposition provides an additional perspective on passing behaviour. This study introduces an interpretable framework for describing passing structure using synchronised event and tracking data. Three complementary descriptors are proposed: Line Bypass Score (LBS), measuring the difference in goal-side opponent counts between passer and receiver; Space Gain Metric (SGM), comparing their nearest-opponent distances; and Structural Disruption Index (SDI), summarising differences in defensive organisation between pass initiation and a sampled state one second later. Following transformation and standardisation, these descriptors are combined into Tactical Impact Value (TIV), a fixed-weight structural composite. Applied to 41,078 open-play passing events from the 2022 FIFA World Cup, the framework identifies four interpretable profiles: Circulatory, Destabilising, Line-breaking, and Space-expanding. The profiles exhibit distinct structural signatures and different distributions of territorial-crossing and recorded attacking indicators. Team, spatial, player, and passer–receiver analyses further demonstrate how the representation can describe passing behaviour across analytical scales. The framework complements outcome-oriented approaches by connecting individual passing actions with measurable characteristics of opposition positioning and defensive organisation.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
Robust Multi-Agent Reinforcement Learning for Small UAS Separation Assurance under GPS Degradation and Spoofing
Alex Zongo, Filippos Fotiadis, Ufuk Topcu et al. (+1 more)
We address robust separation assurance for small Unmanned Aircraft Systems (sUAS) under GPS degradation and spoofing via Multi-
Agent Reinforcement Learning (MARL). In cooperative surveillance, each aircraft (or
agent) broadcasts its GPS-derived position; when such position broadcasts are corrupted, the entire observed air traffic state becomes unreliable. We cast this state observation corruption as a zero-sum game between the
agents and an adversary: with probability R, the adversary perturbs the observed state to maximally degrade each
agent's safety performance. We derive a closed-form expression for this
adversarial perturbation, bypassing the iterative inner optimization of
adversarial training entirely and enabling linear-time evaluation in the state dimension. We show that this expression approximates the exact minimizer of the value function over the modeled uncertainty set with second-order accuracy. We further bound the safety performance gap between clean and corrupted observations, showing that it degrades at most linearly with the corruption probability under Kullback-Leibler regularization. Finally, we integrate the closed-form
adversarial policy into a MARL policy gradient
algorithm to obtain a robust counter-policy for the
agents. In a high-density sUAS simulation, we observe near-zero collision rates under corruption levels up to 35%, outperforming a baseline policy trained without
adversarial perturbations.
Read the full abstract ▾
All authors Alex Zongo, Filippos Fotiadis, Ufuk Topcu, Peng Wei
Machine Learning · Cross-listed from cs.IR
Calibrated Fusion for Heterogeneous Graph-Vector Retrieval in Multi-Hop QA
Andre Bacellar
Graph-augmented
retrieval combines dense similarity with graph-based relevance signals such as Personalized PageRank (PPR), but these scores have different distributions and are not directly comparable. We study this as a score calibration problem for heterogeneous
retrieval fusion in multi-hop question answering. Our method, PhaseGraph, maps vector and graph scores to a common unit-free scale using percentile-rank normalization (PIT) before fusion, enabling stable combination without discarding magnitude information. Across MuSiQue and 2WikiMultiHopQA, calibrated fusion improves held-out last-hop
retrieval on HippoRAG2-style
benchmarks: LastHop@10 increases from 69.1% to 71.0% on MuSiQue (15W/5L, p=0.041, n=514) and LastHop@5 from 51.7% to 53.6% on 2WikiMultiHopQA (11W/2L, p=0.023, n=491), both on independent held-out test splits. Against the official HippoRAG 2 pipeline on 2WikiMultiHopQA, calibrated fusion is ahead at LastHop@10 (+6.3pp, p<10^-3) and behind at LastHop@5 (-8.4pp), a cutoff-dependent cross-over we report in full. A theory-driven ablation shows that percentile-based calibration is directionally more robust than min-max normalization on both tune and test splits (1W/6L, p=0.125), while Boltzmann weighting performs comparably to linear fusion after calibration (0W/3L, p=0.25). These results suggest that score commensuration is a robust design choice, and the exact post-calibration operator appears to matter less on these
benchmarks.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
Haiyue Song and Masao Utiyama
Continual pre-training is widely used to adapt LLMs to target
languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: they must be fixed before training begins, and a suboptimal choice can waste weeks of compute. In this work, we propose OptiMer, which decouples ratio selection from training: we train one CPT model per
dataset, extract each model's distribution vector, which represents the parameter shift induced by that
dataset, and search for optimal composition weights post-hoc via
Bayesian optimization. Experiments on Gemma 3 27B across
languages (Japanese, Chinese) and domains (Math, Code) show that OptiMer consistently outperforms data mixture and model averaging baselines with 15-35 times lower search cost. Key findings reveal that 1) the optimized weights can be interpreted as data mixture ratios, and retraining with these ratios improves data mixture CPT, and 2) the same vector pool can be re-optimized for a given objective without any retraining, producing target-tailored models on demand. Our work establishes that data mixture ratio selection, traditionally a pre-training decision, can be reformulated as a post-hoc optimization over distribution vectors, offering a more flexible paradigm for continual pre-training.
Read the full abstract ▾
Machine Learning
OneComp: One-Line Revolution for Generative AI Model Compression
Yuma Ichikawa, Keiji Kimura, Akihiro Yoshida et al. (+11 more)
Deploying foundation models is increasingly constrained by
memory footprint, latency, and hardware costs. Post-training compression can mitigate these bottlenecks by reducing the precision of model parameters without significantly degrading performance; however, its practical implementation remains challenging as practitioners navigate a fragmented landscape of quantization
algorithms, precision budgets, data-driven calibration strategies, and hardware-dependent execution regimes. We present OneComp, an open-source compression framework that transforms this expert workflow into a reproducible, resource-adaptive pipeline. Given a model identifier and available hardware, OneComp automatically inspects the model, plans mixed-precision assignments, and executes progressive quantization stages, ranging from layer-wise compression to block-wise refinement and global refinement. A key architectural choice is treating the first quantized checkpoint as a deployable pivot, ensuring that each subsequent stage improves the same model and that quality increases as more compute is invested. By converting state-of-the-art compression research into an extensible, open-source, hardware-aware pipeline, OneComp bridges the gap between algorithmic innovation and production-grade model deployment.
Read the full abstract ▾
All authors Yuma Ichikawa, Keiji Kimura, Akihiro Yoshida, Yudai Fujimoto, Hiroki Tokura, Yamato Arai, Yoshiyuki Ishii, Yusei Kawakami, Genki Shikada, Achille Jacquemond, Yoshihiko Fujisawa, Katsuki Fujisawa, Takumi Honda, Akira Sakai
Machine Learning
Geometry-aware similarity metrics for neural representations on Riemannian and statistical manifolds
N Alex Cayco-Gajic and Arthur Pellegrino
Similarity measures such as centred kernel
alignment (CKA) or representational similarity analysis (RSA) are widely used to compare the representational geometries employed by different neural networks to solve the same task. Yet, because existing methods compare the extrinsic geometry of representations in state space, rather than their intrinsic geometry, they may fail to capture subtle yet crucial distinctions between fundamentally different computations. Here, we introduce metric similarity analysis (MSA), a novel method which leverages tools from Riemannian geometry to compare the intrinsic geometry of neural representations under the
manifold hypothesis. We show that MSA can be used to (i) disentangle neural computations of deep networks in rich vs. lazy learning regimes, (ii) compare the dynamics of recurrent neural networks and state-space models, and (iii) investigate the statistical
manifolds of
diffusion models. Overall, we introduce a mathematically grounded and broadly applicable framework to understand the mechanisms behind neural computations by contrasting their intrinsic geometries.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers
Omer Dahary, Benaya Koren, Daniel Garibi et al. (+1 more)
Modern Text-to-Image (T2I)
diffusion models have achieved remarkable semantic
alignment, yet they often suffer from a significant lack of variety, converging on a narrow set of visual solutions for any given
prompt. This typicality bias presents a challenge for creative applications that require a wide range of generative outcomes. We identify a fundamental trade-off in current approaches to diversity: modifying model inputs requires costly optimization to incorporate feedback from the generative path. In contrast, acting on spatially-committed intermediate latents tends to disrupt the forming visual structure, leading to artifacts. In this work, we propose to apply repulsion in the Contextual Space as a novel framework for achieving rich diversity in
Diffusion Transformers. By intervening in the
multimodal attention channels, we apply on-the-fly repulsion during the
transformer's forward pass, injecting the intervention between blocks where text conditioning is enriched with emergent image structure. This allows for redirecting the guidance trajectory after it is structurally informed but before the composition is fixed. Our results demonstrate that repulsion in the Contextual Space produces significantly richer diversity without sacrificing visual fidelity or semantic adherence. Furthermore, our method is uniquely efficient, imposing a small computational overhead while remaining effective even in modern "Turbo" and distilled models where traditional trajectory-based interventions typically fail.
Read the full abstract ▾
All authors Omer Dahary, Benaya Koren, Daniel Garibi, Daniel Cohen-Or
Machine Learning
Immediate Derivatives Suffice for Online Recurrent Adaptation
Aur Shalev Merin
For three decades online recurrent learning has been assumed to require propagating a Jacobian tensor through the network's dynamics at $O(n^4)$ per step. We show it doesn't. Dropping the propagation entirely ($d=0$, $O(n^2)$
memory) matches full RTRL within CI on held-out BCI cross-session drift (TOST equivalent within $\pm 3$ pp at $n=20$, Adam, float64), and across vanilla-RNN synthetic
cells (sine and Lorenz under Adam and SGD) and LSTM/sine under Adam. A decomposition $g_{RTRL} = g_{imm} + g_{past}$ explains why. On BCI, $g_{past}$ concentrates in a single direction (top-1 singular fraction 0.62-0.74 across four optimizers, vs 0.333 for $g_{imm}$), and the four-optimizer full-RTRL-vs-$d=0$ recovery gap tracks each optimizer's per-layer update-magnitude ratio $\|\Delta W_{hh}\|/\|\Delta W_{out}\|$ monotonically. A stationary (no-drift) control collapses both concentrations to ~0.6: the drift-specific signal is the differential, not $g_{past}$'s absolute rank-1 structure. The signature and the behavioral gap both collapse on LSTM, consistent with a mechanism specific to additive linear recurrence. On synthetic sine, $g_{imm}$ is redundant with $g_{past}$, which predicts the synthetic null. Full RTRL's one robust advantage is LARS (+17 to +27 pp), but $d=0$+LARS also fails to adapt independently; the gap is an optimizer$\times$method interaction, not a method-quality claim. We characterize the regime: $d=0$+Adam+float64 is robust; SGD, Adafactor, and float32 have specific fragilities documented in the paper. On the evaluated
cells, the $1000\times$
memory saving at $n=1024$ ($O(n^2)$ vs $O(n^4)$) comes with no measured recovery cost.
Read the full abstract ▾
Machine Learning
Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation
Vitória Barin-Pacela, Shruti Joshi, Isabela Camacho et al. (+2 more)
The linear representation hypothesis states that neural network activations encode high-level concepts as linear mixtures. However, under superposition, this encoding is a projection from a higher-dimensional concept space into a lower-dimensional activation space, and a linear decision boundary in the concept space need not remain linear after projection. In this setting, classical sparse coding methods with per-sample iterative inference leverage compressed sensing guarantees to recover latent factors. Sparse autoencoders (SAEs), on the other hand, amortise sparse inference into a fixed encoder, introducing a systematic gap. We show this amortisation gap persists across training set sizes, latent dimensions, and sparsity levels, causing SAEs to fail under out-of-distribution (OOD) compositional shifts. Through controlled experiments that decompose the failure, we identify dictionary learning as the limiting factor (not the inference procedure): SAE-learned dictionaries point in substantially wrong directions, and replacing the encoder with per-sample FISTA on the same dictionary does not close the gap. An oracle baseline proves the problem is solvable with a good dictionary at all scales tested. Our results, including experiments with real LLM activations (Pythia-70M, Gemma-2-2B) reframe the SAE failure as a dictionary learning challenge, not an inference problem, and point to scalable dictionary learning as the key open problem for sparse inference under superposition.
Read the full abstract ▾
All authors Vitória Barin-Pacela, Shruti Joshi, Isabela Camacho, Simon Lacoste-Julien, David Klindt
Machine Learning
Rethinking Language Model Scaling under Transferable Hypersphere Optimization
Liliang Ren, Yang Liu, Yelong Shen et al. (+1 more)
Scaling laws for large
language models depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizers, and they do not structurally prevent training instability at scale. Recent hypersphere optimization methods constrain weight
matrices to a fixed-norm hypersphere, offering a promising alternative for more stable scaling. We introduce HyperP (Hypersphere Parameterization), the first framework for transferring optimal learning rates across model width, depth, training
tokens, and Mixture-of-Experts (MoE) granularity under the Frobenius-sphere constraint with the
Muon optimizer. We prove that weight decay is a first-order no-op on the Frobenius sphere, show that Depth-$\mu$P remains necessary, and find that the optimal learning rate follows the same data-scaling power law with the "magic exponent" 0.32 previously observed for AdamW. A single base learning rate tuned at the smallest scale transfers across all compute budgets under HyperP, yielding $1.58\times$ compute efficiency over a strong
Muon baseline at $6\times10^{21}$ FLOPs. Moreover, HyperP delivers transferable stability: all monitored instability indicators, including $Z$-values, output RMS, and activation outliers, remain bounded and non-increasing under training FLOPs scaling. We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities for improved granularity scaling, and show that hypersphere optimization enables substantially larger auxiliary load-balancing weights, yielding both strong performance and good expert balance. We release our training codebase at https://github.com/microsoft/ArchScale.
Read the full abstract ▾
All authors Liliang Ren, Yang Liu, Yelong Shen, Weizhu Chen
Machine Learning
Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks
Meitong Liu, Christopher Jung, Rui Li et al. (+2 more)
In transfer learning, the learner leverages auxiliary data to improve generalization on a main task. However, the precise theoretical understanding of when and how auxiliary data help remains incomplete. We provide new insights on this issue in two canonical linear settings: ordinary least squares regression and under-parameterized linear neural networks. For linear regression, we derive exact closed-form expressions for the expected generalization error with bias-variance decomposition, yielding necessary and sufficient conditions for auxiliary tasks to improve generalization on the main task. We also derive globally optimal task weights as outputs of solvable optimization programs, with consistency guarantees for empirical estimates. For linear neural networks with shared representations of width $q \leq K$, where $K$ is the number of auxiliary tasks, we derive a non-asymptotic expectation bound on the generalization error, yielding the first non-vacuous sufficient condition for beneficial auxiliary learning in this setting, as well as principled directions for task weight curation. We achieve this by proving a new column-wise low-rank perturbation bound for random
matrices, which improves upon existing bounds by preserving fine-grained column structures. Our results are verified on synthetic data simulated with controlled parameters.
Read the full abstract ▾
All authors Meitong Liu, Christopher Jung, Rui Li, Xue Feng, Han Zhao