Machine Learning · Cross-listed from cs.CY
AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms
LearnLM Team Google, Eedi: Albert Wang, Aliya Rysbek et al. (+31 more)
One-to-one tutoring is widely considered the gold standard for personalized education, yet it remains prohibitively expensive to scale. To evaluate whether generative AI might help expand access to this resource, we conducted an exploratory randomized controlled trial (RCT) with $N = 165$ students across five UK secondary schools. We integrated LearnLM – a generative AI model fine-tuned for pedagogy – into chat-based tutoring sessions on the Eedi mathematics platform. In the RCT, expert tutors directly supervised LearnLM, with the remit to revise each message it drafted until they would be satisfied sending it themselves. LearnLM proved to be a reliable source of pedagogical instruction, with supervising tutors approving 76.4% of its drafted messages making zero or minimal edits (i.e., changing only one or two characters). This translated into effective tutoring support: students guided by LearnLM performed at least as well as students chatting with human tutors on each learning outcome we measured. In fact, students who received support from LearnLM were 5.5 percentage points more likely to solve novel problems on subsequent topics (with a success rate of 66.2%) than those who received tutoring from human tutors alone (rate of 60.7%). In interviews, tutors highlighted LearnLM's strength at drafting Socratic questions that encouraged deeper reflection from students, with multiple tutors even reporting that they learned new pedagogical practices from the model. Overall, our results suggest that pedagogically fine-tuned AI tutoring systems may play a promising role in delivering effective, individualized learning support at scale.
Read the full abstract ▾
All authors LearnLM Team Google, Eedi: Albert Wang, Aliya Rysbek, Andrea Huber, Anjali Nambiar, Anna Kenolty, Ben Caulfield, Beth Lilley-Draper, Bibi Groot, Brian Veprek, Chelsea Burdett, Claire Willis, Craig Barton, Digory Smith, George Mu, Harriet Walters, Irina Jurenka, Iris Hulls, James Stalley-Moores, Jonathan Caton, Julia Wilkowski, Kaiz Alarakyia, Kevin R. McKee, Liam McCafferty, Lucy Dalton, Markus Kunesch, Pauline Malubay, Rachel Kidson, Rich Wells, Sam Wheeler, Sara Wiltberger, Shakir Mohamed, Simon Woodhead, Vasco Brazão
Machine Learning
BOAD: Discovering Hierarchical Software Engineering Agents via Bandit Optimization
Iris Xu, Guangtao Zeng, Zexue He et al. (+5 more)
Large
language models (LLMs) have shown strong
reasoning and coding capabilities, yet they struggle to generalize to real-world software engineering (SWE) problems that are long-horizon and out of distribution. Existing systems often rely on a single
agent to handle the entire workflow-interpreting issues, navigating large codebases, and implementing fixes-within one
reasoning chain. Such monolithic designs force the model to retain irrelevant context, leading to spurious correlations and poor generalization. Motivated by how human engineers decompose complex problems, we propose structuring SWE
agents as orchestrators coordinating specialized sub-
agents for sub-tasks such as localization, editing, and validation. The challenge lies in discovering effective hierarchies automatically: as the number of sub-
agents grows, the search space becomes combinatorial, and it is difficult to attribute credit to individual sub-
agents within a team. We address these challenges by formulating hierarchy discovery as a multi-armed bandit (MAB) problem, where each arm represents a candidate sub-
agent and the reward measures its helpfulness when collaborating with others. This framework, termed Bandit Optimization for
Agent Design (BOAD), enables efficient exploration of sub-
agent designs under limited evaluation budgets. On SWE-bench-Verified, BOAD outperforms single-
agent and manually designed multi-
agent systems. On SWE-bench-Live, featuring more recent and out-of-distribution issues, our 36B system ranks second on the leaderboard at the time of evaluation, surpassing larger models such as GPT-4 and Claude. These results demonstrate that automatically discovered hierarchical multi-
agent systems significantly improve generalization on challenging long-horizon SWE tasks. Code is available at https://github.com/iamxjy/BOAD-SWE-
Agent.
Read the full abstract ▾
All authors Iris Xu, Guangtao Zeng, Zexue He, Charles Jin, Aldo Pareja, Dan Gutfreund, Chuang Gan, Zhang-Wei Hong
Machine Learning · Cross-listed from cs.CV
Memorization in 3D Shape Generation: An Empirical Study
Shu Pu, Boya Zeng, Kaichen Zhou et al. (+2 more)
Generative models are increasingly used in 3D
vision to synthesize novel shapes, yet it remains unclear whether their generation relies on memorizing training shapes. Understanding their memorization could help prevent training data leakage and improve the diversity of generated results. In this paper, we design an evaluation framework to quantify memorization in 3D generative models and study the influence of different data and modeling designs on memorization. We first apply our framework to quantify memorization in existing methods. Next, through controlled experiments with a latent vector-set (Vecset)
diffusion model, we find that, on the data side, memorization depends on data modality, and increases with data diversity and finer-grained conditioning; on the modeling side, it peaks at a moderate guidance scale and can be mitigated by longer Vecsets and simple rotation augmentation. Together, our framework and analysis provide an empirical understanding of memorization in 3D generative models and suggest simple yet effective strategies to reduce it without degrading generation quality. Our code is available at https://github.com/zlab-princeton/3d_mem.
Read the full abstract ▾
All authors Shu Pu, Boya Zeng, Kaichen Zhou, Mengyu Wang, Zhuang Liu
Machine Learning · Cross-listed from cs.AI
Regret-Based Federated Causal Discovery with Unknown Interventions
Federico Baldo and Charles K. Assaad
Most causal discovery methods recover a completed partially directed acyclic graph representing a Markov equivalence class from observational data. Recent work has extended these methods to
federated settings to address data decentralization and
privacy constraints, but often under idealized assumptions that all clients share the same causal model. Such assumptions are unrealistic in practice, as client-specific policies or protocols, for example, across hospitals, naturally induce heterogeneous and unknown interventions. In this work, we address
federated causal discovery under unknown client-level interventions. We propose I-PERI, a novel
federated algorithm that first recovers the CPDAG of the union of client graphs and then orients additional edges by exploiting structural differences induced by interventions across clients. This yields a tighter equivalence class, which we call the $\mathbf{\Phi}$-Markov Equivalence Class, represented by the $\mathbf{\Phi}$-CPDAG. We provide theoretical guarantees on the convergence of I-PERI, as well as on its
privacy-preserving properties, and present empirical evaluations on synthetic data demonstrating the effectiveness of the proposed
algorithm.
Read the full abstract ▾
Machine Learning
Le Cam Distortion: A Decision-Theoretic Framework for Robust Transfer Learning
Deniz Akdemir
Distribution shift is the defining challenge of real-world machine learning. The dominant paradigm–Unsupervised Domain Adaptation (UDA)–enforces feature invariance, aligning source and target representations via symmetric divergence minimization [Ganin et al., 2016]. We demonstrate that this approach is fundamentally flawed: when domains are unequally informative (e.g., high-quality vs degraded sensors), strict invariance necessitates information destruction, causing "negative transfer" that can be catastrophic in safety-critical applications [Wang et al., 2019]. We propose a decision-theoretic framework grounded in Le Cam's theory of statistical experiments [Le Cam, 1986], using constructive approximations to replace symmetric invariance with directional simulability. We introduce Le Cam Distortion, quantified by the Deficiency Distance $\delta(E_1, E_2)$, as a rigorous upper bound for transfer risk conditional on simulability. Our framework enables transfer without source degradation by learning a kernel that simulates the target from the source. Across five experiments (genomics,
vision, reinforcement learning), Le Cam Distortion achieves: (1) near-perfect frequency estimation in HLA genomics (correlation $r=0.999$, matching classical methods), (2) zero source utility loss in CIFAR-10 image classification (81.2% accuracy preserved vs 34.7% drop for CycleGAN), and (3) safe policy transfer in RL control where invariance-based methods suffer catastrophic collapse. Le Cam Distortion provides the first principled framework for risk-controlled transfer learning in domains where negative transfer is unacceptable: medical imaging, autonomous systems, and precision medicine.
Read the full abstract ▾
Machine Learning
Distribution-Free Process Monitoring with Conformal Prediction
Christopher Burger
Traditional Statistical Process Control (SPC) is essential for quality management but is limited by its reliance on often violated statistical assumptions, leading to unreliable monitoring in modern, complex manufacturing environments. This paper introduces a hybrid framework that enhances SPC by integrating the distribution free, model agnostic guarantees of Conformal Prediction. We propose two novel applications: Conformal-Enhanced Control Charts, which visualize process uncertainty and enable proactive signals like 'uncertainty spikes', and Conformal-Enhanced Process Monitoring, which reframes multivariate control as a formal anomaly detection problem using an intuitive p-value chart. Our framework provides a more robust and statistically rigorous approach to quality control while maintaining the interpretability and ease of use of classic methods.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
The Nonstationarity-Complexity Tradeoff in Return Prediction
Agostino Capponi, Chengpiao Huang, J. Antonio Sidaoui et al. (+2 more)
Does more data improve return prediction? In non-stationary financial
markets, longer training windows improve prediction of complex models but incorporate outdated economic regimes, whereas simpler models require less data and are less vulnerable to changes in economic conditions. We formally characterize this nonstationarity-complexity tradeoff, showing that model complexity and training window length must be jointly optimized. We propose an adaptive selection procedure with formal performance guarantees. Over three decades of U.S. equity
markets, our method improves out-of-sample $R^2$ on industry portfolios by 14% relative to fixed-window and regime-switching
benchmarks, with large gains during recessions.
Read the full abstract ▾
All authors Agostino Capponi, Chengpiao Huang, J. Antonio Sidaoui, Kaizheng Wang, Jiacheng Zou
Machine Learning · Cross-listed from math.DS
From geometry to dynamics: Learning overdamped Langevin dynamics from sparse observations with geometric constraints
Dimitra Maoutsa
How can we learn the laws underlying the dynamics of
stochastic systems when their trajectories are sampled sparsely in time? Existing methods either require temporally resolved high-frequency observations, or rely on
geometric arguments that apply only to conservative systems, limiting the range of dynamics they can recover. Here, we present a new framework that reconciles these two perspectives by reformulating inference as a
stochastic control problem. Our method uses geometry-driven path augmentation, guided by the geometry in the system's invariant density to reconstruct likely trajectories and infer the underlying dynamics without assuming specific parametric models. Applied to overdamped Langevin systems, our approach accurately recovers
stochastic dynamics even from extremely undersampled data, outperforming existing methods in synthetic
benchmarks. This work demonstrates the effectiveness of incorporating
geometric inductive biases into
stochastic system identification methods.
Read the full abstract ▾
Machine Learning
Zhehao Huang, Baijiong Lin, Jingyuan Zhang et al. (+5 more)
Multi-model routing has evolved from an engineering technique into essential infrastructure, yet existing work lacks a systematic, reproducible
benchmark for evaluating
vision-
language models (VLMs). We present VL-RouterBench to assess the overall capability of VLM routing systems systematically. The
benchmark is grounded in raw inference and scoring logs from VLMs and constructs quality and cost
matrices over sample-model pairs. In scale, VL-RouterBench covers 14
datasets across 3 task groups, totaling 30,540 samples, and includes 15 open-source models and 2 API models, yielding 519,180 sample-model pairs and a total input-output
token volume of 34,494,977. The evaluation protocol jointly measures average accuracy, average cost, and throughput, and builds a ranking score from the harmonic mean of normalized cost and accuracy to enable comparison across router configurations and cost budgets. On this
benchmark, we evaluate 10 routing methods and baselines and observe a significant routability gain, while the best current routers still show a clear gap to the ideal Oracle, indicating considerable room for improvement in router architecture through finer visual cues and modeling of textual structure. We will open-source the complete data construction and evaluation toolchain to promote comparability, reproducibility, and practical deployment in
multimodal routing research.
Read the full abstract ▾
All authors Zhehao Huang, Baijiong Lin, Jingyuan Zhang, Jingying Wang, Yuhang Liu, Ning Lu, Tao Li, Xiaolin Huang
Machine Learning
EEG-based Graph-guided Domain Adaptation for Robust Cross-Session Emotion Recognition
Maryam Mirzaei, Farzaneh Shayegh and Hamed Narimani
Accurate recognition of human emotional states is critical for effective human-machine interaction. Electroencephalography (
EEG) offers a reliable source for
emotion recognition due to its high temporal resolution and its direct reflection of neural activity. Nevertheless, variations across recording sessions present a major challenge for model generalization. To address this issue, we propose EGDA, a framework that reduces cross-session discrepancies by jointly aligning the global (marginal) and class-specific (conditional) distributions, while preserving the intrinsic structure of
EEG data through graph regularization. Experimental results on the SEED-IV
dataset demonstrate that EGDA achieves robust cross-session performance, obtaining accuracies of 81.22%, 80.15%, and 83.27% across three transfer tasks, and surpassing several baseline methods. Furthermore, the analysis highlights the Gamma frequency band as the most discriminative and identifies the central-parietal and prefrontal
brain regions as critical for reliable
emotion recognition.
Read the full abstract ▾
Machine Learning
Trustworthy Machine Learning under Distribution Shifts
Zhuo Huang
Machine Learning (ML) has been a foundational topic in artificial intelligence (AI), providing both theoretical groundwork and practical tools for its exciting advancements. From ResNet for visual recognition to
Transformer for
vision-
language alignment, the AI models have achieved superior capability to humans. Furthermore, the scaling law has enabled AI to initially develop general intelligence, as demonstrated by Large
Language Models (LLMs). To this stage, AI has had an enormous influence on society and yet still keeps shaping the future for humanity. However, distribution shift remains a persistent “Achilles' heel”, fundamentally limiting the reliability and general usefulness of ML systems. Moreover, generalization under distribution shift would also cause trust issues for AIs. Motivated by these challenges, my research focuses on Trustworthy Machine Learning under Distribution Shifts, with the goal of expanding AI's robustness, versatility, as well as its responsibility and reliability. We carefully study the three common distribution shifts into: (1) Perturbation Shift, (2) Domain Shift, and (3) Modality Shift. For all scenarios, we also rigorously investigate trustworthiness via three aspects: (1) Robustness, (2) Explainability, and (3) Adaptability. Based on these dimensions, we propose effective solutions and fundamental insights, meanwhile aiming to enhance the critical ML problems, such as efficiency, adaptability, and safety.
Read the full abstract ▾
Machine Learning · Cross-listed from q-fin.TR
Alpha-R1: Alpha Screening with LLM Reasoning via Reinforcement Learning
Zuoyou Jiang, Li Zhao, Rui Sun et al. (+6 more)
Signal decay and regime shifts pose recurring challenges for data-driven investment strategies in non-stationary
markets, where conventional time-series and machine learning approaches often struggle to generalize beyond historical correlations. While large
language models (LLMs) offer strong capabilities for processing unstructured information, their potential to support quantitative factor screening through explicit economic
reasoning remains underexplored. Existing factor-based methods typically reduce alphas to numerical time series, overlooking the semantic rationale that determines when a factor is economically relevant. We present Alpha-R1, an RL-aligned LLM framework for context-aware alpha screening. Its core mechanism, semantic gating, evaluates each candidate factor's semantic profile against a dynamically constructed
market state description, selecting a sparse subset of factors whose economic rationale aligns with current
market conditions. The selection model is trained via group relative policy optimization (GRPO), using realized portfolio returns as the primary reward signal. Under a 12-month out-of-sample evaluation, Alpha-R1 achieves annualized returns of 47.87% on S&P 500 and 40.57% on CSI 300 with Sharpe ratios of 1.62 and 2.23. These results, obtained under a bounded candidate-pool evaluation protocol, provide evidence for second-stage semantic factor reranking in non-stationary
markets. The full implementation and resources are available at https://github.com/FinStep-AI/Alpha-R1.
Read the full abstract ▾
All authors Zuoyou Jiang, Li Zhao, Rui Sun, Ruohan Sun, Zhongjian Li, Jing Li, Daxin Jiang, Zuo Bai, Cheng Hua
Machine Learning
Joint Link Adaptation and Device Scheduling Approach for URLLC Industrial IoT Network: A DRL-based Method with Bayesian Optimization
Wei Gao, Paul Zheng, Peng Wu et al. (+2 more)
In this article, we consider an industrial internet of things (IIoT) network supporting multi-device dynamic ultra-reliable low-latency communication (URLLC) while the channel state information (CSI) is imperfect. A joint link adaptation (LA) and device scheduling (including the order) design is provided, aiming at maximizing the total transmission rate under strict block error rate (BLER) constraints. In particular, a
Bayesian optimization (BO) driven Twin Delayed Deep Deterministic Policy Gradient (TD3) method is proposed, which determines the device served order sequence and the corresponding modulation and coding scheme (MCS) adaptively based on the imperfect CSI. Note that the imperfection of CSI, error sample imbalance in URLLC networks, as well as the parameter sensitivity nature of the TD3
algorithm likely diminish the
algorithm's convergence speed and reliability. To address such an issue, we proposed a BO based training mechanism for the convergence speed improvement, which provides a more reliable learning direction and sample selection method to track the imbalance sample problem. Via extensive simulations, we show that the proposed
algorithm achieves faster convergence and higher sum-rate performance compared to existing solutions.
Read the full abstract ▾
All authors Wei Gao, Paul Zheng, Peng Wu, Yulin Hu, Anke Schmeink
Machine Learning
ML Compass: Navigating Capability, Cost, and Compliance Trade-offs in AI Model Deployment
Vassilis Digalakis Jr, Ramayya Krishnan, Gonzalo Martin Fernandez et al. (+1 more)
We study how organizations should select among competing AI models when user utility, deployment costs, and compliance requirements jointly matter. Widely used capability leaderboards do not translate directly into deployment decisions, creating a capability – deployment gap; to bridge it, we take a systems-level view in which model choice is tied to application outcomes, operating constraints, and a capability-cost frontier. We develop ML Compass, a framework that treats model selection as constrained optimization over this frontier. On the theory side, we characterize optimal model configurations under a parametric frontier and show a three-regime structure in optimal internal measures: some dimensions are pinned at compliance minima, some saturate at maximum levels, and the remainder take interior values governed by frontier
curvature. We derive comparative statics that quantify how budget changes, regulatory tightening, and technological progress propagate across capability dimensions and costs. On the implementation side, we propose a pipeline that (i) extracts low-dimensional internal measures from heterogeneous model descriptors, (ii) estimates an empirical frontier from capability and cost data, (iii) learns a user- or task-specific utility function from interaction outcome data, and (iv) uses these components to target capability-cost profiles and recommend models. We validate ML Compass with two case studies: a general-purpose conversational setting using the PRISM
Alignment dataset and a healthcare setting using a custom
dataset we build using HealthBench. In both environments, our framework produces recommendations – and deployment-aware leaderboards based on predicted deployment value under constraints – that can differ materially from capability-only rankings, and clarifies how trade-offs between capability, cost, and safety shape optimal model choice.
Read the full abstract ▾
All authors Vassilis Digalakis Jr, Ramayya Krishnan, Gonzalo Martin Fernandez, Agni Orfanoudaki
Machine Learning
FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence
Guoan Wan, Tianyu Chen, Fangzheng Feng et al. (+2 more)
Parameter-efficient fine-tuning (PEFT) methods have emerged as a practical solution for adapting large foundation models to downstream tasks, reducing computational and
memory costs by updating only a small subset of parameters. Among them, approaches like LoRA aim to strike a balance between efficiency and expressiveness, but often suffer from slow convergence and limited adaptation capacity due to their inherent low-rank constraints. This trade-off hampers the ability of PEFT methods to capture complex patterns needed for diverse tasks. To address these challenges, we propose FRoD, a novel fine-tuning method that combines hierarchical joint decomposition with rotational degrees of freedom. By extracting a globally shared basis across layers and injecting sparse, learnable perturbations into scaling factors for flexible full-rank updates, FRoD enhances expressiveness and efficiency, leading to faster and more robust convergence. On 20
benchmarks spanning
vision,
reasoning, and
language understanding, FRoD matches full model fine-tuning in accuracy, while using only 1.72% of trainable parameters under identical training budgets.
Read the full abstract ▾
All authors Guoan Wan, Tianyu Chen, Fangzheng Feng, Haoyi Zhou, Runhua Xu
Machine Learning
Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
Zhuo Li, Pengyu Cheng, Zhechao Yu et al. (+7 more)
Reward models (RMs) are essential in reinforcement learning from human feedback (RLHF) to align large
language models (LLMs) with human values. However, RM training data is commonly recognized as low-quality, containing inductive biases that can easily lead to overfitting and reward hacking. For example, more detailed and comprehensive responses are usually human-preferred but with more words, leading response length to become one of the inevitable inductive biases. A limited number of prior RM debiasing approaches either target a single specific type of bias or model the problem with only simple linear correlations, e.g., Pearson coefficients. To mitigate more complex and diverse inductive biases in reward modeling, we introduce a novel information-theoretic debiasing method called Debiasing via Information optimization for RM (DIR). Inspired by the information bottleneck (IB), we maximize the mutual information (MI) between RM scores and human preference pairs, while minimizing the MI between RM outputs and biased attributes of preference inputs. With theoretical justification from information theory, DIR can handle more sophisticated types of biases with non-linear correlations, broadly extending the real-world application scenarios for RM debiasing methods. In experiments, we verify the effectiveness of DIR with three types of inductive biases: response length, sycophancy, and format. We discover that DIR not only effectively mitigates target inductive biases but also enhances RLHF performance across diverse
benchmarks, yielding better generalization abilities. The code and training recipes are available at https://github.com/Qwen-Applications/DIR.
Read the full abstract ▾
All authors Zhuo Li, Pengyu Cheng, Zhechao Yu, Feifei Tong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang
Machine Learning · Cross-listed from cs.AI
Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
Kongcheng Zhang, Qi Yao, Shunyu Liu et al. (+7 more)
Reinforcement Learning (RL) has shown promise for aligning Large
Language Models (LLMs) to follow instructions with various constraints. Despite the encouraging results, RL improvement inevitably relies on sampling successful, high-quality responses; however, the initial model often struggles to generate responses that satisfy all constraints due to its limited capabilities, yielding sparse or indistinguishable rewards that impede learning. In this work, we propose Hindsight instruction Replay (HiR), a novel sample-efficient RL framework for complex instruction following tasks, which employs a select-then-rewrite strategy to replay failed attempts as successes based on the constraints that have been satisfied in hindsight. We perform RL on these replayed samples as well as the original ones, theoretically framing the objective as dual-preference learning at both the instruction- and response-level to enable efficient optimization using only a binary reward signal. Extensive experiments demonstrate that the proposed HiR yields promising results across different instruction following tasks, while requiring less computational budget. Our code and
dataset is available at https://github.com/sastpg/HIR.
Read the full abstract ▾
All authors Kongcheng Zhang, Qi Yao, Shunyu Liu, Wenjian Zhang, Min Cen, Yang Zhou, Wenkai Fang, Yiru Zhao, Baisheng Lai, Mingli Song
Machine Learning
Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion
Vladimer Khasia
Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a
Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r
matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and
memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
Ang Lv, Jin Ma, Yiyuan Ma et al. (+1 more)
Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router's decisions align well with the experts' capabilities, which ultimately limits model performance. To address this, we propose expert-router coupling (ERC) loss, a lightweight auxiliary loss that tightly couples the router's decisions with expert capabilities. Our approach treats each expert's router
embedding as a proxy
token for the
tokens assigned to that expert, and feeds perturbed router
embeddings through the experts to obtain intermediate activations. The ERC loss enforces two constraints on these activations: (1) Each expert must exhibit higher activation for its own proxy
token than for the proxy
tokens of any other expert. (2) Each proxy
token must elicit stronger activation from its corresponding expert than from any other expert. These constraints jointly ensure that each router
embedding faithfully represents its corresponding expert's capability, while each expert specializes in processing the
tokens actually routed to it. The ERC loss is computationally efficient, operating only on $n^2$ activations, where $n$ is the number of experts. This represents a fixed cost independent of batch size, unlike prior coupling methods that scale with the number of
tokens (often millions per batch). Through pre-training MoE-LLMs ranging from 3B to 15B parameters and extensive analysis on trillions of
tokens, we demonstrate the effectiveness of the ERC loss. Moreover, the ERC loss offers flexible control and quantitative tracking of expert specialization levels during training, providing valuable insights into MoEs.
Read the full abstract ▾
All authors Ang Lv, Jin Ma, Yiyuan Ma, Siyuan Qiao
Machine Learning · Cross-listed from cs.MA
Assessing behaviour coverage in a multi-agent system simulation for autonomous vehicle testing
Manuel Franco-Vivo
As autonomous vehicle technology advances, ensuring the safety and reliability of these systems becomes paramount. Consequently, comprehensive testing methodologies are essential to evaluate the performance of autonomous vehicles in diverse and complex real-world scenarios. This study focuses on the behaviour coverage analysis of a multi-
agent system simulation designed for autonomous vehicle testing, and provides a systematic approach to measure and assess behaviour coverage within the simulation environment. By defining a set of driving scenarios, and
agent interactions, we evaluate the extent to which the simulation encompasses a broad range of behaviours relevant to autonomous driving. Our findings highlight the importance of behaviour coverage in validating the effectiveness and robustness of autonomous vehicle systems. Through the analysis of behaviour coverage metrics and coverage-based testing, we identify key areas for improvement and optimization in the simulation framework. Thus, a Model Predictive Control (MPC) pedestrian
agent is proposed, where its objective function is formulated to encourage interesting tests while promoting a more realistic behaviour than other previously studied pedestrian
agents. This research contributes to advancing the field of autonomous vehicle testing by providing insights into the comprehensive evaluation of system behaviour in simulated environments. The results offer valuable implications for enhancing the safety, reliability, and performance of autonomous vehicles through rigorous testing methodologies.
Read the full abstract ▾
Machine Learning · Cross-listed from physics.app-ph
Adaptive Fusion Graph Network for 3D Strain Field Prediction in Solid Rocket Motor Grains
Jiada Huang, Hao Ma, Zhibin Shen et al. (+2 more)
Local high strain in solid rocket motor grains is a primary cause of structural failure. However, traditional numerical simulations are computationally expensive, and existing surrogate models cannot explicitly establish
geometric models and accurately capture high-strain regions. Therefore, this paper proposes an adaptive graph network, GrainGNet, which employs an adaptive pooling dynamic node selection mechanism to effectively preserve the key mechanical features of structurally critical regions, while concurrently utilising feature fusion to transmit deep features and enhance the model's representational capacity. In the joint prediction task involving four sequential conditions–curing and cooling, storage, overloading, and ignition–GrainGNet reduces the mean squared error by 62.8% compared to the baseline graph U-Net model, with only a 5.2% increase in parameter count and an approximately sevenfold improvement in training efficiency. Furthermore, in the high-strain regions of debonding seams, the prediction error is further reduced by 33% compared to the second-best method, offering a computationally efficient and high-fidelity approach to evaluate motor structural safety.
Read the full abstract ▾
All authors Jiada Huang, Hao Ma, Zhibin Shen, Yizhou Qiao, Haiyang Li
Machine Learning
Stochastic Siamese MAE Pretraining for Longitudinal Medical Images
Taha Emre, Arunava Chakravarty, Thomas Pinetz et al. (+9 more)
Temporally aware image representations are crucial for capturing disease progression in 3D volumes of longitudinal medical
datasets. However, recent state-of-the-art self-supervised learning approaches like Masked Autoencoding (MAE), despite their strong representation learning capabilities, lack temporal awareness. In this paper, we propose STAMP (
Stochastic Temporal Autoencoder with Masked Pretraining), a Siamese MAE framework that encodes temporal information through a
stochastic process by conditioning on the time difference between the 2 input volumes. Unlike deterministic Siamese approaches, which compare scans from different time points but fail to account for the inherent uncertainty in disease
evolution, STAMP learns temporal dynamics stochastically by reframing the MAE reconstruction loss as a conditional variational inference objective. We evaluated STAMP on two OCT and one MRI
datasets with multiple visits per patient. STAMP pretrained ViT models outperformed both existing temporal MAE methods and foundation models on different late stage Age-Related Macular Degeneration and Alzheimer's Disease progression prediction which require models to learn the underlying non-deterministic temporal dynamics of the diseases.
Read the full abstract ▾
All authors Taha Emre, Arunava Chakravarty, Thomas Pinetz, Dmitrii Lachinov, Martin J. Menten, Hendrik Scholl, Sobha Sivaprasad, Daniel Rueckert, Andrew Lotery, Stefan Sacu, Ursula Schmidt-Erfurth, Hrvoje Bogunović
Machine Learning · Cross-listed from cs.CV
Towards Integrating Uncertainty for Domain-Agnostic Segmentation
Jesse Brouwers, Xiaoyan Xing and Alexander Timans
Foundation models for
segmentation such as the Segment Anything Model (SAM) family exhibit strong zero-shot performance, but remain vulnerable in shifted or limited-knowledge domains. This work investigates whether uncertainty quantification can mitigate such challenges and enhance model generalisability in a domain-agnostic manner. To this end, we (1) curate UncertSAM, a
benchmark comprising eight
datasets designed to stress-test SAM under challenging
segmentation conditions including shadows, transparency, and camouflage; (2) evaluate a suite of lightweight, post-hoc uncertainty estimation methods; and (3) assess a preliminary uncertainty-guided prediction refinement step. Among evaluated approaches, a last-layer Laplace approximation yields uncertainty estimates that correlate well with
segmentation errors, indicating a meaningful signal. While refinement benefits are preliminary, our findings underscore the potential of incorporating uncertainty into
segmentation models to support robust, domain-agnostic performance. Our
benchmark and code are made publicly available.
Read the full abstract ▾
Machine Learning · Cross-listed from math.ST
A general framework for deep learning
William Kengne and Modou Wade
This paper develops a general approach for deep learning for a setting that includes nonparametric regression and classification. We perform a framework from data that fulfills a generalized Bernstein-type inequality, including independent, $\phi$-mixing, strongly mixing and $\mathcal{C}$-mixing observations. Two
estimators are proposed: a non-penalized deep neural network
estimator (NPDNN) and a sparse-penalized deep neural network
estimator (SPDNN). For each of these
estimators, bounds of the expected excess risk on the class of Hölder smooth functions and composition Hölder functions are established. Applications to independent data, as well as to $\phi$-mixing, strongly mixing, $\mathcal{C}$-mixing processes are considered. For each of these examples, the upper bounds of the expected excess risk of the proposed NPDNN and SPDNN predictors are derived. It is shown that both the NPDNN and SPDNN
estimators are minimax optimal (up to a logarithmic factor) in many classical settings.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
AKG kernel Agent: A Multi-Agent Framework for Cross-Platform Kernel Synthesis
Jinye Du, Quan Yuan, Zuyao Zhang et al. (+16 more)
Modern AI models demand high-performance computation kernels. The growing complexity of LLMs,
multimodal architectures, and recommendation systems, combined with techniques like sparsity and quantization, creates significant computational challenges. Moreover, frequent hardware updates and diverse chip architectures further complicate this landscape, requiring tailored kernel implementations for each platform. However, manual optimization cannot keep pace with these demands, creating a critical bottleneck in AI system development. Recent advances in LLM code generation capabilities have opened new possibilities for automating kernel development. In this work, we propose AKG kernel
agent (AI-driven Kernel Generator), a multi-
agent system that automates kernel generation, migration, and performance tuning. AKG kernel
agent is designed to support multiple domain-specific
languages (DSLs), including Triton, TileLang, CPP, and CUDA-C, enabling it to target different hardware backends while maintaining correctness and portability. The system's modular design allows rapid integration of new DSLs and hardware targets. When evaluated on KernelBench using Triton DSL across GPU and NPU backends, AKG kernel
agent achieves an average speedup of 1.46$\times$ over PyTorch Eager baselines implementations, demonstrating its effectiveness in accelerating kernel development for modern AI workloads.
Read the full abstract ▾
All authors Jinye Du, Quan Yuan, Zuyao Zhang, Yanzhi Yi, Jiahui Hu, Wangyi Chen, Yiyang Zhu, Qishui Zheng, Wenxiang Zou, Xiangyu Chang, Zuohe Zheng, Zichun Ye, Chao Liu, Shanni Li, Renwei Zhang, Yiping Deng, Xinwei Hu, Xuefeng Jin, Jie Zhao
Machine Learning
High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation
Yusuf Kalyoncuoglu and Ratmir Miftachov
State-of-the-art models rely on massive widths despite exhibiting low Intrinsic Dimension (ID). We posit that this redundancy serves the non-convex optimization search rather than the final representation. We validate this hypothesis by decoupling the solution geometry via data-independent random projections, demonstrating that ResNet, ViT, and BERT representations can be compressed by up to 16x with negligible performance degradation of around 1%. Notably, these oblivious projections achieve parity with PCA and learned baselines, confirming the solution
manifold is intrinsically robust. These findings establish the foundation for Subspace-Native
Distillation: a paradigm where student models target this intrinsic
manifold directly, bypassing the high-dimensional optimization bottleneck to realize the
vision of "Train Big, Deploy Small"
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Probabilistic Modelling is Sufficient for Causal Inference
Bruno Mlodozeniec, David Krueger and Richard E. Turner
Causal inference is a key research area in machine learning, yet confusion reigns over the tools needed to tackle it. There are prevalent claims in the machine learning literature that you need a bespoke causal framework or notation to answer causal questions. In this paper, we want to make it clear that you can answer any causal inference question within the realm of probabilistic modelling and inference, without causal-specific tools or notation. Through concrete examples, we demonstrate how causal questions can be tackled by writing down the probability of everything. Lastly, we reinterpret causal tools as emerging from standard probabilistic modelling and inference, elucidating their necessity and utility.
Read the full abstract ▾
Machine Learning
Theoretical Foundations of Scaling Law in Familial Models
Huan Song, Qingfei Zhao, Ting Long et al. (+4 more)
Neural scaling laws have become foundational for optimizing large
language model (LLM) training, yet they typically assume a single dense model output. This limitation effectively overlooks "Familial models, a transformative paradigm essential for realizing ubiquitous intelligence across heterogeneous device-edge-cloud hierarchies. Transcending static architectures, familial models integrate early exits with relay-style inference to spawn G deployable sub-models from a single shared backbone. In this work, we theoretically and empirically extend the scaling law to capture this "one-run, many-models" paradigm by introducing Granularity (G) as a fundamental scaling variable alongside model size (N) and training
tokens (D). To rigorously quantify this relationship, we propose a unified functional form L(N, D, G) and parameterize it using large-scale empirical runs. Specifically, we employ a rigorous IsoFLOP experimental design to strictly isolate architectural impact from computational scale. Across fixed budgets, we systematically sweep model sizes (N) and granularities (G) while dynamically adjusting
tokens (D). This approach effectively decouples the marginal cost of granularity from the benefits of scale, ensuring high-fidelity parameterization of our unified scaling law. Our results reveal that the granularity penalty follows a multiplicative power law with an extremely small exponent. Theoretically, this bridges fixed-compute training with dynamic architectures. Practically, it validates the "train once, deploy many" paradigm, demonstrating that deployment flexibility is achievable without compromising the compute-optimality of dense baselines.
Read the full abstract ▾
All authors Huan Song, Qingfei Zhao, Ting Long, Shuyu Tian, Hongjun An, Jiawei Shao, Xuelong Li
Machine Learning
Task-driven Heterophilic Graph Structure Learning
Ayushman Raghuvanshi, Gonzalo Mateos and Sundeep Prabhakar Chepuri
Graph neural networks (GNNs) often struggle to learn discriminative node representations for heterophilic graphs, where connected nodes tend to have dissimilar labels and feature similarity provides weak structural cues. We propose frequency-guided graph structure learning (FgGSL), an end-to-end graph inference framework that jointly learns homophilic and heterophilic graph structures along with a spectral encoder. FgGSL employs a learnable, symmetric, feature-driven masking function to infer said complementary graphs, which are processed using pre-designed low- and high-pass graph filter banks. A label-based structural loss explicitly promotes the recovery of homophilic and heterophilic edges, enabling task-driven graph structure learning. We derive stability bounds for the structural loss and establish robustness guarantees for the filter banks under graph perturbations. Experiments on six heterophilic
benchmarks demonstrate that FgGSL consistently outperforms state-of-the-art GNNs and graph rewiring methods, highlighting the benefits of combining frequency information with supervised
topology inference.
Read the full abstract ▾
Machine Learning
On the Sample Complexity of Learning for Blind Inverse Problems
Nathan Buskulic, Luca Calatroni, Lorenzo Rosasco et al. (+1 more)
Blind inverse problems arise in many experimental settings where both the signal of interest and the forward operator are (partially) unknown. In this context, methods developed for the non-blind case cannot be adapted in a straightforward manner due to identifiability issues and symmetric solutions inherent to the blind setting. Recently, data-driven approaches have been proposed to address such problems, demonstrating strong empirical performance and adaptability. However, these methods often lack interpretability and are not supported by theoretical guarantees, limiting their reliability in domains such as applied imaging where a blind approach often relates to a calibration of the acquisition device. In this work, we shed light on learning in blind inverse problems within the insightful framework of Linear Minimum Mean Square
Estimators (LMMSEs). We provide a theoretical analysis, deriving closed-form expressions for optimal
estimators and extending classical recovery results to the blind setting. In particular, we establish equivalences with tailored Tikhonov-regularized formulations, where the regularization structure depends explicitly on the distributions of the unknown signal, of the noise, and of the random forward operator. We also show how the reconstruction error converges as the noise and the randomness of the operator diminish when we use a source condition assumption. Furthermore, we derive finite-sample error bounds that characterize the performance of the learned
estimators as a function of the noise level, problem conditioning, and number of available samples. These bounds explicitly quantify the impact of operator randomness and show explicitly the dependence of the associated convergence rates to this randomness factors. Finally, we validate our theoretical findings through illustrative exemplar numerical experiments that confirm the predicted convergence behavior.
Read the full abstract ▾
All authors Nathan Buskulic, Luca Calatroni, Lorenzo Rosasco, Silvia Villa