Machine Learning · Cross-listed from stat.ML
Amortised and provably-robust simulation-based inference
Ayush Bharti, Charita Dellaporta, Yuga Hikida et al. (+1 more)
Complex simulator-based models are now routinely used to perform inference across the sciences and engineering, but existing inference methods are often unable to account for outliers and other extreme values in data which occur due to faulty measurement instruments or human error. In this paper, we introduce a novel approach to simulation-based inference grounded in generalised
Bayesian inference and a neural approximation of a weighted score-matching loss. This leads to a method that is both amortised and provably robust to outliers, a combination not achieved by existing approaches. Furthermore, through a carefully chosen conditional density model, we demonstrate that inference can be further simplified and performed without the need for Markov chain Monte Carlo sampling, thereby offering significant computational advantages, with complexity that is only a small fraction of that of current state-of-the-art approaches.
Read the full abstract ▾
All authors Ayush Bharti, Charita Dellaporta, Yuga Hikida, François-Xavier Briol
Machine Learning
Predictive Associative Memory: Retrieval Beyond Similarity Through Temporal Co-occurrence
Jason Dury
Current approaches to
memory in neural systems rely on similarity-based
retrieval: given a query, find the most representationally similar stored state. This assumption – that useful
memories are similar
memories – fails to capture a fundamental property of biological
memory: association through temporal co-occurrence. We propose Predictive Associative
Memory (PAM), an architecture in which a JEPA-style predictor, trained on temporal co-occurrence within a continuous experience stream, learns to navigate the associative structure of an
embedding space. We introduce an Inward JEPA that operates over stored experience (predicting associatively reachable past states) as the complement to the standard Outward JEPA that operates over incoming sensory data (predicting future states). We evaluate PAM as an associative recall system – testing faithfulness of recall for experienced associations – rather than as a
retrieval system evaluated on generalisation to unseen associations. On a synthetic
benchmark, the predictor's top
retrieval is a true temporal associate 97% of the time (Association Precision@1 = 0.970); it achieves cross-boundary Recall@20 = 0.421 where cosine similarity scores zero; and it separates experienced-together from never-experienced-together states with a discrimination AUC of 0.916 (cosine: 0.789). Even restricted to cross-room pairs where
embedding similarity is uninformative, the predictor achieves AUC = 0.849 (cosine: 0.503, chance). A temporal shuffle control confirms the signal is genuine temporal co-occurrence structure, not
embedding geometry: shuffling collapses cross-boundary recall by 90%, replicated across training seeds. All results are stable across seeds (SD < 0.006) and query selections (SD $\leq$ 0.012).
Read the full abstract ▾
Machine Learning
Efficient Analysis of the Distilled Neural Tangent Kernel
Jamie Mahowald, Brian Bell, Alex Ho et al. (+1 more)
Neural tangent kernel (NTK) methods are computationally limited by the need to evaluate large Jacobians across many data points. Existing approaches reduce this cost primarily through projecting and sketching the Jacobian. We show that NTK computation can also be reduced by compressing the data dimension itself using NTK-tuned
dataset distillation. We demonstrate that the neural tangent space spanned by the input data can be induced by
dataset distillation, yielding a 20-100$\times$ reduction in required Jacobian calculations. We further show that per-class NTK
matrices have low effective rank that is preserved by this reduction. Building on these insights, we propose the distilled neural tangent kernel (DNTK), which combines NTK-tuned
dataset distillation with state-of-the-art projection methods to reduce up NTK computational complexity by up to five orders of magnitude while preserving kernel structure and predictive performance.
Read the full abstract ▾
All authors Jamie Mahowald, Brian Bell, Alex Ho, Michael Geyer
Machine Learning · Cross-listed from physics.ao-ph
Hierarchical Testing of a Hybrid Machine Learning-Physics Global Atmosphere Model
Ziming Chen, L. Ruby Leung, Wenyu Zhou et al. (+9 more)
Machine learning (ML)-based models have demonstrated high skill and computational efficiency, often outperforming conventional physics-based models in weather and subseasonal predictions. While prior studies have assessed their fidelity in capturing synoptic-scale atmospheric dynamics, their performance across timescales and under out-of-distribution forcing, such as +3K or +4K uniform-warming forcings, and the sources of biases remain elusive, to establish the model reliability for Earth science. Here, we design three sets of experiments targeting synoptic-scale phenomena, interannual variability, and out-of-distribution uniform-warming forcings. We evaluate the Neural General Circulation Model (NeuralGCM), a hybrid model integrating a dynamical core with ML-based component, against observations and physics-based Earth system models (ESMs). At the synoptic scale, NeuralGCM captures the
evolution and propagation of extratropical cyclones with performance comparable to ESMs. At the interannual scale, when forced by El Niño-Southern Oscillation sea surface temperature (SST) anomalies, NeuralGCM successfully reproduces associated teleconnection patterns but exhibits deficiencies in capturing nonlinear response. Under out-of-distribution uniform-warming forcings, NeuralGCM simulates similar responses in global-average temperature and precipitation and reproduces large-scale tropospheric circulation features similar to those in ESMs. Notable weaknesses include overestimating the tracks and spatial extent of extratropical cyclones, biases in the teleconnected wave train triggered by tropical SST anomalies, and differences in upper-level warming and stratospheric circulation responses to SST warming compared to physics-based ESMs. The causes of these weaknesses were explored.
Read the full abstract ▾
All authors Ziming Chen, L. Ruby Leung, Wenyu Zhou, Jian Lu, Sandro W. Lubis, Ye Liu, Chuan-Chieh Chang, Bryce E. Harrop, Ya Wang, Mingshi Yang, Gan Zhang, Yun Qian
Machine Learning
HiFloat4 Format for Language Model Inference
Yuanyong Luo, Jing Huang, Yu Cheng et al. (+19 more)
This paper introduces HiFloat4 (HiF4), a block floating-point data format tailored for deep learning. Each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value. The metadata specifies a three-level scaling hierarchy, capturing inter- and intra-group dynamic range while improving the utilization of the representational space. In addition, the large 64-element group size enables
matrix multiplications to be executed in a highly fixed-point manner, significantly reducing hardware area and power consumption. To evaluate the proposed format, we conducted inference experiments on several
language models, including LLaMA, Qwen, Mistral, DeepSeek-V3.1 and LongCat. Results show that HiF4 achieves higher average accuracy than the state-of-the-art NVFP4 format across multiple models and diverse downstream tasks.
Read the full abstract ▾
All authors Yuanyong Luo, Jing Huang, Yu Cheng, Ziwei Yu, Kaihua Tang, Xinda Ma, Xin Wang, Anping Tong, Guipeng Hu, Yun Xu, Mehran Taghian, Peng Wu, Guanglin Li, Yunke Peng, Tianchi Hu, Minqi Chen, Michael Bi Mi, Hu Liu, Xiping Zhou, Junsong Wang, Qiang Lin, Heng Liao
Machine Learning · Cross-listed from astro-ph.IM
DeepRed: an architecture for redshift estimation
Alessandro Meroni, Nicolò Oreste Pinciroli Vago and Piero Fraternali
Estimating
redshift is a central task in astrophysics, but its measurement is costly and time-consuming. In addition, current image-based methods are often validated on homogeneous
datasets. The development and comparison of networks able generalize across different morphologies, ranging from
galaxies to gravitationally-lensed transients, and observational conditions, remain an open challenge. This work proposes DeepRed, a deep learning pipeline that demonstrates how modern computer
vision architectures, including ResNet, EfficientNet, Swin
Transformer, and MLP-Mixer, can estimate
redshifts from images of
galaxies, gravitational lenses, and gravitationally-lensed
supernovae. We compare these architectures and their ensemble to both neural networks (A1, A3, NetZ, and PhotoZ) and a feature-based method (HOG+SVR) on simulated (DeepGraviLens) and real (KiDS, SDSS)
datasets. Our approach achieves state-of-the-art results on all
datasets. On DeepGraviLens, DeepRed achieves a significant improvement in the Normalized Mean Absolute Deviation compared to the best baseline (PhotoZ): 55% on DES-deep (using EfficientNet), 51% on DES-wide (Ensemble), 52% on DESI-DOT (Ensemble), and 46% on LSST-wide (Ensemble). On real observations from the KiDS survey, the pipeline outperforms the best baseline (NetZ), improving NMAD by 16% on a general test set without high-probability lenses (Ensemble) and 27% on high-probability lenses (Ensemble). For non-lensed
galaxies in the SDSS
dataset, the MLP-Mixer architecture achieves a 5% improvement over the best baselines (A3 and NetZ). SHAP shows that the models correctly focus on the objects of interest with over 95% localization accuracy on high-quality images, validating the reliability of the predictions. These findings suggest that deep learning is a scalable, robust, and interpretable solution for
redshift estimation in large-scale surveys.
Read the full abstract ▾
Machine Learning · Cross-listed from cond-mat.dis-nn
Unlearnable phases of matter
Tarun Advaith Kumar, Yijian Zou, Amir-Reza Negari et al. (+2 more)
We identify fundamental limitations in machine learning by demonstrating that non-trivial mixed-state phases of matter are computationally hard to learn. Focusing on unsupervised learning of distributions, we show that autoregressive neural networks fail to learn global properties of distributions characterized by locally indistinguishable (LI) states. We demonstrate that conditional mutual information (CMI) is a useful diagnostic for LI: we show that for classical distributions, long-range CMI of a state implies a spatially LI partner. By introducing a restricted statistical query model, we prove that nontrivial phases with long-range CMI, such as strong-to-weak spontaneous
symmetry breaking phases, are hard to learn. We validate our claims by using recurrent, convolutional, and
Transformer neural networks to learn the syndrome and physical distributions of toric/surface code under bit flip noise. Our findings suggest hardness of learning as a diagnostic tool for detecting mixed-state phases and transitions and error-correction thresholds, and they suggest CMI and more generally “non-local Gibbsness” as metrics for how hard a distribution is to learn.
Read the full abstract ▾
All authors Tarun Advaith Kumar, Yijian Zou, Amir-Reza Negari, Roger G. Melko, Timothy H. Hsieh
Machine Learning
How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?
Nikhil Garg, Jon Kleinberg and Kenny Peng
We introduce a mathematical framework for the linear representation hypothesis (LRH), which asserts that intermediate layers of
language models store features linearly. We separate the hypothesis into two claims: linear representation (features are linearly embedded in
neuron activations) and linear accessibility (features can be linearly decoded). We then ask: How many
neurons $d$ suffice to both linearly represent and linearly access $m$ features? Classical results in compressed sensing imply that for $k$-sparse inputs, $d = O(k\log (m/k))$ suffices if we allow non-linear decoding
algorithms (Candes and Tao, 2006; Candes et al., 2006; Donoho, 2006). However, the additional requirement of linear decoding takes the problem out of the classical compressed sensing, into linear compressed sensing. Our main theoretical result establishes nearly-matching upper and lower bounds for linear compressed sensing. We prove that $d = \Omega_\epsilon(\frac{k^2}{\log k}\log (m/k))$ is required while $d = O_\epsilon(k^2\log m)$ suffices. The lower bound establishes a quantitative gap between classical and linear compressed setting, illustrating how linear accessibility is a meaningfully stronger hypothesis than linear representation alone. The upper bound confirms that
neurons can store an exponential number of features under the LRH, giving theoretical evidence for the "superposition hypothesis" (Elhage et al., 2022). The upper bound proof uses standard random constructions of
matrices with approximately orthogonal columns. The lower bound proof uses rank bounds for near-identity
matrices (Alon, 2003) together with Turán's
theorem (bounding the number of edges in clique-free graphs). We also show how our results do and do not constrain the geometry of feature representations and extend our results to allow decoders with an activation function and bias.
Read the full abstract ▾
Machine Learning
Evaluating Memory Structure in LLM Agents
Alina Shutova, Alexandra Olenina, Ivan Vinogradov et al. (+1 more)
Modern LLM-based
agents and chat assistants rely on long-term
memory frameworks to store reusable knowledge, recall user preferences, and augment
reasoning. As researchers create more complex
memory architectures, it becomes increasingly difficult to analyze their capabilities and guide future
memory designs. Most long-term
memory benchmarks focus on simple fact retention, multi-hop recall, and time-based changes. While undoubtedly important, these capabilities can often be achieved with simple
retrieval-augmented LLMs and do not test complex
memory hierarchies. To bridge this gap, we propose StructMemEval - a
benchmark that tests the
agent's ability to organize its long-term
memory, not just factual recall. We gather a suite of tasks that humans solve by organizing their knowledge in a specific structure: transaction ledgers, to-do lists, trees and others. Our initial experiments show that simple
retrieval-augmented LLMs struggle with these tasks, whereas
memory agents can reliably solve them if prompted how to organize their
memory. However, we also find that modern LLMs do not always recognize the
memory structure when not prompted to do so. This highlights an important direction for future improvements in both LLM training and
memory frameworks.
Read the full abstract ▾
All authors Alina Shutova, Alexandra Olenina, Ivan Vinogradov, Anton Sinitsin
Machine Learning · Cross-listed from cs.CV
Active Zero: Self-Evolving Vision-Language Models through Active Environment Exploration
Jinghan He, Junfeng Fang, Feng Xiong et al. (+5 more)
Self-play has enabled large
language models to autonomously improve through self-generated challenges. However, existing self-play methods for
vision-
language models rely on passive interaction with static image collections, resulting in strong dependence on initial
datasets and inefficient learning. Without the ability to actively seek visual data tailored to their evolving capabilities,
agents waste computational effort on samples that are either trivial or beyond their current skill level. To address these limitations, we propose Active-Zero, a framework that shifts from passive interaction to active exploration of visual environments. Active-Zero employs three co-evolving
agents: a Searcher that retrieves images from open-world repositories based on the model's capability frontier, a Questioner that synthesizes calibrated
reasoning tasks, and a Solver refined through accuracy rewards. This closed loop enables self-scaffolding auto-curricula where the model autonomously constructs its learning trajectory. On Qwen2.5-VL-7B-Instruct across 12
benchmarks, Active-Zero achieves 53.97 average accuracy on
reasoning tasks (5.7% improvement) and 59.77 on general understanding (3.9% improvement), consistently outperforming existing self-play baselines. These results highlight active exploration as a key ingredient for scalable and adaptive self-evolving
vision-
language systems.
Read the full abstract ▾
All authors Jinghan He, Junfeng Fang, Feng Xiong, Zijun Yao, Fei Shen, Haiyun Guo, Jinqiao Wang, Tat-Seng Chua
Machine Learning · Cross-listed from cs.CV
Toward Reliable Tea Leaf Disease Diagnosis Using Deep Learning Model: Enhancing Robustness With Explainable AI and Adversarial Training
Samanta Ghosh, Jannatul Adan Mahi, Shayan Abrar et al. (+4 more)
Tea is a valuable asset for the economy of Bangladesh. So, tea cultivation plays an important role to boost the economy. These valuable plants are vulnerable to various kinds of leaf infections which may cause less production and low quality. It is not so easy to detect these diseases manually. It may take time and there could be some errors in the detection.Therefore, the purpose of the study is to develop an automated deep learning model for tea leaf disease classification based on the teaLeafBD
dataset so that anyone can detect the diseases more easily and efficiently. There are 5,278 high-resolution images in this
dataset. The images are classified into seven categories. Six of them represents various diseases and the rest one represents healthy leaves. The proposed pipeline contains data preprocessing, data splitting,
adversarial training, augmentation, model training, evaluation, and comprehension made possible with Explainable AI strategies. DenseNet201 and EfficientNetB3 were employed to perform the classification task. To prepare the model more robustly, we applied
adversarial training so it can operate effectively even with noisy or disturbed inputs. In addition, Grad-CAM visualization was executed to analyze the model's predictions by identifying the most influential regions of each image. Our experimental outcomes revealed that EfficientNetB3 achieved the highest classification accuracy of 93%, while DenseNet201 reached 91%. The outcomes prove that the effectiveness of the proposed approach can accurately detect tea leaf diseases and provide a practical solution for advanced agricultural management.
Read the full abstract ▾
All authors Samanta Ghosh, Jannatul Adan Mahi, Shayan Abrar, Md Parvez Mia, Asaduzzaman Rayhan, Abdul Awal Yasir, Asaduzzaman Hridoy
Machine Learning
AI-Driven Clinical Decision Support System for Enhanced Diabetes Diagnosis and Management
Mujeeb Ur Rehman, Imran Rehan and Sohail Khalid
Identifying type 2 diabetes mellitus can be challenging, particularly for primary care physicians. Clinical decision support systems incorporating artificial intelligence (AI-CDSS) can assist medical professionals in diagnosing type 2 diabetes with high accuracy. This study aimed to assess an AI-CDSS specifically developed for the diagnosis of type 2 diabetes by employing a hybrid approach that integrates expert-driven insights with machine learning techniques. The AI-CDSS was developed (training
dataset: n = 650) and tested (test
dataset: n = 648) using a
dataset of 1298 patients with and without type 2 diabetes. To generate predictions, the
algorithm utilized key features such as body mass index,
plasma fasting glucose, and hemoglobin A1C. Furthermore, a clinical pilot study involving 105 patients was conducted to assess the diagnostic accuracy of the system in comparison to non-endocrinology specialists. The AI-CDSS showed a high degree of accuracy, with 99.8% accuracy in predicting diabetes, 99.3% in predicting prediabetes, 99.2% in identifying at-risk individuals, and 98.8% in predicting no diabetes. The test
dataset revealed a 98.8% agreement between endocrinology specialists and the AI-CDSS. Type 2 diabetes was identified in 45% of 105 individuals in the pilot study. Compared with diabetes specialists, the AI-CDSS scored a 98.5% concordance rate, greatly exceeding that of nonendocrinology specialists, who had an 85% agreement rate. These findings indicate that the AI-CDSS has the potential to be a useful tool for accurately identifying type 2 diabetes, especially in situations in which diabetes specialists are not readily available.
Read the full abstract ▾
Machine Learning
Learning Glioblastoma Tumor Heterogeneity Using Brain Inspired Topological Neural Networks
Ankita Paul and Wenyi Wang
Accurate prognosis for Glioblastoma (GBM) using deep learning (DL) is hindered by extreme spatial and structural heterogeneity. Moreover, inconsistent MRI acquisition protocols across institutions hinder generalizability of models. Conventional
transformer and DL pipelines often fail to capture the multi-scale morphological diversity such as fragmented necrotic cores, infiltrating margins, and disjoint enhancing components leading to scanner-specific artifacts and poor cross-site prognosis. We propose TopoGBM, a learning framework designed to capture heterogeneity-preserved, scanner-robust representations from multi-parametric 3D MRI. Central to our approach is a 3D convolutional autoencoder regularized by a topological regularization that preserves the complex, non-Euclidean invariants of the tumor's
manifold within a compressed latent space. By enforcing these topological priors, TopoGBM explicitly models the high-variance structural signatures characteristic of aggressive GBM. Evaluated across heterogeneous cohorts (UPENN, UCSF, RHUH) and external validation on TCGA, TopoGBM achieves better performance (C-index 0.67 test, 0.58 validation), outperforming baselines that degrade under domain shift. Mechanistic interpretability analysis reveals that reconstruction residuals are highly localized to pathologically heterogeneous zones, with tumor-restricted and healthy tissue error significantly low (Test: 0.03, Validation: 0.09). Furthermore, occlusion-based attribution localizes approximately 50% of the prognostic signal to the tumor and the diverse peritumoral microenvironment advocating clinical reliability of the unsupervised learning method. Our findings demonstrate that incorporating topological priors enables the learning of morphology-faithful
embeddings that capture tumor heterogeneity while maintaining cross-institutional robustness.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Latent Generative Solvers for Generalizable Long-Term Physics Simulation
Zituo Chen and Sili Deng
Reliable physics simulation demands two capabilities that today's neural PDE solvers do not deliver together: generalization across heterogeneous PDE families, and stability under long autoregressive rollouts. Deterministic operators accumulate error geometrically, while existing probabilistic solvers are confined to a single PDE family or short horizons. We close this gap with the Latent Generative Solver (LGS), three coupled components: (i) a Physics VAE (PhyVAE) compressing twelve PDE families into a shared latent
manifold; (ii) a Pyramidal Flow-Forcing
Transformer (PFlowFT) that generates the next latent by flow matching, conditioned on a per-trajectory context updated on the model's own predictions; and (iii) input noising during training, for which we derive a sufficient-condition contraction bound explaining the observed long-horizon stability. Pretrained on a 2.5\,M-trajectory, 16-system corpus at $128^2$, LGS matches the strongest deterministic baseline at one step, wins on 15/16 systems at both 5- and 10-step rollout, cuts 20-step L2RE from $56.1\%$ to $\mathbf{30.2\%}$, and uses $\mathbf{13}$–$\mathbf{77\times}$ less recurrent dynamics-step compute. It also adapts efficiently to a $256^2$ Kolmogorov flow held out from the pretraining corpus, dropping 1-step L2RE from $0.398$ to $0.129$ in five finetune epochs against U-AFNO's $0.653{\to}0.343$. These empirical claims are limited to the evaluated 2D regular-grid systems and discrete horizons.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IT
Generative AI-Driven Phase Control for RIS-Aided Cell-Free Massive MIMO Systems
Kalpesh K. Patel, Malay Chakraborty, Ekant Sharma et al. (+1 more)
This work investigates a generative artificial intelligence (GenAI) model to optimize the reconfigurable intelligent surface (RIS) phase shifts in RIS-aided
cell-free massive multiple-input multiple-output (mMIMO) systems under practical constraints, including imperfect channel state information (CSI) and spatial correlation. We propose two GenAI based approaches, generative conditional
diffusion model (GCDM) and generative conditional
diffusion implicit model (GCDIM), leveraging the
diffusion model conditioned on dynamic CSI to maximize the sum spectral efficiency (SE) of the system. To
benchmark performance, we compare the proposed GenAI based approaches against an expert
algorithm, traditionally known for achieving near-optimal solutions at the cost of computational efficiency. The simulation results demonstrate that GCDM matches the sum SE achieved by the expert
algorithm while significantly reducing the computational overhead. Furthermore, GCDIM achieves a comparable sum SE with an additional $98\%$ reduction in computation time, underscoring its potential for efficient phase optimization in RIS-aided
cell-free mMIMO systems.
Read the full abstract ▾
All authors Kalpesh K. Patel, Malay Chakraborty, Ekant Sharma, Sandeep Kumar Singh
Machine Learning
Patch the Distribution Mismatch: RL Rewriting Agent for Stable Off-Policy SFT
Jiacheng Wang, Zhijie Liu, Ping Jian et al. (+4 more)
Large
language models are commonly adapted to downstream tasks through supervised fine-tuning (SFT), but substantial distribution mismatch between downstream supervision and a model's generation distribution can intensify catastrophic forgetting. Data rewriting offers a data-centric way to narrow this mismatch before SFT. Existing methods, however, typically sample rewrites from a
prompt-induced conditional distribution, which need not align with the backbone's natural question-answering generation distribution, and fixed templates can reduce output diversity. We formulate data rewriting as a policy-learning problem and train a lightweight LoRA rewriting policy with reinforcement learning. The policy optimizes question-answering-style distributional
alignment and semantic diversity under a hard task-consistency gate, producing verified supervision for downstream SFT. Across three instruction-tuned backbones, the resulting models attain downstream gains broadly comparable to standard SFT while reducing degradation on non-downstream
benchmarks in every evaluated setting. Additional experiments on logical
reasoning and medical question answering provide preliminary evidence that a rewriting policy can be reused across domains for the same backbone.
Read the full abstract ▾
All authors Jiacheng Wang, Zhijie Liu, Ping Jian, Zirong Chen, Ke Ren Liao, Zhen Yang, Zhongbin Guo
Machine Learning
Structurally Separated Uncertainty in Supervised Latent Variable Models
Tanmoy Mukherjee, Marius Kloft, Pierre Marquis et al. (+1 more)
Predictive uncertainty is commonly decomposed into epistemic and aleatoric components, but standard decompositions often produce strongly correlated estimates because both quantities are derived from the same predictive distribution. We study an alternative design principle, structural separation, which assigns epistemic and aleatoric uncertainty to disjoint parameter paths trained with distinct supervision targets: reducible prediction error for epistemic uncertainty and persistent label ambiguity for aleatoric uncertainty. We instantiate this principle in two supervised latent-variable models, a Credal Concept Bottleneck Model and a credal Self-Explaining Neural Network, and prove a gradient-isolation result showing that the two uncertainty heads are not coupled through shared training gradients under the proposed parameterization. Across five ambiguity-aware
benchmarks, structural separation substantially reduces epistemic-aleatoric correlation while preserving predictive performance. Further analyses show that aleatoric estimates track annotator- or corpus-derived ambiguity, while epistemic estimates are more sensitive to prediction error and data availability. These results suggest that supervised latent-variable architectures provide a practical route toward uncertainty estimates that are not merely decorrelated, but operationally distinguishable.
Read the full abstract ▾
All authors Tanmoy Mukherjee, Marius Kloft, Pierre Marquis, Zied Bouraoui
Machine Learning
The Magic Correlations: Understanding Knowledge Transfer from Pretraining to Supervised Fine-Tuning
Simin Fan, Dimitris Paparas, Natasha Noy et al. (+3 more)
Understanding how
language model capabilities transfer from pretraining to supervised fine-tuning (SFT) is fundamental to efficient model development and data curation. In this work, we investigate four core questions: RQ1. To what extent do accuracy and confidence rankings established during pretraining persist after SFT? RQ2. Which
benchmarks serve as robust cross-stage predictors and which are unreliable? RQ3. How do transfer dynamics shift with model scale? RQ4. How well does model confidence align with accuracy, as a measure of calibration quality? Does this
alignment pattern transfer across training stages? We address these questions through a suite of correlation protocols applied to accuracy and confidence metrics across diverse data mixtures and model scales. Our experiments reveal that transfer reliability varies dramatically across capability categories,
benchmarks, and scales – with accuracy and confidence exhibiting distinct, sometimes opposing, scaling dynamics. These findings shed light on the complex interplay between pretraining decisions and downstream outcomes, providing actionable guidance for
benchmark selection, data curation, and efficient model development.
Read the full abstract ▾
All authors Simin Fan, Dimitris Paparas, Natasha Noy, Binbin Xiong, Noveen Sachdeva, Berivan Isik
Machine Learning
Protein Language Model Embeddings Improve Generalization of Implicit Transfer Operators
Panagiotis Antoniadis, Beatrice Pavesi, Simon Olsson et al. (+1 more)
Molecular dynamics (MD) is a central computational tool in physics, chemistry, and biology, enabling quantitative prediction of experimental observables as expectations over high-dimensional molecular distributions such as Boltzmann distributions and transition densities. However, conventional MD is fundamentally limited by the high computational cost required to generate independent samples. Generative molecular dynamics (GenMD) has recently emerged as an alternative, learning surrogates of molecular distributions either from data or through interaction with energy models. While these methods enable efficient sampling, their transferability across molecular systems is often limited. In this work, we show that incorporating auxiliary sources of information can improve the data efficiency and generalization of transferable implicit transfer operators (TITO) for molecular dynamics. We find that coarse-grained TITO models are substantially more data-efficient than Boltzmann Emulators, and that incorporating
protein language model (pLM)
embeddings further improves out-of-distribution generalization. Our approach, PLaTITO, achieves state-of-the-art performance on
equilibrium sampling
benchmarks for out-of-distribution
protein systems, including fast-folding
proteins. We further study the impact of additional conditioning signals such as structural
embeddings, temperature, and large-
language-model-derived
embeddings on model performance.
Read the full abstract ▾
All authors Panagiotis Antoniadis, Beatrice Pavesi, Simon Olsson, Ole Winther
Machine Learning
Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learning
Lintao Wang, Zhuqiang Lu, Yilin Zhu et al. (+6 more)
While large
language models (LLMs) have achieved strong performance through fine-tuning within individual scientific domains, their learning dynamics in multi-disciplinary contexts remains poorly understood, despite the promise of improved generalization and broader applicability through cross-domain knowledge synergy. In this work, we present the first systematic study of multi-disciplinary LLM fine-tuning, constructing a five-discipline corpus and analyzing learning patterns of full fine-tuning, LoRA, LoRA-MoE, and LoRA compositions. Particularly, our study shows that multi-disciplinary learning is substantially more variable than single-discipline training and distills four consistent empirical laws: (1) Balance-then-Diversity: low-resource disciplines degrade performance unless mitigated via diversity-aware upsampling; (2) Merge-then-Align: restoring instruction-following ability is critical for cross-discipline synergy; (3) Optimize-then-Scale: parameter scaling offers limited gains without prior design optimization; and (4) Share-then-Specialize: asymmetric LoRA-MoE yields robust gains with minimal trainable parameters via shared low-rank projection. Together, these laws form a practical recipe for principled multi-discipline fine-tuning and provide actionable guidance for developing generalizable scientific LLMs.
Read the full abstract ▾
All authors Lintao Wang, Zhuqiang Lu, Yilin Zhu, Kun Hu, Zhenfei Yin, Shixiang Tang, Zhiyong Wang, Wanli Ouyang, Xinzhu Ma
Machine Learning
Towards Compressive and Scalable Recurrent Memory
Yunchong Song, Jushi Kai, Liming Lu et al. (+2 more)
Transformers face a quadratic bottleneck in
attention when scaling to long contexts. Recent approaches introduce recurrent
memory to extend context beyond the current window, yet these often face a fundamental trade-off between theoretical principles and practical scalability. To address this, we introduce Elastic
Memory, a novel
memory architecture grounded in the HiPPO framework for online function approximation. Elastic
Memory treats historical sequence as samples from continuous signals, applying optimal online compression to encode them into a fixed-size
memory state. For
retrieval, we propose a flexible
polynomial sampling mechanism that reconstructs a history summary from this compressed state. Elastic
Memory consistently outperformed baselines on long-context (32k+)
datasets across three domains. With equal parameters, it beat Memorizing
Transformer by 16x
memory and outperformed Melodi at all
memory sizes, even when Melodi had 30% more parameters. When scaling model size, Elastic
Memory stayed ahead of all baselines and was significantly faster than Melodi at 4x size. Furthermore, its decoupled design allows for injecting inductive biases at test-time to boost performance.
Read the full abstract ▾
All authors Yunchong Song, Jushi Kai, Liming Lu, Kaixi Qiu, Zhouhan Lin
Machine Learning · Cross-listed from cs.SE
SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents
Danlong Yuan, Wei Wu, Enhan Zhao et al. (+4 more)
Reinforcement learning (RL) has become a key paradigm for training software engineering (SWE)
agents, but existing pipelines typically rely on per-task containers for isolation. At scale, pre-built container images incur substantial storage overhead, slow environment setup, and require container-management privileges. We propose SWE-MiniSandbox, a lightweight, container-free method that enables scalable RL training of SWE
agents without sacrificing isolation. Instead of relying on per-instance containers, SWE-MiniSandbox executes each task in an isolated workspace backed by kernel-level mechanisms, substantially reducing system overhead. It leverages lightweight environment pre-caching techniques to eliminate the need for bulky container images. As a result, our approach lowers disk usage to approximately 5% of that required by container-based pipelines and reduces environment preparation time to about 25% of the container baseline. Empirical results demonstrate that SWE-MiniSandbox achieves evaluation performance comparable to standard container-based pipelines. By removing the dependency on heavy container infrastructure, SWE-MiniSandbox offers a practical and accessible foundation for scaling RL-based SWE
agents, particularly in resource-constrained research environments.
Read the full abstract ▾
All authors Danlong Yuan, Wei Wu, Enhan Zhao, Zhengren Wang, Xueliang Zhao, Huishuai Zhang, Dongyan Zhao
Machine Learning
Sedigheh Eslami, Maksim Gaiduk, Markus Krimmel et al. (+3 more)
In this report, we introduce pplx-embed, a family of
multilingual embedding models that employ multi-stage contrastive learning on a
diffusion-pretrained
language model backbone for web-scale
retrieval. By leveraging bidirectional
attention through
diffusion-based pretraining, our models capture comprehensive bidirectional context within passages, enabling the use of mean pooling and a late chunking strategy to better preserve global context across long documents. We release two model types: pplx-embed-v1 for standard
retrieval, and pplx-embed-context-v1 for contextualized
embeddings that incorporate global document context into passage representations. pplx-embed-v1 achieves competitive performance on the MTEB(
Multilingual, v2), MTEB(Code), MIRACL, BERGEN, and ToolRet
retrieval benchmarks, while pplx-embed-context-v1 sets new records on the ConTEB
benchmark. Beyond public
benchmarks, pplx-embed-v1 demonstrates strong performance on our internal evaluation suite, focusing on real-world, large-scale search scenarios constructed from 1B production web pages. These results validate the models' effectiveness in production environments where
retrieval quality and efficiency are critical at scale.
Read the full abstract ▾
All authors Sedigheh Eslami, Maksim Gaiduk, Markus Krimmel, Louis Milliken, Bo Wang, Denis Bykov
Machine Learning · Cross-listed from cs.RO
YOR: Your Own Mobile Manipulator for Generalizable Robotics
Manan H Anjaria, Mehmet Enes Erciyes, Vedant Ghatnekar et al. (+11 more)
Recent advances in
robot learning have generated significant interest in capable platforms that may eventually approach human-level competence. This interest, combined with the commoditization of actuators, has propelled growth in low-cost robotic platforms. However, the optimal form factor for mobile manipulation, especially on a budget, remains an open question. We introduce YOR, an open-source, low-cost mobile manipulator that integrates an omnidirectional base, a telescopic vertical lift, and two arms with grippers to achieve whole-body mobility and manipulation. Our design emphasizes modularity, ease of assembly using off-the-shelf components, and affordability, with a bill-of-materials cost under 10,000 USD. We demonstrate YOR's capability by completing tasks that require coordinated whole-body control, bimanual manipulation, and autonomous
navigation. Overall, YOR offers competitive functionality for mobile manipulation research at a fraction of the cost of existing platforms. Project website: https://www.yourownrobot.ai/
Read the full abstract ▾
All authors Manan H Anjaria, Mehmet Enes Erciyes, Vedant Ghatnekar, Neha Navarkar, Haritheja Etukuru, Xiaole Jiang, Kanad Patel, Dhawal Kabra, Nicholas Wojno, Radhika Ajay Prayage, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui
Machine Learning · Cross-listed from cs.SD
SCRAPL: Scattering Transform with Random Paths for Machine Learning
Christopher Mitcheltree, Vincent Lostanlen, Emmanouil Benetos et al. (+1 more)
The Euclidean distance between wavelet scattering transform coefficients (known as paths) provides informative gradients for perceptual quality assessment of deep inverse problems in computer
vision,
speech, and audio processing. However, these transforms are computationally expensive when employed as differentiable loss functions for
stochastic gradient descent due to their numerous paths, which significantly limits their use in neural network training. Against this problem, we propose "Scattering transform with Random Paths for machine Learning" (SCRAPL): a
stochastic optimization scheme for efficient evaluation of multivariable scattering transforms. We implement SCRAPL for the joint time-frequency scattering transform (JTFS) which demodulates spectrotemporal patterns at multiple scales and rates, allowing a fine characterization of intermittent auditory textures. We apply SCRAPL to differentiable digital signal processing (DDSP), specifically, unsupervised sound matching of a granular synthesizer and the Roland TR-808 drum machine. We also propose an initialization heuristic based on importance sampling, which adapts SCRAPL to the perceptual content of the
dataset, improving neural network convergence and evaluation performance. We make our code and audio samples available and provide SCRAPL as a Python package.
Read the full abstract ▾
All authors Christopher Mitcheltree, Vincent Lostanlen, Emmanouil Benetos, Mathieu Lagrange
Machine Learning
GENIUS: Generative Fluid Intelligence Evaluation Suite
Ruichuan An, Sihan Yang, Ziyu Guo et al. (+8 more)
Unified
Multimodal Models (UMMs) have shown remarkable progress in visual generation. Yet, existing
benchmarks predominantly assess $\textit{Crystallized Intelligence}$, which relies on recalling accumulated knowledge and learned schemas. This focus overlooks $\textit{Generative Fluid Intelligence (GFI)}$: the capacity to induce patterns, reason through constraints, and adapt to novel scenarios on the fly. To rigorously assess this capability, we introduce $\textbf{GENIUS}$ ($\textbf{GEN}$ Fluid $\textbf{I}$ntelligence Eval$\textbf{U}$ation $\textbf{S}$uite). We formalize $\textit{GFI}$ as a synthesis of three primitives. These include $\textit{Inducing Implicit Patterns}$ (e.g., inferring personalized visual preferences), $\textit{Executing Ad-hoc Constraints}$ (e.g., visualizing abstract metaphors), and $\textit{Adapting to Contextual Knowledge}$ (e.g., simulating counter-intuitive physics). Collectively, these primitives challenge models to solve problems grounded entirely in the immediate context. Our systematic evaluation of 12 representative models reveals significant performance deficits in these tasks. Crucially, our diagnostic analysis disentangles these failure modes. It demonstrates that deficits stem from limited context comprehension rather than insufficient intrinsic generative capability. To bridge this gap, we propose a training-free
attention intervention strategy. Ultimately, $\textbf{GENIUS}$ establishes a rigorous standard for $\textit{GFI}$, guiding the field beyond knowledge utilization toward dynamic, general-purpose
reasoning. Our
dataset and code will be released at: $\href{https://github.com/arctanxarc/GENIUS}{https://github.com/arctanxarc/GENIUS}$.
Read the full abstract ▾
All authors Ruichuan An, Sihan Yang, Ziyu Guo, Wei Dai, Zijun Shen, Haodong Li, Renrui Zhang, Xinyu Wei, Guopeng Li, Wenshan Wu, Wentao Zhang
Machine Learning · Cross-listed from cs.RO
Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows
Shaswat Garg, Matin Moezzi and Brandon Da Silva
Hierarchical goal-conditioned reinforcement learning (H-GCRL) provides a powerful framework for tackling complex, long-horizon tasks by decomposing them into structured subgoals. However, its practical adoption is hindered by poor data efficiency and limited policy expressivity, especially in offline or data-scarce regimes. In this work, Normalizing flow-based hierarchical implicit Q-learning (NF-HIQL), a novel framework that replaces unimodal gaussian policies with expressive normalizing flow policies at both the high- and low-levels of the hierarchy is introduced. This design enables tractable log-likelihood computation, efficient sampling, and the ability to model rich
multimodal behaviors. New theoretical guarantees are derived, including explicit KL-divergence bounds for Real-valued non-volume preserving (RealNVP) policies and PAC-style sample efficiency results, showing that NF-HIQL preserves stability while improving generalization. Empirically, NF-HIQL is evaluted across diverse long-horizon tasks in locomotion, ball-dribbling, and multi-step manipulation from OGBench. NF-HIQL consistently outperforms prior goal-conditioned and hierarchical baselines, demonstrating superior robustness under limited data and highlighting the potential of flow-based architectures for scalable, data-efficient hierarchical reinforcement learning.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.HC
LCIP: Loss-Controlled Inverse Projection of High-Dimensional Image Data
Yu Wang, Frederik L. Dennig, Michael Behrisch et al. (+1 more)
Projections (or dimensionality reduction) methods $P$ aim to map high-dimensional data to typically 2D scatterplots for visual exploration. Inverse projection methods $P^{-1}$ aim to map this 2D space to the data space to support tasks such as data augmentation, classifier analysis, and data imputation. Current $P^{-1}$ methods suffer from a fundamental limitation – they can only generate a fixed surface-like structure in data space, which poorly covers the richness of this space. We address this by a new method that can `sweep' the data space under user control. Our method works generically for any $P$ technique and
dataset, is controlled by two intuitive user-set parameters, and is simple to implement. We demonstrate it by an extensive application involving image manipulation for style transfer.
Read the full abstract ▾
All authors Yu Wang, Frederik L. Dennig, Michael Behrisch, Alexandru Telea
Machine Learning
TabICLv2: A better, faster, scalable, and open tabular foundation model
Jingang Qu, David Holzmüller, Gaël Varoquaux et al. (+1 more)
Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive
benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in
attention improving generalization to larger
datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the
Muon optimizer. On the TabArena and TALENT
benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale
datasets under 50 GB GPU
memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.
Read the full abstract ▾
All authors Jingang Qu, David Holzmüller, Gaël Varoquaux, Marine Le Morvan
Machine Learning
Weight Decay Improves Language Model Plasticity
Tessa Han, Sebastian Bordt, Hanlin Zhang et al. (+1 more)
Large
language models are typically trained in two broad phases: pretraining to produce a base model, followed by further training to improve downstream performance. However, hyperparameter optimization and scaling laws are studied primarily from the perspective of the base model's validation loss, overlooking a crucial model property: downstream adaptability. In this work, we study pretraining from the perspective of model plasticity, that is, the ability of the base model to successfully adapt to downstream tasks upon additional training. We focus on the role of weight decay, a key regularization parameter during pretraining, and show through systematic experiments that larger weight decay increases the plasticity of the pretrained model, resulting in greater performance gains downstream after fine-tuning. This effect can lead to counterintuitive trade-offs where base models that perform worse after pretraining can perform better after further training. Further investigation of weight decay's mechanistic effects on model behavior reveals that it encourages linearly separable representations, regularizes
attention matrices, and reduces overfitting on the training data. Together, these findings highlight the importance of pretrained model plasticity, the limits of using cross-
entropy loss as the sole metric for hyperparameter optimization, and the multifaceted role that a single optimization hyperparameter plays in shaping model behavior.
Read the full abstract ▾
All authors Tessa Han, Sebastian Bordt, Hanlin Zhang, Sham Kakade