Machine Learning · Cross-listed from stat.ML
Universal priors: solving empirical Bayes via Bayesian inference and pretraining
Nick Cannella, Anzo Teh, Yanjun Han et al. (+1 more)
We theoretically justify the recent empirical finding of [Teh et al., 2025] that a
transformer pretrained on synthetically generated data achieves strong performance on empirical Bayes (EB) problems. We take an indirect approach to this question: rather than analyzing the model architecture or training dynamics, we ask why a pretrained Bayes
estimator, trained under a prespecified training distribution, can adapt to arbitrary test distributions. Focusing on Poisson EB problems, we identify the existence of universal priors such that training under these priors yields a near-optimal regret bound of $\widetilde{O}(\frac{1}{n})$ uniformly over all test distributions. Our analysis leverages the classical phenomenon of posterior contraction in
Bayesian statistics, showing that the pretrained
transformer adapts to unknown test distributions precisely through posterior contraction. This perspective also explains the phenomenon of length generalization, in which the test sequence length exceeds the training length, as the model performs
Bayesian inference using a generalized posterior.
Read the full abstract ▾
All authors Nick Cannella, Anzo Teh, Yanjun Han, Yury Polyanskiy
Machine Learning
PolyNODE: Variable-dimension Neural ODEs on M-polyfolds
Per Åhag, Alexander Friedrich, Fredrik Ohlsson et al. (+1 more)
Neural ordinary differential equations (NODEs) are
geometric deep learning models based on dynamical systems and flows generated by vector fields on
manifolds. Despite numerous successful applications, particularly within the flow matching paradigm, all existing NODE models are fundamentally constrained to fixed-dimensional dynamics by the intrinsic nature of the
manifold's dimension. In this paper, we extend NODEs to M-polyfolds (spaces that can simultaneously accommodate varying dimensions and a notion of differentiability) and introduce PolyNODEs, the first variable-dimensional flow-based model in
geometric deep learning. As an example application, we construct explicit M-polyfolds featuring dimensional bottlenecks and PolyNODE autoencoders based on parametrised vector fields that traverse these bottlenecks. We demonstrate experimentally that our PolyNODE models can be trained to solve reconstruction tasks in these spaces, and that latent representations of the input can be extracted and used to solve downstream classification tasks. The code used in our experiments is publicly available at https://github.com/turbotage/PolyNODE .
Read the full abstract ▾
All authors Per Åhag, Alexander Friedrich, Fredrik Ohlsson, Viktor Vigren Näslund
Machine Learning · Cross-listed from stat.ML
Mixture-of-Experts under Finite-Rate Gating: Communication–Generalization Trade-offs
Ali Khalesi and Mohammad Reza Deylam Salehi
Mixture-of-Experts (MoE) architectures decompose prediction tasks into specialized expert sub-networks selected by a gating mechanism. This letter adopts a communication-theoretic view of MoE gating, modeling the gate as a
stochastic channel operating under a finite information rate. Within an information-theoretic learning framework, {we specialize a mutual-information generalization bound and develop a rate-distortion characterization $D(R_g)$ of finite-rate gating, where $R_g:=I(X; T)$, yielding (under a standard empirical rate-distortion optimality condition) $\mathbb{E}[R(W)] \le D(R_g)+\delta_m+\sqrt{(2/m)\, I(S; W)}$. }The analysis yields capacity-aware limits for communication-constrained MoE systems, and numerical simulations on synthetic multi-expert models empirically confirm the predicted trade-offs between gating rate, expressivity, and generalization.
Read the full abstract ▾
Machine Learning
Triplet Feature Fusion for Equipment Anomaly Prediction : An Open-Source Methodology Using Small Foundation Models
Takato Yasuno
Predicting equipment anomalies before they escalate into failures is a critical challenge in industrial facility management. Existing approaches rely either on hand-crafted threshold rules, which lack generalizability, or on large neural models that are impractical for on-site, air-gapped deployments. We present an industrial methodology that resolves this tension by combining open-source small foundation models into a unified 1,116-dimensional Triplet Feature Fusion pipeline. This pipeline integrates: (1) statistical features (x in $R^{28}$) derived from 90-day sensor histories, (2) time-series
embeddings (y in $R^{64}$) from a LoRA-adapted IBM Granite TinyTimeMixer (TTM, 133K parameters), and (3)
multilingual text
embeddings (z in $R^{1024}$) extracted from Japanese equipment master records via
multilingual-e5-large. The concatenated triplet h = [x; y; z] is processed by a LightGBM classifier (< 3 MB) trained to predict anomalies at 30-, 60-, and 90-day horizons. All components use permissive open-source licenses (Apache 2.0 / MIT). The inference-time pipeline runs entirely on CPU in under 2 ms, enabling edge deployment on co-located hardware without cloud dependency. On a
dataset of 64 HVAC units comprising 67,045 samples, the triplet model achieves Precision = 0.992, F1 = 0.958, and ROC-AUC = 0.998 at the 30-day horizon. Crucially, it reduces the False Positive Rate from 0.6 percent (baseline) to 0.1 percent - an 83 percent reduction attributable to equipment-type conditioning via text
embedding z. Cluster analysis reveals that the
embeddings align time-series signatures with distinct fault archetypes, explaining how compact
multilingual representations improve discrimination without explicit categorical encoding.
Read the full abstract ▾
Machine Learning · Cross-listed from physics.ao-ph
IT-DPC-SRI: A Cloud-Optimized Archive of Italian Radar Precipitation (2010-2025)
Gabriele Franch, Elena Tomasi, Uladzislau Azhel et al. (+11 more)
We present IT-DPC-SRI, the first publicly available long-term archive of Italian weather radar precipitation estimates, spanning 16 years (2010–2025). The
dataset contains Surface Rainfall Intensity (SRI) observations from the Italian Civil Protection Department's national radar mosaic, harmonized into a coherent Analysis-Ready Cloud-Optimized (ARCO) Zarr datacube. The archive comprises over one million timesteps at temporal resolutions from 15 to 5 minutes, covering a $1200\times1400$ kilometer domain at 1 kilometer spatial resolution, compressed from 7TB to 51GB on disk. We address the historical fragmentation of Italian radar data - previously scattered across heterogeneous formats (OPERA BUFR, HDF5, GeoTIFF) with varying spatial domains and projections - by reprocessing the entire record into a unified store. The
dataset is accessible as a static versioned snapshot on Zenodo, via cloud-native access on the ECMWF European Weather Cloud, and as a continuously updated live version on the ArcoDataHub platform. This release fills a significant gap in European radar data availability, as Italy does not participate in the EUMETNET OPERA pan-European radar composite. The
dataset is released under a CC BY-SA 4.0 license.
Read the full abstract ▾
All authors Gabriele Franch, Elena Tomasi, Uladzislau Azhel, Giacomo Tomezzoli, Alessandro Camilletti, Virginia Poli, Renata Pelosini, Gianfranco Vulpiani, Gabriella Scipione, Giuseppe Trotta, Matteo Angelinelli, Leif Denby, Irene Livia Kruse, Marco Cristoforetti
Machine Learning · Cross-listed from eess.IV
StrokeNeXt: A Siamese-encoder Approach for Brain Stroke Classification in Computed Tomography Imagery
Leo Thomas Ramos and Angel D. Sappa
We present StrokeNeXt, a model for stroke classification in 2D Computed
Tomography (CT) images. StrokeNeXt employs a dual-branch design with two ConvNeXt encoders, whose features are fused through a lightweight convolutional decoder based on stacked 1D operations, including a bottleneck projection and transformation layers, and a compact classification head. The model is evaluated on a curated
dataset of 6,774 CT images, addressing both stroke detection and subtype classification between ischemic and hemorrhage cases. StrokeNeXt consistently outperforms convolutional and
Transformer-based baselines, reaching accuracies and F1-scores of up to 0.988. Paired statistical tests confirm that the performance gains are statistically significant, while class-wise sensitivity and specificity demonstrate robust behavior across diagnostic categories. Calibration analysis shows reduced prediction error compared to competing methods, and confusion
matrix results indicate low misclassification rates. In addition, the model exhibits low inference time and fast convergence.
Read the full abstract ▾
Machine Learning · Cross-listed from physics.plasm-ph
TokaMind: A Multi-Modal Transformer Foundation Model for Tokamak Plasma Dynamics
Tobia Boschi, Andrea Loreti, Nicola C. Amorisco et al. (+13 more)
We present TokaMind, to our knowledge the first open-source foundation model for tokamak
plasma dynamics, based on a Multi-Modal
Transformer (MMT) and pretrained on heterogeneous diagnostics from the publicly available MAST
dataset. TokaMind supports multiple data modalities (time-series, 2D profiles, and
videos) with different sampling rates, robust missing-signal handling, and efficient task adaptation via selectively loading and freezing four model components. To represent multi-modal signals, we use a lightweight fixed-basis Discrete Cosine Transform
embedding (DCT3D) and provide a clean interface for alternative
embeddings (e.g., Variational Autoencoders). We evaluate TokaMind on the recently introduced MAST
benchmark TokaMark, which comprises 14 tasks with heterogeneous reconstruction and
forecasting objectives. Our results show that fine-tuned TokaMind outperforms the strongest
benchmark baseline on all but one task. Compared with training the same architecture from scratch under a matched epoch budget, warm-start adaptation is most beneficial on demanding downstream settings, including long-horizon
forecasting and high-dimensional
equilibrium objectives. These findings highlight the value of multi-modal pretraining for tokamak
plasma dynamics and provide a practical, extensible foundation for future fusion modeling tasks. Training code and model weights are publicly available at github.com/UKAEA-IBM-STFC-Fusion-FMs/tokamind and huggingface.co/UKAEA-IBM-STFC, respectively.
Read the full abstract ▾
All authors Tobia Boschi, Andrea Loreti, Nicola C. Amorisco, Rodrigo H. Ordonez-Hurtado, Cécile Rousseau, George K. Holt, Eszter Székely, Alexander Whittle, Samuel Jackson, Adriano Agnello, Stanislas Pamela, Alessandra Pascale, Robert Akers, Juan Bernabe Moreno, Vassil Alexandrov, Mykhaylo Zayats
Machine Learning
Near-Optimal Sample Complexity for Online Constrained MDPs
Chang Liu, Yunfan Li and Lin F. Yang
Safety is a fundamental challenge in reinforcement learning (RL), particularly in real-world applications such as autonomous driving, robotics, and healthcare. To address this, Constrained Markov Decision Processes (CMDPs) are commonly used to enforce safety constraints while optimizing performance. However, existing methods often suffer from significant safety violations or require a high sample complexity to generate near-optimal policies. We address two settings: relaxed feasibility, where small violations are allowed, and strict feasibility, where no violation is allowed. We propose a model-based primal-dual
algorithm that balances regret and bounded constraint violations, drawing on techniques from online RL and constrained optimization. For relaxed feasibility, we prove that our
algorithm returns an $\varepsilon$-optimal policy with $\varepsilon$-bounded violation with arbitrarily high probability, requiring $\tilde{O}\left(\frac{SAH^3}{\varepsilon^2}\right)$ learning episodes, matching the lower bound for unconstrained MDPs. For strict feasibility, we prove that our
algorithm returns an $\varepsilon$-optimal policy with zero violation with arbitrarily high probability, requiring $\tilde{O}\left(\frac{SAH^5}{\varepsilon^2\zeta^2}\right)$ learning episodes, where $\zeta$ is the problem-dependent Slater constant characterizing the size of the feasible region. This result matches the lower bound for learning CMDPs with access to a generative model. Our results demonstrate that learning CMDPs in an online setting is as easy as learning with a generative model and is no more challenging than learning unconstrained MDPs when small violations are allowed.
Read the full abstract ▾
Machine Learning
Symmetry in language statistics shapes the geometry of model representations
Dhruva Karkada, Daniel J. Korchinski, Andres Nava et al. (+2 more)
The internal representations learned by
language models consistently exhibit striking
geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional
manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that
language statistics exhibit translation
symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this
symmetry governs these
geometric structures in high-dimensional word
embedding models, and we analytically derive the
manifold geometry of word representations. These predictions empirically match large text
embedding models and large
language models. Moreover, the representational geometry persists at moderate
embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational
manifolds originate in the statistical
symmetries of natural
language.
Read the full abstract ▾
All authors Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri
Machine Learning
Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization
Shangding Gu
Large
language models (LLMs) are increasingly deployed in
privacy-critical and personalization-oriented scenarios, yet the role of context length in shaping
privacy leakage and personalization effectiveness remains largely unexplored. We introduce a large-scale
benchmark, PAPerBench, to systematically study how increasing context length influences both personalization quality and
privacy protection in LLMs. The
benchmark comprises approximately 29,000 instances with context lengths ranging from 1K to 256K
tokens, yielding a total of 377K evaluation questions. It jointly evaluates personalization performance and
privacy risks across diverse scenarios, enabling controlled analysis of long-context model behavior. Extensive evaluations across state-of-the-art LLMs reveal consistent performance degradation in both personalization and
privacy as context length increases. We further provide a theoretical analysis of
attention dilution under context scaling, explaining this behavior as an inherent limitation of soft
attention in fixed-capacity
Transformers. The empirical and theoretical findings together suggest a general scaling gap in current models – long context, less focus. We release the
benchmark to support reproducible evaluation and future research on scalable
privacy and personalization. Code and data are available at https://github.com/SafeRL-Lab/PAPerBench
Read the full abstract ▾
Machine Learning
Rethinking Diffusion Models with Symmetries through Canonicalization with Applications to Molecular Graph Generation
Cai Zhou, Zijie Chen, Zian Li et al. (+7 more)
Many generative tasks in chemistry and science involve distributions invariant to group
symmetries (e.g., permutation and rotation). A common strategy enforces invariance and equivariance through architectural constraints such as equivariant denoisers and invariant priors. In this paper, we challenge this tradition through the alternative canonicalization perspective: first map each sample to an orbit representative with a canonical pose or order, train an unconstrained (non-equivariant)
diffusion or flow model on the canonical slice, and finally recover the invariant distribution by sampling a random
symmetry transform at generation time. Building on a formal quotient-space perspective, our work provides a comprehensive theory of canonical
diffusion by proving: (i) the correctness, universality and superior expressivity of canonical generative models over invariant targets; (ii) canonicalization accelerates training by removing
diffusion score complexity induced by group mixtures and reducing conditional variance in flow matching. We then show that aligned priors and optimal transport act complementarily with canonicalization and further improves training efficiency. We instantiate the framework for molecular graph generation under $S_n \times SE(3)$
symmetries. By leveraging
geometric spectra-based canonicalization and mild positional encodings, canonical
diffusion significantly outperforms equivariant baselines in 3D molecule generation tasks, with similar or even less computation. Moreover, with a novel architecture Canon, CanonFlow achieves state-of-the-art performance on the challenging GEOM-DRUG
dataset, and the advantage remains large in few-step generation.
Read the full abstract ▾
All authors Cai Zhou, Zijie Chen, Zian Li, Jike Wang, Kaiyi Jiang, Pan Li, Rose Yu, Muhan Zhang, Stephen Bates, Tommi Jaakkola
Machine Learning · Cross-listed from astro-ph.SR
Generalization from Low- to Moderate-Resolution Spectra with Neural Networks for Stellar Parameter Estimation: A Case Study with DESI
Xiaosheng Zhao, Yuan-Sen Ting, Rosemary F.G. Wyse et al. (+5 more)
Cross-survey generalization is a critical challenge in
stellar spectral analysis, particularly in cases such as transferring from low- to moderate-resolution surveys. We investigate this problem using pre-trained models, focusing on simple neural networks such as multilayer perceptrons (MLPs), with a case study transferring from LAMOST low-resolution spectra (LRS) to DESI medium-resolution spectra (MRS). Specifically, we pre-train MLPs on either LRS or their
embeddings and fine-tune them for application to DESI
stellar spectra. We compare MLPs trained directly on spectra with those trained on
embeddings derived from
transformer-based models (self-supervised foundation models pre-trained for multiple downstream tasks). We also evaluate different fine-tuning strategies, including residual-head fine-tuning, LoRA, and full fine-tuning. We find that MLPs pre-trained on LAMOST LRS achieve strong performance, even without fine-tuning, and that modest fine-tuning with DESI spectra further improves the results. For iron abundance,
embeddings from a
transformer-based model yield advantages in the metal-rich ([Fe/H] > -1.0) regime, but underperform in the metal-poor regime compared to MLPs trained directly on LRS. We also show that the optimal fine-tuning strategy depends on the specific
stellar parameter under consideration. These results highlight that simple pre-trained MLPs can provide competitive cross-survey generalization, while the role of spectral foundation models for cross-survey
stellar parameter estimation requires further exploration.
Read the full abstract ▾
All authors Xiaosheng Zhao, Yuan-Sen Ting, Rosemary F.G. Wyse, Alexander S. Szalay, Yang Huang, László Dobos, Tamás Budavári, Viska Wei
Machine Learning
Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang et al. (+4 more)
Diffusion language models are a promising alternative to autoregressive models due to their potential for faster generation. Among discrete
diffusion approaches, Masked
diffusion currently dominates, largely driven by strong perplexity on
language modeling
benchmarks. In this work, we present the first scaling law study of uniform-state and interpolating discrete
diffusion methods. We also show that Masked
diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-
entropy objective. We find that perplexity is informative within a
diffusion family but can be misleading across families, where models with worse likelihood scaling may be preferable due to faster and more practical sampling, as reflected by the speed-quality Pareto frontier. These results challenge the view that Masked
diffusion is categorically the future of
diffusion language modeling and that perplexity alone suffices for cross-
algorithm comparison. Scaling all methods to 1.7B parameters, we show that uniform-state
diffusion remains competitive on likelihood-based
benchmarks and outperforms autoregressive and Masked
diffusion models on GSM8K, despite worse validation perplexity. We provide the code, model checkpoints, and
video tutorials on the project page: http://s-sahoo.github.io/scaling-dllms
Read the full abstract ▾
All authors Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux, Jingyu Liu, John Thickstun, Ante Jukic
Machine Learning · Cross-listed from cs.CL
Cold-Start Personalization via Bayesian Adaptive Questioning
Avinandan Bose, Shuyue Stella Li, Faeze Brahman et al. (+6 more)
Cold-start personalization requires inferring preferences from minimal interaction when no user-specific historical data is available. The space of possible preferences is vast, yet users care about only a sparse subset and rarely articulate them upfront; combined with limited interaction budgets, this makes preference elicitation challenging. Our key insight is that preferences exhibit predictable structure across populations; e.g., users who want detailed explanations often also value worked examples. We propose CAPE (Cold-start Adaptive Preference Elicitation), which learns a structured world model of preference correlations offline from complete profiles, then performs training-free
Bayesian inference online to select informative questions and predict complete preference profiles, including dimensions never asked about. Even simple belief model instantiations (e.g., linear regression) substantially outperform end-to-end RL. Across medical, mathematical, social, and commonsense
reasoning, CAPE achieves 80.8%
alignment with ground-truth user preferences versus 68.5% for RL, requires 2.5x to over 15x fewer interactions, and adapts its questions twice as often as RL. Our contribution is a principled decomposition of cold-start personalization that makes
Bayesian preference elicitation practical at scale for LLM systems.
Read the full abstract ▾
All authors Avinandan Bose, Shuyue Stella Li, Faeze Brahman, Pang Wei Koh, Simon Shaolei Du, Yulia Tsvetkov, Maryam Fazel, Lin Xiao, Asli Celikyilmaz
Machine Learning · Cross-listed from cs.RO
BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames
Max Sobol Mark, Jacky Liang, Maria Attarian et al. (+4 more)
Many
robot tasks require attending to the history of past observations. For example, finding an item in a room requires remembering which places have already been searched. However, the best-performing
robot policies typically condition only on the current observation, limiting their applicability to such tasks. Naively conditioning on past observations often fails due to spurious correlations: policies latch onto incidental features of training histories that do not generalize to out-of-distribution trajectories upon deployment. We analyze why policies latch onto these spurious correlations and find that this problem stems from limited coverage over the space of possible histories during training, which grows exponentially with horizon. Existing regularization techniques provide inconsistent benefits across tasks, as they do not fundamentally address this coverage problem. Motivated by these findings, we propose Big Picture Policies (BPP), an approach that conditions on a minimal set of meaningful keyframes detected by a
vision-
language model. By projecting diverse rollouts onto a compact set of task-relevant events, BPP substantially reduces distribution shift between training and deployment, without sacrificing expressivity. We evaluate BPP on four challenging real-world manipulation tasks and three simulation tasks, all requiring history conditioning. BPP achieves 70% higher success rates than the best comparison on real-world evaluations.
Videos are available at https://bigpicturepolicies.github.io/
Read the full abstract ▾
All authors Max Sobol Mark, Jacky Liang, Maria Attarian, Chuyuan Fu, Debidatta Dwibedi, Dhruv Shah, Aviral Kumar
Machine Learning
Efficient Sampling with Discrete Diffusion Models: Sharp and Adaptive Guarantees
Daniil Dmitriev, Zhihan Huang and Yuting Wei
Diffusion models over discrete spaces have recently shown striking empirical success, yet their theoretical foundations remain incomplete. In this paper, we study the sampling efficiency of score-based discrete
diffusion models under a continuous-time Markov chain (CTMC) formulation, with a focus on $\tau$-leaping-based samplers. We establish sharp convergence guarantees for attaining $\varepsilon$ accuracy in Kullback-Leibler (KL) divergence for both uniform and masking noising processes. For uniform discrete
diffusion, we show that the $\tau$-leaping
algorithm achieves an iteration complexity of order $\tilde O(d/\varepsilon)$, with $d$ the ambient dimension of the target distribution, eliminating linear dependence on the vocabulary size $S$ and improving existing bounds by a factor of $d$; moreover, we establish a matching algorithmic lower bound showing that linear dependence on the ambient dimension is unavoidable in general. For masking discrete
diffusion, we introduce a modified $\tau$-leaping sampler whose convergence rate is governed by an intrinsic information-theoretic quantity, termed the effective total correlation, which is bounded by $d \log S$ but can be sublinear or even constant for structured data. As a consequence, the sampler provably adapts to low-dimensional structure without prior knowledge or algorithmic modification, yielding sublinear convergence rates for various practical examples (such as hidden Markov models, image data, and random graphs). Our analysis requires no boundedness or smoothness assumptions on the score
estimator beyond control of the score
entropy loss.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.MA
Distributed Quantum Gaussian Processes for Multi-Agent Systems
Meet Gandhi and George P. Kontoudis
Gaussian Processes (GPs) are a powerful tool for probabilistic modeling, but their performance is often constrained in complex, large-scale real-world domains due to the limited expressivity of classical kernels.
Quantum computing offers the potential to overcome this limitation by
embedding data into exponentially large
Hilbert spaces, capturing complex correlations that remain inaccessible to classical computing approaches. In this paper, we propose a Distributed
Quantum Gaussian Process (DQGP) method in a multi-
agent setting to enhance modeling capabilities and scalability. To address the challenging non-Euclidean optimization problem, we develop a Distributed consensus Riemannian Alternating Direction Method of Multipliers (DR-ADMM)
algorithm that aggregates local
agent models into a global model. We evaluate the efficacy of our method through numerical experiments conducted on a
quantum simulator in classical hardware. We use real-world, non-stationary elevation
datasets of NASA's Shuttle Radar Topography Mission and synthetic
datasets generated by
Quantum Gaussian Processes. Beyond modeling advantages, our framework highlights potential computational speedups that
quantum hardware may provide, particularly in Gaussian processes and distributed optimization.
Read the full abstract ▾
Machine Learning
PDE foundation models are skillful AI weather emulators for the Martian atmosphere
Johannes Schmude, Sujit Roy, Liping Wang et al. (+10 more)
We show that AI foundation models that are pretrained on numerical solutions to a diverse corpus of partial differential equations can be adapted and fine-tuned to obtain skillful predictive weather emulators for the Martian atmosphere. We base our work on the Poseidon PDE foundation model for two-dimensional systems. We develop a method to extend Poseidon from two to three dimensions while keeping the pretraining information. Moreover, we investigate the performance of the model in the presence of sparse initial conditions. Our results make use of four Martian years (approx.~34 GB) of training data and a median compute budget of 13 GPU hours. We find that the combination of pretraining and model extension yields a performance increase of 34.4% on a held-out year. This shows that PDEs-FMs can not only approximate solutions to (other) PDEs but also anchor models for real-world problems with complex interactions that lack a sufficient amount of training data or a suitable compute budget.
Read the full abstract ▾
All authors Johannes Schmude, Sujit Roy, Liping Wang, Theodore van Kessel, Levente Klein, Marcus Freitag, Eloisa Bentivegna, Robert Manson-Sawko, Bjorn Lutjens, Manil Maskey, Campbell Watson, Rahul Ramachandran, Juan Bernabe-Moreno
Machine Learning
Boundary Point Jailbreaking of Black-Box LLMs
Xander Davies, Giorgi Giglemiani, Edmund Lau et al. (+3 more)
Frontier LLMs are safeguarded against attempts to extract harmful information via
adversarial prompts known as "jailbreaks". Recently, defenders have developed classifier-based systems that have survived thousands of hours of human red teaming. We introduce Boundary Point Jailbreaking (BPJ), a new class of automated jailbreak attacks that evade the strongest industry-deployed safeguards. Unlike previous attacks that rely on white/grey-box assumptions (such as classifier scores or gradients) or libraries of existing jailbreaks, BPJ is fully black-box and uses only a single bit of information per query: whether or not the classifier flags the interaction. To achieve this, BPJ addresses the core difficulty in optimising attacks against robust real-world defences: evaluating whether a proposed modification to an attack is an improvement. Instead of directly trying to learn an attack for a target harmful string, BPJ converts the string into a curriculum of intermediate attack targets and then actively selects evaluation points that best detect small changes in attack strength ("boundary points"). We believe BPJ is the first fully automated attack
algorithm that succeeds in developing universal jailbreaks against Constitutional Classifiers, as well as the first automated attack
algorithm that succeeds against GPT-5's input classifier without relying on human attack seeds. BPJ is difficult to defend against in individual interactions but incurs many flags during optimisation, suggesting that effective defence requires supplementing single-interaction methods with batch-level monitoring.
Read the full abstract ▾
All authors Xander Davies, Giorgi Giglemiani, Edmund Lau, Eric Winsor, Geoffrey Irving, Yarin Gal
Machine Learning
Spectral Convolution on Orbifolds for Geometric Deep Learning
Tim Mangliers, Bernhard Mössner and Benjamin Himpel
Geometric deep learning (GDL) deals with supervised learning on data domains that go beyond Euclidean structure, such as data with graph or
manifold structure. Due to the demand that arises from application-related data, there is a need to identify further topological and
geometric structures with which these use cases can be made accessible to machine learning. There are various techniques, such as spectral convolution, that form the basic building blocks for some convolutional neural network-like architectures on non-Euclidean data. In this paper, the concept of spectral convolution on orbifolds is introduced. This provides a building block for making learning on orbifold structured data accessible using GDL. The theory discussed is illustrated using an example from music theory.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery
Ayush Shrivastava, Kirtan Gangani, Laksh Jain et al. (+2 more)
Vision language models (VLMs) achieve strong performance on RGB imagery, but they do not generalize to thermal images. Thermal sensing plays a critical role in settings where visible light fails, including nighttime surveillance, search and rescue, autonomous driving, and medical screening. Unlike RGB imagery, thermal images encode physical temperature rather than color or texture, requiring perceptual and
reasoning capabilities that existing RGB-centric
benchmarks do not evaluate. We introduce ThermEval-B, a structured
benchmark of approximately 55,000 thermal visual question answering pairs designed to assess the foundational primitives required for thermal
vision language understanding. ThermEval-B integrates public
datasets with our newly collected ThermEval-D, the first
dataset to provide dense per-pixel temperature maps with semantic body-part annotations across diverse indoor and outdoor environments. Evaluating 25 open-source and closed-source VLMs, we find that models consistently fail at temperature-grounded
reasoning, degrade under colormap transformations, and default to
language priors or fixed responses, with only marginal gains from prompting or supervised fine-tuning. These results demonstrate that thermal understanding requires dedicated evaluation beyond RGB-centric assumptions, positioning ThermEval as a
benchmark to drive progress in thermal
vision language modeling.
Read the full abstract ▾
All authors Ayush Shrivastava, Kirtan Gangani, Laksh Jain, Mayank Goel, Nipun Batra
Machine Learning
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Carolin Cissee, Raneen Younis and Zahra Ahmadi
Multimodal learning seeks to integrate information from heterogeneous sources, where signals may be shared across modalities, specific to individual modalities, or emerge only through their interaction. While self-supervised
multimodal contrastive learning has achieved remarkable progress, most existing methods predominantly capture redundant cross-modal signals, often neglecting modality-specific (unique) and interaction-driven (synergistic) information. Recent extensions broaden this perspective, yet they either fail to explicitly model synergistic interactions or learn different information components in an entangled manner, leading to incomplete representations and potential information leakage. We introduce COrAL, a principled framework that explicitly and simultaneously preserves redundant, unique, and synergistic information within
multimodal representations. COrAL employs a dual-path architecture with orthogonality constraints to disentangle shared and modality-specific features, ensuring a clean separation of information components. To promote synergy modeling, we introduce asymmetric masking with complementary view-specific patterns, compelling the model to infer cross-modal dependencies rather than rely solely on redundant cues. Extensive experiments on synthetic
benchmarks and diverse MultiBench
datasets demonstrate that COrAL consistently matches or outperforms state-of-the-art methods while exhibiting low performance variance across runs. These results indicate that explicitly modeling the full spectrum of
multimodal information yields more stable, reliable, and comprehensive
embeddings.
Read the full abstract ▾
Machine Learning
MacroGuide: Topological Guidance for Macrocycle Generation
Alicja Maksymiuk, Alexandre Duplessis, Michael Bronstein et al. (+3 more)
Macrocycles are ring-shaped molecules that offer a promising alternative to small-molecule drugs due to their enhanced selectivity and binding affinity against difficult targets. Despite their chemical value, they remain underexplored in generative modeling, likely owing to their scarcity in public
datasets and the challenges of enforcing topological constraints in standard deep generative models. We introduce MacroGuide: Topological Guidance for Macrocycle Generation, a
diffusion guidance mechanism that uses Persistent Homology to steer the sampling of pretrained molecular generative models toward the generation of macrocycles, in both unconditional and conditional (
protein pocket) settings. At each denoising step, MacroGuide constructs a Vietoris-Rips complex from atomic positions and promotes ring formation by optimizing persistent homology features. Empirically, applying MacroGuide to pretrained
diffusion models increases macrocycle generation rates from 1% to 99%, while matching or exceeding state-of-the-art performance on key quality metrics such as chemical validity, diversity, and PoseBusters checks.
Read the full abstract ▾
All authors Alicja Maksymiuk, Alexandre Duplessis, Michael Bronstein, Alexander Tong, Fernanda Duarte, İsmail İlkan Ceylan
Machine Learning · Cross-listed from physics.chem-ph
Faster Molecular Dynamics with Neural Network Potentials via Distilled Multiple Time-Stepping and Non-Conservative Forces
Nicolaï Gouraud, Côme Cattin, Thomas Plé et al. (+3 more)
Following our previous work (J. Phys. Chem. Lett., 2026, 17, 5, 1288-1295), we propose the DMTS-NC approach, a distilled multi-time-step (DMTS) strategy using non-conservative (NC) forces to further accelerate atomistic molecular dynamics simulations using foundation neural network models such as FeNNix-Bio1. There, a dual-level reversible reference system propagator
algorithm (RESPA) formalism couples a target accurate conservative potential to a simplified distilled representation optimized for the production of non-conservative forces. Despite being non-conservative, the distilled architecture is designed to enforce key physical priors, such as equivariance under rotation and cancellation of atomic force components. These choices facilitate the
distillation process and therefore improve drastically the robustness of simulation, significantly limiting abnormal discrepancies between the two models, thus achieving excellent agreement with the forces data. Overall, the DMTS-NC scheme is found to be more stable and efficient than its conservative counterpart with additional speedups reaching 15-30% over DMTS. Requiring no fine-tuning steps, it is easier to implement and can be pushed to the limit of the systems physical resonances to maintain accuracy while providing maximum efficiency. We obtain additional speedup by combining hydrogen mass repartitioning (HMR), High Hydrogen Friction (HHF) to further extended the largest timestep up to 10fs of our schemes while conserving stability and accuracy. As for DMTS, DMTS-NC is applicable to any neural network potential and can be applied to approaches that are computationally heavier than FeNNix-Bio1. We show a proof of principle applying the approach to the
distillation of MACE-OFF23 with consequent speedups ranging from 3.66 to 5.64 compared to single timestep.
Read the full abstract ▾
All authors Nicolaï Gouraud, Côme Cattin, Thomas Plé, Olivier Adjoua, Louis Lagardère, Jean-Philip Piquemal
Machine Learning
Use What You Know: Causal Foundation Models with Partial Graphs
Arik Reuter, Anish Dhir, Cristiana Diaconu et al. (+6 more)
Estimating causal quantities traditionally relies on bespoke
estimators tailored to specific assumptions. Recently proposed Causal Foundation Models (CFMs) promise a more unified approach by amortising causal discovery and inference in a single step. However, in their current state, they do not allow for the incorporation of any domain knowledge, which can lead to suboptimal predictions. We bridge this gap by introducing methods to condition CFMs on causal information, such as the causal graph or more readily available ancestral information. When access to complete causal graph information is too strict a requirement, our approach also effectively leverages partial causal information. We systematically evaluate conditioning strategies and find that injecting learnable biases into the
attention mechanism, together with a graph-convolutional encoder, is a highly effective method to utilise full and partial causal information. Our experiments show that this conditioning allows a general-purpose CFM to match the performance of specialised models trained on specific causal structures. Overall, our approach addresses a central hurdle on the path towards all-in-one causal foundation models: the capability to answer causal queries in a data-driven manner while effectively leveraging any amount of domain expertise.
Read the full abstract ▾
All authors Arik Reuter, Anish Dhir, Cristiana Diaconu, Jake Robertson, Ole Ossen, Frank Hutter, Adrian Weller, Mark van der Wilk, Bernhard Schölkopf
Machine Learning
Locally Adaptive Multi-Objective Learning
Jivat Neet Kaur, Isaac Gibbs and Michael I. Jordan
We consider the general problem of learning a predictor that satisfies multiple objectives of interest simultaneously, a broad framework that captures a range of specific learning goals including calibration, regret, and multiaccuracy. We work in an online setting where the data distribution can change arbitrarily over time. Existing approaches to this problem aim to minimize the set of objectives over the entire time horizon in a worst-case sense, and in practice they do not necessarily adapt to distribution shifts. Earlier work has aimed to alleviate this problem by incorporating additional objectives that target local guarantees over contiguous subintervals. Empirical evaluation of these proposals is, however, scarce. In this article, we consider an alternative procedure that achieves local adaptivity by replacing one part of the multi-objective learning method with an adaptive online
algorithm. Empirical evaluations on
datasets from energy
forecasting and algorithmic
fairness show that our proposed method improves upon existing approaches and achieves unbiased predictions over subgroups, while remaining robust under distribution shift.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SY
Gradient Networks for Universal Magnetic Modeling of Synchronous Machines
Junyi Li, Tim Foissner, Floran Martin et al. (+2 more)
This paper presents a physics-constrained neural network framework for
magnetic modeling of saturable synchronous machines, including spatial harmonics. By
embedding gradient networks into the machine equations to model conservative electromagnetic behavior, the framework satisfies reciprocity and energy conservation by construction, while universally approximating any physically feasible
magnetic characteristic. Unlike lookup tables and black-box neural networks, it guarantees monotonicity, invertibility, and smooth outputs, and remains highly data efficient. The method is validated using measured and finite-element method (FEM) data from a 5.6-kW permanent-magnet (PM) synchronous reluctance machine, and is demonstrated in real-time closed-loop control on an embedded platform. The results confirm accurate, physically consistent, and computationally efficient performance.
Read the full abstract ▾
All authors Junyi Li, Tim Foissner, Floran Martin, Antti Piippo, Marko Hinkkanen
Machine Learning · Cross-listed from eess.SY
Fault Detection in Electrical Distribution System using Autoencoders
Sidharthenee Nayak, Victor Sam Moses Babu, Chandrashekhar Narayan Bhende et al. (+2 more)
In recent times, there has been considerable interest in fault detection within electrical power systems, garnering
attention from both academic researchers and industry professionals. Despite the development of numerous fault detection methods and their adaptations over the past decade, their practical application remains highly challenging. Given the probabilistic nature of fault occurrences and parameters, certain decision-making tasks could be approached from a probabilistic standpoint. Protective systems are tasked with the detection, classification, and localization of faulty voltage and current line magnitudes, culminating in the activation of
circuit breakers to isolate the faulty line. An essential aspect of designing effective fault detection systems lies in obtaining reliable data for training and testing, which is often scarce. Leveraging deep learning techniques, particularly the powerful capabilities of pattern classifiers in learning, generalizing, and parallel processing, offers promising avenues for intelligent fault detection. To address this, our paper proposes an anomaly-based approach for fault detection in electrical power systems, employing deep autoencoders. Additionally, we utilize Convolutional Autoencoders (CAE) for dimensionality reduction, which, due to its fewer parameters, requires less training time compared to conventional autoencoders. The proposed method demonstrates superior performance and accuracy compared to alternative detection approaches by achieving an accuracy of 97.62% and 99.92% on simulated and publicly available
datasets.
Read the full abstract ▾
All authors Sidharthenee Nayak, Victor Sam Moses Babu, Chandrashekhar Narayan Bhende, Pratyush Chakraborty, Mayukha Pal
Machine Learning
Variance-Reduced $(\varepsilon,\delta)-$Unlearning using Forget Set Gradients
Martin Van Waerebeke, Marco Lorenzi, Kevin Scaman et al. (+2 more)
In machine unlearning, $(\varepsilon,\delta)-$unlearning is a popular framework that provides formal guarantees on the effectiveness of the removal of a subset of training data, the forget set, from a trained model. For strongly convex objectives, existing first-order methods achieve $(\varepsilon,\delta)-$unlearning, but they only use the forget set to calibrate injected noise, never as a direct optimization signal. In contrast, efficient empirical heuristics often exploit the forget samples (e.g., via gradient ascent) but come with no formal unlearning guarantees. We bridge this gap by presenting the Variance-Reduced Unlearning (VRU)
algorithm. To the best of our knowledge, VRU is the first first-order
algorithm that directly includes forget set gradients in its update rule, while provably satisfying ($(\varepsilon,\delta)-$unlearning. We establish the convergence of VRU and show that incorporating the forget set yields strictly improved rates, i.e. a better dependence on the achieved error compared to existing first-order $(\varepsilon,\delta)-$unlearning methods. Moreover, we prove that, in a low-error regime, VRU asymptotically outperforms any first-order method that ignores the forget set.Experiments corroborate our theory, showing consistent gains over both state-of-the-art certified unlearning methods and over empirical baselines that explicitly leverage the forget set.
Read the full abstract ▾
All authors Martin Van Waerebeke, Marco Lorenzi, Kevin Scaman, El Mahdi El Mhamdi, Giovanni Neglia
Machine Learning · Cross-listed from stat.ML
Activation-Space Uncertainty Quantification for Pretrained Networks
Richard Bergna, Stefan Depeweg, Sergio Calvo-Ordoñez et al. (+3 more)
Reliable uncertainty estimates are crucial for deploying pretrained models; yet, many strong methods for quantifying uncertainty require retraining, Monte Carlo sampling, or expensive second-order computations and may alter a frozen backbone's predictions. To address this, we introduce Gaussian Process Activations (GAPA), a post-hoc method that shifts
Bayesian modeling from weights to activations. GAPA replaces standard nonlinearities with Gaussian-process activations whose posterior mean exactly matches the original activation, preserving the backbone's point predictions by construction while providing closed-form epistemic variances in activation space. To scale to modern architectures, we use a sparse variational inducing-point approximation over cached training activations, combined with local k-nearest-neighbor subset conditioning, enabling deterministic single-pass uncertainty propagation without sampling, backpropagation, or second-order information. Across regression, classification, image
segmentation, and
language modeling, GAPA matches or outperforms strong post-hoc baselines in calibration and out-of-distribution detection while remaining efficient at test time.
Read the full abstract ▾
All authors Richard Bergna, Stefan Depeweg, Sergio Calvo-Ordoñez, Jonathan Plenk, Alvaro Cartea, Jose Miguel Hernández-Lobato