Machine Learning · Cross-listed from stat.ML
Partition Function Estimation under Bounded f-Divergence
Adam Block and Abhishek Shetty
We study the statistical complexity of estimating partition functions given sample access to a proposal distribution and an unnormalized density ratio for a target distribution. While partition function estimation is a classical problem, existing guarantees typically rely on structural assumptions about the domain or model geometry. We instead provide a general, information-theoretic characterization that depends only on the relationship between the proposal and target distributions. Our analysis introduces the integrated coverage profile, a functional that quantifies how much target mass lies in regions where the density ratio is large. We show that integrated coverage tightly characterizes the sample complexity of multiplicative partition function estimation and provide matching lower bounds. We further express these bounds in terms of $f$-divergences, yielding sharp phase transitions depending on the growth rate of f and recovering classical results as a special case while extending to heavy-tailed regimes. Matching lower bounds establish tightness in all regimes. As applications, we derive improved finite-sample guarantees for importance sampling and self-normalized importance sampling, and we show a strict separation between the complexity of approximate sampling and counting under the same divergence constraints. Our results unify and generalize prior analyses of importance sampling, rejection sampling, and heavy-tailed mean estimation, providing a minimal-assumption theory of partition function estimation. Along the way we introduce new technical tools including new connections between coverage and $f$-divergences as well as a generalization of the classical Paley-Zygmund inequality.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.IV
Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models
Chi-Sheng Chen, Xinyu Zhang, Guan-Ying Chen et al. (+3 more)
Foundation models pretrained on large-scale 3D medical imaging data face challenges when adapted to multiple downstream tasks under continual learning with limited labeled data. We address few-shot continual learning for 3D
brain MRI by combining a frozen pretrained backbone with task-specific Low-Rank Adaptation (LoRA) modules. Tasks arrive sequentially – tumor
segmentation (BraTS) and
brain age estimation (IXI) – with no replay of previous task data. Each task receives a dedicated LoRA adapter; only the adapter and task-specific head are trained while the backbone remains frozen, thereby eliminating catastrophic forgetting by design (BWT=0). In continual learning, sequential full fine-tuning suffers severe forgetting (T1 Dice drops from 0.80 to 0.16 after T2), while sequential linear probing achieves strong T1 (Dice 0.79) but fails on T2 (MAE 1.45). Our LoRA approach achieves the best balanced performance across both tasks: T1 Dice 0.62$\pm$0.07, T2 MAE 0.16$\pm$0.05, with zero forgetting and $<$0.1% trainable parameters per task, though with noted systematic age underestimation in T2 (Wilcoxon $p<0.001$). Frozen foundation models with task-specific LoRA adapters thus offer a practical solution when both tasks must be maintained under few-shot continual learning.
Read the full abstract ▾
All authors Chi-Sheng Chen, Xinyu Zhang, Guan-Ying Chen, Qiuzhe Xie, Fan Zhang, En-Jui Kuo
Machine Learning
Active Value Querying to Minimize Additive Error in Subadditive Set Function Learning
Martin Černý, David Sychrovský, Filip Úradník et al. (+1 more)
Subadditive set functions play a pivotal role in computational economics (especially in combinatorial auctions), combinatorial optimization or artificial intelligence applications such as interpretable machine learning. However, specifying a set function requires assigning values to an exponentially large number of subsets in general, a task that is often resource-intensive in practice, particularly when the values derive from external sources such as retraining of machine learning models. A~simple omission of certain values introduces ambiguity that becomes even more significant when the incomplete set function has to be further optimized over. Motivated by the well-known result about inapproximability of subadditive functions using deterministic value queries with respect to a multiplicative error, we study a problem of approximating an unknown subadditive (or a subclass of thereof) set function with respect to an additive error – i. e., we aim to efficiently close the distance between minimal and maximal completions. Our contributions are threefold: (i) a thorough exploration of minimal and maximal completions of different classes of set functions with missing values and an analysis of their resulting distance; (ii) the development of methods to minimize this distance over classes of set functions with a known prior, achieved by disclosing values of additional subsets in both offline and online manner; and (iii) empirical demonstrations of the
algorithms' performance in practical scenarios.
Read the full abstract ▾
All authors Martin Černý, David Sychrovský, Filip Úradník, Jakub Černý
Machine Learning
Neural Operators Can Discover Functional Clusters
Yicen Li, Jose Antonio Lara Benitez, Ruiyang Hong et al. (+3 more)
Operator learning is reshaping scientific computing by amortizing inference across infinite families of problems. While neural operators (NOs) are increasingly well understood for regression, far less is known for classification and its unsupervised analogue: clustering. We prove that sample-based neural operators can learn any finite collection of classes in an infinite-dimensional reproducing kernel
Hilbert space, even when the classes are neither convex nor connected, under mild kernel sampling assumptions. Our universal clustering
theorem shows that any $K$ closed classes can be approximated to arbitrary precision by NO-parameterized classes in the upper Kuratowski
topology on closed sets, a notion that can be interpreted as disallowing false-positive misclassifications. Building on this, we develop an NO-powered clustering pipeline for functional data and apply it to unlabeled families of ordinary differential equation (ODE) trajectories. Discretized trajectories are lifted by a fixed pre-trained encoder into a continuous feature map and mapped to soft assignments by a lightweight trainable head. Experiments on diverse synthetic ODE
benchmarks show that the resulting practical SNO recovers latent dynamical structure in regimes where classical methods fail, providing evidence consistent with our universal clustering theory.
Read the full abstract ▾
All authors Yicen Li, Jose Antonio Lara Benitez, Ruiyang Hong, Anastasis Kratsios, Paul David McNicholas, Maarten Valentijn de Hoop
Machine Learning · Cross-listed from cs.RO
V-MORALS: Visual Morse Graph-Aided Estimation of Regions of Attraction in a Learned Latent Space
Faiz Aladin, Ashwin Balasubramanian, Lars Lindemann et al. (+1 more)
Reachability analysis has become increasingly important in robotics to distinguish safe from unsafe states. Unfortunately, existing reachability and safety analysis methods often fall short, as they typically require known system dynamics or large
datasets to estimate accurate system models, are computationally expensive, and assume full state information. A recent method, called MORALS, aims to address these shortcomings by using topological tools to estimate Regions of Attraction (ROA) in a low-dimensional latent space. However, MORALS still relies on full state knowledge and has not been studied when only sensor measurements are available. This paper presents Visual Morse Graph-Aided Estimation of Regions of Attraction in a Learned Latent Space (V-MORALS). V-MORALS takes in a
dataset of image-based trajectories of a system under a given controller, and learns a latent space for reachability analysis. Using this learned latent space, our method is able to generate well-defined Morse Graphs, from which we can compute ROAs for various systems and controllers. V-MORALS provides capabilities similar to the original MORALS architecture without relying on state knowledge, and using only high-level sensor data. Our project website is at: https://v-morals.onrender.com.
Read the full abstract ▾
All authors Faiz Aladin, Ashwin Balasubramanian, Lars Lindemann, Daniel Seita
Machine Learning · Cross-listed from stat.ML
Uncovering Physical Drivers of Dark Matter Halo Structures with Auxiliary-Variable-Guided Generative Models
Arkaprabha Ganguli, Anirban Samaddar, Florian Kéruzoré et al. (+4 more)
Deep generative models (DGMs) compress high-dimensional data but often entangle distinct physical factors in their latent spaces. We present an auxiliary-variable-guided framework for disentangling representations of thermal Sunyaev-Zel'dovich (tSZ) maps of dark matter
halos. We introduce
halo mass and concentration as auxiliary variables and apply a lightweight
alignment penalty to encourage latent dimensions to reflect these physical quantities. To generate sharp and realistic samples, we extend latent conditional flow matching (LCFM), a state-of-the-art generative model, to enforce disentanglement in the latent space. Our Disentangled Latent-CFM (DL-CFM) model recovers the established mass-concentration scaling relation and identifies latent space outliers that may correspond to unusual
halo formation histories. By linking latent coordinates to interpretable astrophysical properties, our method transforms the latent space into a diagnostic tool for cosmological structure. This work demonstrates that auxiliary guidance preserves generative flexibility while yielding physically meaningful, disentangled
embeddings, providing a generalizable pathway for uncovering independent factors in complex astronomical
datasets.
Read the full abstract ▾
All authors Arkaprabha Ganguli, Anirban Samaddar, Florian Kéruzoré, Nesar Ramachandra, Julie Bessac, Sandeep Madireddy, Emil Constantinescu
Machine Learning · Cross-listed from cs.CR
Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theory
Meisam Mohammady, Qin Yang, Nicholas Stout et al. (+4 more)
Differentially Private
Stochastic Gradient Descent (DP-SGD) is a cornerstone technique for ensuring
privacy in deep learning, widely used in both training from scratch and fine-tuning large-scale
language models. While DP-SGD predominantly relies on the Gaussian mechanism, the Laplace mechanism remains underutilized due to its reliance on L1 norm clipping. This constraint severely limits its practicality in high-dimensional models because the L1 norm of an n-dimensional gradient can be up to sqrt(n) times larger than its L2 norm. As a result, the required noise scale grows significantly with model size, leading to poor utility or untrainable models. In this work, we introduce Lap2, a new solution that enables L2 clipping for Laplace DP-SGD while preserving strong
privacy guarantees. We overcome the dimensionality-driven clipping barrier by computing coordinate-wise moment bounds and applying majorization theory to construct a tight, data-independent upper bound over the full model. By exploiting the Schur-convexity of the moment accountant function, we aggregate these bounds using a carefully designed majorization set that respects the L2 clipping constraint. This yields a multivariate
privacy accountant that scales gracefully with model dimension and enables the use of thousands of moments. Empirical evaluations demonstrate that our approach significantly improves the performance of Laplace DP-SGD, achieving results comparable to or better than Gaussian DP-SGD under strong
privacy constraints. For instance, fine-tuning RoBERTa-base (125M parameters) on SST-2 achieves 87.88% accuracy at epsilon=0.54, outperforming Gaussian (87.16%) and standard Laplace (48.97%) under the same budget.
Read the full abstract ▾
All authors Meisam Mohammady, Qin Yang, Nicholas Stout, Ayesha Samreen, Han Wang, Christopher J Quinn, Yuan Hong
Machine Learning · Cross-listed from cs.CV
Modelling and Simulation of Neuromorphic Datasets for Anomaly Detection in Computer Vision
Mike Middleton, Teymoor Ali, Hakan Kayan et al. (+6 more)
Limitations on the availability of Dynamic
Vision Sensors (DVS) present a fundamental challenge to researchers of neuromorphic computer
vision applications. In response,
datasets have been created by the research community, but often contain a limited number of samples or scenarios. To address the lack of a comprehensive simulator of neuromorphic
vision datasets, we introduce the Anomalous Neuromorphic Tool for Shapes (ANTShapes), a novel
dataset simulation framework. Built in the Unity engine, ANTShapes simulates abstract, configurable 3D scenes populated by objects displaying randomly-generated behaviours describing attributes such as motion and rotation. The sampling of object behaviours, and the labelling of anomalously-acting objects, is a statistical process following central limit
theorem principles.
Datasets containing an arbitrary number of samples can be created and exported from ANTShapes, along with accompanying label and frame data, through the adjustment of a limited number of parameters within the software. ANTShapes addresses the limitations of data availability to researchers of event-based computer
vision by allowing for the simulation of bespoke
datasets to suit purposes including object recognition and localisation alongside anomaly detection.
Read the full abstract ▾
All authors Mike Middleton, Teymoor Ali, Hakan Kayan, Basabdatta Sen Bhattacharya, Charith Perera, Oliver Rhodes, Elena Gheorghiu, Mark Vousden, Martin A. Trefzer
Machine Learning
Sample Size Calculations for Developing Clinical Prediction Models: Overview and pmsims R package
Diana Shamsutdinova, Felix Zimmer, Oyebayo Ridwan Olaniran et al. (+4 more)
Background: Clinical prediction models are increasingly used to inform healthcare decisions, but determining the minimum sample size for their development remains a critical and unresolved challenge. Inadequate sample sizes can lead to overfitting, poor generalisability, and biased predictions. Existing approaches, such as heuristic rules, closed-form formulas, and simulation-based methods, vary in flexibility and accuracy, particularly for complex data structures and machine learning models. Methods: We review current methodologies for sample size estimation in prediction modelling and introduce a conceptual framework that distinguishes between mean-based and assurance-based criteria. Building on this, we propose a novel simulation-based approach that integrates learning curves, Gaussian Process optimisation, and assurance principles to identify sample sizes that achieve target performance with high probability. This approach is implemented in pmsims, an open-source, model-agnostic R package. Results: Through case studies, we demonstrate that sample size estimates vary substantially across methods, performance metrics, and modelling strategies. Compared to existing tools, pmsims provides flexible, efficient, and interpretable solutions that accommodate diverse models and user-defined metrics while explicitly accounting for variability in model performance. Conclusions: Our framework and software advance sample size methodology for clinical prediction modelling by combining flexibility with computational efficiency. Future work should extend these methods to hierarchical and
multimodal data, incorporate
fairness and stability metrics, and address challenges such as missing data and complex dependency structures.
Read the full abstract ▾
All authors Diana Shamsutdinova, Felix Zimmer, Oyebayo Ridwan Olaniran, Sarah Markham, Daniel Stahl, Gordon Forbes, Ewan Carr
Machine Learning
FedDAG: Clustered Federated Learning via Global Data and Gradient Integration for Heterogeneous Environments
Anik Pramanik, Murat Kantarcioglu, Vincent Oria et al. (+1 more)
Federated Learning (FL) enables a group of clients to collaboratively train a model without sharing individual data, but its performance drops when client data are heterogeneous. Clustered FL tackles this by grouping similar clients. However, existing clustered FL approaches rely solely on either data similarity or gradient similarity; however, this results in an incomplete assessment of client similarities. Prior clustered FL approaches also restrict knowledge and representation sharing to clients within the same cluster. This prevents cluster models from benefiting from the diverse client population across clusters. To address these limitations, FedDAG introduces a clustered FL framework, FedDAG, that employs a weighted, class-wise similarity metric that integrates both data and gradient information, providing a more holistic measure of similarity during clustering. In addition, FedDAG adopts a dual-encoder architecture for cluster models, comprising a primary encoder trained on its own clients' data and a secondary encoder refined using gradients from complementary clusters. This enables cross-cluster feature transfer while preserving cluster-specific specialization. Experiments on diverse
benchmarks and data heterogeneity settings show that FedDAG consistently outperforms state-of-the-art clustered FL baselines in accuracy.
Read the full abstract ▾
All authors Anik Pramanik, Murat Kantarcioglu, Vincent Oria, Shantanu Sharma
Machine Learning · Cross-listed from cs.CC
Spiky Rank and Its Applications to Rigidity and Circuits
Lianna Hambardzumyan, Konstantin Myasnikov, Artur Riazanov et al. (+2 more)
We introduce spiky rank, a new
matrix parameter that enhances blocky rank by combining the combinatorial structure of the latter with linear-algebraic flexibility. A spiky
matrix is block-structured with diagonal blocks that are arbitrary rank-one
matrices, and the spiky rank of a
matrix is the minimum number of such
matrices required to express it as a sum. This measure extends blocky rank to real
matrices and is more robust for problems with both combinatorial and algebraic character. Our conceptual contribution is as follows: we propose spiky rank as a well-behaved candidate
matrix complexity measure and demonstrate its potential through applications. We show that large spiky rank implies high
matrix rigidity, and that spiky rank lower bounds yield lower bounds for depth-2 ReLU
circuits, the basic building blocks of neural networks. On the technical side, we establish tight bounds for random
matrices and develop a framework for explicit lower bounds, applying it to Hamming distance
matrices and spectral expanders. Finally, we relate spiky rank to other
matrix parameters, including blocky rank, sparsity, and the $\gamma_2$-norm.
Read the full abstract ▾
All authors Lianna Hambardzumyan, Konstantin Myasnikov, Artur Riazanov, Morgan Shirley, Adi Shraibman
Machine Learning
Uncertainty-aware Language Guidance for Concept Bottleneck Models
Yangyi Li and Mengdi Huai
Concept Bottleneck Models (CBMs) provide inherent interpretability by first mapping input samples to high-level semantic concepts, followed by a combination of these concepts for the final classification. However, the annotation of human-understandable concepts requires extensive expert knowledge and labor, constraining the broad adoption of CBMs. On the other hand, there are a few works that leverage the knowledge of large
language models (LLMs) to construct concept bottlenecks. Nevertheless, they face two essential limitations: First, they overlook the uncertainty associated with the concepts annotated by LLMs and lack a valid mechanism to quantify uncertainty about the annotated concepts, increasing the risk of errors due to
hallucinations from LLMs. Additionally, they fail to incorporate the uncertainty associated with these annotations into the learning process for concept bottleneck models. To address these limitations, we propose a novel uncertainty-aware CBM method, which not only rigorously quantifies the uncertainty of LLM-annotated concept labels with valid and distribution-free guarantees, but also incorporates quantified concept uncertainty into the CBM training procedure to account for varying levels of reliability across LLM-annotated concepts. We also provide the theoretical analysis for our proposed method. Extensive experiments on the real-world
datasets validate the desired properties of our proposed methods.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IR
Cross-Representation Knowledge Transfer for Improved Sequential Recommendations
Artur Gimranov, Viacheslav Yusupov, Elfat Sabitov et al. (+4 more)
Transformer architectures, capable of capturing sequential dependencies in the history of user interactions, have become the dominant approach in sequential recommender systems. Despite their success, such models consider sequence elements in isolation, implicitly accounting for the complex relationships between them. Graph neural networks, in contrast, explicitly model these relationships through higher order interactions but are often unable to adequately capture their
evolution over time, limiting their use for predicting the next interaction. To fill this gap, we present a new framework that combines
transformers and graph neural networks and aligns different representations for solving next-item prediction task. Our solution simultaneously encodes structural dependencies in the interaction graph and tracks their dynamic change. Experimental results on a number of open
datasets demonstrate that the proposed framework consistently outperforms both pure sequential and graph approaches in terms of recommendation quality, as well as recent methods that combine both types of signals.
Read the full abstract ▾
All authors Artur Gimranov, Viacheslav Yusupov, Elfat Sabitov, Tatyana Matveeva, Anton Lysenko, Ruslan Israfilov, Evgeny Frolov
Machine Learning · Cross-listed from math.NA
On the Limits of Interpretable Machine Learning in Quintic Root Classification
Rohan Thomas and Majid Bani-Yaghoub
Can Machine Learning (ML) autonomously recover interpretable mathematical structure from raw numerical data? We aim to answer this question using the classification of real-root configurations of
polynomials up to degree five as a structured
benchmark. We tested an extensive set of ML models, including decision trees, logistic regression, support vector machines, random forest, gradient boosting, XGBoost, symbolic regression, and neural networks. Neural networks achieved strong in-distribution performance on quintic classification using raw coefficients alone (84.3% + or - 0.9% balanced accuracy), whereas decision trees perform substantially worse (59.9% + or - 0.9%). However, when provided with an explicit feature capturing sign changes at critical points, decision trees match neural performance (84.2% + or - 1.2%) and yield explicit classification rules. Knowledge
distillation reveals that this single invariant accounts for 97.5% of the extracted decision structure. Out-of-distribution, data-efficiency, and noise robustness analyses indicate that neural networks learn continuous, data-dependent
geometric approximations of the decision boundary rather than recovering scale-invariant symbolic rules. This distinction between
geometric approximation and symbolic invariance explains the gap between predictive performance and interpretability observed across models. Although high predictive accuracy is attainable, we find no evidence that the evaluated ML models autonomously recover discrete, human-interpretable mathematical rules from raw coefficients. These results suggest that, in structured mathematical domains, interpretability may require explicit structural inductive bias rather than purely data-driven approximation.
Read the full abstract ▾
Machine Learning · Cross-listed from physics.flu-dyn
Neural ensemble Kalman filter: Data assimilation for compressible flows with shocks
Xu-Hui Zhou, Lorenzo Beronilla, Michael K. Sleeman et al. (+4 more)
Data assimilation (DA) for compressible flows with shocks is challenging because many classical DA methods generate spurious oscillations and nonphysical features near uncertain shocks. We focus here on the ensemble Kalman filter (EnKF). We show that the poor performance of the EnKF may be attributed to the bimodal forecast distribution that can arise in the vicinity of an uncertain shock location; this violates the assumptions underpinning the EnKF, which assume a forecast which is close to Gaussian. To address this issue we introduce the new neural EnKF. The basic idea is to systematically embed neural function approximations within ensemble DA by mapping the forecast ensemble of shocked flows to the parameter space (weights and biases) of a deep neural network (NN) and to subsequently perform DA in that space. The nonlinear mapping encodes sharp and smooth flow features in an ensemble of NN parameters. Neural EnKF updates are therefore well-behaved only if the NN parameters vary smoothly within the neural representation of the forecast ensemble. We show that such a smooth variation of network parameters can be enforced via physics-informed transfer learning, and demonstrate that in so-doing the neural EnKF avoids the spurious oscillations and nonphysical features that plague the EnKF. The applicability of the neural EnKF is demonstrated through a series of systematic numerical experiments with the inviscid Burgers' equation, the Sod shock tube, and a two-dimensional blast wave.
Read the full abstract ▾
All authors Xu-Hui Zhou, Lorenzo Beronilla, Michael K. Sleeman, Hangchuan Hu, Matthias Morzfeld, Andrew M. Stuart, Tamer A. Zaki
Machine Learning
Global Interpretability via Automated Preprocessing: A Framework Inspired by Psychiatric Questionnaires
Eric V. Strobl
Psychiatric questionnaires are highly context sensitive and often only weakly predict subsequent symptom severity, which makes the prognostic relationship difficult to learn. Although flexible nonlinear models can improve predictive accuracy, their limited interpretability can erode clinical trust. In fields such as imaging and omics, investigators commonly address visit- and instrument-specific artifacts by extracting stable signal through preprocessing and then fitting an interpretable linear model. We adopt the same strategy for questionnaire data by decoupling preprocessing from prediction: REFINE (Redundancy-Exploiting Follow-up-Informed Nonlinear Enhancement) uses follow-up information to learn a nonlinear, item-aligned preprocessing of baseline measurements, and then maps these stabilized baseline items to future severity through a linear coefficient
matrix. This construction constrains where nonlinearity enters rather than restricting the end-to-end predictive function class, allowing the prognostic relationship to remain globally interpretable through a single coefficient
matrix rather than through post hoc local attributions. In the population, the resulting predictor recovers the Bayes-optimal conditional mean, so global linear interpretability does not require sacrificing nonlinear predictive flexibility. In experiments, REFINE outperforms other interpretable approaches while preserving clear global attribution of prognostic factors across psychiatric and non-psychiatric longitudinal prediction tasks.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.IV
SALIENT: Frequency-Aware Paired Diffusion for Controllable Long-Tail CT Detection
Yifan Li, Mehrdad Salimitari, Taiyu Zhang et al. (+2 more)
Detection of rare lesions in whole-body CT is fundamentally limited by extreme class imbalance and low target-to-volume ratios, producing precision collapse despite high AUROC. Synthetic augmentation with
diffusion models offers promise, yet pixel-space
diffusion is computationally expensive, and existing mask-conditioned approaches lack controllable attribute-level regulation and paired supervision for accountable training. We introduce SALIENT, a mask-conditioned wavelet-domain
diffusion framework that synthesizes paired lesion-masking volumes for controllable CT augmentation under long-tail regimes. Instead of denoising in pixel space, SALIENT performs structured
diffusion over discrete wavelet coefficients, explicitly separating low-frequency brightness from high-frequency structural detail. Learnable frequency-aware objectives disentangle target and background attributes (structure, contrast, edge fidelity), enabling interpretable and stable optimization. A 3D VAE generates diverse volumetric lesion masks, and a semi-supervised teacher produces paired slice-level pseudo-labels for downstream mask-guided detection. SALIENT improves generative realism, as reflected by higher MS-SSIM (0.63 to 0.83) and lower FID (118.4 to 46.5). In a separate downstream evaluation, SALIENT-augmented training improves long-tail detection performance, yielding disproportionate AUPRC gains across low prevalences and target-to-volume ratios. Optimal synthetic ratios shift from 2x to 4x as labeled seed size decreases, indicating a seed-dependent augmentation regime under low-label conditions. SALIENT demonstrates that frequency-aware
diffusion enables controllable, computationally efficient precision rescue in long-tail CT detection.
Read the full abstract ▾
All authors Yifan Li, Mehrdad Salimitari, Taiyu Zhang, Guang Li, David Dreizin
Machine Learning
Human Supervision as an Information Bottleneck: A Unified Theory of Error Floors in Human-Guided Learning
Alejandro Rodriguez Dominguez
Large
language models are trained primarily on human-generated data and feedback, yet they exhibit persistent errors arising from annotation noise, subjective preferences, and the limited expressive bandwidth of natural
language. We argue that these limitations reflect structural properties of the supervision channel rather than model scale or optimization. We develop a unified theory showing that whenever the human supervision channel is not sufficient for a latent evaluation target, it acts as an information-reducing channel that induces a strictly positive excess-risk floor for any learner dominated by it. We formalize this Human-Bounded Intelligence limit and show that across six complementary frameworks (operator theory, PAC-Bayes, information theory, causal inference, category theory, and game-theoretic analyses of reinforcement learning from human feedback), non-sufficiency yields strictly positive lower bounds arising from the same structural decomposition into annotation noise, preference distortion, and semantic compression. The theory explains why scaling alone cannot eliminate persistent human-aligned errors and characterizes conditions under which auxiliary non-human signals (e.g.,
retrieval, program execution, tools) increase effective supervision capacity and collapse the floor by restoring information about the latent target. Experiments on real preference data, synthetic known-target tasks, and externally verifiable
benchmarks confirm the predicted structural signatures: human-only supervision exhibits a persistent floor, while sufficiently informative auxiliary channels strictly reduce or eliminate excess error.
Read the full abstract ▾
Machine Learning
EvoX: Meta-Evolution for Automated Discovery
Shu Liu, Shubham Agarwal, Monishwaran Maheswaran et al. (+14 more)
Recent work such as AlphaEvolve has shown that combining LLM-driven optimization with evolutionary search can effectively improve programs,
prompts, and
algorithms across domains. In this paradigm, previously evaluated solutions are reused to guide the model toward new candidate solutions. Crucially, the effectiveness of this
evolution process depends on the search strategy: how prior solutions are selected and varied to generate new candidates. However, most existing methods rely on fixed search strategies with predefined knobs (e.g., explore-exploit ratios) that remain static throughout execution. While effective in some settings, these approaches often fail to adapt across tasks, or even within the same task as the search space changes over time. We introduce EvoX, an adaptive
evolution method that optimizes its own
evolution process. EvoX jointly evolves candidate solutions and the search strategies used to generate them, continuously updating how prior solutions are selected and varied based on progress. This enables the system to dynamically shift between different search strategies during the optimization process. Across nearly 200 real-world optimization tasks, EvoX outperforms existing AI-driven evolutionary methods including AlphaEvolve, OpenEvolve, GEPA, and ShinkaEvolve on the majority of tasks.
Read the full abstract ▾
All authors Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z. Pan, Alexander Du, Kurt Keutzer, Alvin Cheung, Alexandros G. Dimakis, Koushik Sen, Matei Zaharia, Ion Stoica
Machine Learning
Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG
Hanning Guo, Hanwen Bi, Farah Abdellatif et al. (+4 more)
Brain foundation models have achieved remarkable advances across a wide range of neuroscience tasks. However, most existing models are limited to a single functional modality, restricting their ability to exploit complementary spatiotemporal dynamics and the collective data scale across different neuroimaging techniques. This limitation largely arises from severe semantic heterogeneity and resolution discrepancies among modalities. To address these challenges, we propose
Brain-OF, an omnifunctional
brain foundation model jointly pretrained on fMRI,
EEG and MEG, capable of handling both unimodal and
multimodal inputs within a unified framework. To reconcile heterogeneous spatiotemporal resolutions, we introduce the Any-Resolution Neural Signal Sampler, which projects diverse
brain signals into a shared semantic space. To further manage semantic shifts, the
Brain-OF backbone integrates DINT
attention with a Sparse Mixture of Experts, where shared experts capture modality-invariant representations and routed experts specialize in modality-specific semantics. Furthermore, to explicitly internalize the characteristics of neural activity through self-supervised learning, we propose Masked Temporal-Frequency Modeling, a dual-domain pretraining objective that jointly reconstructs
brain signals in both the time and frequency domains.
Brain-OF is pretrained on a large-scale corpus comprising around 40
datasets and demonstrates superior performance across diverse downstream tasks, highlighting the benefits of joint
multimodal integration and dual-domain pretraining.
Read the full abstract ▾
All authors Hanning Guo, Hanwen Bi, Farah Abdellatif, Andrei Galbenus, Jon. N. Shah, Abigail Morrison, Jürgen Dammers
Machine Learning
How Far Can a Trainable Frequency Move? Resolution, Grids and Coupling in Trainable-FrequencyQuantum Models
Michael Poppel, Markus Baumann, Sebastian Wölckert et al. (+2 more)
Trainable-frequency
quantum models train the prefactors of their encoding gates so that the
circuit's frequencies can move onto the target's. From the unit start of prior work, training rarely reaches a target more than one resolution step away. We show that the target-driven part of the prefactor gradient is a weighted sum of slopes of the overlap $\sin u/u$, where $u$ is the model-target frequency distance scaled by the window half-width. This gives each frequency a reach: under gradient flow, an isolated real frequency whose coefficient adapts optimally reaches its target from within half a step, for every target at least one step from zero frequency; with a fixed coefficient, a phase-dependent barrier can stop it. A ternary prefactor grid places a model frequency within half a step of every band frequency, with logarithmically many encoding gates. Single-frequency
circuits trained with Adam and jointly trained amplitudes behave as predicted: inside the first side lobe, a frequency starts toward its target exactly when in phase with it in 98% of runs, and only 11 of 768 frequencies starting beyond the lobe reached their target. With several frequencies, the grid start raised successful fits on synthetic targets from 69% to 84% against the best random start. With many target frequencies, least squares on the fixed grid fits near-perfectly, but the
circuit does not reach the coefficients; training the prefactors narrows this shortfall, but with 8 and 24 target frequencies, training the fixed grid ten times longer fitted better. On real series, training the frequencies improved held-out $R^2$ at equal budget by 0.010–0.045 on four of seven series, but a classical screen did not separate these from the others. A classical model with independently trainable frequencies shows the same dependence on the start; our results describe how to train such models, not an advantage of the
circuit.
Read the full abstract ▾
All authors Michael Poppel, Markus Baumann, Sebastian Wölckert, Claudia Linnhoff-Popien, Jonas Stein
Machine Learning · Cross-listed from cs.NE
Isotropic Activation Functions Enable Deindividuated Neurons and Adaptive Topologies
George Bird
Varying the number of
neurons while preserving function is difficult because neural networks' dense connectivity creates strong computational interdependence between units. Elementwise activation functions, particularly their limited equivariance properties, are identified as a central hurdle to achieving this. We propose activation functions designed around chosen equivariances, enabling dense neural networks to adapt via singular value decomposition of their affine maps. This diagonalises layers into one-to-one, ordered connections, making it easier to assess the impact of individual connections on function. Low-impact
neurons can be removed (neurodegeneration), and a thresholded buffer of largely inactive `scaffold'
neurons is maintained (neurogenesis). These
symmetry-led diagonalisation and structural changes prioritise function-invariance: neurogenesis exactly preserves function, neurodegeneration is asymptotically function-preserving with the pruned singular-value threshold. Moreover, this construction enables asymptotic 50% parameter sparsification of dense networks while preserving function analytically. Thus, the approach enables real-time restructuring of the architecture in response to task demands, task appending, removal or changes. Conceptually, this motivates broader applications of
symmetry prescriptions at the small scale, even when no data-driven model-scale
symmetry adherence is required. Following this, we build activations around
symmetry, not from classical assumptions about individual
neurons. Unlike typical elementwise forms, these alternatives feature explicit basis independence and do not individuate
neurons. We leverage this to decompose and reinterpret
neurons across sequential layers, simplifying structure and providing a natural route to adaptive
topologies. Additional developments include a new tunable parameter, the `intrinsic length'.
Read the full abstract ▾
Machine Learning
U-CAN: Utility-Aware Contrastive Attenuation for Efficient Unlearning in Generative Recommendation
Zezheng Wu, Rui Wang, Xinghe Cheng et al. (+4 more)
Generative Recommendation (GenRec) typically leverages Large
Language Models (LLMs) to redefine personalization as an instruction-driven sequence generation task. However, fine-tuning on user logs inadvertently encodes sensitive attributes into model parameters, raising critical
privacy concerns. Existing Machine Unlearning (MU) techniques struggle to navigate this tension due to the Polysemy Dilemma, where
neurons superimpose sensitive data with general
reasoning patterns, leading to catastrophic utility loss under traditional gradient or
pruning methods. To address this, we propose Utility-aware Contrastive AttenuatioN (U-CAN), a precision unlearning framework that operates on low-rank adapters. U-CAN quantifies risk by contrasting activations and focuses on
neurons with asymmetric responses that are highly sensitive to the forgetting set but suppressed on the retention set. To safeguard performance, we introduce a utility-aware calibration mechanism that combines weight magnitudes with retention-set activation norms, assigning higher utility scores to dimensions that contribute strongly to retention performance. Unlike binary
pruning, which often fragments network structure, U-CAN develop adaptive soft attenuation with a differentiable decay function to selectively down-scale high-risk parameters on LoRA adapters, suppressing sensitive
retrieval pathways and preserving the topological connectivity of
reasoning circuits. Experiments on two public
datasets across seven metrics demonstrate that U-CAN achieves strong
privacy forgetting, utility retention, and computational efficiency.
Read the full abstract ▾
All authors Zezheng Wu, Rui Wang, Xinghe Cheng, Yang Shao, Qing Yang, Jiapu Wang, Jingwei Zhang
Machine Learning · Cross-listed from q-bio.MN
Complex Networks and the Drug Repositioning Problem
Felipe Bivort Haiek
In this Master's thesis, the graph properties of a multi-level drug-
protein network are studied, as well as how the network's shape has informed discoveries over the years, identifying primarily crawling discoveries and a smaller number of hopping discoveries. Finally, the network structure is used to inform a network
diffusion recommendation system and to prioritize existing drugs for repurposing against
proteins in organisms that cause Neglected Tropical Diseases.
Read the full abstract ▾
Machine Learning
Model Agreement via Anchoring
Eric Eaton, Surbhi Goel, Marcel Hussing et al. (+4 more)
Numerous lines of aim to control $\textit{model disagreement}$ – the extent to which two machine learning models disagree in their predictions. We adopt a simple and standard notion of model disagreement in real-valued prediction problems, namely the expected squared difference in predictions between two models trained on independent samples, without any coordination of the training processes. We would like to be able to drive disagreement to zero with some natural parameter(s) of the training procedure using analyses that can be applied to existing training methodologies. We develop a simple general technique for proving bounds on independent model disagreement based on $\textit{anchoring}$ to the average of two models within the analysis. We then apply this technique to prove disagreement bounds for four commonly used machine learning
algorithms: (1) stacked aggregation over an arbitrary model class (where disagreement is driven to 0 with the number of models $k$ being stacked) (2) gradient boosting (where disagreement is driven to 0 with the number of iterations $k$) (3) neural network training with architecture search (where disagreement is driven to 0 with the size $n$ of the architecture being optimized over) and (4) regression tree training over all regression trees of fixed depth (where disagreement is driven to 0 with the depth $d$ of the tree architecture). For clarity, we work out our initial bounds in the setting of one-dimensional regression with squared error loss – but then show that all of our results generalize to multi-dimensional regression with any strongly convex loss.
Read the full abstract ▾
All authors Eric Eaton, Surbhi Goel, Marcel Hussing, Michael Kearns, Aaron Roth, Sikata Bela Sengupta, Jessica Sorrell
Machine Learning
A Dataset is Worth 1 MB
Elad Kimchi Shoshani, Leeyam Gabay and Yedid Hoshen
A
dataset server must often distribute the same large payload to many clients, incurring massive communication costs. Since clients frequently operate on diverse hardware and software frameworks, transmitting a pre-trained model is often infeasible; instead,
agents require raw data to train their own task-specific models locally. While
dataset distillation attempts to compress training signals, current methods struggle to scale to high-resolution data and rarely achieve sufficiently small files. In this paper, we propose Pseudo-Labels as Data (PLADA), a method that completely eliminates pixel transmission. We assume
agents are preloaded with a large, generic, unlabeled reference
dataset (e.g., ImageNet-1K, ImageNet-21K) and communicate a new task by transmitting only the class labels for specific images. To address the distribution mismatch between the reference and target
datasets, we introduce a
pruning mechanism that filters the reference
dataset to retain only the labels of the most semantically relevant images for the target task. This selection process simultaneously maximizes training efficiency and minimizes transmission payload. Experiments on 10 diverse
datasets demonstrate that our approach can transfer task knowledge with a payload of less than 1 MB while retaining high classification accuracy, offering a promising solution for efficient
dataset serving.
Read the full abstract ▾
Machine Learning
SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
Simon Roschmann, Paul Krzakala, Sonia Mazelet et al. (+2 more)
The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained
vision and
language models with lightweight
alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful
alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the
alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. SOTAlign effectively leverages unpaired images and text, learning robust joint
embeddings across
datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines. Code is available at https://github.com/ExplainableML/SOTAlign.
Read the full abstract ▾
All authors Simon Roschmann, Paul Krzakala, Sonia Mazelet, Quentin Bouniot, Zeynep Akata
Machine Learning
FlashOptim: Optimizers for Memory-Efficient Training
Jose Javier Gonzalez Ortiz, Abhay Gupta, Christopher Rinard et al. (+1 more)
Standard mixed-precision training of neural networks requires many bytes of accelerator
memory for each model parameter. These bytes reflect not just the parameter itself, but also its gradient and one or more optimizer state variables. With each of these values typically requiring 4 bytes, training even a 7 billion parameter model can be impractical for researchers with less than 100GB of accelerator
memory. We introduce FlashOptim, a suite of optimizations that reduces per-parameter
memory by over 50% while preserving model quality and API compatibility. Our approach introduces two key techniques. First, we improve master weight splitting by finding and exploiting a tight bound on its quantization error. Second, we design companding functions that greatly reduce the error in 8-bit optimizer state quantization. Together with 16-bit gradients, these techniques reduce AdamW
memory from 16 bytes to 7 bytes per parameter, or 5 bytes with gradient release. They also cut model checkpoint sizes by more than half. Experiments with FlashOptim applied to SGD, AdamW, and Lion show no measurable quality degradation on any task from a collection of standard
vision and
language benchmarks, including Llama-3.1-8B finetuning.
Read the full abstract ▾
All authors Jose Javier Gonzalez Ortiz, Abhay Gupta, Christopher Rinard, Davis Blalock
Machine Learning
Mean Estimation from Coarse Data: Characterizations and Efficient Algorithms
Alkis Kalavasis, Anay Mehrotra, Manolis Zampetakis et al. (+2 more)
Coarse data arise when learners observe only partial information about samples; namely, a set containing the sample rather than its exact value. This occurs naturally through measurement rounding, sensor limitations, and lag in economic systems. We study Gaussian mean estimation from coarse data, where each true sample $x$ is drawn from a $d$-dimensional Gaussian distribution with identity covariance, but is revealed only through the set of a partition containing $x$. When the coarse samples, roughly speaking, have “low” information, the mean cannot be uniquely recovered from observed samples (i.e., the problem is not identifiable). Recent work by Fotakis, Kalavasis, Kontonis, and Tzamos [FKKT21] established that sample-efficient mean estimation is possible when the unknown mean is identifiable and the partition consists of only convex sets. Moreover, they showed that without convexity, mean estimation becomes NP-hard. However, two fundamental questions remained open: (1) When is the mean identifiable under convex partitions? (2) Is computationally efficient estimation possible under identifiability and convex partitions? This work resolves both questions. [...]
Read the full abstract ▾
All authors Alkis Kalavasis, Anay Mehrotra, Manolis Zampetakis, Felix Zhou, Ziyu Zhu
Machine Learning
Differentiable Zero-One Loss via Hypersimplex Projections
Camilo Gomez, Pengyang Wang and Liansheng Tang
Recent advances in machine learning have emphasized the integration of structured optimization components into end-to-end differentiable models, enabling richer inductive biases and tighter
alignment with task-specific objectives. In this work, we introduce a novel differentiable approximation to the zero-one loss-long considered the gold standard for classification performance, yet incompatible with gradient-based optimization due to its non-differentiability. Our method constructs a smooth, order-preserving projection onto the n,k-dimensional hypersimplex through a constrained optimization framework, leading to a new operator we term Soft-Binary-Argmax. After deriving its mathematical properties, we show how its Jacobian can be efficiently computed and integrated into binary and multiclass learning systems. Empirically, our approach achieves significant improvements in generalization under large-batch training by imposing
geometric consistency constraints on the output logits, thereby narrowing the performance gap traditionally observed in large-batch training.
Read the full abstract ▾