Machine Learning
Mugi: Value Level Parallelism For Efficient LLMs
Daniel Price, Prabhu Vellaisamy, John Shen et al. (+1 more)
Value level parallelism (VLP) has been proposed to improve the efficiency of large-batch, low-precision general
matrix multiply (GEMM) between symmetric activations and weights. In
transformer based large
language models (LLMs), there exist more sophisticated operations beyond activation-weight GEMM. In this paper, we explore how VLP benefits LLMs. First, we generalize VLP for nonlinear approximations, outperforming existing nonlinear approximations in end-to-end LLM accuracy, performance, and efficiency. Our VLP approximation follows a value-centric approach, where important values are assigned with greater accuracy. Second, we optimize VLP for small-batch GEMMs with asymmetric inputs efficiently, which leverages timely LLM optimizations, including weight-only quantization, key-value (KV) cache quantization, and group query
attention. Finally, we design a new VLP architecture, Mugi, to encapsulate the innovations above and support full LLM workloads, while providing better performance, efficiency and sustainability. Our experimental results show that Mugi can offer significant improvements on throughput and energy efficiency, up to $45\times$ and $668\times$ for nonlinear softmax operations, and $2.07\times$ and $3.11\times$ for LLMs, and also decrease operational carbon for LLM operation by $1.45\times$ and
embodied carbon by $1.48\times$.
Read the full abstract ▾
All authors Daniel Price, Prabhu Vellaisamy, John Shen, Di Wu
Machine Learning
Towards Reliable ML Feature Engineering via Planning in Constrained-Topology of LLM Agents
Himanshu Thakur, Anusha Kamath, Anurag Muthyala et al. (+3 more)
Recent advances in code generation models have unlocked unprecedented opportunities for automating feature engineering, yet their adoption in real-world ML teams remains constrained by critical challenges: (i) the scarcity of
datasets capturing the iterative and complex coding processes of production-level feature engineering, (ii) limited integration and personalization of widely used coding
agents, such as CoPilot and Devin, with a team's unique tools, codebases, workflows, and practices, and (iii) suboptimal human-AI collaboration due to poorly timed or insufficient feedback. We address these challenges with a planner-guided, constrained-
topology multi-
agent framework that generates code for repositories in a multi-step fashion. The LLM-powered planner leverages a team's environment, represented as a graph, to orchestrate calls to available
agents, generate context-aware
prompts, and use downstream failures to retroactively correct upstream artifacts. It can request human intervention at critical steps, ensuring generated code is reliable, maintainable, and aligned with team expectations. On a novel in-house
dataset, our approach achieves 38% and 150% improvement in the evaluation metric over manually crafted and unplanned workflows respectively. In practice, when building features for recommendation models serving over 120 million users, our approach has delivered real-world impact by reducing feature engineering cycles from three weeks to a single day.
Read the full abstract ▾
All authors Himanshu Thakur, Anusha Kamath, Anurag Muthyala, Dhwani Sanmukhani, Smruthi Mukund, Jay Katukuri
Machine Learning
Digital Metabolism: Decoupling Logic from Facts via Regenerative Unlearning – Towards a Pure Neural Logic Core
Mengmeng Peng, Zhenyu Fang and He Sun
Large
language models (LLMs) currently suffer from parameter
entanglement, where general
reasoning capabilities (logic) and specific factual knowledge (facts) exist in a superposition state within shared weights. This coupling leads to the "
memory wall," where computational capacity is squandered on simulating
retrieval, often resulting in
hallucinations. In this paper, we propose "digital metabolism," a thermodynamic hypothesis suggesting that targeted forgetting is necessary for distilling a pure neural logic core. To validate this hypothesis, we introduce the Regenerative Logic-Core Protocol (RLCP), a dual-stream training framework that renders specific factual dependencies linearly undecodable via deep-layer gradient reversal. Applying RLCP to Qwen2.5-0.5B, we observe a distinct phase transition: the model achieves near-zero retention of targeted factual associations (Accuracy < 7%) while exhibiting changes consistent with an emergent "structural crystallization" effect. Empirical analysis on GSM8K reveals that the "metabolized" model spontaneously adopts chain-of-thought (CoT) scaffolding, which we interpret as compensating for the loss of direct associative recall (shifting from $O(1)$ recall to $O(N)$
reasoning). While the causal mechanism underlying this behavioral shift requires further investigation, our findings provide a dynamic weight-level counterpart to architectural innovations like DeepSeek's Engram, paving the way for modular "Neural CPU + Symbolic RAM" architectures.
Read the full abstract ▾
Machine Learning
Towards Tensor Network Models for Low-Latency Jet Tagging on FPGAs
Alberto Coppi, Ema Puljak, Lorenzo Borella et al. (+6 more)
We present a systematic study of Tensor Network (TN) models $\unicode{x2013}$
Matrix Product States (MPS) and Tree Tensor Networks (TTN) $\unicode{x2013}$ for real-time jet tagging in high-energy physics, with a focus on low-latency deployment on Field Programmable Gate Arrays (FPGAs). Motivated by the strict requirements of the HL-LHC Level-1 trigger system, we explore TNs as compact and interpretable alternatives to deep neural networks. Using low-level jet constituent features, our models achieve competitive performance compared to state-of-the-art deep learning classifiers. We investigate post-training quantization to enable hardware-efficient implementations without degrading classification performance or latency. The best-performing models are synthesized to estimate FPGA resource usage, latency, and
memory occupancy, demonstrating sub-microsecond latency and supporting the feasibility of online deployment in real-time trigger systems. Overall, this study highlights the potential of TN-based models for fast and resource-efficient inference in low-latency environments.
Read the full abstract ▾
All authors Alberto Coppi, Ema Puljak, Lorenzo Borella, Daniel Jaschke, Enrique Rico, Maurizio Pierini, Jacopo Pazzini, Andrea Triossi, Simone Montangero
Machine Learning
Unified Optimization of Source Weights and Transfer Quantities in Multi-Source Transfer Learning: An Asymptotic Framework
Qingyue Zhang, Chang Chu, Haohao Fu et al. (+5 more)
In multi-source transfer learning, a key challenge lies in how to appropriately differentiate and utilize heterogeneous source tasks. However, existing multi-source methods typically focus on optimizing either the source weights or the amount of transferred samples, largely neglecting their joint consideration. In this work, we propose a theoretical framework, Unified Optimization of Weights and Quantities (UOWQ), that jointly determines the optimal source weights and transfer quantities for each source task. Specifically, the framework formulates multi-source transfer learning as a parameter estimation problem based on an asymptotic analysis of a Kullback–Leibler divergence–based generalization error measure, leading to two main theoretical findings: 1) using all available source samples is always optimal when the weights are properly adjusted; 2) the optimal source weights are characterized by a principled optimization problem whose structure explicitly incorporates the Fisher information, parameter discrepancy, parameter dimensionality, and transfer quantities. Building on the theoretical results, we further propose a practical
algorithm for multi-source transfer learning, and extend it to multi-task learning settings where each task simultaneously serves as both a source and a target. Extensive experiments on real-world
benchmarks, including DomainNet and Office-Home, demonstrate that UOWQ consistently outperforms strong baselines. The results validate both the theoretical predictions and the practical effectiveness of our framework.
Read the full abstract ▾
All authors Qingyue Zhang, Chang Chu, Haohao Fu, Tianren Peng, Yanru Wu, Guanbo Huang, Yang Li, Shao-Lun Huang
Machine Learning · Cross-listed from cs.CL
LLMs for Game Theory: Entropy-Guided In-Context Learning and Adaptive CoT Reasoning
Tommaso Felice Banfi and Sashenka Gamage
We propose a novel LLM-based framework for
reasoning in discrete, game-theoretic tasks, illustrated with Tic-Tac-Toe. The method integrates in-context learning with
entropy-guided chain-of-thought (CoT)
reasoning and adaptive context
retrieval. The model dynamically adjusts both the number of retrieved examples and
reasoning paths according to
token-level uncertainty: concise
reasoning with minimal context is used when uncertainty is low, whereas higher uncertainty triggers expanded multi-path CoT exploration. Experimental evaluation against a sub-optimal algorithmic opponent shows that
entropy-aware adaptive
reasoning substantially improves decision quality, increasing the average game outcome from \(-11.6\%\) with the baseline LLM to \(+9.5\%\) with
entropy-guided adaptive
reasoning over 100 games (win = +1, tie = 0, loss = -1), while maintaining a relatively low number of LLM queries per game. Statistical validation confirms that the improvement is significant, and correlation analysis reveals a negative association between
token-level
entropy and move optimality. These findings demonstrate that uncertainty-guided adaptive
reasoning effectively enhances LLM performance in sequential decision-making environments.
Read the full abstract ▾
Machine Learning
Analytic Bijections for Smooth and Interpretable Normalizing Flows
Mathis Gerdes and Miranda C. N. Cheng
A key challenge in normalizing flows is finding expressive invertible scalar bijections. Existing approaches face trade-offs: affine transformations are smooth and analytically invertible but lack expressivity; monotonic splines offer local control but are only piecewise smooth and act on bounded domains; residual flows achieve smoothness but need numerical inversion. We introduce three families of analytic bijections that are globally smooth ($C^\infty$), defined on all of $\mathbb{R}$, and analytically invertible in closed form, combining the favorable properties of prior approaches. Beyond serving as drop-in replacements in coupling flows, where they match or exceed spline performance, we develop radial flows: a novel architecture using direct parametrization that transforms the radial coordinate while preserving angular direction. Radial flows exhibit exceptional training stability, produce geometrically interpretable transformations, and on targets with radial structure can achieve comparable quality to coupling flows with $1000\times$ fewer parameters. We provide comprehensive evaluation on 1D and 2D
benchmarks, and demonstrate applicability to higher-dimensional physics problems through experiments on $\phi^4$
lattice field theory, where our bijections outperform affine baselines and enable problem-specific designs that address mode collapse.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
Physically constrained unfolded multi-dimensional OMP for large MIMO systems
Nay Klaimi, Clément Elvira, Philippe Mary et al. (+1 more)
Sparse recovery methods are essential for channel estimation and localization in modern communication systems, but their reliability relies on accurate physical models, which are rarely perfectly known. Their computational complexity also grows rapidly with the dictionary dimensions in large
MIMO systems. In this paper, we propose MOMPnet, a novel unfolded sparse recovery framework that addresses both the reliability and complexity challenges of traditional methods. By integrating deep unfolding with data-driven dictionary learning, MOMPnet mitigates hardware impairments while preserving interpretability. Instead of a single large dictionary, multiple smaller, independent dictionaries are employed, enabling a low-complexity multidimensional Orthogonal Matching Pursuit
algorithm. The proposed unfolded network is evaluated on realistic channel data against multiple baselines, demonstrating its strong performance and potential.
Read the full abstract ▾
All authors Nay Klaimi, Clément Elvira, Philippe Mary, Luc Le Magoarou
Machine Learning
DInf-Grid: A Neural Differential Equation Solver with Differentiable Feature Grids
Navami Kairanda, Shanthika Naik, Marc Habermann et al. (+3 more)
We present a novel differentiable grid-based representation for efficiently solving differential equations (DEs). Widely used architectures for neural solvers, such as sinusoidal neural networks, are coordinate-based MLPs that are both computationally intensive and slow to train. Although grid-based alternatives for implicit representations (e.g., Instant-NGP and K-Planes) train faster by exploiting signal structure, their reliance on linear interpolation restricts their ability to compute higher-order derivatives, rendering them unsuitable for solving DEs. Our approach overcomes these limitations by combining the efficiency of feature grids with radial basis function interpolation, which is infinitely differentiable. To effectively capture high-frequency solutions and enable stable and faster computation of global gradients, we introduce a multi-resolution decomposition with co-located grids. Our proposed representation, DInf-Grid, is trained implicitly using the differential equations as loss functions, enabling accurate modelling of physical fields. We validate DInf-Grid on a variety of tasks, including the Poisson equation for image reconstruction, the Helmholtz equation for wave fields, and the Kirchhoff-Love boundary value problem for cloth simulation. Our results demonstrate a 5-20x speed-up over coordinate-based MLP-based methods, solving differential equations in seconds or minutes while maintaining comparable accuracy and compactness.
Read the full abstract ▾
All authors Navami Kairanda, Shanthika Naik, Marc Habermann, Avinash Sharma, Christian Theobalt, Vladislav Golyanik
Machine Learning
High-accuracy and dimension-free sampling with diffusions
Khashayar Gatmiry, Sitan Chen and Adil Salim
Diffusion models have shown remarkable empirical success in sampling from rich multi-modal distributions. Their inference relies on numerically solving a certain differential equation. This differential equation cannot be solved in closed form, and its resolution via discretization typically requires many small iterations to produce high-quality samples. More precisely, prior works have shown that the iteration complexity of discretization methods for
diffusion models scales polynomially in the ambient dimension and the inverse accuracy $1/\varepsilon$. In this work, we propose a new solver for
diffusion models relying on a subtle interplay between low-degree approximation and the collocation method (Lee, Song, Vempala 2018), and we prove that its iteration complexity scales polylogarithmically in $1/\varepsilon$, yielding the first “high-accuracy” guarantee for a
diffusion-based sampler that only uses (approximate) access to the scores of the data distribution. In addition, our bound does not depend explicitly on the ambient dimension; more precisely, the dimension affects the complexity of our solver through the effective radius of the support of the target distribution only.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection
Amir Mallak, Erfan Aasi, Shiva Sreeram et al. (+3 more)
Recent advances in end-to-end autonomous driving show that policies trained on patch-aligned features extracted from foundation models generalize better to Out-of-Distribution (OOD). We hypothesize that due to the self-
attention mechanism, each patch feature implicitly embeds/contains information from all other patches, represented in a different way and intensity, making these descriptors highly redundant. We quantify redundancy in such (BLIP2) features via PCA and cross-patch similarity: $90$% of variance is captured by $17/64$ principal components, and strong inter-
token correlations are pervasive. Training on such overlapping information leads the policy to overfit spurious correlations, hurting OOD robustness. We present
Stochastic-Patch-Selection (SPS), a simple yet effective approach for learning policies that are more robust, generalizable, and efficient. For every frame, SPS randomly masks a fraction of patch descriptors, not feeding them to the policy model, while preserving the spatial layout of the remaining patches. Thus, the policy is provided with different
stochastic but complete views of the (same) scene: every random subset of patches acts like a different, yet still sensible, coherent projection of the world. The policy thus bases its decisions on features that are invariant to which specific
tokens survive. Extensive experiments confirm that across all OOD scenarios, our method outperforms the state of the art (SOTA), achieving a $6.2$% average improvement and up to $20.4$% in closed-loop simulations, while being $2.4\times$ faster. We conduct ablations over masking rates and patch-feature reorganization, training and evaluating 9 systems, with 8 of them surpassing prior SOTA. Finally, we show that the same learned policy transfers to a physical, real-world car without any tuning.
Read the full abstract ▾
All authors Amir Mallak, Erfan Aasi, Shiva Sreeram, Tsun-Hsuan Wang, Daniela Rus, Alaa Maalouf
Machine Learning
Distributed Perceptron under Bounded Staleness, Partial Participation, and Noisy Communication
Keval Jain, Anant Raj, Saurav Prakash et al. (+1 more)
We study a semi-asynchronous client-server perceptron trained via iterative parameter mixing (IPM-style averaging): clients run local perceptron updates and a server forms a global model by aggregating the updates that arrive in each communication round. The setting captures three system effects in
federated and distributed deployments: (i) stale updates due to delayed model delivery and delayed application of client computations (two-sided version lag), (ii) partial participation (intermittent client availability), and (iii) imperfect communication on both downlink and uplink, modeled as effective zero-mean additive noise with bounded second moment. We introduce a server-side aggregation rule called staleness-bucket aggregation with padding that deterministically enforces a prescribed staleness profile over update ages without assuming any
stochastic model for delays or participation. Under margin separability and bounded data radius, we prove a finite-horizon expected bound on the cumulative weighted number of perceptron mistakes over a given number of server rounds: the impact of delay appears only through the mean enforced staleness, whereas communication noise contributes an additional term that grows on the order of the square root of the horizon with the total noise energy. In the noiseless case, we show how a finite expected mistake budget yields an explicit finite-round stabilization bound under a mild fresh-participation condition.
Read the full abstract ▾
All authors Keval Jain, Anant Raj, Saurav Prakash, Girish Varma
Machine Learning
Communication-Efficient and Privacy-Adaptable Mechanism – a Federated Learning Scheme with Convergence Analysis
Chun Hei Michael Shiu and Chih Wei Ling
Federated learning enables multiple parties to jointly train learning models without sharing their own underlying data, offering a practical pathway to
privacy-preserving collaboration under data-governance constraints. Continued study of
federated learning is essential to address key challenges in it, including communication efficiency and
privacy protection between parties. A recent line of work introduced a novel approach called the Communication-Efficient and
Privacy-Adaptable Mechanism (CEPAM), which achieves both objectives simultaneously. CEPAM leverages the rejection-sampled universal quantizer (RSUQ), a randomized vector quantizer whose quantization error is equivalent to a prescribed noise, which can be tuned to customize
privacy protection between parties. In this work, we theoretically analyze the
privacy guarantees and convergence properties of CEPAM. Moreover, we assess CEPAM's utility performance through experimental evaluations, including convergence profiles compared with other baselines, and accuracy-
privacy trade-offs between different parties.
Read the full abstract ▾
Machine Learning
Data-driven stochastic reduced-order modeling of parametrized dynamical systems
Andrew F. Ilersich, Kevin Course and Prasanth B. Nair
Modeling complex dynamical systems under varying conditions is computationally intensive, often rendering high-fidelity simulations intractable. Although reduced-order models (ROMs) offer a promising solution, current methods often struggle with
stochastic dynamics and fail to quantify prediction uncertainty, limiting their utility in robust decision-making contexts. To address these challenges, we introduce a data-driven framework for learning continuous-time
stochastic ROMs that generalize across parameter spaces and forcing conditions. Our approach, based on amortized
stochastic variational inference, leverages a reparametrization trick for Markov Gaussian processes to eliminate the need for computationally expensive forward solvers during training. This enables us to jointly learn a probabilistic autoencoder and
stochastic differential equations governing the latent dynamics, at a computational cost that is independent of the
dataset size and system stiffness. Additionally, our approach offers the flexibility of incorporating physics-informed priors if available. Numerical studies are presented for three challenging test problems, where we demonstrate excellent generalization to unseen parameter combinations and forcings, and significant efficiency gains compared to existing approaches.
Read the full abstract ▾
Machine Learning
On the origin of neural scaling laws: from random graphs to natural language
Maissam Barkeshli, Alberto Alfarano and Andrey Gromov
Scaling laws have played a major role in the modern AI revolution, providing practitioners predictive power over how the model performance will improve with increasing data, compute, and number of model parameters. This has spurred an intense interest in the origin of neural scaling laws, with a common suggestion being that they arise from power law structure already present in the data. In this paper we study scaling laws for
transformers trained to predict random walks (bigrams) on graphs with tunable complexity. We demonstrate that this simplified setting already gives rise to neural scaling laws even in the absence of power law structure in the data correlations. We further consider dialing down the complexity of natural
language systematically, by training on sequences sampled from increasingly simplified generative
language models, from 4,2,1-layer
transformer language models down to
language bigrams, revealing a monotonic
evolution of the scaling exponents. Our results also include scaling laws obtained from training on random walks on random graphs drawn from Erdös-Renyi and scale-free Barabási-Albert ensembles. Finally, we revisit conventional scaling laws for
language modeling, demonstrating that several essential results can be reproduced using 2 layer
transformers with context length of 50, provide a critical analysis of various fits used in prior literature, demonstrate an alternative method for obtaining compute optimal curves as compared with current practice in published literature, and provide preliminary evidence that maximal update parameterization may be more parameter efficient than standard parameterization.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Are Your Reasoning Models Reasoning or Guessing? A Mechanistic Analysis of Hierarchical Reasoning Models
Zirui Ren and Ziming Liu
Hierarchical
reasoning model (HRM) achieves extraordinary performance on various
reasoning tasks, significantly outperforming large
language model-based reasoners. To understand the strengths and potential failure modes of HRM, we conduct a mechanistic study on its
reasoning patterns and find three surprising facts: (a) Failure of extremely simple puzzles, e.g., HRM can fail on a puzzle with only one unknown
cell. We attribute this failure to the violation of the fixed point property, a fundamental assumption of HRM. (b) "Grokking" dynamics in
reasoning steps, i.e., the answer is not improved uniformly, but instead there is a critical
reasoning step that suddenly makes the answer correct; (c) Existence of multiple fixed points. HRM "guesses" the first fixed point, which could be incorrect, and gets trapped there for a while or forever. All facts imply that HRM appears to be "guessing" instead of "
reasoning". Leveraging this "guessing" picture, we propose three strategies to scale HRM's guesses: data augmentation (scaling the quality of guesses), input perturbation (scaling the number of guesses by leveraging inference randomness), and model bootstrapping (scaling the number of guesses by leveraging training randomness). On the practical side, by combining all methods, we develop Augmented HRM, boosting accuracy on Sudoku-Extreme from 54.5% to 96.9%. On the scientific side, our analysis provides new insights into how
reasoning models "reason".
Read the full abstract ▾
Machine Learning
Single-Stage Huffman Encoder for ML Compression
Aditya Agrawal, Albert Magyar, Hiteshwar Eswaraiah et al. (+5 more)
Training and serving Large
Language Models (LLMs) require partitioning data across multiple accelerators, where collective operations are frequently bottlenecked by network bandwidth. Lossless compression using Huffman codes is an effective way to alleviate the issue, however, its three-stage design requiring on-the-fly frequency analysis, codebook generation and transmission of codebook along with data introduces computational, latency and data overheads which are prohibitive for latency-sensitive scenarios such as die-to-die communication. This paper proposes a single-stage Huffman encoder that eliminates these overheads by using fixed codebooks derived from the average probability distribution of previous data batches. Through our analysis of the Gemma 2B model, we demonstrate that tensors exhibit high statistical similarity across layers and shards. Using this approach we achieve compression within 0.5% of per-shard Huffman coding and within 1% of the ideal Shannon compressibility, enabling efficient on-the-fly compression.
Read the full abstract ▾
All authors Aditya Agrawal, Albert Magyar, Hiteshwar Eswaraiah, Patrick Sheridan, Pradeep Janedula, Ravi Krishnan Venkatesan, Krishna Nair, Ravi Iyer
Machine Learning · Cross-listed from cs.NE
PACEvolve: Enabling Progress-Aware Consistent Evolution
Minghao Yan, Bo Peng, Benjamin Coleman et al. (+13 more)
Self-evolving
agents powered by Large
Language Models (LLMs) have emerged as a promising direction across diverse domains, including code optimization and scientific discovery, yet their core failure modes remain underexplored. Through a comprehensive empirical study, we identify that the model's
reasoning becomes anchored to the local context of current hypotheses, overemphasizing low-level details while neglecting the broader search landscape. As a result, such
agents become prone to context pollution and mode collapse, repeatedly revisiting flawed hypotheses and converging on suboptimal solutions. To address this challenge, we propose Progress-Aware Consistent
Evolution (PACEvolve), a systematic framework for governing
agent memory and search dynamics. PACEvolve overcomes these limitations through three key techniques: (1) Hierarchical Context Management (HCM), which structures historical trajectories while dynamically
pruning branches to preserve a high-signal
memory state; (2) Momentum-Based Backtracking (MBB), which monitors optimization progress to escape local minima; and (3) a self-adaptive Collaborative
Evolution policy (CE) that balances intra-trajectory refinement with inter-trajectory knowledge transfer. By decoupling high-level idea generation from low-level code evaluation, PACEvolve maintains a global view of search momentum and achieves state-of-the-art results across complex evolutionary
benchmarks.
Read the full abstract ▾
All authors Minghao Yan, Bo Peng, Benjamin Coleman, Ziqi Chen, Zhouhang Xie, Shuo Chen, Zhankui He, Noveen Sachdeva, Isabella Ye, Weili Wang, Chi Wang, Ed H. Chi, Fernando Pereira, Wang-Cheng Kang, Derek Zhiyuan Cheng, Beidou Wang
Machine Learning · Cross-listed from stat.ME
Adjusted Similarity Measures and a Violation of Expectations
William L. Lippitt, Edward J. Bedrick and Nichole E. Carlson
Adjusted similarity measures, such as Cohen's kappa for inter-rater reliability and the adjusted Rand index used to compare clustering
algorithms, are a vital tool for comparing discrete labellings. These measures are intended to have the property of 0 expectation under a null distribution and maximum value 1 under maximal similarity to aid in interpretation. Measures are frequently adjusted with respect to the permutation distribution for historic and analytic reasons. There is currently renewed interest in considering other null models more appropriate for context, such as clustering ensembles permitting a random number of identified clusters. The purpose of this work is two – fold: (1) to generalize the study of the adjustment operator to general null models and to a more general procedure which includes statistical standardization as a special case and (2) to identify sufficient conditions for the adjustment operator to produce the intended properties, where sufficient conditions are related to whether and how observed data are incorporated into null distributions. We demonstrate how violations of the sufficient conditions may lead to substantial breakdown, such as by producing a non-positive measure under traditional adjustment rather than one with mean 0, or by producing a measure which is deterministically 0 under statistical standardization.
Read the full abstract ▾
Machine Learning
Ranajoy Sadhukhan, Sheng Cao, Harry Dong et al. (+5 more)
Fine-grained sparsity promises higher parametric capacity without proportional per-
token compute, but often suffers from training instability, load balancing, and communication overhead. We introduce STEM (Scaling
Transformers with
Embedding Modules), a static,
token-indexed approach that replaces the FFN up-projection with a layer-local
embedding lookup while keeping the gate and down-projection dense. This removes runtime routing, enables CPU offload with asynchronous prefetch, and decouples capacity from both per-
token FLOPs and cross-device communication. Empirically, STEM trains stably despite extreme sparsity. It improves downstream performance over dense baselines while reducing per-
token FLOPs and parameter accesses (eliminating roughly one-third of FFN parameters). STEM learns
embedding spaces with large angular spread which enhances its knowledge storage capacity. More interestingly, this enhanced knowledge capacity comes with better interpretability. The
token-indexed nature of STEM
embeddings allows simple ways to perform knowledge editing and knowledge injection in an interpretable manner without any intervention in the input text or additional computation. In addition, STEM strengthens long-context performance: as sequence length grows, more distinct parameters are activated, yielding practical test-time capacity scaling. Across 350M and 1B model scales, STEM delivers up to ~3–4% accuracy improvements overall, with notable gains on knowledge and
reasoning-heavy
benchmarks (ARC-Challenge, OpenBookQA, GSM8K, MMLU). Overall, STEM is an effective way of scaling parametric
memory while providing better interpretability, better training stability and improved efficiency.
Read the full abstract ▾
All authors Ranajoy Sadhukhan, Sheng Cao, Harry Dong, Changsheng Zhao, Attiano Purpura-Pontoniere, Yuandong Tian, Zechun Liu, Beidi Chen
Machine Learning · Cross-listed from stat.ML
Classification Imbalance as Transfer Learning
Eric Xia and Jason M. Klusowski
Classification imbalance arises when one class is much rarer than the other. We frame this setting as transfer learning under label (prior) shift between an imbalanced source distribution induced by the observed data and a balanced target distribution under which performance is evaluated. Within this framework, we study a family of oversampling procedures that augment the training data by generating synthetic samples from an estimated minority-class distribution to roughly balance the classes, among which the celebrated SMOTE
algorithm is a canonical example. We show that the excess risk decomposes into the rate achievable under balanced training (as if the data had been drawn from the balanced target distribution) and an additional term, the cost of transfer, which quantifies the discrepancy between the estimated and true minority-class distributions. In particular, we show that the cost of transfer for SMOTE dominates that of bootstrapping (random oversampling) in moderately high dimensions, suggesting that we should expect bootstrapping to have better performance than SMOTE in general. We corroborate these findings with experimental evidence. More broadly, our results provide guidance for choosing among augmentation strategies for imbalanced classification.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Parametric RDT approach to computational gap of symmetric binary perceptron
Mihailo Stojnic
We study potential presence of statistical-computational gaps (SCG) in symmetric binary perceptrons (SBP) via a parametric utilization of fully lifted random duality theory (fl-RDT) [96]. A structural change from decreasingly to arbitrarily ordered $c$-sequence (a key fl-RDT parametric component) is observed on the second lifting level and associated with satisfiability ($\alpha_c$) – algorithmic ($\alpha_a$) constraints density threshold change thereby suggesting a potential existence of a nonzero computational gap $SCG=\alpha_c-\alpha_a$. The second level estimate is shown to match the theoretical $\alpha_c$ whereas the $r\rightarrow \infty$ level one is proposed to correspond to $\alpha_a$. For example, for the canonical SBP ($\kappa=1$ margin) we obtain $\alpha_c\approx 1.8159$ on the second and $\alpha_a\approx 1.6021$ (with converging tendency towards $\sim 1.59$ range) on the seventh level. Our propositions remarkably well concur with recent literature: (i) in [20] local
entropy replica approach predicts $\alpha_{LE}\approx 1.58$ as the onset of clustering defragmentation (presumed driving force behind locally improving
algorithms failures); (ii) in $\alpha\rightarrow 0$ regime we obtain on the third lifting level $\kappa\approx 1.2385\sqrt{\frac{\alpha_a}{-\log\left ( \alpha_a \right ) }}$ which qualitatively matches overlap gap property (OGP) based predictions of [43] and identically matches local
entropy based predictions of [24]; (iii) $c$-sequence ordering change phenomenology mirrors the one observed in asymmetric binary perceptron (ABP) in [98] and the negative Hopfield model in [100]; and (iv) as in [98,100], we here design a CLuP based
algorithm whose practical performance closely matches proposed theoretical predictions.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.MA
Procedural Fairness in Multi-Agent Bandits
Joshua Caiata, Carter Blair and Kate Larson
In the context of multi-
agent multi-armed bandits (MA-MAB),
fairness is often reduced to outcomes: maximizing welfare, reducing inequality, or balancing utilities. However, evidence in psychology, economics, and Rawlsian theory suggests that
fairness is also about process and who gets a say in the decisions being made. We introduce procedural
fairness as equal voice and formalize it for MA-MABs as equal representation within a policy over arms, using a core-stable Nash welfare objective based on representation rather than utility. Empirical results confirm that
fairness notions based on optimizing for outcomes sacrifice equal voice and representation, while the sacrifice in outcome-based objectives (like equality and utilitarianism) is minimal under procedurally fair policies. We further prove that different
fairness notions prioritize fundamentally different and incompatible values, highlighting that
fairness requires explicit normative choices. This paper argues that procedural legitimacy deserves greater focus as a
fairness objective and provides a framework for putting procedural
fairness into practice.
Read the full abstract ▾
Machine Learning
ProbFM: Probabilistic Time Series Foundation Model with Uncertainty Decomposition
Arundeep Chinta, Lucas Vinh Tran and Jay Katukuri
Time Series Foundation Models (TSFMs) have emerged as a promising approach for zero-shot financial
forecasting, demonstrating strong transferability and data efficiency gains. However, their adoption in financial applications is hindered by fundamental limitations in uncertainty quantification: current approaches either rely on restrictive distributional assumptions, conflate different sources of uncertainty, or lack principled calibration mechanisms. While recent TSFMs employ sophisticated techniques such as mixture models, Student's t-distributions, or conformal prediction, they fail to address the core challenge of providing theoretically-grounded uncertainty decomposition. For the very first time, we present a novel
transformer-based probabilistic framework, ProbFM (probabilistic foundation model), that leverages Deep Evidential Regression (DER) to provide principled uncertainty quantification with explicit epistemic-aleatoric decomposition. Unlike existing approaches that pre-specify distributional forms or require sampling-based inference, ProbFM learns optimal uncertainty representations through higher-order evidence learning while maintaining single-pass computational efficiency. To rigorously evaluate the core DER uncertainty quantification approach independent of architectural complexity, we conduct an extensive controlled comparison study using a consistent LSTM architecture across five probabilistic methods: DER, Gaussian NLL, Student's-t NLL,
Quantile Loss, and Conformal Prediction. Evaluation on cryptocurrency return
forecasting demonstrates that DER maintains competitive
forecasting accuracy while providing explicit epistemic-aleatoric uncertainty decomposition. This work establishes both an extensible framework for principled uncertainty quantification in foundation models and empirical evidence for DER's effectiveness in financial applications.
Read the full abstract ▾
Machine Learning · Cross-listed from quant-ph
Searching for Quantum Effects in the Brain: A Bell-Type Test for Nonclassical Latent Representations in Autoencoders
I. K. Kominis, C. Xie, S. Li et al. (+2 more)
Whether neural information processing is entirely classical or involves
quantum-mechanical elements remains an open question. Here we propose a model-agnostic, information-theoretic test of nonclassicality that bypasses microscopic assumptions and instead probes the structure of neural representations themselves. Using autoencoders as a transparent model system, we introduce a Bell-type consistency test in latent space, and ask whether decoding statistics obtained under multiple readout contexts can be jointly explained by a single positive latent-variable distribution. By shifting the search for
quantum-like signatures in neural systems from microscopic dynamics to experimentally testable constraints on information processing, this work opens a new route for probing the fundamental physics of neural computation. The proposed test identifies violations of classical latent-variable consistency at the level of statistical representations, without assuming a specific underlying physical mechanism.
Read the full abstract ▾
All authors I. K. Kominis, C. Xie, S. Li, M. Skotiniotis, G. P. Tsironis
Machine Learning
Combinatorial Optimization Augmented Machine Learning
Maximilian Schiffer, Heiko Hoppe, Yue Su et al. (+2 more)
Combinatorial optimization augmented machine learning (COAML) has recently emerged as a powerful paradigm for integrating predictive models with combinatorial decision-making. By
embedding combinatorial optimization oracles into learning pipelines, COAML enables the construction of policies that are both data-driven and feasibility-preserving, bridging the traditions of machine learning, operations research, and
stochastic optimization. This paper provides a comprehensive overview of the state of the art in COAML. We introduce a unifying framework for COAML pipelines, describe their methodological building blocks, and formalize their connection to empirical cost minimization. We then develop a taxonomy of problem settings based on the form of uncertainty and decision structure. Using this taxonomy, we review algorithmic approaches for static and dynamic problems, survey applications across domains such as scheduling, vehicle routing,
stochastic programming, and reinforcement learning, and synthesize methodological contributions in terms of empirical cost minimization, imitation learning, and reinforcement learning. Finally, we identify key research frontiers. This survey aims to serve both as a tutorial introduction to the field and as a roadmap for future research at the interface of combinatorial optimization and machine learning.
Read the full abstract ▾
All authors Maximilian Schiffer, Heiko Hoppe, Yue Su, Louis Bouvier, Axel Parmentier
Machine Learning · Cross-listed from cs.AI
Generative AI collective behavior needs an interactionist paradigm
Laura Ferrarotti, Gian Maria Campedelli, Roberto Dessì et al. (+7 more)
In this article, we argue that understanding the collective behavior of
agents based on large
language models (LLMs) is an essential area of inquiry, with important implications in terms of risks and benefits, impacting us as a society at many levels. We claim that the distinctive nature of LLMs–namely, their initialization with extensive pre-trained knowledge and implicit social priors, together with their capability of adaptation through in-context learning–motivates the need for an interactionist paradigm consisting of alternative theoretical foundations, methodologies, and analytical tools, in order to systematically examine how prior knowledge and embedded values interact with social context to shape emergent phenomena in multi-
agent generative AI systems. We propose and discuss four directions that we consider crucial for the development and deployment of LLM-based collectives, focusing on theory, methods, and trans-disciplinary dialogue.
Read the full abstract ▾
All authors Laura Ferrarotti, Gian Maria Campedelli, Roberto Dessì, Andrea Baronchelli, Giovanni Iacca, Kathleen M. Carley, Alex Pentland, Joel Z. Leibo, James Evans, Bruno Lepri
Machine Learning · Cross-listed from cs.CL
Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure
Syed Naveed Mahmood, Md. Rezaur Rahman Bhuiyan, Tasfia Zaman et al. (+5 more)
Entity-level unlearning is usually evaluated by what a model says: whether it stops naming the target, refuses a query, or shifts a Truth Ratio distribution. These output-level tests, however, do not show whether a subject's internal representation has been attenuated. We introduce the Entity Representation Unlearning Framework (ERUF), a representation-aware framework that mines subject-specific activation signatures, suppresses the corresponding activation direction, and distills the behavior into LoRA parameters. Among evaluated baselines, ERUF is the only method that jointly achieves surface-level suppression, internal attenuation, and utility preservation. On TOFU forget10, ERUF achieves FQ = 0.99 and MU = 0.62, matching reported oracle utility while approaching oracle forget quality. Across most standard foundation-model settings, ERUF maintains low leakage and low internal target activation, with SMR between 0.00% and 1.10%, EL10 below 0.06, and utility drift below 3%. On Llama-3.1-8B,
adversarial entity recovery falls from 63.89% to 20.15%, while name-agnostic recovery decreases by 72.7% to 77.4%. Joint surface/internal diagnostics further reveal scale-dependent behavior in
reasoning-prior models that surface metrics alone would miss. We interpret these results as operational evidence of representation-level attenuation, not as a formal guarantee of irreversible deletion.
Read the full abstract ▾
All authors Syed Naveed Mahmood, Md. Rezaur Rahman Bhuiyan, Tasfia Zaman, Jareen Tasneem Khondaker, Md. Sameer Sakib, K. M. Shadman Wadith, Nazia Tasnim, Farig Sadeque
Machine Learning
Kolmogorov Arnold Networks and Multi-Layer Perceptrons: A Paradigm Shift in Neural Modelling
Aradhya Gaonkar, Nihal Jain, Vignesh Chougule et al. (+3 more)
The research undertakes a comprehensive comparative analysis of Kolmogorov-Arnold Networks (KAN) and Multi-Layer Perceptrons (MLP), highlighting their effectiveness in solving essential computational challenges like nonlinear function approximation, time-series prediction, and multivariate classification. Rooted in Kolmogorov's representation
theorem, KANs utilize adaptive spline-based activation functions and grid-based structures, providing a transformative approach compared to traditional neural network frameworks. Utilizing a variety of
datasets spanning mathematical function estimation (quadratic and cubic) to practical uses like predicting daily temperatures and categorizing wines, the proposed research thoroughly assesses model performance via accuracy measures like Mean Squared Error (MSE) and computational expense assessed through Floating Point Operations (FLOPs). The results indicate that KANs reliably exceed MLPs in every
benchmark, attaining higher predictive accuracy with significantly reduced computational costs. Such an outcome highlights their ability to maintain a balance between computational efficiency and accuracy, rendering them especially beneficial in resource-limited and real-time operational environments. By elucidating the architectural and functional distinctions between KANs and MLPs, the paper provides a systematic framework for selecting the most suitable neural architectures for specific tasks. Furthermore, the proposed study highlights the transformative capabilities of KANs in progressing intelligent systems, influencing their use in situations that require both interpretability and computational efficiency.
Read the full abstract ▾
All authors Aradhya Gaonkar, Nihal Jain, Vignesh Chougule, Nikhil Deshpande, Sneha Varur, Channabasappa Muttal
Machine Learning
Process-Guided Concept Bottleneck Model
Reza M. Asiyabi, SEOSAW Partnership and Steven Hancock Casey Ryan
Concept Bottleneck Models (CBMs) improve the explainability of black-box Deep Learning (DL) by introducing intermediate semantic concepts. However, standard CBMs often overlook domain-specific relationships and causal mechanisms, and their dependence on complete concept labels limits applicability in scientific domains where supervision is sparse but processes are well defined. To address this, we propose the Process-Guided Concept Bottleneck Model (PG-CBM), an extension of CBMs which constrains learning to follow domain-defined causal mechanisms through biophysically meaningful intermediate concepts. Using above ground biomass density estimation from Earth Observation data as a case study, we show that PG-CBM reduces error and bias compared to multiple
benchmarks, whilst leveraging multi-source heterogeneous training data and producing interpretable intermediate outputs. Beyond improved accuracy, PG-CBM enhances transparency, enables detection of spurious learning, and provides scientific insights, representing a step toward more trustworthy AI systems in scientific applications.
Read the full abstract ▾