Machine Learning
FFR: Forward-Forward Learning for Regression
Xinyang Liu, Xuanyu Liang, Shiqi Ding et al. (+4 more)
The Forward-Forward (FF)
algorithm offers a computationally efficient and biologically plausible alternative to backpropagation (BP) by training neural networks through purely local, layer-wise optimization. However, FF is inherently designed for classification via contrastive positive-negative sample pairs, and extending it to regression poses fundamental challenges: continuous target space lacks natural "opposites" for contrastive learning, and the standard goodness function carries no information about target magnitude or ordering. We propose FFR (Forward-Forward for Regression), to our knowledge, the first framework to extend FF to real-world regression and demonstrate competitive performance across diverse realworld
datasets. FFR introduces three key innovations: (1) an ordinal competitive goodness function that replaces contrastive pairs with competitive learning between partitioned
neuron groups under distance-aware ordinal supervision; (2) a stratified ladder architecture where shallow layers learn coarse ordinal discrimination and deeper layers refine into fine-grained regression, with multi-scale feature aggregation for inter-layer collaboration; and (3) hierarchical prediction with uncertainty estimation, where multi-scale predictors jointly provide robust predictions and a single-pass uncertainty score. Extensive experimental results show FFR recovers on average 98.5% of BP's accuracy across six real-world regression
benchmarks while reducing peak training
memory to only 27% of BP's at depth 8 and 8% at depth 32, with per-iteration time around 72% of BP's, and substantially outperforms all BP-free competitors.
Read the full abstract ▾
All authors Xinyang Liu, Xuanyu Liang, Shiqi Ding, Boyang Li, Zhiqiang Que, Jiayang Li, Guosheng Hu
Machine Learning · Cross-listed from cs.HC
DiffUNet^2: Bidirectional Conditional Diffusion for Probabilistic Scientific Spatiotemporal Modeling
Mengdi Chu, Jiaxin Yang, Angus G. Forbes et al. (+4 more)
Studying the spatiotemporal
evolution of scientific phenomena often relies on costly simulations and experiments. Machine learning-based surrogate models reduce this cost, but most are limited to deterministic forward prediction. Scientific temporal analysis often requires both forward prediction and backward inference, while temporal
evolution is not always uniquely determined, especially for the inverse problem. We introduce DiffUNet^2, a bidirectional conditional
diffusion model for probabilistic scientific temporal prediction. It supports both forward and backward prediction within a shared model. We evaluate DiffUNet^2 on four scientific temporal
datasets spanning fluid dynamics, chemical reaction dynamics, and material deformation, against deterministic and probabilistic baselines. Results show that DiffUNet^2 achieves strong predictive performance in both temporal directions and high probabilistic ensemble quality compared with existing baselines. To support practical scientific exploration beyond prediction, we further extend DiffUNet^2's generation ability with target-guided state editing, allowing states of interest to be specified and explored in both temporal directions.
Read the full abstract ▾
All authors Mengdi Chu, Jiaxin Yang, Angus G. Forbes, Nathan Debardeleben, Earl Lawrence, Ayan Biswas, Han-Wei Shen
Machine Learning
Contrastive Neural Algorithmic Reasoning for Graph Coloring
Thien Le, Tianyu Zhao and Melanie Weber
Graph coloring seeks to assigns colors to a graph's nodes so that adjacent nodes receive different colors, using as few colors as possible. Here, we study approximate $k$-coloring, where the goal is to use at most $k$ colors while minimizing the number of monochromatic edges. This problem is central to graph theory and has applications in areas such as scheduling and resource allocation. Recent unsupervised GNN approaches optimize each instance directly, precluding generalization across graph sizes and distributions. We instead propose a contrastive learning framework that learns transferable coloring geometry where the
embeddings of same-color nodes align, while adjacent nodes' representations are pushed toward distinct directions. We analyze the resulting population objective over bounded-size graphs. For unit-norm
embeddings, we show that its optima have a line-prototype structure: Representations of nodes of the same color collapse to a shared one-dimensional subspace, and edges connect orthogonal subspaces. This geometry yields stationarity conditions in the supervised setting and is preserved by projected subgradient dynamics under a balanced-coloring assumption. In an unnormalized variant, gradient descent has a max-margin bias governed by a quotient-graph hard-margin problem. Experiments on synthetic and real-world graphs show that contrastive GNN encoders generalize effectively and produce low-conflict colorings, matching and sometimes improving on greedy approaches.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SI
Thomas Maillart, Thibaut Chataing, David Dosu et al. (+3 more)
Understanding and anticipating scientific change requires models that distinguish between endogenous consolidation and exogenous
diffusion of scientific concepts. Using the
quantum computing subtree of concepts in OpenAlex, we construct a temporally resolved concept co-occurrence network and track each concept pair through its upstream citation lineage and downstream
diffusion. We train LightGBM models on distributional and diversity-aware features to predict four outcomes: endogenous reinforcement, exogenous
diffusion, their ratio, and
diffusion entropy. After controlling for overall publication growth of the scientific body, endogenous reinforcement proves largely unpredictable in the primary
quantum-computing
benchmark. In contrast, exogenous
diffusion and
entropy are strongly predictable ($R^2$ up to $0.78\`a) and are driven by upstream heterogeneity, citation breadth, and distributional dispersion, as shown by SHAP analyses; replications on robotics, advanced materials, and neuro implants confirm that exogenous diffusion remains the top-ranked target across fields ($R^2_test \sim 0.60-0.87$), while endogenous predictability rises markedly in neuro implants (R^2_test = 0.83), indicating that the
quantum-computing asymmetry does not generalise uniformly. Case studies reveal that sharp
entropy increases coincide with the opening of new conceptual frontiers, while
entropy collapses signal technological convergence or paradigm displacement. These results demonstrate that conceptual
diffusion is governed by stable structural regularities embedded in semantic and citation environments. By identifying early diversity-based signals of cross-domain uptake, the approach provides a scalable foundation for anticipatory scientometrics, technology foresight, and innovation-oriented policy analysis in rapidly evolving research fields.
Read the full abstract ▾
All authors Thomas Maillart, Thibaut Chataing, David Dosu, Paul Bagourd, Julian Jang-Jaccard, Alain Mermoud
Machine Learning · Cross-listed from physics.app-ph
Beyond Gradient Descent: Adam for Analog Ising Machines
Stijn Van Vooren, Guy Van der Sande and Guy Verschaffelt
As Moore's law reaches its limits, Ising machines offer a promising alternative computing approach for difficult optimization problems. However, many analog, time-continuous Ising machines rely on gradient-descent-like dynamics to find solutions, which can limit speed and robustness. We investigate whether momentum and Adam optimization can improve these systems. Since these optimizers are traditionally formulated in discrete time, we derive continuous-time versions suitable for analog, time-continuous Ising-machine dynamics. On Max-Cut
benchmarks, we find that Adam-based dynamics substantially reduce time-to-target and improve solution quality compared with gradient-descent- and momentum-based dynamics. We further introduce a first-order continuous-time approximation of Adam that is intended as a simpler starting point for future physical implementations and while performing better than the full Adam formulation in a continuous-time setting. We also study a purely algorithmic discrete-time setting, where the performance gap is reduced on easier problem instances, while the Adam-based update rule performs best on harder weighted problem instances. These results identify continuous-time Adam dynamics as a powerful design principle for analog Ising machines.
Read the full abstract ▾
Machine Learning
MAdam: Metric-Aware Multi-Objective Adam
Fengbei Liu, Rachit Saluja, Sunwoo Kwak et al. (+5 more)
Multi-objective optimization (MOO) underlies many machine learning problems, yet MOO solvers across the loss-balancing, gradient-balancing, and Pareto-based families almost universally hand their reconciled directions to Adam~\citep{kingma2015adam}. We show this coupling introduces two systematic gaps between the solver's intent and the optimizer's execution. The first is a weighting mismatch: Adam's second-moment denominator entangles the time-varying preference vector with gradient statistics, marginalizing the preference into a history average and collapsing distinct Pareto trade-offs toward a near-uniform mixture. The second is a
geometric mismatch: Adam's adaptive metric distorts the Euclidean geometry MOO solvers assume, turning aligned objectives into apparent conflicts. To resolve both jointly, we introduce MAdam (Metric-Aware Multi-Objective Adam), a drop-in wrapper that leaves both solver and optimizer unchanged. MAdam preconditions the reconciled direction by the preference-conditioned
curvature of the scalarized objective; on this whitened input, Adam's second moment collapses to identity, so the realized update is governed by the preference-conditioned metric. Across multi-task learning, Pareto-front recovery, physics-informed neural networks, and medical imaging, MAdam improves over Adam in aggregate for every solver family. Our code is available at https://github.com/lfb-1/MAdam.
Read the full abstract ▾
All authors Fengbei Liu, Rachit Saluja, Sunwoo Kwak, Ruibo Wang, Ruining Deng, Heejong Kim, Johannes C. Paetzold, Mert R. Sabuncu
Machine Learning
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Xianliang Li, Zihan Zhang, Weiyang Liu et al. (+1 more)
Muon has recently demonstrated strong empirical performance in large
language model training, but the theoretical role of momentum in
Muon remains unclear. Existing analyses of
Muon either remove momentum to study spectral updates in isolation, or retain momentum without explaining why it improves empirical performance. Our work bridges this gap by showing momentum in
Muon acts as a spectral filter. Under a structured signal-plus-perturbation gradient model, we prove that momentum suppresses perturbations while preserving the dominant signal, thereby enlarging the spectral gap between them. This enlarged gap stabilizes the singular subspaces of the
matrix passed to
Muon's orthogonalization step, making the resulting update more reliable. We further show that applying momentum before orthogonalization achieves provably stronger
alignment with the signal component of the gradient than either reversing this order or simply removing momentum. Experiments across diverse tasks, including LLM pretraining, support our theoretical analysis. More broadly, our theory offers a starting point for understanding the benefits of momentum in other
matrix-based optimizers.
Read the full abstract ▾
All authors Xianliang Li, Zihan Zhang, Weiyang Liu, Han Bao
Machine Learning · Cross-listed from cs.CL
Synthesize and Reward – Reinforcement Learning for Multi-Step Tool Use in Live Environments
Ibrahim Abdelaziz, Asim Munawar, Kinjal Basu et al. (+4 more)
Training LLMs to orchestrate multi-step tool calls is held back by three coupled obstacles: realistic stateful execution environments are costly to build, synthetic training queries are often detached from the server's actual state (so the generated tool calls fail to execute), and recall-based RL rewards incentivize verbose tool-calling patterns. We present PROVE (Programmatic Rewards On Verified Environments), a framework with three contributions: (1) a library of 20 stateful MCP (Model Context Protocol) servers exposing 343 tools, enabling live-execution RL training with session-scoped state isolation; (2) a state-machine data synthesis pipeline that generates multi-turn tool-call trajectories grounded in live-sampled server state, so generated queries reference entities that actually exist; and (3) a multi-component programmatic reward with an adaptive efficiency penalty that counters the verbosity incentive of recall-based rewards. We train four models (Qwen3-4B, Qwen3-8B, Qwen2.5-7B, Granite-4.1-8B) with GRPO on the resulting ~13K training examples. On BFCL Multi-Turn, tau2-bench, and T-Eval, PROVE yields improvements of up to +10.2, +6.8, and +6.5 points respectively, demonstrating that this framework yields consistent gains on multi-step tool orchestration across two model families.
Read the full abstract ▾
All authors Ibrahim Abdelaziz, Asim Munawar, Kinjal Basu, Maxwell Crouse, Chulaka Gunasekara, Suneet Katrekar, Pavan Kapanipathi
Machine Learning · Cross-listed from cs.CV
CoralBay: A Self-Supervised CT Foundation Model
Ioannis Gatopoulos, Nicolas Känzig, Sebastian Otálora et al. (+1 more)
Self-supervised learning has enabled large-scale pre-training on 2D natural images, producing general-purpose visual representations that transfer effectively across tasks. However, many medical imaging modalities, such as CT scans, are inherently three-dimensional and differ fundamentally from natural images in both structure and semantics. Volumetric modalities capture spatial continuity, organ anatomy, and intensity-based tissue properties (e.g., Hounsfield Units), which are not adequately modeled by 2D pre-training. To bridge this gap, we introduce CoralBay, a self-
distillation framework that extends DINO by using a hierarchical 3D Swin backbone and applying self-
distillation to concatenated multi-scale features, enabling data-efficient self-supervised learning of rich spatial representations that encode both global semantics and fine-grained local structure. As a result, CoralBay transfers effectively to a wide range of downstream radiological tasks, demonstrating strong and consistent performance across diverse anatomical targets. In addition, we contribute to the open-source \eva framework by introducing a public, reproducible 3D radiology leaderboard that unifies multiple
datasets and establishes a standardized
benchmark for evaluating volumetric representation learning methods.
Read the full abstract ▾
All authors Ioannis Gatopoulos, Nicolas Känzig, Sebastian Otálora, Fei Tang
Machine Learning
Attribution via Distributional Paths for Information Revelation
Kieran A. Murphy and Shameen Shrestha
Feature attribution methods explain predictions by assigning importance scores to input features. Path-based methods such as Integrated Gradients are especially appealing because they satisfy completeness: attributions sum to the change in model output between a reference state and the input. Yet most path methods define this trajectory in input space, explaining a model through pointwise perturbed inputs along a chosen path. An input-space path integrates the model's raw response at each point it passes through, with no control over the resolution at which a feature is queried; the early, baseline-adjacent part of the trajectory contributes to the explanation on equal footing with the input itself. Here, we lift path attribution from input space to a space of structured probe distributions around the example of interest, and call our method Reveal-IG. Rather than traversing raw input values, Reveal-IG progressively reveals information about the input and attributes changes in the model's expected output along this distributional path. The result is a path-attribution framework that retains completeness with respect to the expected model response, and naturally accommodates multiscale image probes and feature-wise uncertainty in tabular data. Synthetic diagnostics show that Reveal-IG avoids path artifacts that affect input-space methods, and across ImageNet classification and tabular regression it produces stable, signed attributions – leading on metrics that use attribution sign while remaining competitive on the rest.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Frédéric Berdoz, Luca A. Lanzendörfer, Fabian Farestam et al. (+1 more)
Large
reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or
token count. However, identical scores on these metrics can hide fundamentally different
reasoning structures. To address this limitation, we introduce a scalable LRM
benchmark of logic puzzles and a pipeline that converts unstructured traces into verifiable
reasoning graphs of claims and dependencies. This turns
reasoning into a structured, measurable object whose
topology can be quantitatively analyzed. Building on this, we define a
reasoning efficiency metric that quantifies how concentrated the model's logical flow is. Our analysis on open-source
reasoning models shows that structural measurements separate behaviors that
token count and accuracy conflate, providing a practical tool for diagnosing failure modes and comparing how
reasoning scales with puzzle difficulty.
Read the full abstract ▾
All authors Frédéric Berdoz, Luca A. Lanzendörfer, Fabian Farestam, Roger Wattenhofer
Machine Learning · Cross-listed from stat.ML
Privacy-Robust Incrementality Measurement for Advertising Systems under Signal Loss
Prashant Shekhar and Caroline Howard
Advertising platforms use randomized lift tests to measure incrementality, but
privacy-preserving reporting systems degrade the observed signal through match-rate loss, linkability loss, attribution-window loss, aggregation-threshold suppression, randomized reporting noise, and segment-heterogeneous signal loss. This paper formulates
privacy-constrained advertising measurement as a robust causal decision problem under the mentioned signal losses. Given a randomized experiment and an ambiguity set for
privacy-induced degradation, the framework projects the observation-compatible fiber of clean/unfiltered experimental worlds onto the incrementality functional and returns certified, rejected, and unresolved decisions. The main result gives a sharp decision frontier. Reports outside the frontier support uniformly valid certification or rejection, whereas reports inside it contain too little information for any method to uniformly distinguish above-threshold incrementality from non-incrementality. Supporting results give finite-sample certification, sample-complexity guarantees, a minimax lower bound showing that signal loss reduces effective information, and a reporting-granularity tradeoff. On 2.0M Criteo Uplift rows and the 64K-row Hillstrom email experiment, clean conversion lift is positive in both
datasets, with lifts 0.00112 and 0.00495, respectively. Population certification survives mild degradation in Criteo and severe degradation in Hillstrom, while all considered finite-sample stress settings in both
datasets remain unresolved after simultaneous uncertainty and reporting noise are included. Overall, the research contributes a decision-theoretic layer for
privacy-aware incrementality measurement whose output is the strongest causal-claim justified by degraded ads signals.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Visual Instruction Tuning Aligns Modalities through Abstraction
Luis Palacios, Lorenzo Basile, Diego Doimo et al. (+1 more)
Visual instruction tuning effectively adapts a pre-trained Large
Language Model (LLM) to process image information alongside text. Yet, it remains unclear how visual features are embedded into the layer-wise hierarchy of abstractions of the LLM backbone. Across a diverse set of
vision-
language architectures, we show that instruction tuning primarily serves as a bridge,
embedding visual features directly into the intermediate semantic layers of the LLM, bypassing the early layers devoted to unimodal processing. With probing analyses and causal interventions, we show that these intermediate layers are the semantic core of
vision-
language processing and play a critical role in the performance on a broad set of
multimodal benchmarks. In addition, by comparing the geometry of semantically equivalent visual and textual representations, we find that fine-tuning extends and strengthens the existing abstraction phase, aligning visual features with pre-existing textual ones. Finally, we confirm the functional role of this localized
alignment by restricting fine-tuning to intermediate layers alone: this strategy preserves the performance of full fine-tuning on
vision-centric
benchmarks while reducing training time. Our results suggest that
multimodal integration is a localized phenomenon driven by the repurposing of the internal abstraction engine of the LLM.
Read the full abstract ▾
All authors Luis Palacios, Lorenzo Basile, Diego Doimo, Alberto Cazzaniga
Machine Learning · Cross-listed from cs.SI
Explainable Forecasting of Scientific Breakthroughs from Concept Network Dynamics
Thomas Maillart, Thibaut Chataing, Ntorina Antoni et al. (+4 more)
We introduce an explainable machine-learning approach that forecasts the structural precursors of scientific breakthroughs – the emergence and intensification of links between research concepts – by modelling how OpenAlex concept networks evolve over time. Using 59 semantic and topological features, a two-stage LightGBM model jointly predicts the formation and the future weight of concept pairs, adding a regression stage that quantifies expected intensity to prior link-existence forecasts. Relative to the state of the art, the approach improves accuracy and explainability at once: comparative validation across four technology and biomedical domains yields ROC-AUC in [0.954, 0.967] at all horizons without re-tuning, exceeding the roughly 0.90 of prior models, while every forecast rests on structural, auditable features rather than opaque
embeddings. Classification performance is high (AUC about 0.95) and regression remains stable (RMSLE 0.45 to 0.6 over one to five years). Feature attribution shows that structural factors – particularly Adamic-Adar similarity and degree-based Hadamard measures – consistently drive accuracy, suggesting that breakthrough-relevant recombinations emerge in tightly connected sub-networks. Two expert-anchored cases,
quantum annealing and AI-enabled
quantum architectures, show the model surfacing technological convergence consistent with expert expectations. We then outline a three-layer decision architecture – detection, expert translation, institutional integration – that turns these forecasts into evidence-based research strategy and policy, anchored in open data and explainable features.
Read the full abstract ▾
All authors Thomas Maillart, Thibaut Chataing, Ntorina Antoni, David Dosu, Paul Bagourd, Julian Jang-Jaccard, Alain Mermoud
Machine Learning
The Price of Partial Information with Switching Costs: A Complete Characterization for Feedback Graphs
Nicolò Cesa-Bianchi, Tommaso Cesari, Roberto Colomboni et al. (+3 more)
We study how switching costs affect minimax regret under partial feedback. Which features of the feedback preserve the $\sqrt T$ rate, and which force the $T^{2/3}$ rate? We address this question in the framework of feedback graphs, which interpolate between full information and bandits: playing an action reveals the losses of its out-neighbors in a known directed graph. Prior work on feedback graphs with switching costs gave
algorithms with regret of order $T^{2/3}$, but matching lower bounds only for restricted classes of graphs, leaving the question open. We establish a complete characterization of the minimax regret with switching costs for all feedback graphs: $\widetilde\Theta(\sqrt T)$ if the graph is strongly revealing, meaning that every pair of distinct actions has an endpoint that observes the losses of both; $\widetilde\Theta(\gamma^{1/3}T^{2/3})$ for every other observable graph, where $\gamma$ is the domination number; and linear otherwise. In particular, tournaments, in which every action observes its own loss and each pair of actions is joined by one directed edge, sit at the phase transition: they behave like full information, but deleting the edge of any pair forces the rate $T^{2/3}$. Hence an action that observes all losses is neither necessary nor sufficient for the rate $\sqrt T$. Our
algorithm TournamentSteering for the strongly revealing case rests on a new principle: each observed comparison of two actions serves both to estimate losses and to steer play, and the loss reduction from steering pays for the variance of the estimates. For every strongly revealing graph, distributing the comparisons among observing actions keeps this variance, and hence the regret, polylogarithmic in the number of actions, by an argument similar to the greedy construction of logarithmic dominating sets in tournaments.
Read the full abstract ▾
All authors Nicolò Cesa-Bianchi, Tommaso Cesari, Roberto Colomboni, Emmanuel Esposito, Federico Fusco, Tomer Koren
Machine Learning · Cross-listed from cs.CL
Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models
Qi Cao, Takeshi Kojima, Andrew Gambardella et al. (+3 more)
Large
language models (LLMs) demonstrate remarkable performance across diverse tasks, but they often generate responses that appear plausible while being factually incorrect. This problem is compounded by the lack of explicit uncertainty estimates, which makes it difficult for users to judge the reliability of model outputs. Existing uncertainty quantification methods typically rely on indirect signals, such as
entropy across sampled generations. These signals can be difficult to interpret and do not fully leverage the model's ability to assess its own uncertainty. We propose a simple yet effective self-assessment method for uncertainty quantification in LLMs. Our approach groups sampled generations into semantically distinct clusters, converts them into answer options in a structured multiple-choice question, and uses the probability assigned by the LLM to each option as a confidence estimate. Experiments across multiple models and
datasets show that our method consistently outperforms baseline approaches. Notably, it achieves competitive performance with as few as two additional samples, demonstrating both its effectiveness and efficiency.
Read the full abstract ▾
All authors Qi Cao, Takeshi Kojima, Andrew Gambardella, Helinyi Peng, Yutaka Matsuo, Yusuke Iwasawa
Machine Learning
Re-Evaluating Continual Learning with Few-Shot Adaptation
Amogh Inamdar, Matthew So, Vici Milenia et al. (+1 more)
Continual learning methods aim to maximize the stability and plasticity of machine learning models that are trained on a sequence of tasks. The standard measure of stability (i.e., forgetting) is the 0-shot performance of a model on previously learned tasks, and plasticity, the performance on the most recently learned task. However, 0-shot evaluation does not fully measure a model or method's ability to retain learned information or adapt quickly to new information, as it requires perfect recall across multiple tasks. In this paper, we propose few-shot evaluation as a more comprehensive assessment of the stability and plasticity of a continual learning system. We conduct a fine-grained assessment on task sequences for continual image classification and find that this paradigm produces novel insights into the performance of popular continual learning strategies. Through few-shot evaluation with a novel metric – per-shot plasticity – we show that adding `foresight' to continual learning methods via the meta-learning of a short sequence of future tasks induces learning-to-learn behavior over the task sequence.
Read the full abstract ▾
All authors Amogh Inamdar, Matthew So, Vici Milenia, Richard Zemel
Machine Learning
Text-attributed Graph Condensation via Text Selection and Attribute Matching
Haowei Han, Yuxiang Wang, Guojia Wan et al. (+5 more)
Text-Attributed Graph (TAG) is an important type of graph structured data, where each node has a text description. TAG models usually train a Graph Neural Network (GNN) and
language model jointly, which leads to high space and time consumption, especially on large
datasets. To mitigate this, we propose TAGSAM, a condensation method that compresses TAGs while preserving training accuracy. TAGSAM comes with two key designs, i.e., subgraph text Selection and Attribute similarity Matching, which compress the text description and graph
topology of TAG, respectively. For the texts, subgraph text selection selects and merges representative text chunks from multiple related text descriptions by maximizing mutual information. For the graph
topology, popular condensation methods based on Matching Training Trajectories (MTT) suffer from high variance, which hinders accuracy. Our attribute similarity matching mitigates this issue by aligning stable similarity
matrices. We evaluate TAGSAM against six state-of-the-art baselines, where it showcases superior performance. For the same compressed size, TAGSAM improves upon the best-performing baseline by an average of 4.9% in accuracy. Furthermore, it maintains competitive training accuracy even when the TAG is condensed to just 1% size. Our code is available at https://github.com/SundayVHan/TAGSAM
Read the full abstract ▾
All authors Haowei Han, Yuxiang Wang, Guojia Wan, Hao Wang, Shanshan Feng, Hao Huang, Jiawei Jiang, Xiao Yan
Machine Learning
Online Learning with Gradient-Variation Interval Regret
Yan-Feng Xie, Shuche Wang, Peng Zhao et al. (+1 more)
This paper investigates non-stationary online learning using the metric of interval regret, which requires an online
algorithm to perform well over every time interval. We propose the first online learning
algorithm that achieves an interval regret bound scaling with gradient variation, a fundamental measure of the cumulative change in online function gradients, which relates to various problem-dependent quantities and is closely connected to
stochastic optimization and other problems. Our method employs a simple and efficient two-layer online ensemble structure that achieves strong theoretical guarantees. Specifically, it enjoys a regret bound that simultaneously adapts to various problem-dependent quantities while also preserving the minimax-optimal rate in the worst case. Moreover, recognizing the challenge of hyperparameter tuning, we introduce a Lipschitz- and smoothness-agnostic variant that automatically adapts to these potentially unknown constants. This is primarily enabled by a novel Lipschitz-adaptive meta
algorithm, which may be of independent interest. Beyond interval regret, our method also yields broader implications: it provides versatile bounds for interval dynamic regret, a stronger measure that competes with changing comparators over any interval, and yields the first piecewise characterization for
stochastic extended
adversarial optimization. Theoretical findings are validated by experiments.
Read the full abstract ▾
All authors Yan-Feng Xie, Shuche Wang, Peng Zhao, Zhi-Hua Zhou
Machine Learning
Dynamic Short Convolutions Improve Transformers
Oliver Sieberling, Bharat Runwal, Rameswar Panda et al. (+1 more)
Transformers have become the dominant architecture for large
language models, largely due to the scalability and flexibility of
attention, feed-forward layers, residual connections, and normalization. This paper introduces dynamic short convolutions as an additional neural network primitive for improving
Transformers. Unlike static short convolutions, dynamic convolutions use input-dependent filters, which preserves the locality bias of convolution while increasing expressivity. Motivating experiments show that applying dynamic short convolutions to key, query, and value representations improves performance on challenging associative recall tasks compared with static convolutional variants. Across
language-modeling experiments ranging from 150M to 2B parameters, dynamic convolutions consistently outperform standard
Transformers and
Transformers augmented with static short convolutions. Fitting scaling laws indicates a 1.33$\times$ compute advantage over compute-matched
Transformers when dynamic convolutions are applied to the key, query, and value vectors, and a 1.60$\times$ advantage when adding dynamic convolutions after every linear layer. Dynamic convolutions also offer improvements on linear RNNs (Mamba-2/Gated DeltaNet) and mixture-of-experts architectures. We make these gains practical with custom Triton kernels that enable efficient training with a manageable end-to-end slowdown. These results suggest that dynamic short convolutions are a scalable, hardware-efficient, and expressive primitive for advancing
Transformer-based
language models.
Read the full abstract ▾
All authors Oliver Sieberling, Bharat Runwal, Rameswar Panda, Yoon Kim
Machine Learning
Finding Needles in the Haystack: Transductive Active Labeling in Ecology
Rupa Kurinchi-Vendhan and Sara Beery
Active learning is now standard practice in labeling ecological data, enabling ecologists to quickly process large volumes of field data to understand and monitor natural environments. Current practices evaluate active learning inductively, estimating predictive performance on a held-out test set. We argue that this evaluation is misaligned with most ecological tasks, where the goal is to transductively label an entire pool of data as efficiently as possible. We demonstrate that ignoring the human-in-the-loop underestimates the importance of continuing to label, particularly for classes in the long tail which may be of disproportionate ecological importance (rare species, uncommon behaviors, etc.). Our analysis shows that, for this long tail, the transductive objective shifts importance from prediction to discovery: the true challenge becomes finding "needles in the haystack," examples of rare classes that are embedded within dense regions of abundant classes in the latent geometry, which we quantify with a novel metric of sampling difficulty. Finally, to translate these insights to practical ecological workflows, we propose a conservative hybrid stopping criterion inspired by ecological rarefaction curves, and show that combining predictive performance with discovery criteria reduces premature stopping on long-tailed pools, improving rare-class recovery when discovery, not classification, is the limiting factor.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
A Quantitative Approximation Framework for Flow Distillation in Diffusion Models
Weiguo Gao, Ming Li, Lei Shi et al. (+1 more)
We develop a quantitative framework for
diffusion distillation by viewing few step sampling as approximation through compositions of learned flow maps. For trajectory
distillation of the probability flow ODE, we show that low noise
multimodal regimes separate score approximability from dynamical stability: the score remains efficiently approximable, while small local errors may be strongly amplified by stiff flow dynamics. In a Gaussian mixture Ornstein–Uhlenbeck model, we prove time uniform \(L^p(p_t)\) score approximation by ReLU and ReQU networks with explicit polylogarithmic complexity, and derive a computable Lipschitz bound \(L(t)\) for the flow velocity. The stability factor \(\exp\bigl(\int_s^t L(u)\mathrm du\bigr)\) can grow exponentially as noise decreases and mixture separation increases. Comparing this certificate with a certified local Lipschitz budget for one step students identifies regimes of direct
distillation difficulty, without implying an approximation lower bound. We also show that deep residual compositions control global transport error through propagated local errors, and that equalizing cumulative stability yields an optimal nonuniform
segmentation. With eight segments, this grid reduces final mean relative MSE by up to \(51.9\%\) versus uniform grids.
Read the full abstract ▾
All authors Weiguo Gao, Ming Li, Lei Shi, Hanfei Zhou
Machine Learning
TreeFlash: Parallel AR-Approximation for Faster Speculative Decoding
Peer Rheinboldt, Frédéric Berdoz and Roger Wattenhofer
One-shot block drafters for speculative decoding generate the full draft in a single forward pass, achieving strong throughput by eliminating sequential
token generation. However, they predict each draft
token conditioned only on the prefix context, with no dependence on previously drafted
tokens. This non-autoregressive conditioning causes the drafter's distribution to diverge from the verifier's true autoregressive distribution as draft depth grows. This problem becomes more severe in tree-based drafting, where distinct branches are forced to share the same marginal distribution for subsequent
tokens. We propose TreeFlash, which addresses this by incorporating an MLP layer conditioned on the drafter's hidden state and the previous
token to approximate an autoregressive distribution. TreeFlash retains the $\mathcal{O}(1)$ decoding time complexity of one-shot drafters by employing a two-stage approximation mechanism. TreeFlash achieves state-of-the-art performance across a variety of tasks and models, improving over marginal tree drafting by $12\%$ higher block efficiency and $9\%$ higher speedup.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CR
AI Agents Enable Adaptive Computer Worms
Jonas Guan, Tom Blanchard, Hanna Foerster et al. (+3 more)
A computer worm is malware that spreads on a network by replicating itself from one machine to another. Traditional worms, like WannaCry, exploited predetermined vulnerabilities, and their spread can be halted by patching those vulnerabilities. Here we show that artificial intelligence (AI)
agents enable a fundamentally new threat: a worm that generates tailored attack strategies to each target it encounters. The worm parasitically uses compromised machines to run open-weight large
language models (LLMs) to sustain its
reasoning, or extend its reach for further attacks. Deployed on a network of machines spanning Linux, Windows, and IoT (Internet of Things) devices, the worm propagated by exploiting common, real-world corporate network vulnerabilities. Since the worm is powered by stolen compute, the attacker's marginal cost per new infection is zero. This creates a destabilizing economic asymmetry between attackers and defenders. Moreover, because the worm requires no commercial AI platform, centralized safety controls, such as service refusals or rate limiting, are structurally irrelevant. Our results demonstrate that self-sustaining AI-driven cyber-threats are no longer theoretical. We must prepare for autonomous generative adversaries: malware systems that propagate without human operators and are defined not by fixed exploit code, but by the capacity to reason about targets, adapt to observations, and synthesize attack logic in real time.
Read the full abstract ▾
All authors Jonas Guan, Tom Blanchard, Hanna Foerster, Hengrui Jia, Gabriel Huang, Nicolas Papernot
Machine Learning
PURGE: Projected Unlearning via Retain-Guided Erasure
Vedant Jawandhia, Daksh Ahuja, Ghufran Alam Siddiqui et al. (+3 more)
We propose PURGE, a machine unlearning
algorithm built on a simple but an under-exploited observation: continual learning (CL) and machine unlearning (MU) which are fundamentally dual problems. CL tries to learn new tasks without forgetting old ones; MU tries to erase specific data without hurting retained performance representing the same underlying tension in opposite directions. PURGE leverages this duality by adapting gradient projection from A-GEM (Chaudhry et al., 2019) so that every unlearning step is constrained to not increase the retain-set loss. On top of this, it performs multi-layer representation erasure, pushing forget-set activations in intermediate layers towards the retain distribution to remove information from hidden representations rather than just suppressing it at the output. A key design choice is the retain-confusion target: rather than pushing forget outputs toward the uniform distribution, which we found to be surprisingly easy for membership inference attacks to detect, we instead target the model's natural confusion pattern on retain data. This makes the unlearned model hard to distinguish from one retrained from scratch. Two self-regulating stopping criteria (a retain-loss budget and a forget-accuracy target) let the
algorithm decide on its own when to stop, removing the need for manual epoch tuning. In experiments on five
datasets (CIFAR-10, MNIST, SVHN, STL10, PathMNIST) across 22 class-level forgetting tasks, PURGE consistently keeps retain accuracy above 96% while achieving MIA AUROC close to 0.5 (the ideal), outperforming gradient ascent, KL-uniform, and several published baselines on the
privacy-utility frontier.
Read the full abstract ▾
All authors Vedant Jawandhia, Daksh Ahuja, Ghufran Alam Siddiqui, Prashant Trivedi, Yash Sinha, Pratik Narang
Machine Learning
Easy-to-Use Shielding for Reinforcement Learning
Stefan Pranger and Bettina Könighofer
Safe exploration is a key challenge in Reinforcement Learning (RL) that aims to prevent
agents from making harmful decisions while exploring their environment. Safe exploration is a key challenge in Reinforcement Learning (RL) that aims to prevent
agents from making harmful decisions while exploring their environment. Shielding is one such technique that assumes domain knowledge in the form of an environment model to decide upon action safety. Although well-established, shielding has seen limited adoption in RL due to the lack of accessible end-to-end infrastructure connecting formal shield synthesis with standard RL frameworks. Applying shielding typically requires expertise in formal methods and substantial engineering effort, keeping it outside the typical RL workflow. We address this by extending our shield synthesis tool Tempest into a practical backend for safe RL. Our core contribution is tempestpy, a Python library that integrates Tempest-based shield synthesis directly into the Gymnasium API, allowing shields to be synthesized and deployed within existing RL pipelines. This lowers the barrier to entry for shielding and turns formal safe-exploration methods into a usable component for RL practitioners. We also extend Tempest's algorithmic support to compute sound shields for
stochastic multiplayer games, preserving formal safety guarantees. We demonstrate the resulting workflow end to end and evaluate shielded and unshielded RL across multiple environments. To facilitate modeling, we provide symbolic models for MiniGrid and introduce MiniGridSafe, a collection of playground environments designed to make shielding easily accessible and experimentally transparent. MiniGridSafe extends MiniGrid with safety-oriented scenarios featuring probabilistic transitions and additional
agents, enabling the study of challenging safety aspects in a simple and intuitive setting.
Read the full abstract ▾
Machine Learning
Trading Human Curation for Synthetic Augmentation in RLVR
Akshansh, Leonardo Rosa Rodrigues, Michael Korostelev et al. (+2 more)
The supply of high-quality training tasks is a central bottleneck for reinforcement learning from verifiable rewards (RLVR) on agentic
language models. Each task requires a sandboxed setup, a
prompt, and a hand-authored reward function, and only tasks that pass a quality bar produce useful training signal. Hand-curation at this quality bar does not scale economically to the task counts effective RL training requires, and the substitution rate between automatically generated task variants and human-authored ones is not yet established. We investigate using pre-specified, gate-filtered augmentations of a small hand-authored base as a substitute for additional human curation during RLVR. We formalize the cost-adjusted trade rate $\rho_{\text{cost}}$ between augmented and human-authored tasks, measure it through a controlled ablation across training corpora with varying augmentation share, and characterize the end-to-end economics of the augmentation pipeline. Substituting augmented content for additional human-authored tasks retains aggregate held-out generalization on a ten-
benchmark suite spanning code, instruction following,
reasoning, and multi-turn agentic function-calling. The cost-adjusted trade rate $\rho_{\text{cost}}$ between gated synthetic and human-authored RLVR tasks stays in $[1.4\times, 11.6\times]$ across the plausible $c_{\text{human}}/c_{\text{aug}}$ range.
Read the full abstract ▾
All authors Akshansh, Leonardo Rosa Rodrigues, Michael Korostelev, Youssef Hassan, Mark E. Whiting
Machine Learning
Limit Analysis of Graph Neural Networks with Wireless Conflict Graphs
Romina Garcia Camargo, Zhiyang Wang and Alejandro Ribeiro
Graph Neural Networks (GNNs) have emerged as a powerful tool for wireless resource allocation that leverages the underlying graph structure of communication networks. Their transferability property enables models trained on small-scale graphs to generalize to large-scale deployments with little performance deterioration, a desirable property for currently growing networks. Wireless networks are sparse regimes, where a single node is connected to a small number of other users. This work establishes theoretical results for transferability of GNNs over graphs derived from sparse Random
Geometric Graphs (RGGs). In particular, we focus on conflict graphs of RGGs used to model interference among links. Our approach considers the closeness between RGGs and Deterministic Grid Graphs (DGG) to establish bounds in the performance loss when a model is transferred across scales. We validate our theoretical findings through the problem of link scheduling, demonstrating that our learned policies consistently outperform existing
benchmarks at scale. Finally, we examine the impact of our theoretical assumptions on empirical performance.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting
Georgios Tsoumplekas, Stella Bounareli and Vasileios Argyriou
Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained
diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding
prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a
prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target
prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large
Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/
Prompt-Aware-Multi-LoRA-Composition.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
Yuetian Lu, Ali Modarressi, Yihong Liu et al. (+1 more)
Activation patching can identify a mixture-of-experts (MoE) block whose clean output restores a corrupted factual prediction. However, because the block output combines contributions from multiple routed experts, block-level rescue does not establish whether the recovery localizes to an individual expert or depends on the routed expert set. We study this question on single-
token COUNTERFACT contrasts by corrupting subject-
token embeddings, restoring clean block outputs, and then restoring clean-minus-noised expert updates under fixed routing. In Qwen3-30B-A3B-Base, a discovery sweep selects layer 44, and held-out analysis identifies L44E069 as a recurrent routed contributor with positive specificity over same-layer active experts. Its effect is fact-matched and improves true-
token probability and rank, which explains part of the layer rescue. In Mixtral-8x7B-v0.1, the selected recurrent singleton is not specific; matched-size controls instead show specificity of the clean-routed top-2 set. These findings show that successful MoE-block restoration does not necessarily imply localization to a single expert.
Read the full abstract ▾
All authors Yuetian Lu, Ali Modarressi, Yihong Liu, Hinrich Schütze