Machine Learning · Cross-listed from stat.ML
Finite-Particle Convergence Rates for Conservative and Non-Conservative Drifting Models
Krishnakumar Balasubramanian
We analyze finite-particle drifting models for one-step generative modeling. For a conservative velocity given by the difference of the kernel-smoothed data and model scores, a joint-
entropy identity yields continuous-time bounds for the smoothed Fisher discrepancy and the squared particle velocity. The finite-particle correction involves reciprocal kernel density estimates. We give local-occupancy conditions and exact expectation bounds, and show that these expectations diverge for full-support initial densities in a shrinking-bandwidth regime. Keeping the bandwidth dependence of the quadrature constants explicit yields a conditional root residual-velocity rate of $N^{-1/(d+4)}$ under uniform regularity, with a corresponding rate under weaker quadrature growth conditions. We also analyze the original displacement field with the exact Laplace kernel. A companion kernel gives a weighted coercivity estimate that accommodates the unbounded empirical scale and the kernel's nondifferentiability. Localization controls the particle velocity, while relative scale
alignment absorbs the mismatch into dissipation. Both analyses quantify the size of an additional drift correction under explicit regularity and stability assumptions.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems
Qianshu Cai, Yonggang Zhang, Xianzhang Jia et al. (+5 more)
Autonomous agentic systems are largely static after deployment: they do not learn from user interactions, and recurring failures persist until the next human-driven update ships a fix. Self-evolving
agents have emerged in response, but all confine
evolution to text-mutable artifacts – skill files,
prompt configurations,
memory schemas, workflow graphs – and leave the
agent harness untouched. Since routing, hook ordering, state invariants, and dispatch live in code rather than in any text artifact, an entire class of structural failure is physically unreachable from the text layer. We argue that source-level adaptation is a fundamentally more general medium: it is Turing-complete, a strict superset of every text-mutable scope, takes effect deterministically rather than through base-model compliance, and does not erode under long-context drift. We present MOSS, a system that performs self-rewriting at the source level on production agentic substrates. Each
evolution is anchored to an automatically curated batch of production-failure evidence and proceeds through a deterministic multi-stage pipeline; code modification is delegated to a pluggable external coding-
agent CLI while MOSS retains stage ordering and verdicts. Candidates are verified by replaying the batch against the candidate image in ephemeral trial workers, then promoted via user-consent-gated, in-place container swap with health-probe-gated rollback. On OpenClaw, MOSS lifts a four-task mean grader score from 0.25 to 0.61 in a single cycle without human intervention.
Read the full abstract ▾
All authors Qianshu Cai, Yonggang Zhang, Xianzhang Jia, Huajiang Zheng, Wei Xue, Jun Song, Xinmei Tian, Yike Guo
Machine Learning · Cross-listed from cs.AI
LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systems
Sadia Asif, Mohammad Mohammadi Amiri, Momin Abbas et al. (+2 more)
Large
language model (LLM)-based multi-
agent systems increasingly rely on intermediate communication to coordinate complex tasks. While most existing systems communicate through natural
language, recent work shows that latent communication, particularly through
transformer key-value (KV) caches, can improve efficiency and preserve richer task-relevant information. However, KV caches also encode contextual inputs, intermediate
reasoning states, and
agent-specific information, creating an opaque channel through which sensitive content may propagate across
agents without explicit textual disclosure. To address this, we introduce LCGuard (Latent Communication Guard), a framework for safe KV-based latent communication in multi-
agent LLM systems. LCGuard treats shared KV caches as latent working
memory and learns representation-level transformations before cache artifacts are transmitted across
agents. We formalize representation-level sensitive information leakage operationally through reconstruction: a shared cache artifact is unsafe if an
adversarial decoder can recover
agent-specific sensitive inputs from it. This leads to an
adversarial training formulation in which the adversary learns to reconstruct sensitive inputs, while LCGuard learns transformations that preserve task-relevant semantics and reduce reconstructable information. Empirical evaluations across multiple model families and multi-
agent benchmarks show that LCGuard consistently reduces reconstruction-based leakage and attack success rates while maintaining competitive task performance compared to standard KV-sharing baselines.
Read the full abstract ▾
All authors Sadia Asif, Mohammad Mohammadi Amiri, Momin Abbas, Prasanna Sattigeri, Karthikeyan Natesan Ramamurthy
Machine Learning · Cross-listed from cs.SE
FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection
Huanchi Wang, Zihang Huang, Yifang Tian et al. (+3 more)
Production systems generate millions of log lines daily, yet most anomaly detectors operate at the session or window-level, flagging groups of lines rather than identifying the specific message responsible. This coarse granularity forces operators to inspect many routine lines per alert. Message-level detection offers finer granularity, but remains challenging. A single event template may correspond to both normal and anomalous messages, failures arise from heterogeneous subsystems, and line-level labeling at scale is impractical. Although large
language models (LLMs) can reason over log semantics, applying them to every line is too costly for continuous monitoring. We present FAME (Failure-Aware Mixture-of-Experts), a label-efficient message-level mixture-of-experts framework that uses an LLM only once offline. We annotate at most K labeled lines per template to derive binary normal/anomaly indicators and representative examples. The LLM proposes a partition of templates into failure domains, and a certification step validates the proposal before training. FAME trains a lightweight router and domain experts that run on-premise and output anomaly predictions and failure-domain labels. On BGL, FAME achieves F1 = 98.16 at K = 100 reducing annotation effort by 76x and detects 97.7% of anomalies from unseen EventIDs. On Thunderbird, FAME reaches F1 = 99.95 with perfect recall.
Read the full abstract ▾
All authors Huanchi Wang, Zihang Huang, Yifang Tian, Kristina Dzeparoska, Hans-Arno Jacobsen, Alberto Leon-Garcia
Machine Learning
SDPM: Survival Diffusion Probabilistic Model for Continuous-Time Survival Analysis
Stanislav R. Kirpichenko, Andrei V. Konstantinov and Lev V. Utkin
Survival analysis aims to estimate a time-to-event distribution from data with censored observations. Many existing methods either impose structural assumptions on the hazard function or discretize the time axis, which may limit flexibility and introduce approximation errors. We propose the Survival
Diffusion Probabilistic Model (SDPM), a generative approach to continuous-time survival analysis. SDPM models the conditional distribution of the survival outcome, represented by the pair of observed time and censoring indicator, $\mathbb{P}(T,\delta \mid \mathbf{x})$, using a denoising
diffusion model. Under the assumption of conditionally independent censoring, conditional samples generated by the model can be transformed into survival function estimates using the Kaplan-Meier
estimator. This formulation avoids parametric assumptions on the event-time distribution and does not require a discretization of the output time space. The model operates in a transformed target space, using standardized log-times and a continuous Gaussian-mixture representation of the censoring indicator. We evaluate SDPM on ten real survival
datasets and compare it with five strong baselines, including tree-based, boosting-based, and neural survival models. Results show that SDPM achieves competitive predictive performance across C-index, integrated time-dependent AUC, and integrated Brier score. A study on synthetic Cox-Weibull data demonstrates that SDPM can recover the shape of an underlying continuous survival distribution more accurately than a strong nonparametric baseline when sufficiently many samples are generated. An ablation study confirms the importance of the proposed target-space transformations, which improve event-rate calibration, reduce invalid generated times, and provide consistent gains in predictive discrimination. Codes implementing the proposed model are publicly available.
Read the full abstract ▾
Machine Learning
MambaGaze: Bidirectional Mamba with Explicit Missing Data Modeling for Cognitive Load Assessment from Eye-Gaze Tracking Data
Amir Mousavi, Mohammad Sadegh Sirjani, Erfan Nourbakhsh et al. (+5 more)
Real-time cognitive load assessment from eye-tracking signals could enable adaptive human-centered AI in safety-critical applications such as driver vigilance monitoring or automated flight deck assistance, yet two challenges persist: handling frequent data missingness from blinks and tracking failures, and efficiently modeling long-range temporal dependencies. We propose MambaGaze (Bi-Mamba), a framework that addresses these challenges through (1) XMD encoding, which augments raw features with observation masks and time-deltas to explicitly model data uncertainty, and (2) bidirectional Mamba-2, which captures temporal dependencies with linear computational complexity. Experiments on CLARE and CL-Drive
datasets under leave-one-subject-out evaluation show that MambaGaze achieves 77.1% accuracy and 59.2% macro-F1 on CLARE, and 69.4% accuracy and 51.5% macro-F1 on CL-Drive, attaining the highest average LOSO macro-F1 (55.3%) across all ten compared models. Input-stream ablation indicates that log-scaled time-deltas are the strongest single channel in our setting, and combining all three XMD streams provides consistent gains of 5-20 pp macro-F1. Edge deployment
benchmarks on three NVIDIA Jetson Orin platforms show real-time inference at 27-36 FPS with power consumption below 6.6 W, supporting feasibility for embedded cognitive load monitoring.
Read the full abstract ▾
All authors Amir Mousavi, Mohammad Sadegh Sirjani, Erfan Nourbakhsh, Mimi Xie, Rocky Slavin, Leslie Neely, John Davis, John Quarles
Machine Learning
CogAdapt: Adapting Clinical ECG Foundation Models for Wearable Cognitive Load Assessment
Amir Mousavi, Erfan Nourbakhsh, Mohammad Sadegh Sirjani et al. (+5 more)
Assessing cognitive load continuously and at low latency would help adaptive human-computer interaction, but it remains hard because labeled data are scarce and models generalize poorly across subjects. Recent ECG foundation models, pre-trained on millions of clinical diagnostic ECG recordings, yet they do not apply directly to wearable devices when the sensor configuration and the task both differ. We present CogAdapt, a framework that adapts a clinical ECG foundation model to wearable cognitive load assessment. CogAdapt has two parts. LeadBridge is a learnable adapter that maps 3-lead wearable signals to a 12-lead-compatible representation. ProFine is a progressive fine-tuning strategy that unfreezes encoder layers in stages while limiting representational drift in the pre-trained model. On two public
datasets (CLARE and CL-Drive) under leave-one-subject-out cross-validation, CogAdapt reaches macro-F1 of 0.626 and 0.768, improving over from-scratch baselines by 11.2 and 16.1 percentage points. The results show that a clinical ECG pretraining can support subject-independent cognitive load assessment from wearable sensors.
Read the full abstract ▾
All authors Amir Mousavi, Erfan Nourbakhsh, Mohammad Sadegh Sirjani, Mimi Xie, Rocky Slavin, Leslie Neely, John Davis, John Quarles
Machine Learning
Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
Samson Gourevitch, Yazid Janati, Dario Shariatian et al. (+4 more)
Discrete
diffusion models are often trained through clean-data prediction, but the prediction can be used in different ways to define the reverse dynamics. In Masked
Diffusion Models (MDM) these choices largely coincide, whereas in Uniform
Diffusion Models (UDM) they do not. We show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior, but by a leave-one-out posterior that predicts each clean
token without using its own noisy observation. This identifies a mismatch between the plug-in ELBO and the usual cross-
entropy denoising objective. We characterize the leave-one-out target and derive exact conversions between the denoiser, the leave-one-out posterior, and the score. These conversions allow us to disentangle parameterization and training objective. Our results also lead to inference improvements without any additional training through an informed predictor-corrector sampler and improved temperature sampling based on the leave-one-out predictor. We further introduce an absorbing-state reformulation of uniform
diffusion that preserves the UDM joint law while decomposing it into masked-
diffusion-like sampling operations, with simpler denoising posteriors, carry-over unmasking, and a natural remasking mechanism. On
language modeling, leave-one-out parameterizations consistently improve UDM generation, while the absorbing construction matches or surpasses masked
diffusion. These results suggest that the empirical gap between masked and uniform
diffusion is driven less by the choice of marginals themselves than by parameterization and sampling design. The code and models can be found at https://github.com/samsongourevitch/rev_udm.
Read the full abstract ▾
All authors Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli, Eric Moulines, Eric P. Xing, Alain Durmus
Machine Learning
Lumberjack: Better Differentially Private Random Forests through Heavy Hitter Detection in Trees
Christian Janos Lebeda, David Erb, Tudor Cebere et al. (+1 more)
Random forests are widely used in fields involving sensitive tabular data, but existing approaches to enforcing differential
privacy (DP) typically degrade performance to the point of impracticality. In this paper, we introduce Lumberjack, a differentially private random forest
algorithm that achieves substantially higher utility by constructing large random decision trees and then applying aggressive,
privacy-preserving
pruning to retain only sufficiently populated nodes. A key component of our approach is a novel $(\varepsilon,\delta)$-DP heavy hitter detection
algorithm for hierarchical data, whose error is $O_{\varepsilon,\delta}(\sqrt{\log h})$ for trees of height $h$ and may be of independent interest. This favorable scaling enables the use of significantly deeper trees than in prior work, leading to improved expressiveness under
privacy constraints. Our empirical evaluation on
benchmark datasets shows that Lumberjack consistently outperforms prior DP random forest methods, establishing a new state of the art. In particular, our approach yields substantial improvements in the
privacy-utility trade-off for practical
privacy budgets. Our findings suggest that carefully designed DP random forests can close much of the utility gap, highlighting a promising and underexplored direction for future research.
Read the full abstract ▾
All authors Christian Janos Lebeda, David Erb, Tudor Cebere, Aurélien Bellet
Machine Learning
Cyber-Physical Anomaly Detection in IoT-Enabled Smart Grids Using Machine Learning and Metaheuristic Feature Optimization
Adis Alihodžić, Eva Tuba and Milan Tuba
Modern smart grids rely on dense measurement infrastructures, communication links, and intelligent field devices. Although this improves supervision and control, it also increases vulnerability to cyber-physical disruptions. Operators must distinguish physical incidents, such as faults or line disturbances, from malicious actions, such as false data injection or unauthorized command execution. This chapter investigates this problem using the well-known MSU/ORNL Power System Attack
Dataset. The proposed method combines machine learning with genetic-
algorithm-based feature selection. The objective is twofold: to classify attack and natural events accurately, and to determine whether a reduced set of physically informative PMU/IED measurements can support reliable detection. Several baseline models are evaluated, including logistic regression, RBF-SVM, XGBoost, Random Forest, and Extra Trees. The results show that tree-based ensemble models are the most effective for the considered
dataset, with Extra Trees providing the strongest full-feature baseline. After feature selection, the GA + Extra Trees model reduces the clean PMU feature space from 112 attributes to an average of 27.4 attributes over five runs, while increasing macro-F1 from 0.9118 to 0.9212 and ROC-AUC from 0.9791 to 0.9837. These results indicate that many synchronized electrical measurements are redundant. A compact subset of phasor-based features can still provide accurate and interpretable anomaly detection in smart grids.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
Superhuman Safe and Agile Racing through Multi-Agent Reinforcement Learning
Ismail Geles, Leonard Bauersfeld, Markus Wulfmeier et al. (+1 more)
Autonomous systems have achieved superhuman performance in isolation or simulation, yet they remain brittle in shared, dynamic real-world spaces. This failure stems from the dominant single-
agent paradigm for physical applications, where other actors are ignored or treated as environmental noise, preventing effective coordination. Here we show that multi-
agent reinforcement learning provides the essential safety scaffolding required for real-world interaction. Using high-speed quadrotor racing as a high-stakes testbed, we train
agents to navigate complex aerodynamic interactions and strategic maneuvering with a variable number of racers. Through league-based self-play,
agents evolve sophisticated anticipatory behaviors, including proactive collision avoidance, overtaking, and handling multi-
agent physical interactions, including aerodynamic downwash. Our
agents outperform a champion-level human pilot in multi-player races at speeds exceeding 22 m/s, while simultaneously reducing collision rates by 50 % compared to state-of-the-art single-
agent baselines. Crucially, training with diverse artificial
agents enables zero-shot generalization to safer human interaction. These results suggest that the path to robust robotic co-existence lies not in isolated safety constraints, but in the rigorous demands of multi-
agent interaction. Multimedia materials are available at: https://rpg.ifi.uzh.ch/marl
Read the full abstract ▾
All authors Ismail Geles, Leonard Bauersfeld, Markus Wulfmeier, Davide Scaramuzza
Machine Learning
Plug-in Losses for Evidential Deep Learning: A Simplified Framework for Uncertainty Estimation that Includes the Softmax Classifier
Berk Hayta, Hannah Laus, Simon Mittermaier et al. (+1 more)
Real-world sensor-based learning systems require uncertainty estimation that is both reliable and computationally efficient. Evidential Deep Learning (EDL) provides single-pass uncertainty estimation by modeling the class probabilities via Dirichlet distributions, where the Dirichlet parameters are predicted by a learned neural network mapping. However, this approach can lead to computational challenges, as Dirichlet expected objectives are more complex than standard supervised learning losses, complicating their analysis and implementation. We address this issue by approximating the objective of the first-order empirical risk minimization problem induced by EDL with a plug-in loss evaluated at the Dirichlet mean and show that, under mild assumptions, the approximation error decays with growing evidence for a broad class of loss functions, including mean-squared error and cross-
entropy loss. As a special case, our analysis provides justification for the use of softmax in the context of uncertainty estimation, since under a particular evidence-to-Dirichlet mapping, our framework includes the standard softmax classifier. We validate the proposed simplified objectives on the Google
Speech Commands
dataset and show that they achieve predictive accuracy and selective prediction performance comparable to classical EDL, while being simpler to implement using standard deep learning losses and training pipelines. To the best of our knowledge, this empirical analysis is the first to obtain coverage-accuracy trade-offs for
speech recognition tasks through EDL.
Read the full abstract ▾
All authors Berk Hayta, Hannah Laus, Simon Mittermaier, Felix Krahmer
Machine Learning
SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation
Javad Parsa, Enis Simsar, Amir Joudaki et al. (+2 more)
Parameter-efficient fine-tuning enables fast personalization of text-to-image
diffusion models to user-provided concepts (objects, people, or styles), but composing multiple such concepts remains challenging due to representation interference. Existing modular methods, usually built on low-rank adaptation (LoRA), either rely on expensive post-hoc fusion or freeze the LoRA adaptation subspaces, which limit expressiveness and concept fidelity. To address this trade-off, we propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework that jointly optimizes both LoRA factors via bilevel optimization while keeping each new basis orthogonal to all previously learned ones. Theoretically, we establish monotone descent and convergence to a critical point of the constrained problem, and model the residual layer activations as a
matrix sub-Gaussian process to derive a high-probability bound on catastrophic forgetting in multi-layer nonlinear networks. This bound depends on the basis only through a residual interference energy, and within the feasible subspace we prove that a data-adapted basis minimizes it, whereas a random frozen basis is suboptimal in expectation. Experiments on Stable
Diffusion with up to 101 concepts show that SeqLoRA improves identity preservation over fusion-based methods, attains the lowest cross-concept leakage in multi-concept compositions, requires no fusion step, and scales to concept counts at which fusion runs out of
memory.
Read the full abstract ▾
All authors Javad Parsa, Enis Simsar, Amir Joudaki, Thomas Hofmann, André M. H. Teixeira
Machine Learning
Ternary Decision Trees with Locally-Adaptive Uncertainty Zones
William Smits
Decision trees assign identical confidence to instances near and far from each split threshold. We introduce ternary decision trees, which augment each split node with an uncertainty zone of half-width delta. A decision-theoretic framework characterises the optimal zone width delta* as the solution to a node-local cost-minimisation problem; four formal properties are established: accuracy decomposition, a sufficiency condition for decided accuracy improvement, an exact efficiency characterisation (eta = Dec-Acc minus Acc_u, the accuracy gap between decided and boundary-uncertain predictions), and asymptotic consistency of the margin method. Instances within the zone receive predictions by weighted blending of both child subtrees and are flagged as boundary-uncertain. We propose and evaluate five delta-estimation methods: quality-plateau (plateau width of the split criterion curve), class-overlap (empirical class-distribution overlap), gain-ratio (split quality relative to split
entropy), node-bootstrap (threshold variance under node-level resampling), and margin (SVM-inspired distance to the nearest cross-class training example). All methods reuse statistics already computed during standard CART split finding, requiring no external noise specification. Evaluated across 71 of the 72 OpenML-CC18
datasets with 5-fold cross-validation, all five methods with probabilistic routing significantly outperform standard CART on decided accuracy (Wilcoxon signed-rank, p < 0.001). The margin method achieves the best efficiency (0.104 accuracy gain per unit flagging rate), wins on 42 of 72
datasets, and requires zero hyperparameters. Analysis on Breiman synthetic
benchmarks confirms margin is self-calibrating on clean data. On mammography, node-bootstrap achieves +0.71% decided accuracy by flagging 10.8% of cases as boundary-uncertain.
Read the full abstract ▾
Machine Learning
Proxy-Based Approximation of Shapley and Banzhaf Interactions
Santo M. A. R. Thies, Hubert Baniecki, R. Teal Witter et al. (+3 more)
Shapley and Banzhaf interactions capture the complex dynamics inherent in modern machine learning applications. However, current
estimators for these higher-order interactions trade off between speed and accuracy. To overcome this limitation, we introduce ProxySHAP. ProxySHAP reconciles the high sample efficiency of tree-based proxy models with a principled path to consistency via residual correction. On a theoretical level, we derive a
polynomial-time generalization of interventional TreeSHAP to compute exact interaction indices for tree ensembles, successfully bypassing exponential tree-depth dependencies in prior methods. Furthermore, we formally analyze the residual adjustment strategy, characterizing the specific conditions under which Maximum Sample Reuse (MSR) corrects proxy bias without its variance scaling exponentially with interaction size. Extensive benchmarking demonstrates that ProxySHAP sets a new state-of-the-art standard for approximation quality, including in large-scale applications with thousands of features. By achieving the lowest error in both small- and large-budget regimes, ProxySHAP significantly outperforms the prior best
estimators ProxySPEX and KernelSHAP-IQ, while also delivering superior performance on downstream explainability tasks.
Read the full abstract ▾
All authors Santo M. A. R. Thies, Hubert Baniecki, R. Teal Witter, Eyke Hüllermeier, Maximilian Muschalik, Fabian Fumagalli
Machine Learning
The Distillation Game: Adaptive Evaluations & Efficient Defenses
Youssef Allouah, Mahdi Haghifam, Sanmi Koyejo et al. (+1 more)
Distillation attacks create a deployment trade-off for model providers: the same outputs that make a model more useful can also make it easier to imitate. We study this trade-off through a minimax game between a utility-constrained teacher and an adaptive student. Our framework yields tractable one-sided response rules: an adaptive evaluation rule in which the student reweights high-value examples, and a teacher-side defense template that suppresses outputs most useful for
distillation. From a cheap proxy for example value, we derive Product-of-Experts (PoE), a simple forward-pass-only defense that combines the teacher with a proxy student during generation. Empirically, adaptive evaluation reveals a large passive–adaptive gap: on state-of-the-art defenses, adaptive students recover substantially more capability than passive evaluation suggests on GSM8K and MATH. Under this stronger evaluation, the apparent robustness gap between expensive defenses and PoE narrows considerably, while PoE remains substantially cheaper and preserves higher-quality
reasoning traces. Overall, our results suggest that strong
distillation remains difficult to stop, and that progress on antidistillation should be judged against adaptive students rather than passive ones. Our code is available at: https://github.com/ysfalh/
distillation-game.
Read the full abstract ▾
All authors Youssef Allouah, Mahdi Haghifam, Sanmi Koyejo, Reza Shokri
Machine Learning · Cross-listed from math.OC
Optimization over the intersection of manifolds
Yan Yang, Bin Gao and Ya-xiang Yuan
Optimization over the intersection of two
manifolds arises in a broad range of applications, but is hindered by the coupled geometry of the feasible region. In this paper, we prove that the regularities – clean intersection and intrinsic transversality – are equivalent, which yields a tractable projection onto the tangent space of the intersection. Therefore, we propose a
geometric method that employs a retraction on only one
manifold and updates the iterate along two orthogonal directions. Specifically, the iterates stay on one
manifold, and the two directions are responsible for asymptotically approaching the other
manifold and decreasing the objective function, respectively. Under intrinsic transversality, we derive the convergence rate for both the feasibility and optimality measures, and show that every accumulation point is first-order stationary. Numerical experiments on problems stemming from sparse and low-rank optimization, including fitting spherical data, approximating
hyperbolic embeddings on real data, and computing compressed modes, demonstrate the effectiveness of the proposed method.
Read the full abstract ▾
Machine Learning
Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation
Dong Nie
Large
language model post-training methods such as supervised fine-tuning (SFT), reinforcement learning (RL), and
distillation are often analyzed through their loss functions: maximum likelihood, policy gradients, forward KL, reverse KL, or related objective-level variants. We study a complementary factor: the state distribution on which supervision is applied. For an autoregressive policy, a state is a
prompt plus generated prefix. SFT trains on fixed
dataset states, while RL and on-policy
distillation (OPD) train on states induced by the current learner. We formalize post-training as state-distribution shaping and run a controlled smallscale study using Qwen3-0.6B-Base on GSM8K, with TruthfulQA and MMLU as retention evaluations. Our results show three phenomena. First, a mild SFT run improves GSM8K with little forgetting, while a stress SFT run causes substantial retention loss. Second, OPD from a degraded SFT teacher surpasses that teacher on GSM8K, TruthfulQA, and MMLU, despite using the teacher as its only supervision source. Third, a lightweight on-policy RL run improves GSM8K while preserving retention. These results support a state-centric view of post-training: the source and locality of training states can be as important as the form of the supervision signal.
Read the full abstract ▾
Machine Learning
Multiple Neural Operators Achieve Near-Optimal Rates for Multi-Task Learning
Adrien Weihs and Hayden Schaeffer
We study the approximation and statistical complexity of learning collections of operators in a shared multi-task setting, with a focus on the Multiple Neural Operators (MNO) architecture. For broad classes of Lipschitz multiple operator maps, we derive near-optimal upper bounds for approximation and statistical generalization. On the lower-bound side, we establish a curse of parametric complexity and prove corresponding minimax rates. Together, these results show that shared representations across tasks do not increase the overall cost: multi-task operator learning follows the same scaling laws as single operator learning. We also compare MNO with a multi-task extension of DeepONet based on concatenated task inputs and show that, from a worst-case approximation-complexity perspective, both architectures satisfy essentially the same asymptotic rates.
Read the full abstract ▾
Machine Learning
The Value of Covariance Matching in Gaussian DDPMs and the Lanczos Sampler
Md Sahil Akhtar, Aymane El Gadarri, Vivek F. Farias et al. (+1 more)
A central error measure in Gaussian DDPMs is the path-space KL divergence between the exact reverse chain and the learned Gaussian reverse process. This quantity is especially relevant for procedures such as classifier guidance, which perturb the entire reverse trajectory rather than only the terminal sample. Prior analyses show that standard isotropic reverse covariances suffer an unavoidable $\Omega(1/T)$ path-KL error as the number of denoising steps $T$ grows. We show that matching the full posterior covariance breaks this barrier, yielding an order-wise improvement that reduces the path KL to $O(1/T^2)$. To make full covariance matching practical, we introduce the Lanczos Gaussian sampler (LGS), a training-free,
matrix-free method for sampling from the optimal reverse covariance using only covariance-vector products, which are available through Jacobian-vector products of the posterior mean. LGS avoids dense covariance storage and auxiliary covariance models. We prove that LGS approximation error decays exponentially in the number of Lanczos steps, where each Lanczos step requires a single Jacobian-vector product. Empirically, using only just three such steps improves sample quality over strong diagonal-covariance baselines, including OCM-DDPM, across standard image
benchmarks. This identifies full covariance matching as both theoretically valuable and practically accessible for fast DDPM sampling.
Read the full abstract ▾
All authors Md Sahil Akhtar, Aymane El Gadarri, Vivek F. Farias, Adam D. Jozefiak
Machine Learning
Reading Task Failure Off the Activations: A Sparse-Feature Audit of GPT-2 Small on Indirect Object Identification
Mahdi Nasermoghadasi
We report a small, reproducible audit of which sparse-autoencoder (SAE) features of GPT-2 small fire differently on failed versus successful trials of the Indirect Object Identification (IOI) task. On 300
prompts, GPT-2 small reaches 79.7% accuracy; 146 of the 24,576 features in the layer-8 residual-stream SAE release of Bloom (2024) clear a Holm-corrected significance threshold and 105 reach a large effect size (|Cohen's d| > 0.8). The strongest single correlate of failure – feature 17,491, d=+2.93, Neuronpedia label 'cryptographic keys' – is essentially silent except when the
prompt's transferred object is 'the keys,' on which GPT-2 small fails 93.3% of the time vs. 7.5% on the other seven objects (Fisher exact p = 8.79 x 10^-33). We put this correlate through three controls that a mechanistic claim should pass. (i) A causal ablation: zeroing feature 17,491 in the residual stream across all
token positions of the 45 keys
prompts does not restore accuracy (6.7% -> 4.4%); the feature is a correlate, not a sufficient cause at this layer. (ii) A representation baseline: a logistic regression on the raw 768-dimensional residual stream reaches 5-fold ROC AUC = 0.929, matching the top-100 SAE features (0.927); the SAE basis adds interpretability, not predictive power. (iii) A seed-robustness check: across five random seeds the keys-subset failure rate stays in 75.0–93.3% (the behavioural effect is real), but feature 17,491 is the top-|d| feature in only 1 of 5 runs. The methodological contribution is therefore the audit pipeline (cheap, model-agnostic, surfaces named correlates) rather than any single feature found through it. We release the code, the 300-
prompt corpus, the 300x24,576 activation
matrix, the ablation and baseline scripts, and the figures. The full pipeline runs on a laptop (Apple M3 Max, no discrete GPU).
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SD
Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators
Zachary Novack, Stephen Brade, Haven Kim et al. (+8 more)
Interactive streaming music generation promises the use of generative models for live performance and co-creation that is impossible with offline models. However, SOTA models exist in the discrete-AR regime, requiring industrial levels of compute for both training and inference. In this work, we investigate whether audio
diffusion models, with their wide support in the open-source community but non-streaming bidirectional nature, can be repurposed efficiently into interactive models accessible on consumer hardware. By taking a critical look at the modern pipeline for block-wise outpainting
diffusion, we identify critical inefficiencies during inference that result in strictly worse computational efficiency than their discrete-AR counterparts. We propose Live Music
Diffusion Models (LMDMs), a simple modification of the generative
diffusion process that recovers, and then outperforms, the inference complexity of the discrete Live Music Models (LMMs) through block-wise KV Caching. Unlike LMMs, LMDMs further enable stable post-training
alignment through our novel ARC-Forcing paradigm, reducing error accumulation without any explicit RL or reward models. We demonstrate the application of LMDMs in a number of creative domains, including text-conditioned generation, sketch-based music synthesis, and jamming. We finally show how LMDMs can be used as a generative instrument in a real artist-AI collaboration, utilizing LMDMs as a "generative delay" to transform musicians' improvisation live for variable timbral effects while running locally on a consumer gaming laptop.
Read the full abstract ▾
All authors Zachary Novack, Stephen Brade, Haven Kim, Hugo Flores García, Nithya Shikarpur, Chinmay Talegaonkar, Suwan Kim, Valerie K. Chen, Julian McAuley, Taylor Berg-Kirkpatrick, Cheng-Zhi Anna Huang
Machine Learning · Cross-listed from cs.AI
AMEL: Accumulated Message Effects on LLM Judgments
Sid-Ali Temkit
Large
language models are routinely used as automated evaluators: to review code, moderate content, or score outputs, often with many items passing through one conversation. We ask whether the polarity of prior conversation history biases subsequent judgments, an effect we call the accumulated message effect on LLM judgments (AMEL). Across 84,088 API calls to 12 models from 5 providers (OpenAI, Anthropic, Google, DeepSeek, and four open-source models), we present identical test items in isolation or following histories saturated with predominantly positive or negative evaluations. Models shift toward the conversation's prevailing polarity (d = -0.17, p < 10^-53). The effect concentrates on items where the model is genuinely uncertain at baseline (d = -0.36 for high-
entropy items, vs d = -0.15 when the baseline is deterministic). Bias does not grow with context length: 5 prior turns and 50 produce the same shift (Spearman |r| < 0.01; OLS slope p = 0.80). And there is a negativity asymmetry: paired per item, negative histories induce 1.52x more bias than positive (t = 13.03, p < 10^-36, n = 2,733). Scaling helps but does not solve it (Anthropic: Haiku -0.22 to Opus -0.17; OpenAI: Nano -0.34 to GPT-5.2 -0.17). Three follow-ups narrow the mechanism. The
token probability distribution shifts continuously, not at a threshold. The negativity asymmetry has both
token-level and semantic components, though attributing the balance is exploratory at our sample sizes. Position does not matter: five biased turns anywhere in a 50-turn history produce the same shift. The simplest fix for evaluation pipelines is a fresh context per item; when batching is unavoidable, balancing the history helps.
Read the full abstract ▾
Machine Learning
Abstraction for Offline Goal-Conditioned Reinforcement Learning
Clarisse Wibault, Alexander Goldie, Antonio Villares et al. (+2 more)
Markov Decision Processes (MDPs) often exhibit significant redundancy due to
symmetries and shared structure across state-goal pairs in real-world Goal-Conditioned Reinforcement Learning (GCRL). While hierarchical policies have been motivated for horizon reduction via temporal abstraction in offline GCRL, we demonstrate that hierarchy also enables absolute abstraction. By introducing relativised options as well as distinct representations for different levels of the hierarchy, we demonstrate how an
agent can reuse experience across similar contexts of the state-space. Based on this framework, we introduce two simple
algorithms for learning relativised options and abstracting from the absolute frame of reference. Our experiments show that such inductive biases significantly improve performance in offline GCRL.
Read the full abstract ▾
All authors Clarisse Wibault, Alexander Goldie, Antonio Villares, Maike Osborne, Jakob Foerster
Machine Learning
Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals
Shuo Yang, Jinda Lu, Chiyu Ma et al. (+8 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a central paradigm for scaling LLM
reasoning, yet its optimization often suffers from training instability and suboptimal convergence. Through a systematic dissection of clipping-based GRPO-style objectives, we identify the rigid clipping decision induced by hard clipping as a key practical bottleneck in the studied RLVR setups. Specifically, our analysis suggests that informative signals can lie in the near-boundary region just beyond the clipping threshold, and are therefore discarded by the standard hard-clipping rule. Notably, once this bottleneck is precisely identified, even simple
stochastic perturbations at the boundary can recover meaningful performance gains. Building on this finding, we propose Near-boundary
Stochastic Rescue (NSR), a minimal, plug-and-play modification that stochastically retains these slightly out-of-bound
tokens to recover lost signals. While NSR, via
stochastic sampling, can be interpreted as inducing an implicit gradient decay in expectation, our ablations reveal that its
stochastic, boundary-local rescue mechanism is consistently more effective than deterministic gradient decay. Validated by extensive experiments across model sizes from 7B to 30B and both dense and MoE architectures, as a plug-and-play solution, NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
Read the full abstract ▾
All authors Shuo Yang, Jinda Lu, Chiyu Ma, Kexin Huang, Haoming Meng, Qihui Zhang, Yuyang Liu, Bolin Ding, Guoyin Wang, Li Yuan, Jingren Zhou
Machine Learning
Latent Spectroscopy: Posterior Collapse as a Feature
Johannes Hirn
We show that, in linear Gaussian VAEs, posterior collapse is a form of latent feature selection. Feature importance is set by each latent coordinate's contribution to reconstruction, itself given by the corresponding PCA
eigenvalue. Varying the regularizer strength $\beta$ reveals a ranked spectrum of collapse events, with thresholds set by the utility/PCA spectrum. A mode-by-mode analysis identifies the scale-invariant signal fraction as an order parameter obeying Landau scaling near collapse. WorldClim experiments confirm both the utility-threshold calibration and the predicted near-collapse scaling.
Read the full abstract ▾
Machine Learning
ChronoVAE-HOPE: Beyond Attention – A Next-Generation VAE Foundation Model for Specialized Time Series Classification
José Alberto Rodríguez, Luis Balderas, Miguel Lastra et al. (+2 more)
Time Series Foundation Models (TSFMs) have become a new component of the state-of-the-art in general time series
forecasting. However, adapting them to specialized classification tasks remains constrained by two interconnected challenges: the quadratic cost of standard
attention mechanisms and the inability to disentangle the structural components underlying time series variability. This technical report introduces ChronoVAE-HOPE, a next-generation TSFM that reconciles massive generalization with structured latent representation for time series classification. The core of the proposal is a Variational Autoencoder (VAE) framework built upon the HOPE Block, which replaces quadratic
attention with a dual-
memory system: Titans modules for dynamic short-term retention and a Continuum
Memory System (CMS) for the abstraction of long-term historical context. A key architectural novelty is the disentangled latent space, which factorizes representations into independent trend and seasonal components via dedicated encoder heads and separate decoder pathways. ChronoVAE-HOPE undergoes self-supervised pre-training on the Monash archive, combining a Masked Time Series Modeling (MTSM) auxiliary objective with a disentangled VAE reconstruction loss. The pre-trained encoder is subsequently frozen and used to generate fixed-length
embeddings for downstream classification on the UCR
benchmark datasets. Empirical results demonstrate strong performance across diverse temporal domains, particularly in settings characterized by strict causal structure. ChronoVAE-HOPE establishes a robust and interpretable framework for the adaptation of foundation models to time series classification through structured generative representations.
Read the full abstract ▾
All authors José Alberto Rodríguez, Luis Balderas, Miguel Lastra, Antonio Arauzo-Azofra, José M. Benítez
Machine Learning · Cross-listed from cs.CV
Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models
Piotr Kubaty, Patryk Marszałek, Łukasz Struski et al. (+3 more)
Vision-
language models learn powerful
multimodal embeddings, yet their internal semantics remain opaque. While sparse autoencoders (SAEs) can extract interpretable features, they rely on expanding the representation dimension, which compromises the original geometry and introduces redundancy. We introduce CEDAR (Conceptual
Embedding Disentanglement via Adaptive Rotation), a post-hoc method that reveals the compositional structure of pretrained
embeddings without increasing dimensionality. By learning an invertible transformation with a top-$k$ sparsity bottleneck, CEDAR concentrates semantic information into axis-aligned disentangled coordinates. In CLIP-like architecture, individual coordinates can be interpreted with textual concepts, while for generative models such as BLIP, they can be decoded into natural
language descriptions. Experiments demonstrate that CEDAR achieves a competitive reconstruction-sparsity trade-off while producing explanations that are more interpretable and better aligned with human perception. Our results suggest that the apparent
entanglement in
vision-
language representations can be resolved through a suitable change of basis, eliminating the need for overcomplete expansions.
Read the full abstract ▾
All authors Piotr Kubaty, Patryk Marszałek, Łukasz Struski, Adam Wróbel, Jacek Tabor, Marek Śmieja
Machine Learning · Cross-listed from math.CO
Holographic functions and neural networks
Balazs Szegedy
A fuzzy Boolean function is a map $f:\cube^n\to [0,1]$, where $n\in\mathbb N$. We introduce and compare three ways of saying that such a function has bounded complexity. The first is a sampling property: the value $f(x)$ can be recovered, up to small error and with high probability, from the values of a bounded number of randomly chosen coordinates of $x$. We call this the
holographic property. The second is a structural property: $f$ is uniformly close to a bounded-degree
polynomial in boundedly many bounded linear coordinate forms. The third is computational: $f$ is uniformly close to the output of a neural network with a bounded number of non-input
neurons, bounded Lipschitz activation functions and bounded incoming weights. We prove that these three properties are equivalent up to quantitative changes of the parameters. The implication from holography to
polynomial structure uses a variant of a weak version of hypergraph regularity.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation
Zhenyu Lu, Liupeng Li, Jinpeng Wang et al. (+4 more)
While large
language models provide strong compositional
reasoning, existing
reasoning segmentation pipelines fail to transparently connect this
reasoning to visual perception. Current methods, such as latent query
alignment, are end-to-end yet opaque "black boxes". Conversely, textual localization readout is merely readable, not truly interpretable, often functioning as an unconstrained post-hoc step. To bridge this interpretability gap, we propose SegCompass, an end-to-end model that leverages a Sparse Autoencoder (SAE) to forge an explicit, interpretable, and differentiable
alignment pathway. Given an image-instruction pair, SegCompass first generates a chain-of-thought (CoT) trace. The core of our method is an SAE that maps both the CoT and visual
tokens into a shared, high-dimensional sparse concept space. A query codebook selects salient concepts from this space, which are then spatially grounded by a slot mapper into a multi-slot heatmap that guides the final mask decoder. The entire model is trained jointly, unifying reinforcement learning for the
reasoning path with standard
segmentation supervision. This SAE-driven interface provides a "white-box" connection that is significantly more traceable than latent queries and more coherent than textual readouts. Extensive experiments on five challenging
benchmarks demonstrate that SegCompass matches or surpasses state-of-the-art performance. Crucially, our visual and quantitative analyses show a strong correlation between the quality of the learned sparse concepts and final mask accuracy, confirming that SegCompass achieves superior results through its enhanced and inspectable
alignment. Code is available at https://github.com/ZhenyuLU-Heliodore/SegCompass.
Read the full abstract ▾
All authors Zhenyu Lu, Liupeng Li, Jinpeng Wang, Haoqian Kang, Yan Feng, Ke Chen, Yaowei Wang