Machine Learning
Beyond the Class Subspace: Teacher-Guided Training for Reliable Out-of-Distribution Detection in Single-Domain Models
Hong Yang, Devroop Kar, Qi Yu et al. (+2 more)
Out-of-distribution (OOD) detection methods perform well on multi-domain
benchmarks, yet many practical systems are trained on single-domain data. We show that this regime induces a
geometric failure mode, Domain-Sensitivity Collapse (DSC): supervised training compresses features into a low-rank class subspace and suppresses directions that carry domain-shift signal. We provide theory showing that, under DSC, distance- and logit-based OOD scores lose sensitivity to domain shift. We then introduce Teacher-Guided Training (TGT), which distills class-suppressed residual structure from a frozen multi-domain teacher (DINOv2) into the student during training. The teacher and auxiliary head are discarded after training, adding no inference overhead. Across eight single-domain
benchmarks, TGT yields large far-OOD FPR@95 reductions for distance-based scorers: MDS improves by 11.61 pp, ViM by 10.78 pp, and kNN by 12.87 pp (ResNet-50 average), while maintaining or slightly improving in-domain OOD and classification accuracy.
Read the full abstract ▾
All authors Hong Yang, Devroop Kar, Qi Yu, Travis Desell, Alex Ororbia
Machine Learning · Cross-listed from math.NA
A Machine Learning-Enhanced Hopf-Cole Formulation for Nonlinear Gas Flow in Porous Media
V. S. Maduri and K. B. Nakshatrala
Accurate modeling of gas flow through porous media is critical for many technological applications, including reservoir performance prediction, carbon capture and sequestration, and fuel
cells and batteries. However, such modeling remains challenging due to strong nonlinear behavior and uncertainty in model parameters. In particular, gas slippage effects described by the Klinkenberg model introduce pressure-dependent permeability, which complicates numerical simulation and obscures deviations from classical Darcy flow behavior. To address these challenges, we present an integrated modeling framework for gas transport in porous media that combines a Klinkenberg-enhanced constitutive relation,
Hopf-Cole-transformed mixed-form linear governing equations, a shared-trunk neural network architecture, and a Deep Least-Squares (DeepLS) solver. The
Hopf-Cole transformation reformulates the original nonlinear flow equations into an equivalent linear system closely related to the Darcy model, while the mixed formulation, together with a shared-trunk neural architecture, enables simultaneous and accurate prediction of both pressure and velocity fields. A rigorous convergence analysis is performed both theoretically and numerically, establishing the stability and convergence properties of the proposed solver. Importantly, the proposed framework also naturally facilitates inverse modeling of pressure-dependent permeability and slippage parameters from limited or indirect observations, enabling efficient estimation of flow properties that are difficult to measure experimentally. Numerical results demonstrate accurate recovery of flow dynamics and parameters across a wide range of pressure regimes, highlighting the framework's robustness, accuracy, and computational efficiency for gas transport modeling and inversion in tight formations.
Read the full abstract ▾
Machine Learning
Differentiable Thermodynamic Phase-Equilibria for Machine Learning
Karim K. Ben Hicham, Moreno Ascani, Jan G. Rittig et al. (+1 more)
Accurate prediction of phase
equilibria remains a central challenge in chemical engineering. Physics-consistent machine learning methods that incorporate thermodynamic structure into neural networks have recently shown strong performance for activity-coefficient modeling. However, extending such approaches to
equilibrium data arising from an extremum principle, such as
liquid-
liquid equilibria, remains difficult. Here we present DISCOMAX, a differentiable
algorithm for phase-
equilibrium calculation that guarantees thermodynamic consistency at both training and inference, only subject to a user-specified discretization. The method combines discrete enumeration of feasible phase states with masked softmax aggregation in the backward pass, with the propagation of the true
equilibrium state in the forward pass, using a straight-through gradient
estimator to enable physics-consistent end-to-end learning of neural \gls{gE}-models. We show that this approach bears analogy to statistical thermodynamics, and we evaluate it on binary
liquid-
liquid equilibrium data where it outperforms existing surrogate-based methods, while offering a general framework for learning from different kinds of
equilibrium data.
Read the full abstract ▾
All authors Karim K. Ben Hicham, Moreno Ascani, Jan G. Rittig, Alexander Mitsos
Machine Learning · Cross-listed from q-bio.GN
A Standardized Framework For Evaluating Gene Expression Generative Models
Andrea Rubbi, Andrea Giuseppe Di Francesco, Mohammad Lotfollahi et al. (+1 more)
The rapid development of generative models for single-
cell gene expression data has created an urgent need for standardised evaluation frameworks. Current evaluation practices suffer from inconsistent metric implementations, incomparable hyperparameter choices, and a lack of biologically-grounded metrics. We present Generated Genetic Expression Evaluator (GGE), an open-source Python framework that addresses these challenges by providing a comprehensive suite of distributional metrics with explicit computation space options and biologically-motivated evaluation through differentially expressed gene (DEG)-focused analysis and perturbation-effect correlation, enabling standardized reporting and reproducible benchmarking. Through extensive analysis of the single-
cell generative modeling literature, we identify that no standardized evaluation protocol exists. Methods report incomparable metrics computed in different spaces with different hyperparameters. We demonstrate that metric values vary substantially depending on implementation choices, highlighting the critical need for standardization. GGE enables fair comparison across generative approaches and accelerates progress in perturbation response prediction, cellular identity modeling, and counterfactual inference.
Read the full abstract ▾
All authors Andrea Rubbi, Andrea Giuseppe Di Francesco, Mohammad Lotfollahi, Pietro Liò
Machine Learning · Cross-listed from stat.ML
A Unified Latent Space Disentanglement VAE Framework with Robust Disentanglement Effectiveness Evaluation
Xiaoan Lang, Md Mostafizer Rahman and Fang Liu
Evaluating and interpreting latent representations, such as variational autoencoders (VAEs), remains a significant challenge for diverse data types, especially when ground-truth generative factors are unknown. To address this, we unify several state-of-the-art disentangled VAE approaches for latent space disentanglement into one framework – bfVAE. To assess the effectiveness of a disentangled VAE model and enhance latent space interpretability, we propose Feature Variance Heterogeneity via Latent Traversal (FVH-LT) and Dirty Block Sparse Regression in Latent Space (DBSR-LS). To ensure robust interpretability of learned latent space, we develop a greedy
alignment strategy (GAS) that mitigates label switching and aligns latent dimensions across runs to set the foundation of result aggregation. We also introduce a convenient scalar latent space separation index (LSSI) based on the GAS-aligned outputs of FVH-LT and DBSR-LS to summarize the overall latent structural separation without knowledge of the ground-truth generative factors. We compare bfVAE to five VAE models and validate the effectiveness FVH-LT, DBSR-LS, and LSSI in on seven tabular and image
datasets. Under our examined experimental settings, bfVAE provides a more flexible disentanglement framework achieves more favorable overall trade-off between disentanglement and reconstruction than the
benchmark VAE models; FVH-LT and DBSR-LS reliably uncover semantically meaningful and domain-relevant latent structures and generally yield consistent results; and LSSI makes an effective quantitative summary of latent structural separation.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.AS
Cough activity detection for automatic tuberculosis screening
Joshua Jansen van Vüren, Devendra Singh Parihar, Daphne Naidoo et al. (+4 more)
The automatic identification of cough segments in audio through the determination of start and end points is pivotal to building scalable screening tools in health technologies for pulmonary related diseases. We propose the application of two current pre-trained architectures to the task of cough activity detection. A
dataset of recordings containing cough from patients symptomatic for tuberculosis (TB) who self-present at community-level care centres in South Africa and Uganda is employed. When automatic start and end points are determined using XLS-R, an average precision of 0.96 and an area under the receiver-operating characteristic of 0.99 are achieved for the test set. We show that best average precision is achieved by utilising only the first three layers of the network, which has the dual benefits of reduced computational and
memory requirements, pivotal for smartphone-based applications. This XLS-R configuration is shown to outperform an audio spectrogram
transformer (AST) as well as a logistic regression baseline by 9% and 27% absolute in test set average precision respectively. Furthermore, a downstream TB classification model trained using the coughs automatically isolated by XLS-R comfortably outperforms a model trained on the coughs isolated by AST, and is only narrowly outperformed by a classifier trained on the ground truth coughs. We conclude that the application of large pre-trained
transformer models is an effective approach to identifying cough end-points and that the integration of such a model into a screening tool is feasible.
Read the full abstract ▾
All authors Joshua Jansen van Vüren, Devendra Singh Parihar, Daphne Naidoo, Kimsey Zajac, Willy Ssengooba, Grant Theron, Thomas Niesler
Machine Learning
Monitoring and Prediction of Mood in Elderly People during Daily Life Activities
Daniel Bautista-Salinas, Joaquín Roca González, Inmaculada Méndez et al. (+1 more)
We present an intelligent wearable system to monitor and predict mood states of elderly people during their daily life activities. Our system is composed of a wristband to record different physiological activities together with a mobile app for ecological momentary assessment (EMA). Machine learning is used to train a classifier to automatically predict different mood states based on the smart band only. Our approach shows promising results on mood accuracy and provides results comparable with the state of the art in the specific detection of happiness and activeness.
Read the full abstract ▾
All authors Daniel Bautista-Salinas, Joaquín Roca González, Inmaculada Méndez, Oscar Martinez Mozos
Machine Learning · Cross-listed from stat.ML
Trustworthy Predictive Distributions for Tail Events with Semiparametric Diagnostic Transport Maps
Elizabeth Cucuzzella, Rafael Izbicki and Ann B. Lee
Machine learning forecast systems are moving beyond point predictions to full predictive distributions for future outcomes y conditional on complex inputs x. However, these distributions are often locally miscalibrated, especially for high-stakes tail events where accurate uncertainty quantification is most needed to establish trust in models. Local miscalibration occurs because training data often lack examples of low-frequency events. The goal of this paper is to describe a simple, yet flexible framework that, at deployment, produces interpretable diagnostics and a robust correction mechanism of predictive distributions when train examples are limited. With this goal in mind, we introduce a semiparametric version of the Local Amortized Diagnostic and Reshaping (LADaR) framework that posits a
covariate-dependent parametric model for a diagnostic transport map regressed nonparametrically on inputs to describe how to correct tail probabilities across the feature space to match calibration data. These maps provide the user with local, real-time diagnostics and a reshaped predictive distribution that can be related back to physical processes in the input space. We apply these semiparametric diagnostic transport maps to short-term tropical cyclone intensity
forecasting to detect evolutionary modes linked to local miscalibration in the National Hurricane Center's forecasts and improve predictions for severe weather hazards.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Markovian Generation Chains in Large Language Models
Mingmeng Geng, Amr Mohamed, Guokan Shang et al. (+2 more)
The widespread use of large
language models (LLMs) raises an important question: how do texts evolve when they are repeatedly processed by LLMs? In this paper, we define this iterative inference process as Markovian generation chains, where each step takes a specific
prompt template and the previous output as input, without including any prior data. In iterative rephrasing and round-trip translation experiments, the output either converges to a small recurrent set or continues to produce novel sentences within a finite horizon. Through sentence-level Markov chain modeling and analysis of simulated data, we show that iterative process can either increase or reduce the diversity depending on factors such as the temperature parameter and the initial input sentence. These results offer valuable insights into the dynamics of iterative LLM inference and their implications for multi-
agent LLM systems.
Read the full abstract ▾
All authors Mingmeng Geng, Amr Mohamed, Guokan Shang, Michalis Vazirgiannis, Thierry Poibeau
Machine Learning · Cross-listed from cs.AI
Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios
Linus Folkerts, Will Payne, Simon Inman et al. (+11 more)
We evaluate the autonomous cyber-attack capabilities of frontier AI models on two purpose-built cyber ranges-a 32-step corporate network attack and a 7-step industrial control system attack-that require chaining heterogeneous capabilities across extended action sequences. By comparing seven models released over an eighteen-month period (August 2024 to February 2026) at varying inference-time compute budgets, we observe two capability trends. First, model performance scales log-linearly with inference-time compute, with no observed plateau-increasing from 10M to 100M
tokens yields gains of up to 59%, requiring no specific technical sophistication from the operator. Second, each successive model generation outperforms its predecessor at fixed
token budgets: on the corporate network range, average steps completed at 10M
tokens rose from 1.7 (GPT-4o, August 2024) to 9.8 (Opus 4.6, February 2026). The best single run completed 22 of 32 steps, corresponding to roughly 6 of the estimated 14 hours a human expert would need. On the industrial control system range, performance remains limited, though the most recent models are the first to reliably complete steps, averaging 1.2-1.4 of 7 (max 3).
Read the full abstract ▾
All authors Linus Folkerts, Will Payne, Simon Inman, Philippos Giavridis, Joe Skinner, Sam Deverett, James Aung, Ekin Zorer, Michael Schmatz, Mahmoud Ghanem, John Wilkinson, Alan Steer, Vy Hong, Jessica Wang
Machine Learning · Cross-listed from cs.CR
Security-by-Design for LLM-Based Code Generation: Leveraging Internal Representations for Concept-Driven Steering Mechanisms
Maximilian Wendlinger, Daniel Kowatsch, Konstantin Böttinger et al. (+1 more)
Large
Language Models (LLMs) show remarkable capabilities in understanding natural
language and generating complex code. However, as practitioners adopt CodeLLMs for increasingly critical development tasks, research reveals that these models frequently generate functionally correct yet insecure code, posing significant security risks. While multiple approaches have been proposed to improve security in AI-based code generation, combined
benchmarks show these methods remain insufficient for practical use, achieving only limited improvements in both functional correctness and security. This stems from a fundamental gap in understanding the internal mechanisms of code generation and the root causes of security vulnerabilities, forcing researchers to rely on heuristics and empirical observations. In this work, we investigate the internal representation of security concepts in CodeLLMs, revealing that models are often aware of vulnerabilities as they generate insecure code. Through systematic evaluation, we demonstrate that CodeLLMs can distinguish between security subconcepts, enabling a more fine-grained analysis than prior black-box approaches. Leveraging these insights, we propose Secure Concept Steering for CodeLLMs (SCS-Code). During
token generation, SCS-Code steers LLMs' internal representations toward secure and functional code output, enabling a lightweight and modular mechanism that can be integrated into existing code models. Our approach achieves superior performance compared to state-of-the-art methods across multiple secure coding
benchmarks.
Read the full abstract ▾
All authors Maximilian Wendlinger, Daniel Kowatsch, Konstantin Böttinger, Philip Sperl
Machine Learning
Reference-Guided Machine Unlearning
Jonas Mirlach, Sonia Laguna and Julia E. Vogt
Machine unlearning aims to remove the influence of specific training data from a model while preserving its general utility. In
vision, many approximate unlearning methods pursue this goal through degradation-based heuristics, such as loss maximization or random labeling. Yet making a model worse on forget samples is not the same as making it behave as if those examples had never been seen: these signals can be poorly conditioned, destabilize optimization, and harm generalization. We argue that approximate unlearning should instead prioritize distributional indistinguishability, aligning the model's predictive behavior on forget data with that on truly unseen data. Motivated by this principle, we propose Reference-Guided Unlearning (ReGUn), a
vision unlearning framework that uses disjoint held-out data to construct a principled, class-conditioned reference distribution for
distillation. Rather than explicitly degrading predictions on forget examples, ReGUn guides them toward non-member behavior through held-out supervision. Across multiple architectures, natural image
datasets, and forget fractions, ReGUn consistently improves the forgetting-utility trade-off relative to standard approximate baselines while closely matching retrain-like membership inference behavior. As one instantiation of this principle, the results suggest that simple objectives designed around indistinguishability can provide an effective alternative to complex degradation-based unlearning procedures.
Read the full abstract ▾
Machine Learning
Representation Finetuning for Continual Learning
Haihua Luo, Xuming Ran, Tommi Kärkkäinen et al. (+4 more)
The world is inherently dynamic, and continual learning aims to enable models to adapt to ever-evolving data streams. While pre-trained models have shown powerful performance in continual learning, they still require finetuning to adapt effectively to downstream tasks. However, prevailing Parameter-Efficient Fine-Tuning (PEFT) methods operate through empirical, black-box optimization at the weight level. These approaches lack explicit control over representation drift, leading to sensitivity to domain shifts and catastrophic forgetting in continual learning scenarios. In this work, we introduce Continual Representation Learning (CoRe), a novel framework that for the first time shifts the finetuning paradigm from weight space to representation space. Unlike conventional methods, CoRe performs task-specific interventions within a low-rank linear subspace of hidden representations, adopting a learning process with explicit objectives, which ensures stability for past tasks while maintaining plasticity for new ones. By constraining updates to a low-rank subspace, CoRe achieves exceptional parameter efficiency. Extensive experiments across multiple continual learning
benchmarks demonstrate that CoRe not only preserves parameter efficiency but also significantly outperforms existing state-of-the-art methods. Our work introduces representation finetuning as a new, more effective and interpretable paradigm for continual learning.
Read the full abstract ▾
All authors Haihua Luo, Xuming Ran, Tommi Kärkkäinen, Huiyan Xue, Zhonghua Chen, Qi Xu, Fengyu Cong
Machine Learning · Cross-listed from cs.CR
DNS-GT: A Graph-based Transformer Approach to Learn Embeddings of Domain Names from DNS Queries
Massimiliano Altieri, Ronan Hamon, Roberto Corizzo et al. (+2 more)
Network intrusion detection systems play a crucial role in the security strategy employed by organisations to detect and prevent cyberattacks. Such systems usually combine pattern detection signatures with anomaly detection techniques powered by machine learning methods. However, the commonly proposed machine learning methods present drawbacks such as over-reliance on labeled data and limited generalization capabilities. To address these issues,
embedding-based methods have been introduced to learn representations from network data, such as DNS traffic, mainly due to its large availability, that generalise effectively to many downstream tasks. However, current approaches do not properly consider contextual information among DNS queries. In this paper, we tackle this issue by proposing DNS-GT, a novel
Transformer-based model that learns
embeddings for domain names from sequences of DNS queries. The model is first pre-trained in a self-supervised fashion in order to learn the general behavior of DNS activity. Then, it can be finetuned on specific downstream tasks, exploiting interactions with other relevant queries in a given sequence. Our experiments with real-world DNS data showcase the ability of our method to learn effective domain name representations. A quantitative evaluation on domain name classification and botnet detection tasks shows that our approach achieves better results compared to relevant baselines, creating opportunities for further exploration of large-scale
language models for intrusion detection systems. Our code is available at: https://github.com/m-altieri/DNS-GT.
Read the full abstract ▾
All authors Massimiliano Altieri, Ronan Hamon, Roberto Corizzo, Michelangelo Ceci, Ignacio Sanchez
Machine Learning
Bayesian Optimization of Partially Known Systems using Hybrid Models
Eike Cramer, Luis Kutschat, Oliver Stollenwerk et al. (+2 more)
Bayesian optimization (BO) has gained
attention as an efficient
algorithm for black-box optimization of expensive-to-evaluate systems, where the BO
algorithm iteratively queries the system and suggests new trials based on a probabilistic model fitted to previous samples. Still, the standard BO loop may require a prohibitively large number of experiments to converge to the optimum, especially for high-dimensional and nonlinear systems. We present a hybrid model-based BO formulation that combines the iterative
Bayesian learning of BO with partially known mechanistic physical models. Instead of learning a direct mapping from inputs to the objective, we write all known equations for a physics-based model and infer expressions for variables missing equations using a probabilistic model, in our case, a Gaussian process (GP). The final formulation then includes the GP as a constraint in the hybrid model, thereby allowing other physics-based nonlinear and implicit model constraints. This hybrid model formulation yields a constrained, nonlinear
stochastic program, which we discretize using the sample-average approximation. In an in-silico optimization of a single-stage
distillation, the hybrid BO model based on mass conservation laws yields significantly better designs than a standard BO loop. Furthermore, the hybrid model converges in as few as one iteration, depending on the initial samples, whereas, the standard BO does not converge within 25 for any of the seeds. Overall, the proposed hybrid BO scheme presents a promising optimization method for partially known systems, leveraging the strengths of both mechanistic modeling and data-driven optimization.
Read the full abstract ▾
All authors Eike Cramer, Luis Kutschat, Oliver Stollenwerk, Joel A. Paulson, Alexander Mitsos
Machine Learning · Cross-listed from cs.AI
PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence
Yuanda Xu, Hejian Sang, Zhengze Zhou et al. (+2 more)
Standard LLM
distillation treats all training problems equally – wasting compute on problems the student has already mastered or cannot yet solve. We empirically show that this inefficiency has a precise gradient-level signature: the cross-problem gradient signal-to-noise ratio (SNR) follows a bell curve over student pass rate, collapsing at both extremes. We propose PACED, which weights each problem by $w(p) = p(1{-}p)$ where $p$ is the student's empirical pass rate – concentrating training on the zone of proximal development. This requires only student rollouts, no architectural changes, and no hyperparameters. We prove the Beta kernel $w(p) = p^\alpha(1{-}p)^\beta$ is the leading-order optimal weight family arising from the SNR boundary-collapse structure, and is minimax-robust under misspecification (worst-case efficiency loss $O(\delta^2)$). Across Qwen3, Qwen2.5, and Llama-3 families, PACED sets a new state of the art in our experimental setting on MATH-500, AIME~2024, and AIME~2025, improving over unweighted
distillation by up to $\mathbf{+8.2}$ and over the strong AKL baseline by up to $\mathbf{+3.6}$, while reducing forgetting to $\mathbf{1.4\%}$ and $\mathbf{0.6\%}$ in
distillation and self-
distillation. A two-stage forward-then-reverse KL schedule pushes gains further to $\mathbf{+5.8}$ over standard forward KL on the hardest
benchmark.
Read the full abstract ▾
All authors Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang
Machine Learning
Huntington Disease Automatic Speech Recognition with Biomarker Supervision
Charles L. Wang, Cady Chen, Ziwei Gong et al. (+1 more)
Automatic
speech recognition (ASR) for pathological
speech remains underexplored, especially for Huntington's disease (HD), where irregular timing, unstable phonation, and articulatory distortion challenge current models. We present a systematic HD-ASR study using a high-fidelity clinical
speech corpus not previously used for end-to-end ASR training. We compare multiple ASR families under a unified evaluation, analyzing WER as well as substitution, deletion, and insertion patterns. HD
speech induces architecture-specific error regimes, with Parakeet-TDT outperforming encoder-decoder and CTC baselines. HD-specific adaptation reduces WER from 6.99% to 4.95% and we also propose a method for using biomarker-based auxiliary supervision and analyze how error behavior is reshaped in severity-dependent ways rather than uniformly improving WER. We open-source all code and models.
Read the full abstract ▾
All authors Charles L. Wang, Cady Chen, Ziwei Gong, Julia Hirschberg
Machine Learning · Cross-listed from hep-th
Learning to Unscramble: Simplifying Symbolic Expressions via Self-Supervised Oracle Trajectories
David Shih
We present a new self-supervised machine learning approach for symbolic simplification of complex mathematical expressions. Training data is generated by scrambling simple expressions and recording the inverse operations, creating oracle trajectories that provide both goal states and explicit paths to reach them. A permutation-equivariant,
transformer-based policy network is then trained on this data step-wise to predict the oracle action given the input expression. We demonstrate this approach on two problems in high-energy physics: dilogarithm reduction and spinor-helicity scattering amplitude simplification. In both cases, our trained policy network achieves near perfect solve rates across a wide range of difficulty levels, substantially outperforming prior approaches based on reinforcement learning and end-to-end regression. When combined with contrastive grouping and beam search, our model achieves a 100% full simplification rate on a representative selection of 5-point
gluon tree-level amplitudes in Yang-Mills theory, including expressions with over 200 initial terms.
Read the full abstract ▾
Machine Learning
Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers
Orit Davidovich and Zohar Ringel
We formally define algorithmic capture of combinatorial tasks as the ability of a
transformer to extrapolate to arbitrary task sizes with controllable error and logarithmic sample adaptation, providing a sharp scaling criterion for distinguishing logic internalization from statistical interpolation. Empirically, across scaling ranges spanning up to 2.5 orders of magnitude, we observe evidence of capture and non-capture. By analyzing infinite-width
transformers in both the lazy and rich regimes, we derive upper bounds on the inference-time computational complexity of the combinatorial tasks these networks can capture. We show that, despite their universal expressivity,
transformers possess an inductive bias that disfavors higher-complexity algorithmic procedures within the efficient
polynomial-time heuristic scheme class, consistent with successful capture on simpler combinatorial tasks such as induction heads, sort, and string matching.
Read the full abstract ▾
Machine Learning
Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models
Xiangwen Wang, Ananth Balashankar and Varun Chandrasekaran
Large
language models remain vulnerable to jailbreak attacks, yet we still lack a systematic understanding of how jailbreak success scales with attacker effort across methods, model families, and harm types. We initiate a scaling-law framework for jailbreaks by treating each attack as a compute-bounded optimization procedure and measuring progress on a shared FLOPs axis. Our systematic evaluation spans four representative jailbreak paradigms, covering optimization-based attacks, self-refinement prompting, sampling-based selection, and genetic optimization, across multiple model families and scales on a diverse set of harmful goals. We investigate scaling laws that relate attacker budget to attack success score by fitting a simple saturating exponential function to FLOPs–success trajectories, and we derive comparable efficiency summaries from the fitted curves. Empirically, prompting-based paradigms tend to be the most compute-efficient compared to optimization-based methods. To explain this gap, we cast
prompt-based updates into an optimization view and show via a same-state comparison that
prompt-based attacks more effectively optimize in
prompt space. We also show that attacks occupy distinct success–stealthiness operating points with prompting-based methods occupying the high-success, high-stealth region. Finally, we find that vulnerability is strongly goal-dependent: harms involving misinformation are typically easier to elicit than other non-misinformation harms.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.MM
Catalogue-Grounded Multimodal Attribution for Museum Video under Resource and Regulatory Constraints
Minsak Nanang, Adrian Hilton and Armin Mustafa
Museum audiovisual archives are important records of visual culture, yet in-gallery
video remains difficult to search because it lacks catalogue-compatible metadata. We propose CaGMulA, catalogue-grounded
multimodal attribution for museum
video: a locally deployable pipeline that combines
video-
language models, structured catalogue
retrieval, and abstention-aware decision policies. The system summarises visible artworks, generates catalogue-style descriptions, and attributes title and artist only when evidence is sufficient. We evaluate VideoLLaMA2 and Qwen2-VL across four model configurations and four attribution modes on 18 in-gallery
videos. Open-ended generation fails under conservative conditions, producing 0% coverage across models. By contrast, closed-set catalogue
retrieval reaches 100% coverage with 62% accepted title–artist correctness, while a verifier-gated hybrid mode reduces coverage to 50% but improves accepted correctness to 78%. The results show that reliable museum-
video attribution is driven less by unrestricted generation than by catalogue grounding, visual
retrieval, and selective abstention for curatorial review.
Read the full abstract ▾
Machine Learning
Attention Gathers, MLPs Compose: A Causal Analysis of an Action-Outcome Circuit in VideoViT
Sai V R Chereddy
The paper explores how
video models trained for classification tasks represent nuanced, hidden semantic information that may not affect the final outcome, a key challenge for Trustworthy AI models. Through Explainable and Interpretable AI methods, specifically mechanistic interpretability techniques, the internal
circuit responsible for representing the action's outcome is reverse-engineered in a pre-trained
video vision transformer, revealing that the "Success vs Failure" signal is computed through a distinct amplification cascade. While there are low-level differences observed from layer 0, the abstract and semantic representation of the outcome is progressively amplified from layers 5 through 11. Causal analysis, primarily using activation patching supported by ablation results, reveals a clear division of labor:
Attention Heads act as "evidence gatherers", providing necessary low-level information for partial signal recovery, while MLP Blocks function as robust "concept composers", each of which is the primary driver to generate the "success" signal. This distributed and redundant
circuit in the model's internals explains its resilience to simple ablations, demonstrating a core computational pattern for processing human-action outcomes. Crucially, the existence of this sophisticated
circuit for representing complex outcomes, even within a model trained only for simple classification, highlights the potential for models to develop forms of 'hidden knowledge' beyond their explicit task, underscoring the need for mechanistic oversight for building genuinely Explainable and Trustworthy AI systems intended for deployment.
Read the full abstract ▾
Machine Learning
Procedural Fairness via Group Counterfactual Explanation
Gideon Popoola and John Sheppard
Fairness in machine learning research has largely focused on outcome-oriented
fairness criteria such as Equalized Odds, while comparatively less
attention has been given to procedural-oriented
fairness, which addresses how a model arrives at its predictions. Neglecting procedural
fairness means it is possible for a model to generate different explanations for different protected groups, thereby eroding trust. In this work, we introduce Group Counterfactual Integrated Gradients (GCIG), an in-processing regularization framework that enforces explanation invariance across groups, conditioned on the true label. For each input, GCIG computes explanations relative to multiple Group Conditional baselines and penalizes cross-group variation in these attributions during training. GCIG formalizes procedural
fairness as Group Counterfactual explanation stability and complements existing
fairness objectives that constrain predictions alone. We compared GCIG empirically against six state-of-the-art methods, and the results show that GCIG substantially reduces cross-group explanation disparity while maintaining competitive predictive performance and accuracy-
fairness trade-offs. Our results also show that aligning model
reasoning across groups offers a principled and practical avenue for advancing
fairness beyond outcome parity.
Read the full abstract ▾
Machine Learning
H2LooP Spark Preview: Continual Pretraining of Large Language Models for Low-Level Embedded Systems Code
Amit Singh, Vedant Nipane, Pulkit Agrawal et al. (+2 more)
Large
language models (LLMs) demonstrate strong code generation abilities in general-purpose programming
languages but remain limited in specialized domains such as low-level embedded systems programming. This domain involves hardware register manipulation, vendor-specific SDKs, real-time operating system APIs, and hardware abstraction layers that are underrepresented in standard pretraining corpora. We introduce H2LooP Spark Preview, a continual pretraining (CPT) pipeline that adapts the OLMo-3-7B-a fully open
language model to the embedded systems domain using BF16 LoRA with rank-stabilized scaling on 8 NVIDIA H100 GPUs. Our training corpus is constructed from repository-datasheet pairs covering 100B
tokens of raw embedded systems data across 117 manufacturers, processed using the hierarchical datasheet-to-code mapping approach proposed in SpecMap (Nipane et al., 2026). The resulting curated
dataset split contains 23.5B
tokens across 13 embedded domains. Continual pretraining with high-rank LoRA (r=512) yields substantial gains, reducing in-domain perplexity by 70.4% and held-out repository perplexity by 66.1%. On generative code completion
benchmarks spanning 13 embedded domains, our 7B model outperforms Claude Opus 4.6 and Qwen3-Coder-30B on 8 categories in
token accuracy, showing that targeted continual pretraining enables smaller open-weight models to rival frontier systems on specialized technical tasks. We release the production training checkpoint on Huggingface as an open-source artifact.
Read the full abstract ▾
All authors Amit Singh, Vedant Nipane, Pulkit Agrawal, Jatin Kishnani, Sairanjan Mishra
Machine Learning · Cross-listed from stat.ML
Deep regression learning from dependent observations with minimum error entropy principle
William Kengne and Modou Wade
This paper considers nonparametric regression from strongly mixing observations. The proposed approach is based on deep neural networks with minimum error
entropy (MEE) principle. We study two
estimators: the non-penalized deep neural network (NPDNN) and the sparse-penalized deep neural network (SPDNN) predictors. Upper bounds of the expected excess risk are established for both
estimators over the classes of Hölder and composition Hölder functions. For the models with Gaussian error, the rates of the upper bound obtained match (up to a logarithmic factor) with the lower bounds established in \cite{schmidt2020nonparametric}, showing that both the MEE-based NPDNN and SPDNN
estimators from strongly mixing data can achieve the minimax optimal convergence rate.
Read the full abstract ▾
Machine Learning
Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
Jongwoo Ko, Sara Abdali, Young Jin Kim et al. (+2 more)
On-policy
distillation (OPD) is a promising approach for transferring
reasoning capabilities to capacity-constrained models, yet sampled-
token OPD often suffers from
entropy collapse and negative transfer. We identify uniform credit allocation as a key bottleneck: existing objectives obtain dense teacher feedback on student-generated trajectories, but apply it uniformly across sampled
tokens despite highly heterogeneous
token-level signals. Most
tokens carry redundant near-zero credit, while rare heavy-tailed negative credits can dominate updates and prematurely suppress plausible
reasoning trajectories. We introduce REOPOLD (Relaxed On-Policy
Distillation), a framework that relaxes this uniform assignment by controlling where teacher feedback is applied, how strongly it affects the update, and when the allocation rule changes during training. Across diverse
reasoning tasks, REOPOLD improves sampled-
token OPD over recent post-training baselines with up to 12x higher sample efficiency, and extends to cross-vocabulary
distillation, self-
distillation, and test-time scaling.
Read the full abstract ▾
All authors Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron
Machine Learning · Cross-listed from math.ST
Conformal e-prediction in the presence of confounding
Vladimir Vovk and Ruodu Wang
This note extends conformal e-prediction to cover the case where there is observed confounding between the random object $X$ and its label $Y$. We consider both the case where the observed data is IID and a case where some dependence between observations is permitted.
Machine Learning
Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning
Shirin Amiraslani and Xin Gao
Sequence modeling tasks can involve intrinsic higher-order dependencies, while standard self-
attention assigns scores to
token pairs and does not explicitly parameterize such interactions. We introduce Higher-Order Modular
Attention (HOMA), which fuses pairwise
attention with an explicit triadic
attention pathway made tractable through overlapping blocks, local windows, and a low-rank projection. We compare HOMA with matched pairwise and purely triadic baselines on controlled PARITY and MATCH3 tasks, as well as TAPE
benchmarks. HOMA is competitive with or outperforms the baselines, with its clearest advantages when the underlying dependencies extend beyond the explicitly modeled triadic order. These advantages are accompanied in several settings by faster convergence and improved parameter efficiency, with the learned nonlinear fusion providing an effective mechanism for combining the pairwise and triadic representations. Overall, our results provide empirical evidence that HOMA is an effective
attention design when task structure extends beyond pairwise interactions.
Read the full abstract ▾
Machine Learning
Beyond Barren Plateaus: A Scalable Quantum Convolutional Architecture for High-Fidelity Image Classification
Radhakrishnan Delhibabu
While
Quantum Convolutional Neural Networks (QCNNs) offer a theoretical paradigm for
quantum machine learning, their practical implementation is severely bottlenecked by barren plateaus – the exponential vanishing of gradients – and poor empirical accuracy compared to classical counterparts. In this work, we propose a novel QCNN architecture utilizing localized cost functions and a hardware-efficient tensor-network initialization strategy to provably mitigate barren plateaus. We evaluate our scalable QCNN on the MNIST
dataset, demonstrating a significant performance leap. By resolving the gradient vanishing issue, our optimized QCNN achieves a classification accuracy of 98.7%, a substantial improvement over the baseline QCNN accuracy of 52.32% found in unmitigated models. Furthermore, we provide empirical evidence of a parameter-efficiency advantage, requiring $\mathcal{O}(\log N)$ fewer trainable parameters than equivalent classical CNNs to achieve $>95\%$ convergence. This work bridges the gap between theoretical
quantum utility and practical application, providing a scalable framework for
quantum computer
vision tasks without succumbing to loss landscape concentration.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks
ZeYu Li, FengLei Fan and TieYong Zeng
This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and $L^p$ functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general $L^p$ functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.
Read the full abstract ▾