Machine Learning · Cross-listed from cs.CV
VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generations
Maitreya Patel, Jingtao Li, Weiming Zhuang et al. (+2 more)
We introduce an efficient, resolution-agnostic autoregressive (AR) image synthesis approach that generalizes to arbitrary resolutions and aspect ratios, narrowing the gap to
diffusion models at scale. At its core is VibeToken, a novel resolution-agnostic 1D
Transformer-based image tokenizer that encodes images into a dynamic, user-controllable sequence of 32-256
tokens, achieving a state-of-the-art efficiency and performance trade-off. Building on VibeToken, we present VibeToken-Gen, a class-conditioned AR generator with out-of-the-box support for arbitrary resolutions while requiring significantly fewer compute resources. Notably, VibeToken-Gen synthesizes 1024x1024 images using only 64
tokens and achieves 3.94 gFID; by comparison, a
diffusion-based state-of-the-art alternative requires 1,024
tokens and attains 5.87 gFID. In contrast to fixed-resolution AR models such as LlamaGen – whose inference FLOPs grow quadratically with resolution (11T FLOPs at 1024x1024) – VibeToken-Gen maintains a constant 179G FLOPs (63.4x efficient) independent of resolution. We hope VibeToken can help unlock the wide adoption of AR visual generative models in production use cases.
Read the full abstract ▾
All authors Maitreya Patel, Jingtao Li, Weiming Zhuang, Yezhou Yang, Lingjuan Lv
Machine Learning · Cross-listed from cond-mat.quant-gas
Uncovering Exotic Paired States in the 2D Spin-Imbalanced Fermi Gas with Neural Wave Functions
Wan Tong Lou, Gino Cassella, Andres Perez Fadon et al. (+5 more)
We study the zero-temperature phase diagram of the 2D
spin-imbalanced
Fermi gas with short-ranged attractive interactions using the recently developed neural network variational Monte Carlo method with the AGPs FermiNet Ansatz. The Fulde-Ferrell-Larkin-Ovchinnikov phase is observed in the weakly interacting BCS limit and a polarised superfluid is seen in the strongly interacting BEC limit. When the interactions are strong, the minority-
spin momentum density is reduced almost to zero in the momentum-space region occupied by the unpaired majority-
spin electrons. When the interactions are very strong, phase separation occurs, with regions containing bosonic pairs and unpaired regions occupied by the remaining majority-
spin particles. In addition, we observe translational
symmetry breaking at intermediate interaction strengths, where the system forms an exotic
crystal of Cooper pairs in a
Fermi fluid of unpaired majority-
spin particles. We provide a possible explanation for the formation of the crystalline phase, explain the origins of the k-space momentum-density hole when the pairs are tightly bound, and discuss how our approach opens new directions for future work.
Read the full abstract ▾
All authors Wan Tong Lou, Gino Cassella, Andres Perez Fadon, Halvard Sutterud, David Pfau, James S. Spencer, Johannes Knolle, W.M.C. Foulkes
Machine Learning · Cross-listed from eess.SP
Monitoring exposure-length variations in submarine power cables using distributed fiber-optic sensing
Sakiko Mishima, Yoshiyuki Yajima, Noriyuki Tonami et al. (+4 more)
This study proposes an anomaly-detection framework for monitoring exposure-length variations in submarine free-span cables using Distributed Acoustic Sensing (DAS), which is one of the distributed fiber-optic sensing technologies. To address environmental variability and limited training data in offshore environments, a regression-based feature extraction method was introduced to derive low-dimensional latent representations that retain exposure length-dependent vibration characteristics while suppressing environmental influences. The extracted features were used for one-class Support Vector Machine (SVM)-based anomaly detection. The proposed framework was evaluated through wave-tank experiments with exposure lengths ranging from 2 to 10 m. Experimental results showed that anomaly scores decreased approximately monotonically with increasing exposure-length change, exhibiting a strong correlation ($r = -0.83$). The binary classification achieved an F1 score of 0.82 despite training with only small-sample
datasets. These findings demonstrate that exposure-length variations can be reliably detected under severe data limitations, supporting the potential of DAS-based cable condition monitoring.
Read the full abstract ▾
All authors Sakiko Mishima, Yoshiyuki Yajima, Noriyuki Tonami, Tomoyuki Hino, Shugo Aibe, Junichiro Saikawa, Koji Mizuguchi
Machine Learning
Transformer Approximations from ReLUs
Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee et al. (+1 more)
We provide a systematic recipe for translating ReLU approximation results to softmax
attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax
transformer models.
Read the full abstract ▾
All authors Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee, Han Liu
Machine Learning · Cross-listed from cs.CV
Learning Illumination Control in Diffusion Models
Nishit Anand, Manan Suri, Christopher Metzler et al. (+2 more)
Controlling illumination in images is essential for photography and visual content creation. While closed-source models have demonstrated impressive illumination control, open-source alternatives either require heavy control inputs like depth maps or do not release their data and code. We present a fully open-source and reproducible pipeline for learning illumination control in
diffusion models. Our approach builds a data engine that transforms well-lit images into supervised training triplets consisting of a poorly-illuminated input image, a natural
language lighting instruction, and a well-illuminated output image. We finetune a
diffusion model on this data and demonstrate significant improvements over baseline SD 1.5, SDXL, and FLUX.1-dev models in perceptual similarity, structural similarity, and identity preservation. Our work provides a reproducible solution built entirely with open-source tools and publicly available data. We release all our code, data, and model weights publicly.
Read the full abstract ▾
All authors Nishit Anand, Manan Suri, Christopher Metzler, Dinesh Manocha, Ramani Duraiswami
Machine Learning · Cross-listed from cs.RO
MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives
Tingwu Wang, Olivier Dionne, Michael De Ruyter et al. (+13 more)
Despite transformative advances in generative motion synthesis, real-time interactive motion control remains dominated by traditional techniques. In this work, we identify two key challenges in bridging research and production: 1) Real-time scalability: Industry applications demand real-time generation of a vast repertoire of motion skills, while generative methods exhibit significant degradation in quality and scalability under real-time computation constraints, and 2) Integration: Industry applications demand fine-grained multi-modal control involving velocity commands, style selection, and precise keyframes, a need largely unmet by existing text- or tag-driven models. To overcome these limitations, we introduce MotionBricks: a large-scale, real-time generative framework with a two-fold solution. First, we propose a large-scale modular latent generative backbone tailored for robust real-time motion generation, effectively modeling a
dataset of over 350,000 motion clips with a single model. Second, we introduce smart primitives that provide a unified, robust, and intuitive interface for authoring both
navigation and object interaction. Applications can be designed in a plug-and-play manner like assembling bricks without expert animation knowledge. Quantitatively, we show that MotionBricks produces state-of-the-art motion quality on open-source and proprietary
datasets of various scales, while also achieving a real-time throughput of 15,000 FPS with 2ms latency. We demonstrate the flexibility and robustness of MotionBricks in a complete production-level animation demo, covering
navigation and object-scene interaction across various styles with a unified model. To showcase our framework's application beyond animation, we deploy MotionBricks on the Unitree G1
humanoid robot to demonstrate its flexibility and generalization for real-time robotic control.
Read the full abstract ▾
All authors Tingwu Wang, Olivier Dionne, Michael De Ruyter, David Minor, Davis Rempe, Kaifeng Zhao, Mathis Petrovich, Ye Yuan, Chenran Li, Zhengyi Luo, Brian Robison, Xavier Blackwell, Bernardo Antoniazzi, Xue Bin Peng, Yuke Zhu, Simon Yuen
Machine Learning
On the Trainability of Masked Diffusion Language Models via Blockwise Locality
Yuxiang Wang, Yu Xiang, Baojian Zhou et al. (+4 more)
Masked
diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large
language models (AR-LLMs), yet their optimization can be substantially less stable. We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation: in-context linear regression, graph path-finding, and Sudoku solving. We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku. To mitigate these instabilities, we propose two locality aware blockwise models, namely Jigsaw and Scatter, that inject left-to-right inductive bias by enforcing autoregressive locality within blocks while preserving iterative refinement at the block level. Empirically, Jigsaw matches AR-LLM stability on linear regression and remains strong on Sudoku, while Scatter retains
diffusion's planning advantage on path-finding. Our results indicate that standard random-masking MDMs, even with blockwise variants, may be a suboptimal instantiation of
diffusion LMs for ordered generation, motivating models beyond random masking.
Read the full abstract ▾
All authors Yuxiang Wang, Yu Xiang, Baojian Zhou, Qifang Zhao, Keyue Jiang, Yanghua Xiao, Xiaoxiao Xu
Machine Learning · Cross-listed from cs.SE
FGDM: Reasoning Aware Multi-Agentic Framework for Software Bug Detection using Chain of Thought and Tree of Thought Prompting
Srita Padmanabhuni, Bhargavi Karuturi, Jerusha Karen Indupalli et al. (+2 more)
Deep Learning methods are becoming prominent in automated software bug detection; however, they lack the global understanding of the given code. Consequently, their performance tends to degrade, especially when they are applied to large interconnected code bases or complex modular programs. Recently, Large
Language Models (LLMs) have proven to be effective at capturing dependencies among multiple interconnected modules in the codebase. This motivated us to propose the Flow-Graph-Driven Multi-
Agent Framework (FGDM), which is composed of four
agents that operate in a sequential manner. The framework converts the received code to a flow graph, identifies the erroneous segments, and further generates the repaired code. All the employed
agents utilize Chain-of-Thought (COT) and Tree-of-Thoughts (TOT)
prompts. Additionally, we also integrated with the FAISS vector database to retrieve similar previous bugs and their repairs. We demonstrated the efficacy of the proposed framework over 100 programs from several projects, including Ansible, Black, FastAPI, Keras, Luigi, Matplotlib, Pandas, Scrapy, SpaCy, and Tornado in both C and Python programs. Our experiments demonstrate that the FGDM outperforms the extant approaches and yielded reductions with a mean of 24.33 and 8.37 in Levenshtein distance and similarities of 0.951 and 0.974 in cosine similarity for Python and C, respectively.
Read the full abstract ▾
All authors Srita Padmanabhuni, Bhargavi Karuturi, Jerusha Karen Indupalli, Santhan Reddy Chilla, Vivek Yelleti
Machine Learning
Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity
Bojie Li
Closed-source frontier labs do not disclose parameter counts. Storing F facts requires at least F/(bits per parameter) weights, so factual recall lower-bounds parameter count–an intrinsic, serving-independent signal, though (as we show) a coarse one. We introduce Incompressible Knowledge Probes (IKPs), a
benchmark of 1,400 factual questions spanning 7 tiers of obscurity, designed to isolate knowledge that cannot be derived by
reasoning or compressed by architectural improvements. We score with no
hallucination penalty (lambda = 0: IKP accuracy is simply the fraction of probed facts answered correctly), which removes both the penalty hyperparameter and the per-tier flooring choice; a full lambda x flooring ablation shows the calibration is robust across scoring choices while individual estimates are not, motivating the no-penalty default. We calibrate a log-linear mapping from IKP accuracy to parameter count on 93 open-weight models (135M-1,600B) spanning 19 vendors, achieving R^2 = 0.910; leave-one-out cross-validation confirms generalization (median fold error 1.48x, 72% within 2x, 86% within 3x). The instrument is deliberately coarse–its 90% prediction interval spans ~3x in either direction, wider than inference economics–so IKP recovers order-of-magnitude effective capacity and relative rankings, not precise parameter counts. For Mixture-of-Experts models, total parameters predict knowledge (R^2 = 0.67) better than active parameters (R^2 = 0.41). We evaluate 201 models from 27 vendors on a curated probe set (1,311 of 1,400 probes surviving name-collision and label-ambiguity filters) and report effective knowledge capacity for all major proprietary frontier models as prediction bands rather than point estimates; for heavily safety-tuned models these are lower bounds, since refusal policy can suppress tens of percentage points of otherwise-answerable capacity.
Read the full abstract ▾
Machine Learning
Negative Ontology of True Target for Machine Learning: Towards Recognition, Evaluation and Learning under Democratic Supervision
Yongquan Yang
This article philosophically examines how a shift in the assumed ontology of the true target (TT) can lead to a new paradigm for machine learning (ML)-based predictive modelling. By systematically analysing the existence assumption of the TT underlying mainstream ML paradigms, we adopt a negative ontology perspective, explicitly positing that the TT does not objectively exist in the real world as a universally accessible object. On this basis, we define Democratic Supervision as an alternative supervisory principle for ML, in which no single source is assumed to possess an objectively privileged target. We further introduce Multiple Inaccurate True Targets (MIATTs) as an instance-level realization of Democratic Supervision. Building upon MIATTs, we establish the logic-driven generation and assessment for MIATTs construction (recognition with MIATTs), formulate logical assessment formula for evaluation with MIATTs, and develop undefinable true target learning for learning with MIATTs. These components are integrated to formulate the Recognition, Evaluation, Learning with MIATTs (REL-MIATTs) framework. We further characterize REL-MIATTs from the perspective of Cognitive Machine Learning: a single cycle provides an elementary cognitive learning unit, MIATTs introduce complementary supervisory perspectives, and iterative cycles support the
evolution of learning through accumulated experience, feedback, and state updating. This provides a basis for human-AI co-
evolution. We examine how REL-MIATTs support human-AI co-
evolution and adaptive knowledge discovery in a synthetic controlled environment. A real-world application further demonstrates the potential of the framework for supporting individual education and professional development, providing empirical evidence for the feasibility of Democratic Supervision, REL-MIATTs, and its broader implications for continuous human-AI co-
evolution.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SE
A systematic literature Review for Transformer-based Software Vulnerability detection
Fiza Naseer, Javed Ali Khan, Muhammad Yaqoob et al. (+2 more)
Context: Software vulnerabilities pose significant security threats to software systems, especially as software is increasingly used across many areas of daily life, including health, government, and finance. Recently,
transformer-based models have demonstrated promising results in automatic software vulnerability identification due to their robust contextual modelling and representation learning capabilities. Objectives: While numerous systematic literature reviews (SLRs) have examined machine learning and deep learning methods for identifying vulnerabilities, a more
transformer-centric analysis remains to be explored. This SLR critically analysed 80 studies published between 2021 and 2025 that utilised
transformer models to identify software vulnerabilities. Methods: Using Kitchenhams SLR guidelines, we methodically evaluate current research from various perspectives, encompassing study trends,
datasets and sources, programming
languages,
transformer frameworks, detection detail levels, assessment metrics, reference models, types of vulnerabilities, and experimental configurations. Results: We classify
transformer models into encoder, decoder, and combined architectures and analyse both pre-trained and fine-tuned versions utilized on source code, logs, and smart contracts. The results emphasise prevailing research trends, frequently utilised
benchmarks, and main baselines. It also uncovers crucial technical issues like data imbalance, interpretability, scalability, and generalization across programming
languages. Conclusion: By integrating current evidence and recognising unaddressed research areas, this SLR provides a consolidated resource for researchers and professionals seeking to develop more reliable, precise, and interpretable
transformer-based vulnerability identification systems.
Read the full abstract ▾
All authors Fiza Naseer, Javed Ali Khan, Muhammad Yaqoob, Alexios Mylonas, Ishaya Gambo
Machine Learning
Heterogeneous Variational Inference for Markov Degradation Hazard Models: Discretized Mixture with Interpretable Clusters
Takato Yasuno
Bayesian finite mixture models can identify discrete risk clusters (low-risk vs. high-risk equipment), but face three critical bottlenecks: (1) insufficient degradation signals from coarse state discretization, (2) unstable cluster identification when data inherently supports fewer clusters than explored, and (3) computational infeasibility of Markov Chain Monte Carlo (MCMC) methods for production deployment (7+ hours per model). We propose a practical framework combining (1) 8-state global percentile discretization that amplifies degradation events, (2) 30-dimensional feature engineering integrating statistical trends (22 features), continuous health indicators, and text
embeddings (PCA-compressed to 3 dimensions), (3) interpretable model selection rules enforcing minimum cluster share and separation alongside WAIC, and (4) Automatic Differentiation Variational Inference (ADVI) with full-rank covariance for stable, fast estimation. Applied to 280 industrial pump equipment with 104,703 inspection records, we demonstrate: (1) Random effect models (baseline) show ADVI and NUTS produce nearly identical estimates with 15$\times$ speedup, validating ADVI accuracy. (2) Finite mixture models identify optimal number of clusters with interpretability constraints. (3) NUTS exhibits severe convergence issues and label switching, while ADVI provides stable results in 84$\times$ less time. We contributed that (1) First demonstration that fine-grained state discretization (8-state) is essential for mixture model stability in survival analysis.(2) Comprehensive feature engineering strategy combining statistical, continuous, and semantic signals. (3) Practical interpretability rules preventing overfitting in automated model selection. (4) Empirical evidence that ADVI outperforms NUTS for finite mixture models in terms of convergence, stability, and computational efficiency.
Read the full abstract ▾
Machine Learning
Time-varying Interaction Graph ODE for Dynamic Graph Representation Learning
Xiaoyi Wang, Zhiqiang Wang, Jianqing Liang et al. (+4 more)
Graph neural Ordinary Differential Equations (ODE) combine neural ODE with the message passing mechanism of Graph Neural Networks (GNN), providing a continuous-time modeling method for graph representation learning. However, in dynamic graph scenarios, existing graph neural ODEs typically employ a unified message passing mechanism, assuming that inter-node interactions share the same message passing function at any time, which makes it challenging to capture the diversity and time-varying nature of inter-node interaction patterns. To address this, we propose Time-varying Interaction Graph Ordinary Differential Equations (TI-ODE). The core idea of TI-ODE is to decompose the
evolution function of a graph ODE into a set of learnable interaction basis functions, where each basis function corresponds to a distinct type of inter-node interaction. These basis functions are dynamically combined through time-dependent learnable weights, enabling inter-node interaction patterns to adaptively evolve over time. Experimental results on six dynamic graph
datasets demonstrate that TI-ODE consistently outperforms existing methods and achieves state-of-the-art performance on attribute prediction tasks, and experiments on the Covid
dataset further verify the interpretability and generalizability of our TI-ODE. Furthermore, we demonstrate both theoretically and empirically that TI-ODE exhibits superior robustness compared to models utilizing a unified message-passing mechanism.
Read the full abstract ▾
All authors Xiaoyi Wang, Zhiqiang Wang, Jianqing Liang, Xingwang Zhao, Chuangyin Dang, Zhen Jin, Jiye Liang
Machine Learning
A Comparative Analysis on the Performance of Upper Confidence Bound Algorithms in Adaptive Deep Neural Networks
Grigorios Papanikolaou, Ioannis Kontopoulos and Konstantinos Tserpes
Edge computing environments impose strict constraints on energy consumption and latency, making the deployment of deep neural networks a significant challenge. Therefore, smart and adaptive inference strategies that dynamically balance computational cost or latency with predictive accuracy are critical in edge computing scenarios. In this work, we build on Adaptive Deep Neural Networks (ADNNs) that employ the Multi-Armed Bandit (MAB) framework. Current literature leverages the first version of the Upper Confidence Bound (UCB1) strategy to dynamically select the optimal confidence threshold, enabling efficient early exits without sacrificing accuracy. However, we introduce four additional Upper Confidence Bound strategies in ADNNs, namely UCB-V, UCB-Tuned, UCB-Bayes, and UCB-BwK, and perform, for the first time, a comparative study of these strategies with respect to trade-offs between accuracy, energy consumption, and latency. The proposed UCB strategies are employed on the ResNet and MobileViT neural networks, and are evaluated on the
benchmark datasets of CIFAR-10, CIFAR-10.1, and CIFAR-100. Experimental results demonstrate that all strategies achieve sub-linear cumulative regret, with UCB-Bayes converging the fastest, followed by UCB-Tuned and UCB-V. Finally, UCB-V and UCB-Tuned dominate the Pareto Frontiers of accuracy-latency and accuracy-energy trade-offs. The implementation code is available here: https://github.com/gr3gor1/MAB_UCB
Read the full abstract ▾
Machine Learning
Maixent Chenebaux
We present Nautile-370M, a 371-million-parameter small
language model designed for efficient
reasoning under strict parameter and inference budgets. Nautile-370M uses a hybrid backbone in which two SeqCond
Attention (SCA) layers, a linear-time spectral sequence operator inspired by SeqCondenser, alternate with one
transformer layer. This design aims to retain the long-context efficiency and state-tracking benefits of structured sequential models while preserving the expressive
token-to-
token routing of
attention. The model was trained on a single Cloud TPU v4-64 pod slice provided through the Google TPU Research Cloud (TRC) program; the subsequent reinforcement learning stage was carried out on a single NVIDIA DGX Spark. We prove that the SCA readout mechanism can exactly retrieve any individual
token from the prefix summary and can reproduce any output of softmax
attention as a special case, establishing that SCA is at least as expressive as full self-
attention in the continuous limit. We also describe the training data pipeline and outline a reinforcement learning stage specialized for
reasoning, verification, and response quality.
Read the full abstract ▾
Machine Learning
minAction.net: Energy-First Neural Architecture Design – From Biological Principles to Systematic Validation
Martin G. Frasch
Modern machine learning optimizes for accuracy without explicit treatment of internal computational cost, even though physical and biological systems operate under intrinsic energy constraints. We evaluate energy-aware learning across 2,203 experiments spanning
vision, text, neuromorphic, and physiological
datasets with 10 seeds per configuration and factorial statistical analysis. Three findings emerge. First, architecture alone explains negligible variance in accuracy (partial eta^2 = 0.001), while the architecture x
dataset interaction is large (partial eta^2 = 0.44, p < 0.001), demonstrating that optimal architecture depends critically on task modality and rejecting the assumption of a universal best architecture. Second, a controlled lambda-sweep across lambda in {0, 1e-5, 1e-4, 1e-3, 1e-2} validates a single-parameter energy-regularized objective L = L_CE + lambda * E(theta, x): across this range, internal activation energy decreases by approximately three orders of magnitude relative to the unregularized lambda=0 baseline, with negligible accuracy change (<0.5 percentage points) on both MNIST and Fashion-MNIST. Third, energy-first architectures inspired by an action-principle framework yield 5-33% within-modality training-efficiency gains over conventional baselines. These results emerge from a research program that interprets learning through a structural correspondence between the action functional in classical mechanics, free energy in statistical physics, and KL-regularized objectives in variational inference. We frame this correspondence as a design hypothesis, not a derivation.
Read the full abstract ▾
Machine Learning
Intrinsic Mutual Information as a Modulator for Preference Optimization
Peng Liao, Peijia Zheng, Lingbo Li et al. (+2 more)
Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large
Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15%. Our code is available at https://github.com/liavonpenn/rmipo.
Read the full abstract ▾
All authors Peng Liao, Peijia Zheng, Lingbo Li, Shangsong Liang, Lin Chen
Machine Learning
Query-Efficient Quantum Approximate Optimization via Graph-Conditioned Trust Regions
Molena Huynh
In low-depth implementations of the
Quantum Approximate Optimization
Algorithm (QAOA), the dominant cost is often the number of objective evaluations rather than
circuit depth. We introduce a graph-conditioned trust-region method for reducing this query cost. A graph neural network predicts a Gaussian distribution N(mu, Sigma) over QAOA angles. The mean initializes a local optimizer, the covariance defines an ellipsoidal trust region that constrains the search, and the predicted uncertainty determines an instance-dependent evaluation budget. Thus the learned distribution defines a search policy rather than only an initial parameter estimate. Under explicit assumptions on local smoothness,
curvature, calibration, and noise, we derive bounds on objective degradation within the trust region, lower bounds on gradient variance, preservation of expected objective ordering under depolarizing noise, and finite-sample coverage guarantees. We evaluate the method for MaxCut at depth p = 2 on Erdos-Renyi, 3-regular, Barabasi-Albert, and Watts-Strogatz graphs with n = 8-16 vertices. Relative to random restarts and the strongest learned point-prediction baseline, the method reduces the mean number of
circuit evaluations from 343 and 85 to 45 +/- 7, while maintaining sampled approximation ratios within 3 percentage points of concentration-based heuristics. The method does not improve absolute approximation ratios; its advantage is reduced query cost at comparable solution quality. The predictive uncertainty is calibrated in the experiments, with ECE = 0.052 and Spearman correlation rho = 0.770, and the learned trust regions transfer to graph sizes not used during training. The results identify a low-depth, query-dominated regime in which graph-conditioned trust regions reduce the query cost of QAOA without modifying the ansatz.
Read the full abstract ▾
Machine Learning
Architecture Determines Observability of Transformers
Thomas Carmichael
Autoregressive
transformers make confident errors that output-confidence monitoring cannot catch. Activation monitors catch them only when training leaves a decision-quality signal beyond what the output already exposes. This signal is an architectural property of the trained model, fixed upstream of any monitor. Controlling for output confidence removes 60.3% of the raw activation-probe signal on average across 14 models. Raw probe signal is mostly output confidence, and output-side readouts cannot recover the residual. What remains depends on architecture and training. In Pythia's controlled training, both matched-width configurations form the signal early. One preserves it through convergence while another erases it as perplexity continues to improve. Capability and observability are not inherently in tension. Across independently trained families this pattern persists, even as the collapse point shifts. Where the signal survives, monitoring catches what confidence cannot. On downstream QA, a WikiText-trained probe with no task-specific tuning catches about one in eight confident errors that output-confidence monitoring misses, at a 20% flag rate. These results establish signal engineering as a training-time design axis alongside loss and capability. Architecture sets the conditions for observability, and training determines what remains readable.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.HC
Personalized Worked Example Generation from Student Code Submissions Using Pattern-based Knowledge Components
Griffin Pitts, Muntasir Hoq, Peter Brusilovsky et al. (+4 more)
Adaptive programming practice often relies on fixed libraries of worked examples and practice problems, which require substantial authoring effort and may not correspond well to the logical errors and partial solutions students produce while writing code. As a result, students may receive learning content that does not directly address the concepts they are working to understand, while instructors must either invest additional effort in expanding content libraries or accept a coarse level of personalization. We present an approach for knowledge-component (KC) guided educational content generation using pattern-based KCs extracted from student code. Given a problem statement and student submissions, our pipeline extracts recurring structural KC patterns from students' code through AST-based analysis and uses them to condition a generative model. In this study, we apply this approach to worked example generation, and compare baseline and KC-conditioned outputs through expert evaluation. Results suggest that KC-conditioned generation improves topical focus and relevance to students' underlying logical errors, providing evidence that KC-based steering of generative models can support personalized learning at scale.
Read the full abstract ▾
All authors Griffin Pitts, Muntasir Hoq, Peter Brusilovsky, Narges Norouzi, Arto Hellas, Juho Leinonen, Bita Akram
Machine Learning
The Optimal Sample Complexity of Multiclass and List Learning
Chirag Pabbaraju
While the optimal sample complexity of binary classification in terms of the VC dimension is well-established, determining the optimal sample complexity of multiclass classification had remained open. The appropriate complexity parameter for multiclass classification is the DS dimension, and despite significant efforts, a gap of $\sqrt{\text{DS}}$ had persisted between the upper and lower bounds on sample complexity. Recent work by Hanneke et al. (2026) shows a novel algebraic characterization of multiclass hypothesis classes in terms of their DS dimension. Building up on this, we show that the maximum hypergraph density of any multiclass hypothesis class is upper-bounded by its DS dimension. This proves a longstanding
conjecture of Daniely and Shalev-Shwartz (2014). As a consequence, we determine the optimal dependence of the sample complexity on the DS dimension for multiclass as well as list learning.
Read the full abstract ▾
Machine Learning
Conflict-Aware Harmonized Rotational Gradient for Multiscale Kinetic Regimes
Zhangyong Liang
In this paper, we propose a harmonized rotational gradient method, termed HRGrad, for simultaneously tackling multiscale time-dependent kinetic problems with varying small parameters. These parameters exhibit asymptotic transitions from microscopic to macroscopic physics, making it a challenging multi-task problem to solve over all ranges simultaneously. Solving tasks in different asymptotic regions often encounter gradient conflicts, which can lead to the failure of multi-task learning. To address this challenge, we explicitly encode a hidden representation of these parameters, ensuring that the corresponding solving tasks are serialized for simultaneous training. Furthermore, to mitigate gradient conflicts, we segment the prediction results to construct task losses and introduce a novel gradient
alignment metric to ensure a positive dot product between the final update and each loss-specific gradient. This metric maintains consistent optimization rates for all task losses and dynamically adjusts gradient magnitudes based on conflict levels. Moreover, we provide a mathematical proof demonstrating the convergence of the HRGrad method, which is evaluated across a range of challenging asymptotic-preserving neural networks (APNNs) scenarios. We conduct an extensive set of experiments encompassing the Bhatnagar-Gross-Krook (BGK) equation and the linear transport equation in all ranges of Knudsen number. Our results indicate that HRGrad effectively overcomes the `failure modes' of APNNs in these problems.
Read the full abstract ▾
Machine Learning
Learning to Think from Multiple Thinkers: How Chain-of-Thought Heterogeneity Affects Efficient Learnability
Nirmit Joshi, Roey Magen, Nathan Srebro et al. (+2 more)
Chain-of-thought (CoT) supervision makes computationally hard learning problems tractable by exposing intermediate steps that are easy to learn via next-
token prediction. However, in practice, CoT
datasets draw on heterogeneous sources with different
reasoning styles, and substantial effort goes into curating supervision suitable for next-
token training. What role does this curation play in enabling the computational advantage of CoT supervision? In this work, we take a first step toward understanding this question from a learning-theoretic perspective. We introduce several models of $heterogeneous$ $CoT$ $supervision$ from multiple sources, each providing correct traces with the same final answer but different intermediate steps. Assuming that fitting a next-
token predictor to $homogeneous$ traces from any single source is computationally tractable, we ask when this suffices for efficient learning from heterogeneous supervision. We show $cryptographic$ $hardness$ of learning from $passively$ collected traces from multiple sources, even though learning from homogeneous CoTs is easy. This hardness can stem either from $hard$-$to$-$learn$ $contribution$ $biases$ determining which sources contribute traces on different parts of the domain, or from receiving an $unfiltered$ heterogeneous mixture (though a filtered subset of homogeneous traces is sufficient for efficient learning). On the positive side, inspired by iterative and adaptive fine-tuning of
language models, we give a computationally efficient learning
algorithm that $actively$ and $adaptively$ collects CoTs from multiple sources with a per-source CoT budget completely independent of the target error $\varepsilon$. We argue that our results illuminate data collection and curation practices central to making CoT supervision effective for learning.
Read the full abstract ▾
All authors Nirmit Joshi, Roey Magen, Nathan Srebro, Nikolaos Tsilivis, Gal Vardi
Machine Learning
SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning
Zijian Guo, İlker Iş{\i}k, H. M. Sabbir Ahmad et al. (+1 more)
Specification-guided reinforcement learning (RL) provides a principled framework for encoding complex, temporally extended tasks using formal specifications such as linear temporal logic (LTL). While recent methods have shown promising results, their ability to generalize across unseen specifications and diverse environments remains insufficiently understood. In this work, we introduce SpecRLBench, a
benchmark designed to evaluate the generalization capabilities of LTL-based specification-guided RL methods. The
benchmark spans multiple difficulty levels across
navigation and manipulation domains, incorporating both static and dynamic environments, diverse
robot dynamics, and varied observation modalities. Through extensive empirical evaluation, we characterize the strengths and limitations of existing approaches and reveal the challenges that emerge as specification and environment complexity increase. SpecRLBench provides a structured platform for systematic comparison and supports the development of more generalizable specification-guided RL methods. Code is available at https://github.com/BU-DEPEND-Lab/SpecRLBench.
Read the full abstract ▾
All authors Zijian Guo, İlker Iş{\i}k, H. M. Sabbir Ahmad, Wenchao Li
Machine Learning · Cross-listed from cs.CL
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh et al. (+7 more)
Hybrid sequence models that combine efficient
Transformer components with linear sequence modeling blocks are a promising alternative to pure
Transformers, but most are still pretrained from scratch and therefore fail to reuse existing
Transformer checkpoints. We study upcycling as a practical path to convert pretrained
Transformer LLMs into hybrid architectures while preserving short-context quality and improving long-context capability. We call our solution HyLo (HYbrid LOng-context): a long-context upcycling recipe that combines architectural adaptation with efficient
Transformer blocks, Multi-Head Latent
Attention (MLA), and linear blocks (Mamba2 or Gated DeltaNet), together with staged long-context training and teacher-guided
distillation for stable optimization. HyLo extends usable context length by up to $32\times$ through efficient post-training and reduces KV-cache
memory by more than $90\%$, enabling up to 2M-
token prefill and decoding in our vLLM inference stack, while comparable Llama baselines run out of
memory beyond 64K context. Across 1B- and 3B-scale settings (Llama- and Qwen-based variants), HyLo delivers consistently strong short- and long-context performance and significantly outperforms state-of-the-art upcycled hybrid baselines on long-context evaluations such as RULER. Notably, at similar scale, HyLo-Qwen-1.7B trained on only 10B
tokens significantly outperforms JetNemotron (trained on 400B
tokens) on GSM8K, Lm-Harness common sense
reasoning and RULER-64K.
Read the full abstract ▾
All authors Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh, Aref Jafari, Akash Haridas, Mingyu Yang, Vansh Bhatia, Guihong Li, Vikram Appia, Emad Barsoum
Machine Learning
Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Models
Hailing Cheng, Tao Huang, Chen Zhu et al. (+1 more)
Training large neural networks with data-parallel
stochastic gradient descent allocates N GPU replicas to compute effectively identical updates – a practice that leaves the rich space of learning rate configurations entirely unexplored during training. We propose Hyperparameter-Divergent Ensemble Training (HDET), a method that repurposes these replicas for simultaneous learning rate exploration at negligible communication overhead. HDET operates in alternating phases: a fan-out stage in which replicas train independently under a structured, symmetric spread of learning rates, and a converge stage in which parameters are averaged across all replicas via AllReduce every T steps. Building on this ensemble substrate, we further propose an automatic learning rate (auto-LR) controller that treats the relative training loss across replicas as a performance signal, updating the shared base schedule toward higher-performing configurations via a momentum-based gradient-free meta-update. The combined method produces a self-adapting learning rate schedule that improves both optimization quality and generalization without additional hyperparameter sweeps or training budget. Crucially, the framework generalizes beyond learning rate: any scalar hyperparameter that does not alter model architecture – such as dropout rate,
attention scale temperature, or weight-decay coefficient – can be explored across replicas using the same fan-out/converge protocol, with inter-replica loss differences serving as zero-order hypergradients that guide the search direction. HDET is implemented as a drop-in replacement for PyTorch's OneCycleLR scheduler, requiring no changes to model architecture, optimizer, or data pipeline.
Read the full abstract ▾
All authors Hailing Cheng, Tao Huang, Chen Zhu, Antonio Alonso
Machine Learning · Cross-listed from eess.SY
Exploiting Differential Flatness for Efficient Learning-based Model Predictive Control of Constrained Multi-Input Control Affine Systems
Tobias A. Farger, Adam W. Hall and Angela P. Schoellig
Learning-based control techniques use data from past trajectories to control systems with uncertain dynamics. However, learning-based controllers are often computationally inefficient, limiting their practicality. To address this limitation, we propose a learning-based controller that exploits differential flatness, a property of many robotic systems. Recent research on using flatness for learning-based control either is limited in that it (i) ignores input constraints, (ii) applies only to single-input systems, or (iii) is tailored to specific platforms. In contrast, our approach uses a system extension and block-diagonal cost formulation to control general multi-input, nonlinear, affine systems. Furthermore, it satisfies input and half-space flat state constraints and guarantees probabilistic Lyapunov decrease using only two sequential convex optimizations. We show that our approach performs similarly to, but is multiple times more efficient than, a Gaussian process model predictive controller in simulation, and achieves competitive tracking in real hardware experiments.
Read the full abstract ▾
Machine Learning · Cross-listed from econ.EM
Energy-Arena: A Dynamic Benchmark for Operational Energy Forecasting
Max Kleinebrahm, Jonathan Berrisch, Philipp Eiser et al. (+11 more)
Energy
forecasting research faces a persistent comparability gap that makes it difficult to measure consistent progress over time. Reported accuracy gains are often not directly comparable because models are evaluated under study-specific
datasets, time periods, information sets, and scoring setups, while widely used
benchmarks and competition
datasets are typically tied to fixed historical windows. This paper introduces the Energy-Arena, a dynamic benchmarking platform for operational energy time series
forecasting that provides a continuously updated reference point as energy systems evolve. The platform operates as an open, API-based submission system and standardizes challenge definitions and submission deadlines aligned with operational constraints. Performance is reported on rolling evaluation windows via persistent leaderboards. By moving from retrospective backtesting to forward-looking benchmarking, the Energy-Arena enforces standardized ex-ante submission and ex-post evaluation, thereby improving transparency by preventing information leakage and retroactive tuning. The platform is publicly available at Energy-Arena.org.
Read the full abstract ▾
All authors Max Kleinebrahm, Jonathan Berrisch, Philipp Eiser, Wolf Fichtner, Veit Hagenmeyer, Matthias Hertel, Nils Koster, Sebastian Lerch, Ralf Mikut, Jan Priesmann, Melanie Schienle, Benjamin Schaefer, Jann Weinand, Florian Ziel
Machine Learning · Cross-listed from cs.CL
Contextual Linear Activation Steering of Language Models
Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan et al. (+1 more)
Linear activation steering is a powerful approach for controlling and specializing large
language models by adding linear representations of concepts to their activations at inference. While effective, existing methods often steer with a fixed strength, independent of how much steering any given input requires. In this work, we introduce Contextual Linear Activation Steering (CLAS), which decomposes activation steering into a fixed steering direction and a context-dependent steering coefficient. This decomposition preserves the interpretability of the steering direction while enabling the steering strength to vary with context. It also enables CLAS to perform selective steering: steering multiple behaviors across different portions of a single generation, which cannot be achieved using a fixed steering strength. In addition, across a wide range of steering
benchmarks and
language models, CLAS consistently outperforms standard linear activation steering and matches or exceeds ReFT and LoRA in settings with limited labeled data. Notably, CLAS achieves this using directions that also serve as reliable concept monitors, a property that directions learned via fine-tuning lack. We therefore propose CLAS as an interpretable and accurate method for controlling and specializing large
language models.
Read the full abstract ▾
All authors Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan, Mikhail Belkin
Machine Learning
Diffusion-Guided Feature Selection via Nishimori Temperature: Noise-Based Spectral Embedding
Vasiliy S. Usatyuk, Denis A. Sapozhnikov and Sergey I. Egorov
We propose Noise-Based Spectral
Embedding (NBSE), a physics-informed framework for selecting informative features from high-dimensional data without greedy search. NBSE constructs a sparse similarity graph on the samples and identifies the Nishimori temperature $\beta_N$ the critical inverse temperature at which the Bethe Hessian becomes singular. The corresponding smallest eigenvector captures the dominant mode of an intrinsically degree-corrected
diffusion process, naturally reweighting nodes to prevent hub dominance. By transposing the data
matrix and applying NBSE in feature space, we obtain a one-dimensional spectral
embedding that reveals groups of redundant or semantically related dimensions; balanced binning then selects one representative per group. We prove that coloured Gaussian perturbations shift $\beta_N$ by at most $O(\bar\sigma^2)$, guaranteeing robustness to measurement noise. Experiments on ImageNet
embeddings from MobileNetV2 and EfficientNet-B4 show that NBSE preserves classification accuracy even under aggressive compression: on EfficientNet-B4 the accuracy drop is below $1\%$ when retaining only $30\%$ of features, outperforming ANOVA $F$-test and random selection by up to $6.8\%$.
Read the full abstract ▾