Machine Learning · Cross-listed from stat.ML
Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations
Yuhao Liu, Zilin Wang, Lei Wu et al. (+1 more)
Smooth activation functions are ubiquitous in modern deep learning, yet their theoretical advantages over non-smooth counterparts remain poorly understood. In this work, we study both approximation and statistical properties of neural networks with smooth activations for learning functions in the Sobolev space $W^{s,\infty}([0,1]^d)$ with $s>0$. We prove that constant-depth networks equipped with smooth activations achieve smoothness adaptivity: increasing width alone suffices to attain the minimax-optimal approximation and estimation error rates (up to logarithmic factors). In contrast, for non-smooth activations such as ReLU, smoothness adaptivity is fundamentally limited by depth: the attainable approximation order is bounded by depth, and higher-order smoothness requires proportional depth growth. These results identify activation smoothness as a fundamental mechanism, complementary to depth, for achieving optimal rates over Sobolev function classes. Technically, our analysis is based on a multi-scale approximation framework that yields explicit neural network approximators with controlled parameter norms and model size. This complexity control ensures statistical learnability under empirical risk minimization (ERM) and avoids the impractical $\ell^0$-sparsity constraints commonly required in prior analyses.
Read the full abstract ▾
All authors Yuhao Liu, Zilin Wang, Lei Wu, Shaobo Zhang
Machine Learning
PerturbDiff: Functional Diffusion for Single-Cell Perturbation Modeling
Xinyu Yuan, Xixian Liu, Ya Shi Zhang et al. (+3 more)
Building Virtual
Cells that can accurately simulate cellular responses to perturbations is a long-standing goal in systems biology. A fundamental challenge is that high-throughput single-
cell sequencing is destructive: the same
cell cannot be observed both before and after a perturbation. Thus, perturbation prediction requires mapping unpaired control and perturbed populations. Existing models address this by learning maps between distributions, but typically assume a single fixed response distribution when conditioned on observed cellular context (e.g.,
cell type) and the perturbation type. In reality, responses vary systematically due to unobservable latent factors such as microenvironmental fluctuations and complex batch effects, forming a
manifold of possible distributions for the same observed conditions. To account for this variability, we introduce PerturbDiff, which shifts modeling from individual
cells to entire distributions. By
embedding distributions as points in a
Hilbert space, we define a
diffusion-based generative process operating directly over probability distributions. This allows PerturbDiff to capture population-level response shifts across hidden factors.
Benchmarks on established
datasets show that PerturbDiff achieves state-of-the-art performance in single-
cell response prediction and generalizes substantially better to unseen perturbations. See our project page (https://katarinayuan.github.io/PerturbDiff-ProjectPage/), where code and data will be made publicly available (https://github.com/DeepGraphLearning/PerturbDiff).
Read the full abstract ▾
All authors Xinyu Yuan, Xixian Liu, Ya Shi Zhang, Zuobai Zhang, Hongyu Guo, Jian Tang
Machine Learning · Cross-listed from cs.AI
SkillOrchestra: Learning to Route Agents via Skill Transfer
Jiayu Wang, Yifei Ming, Zixuan Ke et al. (+3 more)
Compound AI systems promise capabilities beyond those of individual models, yet their success depends critically on effective orchestration. Existing routing approaches face two limitations: (1) input-level routers make coarse query-level decisions that ignore evolving task requirements; (2) RL-trained orchestrators are expensive to adapt and often suffer from routing collapse, repeatedly invoking one strong but costly option in multi-turn scenarios. We introduce SkillOrchestra, a framework for skill-aware orchestration. Instead of directly learning a routing policy end-to-end, SkillOrchestra learns fine-grained skills from execution experience and models
agent-specific competence and cost under those skills. At deployment, the orchestrator infers the skill demands of the current interaction and selects
agents that best satisfy them under an explicit performance-cost trade-off. Extensive experiments across ten
benchmarks demonstrate that SkillOrchestra outperforms SoTA RL-based orchestrators by up to 22.5% with 700x and 300x learning cost reduction compared to Router-R1 and ToolOrchestra, respectively. These results show that explicit skill modeling enables scalable, interpretable, and sample-efficient orchestration, offering a principled alternative to data-intensive RL-based approaches. The code is available at: https://github.com/jiayuww/SkillOrchestra.
Read the full abstract ▾
All authors Jiayu Wang, Yifei Ming, Zixuan Ke, Shafiq Joty, Aws Albarghouthi, Frederic Sala
Machine Learning · Cross-listed from cs.CV
Personalized Longitudinal Medical Report Generation via Temporally-Aware Federated Adaptation
He Zhu, Ren Togo, Takahiro Ogawa et al. (+8 more)
Longitudinal medical report generation is clinically important yet remains challenging due to strict
privacy constraints and the evolving nature of disease progression. Although
federated learning (FL) enables collaborative training without data sharing, existing FL methods largely overlook longitudinal dynamics by assuming stationary client distributions, making them unable to model temporal shifts across visits or patient-specific heterogeneity-ultimately leading to unstable optimization and suboptimal report generation. We introduce
Federated Temporal Adaptation (FTA), a
federated setting that explicitly accounts for the temporal
evolution of client data. Building upon this setting, we propose FedTAR, a framework that integrates demographic-driven personalization with time-aware global aggregation. FedTAR generates lightweight LoRA adapters from demographic
embeddings and performs temporal residual aggregation, where updates from different visits are weighted by a meta-learned temporal policy optimized via first-order MAML. Experiments on J-MID (1M exams) and MIMIC-CXR demonstrate consistent improvements in linguistic accuracy, temporal coherence, and cross-site generalization, establishing FedTAR as a robust and
privacy-preserving paradigm for
federated longitudinal modeling.
Read the full abstract ▾
All authors He Zhu, Ren Togo, Takahiro Ogawa, Kenji Hirata, Minghui Tang, Takaaki Yoshimura, Hiroyuki Sugimori, Noriko Nishioka, Yukie Shimizu, Kohsuke Kudo, Miki Haseyama
Machine Learning
PaReGTA: A Temporally Aware LLM-Based Patient Representation Framework for EHR Analytics
Kihyuk Yoon, Lingchao Mao, Catherine Chong et al. (+3 more)
Temporal information in structured electronic health records (EHRs) is often lost in sparse one-hot or count-based representations, while sequence models can be costly and data-hungry. We propose PaReGTA, an LLM-based encoding framework that (i) converts longitudinal EHR events into visit-level templated text with explicit temporal cues, (ii) learns domain-adapted visit
embeddings via lightweight contrastive fine-tuning of a sentence-
embedding model, and (iii) aggregates visit
embeddings into a fixed-dimensional patient representation using hybrid temporal pooling that captures both recency and globally informative visits. The resulting fixed-dimensional patient representations can be used with conventional downstream machine-learning models. To examine factor-level sensitivity, we use PaReGTA-RSS (Representation Shift Score), a prespecified factor-removal analysis that recomputes patient representations after removing clinically defined factor groups and quantifies the resulting change in the fitted logit of a fixed logistic-regression model. We evaluated PaReGTA in a cohort of 39,088 patients with migraine from the All of Us Research Program (AoU) on a retrospective patient-level classification task with an EHR-derived target. In an exploratory comparison on the fixed held-out test cohort of 7,818 patients, PaReGTA-Gap + LightGBM, used as a post hoc analytical reference, had the highest reported AUC, accuracy, and F1 values among the evaluated sparse, BERT-based EHR, and recurrent approaches in this cohort.
Read the full abstract ▾
All authors Kihyuk Yoon, Lingchao Mao, Catherine Chong, Todd J. Schwedt, Chia-Chun Chiang, Jing Li
Machine Learning
Representation Stability in a Minimal Continual Learning Agent
Vishnu Subramanian
Continual learning systems are increasingly deployed in environments where retraining or reset is infeasible, yet many approaches emphasize task performance rather than the
evolution of internal representations over time. In this work, we study a minimal continual learning
agent designed to isolate representational dynamics from architectural complexity and optimization objectives. The
agent maintains a persistent state vector across executions and incrementally updates it as new textual data is introduced. We quantify representational change using cosine similarity between successive normalized state vectors and define a stability metric over time intervals. Longitudinal experiments across eight executions reveal a transition from an initial plastic regime to a stable representational regime under consistent input. A deliberately introduced semantic perturbation produces a bounded decrease in similarity, followed by recovery and restabilization under subsequent coherent input. These results demonstrate that meaningful stability plasticity tradeoffs can emerge in a minimal, stateful learning system without explicit regularization, replay, or architectural complexity. The work establishes a transparent empirical baseline for studying representational accumulation and adaptation in continual learning systems.
Read the full abstract ▾
Machine Learning
NEXUS: A compact neural architecture for high-resolution spatiotemporal air quality forecasting in Delhi National Capital Region
Rampunit Kumar and Aditya Maheshwari
Urban air pollution in megacities poses critical public health challenges, particularly in Delhi National Capital Region (NCR) where severe degradation affects millions. We present NEXUS (Neural Extraction and Unified Spatiotemporal) architecture for
forecasting carbon monoxide, nitrogen
oxide, and sulfur dioxide. Working with four years (2018–2021) of atmospheric data across sixteen spatial grids, NEXUS achieves R$^2$ exceeding 0.94 for CO, 0.91 for NO, and 0.95 for SO$_2$ using merely 18,748 parameters – substantially fewer than SCINet (35,552), Autoformer (68,704), and FEDformer (298,080). The architecture integrates patch
embedding, low-rank projections, and adaptive fusion mechanisms to decode complex atmospheric chemistry patterns. Our investigation uncovers distinct diurnal rhythms and pronounced seasonal variations, with winter months experiencing severe pollution episodes driven by temperature inversions and agricultural biomass burning. Analysis identifies critical meteorological thresholds, quantifies wind field impacts on pollutant dispersion, and maps spatial heterogeneity across the region. Extensive ablation experiments demonstrate each architectural component's role. NEXUS delivers superior predictive performance with remarkable computational efficiency, enabling real-time deployment for air quality monitoring systems.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
Denoising Particle Filters: Learning State Estimation with Single-Step Objectives
Lennart Röstel and Berthold Bäuml
Learning-based methods commonly treat state estimation in robotics as a sequence modeling problem. While this paradigm can be effective at maximizing end-to-end performance, models are often difficult to interpret and expensive to train, since training requires unrolling sequences of predictions in time. As an alternative to end-to-end trained state estimation, we propose a novel particle filtering
algorithm in which models are trained from individual state transitions, fully exploiting the Markov property in robotic systems. In this framework, measurement models are learned implicitly by minimizing a denoising score matching objective. At inference, the learned denoiser is used alongside a (learned) dynamics model to approximately solve the
Bayesian filtering equation at each time step, effectively guiding predicted states toward the data
manifold informed by measurements. We evaluate the proposed method on challenging robotic state estimation tasks in simulation, demonstrating competitive performance compared to tuned end-to-end trained baselines. Importantly, our method offers the desirable composability of classical filtering
algorithms, allowing prior information and external sensor models to be incorporated without retraining.
Read the full abstract ▾
Machine Learning
Spectral Phase Encoding for Quantum Kernel Methods
Pablo Herrero Gómez, Antonio Jimeno Morenilla, David Muñoz-Hernández et al. (+1 more)
Quantum kernel methods are promising for near-term
quantum ma- chine learning, yet their behavior under data corruption remains insuf- ficiently understood. We analyze how
quantum feature constructions degrade under controlled additive noise. We introduce Spectral Phase Encoding (SPE), a hybrid construc- tion combining a discrete
Fourier transform (DFT) front-end with a diagonal phase-only
embedding aligned with the geometry of diagonal
quantum maps. Within a unified framework, we compare QK-DFT against alternative
quantum variants (QK-PCA, QK-RP) and classi- cal SVM baselines under identical clean-data hyperparameter selection, quantifying robustness via
dataset fixed-effects regression with wild cluster bootstrap inference across heterogeneous real-world
datasets. Across the
quantum family, DFT-based preprocessing yields the smallest degradation rate as noise increases, with statistically sup- ported slope differences relative to PCA and RP. Compared to classical baselines, QK-DFT shows degradation comparable to linear SVM and more stable than RBF SVM under matched tuning. Hardware exper- iments confirm that SPE remains executable and numerically stable for overlap estimation. These results indicate that robustness in quan- tum kernels depends critically on structure-aligned preprocessing and its interaction with diagonal
embeddings, supporting a robustness-first perspective for NISQ-era
quantum machine learning.
Read the full abstract ▾
All authors Pablo Herrero Gómez, Antonio Jimeno Morenilla, David Muñoz-Hernández, Higinio Mora Mora
Machine Learning
Evaluating the Impact of Data Anonymization on Image Retrieval
Marvin Chen, Manuel Eberhardinger and Johannes Maucher
With the growing importance of
privacy regulations such as the General Data Protection Regulation, anonymizing visual data is becoming increasingly relevant across institutions. However, anonymization can negatively affect the performance of Computer
Vision systems that rely on visual features, such as Content-Based Image
Retrieval (CBIR). Despite this, the impact of anonymization on CBIR has not been systematically studied. This work addresses this gap, motivated by the DOKIQ project, an artificial intelligence-based system for document verification actively used by the State Criminal Police Office Baden-Württemberg. We propose a simple evaluation framework:
retrieval results after anonymization should match those obtained before anonymization as closely as possible. To this end, we systematically assess the impact of anonymization using two public
datasets and the internal DOKIQ
dataset. Our experiments span three anonymization methods, four anonymization degrees, and four training strategies, all based on the state of the art backbone Self-
Distillation with No Labels (DINO)v2. Our results reveal a pronounced
retrieval bias in favor of models trained on original data, which produce the most similar
retrievals after anonymization. The findings of this paper offer practical insights for developing
privacy-compliant CBIR systems while preserving performance.
Read the full abstract ▾
Machine Learning
Compositional Planning with Jumpy World Models
Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni et al. (+3 more)
The ability to plan with temporal abstractions is central to intelligent decision-making. Rather than
reasoning over primitive actions, we study
agents that compose pre-trained policies as temporally extended actions, enabling solutions to complex tasks that no constituent alone can solve. Such compositional planning remains elusive as compounding errors in long-horizon predictions make it challenging to estimate the visitation distribution induced by sequencing policies. Motivated by the
geometric policy composition framework introduced in arXiv:2206.08736, we address these challenges by learning predictive models of multi-step dynamics – so-called jumpy world models – that capture state occupancies induced by pre-trained policies across multiple timescales in an off-policy manner. Building on Temporal Difference Flows (arXiv:2503.09817), we enhance these models with a novel consistency objective that aligns predictions across timescales, improving long-horizon predictive accuracy. We further demonstrate how to combine these generative predictions to estimate the value of executing arbitrary sequences of policies over varying timescales. Empirically, we find that compositional planning with jumpy world models significantly improves zero-shot performance across a wide range of base policies on challenging manipulation and
navigation tasks, yielding, on average, a 200% relative improvement over planning with primitive actions on long-horizon tasks.
Read the full abstract ▾
All authors Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni, Marc G. Bellemare, Alessandro Lazaric, Ahmed Touati
Machine Learning
VecFormer: Towards Efficient and Generalizable Graph Transformer with Graph Token Attention
Jingbo Zhou, Jun Xia, Siyuan Li et al. (+11 more)
Graph
Transformer has demonstrated impressive capabilities in the field of graph representation learning. However, existing approaches face two critical challenges: (1) most models suffer from exponentially increasing computational complexity, making it difficult to scale to large graphs; (2)
attention mechanisms based on node-level operations limit the flexibility of the model and result in poor generalization performance in out-of-distribution (OOD) scenarios. To address these issues, we propose VecFormer (the Vector Quantized Graph
Transformer), an efficient and highly generalizable model for node classification, particularly under OOD settings. VecFormer adopts a two-stage training paradigm. In the first stage, two codebooks are used to reconstruct the node features and the graph structure, aiming to learn the rich semantic Graph Codes. In the second stage,
attention mechanisms are performed at the Graph
Token level based on the transformed cross codebook, reducing computational complexity while enhancing the model's generalization capability. Extensive experiments on
datasets of various sizes demonstrate that VecFormer outperforms the existing Graph
Transformer in both performance and speed.
Read the full abstract ▾
All authors Jingbo Zhou, Jun Xia, Siyuan Li, Yunfan Liu, Wenjun Wang, Yufei Huang, Changxi Chi, Mutian Hong, Zhuoli Ouyang, Shu Wang, Zhongqi Wang, Xingyu Wu, Chang Yu, Stan Z. Li
Machine Learning
Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models
Luhan Tang, Longxuan Yu, Shaorong Zhang et al. (+1 more)
Discrete
diffusion language models (dLLMs) provide a fast and flexible alternative to autoregressive models (ARMs) via iterative denoising with parallel updates. However, their evaluation is challenging: existing metrics conflate denoiser approximation error with sampler-induced error from the sampling dynamics, a problem that does not arise for ARMs whose autoregressive sampling exactly reflects the learned probability model. We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain, isolating sampler-induced error in a controlled setting. We show that few-step discrete
diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length. Moreover, improvements in negative log-likelihood (NLL), generative perplexity (GenPPL), or MAUVE do not imply correct sampling. Code is available at https://luhantang.github.io/dllm_sampler
Read the full abstract ▾
All authors Luhan Tang, Longxuan Yu, Shaorong Zhang, Greg Ver Steeg
Machine Learning · Cross-listed from cs.SE
Workflow-Level Design Principles for Trustworthy GenAI in Automotive System Engineering
Chih-Hong Cheng, Brian Hsuan-Cheng Liao, Adam Molin et al. (+1 more)
The adoption of large
language models in safety-critical system engineering is constrained by trustworthiness, traceability, and
alignment with established verification practices. We propose workflow-level design principles for trustworthy GenAI integration and demonstrate them in an end-to-end automotive pipeline, from requirement delta identification to SysML v2 architecture update and re-testing. First, we show that monolithic ("big-bang") prompting misses critical changes in large specifications, while section-wise decomposition with diversity sampling and lightweight NLP sanity checks improves completeness and correctness. Then, we propagate requirement deltas into SysML v2 models and validate updates via compilation and static analysis. Additionally, we ensure traceable regression testing by generating test cases through explicit mappings from specification variables to architectural ports and states, providing practical safeguards for GenAI used in safety-critical automotive engineering.
Read the full abstract ▾
All authors Chih-Hong Cheng, Brian Hsuan-Cheng Liao, Adam Molin, Hasan Esen
Machine Learning
Variational Inference for Bayesian MIDAS Regression
Luigi Simeone
We develop a Coordinate Ascent Variational Inference (CAVI)
algorithm for
Bayesian Mixed Data Sampling (MIDAS) regression with linear weight parameterizations. The model separates impact coeffcients from weighting function parameters through a normalization constraint, creating a bilinear structure that renders generic
Hamiltonian Monte Carlo samplers unreliable while preserving conditional conjugacy exploitable by CAVI. Each variational update admits a closed-form solution: Gaussian for regression coefficients and weight parameters, Inverse-Gamma for the error variance. The
algorithm propagates uncertainty across blocks through second moments, distinguishing it from naive plug-in approximations. In a Monte Carlo study spanning 21 data-generating configurations with up to 50 predictors, CAVI produces posterior means nearly identical to a block Gibbs sampler
benchmark while achieving speedups of 107x to 1,772x (Table 9). Generic automatic differentiation VI (ADVI), by contrast, produces bias 714 times larger while being orders of magnitude slower, confirming the value of model-specific derivations. Weight function parameters maintain excellent calibration (coverage above 92%) across all configurations. Impact coefficient credible intervals exhibit the underdispersion characteristic of mean-field approximations, with coverage declining from 89% to 55% as the number of predictors grows a documented trade-off between speed and interval calibration that structured variational methods can address. An empirical application to realized
volatility forecasting on S&P 500 daily returns cofirms that CAVI and Gibbs sampling yield virtually identical point forecasts, with CAVI completing each monthly estimation in under 10 milliseconds.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Manifold-Aligned Generative Transport
Xinyu Tian and Xiaotong Shen
Many high-dimensional
datasets concentrate near a low-dimensional structure embedded in the ambient space. Generative models for such data must control off-support mass while remaining computationally practical.
Diffusion models use iterative denoising at inference, whereas standard normalizing flows require invertible, dimension-preserving maps. We propose MAGT (
Manifold-Aligned Generative Transport), a direct transport from a low-dimensional base distribution to the data space. Its core objective compares the data and generator-induced scores at a selected Gaussian smoothing level. A posterior identity expresses this score through a latent conditional mean, which is approximated by self-normalized importance sampling over a finite anchor set. After training, generation requires one evaluation of the transport, whose image also carries an intrinsic density with respect to
manifold volume. We establish a minimax-optimal Wasserstein convergence rate for an explicitly constructed localized spline-RePU coordinate transport
estimator, and treat finite-anchor approximation separately. Experiments on synthetic, image, and tabular
benchmarks compare fidelity, support
alignment, and sampling cost with
diffusion, flow-matching, and
adversarial baselines.
Read the full abstract ▾
Machine Learning
ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
Ayush Nangia, Shikhar Mishra, Aman Gokrani et al. (+1 more)
We introduce ISO-Bench, a
benchmark for coding
agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an
agent with a codebase and bottleneck description, whereby the
agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing
benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding
agents, we find no single
agent dominates across codebases. Surprisingly,
agents often identify correct bottlenecks but fail to execute working solutions. We also show that
agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model.
Read the full abstract ▾
All authors Ayush Nangia, Shikhar Mishra, Aman Gokrani, Paras Chopra
Machine Learning
Detecting High-Potential SMEs with Heterogeneous Graph Neural Networks
Yijiashun Qi, Hanzhe Guo and Yijiazhen Qi
Small and Medium Enterprises (SMEs) constitute 99.9% of U.S. businesses and generate 44% of economic activity, yet systematically identifying high-potential SMEs remains an open challenge. We introduce SME-HGT, a Heterogeneous Graph
Transformer framework that predicts which SBIR Phase I awardees will advance to Phase II funding using exclusively public data. We construct a heterogeneous graph with 32,268 company nodes, 124 research topic nodes, and 13 government agency nodes connected by approximately 99,000 edges across three semantic relation types. SME-HGT achieves an AUPRC of 0.621 0.003 on a temporally-split test set, outperforming an MLP baseline (0.590 0.002) and R-GCN (0.608 0.013) across five random seeds. At a screening depth of 100 companies, SME-HGT attains 89.6% precision with a 2.14 lift over random selection. Our temporal evaluation protocol prevents information leakage, and our reliance on public data ensures reproducibility. These results demonstrate that relational structure among firms, research topics, and funding agencies provides meaningful signal for SME potential assessment, with implications for policymakers and early-stage investors.
Read the full abstract ▾
Machine Learning
Interpolation-Driven Machine Learning Approaches for Plume Shine Dose Estimation: A Comparison of XGBoost, Random Forest, and TabNet
Biswajit Sadhu, Kalpak Gupte, Trijit Sadhu et al. (+1 more)
Despite the success of machine learning (ML) in surrogate modeling, its use in radiation dose assessment is limited by safety-critical constraints, scarce training-ready data, and challenges in selecting suitable architectures for physics-dominated systems. Within this context, rapid and accurate plume shine dose estimation serves as a practical test case, as it is critical for
nuclear facility safety assessment and radiological emergency response, while conventional
photon-transport-based calculations remain computationally expensive. In this work, an interpolation-assisted ML framework was developed using discrete dose
datasets generated with the pyDOSEIA suite for 17 gamma-emitting radionuclides across varying downwind distances, release heights, and atmospheric stability categories. The
datasets were augmented using shape-preserving interpolation to construct dense, high-resolution training data. Two tree-based ML models (Random Forest and XGBoost) and one deep learning (DL) model (TabNet) were evaluated to examine predictive performance and sensitivity to
dataset resolution. All models showed higher prediction accuracy with the interpolated high-resolution
dataset than with the discrete data; however, XGBoost consistently achieved the highest accuracy. Interpretability analysis using permutation importance (tree-based models) and
attention-based feature attribution (TabNet) revealed that performance differences stem from how the models utilize input features. Tree-based models focus mainly on dominant geometry-dispersion features (release height, stability category, and downwind distance), treating radionuclide identity as a secondary input, whereas TabNet distributes
attention more broadly across multiple variables. For practical deployment, a web-based GUI was developed for interactive scenario evaluation and transparent comparison with
photon-transport reference calculations.
Read the full abstract ▾
All authors Biswajit Sadhu, Kalpak Gupte, Trijit Sadhu, S. Anand
Machine Learning
Advantage-based Temporal Attack in Reinforcement Learning
Shenghong He
Extensive research demonstrates that Deep Reinforcement Learning (DRL) models are susceptible to adversarially constructed inputs (i.e.,
adversarial examples), which can mislead the
agent to take suboptimal or unsafe actions. Recent methods improve attack effectiveness by leveraging future rewards to guide
adversarial perturbation generation over sequential time steps (i.e., reward-based attacks). However, these methods are unable to capture dependencies between different time steps in the perturbation generation process, resulting in a weak temporal correlation between the current perturbation and previous perturbations.In this paper, we propose a novel method called Advantage-based
Adversarial Transformer (AAT), which can generate
adversarial examples with stronger temporal correlations (i.e., time-correlated
adversarial examples) to improve the attack performance. AAT employs a multi-scale causal self-
attention (MSCSA) mechanism to dynamically capture dependencies between historical information from different time periods and the current state, thus enhancing the correlation between the current perturbation and the previous perturbation. Moreover, AAT introduces a weighted advantage mechanism, which quantifies the effectiveness of a perturbation in a given state and guides the generation process toward high-performance
adversarial examples by sampling high-advantage regions. Extensive experiments demonstrate that the performance of AAT matches or surpasses mainstream
adversarial attack baselines on Atari, DeepMind Control Suite and Google football tasks.
Read the full abstract ▾
Machine Learning
Leap+Verify: Regime-Adaptive Speculative Weight Prediction for Accelerating Neural Network Training
Jeremy McEntire
We introduce Leap+Verify, a framework that applies speculative execution – predicting future model weights and validating predictions before acceptance – to accelerate neural network training. Inspired by speculative decoding in
language model inference and by the Automatically Scalable Computation (ASC) architecture for program execution, Leap+Verify decomposes training into three dynamically detected regimes (chaotic, transition, stable) using activation-space cosine similarity as a real-time Lyapunov proxy signal. Within each regime, analytic weight predictors (momentum, linear, quadratic extrapolation) attempt to forecast model parameters K training steps ahead; predictions are accepted only when validated against a held-out loss criterion. We evaluate Leap+Verify on GPT-2 124M and Qwen 2.5-1.5B trained on WikiText-103 across five random seeds, sweeping prediction depth K in {5, 10, 25, 50, 75, 100}. Momentum-based prediction (Adam moment extrapolation) fails catastrophically at both scales, with predicted losses exceeding actuals by 100-10,000x – a universal norm explosion in optimizer-state extrapolation. Finite-difference predictors (linear, quadratic) succeed where momentum fails: at 124M, they achieve 24% strict acceptance at K=5 in stable regimes; at 1.5B, they achieve 37% strict acceptance in transition regimes. The scale-dependent finding is in regime distribution: GPT-2 124M spends 34% of training in stable regime, while Qwen 1.5B spends 64% in chaotic regime and reaches stable in only 0-2 of 40 checkpoints. Larger models are more predictable when predictable, but less often predictable – the practical bottleneck shifts from predictor accuracy to regime availability. Cross-seed results are highly consistent (less than 1% validation loss variance), and the three-regime framework produces identical phase boundaries (plus or minus 50 steps) across seeds.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Goal-Oriented Influence-Maximizing Data Acquisition for Learning and Optimization
Weichi Yao, Bianca Dumitrascu, Bryan R. Goldsmith et al. (+1 more)
Active data acquisition is central to many learning and optimization tasks in deep neural networks, yet remains challenging because most approaches rely on predictive uncertainty estimates that are difficult to obtain reliably. To this end, we propose Goal-Oriented Influence- Maximizing Data Acquisition (GOIMDA), an active acquisition
algorithm that avoids explicit posterior inference while remaining uncertainty-aware through inverse
curvature. GOIMDA selects inputs by maximizing their expected influence on a user-specified goal functional, such as test loss, predictive
entropy, or the value of an optimizer-recommended design. Leveraging first-order influence functions, we derive a tractable acquisition rule that combines the goal gradient, training-loss
curvature, and candidate sensitivity to model parameters. We show theoretically that, for generalized linear models, GOIMDA approximates predictive-
entropy minimization up to a correction term accounting for goal
alignment and prediction bias, thereby, yielding uncertainty-aware behavior without maintaining a
Bayesian posterior. Empirically, across learning tasks (including image and text classification) and optimization tasks (including noisy global optimization
benchmarks and neural-network hyperparameter tuning), GOIMDA consistently reaches target performance with substantially fewer labeled samples or function evaluations than uncertainty-based active learning and Gaussian-process
Bayesian optimization baselines.
Read the full abstract ▾
All authors Weichi Yao, Bianca Dumitrascu, Bryan R. Goldsmith, Yixin Wang
Machine Learning
The Sample Complexity of Replicable Realizable PAC Learning
Kasper Green Larsen, Markus Engelund Mathiasen, Chirag Pabbaraju et al. (+1 more)
In this paper, we consider the problem of replicable realizable PAC learning. We construct a particularly hard learning problem and show a sample complexity lower bound with a close to $(\log|H|)^{3/2}$ dependence on the size of the hypothesis class $H$. Our proof uses several novel techniques and works by defining a particular Cayley graph associated with $H$ and analyzing a suitable random walk on this graph by examining the spectral properties of its adjacency
matrix. Furthermore, we show an almost matching upper bound for the lower bound instance, meaning if a stronger lower bound exists, one would have to consider a different instance of the problem.
Read the full abstract ▾
All authors Kasper Green Larsen, Markus Engelund Mathiasen, Chirag Pabbaraju, Clement Svendsen
Machine Learning · Cross-listed from cs.CL
Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
Jeffrey Li, Josh Gardner, Doug Kang et al. (+10 more)
One of the first pre-processing steps for constructing web-scale LLM pretraining
datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source
datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard
language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple intervention: by taking a Union over different extractors, we can increase the
token yield of DCLM-Baseline by up to 71% while maintaining
benchmark performance. We further show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance, with differences of up to 10 percentage points (p.p.) on WikiTQ and 3 p.p. on HumanEval.
Read the full abstract ▾
All authors Jeffrey Li, Josh Gardner, Doug Kang, Fangping Shi, Karanjeet Singh, Chun-Liang Li, Herumb Shandilya, David Hall, Oncel Tuzel, Percy Liang, Ludwig Schmidt, Hadi Pour Ansari, Fartash Faghri
Machine Learning · Cross-listed from cs.CV
Can a Teenager Fool an AI? Evaluating Low-Cost Cosmetic Attacks on Age Estimation Systems
Xingyu Shen, Tommy Duong, Xiaodong An et al. (+6 more)
Age estimation systems are increasingly deployed as gatekeepers for age-restricted online content, yet their robustness to cosmetic modifications has not been systematically evaluated. We investigate whether simple, household-accessible cosmetic changes, including beards, grey hair, makeup, and simulated wrinkles, can cause AI age
estimators to classify minors as adults. To study this threat at scale without ethical concerns, we simulate these physical attacks on 329 facial images of individuals aged 10 to 21 using a VLM image editor (Gemini 2.5 Flash Image). We then evaluate eight models from our prior
benchmark: five specialized architectures (MiVOLO, Custom-Best, Herosan, MiViaLab, DEX) and three
vision-
language models (Gemini 3 Flash, Gemini 2.5 Flash, GPT-5-Nano). We introduce the Attack Conversion Rate (ACR), defined as the fraction of images predicted as minor at baseline that flip to adult after attack, a population-agnostic metric that does not depend on the ratio of minors to adults in the test set. Our results reveal that a synthetic beard alone achieves 28 to 69 percent ACR across all eight models; combining all four attacks shifts predicted age by +7.7 years on average across all 329 subjects and reaches up to 83 percent ACR; and
vision-
language models exhibit lower ACR (59 to 71 percent) than specialized models (63 to 83 percent) under the full attack, although the ACR ranges overlap and the difference is not statistically tested. These findings highlight a critical vulnerability in deployed age-verification pipelines and call for
adversarial robustness evaluation as a mandatory criterion for model selection.
Read the full abstract ▾
All authors Xingyu Shen, Tommy Duong, Xiaodong An, Zengqi Zhao, Zebang Hu, Haoyu Hu, Ziyou Wang, Finn Guo, Simiao Ren
Machine Learning · Cross-listed from cs.RO
Cost-Aware Diffusion Active Search
Arundhati Banerjee and Jeff Schneider
Active search for recovering objects of interest through online, adaptive decision making with autonomous
agents requires trading off exploration of unknown environments with exploitation of prior observations in the search space. Prior work has proposed information gain and Thompson sampling based myopic, greedy approaches for
agents to actively decide query or search locations when the number of targets is unknown. Decision making
algorithms in such partially observable environments have also shown that
agents capable of lookahead over a finite horizon outperform myopic policies for active search. Unfortunately, lookahead
algorithms typically rely on building a computationally expensive search tree that is simulated and updated based on the
agent's observations and a model of the environment dynamics. Instead, in this work, we leverage the sequence modeling abilities of
diffusion models to sample lookahead action sequences that balance the exploration-exploitation trade-off for active search without building an exhaustive search tree. We identify the optimism bias in prior
diffusion based reinforcement learning approaches when applied to the active search setting and propose mitigating solutions for efficient cost-aware decision making with both single and multi-
agent teams. Our proposed
algorithm outperforms standard baselines in offline reinforcement learning in terms of full recovery rate and is computationally more efficient than tree search in cost-aware active decision making.
Read the full abstract ▾
Machine Learning
Grokking Finite-Dimensional Algebra
Pascal Jr Tikeng Notsawo, Guillaume Dumas and Guillaume Rabusseau
This paper investigates the grokking phenomenon, which refers to the sudden transition from a long memorization to generalization observed during neural networks training, in the context of learning multiplication in finite-dimensional
algebras (FDA). While prior work on grokking has focused mainly on group operations, we extend the analysis to more general algebraic structures, including non-associative, non-commutative, and non-unital
algebras. We show that learning group operations is a special case of learning FDA, and that learning multiplication in FDA amounts to learning a bilinear product specified by the
algebra's structure tensor. For
algebras over the reals, we connect the learning problem to
matrix factorization with an implicit low-rank bias, and for
algebras over finite fields, we show that grokking emerges naturally as models must learn discrete representations of algebraic elements. This leads us to experimentally investigate the following core questions: (i) how do algebraic properties such as commutativity, associativity, and unitality influence both the emergence and timing of grokking, (ii) how structural properties of the structure tensor of the FDA, such as sparsity and rank, influence generalization, and (iii) to what extent generalization correlates with the model learning latent
embeddings aligned with the
algebra's representation. Our work provides a unified framework for grokking across algebraic structures and new insights into how mathematical structure governs neural network generalization dynamics.
Read the full abstract ▾
Machine Learning
A Statistical Approach for Modeling Irregular Multivariate Time Series with Missing Observations
Dingyi Nie, Yixing Wu and C.-C. Jay Kuo
Irregular multivariate time series with missing values present significant challenges for predictive modeling in domains such as healthcare. While deep learning approaches often focus on temporal interpolation or complex architectures to handle irregularities, we propose a simpler yet effective alternative: extracting time-agnostic summary statistics to eliminate the temporal axis. Our method computes four key features per variable-mean and standard deviation of observed values, as well as the mean and variability of changes between consecutive observations to create a fixed-dimensional representation. These features are then utilized with standard classifiers, such as logistic regression and XGBoost. Evaluated on four biomedical
datasets (PhysioNet Challenge 2012, 2019, PAMAP2, and MIMIC-III), our approach achieves state-of-the-art performance, surpassing recent
transformer and graph-based models by 0.5-1.7% in AUROC/AUPRC and 1.1-1.7% in accuracy/F1-score, while reducing computational complexity. Ablation studies demonstrate that feature extraction-not classifier choice-drives performance gains, and our summary statistics outperform raw/imputed input in most
benchmarks. In particular, we identify scenarios where missing patterns themselves encode predictive signals, as in sepsis prediction (PhysioNet, 2019), where missing indicators alone can achieve 94.2% AUROC with XGBoost, only 1.6% lower than using original raw data as input. Our results challenge the necessity of complex temporal modeling when task objectives permit time-agnostic representations, providing an efficient and interpretable solution for irregular time series classification.
Read the full abstract ▾
Machine Learning
Beyond Accuracy: A Unified Random Matrix Theory Diagnostic Framework for Crash Classification Models
Ibne Farabi Shihab, Sanjeda Akter and Anuj Sharma
Crash classification models in transportation safety are typically evaluated using accuracy, F1, or AUC, metrics that cannot reveal whether a model is silently overfitting. We introduce a spectral diagnostic framework grounded in Random
Matrix Theory (RMT) and Heavy-Tailed Self-Regularization (HTSR) that spans the ML taxonomy: weight
matrices for BERT/ALBERT/Qwen2.5, out-of-fold increment
matrices for XGBoost/Random Forest, empirical Hessians for Logistic Regression, induced affinity
matrices for Decision Trees, and Graph
Laplacians for KNN. Evaluating nine model families on two Iowa DOT crash classification tasks (173,512 and 371,062 records respectively), we find that the power-law exponent $\alpha$ provides a structural quality signal: well-regularized models consistently yield $\alpha$ within $[2, 4]$ (mean $2.87 \pm 0.34$), while overfit variants show $\alpha < 2$ or spectral collapse. We observe a strong rank correlation between $\alpha$ and expert agreement (Spearman $\rho = 0.89$, $p < 0.001$), suggesting spectral quality captures model behaviors aligned with expert
reasoning. We propose an $\alpha$-based early stopping criterion and a spectral model selection protocol, and validate both against cross-validated F1 baselines. Sparse Lanczos approximations make the framework scalable to large
datasets.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Ada-RS: Adaptive Rejection Sampling for Selective Thinking
Yirou Ge, Yixi Li, Alec Chiu et al. (+8 more)
Large
language models (LLMs) are increasingly being deployed in cost and latency-sensitive settings. While chain-of-thought improves
reasoning, it can waste
tokens on simple requests. We study selective thinking for tool-using LLMs and introduce Adaptive Rejection Sampling (Ada-RS), an
algorithm-agnostic sample filtering framework for learning selective and efficient
reasoning. For each given context, Ada-RS scores multiple sampled completions with an adaptive length-penalized reward then applies
stochastic rejection sampling to retain only high-reward candidates (or preference pairs) for downstream optimization. We demonstrate how Ada-RS plugs into both preference pair (e.g. DPO) or grouped policy optimization strategies (e.g. DAPO). Using Qwen3-8B with LoRA on a synthetic tool call-oriented e-commerce
benchmark, Ada-RS improves the accuracy-efficiency frontier over standard
algorithms by reducing average output
tokens by up to 80% and reducing thinking rate by up to 95% while maintaining or improving tool call accuracy. These results highlight that training-signal selection is a powerful lever for efficient
reasoning in latency-sensitive deployments.
Read the full abstract ▾
All authors Yirou Ge, Yixi Li, Alec Chiu, Shivani Shekhar, Zijie Pan, Avinash Thangali, Yun-Shiuan Chuang, Chaitanya Kulkarni, Uma Kona, Linsey Pang, Prakhar Mehrotra