Machine Learning
Pareto DNN Verification: Fast for Most Queries
Yizhak Y. Elboher, Avraham Raviv, Amihay Elboher et al. (+3 more)
Formal verification of deep neural networks (DNNs) faces a fundamental scalability challenge: verifying robustness is NP-hard and state-of-the-art tools often time out on modern architectures. We propose PAreto VErification, or PAVE: by augmenting DNNs with early exits (EEs) and verifying exit-by-exit, we solve a large fraction of robustness queries—roughly 80% in our evaluation—in a small fraction of the time required for direct verification. The key insight is that many inputs in a robustness neighborhood exit at an early layer. Verifying this early-exit behavior is often a tractable problem,allowing verification to terminate long before the full network is analyzed. EEs can be added without modifying the base network, and incur at most a 0.3% accuracy change across our tested architectures. We formalize the novel robustness property for EE networks, prove it is Fixed-Parameter Tractable under a trace stability condition (which commonly holds in practice), and present a sound and complete verification
algorithm. Two sound optimizations — break (early termination when the winner dominates) and continue (skipping the inner loop when no runner-up can win) — reduce SAFE-case verification time by up to 10x. Experiments on MNIST, CIFAR-10, and CIFAR-100 with networks from 600K to 33M parameters demonstrate that PAVE enables solving queries that were not solved otherwise, including formal verification of ResNet-18 on CIFAR-100.
Read the full abstract ▾
All authors Yizhak Y. Elboher, Avraham Raviv, Amihay Elboher, Zhouxing Shi, Hillel Kugler, Guy Katz
Machine Learning
Stabilizing Multimodal Autoencoders: A Theoretical and Empirical Analysis of Fusion Strategies
Diyar Altinses and Andreas Schwung
In recent years, the development of
multimodal autoencoders has gained significant
attention due to their potential to handle
multimodal complex data types and improve model performance. Understanding the stability and robustness of these models is crucial for optimizing their training, architecture, and real-world applicability. This paper presents an analysis of Lipschitz properties in
multimodal autoencoders, combining both theoretical insights and empirical validation to enhance the training stability of these models. We begin by deriving the theoretical Lipschitz constants for aggregation methods within the
multimodal autoencoder framework. We then introduce a regularized
attention-based fusion method, developed based on our theoretical analysis, which demonstrates improved stability and performance during training. Through a series of experiments, we empirically validate our theoretical findings by estimating the Lipschitz constants across multiple trials and fusion strategies. Our results demonstrate that our proposed fusion function not only aligns with theoretical predictions but also outperforms existing strategies in terms of consistency, convergence speed, and accuracy. This work provides a solid theoretical foundation for understanding fusion in
multimodal autoencoders and contributes a solution for enhancing their performance.
Read the full abstract ▾
Machine Learning · Cross-listed from astro-ph.SR
A Physics Informed Neural Network For Deriving MHD State Vectors From Global Active Regions Observations
Subhamoy Chatterjee and Mausumi Dikpati
Solar active regions (ARs) do not appear randomly but cluster along longitudinally warped toroidal bands ('toroids') that encode information about
magnetic structures in the tachocline, where global-scale organization likely originates. Global MagnetoHydroDynamic Shallow-Water Tachocline (MHD-SWT) models have shown potential to simulate such toroids, matching observations qualitatively. For week-scale early prediction of
flare-producing AR emergence, forward-integration of these toroids is necessary. This requires model initialization with a dynamically self-consistent MHD state-vector that includes
magnetic, flow fields, and shell-thickness variations. However, synoptic magnetograms provide only
geometric shape of toroids, not the state-vector needed to initialize MHD-SWT models. To address this challenging task, we develop PINNBARDS, a novel Physics-Informed Neural Network (PINN)-Based AR Distribution Simulator, that uses observational toroids and MHD-SWT equations to derive initial state-vector. Using Feb-14-2024 SDO/HMI synoptic map, we show that PINN converges to physically consistent, predominantly antisymmetric toroids, matching observed ones. Although surface data provides north and south toroids' central latitudes, and their latitudinal widths, they cannot determine tachocline field strengths, connected to AR emergence. We explore here solutions across a broad parameter range, finding hydrodynamically-dominated structures for weak fields (~2 kG) and overly rigid behavior for strong fields (~100 kG). We obtain best agreement with observations for 20-30 kG toroidal fields, and ~10 degree bandwidth, consistent with low-order longitudinal mode excitation. To our knowledge, PINNBARDS serves as the first method for reconstructing state-vectors for hidden tachocline
magnetic structures from surface patterns; potentially leading to weeks ahead prediction of
flare-producing AR-emergence.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
TrashDet: Iterative Neural Architecture Search for Efficient Waste Detection
Tony Tran and Bin Hu
This paper addresses trash detection on the TACO
dataset under strict TinyML constraints using an iterative hardware-aware neural architecture search framework targeting edge and IoT devices. The proposed method constructs a Once-for-All-style ResDets supernet and performs iterative evolutionary search that alternates between backbone and neck/head optimization, supported by a population passthrough mechanism and an accuracy predictor to reduce search cost and improve stability. This framework yields a family of deployment-ready detectors, termed TrashDets. On a five-class TACO subset (paper, plastic, bottle, can, cigarette), the strongest variant, TrashDet-l, achieves 19.5 mAP50 with 30.5M parameters, improving accuracy by up to 3.6 mAP50 over prior detectors while using substantially fewer parameters. The TrashDet family spans 1.2M to 30.5M parameters with mAP50 values between 11.4 and 19.5, providing scalable detector options for diverse TinyML deployment budgets on resource-constrained hardware. On the MAX78002 microcontroller with the TrashNet
dataset, two specialized variants, TrashDet-ResNet and TrashDet-MBNet, jointly dominate the ai87-fpndetector baseline, with TrashDet-ResNet achieving 7525~$\mu$J energy per inference at 26.7 ms latency and 37.45 FPS, and TrashDet-MBNet improving mAP50 by 10.2%; together they reduce energy consumption by up to 88%, latency by up to 78%, and average power by up to 53% compared to existing TinyML detectors.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent
Haipeng Luo, Huawen Feng, Qingfeng Sun et al. (+6 more)
Large
Reasoning Models (LRMs) like o3 and DeepSeek-R1 have achieved remarkable progress in
reasoning tasks with long cot. However, they remain computationally inefficient and struggle with accuracy when solving problems requiring complex mathematical operations. In this work, we present AgentMath, an
agent framework that seamlessly integrates
language models'
reasoning capabilities with code interpreters' computational precision to efficiently tackle complex mathematical problems. Our approach introduces three key innovations: (1) An automated method that converts natural
language chain-of-thought into structured tool-augmented trajectories, generating high-quality supervised fine-tuning (SFT) data to alleviate data scarcity; (2) A novel agentic reinforcement learning (RL) paradigm that dynamically interleaves natural
language generation with real-time code execution. This enables models to autonomously learn optimal tool-use strategies through multi-round interactive feedback, while fostering emergent capabilities in code refinement and error correction; (3) An efficient training system incorporating innovative techniques, including request-level asynchronous rollout scheduling, agentic partial rollout, and prefix-aware weighted load balancing, achieving 4-5x speedup and making efficient RL training feasible on ultra-long sequences with scenarios with massive tool invocation. The evaluations show that AgentMath achieves state-of-the-art performance on challenging mathematical competition
benchmarks including AIME24, AIME25, and HMMT25. Specifically, AgentMath-30B-A3B attains 90.6%, 86.4%, and 73.8% accuracy respectively, surpassing OpenAI-o3-mini and Claude-Opus-4.0-Thinking while remaining competitive with OpenAI-o3, Gemini-2.5-Pro, and DeepSeek-R1-671B-0528.These results validate the effectiveness of our approach and pave the way for building scalable mathematical
reasoning agents.
Read the full abstract ▾
All authors Haipeng Luo, Huawen Feng, Qingfeng Sun, Can Xu, Kai Zheng, Yufei Wang, Tao Yang, Han Hu, Yansong Tang
Machine Learning · Cross-listed from cs.NI
AI-Driven Green Cognitive Radio Networks for Sustainable 6G Communication
Anshul Sharma, Shujaatali Badami, Biky Chouhan et al. (+3 more)
The 6G wireless aims at the Tb/s peak data rates are expected, a sub-millisecond latency, massive Internet of Things/vehicle connectivity, which requires sustainable access to audio over the air and energy-saving functionality. Cognitive
Radio Networks CCNs help in alleviating the problem of spectrum scarcity, but classical sensing and allocation are still energy-consumption intensive, and sensitive to rapid spectrum variations. Our framework which centers on AI driven green CRN aims at integrating deep reinforcement learning (DRL) with transfer learning, energy harvesting (EH), reconfigurable intelligent surfaces (RIS) with other light-weight genetic refinement operations that optimally combine sensing timelines, transmit power, bandwidth distribution and RIS phase selection. Compared to two baselines, the utilization of MATLAB + NS-3 under dense loads, a traditional CRN with energy sensing under fixed policies, and a hybrid CRN with cooperative sensing under heuristic distribution of resource, there are (25-30%) fewer energy reserves used, sensing AUC greater than 0.90 and +6-13 p.p. higher PDR. The integrated framework is easily scalable to large IoT and vehicular applications, and it provides a feasible and sustainable roadmap to 6G CRNs. Index Terms–Cognitive
Radio Networks (CRNs), 6G, Green Communication, Energy Efficiency, Deep Reinforcement Learning (DRL), Spectrum Sensing, RIS, Energy Harvesting, QoS, IoT.
Read the full abstract ▾
All authors Anshul Sharma, Shujaatali Badami, Biky Chouhan, Pushpanjali Pandey, Brijeena Rana, Navneet Kaur
Machine Learning
FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
Saeed Mohammadzadeh, Erfan Hamdi, Joel Shor et al. (+1 more)
As LLMs advance their
reasoning capabilities about the physical world, the absence of rigorous
benchmarks for evaluating their ability to generate scientifically valid physical models has become a critical gap. Computational mechanics, which develops and applies mathematical models and numerical methods to predict the behavior of physical systems under forces, deformation, and constraints, provides an ideal foundation for structured scientific
reasoning evaluation. Problems follow clear mathematical structure, enforce strict physical and numerical constraints, and support objective verification. The discipline requires constructing explicit models of physical systems and
reasoning about geometry, spatial relationships, and material behavior, connecting directly to emerging AI goals in physical
reasoning and world modeling. We introduce FEM-Bench, a computational mechanics
benchmark designed to evaluate the ability of LLMs to generate correct finite element method (FEM) and related code. FEM-Bench 2025 contains a suite of introductory but nontrivial tasks aligned with material from a first graduate course on computational mechanics. These tasks capture essential numerical and physical modeling challenges while representing only a small fraction of the complexity present in the discipline. Despite their simplicity, state-of-the-art LLMs do not reliably solve all of them. In a five attempt run, the best performing model at function writing, Gemini 3 Pro, completed 30/33 tasks at least once and 26/33 tasks all five times. The best performing model at unit test writing, GPT-5, had an Average Joint Success Rate of 73.8%. Other popular models showed broad performance variation. FEM-Bench establishes a structured foundation for evaluating AI-generated scientific code, and future iterations will incorporate increasingly sophisticated tasks to track progress as models evolve.
Read the full abstract ▾
All authors Saeed Mohammadzadeh, Erfan Hamdi, Joel Shor, Emma Lejeune
Machine Learning · Cross-listed from cs.CR
Real-World Adversarial Attacks on RF-Based Drone Detectors
Omer Gazit, Yael Itzhakev, Yuval Elovici et al. (+1 more)
Radio frequency (RF) based systems are increasingly used to detect drones by analyzing their RF signal patterns, converting them into spectrogram images which are processed by object detection models. Existing RF attacks against image based models alter digital features, making over-the-air (OTA) implementation difficult due to the challenge of converting digital perturbations to transmittable waveforms that may introduce synchronization errors and interference, and encounter hardware limitations. We present the first physical attack on RF image based drone detectors, optimizing class-specific universal complex baseband (I/Q) perturbation waveforms that are transmitted alongside legitimate communications. We evaluated the attack using RF recordings and OTA experiments with four types of drones. Our results show that modest, structured I/Q perturbations are compatible with standard RF chains and reliably reduce target drone detection while preserving detection of legitimate drones.
Read the full abstract ▾
All authors Omer Gazit, Yael Itzhakev, Yuval Elovici, Asaf Shabtai
Machine Learning · Cross-listed from cs.AI
Runtao Liu, Ziyi Liu, Jiaqi Tang et al. (+4 more)
Recent advances in
multimodal LLMs and systems that use tools for long-
video QA point to the promise of
reasoning over hour-long episodes. However, many methods still compress content into lossy summaries or rely on limited toolsets, weakening temporal grounding and missing fine-grained cues. We propose a multi-
agent framework in which a master LLM coordinates a grounding
agent to localize question-relevant segments and a
vision agent to extract targeted textual observations. The master
agent plans with a step limit, and is trained with reinforcement learning to encourage concise, correct, and efficient multi-
agent cooperation. This design helps the master
agent focus on relevant clips via grounding, complements subtitles with visual detail, and yields interpretable trajectories. On our proposed LongTVQA and LongTVQA+ which are episode-level
datasets aggregated from TVQA/TVQA+, our multi-
agent system significantly outperforms strong non-
agent baselines. Experiments also show reinforcement learning further strengthens
reasoning and planning for the trained
agent. Code and data will be shared at https://longvideoagent.github.io/.
Read the full abstract ▾
All authors Runtao Liu, Ziyi Liu, Jiaqi Tang, Yue Ma, Renjie Pi, Jipeng Zhang, Qifeng Chen
Machine Learning · Cross-listed from cs.CV
FedPOD: the deployable units of training for federated learning
Daewoon Kim, Si Young Yie and Jae Sung Lee
This paper proposes FedPOD, which ranked first in the 2024
Federated Tumor
Segmentation (FeTS) Challenge, for optimizing learning efficiency and communication cost in
federated learning among multiple clients. Inspired by FedPIDAvg, we define a round-wise task for FedPOD to enhance training efficiency. FedPIDAvg achieved performance improvement by incorporating the training loss reduction for prediction
entropy as weights using differential terms. Furthermore, by modeling data distribution with a Poisson distribution and using a PID controller, it reduced communication costs even in skewed data distribution. However, excluding participants classified as outliers based on the Poisson distribution can limit data utilization. Additionally, PID controller requires the same participants to be maintained throughout the
federated learning process as it uses previous rounds' learning information in the current round. In our approach, FedPOD addresses these issues by including participants excluded as outliers, eliminating dependency on previous rounds' learning information, and applying a method for calculating validation loss at each round. In this challenge, FedPOD presents comparable performance to FedPIDAvg in metrics of Dice score, 0.78, 0.71 and 0.72 for WT, ET and TC in average, and projected convergence score, 0.74 in average. Furthermore, the concept of FedPOD draws inspiration from Kubernetes' smallest computing unit, POD, designed to be compatible with Kubernetes auto-scaling. Extending round-wise tasks of FedPOD to POD units allows flexible design by applying scale-out similar to Kubernetes' auto-scaling. This work demonstrated the potentials of FedPOD to enhance
federated learning by improving efficiency, flexibility, and performance in metrics.
Read the full abstract ▾
Machine Learning
Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures
Yedi Zhang, Andrew Saxe and Peter E. Latham
Neural networks trained with gradient descent often learn solutions of increasing complexity over time, a phenomenon known as simplicity bias. Despite being widely observed across architectures, existing theoretical treatments lack a unifying framework. We present a theoretical framework that explains a simplicity bias arising from saddle-to-saddle learning dynamics for a general class of neural networks, incorporating fully-connected, convolutional, and
attention-based architectures. Here, simple means expressible with few hidden units, i.e., hidden
neurons, convolutional kernels, or
attention heads. Specifically, we show that linear networks learn solutions of increasing rank, ReLU networks learn solutions with an increasing number of kinks, convolutional networks learn solutions with an increasing number of convolutional kernels, and self-
attention models learn solutions with an increasing number of
attention heads. By analyzing fixed points, invariant
manifolds, and dynamics of gradient descent learning, we show that saddle-to-saddle dynamics operates by iteratively evolving near an invariant
manifold, approaching a saddle, and switching to another invariant
manifold. Our analysis also disentangles data-induced and initialization-induced saddle-to-saddle dynamics. In particular, the former leads to low-rank weights while the latter to sparse weights. Equipped with the theory, we predict the effects of data distribution and weight initialization on the duration and number of plateaus in learning. Overall, our theory offers a framework for understanding when and why gradient descent progressively learns increasingly complex solutions.
Read the full abstract ▾
Machine Learning
Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning
Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel et al. (+12 more)
Large-scale autoregressive models pretrained on next-
token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. During RL, these models explore by generating new outputs, one
token at a time. However, sampling actions
token-by-
token can result in highly inefficient learning, particularly when rewards are sparse. Here, we show that it is possible to overcome this problem by acting and exploring within the internal representations of an autoregressive model. Specifically, to discover temporally-abstract actions, we introduce a higher-order, non-causal sequence model whose outputs control the residual stream activations of a base autoregressive model. On grid world and MuJoCo-based tasks with hierarchical structure, we find that the higher-order model learns to compress long activation sequence chunks onto internal controllers. Critically, each controller executes a sequence of behaviorally meaningful actions that unfold over long timescales and are accompanied with a learned termination condition, such that composing multiple controllers over time leads to efficient exploration on novel tasks. We show that direct internal controller reinforcement, a process we term "internal RL", enables learning from sparse rewards in cases where standard RL finetuning fails. Our results demonstrate the benefits of latent action generation and reinforcement in autoregressive models, suggesting internal RL as a promising avenue for realizing hierarchical RL within foundation models.
Read the full abstract ▾
All authors Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel, Angelika Steger, Maciej Wolczyk, Johannes von Oswald, Nino Scherrer, Kaitlin Maile, Guillaume Lajoie, Blake A. Richards, Rif A. Saurous, James Manyika, Blaise Agüera y Arcas, Alexander Meulemans, João Sacramento
Machine Learning
Relu and softplus neural nets as zero-sum turn-based games
Stephane Gaubert and Yiannis Vlassopoulos
We show that the output of a ReLU neural network can be interpreted as the value of a zero-sum, turn-based, stopping game, which we call the ReLU net game. The game runs in the direction opposite to that of the network, and the input of the network serves as the terminal reward of the game. In fact, evaluating the network is the same as running the Shapley-Bellman backward recursion for the value of the game. Using the expression of the value of the game as an expected total payoff with respect to the path measure induced by the transition probabilities and a pair of optimal policies, we derive a discrete Feynman-Kac-type path-integral formula for the network output. This game-theoretic representation can be used to derive bounds on the output from bounds on the input, leveraging the monotonicity of Shapley operators, and to verify robustness properties using policies as certificates. Moreover, training the neural network becomes an inverse game problem: given pairs of terminal rewards and corresponding values, one seeks transition probabilities and rewards of a game that reproduces them. Finally, we show that a similar approach applies to neural networks with Softplus activation functions, where the ReLU net game is replaced by its entropic regularization.
Read the full abstract ▾
Machine Learning
Improving ML Training Data with Gold-Standard Quality Metrics
Leslie Barrett and Michael W. Sherman
Hand-tagged training data is essential to many machine learning tasks. However, training data quality control has received little
attention in the literature, despite data quality varying considerably with the tagging exercise. We propose methods to evaluate and enhance the quality of hand-tagged training data using statistical approaches to measure tagging consistency and agreement. We show that agreement metrics give more reliable results if recorded over multiple iterations of tagging, where declining variance in such recordings is an indicator of increasing data quality. We also show one way a tagging project can collect high-quality training data without requiring multiple tags for every work item, and that a tagger burn-in period may not be sufficient for minimizing tagger errors.
Read the full abstract ▾
Machine Learning
Performative Policy Gradient: Optimality in Performative Reinforcement Learning
Debabrota Basu, Udvas Das, Brahim Driss et al. (+1 more)
Post-deployment machine learning
algorithms often influence the environments they act in, and thus shift the underlying dynamics that the standard reinforcement learning (RL) methods ignore. While designing optimal
algorithms in this performative setting has recently been studied in supervised learning, the RL counterpart remains under-explored. In this paper, we prove the performative counterparts of the performance difference lemma and the policy gradient
theorem in RL, and further introduce the Performative Policy Gradient
algorithm (PePG). PePG is the first policy gradient
algorithm designed to account for performativity in RL. Under softmax parametrisation, and also with and without
entropy regularisation, we prove that PePG converges to performatively optimal policies, i.e. policies that remain optimal under the distribution shifts induced by themselves. Thus, PePG significantly extends the prior works in Performative RL that achieves performative stability but not optimality. Furthermore, our empirical analysis on standard performative RL environments validate that PePG outperforms the existing performative RL
algorithms aiming for stability.
Read the full abstract ▾
All authors Debabrota Basu, Udvas Das, Brahim Driss, Uddalak Mukherjee
Machine Learning
Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
Rui Pan, Zhuofu Chen, Hongyi Liu et al. (+2 more)
Diffusion Large
Language Models (dLLMs) offer fast, parallel
token generation, but their standalone use is plagued by an inherent efficiency-quality tradeoff. We show that, if carefully applied, the attributes of dLLMs can actually be a strength for drafters in speculative decoding with autoregressive (AR) verifiers. Our core insight is that dLLM's speed from parallel decoding drastically lowers the risk of costly rejections, providing a practical mechanism to effectively realize the (elusive) lengthy drafts that lead to large speedups with speculative decoding. We present FailFast, a dLLM-based speculative decoding framework that realizes this approach by dynamically adapting its speculation length. It "fails fast" by spending minimal compute in hard-to-speculate regions to shrink speculation latency and "wins big" by aggressively extending draft lengths in easier regions to reduce verification latency (in many cases, speculating and accepting 70
tokens at a time!). Without any fine-tuning, FailFast delivers lossless acceleration of AR LLMs and achieves up to 4.9$\times$ speedup over vanilla decoding, 1.7$\times$ over the best naive dLLM drafter, and 1.7$\times$ over EAGLE-3 across diverse models and workloads. We open-source FailFast at https://github.com/ruipeterpan/failfast.
Read the full abstract ▾
All authors Rui Pan, Zhuofu Chen, Hongyi Liu, Arvind Krishnamurthy, Ravi Netravali
Machine Learning · Cross-listed from cs.CV
LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving
Long Nguyen, Micha Fauth, Bernhard Jaeger et al. (+4 more)
Simulators can generate virtually unlimited driving data, yet imitation learning policies in simulation still struggle to achieve robust closed-loop performance. Motivated by this gap, we empirically study how misalignment between privileged expert demonstrations and sensor-based student observations can limit the effectiveness of imitation learning. More precisely, experts have significantly higher visibility (e.g., ignoring occlusions) and far lower uncertainty (e.g., knowing other vehicles' actions), making them difficult to imitate reliably. Furthermore, navigational intent (i.e., the route to follow) is under-specified in student models at test time via only a single target point. We demonstrate that these asymmetries can measurably limit driving performance in CARLA and offer practical interventions to address them. After careful modifications to narrow the gaps between expert and student, our TransFuser v6 (TFv6) student policy achieves a new state of the art on all major publicly available CARLA closed-loop
benchmarks, reaching 95 DS on Bench2Drive and more than doubling prior performances on Longest6~v2 and Town13. Additionally, by integrating perception supervision from our
dataset into a shared sim-to-real pipeline, we show consistent gains on the NAVSIM and Waymo
Vision-Based End-to-End driving
benchmarks. Our code, data, and models are publicly available at https://github.com/autonomousvision/lead.
Read the full abstract ▾
All authors Long Nguyen, Micha Fauth, Bernhard Jaeger, Daniel Dauner, Maximilian Igl, Andreas Geiger, Kashyap Chitta
Machine Learning · Cross-listed from stat.ML
Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attention
Yingzhen Yang
We study the problem of learning a low-degree spherical
polynomial of degree $\ell_0 = \Theta(1) \ge 1$ defined on the unit sphere in $\RR^d$ by training an over-parameterized two-layer neural network (NN) with channel
attention in this paper. Our main result is the significantly improved sample complexity for learning such low-degree
polynomials. We show that, for any regression risk $\eps \in (0,1)$, a carefully designed two-layer NN with channel
attention and finite width trained by the vanilla gradient descent (GD) requires the lowest sample complexity of $n \asymp \Theta(d^{\ell_0}/\eps)$ with high probability, in contrast with the representative sample complexity $\Theta\pth{d^{\ell_0} \max\set{\eps^{-2},\log d}}$, where $n$ is the training data size. Moreover, such sample complexity is not improvable since the trained network renders a sharp rate of the nonparametric regression risk of the order $\Theta(d^{\ell_0}/{n})$ with high probability. On the other hand, the minimax optimal rate for the regression risk with a kernel of rank $\Theta(d^{\ell_0})$ is $\Theta(d^{\ell_0}/{n})$, so that the rate of the nonparametric regression risk of the network trained by GD is minimax optimal. Training the two-layer NN with channel
attention proceeds in two stages: (1) a provable learnable channel selection
algorithm, as a learnable harmonic-degree selection process, identifies the ground truth channel number in the target function, $\ell_0$, from $L \ge \ell_0$ channels in the first-layer activation; (2) the second layer is trained by standard GD using the selected channels. To the best of our knowledge, this is the first time a minimax optimal risk bound is obtained by training an over-parameterized but finite-width neural network with feature learning capability to learn low-degree spherical
polynomials.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
Over-the-Air Goal-Oriented Communications
Kyriakos Stylianopoulos, Paolo Di Lorenzo and George C. Alexandropoulos
Goal-oriented communications offer an attractive alternative to the Shannon-based communication paradigm, where the data is never reconstructed at the Receiver (RX) side. Rather, focusing on the case of edge inference, the Transmitter (TX) and the RX cooperate to exchange features of the input data that will be used to predict an unseen attribute of them, leveraging information from collected data sets. This chapter demonstrates that the wireless channel can be used to perform computations over the data, when equipped with programmable metasurfaces. The end-to-end system of the TX, RX, and MS-based channel is treated as a single deep neural network which is trained through backpropagation to perform inference on unseen data. Using Stacked Intelligent Metasurfaces (SIM), it is shown that this Metasurfaces-Integrated Neural Network (MINN) can achieve performance comparable to fully digital neural networks under various system parameters and data sets. By offloading computations onto the channel itself, important benefits may be achieved in terms of energy consumption, arising from reduced computations at the transceivers and smaller transmission power required for successful inference.
Read the full abstract ▾
Machine Learning · Cross-listed from econ.EM
ScoreMatchingRiesz: Score Matching for Debiased Machine Learning and Policy Path Estimation
Masahiro Kato
We propose ScoreMatchingRiesz, a family of Riesz representer
estimators based on score matching. The Riesz representer is a key nuisance component in debiased machine learning, enabling $\sqrt{n}$-consistent and asymptotically efficient estimation of causal and structural targets via Neyman-orthogonal scores. We formulate Riesz representer estimation as a score estimation problem. This perspective stabilizes representer estimation by allowing us to leverage denoising score matching and telescoping density ratio estimation. We also introduce the policy path, a parameter that captures how policy effects evolve under continuous treatments. We show that the policy path can be estimated via score matching by smoothly connecting average marginal effect (AME) and average policy effect (APE) estimation, which improves the interpretability of policy effects.
Read the full abstract ▾
Machine Learning
Explainable time-series forecasting with sampling-free SHAP for Transformers
Matthias Hertel, Sebastian Pütz, Ralf Mikut et al. (+2 more)
Time-series forecasts are essential for planning and decision-making in many domains. Explainability is key to building user trust and meeting transparency requirements. Shapley Additive Explanations (SHAP) is a popular explainable AI framework, but it lacks efficient implementations for time series and often assumes feature independence when sampling counterfactuals. We introduce SHAPformer, an accurate, fast and sampling-free explainable time-series
forecasting model based on the
Transformer architecture. It leverages
attention manipulation to make predictions based on feature subsets. SHAPformer generates explanations in under one second, several orders of magnitude faster than the SHAP Permutation Explainer. On synthetic data with ground truth explanations, SHAPformer provides explanations that are true to the data. Applied to real-world electrical load data, it achieves competitive predictive performance and delivers meaningful local and global insights, such as identifying the past load as the key predictor and revealing a distinct model behavior during the Christmas period.
Read the full abstract ▾
All authors Matthias Hertel, Sebastian Pütz, Ralf Mikut, Veit Hagenmeyer, Benjamin Schäfer
Machine Learning
Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
Tyler Clark, Christine Evers and Jonathon Hare
Recurrent off-policy deep reinforcement learning models achieve state-of-the-art performance but are often sidelined due to their high computational demands. In response, we introduce RISE (Recurrent Integration via Simplified Encodings), a novel approach that can leverage recurrent networks in any image-based off-policy RL setting without significant computational overheads via using both learnable and non-learnable encoder layers. When integrating RISE into leading non-recurrent off-policy RL
algorithms, we observe a 35.6% human-normalized interquartile mean (IQM) performance improvement across the Atari
benchmark. We analyze various implementation strategies to highlight the versatility and potential of our proposed framework.
Read the full abstract ▾
Machine Learning · Cross-listed from q-fin.ST
The Aligned Economic Index & The State Switching Model
Ilias Aarab
A growing empirical literature suggests that equity-premium predictability is state dependent, with much of the
forecasting power concentrated around recessionary periods (Henkel et al., 2011; Dangl and Halling, 2012; Devpura et al., 2018). I study U.S. stock return predictability across economic regimes and document strong evidence of time-varying expected returns across both expansionary and contractionary states. I contribute in two ways. First, I introduce a state-switching predictive regression in which the
market state is defined in real time using the slope of the yield curve. Relative to the standard one-state predictive regression, the state-switching specification increases both in-sample and out-of-sample performance for the set of popular predictors considered by Welch and Goyal (2008), improving the out-of-sample performance of most predictors in economically meaningful ways. Second, I propose a new aggregate predictor, the Aligned Economic Index, constructed via partial least squares (PLS). Under the state-switching model, the Aligned Economic Index exhibits statistically and economically significant predictive power in sample and out of sample, and it outperforms widely used
benchmark predictors and alternative predictor-combination methods.
Read the full abstract ▾
Machine Learning
Machine Learning to Predict Digital Frustration from Clickstream Data
Jibin Joseph
Many businesses depend on their mobile apps and websites, so user frustration while trying to complete a task on these channels can cause lost sales and complaints. In this research, I use clickstream data from a real e-commerce site to predict whether a session is frustrated or not. Frustration is defined using certain rules based on rage bursts, back and forth
navigation (U turns), cart churn, search struggle, and long wandering sessions, and applies these rules to 5.4 million raw clickstream events (304,881 sessions). From each session, I build tabular features and train standard classifier models. I also use the full event sequence to train a discriminative LSTM classifier. XGBoost reaches about 90% accuracy, ROC AUC of 0.9579, while the LSTM performs best with about 91% accuracy and a ROC AUC of 0.9705. Finally, the research shows that with only the first 20 to 30 interactions, the LSTM already predicts frustration reliably.
Read the full abstract ▾
Machine Learning
Simplifying Multi-Task Architectures Through Task-Specific Normalization
Mihai Suteu and Ovidiu Serban
Multi-task learning (MTL) aims to leverage shared knowledge across tasks to improve generalization and parameter efficiency, yet balancing resources and mitigating interference remain open challenges. Architectural solutions often introduce elaborate task-specific modules or routing schemes, increasing complexity and overhead. In this work, we show that normalization layers alone are sufficient to address many of these challenges. Simply replacing shared normalization with task-specific variants already yields competitive performance, questioning the need for complex designs. Building on this insight, we propose Task-Specific Sigmoid Batch Normalization (TS$\sigma$BN), a lightweight mechanism that enables tasks to softly allocate network capacity while fully sharing feature extractors. TS$\sigma$BN improves stability across CNNs and
Transformers, matching or exceeding performance on NYUv2, Cityscapes, CelebA, and PascalContext, while remaining highly parameter-efficient. Moreover, its learned gates provide a natural framework for analyzing MTL dynamics, offering interpretable insights into capacity allocation, filter specialization, and task relationships. Our findings suggest that complex MTL architectures may be unnecessary and that task-specific normalization offers a simple, interpretable, and efficient alternative.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SD
AUDRON: A Deep Learning Framework with Fused Acoustic Signatures for Drone Type Recognition
Rajdeep Chatterjee, Sudip Chakrabarty, Trishaani Acharjee et al. (+1 more)
Unmanned aerial vehicles (
UAVs), commonly known as drones, are increasingly used across diverse domains, including logistics, agriculture, surveillance, and defense. While these systems provide numerous benefits, their misuse raises safety and security concerns, making effective detection mechanisms essential. Acoustic sensing offers a low-cost and non-intrusive alternative to
vision or radar-based detection, as drone propellers generate distinctive sound patterns. This study introduces AUDRON (AUdio-based Drone Recognition Network), a hybrid deep learning framework for drone sound detection, employing a combination of Mel-Frequency Cepstral Coefficients (MFCC), Short-Time
Fourier Transform (STFT) spectrograms processed with convolutional neural networks (CNNs), recurrent layers for temporal modeling, and autoencoder-based representations. Feature-level fusion integrates complementary information before classification. Experimental evaluation demonstrates that AUDRON effectively differentiates drone acoustic signatures from background noise, achieving high accuracy while maintaining generalizability across varying conditions. AUDRON achieves 98.51 percent and 97.11 percent accuracy in binary and multiclass classification. The results highlight the advantage of combining multiple feature representations with deep learning for reliable acoustic drone detection, suggesting the framework's potential for deployment in security and surveillance applications where visual or radar sensing may be limited.
Read the full abstract ▾
All authors Rajdeep Chatterjee, Sudip Chakrabarty, Trishaani Acharjee, Deepanjali Mishra
Machine Learning
BRIDGE: Budget-aware Reasoning via Intermediate Distillation with Guided Examples
Xuan-An Le, Minh-Nam Tran and Son Nguyen
Distilling knowledge from large proprietary models (e.g., GPT-4) to tiny deployable models (less than 1B parameters) faces a critical capacity-budget trap: the 1000x capacity gap between teachers and students prevents effective direct transfer, while API costs prohibit extensive data collection. We introduce BRIDGE (Budget-Aware
Reasoning via Intermediate
Distillation), a two-phase framework that resolves these constraints through strategic intermediation and budget asymmetry. In Phase 1, a mid-sized Teacher Assistant (TA; e.g., about 7B) learns from the black-box teacher on a strictly limited subset of data (e.g., 3-5%), selected via a zero-API-cost pipeline that balances entropic difficulty and semantic diversity using only local TA inference. In Phase 2, we exploit this asymmetry-teacher queries are expensive, whereas TA inference is free to amplify supervision: the refined TA generates synthetic rationales for the full
dataset to train the tiny student. Crucially, we apply an instruction-tuning curriculum to establish behavioral
alignment in the tiny student before transferring
reasoning. Our theoretical analysis shows that BRIDGE yields tighter generalization bounds than direct
distillation when data is abundant. Experiments across medical, legal, and financial
benchmarks demonstrate consistent improvements: BRIDGE delivers student performance gains of 28-41%, closing the capability gap with proprietary teachers by 12-16% while using 10x fewer teacher queries. Notably, BRIDGE defies the conventional cost-performance frontier, surpassing direct
distillation baselines that use 100% of the budget while consuming only 5% of the resources.
Read the full abstract ▾
Machine Learning
GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer
Corey Adams, Rishikesh Ranade, Ram Cherukuri et al. (+1 more)
We present GeoTransolver, a multiscale geometry-aware physics
attention transformer for Computer Aided Engineering (CAE). GeoTransolver extends the Transolver backbone with GALE (Geometry-Aware Latent
Embeddings)
attention, which pairs physics-aware self-
attention on learned state slices with cross-
attention to a shared geometry and global context computed via multi-scale ball queries (inspired by Domino) and reused in every block. Implemented and released in NVIDIA PhysicsNeMo, GeoTransolver persistently projects geometry and global parameters, into physical state spaces to anchor computations to domain structure and operating regimes. We
benchmark on DrivAerML, SHIFT-SUV, and SHIFT-Wing against Domino, Transolver (PhysicsNeMo implementation), and literature-reported AB-UPT, evaluating drag/lift R2 and relative L1 errors on field variables. As an additional nonlinear structural mechanics application, we also report Transolver and GeoTransolver results on bumper-beam and full-vehicle Body-in-White (BIW) crash-dynamics
benchmarks, evaluating relative L2 trajectory error and probe-level kinematic MSE. GeoTransolver delivers improved accuracy, robustness to geometry and regime shifts, and favorable data efficiency; we include DrivAerML ablations and qualitative contour and design-trend results, advancing operator learning for high-fidelity surrogates on complex, irregular, non-linear domains.
Read the full abstract ▾
All authors Corey Adams, Rishikesh Ranade, Ram Cherukuri, Sanjay Choudhry
Machine Learning · Cross-listed from stat.ML
Avoiding the Price of Adaptivity: Inference in Linear Contextual Bandits via Stability
Samya Praharaj and Koulik Khamaru
Statistical inference in contextual bandits is challenging due to the adaptive, non-i.i.d. nature of the data. A growing body of work shows that classical least-squares inference can fail under adaptive sampling, and that valid confidence intervals for linear functionals typically require an inflation of order $\sqrt{d \log T}$. This phenomenon – often termed the price of adaptivity – reflects the intrinsic difficulty of reliable inference under general contextual bandit policies. A key structural condition that overcomes this limitation is the stability condition of Lai and Wei, which requires the empirical feature covariance to converge to a deterministic limit. When stability holds, the ordinary least-squares
estimator satisfies a central limit
theorem, and classical Wald-type confidence intervals remain asymptotically valid under adaptation, without incurring the $\sqrt{d \log T}$ price of adaptivity. In this paper, we propose and analyze a regularized EXP4
algorithm for linear contextual bandits. Our first main result shows that this procedure satisfies the Lai–Wei stability condition and therefore admits valid Wald-type confidence intervals for linear functionals. We additionally provide quantitative rates of convergence in the associated central limit
theorem. Our second result establishes that the same
algorithm achieves regret guarantees that are minimax optimal up to logarithmic factors, demonstrating that stability and statistical efficiency can coexist within a single contextual bandit method. As an application of our theory, we show how it can be used to construct confidence intervals for the conditional average treatment effect (CATE) under adaptively collected data. Finally, we complement our theory with simulations illustrating the empirical normality of the resulting
estimators and the sharpness of the corresponding confidence intervals.
Read the full abstract ▾
Machine Learning
Clust-PSI-PFL: A Population Stability Index Approach for Clustered Non-IID Personalized Federated Learning
Daniel M. Jimenez-Gutierrez, Mehrdad Hassanzadeh, David Solans et al. (+5 more)
Federated learning (FL) supports
privacy-preserving, decentralized machine learning (ML) model training by keeping data on client devices. However, non-independent and identically distributed (non-IID) data across clients biases updates and degrades performance. To alleviate these issues, we propose Clust-PSI-PFL, a clustering-based personalized FL framework that uses the Population Stability Index (PSI) to quantify the level of non-IID data. We compute a weighted PSI metric, $WPSI^L$, which we show to be more informative than common non-IID metrics (Hellinger, Jensen-Shannon, and Earth Mover's distance). Using PSI features, we form distributionally homogeneous groups of clients via K-means++; the number of optimal clusters is chosen by a systematic silhouette-based procedure, typically yielding few clusters with modest overhead. Across six
datasets (tabular, image, and text modalities), two partition protocols (Dirichlet with parameter $\alpha$ and Similarity with parameter S), and multiple client sizes, Clust-PSI-PFL delivers up to 18% higher global accuracy than state-of-the-art baselines and markedly improves client
fairness by a relative improvement of 37% under severe non-IID data. These results establish PSI-guided clustering as a principled, lightweight mechanism for robust PFL under label skew.
Read the full abstract ▾
All authors Daniel M. Jimenez-Gutierrez, Mehrdad Hassanzadeh, David Solans, Mohammed Elbamby, Nicolas Kourtellis, Aris Anagnostopoulos, Ioannis Chatzigiannakis, Andrea Vitaletti