Machine Learning
Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation
Srujan P Mule, Aniketh Garikaparthi and Manasi Patwardhan
As
language models accelerate scientific research by automating hypothesis generation and implementation, a new bottleneck emerges: evaluating and filtering hundreds of AI-generated ideas without exhaustive experimentation. We ask whether LMs can learn to forecast the empirical success of research ideas before any experiments are run. We study comparative empirical
forecasting: given a
benchmark-specific research goal and two candidate ideas, predict which will achieve better
benchmark performance. We construct a
dataset of 11,488 idea pairs grounded in objective outcomes from PapersWithCode. While off-the-shelf 8B-parameter models struggle (30% acc.), SFT dramatically boosts performance to 77.1%, outperforming GPT-5 (61.1%). By framing evaluation as a
reasoning task via Reinforcement Learning with Verifiable Rewards (RLVR), we train models to discover latent
reasoning paths, achieving 71.35% acc. with interpretable justifications. Through additional ablations and out-of-distribution tests, we show robustness to surface-level heuristics and transfer to both a cross-domain time-split test set and an independently constructed test set. Our results demonstrate that compute-efficient small
language models can serve as effective, objective verifiers, offering a scalable path for autonomous scientific discovery.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Long-Context Reasoning Through Proxy-Based Chain-of-Thought Tuning
Miao Li, Irina Saparina, Alexander Gurung et al. (+1 more)
Recent large
language models support inputs of up to 10 million
tokens, yet they perform poorly on long-context tasks that require complex
reasoning. Such tasks can be solved using only a subset of the input – a proxy context – rather than the full sequence. Despite sharing the same underlying
reasoning process, models exhibit a significant performance disparity between proxy and full contexts. To improve long-context
reasoning, we propose ProxyCoT, a novel training framework that transfers
reasoning capabilities from short proxy contexts to full long contexts. Specifically, we first obtain high-quality chain-of-thought
reasoning traces on proxy contexts through reinforcement learning or
distillation from a larger teacher model, and then ground the generated traces in full long contexts with supervised fine-tuning. Experiments across different
datasets demonstrate that ProxyCoT consistently outperforms strong baselines with reduced computational overhead. Furthermore, models trained with ProxyCoT generalize their long-context
reasoning capabilities to out-of-domain tasks.
Read the full abstract ▾
All authors Miao Li, Irina Saparina, Alexander Gurung, Mirella Lapata
Machine Learning · Cross-listed from cs.HC
Beyond Adoption Intention How Trust in Augmented Analytics Relates to Perceived Decision Quality Among Non-Technical BI Users
Thuy Pham Thi Phuong, Hieu Vu Le Trung, Ha Nguyen Manh et al. (+2 more)
Augmented analytics has transformed how Business Intelligence (BI) systems support decision-making, shifting non-technical managers from manual analysis toward dependence on automated insights. Current BI research often overlooks the cognitive mechanisms and the direct impact of AI-enabled analytics on decision quality. This study employs the theory of cognitive delegation to investigate the association between trust in augmented analytics and perceived decision quality among non-technical BI users. Data were collected from 250 business professionals across various organizational roles in Vietnam between January and March 2025 and analyzed using partial least squares structural equation modeling (PLS-SEM). Findings indicate that augmented analytics capabilities are positively associated with perceived ease of use, usefulness, and trust in BI systems. Trust and usefulness are jointly associated with BI adoption intention and perceived decision quality. Notably, trust is positively related to perceived decision quality, as observed within the studied sample of non-specialist users. By framing augmented analytics as cognitive delegation, this study expands BI adoption research to include perceived decision outcomes and contributes to the understanding of human-AI interaction in organizations.
Read the full abstract ▾
All authors Thuy Pham Thi Phuong, Hieu Vu Le Trung, Ha Nguyen Manh, Nhi Tran Pham Yen, Lan Hoang Thi
Machine Learning · Cross-listed from cs.AI
CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing
Cheng Qian, Hyeonjeong Ha, Jiayu Liu et al. (+10 more)
Recent advances in large
language models have led to strong performance on
reasoning and environment-interaction tasks, yet their ability for creative problem-solving remains underexplored. We study this capability through the lens of creative tool use, where a model repurposes available objects by
reasoning about their affordances and attributes rather than relying on canonical usage. As a first step, we introduce CreativityBench, a
benchmark for evaluating affordance-based creativity in LLMs. To this end, we build a large-scale affordance knowledge base (KB) with 4K entities and 150K+ affordance annotations, explicitly linking objects, parts, attributes, and actionable uses. Building on this KB, we generate 14K grounded tasks that require identifying non-obvious yet physically plausible solutions under constraints. Evaluations across 10 state-of-the-art LLMs, including closed and open-source models, show that models can often select a plausible object, but fail to identify the correct parts, their affordances, and the underlying physical mechanism needed to solve the task, leading to a significant drop in performance. Furthermore, improvements from model scaling quickly saturate, strong general
reasoning does not reliably translate to creative affordance discovery, and common inference-time strategies such as Chain-of-Thought yield limited gains. These results suggest that creative tool use remains a major challenge for current models, and that CreativityBench provides a useful testbed for studying this missing dimension of intelligence, with potential implications for planning and
reasoning modules in future
agents.
Read the full abstract ▾
All authors Cheng Qian, Hyeonjeong Ha, Jiayu Liu, Jeonghwan Kim, Jiateng Liu, Bingxuan Li, Aditi Tiwari, Dwip Dalal, Zhenhailong Wang, Xiusi Chen, Mahdi Namazifar, Yunzhu Li, Heng Ji
Machine Learning
Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
Kazuki Egashira, Mark Vero, Jasper Dekoninck et al. (+3 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the
reasoning capabilities of large
language models (LLMs). While RLVR is designed for tasks with verifiable ground-truth answers, real-world verifiers (e.g., static code checkers) can introduce errors into the reward signal. Prior analyses have largely treated such errors as random and independent across samples, concluding that errors merely slow training with limited effect on final performance. However, practical verifiers tend to exhibit systematic errors. This introduces a risk of models learning unwanted consistent behavior from a structurally incorrect reward signal. In this work, we study the impact of such systematic verification errors on RLVR. Through controlled experiments on arithmetic tasks, we show that systematic false negatives lead to similar effects as random noise. On the other hand, systematic false positives can cause a wide range of behaviors from sub-optimal plateaus to performance collapse. Crucially, these outcomes are not determined by the overall error rate but by the specific pattern of introduced errors, making pre-hoc mitigation difficult. Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes and that verifier quality has to be understood beyond its sample-level error rate.
Read the full abstract ▾
All authors Kazuki Egashira, Mark Vero, Jasper Dekoninck, Florian E. Dorner, Robin Staab, Martin Vechev
Machine Learning · Cross-listed from cs.CV
Memorization In Stable Diffusion Is Unexpectedly Driven by CLIP Embeddings
Bumjun Kim and Albert No
Understanding how textual
embeddings contribute to memorization in text-to-image
diffusion models is crucial for both interpretability and safety. This paper investigates an unexpected behavior of CLIP
embeddings in Stable
Diffusion, revealing that the model disproportionately relies on specific
embeddings. We categorize input
tokens as <startoftext>, <
prompt>, <endoftext> and <pad> with corresponding
embeddings $\mathbf{v}^{\mathbf{sot}}, \mathbf{v}^{\mathbf{pr}}, \mathbf{v}^{\mathbf{eot}}, \mathbf{v}^{\mathbf{pad}}$. We discover that $\mathbf{v}^{\mathbf{pr}}$ contribute minimally to generation in memorized cases. In contrast, $\mathbf{v}^{\mathbf{pad}}$ strongly affect memorization due to their structural duplication of $\mathbf{v}^{\mathbf{eot}}$, the only
embedding explicitly optimized during CLIP training. This duplication unintentionally amplifies the influence of $\mathbf{v}^{\mathbf{eot}}$, causing the model to over-rely on it, thereby driving memorization. Based on these observations, we propose two simple yet effective inference-time mitigation strategies: (1) Replacing the tokenizer's default <pad> from <eot> to the !
token before
embedding, and masking the $\mathbf{v}^{\mathbf{eot}}$; (2) Partial masking of $\mathbf{v}^{\mathbf{pad}}$. Both suppress memorization without degrading quality, and are readily deployable without prior detection.
Read the full abstract ▾
Machine Learning
On the Invariants of Softmax Attention
Wonsuk Lee
Softmax
attention maps every query–key interaction into a probability distribution, but the underlying structure remains largely unexplored. We define the energy field, the row-centered
attention logit, and show that it exhibits invariant properties across models, architectures, and inputs. Two classes of invariants emerge. Mechanism-level invariants follow from the algebraic structure of softmax
attention. They include a per-row zero-sum constraint, a rank bound determined by the head dimension, and spectral signatures that follow from them. Model-level regularities are not required by the mechanism, yet hold in every autoregressive
language model we test, spanning several architecture families. The energy field distributes its variance over key positions without concentrating at a few. This delocalization traces to a property of the key
matrix we call key incoherence. These invariants have practical consequences. The rank bound confines the energy field to a low-dimensional subspace. Key incoherence yields a per-head training monitor. All results are verified at multiple context lengths and input texts.
Read the full abstract ▾
Machine Learning
OpsLLM: Construction of Large Language Model for Software Operations with Multi-stage Learning
Jingkai He, Pengfei Chen, Chenghui Wu et al. (+7 more)
In the field of software operations, Large
Language Models (LLMs) have attracted increasing
attention. However, existing research has not yet achieved efficient and effective endto-end intelligent operations due to low-quality data, fragmented knowledge and insufficient learning. To explore the potential of LLMs in software operations, we propose OpsLLM, a domainspecific LLM that supports both knowledge-based question answering (QA) and root cause analysis (RCA). Moreover, we disclose the detailed workflow for building LLMs specifically in the software operations domain. First, a Human-in-the-Loop mechanism is introduced to curate high-quality data from a large collection of operational data and construct a fine-tuning
dataset. Then, based on the data, supervised fine-tuning is conducted to achieve a base model. Furthermore, we introduce a domain process reward model (DPRM) during the reinforcement learning stage to optimize the accuracy and reliability of the fine-tuned model on RCA tasks. Experimental results on the tasks with diverse difficulties demonstrate that OpsLLMs effectively learns and aligns with the operational domain knowledge infused, outperforming existing open-source and closed-source LLMs in accuracy with improvements of 0.2%~11.9% on QA tasks and 8.5%~70.3% on RCA tasks, while exhibiting strong transferability. Moreover, we will open-source three versions of OpsLLM with 7B, 14B and 32B parameters, along with a 15K fine-tuning
dataset.
Read the full abstract ▾
All authors Jingkai He, Pengfei Chen, Chenghui Wu, Shuang Liang, Ye Li, Gou Tan, Xidao Wen, Chuanfu Zhang, Fang Situ, Qi Zhou
Machine Learning
eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization
Pei-Chun Su
We show that the key-value (KV) cache in
transformer attention heads admits a natural decomposition into a low-rank shared context component and a full-rank per-
token residual, well described by the spiked random
matrix model. This observation leads to eOptShrinkQ, a two-stage compression pipeline: optimal singular value shrinkage (eOptShrink) automatically extracts the shared structure, and the residual – which satisfies the thin shell property with delocalized coordinates – is quantized by TurboQuant~\citep{zandieh2025turboquant}, a recently proposed per-vector scalar quantizer with near-optimal distortion guarantees. By restoring the isotropy that scalar quantization assumes, spectral denoising eliminates the need for both outlier handling and dedicated inner product bias correction, freeing those bits for improved reconstruction. The theoretical grounding in random
matrix theory provides three guarantees: automatic rank selection via the BBP phase transition, provably near-zero inner product bias on the residual, and coordinate delocalization ensuring near-optimal quantization distortion. Experimentally, we validate eOptShrinkQ on Llama-3.1-8B and Ministral-8B across three levels: per-head MSE and inner product fidelity, where eOptShrinkQ saves nearly one bit per entry over TurboQuant at equivalent quality; end-to-end on LongBench (16 tasks), where eOptShrinkQ at $\sim$2.2 bits per entry outperforms TurboQuant at 3.0 bits; and multi-needle
retrieval, where eOptShrinkQ at 2.2 bits closely matches or exceeds uncompressed FP16, suggesting that spectral denoising can act as a beneficial regularizer for
retrieval-intensive tasks.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
Sampling Matters: The Effect of ECG Frequency on Deep Learning-Based Atrial Fibrillation Detection
Arjan Mahmuod, Adrian Rod Hammerstad, Muzaffar Yousef et al. (+5 more)
Deep learning models for atrial fibrillation (AF) detection are increasingly trained on heterogeneous electrocardiogram (ECG)
datasets with varying sampling frequencies, yet the specific consequences of these discrepancies on model performance, calibration, and robustness remain insufficiently characterized. To address this, we conducted a systematic
benchmark using 12-lead, 10-second recordings from the PTB-XL
dataset, resampled to target frequencies of 62, 100, 250, and 500 Hz, to evaluate a standard 1-D Convolutional Neural Network (CNN) and a hybrid CNN-Long Short-Term
Memory (LSTM) architecture under a rigorous patient-safe cross-validation framework. Our analysis reveals that sampling frequency significantly impacts detection metrics in an architecture-dependent manner; the hybrid CNN-LSTM model demonstrated optimal performance and consistent calibration at intermediate frequencies (100-250 Hz), whereas the 1-D CNN baseline exhibited marked degradation in accuracy and sensitivity at 500 Hz, suggesting increased susceptibility to high-frequency noise. We conclude that ECG sampling frequency is a critical, underappreciated factor in arrhythmia detection, and future foundation models must explicitly control for temporal resolution to ensure clinical reliability and reproducibility.
Read the full abstract ▾
All authors Arjan Mahmuod, Adrian Rod Hammerstad, Muzaffar Yousef, Yngve Sebastian Heill, Jonas L. Isaksen, Jørgen K. Kanters, Pal Halvorsen, Vajira Thambawita
Machine Learning · Cross-listed from cs.NE
Fuzzy Encoding-Decoding to Improve Spiking Q-Learning Performance in Autonomous Driving
Aref Ghoreishee, Abhishek Mishra, Lifeng Zhou et al. (+3 more)
This paper develops an end-to-end fuzzy encoder-decoder architecture for enhancing
vision-based multi-modal deep spiking Q-networks in autonomous driving. The method addresses two core limitations of spiking reinforcement learning: information loss stemming from the conversion of dense visual inputs into sparse spike trains, and the limited representational capacity of spike-based value functions, which often yields weakly discriminative Q-value estimates. The encoder introduces trainable fuzzy membership functions to generate expressive, population-based spike representations, and the decoder uses a lightweight neural decoder to reconstruct continuous Q-values from spiking outputs. Experiments on the HighwayEnv
benchmark show that the proposed architecture substantially improves decision-making accuracy and closes the performance gap between spiking and non-spiking multi-modal Q-networks. The results highlight the potential of this framework for efficient and real-time autonomous driving with spiking neural networks.
Read the full abstract ▾
All authors Aref Ghoreishee, Abhishek Mishra, Lifeng Zhou, John Walsh, Anup Das, Nagarajan Kandasamy
Machine Learning · Cross-listed from cs.AI
Support Sufficiency as Consequence-Sensitive Compression in Belief Arbitration
Mark Walsh
When a system commits to a hypothesis, much of the evidential structure behind that commitment is lost to compression. Standard accounts assume that selected content and scalar confidence suffice for downstream control. This paper argues that they do not, and that determining what must survive compression is itself a consequence-sensitive problem. We develop a recurrent arbitration architecture in which active constraint fields jointly determine a hypothesis geometry over candidates. Rather than carrying that geometry forward in full, the system compresses it into a support-aware control state whose resolution is regulated by current consequence geometry, arbitration
memory, and resource constraints. A bounded objective formalizes the tradeoff. Too little retained support collapses policy-relevant distinctions, producing controllers that select content adequately while misrouting verification, abstention, and recovery. Too much retained support fragments learning across overly fine contexts, degrading adaptation even as discrimination improves. These failure modes yield ordered controller predictions confirmed by a minimal repeated-interaction simulation. Adaptive controllers that regulate support resolution outperform all fixed-resolution controllers in cumulative utility. Agile adaptive control outperforms sluggish adaptive control. Fixed high-resolution control achieves the best commitment accuracy but still trails adaptive controllers because resource cost and learning fragmentation offset the gains from richer retention. Support sufficiency should be understood not as a static representational threshold, but as a dynamic compression criterion. Robust arbitration depends on preserving the smallest support structure adequate for policy under the current consequence landscape, and on regulating that structure as conditions change across repeated cycles of inference and action.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CY
Quantifying how AI Panels improve precision
Nicholas CL Beale
AI in applications like screening job applicants had become widespread, and may contribute to unemployment especially among the young. Biases in the AIs may become baked into the job selection process, but even in their absence, reliance on a single AI is problematic. In this paper we derive a simple formula to estimate, or at least place an upper bound on, the precision of such approaches for data resembling realistic CVs: $P(q) \approx \frac{\rho n^b + q(1-\rho)}{1 + (n^b - 1)\rho}$ where $b \approx q^* + 0.8 (1 - \rho)$ and $q^*$ is $q$ clipped to $[0.07, 0.22]$ where $P(q)$ is the precision of the top $q$
quantile selected by a panel of $n$ AIs and $\rho$ is their average pairwise correlation. This equation provides a basis for considering how many AIs should be used in a Panel, depending on the importance of the decision. A quantitative discussion of the merits of using a diverse panel of AIs to support decision-making in such areas will move away from dangerous reliance on single AI systems and encourage a balanced assessment of the extent to which diversity needs to be built into the AI parts of the socioeconomic systems that are so important for our future.
Read the full abstract ▾
Machine Learning
Dimensional Criticality at Grokking Across MLPs and Transformers
Ping Wang
Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems. Grokking in deep neural networks provides a striking example – an abrupt transition from memorization to generalization long after training accuracy saturates – yet robust macroscopic signatures of this transition remain elusive. Here we introduce TDU–OFC (Thresholded
Diffusion Update–Olami-Feder-Christensen), an offline avalanche probe that converts gradient snapshots into cascade statistics and extracts a macroscopic observable – the time-resolved effective cascade dimension $D(t)$ – via grokking-aligned finite-size scaling. Across
Transformers trained on modular addition and MLPs trained on XOR, we discover a localized dynamical crossing of the Gaussian
diffusion baseline $D=1$ precisely at the generalization transition. The crossing direction is task-dependent: modular addition descends through $D=1$ (approaching from $D>1$), while XOR ascends (from $D<1$). This opposite-direction convergence is consistent with attraction toward a candidate shared critical
manifold, rather than trivial residence near $D \approx 1$. Negative controls confirm this picture: ungrokked runs remain supercritical ($D>1$) and never enter the post-transition regime. In addition, avalanche distributions exhibit heavy tails and finite-size scaling consistent with the dimensional exponent extracted from $D(t)$. Shadow-probe controls ($\alpha_{\mathrm{train}}=0$) confirm that $D(t)$ is non-invasive, and grokked trajectories diverge from ungrokked ones in $D(t)$ some $100$–$200$ epochs before the behavioral transition.
Read the full abstract ▾
Machine Learning
(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models
Maksim Zhdanov, Ana Lucic, Max Welling et al. (+1 more)
We introduce Mosaic, a probabilistic weather
forecasting model that addresses three failure modes of spectral degradation in ML-based weather prediction: spectral damping (statistical), high-frequency aliasing (architectural), and residual high-frequency leakage (parametric). Mosaic generates ensemble members through learned functional perturbations and operates on native-resolution grids via mesh-aligned block-sparse
attention, a hardware-aligned mechanism that captures long-range dependencies at linear cost by sharing keys and values across spatially adjacent queries. At 1.5{\deg} resolution with 214M parameters, Mosaic matches or outperforms models trained on 6$\times$ finer resolution on key variables and achieves state-of-the-art results among 1.5{\deg} models, producing well-calibrated ensembles whose individual members exhibit near-perfect spectral
alignment across all resolved frequencies. A 24-member, 10-day forecast takes under 12s on a single H100~GPU. Code is available at https://github.com/maxxxzdn/mosaic.
Read the full abstract ▾
All authors Maksim Zhdanov, Ana Lucic, Max Welling, Jan-Willem van de Meent
Machine Learning
Non-Stationarity in the Embedding Space of Time Series Foundation Models
Jinmyeong Choi, Brad Shook and Artur Dubrawski
Time series foundation models (TSFMs) are widely used as generic feature extractors, yet the notion of non-stationarity in their
embedding spaces remains poorly understood. Recent work often conflates non-stationarity with distribution shift, blurring distinctions fundamental to classical time-series analysis and long-standing methodologies such as statistical process control (SPC). In SPC, non-stationarity signals a process leaving a stable regime - via shifts in mean, variance, or emerging trends - and detecting such departures is central to quality monitoring and change-point analysis. Motivated by this diagnostic tradition, we study how different forms of distributional non-stationarity - mean shifts, variance changes, and linear trends - become linearly accessible in TSFM
embedding spaces under controlled conditions. We further examine temporal non-stationarity arising from persistence, which reflects violations of weak stationarity due to long-
memory or near-unit-root behavior rather than explicit distributional shifts. By sweeping shift strength and probing multiple TSFMs, we find that
embedding-space detectability of non-stationarity degrades smoothly and that different models exhibit distinct, model-specific failure modes.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
A methodology to rank importance of frequencies and channels in electromyography data with Decision Tree classifiers
Albert A. Nasybullin, Nursultan Abdullaev, Maksim A. Baranov et al. (+2 more)
This study presents a methodology for identifying the most informative frequencies and channels in electromyography (EMG) data to evaluate muscle recovery using Decision Tree classifiers. EMG signals, recorded from the vastus lateralis muscle during squat exercises, were analyzed across varying rest intervals to assess optimal recovery periods. By employing single Decision Tree classifiers, the study enhances interpretability, offering insights into feature importance - essential for applications in medical and sports settings where transparency is critical. The experimental protocol utilized a grid search for hyperparameter tuning and cross-validation to address class imbalance, ultimately achieving a reliable classification of rest intervals based on power spectral density features. The results indicate that a limited subset of highly informative features provides sufficient accuracy, suggesting that streamlined, interpretable models are effective for the evaluation of muscle recovery. This approach can guide future research in developing compact, robust models adapted to EMG-based diagnostics.
Read the full abstract ▾
All authors Albert A. Nasybullin, Nursultan Abdullaev, Maksim A. Baranov, Viacheslav V. Koshman, Vitaly A. Mahonin
Machine Learning · Cross-listed from cs.NE
Sharpness-Aware Surrogate Training for On-Sensor Spiking Neural Networks
Maximilian Nicholson
Spiking neural networks (SNNs) are a natural computational model for on-sensor and near-sensor
vision, where event driven processors must operate under strict power budgets with hard binary spikes. However, models trained with surrogate gradients often degrade sharply when the smooth surrogate nonlinearity is replaced by a hard threshold at deployment; a surrogate-to-hard transfer gap that directly limits on-sensor accuracy. We study Sharpness-Aware Surrogate Training (SAST), which applies Sharpness-Aware Minimization (SAM) to a surrogate-forward SNN so that the training objective is smooth and the gradient is exact, and position it as one gap-reduction strategy under the tested settings rather than the only viable mechanism. Under explicit contraction assumptions we provide state-stability, input-Lipschitz, and smoothness bounds, together with a corresponding nonconvex convergence result. On two event-camera
benchmarks, swap-only hard-spike accuracy improves from 65.7% to 94.7% on N-MNIST and from 31.8% to 63.3% on DVS Gesture. Under a hardware-aware inference simulation (INT8/INT4 weight quantization, fixed-point membrane potentials, discrete leak factors), SAST remains strong: on N-MNIST, hard-spike accuracy improves from 47.6% to 96.9% (INT8) and from 43.2% to 81.0% (INT4), while on DVS Gesture it improves from 25.3% to 47.6% (INT8) and from 26.0% to 43.8% (INT4). SynOps also decrease under the same hardware-aware setting, including 1734k$\rightarrow$1315k (N-MNIST, INT8) and 86221k$\rightarrow$4323k (DVS Gesture, INT8). These results suggest that SAST is a promising component in a broader toolbox for on-sensor spiking inference under the tested settings.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Face Density as a Proxy for Data Complexity: Quantifying the Hardness of Instance Count
Abolfazl Mohammadi-Seif and Ricardo Baeza-Yates
Machine learning progress has historically prioritized model-centric innovations, yet achievable performance is frequently capped by the intrinsic complexity of the data itself. In this work, we isolate and quantify the impact of instance density (measured by face count) as a primary driver of data complexity. Rather than simply observing that “crowded scenes are harder,” we rigorously control for class imbalance to measure the precise degradation caused by density alone. Controlled experiments on the WIDER FACE and Open Images
datasets, restricted to exactly 1 to 18 faces per image with perfectly balanced sampling, reveal that model performance degrades monotonically with increasing face count. This trend holds across classification, regression, and detection paradigms, even when models are fully exposed to the entire density range. Furthermore, we demonstrate that models trained on low-density regimes fail to generalize to higher densities, exhibiting a systematic under-counting bias, with error rates increasing by up to 4.6x, which suggests density acts as a domain shift. These findings establish instance density as an intrinsic, quantifiable dimension of data hardness and motivate specific interventions in curriculum learning and density-stratified evaluation.
Read the full abstract ▾
Machine Learning
Spectral Edge Dynamics Reveal Functional Modes of Learning
Yongzhong Xu
Training dynamics during grokking concentrate along a small number of dominant update directions – the spectral edge – which reliably distinguishes grokking from non-grokking regimes. We show that standard mechanistic interpretability tools (head attribution, activation probing, sparse autoencoders) fail to capture these directions: their structure is not localized in parameter or feature space. Instead, each direction induces a structured function over the input domain, revealing low-dimensional functional modes invisible to representation-level analysis. For modular addition, all leading directions collapse to a single
Fourier mode. For multiplication, the same collapse appears only in the discrete-log basis, yielding a 5.9x improvement in concentration. For subtraction, the edge spans a small multi-mode family. For $x^2+y^2$, no single harmonic basis suffices, but cross-terms of additive and multiplicative features provide a 4x variance boost, consistent with the decomposition (a+b)^2 - 2ab. Multitask training amplifies this compositional structure, with the $x^2+y^2$ spectral edge inheriting the addition
circuit's characteristic frequency (2.3x concentration increase). These results suggest that training discovers low-dimensional functional modes over the input domain, whose structure depends on the algebraic
symmetry of the task. These results suggest that spectral edge dynamics identify low-dimensional functional subspaces governing learning, whose representation depends on the algebraic structure of the task. Simple harmonic structure emerges only when the task admits a
symmetry-adapted basis; more complex tasks require richer functional descriptions.
Read the full abstract ▾
Machine Learning
FLeX: Fourier-based Low-rank EXpansion for multilingual transfer
Gaurav Narasimhan
Cross-lingual code generation is critical in enterprise environments where multiple programming
languages coexist. However, fine-tuning large
language models (LLMs) individually for each
language is computationally prohibitive. This paper investigates whether parameter-efficient fine-tuning methods and optimizer enhancements can improve cross-lingual transfer from Python to
languages like Java. We fine-tune the Code Llama 7B model using low-rank adaptation (LoRA) to optimize a small subset of parameters and compare Adam and Sophia optimizers, while exploring a novel
Fourier-based regularization technique. Our contributions include: (1)demonstrating that LoRA fine-tuning on a small, high-quality
dataset (MBPP) can exceed the pass@1 performance of the more broadly fine-tuned Code Llama-Python-7B model (40.1% vs. 38.4%); (2) showing that while Sophia achieves faster convergence than Adam, final pass@1 scores show marginal differences; and (3) presenting evidence that
Fourier-based regularization during fine-tuning significantly improves cross-lingual transfer, achieving 42.1% pass@1 on Java tasks compared to the 34.2% baseline. These findings suggest that combining LoRA, optimized training methods, and frequency-domain regularization can efficiently adapt single-
language LLMs to perform well across multiple programming
languages.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Toward Reducing Unproductive Container Moves: Predicting Service Requirements and Dwell Times
Elena Villalobos, Adolfo De Unánue T., Fernanda Sobrino et al. (+4 more)
This article presents the results of a data science study conducted at a container terminal, aimed at reducing unproductive container moves through the prediction of service requirements and container dwell times. We develop and evaluate machine learning models that leverage historical operational data to anticipate which containers will require pre-clearance handling services prior to cargo release and to estimate how long they are expected to remain in the terminal. As part of the data preparation process, we implement a classification system for cargo descriptions and perform deduplication of consignee records to improve data consistency and feature quality. These predictive capabilities provide valuable inputs for strategic planning and resource allocation in yard operations. Across multiple temporal validation periods, the proposed models consistently outperform existing rule-based heuristics and random baselines in precision and recall. These results demonstrate the practical value of predictive analytics for improving operational efficiency and supporting data-driven decision-making in container terminal logistics.
Read the full abstract ▾
All authors Elena Villalobos, Adolfo De Unánue T., Fernanda Sobrino, David Aké, Stephany Cisneros, Jorge Lecona, Alejandra Matadamaz
Machine Learning
Extending Tabular Denoising Diffusion Probabilistic Models for Time-Series Data Generation
Umang Dobhal, Christina Garcia and Sozo Inoue
Diffusion models are increasingly being utilised to create synthetic tabular and time series data for
privacy-preserving augmentation. Tabular Denoising
Diffusion Probabilistic Models (TabDDPM) generate high-quality synthetic data from heterogeneous tabular
datasets but assume independence between samples, limiting their applicability to time-series domains where temporal dependencies are critical. To address this, we propose a temporal extension of TabDDPM, introducing sequence awareness through the use of lightweight temporal adapters and context-aware
embedding modules. By reformulating sensor data into windowed sequences and explicitly modeling temporal context via timestep
embeddings, conditional activity labels, and observed/missing masks, our approach enables the generation of temporally coherent synthetic sequences. Compared to baseline and interpolation techniques, validation using bigram transition
matrices and autocorrelation analysis shows enhanced temporal realism, diversity, and coherence. On the WISDM accelerometer
dataset, the suggested system produces synthetic time-series that closely resemble real world sensor patterns and achieves comparable classification performance (macro F1-score 0.64, accuracy 0.71). This is especially advantageous for minority class representation and preserving statistical
alignment with real distributions. These developments demonstrate that
diffusion based models provide effective and adaptable solutions for sequential data synthesis when they are equipped for temporal
reasoning. Future work will explore scaling to longer sequences and integrating stronger temporal architectures.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
EAGLE: Edge-Aware Graph Learning for Proactive Delivery Delay Prediction in Smart Logistics Networks
Zhiming Xue, Menghao Huo and Yujue Wang
Modern logistics networks generate rich operational data streams at every warehouse node and transportation lane – from order timestamps and routing records to shipping manifests – yet predicting delivery delays remains predominantly reactive. Existing predictive approaches typically treat this problem either as a tabular classification task, ignoring network
topology, or as a time-series anomaly detection task, overlooking the spatial dependencies of the supply chain graph. To bridge this gap, we propose a hybrid deep learning framework for proactive supply chain risk management. The proposed method jointly models temporal order-flow dynamics via a lightweight
Transformer patch encoder and inter-hub relational dependencies through an Edge-Aware Graph
Attention Network (E-GAT), optimized via a multi-task learning objective. Evaluated on the real-world DataCo Smart Supply Chain
dataset, our framework achieves consistent improvements over baseline methods, yielding an F1-score of 0.8762 and an AUC-ROC of 0.9773. Across four independent random seeds, the framework exhibits a cross-seed F1 standard deviation of only 0.0089 – a 3.8 times improvement over the best ablated variant – achieving the strongest balance of predictive accuracy and training stability among all evaluated models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IR
Spike Hijacking in Late-Interaction Retrieval
Karthik Suresh, Tushar Vatsa, Tracy King et al. (+2 more)
Late-interaction
retrieval models rely on hard maximum similarity (MaxSim) to aggregate
token-level similarities. Although effective, this winner-take-all pooling rule may structurally bias training dynamics. We provide a mechanistic study of gradient routing and robustness in MaxSim-based
retrieval. In a controlled synthetic environment with in-batch contrastive training, we demonstrate that MaxSim induces significantly higher patch-level gradient concentration than smoother alternatives such as Top-k pooling and softmax aggregation. While sparse routing can improve early discrimination, it also increases sensitivity to document length: as the number of document patches grows, MaxSim degrades more sharply than mild smoothing variants. We corroborate these findings on a real-world multi-vector
retrieval benchmark, where controlled document-length sweeps reveal similar brittleness under hard max pooling. Together, our results isolate pooling-induced gradient concentration as a structural property of late-interaction
retrieval and highlight a sparsity-robustness tradeoff. These findings motivate principled alternatives to hard max pooling in multi-vector
retrieval systems.
Read the full abstract ▾
All authors Karthik Suresh, Tushar Vatsa, Tracy King, Asim Kadav, Michael Friedrich
Machine Learning
DualDiffusion: A Speculative Decoding Strategy for Masked Diffusion Models
Satyam Goyal, Kushal Patel, Tanush Mittal et al. (+1 more)
Masked
Diffusion Models (MDMs) offer a promising alternative to autoregressive
language models by enabling parallel
token generation and bidirectional context modeling. However, their inference speed is significantly limited by the inability to cache key-value pairs due to bidirectional
attention, requiring $O(N^2)$ computations at each generation step. While recent methods like FastDLLM and DkvCache improve inference speed through
attention approximations and caching strategies, they achieve speedups at the cost of generation quality. We propose DualDiffusion, a speculative decoding framework for MDMs that combines fast drafter models (using efficient approximations) with slower, more accurate verifier models. By running multiple steps of a lightweight drafter followed by a single verification step, DualDiffusion achieves a superior Pareto frontier between generation steps and accuracy compared to existing approaches. We evaluate our method on MMLU and GSM8K, demonstrating that DualDiffusion maintains high accuracy while reducing the number of generation steps required, effectively pushing the quality-efficiency trade-off curve for masked
diffusion language models.
Read the full abstract ▾
All authors Satyam Goyal, Kushal Patel, Tanush Mittal, Arjun Laxman
Machine Learning
Improving Sparse Memory Finetuning
Satyam Goyal, Anirudh Kanchi, Garv Shah et al. (+1 more)
Large
Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities. Standard approaches to updating models, like full finetuning or parameter-efficient methods (e.g., LoRA), face a fundamental trade-off: catastrophic forgetting. They modify shared dense representations, causing interference across tasks. Sparse
Memory Finetuning (SMF) offers a promising alternative by localizing updates to a small subset of parameters in explicit
memory layers. In this work, we present an open-source pipeline to retrofit existing pretrained models (Qwen-2.5-0.5B) with sparse
memory modules, enabling effective continual learning on consumer hardware. We extend prior work by introducing a theoretically grounded slot-selection mechanism based on Kullback-Leibler (KL) divergence, which prioritizes
memory updates for informationally "surprising"
tokens relative to a background distribution. Our experiments demonstrate that our retrofitted models can acquire new factual knowledge with minimal forgetting of held-out capabilities, validating the sparse update hypothesis in a practical setting.
Read the full abstract ▾
All authors Satyam Goyal, Anirudh Kanchi, Garv Shah, Prakhar Gupta
Machine Learning
Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks
Anas Jnini, Elham Kiyani, Khemraj Shukla et al. (+5 more)
Efficient and robust optimization is essential for neural networks, enabling scientific machine learning models to converge rapidly to very high accuracy – faithfully capturing complex physical behavior governed by differential equations. In this work, we present advanced optimization strategies to accelerate the convergence of physics-informed neural networks (PINNs) for challenging partial (PDEs) and ordinary differential equations (ODEs). Specifically, we provide efficient implementations of the Natural Gradient (NG) optimizer, Self-Scaling BFGS and Broyden optimizers, and demonstrate their performance on problems including the Helmholtz equation, Stokes flow, inviscid Burgers equation, Euler equations for high-speed flows, and stiff ODEs arising in pharmacokinetics and pharmacodynamics. Beyond optimizer development, we also propose new PINN-based methods for solving the inviscid Burgers and Euler equations, and compare the resulting solutions against high-order numerical methods to provide a rigorous and fair assessment. Finally, we address the challenge of scaling these quasi-Newton optimizers for batched training, enabling efficient and scalable solutions for large data-driven problems.
Read the full abstract ▾
All authors Anas Jnini, Elham Kiyani, Khemraj Shukla, Jorge F. Urban, Nazanin Ahmadi Daryakenari, Johannes Muller, Marius Zeinhofer, George Em Karniadakis
Machine Learning · Cross-listed from cs.AI
From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AI
Christopher Koch
Agentic AI systems plan, use tools, maintain state, and produce multi-step trajectories with external effects. Those properties create a governance problem that differs materially from single-turn generative AI: important risks emerge dur- ing execution, not only at model development or deployment time. Governance standards such as ISO/IEC 42001, ISO/IEC 23894, ISO/IEC 42005, ISO/IEC 5338, ISO/IEC 38507, and the NIST AI Risk Management Framework are therefore highly relevant to agentic AI, but they do not by themselves yield implementable runtime guardrails. This paper proposes a layered translation method that connects standards-derived governance objectives to four control layers: governance objectives, design- time constraints, runtime mediation, and assurance feedback. It distinguishes governance objectives, technical controls, runtime guardrails, and assurance evidence; introduces a control tuple and runtime-enforceability rubric for layer assignment; and demonstrates the method in a procurement-
agent case study. The central claim is modest: standards should guide control placement across architecture, runtime policy, human escalation, and audit, while runtime guardrails are reserved for controls that are observable, determinate, and time-sensitive enough to justify execution-time intervention.
Read the full abstract ▾