Machine Learning · Cross-listed from eess.SP
Buffer-Parameterized Machine Learning Surrogate Models for Cross-Technology Signal Integrity Analysis and Optimization
Julian Withöft, Werner John, Emre Ecik et al. (+2 more)
Signal integrity (SI) analysis in printed
circuit board (PCB) interconnects faces increasing complexity due to diverse integrated
circuit (IC) buffer technologies, varying operating conditions, and manufacturing tolerances. Existing machine learning (ML) surrogate models for the prediction of SI metrics such as the inner eye contour, eye-height (EH), eye-width (EW), and transient waveform features often rely on fixed buffer parameters, limiting the adaptability across design and technology variations. This paper introduces a buffer-parameterized ML surrogate modeling methodology capable of handling cross-technology variations without retraining by treating IC buffer characteristics, e.g., clock frequency, supply voltage, rise/fall times, jitter, and internal resistors and capacitors, as model inputs in addition to the PCB parameters. To identify the optimal surrogate architecture for this high-dimensional space, tree-based methods (RFR/GBM), kernel methods (SVR/KRR), Gaussian process regression (GPR), and neural networks are compared. The buffer-parameterized approach is validated using a complex interconnect with a total of 44 design parameters. In addition to demonstrating strong generalization, the results show that neural networks outperform other ML models on large
datasets, while the anisotropic GPR excels in low-data regimes. Finally, the practical value of the ML surrogate models is demonstrated through a cross-technology design space exploration and optimization scenario, which indicates large computational speedups for eye mask compliance checking compared to simulation.
Read the full abstract ▾
All authors Julian Withöft, Werner John, Emre Ecik, Ralf Brüning, Jürgen Götze
Machine Learning
Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMs
Junyi Wu, Tianchen Zhao, Shaoqiu Zhang et al. (+3 more)
Unlike autoregressive models, which generate one
token at a time, dLLMs denoise a chunk of [MASK]
tokens jointly and sample one or more
tokens per step; despite enabling parallel decoding, this process incurs substantial computational cost due to the large chunk size of masked
tokens. We observe that much of this cost is spent on repeatedly processing the preceding context and many [MASK]
tokens with the same feature representations, indicating considerable computational redundancy. In this work, we revisit dLLM's redundancy from the perspective of [MASK]
tokens. Through systematic analysis, we verify the redundancy of [MASK]
tokens while revealing their critical role in providing structural information. Guided by these findings, we propose position-preserving [MASK]
token compression and terminal-aware augmentation. By compressing redundant [MASK] computation, this approach accelerates decoding and further provides a natural extension toward context-folding-like long-context scaling under limited input-length constraints for full-sequence dLLMs such as LLaDA-8B-Instruct and LLaDA-1.5. Moreover, for block dLLMs such as LLaDA2.0-mini, it augments the context with a protected terminal [MASK]
token to enhance generation quality with negligible overhead.
Read the full abstract ▾
All authors Junyi Wu, Tianchen Zhao, Shaoqiu Zhang, Linfeng Zhang, Guohao Dai, Yu Wang
Machine Learning
Tabular Foundation Models for Credit Risk Prediction: A Dedicated Review and Empirical Comparison
Bart Baesens, Andreas Goethals, Stefan Lessmann et al. (+10 more)
Predictive models are central to credit risk management, where the accuracy of default-probability and loss estimates affects the profitability of lending and the stability of the financial system. Decades of benchmarking research have consolidated the state of the art, with gradient-boosting models often representing the strongest machine-learning
benchmark, while linear and logistic models remain important regulatory and industry reference models. The recent advent of tabular data foundation models introduces a new paradigm for predictive modeling. Rather than fitting a model to a target
dataset, they use in-context learning to integrate information from the target
dataset with prior knowledge acquired through large-scale pretraining on diverse collections of (synthetic) tabular data. We
conjecture that access to information beyond the target data is beneficial in small-data settings and may help address longstanding challenges, including low-default portfolios. Whether this promise holds in practice remains an open question. This paper
benchmarks state-of-the-art tabular foundation models against a broad set of classical and advanced machine learning competitors on two core tasks: Probability of Default (PD) and Loss Given Default (LGD) modeling. We evaluate $33$ classification methods across $14$ PD
datasets and $27$ regression methods across $7$ LGD
datasets, spanning a range of performance indicators and experimental conditions. The most recent tabular foundation models, in particular TabPFN-3 and TabICLv2, achieve the strongest average results, with the largest gains in small-sample and strongly imbalanced settings. These results are notable because these models are used with their pretrained parameters fixed and without downstream hyperparameter optimization or task-specific parameter updates.
Read the full abstract ▾
All authors Bart Baesens, Andreas Goethals, Stefan Lessmann, Simon De Vos, Cristián Bravo, David Martens, Victor Medina-Olivares, Christophe Mues, Maria Oskarsdóttir, Seppe vanden Broucke, Tony Van Gestel, Tim Verdonck, Wouter Verbeke
Machine Learning · Cross-listed from math.OC
Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers
Tim Tsz-Kit Lau and Weijie Su
A striking
geometric disparity has long persisted in the practice of deep learning. While modern neural network architectures naturally exhibit rich
symmetry and equivariance properties, popular optimizers such as Adam and its variants operate inherently coordinate-wise, rendering them unable to respect the equivariance structures of the parameter space. We address this disparity by introducing a
symmetry-compatible principle for optimizer design: the gradient update rule should be equivariant under the
symmetry group acting on the corresponding weight block. Following this principle, we first provide a unified perspective on bi-orthogonally equivariant updates for general
matrix layers, as employed by
stochastic spectral descent,
Muon, Scion, and polar gradient methods. More importantly, by moving from orthogonal groups to permutation and shared-shift
symmetries, we derive
symmetry-compatible optimizers for parameter blocks whose
symmetries differ from those of general
matrix layers:
embedding and LM head
matrices, SwiGLU MLP projections, and MoE router
matrices. These constructions include one-sided spectral, row-norm, hybrid row-norm/spectral, row-aware, column-aware, centered row-norm, and left-spectral updates. They yield an end-to-end layerwise optimizer stack in which each major
matrix-valued parameter class is assigned an update whose equivariance matches its
symmetry group. We corroborate this principle through pre-training experiments on dense and sparse MoE
language models, including Qwen3-0.6B-style, Gemma 3 1B-style, OLMoE-1B-7B-style, and downsized gpt-oss architectures. Across these experiments,
symmetry-compatible update rules consistently improve final validation loss, reduce expert load imbalance in sparse MoE models, and in several cases control final vocabulary-logit growth, improve router stability, and overall training stability over the corresponding AdamW updates.
Read the full abstract ▾
Machine Learning
pyforce-1.0.0: Python Framework for data-driven model Order Reduction of multi-physiCs problEms
Stefano Riva, Yantao Luo, Carolina Introini et al. (+1 more)
pyforce is a Python package implementing Data-Driven Reduced Order Modelling techniques for applications to multi-physics problems, mainly set in the
Nuclear Engineering world. The package is part of the ROSE (Reduced Order modelling with data-driven techniques for multi-phySics problEms): mathematical
algorithms aimed at reducing the complexity of multi-physics models (for
nuclear reactors applications), at searching for optimal sensor positions and at integrating real measures to improve the knowledge on the physical systems. With respect to the previous original implementation based on dolfinx package (v0.6.0), version 1.0.0 of pyforce has been completely re-written using pyvista as backend for mesh importing, computing integrals, and visualisation of results; in addition, functions are stored as numpy arrays, improving the ease of use of the package. This choice allows to use pyforce with any software solver able to export results in VTK format.
Read the full abstract ▾
All authors Stefano Riva, Yantao Luo, Carolina Introini, Antonio Cammi
Machine Learning
The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
Moritz Brösamle and Stephan Eckstein
Existing expressivity results for
transformers typically rely on hardmax
attention, high precision, and other architectural modifications that disconnect them from the models used in practice. We bridge this gap by analyzing standard
transformer decoders with softmax
attention and rounding of activations and
attention weights, while allowing depth and width to grow logarithmically with the context length. As an intermediate step, we construct hardmax
transformers with ternary activations and well-separated
attention scores that simulate Turing machines using Chain-of-Thought (CoT). This lets us convert the constructions to equivalent softmax
transformers without the unrealistic parameter magnitudes or activation precision that prior approaches would require. Using the same technique, we analyze a recently proposed summarized CoT paradigm and show that it simulates Turing machines more efficiently, with model size scaling logarithmically in a space bound rather than a time bound. We empirically test predictions made by our results on a Sudoku
reasoning task and find better
alignment with learnability than for prior high-precision results. Our code is available at https://github.com/moritzbroe/
transformer-expressivity.
Read the full abstract ▾
Machine Learning
Equilibrium Selection in Multi-Agent Policy Gradients via Opponent-Aware Basin Entry
Yevhen Shcherbinin, Arina Redina, Maxim Kalpin et al. (+1 more)
Multi-
agent policy-gradient methods have been shown to converge locally near stable Nash
equilibria. Local convergence, however, does not determine which
equilibrium is reached. We study this question through basin-entry probability with respect to a target set of
equilibria selected by an external criterion, such as payoff dominance. For finite-unroll Meta-MAPG, we show that the update decomposes into ordinary policy gradient plus own-learning and peer-learning corrections, with controlled sampling noise and finite-unroll bias. We identify the peer-learning correction as the main
equilibrium-selection mechanism: under a local
alignment condition, the probability of entering the certified attraction region of the target stable-Nash set increases, relative to ordinary policy gradient. Because persistent correction may shift zero-update points of the original game, annealing the correction after entering the basin recovers ordinary policy-gradient dynamics and inherits local stable-Nash convergence guarantees. Experiments in Stag Hunt, iterated Prisoner's Dilemma, and preliminary neural-policy coordination environments support this basin-entry view, showing increased entry into cooperative basins under peer-aware updates.
Read the full abstract ▾
All authors Yevhen Shcherbinin, Arina Redina, Maxim Kalpin, Vlad Kochetov
Machine Learning · Cross-listed from cs.AI
LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning
Sangjun Bae, Yisak Park, Sanghyeon Lee et al. (+1 more)
Communication is a key component in multi-
agent reinforcement learning (MARL) for mitigating partial observability, yet prior approaches often rely on inefficient information exchange or fail to transmit sufficient state information. To address this, we propose LLM-driven Multi-
Agent Communication (LMAC), which leverages an LLM's
reasoning capability to design a communication protocol that enables all
agents to reconstruct the underlying state as accurately and uniformly as possible. LMAC iteratively refines the protocol using an explicit state-awareness criterion, improving state recovery while narrowing differences in
agents' knowledge. Experiments on diverse MARL
benchmarks show that LMAC improves state reconstruction across
agents and yields substantial performance gains over prior communication baselines.
Read the full abstract ▾
All authors Sangjun Bae, Yisak Park, Sanghyeon Lee, Seungyul Han
Machine Learning · Cross-listed from stat.ML
Wasserstein bounds for denoising diffusion probabilistic models via the Föllmer process
Yuta Koike
This paper studies sampling error bounds for denoising
diffusion probabilistic models (DDPMs) in the 2-Wasserstein distance. Our contributions are threefold. (i) Under general Lipschitz-type conditions on the score function and for a broad class of variance schedules, including the cosine schedule, we establish sharp upper bounds that are optimal in both the dimension and the number of steps, and recover several sharp error bounds previously obtained in the literature. (ii) We prove that the same Lipschitz-type conditions, which encompass those commonly imposed on the (learned) score, imply a logarithmic Sobolev inequality and hence a quadratic transportation cost inequality for the DDPM. As a consequence, in settings covered by existing work, an optimal Wasserstein bound, up to a logarithmic factor, follows from the recently obtained sharp error bound in the Kullback-Leibler divergence under
geometric-type variance schedules. (iii) We show that for general log-concave target distributions, the optimal Wasserstein error bound remains attainable even without a quadratic transportation cost inequality for the target. Our analysis is based on viewing the DDPM sampler as a discretization of the Föllmer process rather than the conventional reverse Ornstein-Uhlenbeck process.
Read the full abstract ▾
Machine Learning
Teger: Spatiotemporal Covariance for Probabilistic Traffic Forecasting
Seyed Mohamad Moghadas, Esther Rodrigo Bonet, Bruno Cornelis et al. (+1 more)
Traffic conditions drift – demand patterns, incident dynamics, and sensor behavior shift over a deployment's lifetime – so a joint uncertainty estimate fit once at training time and left static will miscalibrate as conditions change. We present TEGER, a residual covariance model that keeps a forecaster's joint predictive uncertainty current at test time through closed-form updates, not retraining. A fixed sensor graph supplies a low-dimensional spatial precision factor encoding which sensors' errors move together; at inference, Gaussian conditioning corrects The next forecast's mean and covariance from only the most recently observed residuals, and an exponential moving-average
volatility term rescales marginal uncertainty to track local drift while preserving the learned correlation structure. Neither update touches the
forecasting backbone's weights, so the same mechanism attaches to a frozen time-series foundation model: no gradient passes through it, turning a static point forecast into one with continually refreshed, correlated uncertainty at negligible added cost. In four traffic
datasets, the fixed-graph covariance improves 60-minute $CRPS_{sum}$ over a temporal-only baseline in different configurations of the backbone and
datasets. Applied post hoc to a frozen time-series foundation model, Chronos, the same test-time correction reduces $CRPS_{sum}$ from 0.1798 to 0.1736 without foundation-model fine-tuning. These results support closed-form test-time residual correction, rather than a covariance fixed once at training time.
Read the full abstract ▾
All authors Seyed Mohamad Moghadas, Esther Rodrigo Bonet, Bruno Cornelis, Adrian Munteanu
Machine Learning · Cross-listed from cs.CV
The MixCount Dataset: Bridging the Data Gap for Open-Vocabulary Object Counting
Corentin Dumery, Niki Amini-Naieni, Shervin Naini et al. (+1 more)
Object counting is a foundational
vision task with over a decade of dedicated research, yet state-of-the-art models still fail systematically in the mixed-object setting that dominates real-world applications such as industrial inspection and product sorting. We show that this gap is strongly driven by limitations in existing training and evaluation data: real counting
datasets are prohibitively expensive to annotate and suffer from labeling noise, while existing synthetic alternatives lack diversity and realism. We address this with MixCount, a
dataset and
benchmark for mixed-object counting designed to target the failure modes of current counting models. To overcome the high cost of constructing and labeling such data, we develop an automatic generation pipeline that synthesizes images, fine-grained textual descriptions, and pixel-perfect counting annotations at scale, eliminating the labeling ambiguity that plagues prior
datasets. Evaluating state-of-the-art counting models on MixCount exposes severe degradation in the mixed-object setting. More importantly, training these models on our synthesized data yields substantial gains on real-world
benchmarks, reducing MAE by 20.14% on FSC-147 and by 18.3% on PairTally. These results establish MixCount as both a
benchmark and a training
dataset for fine-grained counting, and demonstrate that our pipeline, which produces effectively unlimited labeled data, helps address a long-standing bottleneck in counting models.
Read the full abstract ▾
All authors Corentin Dumery, Niki Amini-Naieni, Shervin Naini, Pascal Fua
Machine Learning
FLAG: Foundation model representation with Latent diffusion Alignment via Graph for spatial gene expression prediction
Qi Si, Penglei Wang, Yushuai Wu et al. (+5 more)
Predicting spatial gene expression from routine H&E enables large-scale molecular profiling, yet current models treat this as isolated pointwise tasks, thereby overlooking essential biological structures like gene coordination and spatial distribution. To preserve these relationships, we introduce FLAG, a
diffusion-based framework that redefines this task as structured distribution modeling. At the same time, we identify the critical Gene Dimension Curse, where joint modeling gene expression and their spatial interactions fail in high-dimensional spaces, and FLAG solves this challenge by integrating a spatial graph encoder for topological consistency and utilizing Gene Foundation Model (GFM)
alignment for gene-gene fidelity in the generation process. To rigorously assess model performance, we propose a set of novel structural evaluation metrics, including Gene Structural Correlation (GSC) and Spatial Structural Correlation (SSC). Our experiments demonstrate that FLAG is highly competitive in traditional accuracy (PCC/MSE) while achieving significantly enhanced structural fidelity in capturing both gene-gene and gene-spatial relationships. The code is available at https://github.com/darkflash03/FLAG.
Read the full abstract ▾
All authors Qi Si, Penglei Wang, Yushuai Wu, Yifeng Jiao, Xuyang Liu, Xin Guo, Yuan Qi, Yuan Cheng
Machine Learning
Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in Globally Capped KV Eviction
Gabriel Garcia
We study KV cache eviction under a shared globally capped decode-time harness. Seven policies (LRU, H2O, SnapKV, StreamingLLM, Ada-KV, QUEST, Random) share a
prompt-boundary vulnerability: without structural protection, they collapse to near-zero quality on six pure-
transformer models (F1$\leq$0.064). Reserving 10% of cache at each boundary recovers 69–90% of the $C{=}2{,}048$ reference-ceiling quality on seven LongBench models at $C{=}256$ (13% retention); a ten-model panel spans 68–98%. An
attention-mass pilot (Qwen2.5-3B, $N{=}30$) suggests why: the position-0 sink holds ${\sim}75\%$ of prefix mass, while other boundary
tokens sit near ${\sim}0.41{\times}$ uniform expectation, so
attention scorers retain the sink but still drop structurally critical
tokens. With protection, simplified score-isolation variants are TOST-equivalent to LRU at $K{=}32$ ($\Delta{=}0.02$); at $K{=}8$,
attention policies pairwise converge yet beat LRU by 0.011–0.021 F1 across $C{=}256$ and $C{=}512$. Faithful Ada-KV/QUEST add ${\sim}0.03$–$0.04$ F1 on Mistral-7B and Phi-3.5 beyond simplified variants. A NIAH-32K regime-transfer pilot on Qwen3-4B (decode vs.\ prefill, $C{\in}\{512,2048\}$) shows near-identical protection lifts (ratio 0.99–1.00). At 64K, protection helps but recovery is modest; faithful per-head scoring matches full-cache ceiling on Gemma-3-4B at 6.3% retention only when the model already supports strong 64K
retrieval without eviction. Overall: protection dominates; scoring differences are secondary once boundaries are guarded; per-head allocation gives a further modest gain.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Connections between the Föllmer process and the denoising diffusion probabilistic model
Yuta Koike
The Föllmer process is a Brownian motion conditioned to have a pre-specified distribution at time 1. This process can be interpreted as an “augmented” time-compressed version of the reverse
stochastic differential equation (SDE) corresponding to the denoising
diffusion probabilistic model (DDPM). While this fact has been indirectly used to analyze DDPM sampling errors via discretization of the reverse SDE, the connection between direct discretization of the Föllmer process and the DDPM sampler has not yet been fully explored. This paper clarifies this point while surveying relevant results from the literature. We show that discretized Föllmer processes give natural hyper-parameter settings of the DDPM sampler while accommodating a broader class of variance schedules than discretized reverse SDEs. Moreover, this allows us to systematically recover state-of-the-art results on DDPM sampling error bounds, along with slight improvements.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
New Insight of Variance reduce in Zero-Order Hard-Thresholding: Mitigating Gradient Error and Expansivity Contradictions
Xinzhe Yuan, William de Vazelhes, Bin Gu et al. (+1 more)
Hard-thresholding is an important type of
algorithm in machine learning that is used to solve $\ell_0$ constrained optimization problems. However, the true gradient of the objective function can be difficult to access in certain scenarios, which normally can be approximated by zeroth-order (ZO) methods. The SZOHT
algorithm is the only
algorithm tackling $\ell_0$ sparsity constraints with ZO gradients so far. Unfortunately, SZOHT has a notable limitation on the number of random directions % in ZO gradients due to the inherent conflict between the deviation of ZO gradients and the expansivity of the hard-thresholding operator. This paper approaches this problem by considering the role of variance and provides a new insight into variance reduction: mitigating the unique conflicts between ZO gradients and hard-thresholding. Under this perspective, we propose a generalized variance reduced ZO hard-thresholding
algorithm as well as the generalized convergence analysis under standard assumptions. The theoretical results demonstrate the new
algorithm eliminates the restrictions on the number of random directions, leading to improved convergence rates and broader applicability compared with SZOHT. Finally, we illustrate the utility of our method on a ridge regression problem as well as black-box
adversarial attacks.
Read the full abstract ▾
All authors Xinzhe Yuan, William de Vazelhes, Bin Gu, Huan Xiong
Machine Learning · Cross-listed from cond-mat.mtrl-sci
Real-time Multi-instrument Autonomous Discovery of Novel Phase-change Memory Materials
Chih-Yu Lee, Haotong Liang, Ryan Kim et al. (+4 more)
Autonomous labs enable the integration of automated experiment execution, data analysis and decision making. The main challenge remains the integration of diverse data streams from multiple instruments, where the data is often heterogeneous and unsynchronized. The standard learning process of undetermined synthesis-process-structure-property relationships (SPSPR) usually relies on post-experiment analysis after data is fully collected, not during live experiments, and decision making is carried out independently across characterization equipment. Here, we demonstrate the Multi-instrument Autonomous Discovery (MAD) framework – combining structural property mapping and functional property optimization simultaneously in a closed-loop manner. As an example, we applied MAD to phase change
memory (PCM) materials, and, in particular on the Mn-Sb-Te ternary, a previously unexplored materials system for PCM. A multi-output model is employed to merge data from x-ray diffraction (XRD) and electrical resistance measurements simultaneously through a co-regionalization kernel that models the relationship between them. The output probabilistic posterior and uncertainty quantification facilitate decision making with shared knowledge, while the goals are different across tasks. We aimed to maximize the knowledge of
crystal structure distribution using non-negative
matrix factorization (NMF), while in parallel, we find the composition with the maximum resistance value, an important figure of merit for PCM. Leveraging MAD, we found promising electrical PCMs and identified the SPSPR within 25 closed-loop iterations, corresponding to a seven-fold speed-up. The framework opens a new path of study in large-scale autonomous facilities, where future experiments can be run in parallel together, not independently.
Read the full abstract ▾
All authors Chih-Yu Lee, Haotong Liang, Ryan Kim, Austin McDannald, Carlos A Rios Ocampo, A. Gilad Kusne, Ichiro Takeuchi
Machine Learning
Ziheng Ren, Zhanming Shen, Hao Wang et al. (+2 more)
Federated fine-tuning of Large
Language Models faces severe statistical heterogeneity. However, existing model-level defenses often overlook the root cause: intrinsic data distribution mismatches. In this work, we first establish
Federated Self-
Distillation (FedSD) as a fundamental and potent strategy. By projecting client representations into a smoothed “model-understanding space,” FedSD alone serves as a universal booster, demonstrating superior performance over conventional
algorithms. Despite its success, we identify a subtle trade-off termed the Rewrite Paradox – unconstrained self-
distillation can inadvertently increase
hallucinations and redundancy. To refine this paradigm, we further propose FedSDR (
Federated Self-
Distillation with Rectification), the ultimate reinforced framework. It augments FedSD with a dual-stream mechanism: a local LoRA-S (Smoothing) branch to implicitly absorb heterogeneity via distilled data, and a parallel global LoRA-R (Rectification) branch anchored to raw data to enforce factual correctness. By selectively aggregating only LoRA-R, FedSDR yields a globally aligned and faithful model. Extensive experiments verify its superior performance.
Read the full abstract ▾
All authors Ziheng Ren, Zhanming Shen, Hao Wang, Ning Liu, You Song
Machine Learning
Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learning
Sunwoo Lee, Mingu Kang, Yonghyeon Jo et al. (+1 more)
Cooperation is central to multi-
agent reinforcement learning (MARL), yet learned coordination can be fragile when external perturbations disrupt inter-
agent interactions. Prior robust MARL methods have primarily considered value-oriented attacks, leaving a gap in robustness when interaction structures themselves are corrupted. In this paper, we propose an interaction-breaking
adversarial learning (IBAL) framework that takes an information-theoretic view to construct attacks that impede coordination by perturbing
agents' observations and actions, and trains
agents to perform reliably under such disruptions. Empirically, our approach improves robustness over existing robust MARL baselines across diverse attack settings and yields stronger performance even under
agent-missing scenarios. Our code is available at https://sunwoolee0504.github.io/IBAL.
Read the full abstract ▾
All authors Sunwoo Lee, Mingu Kang, Yonghyeon Jo, Seungyul Han
Machine Learning
Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise
Linyu Liu and Pinyan Lu
Highly over-parameterized models can simultaneously memorize noisy labels and generalize well, yet how these behaviors coexist remains poorly understood. In this work, we investigate the underlying mechanisms of this coexistence using modular arithmetic tasks under heavy label noise. Through extensive experiments on two-layer neural networks, we find that larger models tend to generalize better under appropriate optimization and model configurations, while noisy labels are memorized faster than clean data. Over-parameterized models internally form a generalization structure, but its expression in the output is suppressed by the need to fit noisy labels. Remarkably, even with 80% label noise, near-perfect test accuracy can be achieved by extracting this internal structure using frequency-based methods. We further propose a task-agnostic method to partition networks into generalization and memorization components. Although this subnetwork improves generalization, it is limited compared with frequency-based extraction, indicating that the generalization structure is distributed across
neurons and motivating the development of new tools to retrieve generalizable knowledge from over-parameterized networks.
Read the full abstract ▾
Machine Learning
Federated Learning by Utility-Constrained Stochastic Aggregation for Improving Rational Participation
M Yashwanth, Arunabh Singh, Ashok Nayak et al. (+2 more)
Federated Learning (FL)
algorithms implicitly assume that clients passively comply with server-side orchestration by sharing local model updates upon server request. However, this overlooks an important aspect in real-world cross-silo environments: clients are often rational
agents who may prioritize their utilities such as local model performance over that of the global model. In settings with significant statistical heterogeneity, rational clients may opt out of the federation if the perceived benefits of collaboration fail to meet their local utility thresholds. Such attrition degrades the global model performance and can lead to the collapse of the
federated training process. In this work, we introduce FedUCA, (
Federated Learning by Utility-Constrained
Stochastic Aggregation for Improving Rational Participation), a framework that formalizes the server's role as an optimizer seeking to maximize global model performance by sustaining client participation. We substantiate our framework through extensive experiments on standard
datasets demonstrating that by prioritizing participation feasibility, FedUCA achieves significantly higher client retention and, consequently, a superior global model performance.
Read the full abstract ▾
All authors M Yashwanth, Arunabh Singh, Ashok Nayak, Sai Kiran Bulusu, Anirban Chakraborty
Machine Learning
LogRouter: Adaptive Two-Level LLM Routing for Log Question Answering in Big Data Systems
Mert Coskuner, Merve Zeybel and Melik Mert Dolan
Production log analytics in self-hosted, resource-constrained environments requires natural-
language access to massive log streams without the cost of routing every query through a large
language model. We present LogRouter, an end-to-end log question-answering system deployed on TUBITAK BILGEM's national big data platform that combines a PySpark-based Drain3 ingestion pipeline, GPU-accelerated
embeddings, and dual-index storage in Apache Druid and PostgreSQL with pgvector. A two-level cost-aware router dispatches each query along one of four execution paths: direct response, Druid keyword search, template lookup with SQL generation, and pgvector semantic
retrieval, while a Level-2 router selects either a 14B-class or 32B-class generator for the semantic path. A dedicated coder LLM handles text-to-SQL generation. We evaluate the system on four LogHub
datasets (Linux, Apache, Windows, and Mac; 70 questions in total) under both an online full-pipeline configuration and an offline configuration that isolates the generator. The router reaches 88.4% mean accuracy across
datasets and 94.7% on Linux, while the full pipeline attains a mean ROUGE-1 of 0.373, BERTScore of 0.879, RAGAS Faithfulness of 0.779, and an end-to-end latency of 18.6 s. In an apples-to-apples offline comparison, the routed system reduces mean latency by 55% versus a Fixed-32B baseline (46.3 s vs. 102.1 s) while preserving Answer Correctness within 5.8 points and exceeding a Fixed-14B baseline on RAGAS Faithfulness across every
dataset. Cost-aware dispatching is therefore a practical mechanism for production log QA: routing recovers most of the quality of an always-32B configuration at less than half the latency, and the L1 keyword vocabulary makes that routing decision with high precision without a learned classifier.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
SAS: Semantic-aware Sampling for Generative Dataset Distillation
Mingzhuo Li, Guang Li, Linfeng Ye et al. (+4 more)
Deep neural networks have achieved impressive performance across a wide range of tasks, but this success often comes with substantial computational and storage costs due to large-scale training data.
Dataset distillation addresses this challenge by constructing compact yet informative
datasets that enable efficient model training while maintaining downstream performance. However, most existing approaches primarily emphasize matching data distributions or downstream training statistics, with limited
attention to preserving high-level semantic information in the distilled data. In this work, we introduce a semantic-aware perspective for
dataset distillation by leveraging Contrastive
Language-Image Pretraining (CLIP) as a semantic prior for post-sampling. Our goal is to obtain distilled
datasets that are not only compact but also semantically class-discriminative and diverse. To this end, we design three semantic scoring functions that quantify class relevance, inter-class separability, and intra-set diversity in a pretrained semantic space. Based on image pools generated by existing
distillation methods, we further develop a two-stage strategy for effective sampling: the first stage filters semantically discriminative samples to form a reliable candidate set, and the second stage performs a dynamic diversity-aware selection to reduce redundancy while preserving semantic coverage. Extensive experiments across multiple
datasets, image pools, and downstream models demonstrate consistent performance gains, highlighting the effectiveness of incorporating semantic information into
dataset distillation.
Read the full abstract ▾
All authors Mingzhuo Li, Guang Li, Linfeng Ye, Jiafeng Mao, Takahiro Ogawa, Konstantinos N. Plataniotis, Miki Haseyama
Machine Learning
Uncertainty Reliability Under Domain Shift: An Investigation for Data-Driven Blood Pressure Estimation in Photoplethysmography
Mohammad Moulaeifard, Ciaran Bench, Philip J. Aston et al. (+1 more)
Uncertainty quantification (UQ) is critical for safety-critical domains like healthcare, yet it is rarely evaluated under realistic out-of-distribution (OOD) conditions. Here, we assessed predictive performance and uncertainty reliability for deep learning-based blood pressure (BP) estimation from photoplethysmography (PPG) signals under both in-distribution (ID) and OOD settings. Using an XResNet1D-50 trained on PulseDB and tested on four external
datasets, we compared deep ensembles (DE) and Monte Carlo dropout (MCD) with Gaussian negative log-likelihood (GNLL) and mean squared error (MSE) losses, optionally followed by post-hoc recalibration via conformal prediction (CP), temperature scaling (TS), and isotonic regression (IR). The key findings of our study are as follows: (1) DE provides stronger predictive robustness under domain shift than MCD, an advantage that becomes clear primarily under external shift. (2) Recalibrated GNLL-based methods yield the best uncertainty calibration (e.g., GNLL+DE+CP for systolic blood pressure (SBP), GNLL+DE+TS for diastolic blood pressure (DBP)), while MSE-based uncertainty requires recalibration to become practically useful. (3) Across settings, CP and TS offer the most consistent gains, with IR remaining competitive in several cases. Overall, our results identify DE-based methods as most robust for predictive performance under domain shift, GNLL as strongest for native UQ, and recalibration as essential for making MSE-based uncertainty practical. These findings highlight the need to jointly assess predictive accuracy and calibration on external data for trustworthy cuffless BP estimation
Read the full abstract ▾
All authors Mohammad Moulaeifard, Ciaran Bench, Philip J. Aston, Nils Strodthoff
Machine Learning
Scalable Decision-Focused Learning through Cost-Sensitive Regression
Noah Schutte, Senne Berden, Tias Guns et al. (+2 more)
Many real-world combinatorial problems involve uncertain parameters, which can be predicted given contextual features and historical data. These `predict-then-optimize' or `contextual optimization' problems have gained significant
attention: end-to-end training methods can now minimize the downstream task cost rather than the predictive error. However, despite their effectiveness, these decision-focused learning (DFL) approaches often rely on repeated solving of the underlying combinatorial optimization problem during training, making them computationally expensive and difficult to scale. We reframe the learning problem as a cost-sensitive multi-output regression problem: multi-output due to the combinatorial problem having multiple uncertain parameters, and cost-sensitive due to the downstream task cost being the real target. Our technical contribution is the formalization of multiple loss function components that follow from this reframing: cost-insensitive normalization, decision-aware asymmetric penalization of over- and underpredictions, and instance-based costs that mimic the true downstream task-based loss locally. These components require zero or one solve per training data instance, while requiring no further solves during training. Experiments show that the combination of loss components achieves comparable downstream task quality to the state of the art, while being significantly more efficient, enabling scaling to problem sizes that have not been tackled before with DFL.
Read the full abstract ▾
All authors Noah Schutte, Senne Berden, Tias Guns, Krzysztof Postek, Neil Yorke-Smith
Machine Learning
RL4RLA: Teaching ML to Discover Randomized Linear Algebra Algorithms Through Curriculum Design and Graph-Based Search
Jinglong Xiong, Xiaotian Liu, Ruoxin Wang et al. (+4 more)
Randomized linear
algebra (RLA)
algorithms are a modern class of numerical linear
algebra techniques that play an essential role in scientific computing and machine learning, with broad and growing adoption. However, their discovery remains mostly a manual process that requires deep expert knowledge and inspiration. While Reinforcement Learning (RL) offers a pathway to automation, standard approaches struggle with sparse reward landscapes and vast search spaces inherent to high-performing RLA
algorithms. In this paper, we present RL4RLA, a general RL framework that automates the discovery of interpretable, symbolic RLA
algorithms. Unlike black-box approaches, our method builds explicit
algorithms from basic linear
algebra primitives, ensuring verifiable and implementable representations. To enable efficient discovery, we introduce: (1) a numerical curriculum that progressively increments problem difficulty to encode inductive bias specific to the RLA domain; (2) Monte Carlo Graph Search, which optimizes exploration by identifying and merging equivalent partial
algorithms. We demonstrate that RL4RLA rediscovers state-of-the-art methods, including sketch-and-precondition solvers, Randomized Kaczmarz, and Newton Sketch, and can be targeted to produce
algorithms optimized for specific trade-offs between accuracy, speed, and stability. Code is available at https://github.com/Tim-Xiong/RL4RLA.
Read the full abstract ▾
All authors Jinglong Xiong, Xiaotian Liu, Ruoxin Wang, Zihang Liu, Yefan Zhou, Yujun Yan, Yaoqing Yang
Machine Learning · Cross-listed from cs.CL
Bridging the Gap: Converting Read Text to Conversational Dialogue
Parshav Singla, Agnik Banerjee, Aaditya Arora et al. (+5 more)
In recent advancements within
speech processing, converting read
speech to conversational
speech has gained significant
attention. The primary challenge in this domain is maintaining naturalness and intelligibility while minimizing computational overhead for real-time applications. Traditional read
speech often lacks the nuanced prosodic variation essential for natural conversational interactions, posing challenges for applications in virtual assistants, customer service, and
language learning tools. This paper introduces a novel approach, Prosodic Adjustment with Conversational Context (PACC), aimed at converting read
speech into natural conversational
speech used in various modern applications. PACC utilizes advanced deep neural networks to analyze and modify prosodic features such as intonation, stress, and rhythm. Unlike conventional methods, our approach uses High-Fidelity Generative
Adversarial Networks (HiFi-GAN) for
speech synthesis. Our experimental results demonstrate significant improvements in
speech conversion, enhancing naturalness and achieving better model accuracy with additional training on
speech datasets. This research establishes new
benchmarks in
speech conversion tasks and Mean Opinion Score (MOS) evaluation for testing model accuracy, and we show that our approach can be successfully extended to other
speech conversion applications.
Read the full abstract ▾
All authors Parshav Singla, Agnik Banerjee, Aaditya Arora, Shruti Aggarwal, Anil Kumar Verma, Vikram C M, Raj Prakash Gohil, Gopal Kumar Agarwal
Machine Learning
MARR: Module-Adaptive Residual Reconstruction for Low-Bit Post-Training Quantization
Le Su, Xing Luo and Zhi Jin
Recently, residual reconstruction-based model quantization methods have achieved promising performance in low-bit post-training quantization (PTQ) by introducing cross-layer residuals to reduce error accumulated from previous layers.However, these residuals may also introduce additional bias arising from the Hessian-approximation (HA) assumption underlying reconstruction-based PTQ, leading to suboptimal quantization performance.In this work, we analyze that multiplying the residual term by a scaling coefficient provides a direct way to mitigate the HA bias associated with residual strength, while preserving accumulated-error correction. More importantly, we observe that this trade-off is module-dependent, making a single global residual strength insufficient to balance effective correction and residual-related bias across modules.Based on these observations, we propose Module-Adaptive Residual Reconstruction (MARR), which assigns a module-specific scaling coefficient to adaptively balance accumulated-error correction and residual-related HA bias for each module.To avoid expensive per-module coefficient search and obtain a stable coefficient estimate, we design a Proportional-Integral-Derivative (PID)-based adaptive update strategy that uses reconstruction error as feedback to progressively refine this coefficient. Experiments on several typical large
language models (LLMs) and
vision transformers (ViTs) demonstrate the effectiveness of MARR under low-bit quantization (less than or equal to 4-bit), achieving up to 20.2% performance gains on LLMs and up to 4.6% relative gains on ViTs over the residual reconstruction state-of-the-art methods.Code will be made publicly available upon acceptance.
Read the full abstract ▾
Machine Learning
SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Models
Chengjie Hong, Feixiang He, Yiheng Zeng et al. (+2 more)
We propose a new method for compressing physics foundation models (PFMs) which is a new trend in AI for Science. While model compression is essential for reducing
memory use and accelerating inference in large foundation models, it remains under-explored for PFMs, where preserving physical fidelity is crucial. The challenge lies in the functional nature of physics data, where partial derivatives encode spatiotemporal dynamics and exhibit high sensitivity to compression. Conventional compression methods ignore this structure, often causing severe performance degradation or failure. To address this, we introduce a sensitivity-aware fidelity-enforcing compression framework that explicitly models loss-aware layer sensitivity in the output function space during compression. This provides a new route to compressing scientific foundation models while preserving accuracy and physical fidelity. Experiments show substantial gains over existing methods across multiple models and
datasets, achieving significantly higher compression ratios while maintaining accuracy, in some cases by orders of magnitude. More broadly, the work potentially leads to a new subfield of efficient, deployable, and sustainable scientific foundation models in AI for Science.
Read the full abstract ▾
All authors Chengjie Hong, Feixiang He, Yiheng Zeng, Lulu Kang, He Wang
Machine Learning
Function graph transformers universally approximate operators between function spaces
Takashi Furuya, David Mis, Ivan Dokmanić et al. (+2 more)
We study the approximation of nonlinear operators between function spaces by
transformers. Our approach is to lift functions to measures supported on their graphs and leverage a recently introduced measure-theoretic view of
transformers. A function $h$ is represented by its graph measure $\gamma_h$, with finite
tokens $\{(x_j,h(x_j))\}_{j=1}^N$ being its empirical approximations. We show that this framework elegantly models discretization refinement via convergence of measures and provides a natural setting for operator learning. Within this framework, we introduce function graph
transformers, a graph-preserving subclass of measure-theoretic
transformers that maps graph measures to graph measures, which is to say that outputs remain single-valued functions. Crucially, this additional structure does not reduce generality: we prove that the resulting graph-preserving maps can be approximated by finite compositions of standard softmax self-
attention layers and pointwise MLPs, yielding universal approximation results for broad classes of nonlinear operators. Unlike existing theoretical approaches to operator learning with
transformers, the measure-theoretic framework also accommodates regularized negative-order Sobolev inputs for which discretization invariance is particularly challenging, as well as query points on different output domains. Overall, function graph
transformers provide a continuum viewpoint and mathematical toolkit for
transformer-based operator learning, clarifying the roles of positional encodings, graph structure, regularization, and ensuring consistency across discretizations.
Read the full abstract ▾
All authors Takashi Furuya, David Mis, Ivan Dokmanić, Maarten V. de Hoop, Matti Lassas
Machine Learning
Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning
Zhanyue Qin, Jia Feng, Yibo Lyu et al. (+4 more)
Code
reasoning refers to the task of predicting the output of a program given its source code and specific inputs. It can measure the
reasoning capability of large
language models (LLMs) and also benefit downstream tasks such as code generation and mathematical
reasoning. Existing work has verified the effectiveness of reinforcement learning on the task. However, these methods design rewards solely based on final outputs or coarse-grained signals, and neglect the inherent consistency of the stepwise
reasoning process in the task. Therefore, these methods often result in sparse reward or reward hacking, which limits the full play of enhanced learning capabilities. To alleviate these issues, we propose CodeThinker, a consistency-driven reinforcement learning framework for code
reasoning. Specifically, CodeThinker has three key components: (1) a stepwise
reasoning-aware model training module, which utilizes a consistency tracing paradigm as a template to synthesize training data that captures the stepwise
reasoning process; (2) a dynamic beam sampling strategy, which aims to improve the quality of sampled outputs under a fixed sampling budget; and (3) a consistency reward mechanism that can effectively alleviate reward hacking. Experiments on three popular
benchmarks show that CodeThinker achieves state-of-the-art performance across multiple LLMs. For instance, it outperforms the strongest baseline by 4.3% in accuracy when deployed on Qwen2.5-Coder-7B-Instruct. We also validate the effectiveness of CodeThinker on downstream tasks. Results show that, without additional training, CodeThinker obtains average accuracy gains of 5.33 and 3.11 percentage points on mathematical
reasoning and code
reasoning tasks covering 17 programming
languages, respectively.
Read the full abstract ▾
All authors Zhanyue Qin, Jia Feng, Yibo Lyu, Yun Peng, Dianbo Sui, Cuiyun Gao, Qing Liao