Machine Learning
Fed-ReMasker: Federated Tabular Imputation under Feature-Level Missingness
Ioannis Papathanail, Rooholla Poursoleymani, Lubnaa Abdur Rahman et al. (+1 more)
Multi-center clinical studies and biomedical research collaborations increasingly seek to utilize data across centers to build models that generalize beyond any single center. This creates two distinct challenges: data protection regulations may restrict the sharing of raw patient data across institutions, while centers may collect only partially overlapping sets of features under different protocols.
Federated learning enables collaborative model training without centralizing raw data. However, existing
federated imputation methods rarely evaluate feature-level missingness, in which entire features are unobserved at some centers. To address this setting, we adapt the ReMasker masked autoencoder to
federated learning (Fed-ReMasker), enabling centers to impute features never observed locally by leveraging knowledge learned across collaborating centers. We evaluate Fed-ReMasker in a
benchmark spanning synthetic
datasets with linear and nonlinear relationships and real-world tabular
datasets, including clinical data. The
benchmark varies the number of centers, the missingness ratios, and client heterogeneity. Fed-ReMasker achieves the lowest imputation error in 93.2% of value-level and 96.7% of feature-level scenarios in the homogeneous
benchmark. It also remains robust to client heterogeneity using simple
federated averaging, outperforming all baselines in all 36 value-level scenarios and each baseline in at least 35 of 36 feature-level scenarios, and comes within 3.0% on average of a centralized model trained on the pooled data.
Read the full abstract ▾
All authors Ioannis Papathanail, Rooholla Poursoleymani, Lubnaa Abdur Rahman, Stavroula Georgia Mougiakakou
Machine Learning · Cross-listed from cs.CV
Visual Tripwires: Anticipating Failure in Deep Vision Systems
Anoushka Harit, Rehan Zuberi, William Prew et al. (+1 more)
Deep
vision systems remain vulnerable to corruption, occlusion, and distribution shift despite strong
benchmark performance. Existing reliability methods typically evaluate uncertainty at individual time steps and do not explicitly model how a system progresses toward failure. We introduce Visual Tripwires, a predictive reliability framework that uses temporal instability in model behaviour to anticipate impending failure. Our central hypothesis is that predictive degradation develops progressively through measurable changes in latent representations, prediction trajectories, and
attention structure. Visual Tripwires captures these changes using representation drift, prediction oscillation, trajectory
curvature, and
attention entropy. A lightweight tripwire predictor aggregates these signals over a temporal window to estimate the probability of failure within a future prediction horizon. Experiments across multiple
datasets, architectures, and progressive perturbation settings show that the proposed instability signals emerge before predictive degradation and provide earlier and more accurate failure warnings than conventional uncertainty estimation methods. These results demonstrate that temporal instability contains useful information about future model reliability and provides a practical basis for early warning in deep
vision systems.
Read the full abstract ▾
All authors Anoushka Harit, Rehan Zuberi, William Prew, Florian Markowetz
Machine Learning · Cross-listed from cs.AI
Discovery of fully efficient fault indicators along a data-based diagnosis process
Igor Bezmaternykh, Louise Travé-Massuyès and Elodie Chanthery
The integration of model-based and data-driven paradigms provides a powerful framework for fault diagnosis by combining the interpretability of analytical redundancy relations, i.e., input-output relations that are used as diagnosis indicators in model-based diagnosis, with the adaptability of learning techniques. DT4X is a recent diagnosis
algorithm that uses symbolic regression to generate multivariate relations leveraging some properties of analytical redundancy relations and uses them as split functions in a decision tree. However, its symbolic regression procedure optimizes only the separation between two selected classes at each node, often fragmenting the remaining classes and degrading both interpretability and diagnosis performance. This paper introduces DT4X+, an enhanced version of DT4X that modifies the construction of training sets and the symbolic-regression loss so that expressions separate the target classes while preserving the coherence of non-target classes. The resulting relations become fully consistent with ARR properties and lead to more informative splits, improved robustness, and better performance on dynamic-system
datasets. Experiments conducted on several
benchmark systems demonstrate the benefits of this enhanced formulation.
Read the full abstract ▾
Machine Learning
LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations
Sandra Arcos-Holzinger, Debashish Chakraborty, Rohita Mocharla et al. (+7 more)
We propose LAYERSCOPE, a label-free, layerwise framework that aims to characterize a model's learned representations in
video and
multimodal settings. Evaluating downstream performance using representations from final or intermediate layers typically requires large amounts of labeled data, repeated task-specific evaluations, and substantial computation. To address these limitations, LAYERSCOPE uses local, global, distributional, and correspondence-based
geometric metrics to compare layerwise representation structure within and across models without requiring task-specific labels. We evaluate seven architecturally diverse models across
video and
multimodal classification, clustering, and text-to-
video retrieval tasks from MVEB/MVEB+. We find that intermediate-layer representations can outperform final-layer and model-default outputs. We also find that no single
geometric metric consistently predicts downstream performance, but note that distinct layerwise
geometric signatures emerge across model families. LID shows task-dependent relationships with performance, while RankMe provides the strongest measure for classification and clustering, but is not a universal layer selector. We also find that pairing-aware metrics explain
retrieval better than distributional distances alone. LAYERSCOPE therefore offers a framework for comparing representations across models and layers, enabling a more systematic evaluation in
video and
multimodal settings.
Read the full abstract ▾
All authors Sandra Arcos-Holzinger, Debashish Chakraborty, Rohita Mocharla, Will Walden, Andrew Yates, Reno Kriz, Sarah M. Erfani, James Bailey, Vishal M. Patel, Sanjeev Khudanpur
Machine Learning
Curriculum Learning with GNN-based Reinforcement Learning for Job Shop Scheduling
Jayakrishnan K. Vasudevan, Jonathan Hoss and Noah Klarmann
The job shop scheduling problem is a challenging combinatorial optimization problem, and recent reinforcement learning approaches using graph neural networks have shown promise for learning scheduling policies directly from problem instances. However, training on large instances remains computationally expensive, and generalization across instance sizes remains challenging. This paper studies curriculum learning for graph neural network-based reinforcement learning in the job shop scheduling problem by comparing it with single-size training across three target sizes: 20 x 20, 25 x 25, and 30 x 30. In the curriculum setting, the policy is first trained on smaller instances and then progressively adapted to larger target sizes, allowing scheduling behavior learned in earlier stages to support learning on larger instances. Models are evaluated on unseen instances from 8 x 8 to 30 x 30 using the optimality gap, considering both generalization across all evaluation sizes and specialization on the target size. Results show that curriculum learning consistently reduces wall-clock training time, with larger benefits as the target size increases. The strongest advantage is observed at 30 x 30, where curriculum learning reduces the mean optimality gap across all evaluation sizes by approximately 8.1 percentage points, reduces the target-size mean optimality gap by approximately 8.6 percentage points, and saves approximately 50 hours of training time.
Read the full abstract ▾
Machine Learning
Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts
Pit Neitemeier, Jiaze Li, Alessio Serra et al. (+2 more)
Mixture-of-Experts (MoE) training requires global load balance to prevent expert under-utilization and local balance for efficient expert-parallel execution. Existing distributed
Quantile Balancing (QB) uses shard-dependent or approximate global
quantiles, while
token-independent expert biases cannot ensure microbatch-level balance. We introduce Exact
Quantile Balancing (EQB), which computes exact global-batch BF16
quantiles with negligible communication, and Load-Error Injection (LEI), which injects local load errors directly into router-score gradients. On 7.5B-parameter MoEs trained for up to 500B
tokens, EQB improves global balance and downstream performance over naive QB, while LEI improves local balance and outperforms the GShard loss at comparable quality.
Read the full abstract ▾
All authors Pit Neitemeier, Jiaze Li, Alessio Serra, Philipp Scholl, Sohir Maskey
Machine Learning · Cross-listed from math.NA
Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison
Rahul Krishnan and Volker Schulz
The key-value (KV) cache of autoregressive
transformers can be viewed as a fourth-order tensor spanning
attention heads,
tokens, features, and grouped layers. We measure the singular-value spectra of all four mode unfoldings on Mistral-7B-v0.3 and LLaMA-2-13B and compare four standard tensor decompositions: Tucker, CP, tensor train, and t-SVD, at matched storage. The spectra partition the four axes into two classes. The
token and feature modes carry low-rank structure, particularly for keys. The head and layer modes are nearly full-rank and resist compression at any practical error level. Among the four decompositions, Tucker achieves the lowest reconstruction error at every compression ratio from $2\times$ to $5\times$, because it can leave the full-rank modes untouched. Comparisons with two-dimensional unfolding baselines show that the preferred representation differs between keys and values: 2D methods achieve lower key error, while four-way Tucker achieves lower value error at matched storage. A mode-pinning
theorem certifies the full-rank preservation from the measured spectra alone. Two further spectral properties affect the compressible modes without touching the full-rank ones: values reach a higher error floor than keys at every ratio, and post-RoPE keys lose $41\%$ - $64\%$ of their pre-RoPE compressibility on both models.
Read the full abstract ▾
Machine Learning
PISCES: Physics-Informed Solar-wind Convolutional autoEncoder for Space-weather Anomaly Detection and Early Warning
Kevin Lee and Alison J. March
Space weather early warning depends on detecting
solar wind transients in in-situ measurements at the first Sun-Earth Lagrange point (L1), before they reach Earth. Fixed thresholds can miss combined
magnetic and
plasma structure, and many learning methods provide a single anomaly score. We present the Physics-Informed
Solar-wind Convolutional autoEncoder for Space-weather (PISCES), a convolutional autoencoder trained without catalog labels on OMNI
solar wind measurements under physics constraints. Its loss includes
magnetic field consistency, an empirical relation between temperature and velocity, the Parker spiral angle, and penalties on changes between consecutive one-minute samples in derived quantities calculated from the reconstruction. At inference, PISCES separates the anomaly score into
magnetic and
plasma reconstruction errors, physics relations, and residual corrections, and reports the magnitude of each contribution. Attenuation of the skip connections, selected on validation data, improves average precision for the trained models, while the untrained scores remain nearly the same. The trained models also give a more consistent ordering of these physical contributions. After smoothing with a trailing median, the alarms can precede independently observed sudden commencements, including positive sudden impulses.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Improving Ensemble Filters with Flow Matching
Haoyuan Chen and Alexandre Thiéry
Data assimilation estimates a dynamical state from partial and noisy observations. Classical ensemble filters are efficient but restrict analysis updates through finite sample covariance and affine Gaussian distribution. We introduce the Flow Ensemble Filter (FlowEF), which uses conditional flow matching to transport the forecast ensemble from a classical baseline filter to an analysis ensemble. FlowEF uses a localized Gaussian source during training, transports forecast ensemble members from a baseline filter at deployment, and conditions its velocity field on ensembles from that baseline filter and the observation. The proposed model therefore learns a nonlinear update while mapping each baseline ensemble independently. For sparsely observed dynamical systems, FlowEF improves both deterministic and probabilistic metrics over all four classical ensemble filters. It also achieves the best performance among the state-of-the-art generative data assimilation models.
Read the full abstract ▾
Machine Learning · Cross-listed from cond-mat.mtrl-sci
SoLiD26: A First Principles Solid-Liquid Interface Dataset for Machine-learned Interatomic Potentials
Jonas Busk, Emil J. P. Frost, Yogeshwaran Krishnan et al. (+7 more)
Machine-learned interatomic potentials (MLIPs) enable simulations of larger atomistic systems over longer timescales than traditional ab initio methods but require large, diverse training data. Existing
datasets lack comprehensive sampling of solid-
liquid interfaces, which exhibit complex phenomena such as solvent layering and surface reconstruction critical for energy and catalytic materials. Here, we present SoLiD26, a curated solid-
liquid interface
dataset of 15.4 million first-principles structures with up to 576 atoms and 15 chemical elements for training and evaluating MLIPs. The structures were compiled from density functional theory (DFT) calculations, with most configurations originating from ab initio molecular dynamics (AIMD) simulations. SoLiD26 includes aqueous coinage metal interfaces, electrode-electrolyte systems, and selected bulk reference structures, calculated with VASP using the PBE functional and D3 dispersion corrections. The application of SoLiD26 for training and evaluating MLIPs is demonstrated with a suite of MACE models on a simple training, validation and test split. This
dataset provides a valuable resource for advancing MLIP development targeted at modelling solid-
liquid interfaces in electrochemical and catalytic systems.
Read the full abstract ▾
All authors Jonas Busk, Emil J. P. Frost, Yogeshwaran Krishnan, Henrik H. Kristoffersen, August E. G. Mikkelsen, Xueping Qin, Xin Yang, Heine A. Hansen, Arghya Bhowmik, Tejs Vegge
Machine Learning · Cross-listed from cs.CL
Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints
Imtiaz Ul Hassan, Öykü Akbulut, Onur Kaya et al. (+4 more)
Most Turkish-capable large
language models (LLMs) are evaluated using general-purpose
benchmarks rather than long, structurally complex domain documents. This paper evaluates five open-weight 7B-8B models for Turkish document question answering under a resource-constrained local deployment setting. The primary
benchmark contains 100 systematically validated questions derived from a 109-page industrial R&D report, and the evaluation protocol is replicated using a second 112-page public-sector report and an independently constructed 100-question set. All models are evaluated locally on an NVIDIA RTX 3050 laptop GPU with 6 GB VRAM using controlled prompting, decoding, and 4-bit quantisation. The principal methodological contribution is an evidence-annotated evaluation protocol that separates
retrieval failure from downstream model
reasoning failure without requiring additional model calls. On the primary
benchmark, end-to-end accuracy ranges from 49% to 75%. Seven lexical, dense, and hybrid
retrieval configurations are additionally compared using 95% Wilson intervals and exact paired McNemar tests; none significantly outperforms the character TF-IDF baseline on either document. Evidence recall saturates differently across the two reports, showing that
retrieval and effective context capacity can be binding constraints for some documents but not others. These results demonstrate that model selection,
retrieval behaviour, and hardware limits must be evaluated separately when deploying open-weight LLMs for Turkish domain documents.
Read the full abstract ▾
All authors Imtiaz Ul Hassan, Öykü Akbulut, Onur Kaya, Ardhendu Behera, Swagat Kumar, Peter Matthew, Yonghuai Liu
Machine Learning
Shared Global KV with Layer-Specific Local History
Xinglang Xian
Decoder-only
Transformer language models cache keys and values (KV) to reuse past computation during generation. Sharing KV across layers saves storage but reduces the diversity of representations available across depth. We study what local
memory should retain alongside shared global KV, separating historical content from the input source used to form it. At 126M parameters and 2K context, an eight-seed study finds about 1.4% lower held-out test perplexity with local history than with a current-
token local branch. Capacity, entry-count and training-compute controls support the value of historical content. In a two-seed comparison, this value persists when adjacent layers share local inputs while retaining independent projections; source sharing also shortens exact cache-construction dependencies. Against GQA and adjacent-layer KV sharing, equal bounded learning-rate searches and new-seed confirmation yield better same-source likelihood with larger caches and higher long-request latency. The ordering against adjacent-layer sharing persists after equal-
token adaptation to 8K, with a short-context cost. The eight-seed external-book history effect remains uncertain, and downstream outcomes vary by task. We derive a sufficient suffix schedule that reduces upper-layer construction work while preserving the complete cache in exact arithmetic.
Read the full abstract ▾
Machine Learning
Learning from Failures: Heterogeneous Graph Memory for Small Language Model Tool-Using Agents
Jiaxing Li, Lei Song, Rui Dong et al. (+1 more)
Small and medium-sized
language models offer cost-effective executors for tool-using
agents, making them attractive for local and large-scale deployment. However, in long-horizon and stateful environments, they often make structural errors such as missing required observations, performing premature writes, repeating failed calls, and violating action preconditions. These errors can lead to incorrect state updates, policy violations, and costly or irreversible consequences, making reliable tool execution a critical deployment challenge. Existing fine-tuning approaches require substantial data and computation, while flat
memory may retrieve failed actions without preserving their causal context or safety conditions. In this paper, we propose FRESH, a Failure-aware
Retrieval framework over Experience-Structured Heterogeneous graphs, which transforms historical successes and failures into structured external experience for tool-using
agents. By explicitly modeling the dependencies among tasks, actions, errors, repairs, and execution conditions, FRESH helps frozen
language models reuse reliable strategies, avoid recurring failures, and make safer decisions in stateful tool interactions. Experiments on $\tau$-Bench and AppWorld with multiple open-source models show that FRESH consistently improves task success and tool-use reliability over no-
memory agents and representative
memory-based baselines.
Read the full abstract ▾
All authors Jiaxing Li, Lei Song, Rui Dong, Youyong Kong
Machine Learning · Cross-listed from cs.CV
Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces
Andrew Bond, Ege Erdem Özlü, Tuna Çimen et al. (+4 more)
Methods operating on
Vision Transformer (ViT) feature spaces typically rely on Euclidean distance or cosine similarity. This assumes that every direction is equally meaningful, but there is no reason to believe the true task geometry has this property. The task-sensitive geometry of the feature space is given by the pullback metric $g(F) = J(F)^\top J(F)$, where $J$ is the Jacobian of the decoder's output fed to a task-specific distance, with respect to the features. Storing the full $g$ is infeasible at modern scales, and for dense outputs such as depth maps even forming $J$ is impractical. We show that whether a low-rank approximation of this metric can be learned depends on the model-decoder pair, and we characterize this with a
matrix-free diagnostic $\kappa_{cap}(r)$ computable with a low number of Jacobian-vector products. For tractable pairs, we develop the Spectral Pullback Network (SPN), which learns a low-rank version of the metric from randomized power iteration, and we distill it into a $310$K-parameter importance head that predicts
token importance directly from the features. When the Jacobian spectrum is too spread out for a low-rank approximation, passing the decoder's input features through a VAE bottleneck can restore tractability. Across DPT, DINOv2, CLIP, and VGGT backbones, $\kappa_{cap}(r)$ predicts which learned-metric architectures are viable. The importance head reaches Spearman $\rho = 0.998$ on DINOv2 CLS, and our
geometric token pruning reduces the additional depth error of ToMe-based
token selection by $25\%$ on DPT depth at prune ratio $0.5$, without fine-tuning the ViT. Project page: https://cyberiada.github.io/TaskInducedViTs/
Read the full abstract ▾
All authors Andrew Bond, Ege Erdem Özlü, Tuna Çimen, Ilkin Umut Melanlioglu, Tolga Birdal, Erkut Erdem, Aykut Erdem
Machine Learning
PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue
Chenxuan Li, Jiayi Wan, Xinrong Chen et al. (+3 more)
Large
language models (LLMs) have made substantial progress on medical question-answering, yet effective medical dialogue also requires learning to ask questions that uncover relevant patient information. To train such dialogue policies, a common pipeline combines supervised fine-tuning with reinforcement learning (RL) based on final diagnostic correctness. However, this outcome-based supervision does not directly distinguish the contributions of individual questions and provides no question-level feedback for unexecuted alternatives. To address this gap, we introduce PCQC (Privileged Counterfactual Question Credit), which uses privileged patient information during training to learn from questions never asked. During training, PCQC makes alternative questions directly comparable at the same dialogue state by using privileged patient facts to construct their answers. A frozen diagnostic scorer evaluates the diagnostic utility of each resulting question-answer pair by how strongly it supports the correct diagnosis. PCQC turns these comparisons into relative question credit that teaches the policy which questions to favor, directly supervising both executed and unexecuted questions alongside outcome-based RL without requiring complete rollouts for the unexecuted alternatives. Extensive experiments across four medical
benchmarks demonstrate that PCQC achieves 63.10% mean diagnostic accuracy, outperforming GRPO and ATPO by 4.38 and 4.21 percentage points, respectively. These gains are achieved with 33.1% fewer inquiry turns than GRPO.
Read the full abstract ▾
All authors Chenxuan Li, Jiayi Wan, Xinrong Chen, Zhongyu Zhao, Xuecheng Shang, Peixing Wan
Machine Learning
Relative Discharge Stage (RDS) Classification: A Practical Indicator of Battery Discharge Progress
Khoa Tran, Tri Le, Hung-Cuong Trinh et al. (+1 more)
Accurate remaining discharge time (RDT) prediction is challenging in real-world battery applications because future load profiles are unknown and highly dynamic. To address the uncertainty of continuous RDT regression, this paper introduces Relative Discharge Stage (RDS), a battery-management indicator that represents the remaining discharge condition using five interpretable classes: Normal, Good, Moderate, Low, and Recharge Required. Unlike state of charge (SOC), which reflects the current charge level, RDS characterizes the remaining discharge process without requiring future-current information during inference. A physics-informed RDS classification framework is proposed, combining SOC estimation with lightweight temporal learning. The SOC-estimation component includes second-order ECM state and terminal-voltage prediction, hysteresis and OCV temperature correction, core-temperature estimation, and AEKF state correction, supported by OCV evaluation, online STC-ECM parameter adaptation, and pretrained neural residual-voltage correction. The measured current, terminal voltage, surface temperature, and estimated SOC are arranged into a sliding observation window and processed by a lightweight temporal convolutional network. Experiments on two public lithium-ion battery
datasets demonstrate robust RDS classification, with accuracy exceeding 80% under varying load and thermal conditions.
Read the full abstract ▾
All authors Khoa Tran, Tri Le, Hung-Cuong Trinh, Hung Tran-Nam
Machine Learning
Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
Ali Aliev and Maxim Rakhuba
In this paper, we are concerned with
matrices formed by block-diagonal factors interleaved with fixed permutations – a flexible family of structured
matrices. This class has recently drawn interest in deep learning architectures for its balanced expressivity-efficiency trade-off, yet efficient computational strategies for working with it remain to be found. We approach this problem through Riemannian geometry and examine under what conditions this class admits a smooth
manifold structure. For the practically important case of orthogonal two-factor
matrices, we derive the essential Riemannian tools and propose efficient
algorithms for their implementation. The
algorithms leverage automatic differentiation, support parameter sharing within each factor, and avoid explicit dense
matrix construction. We test them within the Riemannian optimization framework on the best
matrix approximation problem and for parameter-efficient fine-tuning of large
language models. Beyond the two-factor setting, we study the
geometric and
matrix-theoretic properties of factorizations with a larger number of block-diagonal factors.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets
Beomgu Kang, SoJin Yun, Hojoon Kim et al. (+1 more)
KV-cache eviction is typically evaluated through average quality-
memory trade-offs, yet a small average loss can hide requests whose utility degrades materially. We reformulate eviction as a deployment risk-control problem: a material degradation occurs when eviction lowers task utility by more than a deployment-specified tolerance relative to full-KV inference on the same request, and deployment risk is the population frequency of such events. Given a reliability contract specifying a target risk level and confidence requirement, we use a compressor-agnostic post-hoc certification procedure to select a retention policy from calibration data with a finite-sample guarantee, falling back to full KV when no compressed policy is certified. Across multiple eviction methods, Llama and Mistral models, and LongBench and RULER-32K, the same contract supports substantially different levels of eviction: on Llama, it certifies SnapKV at 75% retention on LongBench but no tested compressed policy on RULER-32K, triggering full-KV fallback. Policies with empirical degradation rates below the 5% target can still fail finite-sample certification; on Llama LongBench, empirical thresholding selects uncertified policies that retain 5-10 percentage points less cache across fixed-budget methods. The proposed framework converts a deployment-level reliability requirement into a KV-
memory operating point.
Read the full abstract ▾
All authors Beomgu Kang, SoJin Yun, Hojoon Kim, Hyunseok Seo
Machine Learning · Cross-listed from cs.CL
Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Rasmus Aagaard and Nicki Skafte Detlefsen
Pruning large pre-trained
transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as
pruning the decoder led to significant end-to-end transcription speedups. For instance, the whisper-large-v3-turbo variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some
attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to $18.5\%$ of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual
speech data to recover performance degradation caused by the zero-shot layer
pruning. Mean WER across four
languages increases to $20.1\%$ after
distillation, compared to $21.9\%$ zero-shot, going from a baseline of $18.2\%$. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions
Michael Lawrence Castanares, Princess Ventures and Allan Tan
The surge in AI-assisted generation of educational materials has outpaced our capacity to validate their pedagogical quality. Automated evaluation using Bloom Classifier models is a promising approach to assess educational materials at scale. These models show high accuracy within-distribution
dataset (IID
Dataset). However, applying the same models to new out-of-distribution (OOD)
datasets such as AI-assisted generated questions could show performance degradation. To identify robust classifiers under
dataset shift, we evaluated traditional Machine Learning (ML),
transformer, and Large
Language models on the Bloom level classification task. We also explored feature-engineering strategies incorporating NLP metrics, appending the learning objectives as part of the input, and text splicing to stabilize OOD performance. Our baseline tests show that TFPOS-IDF ML models perform poorly on OOD (Macro F1-score 0.48) compared to BERT (0.55) and LLMs (0.79). Text splicing improved macro F1-score performance of ML and BERT models (0.59 and 0.62, respectively). Appending the learning objectives with the input increased model performance on specific
dataset. Model retraining provided the largest improvement across models and
datasets. Overall, these findings highlight the trade-off on the use of pre-trained models with novel AI-assisted educational questions and how strategic feature enhancements help address loss in performance.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
Less Language, More Latents: Annotation-Efficient VLAs for Driving
Alexey Zakharov, Kemal Oksuz and Puneet K. Dokania
Vision-
language-action models (VLA) promise human-steerable autonomous driving, but their training is bottlenecked by the scarcity of frames paired with natural-
language instructions: while camera streams and expert trajectories are logged at scale,
language annotations (e.g., turn left at the intersection) remain scarce and expensive to acquire. To address this challenge, we introduce Latent Action Driving Annotations (LADA), a three-stage pipeline that transforms abundant unlabelled observation-trajectory pairs into a substrate for
language-conditioned control. First, we train a latent action model with a vector-quantised bottleneck, producing a compact codebook of high-level vehicle intents. Second, a small
language-annotated subset is used to train a
vision-
language translator to map observations and
language instructions into this codebook. Third, we train a driving VLA on observation-latent-action pairs over the full unlabelled corpus. Using fewer than 5% of
language annotations and without leveraging any auxiliary chain-of-thought
reasoning or visual question answering streams, LADA achieves a Driving Score of 87.98 and a Success Rate of 70.46% on the closed-loop Bench2Drive
benchmark, matching or surpassing fully supervised baselines.
Read the full abstract ▾
Machine Learning
Limiting-Kernel Q($\lambda$): Bridging Short and Long Horizons
Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani et al. (+1 more)
In reinforcement learning, more accurate policy evaluation has been frequently shown to improve downstream policy optimization performance. For tractability, most value
estimators rely on $n$-step truncation and thereby omit the long-horizon behavior of the transition dynamics. Methods that instead capture this behavior through the global structure of the dynamics can converge faster, but their computational and
memory requirements limit their scalability to large or continuous state spaces. To reconcile these limitations, we introduce Limiting-Kernel Q($\lambda$) (LKQL), an off-policy value
estimator that uses the limiting kernel to approximate the long-horizon terms omitted by truncation. LKQL has the same order of complexity as $n$-step
estimators and integrates directly into both on- and off-policy actor-critic
algorithms. We prove that, under aperiodicity and in the near-on-policy regime, the operator underlying LKQL improves the policy evaluation convergence rate over its truncated counterpart for sufficiently large $n$, and that LKQL itself converges almost surely to the optimal values in finite Markov decision processes (MDPs) under a fixed behavior policy. On the MuJoCo continuous-control
benchmark, we show that LKQL improves over $n$-step baselines in most settings, particularly on long-horizon tasks.
Read the full abstract ▾
All authors Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani, Mohamad Amin Sharifi Kolarijani
Machine Learning
MENO: Memory-Efficient Neural Operator
Shengyang Xu, Weijun Zhang, Jun Hu et al. (+1 more)
We propose the
Memory-Efficient Neural Operator (MENO) as a high-performance PDE neural solver based on the
Manifold Function Encoder (MFE). MENO features three primary advantages: (1) MENO has a significantly smaller
memory footprint and much faster training speed than other popular architectures, with the
memory footprint being independent of the data resolution, and therefore holds the potential for scaling up to large-scale models. (2) MENO can accept PDE inputs of arbitrary form, including arbitrary
geometric domains and arbitrary discretizations. In particular, it is capable of handling cross-geometry scenarios, i.e., where the input functions and the output solutions are defined on different
manifolds. (3) MENO exhibits strong generalization capability, and achieves the best accuracy on most of the
benchmarks we tested, compared with the results reported in the literature. The code is available on GitHub at https://github.com/jpzxshi/MENO, and all numerical examples in this paper can be run with a single command to reproduce the reported results.
Read the full abstract ▾
All authors Shengyang Xu, Weijun Zhang, Jun Hu, Pengzhan Jin
Machine Learning
Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang et al. (+1 more)
Newton-Schulz (NS) iteration has recently been used in the
Muon optimizer to transform update
matrices during the training of large
language models. Motivated by its spectral effect, we investigate applying NS directly to
Transformer attention representations. We introduce Newton-Schulz
Attention (NS-Attn.), a parameter-free transformation applied to the output of each
attention head. Each head output is arranged as a feature-by-
token matrix and normalized by its Frobenius norm. We then apply a finite NS
polynomial step and restore the original norm. The objective is to reduce spectral concentration and increase effective rank before standard head merging and output projection. Across ViT and Swin on CIFAR-10 and CIFAR-100, NS-Attn. improves final-epoch accuracy in all 12 matched-seed comparisons, with mean gains of 0.25–0.83 percentage points. ViT ablations show higher mean accuracy with one iteration than with two. Spectral analysis further shows reduced leading-
eigenvalue concentration and increased effective rank. These gains incur additional inference latency.
Read the full abstract ▾
All authors Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang, Ronghui Mu
Machine Learning · Cross-listed from cs.CV
FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathology
Anh-Tien Nguyen, Trung DQ. Dang, Nghiem Tuong Diep et al. (+9 more)
Pathology
vision-
language foundation models vary in performance across diseases and tasks, with no single model consistently performing best. The high cost of expert pathology annotation can also limit the labeled data available for task-specific adaptation. Combining complementary pretrained representations is a potential approach to these limitations, yet learning an effective fusion from few labeled examples remains challenging. We introduce Few-shot Fusion Foundation Models of Computational Pathology (FFM-CP), which is a framework that combines multiple pathology
vision-
language models in the few-shot learning setting. The framework first aligns heterogeneous representations using a closed-form Orthogonal Procrustes transformation estimated from corresponding support images. This
alignment preserves within-model feature geometry without training an additional
alignment network. Within the aligned space, a unified graph enables information exchange across backbones by jointly refining support-image features and visual and textual class prototypes. These refined representations support complementary text-prototype and case-
retrieval branches that capture semantic class knowledge and within-class visual variation, respectively. Each branch learns to combine predictions from all ordered backbone pairs, allowing queries encoded by one model to draw on evidence represented by another. We evaluate three backbone combinations on six histopathology
datasets at 4, 8, and 16 shots per class. FFM-CP achieves higher mean macro-F1 than the strongest individually adapted member of each fused set in 50 of 54 comparisons. These findings suggest that combining complementary pretrained representations can improve histopathological classification when annotations are limited.
Read the full abstract ▾
All authors Anh-Tien Nguyen, Trung DQ. Dang, Nghiem Tuong Diep, Bui Ngoc Han Nguyen, Tan-Ha Mai, Miriam Cindy Maurer, Phuong Hoa Nguyen, Thi Thuy Uyen Nguyen, Youngjun Park, Daniel Sonntag, Duy Minh Ho Nguyen, Anne-Christin Hauschild
Machine Learning
What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates
Tian Zhou, Beverly Jin, Linxiao Yang et al. (+6 more)
A tabular foundation model must discover which distinctions matter for each new table without updating its parameters. We develop in-situ representation refinement: support labels guide changes to the episode's representations, improving the information available to later queries. A regularized leave-one-out objective yields a support correction and its query extension. The leading term separates
attention-based reading from state-dependent scaling, motivating RefineICL: an
attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed
memory. A direct intervention tests the role of evolving support states: removing one intermediate support update while preserving the block's query output increases final query cross-
entropy in all 72 tested episodes. RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A
benchmark-informed continuation reaches 1644.8 Elo on the 38-
dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation. It also improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference
memory at L8. These results connect learning within a forward pass to representation refinement and show how this view guides a competitive,
memory-efficient model.
Read the full abstract ▾
All authors Tian Zhou, Beverly Jin, Linxiao Yang, Xue Wang, Wenwei Wang, Bingqing Peng, Mengni Ye, Jinjie Gu, Liang Sun
Machine Learning
Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization
Jiaxi Wu, Tiantian Zhang, Yuxing Wang et al. (+2 more)
Reinforcement learning (RL) achieves strong performance in sequential decision-making but remains brittle under dynamic uncertainty and distributional shifts. Robust
Adversarial Reinforcement Learning (RARL) improves robustness via worst-case perturbations, but existing approaches frequently suffer from unstable optimization and degraded value estimation. In particular, overly aggressive adversaries can drive the
agent toward uninformative failure states, while
adversarial perturbations amplify disagreement between double critics and introduce biased value targets. We propose a unified framework, RACER (Risk-sensitive robust
Adversarial critic ConsistEncy-regularized Reinforcement learning), that revisits
adversarial RL from a risk-sensitive perspective. First, we introduce a state-dependent
adversarial objective that adaptively regulates perturbation strength, suppressing harmful disturbances while preserving informative exploration. Second, we propose critic consistency regularization to reduce disagreement between Q-value
estimators and stabilize learning. Comprehensive experiments on challenging continuous control
benchmarks demonstrate that RACER consistently improves performance, robustness, and training stability over strong robust RL baselines.
Read the full abstract ▾
All authors Jiaxi Wu, Tiantian Zhang, Yuxing Wang, Yongzhe Chang, Xueqian Wang
Machine Learning
Private Decentralized Optimization with Noise Reduction and Bias Correction
Yizhao Fan, Wenjian Luo and Jiaojiao Zhang
Private decentralized learning is affected by sampling noise,
privacy noise, and decentralized bias under heterogeneous data. We propose Private Recursive Decentralized Optimization (PRDO). PRDO uses recursive estimation with same-batch gradient differences to reduce estimation errors caused by sampling and
privacy noise, while its Exact
Diffusion component corrects decentralized bias arising from data heterogeneity. Our analysis establishes a nonconvex convergence bound without assuming uniformly bounded data heterogeneity across nodes. It further gives a sufficient condition under which recursive gradient differences yield strictly lower query sensitivity than private Exact
Diffusion, together with an example that rigorously satisfies this condition. Experiments show improved accuracy over the evaluated baselines.
Read the full abstract ▾
Machine Learning
FLEET: From Logits Entropy to Enhanced Trajectories in Text Generation
Oleksii Streltsov and Oleksandra Vitko
Solutions based on large
language models (LLMs) often rely on temperature sampling to improve accuracy and stability by aggregating multiple samples from the completion distribution. However, this memoryless approach is inherently suboptimal: because it lacks awareness of prior generations and their evaluations, it produces an increasing proportion of semantically duplicate answers as more samples are drawn, leading to diminishing returns. To address this limitation, we introduce FLEET, a novel method that integrates a
memory mechanism into the generation process. FLEET represents each generation as a sparse trajectory through states whose
entropy exceeds a predefined threshold and uses these trajectories to infer per-
token utility scores that adjust the logits.
Benchmark evaluations demonstrate that FLEET achieves the same accuracy as the repeated sampling baseline, with a 3x speedup, and substantially improves accuracy on complex coding tasks (LiveCodeBench Pass@32 increases from 59.9% to 66.2%) under the same budget. Furthermore, in the greedy-decoding configuration evaluated here, the approach is deterministic and uses a single calibration pass to derive its principal hyperparameters, requiring only minimal modifications to existing LLM pipelines.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
FedIncome: Federated Learning for Income Estimation in Digital Lending Under Data Sovereignty Constraints
Sultan Amed, Tanmay Sen and Sayantan Banerjee
Verified income is often unavailable in digital loan applications, forcing lenders to rely on reported income and potentially leading to over-lending, overly conservative offers, or rejection of creditworthy applicants. Cross-institutional data-sharing constraints make this problem especially difficult for smaller lenders with limited training data. We introduce FedIncome, a
federated learning framework for income estimation that enables institutions to train a shared model without pooling raw borrower records. Using more than one million LendingClub loans partitioned into $50$ state-level clients, we simulate a heterogeneous lending consortium. The best
federated model achieves out-of-time $R^2=0.608$, compared with $0.619$ for a pooled centralised
benchmark. Small-sample clients obtain an average out-of-time $R^2$ improvement of $3.8$ percentage points relative to the pooled centralised
benchmark, while the fitted client-level relationship places the empirical crossover at approximately $4,790$ training observations in this setting. When pooling is infeasible and the relevant alternative is local-only training, federation improves out-of-time performance across all sample-size groups, with the largest gains for data-scarce clients. We also combine
federated income estimates with state- and income-specific debt-to-income thresholds. In a retrospective decision analysis, replacing reported income with the
federated estimate increases simulated approval rates with only modest changes in observed default rates. FedIncome supports collaborative learning under data-locality constraints with little aggregate loss relative to pooled training and larger gains relative to local-only estimation.
Read the full abstract ▾