Machine Learning
RCAP: Robust, Class-Aware, Probabilistic Dynamic Dataset Pruning
Atif Hassan, Swanand Khare and Jiaul H. Paik
Dynamic data
pruning techniques aim to reduce computational cost while minimizing information loss by periodically selecting representative subsets of input data during model training. However, existing methods often struggle to maintain strong worst-group accuracy, particularly at high
pruning rates, across balanced and imbalanced
datasets. To address this challenge, we propose RCAP, a Robust, Class-Aware, Probabilistic dynamic
dataset pruning algorithm for classification tasks. RCAP applies a closed-form solution to estimate the fraction of samples to be included in the training subset for each individual class. This fraction is adaptively adjusted in every epoch using class-wise aggregated loss. Thereafter, it employs an adaptive sampling strategy that prioritizes samples having high loss for populating the class-wise subsets. We evaluate RCAP on six diverse
datasets ranging from class-balanced to highly imbalanced using five distinct models across three training paradigms: training from scratch, transfer learning, and fine-tuning. Our approach consistently outperforms state-of-the-art
dataset pruning methods, achieving superior worst-group accuracy at all
pruning rates. Remarkably, with only $10\%$ data, RCAP delivers $>1\%$ improvement in performance on class-imbalanced
datasets compared to full data training while providing an average $8.69\times$ speedup. The code can be accessed at https://github.com/atif-hassan/RCAP-dynamic-
dataset-
pruning
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
TacCoRL: Integrating Tactile Feedback into VLA via Simulation
Siyu Ma, Yuqi Liang, Chang Yu et al. (+5 more)
Vision-
language-action (VLA) models provide strong visual,
language, and action priors for
robot manipulation, but visual observations alone often miss the local contact state required for contact-rich tasks. We present TacCoRL, a scalable framework that injects Tactile feedback into VLA policies and improves them through sim-real Co-training and simulation-based reinforcement learning (RL), without requiring large-scale tactile pretraining or extensive real-world contact exploration. The key idea is not only adding touch as an input, but learning how contact readings should modulate action responses in near-failure states that are rare in demonstrations and risky to collect on hardware. We use a real-aligned simulator as a closed-loop training environment for contact interaction. Mixed simulated and real trajectories first warm-start tactile-conditioned actions in the pretrained policy. Reinforcement learning with verifiable task rewards then optimizes the policy using simulated contact rollouts. It reinforces tactile-conditioned actions that lead to task completion, while a supervised objective on real trajectories keeps the refined policy anchored to deployment visual, tactile, and action distributions. The resulting policy transfers directly to the real
robot without privileged simulation state or online real-world RL. Across four bimanual contact-rich tasks, the final visuo-tactile policy achieves an average success rate of 72.5%, compared to baseline of 50.0%. Result
videos and more details are available at https://tac-corl.github.io/
Read the full abstract ▾
All authors Siyu Ma, Yuqi Liang, Chang Yu, Yunuo Chen, Hao Su, Yixin Zhu, Yin Yang, Chenfanfu Jiang
Machine Learning · Cross-listed from stat.ML
Renewable Lasso without Batch-Number Constraints: A Gradient-Enhanced Approach
Junzhuo Gao, Ling Peng, Xu Guo et al. (+1 more)
We study online estimation for high-dimensional generalized linear models with streaming data. First, for the non-distributed setting, we propose a gradient-enhanced surrogate loss that approximates the cumulative loss using only historical summaries, which modifies and improves upon the existing renewable estimation approach for the same model in the high-dimensional setting, and removes the batch-number constraint in previous studies. We then extend the method to distributed streaming data under the master-client architecture, where batches are partitioned across sites and only summaries (gradient vectors) are exchanged. Instead of directing applying the popular method of Jordan et al. (2019) to the surrogate quadratic loss, our adjusted approach does not require the clients to compute the full surrogate loss. We derive non-asymptotic error bounds under the high-dimensional scaling, without the stringent constraint on the number of batches in the previous studies. Simulation results under linear and logistic models, together with a real-data application, show improved accuracy over existing renewable
estimators.
Read the full abstract ▾
All authors Junzhuo Gao, Ling Peng, Xu Guo, Heng Lian
Machine Learning · Cross-listed from astro-ph.EP
Machine-learning clustering of close-in exoplanet populations: links to pebble accretion
Yi Duann, Anders Johansen, Haiyang S. Wang et al. (+1 more)
Close-in
exoplanets exhibit a wide range of orbital architectures and physical properties shaped by both formation conditions and migration processes. Although population-synthesis models predict distinct planetary populations, establishing a quantitative connection between observed
exoplanets and synthetic populations remains challenging. We investigate the intrinsic organisation of close-in
exoplanets using physically motivated dynamical parameters and connect the resulting populations to pebble-
accretion formation pathways. A two-stage Gaussian mixture model (GMM) is applied to an observed sample of close-in
exoplanets, performing unsupervised probabilistic clustering in a feature space dominated by dynamical descriptors of
planet-
star interactions. The resulting clusters are mapped onto a pebble-
accretion synthetic population within a statistically motivated three-dimensional parameter space. Formation-related quantities, including gas availability, gas fraction, and ice-rock mass ratio, are then used to interpret the mapped populations. We identify statistically supported sub-populations without imposing predefined classification boundaries, including very-massive gas
giants, hot
giants, warm-Jupiter-dominated systems, and lower-mass
giants. The mapped synthetic populations reveal systematic differences in formation timing, gas
accretion, and solid growth histories. In particular, very-massive gas
giants are preferentially associated with earlier formation epochs than hot-
giant and warm-Jupiter-dominated populations. These results demonstrate that physically motivated machine-learning approaches can provide a statistically robust framework for linking observed
exoplanet populations to theoretical
planet formation pathways.
Read the full abstract ▾
All authors Yi Duann, Anders Johansen, Haiyang S. Wang, H. Jens Hoeijmakers
Machine Learning
Non-Gaussianity Reveals Interpretable Directions in Language Models
Sida Liu, Feijiang Han, Sarah Wiegreffe et al. (+1 more)
Finding human-interpretable directions in
language-model activations is important for understanding and controlling model behavior. These directions are usually discovered through indirect learning objectives such as sparse reconstruction, but it remains unclear what existing statistical structure distinguishes interpretable directions and whether that structure can be used directly for feature discovery. We find that non-Gaussianity provides such a signature: projection distributions of natural-text residual-stream activations along sparse autoencoder (SAE) decoder directions are substantially more non-Gaussian than those along random directions, with most contexts forming a concentrated background and recurring patterns often occupying one tail. This raises a natural question: if interpretable directions tend to have non-Gaussian projection distributions, can we discover them by searching for non-Gaussian directions directly in activation space, without training another sparse dictionary? To test this hypothesis, we introduce ICA Lens, a practical workflow for efficient discovery, reliable annotation, and faithful evaluation of non-Gaussian directions in LLM residual-stream activations. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find that: (1) the recovered non-Gaussian directions capture coherent lexical, syntactic, and semantic patterns under both human and LLM-based evaluation; (2) non-Gaussianity is positively associated with both human-label confidence and automated interpretability; and (3) the resulting coordinates are competitive with SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7-43 times less
memory. These results show that non-Gaussianity is a direct unsupervised signal for discovering interpretable and useful structure in activations.
Read the full abstract ▾
All authors Sida Liu, Feijiang Han, Sarah Wiegreffe, Furong Huang
Machine Learning · Cross-listed from cs.CL
Substrate Asymmetry in User-Side Memory: A Diagnostic Framework
Youwang Deng
User-side
memory in LLMs is typically scored as a single "personalization" capability: given a user's history, is the output more user-aware? We show this aggregate metric hides opposite-direction failures.
Memory factorises into at least three orthogonal axes – behavioral consistency (style, voice), factual presence (recall facts in history), and factual absence (abstain when a fact is absent) – and no single substrate wins all three. Comparing per-user gamma-LoRA (a small LoRA adapter trained on each user's history; gamma denotes per-user, not per-task) against BGE-large dense top-K
retrieval on a controlled 50-user synthetic corpus and a real-data probe (LaMP-3), we find gamma-LoRA decisively wins behavioral style while RAG decisively wins factual absence – and the same query-projection
cells in
attention layers 21-35 causally load-bear both effects in opposite directions (zeroing those LoRA weights raises absence-probe TPR by +33 pp and drops presence-probe TPR by 20 pp). On the more heavily RLHF-tuned Llama-3.1-8B-Instruct the asymmetry strengthens, not heals: parametric
memory's behavioral advantage collapses while its absence-calibration deficit against
retrieval widens – an
alignment tax on parametric user-
memory. On real-data LaMP-3, gamma-LoRA underperforms a majority baseline; a 9-condition mitigation sweep diagnoses this as instruction-following collapse, not substrate failure (a 9x2 cross-product shows the eval-time {1..5} logit mask drives main_acc to >=0.995 on every recipe), and the best training-time fix replicates bit-identically on Llama. Finally, substrate-selection routing is question-classification, not calibration: a 110M DistilBERT on the question text alone beats every logit-based router. We contribute the diagnostic framework, the diagnosed real-data negative, the
alignment-tax replication, and the routing-as-classification finding.
Read the full abstract ▾
Machine Learning
Capacity-Constrained Online Convex Optimization with Delayed Feedback
Alexander Ryabchenko, Idan Attias and Daniel M. Roy
Online learning with delayed feedback typically assumes that the learner can track all pending rounds until their feedback arrives. In practice, tracking resources are finite, and feedback from untracked rounds is permanently lost. In this paper, we study delayed online convex optimization (OCO) under a hard capacity constraint, where at most $C$ pending rounds can be tracked at any time. To model delay information, we introduce a semi-clairvoyant model that refines the clairvoyant assumption from prior work: rather than requiring delays to be known at prediction time, the learner observes delay expirations online, consistent with the classical unconstrained delayed setting. Our approach proceeds via a reduction to a novel “delayed and weighted” OCO problem, using a scheduler that randomizes tracking decisions and importance-weights the resulting observations. For this base problem, we propose and analyze Delayed-Weighted FTRL and its bandit analogue, establishing regret bounds that explicitly characterize the interaction between time-varying weights and delayed feedback. Combining these base learners with our schedulers yields the first regret guarantees for capacity-constrained OCO under convex and strongly convex losses, for both first-order and bandit feedback. For first-order feedback, capacity $C = \Omega(\log T)$ suffices to recover standard delayed OCO rates up to logarithmic factors. For bandit feedback, the regret rates are modulated by powers of $(1 + \sigma_{\text{max}}/C)$, where $\sigma_{\text{max}}$ is the maximum number of pending observations at any time. This allows the regret bound to degrade gracefully when $C < \sigma_{\text{max}}$, while remaining sublinear.
Read the full abstract ▾
Machine Learning
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
Leyi Pan, Shuchang Tao, Yunpeng Zhai et al. (+5 more)
On-policy self-
distillation (OPSD) provides dense,
token-level supervision for
reasoning models by aligning a model's own distribution with that under privileged context, typically a verified solution. However, we show that the resulting distributional gap concentrates on style
tokens rather than task-bearing ones, as the hinted model tends to produce shorter, more direct outputs. We term this pathology privilege-induced style drift, which can destabilize training and shorten responses. To address this, we propose RLCSD (Reinforcement Learning with Contrastive on-policy Self-
Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing style shifts induced by hints regardless of correctness and yielding a signal more concentrated on task-bearing
tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical
reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods, with additional results on agentic tasks supporting broader applicability. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to broader cross-model on-policy
distillation.
Read the full abstract ▾
All authors Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, Lijie Wen
Machine Learning
A Data-Centric Framework for Detecting and Correcting Corrupted Labels
Ha-Linh Nguyen, Hong-Anh Nguyen, Minh-Duc La et al. (+3 more)
The performance of machine learning and deep learning models largely depends on the quality of the training data. However, the quality of the real-world
datasets is often compromised by noisy labels, which can substantially degrade model accuracy and reliability. To address this challenge, we propose Relabeler, an end-to-end data-centric framework for detecting and correcting corrupted labels. For corrupted label detection, Relabeler jointly leverages both local and global relationships among data instances to identify potentially noisy samples. After detecting suspicious instances, Relabeler further performs label correction by estimating the most probable clean label for each instance based on both its input features and observed noisy label. Extensive experiments across multiple
datasets, noise types, and noise rates demonstrate that Relabeler consistently outperforms state-of-the-art baselines, achieving up to 58% improvement in label correction precision and 6% improvement in downstream task performance.
Read the full abstract ▾
All authors Ha-Linh Nguyen, Hong-Anh Nguyen, Minh-Duc La, Thu-Trang Nguyen, Son Nguyen, Hieu Dinh Vo
Machine Learning
Noise-Aware Framework for Correcting Corrupted Labels
Ha-Linh Nguyen, Hong-Anh Nguyen, Minh-Duc La et al. (+4 more)
High-quality labeled data is essential for training reliable ML/DL models. However, real-world
datasets often contain a considerable proportion of corrupted labels, which can severely degrade model performance. To address this problem, we propose CANOLA, a novel framework for correcting corrupted labels through noise-aware learning and iterative label refinement. CANOLA explicitly estimates the underlying noise distribution of the
dataset and incorporates this information into the training of a noise-aware Deep Neural Network. By incorporating noise characteristics during learning, CANOLA enables the model to down-weight unreliable supervision signals and focus on trustworthy patterns, thereby improving robustness and generalization. Label correction is performed via cautious, iterative soft label refinement, in which model predictions are blended with observed labels to prevent premature or erroneous updates. This progressive refinement allows the
dataset to be repaired in a stable and controlled manner. We evaluate CANOLA on six widely used
datasets under realistic noisy labeling scenarios. Experimental results show that CANOLA consistently outperforms SOTA label correction methods, achieving relative improvements ranging from 19% to 52% in error reduction. Moreover, models trained on
datasets corrected by CANOLA obtain substantial downstream performance gains. Even simple classifiers trained on CANOLA's corrected data can outperform complex model-centric approaches by margins of up to 67%.
Read the full abstract ▾
All authors Ha-Linh Nguyen, Hong-Anh Nguyen, Minh-Duc La, Phong Lam, Thu-Trang Nguyen, Son Nguyen, Hieu Dinh Vo
Machine Learning
Spectrally Regularized Latent Flow Matching for Turbulence Generation
Khalid Rafiq and Aditya G. Nair
Latent
diffusion and flow matching have emerged as leading approaches for synthetic
turbulence generation, yet they systematically under-represent dissipation-range amplitudes. We introduce a latent flow matching framework with a spectrally regularized compression stage that directly targets this failure mode. On a 256^2 DNS
dataset at Re_f \approx 2250, replacing an MSE-trained VAE with a zone-weighted log-spectral objective raises deep-dissipation retained spectral power from 25% to 94% in reconstruction and from 20% to 79% in unconditional generation. The improved latent representation also yields a substantially better sampling cost-fidelity tradeoff: the MSE-trained latent space imposes a fundamental quality ceiling near DD bias -0.70 that no integrator or step-count can overcome, while the spectrally regularized latent space reaches DD bias -0.117 at just 20 function evaluations. Mechanistically, encoder-decoder swap experiments show that the improvement is driven primarily by encoder-induced latent reorganization rather than decoder capacity, while a support-amplitude decomposition reveals that MSE-trained models behave as conservative suppression models, minimizing pointwise error by attenuating intermittent high-wavenumber structure. Both pipelines recover the second-order structure function and the correct sign of S_3, indicating the correct cascade direction without explicit supervision. A small residual gap in the magnitude of S_3 suggests that phase-coherent triadic organization remains a complementary axis to amplitude fidelity for future generative
turbulence models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
DroneShield-AI: A Multi-Modal Sensor Fusion Framework for Real-Time Autonomous Drone Threat Detection, Behavioral Intent Classification, and Swarm Intelligence in Contested Airspace
Marius Bayizere
Unmanned Aerial Vehicle (
UAV) threats have emerged as a defining security challenge of the 21st century. This paper presents DroneShield-AI, a unified open framework integrating six processing layers: RF signal classification, acoustic motor-signature detection, YOLOv8-based visual detection, evidence-weighted sensor fusion, a Behavioral Intent Classification Engine (BICE), and a Graph Neural Network Swarm Intelligence Module (GNN-SIM). This v2 revision reports measured results on the completed implementation (495 automated tests), superseding the v1 simulation-only preprint. On real public data: RF presence detection reaches F1 0.9924; acoustic detection reaches 98.12% accuracy, though a 4-feature statistical baseline reaches 93.25% on the same split, so the model's drone-specific contribution is approximately 4.9 points, a property of
dataset provenance rather than a leakage bug; visual detection (YOLOv8) reaches AP50 0.8782 on a held-out test split, from a model still improving when training stopped. No fused-accuracy figure is reportable, since the three real-data layers use independently collected, unsynchronized
datasets with no shared ground truth. BICE and GNN-SIM remain evaluated on physics-calibrated simulation data pending real
adversarial and swarm field data: BICE shows a consistent robustness advantage over a trivial baseline under distribution shift; GNN-SIM underperforms its own trivial baseline in-distribution but independently replicates BICE's noise-robustness advantage, regarded as this work's most durable finding. All code, trained checkpoints, and the full test suite are publicly released.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Parameter-Efficient Adapter Tuning for Tabular-Image Multimodal Learning
Jiaqi Luo
Tabular-image
multimodal learning aims to improve predictive modeling by jointly using structured tabular attributes and visual data. Although pretrained encoders provide strong modality-specific representations, full fine-tuning can be computationally expensive, while keeping encoders frozen may limit task-specific adaptation. We propose the Tabular-Image Adapter (TI-Adapter), a modality-specific adapter-based fine-tuning framework for efficient
multimodal adaptation. TI-Adapter freezes the pretrained tabular encoder and learns an adapter after the extracted tabular
embedding, while adapting the image branch with
embedding-level and bottleneck-level adapters instead of full fine-tuning. Experiments on 20 tabular-image
datasets show that TI-Adapter achieves competitive or better predictive performance than full fine-tuning while using substantially fewer trainable parameters. Ablation studies further demonstrate the importance of adapter placement for balancing performance and practical efficiency.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents
Hao-Lun Hsu, Nikki Lijing Kuang, Boyi Liu et al. (+2 more)
Large
language model (LLM)
agents struggle with long-horizon tasks due to their inherent statelessness, requiring all task-relevant information to be encoded in growing input contexts. The resulting degraded
reasoning quality, increased inference cost, and higher latency necessitate efficient working
memory mechanisms. However, existing approaches either rely on lossy compression or similarity-based
retrieval, which often fail to capture temporal structure and causal dependencies required for multi-step agentic tasks. In this work, we present HORMA, a Hierarchical Organize-and-Retrieve
Memory Agent that organizes experience into a file-system-like hierarchical structure, where summarized entities are linked to the corresponding raw trajectories, enabling efficient access without losing detailed information. HORMA decomposes working
memory into two stages: structured
memory construction and
navigation-based
retrieval. The construction module iteratively refines how experiences are structured by distinguishing between failures caused by missing information and those caused by misleading or overloaded context. The
navigation module retrieves task-relevant context by traversing the hierarchy using a lightweight
agent trained with reinforcement learning to select minimal yet sufficient context, thereby reducing latency along the critical execution path. Across ALFWorld, LoCoMo, and LongMemEval, HORMA improves task performance under constrained context budgets while requiring at most 22.17% of the baseline
token usage in long conversation tasks. Compared to existing methods, it consistently achieves better efficiency-performance trade-offs and generalizes effectively to unseen tasks.
Read the full abstract ▾
All authors Hao-Lun Hsu, Nikki Lijing Kuang, Boyi Liu, Zhewei Yao, Yuxiong He
Machine Learning · Cross-listed from cs.CE
Neural-Parameterized Cellular Automata for Wildfire Spread
Maksym Zhenirovskyy, Ion Matei, Rohit Vuppala et al. (+2 more)
Traditional wildfire models rely on rigid, low-dimensional parameters and static fuel maps, frequently underpredicting fire spread. To address this weakness, we introduce a hybrid deep-learning parameterized Probabilistic Cellular Automata (CA) framework implemented in JAX. Our approach employs a Multi-Scale Convolutional Neural Network to dynamically generate spatially varying parameters that govern fire-spread probability, wind
alignment, and slope influence. This hybrid design captures complex, nonlinear environmental interactions while preserving the physical interpretability of the underlying three-state CA. The JAX implementation enables hardware acceleration and gradient-based parameter calibration. Evaluated on six large-scale wildfires in the western United States, the model maintains IoU > 0.6 over 72-hour forecast horizons after a 10-day data assimilation window during which the model is fitted incrementally to observed perimeters; the resulting forecast is a conditional projection of fire growth under the suppression regime already ncoded in those observations.
Read the full abstract ▾
All authors Maksym Zhenirovskyy, Ion Matei, Rohit Vuppala, Takuya Kurihana, Hon Yung Wonga
Machine Learning · Cross-listed from cs.SD
SpAArSIST: Sparsified AASIST for Efficient and Reliable Anti-Spoofing
Anton Firc, Vojtěch Staněk, Zbyněk Lička et al. (+2 more)
We present SpAArSIST, a deployment-oriented refinement of the widely used AASIST graph pooling backend for self-supervised learning (SSL) based anti-spoofing. Motivated by redundant operations in public implementations, we replace learned pooling and stack-node
attention with explicit, lightweight choices: separate train and inference graph pooling ratios $(k_{\mathrm{tr}},k_{\mathrm{inf}})$, magnitude-based node scoring, and mean aggregation of graph nodes. The best overall configuration (rank 1) cuts backend compute by 20.7% (195.045M $\rightarrow$ 154.706M MACs) and model size by 4.1% (611.8k $\rightarrow$ 586.4k params), while improving out-of-domain robustness on In-the-Wild to 2.82% EER and 0.078 minDCF (from 4.64% and 0.133) and remaining competitive on ASVspoof5. We further provide a composite selection score that summarizes accuracy, calibration, and compute to support balanced deployment-oriented model choice.
Read the full abstract ▾
All authors Anton Firc, Vojtěch Staněk, Zbyněk Lička, Kamil Malinka, Martin Perešíni
Machine Learning · Cross-listed from quant-ph
Jian Xu, Chao Li, Delu Zeng et al. (+2 more)
Standard dot-product self-
attention computes, in a single layer, only pairwise (order-2) interactions between
tokens; representing a generic order-$k$ interaction is known to require either super-quadratic resources in one layer or composition across depth. We introduce
Quantum Higher-Order
Attention (QHA), a shallow, hardware-realizable
quantum attention head that, via data re-uploading and an all-to-all non-Clifford entangler, synthesizes order-$k$
token interactions inside the
circuit and exposes them through a local single-
qubit read-out. We prove (i) an expressivity separation: any single standard self-
attention layer with
embedding dimension $m$, $H$ heads and $p$-bit precision satisfying $mHp=o(N/\log\log N)$ cannot represent the order-$k$ correlation family that one QHA head represents with
circuit depth $O(\log k)$ ($O(k)$ two-
qubit gates); and (ii) a trainability guarantee for its local-design instantiation: with a local read-out and $O(\log n)$ depth the gradient variance is $\Omega(1/\mathrm{poly}(n))$ (no barren plateau), which we confirm empirically – while being explicit that the more expressive all-to-all instantiation we
benchmark is trained empirically and shows exponentially decaying gradients. Empirically, at a $6.5\times$ smaller parameter budget, QHA generalizes hidden-subset parity of every order $k\le6$ from disjoint inputs, whereas the larger classical
attention head collapses past order~2; consistent with theory, the size of the advantage tracks the target's
Fourier degree - largest for parity and shrinking when low-order structure is present. As an application, QHA serves as a compact high-order interaction detector across three domains - genetic epistasis, learning-parity-with-noise, and graph triangle detection - reaching the noise ceiling at the smallest parameter budget where field-standard linear methods fail.
Read the full abstract ▾
All authors Jian Xu, Chao Li, Delu Zeng, John Paisley, Qibin Zhao
Machine Learning · Cross-listed from cs.SI
Probabilistic Salary Prediction with Graph Attention Networks and a Mixture Density Network
Zhipei Qin, Mohammad Shokri, N. van Weeren et al. (+1 more)
Accurate salary prediction is critical for bridging the information gap between employers and job seekers in modern labor
markets. Existing approaches predominantly yield a single point estimate and treat job attributes such as location, occupation, and industry as independent categorical features, ignoring both the inherent uncertainty and multi-modality of real-world compensation data and the rich hierarchical and semantic-similarity relationships that govern pay norms. In this paper we propose GAT-MDN, a unified framework that addresses both limitations simultaneously. For each of the three attribute domains we construct a domain-specific graph whose edges encode (i) hierarchical parent-child containment and (ii) weighted similarity links derived from a pre-trained Sentence-
Transformer. Parallel Graph
Attention Networks (GATs) with edge-feature-aware
attention learn rich, context-sensitive node representations from these multi-relational graphs. A priority-based hierarchical selection module then assembles a composite feature vector that gracefully handles missing or coarse attributes, and a Mixture Density Network (MDN) head maps this vector to the parameters of a Gaussian Mixture Model (GMM), yielding a full conditional salary distribution. Extensive experiments on a real-world Dutch job-posting
dataset of over 1 million records demonstrate that GAT-MDN significantly outperforms a non-graph MLP-MDN baseline in both Negative Log-Likelihood (NLL) and Mean Squared Error (MSE).
Read the full abstract ▾
All authors Zhipei Qin, Mohammad Shokri, N. van Weeren, F.W. Takes
Machine Learning · Cross-listed from cs.CV
Learning Instance-Adaptive Low-Rank Orthogonal Subspaces for Clothes-Changing Person Re-Identification
Dong-Woo Kim and Tae-Kyun Kim
Clothes-changing person re-identification (CC-ReID) aims to recognize individuals despite drastic appearance changes caused by clothing variation. While existing methods rely on
adversarial learning to disentangle clothing features, we propose Ortho-ReID, which explicitly models a low-rank clothing subspace from VLM text descriptions and extracts clothing-invariant representations via direct
geometric constraints. A critical component is our
transformer-based Basis Maker, which refines a shared, low-dimensional clothing prior into an instance-adaptive low-rank subspace through cross-
attention with image patches, enabling robust clothing feature extraction even under varying visibility conditions. This instance-adaptive subspace is supervised via
alignment with clothing text
embeddings, while identity features are extracted via a learnable projection head and geometrically constrained to be strictly orthogonal to it. Extensive experiments demonstrate state-of-the-art performance on PRCC (+5.9% top-1), Celeb-reID-light (+3.5%), and LaST (+5.3%), with competitive results on LTCC.
Read the full abstract ▾
Machine Learning
Bergson: An Open Source Library for Data Attribution
Lucia Quirke, Louis Jaburi, David Johnston et al. (+6 more)
Data attribution is a promising field in interpretability that aims to explain model behavior through the influence of its training data, with applications including debugging undesirable model behavior and training
dataset curation. However, significant engineering effort is required to perform it at scale, and many cutting edge techniques lack open-source tooling and support. Bergson is an open source library that aims to enable faster progress in the field by providing a host of techniques that scale to very large
language models and pre-training
datasets. The library natively supports on-disk gradient stores and multi-node distributed training, and provides quality of life tools for researchers. Finally, we introduce the first open-source implementations of three leading data attribution methods: MAGIC, SOURCE, and TrackStar. The library is available at https://github.com/EleutherAI/bergson .
Read the full abstract ▾
All authors Lucia Quirke, Louis Jaburi, David Johnston, William Z. Li, Gonçalo Paulo, Guillaume Martres, Girish Gupta, Nora Belrose, Stella Biderman
Machine Learning
Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics
Katherine Rosenfeld and Maike Sonnewald
Generative AI emulators are increasingly used in scientific domains where we already have strong theory,
benchmarks, and physical intuition. This raises a central evaluation and interpretability question: when a foundation-style model can reproduce known continuum dynamics, what internal mechanism supports that behavior, is the internal behaviour consistent with known physics, and how does it relate to where the emulator succeeds or fails? We investigate a cross-domain foundation model for continuum dynamics, Walrus by Polymathic, using mechanistic interpretability guided by physical principles. We apply a sparse autoencoder (SAE) to probe a selected layer, and address the practical challenge of triaging a large feature set (over 20,000) using enstrophy as a physically grounded metric. As a deliberately simple testbed, we focus on shear flow and compare feature recruitment across multiple shear-flow setups, i.e. parameter values in the numerical simulation. Across setups we find evidence of piecewise consistency, with subsets of features recurring in similar roles, but this structure is intermittent and does not map cleanly onto standard physical decompositions. In parallel, direct comparisons between numerical simulation and the emulator reveal systematic output-level discrepancies, including regimes where energy/structures become too diffuse or too localized. We connect parts of these discrepancies to changes in specific SAE feature usage. Our work highlights open questions for scientific foundation models: how to robustly prioritize mechanistically meaningful features, how to separate stable structure from analysis artifacts (including single-layer and SAE limitations), and how to use established
benchmarks to decide when "different" internal representations are genuinely informative rather than merely effective.
Read the full abstract ▾
Machine Learning
IAPO: Input Attribution-Aware Policy Optimization for Tool Use in Small Multimodal Agents
Yifan Yang, Zhen Zhang, Jiayi Tian et al. (+2 more)
This paper investigates reinforcement learning (RL) methods for improving tool-calling capabilities in
multimodal small
language model (SLM)
agents. While existing works have explored various reward designs to improve agentic tool-calling ability, these approaches face inherent limitations for SLM training, especially under
multimodal scenarios. First, many existing methods evaluate tool use correctness through exact matching against certain ground-truth or predefined formats. However, this assumption is often unsuitable for
multimodal tasks, where multiple tool use paths may be valid and annotated tool trajectories are typically unavailable. Second, such sparse and brittle binary rewards provide little guidance on how to improve the underlying decision process, making them particularly difficult for
multimodal SLM to learn from. To address these issues, we propose Input Attribution-Aware Policy Optimization (IAPO), an RL
algorithm for improving tool use in
multimodal SLM by aligning the model's attribution across input components with that of a stronger teacher. Experiments on Qwen2.5-VL-3B show that the proposed method improves visual question answering accuracy by an average of 3% across six test sets compared with existing visual tool use work, by helping the model attend to the most relevant input evidence.
Read the full abstract ▾
All authors Yifan Yang, Zhen Zhang, Jiayi Tian, Liyan Tan, Zheng Zhang
Machine Learning
DeepRHP: A Hybrid Variational Autoencoder for Designing Random Heteropolymers as Protein Mimics
Shuni Li, Zhiyuan Ruan, Andy Shen et al. (+3 more)
Synthetic random heteropolymers (RHPs), consisting of a predefined set of monomers, offer an approach toward the design of
protein-like materials. These RHPs, if designed appropriately, can mimic
protein behavior and function. As such, there is a need for computational tools to efficiently guide RHP design. We bridge this gap by developing DeepRHP, a modified variational autoencoder (VAE) model under a semi-supervised framework. By equipping a classical VAE with an additional feature-based VAE, DeepRHP forces the latent space to capture structures of critical chemical features as well as individual RHP sequence patterns. In this sense, our method is versatile by allowing any relevant features to be incorporated in a hybrid manner. We demonstrate the effectiveness of DeepRHP by suggesting potential monomer compositions that stabilize membrane
proteins (e.g. Aquaporin Z) in non-native environments and cross-validating our prediction with published results. The concordance between our model and true RHP function suggests strong potential in utilizing hybrid autoencoder architectures to guide RHP design for
proteins and other biological compounds.
Read the full abstract ▾
All authors Shuni Li, Zhiyuan Ruan, Andy Shen, Ivan Jayapurna, Ting Xu, Haiyan Huang
Machine Learning
Structure-Preserving Neural Surrogates with Tractable Uncertainty Quantification
Handi Zhang, Adrienne M. Propp, Brooks Kinch et al. (+2 more)
Recent advances in scientific machine learning provide a means of near-real-time solution to partial differential equations (PDEs), but lack the theoretical underpinnings of conventional simulators that support contemporary verification and validation. In this work, we construct data-driven reduced-order models that serve as structure-preserving, real-time surrogates. Remarkably, the exterior calculus that imposes physical conservation structure also exposes topological structure that we use to build a Gaussian process (GP) representation of uncertainty in state-flux relationships, ultimately yielding a Dirichlet-to-Neumann map for quantities of interest with closed-form expressions for posterior uncertainty. We specifically propose structure-preserving $H(\mathrm{div})$–$L^2$ subspaces of conventional Raviart–Thomas and $dgP_0$ elements prescribed by a lightweight
transformer. Reduced-order dynamics consistent with this subspace are learned by posing a conservation law in which a GP describes the fluxes between volumes. This work hinges on a novel interface between mixed FEM spaces and GP regression; when training is posed as the optimal recovery problem (ORP), the resulting GP regression can be written as an optimization problem with equality constraints that impose a conservation structure, amenable to a fast Schur-complement training strategy. The trained model can then be solved in real time with closed-form
estimators for boundary fluxes driven by prescribed Dirichlet data. The paper includes RKHS posterior error bounds for linear functionals to support uncertainty quantification, as well as numerical experiments demonstrating the accuracy of the posterior distribution as a surrogate for error estimation.
Read the full abstract ▾
All authors Handi Zhang, Adrienne M. Propp, Brooks Kinch, Houman Owhadi, Nathaniel Trask
Machine Learning
Tree-Structured Orthonormal Decomposition of the Aitchison Simplex
Daisuke Yamada, Qijun Zhang, Travis Pence et al. (+3 more)
Compositional data – vectors encoding relative proportions – arise across scientific domains, including ecology, geochemistry, and genomics. The features in these data often come with known hierarchical structure (e.g., taxonomies, phylogenies, ontologies), yet existing methods either ignore this structure, discard the intrinsic Aitchison geometry, are designed for binary trees, or yield incomplete coordinate systems. We describe PolyILR, a canonical orthonormal decomposition of the Aitchison tangent space aligned with any tree
topology. Our construction defines a weighted local geometry at each internal node capturing full branching structure, then lifts these to a global orthonormal basis where every coordinate corresponds to a specific tree location. On microbiome and single-
cell benchmarks, PolyILR yields stable, interpretable features and enables inference at multiscale tree resolution. We also establish a novel theoretical connection to softmax classifiers, suggesting possible applications to probabilistic modeling.
Read the full abstract ▾
All authors Daisuke Yamada, Qijun Zhang, Travis Pence, Barbara B. Bendlin, Federico Rey, Vikas Singh
Machine Learning
TAROT: Task-Adaptive Refinement of LLM-prior Graphs for Few-shot Tabular Learning
Ruxue Shi, Yili Wang, Mengnan Du et al. (+3 more)
Few-shot tabular learning provides a cost-effective approach for real-world applications where annotation is costly and collecting sufficient samples for new tasks is difficult. Existing Traditional and LLM-based methods have demonstrated effectiveness in few-shot scenarios. However, traditional methods need additional training on unlabeled or generated data, which incur significant computational overhead. In addition, LLM-based methods that directly feed raw tabular data into LLMs raise
privacy and compliance concerns. More importantly, both paradigms largely overlook the semantic relationships between features, which provide structural and semantic prior for constructing a semantic graph. Semantic graph is essential for modeling meaningful feature interactions in few-shot scenarios. In this paper, we propose TAROT, a GNN-based framework that encodes the structural and semantic prior by constructing and refining a task-adaptive semantic graph from this prior, thereby improving predictive performance in few-shot tabular learning. TAROT first encodes heterogeneous tabular data into unified node semantic representations via a Unified Semantic Tabular Node Encoder (USTNE). Then, it
prompts LLMs to infer the semantic relationship between features based on the task description and feature names to construct a semantic graph. To mitigate structural noise introduced by the
hallucination of LLMs, TAROT introduces Task-adaptive Semantic Graph Refinement that prunes spurious or task-unrelated edges and adds missing task-related ones, aligning the graph structure with the downstream objective. Finally, a GNN performs message passing over the refined graph to capture task-related semantic dependencies for prediction. Extensive experiments on various few-shot tabular learning
benchmarks demonstrate the superior performance of TAROT, establishing it as a state-of-the-art approach in this domain.
Read the full abstract ▾
All authors Ruxue Shi, Yili Wang, Mengnan Du, Hangting Ye, Yi Chang, Xin Wang
Machine Learning · Cross-listed from math.DS
Integral Formulation of QENDy for Robust Nonlinear System Identification
Nikhil Saran, Sushant Pokhriyal, Stefan Klus et al. (+2 more)
This manuscript proposes an integral formulation of the newly defined quadratic
embedding method for identifying nonlinear systems (QENDy). In the original
algorithm, trajectory data points along with their time derivatives are used. Methods for calculating time derivatives make the
algorithm sensitive to noise. Our integral formulation does not use the time derivatives. This results in a more robust method to learn the dynamics.
Read the full abstract ▾
All authors Nikhil Saran, Sushant Pokhriyal, Stefan Klus, Rushikesh Kamalapurkar, Joel A. Rosenfeld
Machine Learning
When Context Returns: Toward Robust Internalization in On-Policy Distillation
Xun Wang, Ruishuo Chen, Zhuoran Li et al. (+2 more)
Recent work has shown that on-policy
distillation can internalize privileged context, such as system
prompts or task hints, into a student model so that the context is no longer needed at inference time. However, we identify a counterintuitive and previously unstudied phenomenon: reintroducing the original privileged context to the distilled student often degrades its performance, even on instances it already solves correctly without context. We term this phenomenon context-induced degradation and argue that robust internalization requires not only matching the teacher's context-conditioned behavior, but also remaining stable when the privileged context is reintroduced, a desirable property we call context invariance. To promote this property, we formulate a novel view-robust internalization risk and propose No-Context Anchoring (NCA), a lightweight yet effective consistency regularizer that uses the student's stop-gradient no-context output as an anchor and aligns its context-conditioned output via forward KL divergence. Across 14 configurations spanning diverse domains and model families, NCA improves context-conditioned accuracy in most settings and reduces context harm in 12 out of 14, while preserving or improving no-context performance, demonstrating greater robustness to context reintroduction.
Read the full abstract ▾
All authors Xun Wang, Ruishuo Chen, Zhuoran Li, Yu Chen, Longbo Huang
Machine Learning
TimeRouter: Efficient and Adaptive Routing of Time-Series Foundation Models
Kanghui Ning, Yushan Jiang, Kashif Rasul et al. (+3 more)
Time-series foundation models (TSFMs) are increasingly explored as predictive experts within emerging agentic time-series systems. However, TSFMs exhibit heterogeneous inductive biases, and no single model consistently dominates across
forecasting regimes, making expert selection a critical challenge. Existing systems often delegate this decision to LLM-based controllers, incurring substantial inference overhead. We present TimeRouter, an efficient routing framework that leverages empirical complementarity across a pool of pretrained TSFMs through lightweight discriminative routing, selective gating, and ensemble fallback. Concretely, TimeRouter combines a learned routing head, a selective gate, and an ensemble fallback, enabling adaptive expert selection without invoking an LLM at inference time. TimeRouter achieves state-of-the-art performance on the GIFT-EVAL leaderboard, with an LB MASE of 0.6765. Beyond
benchmark performance, our ablation studies provide empirical insights into TSFM routing design, highlighting the importance of pool composition and selective gating. Taken together, these results position TimeRouter as a modular and lightweight routing layer for future agentic time-series systems built upon foundation-model pools. Our code is available at https://github.com/UConn-DSIS/TimeRouter.
Read the full abstract ▾
All authors Kanghui Ning, Yushan Jiang, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, Dongjin Song
Machine Learning · Cross-listed from quant-ph
Family-Aware Residual Architecture for Predicting Quantum Circuit Simulation Performance
Honjar Xing, Yehong Jiang, Xianbang Wang et al. (+2 more)
Approximate tensor-network simulators enable classical simulation of
quantum circuits beyond the reach of exact methods, but selecting optimal approximation parameters – such as bond dimension thresholds – remains a costly trial-and-error process. We present a family-aware neural architecture that predicts both the minimum approximation threshold required to achieve target fidelity and the expected wall-clock runtime for
quantum circuit simulation, given only the
circuit's OpenQASM description and execution context. Our key insight is that
quantum circuits from different algorithmic families (e.g., QFT, Grover, VQE) exhibit fundamentally distinct simulation cost profiles due to their differing
entanglement structures. We employ family-conditioned residual corrections – additive, family-specific adjustments atop a shared backbone, drawing on established conditional computation techniques – enabling the model to capture both universal
circuit properties and algorithmic nuances. The architecture incorporates a pretrained family classifier (97.5% accuracy) and domain-informed
algorithm fingerprint features derived from gate-composition heuristics. Evaluated on
circuits spanning 7–130
qubits across 10
algorithm families, our system achieves 79.5% exact threshold accuracy (91.2% within one rung) and $R^2 = 0.82$ runtime correlation, with inference completing in approximately 50 ms – replacing trial-and-error simulation runs that may take minutes to hours. Ablation studies confirm that family-aware modeling provides the single largest performance improvement (+3.2 percentage points), validating the hypothesis that
algorithm family is a first-class feature for simulation cost prediction.
Read the full abstract ▾
All authors Honjar Xing, Yehong Jiang, Xianbang Wang, Zehua Wang, Zhicheng Jiang