Machine Learning
Measurement noise limits the advantage of nonlinear models over linear models in biomedical prediction
Marc-Andre Schulz and Kerstin Ritter
On biomedical tabular data, flexible models such as deep networks, gradient-boosted trees, and kernel methods are repeatedly matched or beaten by linear and logistic regression given the same features. The usual reaction is to treat this as a model-side shortfall, to be fixed with more data, a better architecture, or tuning, on the assumption that the nonlinear structure is there and the model has failed to capture it. We argue that these fixes cannot help when the binding limit is the measurement rather than the model, as it frequently is in biomedicine. Additive noise blurs the population-optimal predictor, and because blurring removes a function's fine, rapidly varying detail before its broad shape, it erases nonlinear structure faster than linear structure. A degree-$k$ interaction is attenuated by the $k$-th power of feature reliability, while the linear part is attenuated only once. At the reliabilities typical of biomedical measurement, the nonlinear advantage can vanish even when the underlying biology is strongly nonlinear, and what the noise removes cannot be recovered by a larger cohort or a more flexible model, only by better measurement. The nonlinearity is hidden, not absent, and a tie between linear and flexible models is not by itself a verdict on the biology. These pieces are classical, drawn from measurement-error statistics, psychometrics, and Gaussian analysis, and we assemble them into an exact excess-risk identity. Measurement reliability is one of three conditions, alongside sample size and feature representation, that must align for a flexible model to help, and together they leave only a narrow window that most biomedical tasks fall outside. Across 140 UK Biobank tasks, the gap between flexible and linear models, where it exists, carries the predicted noise signature, and the three conditions can be separated by intervention but not by a
benchmark alone.
Read the full abstract ▾
Machine Learning
P$^2$CE: Model-Agnostic Plausible Pareto-Optimal Counterfactual Explanations
Arthur Hendricks Mendes de Oliveira, Giovani Valdrighi and Marcos Medeiros Raimundo
The increasing use of machine learning
algorithms in social applications has raised concerns about
fairness and transparency, leading to the development of counterfactual explanations. These explanations supports individuals to understand and potentially alter unfavorable decisions in areas such as loan applications, job selections, and more, by providing actionable changes to input features that would lead to a desired outcome. Existing methods often struggle to balance feasibility, plausibility, and computational efficiency. To address this, we introduce P$^2$CE, an
algorithm for generating plausible Pareto-optimal counterfactual explanations, offering users a diverse set of optimal trade-offs between different notions of feasibility. P$^2$CE employs an auxiliary isolation forest outlier detector to ensure that explanations are in accordance with the data distribution and leverages SHAP values to obtain optimal results with short computing times, regardless of the underlying model. Our
algorithm was empirically evaluated on three
datasets, demonstrating superior performance in terms of both solution quality and computational efficiency compared to related techniques.
Read the full abstract ▾
Machine Learning
MOLAR: Learning Multimodal Molecular Representations from Noisy Labels
Yingxu Wang, Kunyu Zhang, Nan Yin et al. (+2 more)
Motivation: Noisy labels are a common challenge in molecular property prediction because molecular annotations are often obtained from assays, curated databases, or weak annotation pipelines rather than directly observed clean biological states. Treating recorded labels as reliable supervision can cause models to memorize corrupted observations and learn misleading molecular evidence. In
multimodal molecular representation learning, this issue can be amplified by graph-text fusion or
alignment, which may propagate label-induced errors across modalities. Results: We propose MOLAR, a noise-aware framework for learning
multimodal molecular representations from noisy labels. MOLAR separates latent clean-property inference from recorded-label observation: graph and text views contribute residual evidence to a clean-property distribution, and a categorical label-observation channel maps this distribution to recorded labels for training. This formulation derives posterior label reliability and modality-specific molecular evidence from the model. Experiments on naturally noisy molecular
benchmarks and controlled label-flipping
benchmarks show that MOLAR consistently outperforms representative baselines. Visualization analyses further show that MOLAR provides interpretable reliability and modality-evidence diagnostics.
Read the full abstract ▾
All authors Yingxu Wang, Kunyu Zhang, Nan Yin, Yu Li, Eran Segal
Machine Learning
LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents
Haoyang Fang, Wei Zhu, Boran Han et al. (+11 more)
RL post-training strategies are
dataset-dependent and reveal a recurring empirical pattern: capacity parameters accumulate monotonically across stages, while regularization parameters predominantly oscillate in response to shifting training dynamics. This distinction highlights a potential flaw in fixed training schedules: by forcing all parameters along rigid paths, they fail to capture the dynamic exploration-exploitation tradeoffs that regularization must track. We uncover this through LLMZero, an agentic system that optimizes training trajectories via tree search by diagnosing pathologies at each checkpoint and proposing coordinated multi-parameter transitions. Across four diverse GRPO tasks, LLMZero discovers strategies that improve over the base model by 9% to 140% and over grid search by 6% to 15% (relative), consistently outperforming random search and a skill-based
agent under a matched compute budget. The capacity–regularization asymmetry is consistent across all four tasks, offering a candidate design heuristic for multi-stage training.
Read the full abstract ▾
All authors Haoyang Fang, Wei Zhu, Boran Han, Alex Zhang, Zhenyu Pan, Shuo Yang, Shuai Zhang, Jiading Gai, Peng Tang, Cuixiong Hu, Xuan Zhu, Huzefa Rangwala, George Karypis, Bernie Wang
Machine Learning
SCOPE-FL: A Strategy-proof Chain-based Optimal pareto efficient Federated Learning System
Seyed Salar Ghazi, Kaiwen Zhang, Mehdi feizi et al. (+1 more)
Hierarchical
Federated Learning (HFL) enables scalable collaborative model training across distributed devices while preserving data
privacy. However, existing HFL client selection mechanisms suffer from a fundamental strategic inefficiency. By prioritizing stability over Pareto efficiency (PE), they produce suboptimal resource allocations, and without strategy proofness (SP), participants are incentivized to misrepresent their true preferences, both failures degrading system overall welfare in the Pareto sense in practice. To address it, we propose SCOPE-FL (Strategy-proof Chain-based Optimal pareto efficient
Federated Learning), a synchronous HFL framework that formulates client selection as a two-sided school choice problem solved through the Top Trading Cycle (TTC)
algorithm that simultaneously guarantees PE and SP. For reward distribution, SCOPE-FL employs a scalable Shapley value approximation based on One-Round Reconstruction (OR), ensuring compensation proportional to each client's contribution. The entire mechanism executes via blockchain smart contracts, providing the tamper-proof environment required for the SP guarantees to hold in practice. A comprehensive evaluation on MNIST, Fashion-MNIST, and CIFAR-10 demonstrates that SCOPE-FL outperforms state-of-the-art approaches, including DA, IAS, and other methods across model accuracy, convergence rate, and reward efficiency, while achieving communication latency comparable to DA and blockchain overhead significantly lower than DA at scale.
Read the full abstract ▾
All authors Seyed Salar Ghazi, Kaiwen Zhang, Mehdi feizi, Hans-Arno Jacobsen
Machine Learning
When Can We Trust the Sparse Lens? A Certification Framework for SAE Faithfulness
Dibyanayan Bandyopadhyay and Asif Ekbal
Sparse autoencoders (SAEs) are increasingly used to study internal representations in
language models (LMs), but it remains unclear when an SAE-based representation reliably preserves the underlying model's predictive behavior. We introduce a post-hoc certification framework for this purpose. At a chosen layer, we replace the model's hidden activation with its pretrained SAE reconstruction and derive a bound on the expected loss of the original frozen LM through this SAE-based proxy. The resulting certificate depends on three quantities measured on held-out data: the proxy risk, the reconstruction loss gap, and support mismatch between the calibration feature pool and new inputs. We further derive a more conservative exact-$P$ extension whose proxy-risk concentration term is uniform over all feature pools of the same size. Empirically, the certificates become non-vacuous for GPT-2 Small, Gemma-2B, and Llama-3-8B. A detailed layerwise study of Llama-3-8B shows that later layers are easier to certify, driven mainly by improved reconstruction fidelity and weaker downstream amplification of reconstruction error. GPT-2 Small shows much weaker layer dependence, indicating that this pattern is not universal across model-SAE pairs. Overall, the framework provides a practical way to determine whether an SAE representation preserves enough of a frozen LM's predictive behavior to support certification of the original model. Code is available at: https://github.com/newcodevelop/sparse-lens-certification.
Read the full abstract ▾
Machine Learning
Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting
Yingshuo Wang, Xian Sun, Lingdong Kong et al. (+4 more)
Standard
benchmarks evaluate time series foundation models (TSFMs) using aggregate metrics, but these can mask severe failures in critical operating regimes. We introduce regime-stratified evaluation and apply it to three TSFMs on two standard traffic speed
benchmarks. Traffic exhibits abrupt regime switching between free-flow and congested states, producing bimodal speed distributions during transitions. When we stratify by traffic regime, both accuracy and prediction-interval coverage degrade sharply during transitions: transition-regime MAE reaches 11 mph (versus 3 mph overall), and empirical coverage of 90% prediction intervals drops as low as 55%. These failures are invisible in aggregate metrics because free-flow observations dominate the sample. A simple historical conditional baseline (sampling from per-sensor training distributions) achieves better transition coverage than any TSFM, but has far worse overall accuracy. We propose bimodal mixture augmentation (BMA), a post-hoc method that combines TSFM forecasts with historical distributional knowledge, approaching the historical baseline's transition coverage while preserving the TSFM's accuracy. Our results suggest that TSFM
benchmarks should incorporate regime-aware evaluation to surface failures that aggregate metrics hide.
Read the full abstract ▾
All authors Yingshuo Wang, Xian Sun, Lingdong Kong, Wei Gao, Yanhang Li, Zhichao Fan, Zexin Zhuang
Machine Learning · Cross-listed from eess.IV
Structural MRI Synthesis for Alzheimer's Disease via Conditional Diffusion on Anatomical Masks
Muge Zhang, Muhammad Ali Khaliq, Jamal Alsakran et al. (+2 more)
Recent advances in generative machine learning models have significantly improved medical imaging, offering promising solutions for data augmentation,
privacy preservation, and improved model generalization. However, synthesizing high-quality structural MRI data for Alzheimer's Disease (AD) remains challenging due to the subtle, region-specific, and progressive anatomical changes associated with neurodegeneration. In this paper, we extend the Med-DDPM conditional
diffusion model – originally designed for
brain tumor synthesis – to generate 3D structural MRIs specifically tailored to AD. We adopted Med-DDPM due to its established stability and structural fidelity compared to other generative models, which makes it particularly suitable for capturing the subtle anatomical changes characteristic of AD. Our approach conditions the
diffusion process on anatomical
segmentation masks derived from the ADNI
dataset, incorporating key AD-relevant
brain structures into the generation process. We systematically evaluate the quality and utility of the synthetic images by training
segmentation models on real, synthetic, and hybrid (mixed)
datasets. Experimental results demonstrate that
segmentation models trained exclusively on synthetic data achieve comparable Dice scores (0.6532) to those trained on real data (0.6513), while exhibiting significantly enhanced recall. Notably, models trained on hybrid
datasets (mixing real and synthetic images) outperform both real and synthetic-only baselines, achieving a Dice score of 0.7244. These findings underscore the successful use of conditional
diffusion models for generating anatomically accurate, AD-specific synthetic MRIs, and highlight their potential for enhancing training data availability, improving diagnostic accuracy, and promoting research reproducibility in neuroimaging studies.
Read the full abstract ▾
All authors Muge Zhang, Muhammad Ali Khaliq, Jamal Alsakran, Byeong Kil Lee, Jeeho Ryoo
Machine Learning
ThousandWorlds: A benchmark for climate emulation of potentially habitable exoplanets
Edward T. Stevenson, Mei Ting Mak, Eric Wolf et al. (+4 more)
The search for life beyond Earth will depend on detecting faint signatures in the atmospheres of potentially habitable
exoplanets. Interpreting those signatures requires understanding the host
planet's
climate: the same molecule may signal life on one
planet and abiotic chemistry on another. Global
climate models (GCMs) provide this understanding, but individual runs can require up to millions of core-hours and substantial domain expert time. Machine-learning emulators could remove this bottleneck, but progress has been limited by the absence of a curated, multi-model exoclimate
dataset. We introduce ThousandWorlds, an ML-ready
benchmark for exoclimate emulation and for the broader regime of low-data, multi-simulator, parameter-to-field regression. The
dataset contains approximately 1,700 simulations from five GCMs, mapping eight
planet parameters to 3D atmospheric fields including temperature, humidity, winds, clouds, and radiation. Three nested subsets define progressively harder challenges: single-simulator regression, multi-simulator regression with complete observations, and multi-simulator regression with structured missingness. We propose two evaluation protocols: one for ranking methods, and one that measures performance relative to the disagreement between GCMs themselves. We evaluate ten baselines spanning simple methods, trees, deep learning, and Gaussian processes. GP-based methods perform best, suggesting that ThousandWorlds exposes a regime where off-the-shelf deep learning does not yet succeed. Data: https://doi.org/10.57967/hf/8695. Code: https://github.com/edstevenson/ThousandWorlds.
Read the full abstract ▾
All authors Edward T. Stevenson, Mei Ting Mak, Eric Wolf, Denis E. Sergeev, Tobi Hammond, N. J. Mayne, Miles Cranmer
Machine Learning
Self-CTRL: Self-Consistency Training with Reinforcement Learning
Itamar Pres, Laura Ruis, Melat Ghebreselassie et al. (+2 more)
Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users. This paper describes Self-Consistency Training with Reinforcement Learning (Self-CTRL), a method that optimizes for consistency between a LM's self-explanations and behavior on related inputs by updating explanations to better predict behavior or updating behavior to better match explanations. We apply our method in two domains. First, we study a formal probabilistic
reasoning task in which LMs must learn to imitate a family of biased samplers and evaluated on their ability to report the associated biases. We find that consistency training improves the correlation between self-reported and behaviorally-measured latent biases from $R^2=0.24$ to $R^2=0.64$ on a set of held-out distributions, matching the generalization of direct ground-truth supervision. Second, we study a constitutional AI domain in which LMs must describe when they will refuse or comply with user requests. Here, Self-CTRL produces rules that faithfully describe the model's behavior on held-out requests, improving the refusal predictions of a third-party auditor model from $36\%$ to $92\%$. In the other direction, behavior updates improve
alignment, reducing HarmBench failure rate from $15.0\%$ to $0.5\%$ without substantially increasing refusal on harmless
prompts. By aligning explanations and behavior, our work provides a general recipe for training AI models to be safer, more transparent, and more controllable.
Read the full abstract ▾
All authors Itamar Pres, Laura Ruis, Melat Ghebreselassie, Belinda Z. Li, Jacob Andreas
Machine Learning
Neural Network Implementation of the Renormalization Group for Fault Diagnosis with Class Imbalance
Evgeny Nikulchev and Dmitry Ilin
The application of machine learning models in practical tasks faces challenges such as class imbalance and multidimensional noise. This paper proposes RGNet, a neural network architecture based on the concept of the renormalization group (RG), for hierarchical coarse-graining of the feature space. The model sequentially compresses the input dimensionality and concatenates all scales before classification, allowing it to capture both local details and global patterns. The notion of RG-flows is introduced - interpretable low-dimensional representations whose visualization via t-SNE reveals a discrete curvilinear structure confirming the effectiveness of coarse-graining. Experimental results are presented on the imbalanced AI4I
dataset. The obtained results demonstrate that RGNet is a universal, interpretable, and competitive solution for fault prediction in applications with imbalanced classes.
Read the full abstract ▾
Machine Learning
Why SWAVE May Not Be All You Need:A Concept-Evolution Retrospective on Complex-Valued Recurrent Language Models
Ramprasath Ganesaraja, Swathika N and Sahil Dilip Panse
SWave is a complex-valued recurrent
language model (169.26M parameters, D=384, L=16, T=2048) trained on FineWeb-Edu using 2xH100 NVL. It was designed around three founding premises: that representing
language as complex waves rather than real-valued numbers enables richer information encoding; that a Cayley-parameterised
unitary transition provides a mathematical guarantee against state decay or explosion; and that a hidden state which rotates rather than shrinks preserves signal integrity over arbitrarily long contexts. The core of SWave evolved substantially across three development phases. The Resonance Head was found to structurally admit imaginary-channel collapse as a global loss minimum (a failure mode we term cos-domination collapse) and was superseded by an untied head with independent real and imaginary
embedding tables from the Phase-Associative
Memory (PAM) architecture. This resolved the degenerate minimum and enabled stable 200,000-step training (best-step PPL 22.0 at step 89,861). ComplexNorm and the Wave Propagation Scan proved load-bearing throughout all three phases and were retained to the final architecture. ProtectGatedScan was reframed as a structural prior rather than a learned behaviour. The four multi-scale retention concepts showed no measurable improvement under controlled evaluation and were found non-load-bearing. The ComplexGatedUnit was superseded by a real-valued squared-ReLU channel mixer with fewer parameters. The auxiliary training objectives showed no benefit once structural constraints were resolved. The investigation yields a formal characterisation of cos-domination collapse, a parallel scan with a log-space backward pass for numerical stability, six transferable engineering principles for complex-valued recurrent training, and a plan-to-code traceability methodology for catching structural divergences that conventional test suites miss.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SD
Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecs
Ali Asaria, Tony Salomone and Deep Gandhi
Open autoregressive neural-codec text-to-
speech (TTS) models sound excellent on typical inputs yet suffer
stochastic catastrophic failures: on a meaningful fraction of utterances they emit silence, terminate early, or collapse into repetitive or hallucinated content. We show this failure mode is cheap to remove. Under a single format-robust metric (a catastrophic-failure rate via an ASR round-trip), best-of-N ASR self-verification drives failures to near-zero: no observed failures remain by N=2 on a standard corpus (LibriSpeech) and by N=4 on a hard
prompt set. This is not an artifact of one model: the reduction replicates across four open codec-TTS systems and three neural codecs (XCodec2, SNAC, Mimi), reaching the near-zero floor by N=2 on three of the four. We then make the fix free at inference time by distilling the self-verified behaviour into the model, which recovers much of the robustness in single-shot decoding, closing ~52-58% of the failure mass on hard inputs at no test-time cost. The
distillation gain concentrates where it is needed (hard inputs); on already-reliable prose there is no headroom and no detectable change. A controlled comparison adds a clean negative: offline direct preference optimization (DPO/IPO) does not beat plain supervised
distillation, and an online iterative variant is promising but not statistically separable at our evaluation size. We report honestly the one model that resists (a larger Llasa where scale did not obviously help) and a rare-word capability ceiling that no self-
distillation method overcomes
Read the full abstract ▾
Machine Learning
SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behavior
Mingyue Cui, Linghui Shen and Xingyi Yang
Sparse Autoencoders (SAEs) decompose residual-stream activations into interpretable features. Recent latent-space defenses increasingly rely on these decompositions, assuming that identified "unsafe" SAE features serve as actionable handles for monitoring and intervention. In this paradigm, clamping a specific harmful feature is expected to reliably prevent model misbehavior. However, we show that this success may hide a recoverable failure mode: the clamp may block one visible route to a behavior without eliminating the behavior itself. We formulate this vulnerability as post-intervention recovery, a constrained residual-space optimization problem. Starting from the post-intervention residual state, we optimize residual perturbations to recover the pre-intervention behavior while preserving the post-intervention values of the targeted SAE features. Even under a strong threat model where the intervention remains active throughout optimization and generation, recovery remains possible. To rule out that recovery simply undoes the intervention, we use encoder-orthogonal updates for single-layer interventions and the corresponding feature-map Jacobian in the cross-layer setting. Across TPP, unlearning, IOI, and refusal steering experiments, this stress test reveals recoverable behavior despite successful feature-level intervention. Especially in the safety-critical refusal-steering setting, we achieve a 95.8% recovery rate on valid samples while keeping defended-feature relative drift to 0.131, substantially below suffix-based baselines. A recovery-path attribution analysis further localizes this recovery to the SAE reconstruction residual, the component left unexplained by the SAE. These results expose a gap between feature-level control and behavioral completeness: SAE features can support causal intervention, but controlling them does not guarantee control over the underlying behavior.
Read the full abstract ▾
Machine Learning
ASTRA: A Scalable Next-Generation ATCO Training Simulator with Autonomous Simpilots
Ethan Chew, Enjia Wu, Iruss Eng et al. (+11 more)
Air Traffic Control Operators (ATCOs) are vital in ensuring the safe, orderly, and efficient flow of air traffic, yet training capacity is constrained by reliance on specialized human trainers known as simpilots, who must role-play both pilots and ATCOs in a simulated airspace. Existing automated solutions rely on Western-centric
speech models that perform poorly in Singaporean operational contexts, with off-the-shelf systems exhibiting Word Error Rates (WER) of up to 107.80% on Singaporean-accented aviation
speech. We introduce ASTRA, an end-to-end training simulator that automates these simpilot roles through a pipeline that transcribes ATCO
speech, interprets instructions, and generates appropriate pilot and ATCO responses using locally adapted voice models. Our fine-tuned Automatic
Speech Recognition (ASR) pipeline reduces WER to 23.45%, substantially outperforming existing approaches in this domain. Beyond traffic simulation, ASTRA incorporates an AI-assisted performance evaluation framework that assesses trainee radiotelephony communications across accuracy, brevity, and completeness, achieving post-optimization scores of 91.7%, 88.2%, and 86.9%, respectively. Built on open-source foundations such as DSPy and Unsloth, this approach enables scalable, standardized ATCO assessment while reducing instructor workload.
Read the full abstract ▾
All authors Ethan Chew, Enjia Wu, Iruss Eng, Ian Lim, Ranen Sim, Brandon Koh, Kaleb Nim, Caden Toh, Wei Dong Soin, Darius Koh, Galen Tay, Prannaya Gupta, Jonathan Koong, Yong Zhi Lim
Machine Learning
Enhanced Graph Neural Networks using K-Hop Gaussian Diffusion
Xuling Zhang, Peng Wang, Daiyan Li et al. (+3 more)
Most graph neural network (GNN) cores rely on graph convolutions, typically implemented as message passing between direct (single-hop) neighbors. In many real-world graphs, edges can be noisy or poorly defined, limiting information propagation to local neighborhoods. Existing
diffusion kernels, such as Personalized PageRank (PPR) and Heat Kernel, alleviate this issue through global propagation, but still struggle with complex local structures and distant node noise. To address these limitations, we propose a K-Hop Gaussian (KHG)
diffusion kernel as a preprocessing module for graph data. KHG introduces multi-hop
diffusion with Gaussian weighting for remote nodes, balancing local and global information propagation before applying standard GNNs. Experiments on multiple
benchmark datasets demonstrate that KHG significantly outperforms traditional message-passing GNNs, as well as PPR and Heat Kernel
diffusion, particularly in noisy or structurally complex graphs.
Read the full abstract ▾
All authors Xuling Zhang, Peng Wang, Daiyan Li, Aoran Huang, Zeiwei Chen, Yongkui Yang
Machine Learning
A Survey on Data-Driven Models for Soil Moisture Regression and Classification
Ilektra Tsimpidi, George Georgoulas, Vidya Sumathy et al. (+1 more)
Soil Moisture (SM) modelling constitutes a complex spatiotemporal learning problem characterised by nonlinear environmental interactions, heterogeneous data sources, and limited ground observations. Physics-based approaches, such as water balance models, rely on explicit hydrological equations and high-quality inputs, but their computational cost and scalability limitations restrict large-scale deployment. Data-driven artificial intelligence (AI) methods have emerged as flexible alternatives, enabling the extraction of empirical relationships between soil moisture and environmental variables with reduced modelling assumptions. This work presents a structured survey of AI-based models for soil moisture estimation and classification. Existing approaches are organized into five categories: (a) statistical time-series models, (b) geostatistical methods (c) classical machine learning (ML) models, (d) Deep Learning (DL) models and (e) Probabilistic/
Bayesian methods. These models leverage historical soil moisture records, meteorological variables, vegetation indices, topography, soil characteristics, and geolocation data to perform regression or classification tasks.
Read the full abstract ▾
All authors Ilektra Tsimpidi, George Georgoulas, Vidya Sumathy, George Nikolakopoulos
Machine Learning
Ghost Attractor Networks: Basin-Structured Dynamical Decoders for Closed-Loop Sequential Generation
Tianyu Wang, Ying Wang, Zhihao Liu et al. (+2 more)
Sequential output generation with large-scale
Transformer and
diffusion decoders pays a
memory cost that grows with sequence length, plus iterative per-step computation. Replacing them with small feed-forward decoders restores efficiency but produces unstructured latent representations that limit closed-loop control: phase-conditioned action generation and cross-step latent carry-over both require a latent geometry with stable basins. This article proposes Ghost Attractor Networks, a theoretically derived dynamical decoder whose latent evolves under a learned potential with drift and produces a basin-attractor structure by construction. Three desiderata (multi-modality, decoder-level single-pass switching, and constant
memory) motivate the potential-drift form, and mode transitions arise as saddle-node bifurcations with ghost-attractor escape. A hierarchical phase-space decomposition separates first-order basin convergence from second-order proprioceptive refinement. Empirically, a Ghost trained end-to-end with a behavioral-cloning and contrastive objective exhibits the predicted gradient-flow contraction in its potential, with the gradient norm decaying by 67 percent across five integration steps on 1430 held-out samples. Ghost is evaluated as a robotic action decoder. A 2.3-million-parameter Ghost matches the offline accuracy of a 1.07-billion-parameter
Diffusion Transformer at 462 times fewer parameters and 32 times lower latency, and beats five alternative 2M-parameter decoders (MLP, Neural ODE, CVAE,
Transformer, 1-step
Diffusion) on offline mean squared error by 5.9 to 29 percent. On the LIBERO-10 closed-loop
benchmark, phase conditioning on Ghost's basin-structured latent yields a 13.5 percentage-point success-rate gain over a feed-forward MLP baseline, and persistent-latent ensembling reaches a 95.7 percent final success rate.
Read the full abstract ▾
All authors Tianyu Wang, Ying Wang, Zhihao Liu, Xi Vincent Wang, Lihui Wang
Machine Learning · Cross-listed from cs.CR
TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization
William Kalikman, Ivo Petrov, Dimitar I. Dimitrov et al. (+1 more)
Federated learning allows multiple clients to jointly train a shared model by sending gradient updates to a central server while keeping raw inputs local. However, prior gradient inversion attacks show that these updates can reveal enough information to reconstruct client inputs. Existing attacks on
transformers either optimize dummy inputs to match the true client updates, which is costly and unstable for modern models, or exploit the low rank of
attention gradients to identify a subspace containing the true layer
embeddings, followed by a discrete membership test for candidate
tokens. However, this
token test is brittle under numerical noise, i.e., from quantization or Differential
Privacy (DP), and scales poorly for encoder models with non-causal
attention. We introduce TIGER, a continuous gradient inversion attack that turns this subspace signal into a differentiable objective. Instead of searching over
tokens or matching full gradients, TIGER directly optimizes
token embeddings to minimize their distance to the subspace. Our experiments demonstrate that on encoder-only models, TIGER substantially improves both reconstruction quality and runtime over existing attacks, while on decoder models, TIGER is more robust than prior subspace-based attacks, enabling the first successful reconstructions in DP-defended
federated learning settings.
Read the full abstract ▾
All authors William Kalikman, Ivo Petrov, Dimitar I. Dimitrov, Martin Vechev
Machine Learning
SAGE: Retain-Aware Post-Hoc Sanitization of Final Unlearning Vector
Jingyuan Zhang, Yucheng Bai, Peixi Wen et al. (+6 more)
Large
Language Model (LLM) unlearning aims to remove undesirable knowledge or behaviors while preserving retained capabilities. Current unlearning methods all involve a trade-off between unlearning and retention. We have found that the retention activation bias can also be used to quantify the damage an unlearning method inflicts on retention, without considering the specific implementation of the unlearning process. This allows us to restore retention performance for any unlearning method using a post-hoc approach. Therefore, we propose a complementary post-hoc setting to sanitize the final update vector without rerunning the original unlearning pipeline. In this setting, we design SAGE, Spectral Activation-GEometry Sanitization, a source-agnostic correction for final unlearning updates. SAGE collects real module inputs from a small retain proxy, extracts their dominant activation geometry, and solves a source-anchored optimization objective in closed form, which suppresses update components aligned with high-energy retained directions while preserving the source method's forgetting carrier. Across multiple unlearning methods, model scales, and
benchmarks, SAGE consistently relieves the retain-forget trade-off, identifying post-hoc sanitization of final vectors as a practical and underexplored axis for machine unlearning.
Read the full abstract ▾
All authors Jingyuan Zhang, Yucheng Bai, Peixi Wen, Zhehao Huang, Zhengbao He, Hanling Tian, Xinwen Cheng, Haiyin Ran, Xiaolin Huang
Machine Learning
TRIDENT: Breaking the Hybrid-Safety-Physics Coupling for Provably Safe Multi-Agent Reinforcement Learning
Zijie Meng, Ziwei Li, Yufei Liu et al. (+5 more)
Safe coordination in networked cyber-physical systems forces learning
algorithms to simultaneously handle hybrid discrete-continuous actions, hard training-time safety constraints, and physics-governed dynamics. We show that these three features form a directed cycle of biases that defeats any naive composition of off-the-shelf modules, and formalize this as a three-way coupling lemma. We then introduce TRIDENT, the first MARL framework whose three components are co-designed to cancel each leak: a Richardson-Romberg gradient correction reducing Gumbel-Softmax bias from O(tau) to O(tau^2), a Lyapunov-constrained sequential trust-region update enforcing per-iterate feasibility, and a physics-informed residual critic that decomposes value rather than reward. We prove an O~(1/sqrt(K)) convergence rate to a constrained Nash
equilibrium and an O(sqrt(K)) cumulative-violation bound. On multi-
UAV mobile-edge computing, autonomous intersection management, and a hybrid SMAC variant, TRIDENT cuts training-time violations by 95.5% over MADDPG and 76.3% over MACPO, while improving reward by 13.5% over the strongest unconstrained baseline.
Read the full abstract ▾
All authors Zijie Meng, Ziwei Li, Yufei Liu, Zhiyu Li, Jiyuan Liu, Wenhua Nie, Bingcai Wei, Miao Zhang
Machine Learning
DRIFT: Data Selection for LLM Instruction Tuning via On-Policy Attribution
Zefan Wang, Lincheng Li, Tianyu Yu et al. (+1 more)
Data selection for instruction tuning is usually studied for efficiency, reaching strong performance with a fraction of the training data. We ask whether data selection can instead raise the performance ceiling, producing a better model than training on all the available data. To test this, we take a large
language model (LLM) that has already completed supervised fine-tuning (SFT) on its full corpus and train it further on examples selected from that same corpus. Any gain therefore lifts the model above the ceiling set by full-data training. Existing selection methods yield little or no gain in this setting. We propose DRIFT (Data Refinement via On-Policy Influence Functions for Supervised Fine-Tuning), a data attribution method built on influence functions. DRIFT samples the model's own responses to validation queries and weights them by correctness to form a validation objective. It ranks the examples of the original corpus by their estimated influence on this objective, and the model then continues SFT on the top-ranked subset. This design is motivated by the proximity gap in influence function theory: we hypothesize that these on-policy responses are better validation targets than external references in terms of locality. DRIFT also corrects influence scores for their dependence on gradient norm within each validation task. DRIFT outperforms all evaluated data selection baselines on Olmo3-7B-Instruct-SFT and OpenR1-Distill-7B, raising average accuracy across eight
benchmarks from 39.17 to 40.14 and from 57.23 to 58.66, respectively, with gains on
benchmarks not used for attribution.
Read the full abstract ▾
All authors Zefan Wang, Lincheng Li, Tianyu Yu, Yuan Yao
Machine Learning
Fisher Width: A Geometric Measure of Complexity on Statistical Manifolds
Vu Khac Ky
Gaussian width is a central
geometric complexity measure in high-dimensional probability, compressed sensing, convex optimization, and learning theory. It quantifies the average extent of a set along random directions, thereby capturing the effective dimension of constraint sets, hypothesis classes, and descent cones. However, this notion is intrinsically Euclidean. Statistical models instead carry a natural Riemannian geometry induced by the Fisher information metric, where directions are scaled according to statistical distinguishability rather than ambient Euclidean length. We introduce Fisher width, a Fisher-
geometric analogue of Gaussian width for statistical
manifolds. At a parameter point $\theta$, Fisher width replaces the Euclidean identity by the local metric tensor $G(\theta)^{1/2}$, measuring the Gaussian width of the Fisher-rescaled set. This makes the resulting quantity sensitive to local statistical
curvature and invariant under smooth reparameterizations. We develop the basic theory of Fisher width, showing that it retains key structural features of Gaussian width, including concentration, metric perturbation stability, and spectral comparison bounds with the Euclidean baseline, while also capturing anisotropic
geometric effects invisible to Euclidean measures. As an application, we prove a generalization bound for Fisher-Lipschitz hypothesis classes and propose computable
estimators, which we evaluate empirically on MNIST across three model classes. Fisher width is to statistical
manifolds what Gaussian width is to Euclidean convex bodies. This work lays the foundation for studying complexity and learning on curved statistical
manifolds.
Read the full abstract ▾
Machine Learning · Cross-listed from math.NA
Starter-Iterator Neural Operator: A Unified Architecture for High-Fidelity Forward and Inverse PDE Problems
Kuilin Qin, Lianfang Wang, Xu Sun et al. (+4 more)
Operator learning is an emerging field at the intersection of machine learning and scientific computing. By learning mappings between function spaces, neural operators provide data-driven surrogate models for families of partial differential equations (PDEs). Once trained, these models can evaluate solution operators efficiently, making them suitable for many-query applications such as real-time prediction and parameter sweeps. However, maintaining high approximation accuracy and stable long-term predictions remains challenging for complex forward and inverse problems. To address these challenges, we propose the Starter-Iterator Neural Operator (SINO), which incorporates the initialization and residual-correction structures of classical iterative solvers into neural operator learning. The frequency-domain Starter captures dominant global spectral features and provides an informed initial approximation, while the latent-space Iterator applies successive residual-based corrections to refine local and multiscale solution structures. Experiments on representative time-dependent PDEs, including the Navier-Stokes and acoustic wave equations, together with applications to image super-resolution and weather
forecasting, show that SINO achieves competitive accuracy and stable performance across the
benchmarks considered in this work.
Read the full abstract ▾
All authors Kuilin Qin, Lianfang Wang, Xu Sun, Jiwei Jia, Yu Wang, Yong Wang, Yuping Duan
Machine Learning
Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
Yifu Ding, Jiacheng Wang, Ge Yang et al. (+4 more)
Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial
memory footprint and inference overhead. Prior compression methods mainly operate at the expert level, either removing entire experts or ranking experts by coarse-grained importance scores. However, such expert-wise decisions are often too coarse to capture fine-grained redundancy, leading to misallocated
pruning budgets and limited compression. To address this problem, we observe that information within MoE experts is highly concentrated in a small subset of channels, leaving substantial redundancy even in experts deemed important. Based on this observation, we propose a structural
pruning framework tailored for MoE models. Our method reformulates prune-ratio allocation as a channel-score coverage maximization problem and solves it efficiently using an attribution-based approximation. Experiments on DeepSeek and Qwen MoE models show that our method preserves model accuracy under 50% or 25% structured
pruning when combined with 4-bit quantization. On Qwen3-30B-A3B, our approach reduces
memory footprint by 5.27$\times$ and consistently outperforms state-of-the-art baselines across diverse
benchmarks.
Read the full abstract ▾
All authors Yifu Ding, Jiacheng Wang, Ge Yang, Yongcheng Jing, Jinyang Guo, Xianglong Liu, Dacheng Tao
Machine Learning
A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks
Taiki Miyagawa
We develop a mathematically explicit link between shock-wave theory and the
symmetry-quotiented learning dynamics of
stochastic gradient descent, drawing on differential geometry, Lie group theory, and fluid mechanics. Specifically, after quotienting parameter
symmetries and applying local-
entropy coarse-graining, the effective dynamics satisfy a viscous Hamilton–Jacobi equation on the quotient
manifold. Moreover, under the assumption that the raw parameter dynamics can be summarized by a gradient field on the quotiented space, the gradient of the coarse-grained loss function obeys a Burgers-type equation, and shock formation can be established rigorously. We apply our theory to multilayer perceptrons, convolutional neural networks,
Transformers, and mean-field networks, and show that they obey the Hamilton–Jacobi or Burgers-type equations. We
conjecture that this framework also yields practical diagnostics for deep learning. In architectures such as
Transformers, raw parameter norms are often distorted by
symmetry redundancy and may therefore be misleading, whereas
symmetry-corrected quotient observables provide a principled basis for monitoring,
forecasting, and controlling training-phase transitions.
Read the full abstract ▾
Machine Learning · Cross-listed from q-bio.OT
Protein-Based Fish Species Identification: Dataset, Models, and Insights from Native Bangladeshi Fish
Md Nasiat Hasan Fahim, Md. Abid Ullah Muhib and Mohammad Shahidur Rahman
Correct identification of fish species is highly significant for food security, economic development, and
climate resilience in Bangladesh.
Protein sequences directly reflect functional and evolutionary constraints which are important for species authentication and biodiversity monitoring. Yet there exists no
benchmark for native Bangladeshi fish species identification from
protein sequence. In this study, we addressed this gap by introducing the first curated
dataset for nine native Bangladeshi fish species of 2845 high quality
protein sequences. We also established the first
protein sequence classification baseline for this domain through a systematic benchmarking of seven architectural paradigms. Moreover, we propose a realistic deployable novel hybrid architecture of MotifCNN and
Transformer with Terminal-Aware Positional-Encoding (MotifCNN-
Transformer+TA-PE). Our novel architecture achieves 79.80% accuracy with macro-F1 of 0.80. The highest 83.04% accuracy is achieved by finetuned
protein language model ProtBERT that has 420M parameters and requires dual 16GB GPUs for inference. According to McNemar's test, ProtBERT's 3.24% accuracy gain over our MotifCNN-
Transformer+TA-PE is statistically insignificant (p = 0.1120). Our novel architecture beats it among six of the nine classes in per class identification. Also our MotifCNN-
Transformer+TA-PE is approximately 5x faster, 42x smaller, and supports 16x larger batch size than ProtBERT and has GPU free inference, making it more practical for deployment in resources constrained areas such as rural Bangladesh. Beyond this, our foundational work shows effects of phylogenetic relationships on sequence similarity and establishes pathways for fisheries management, food authentication and biodiversity conservation in South Asia's
protein dependent economy.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Shanda Li, Qiuhong Anna Wei, Jingwu Tang et al. (+5 more)
Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced
benchmarks to evaluate whether LLM
agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-
agent configurations. Our results show that LLM
agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best
agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for $\sim$90% of papers in the study. Further analysis shows that
agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM
agents on real-world reproducibility
auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.
Read the full abstract ▾
All authors Shanda Li, Qiuhong Anna Wei, Jingwu Tang, Valerie Chen, Nihar B Shah, Tim Dettmers, Yiming Yang, Ameet Talwalkar
Machine Learning
Sign-Rank, Index, and List Replicability: Connections and Separations
Ari Blondal, Hamed Hatami, Pooya Hatami et al. (+2 more)
In learning theory, the sign-rank of a binary concept class captures the smallest dimension in which it can be represented by points and halfspaces. Despite tremendous interest, lower bounds on sign-rank are notoriously difficult to come by. Two recent approaches to the problem establish lower bounds on sign-rank by measures that are easier to analyze: the $\mathbb{Z}_2$-index and the list replicability number. We order these measures, showing that the $\mathbb{Z}_2$-index is upper-bounded by a linear function of the list replicability number. As a main consequence, we obtain a strong separation between sign-rank and $\mathbb{Z}_2$-index, thereby resolving a question of Frick, Hosseini, and Vasileuski. This motivates a thorough study of list replicability, the stronger of the two lower-bounding measures. We establish upper bounds on the list replicability number by two combinatorial measures: height and minimum
star number. We also prove a fundamental composition result, showing that the product of two concept classes has list replicability number bounded by the sum of the list replicability numbers of the two classes.
Read the full abstract ▾
All authors Ari Blondal, Hamed Hatami, Pooya Hatami, Chavdar Lalov, Sivan Tretiak
Machine Learning · Cross-listed from cs.CV
Adaptive Volumetric Mechanical Property Fields Invariant to Resolution
Rishit Dagli, Donglai Xiang, Vismay Modi et al. (+4 more)
Accurate mechanical properties (or materials) Young's modulus ($E$), Poisson's ratio ($\nu$) and density ($\rho$) are essential for reliable physics simulation of digital worlds, but most 3D assets lack this information. We propose AdaVoMP, a method for predicting accurate dense spatially-varying ($E$, $\nu$, $\rho$) for input 3D objects across representations, improving the resolution, accuracy, and
memory efficiency over the state-of-the-art. The foundation of our technique is a sparse and adaptive voxel structure SAV that efficiently represents both the input 3D shape and the material field output. We replace the fixed-voxel model of the most accurate prior method, VoMP, with a novel sparse
transformer encoder-decoder model that learns to generate a unique SAV autoregressively for every input shape to represent its materials, achieving a resolution $16^3\times$ higher than prior art. Experiments show that AdaVoMP estimates more accurate volumetric properties, even with lesser test-time compute than all prior art. This allows us to convert high-resolution complex 3D objects into simulation-ready assets, resulting in realistic deformable simulations.
Read the full abstract ▾
All authors Rishit Dagli, Donglai Xiang, Vismay Modi, Xuning Yang, Gavriel State, David I.W. Levin, Maria Shugrina