Machine Learning
Fair and Calibrated Toxicity Detection with Robust Training and Abstention
Mokshit Surana
Fairness in toxicity classification involves three integrated axes: ranking, calibration, and abstention. Training-time interventions and post-hoc safety mechanisms cannot be evaluated independently because the former determines the efficacy of the latter. We compare Empirical Risk Minimization (ERM), instance-level reweighting, and Group DRO across these axes, combined with temperature scaling, confidence-based abstention, and per-identity threshold optimization. Evaluation uses subgroup AUC, BPSN/BNSP AUC, error gaps, and per-subgroup Expected Calibration Error (ECE) with bootstrap CIs ($n = 1000$). We report four findings. (1) Calibration disparity is a hidden
fairness violation. ERM has near-perfect aggregate calibration ($0.013$) but is significantly miscalibrated across all identity subgroups ($+0.029$ to $+0.134$). (2) Training interventions reshape rather than eliminate disparity. Reweighted ERM improves ranking (BPSN AUC $+0.06$ to $+0.12$) but worsens the calibration-
fairness gap by up to $+0.232$. Group DRO eliminates calibration disparity but only by becoming uniformly miscalibrated globally (ECE $0.118$). (3) Post-hoc methods inherit training failure modes. Temperature scaling fails because miscalibration is non-uniform. Confidence-based abstention works under ERM but breaks under DRO, where the risk-coverage curve rises with deferral. (4) Abstention itself is unfair. Confidence-based deferral helps background content far more than identity-mentioning content. We argue that SRAI
fairness requires a multi-axis framework: methods that differ only in aggregate ranking can differ sharply in failure modes that determine real-world harm.
Read the full abstract ▾
Machine Learning
AttnGen: Attention-Guided Saliency Learning for Interpretable Genomic Sequence Classification
Rayhaneh Shabani Nia and Ali Karkehabadi
Deep neural networks have achieved strong performance in genomic sequence classification; however, relating their predictions to biologically meaningful sequence patterns remains challenging. In this work, we present AttnGen, an
attention-guided training framework that embeds interpretability directly into the optimization process. AttnGen computes nucleotide-level importance scores using an
attention mechanism and progressively suppresses low-contribution positions during training. This encourages the model to focus its predictions on a compact set of informative regions while reducing reliance on noisy sequence elements. We evaluate AttnGen on the standardized demo_human_or_worm
benchmark, a binary classification task over 200-nucleotide sequences. With moderate masking, AttnGen achieves a validation accuracy of 96.73%, outperforming a conventional CNN baseline with 95.83% accuracy, while also exhibiting faster convergence and improved training stability. To assess whether the learned importance scores reflect functionally relevant signal, we conduct perturbation-based analysis by removing high-saliency nucleotides. This causes accuracy to drop from 96.9% to near chance level on a 3,000-sequence evaluation set, indicating that the model relies on a relatively small subset of informative positions. Our analysis shows that masking 10–20% of positions provides the most favorable trade-off between predictive performance and interpretability. These results suggest that
attention-guided masking not only improves classification performance but also reshapes how models distribute importance across sequence positions. Although this study focuses on short genomic sequences, the proposed approach may extend to more complex interpretable sequence modeling settings.
Read the full abstract ▾
Machine Learning
SurF: A Generative Model for Multivariate Irregular Time Series Forecasting
Mohammad R. Rezaei, Tejas Balaji and Rahul G. Krishnan
Irregularly sampled multivariate event streams remain a difficult modality for generative modeling: tokenization-based approaches break down when inter-event intervals vary by orders of magnitude. We (i) propose SurF, a generative model that uses the Time Rescaling
Theorem (TRT) as a learnable bijection between event sequences and i.i.d.\ unit-rate exponential noise, enabling a single model to be trained across heterogeneous event-stream
datasets; (ii) three efficient parameterizations of the cumulative intensity that scale to long sequences; and (iii) a
Transformer-based encoder for multi-
dataset pretraining. On six real-world
benchmarks, SurF achieves the best reported time RMSE on Earthquake, Retweet, and Taobao, and is within trial-level noise of the strongest specialist on the remaining three. Under a strict leave-one-out protocol, the held-out checkpoint beats every classical and neural-autoregressive baseline on $5/6$
datasets and beats every baseline on Amazon and Earthquake, an initial step toward foundation models over asynchronous event streams (Code is available at https://github.com/MrRezaeiUofT/SurF).
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
CurveBench: A Benchmark for Exact Topological Reasoning over Nested Jordan Curves
Amirreza Mohseni, Mona Mohammadi, Morteza Saghafian et al. (+1 more)
We introduce CurveBench, a
benchmark for visual topological
reasoning. Given an image of disjoint closed curves, a model must recover the tree of adjacent regions rooted at the exterior. The
benchmark contains 756 hand-drawn images, ranging from simple configurations to maze-like and densely packed arrangements. Each image has a manually verified reference tree. We evaluate predictions by exact tree matching with the exterior root fixed. Gemini 3.1 Pro, the strongest model in our evaluation, achieves 71.1% exact-match accuracy on CurveBench-Easy and 28.7% on CurveBench-Hard. A Qwen3-VL-8B checkpoint trained using reinforcement learning with verifiable rewards reaches 31.7% on Easy, compared with 21.7% for its Thinking base model at a 16,384-
token generation budget. The same comparison yields a gain of 1.1 percentage points on Hard.
Read the full abstract ▾
All authors Amirreza Mohseni, Mona Mohammadi, Morteza Saghafian, Naser Talebizadeh Sardari
Machine Learning
Comparative Evaluation of Machine Learning Approaches for Minority-Class Financial Distress Prediction Under Class Imbalance Constraints
Karan Sehgal and Khawar Naveed Bhatti
Financial distress prediction remains a significant challenge in enterprise risk analysis due to the highly imbalanced nature of real-world financial
datasets, where bankrupt or distressed firms typically constitute only a small minority of observations. This paper presents a comparative evaluation of classical statistical methods, ensemble learning approaches, and exploratory neural models for minority-class financial distress prediction under class imbalance constraints. The study incorporates structured preprocessing, imbalance mitigation using the Synthetic Minority Oversampling Technique (SMOTE), comparative evaluation across ensemble learning architectures including XGBoost, CatBoost, LightGBM, Random Forest, and explainability analysis using SHAP-based feature attribution methods. Experimental evaluation demonstrates that gradient-boosting approaches achieved improved minority-class sensitivity relative to baseline statistical classifiers under severe imbalance conditions. The workflow additionally emphasises reproducibility, interpretability, auditability, and governance-oriented machine learning evaluation within enterprise financial risk environments. The work is positioned as an applied engineering evaluation intended to support reproducible and interpretable machine learning workflows for financial distress prediction under severe class imbalance constraints.
Read the full abstract ▾
Machine Learning
Reliability-Gated Source Anchoring for Continual Test-Time Adaptation
Vikash Singh, Debargha Ganguly, Weicong Chen et al. (+8 more)
Continual test-time adaptation (CTTA) updates a pretrained model online on an unlabeled, non-stationary stream while anchoring it to a frozen source checkpoint. This anchor is useful only when the source remains reliable. On CCC-Hard, however, a ResNet-50 source falls to approximately $1.3\%$ top-$1$ accuracy, while existing source-anchored CTTA methods continue applying the same anchor strength. We call this failure mode blind anchoring and propose RMemSafe, a reliability-gated extension of ROID that uses the frozen source's normalized predictive
entropy to attenuate all explicit source-coupled uses in the objective. When the source posterior approaches uniformity, the gate closes: the source anchor and agreement filter vanish, and the objective reduces to a source-agnostic fallback comprising ROID's base losses plus marginal calibration. Combined with ASR, RMemSafe achieves the lowest error on $8$ of $9$ matched-split continual-corruption
cells and is the best reset-based method on all $9$, improving ROID+ASR by $1.05$~pp on ResNet-50 and $0.48$~pp on ViT-B/16. A controlled source-degradation sweep shows a $1.13{\times}$ shallower harm slope than ROID+ASR, consistent with the graceful-decay prediction. The
entropy gate detects high-
entropy source collapse, not confidently wrong low-
entropy sources; this scope is explicitly evaluated and discussed.
Read the full abstract ▾
All authors Vikash Singh, Debargha Ganguly, Weicong Chen, Sabyasachi Sahoo, Sreehari Sankar, Biyao Zhang, Mohsen Hariri, Shouren Wang, Osama Zafar, Christian Gagné, Vipin Chaudhary
Machine Learning · Cross-listed from cs.AI
MathAtlas: A Benchmark for Autoformalization in the Wild
Nilay Patel, Noah Arias, Davit Babayan et al. (+7 more)
Current autoformalization
benchmarks are largely focused on olympiad or undergraduate mathematics, while graduate and research-level mathematics remains underexplored. In this paper, we introduce MathAtlas, the first large-scale autoformalization
benchmark of in the wild graduate-level mathematics, containing ~52k
theorems, definitions, exercises, examples, and proofs extracted from 103 graduate mathematics textbooks. MathAtlas is enriched with a mathematical dependency graph containing ~178k relations, and is the first autoformalization
benchmark to include such relations, facilitating evaluation and development of dependency-aware autoformalization systems. Our extensive experiments show that MathAtlas is high quality but extremely challenging: strong baselines achieve at most 9.8% correctness on
theorem statements and 16.7% on definitions. Furthermore, we find performance of state-of-the-art models degrades substantially with dependency depth: on MA-Hard, a subset of 700 entities with the deepest dependency trees, the best model achieves only 2.6% correctness for autoformalization on this challenging
dataset. We release MathAtlas to the community as a
benchmark set for large-scale autoformalization of graduate-level mathematics in the wild.
Read the full abstract ▾
All authors Nilay Patel, Noah Arias, Davit Babayan, Victoria Cochran, Timothy Libman, Hafsah Mahmood, Liam McCarty, Soli Munoz, Laurel Willey, Jeffrey Flanigan
Machine Learning · Cross-listed from cs.AI
Network-Aware Bilinear Tokenization for Brain Functional Connectivity Representation Learning
Leo Milecki, Qingyu Hu, Bahram Jafrasteh et al. (+2 more)
Masked autoencoders (MAEs) have recently shown promise for self-supervised representation learning of resting-state
brain functional connectivity (FC). However, a fundamental question remains unresolved: how should FC
matrices be tokenized to align with the intrinsic modular organization of large-scale
brain networks? Existing approaches typically adopt region-centric or graph-based schemes that treat FC as structurally homogeneous elements and overlook the large-scale network
brain organization. We introduce NERVE (Network-Aware Representations of
Brain Functional Connectivity via Bilinear Tokenization), a self-supervised learning framework that redefines FC tokenization by partitioning FC
matrices into patches of intra- and inter-network connectivity blocks. Unlike image-based MAE, where fixed-size patches share a common tokenizer, FC patches defined by network pairs are heterogeneous in size and correspond to distinct functional roles. To resolve this problem, NERVE embeds FC patches through a novel structured bilinear factorization. This formulation preserves network identity and reduces parameter complexity from quadratic to linear scaling in the number of networks. We evaluate NERVE across three large-scale developmental cohorts (ABCD, PNC, and CCNP) for behavior and psychopathology prediction. Compared to structurally agnostic MAE variants and graph-based self-supervised baselines, the proposed network-aware formulation yields more stable and transferable representations, particularly in cross-cohort evaluation. Ablation studies confirm that the proposed bilinear network
embedding and anatomically grounded parcellation are critical for performance. These findings highlight the importance of incorporating domain-specific structural priors into self-supervised learning for functional connectomics. Code is available at: https://github.com/leomlck/NERVE.
Read the full abstract ▾
All authors Leo Milecki, Qingyu Hu, Bahram Jafrasteh, Mert R. Sabuncu, Qingyu Zhao
Machine Learning · Cross-listed from stat.ME
Wahkon: A Statistically Principled Deep RKHS Superposition Network
Yongkai Chen, Wenxuan Zhong and Ping Ma
Deep learning excels at prediction but often lacks finite-sample guarantees and calibrated uncertainty; RKHS (Reproducing Kernel
Hilbert Space)-based methods provide those guarantees but struggle to adapt in high dimensions. We propose Wahkon, a deep RKHS superposition network that unifies Kolmogorov's superposition principle with RKHS regularization in the smoothing-spline tradition of Wahba. This yields a finite-dimensional deep representer
theorem that makes training tractable and provides explicit layerwise complexity control. We show the penalized
estimator is exactly the MAP (maximum a posteriori) estimate under a hierarchical Gaussian-process prior, extending the spline/GP duality to deep compositions. Using metric-
entropy arguments, we establish minimax-optimal convergence rates under mild smoothness and clarify how depth and width trade off with regularity. Empirically, Wahkon outperforms multilayer perceptrons, Neural Tangent Kernels, and Kolmogorov–Arnold Networks across simulation
benchmarks and a single-
cell CITE-seq study. By unifying Kolmogorov's superposition principle with RKHS regularization, Wahkon delivers accuracy, interpretability, and statistical rigor in a single framework.
Read the full abstract ▾
Machine Learning
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
Gergely Szilvasy, Manuel Faysse, Maria Lomeli et al. (+5 more)
Under modern test-time compute and agentic paradigms,
language models process ever-longer sequences. Efficient text generation with
transformer architectures is increasingly constrained by the Key-Value cache
memory footprint and bandwidth. To address this limitation, we introduce Self-Pruned Key-Value
Attention (SP-KV), a mechanism designed to predict future KV utility in order to reduce the size of the long-term KV cache. This strategy operates at a fine granularity: a lightweight utility predictor scores each key-value pair, and while recent KVs are always available via a local window, older pairs are written in the cache and used in global
attention only if their predicted utility surpasses a given threshold. The LLM and the utility predictor are trained jointly end-to-end exclusively through next-
token prediction loss, and are adapted from pretrained LLM checkpoints. Rather than enforcing a fixed compression ratio, SP-KV performs dynamic sparsification: the mechanism adapts to the input and typically reduces the KV cache size by a factor of $3$ to $10\times$, longer sequences often being more compressible. This leads to vast improvements in
memory usage and decoding speed, with little to no degradation of validation loss nor performance on a broad set of downstream tasks. Beyond serving as an effective KV-cache reduction mechanism, our method reveals structured layer- and head-specific sparsity patterns that we can use to guide the design of hybrid local-global
attention architectures.
Read the full abstract ▾
All authors Gergely Szilvasy, Manuel Faysse, Maria Lomeli, Matthijs Douze, Pierre-Emmanuel Mazaré, Loïc Cabannes, Wen-tau Yih, Hervé Jégou
Machine Learning · Cross-listed from cs.AI
Enhanced and Efficient Reasoning in Large Learning Models
Leslie G. Valiant
In current Large
Language Models we can trust the production of smoothly flowing prose on the basis of the principles of machine learning. However, there is no comparably principled basis to justify trust in the content of the text produced. It appears to be conventional wisdom that addressing this issue by adding more principled
reasoning is not computationally affordable. Here we propose a principled method of
reasoning that is efficient enough to be practical for large
language models. Further, the method allows the retention of much of the currently used software and hardware base. Our method for improving the functioning of large
language models consists of a first stage of preprocessing that recodes the data to a Unary Relational Integracode that is more explicit about the relationships among the objects described in the text, followed as a second stage by a standard but possibly streamlined machine learning process that then also learns to predict these relationships. The method may be viewed as realizing a world model and applying beyond natural
language, to
vision and actions, for example, where the multiple properties of an object referred to in an input are brought together explicitly, rather than remaining distributed in the various references to it in the input. We articulate its advantages in terms of Robust Logic, a system for performing principled chaining on learned, and hence uncertain, information. We show that this recoding has the surprising and fortuitous property that, while succinct, it makes the task of learning a core subset of relational rules that hold in the world described in the training data
polynomial time learnable in a defined sense, the
polynomial depending on the complexity of the rule. This gives support for sound
reasoning within each single call of the learned classifier as well as between multiple calls.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Sheaf-Theoretic Transport and Obstruction for Detecting Scientific Theory Shift in AI Agents
David N. Olivieri and Roque J. Hernández
Scientific theory shift in AI
agents requires more than fitting equations to data. An artificial scientific
agent must detect whether an existing representational framework remains transportable into a new regime, or whether its
language has become locally-to-globally obstructed and must be extended. This paper develops a finite sheaf-theoretic framework for detecting theory-shift candidates through transport and obstruction. Contexts are organized as a local-to-global structure in which source, overlap, target, and validation charts are fitted, restricted, and tested for gluing. Obstruction measures failure of coherence through residual fit, overlap incompatibility, constraint violation, limiting-relation failure, and representational cost. We evaluate the framework on a controlled transition-card
benchmark designed to separate deformation within a source
language from extension of that
language. The main result is direct obstruction ranking: the intended deformation or extension is usually the lowest-obstruction candidate, and transition type is separated in the
benchmark. A constellation kernel over the same signatures is included only as a secondary representational-similarity probe. The aim is not to reconstruct historical paradigm shifts or solve open-ended autonomous theory invention, but to isolate a finite diagnostic subproblem for AI
agents: detecting when representational transport fails and extension becomes the coherent next move.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SD
Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study
Wuao Liu, Mustafa Chasmai, Subhransu Maji et al. (+1 more)
Bioacoustic recognition requires fine-grained acoustic understanding to distinguish similar-sounding species. However, many large-scale data repositories such as iNaturalist are weakly annotated, often with only a single positive species label per recording, making supervised learning particularly challenging. Inspired by advances in computer
vision, recent approaches have shifted toward self-supervised learning to capture the underlying structure of audio without relying on exhaustive annotations. In particular, masked autoencoders (MAE) have shown strong transferability on massive audio corpora, yet their effectiveness in more modest bioacoustic settings remains underexplored. In this work, we conduct a systematic study of MAE pretraining for species classification on iNatSounds, analyzing the impacts of pretraining data scale, domain specificity, data curation, and transfer strategies. Consistent with prior work, we find that models pretrained on diverse general audio data achieve the best transfer performance on iNatSounds. Contrary to observations from large-scale audio
benchmarks, we find that (1) additional masked reconstruction pretraining on domain-specific data provides limited benefits and may even degrade performance relative to off-the-shelf models, and (2) selective data filtering offers a negligible advantage when the overall data scale is limited. Our results indicate that, in moderate-sized fine-grained bioacoustic settings, pretraining scale dominates objective design. These findings further clarify when MAE-based pretraining is effective and provide practical guidance for model selection under limited supervision.
Read the full abstract ▾
All authors Wuao Liu, Mustafa Chasmai, Subhransu Maji, Grant Van Horn
Machine Learning
R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning
Sanghyeob Song, Donghyeok Lee, Jinsik Kim et al. (+1 more)
For reinforcement learning in data-scarce domains like real-world robotics, intensive data reuse enhances efficiency but induces overfitting. While prior works focus on critic bias, representation-level instability in Self-Predictive Learning (SPL) under high Update-to-Data (UTD) regimes remains underexplored. To bridge this gap, we propose Robust Representation via Redundancy Reduction (R2R2), a regularization method within SPL. We theoretically identify that standard zero-centering conflicts with SPL's spectral properties and design a non-centered objective accordingly. We verify R2R2 on SPL-native
algorithms like TD7. Furthermore, to demonstrate its orthogonality to prior advancements, we extend the state-of-the-art SimbaV2, which originally lacks SPL, by integrating a tailored SPL module, termed SimbaV2-SPL. Experiments across 11 continuous control tasks confirm that R2R2 effectively mitigates overfitting; specifically, at a UTD ratio of 20, it improves TD7 by ~22% and provides additional gains on top of SimbaV2-SPL, which itself establishes a new state-of-the-art. The code can be found at: https://github.com/songsang7/R2R2
Read the full abstract ▾
All authors Sanghyeob Song, Donghyeok Lee, Jinsik Kim, Sungroh Yoon
Machine Learning · Cross-listed from econ.EM
Regret Equals Covariance: A Closed-Form Characterization for Stochastic Optimization
Irene Aldridge
Quantifying regret typically involves repeatedly re-solving a
stochastic optimization problem across simulated scenarios. Such an approach is costly: the cost scales with the number of scenarios and the cost of each solve. We propose that regret can instead be measured using the joint moments of the cost vector $c$ and the optimal decision $\pi^*(c)$. Working with the optimal concave and Lipschitz under only a feasible region value function $v(c) = \min_{z\in Z} c^\top z$, we show that regret always decomposes as $\mathrm{Regret} = -\mathrm{Cov}(c,\pi^*(c)) - R(c)$ for a residual $R(c)$. We further show that $R(c)=0$ when $\pi^*$ is affine in $c$, which holds for unconstrained and equality-constrained (e.g., budget-constrained Markowitz) quadratic programs and many, but not all, linear programs. For all cases, we give a distribution-free bound $\mathrm{Regret} \le D\sqrt{\mathrm{tr}(\Sigma_c)}$ that depends only on the radius of the feasible region and the cost covariance. We also derive concentration and asymptotic-normality results for the empirical regret, the latter under an explicit and checkable non-degeneracy condition.
Read the full abstract ▾
Machine Learning
Dywave: Event-Aligned Dynamic Tokenization for Heterogeneous IoT Sensing Signals
Tomoyoshi Kimura, Denizhan Kara, Jinyang Li et al. (+6 more)
Internet of Things (IoT) systems continuously collect heterogeneous sensing signals from ubiquitous sensors to support intelligent applications such as human activity analysis,
emotion monitoring, and environmental perception. These signals are inherently non-stationary and multi-scale, posing unique challenges for standard tokenization techniques. This paper proposes Dywave, a dynamic tokenization framework for IoT sensing signals that constructs compact input representations aligned with intrinsic temporal structures and underlying physical events. Dywave leverages wavelet-based hierarchical decomposition, identifies meaningful temporal boundaries corresponding to underlying semantic events, and adaptively compresses redundant intervals while preserving temporal coherence. Extensive evaluations on five real-world IoT sensing
datasets across activity recognition, stress assessment, and nearby object detection demonstrate that Dywave outperforms state-of-the-art methods by up to 12% in accuracy, while improving computational efficiency by reducing input
token lengths by up to 75% across mainstream sequence models. Moreover, Dywave exhibits improved robustness to domain shifts and varying sequence lengths.
Read the full abstract ▾
All authors Tomoyoshi Kimura, Denizhan Kara, Jinyang Li, Hongjue Zhao, Yigong Hu, Yizhuo Chen, Xiaomin Ouyang, Shengzhong Liu, Tarek Abdelzaher
Machine Learning · Cross-listed from cs.CL
Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding
Shuoyang Sun, Chang Dai, Hao Fang et al. (+6 more)
Speculative decoding has become a widely adopted technique for accelerating large
language model (LLM) inference by drafting multiple candidate
tokens and verifying them with a target model in parallel. Its efficiency, however, critically depends on the average accepted length $\tau$, i.e., how many draft
tokens survive each verification step. In this work, we identify a new mechanism-level vulnerability in model-based speculative decoding: the drafter is trained to approximate the target model distribution, but this approximation is inevitably imperfect. Such a drafter-target mismatch creates a hidden attack surface where small perturbations can preserve the target model's visible behavior while substantially reducing draft-
token acceptability. We propose Mistletoe, a stealthy acceleration-collapse attack against speculative decoding. Mistletoe directly targets the acceptance mechanism of speculative decoding. It jointly optimizes a degradation objective that decreases drafter-target agreement and a semantic-preservation objective that constrains the target model's output distribution. To resolve the conflict between these objectives, we introduce a null-space projection mechanism, where degradation gradients are projected away from the local semantic-preserving direction, suppressing draft acceptance while minimizing semantic drift. Experiments on various speculative decoding systems show that Mistletoe substantially reduces average accepted length $\tau$, collapses speedup, and lowers averaged
token throughput, while preserving output quality and perplexity. Our work highlights that speculative decoding introduces a mechanism-level attack surface beyond existing output robustness, calling for more robust designs of LLM acceleration systems.
Read the full abstract ▾
All authors Shuoyang Sun, Chang Dai, Hao Fang, Kuofeng Gao, Xinhao Zhong, Yi Sun, Fan Mo, Shu-Tao Xia, Bin Chen
Machine Learning
Support Before Frequency in Discrete Diffusion
Adrian Müller, Antoine Gonon, Zebang Shen et al. (+2 more)
Discrete
diffusion models are increasingly competitive for
language modeling, yet it remains unclear how their denoising objectives organize learning. Although these objectives target the full data distribution, we show that the exact reverse process induces a hierarchy between coarse support information and finer frequency information. For uniform and absorbing (a.k.a. masking)
diffusion, we prove that, in the small-noise regime of the final denoising steps, each single-
token reverse edit decomposes into a leading scale, determined by whether it moves toward the data support (e.g., grammatically valid sentences), and a finer coefficient, determining relative probabilities within the same scale. Thus, recovering validity structure only requires learning the correct order of magnitude of reverse probabilities, whereas recovering data frequencies requires coefficient-level estimation. The separation is mechanism-dependent: uniform
diffusion exhibits a trichotomy into validity-improving, validity-preserving, and validity-worsening edits, while absorbing
diffusion places its leading-order mass on validity-improving moves. Experiments on a masked
language diffusion model and synthetic regular-
language tasks support these predictions: support-localization emerges earlier than within-support frequency ranking, and the contrast between uniform and absorbing
diffusion matches the predicted rate separation. Together, our results suggest that discrete
diffusion models learn data support before data frequencies.
Read the full abstract ▾
All authors Adrian Müller, Antoine Gonon, Zebang Shen, Ya-Ping Hsieh, Niao He
Machine Learning · Cross-listed from q-fin.CP
Synthetic American Option Pricing via Jump-HMM-Driven Heston Implied Volatility
Julia Sun, Zheyu Jin, Jiawei Zhang et al. (+1 more)
Valuing American options along simulated stock paths requires an implied
volatility (IV) for every option on every date. A stock-return model alone does not provide it. We built a simulator that assigned IV to American options on simulated dates for any chosen stock model. We fitted parametric and neural IV surfaces to vendor option quotes for 31 tickers. Each surface predicted IV from moneyness and time to expiration. Stock paths came from a jump hidden Markov model with capped daily returns. Each day, every option's implied variance moved partway toward its surface prediction, following the form of the Heston variance equation. Random shocks tended to raise IV when the stock fell. A binomial tree converted IV into option values and price sensitivities. Neural surfaces fitted by sector or ticker matched the quotes more closely than one parametric surface. Their errors still varied by date and carried into dollar prices. Repricing identical stock paths under five ways of updating IV changed option values before expiration, and the worst simulated short-position losses at a fixed horizon. Payoffs at expiration did not change. In forecasts of later Goldman Sachs and Eli Lilly option prices,
stochastic IV did little better than fixed IV. Rerunning the forecasts with the realized stock paths pointed to stock prediction as a major source of option-price error. The hidden Markov model and an adaptive-
volatility model predicted stock prices about as accurately as assuming no change, and tuning found no gain from predicting direction. The adaptive-
volatility model improved predicted price ranges for Eli Lilly but not Goldman Sachs. The simulator supported reproducible comparisons of IV and stock assumptions but did not forecast better than simpler alternatives. Complete option histories and prices consistent across strikes are needed before simulated prices can replace
market data.
Read the full abstract ▾
All authors Julia Sun, Zheyu Jin, Jiawei Zhang, Jeffrey D. Varner
Machine Learning
HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts
Tao Zhong, Dongzhe Zheng and Christine Allen-Blanchette
Sparse Mixture-of-Experts (MoE) layers route
tokens through a handful of experts, and learning-free compression of these layers reduces inference cost without retraining. A subtle obstruction blocks every existing compressor in this family: three experts can each be pairwise compatible yet form an irreducible cycle when merged together, so any score that ranks experts on pairwise signals is structurally blind to which triples are jointly mergeable. We show the obstruction is a precise mathematical object, the harmonic kernel of the simplicial
Laplacian on a 2-complex whose vertices are experts, whose edges carry KL merge barriers, and whose faces carry triplet barriers; Hodge-decomposing the edge-barrier signal isolates the kernel exactly. We turn the diagnostic into a selection objective: HodgeCover greedily covers the harmonic-critical edges and triplet-critical triangles, and a hybrid variant of HodgeCover pairs it with off-the-shelf weight
pruning on survivors. On three open-weight Sparse MoE backbones under aggressive expert reduction, HodgeCover matches state-of-the-art learning-free baselines on the expert-reduction axis, leads on the aggressive-compression frontier of the hybrid axis, and uniquely balances retained mass across all four Hodge components. These results show that exposing the harmonic kernel of a learned MoE structure changes which compressor wins at the regime that matters most.
Read the full abstract ▾
Machine Learning
Neural Fields for NV-Center Inverse Sensing
Zhixuan Zhao, Tao Zhong, Yixun Hu et al. (+2 more)
Inverse problems in scientific sensing are often solved with either hand-designed regularizers or supervised networks trained on simulated labels, yet both can fail when the forward model is nonlinear, spectrally coupled, and physically delicate. We study this issue for noise sensing based on nitrogen-vacancy (NV) centers in diamond, where a
quantum sensor measures
magnetic-noise spectra generated by sparse
spin sources. We show that replacing a common scalar/coherent forward approximation with a tensor power-summed dipolar operator changes the inverse landscape and exposes a center-collapse failure mode in free-density optimization. We propose NeTMY, an amortization-free coordinate neural field coupled to the differentiable NV forward model, with annealed positional encoding, multiscale optimization, sparsity/gating, and spectrum-fidelity losses. Across sparse synthetic reconstructions generated by the corrected operator, NeTMY achieves the best localization and distributional metrics in the tested
benchmark. Mechanism experiments show that NeTMY does not directly execute the raw density-space gradient; its parameterization smooths and redistributes updates, mitigating the center-collapse pathology. These results position NV
quantum sensing as a useful testbed for physics-faithful neural inverse problems.
Read the full abstract ▾
All authors Zhixuan Zhao, Tao Zhong, Yixun Hu, Nathalie P. de Leon, Christine Allen-Blanchette
Machine Learning
TabPFN-3: Technical Report
Léo Grinsztajn, Klemens Flöge, Oscar Key et al. (+38 more)
Tabular data underpins most high-value prediction problems in science and industry, and TabPFN has driven the foundation model revolution for this modality. Designed with feedback from our users, TabPFN-3 builds on this foundation to scale state-of-the-art performance to
datasets with 1M training rows and substantially reduce training and inference time. Pretrained exclusively on synthetic data from our prior, TabPFN-3 dramatically pushes the frontier of tabular prediction and brings substantial gains on time series, relational, and tabular-text data. On the standard tabular
benchmark TabArena, a forward pass of TabPFN-3 outperforms all other models, including tuned and ensembled baselines, by a significant margin, and pareto-dominates the speed/performance frontier. On more diverse
datasets, TabPFN-3 ranks first on
datasets with many classes, and beats 8-hour-tuned gradient-boosted-tree baselines on
datasets up to 1M training rows and 200 features. TabPFN-3 introduces test-time compute scaling to tabular foundation models. Our API offering TabPFN-3-Plus (Thinking) exploits this to beat all non-TabPFN models by over 200 Elo on TabArena, rising to 420 Elo on the largest data subset, and outperforms AutoGluon 1.5 extreme while being 10x faster, without using LLMs, real data, internet search or any other model besides TabPFN. TabPFN-3 extends the capabilities of our models, enabling SOTA prediction on relational data (new SOTA foundation model on RelBenchV1) and tabular-text data (SOTA on TabSTAR via TabPFN-3-Plus); and improves existing integrations: a specialized checkpoint, TabPFN-TS-3, ranks 2nd on the time-series
benchmark fev-bench, and SHAP-value computation is up to 120x faster. TabPFN-3 achieves this performance while being up to 20x faster than TabPFN-2.5. In addition, a reduced KV cache and row-chunking scale to 1M rows on one H100 with fast inference speed.
Read the full abstract ▾
All authors Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Mihir Manium, Shi Bin Hoo, Magnus Bühler, Anurag Garg, Dominik Safaric, Jake Robertson, Benjamin Jäger, Simone Alessi, Adrian Hayler, Vladyslav Moroshan, Lennart Purucker, Philipp Singer, Alan Arazi, Julien Siems, Jan Hendrik Metzen, Georg Grab, Nick Erickson, Siyuan Guo, Eliott Kalfon, Simon Bing, David Salinas, Clara Cornu, Lilly Charlotte Wehrhahn, Diana Kriuchkova, Kursat Kaya, Lydia Sidhoum, Marie Salmon, Jerry Chen, Madelon Hulsebos, Yann LeCun, Samuel Müller, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, Frank Hutter
Machine Learning
Towards Resource-Efficient LLMs: End-to-End Energy Accounting of Distillation Pipelines
Katherine Lambert and Sasha Luccioni
The rise in deployment of large
language models has driven a surge in GPU demand and datacenter scaling, raising concerns about electricity use, grid stress, and the impacts of modern AI workloads.
Distillation is often promoted as one of the most effective paths to obtain cheaper, more efficient models, yet these claims rarely account for the full end-to-end energy and resource costs, including crucial teacher-side workloads such as data generation, logit caching, and evaluation. We present a comprehensive energy accounting framework that measures the complete computational cost of
distillation pipelines via detailed stage-wise tracking of GPU device power consumption. In our experiments, we separate and log empirical energy use across distinct phases and systematically measure the energy and emissions of two common
distillation methods: the classic logit-based knowledge
distillation and synthetic-data supervised fine-tuning, constructing energy-quality Pareto frontiers that expose the previously ignored costs. From these measurements and analyses, we derive practical design rules for selecting
distillation methods and hyperparameters under energy and budget constraints, and release an open-source measurement harness and accounting protocol to provide a standardized foundation for comparable, reproducible
distillation research, explicitly accountable for complete pipeline energy impact.
Read the full abstract ▾
Machine Learning · Cross-listed from quant-ph
Winning Lottery Tickets in Neural Networks via a Quantum-Inspired Classical Algorithm
Natsuto Isogai, Hayata Yamasaki, Sho Sonoda et al. (+1 more)
Quantum machine learning (QML) aims to accelerate machine learning tasks by exploiting
quantum computation. Previous work studied a QML
algorithm for selecting sparse subnetworks from large shallow neural networks. Instead of directly solving an optimization problem over a large-scale network, this
algorithm constructs a sparse subnetwork by sampling hidden nodes from an optimized probability distribution defined using the ridgelet transform. The
quantum algorithm performs this sampling in time $O(D)$ in the data dimension $D$, whereas a naive classical implementation relies on handling exponentially many candidate nodes and hence takes $\exp[O(D)]$ time. In this work, we construct and analyze a
quantum-inspired fully classical
algorithm for the same sampling task. We show that our
algorithm runs in time $O(\operatorname{poly}(D))$, thereby removing the exponential dependence on $D$ from the previous classical approach. Numerical simulations show that the proposed sampler achieves empirical risk comparable to exact sampling from the optimized distribution and substantially lower than sampling from the non-optimized uniform distribution, while also exhibiting exponentially improved runtime scaling compared with the conventional classical implementation. These successful dequantization results show that sparse subnetwork selection via optimized sampling can be achieved classically with
polynomial data-dimension scaling on conventional computers without
quantum hardware, providing an alternative to the existing
quantum algorithm.
Read the full abstract ▾
All authors Natsuto Isogai, Hayata Yamasaki, Sho Sonoda, Mio Murao
Machine Learning
WarmPrior: Straightening Flow-Matching Policies with Temporal Priors
Sinjae Kang, Chanyoung Kim, Kaixin Wang et al. (+2 more)
Generative policies based on
diffusion and flow matching have become a dominant paradigm for visuomotor robotic control. We show that replacing the standard Gaussian source distribution with WarmPrior, a simple temporally grounded prior constructed from readily available recent action history, consistently improves success rates on robotic manipulation tasks. We trace this gain to markedly straighter probability paths, echoing the effect of optimal-transport couplings in Rectified Flow. Beyond standard behavior cloning, WarmPrior also reshapes the exploration distribution in prior-space reinforcement learning, improving both sample efficiency and final performance. Collectively, these results identify the source distribution as an important and underexplored design axis in generative
robot control.
Read the full abstract ▾
All authors Sinjae Kang, Chanyoung Kim, Kaixin Wang, Li Zhao, Kimin Lee
Machine Learning
Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
Darius A. Faroughy, Sofia Palacios Schweitzer, Ian Pang et al. (+2 more)
Autonomous
language-model
agents are increasingly evaluated on long-horizon tool-use tasks, but existing
benchmarks rarely capture the complexity and nuance of real scientific work. To address this gap, we introduce
Collider-Bench, a
benchmark for evaluating whether LLM
agents can reproduce experimental analyses from the Large
Hadron Collider (LHC) using only public papers and open scientific software. Such analyses are often difficult to reproduce because the public toolchain only approximates the software used internally by the experimental collaborations, while the published papers inevitably omit implementation details needed for a faithful reconstruction.
Agents must therefore rely on physical
reasoning, domain knowledge, and trial-and-error to fill these gaps. Each task requires the
agent to turn a published analysis into an executable simulation-and-selection pipeline and submit predicted collision event yields in specified signal regions. These predictions are evaluated with standard histogram metrics that provide continuous fidelity scores without a hand-written rubric. We also report the computational cost incurred by each
agent per task. Finally, we evaluate the codebase and full session trace using an LLM judge to catch qualitative failure modes such as fabrications,
hallucinations and duplications. We release an initial set of tasks drawn from LHC searches, together with a containerized sandbox and event simulation tools. We evaluate across a capability ladder of general purpose coding
agents. Our results show that on average no
agent reliably beats the physicist-in-the-loop solution.
Read the full abstract ▾
All authors Darius A. Faroughy, Sofia Palacios Schweitzer, Ian Pang, Siddharth Mishra-Sharma, David Shih
Machine Learning
A Unified Geometric Framework for Weighted Contrastive Learning
Raphael Vock, Edouard Duchesnay and Benoit Dufumier
Contrastive learning (CL) aims to preserve relational structure between samples by learning representations that reflect a similarity graph. Yet, the geometry of the resulting
embeddings remains poorly understood. Here we show that weighted InfoNCE objectives can be interpreted as Distance Geometry Problems, where the weighting scheme specifies the target geometry to be realized by the representation. This viewpoint yields exact characterizations of the optimal
embeddings for several supervised and weakly supervised objectives. In supervised classification, both SupCon and Soft SupCon (a dense relaxation of it where pairs from distinct classes have small non-zero similarity) collapse samples within each class to a single prototype. However, while balanced SupCon recovers the classical regular simplex geometry, class imbalance breaks this
symmetry: SupCon induces non-uniform inter-class similarities depending on class sizes, whereas Soft SupCon preserves a regular simplex geometry regardless of class imbalance. In continuous-label settings, our framework reveals a different failure mode: y-Aware CL generally cannot attain its entropic optimum unless the labels lie on a hypersphere, exposing a mismatch between Euclidean label weights and spherical latent similarity. By contrast, geometrically consistent choices such as Euclidean-Euclidean weighting or X-CLR admit unique optimal
embeddings. Our results show that the choice of weighting scheme determines whether contrastive learning is geometrically realizable, degenerate, or inconsistent, providing a principled framework for designing contrastive objectives.
Read the full abstract ▾
Machine Learning
EMA: Efficient Model Adaptation for Learning-based Systems
Daiyang Yu, Xinyu Chen, Yihan Zhang et al. (+3 more)
Machine learning (ML) is increasingly applied to optimize system performance in tasks such as resource management and network simulation. Unlike traditional ML tasks (e.g., image classification), networked systems often operate in heterogeneous, long-running, and dynamic environment states, where input conditions (e.g., network loads) and operational objectives can shift over time and across settings. Existing learning-based systems offer little support for adaptation, resulting in costly model training, extensive data collection, degraded system performance, and slow responsiveness. This paper presents EMA, the first model adaptation system supporting learning-based systems to adapt to evolving environments with minimal operational overhead. EMA takes a system-driven, data-centric approach that accommodates diverse system and model designs while addressing two key deployment challenges. First, it reduces expensive model training by introducing state
transformers that align the input state of a new environment with previously similar states, allowing models to warm-start adaptation. Second, it addresses the often-overlooked yet costly process of data labeling–collecting ground truth for exploring and training on various system decisions–by prioritizing labeling high-utility data while balancing the tradeoff between training and labeling cost. Evaluations on eight representative learning-based systems show that EMA reduces adaptation costs (e.g., GPU training time) by 14.9-42.4% while improving system performance (e.g., network throughput) by 6.9-31.3%.
Read the full abstract ▾
All authors Daiyang Yu, Xinyu Chen, Yihan Zhang, Yan Liang, Yaqi Qiao, Fan Lai
Machine Learning
EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents
Jiaqi Liu, Xinyu Ye, Peng Xia et al. (+4 more)
Long-term
memory is essential for LLM
agents that operate across multiple sessions, yet existing
memory systems treat
retrieval infrastructure as fixed: stored content evolves while scoring functions, fusion strategies, and answer-generation policies remain frozen at deployment. We argue that truly adaptive
memory requires co-
evolution at two levels: the stored knowledge and the
retrieval mechanism that queries it. We present EvolveMem, a self-evolving
memory architecture that exposes its full
retrieval configuration as a structured action space optimized by an LLM-powered diagnosis module. In each
evolution round, the module reads per-question failure logs, identifies root causes, and proposes targeted configuration adjustments; a guarded meta-analyzer applies them with automatic revert-on-regression and explore-on-stagnation safeguards. This closed-loop self-
evolution realizes an AutoResearch process: the system autonomously conducts iterative research cycles on its own architecture, replacing manual configuration tuning. Starting from a minimal baseline, the process converges autonomously, discovering effective
retrieval strategies including entirely new configuration dimensions not present in the original action space. On LoCoMo, EvolveMem outperforms the strongest baseline by 25.7% relative and achieves a 78.0% relative improvement over the minimal baseline. On MemBench, EvolveMem exceeds the strongest baseline by 18.9% relative. Evolved configurations transfer across
benchmarks with positive rather than catastrophic transfer, indicating that the self-
evolution process captures universal
retrieval principles rather than
benchmark-specific heuristics. Code is available at https://github.com/aiming-lab/SimpleMem.
Read the full abstract ▾
All authors Jiaqi Liu, Xinyu Ye, Peng Xia, Zeyu Zheng, Cihang Xie, Mingyu Ding, Huaxiu Yao
Machine Learning
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
Daniel M. Jimenez-Gutierrez, Enrique Zuazua, Georgios Kellaris et al. (+3 more)
The recent success of large
language models (LLMs) has been largely driven by vast public
datasets. However, the next frontier for LLM development lies beyond public data. Much of the world's most valuable information is private, especially in highly regulated sectors such as healthcare and finance, where data include patient histories or customer communications. Unlocking this data could represent a major leap forward, enabling LLMs with deeper domain expertise and stronger real-world utility. Yet, these data cannot be shared because they are distributed across institutions and constrained by
privacy, regulatory, and organizational barriers. Moreover, institutional
datasets are typically non-independent and identically distributed (non-IID), differing across sites in population characteristics, data modalities, documentation patterns, and task-specific label distributions. In this paper, we demonstrate a practical approach to unlocking private and distributed institutional data for LLM adaptation through
federated collaboration across data silos. Built on the Sherpa.ai
Federated Learning platform, our framework enables nodes to jointly fine-tune a shared LLM without exchanging private data. We evaluate this approach through a cross-domain
benchmark in healthcare and finance, using four closed-ended question answering and classification
datasets: MedQA, MedMCQA, FPB, and FiQA-SA. We compare three parameter-efficient fine-tuning (PEFT) strategies-LoRA, QLoRA, and IA3-across pretrained backbones under non-IID settings reflecting institutional data heterogeneity. Our results show that
federated fine-tuning performs close to centralized training and outperforms isolated single-institution learning. From a Green AI perspective, QLoRA and IA3 improve efficiency with limited accuracy degradation, supporting
federated PEFT as a viable approach for adapting LLMs where data cannot be shared.
Read the full abstract ▾
All authors Daniel M. Jimenez-Gutierrez, Enrique Zuazua, Georgios Kellaris, Joaquin del Rio, Oleksii Sliusarenko, Xabi Uribe-Etxebarria