Machine Learning
Koa-decision: Fast and Consistent Structured Decision Making with Generative LLMs
Shenghong Dai, Shiva Kumar Pentyala, Yingchi Liu et al. (+6 more)
Industry applications often demand low-latency classification, yet current large
language model (LLM) approaches remain poorly suited for latency-critical applications. Existing prompting and constrained decoding produce verbose, multi-
token outputs that require expensive
token-by-
token generation, while encoder-based models achieve faster inference but sacrifice task flexibility. We propose Koa-decision, a framework for low-latency atomic actions – fast, single-step decisions such as classification, semantic endpointing, Boolean checks, and scoring – formulated as constrained generation with single-
token outputs. By introducing atomic label
tokens and applying supervised fine-tuning, our method reduces classification to a deterministic one-step decoding problem. Across standard
benchmarks, Koa-decision delivers competitive accuracy with consistently low and stable latency. On a production intent-routing
benchmark, Koa-decision reaches 85.5% accuracy – competitive with the strongest frontier models (Claude-4.8-Opus, Gemini-Pro-3.1) and ahead of GPT-5 and Gemini-2.5-Pro – while answering in about half a second, several-fold faster than every frontier model (up to ~7.5x at the median) under identical serving conditions. Against the dedicated single-
token system Jev/TypeSafe, Koa-decision is competitive on accuracy and faster at the median, while also handling
multimodal inputs and multi-label outputs that single-label text systems do not.
Read the full abstract ▾
All authors Shenghong Dai, Shiva Kumar Pentyala, Yingchi Liu, Shubham Mehrotra, Suman Banerjee, James Zhu, Bin Bi, Sitaram Asur, Phil Mui
Machine Learning
LoopICL: Looping a single transformer block to solve tabular tasks
Amir Rezaei Balef and Katharina Eggensperger
Tabular foundation models using in-context learning have recently surpassed gradient-boosted trees on predictive tabular tasks. However, recent mechanistic insights suggest that parameters in these models are largely redundant. We introduce LoopICL, a looped
transformer whose core design decouples parameter count from computational depth. LoopICL consists of a single block, processing data through two coupled streams: a
cell stream capturing per-
cell feature representations and a row stream capturing in-context example representations, jointly refined through within-column and cross-column
attention. During pre-training, we vary loop counts, allowing the block to be unrolled for a varying number of iterations at test-time and use a learned exit-gate to automatically exit. In its standard setting, LoopICL performs competitively with TabICLv2 on TabArena and TALENT at the same computational cost (FLOPs), while using nearly 90% fewer parameters. Furthermore, its recurrent design enables users to also trade off inference cost and performance, providing a resource-aware TFM.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
Fundamental Limits of Transferability and Equivariance in Algebraic Signal Models I: Finite Dimensions
Alejandro Parada-Mayorga
We study the fundamental limits of transferability in algebraic signal processing through homomorphisms between algebraic signal models. Homomorphisms are linear maps between the signal spaces of two models that commute with filtering, so filtering a signal and transferring it across domains can be done in either order. The existence of such maps is governed entirely by coincidences among the filtered
eigenvalues of the two models' shift operators, but existence alone is insufficient: the space of homomorphisms always contains trivial elements that destroy all information. We introduce the spectral transfer efficiency $\eta(\theta)\in[0,1]$ to quantify information-preserving quality, prove that every homomorphism decomposes into unconstrained blocks over coincidence classes, derive the dimension of the homomorphism space, and characterize exactly when lossless transfer is achievable. Beyond normal shift operators, we quantify a departure-from-normality penalty and show how filter derivatives can repair spectral defectiveness. The theory yields concrete consequences in three settings: for sampling,
eigenvalue interlacing converts transferability under subsampling into an explicit filter design constraint; for compressed sensing, $\eta(\theta)$ controls the restricted isometry constant and the coherence of the resulting measurements, and recovery decouples across coincidence classes; and for machine learning, spectral aliasing emerges as the controlled
symmetry breaking that makes transfer between mismatched domains possible at all.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Aditya Sharma and Divya Saxena
Contrastive
vision-
language models learn shared
embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal
alignment, whereas removing gap-related structure can degrade image-text
retrieval. In this paper, we provide a unified
geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal
retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal
retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based
vision-
language systems across evaluated downstream tasks.
Read the full abstract ▾
Machine Learning
Data Unlearning via Inverse Distillation
Aleksei Leonov, Nikita Kornilov, Zhenhe Zhang et al. (+3 more)
Multi-step matching models, including flow and
diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training
datasets. We introduce Inverse
Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate
distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10
datasets under flow-matching and score-based
diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and
distillation in unconditional flow-matching and score-based models.
Read the full abstract ▾
All authors Aleksei Leonov, Nikita Kornilov, Zhenhe Zhang, Evgeny Burnaev, Iaroslav Koshelev, Alexander Korotin
Machine Learning
PHASE: A Physiology-Guided Hierarchical Foundation Model for Intracranial EEG
Yipeng Zhang, Chenda Duan, Yuanyi Ding et al. (+10 more)
Clinicians and neuroscientists have long analyzed intracranial electroencephalography (iEEG) through directly measurable physiological characteristics, which carry much of the information that downstream tasks depend on. Recent iEEG foundation models learn by reconstructing or predicting their inputs, which leaves the retention of these characteristics implicit. They are also evaluated mainly on cognitive decoding and a narrow clinical task, i.e., seizure detection. On a broad, clinically relevant
benchmark such as Omni-iEEG, they remain below task-specific models when used frozen. We introduce PHASE, a physiology-guided foundation model that makes these characteristics explicit learning targets, pairing them with masked latent prediction in a temporal stage (PHASE-T) within each channel and a spatiotemporal stage (PHASE-ST) across synchronized channels. PHASE is pretrained on heterogeneous recordings from 222 participants at nine clinical sites. On all five Omni-iEEG clinical tasks, frozen PHASE-T outperforms every evaluated foundation model by up to 31%, and fine-tuned PHASE-T surpasses the task-specific models, setting a new state of the art. PHASE-T benefits from physiological supervision, outperforming variants trained with latent prediction alone or auxiliary waveform reconstruction on every task in matched ablations. PHASE-T generalizes to unseen institutions, outperforming the compared models with few or no local labels. PHASE-ST further improves seizure-onset-zone identification over PHASE-T and, when frozen, decodes sound volume and pitch on BrainTreebank better than published models. Beyond task performance, PHASE learns to encapsulate the physiological characteristics clinicians recognize, from seizure onset and its propagation to anatomical region identity, even though its pretraining contains no ictal recordings or anatomical labels.
Read the full abstract ▾
All authors Yipeng Zhang, Chenda Duan, Yuanyi Ding, Tianyi Wang, Atsuro Daida, Masaki Izumi, Yuta Tanoue, Naoto Kuroda, Shaun A. Hussain, Nishant Sinha, Eishi Asano, Hiroki Nariai, Vwani Roychowdhury
Machine Learning
Early Learning Shapes Later Directions Of Representation Change In Continual Learning
Yuantao Deng, Jinnuo Liu, Kaizhen Tan et al. (+1 more)
Representations continually change as a network learns new tasks. We ask whether early representational changes naturally form a
geometric structure that continues to shape later learning. We identify a low-dimensional subspace of early representation drift, which we call a scaffold, and test whether it is reused across subsequent tasks. Across four pretrained visual encoders and two
datasets, later representational changes consistently favor this early-defined subspace over matched random alternatives. This reuse is history-dependent: when networks experience different early tasks but identical later training inputs, each network preferentially reuses the scaffold induced by its own learning history. The same preference appears in individual optimizer updates, even though the network's dominant local response directions shift away from the original scaffold. Finally, constraining motion within the scaffold slows new-task acquisition more than matched random constraints, while effects on old-task retention are less consistent. In summary, these results suggest that early experience leaves a persistent
geometric imprint on how neural networks adapt to future tasks.Code is available at https://github.com/YuantaoDeng/latent-scaffold.
Read the full abstract ▾
All authors Yuantao Deng, Jinnuo Liu, Kaizhen Tan, Yuchen Liu
Machine Learning
GEM-KMeans: Memory-Efficient and Accurate Clustering on Massive Scale with GPU Optimization
Peng Xu, Nihar Koganti, Volodymyr Kindratenko et al. (+1 more)
Memory-efficient scaling on clustering problems without sacrificing statistical accuracy is of central interest for large-scale data analysis and machine learning problems. Nonnegative low-rank (NLR)
matrix factorization for $K$-means is a scalable clustering method, which connects to semidefinite relaxations with optimal average-case exact recovery guarantees. However, a direct GPU implementation of NLR requires multiple large factor-sized buffers and substantial data movements that are essentially
memory-bound. In this paper, we introduce GEM-KMeans, a spectrally normalized yet mathematically equivalent NLR formulation that fuses the gradient update, nonnegative projection, and sufficient statistics for normalization and iterate movement into a
matrix-multiplication epilogue. Instead of retaining three massive factor-sized arrays, our IO-aware GPU implementation materializes only one single factor with small tile-reduction arrays as additional storage in the High Bandwidth
Memory (HBM). We derive explicit
memory costs and spectrally normalized smoothness bounds for optimizing the clustering objective function. Accurate clustering is demonstrated at massive scales on synthetic and real
datasets, where performance gains of GEM-KMeans over existing GPU-accelerated Lloyd's
algorithms involve data-dependent runtime tradeoffs.
Read the full abstract ▾
All authors Peng Xu, Nihar Koganti, Volodymyr Kindratenko, Xiaohui Chen
Machine Learning · Cross-listed from cs.DC
Mixture-of-Kittens: MoE Megakernel for NVL72s
Stuart H. Sul, Nash Brown, Henry Wildermuth et al. (+3 more)
AI accelerator systems are rapidly consolidating into scale-up architectures, where tens to thousands of GPUs communicate over high-bandwidth, single-hop fabrics. We find that existing Mixture-of-Experts (MoE) training systems, optimized for conventional scale-out networks, transfer poorly to this setting, often running slower than a naive baseline built with PyTorch and NCCL. With industry roadmaps pointing toward even larger scale-up domains, understanding the performance tradeoffs of this hardware regime is increasingly important. We present Mixture-of-Kittens (MoK), an MoE training system designed for Nvidia NVL72. MoK builds on three insights that unlock performance on scale-up domains: (1) choosing push- or pull-based communication per operator, (2) restructuring the computation-communication overlap, and (3) fully eliminating CPU-GPU synchronization. MoK distills these insights into a single deterministic training megakernel that fuses
token dispatch, shared and routed expert FFNs, and
token combine. Across MoE layer shapes from four widely used open-weight models, MoK delivers up to $2.37\times$ the throughput of the strongest publicly available baseline. In a production run on 512 GPUs spanning multiple GB300 NVL72 racks, MoK improves end-to-end training throughput by $1.41\times$.
Read the full abstract ▾
All authors Stuart H. Sul, Nash Brown, Henry Wildermuth, William Lin, Federico Cassano, Christopher Ré
Machine Learning
FLOORA: A Human-Aligned Domain-Specific Language Model for Architectural Design
Sahand Rezaei-Shoshtari, Patryk Wozniczka, Shu Ishida et al. (+14 more)
Foundation models are powerful generators, but many engineering domains require structured representations that general-purpose systems handle poorly. We introduce FLOORA (Floor Layout Optimization with RL
Alignment), a family of small domain-specific
language (DSL) models for architectural layout generation. With specialized data and
alignment, our 0.6B model outperforms much larger frontier models, achieving VLM judge win rates up to 92.0% on out-of-distribution real-world buildings and 96.0% on synthetic buildings. Human evaluations further corroborate these results, with FLOORA selected as the best model in 89.3% of evaluations. FLOORA combines a
token-efficient DSL, custom tokenization, domain-specific pretraining, supervised fine-tuning (SFT), and reinforcement learning (RL) with learned human-preference and verifiable rewards. This pipeline improves architectural and
geometric validity, supported by extensive empirical evaluation and ablation studies. Although focused on architecture, our results suggest that similar domain-specific recipes may be useful in other engineering domains with structured, verifiable outputs.
Datasets, models, and inference code are available at https://github.com/AutodeskAILab/floora.
Read the full abstract ▾
All authors Sahand Rezaei-Shoshtari, Patryk Wozniczka, Shu Ishida, Gregg Streuber, Farnoosh Javadi, Jeffrey Landes, Angela Ju, Muhammad Azam, Bryan Lim, Johan Luttun, Indrajeet Haldar, Jonathan Shaw, Beatriz Guerra, Ivan Sosnovik, James Stoddart, Robert Giaquinto, Adam Gaier
Machine Learning
Understanding Decision-Making Mechanisms in Neural Routing Solvers
Fatemeh Askari, Mazdak Teymourian, Mohammad Izadi et al. (+1 more)
Neural Combinatorial Optimization (NCO) has achieved strong empirical success, yet the internal mechanisms driving model decisions remain largely unexplored. In this paper, we investigate three representative autoregressive NCO models spanning two encoder-decoder configurations: AM and POMO (heavy-encoder, light-decoder), and LEHD (light-encoder, heavy-decoder). Through behavioral analyses, representation probing, and causal interventions, we examine how these models construct solutions and use internal representations during decoding. Our results suggest that AM and POMO predominantly follow a persistent
geometric pattern throughout solution construction, whereas LEHD contains linearly accessible information about multiple future actions. Causal experiments further provide evidence for the role of future-node representations in LEHD's decision-making. We also observe that LEHD relies strongly on the current-node representation for immediate local decisions, while the start-node representation plays a broader navigational role over the subsequent route. Cross-instance
alignment analyses additionally indicate that LEHD maps current-node representations into a relatively shared latent region, which may provide a stable reference for evaluating subsequent decisions. Across the Traveling Salesman Problem and the Capacitated Vehicle Routing Problem, these results reveal distinct decision-making patterns across these architecturally distinct solvers and provide a foundation for more interpretable analyses of NCO solvers. Code and additional visualizations are provided in the https://github.com/NCO-Interpretability/NCO-Interpretability.
Read the full abstract ▾
All authors Fatemeh Askari, Mazdak Teymourian, Mohammad Izadi, Mahdieh Soleymani Baghshah
Machine Learning
Introducing the CZAR Loss: A Tailored Objective Function for Financial Log-Return Predictions
Joel Pfeffer, J. M. Diederik Kruijssen, Florian Stecker et al. (+1 more)
In quantitative finance, standard regression losses are misaligned with the economics of return prediction. As the conditional mean of financial log-returns is close to zero, symmetric losses such as the mean squared and mean absolute errors make the constant zero forecast a near-optimal solution, penalizing models with genuine but noisy directional skill. This applies both during training, where predictions shrink toward zero, and during evaluation, where trivial forecasters can lead loss-based rankings. Under a Gaussian linear prediction model, we show that all symmetric monotonic losses share a universal breakeven directional accuracy against the zero predictor, which rises sharply and becomes unobtainable as the prediction noise approaches the standard deviation of the returns. We introduce the CZAR (Composite Zero-Agnostic Return) loss function, a piecewise quadratic loss built around five key requirements: convexity in the prediction, asymmetry with true return direction that vanishes at zero, near-linear penalization of undershoots and wrong-direction predictions, divergence for large errors, and an adaptive loss floor for evaluation. CZAR is provably convex in the prediction at fixed true value, has closed-form gradient and Hessian suitable for custom objectives in gradient-boosted libraries, and its four hyperparameters reduce to a single choice through correlated defaults. In idealized tests, the minimum directional accuracy required for a CZAR-evaluated forecaster to outperform the zero predictor under mean log loss remains near the 50% chance level, whereas the corresponding threshold for symmetric losses rises sharply with prediction noise. This advantage persists under heavy-tailed return distributions. In a LightGBM experiment on intraday BTC log-returns, CZAR-trained models reduce the `zero-returns bias' and improve directional accuracy on large-magnitude returns.
Read the full abstract ▾
All authors Joel Pfeffer, J. M. Diederik Kruijssen, Florian Stecker, Steven N. Longmore
Machine Learning
ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
Hsiao-Ru Pan, Florent Draye and Bernhard Schölkopf
Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage
estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage-value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic
algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical
reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Mirror-Score: Calibrated, Inference-only Scoring Exposes the Limits of Sequence-compatibility Ranking in D-peptide Design
Jiada Li
D-peptides combine protease resistance with high target specificity, but computational design of D-peptide binders remains immature. Mirror-Peptidizer introduced an in silico mirror-image screening pipeline using target reflection, backbone generation, and ProteinMPNN sequence design, but its raw ProteinMPNN negative log-likelihood (NLL) ranking was not validated against measured affinities, and only 4 of 9 tested MDM2 designs bound detectably. We introduce Mirror-Score, a calibrated, inference-only scoring framework for heterochiral D-peptide/L-
protein complexes, and a public
benchmark of 31
crystal complexes across four target families, including 18 with literature-verified affinities. Raw ProteinMPNN NLL is not a valid affinity ranker: its pooled Spearman correlation with affinity is 0.19, and correlations reverse between MDM2/CHIP (+0.62) and gp41 (-0.70). We therefore evaluate Boltz-2 mirror-space cofolding confidence. For the complete viral-entry family (7 structures representing 3 peptides), interface predicted local distance difference test (pLDDT) achieves structure-level leave-one-out Spearman rho = 0.90 (p = 0.006) and correctly orders all three peptides by affinity, whereas NLL fails (structure-level rho = 0.18). Because only three independent chemotypes are represented, this result indicates directional consistency rather than a statistically validated predictor. Cross-family calibration does not transfer at current sample sizes, supporting family-matched calibration as the practical deployment mode. We also specify a prospective design protocol for the antimicrobial-resistance targets LasR and LecB from Pseudomonas aeruginosa, including mirrored structures, ligand-derived hotspot maps,
diffusion-model-ready inputs, and Mirror-Score ranking. Code,
benchmark data, structures, and analysis scripts are openly available at https://github.com/Jiadalee/Mirror-Score.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
PowerZooJax: A JAX-based Power System Benchmark for Reinforcement Learning
Zhanhua Pan, Xiao Liu, Zhilong Cao et al. (+2 more)
Power system operation is a safety-critical sequential decision-making problem, making it a natural testbed for reinforcement learning (RL). However, existing RL environments for power systems are often narrow in scope and computationally limited by CPU-based simulation workflows, making large-scale evaluation difficult. We introduce PowerZooJax, a JAX-based
benchmark suite for RL in power system operation. It provides five constrained Markov decision process tasks spanning generation, transmission, distribution, distributed energy resources, and data center microgrid. By rewriting power flow, economic dispatch,
market clearing, and device dynamics as JAX computation graphs, PowerZooJax keeps the entire training and evaluation loop on the GPU. Experiments show substantial speedups over CPU-based simulations and demonstrate standardized evaluation of policy returns, safety violations, and out-of-distribution stress conditions. Our open-source
benchmark is available at: https://github.com/powerzoojax/PowerZooJax.
Read the full abstract ▾
All authors Zhanhua Pan, Xiao Liu, Zhilong Cao, Jianhong Wang, Dawei Qiu
Machine Learning
Improving scalable oversight with co-trained monitors
Joseph H. Rudoler, Kevin Tan, Benedict Tessler et al. (+2 more)
Worker-monitor setups are a promising approach to AI oversight, but training workers against fixed monitors can incentivize monitor evasion. We study whether this failure mode can be avoided by co-training the monitor alongside the worker, and explore both supervised and self-supervised approaches. In the supervised setting, we prove a characterization: monitoring is possible with vanishing error and query rates exactly when the class of possible monitor functions has finite Littlestone dimension. This connects worker monitoring with an established literature on
adversarial online learning. For self-supervision, we propose a co-training procedure based on test-time
distillation: the monitor uses additional test-time compute to generate training labels, then trains its standard-compute policy on those labels. For majority-vote labels, we give a finite-sample sharpening guarantee under adaptive worker distributions with action coverage, that shows that the monitor's verdicts converge to its initial modal verdicts. We stress-test the former in code-security settings where the worker is trained adversarially to fool the monitor. Our results suggest that adaptive monitors are better at keeping pace with evolving worker strategies, while fixed monitors are more vulnerable to evasion.
Read the full abstract ▾
All authors Joseph H. Rudoler, Kevin Tan, Benedict Tessler, Timothy Kong, Enric Boix Adserà
Machine Learning
Neural networks for spectral optimization
Alexis de Villeroché, Beniamin Bogosel, Stéphane Breuils et al. (+2 more)
Given a functional dependent on the spectrum of a differential operator, we address the problem of finding a domain which optimizes this functional. PDE solvers might be used to tackle this optimization. It is however computationally expensive. We propose two neural network models which learn the spectrum directly from the geometry of the domain and can be used to optimize the domain from one or more
eigenvalues. We investigate two representations. The first encodes the domain through
Fourier coefficients and a light MLP, which is efficient on
star-shaped geometries, achieving a precision of 0.2%. Through a rescaling of the coefficients the designed models satisfy the scaling law of the
eigenvalues. Additionally, averaging the outputs of the trained surrogates over rotations and reflections induces invariance for these transformations. The second is a model that takes the landscape function, the indicator function and the gradient of the landscape function. A Gram-Schmidt process produces orthogonal eigenfunctions as output of the model along with the associated
eigenvalues. The landscape model reaches 1% mean relative error on the first ten
eigenvalues, compared with 4% for an FNO model. Replacing the landscape by an SDF worsened both prediction and optimization errors. The trained model also generalizes from synthetic shapes to domains given as classical image
dataset. The resulting surrogates of both approaches recover classical spectral optima such as the disk for the first
eigenvalue or the conjectured minima of higher
eigenvalues. This confirms that our models produce accurate differentiable estimates of
eigenvalues, which can be used in shape optimization problems involving spectral quantities.
Read the full abstract ▾
All authors Alexis de Villeroché, Beniamin Bogosel, Stéphane Breuils, Dorin Bucur, Jacques-Olivier Lachaud
Machine Learning · Cross-listed from hep-ph
Hybrid Neural Simulation-Based Inference for Robust Applications and Limited-Budget Scenarios
Sean Benevedes, Mani Dehghan, Aishik Ghosh et al. (+1 more)
We develop two hybrid techniques that approach the performance of neural simulation-based inference (NSBI) analyses while substantially reducing the computational cost of inference and preserving some or all of the reliability guarantees of parametric methods. The first approach is broadly applicable, while the second is tailored to a class of particle physics analyses that admit a semi-parametric NSBI formulation. With only a modest compromise in raw sensitivity, these methods represent an important step toward computationally efficient NSBI in offline analyses and also open the door to the exploration of trigger-level applications in the future. Based on our comparison studies, we recommend the use of our first approach, Latent Categories, for robust and efficient inference.
Read the full abstract ▾
All authors Sean Benevedes, Mani Dehghan, Aishik Ghosh, Tae Hyoun Park
Machine Learning · Cross-listed from cs.CR
Multi-Class, Multi-Tier Network Intrusion Detection: A Comprehensive and Reproducible Benchmark
Yufeng Xin, Bryant Goseland and Mohamed Rahouti
Machine learning (ML) and deep learning (DL) have dominated Intrusion Detection System (IDS) research in recent years. Unfortunately, many existing studies have produced inflated results and unreliable
benchmarks due to critical oversights and mistakes in the ML and DL pipeline, from data collection and labeling to feature engineering and model training and evaluation. CIC-IDS2017 is a standard
benchmark for network intrusion detection. Still, many published results on this
dataset are difficult to compare due to labeling errors, inconsistent flow extraction, potential leakage, and performance evaluation metrics dominated by benign traffic. In this paper, we present a comprehensive
benchmark with corrected PCAP-level labeling and a complete evaluation pipeline with diverse ML models. We evaluate eleven tabular classifiers at three nested levels: binary attack detection, nine-class attack-family attribution, and fifteen-class fine-grained classification. A soft-voting ensemble of Random Forest, XGBoost, and LightGBM obtains the best fine-tier macro-F1 of 0.955, with coarse and binary macro-F1 scores of 0.980 and 0.999, respectively. We further conducted a feature selection study based on an analysis of feature importance. This comprehensive
benchmark pipeline is configurable and open-source, enabling new feature extraction and model plugins for new
datasets. Future work should use this pipeline as a reference point for richer features, rare-class analysis, and model generalization towards new
datasets and attack classes.
Read the full abstract ▾
Machine Learning · Cross-listed from math.OC
Dual-Anchor Acceleration Is Near-Optimal for Stochastic Monotone Root-Finding
TaeHo Yoon and Nicolas Loizou
Among distinct optimal acceleration mechanisms for deterministic monotone root-finding problems and fixed-point problems, dual-anchoring has recently been shown to admit a more robust direct
stochastic extension than standard anchor acceleration. However, without additional strong monotonicity, the existing
stochastic dual-anchoring guarantee has two limitations: first, it requires cocoercivity in expectation, and second, it attains only $O(\epsilon^{-3})$ oracle complexity, leaving a gap to the near-optimal $\tilde{O}(\epsilon^{-2})$ complexity achieved by other methods. In this work, we address both of these limitations by combining dual-anchoring with
stochastic resolvent approximation and optimized variance control. For unbiased
stochastic oracles with variance bounded by $\sigma^2$, where sample operators are monotone and uniformly $L$-Lipschitz, our
algorithm finds a point with $\epsilon$-residual with a near-optimal oracle complexity of $O ( (LD / \epsilon) \ell + (\sigma^2 / \epsilon^2) \ell^2)$, where $\ell = \log (1 + LD / \epsilon)$ and $D$ is the initial distance to a solution. This result improves the best known oracle complexity in the noise-dominated regime under these samplewise assumptions, reducing the poly-logarithmic factor from cubic to quadratic.
Read the full abstract ▾
Machine Learning
TORQUE: Optimizing What (not) to Quantize Before and After Rotation
Ran Ben Basat, Michael Mitzenmacher and Shay Vargaftik
Uniform random rotations are an effective preprocessing step for quantization: they make normalized coordinate distributions approximately Gaussian, enabling the use of codebooks optimized offline. We introduce TORQUE, a framework that improves on previous quantization works that use random rotations by jointly optimizing how many and which coordinates to preserve at high precision both before and after rotation, under a fixed overall expected bit budget. Intuitively, before rotation, preserving large input coordinates at high precision can reduce overall error by preventing the rotation from spreading their values across many coordinates. Likewise, after rotation, preserving a small fraction of the largest-magnitude coordinates at high precision allows the remaining values to be quantized more accurately using codebooks optimized offline for the resulting truncated Gaussian distribution. We derive a quantization error upper bound and prove that top-$k$ pre-rotation retention minimizes it for each $k$. This reduces the search over coordinate subsets to an optimization over $k$, enabling a fast optimizer that uses offline codebooks and parallel parameter selection for practical implementation. We demonstrate an improved tradeoff between reconstruction accuracy and storage cost through numerical evaluation under the Gaussian model and experiments on nearest-neighbor
retrieval, KV-cache compression, and activation compression.
Read the full abstract ▾
Machine Learning
Preferent Compression Bounds Are Tight
Dario Paccagnan and Marius Tirlea
The lack of rigorous safety and performance certificates remains a key bottleneck to the deployment of modern learning-based methods. Sample compression has recently emerged as a powerful tool for deriving such certificates, with particularly sharp bounds available for
algorithms satisfying a so-called preference property – also known as stability in the learning theory literature. These bounds find direct application across domains as different as the Scenario Approach, Pick-to-Learn, and Support Vector methods. However, whether they are tight has remained an open problem. In this paper we resolve this question affirmatively and show that the state-of-the-art bound for preferent compressions is provably tight. We establish this by exhibiting an explicit construction based on the uniform distribution and order statistics that attains the bound in the limit. Along the way, we also provide a considerably shorter and more accessible proof of this bound, requiring only elementary counting arguments and no infinite-dimensional duality.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
In-Context Learning for Robots: Methods and Applications
Haojian Huang, Zexi Li, Junhao Guo et al. (+36 more)
General-purpose
robots must infer what a new task requires and translate that understanding into appropriate physical action. In-context learning (ICL) for
robots supports this process by using demonstrations and interaction to direct existing competence with neural parameters held fixed during deployment. We organize this literature review around the interfaces connecting contextual evidence to execution, distinguishing four families: context-conditioned policies,
geometric demonstration transfer, world-model-based control, and skill- and
agent-based execution. Comparing these interfaces clarifies their transfer assumptions and the roles of training, correspondence, and
memory in making context useful. Across manipulation and navigation, we examine how these mechanisms preserve taught requirements as objects, environments, and execution conditions change. This analysis links method design to evaluation practices that distinguish responsiveness to teaching, physical transfer, and benefits from retained experience. The resulting agenda connects compositional task acquisition and faithful transfer with physical recursive self-improvement, in which experience improves the ability to learn subsequent tasks.
Read the full abstract ▾
All authors Haojian Huang, Zexi Li, Junhao Guo, Yehang Zhang, Wenxuan Peng, Bohan Zhou, Weilin Ruan, Leyi Wu, Chenxu Wang, Jianchong Su, Binghui Xie, Wosong Chen, Yingjie Xu, Tianhao Zhou, Suzeyu Chen, Pukun Zhao, Jiaqi He, Xinyi Li, Runze Li, Peiran Dong, Shaoxiang Dang, Jing Huang, Yingbing Chen, Yifan Chang, Tianyi Zhang, Shiyuan Deng, Haozhi Wang, Yangkai Wei, Wenqian Li, Han Yang, Kaiwen Zhou, Huaping Liu, James Cheng, Rui Shao, Donglin Wang, Yaochu Jin, Jianye Hao, Ying-Cong Chen, Yinchuan Li
Machine Learning
Making Cross-Continental Federated Learning Repeatable with FLIP: a Multi-Application Study
Rafael Garcia-Dias, Alexandre Triay Bagur, Chayanin Tangwiriyasakul et al. (+20 more)
Federated learning (FL) in healthcare remains challenging, as the overhead of rebuilding governance guarantees for every collaboration stops most projects at the proof-of-concept stage. Here we present FLIP (
Federated Learning Interoperability Platform), an open-source, multi-application platform that makes FL training and evaluation repeatable. FLIP implements common FL workflows as a set of composable services: cohort queries against per-site structured databases, on-demand DICOM
retrieval from institutional PACS, per-site project approval, and reusable FL job types. To demonstrate FLIP, we ran two distinct use cases,
federated fine-tuning and
federated evaluation, on synthetic chest X-ray cohorts across two client nodes based in the United Kingdom (UK) and Thailand. In FLIP, each institution independently approves its participation in each project and operates its own node under local IT security processes. This study makes an operational rather than an algorithmic claim. It does not compare
federated with centralised training; for that question, we refer the reader to existing systematic reviews and meta-analyses. The central result is evidence that such platforms enable international FL collaboration and improve repeatability, auditability, and site-specific governance. We also present a comprehensive comparison of existing platforms to help researchers and operators choose the right platform for their use case.
Read the full abstract ▾
All authors Rafael Garcia-Dias, Alexandre Triay Bagur, Chayanin Tangwiriyasakul, Virginia Fernandez, Parhom Esmaeili, Piyalitt Ittichaiwong, Yang Li, Lawrence Adams, Wason Buncharoen, Martin Chapman, Benjamaporn Chayanond, Sadthavud Chunrod, Tanawat Fongsri, Kass Gibson, Supat Plungprasertkul, Supawit Tangpanithandee, Kanyakorn Veerakanjana, Vicky Goh, Michela Antonelli, Joe Zhang, Kongkiat Kespechara, Sebastien Ourselin, M. Jorge Cardoso
Machine Learning · Cross-listed from cs.CL
Causal and Interpretable Structures in LLM Compositional Tasks
Gurbir Arora, Toni J.B. Liu, Jiajun Bao et al. (+2 more)
Large
language models are able to solve tasks whose answers depend on not only individual input
tokens, but also on relations among them. How is such relational information represented and processed across
transformer layers? We study activations from ensembles of
prompts that require inferring relationships between three
tokens corresponding to a cyclic concept (months, hours, weekdays, and musical notes) to correctly predict the next
token. Across model families (Llama, Qwen, Gemma, and Mistral) and cyclic concepts, we find a consistent layerwise progression in how the joint dependence among the
tokens is geometrically organized and causally used: intermediate layers use a joint representation based on the inferred relationship between two
tokens, while later layers use a joint representation associated with all three
tokens to correctly complete the task. We also find other relationships between
tokens that are geometrically structured but remain causally inert in the next-
token prediction. Crucially, when taken together, these
geometric and causal investigations reveal the representation-level mechanism that progressively organizes and composes the relational information to form the answer. More surprisingly, restricting the models to such causally relevant joint representations improves next-
token prediction accuracy.
Read the full abstract ▾
All authors Gurbir Arora, Toni J.B. Liu, Jiajun Bao, Raphaël Sarfati, Christopher J. Earls
Machine Learning · Cross-listed from astro-ph.IM
SIFARI: Self-Supervised Interferometric Fitting for Astronomical Radio Imaging
Shunyuan Mao, Andrea Isella, Paris Perdikaris et al. (+2 more)
Radio-interferometric images are reconstructed from sparsely sampled visibilities, and CLEAN-based imaging can struggle with spatial filtering, complex morphologies, and uncertainty quantification. Alternative methods that fit visibilities directly can address some of these limitations but often require manual choices of image priors and model hyperparameters. We present SIFARI (Self-Supervised Interferometric Fitting for Astronomical
Radio Imaging), a self-supervised neural network workflow that represents sky brightness as a continuous function of position and fits measured visibilities without an external image training set or explicit spatial regularizer. An empirical rule sets the
Fourier feature scale from the visibilities before training, controlling how readily the network fits fine structure. Sampling network weights with
Stochastic Weight Averaging-Gaussian (SWAG) gives approximate brightness uncertainty estimates, which we combine with a thermal-noise floor to construct spatially resolved signal-to-noise maps. In synthetic ALMA tests, SIFARI yields an effective point-source response about eight times narrower than the natural-weighting CLEAN restoring beam and recovers more extended flux than CLEAN when short baselines are missing. It also achieves higher image fidelity than the restored CLEAN images in all three morphology
benchmarks. Applied to ALMA observations of PDS 70, SIFARI recovers the bright outer ring together with faint compact emission in the central cavity. For long-baseline-only WISPIT 2 data, SIFARI supplies a sky model for phase self-calibration where the CLEAN model is inadequate. The restored, self-calibrated SIFARI image has approximately 30% lower RMS noise than the CLEAN image made from the original visibilities without self-calibration.
Read the full abstract ▾
All authors Shunyuan Mao, Andrea Isella, Paris Perdikaris, Li-Ta Lo, Hui Li
Machine Learning
ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
Zhiwei Zhang, Huayu Deng, Fei Zhao et al. (+4 more)
Large
language model post-training generates self-generated rollouts through reinforcement learning and on-policy
distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We introduce ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations. Across domain-specific reinforcement learning, multi-teacher on-policy
distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering. On Qwen3.6-35B-A3B, ROSS improves the six-
benchmark MOPD average from 58.40% to 62.20% and SWE-bench Verified from 64.20% to 68.40%. These results show that self-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine-tuning (SFT), without additional policy rollouts.
Read the full abstract ▾
All authors Zhiwei Zhang, Huayu Deng, Fei Zhao, Jiayan Fu, Bin Liang, Kam-Fai Wong, Mu Chuan
Machine Learning
Intrinsic Associative Memory on Riemannian Manifolds: Curvature, Capacity, and Emergent Modes
Krishnakumar Balasubramanian and Zhaoyang Shi
Geometry does more than constrain an associative
memory:
curvature determines what it remembers and which states it creates. We develop intrinsic dense associative
memories on Riemannian
manifolds by casting
memory as Epanechnikov kernel-density mode seeking. We compare geodesic and volume-corrected energies and show that
curvature separates their behavior. We prove that geodesic
memory always retains an isolated pattern, while corrected
memory obeys a sharp Ricci-
curvature threshold: positive
curvature can erase
memories in high dimensions, while negative
curvature reinforces them. We derive geodesic capacity scalings of $q_\beta^{-1/2}$ for retaining every pattern and $q_\beta^{-1}$ for a typical one, where $q_\beta$ is the pairwise kernel-overlap probability. We show how overlap creates novel
memories: designed $N$-pattern configurations realize all $2^N-1$ subset modes, but random data at the storage threshold yield only a Poisson number. We establish exact one-step recall using Riemannian mean shift. In simulations, we recover the predicted
curvature transition and every designed mode. On WordNet's full noun hierarchy, we demonstrate that volume correction improves low-capacity
retrieval. Together, our work shows that
curvature is a design variable for associative
memory, not merely a property of the data.
Read the full abstract ▾
Machine Learning
FluxLite: Inference-Time Proposal Control for Discrete Diffusion Models
Yinuo Ren, Haoxuan Chen, Grant M. Rotskoff et al. (+2 more)
Many inference-time tasks for pretrained discrete
diffusion models and
diffusion language models reduce to drawing samples from a tilted version of the pretrained distribution. Feynman-Kac sequential Monte Carlo (SMC) makes this correction exact in principle, but its prescribed weights routinely degenerate when the proposal dynamics are misaligned with the tilt, capping the practical gains from additional particles. We introduce FluxLite, a lightweight, training-free proposal-control framework for discrete
diffusion. On the sparse directed graph of pretrained reverse rates, any sparse jump-rate perturbation can be exactly compensated by a $q_t$-weighted graph-divergence term in the Feynman-Kac potential; the target path is therefore preserved while the residual reweighting variance becomes a local convex objective. We instantiate this principle as two practical samplers: a one-hop local reallocation rule (HEU) and a small nonnegative quadratic program over pretrained-rate bases (D-VCG). We further prove population stability under the standard score-
entropy training loss, identifying a tilted-path coverage factor that governs robustness to score error, together with finite-particle convergence for a fixed controlled Feynman-Kac recursion. Empirically, FluxLite improves over standard Feynman-Kac SMC baselines by up to two orders of magnitude in terminal KL on an analytically tractable finite-state CTMC
benchmark, and reduces row-correlation MSE on 2D Ising sampling by 5-7x in
geometric mean and up to 55x at peak.
Read the full abstract ▾
All authors Yinuo Ren, Haoxuan Chen, Grant M. Rotskoff, Jiequn Han, Lexing Ying
Machine Learning
EnJoi: Ensemble Joint Score Filter for Generative Data Assimilation
Julien Moreau and Marc Lelarge
Data Assimilation (DA) aims to recover the full state of a dynamical system that is only partially observed. A solution is to use Score-based models to generate physically consistent trajectories that agree with the observations. These Autoregressive
Diffusion models are trained by conditioning on the previous state; however, they do not take into account the uncertainty of their past predictions. We propose a new
diffusion-based assimilation
algorithm that dynamically balances the confidence in the current state and the new observations. Crucially, we choose to learn the distribution of the joint state containing both the past and future. This allows us to use a modified version of En4DVar, a classical DA
algorithm that relies on the covariance of an ensemble of particles. Experiments on fluid and traffic flow simulations show improved reconstruction performance, especially in situations where observations are sparse and non-homogeneous.
Read the full abstract ▾