Machine Learning
Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates
Saumya Goyal, Rohith Rongali, Ritabrata Ray et al. (+1 more)
We study learning to learn through the lens of hyperparameter tuning. We propose the Langevin Gradient Descent
Algorithm (LGD), which approximates the mean of the posterior distribution defined by the loss function and regularizer of a regression task with convex objective. For classification tasks, the LGD
algorithm estimates the posterior probabilities of each class on the test set. We prove the existence of an optimal hyperparameter configuration for which the LGD
algorithm achieves the Bayes' optimal solution for squared loss on regression tasks, and for which LGD closely approximates the posterior probabilities for well-specified classification tasks. Subsequently, we study generalization guarantees on meta learning optimal hyperparameters for the LGD
algorithm from a given set of tasks in the data-driven setting. For a number of parameters $d$ and hyperparameter dimension $h$, we show a pseudo-dimension bound of $O(dh)$, up to logarithmic terms under mild assumptions on LGD. This matches the dependence of the bounds on number of parameters obtained in prior work for linear regression using the elastic net, which only allows for $h=2$ hyperparameters, and extends their bounds to regression on convex loss. Compared to bounds on regularized logistic regression that allow for only $h=1$ hyperparameter, our bounds improve greatly on the dependence on samples per task at the cost of worse dependence on the number of parameters by accounting for hardware-aware procedures. Finally, we show empirical evidence of the success of LGD and the meta learning procedure for few-shot learning on linear and logistic regression using synthetically created
datasets.
Read the full abstract ▾
All authors Saumya Goyal, Rohith Rongali, Ritabrata Ray, Barnabás Póczos
Machine Learning · Cross-listed from cs.MA
Learning Probabilistic Responsibility Allocations for Multi-Agent Interactions
Isaac Remy, Caleb Chang and Karen Leung
Human behavior in interactive settings is shaped not only by individual objectives but also by shared constraints with others, such as safety. Understanding how people allocate responsibility, i.e., how much one deviates from their desired policy to accommodate others, can inform the design of socially compliant and trustworthy autonomous systems. In this work, we introduce a method for learning a probabilistic responsibility allocation model that captures the
multimodal uncertainty inherent in multi-
agent interactions. Specifically, our approach leverages the latent space of a conditional variational autoencoder, combined with techniques from multi-
agent trajectory
forecasting, to learn a distribution over responsibility allocations conditioned on scene and
agent context. Although ground-truth responsibility labels are unavailable, the model remains tractable by incorporating a differentiable optimization layer that maps responsibility allocations to induced controls, which are available. We evaluate our method on the INTERACTION driving
dataset and demonstrate that it not only achieves strong predictive performance but also provides interpretable insights, through the lens of responsibility, into patterns of multi-
agent interaction.
Read the full abstract ▾
Machine Learning
Synthetic Tabular Generators Fail to Preserve Behavioral Fraud Patterns: A Benchmark on Temporal, Velocity, and Multi-Account Signals
Bhavana Sajja
We introduce behavioral fidelity – a third evaluation dimension for synthetic tabular data that measures whether generated data preserves the temporal, sequential, and structural behavioral patterns that distinguish real-world entity activity. Existing frameworks evaluate statistical fidelity (marginal distributions and correlations) and downstream utility (classifier AUROC on synthetic-trained models), but neither tests for the behavioral signals that operational detection and analysis systems actually rely on. We formalize a taxonomy of four behavioral fraud patterns (P1-P4) covering inter-event timing, burst structure, multi-account graph motifs, and velocity-rule trigger rates; define a degradation ratio metric calibrated to a real-data noise floor (1.0 = matches real variability, k = k-times worse); and prove that row-independent generators – the dominant paradigm – are structurally incapable of reproducing P3 graph motifs (Proposition 1) and produce non-positive within-entity IET autocorrelation (Proposition 2), making the positive burst fingerprint of fraud sequences unachievable regardless of architecture or training data size. We
benchmark CTGAN, TVAE, GaussianCopula, and TabularARGN on IEEE-CIS Fraud Detection and the Amazon Fraud
Dataset. All four fail severely: on IEEE-CIS composite degradation ratios range from 24.4x (TVAE) to 39.0x (GaussianCopula); on Amazon FDB, row-independent generators score 81.6-99.7x, while TabularARGN achieves 17.2x. We document generator-specific failure modes and their resolutions. The P1-P4 framework extends to any domain with entity-level sequential tabular data, including healthcare and network security. We release our evaluation framework as open source.
Read the full abstract ▾
Machine Learning
Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin
Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung et al. (+1 more)
Grokking - the delayed transition from memorisation to generalisation in neural networks - remains poorly understood. We study this phenomenon through the geometry of learned representations and identify a consistent empirical signature preceding generalisation: collapse of the spectral
entropy of the representation covariance
matrix. Across modular arithmetic tasks and multiple random seeds, spectral
entropy decreases gradually during training and crosses a stable task-specific threshold before test accuracy rises. A representation-mixing intervention that delays this collapse also delays grokking, including under norm-matched controls, indicating that the effect is not explained by parameter norm alone. We further show that the
entropy gap predicts the remaining time until grokking with useful out-of-sample accuracy. To probe the structure underlying this transition, we introduce a
Fourier-
alignment observable for cyclic-group tasks.
Entropy collapse is strongly coupled to the emergence of
Fourier-aligned representations, suggesting that spectral
entropy tracks concentration of the representation into task-structured directions rather than generic compression alone. The same qualitative dynamics appear in non-abelian group composition tasks, while MLP controls show that
entropy collapse by itself is insufficient for grokking in the absence of appropriate inductive bias. Taken together, the results support a view of grokking as a representational phase transition with an observable
geometric signature. We discuss the scope and limitations of this interpretation, connections to recent feature-learning and spectral-dynamics work, and directions for testing whether similar transitions appear in larger-scale learning systems.
Read the full abstract ▾
All authors Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung, Phan Thanh Duc
Machine Learning
LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety
Junxiao Yang, Haoran Liu, Jinzhe Tu et al. (+9 more)
Large
language models (LLMs) often demonstrate strong safety performance in high-resource
languages, yet exhibit severe vulnerabilities when queried in low-resource
languages. We attribute this gap to a mismatch between
language-agnostic semantic understanding ability and
language-dominant safety
alignment biased toward high-resource
languages. Consistent with this hypothesis, we empirically identify the semantic bottleneck in LLMs, an intermediate layer in which the geometry of model representations is governed primarily by shared semantic content rather than
language identity. Building on this observation, we propose
Language-Agnostic Semantic
Alignment (LASA), which anchors safety
alignment directly in semantic bottlenecks. Experiments show that LASA substantially improves safety across all
languages: average attack success rate (ASR) drops from 24.7% to 2.8% on LLaMA-3.1-8B-Instruct and remains around 3-4% across Qwen2.5 and Qwen3 Instruct models (7B-32B). Together, our analysis and method offer a representation-level perspective on LLM safety, suggesting that safety
alignment requires anchoring safety understanding not in surface text, but in the model's
language-agnostic semantic space.
Read the full abstract ▾
All authors Junxiao Yang, Haoran Liu, Jinzhe Tu, Jiale Cheng, Zhexin Zhang, Shiyao Cui, Jiaqi Weng, Jialing Tao, Hui Xue, Hongning Wang, Han Qiu, Minlie Huang
Machine Learning
XANE(3): An E(3)-Equivariant Graph Neural Network for Accurate Prediction of XANES Spectra from Atomic Structures
Vitor F. Grizzi, Luke N. Pretzie, Jiayi Xu et al. (+1 more)
We present XANE(3), a physics-based E(3)-equivariant graph neural network for predicting X-ray absorption near-edge structure (XANES) spectra directly from atomic structures. The model combines tensor-product message passing with spherical harmonic edge features, absorber-query
attention pooling, custom equivariant layer normalization, adaptive gated residual connections, and a spectral readout based on a multi-scale Gaussian basis with an optional sigmoidal background term. To improve line-shape fidelity, training is performed with a composite objective that includes pointwise spectral reconstruction together with first- and second-derivative matching terms. We evaluate the model on a
dataset of 5,941 FDMNES simulations of iron
oxide surface facets and obtain a spectrum mean squared error of $1.0 \times 10^{-3}$ on the test set. The model accurately reproduces the main edge structure, relative peak intensities, pre-edge features, and post-edge oscillations. Ablation studies show that the derivative-aware objective, custom equivariant normalization, absorber-conditioned
attention pooling, adaptive gated residual mixing, and global background term each improve performance. Interestingly, a capacity-matched scalar-only variant achieves comparable pointwise reconstruction error but reduced derivative-level fidelity, indicating that explicit tensorial channels are not strictly required for low intensity error on this
dataset, although they remain beneficial for capturing finer spectral structure. These results establish XANE(3) as an accurate and efficient surrogate for XANES simulation and offer a promising route toward accelerated spectral prediction, ML-assisted spectroscopy, and data-driven materials discovery.
Read the full abstract ▾
All authors Vitor F. Grizzi, Luke N. Pretzie, Jiayi Xu, Cong Liu
Machine Learning · Cross-listed from cs.CV
Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models
Md Tanvirul Alam
Large
vision-
language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from rule-mapping failures. We study this behavior as semantic fixation: preserving a default interpretation even when the
prompt specifies an alternative, equally valid mapping. To isolate this effect, we introduce VLM-Fix, a controlled
benchmark over four abstract strategy games that evaluates identical terminal board states under paired standard and inverse rule formulations. Across 14 open and closed VLMs, accuracy consistently favors standard rules, revealing a robust semantic-fixation gap.
Prompt interventions support this mechanism: neutral alias
prompts substantially narrow the inverse-rule gap, while semantically loaded aliases reopen it. Post-training is strongly rule-aligned: training on one rule improves same-rule transfer but hurts opposite-rule transfer, while joint-rule training improves broader transfer. To test external validity beyond synthetic games, we evaluate analogous defamiliarization interventions on VLMBias and observe the same qualitative pattern. Finally, late-layer activation steering partially recovers degraded performance, indicating that semantic-fixation errors are at least partly editable in late representations. Project page, code, and
dataset available at https://maveryn.github.io/vlm-fix/.
Read the full abstract ▾
Machine Learning
SOLARIS: Speculative Offloading of Latent-bAsed Representation for Inference Scaling
Zikun Liu, Liang Luo, Qianru Li et al. (+31 more)
Recent advances in recommendation scaling laws have led to foundation models of unprecedented complexity. While these models offer superior performance, their computational demands make real-time serving impractical, often forcing practitioners to rely on knowledge
distillation-compromising serving quality for efficiency. To address this challenge, we present SOLARIS (Speculative Offloading of Latent-bAsed Representation for Inference Scaling), a novel framework inspired by speculative decoding. SOLARIS proactively precomputes user-item interaction
embeddings by predicting which user-item pairs are likely to appear in future requests, and asynchronously generating their foundation model representations ahead of time. This approach decouples the costly foundation model inference from the latency-critical serving path, enabling real-time knowledge transfer from models previously considered too expensive for online use. Deployed across Meta's advertising system serving billions of daily requests, SOLARIS achieves 0.67% revenue-driving top-line metrics gain, demonstrating its effectiveness at scale.
Read the full abstract ▾
All authors Zikun Liu, Liang Luo, Qianru Li, Zhengyu Zhang, Wei Ling, Jingyi Shen, Zeliang Chen, Yaning Huang, Jingxian Huang, Abdallah Aboelela, Chonglin Sun, Feifan Gu, Fenggang Wu, Hang Qu, Huayu Li, Jill Pan, Kaidi Pei, Laming Chen, Longhao Jin, Qin Huang, Tongyi Tang, Varna Puvvada, Wenlin Chen, Xiaohan Wei, Xu Cao, Yantao Yao, Yuan Jin, Yunchen Pu, Yuxin Chen, Zijian Shen, Zhengkai Zhang, Jing Zhu, Dong Liang, Ellie Wen
Machine Learning · Cross-listed from eess.SY
Parametric Interpolation of Dynamic Mode Decomposition for Predicting Nonlinear Systems
Ananda Chakrabarti, Haitham H. Saleh, Indranil Nayak et al. (+3 more)
We present parameter-interpolated dynamic mode decomposition (piDMD), a parametric reduced-order modeling framework that embeds known parameter-affine structure directly into the DMD regression step. Unlike existing parametric DMD methods which interpolate modes,
eigenvalues, or reduced operators and can be fragile with sparse training data or multi-dimensional parameter spaces, piDMD learns a single parameter-affine Koopman surrogate reduced order model (ROM) across multiple training parameter samples and predicts at unseen parameter values without retraining. We validate piDMD on fluid flow past a cylinder, electron beam oscillations in transverse
magnetic fields, and virtual cathode oscillations – the latter two being simulated using an electromagnetic particle-in-
cell (EMPIC) method. Across all
benchmarks, piDMD achieves accurate long-horizon predictions and improved robustness over state-of-the-art interpolation-based parametric DMD baselines, with less training samples and with multi-dimensional parameter spaces.
Read the full abstract ▾
All authors Ananda Chakrabarti, Haitham H. Saleh, Indranil Nayak, Balasubramaniam Shanker, Fernando L. Teixeira, Debdipta Goswami
Machine Learning · Cross-listed from cs.AI
Spatial Atlas: Compute-Grounded Reasoning for Spatial-Aware Research Agent Benchmarks
Arun Sharma
We describe compute-grounded
reasoning (CGR), a design pattern in which code computes selected sub-problems from explicit intermediate representations before a
language model answers. Spatial Atlas implements CGR as an
Agent2
Agent (A2A) server with a spatial question-answering handler and a machine-learning engineering handler. The spatial handler asks a
language model to extract a scene graph, and code then fills in missing distances and checks the extracted safety rules. A separate
benchmark driver can also run a strict metric bridge. It computes the gap for horizontal-gap questions from
segmentation masks and a reconstructed point map, and it passes that gap to the answering model as a fact. The bridge returns a fixed unavailable answer when an evidence check fails, and it never falls back to model-estimated coordinates. The ML-engineering handler generates pipeline code, parses validation scores, and caps the number of repair and refinement passes. Its code execution is off by default. The repository also provides four run modes that can write label-free journals, a shuffled-image control mapping, and journal validators that reject label-bearing fields. We report one private label-free operational run in which four paths each wrote eight prediction rows with zero retries. Labels stayed sealed, and no score was computed, so this run establishes operational integrity only. We report no FieldWorkArena result because the
benchmark data were not accessible. We also omit every performance, latency, and resource-use number that lacks a reproducible run artifact.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
A Nonparametric Adaptive EWMA Control Chart for Binary Monitoring of Multiple Stream Processes
Faruk Muritala, Austin Brown, Dhrubajyoti Ghosh et al. (+1 more)
Monitoring binomial proportions across multiple independent streams is a critical challenge in Statistical Process Control (SPC), with applications from manufacturing to cybersecurity. While EWMA charts offer sensitivity to small shifts, existing implementations rely on asymptotic variance approximations that fail during early-phase monitoring. We introduce a Cumulative Standardized Binomial EWMA (CSB-EWMA) chart that overcomes this limitation by deriving the exact time-varying variance of the EWMA statistic for binary multiple-stream data, enabling adaptive control limits that ensure statistical rigor from the first sample. Through extensive simulations, we identify optimal smoothing ({\lambda}) and limit (L) parameters to achieve target in-control average run length (ARL0) of 370 and 500. The CSB-EWMA chart demonstrates rapid shift detection across both ARL0 targets, with out-of-control average run length (ARL1) dropping to 3-7 samples for moderate shifts ({\delta}=0.2), and exhibits exceptional robustness across different data distributions, with low ARL1 Coefficients of Variation (CV < 0.10 for small shifts) for both ARL0 = 370 and 500. This work provides practitioners with a distribution-free, sensitive, and theoretically sound tool for early change detection in binomial multiple-stream processes.
Read the full abstract ▾
All authors Faruk Muritala, Austin Brown, Dhrubajyoti Ghosh, Sherry Ni
Machine Learning
Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
Zixuan Liu, Xiaolin Sun and Zizhan Zheng
Designing robust reinforcement learning (RL)
agents in the presence of imperfect reward signals remains a core challenge. In practice,
agents are often trained with proxy rewards that only approximate the true objective, leaving them vulnerable to reward hacking, where high proxy returns arise from unintended or exploitative behaviors. Recent work formalizes this issue using r-correlation between proxy and true rewards, but existing methods like occupancy-regularized policy optimization (ORPO) optimize against a fixed proxy and do not provide strong guarantees against broader classes of correlated proxies. In this work, we formulate reward hacking as a robust policy optimization problem over the space of all r-correlated proxy rewards. We derive a tractable max-min formulation, where the
agent maximizes performance under the worst-case proxy consistent with the correlation constraint. We further show that when the reward is a linear function of known features, our approach can be adapted to incorporate this prior knowledge, yielding both improved policies and interpretable worst-case rewards. Experiments across several environments show that our
algorithms consistently outperform ORPO in worst-case returns, and offer improved robustness and stability across different levels of proxy-true reward correlation. These results show that our approach provides both robustness and transparency in settings where reward design is inherently uncertain. The code is available at https://github.com/ZixuanLiu4869/reward_hacking.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.ET
Robust Reasoning and Learning with Brain-Inspired Representations under Hardware-Induced Nonlinearities
William Youngwoo Chung, Hamza Errahmouni Barkam, Tamoghno Das et al. (+1 more)
Traditional machine learning depends on high-precision arithmetic and near-ideal hardware assumptions, which is increasingly challenged by variability in aggressively scaled
semiconductor devices. Compute-in-
memory (CIM) architectures alleviate data-movement bottlenecks and improve energy efficiency yet introduce nonlinear distortions and reliability concerns. We address these issues with a hardware-aware optimization framework based on Hyperdimensional Computing (HDC), systematically compensating for non-ideal similarity computations in CIM. Our approach formulates encoding as an optimization problem, minimizing the Frobenius norm between an ideal kernel and its hardware-constrained counterpart, and employs a joint optimization strategy for end-to-end calibration of hypervector representations. Experimental results demonstrate that our method when applied to QuantHD achieves 84% accuracy under severe hardware-induced perturbations, a 48% increase over naive QuantHD under the same conditions. Additionally, our optimization is vital for graph-based HDC reliant on precise variable-binding for interpretable
reasoning. Our framework preserves the accuracy of RelHD on the Cora
dataset, achieving a 5.4$\times$ accuracy improvement over naive RelHD under nonlinear environments. By preserving HDC's robustness and symbolic properties, our solution enables scalable, energy-efficient intelligent systems capable of classification and
reasoning on emerging CIM hardware.
Read the full abstract ▾
All authors William Youngwoo Chung, Hamza Errahmouni Barkam, Tamoghno Das, Mohsen Imani
Machine Learning · Cross-listed from cs.CV
OpenTME: An Open Dataset of AI-powered H&E Tumor Microenvironment Profiles from TCGA
Maaike Galama, Nina Kozar-Gillan, Christina Embacher et al. (+18 more)
The tumor microenvironment (TME) plays a central role in cancer progression, treatment response, and patient outcomes, yet large-scale, consistent, and quantitative TME characterization from routine hematoxylin and eosin (H&E)-stained histopathology remains scarce. We introduce OpenTME, an open-access
dataset of pre-computed TME profiles derived from 3,634 H&E-stained whole-slide images across five cancer types (bladder, breast, colorectal, liver, and lung cancer) from The Cancer Genome Atlas (TCGA). All outputs were generated using Atlas H&E-TME, an AI-powered application built on the Atlas family of pathology foundation models, which performs tissue quality control, tissue
segmentation,
cell detection and classification, and spatial neighborhood analysis, yielding over 4,500 quantitative readouts per slide at
cell-level resolution. OpenTME is available for non-commercial academic research on Hugging Face. We will continue to expand OpenTME over time and anticipate it will serve as a resource for biomarker discovery, spatial biology research, and the development of computational methods for TME analysis.
Read the full abstract ▾
All authors Maaike Galama, Nina Kozar-Gillan, Christina Embacher, Todd Dembo, Cornelius Böhm, Evelyn Ramberger, Julika Ribbat-Idel, Rosemarie Krupar, Verena Aumiller, Miriam Hägele, Kai Standvoss, Gerrit Erdmann, Blanca Pablos, Ari Angelo, Simon Schallenberg, Andrew Norgan, Viktor Matyas, Klaus-Robert Müller, Maximilian Alber, Lukas Ruff, Frederick Klauschen
Machine Learning · Cross-listed from cs.CL
Robust Explanations for User Trust in Enterprise NLP Systems
Guilin Zhang, Kai Zhao, Jeffrey Friedman et al. (+3 more)
Robust explanations are increasingly required for user trust in enterprise NLP, yet pre-deployment validation is difficult in the common case of black-box deployment (API-only access) where representation-based explainers are infeasible and existing studies provide limited guidance on whether explanations remain stable under real user noise, especially when organizations migrate from encoder classifiers to decoder LLMs. To close this gap, we propose a unified black-box robustness evaluation framework for
token-level explanations based on leave-one-out occlusion, and operationalize explanation robustness with top-
token flip rate under realistic perturbations (swap, deletion, shuffling, and back-translation) at multiple severity levels. Using this protocol, we conduct a systematic cross-architecture comparison across three
benchmark datasets and six models spanning encoder and decoder families (BERT, RoBERTa, Qwen 7B/14B, Llama 8B/70B; 64,800 cases). We find that decoder LLMs produce substantially more stable explanations than encoder baselines (73% lower flip rates on average), and that stability improves with model scale (44% gain from 7B to 70B). Finally, we relate robustness improvements to inference cost, yielding a practical cost-robustness tradeoff curve that supports model and explanation selection prior to deployment in compliance-sensitive applications.
Read the full abstract ▾
All authors Guilin Zhang, Kai Zhao, Jeffrey Friedman, Xu Chu, Amine Anoun, Jerry Ting
Machine Learning
Interpretable DNA Sequence Classification via Dynamic Feature Generation in Decision Trees
Nicolas Huynh, Krzysztof Kacprzyk, Ryan Sheridan et al. (+2 more)
The analysis of DNA sequences has become critical in numerous fields, from evolutionary biology to understanding gene regulation and disease mechanisms. While deep neural networks can achieve remarkable predictive performance, they typically operate as black boxes. Contrasting these black boxes, axis-aligned decision trees offer a promising direction for interpretable DNA sequence analysis, yet they suffer from a fundamental limitation: considering individual raw features in isolation at each split limits their expressivity, which results in prohibitive tree depths that hinder both interpretability and generalization performance. We address this challenge by introducing DEFT, a novel framework that adaptively generates high-level sequence features during tree construction. DEFT leverages large
language models to propose biologically-informed features tailored to the local sequence distributions at each node and to iteratively refine them with a reflection mechanism. Empirically, we demonstrate that DEFT discovers human-interpretable and highly predictive sequence features across a diverse range of genomic tasks.
Read the full abstract ▾
All authors Nicolas Huynh, Krzysztof Kacprzyk, Ryan Sheridan, David Bentley, Mihaela van der Schaar
Machine Learning · Cross-listed from cs.CL
Haocheng Xi, Harman Singh, Yuezhou Hu et al. (+9 more)
Block-wise
diffusion language models (DLMs) generate multiple
tokens in any order, offering a promising alternative to the autoregressive decoding pipeline. However, they still remain bottlenecked by
memory-bound
attention in long-context scenarios. Naive sparse
attention fails on DLMs due to a KV Inflation problem, where different queries select different prefix positions, making the union of accessed KV pages large. To address this, we observe that between consecutive denoising steps, only a small fraction of active
tokens exhibit significant hidden-state changes, while the majority of stable
tokens remain nearly constant. Based on this insight, we propose LOSA (Locality-aware Sparse
Attention), which reuses cached prefix-
attention results for stable
tokens and applies sparse
attention only to active
tokens. This substantially shrinks the number of KV indices that must be loaded, yielding both higher speedup and higher accuracy. Across multiple block-wise DLMs and
benchmarks, LOSA preserves near-dense accuracy while significantly improving efficiency, achieving up to +9 points in average accuracy at aggressive sparsity levels while maintaining 1.54x lower
attention density. It also achieves up to 4.14x
attention speedup on RTX A6000 GPUs, demonstrating the effectiveness of the proposed method.
Read the full abstract ▾
All authors Haocheng Xi, Harman Singh, Yuezhou Hu, Coleman Hooper, Rishabh Tiwari, Aditya Tomar, Minjae Lee, Wonjun Kang, Michael Mahoney, Chenfeng Xu, Kurt Keutzer, Amir Gholami
Machine Learning
VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation
Eli Corn and Daphna Weinshall
Deep learning models may converge to suboptimal solutions despite strong validation accuracy, masking an optimization failure we term Trajectory Deviation. This is because as training proceeds, models can abandon high generalization states for specific data sub-populations, thus discarding previously learned latent features without triggering classical overfitting signals. To address this problem we introduce VISTA, an online self-
distillation framework that enforces consistency along the optimization trajectory. Using a validation-informed Marginal Coverage score, VISTA identifies expert anchors, which are earlier model states that retain specialized competence over distinct data regions. A coverage-weighted ensemble of these anchors is integrated online during training, regularizing the loss landscape and preserving mastered knowledge. When evaluated across multiple
benchmarks, VISTA demonstrates improved robustness and generalization over standard training and prior self-
distillation methods, while a lightweight implementation reduces storage overhead by 90% without performance loss.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
On the continuum limit of t-SNE for data visualization
Jeff Calder, Zhonggan Huang, Ryan Murray et al. (+1 more)
This work is concerned with the continuum limit of a graph-based data visualization technique called the t-Distributed
Stochastic Neighbor
Embedding (t-SNE), which is widely used for visualizing data in a variety of applications, but is still poorly understood from a theoretical standpoint. The t-SNE
algorithm produces visualizations by minimizing the Kullback-Leibler divergence between similarity
matrices representing the high dimensional data and its low dimensional representation. We prove that as the number of data points $n \to \infty$, after a natural rescaling and in applicable parameter regimes, the Kullback-Leibler divergence is consistent as the number of data points $n \to \infty$ and the similarity graph remains sparse with a continuum variational problem that involves a non-convex gradient regularization term and a penalty on the magnitude of the probability density function in the visualization space. These two terms represent the continuum limits of the attraction and repulsion forces in the t-SNE
algorithm. Due to the lack of convexity in the continuum variational problem, the question of well-posedeness is only partially resolved. We show that when both dimensions are $1$, the problem admits a unique smooth minimizer, along with an infinite number of discontinuous minimizers (interpreted in a relaxed sense). This aligns well with the empirically observed ability of t-SNE to separate data in seemingly arbitrary ways in the visualization. The energy is also very closely related to the famously ill-posed Perona-Malik equation, which is used for denoising and simplifying images. We present numerical results validating the continuum limit, provide some preliminary results about the delicate nature of the limiting energetic problem in higher dimensions, and highlight several problems for future work.
Read the full abstract ▾
All authors Jeff Calder, Zhonggan Huang, Ryan Murray, Adam Pickarski
Machine Learning · Cross-listed from cs.DS
Simpler Logarithmic Approximation Algorithms for the Optimal Decision Tree and Adaptive Set Cover
Michał Szyfelbein
We study the problem of constructing a decision tree that identifies an unknown hypothesis from a given ground set, under both average- and worst-case cost. The Optimal Decision Tree (ODT) problem has been extensively studied in the literature, and $\mathcal{O}(\log n)$-approximation guarantees are known for each criterion, including a simultaneous guarantee for both (here $n$ is the number of hypotheses) [ICML]. Although most of the existing
algorithm use simple greedy strategies, their analyses rely on intricate arguments. Since ODT is a fundamental problem with wide-ranging applications, we are interested in developing
algorithms that are both effective and easy to understand. We present a new greedy
algorithm with a simplified analysis that simultaneously achieves approximation factors below $3.65\log_2 n$ for worst-case cost and $7.3\log_2 n$ for average-case cost. Our
algorithm for the Optimal Decision Tree behaves greedily with respect to the hereby introduced Separating Subfamily problem which asks for the cheapest subfamily of tests which partitions the hypotheses into pieces of small enough size. We show that the Separating Subfamily can itself be reduced to an instance of the well-known Maximum Coverage problem. At the core of our approach lies exploiting properties of cutting a clique into small pieces, where edges represent pairs of hypotheses to be separated. In the spirit of simplicity, we also provide a new and straightforward application of ODT: an $\mathcal{O}(\log (k\cdot n))$-approximation for the Adaptive Set Cover problem, where $k$ is the number of possible realizations.
Read the full abstract ▾
Machine Learning
TriFit: Trimodal Fusion with Protein Dynamics for Mutation Fitness Prediction
Seungik Cho
Predicting the functional impact of single amino acid substitutions (SAVs) is central to understanding genetic disease and engineering therapeutic
proteins. While
protein language models and structure-based methods have achieved strong performance on this task, they systematically neglect
protein dynamics; residue flexibility, correlated motions, and allosteric coupling are well-established determinants of mutational tolerance in structural biology, yet have not been incorporated into supervised variant effect predictors. We present TriFit, a
multimodal framework that integrates sequence, structure, and
protein dynamics through a four-expert Mixture-of-Experts (MoE) fusion module with trimodal cross-modal contrastive learning. Sequence
embeddings are extracted via masked marginal scoring with ESM-2 (650M); structural
embeddings from AlphaFold2-predicted C-alpha geometries; and dynamics
embeddings from Gaussian Network Model (GNM) B-factors, mode shapes, and residue-residue cross-correlations. The MoE router adaptively weights modality combinations conditioned on the input, enabling
protein-specific fusion without fixed modality assumptions. On the ProteinGym substitution
benchmark (217 DMS assays, 696k SAVs), TriFit achieves AUROC 0.897 +/- 0.0002, outperforming all supervised baselines including Kermut (0.864) and ProteinNPT (0.844), and the best zero-shot model ESM3 (0.769). Ablation studies confirm that dynamics provides the largest marginal contribution over pairwise modality combinations, and TriFit achieves well-calibrated probabilistic outputs (ECE = 0.044) without post-hoc correction.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Identity as Attractor: Geometric Evidence for Persistent Agent Architecture in LLM Activation Space
Vladimir Vasilenko
Large
language models map semantically related
prompts to similar internal representations – a phenomenon interpretable as attractor-like dynamics. We ask whether the identity document of a persistent cognitive
agent (its cognitive_core) exhibits analogous attractor-like behavior. We present a controlled experiment on Llama 3.1 8B Instruct, comparing hidden states of an original cognitive_core (Condition A), seven paraphrases (Condition B), and seven structurally matched controls (Condition C). Mean-pooled states at layers 8, 16, and 24 show that paraphrases converge to a tighter cluster than controls (Cohen's d > 1.88, p < 10^{-27}, Bonferroni-corrected). Replication on Gemma 2 9B confirms cross-architecture generalizability. Ablations suggest the effect is primarily semantic rather than structural, and that structural completeness appears necessary to reach the attractor region. An exploratory experiment shows that reading a scientific description of the
agent shifts internal state toward the attractor – closer than a sham preprint – distinguishing knowing about an identity from operating as that identity. These results provide representational evidence that
agent identity documents induce attractor-like geometry in LLM activation space.
Read the full abstract ▾
Machine Learning
UCS: Estimating Unseen Coverage for Improved In-Context Learning
Jiayi Xin, Xiang Li, Evan Qiang et al. (+4 more)
In-context learning (ICL) performance depends critically on which demonstrations are placed in the
prompt, yet most existing selectors prioritize heuristic notions of relevance or diversity and provide limited insight into the coverage of a demonstration set. We propose Unseen Coverage Selection (UKS), a training-free, subset-level coverage prior motivated by the principle that a good demonstration set should expose the model to latent cluster unrevealed by the currently selected subset. UCS operationalizes this idea by (1) inducing discrete latent clusters from model-consistent
embeddings and (2) estimating the number of unrevealed clusters within a candidate subset via a Smoothed Good–Turing
estimator from its empirical frequency spectrum. Unlike previous selection methods, UCS is coverage-based and training-free, and can be seamlessly combined with both query-dependent and query-independent selection baselines via a simple regularized objective. Experiments on multiple intent-classification and
reasoning benchmarks with frontier Large
Language Models show that augmenting strong baselines with UCS consistently improves ICL accuracy by up to 2-6% under the same selection budget, while also yielding insights into task- and model-level latent cluster distributions. Code is available at https://github.com/Raina-Xin/UCS.
Read the full abstract ▾
All authors Jiayi Xin, Xiang Li, Evan Qiang, Weiqing He, Tianqi Shang, Weijie J. Su, Qi Long
Machine Learning
Sample Complexity of Autoregressive Reasoning: Chain-of-Thought vs. End-to-End
Steve Hanneke, Idan Mehalel and Shay Moran
Modern large
language models generate text autoregressively, producing
tokens one at a time. To study the learnability of such systems, Joshi et al. (COLT 2025) introduced a PAC-learning framework for next-
token generators, the primitive underlying autoregressive models. In this framework, an unknown next-
token generator maps a sequence of
tokens to the next
token and is iteratively applied for $T$ steps, producing a chain of
tokens whose final
token constitutes the model's output. The learning task is to learn the input-output mapping induced by this autoregressive process. Depending on the available supervision, training examples may reveal only the final output (End-to-End supervision) or the entire generated chain (Chain-of-Thought supervision). This raises two natural questions: how the sample complexity depends on the generation length $T$, and how much Chain-of-Thought supervision can reduce this dependence. In this work we give a nearly complete answer to both questions by uncovering a taxonomy of how the sample complexity scales with $T$. For End-to-End learning, we show that the landscape is remarkably rich: subject to mild conditions, essentially any growth rate $r(T)$ between constant and linear can arise as the sample complexity, and combined with the linear upper bound of Joshi et al., this yields an essentially complete characterization. In contrast, under Chain-of-Thought supervision we show that the sample complexity is independent of $T$, demonstrating that access to intermediate
reasoning steps can eliminate the dependence on the generation length altogether. Our analysis introduces new combinatorial tools, and as corollaries we resolve several open questions posed by Joshi et al. regarding the dependence of learnability on the generation length and the role of Chain-of-Thought supervision.
Read the full abstract ▾
Machine Learning
BayMOTH: Bayesian optiMizatiOn with meTa-lookahead – a simple approacH
Rahman Ejaz, Varchas Gopalaswamy, Ricardo Luna et al. (+5 more)
Bayesian optimization (BO) has for sequential optimization of expensive black-box functions demonstrated practicality and effectiveness in many real-world settings. Meta-
Bayesian optimization (meta-BO) focuses on improving the sample efficiency of BO by making use of information from related tasks. Although meta-BO is sample-efficient when task structure transfers, poor
alignment between meta-training and test tasks can cause suboptimal queries to be suggested during online optimization. To this end, we propose a simple meta-BO
algorithm that utilizes related-task information when determined useful, falling back to lookahead otherwise, within a unified framework. We demonstrate competitiveness of our method with existing approaches on function optimization tasks, while retaining strong performance in low task-relatedness regimes where test tasks share limited structure with the meta-training set.
Read the full abstract ▾
All authors Rahman Ejaz, Varchas Gopalaswamy, Ricardo Luna, Aarne Lees, Vineet Gundecha, Christopher Kanan, Soumyendu Sarkar, Riccardo Betti
Machine Learning
Loss-Driven Bayesian Active Learning
Zhuoyue Huang, Freddie Bickford Smith and Tom Rainforth
The central goal of active learning is to gather data that maximises downstream predictive performance, but popular approaches have limited flexibility in customising this data acquisition to different downstream problems and losses. We propose a rigorous loss-driven approach to
Bayesian active learning that allows data acquisition to directly target the loss associated with a given decision problem. In particular, we show how any loss can be used to derive a unique objective for optimal data acquisition. Critically, we then show that any loss taking the form of a weighted Bregman divergence permits analytic computation of a central component of its corresponding objective, making the approach applicable in practice. In regression and classification experiments with a range of different losses, we find our approach reduces test losses relative to existing techniques.
Read the full abstract ▾
Machine Learning
Offline-Online Reinforcement Learning for Linear Mixture MDPs
Zhongjun Zhang and Sean R. Sinclair
We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift. In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment, while in the online phase the learner interacts with the target environment. We propose an
algorithm that adaptively leverages offline data. When the offline data are informative, either due to sufficient coverage or small environment shift, the
algorithm provably improves over purely online learning. When the offline data are uninformative, it safely ignores them and matches the online-only performance. We establish regret upper bounds that explicitly characterize when offline data are beneficial, together with nearly matching lower bounds. Numerical experiments further corroborate our theoretical findings.
Read the full abstract ▾
Machine Learning
Exploring Concept Subspace for Self-explainable Text-Attributed Graph Learning
Xiaoxue Han, Libo Zhang, Zining Zhu et al. (+1 more)
We introduce Graph Concept Bottleneck (GCB) as a new paradigm for self-explainable text-attributed graph learning. GCB maps graphs into a subspace, concept bottleneck, where each concept is a meaningful phrase, and predictions are made based on the activation of these concepts. Unlike existing interpretable graph learning methods that primarily rely on subgraphs as explanations, the concept bottleneck provides a new form of interpretation. To refine the concept space, we apply the information bottleneck principle to focus on the most relevant concepts. This not only yields more concise and faithful explanations but also explicitly guides the model to "think" toward the correct decision. We empirically show that GCB achieves intrinsic interpretability with accuracy on par with black-box Graph Neural Networks. Moreover, it delivers better performance under distribution shifts and data perturbations, showing improved robustness and generalizability, benefitting from concept-guided prediction.
Read the full abstract ▾
All authors Xiaoxue Han, Libo Zhang, Zining Zhu, Yue Ning
Machine Learning · Cross-listed from cs.NI
A Geometric Algebra-informed NeRF Framework for Generalizable Wireless Channel Prediction
Jingzhou Shen, Luis Lago Enamorado, Shiwen Mao et al. (+1 more)
In this paper, we propose the
geometric algebra-informed neural radiance fields (GAI-NeRF), a novel framework for wireless channel prediction that leverages
geometric algebra attention mechanisms to capture ray-object interactions in complex propagation environments. Our approach incorporates global
token representations, drawing inspiration from
transformer architectures in
language and
vision domains, to aggregate learned spatial-electromagnetic features and enhance scene understanding. We identify limitations in conventional static ray tracing modules that hinder model generalization and address this challenge through a new ray tracing architecture. This design enables effective generalization across diverse wireless scenarios while maintaining computational efficiency. Experimental results demonstrate that GAI-NeRF achieves superior performance in channel prediction tasks by combining
geometric algebra principles with neural scene representations, offering a promising direction for next-generation wireless communication systems. Moreover, GAI-NeRF greatly outperforms existing methods across multiple wireless scenarios. To ensure comprehensive assessment, we further evaluate our approach against multiple
benchmarks using newly collected real-world indoor
datasets tailored for single-scene downstream tasks and generalization testing, confirming its robust performance in unseen environments and establishing its high efficacy for wireless channel prediction.
Read the full abstract ▾
All authors Jingzhou Shen, Luis Lago Enamorado, Shiwen Mao, Xuyu Wang
Machine Learning
Multi-Head Residual-Gated DeepONet for Coherent Nonlinear Wave Dynamics
Zhiwei Fan, Yiming Pan and Daniel Coca
Coherent nonlinear wave dynamics are often strongly shaped by a compact set of physically meaningful descriptors of the initial state. Traditional neural operators typically treat the input-output mapping as a largely black-box high-dimensional regression problem, without explicitly exploiting this structured physical context. Common feature-integration strategies usually rely on direct concatenation or FiLM-style affine modulation in hidden latent spaces. Here we introduce a different paradigm, loosely inspired by the complementary roles of state
evolution and physically meaningful observables in
quantum mechanics: the wave field is learned through a standard DeepONet state pathway, while compact physical descriptors follow a parallel conditioning pathway and act as residual modulation factors on the state prediction. Based on this idea, we develop a Multi-Head Residual-Gated DeepONet (MH-RG), which combines a pre-branch residual modulator, a branch residual gate, and a trunk residual gate with a low-rank multi-head mechanism to capture multiple complementary conditioned response patterns without prohibitive parameter growth. We evaluate the framework on representative
benchmarks including highly nonlinear conservative wave dynamics and dissipative trapped dynamics and further perform detailed mechanistic analyses of the learned multi-head gating behavior. Compared with feature-augmented baselines, MH-RG DeepONet achieves consistently lower error while better preserving phase coherence and the fidelity of physically relevant dynamical quantities.
Read the full abstract ▾