Machine Learning · Cross-listed from math.OC
Riemannian Dueling Optimization
Yuxuan Ren, Abhishek Roy and Shiqian Ma
Dueling optimization considers optimizing an objective with access to only a comparison oracle of the objective function. It finds important applications in emerging fields such as recommendation systems and robotics. Existing works on dueling optimization mainly focused on unconstrained problems in the Euclidean space. In this work, we study dueling optimization over Riemannian
manifolds, which covers important applications that cannot be solved by existing dueling optimization
algorithms. In particular, we propose a Riemannian Dueling Normalized Gradient Descent (RDNGD) method and establish its iteration complexity when the objective function is geodesically L-smooth or geodesically (strongly) convex. We also propose a projection-free
algorithm, named Riemannian Dueling Frank-Wolfe (RDFW) method, to deal with the situation where projection is prohibited. We establish the iteration and oracle complexities for RDFW. We illustrate the effectiveness of the proposed
algorithms through numerical experiments on both synthetic and real applications.
Read the full abstract ▾
Machine Learning · Cross-listed from math.GN
Universality of shallow and deep neural networks on non-Euclidean spaces
Vugar Ismailov
We study shallow and deep neural networks whose inputs range over a general topological space. The model is built from a prescribed family of continuous feature maps and reduces to multilayer feedforward networks in the Euclidean case. We focus on the universal approximation property and establish general conditions under which such networks are dense in spaces of continuous vector-valued functions on arbitrary topological spaces and, in particular, locally convex spaces. Universality results obtained in the arbitrary-width case extend classical approximation
theorems to non-Euclidean spaces. We also consider the deep narrow setting, in which the width of each hidden layer is uniformly bounded while the depth is allowed to grow. We identify conditions under which such networks retain the universal approximation property. As a concrete example, we employ Ostrand's extension of the Kolmogorov superposition
theorem to derive an explicit universality result for products of compact metric spaces, with width bounds expressed in terms of topological dimension.
Read the full abstract ▾
Machine Learning
BamaER: A Behavior-Aware Memory-Augmented Model for Exercise Recommendation
Qing Yang, Yuhao Jiang, Rui Wang et al. (+6 more)
Exercise recommendation focuses on personalized exercise selection conditioned on students' learning history, personal interests, and other individualized characteristics. Despite notable progress, most existing methods represent student learning solely as exercise sequences, overlooking rich behavioral interaction information. This limited representation often leads to biased and unreliable estimates of learning progress. Moreover, fixed-length sequence
segmentation limits the incorporation of early learning experiences, thereby hindering the modeling of long-term dependencies and the accurate estimation of knowledge mastery. To address these limitations, we propose BamaER, a Behavior-aware
memory-augmented Exercise Recommendation framework that comprises three core modules: (i) the learning progress prediction module that captures heterogeneous student interaction behaviors via a tri-directional hybrid encoding scheme; (ii) the
memory-augmented knowledge tracing module that maintains a dynamic
memory matrix to jointly model historical and current knowledge states for robust mastery estimation; and (iii) the exercise filtering module that formulates candidate selection as a diversity-aware optimization problem, solved via the Hippopotamus Optimization
Algorithm to reduce redundancy and improve recommendation coverage. Experiments on five real-world educational
datasets show that BamaER consistently outperforms state-of-the-art baselines across a range of evaluation metrics.
Read the full abstract ▾
All authors Qing Yang, Yuhao Jiang, Rui Wang, Jipeng Guo, Yejiang Wang, Xinghe Cheng, Zezheng Wu, Jiapu Wang, Jingwei Zhang
Machine Learning
IT-OSE: Exploring Optimal Sample Size for Industrial Data Augmentation
Mingchun Sun, Rongqiang Zhao, Zhennan Huang et al. (+2 more)
In industrial scenarios, data augmentation is an effective approach to improve model performance. However, its benefits are not unidirectionally beneficial. There is no theoretical research or established estimation for the optimal sample size (OSS) in augmentation, nor is there an established metric to evaluate the accuracy of OSS or its deviation from the ground truth. To address these issues, we propose an information-theoretic optimal sample size estimation (IT-OSE) to provide reliable OSS estimation for industrial data augmentation. An interval coverage and deviation (ICD) score is proposed to evaluate the estimated OSS intuitively. The relationship between OSS and dominant factors is theoretically analyzed and formulated, thereby enhancing the interpretability. Experiments show that, compared to empirical estimation, the IT-OSE increases accuracy in classification tasks across baseline models by an average of 4.38%, and reduces MAPE in regression tasks across baseline models by an average of 18.80%. The improvements in downstream model performance are more stable. ICDdev in the ICD score is also reduced by an average of 49.30%. The determinism of OSS is enhanced. Compared to exhaustive search, the IT-OSE achieves the same OSS while reducing computational and data costs by an average of 83.97% and 93.46%. Furthermore, practicality experiments demonstrate that the IT-OSE exhibits generality across representative sensor-based industrial scenarios.
Read the full abstract ▾
All authors Mingchun Sun, Rongqiang Zhao, Zhennan Huang, Songyu Ding, Jie Liu
Machine Learning · Cross-listed from cs.CV
OMNI-Dent: Towards an Accessible and Explainable AI Framework for Automated Dental Diagnosis
Leeje Jang, Yao-Yi Chiang, Angela M. Hastings et al. (+8 more)
Accurate dental diagnosis is essential for oral healthcare, yet many individuals lack access to timely professional evaluation. Existing AI-based methods primarily treat diagnosis as a visual pattern recognition task and do not reflect the structured clinical
reasoning used by dental professionals. These approaches also require large amounts of expert-annotated data and often struggle to generalize across diverse real-world imaging conditions. To address these limitations, we present OMNI-Dent, a data-efficient and explainable diagnostic framework that incorporates clinical
reasoning principles into a
Vision-
Language Model (VLM)-based pipeline. The framework operates on multi-view smartphone photographs,embeds diagnostic heuristics from dental experts, and guides a general-purpose VLM to perform tooth-level evaluation without dental-specific fine-tuning of the VLM. By utilizing the VLM's existing visual-linguistic capabilities, OMNI-Dent aims to support diagnostic assessment in settings where curated clinical imaging is unavailable. Designed as an early-stage assistive tool, OMNI-Dent helps users identify potential abnormalities and determine when professional evaluation may be needed, offering a practical option for individuals with limited access to in-person care.
Read the full abstract ▾
All authors Leeje Jang, Yao-Yi Chiang, Angela M. Hastings, Patimaporn Pungchanchaikul, Martha B. Lucas, Emily C. Schultz, Jeffrey P. Louie, Mohamed Estai, Wen-Chen Wang, Ryan H.L. Ip, Boyen Huang
Machine Learning · Cross-listed from cs.AI
DLLM-Searcher: Adapting Diffusion Large Language Model for Search Agents
Jiahao Zhao, Shaoxuan Xu, Zhongxiang Sun et al. (+6 more)
Recently,
Diffusion Large
Language Models (dLLMs) have demonstrated unique efficiency advantages, enabled by their inherently parallel decoding mechanism and flexible generation paradigm. Meanwhile, despite the rapid advancement of Search
Agents, their practical deployment is constrained by a fundamental limitation, termed as 1) Latency Challenge: the serial execution of multi-round
reasoning, tool calling, and tool response waiting under the ReAct
agent paradigm induces severe end-to-end latency. Intuitively, dLLMs can leverage their distinctive strengths to optimize the operational efficiency of
agents under the ReAct
agent paradigm. Practically, existing dLLM backbones face the 2)
Agent Ability Challenge. That is, existing dLLMs exhibit remarkably weak
reasoning and tool-calling capabilities, preventing these advantages from being effectively realized in practice. In this paper, we propose DLLM-Searcher, an optimization framework for dLLM-based Search
Agents. To solve the
Agent Ability Challenge, we design a two-stage post-training pipeline encompassing Agentic Supervised Fine-Tuning (Agentic SFT) and Agentic Variance-Reduced Preference Optimization Agentic VRPO, which enhances the backbone dLLM's information seeking and
reasoning capabilities. To mitigate the Latency Challenge, we leverage the flexible generation mechanism of dLLMs and propose a novel
agent paradigm termed Parallel-
Reasoning and Acting P-ReAct. P-ReAct guides the model to prioritize decoding tool_call instructions, thereby allowing the model to keep thinking while waiting for the tool's return. Experimental results demonstrate that DLLM-Searcher achieves performance comparable to mainstream LLM-based search
agents and P-ReAct delivers approximately 15% inference acceleration. Our code is available at https://anonymous.4open.science/r/DLLM-Searcher-553C
Read the full abstract ▾
All authors Jiahao Zhao, Shaoxuan Xu, Zhongxiang Sun, Fengqi Zhu, Jingyang Ou, Yuling Shi, Chongxuan Li, Xiao Zhang, Jun Xu
Machine Learning
TransConv-DDPM: Enhanced Diffusion Model for Generating Time-Series Data in Healthcare
Md Shahriar Kabir, Sana Alamgeer, Minakshi Debnath et al. (+1 more)
The lack of real-world data in clinical fields poses a major obstacle in training effective AI models for diagnostic and preventive tools in medicine. Generative AI has shown promise in increasing data volume and enhancing model training, particularly in computer
vision and natural
language processing (NLP) domains. However, generating physiological time-series data, a common type in medical AI applications, presents unique challenges due to its inherent complexity and variability. This paper introduces TransConv-DDPM, an enhanced generative AI method for biomechanical and physiological time-series data generation. The model employs a denoising
diffusion probabilistic model (DDPM) with U-Net, multi-scale convolution modules, and a
transformer layer to capture both global and local temporal dependencies. We evaluated TransConv-DDPM on three diverse
datasets, generating both long and short-sequence time-series data. Quantitative comparisons against state-of-the-art methods, TimeGAN and
Diffusion-TS, using four performance metrics, demonstrated promising results, particularly on the SmartFallMM and
EEG datasets, where it effectively captured the more gradual temporal change patterns between data points. Additionally, a utility test on the SmartFallMM
dataset revealed that adding synthetic fall data generated by TransConv-DDPM improved predictive model performance, showing a 13.64% improvement in F1-score and a 14.93% increase in overall accuracy compared to the baseline model trained solely on fall data from the SmartFallMM
dataset. These findings highlight the potential of TransConv-DDPM to generate high-quality synthetic data for real-world applications.
Read the full abstract ▾
All authors Md Shahriar Kabir, Sana Alamgeer, Minakshi Debnath, Anne H. H. Ngu
Machine Learning
Lagged backward-compatible physics-informed neural networks for unsaturated soil consolidation analysis
Dong Li, Shuai Huang, Yapeng Cao et al. (+3 more)
This study develops a Lagged Backward-Compatible Physics-Informed Neural Network (LBC-PINN) for simulating and inverting one-dimensional unsaturated soil consolidation under long-term loading. To address the challenges of coupled air and water pressure dissipation across multi-scale time domains, the framework integrates logarithmic time
segmentation, lagged compatibility loss enforcement, and segment-wise transfer learning. In forward analysis, the LBC-PINN with recommended
segmentation schemes accurately predicts pore air and pore water pressure
evolution. Model predictions are validated against finite element method (FEM) results, with mean absolute errors below 1e-2 for time durations up to 1e10 seconds. A simplified
segmentation strategy based on the characteristic air-phase dissipation time improves computational efficiency while preserving predictive accuracy. Sensitivity analyses confirm the robustness of the framework across air-to-water permeability ratios ranging from 1e-3 to 1e3.
Read the full abstract ▾
All authors Dong Li, Shuai Huang, Yapeng Cao, Yujun Cui, Xiaobin Wei, Hongtao Cao
Machine Learning · Cross-listed from cs.DC
PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
Rui Ning, Wei Zhang and Fan Lai
Attention efficiency is critical to large
language model (LLM) inference. While prior advances optimize
attention execution for individual requests (e.g., FlashAttention), production LLM serving relies on batching requests with highly heterogeneous sequence lengths for high serving throughput. This mismatch induces severe computation and I/O imbalance, exacerbates stragglers, and underutilizes GPU resources. We present PackInfer, a kernel-level
attention framework that enables compute- and I/O-aware execution for heterogeneous batched inference. PackInfer orchestrates batched requests into load-balanced execution groups, effectively saturating GPU utilization by packing multiple requests into unified kernel launches. By constructing
attention kernels directly over packed query-key regions, PackInfer eliminates redundant computation and balances thread-block execution. It then incorporates I/O-aware grouping that co-locates shared-prefix requests and reorganizes KV caches into group-contiguous layouts, reducing
memory fragmentation and redundant data movement as generation evolves. Evaluations on real-world workloads show that PackInfer reduces inference latency by 13.0-20.1%, and improves throughput by 20% compared to the state-of-the-art FlashAttention.
Read the full abstract ▾
Machine Learning
Causal Discovery from Heteroscedastic Stochastic Dynamical Systems under Imperfect Physical Models
Jianhong Chen, Naichen Shi and Xubo Yue
Causal discovery is a data-driven paradigm for analyzing complex systems, while physics-based models, such as ordinary differential equations (ODEs), provide mechanistic structure for real-world dynamical processes. Integrating these paradigms can improve identifiability, stability, and robustness. However, real dynamical systems often exhibit cyclic interactions and nonstationarity, whereas many causal discovery methods rely on acyclicity, stationarity, or
equilibrium assumptions. We propose an integrative causal discovery framework for dynamical systems that leverages partial physical knowledge through
stochastic differential equations (SDEs). The drift term encodes known ODE dynamics, while the
diffusion term captures unknown causal couplings beyond the prescribed physics. We develop a scalable sparsity-inducing maximum quasi-likelihood
estimator with a theoretically justified stabilization technique to improve the optimization landscape. Under mild conditions, we establish causal graph recovery guarantees for both stable and unstable SDEs. We also analyze robustness of our causal graph estimate to ODE misspecification and clarify how the introduced stabilization technique balances numerical stability and statistical recoverability. Experiments on linear SDEs and nonlinear
benchmarks, including Lotka-Volterra and Lorenz dynamics with acyclic and cyclic structures, show improved graph recovery and robustness over data-driven baselines. We also demonstrate practical utility on real-world epidemic data by reconstructing
stochastic SIR dynamics within our causal discovery framework.
Read the full abstract ▾
Machine Learning
LISA: Laplacian In-context Spectral Analysis
Julio Candanedo
We propose
Laplacian In-context Spectral Analysis (LISA), a method for inference-time adaptation of
Laplacian-based time-series models using only an observed prefix. LISA combines delay-coordinate
embeddings and
Laplacian spectral learning to produce
diffusion-coordinate state representations, together with a frozen nonlinear decoder for one-step prediction. We introduce lightweight latent-space residual adapters based on either Gaussian-process regression or an
attention-like Markov operator over context windows. Across
forecasting and autoregressive rollout experiments, LISA improves over the frozen baseline and is often most beneficial under changing dynamics. This work links in-context adaptation to nonparametric spectral methods for dynamical systems.
Read the full abstract ▾
Machine Learning
DCER: Dual-Stage Compression and Energy-Based Reconstruction
Yiwen Wang and Jiahao Qin
Multimodal fusion faces two robustness challenges: noisy inputs degrade representation quality, and missing modalities cause prediction failures. We propose DCER, a unified framework addressing both challenges through dual-stage compression and energy-based reconstruction. The compression stage operates at two levels: within-modality frequency transforms (wavelet for audio, DCT for
video) remove noise while preserving task-relevant patterns, and cross-modality bottleneck
tokens force genuine integration rather than modality-specific shortcuts. For missing modalities, energy-based reconstruction recovers representations via gradient descent on a learned energy function, with the final energy providing intrinsic uncertainty quantification (h̊o} > 0.72 correlation with prediction error). Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate state-of-the-art performance across all
benchmarks, with a U-shaped robustness pattern favoring
multimodal fusion at both complete and high-missing conditions. The code will be available on Github.
Read the full abstract ▾
Machine Learning
Mind the Performance Gap: Capability-Behavior Trade-offs in Feature Steering
Eitan Sprejer, Oscar Agustín Stanchi, María Victoria Carro et al. (+2 more)
Feature steering has emerged as a promising approach for controlling LLM behavior through direct manipulation of internal representations, offering advantages over
prompt engineering. However, its practical effectiveness in real-world applications remains poorly understood, particularly regarding potential trade-offs with output quality. We show that feature steering methods substantially degrade model performance even when successfully controlling target behaviors, a critical trade-off. Specifically, we evaluate Goodfire's Auto Steer against
prompt engineering baselines across 14 steering queries (covering innocuous and safety-relevant behaviors) on 171 Massive Multitask
Language Understanding (MMLU) questions using Llama-8B and Llama-70B, measuring accuracy, coherence, and behavioral control. Our findings show that Auto Steer successfully modifies target behaviors (achieving scores of 3.33 vs. 2.98 for prompting on Llama-8B and 3.57 vs. 3.10 on Llama-70B), but causes dramatic performance degradation: accuracy on the MMLU questions drops from 66% to 46% on Llama-8B and 87% to 73% on Llama-70B, with coherence falling from 4.62 to 2.24 and 4.94 to 3.89 respectively. Simple prompting achieves the best overall balance. These findings highlight limitations of current feature steering methods for practical deployment where task performance cannot be sacrificed. More broadly, our work demonstrates that mechanistic control methods face fundamental capability-behavior trade-offs that must be empirically characterized before deployment.
Read the full abstract ▾
All authors Eitan Sprejer, Oscar Agustín Stanchi, María Victoria Carro, Denise Alejandra Mester, Iván Arcuschin
Machine Learning
Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability
Kingsuk Maitra
The Mechanistic Interpretability (MI) program has mapped the
Transformer as a precise computational graph. We extend this graph with a conservation law and time-varying AC dynamics, viewing it as a physical
circuit. We introduce Momentum
Attention, a
symplectic augmentation
embedding physical priors via the kinematic difference operator $p_t = q_t - q_{t-1}$, implementing the
symplectic shear $\hat{q}_t = q_t + \gamma p_t$ on queries and keys. We identify a fundamental
Symplectic-Filter Duality: the physical shear is mathematically equivalent to a High-Pass Filter. This duality is our cornerstone contribution – by injecting kinematic momentum, we sidestep the topological depth constraint ($L \geq 2$) for induction head formation. While standard architectures require two layers for induction from static positions, our extension grants direct access to velocity, enabling Single-Layer Induction and Spectral Forensics via Bode Plots. We formalize an Orthogonality
Theorem proving that DC (semantic) and AC (mechanistic) signals segregate into orthogonal frequency bands when Low-Pass RoPE interacts with High-Pass Momentum. Validated through 5,100+ controlled experiments (documented in Supplementary Appendices A–R and 27 Jupyter notebooks), our 125M Momentum model exceeds expectations on induction-heavy tasks while tracking a 350M baseline within $\sim$2.9% validation loss. Dedicated associative recall experiments reveal a scaling law $\gamma^* = 4.17 \times N^{-0.74}$ establishing momentum-depth fungibility. We offer this framework as a complementary analytical toolkit connecting Generative AI,
Hamiltonian Physics, and Signal Processing.
Read the full abstract ▾
Machine Learning · Cross-listed from q-bio.GN
Beyond Independent Genes: Learning Module-Inductive Representations for Single-Cell Gene Perturbation Prediction
Jiafa Ruan, Ruijie Quan, Liyang Xu et al. (+2 more)
Predicting transcriptional responses to genetic perturbations is a central problem in functional genomics. In practice, perturbation responses are rarely gene-independent but instead manifest as coordinated, program-level transcriptional changes among functionally related genes. However, most existing methods do not explicitly model such coordination, due to gene-wise modeling paradigms and reliance on static biological priors that cannot capture dynamic program reorganization. To address these limitations, we propose scBIG, a module-inductive perturbation prediction framework that explicitly models coordinated gene programs. scBIG induces coherent gene programs from data via Gene-Relation Clustering, captures inter-program interactions through a Gene-Cluster-Aware Encoder, and preserves modular coordination using structure-aware
alignment objectives. These structured representations are then modeled using conditional flow matching to enable flexible and generalizable perturbation prediction. Extensive experiments on multiple single-
cell perturbation
benchmarks show that scBIG consistently outperforms state-of-the-art methods, particularly on unseen and combinatorial perturbation settings, achieving an average improvement of 6.7% over the strongest baselines. The code is available at https://github.com/ttruan2426-dot/scBIG.
Read the full abstract ▾
All authors Jiafa Ruan, Ruijie Quan, Liyang Xu, Zongxin Yang, Yi Yang
Machine Learning · Cross-listed from cs.CR
Privacy Amplification Persists under Unlimited Synthetic Data Release
Clément Pierquin, Aurélien Bellet, Marc Tommasi et al. (+1 more)
We study
privacy amplification by synthetic data release, a phenomenon in which differential
privacy guarantees are improved by releasing only synthetic data rather than the private generative model itself. Recent work by Pierquin et al. (2025) established the first formal amplification guarantees for a linear generator, but they apply only in asymptotic regimes where the model dimension far exceeds the number of released synthetic records, limiting their practical relevance. In this work, we show a surprising result: under a bounded-parameter assumption,
privacy amplification persists even when releasing an unbounded number of synthetic records, thereby improving upon the bounds of Pierquin et al. (2025). Our analysis provides structural insights that may guide the development of tighter
privacy guarantees for more complex release mechanisms.
Read the full abstract ▾
All authors Clément Pierquin, Aurélien Bellet, Marc Tommasi, Matthieu Boussard
Machine Learning · Cross-listed from cs.AI
Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RL
Xiaofeng Lin, Sirou Zhu, Yilei Chen et al. (+6 more)
Large
language models (LLMs) achieve strong performance when all task-relevant information is available upfront, as in static prediction and instruction-following problems. However, many real-world decision-making tasks are inherently online: crucial information must be acquired through interaction, feedback is delayed, and effective behavior requires balancing information collection and exploitation over time. While in-context learning enables adaptation without weight updates, existing LLMs often struggle to reliably leverage in-context interaction experience in such settings. In this work, we show that this limitation can be addressed through training. We introduce ORBIT, a multi-task, multi-episode meta-reinforcement learning framework that trains LLMs to learn from interaction in context. After meta-training, a relatively small open-source model (Qwen3-14B) demonstrates substantially improved in-context online learning on entirely unseen environments, matching the performance of GPT-5.2 and outperforming standard RL fine-tuning by a large margin. Scaling experiments further reveal consistent gains with model size, suggesting significant headroom for learn-at-inference-time decision-making
agents. Code reproducing the results in the paper can be found at https://github.com/XiaofengLin7/ORBIT.
Read the full abstract ▾
All authors Xiaofeng Lin, Sirou Zhu, Yilei Chen, Mingyu Chen, Hejian Sang, Ioannis Paschalidis, Zhipeng Wang, Aldo Pacchiano, Xuezhou Zhang
Machine Learning
A Probabilistic Framework for Solving High-Frequency Helmholtz Equations via Diffusion Models
Yicheng Zou, Samuel Lanthaler and Hossein Salahshoor
Deterministic neural operators perform well on many PDEs but can struggle with the approximation of high-frequency wave phenomena, where strong input-to-output sensitivity makes operator learning challenging, and spectral bias blurs oscillations. We argue for adopting a probabilistic approach for approximating waves in high-frequency regime, and develop our probabilistic framework using a score-based conditional
diffusion operator. After demonstrating a stability analysis of the Helmholtz operator, we present our numerical experiments across a wide range of frequencies, benchmarked against other popular data-driven and machine learning approaches for waves. We show that our probabilistic neural operator consistently produces robust predictions with the lowest errors in $L^2$, $H^1$, and energy norms. Moreover, unlike all the other tested deterministic approaches, our framework remarkably captures uncertainties in the input sound speed map propagated to the solution field. We envision that our results position probabilistic operator learning as a principled and effective approach for solving complex PDEs such as Helmholtz in the challenging high-frequency regime.
Read the full abstract ▾
Machine Learning
Principles of Lipschitz continuity in neural networks
Róisín Luo
Deep learning has achieved remarkable success across a wide range of domains, significantly expanding the frontiers of what is achievable in artificial intelligence. Yet, despite these advances, critical challenges remain – most notably, ensuring robustness to small input perturbations and generalization to out-of-distribution data. These critical challenges underscore the need to understand the underlying fundamental principles that govern robustness and generalization. Among the theoretical tools available, Lipschitz continuity plays a pivotal role in governing the fundamental properties of neural networks related to robustness and generalization. It quantifies the worst-case sensitivity of network's outputs to small input perturbations. While its importance is widely acknowledged, prior research has predominantly focused on empirical regularization approaches based on Lipschitz constraints, leaving the underlying principles less explored. This thesis seeks to advance a principled understanding of the principles of Lipschitz continuity in neural networks within the paradigm of machine learning, examined from two complementary perspectives: an internal perspective – focusing on the temporal
evolution of Lipschitz continuity in neural networks during training (i.e., training dynamics); and an external perspective – investigating how Lipschitz continuity modulates the behavior of neural networks with respect to features in the input data, particularly its role in governing frequency signal propagation (i.e., modulation of frequency signal propagation).
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Efficient Subgroup Analysis via Optimal Trees with Global Parameter Fusion
Zhongming Xie, Joseph Giorgio and Jingshen Wang
Identifying and making statistical inferences on differential treatment effects (commonly known as subgroup analysis in clinical research) is central to precision health. Subgroup analysis allows practitioners to pinpoint populations for whom a treatment is especially beneficial or protective, thereby advancing targeted interventions. Tree based recursive partitioning methods are widely used for subgroup analysis due to their interpretability. Nevertheless, these approaches encounter significant limitations, including suboptimal partitions induced by greedy heuristics and overfitting from locally estimated splits, especially under limited sample sizes. To address these limitations, we propose a fused optimal causal tree method that leverages mixed integer optimization (MIO) to facilitate precise subgroup identification. Our approach ensures globally optimal partitions and introduces a parameter fusion constraint to facilitate information sharing across related subgroups. This design substantially improves subgroup discovery accuracy and enhances statistical efficiency. We provide theoretical guarantees by rigorously establishing out of sample risk bounds and comparing them with those of classical tree based methods. Empirically, our method consistently outperforms popular baselines in simulations. Finally, we demonstrate its practical utility through a case study on the Health and Aging
Brain Study Health Disparities (HABS-HD)
dataset, where our approach yields clinically meaningful insights.
Read the full abstract ▾
Machine Learning · Cross-listed from cond-mat.mtrl-sci
Thermodynamic assessment of machine learning models for solid-state synthesis prediction
Jane Schlesinger, Simon Hjaltason, Nathan J. Szymanski et al. (+1 more)
Machine learning models have recently emerged to predict whether hypothetical solid-state materials can be synthesized. These models aim to circumvent direct first-principles modeling of solid-state phase transformations, instead learning from large databases of successfully synthesized materials. Here, we assess the
alignment of several recently introduced synthesis prediction models with material and reaction thermodynamics, quantified by the energy with respect to the convex hull and a metric accounting for thermodynamic selectivity of enumerated synthesis reactions. A
dataset of successful synthesis recipes was used to determine the likely bounds on both quantities beyond which materials can be deemed unlikely to be synthesized. With these bounds as context, thermodynamic quantities were computed using the CHGNet foundation potential for thousands of new hypothetical materials generated using the Chemeleon generative model. Four recently published machine learning models for synthesizability prediction were applied to this same
dataset, and the resultant predictions were considered against computed thermodynamics. We find these models generally overpredict the likelihood of synthesis, but some model scores do trend with thermodynamic heuristics, assigning lower scores to materials that are less stable or do not have an available synthesis recipe that is calculated to be thermodynamically selective. In total, this work identifies existing gaps in machine learning models for materials synthesis and introduces a new approach to assess their quality in the absence of extensive negative examples (failed syntheses).
Read the full abstract ▾
All authors Jane Schlesinger, Simon Hjaltason, Nathan J. Szymanski, Christopher J. Bartel
Machine Learning
Stroke Lesions as a Rosetta Stone for Language Model Interpretability
Julius Fridriksson, Roger D. Newman-Norlund, Saeed Ahmadi et al. (+10 more)
Large
language models (LLMs) have achieved remarkable capabilities, yet methods to verify which model components are truly necessary for
language function remain limited. Current interpretability approaches rely on internal metrics and lack external validation. Here we present the
Brain-LLM Unified Model (BLUM), a framework that leverages lesion-symptom mapping, the gold standard for establishing causal
brain-behavior relationships for over a century, as an external reference structure for evaluating LLM perturbation effects. Using data from individuals with chronic post-stroke aphasia (N = 410), we trained symptom-to-lesion models that predict
brain damage location from behavioral error profiles, applied systematic perturbations to
transformer layers, administered identical clinical assessments to perturbed LLMs and human patients, and projected LLM error profiles into human lesion space. LLM error profiles were sufficiently similar to human error profiles that predicted lesions corresponded to actual lesions in error-matched humans above chance in 67% of picture naming conditions (p < 10^{-23}) and 68.3% of sentence completion conditions (p < 10^{-61}), with semantic-dominant errors mapping onto ventral-stream lesion patterns and phonemic-dominant errors onto dorsal-stream patterns. These findings open a new methodological avenue for LLM interpretability in which clinical neuroscience provides external validation, establishing human lesion-symptom mapping as a reference framework for evaluating artificial
language systems and motivating direct investigation of whether behavioral
alignment reflects shared computational principles.
Read the full abstract ▾
All authors Julius Fridriksson, Roger D. Newman-Norlund, Saeed Ahmadi, Regan Willis, Nadra Salman, Kalil Warren, Xiang Guan, Yong Yang, Srihari Nelakuditi, Rutvik Desai, Leonardo Bonilha, Jeff Charney, Chris Rorden
Machine Learning
Agentic AI-Empowered Dynamic Survey Framework
Furkan Mumcu, Lokman Bekit, Michael J. Jones et al. (+2 more)
Survey papers play a central role in synthesizing and organizing scientific knowledge, yet they are increasingly strained by the rapid growth of research output. As new work continues to appear after publication, surveys quickly become outdated, contributing to redundancy and fragmentation in the literature. We reframe survey writing as a long-horizon maintenance problem rather than a one-time generation task, treating surveys as living documents that evolve alongside the research they describe. We propose an agentic Dynamic Survey Framework that supports the continuous updating of existing survey papers by incrementally integrating new work while preserving survey structure and minimizing unnecessary disruption. Using a retrospective experimental setup, we demonstrate that the proposed framework effectively identifies and incorporates emerging research while preserving the coherence and structure of existing surveys, even over extended sequences of updates.
Read the full abstract ▾
All authors Furkan Mumcu, Lokman Bekit, Michael J. Jones, Anoop Cherian, Yasin Yilmaz
Machine Learning
An Empirical Survey and Benchmark of Learned Distance Indexes for Road Networks
Gautam Choudhary, Libin Zhou, Yeasir Rayhan et al. (+1 more)
The calculation of shortest-path distances in road networks is a core operation in
navigation systems, location-based services, and spatial analytics. Although classical
algorithms, e.g., Dijkstra's
algorithm, provide exact answers, their latency is prohibitive for modern real-time, large-scale deployments. Over the past two decades, numerous distance indexes have been proposed to speed up query processing for shortest distance queries. More recently, with advances in machine learning (ML), researchers have designed and proposed ML-based distance indexes to answer approximate shortest path and distance queries efficiently. However, a comprehensive and systematic evaluation of these ML-based approaches is lacking. This paper presents the first empirical survey of ML-based distance indexes on road networks, evaluating them along four key dimensions: training time, query latency, storage, and accuracy. Using 13 real-world road networks (7 with workload-driven queries, derived from trajectory data, and 6 with synthetically generated queries), we
benchmark ten representative ML techniques and compare them against classical non-ML baselines, highlighting key insights and practical trade-offs. We release a unified open-source codebase to support reproducibility and future research on learned indexes.
Read the full abstract ▾
All authors Gautam Choudhary, Libin Zhou, Yeasir Rayhan, Walid G. Aref
Machine Learning
SEIS: Subspace-based Equivariance and Invariance Scores for Neural Representations
Huahua Lin, Katayoun Farrahi and Xiaohao Cai
Understanding how neural representations respond to
geometric transformations is essential for evaluating whether learned features preserve meaningful spatial structure. Existing approaches primarily assess robustness primarily by comparing model outputs under transformed inputs, offering limited insight into how
geometric information is organized within internal representations and failing to distinguish between information loss and re-encoding. In this work, we introduce SEIS (Subspace-based Equivariance and Invariance Scores), a subspace metric for analyzing layer-wise feature representations under
geometric transformations, disentangling equivariance from invariance without requiring labels or explicit knowledge of the transformation. Through controlled experiments across diverse architectures, we uncover several consistent patterns. First, convolutional encoders exhibit a depth-wise transition from strong equivariance to increasing invariance, with both properties stabilizing within the first few training epochs. In
segmentation decoders, however, equivariance tends to recover in later layers. Second, this trade-off is not intrinsic but is shaped by training decisions: data augmentation actively strengthens both equivariance and invariance simultaneously, and multi-task learning induces synergistic gains in both properties beyond what either task achieves alone. Extending our analysis beyond convolutional networks, we find that
transformer-based models exhibit distinct
geometric behaviors, while MLP-Mixers display intermediate characteristics.
Read the full abstract ▾
Machine Learning
Partition Tree: Conditional Density Estimation over General Outcome Spaces
Felipe Angelim and Alessandro Leite
We propose Partition Tree, a tree-based framework for conditional density estimation over general outcome spaces that supports both continuous and categorical variables within a unified formulation. Our approach models conditional distributions as piecewise-constant densities on data-adaptive partitions and learns trees by greedily minimizing a surrogate for conditional log-loss. This yields a scalable alternative to existing probabilistic trees that does not make parametric assumptions about the target distribution. We further introduce Partition Forest, an ensemble extension obtained by averaging conditional densities. Empirically, we demonstrate improved probabilistic prediction over CART-style trees and competitive performance compared to state-of-the-art probabilistic tree methods and Random Forests.
Read the full abstract ▾
Machine Learning
DADP: Domain Adaptive Diffusion Policy
Pengcheng Wang, Qinghang Liu, Haotian Lin et al. (+4 more)
Learning domain adaptive policies that can generalize to unseen transition dynamics, remains a fundamental challenge in learning-based control. Substantial progress has been made through domain representation learning to capture domain-specific information, thus enabling domain-aware decision making. We analyze the process of learning domain representations through dynamical prediction and find that selecting contexts adjacent to the current step causes the learned representations to entangle static domain information with varying dynamical properties. Such mixture can confuse the conditioned policy, thereby constraining zero-shot adaptation. To tackle the challenge, we propose DADP (Domain Adaptive
Diffusion Policy), which achieves robust adaptation through unsupervised disentanglement and domain-aware
diffusion injection. First, we introduce Lagged Context Dynamical Prediction, a strategy that conditions future state estimation on a historical offset context; by increasing this temporal gap, we unsupervisedly disentangle static domain representations by filtering out transient properties. Second, we integrate the learned domain representations directly into the generative process by biasing the prior distribution and reformulating the
diffusion target. Extensive experiments on challenging
benchmarks across locomotion and manipulation demonstrate the superior performance, and the generalizability of DADP over prior methods. More visualization results are available on the https://outsider86.github.io/DomainAdaptiveDiffusionPolicy/.
Read the full abstract ▾
All authors Pengcheng Wang, Qinghang Liu, Haotian Lin, Yiheng Li, Guojian Zhan, Masayoshi Tomizuka, Yixiao Wang
Machine Learning
The Illusion of Generalization in Tabular Language Models
Aditya Gorla and Ratish Puduppully
Tabular
Language Models (TLMs) have been claimed to achieve strong generalization for tabular prediction. We conduct a systematic re-evaluation of Tabula-8B as a representative TLM, utilizing 165
datasets from the UniPredict
benchmark. Our investigation reveals three findings. First, binary and categorical classification achieve near-zero median lift over majority-class baselines and strong aggregate performance is driven entirely by quartile classification tasks. Second, top-performing
datasets exhibit pervasive contamination, including complete train-test overlap and task-level leakage that evades standard deduplication. Third, instruction-tuning without tabular exposure recovers 92.2% of standard classification performance and on quartile classification, format familiarity closes 71.3% of the gap with the residual attributable to contaminated
datasets. These findings suggest claimed generalization likely reflects evaluation artifacts rather than learned tabular
reasoning. We conclude with recommendations for strengthening TLM evaluation.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.DB
PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models
Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik et al. (+4 more)
Relational Foundation Models (RFMs) facilitate data-driven decision-making by learning from complex multi-table databases. However, the diverse relational databases needed to train such models are rarely public due to
privacy constraints. While there are methods to generate synthetic tabular data of arbitrary size, incorporating schema structure and primary-foreign key connectivity for multi-table generation remains challenging. Here we introduce PLUREL, a framework to synthesize multi-tabular relational databases from scratch. In a step-by-step fashion, PLUREL models (1) schemas with directed graphs, (2) inter-table primary-foreign key connectivity with bipartite graphs, and, (3) feature distributions in tables via conditional causal mechanisms. The design space across these stages supports the synthesis of a wide range of diverse databases, while being computationally lightweight. Using PLUREL, we observe for the first time that (1) RFM pretraining loss exhibits power-law scaling with the number of synthetic databases and total pretraining
tokens, (2) scaling the number of synthetic databases improves generalization to real databases, and (3) synthetic pretraining yields strong base models for continued pretraining on real databases. Overall, our framework and results position synthetic data scaling as a promising paradigm for RFMs.
Read the full abstract ▾
All authors Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik, Vijay Prakash Dwivedi, Johannes Hoffart, Carlos Guestrin, Jure Leskovec
Machine Learning · Cross-listed from cs.AI
Axiomatic Foundations of Counterfactual Explanations
Leila Amgoud and Martin Cooper
Explaining autonomous and intelligent systems is critical in order to improve trust in their decisions. Counterfactuals have emerged as one of the most compelling forms of explanation. They address “why not” questions by revealing how decisions could be altered. Despite the growing literature, most existing explainers focus on a single type of counterfactual and are restricted to local explanations, focusing on individual instances. There has been no systematic study of alternative counterfactual types, nor of global counterfactuals that shed light on a system's overall
reasoning process. This paper addresses the two gaps by introducing an axiomatic framework built on a set of desirable properties for counterfactual explainers. It proves impossibility
theorems showing that no single explainer can satisfy certain axiom combinations simultaneously, and fully characterizes all compatible sets. Representation
theorems then establish five one-to-one correspondences between specific subsets of axioms and the families of explainers that satisfy them. Each family gives rise to a distinct type of counterfactual explanation, uncovering five fundamentally different types of counterfactuals. Some of these correspond to local explanations, while others capture global explanations. Finally, the framework situates existing explainers within this taxonomy, formally characterizes their behavior, and analyzes the computational complexity of generating such explanations.
Read the full abstract ▾