Machine Learning
MarkovScale: Towards Optimal Sequential Scaling at Inference Time
Youkang Wang, Jian Wang, Rubing Chen et al. (+3 more)
Sequential scaling is a prominent inference-time scaling paradigm, yet its performance improvements are typically modest and not well understood, largely due to the prevalence of heuristic, non-principled approaches that obscure clear optimality bounds. To address this, we propose a principled framework that models sequential scaling as a two-state Markov process. This approach reveals the underlying properties of sequential scaling and yields closed-form solutions for essential aspects, such as the specific conditions under which accuracy is improved and the theoretical upper, neutral, and lower performance bounds. Leveraging this formulation, we develop MarkovScale, a practical system that applies these optimality criteria to achieve a theoretically grounded balance between accuracy and efficiency. Comprehensive experiments across 3 backbone LLMs, 5
benchmarks, and over 20 configurations show that MarkovScale consistently outperforms state-of-the-art parallel and sequential scaling methods, representing a significant step toward optimal and resource-efficient inference in LLMs. The source code will be open upon acceptance at https://open-upon-acceptance.
Read the full abstract ▾
All authors Youkang Wang, Jian Wang, Rubing Chen, Tianyi Zeng, Xiao-Yong Wei, Qing Li
Machine Learning
Single-Edge Node Injection Threats to GNN-Based Security Monitoring in Industrial Graph Systems
Wenjie Liang, Ranhui Yan, Jia Cai et al. (+1 more)
Graph neural networks (GNNs) are increasingly adopted in industrial graph-based monitoring systems (e.g., Industrial internet of things (IIoT) device graphs, power-grid
topology models, and manufacturing communication networks) to support anomaly detection, state estimation, and asset classification. In such settings, an adversary that compromises a small number of edge devices may inject counterfeit nodes (e.g., rogue sensors, virtualized endpoints, or spoofed substations) to bias downstream decisions while evading
topology- and homophily-based sanitization. This paper formulates deployment-oriented node-injection attacks under constrained resources and proposes the Single-Edge Graph Injection Attack (SEGIA), in which each injected node attaches to the operational graph through a single edge. SEGIA integrates a pruned SGC surrogate, multi-hop neighborhood sampling, and reverse graph convolution-based feature synthesis with a similarity-regularized objective to preserve local homophily and survive edge
pruning. Theoretical analysis and extensive evaluations across
datasets and defenses show at least $25\%$ higher attack success than representative baselines under substantially smaller edge budgets. These results indicate a system-level risk in industrial GNN deployments and motivate lightweight admission validation and neighborhood-consistency monitoring.
Read the full abstract ▾
All authors Wenjie Liang, Ranhui Yan, Jia Cai, You-Gan Wang
Machine Learning · Cross-listed from cs.SE
SPELL: Synthesis of Programmatic Edits using LLMs
Daniel Ramos, Catarina Gamboa, Inês Lynce et al. (+3 more)
Library migration is a common but error-prone task in software development. Developers may need to replace one library with another due to reasons like changing requirements or licensing changes. Migration typically entails updating and rewriting source code manually. While automated migration tools exist, most rely on mining examples from real-world projects that have already undergone similar migrations. However, these data are scarce, and collecting them for arbitrary pairs of libraries is difficult. Moreover, these migration tools often miss out on leveraging modern code transformation infrastructure. In this paper, we present a new approach to automated API migration that sidesteps the limitations described above. Instead of relying on existing migration data or using LLMs directly for transformation, we use LLMs to extract migration examples. Next, we use an
Agent to generalize those examples to reusable transformation scripts in PolyglotPiranha, a modern code transformation tool. Our method distills latent migration knowledge from LLMs into structured, testable, and repeatable migration logic, without requiring preexisting corpora or manual engineering effort. Experimental results across Python libraries show that our system can generate diverse migration examples and synthesize transformation scripts that generalize to real-world codebases.
Read the full abstract ▾
All authors Daniel Ramos, Catarina Gamboa, Inês Lynce, Vasco Manquinho, Ruben Martins, Claire Le Goues
Machine Learning
OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases
Zixiao Wang, Yifei Shen and Huishuai Zhang
Many optimizers can be interpreted as steepest-descent methods under norm-induced geometries, and thus inherit corresponding implicit biases. We introduce \nameA{} (\fullname{}), which combines spectral control from orthogonalized update directions with $\ell_\infty$-style coordinate control from sign updates. \nameA{} forms a Lion-style momentum direction, approximately orthogonalizes it via a few Newton–Schulz iterations, and then applies an entrywise sign, providing an efficient approximation to taking a maximal step over the intersection of the spectral and $\ell_\infty$ constraint sets (a scaled Hadamard-like set for
matrix parameters). Despite the strong nonlinearity of orthogonalization and sign, we prove convergence under a mild, empirically verified diagonal-isotropy assumption. Across large-scale
language and
vision training, including GPT-2 and Llama pretraining, SiT image pretraining, and supervised fine-tuning, \nameA{} matches or outperforms AdamW and
Muon under comparable tuning while using only momentum-level optimizer state, and it mitigates optimizer mismatch when fine-tuning AdamW-pretrained checkpoints.
Read the full abstract ▾
Machine Learning
On the Expressive Power of Permutation-Equivariant Weight-Space Networks
Adir Dayan, Yam Eitan and Haggai Maron
Weight-space learning studies neural architectures that operate directly on the parameters of other neural networks. Motivated by the growing availability of pretrained models, recent work has demonstrated the effectiveness of weight-space networks across a wide range of tasks. SOTA weight-space networks rely on permutation-equivariant designs to improve generalization. However, this may negatively affect expressive power, warranting theoretical investigation. Importantly, unlike other structured domains, weight-space learning targets maps operating on both weight and function spaces, making expressivity analysis particularly subtle. While a few prior works provide partial expressivity results, a comprehensive characterization is still missing. In this work, we address this gap by developing a systematic theory for expressivity of weight-space networks. We first prove that all prominent permutation-equivariant networks are equivalent in expressive power. We then establish universality in both weight- and function-space settings under mild, natural assumptions on the input weights, and characterize the edge-case regimes where universality no longer holds. Guided by our theoretical results, we show that slight modifications to existing weight-space models yield a 34% improvement over prior SOTA, demonstrating the practical relevance of our framework.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
PDE-Constrained Optimization for Neural Image Segmentation with Physics Priors
Seema K. Poudel and Sunny K. Khadka
Segmentation of microscopy images constitutes an ill-posed inverse problem due to measurement noise, weak object boundaries, and limited labeled data. Although deep neural networks provide flexible nonparametric
estimators, unconstrained empirical risk minimization often leads to unstable solutions and poor generalization. In this work, image
segmentation is formulated as a PDE-constrained optimization problem that integrates physically motivated priors into deep learning models through variational regularization. The proposed framework minimizes a composite objective function consisting of a data fidelity term and penalty terms derived from reaction-
diffusion equations and phase-field interface energies, all implemented as differentiable residual losses. Experiments are conducted on the LIVECell
dataset, a high-quality, manually annotated collection of phase-contrast microscopy images. Training is performed on two
cell types, while evaluation is carried out on a distinct, unseen
cell type to assess generalization. A UNet architecture is used as the unconstrained baseline model. Experimental results demonstrate consistent improvements in
segmentation accuracy and boundary fidelity compared to unconstrained deep learning baselines. Moreover, the PDE-regularized models exhibit enhanced stability and improved generalization in low-sample regimes, highlighting the advantages of incorporating structured priors. The proposed approach illustrates how PDE-constrained optimization can strengthen data-driven learning frameworks, providing a principled bridge between variational methods, statistical learning, and scientific machine learning.
Read the full abstract ▾
Machine Learning
Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning
Dylan Zhang, Yufeng Xu, Haojin Wang et al. (+2 more)
Post-training of
reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone. We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts. We propose PEAR (Policy Evaluation-inspired
Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the
token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected. We conduct controlled experiments on verifiable
reasoning games and mathematical
reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.
Read the full abstract ▾
All authors Dylan Zhang, Yufeng Xu, Haojin Wang, Qingzhi Chen, Hao Peng
Machine Learning · Cross-listed from quant-ph
The Quantum Learning Menagerie (A survey on Quantum learning for Classical concepts)
Sagnik Chatterjee
This paper surveys various results in the field of
Quantum Learning theory, specifically focusing on learning
quantum-encoded classical concepts in the Probably Approximately Correct (PAC) framework. The cornerstone of this work is the emphasis on query, sample, and time complexity separations between classical and
quantum learning that emerge under learning with query access to different labeling oracles. This paper aims to consolidate all known results in the area under the above umbrella and underscore the limits of our understanding by leaving the reader with 23 open problems.
Read the full abstract ▾
Machine Learning
LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents
Hyesung Jeon, Hyeongju Ha and Jae-Joon Kim
Role specialization in multi-LLM
agent systems is often realized via multi-LoRA, where
agents share a pretrained backbone and differ only by lightweight adapters. Despite sharing base model weights, each
agent independently builds and stores its own KV cache for the same long, tool-augmented trajectories, incurring substantial
memory and compute overhead. Existing KV cache sharing methods largely overlook this multi-LoRA setting. We observe that, cache differences across
agents are dominated by adapter outputs, while activations from the shared pretrained backbone remain highly similar. Based on this observation, we propose LRAgent, a KV cache sharing framework for multi-LoRA
agents. It decomposes the cache into two components, a shared base component derived from pretrained weights and an adapter-dependent component derived from LoRA weights. LRAgent reduces
memory overhead by sharing the base component across
agents and storing the adapter component in its inherent low-rank form. It also reduces computational overhead by sharing the low-rank cache, enabled by a shared-A multi-LoRA architecture. This avoids redundant computations for contexts that have already been processed by other
agents. To efficiently reconstruct adapter contributions at runtime, we introduce Flash-LoRA-
Attention, a kernel that reorders
attention computation to avoid materializing the low-rank cache to full dimension. LRAgent achieves throughput and time-to-first-
token latency close to fully shared caching, while preserving accuracy near the non-shared caching baseline across agentic question-answering
benchmarks.
Read the full abstract ▾
Machine Learning
SwiftRepertoire: Few-Shot Immune-Signature Synthesis via Dynamic Kernel Codes
Rong Fu, Yang Li, Yabin Jin et al. (+3 more)
Repertoire-level analysis of T
cell receptors offers a biologically grounded signal for disease detection and immune monitoring, yet practical deployment is impeded by label sparsity, cohort heterogeneity, and the computational burden of adapting large encoders to new tasks. We introduce a framework that synthesizes compact task-specific parameterizations from a learned dictionary of prototypes conditioned on lightweight task descriptors derived from repertoire probes and pooled
embedding statistics. This synthesis produces small adapter modules applied to a frozen pretrained backbone, enabling immediate adaptation to novel tasks with only a handful of support examples and without full model fine-tuning. The architecture preserves interpretability through motif-aware probes and a calibrated motif discovery pipeline that links predictive decisions to sequence-level signals. Together, these components yield a practical, sample-efficient, and interpretable pathway for translating repertoire-informed models into diverse clinical and research settings where labeled data are scarce and computational resources are constrained.
Read the full abstract ▾
All authors Rong Fu, Yang Li, Yabin Jin, Jiekai Wu, Chunlei Meng, Simon Fong
Machine Learning
Superposition unifies power-law training dynamics
Zixin Jessie Chen, Hao Chen, Yizhou Liu et al. (+1 more)
We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large
language models.
Read the full abstract ▾
All authors Zixin Jessie Chen, Hao Chen, Yizhou Liu, Jeff Gore
Machine Learning
Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection
Zhiwei Ling, Hailiang Zhao, Chao Zhang et al. (+8 more)
Federated Learning (FL) enables collaborative model training across large-scale distributed service nodes while preserving data
privacy, making it a cornerstone of intelligent service systems in edge-cloud environments. However, in real-world service-oriented deployments, data generated by heterogeneous users, devices, and application scenarios are inherently non-IID. This severe data heterogeneity critically undermines the convergence stability, generalization ability, and ultimately the quality of service delivered by the global model. To address this challenge, we propose FLood, a novel FL framework inspired by out-of-distribution (OOD) detection. FLood dynamically counteracts the adverse effects of heterogeneity through a dual-weighting mechanism that jointly governs local training and global aggregation. At the client level, it adaptively reweights the supervised loss by upweighting pseudo-OOD samples, thereby encouraging more robust learning from distributionally misaligned or challenging data. At the server level, it refines model aggregation by weighting client contributions according to their OOD confidence scores, prioritizing updates from clients with higher in-distribution consistency and enhancing the global model's robustness and convergence stability. Extensive experiments across multiple
benchmarks under diverse non-IID settings demonstrate that FLood consistently outperforms state-of-the-art FL methods in both accuracy and generalization. Furthermore, FLood functions as an orthogonal plug-in module: it seamlessly integrates with existing FL
algorithms to boost their performance under heterogeneity without modifying their core optimization logic. These properties make FLood a practical and scalable solution for deploying reliable intelligent services in real-world
federated environments.
Read the full abstract ▾
All authors Zhiwei Ling, Hailiang Zhao, Chao Zhang, Xiang Ao, Ziqi Wang, Cheng Zhang, Zhen Qin, Xinkui Zhao, Kingsum Chow, Yuanqing Wu, MengChu Zhou
Machine Learning
SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models
Xin Nie, Haicheng Zhang, Liang Dong et al. (+3 more)
Mixed-precision quantization is a promising approach for compressing large
language models under tight
memory budgets. However, existing mixed-precision methods typically suffer from one of two limitations: they either rely on expensive discrete optimization to determine precision allocation, or introduce hardware inefficiencies due to irregular
memory layouts. We propose SFMP, a search-free and hardware-friendly mixed-precision quantization framework for large
language models. The framework is built upon four novel ideas: Fractional bit-width, which extends integer bit-width for weight
matrix to fractional value and transforms discrete precision allocation as a continuous problem; 2)Block-wise mixed-precision, enabling fine-grained precision within weight
matrices while remaining hardware-friendly; 3)Row-column weight reordering, which aggregates salient weights via row and column reordering, incurring only a small activation reordering overhead during inference; 4)Unified GEMM kernel, which supports mixed-precision GEMM at arbitrary average bit-width. Extensive experiments demonstrate that SFMP outperforms state-of-the-art layer-wise mixed-precision methods under the same
memory constraints, while significantly reducing quantization cost and improving inference efficiency. Code is available at https://github.com/Nkniexin/SFMP
Read the full abstract ▾
All authors Xin Nie, Haicheng Zhang, Liang Dong, Beining Feng, Jinhong Weng, Guiling Sun
Machine Learning · Cross-listed from cs.NE
The Stacked Autoencoder Evolution Hypothesis
Hiroyuki Iizuka
This study introduces a novel theoretical framework, the Stacked Autoencoder
Evolution Hypothesis, which proposes that biological evolutionary systems operate through multi-layered self-encoding and decoding processes, analogous to stacked autoencoders in deep learning. Rather than viewing
evolution solely as gradual changes driven by mutation and selection, this hypothesis suggests that self-replication inherently compresses and reconstructs genetic information across hierarchical layers of abstraction. This layered structure enables evolutionary systems to explore diverse possibilities not only at the sequence level but also across progressively more abstract layers of representation, making it possible for even simple mutations to navigate these higher-order spaces.Such a mechanism may explain punctuated evolutionary patterns and changes that can appear as if they are goal-directed in natural
evolution, by allowing mutations at deeper latent layers to trigger sudden, large-scale phenotypic shifts. To illustrate the plausibility of this mechanism, artificial chemistry simulations were conducted, demonstrating the spontaneous emergence of hierarchical autoencoder structures. This framework offers a new perspective on the informational dynamics underlying both continuous and discontinuous evolutionary change.
Read the full abstract ▾
Machine Learning
Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models
Kaiyuan Cui, Yige Li, Yutao Wu et al. (+4 more)
Vision-
language models (VLMs) extend large
language models (LLMs) with
vision encoders, enabling text generation conditioned on both images and text. However, this
multimodal integration expands the attack surface by exposing the model to image-based jailbreaks crafted to induce harmful responses. Existing gradient-based jailbreak methods transfer poorly, as
adversarial patterns overfit to a single white-box surrogate and fail to generalise to black-box models. In this work, we propose Universal and transferable jailbreak (UltraBreak), a framework that constrains
adversarial patterns through transformations and regularisation in the
vision space, while relaxing textual targets through semantic-based objectives. By defining its loss in the textual
embedding space of the target LLM, UltraBreak discovers universal
adversarial patterns that generalise across diverse jailbreak objectives. This combination of
vision-level regularisation and semantically guided textual supervision mitigates surrogate overfitting and enables strong transferability across both models and attack targets. Extensive experiments show that UltraBreak consistently outperforms prior jailbreak methods. Further analysis reveals why earlier approaches fail to transfer, highlighting that smoothing the loss landscape via semantic objectives is crucial for enabling universal and transferable jailbreaks. The code is publicly available in our \href{https://github.com/kaiyuanCui/UltraBreak}{GitHub repository}.
Read the full abstract ▾
All authors Kaiyuan Cui, Yige Li, Yutao Wu, Xingjun Ma, Sarah Erfani, Christopher Leckie, Hanxun Huang
Machine Learning · Cross-listed from cs.IR
Unifying Ranking and Generation in Query Auto-Completion via Retrieval-Augmented Generation and Multi-Objective Alignment
Kai Yuan, Anthony Zheng, Jia Hu et al. (+9 more)
Query Auto-Completion (QAC) suggests query completions as users type, helping them articulate intent and reach results more efficiently. Existing approaches face fundamental challenges: traditional retrieve-and-rank pipelines have limited long-tail coverage and require extensive feature engineering, while recent generative methods suffer from
hallucination and safety risks. We present a unified framework that reformulates QAC as end-to-end list generation through
Retrieval-Augmented Generation (RAG) and multi-objective Direct Preference Optimization (DPO). Our approach combines three key innovations: (1) reformulating QAC as end-to-end list generation with multi-objective optimization; (2) defining and deploying a suite of rule-based, model-based, and LLM-as-judge verifiers for QAC, and using them in a comprehensive methodology that combines RAG, multi-objective DPO, and iterative critique-revision for high-quality synthetic data; (3) a hybrid serving architecture enabling efficient production deployment under strict latency constraints. Evaluation on a large-scale commercial search platform demonstrates substantial improvements: offline metrics show gains across all dimensions, human evaluation yields +0.40 to +0.69 preference scores, and a controlled online experiment achieves 5.44% reduction in keystrokes and 3.46% increase in suggestion adoption, validating that unified generation with RAG and multi-objective
alignment provides an effective solution for production QAC. This work represents a paradigm shift to end-to-end generation powered by large
language models, RAG, and multi-objective
alignment, establishing a production-validated framework that can benefit the broader search and recommendation industry.
Read the full abstract ▾
All authors Kai Yuan, Anthony Zheng, Jia Hu, Divyanshu Sheth, Hemanth Velaga, Kylee Kim, Matteo Guarrera, Besim Avci, Jianhua Li, Xuetao Yin, Rajyashree Mukherjee, Sean Suchter
Machine Learning
How Does Unfaithful Reasoning Emerge from Autoregressive Training? A Study of Synthetic Experiments
Fuxin Wang, Amr Alazali and Yiqiao Zhong
Chain-of-thought (CoT)
reasoning generated by large
language models (LLMs) is often unfaithful: intermediate steps can be logically inconsistent or fail to reflect the causal relationship leading to the final answer. Despite extensive empirical observations, a fundamental understanding of CoT is lacking–what constitutes faithful CoT
reasoning, and how unfaithfulness emerges from autoregressive training. We study these questions using well-controlled synthetic experiments, training small
transformers on noisy data to solve modular arithmetic expressions step by step, a task we term Arithmetic Expression
Reasoning. We find that models can learn faithful
reasoning that causally follows the underlying arithmetic rules, but only when the training noise is below a critical threshold, a phenomenon attributable to simplicity bias. At higher noise levels, training dynamics exhibit a transition from faithful stepwise
reasoning to unfaithful skip-step
reasoning via an intermediate mixed mode characterized by a transient increase in prediction
entropy. Mechanistic analysis reveals that models learn to encode internal uncertainty by resolving inconsistent
reasoning steps, which suggests the emergence of implicit self-verification from autoregressive training.
Read the full abstract ▾
Machine Learning
LASS-ODE: Scaling ODE Computations to Connect Foundation Models with Dynamical Physical Systems
Haoran Li, Chenhan Xiao, Lihao Mai et al. (+2 more)
Foundation models have transformed
language,
vision, and time series data analysis, yet progress on dynamic predictions for physical systems remains limited. Given the complexity of physical constraints, two challenges stand out. $(i)$ Physics-computation scalability: physics-informed learning can enforce physical regularization, but its computation (e.g., ODE integration) does not scale to extensive systems. $(ii)$ Knowledge-sharing efficiency: the
attention mechanism is primarily computed within each system, which limits the extraction of shared ODE structures across systems. We show that enforcing ODE consistency does not require expensive nonlinear integration: a
token-wise locally linear ODE representation preserves physical fidelity while scaling to foundation-model regimes. Thus, we propose novel
token representations that respect locally linear ODE
evolution. Such linearity substantially accelerates integration while accurately approximating the local data
manifold. Second, we introduce a simple yet effective inter-system
attention that augments
attention with a common structure hub (CSH) that stores shared
tokens and aggregates knowledge across systems. The resulting model, termed LASS-ODE (\underline{LA}rge-\underline{S}cale \underline{S}mall \underline{ODE}), is pretrained on our $40$GB ODE trajectory collections to enable strong in-domain performance, zero-shot generalization across diverse ODE systems, and additional improvements through fine-tuning.
Read the full abstract ▾
All authors Haoran Li, Chenhan Xiao, Lihao Mai, Yang Weng, Erik Blasch
Machine Learning
Predicting Anemia Among Under-Five Children in Nepal Using Machine Learning and Deep Learning
Deepak Bastola, Pitambar Acharya, Dipak Dulal et al. (+2 more)
Childhood anemia remains a major public health challenge in Nepal and is associated with impaired growth, cognition, and increased morbidity. Using World Health Organization hemoglobin thresholds, we defined anemia status for children aged 6-59 months and formulated a binary classification task by grouping all anemia severities as anemic versus not anemic. We analyzed Nepal Demographic and Health Survey (NDHS 2022) microdata comprising 1,855 children and initially considered 48 candidate features spanning demographic, socioeconomic, maternal, and child health characteristics. To obtain a stable and substantiated feature set, we applied four features selection techniques (Chi-square, mutual information, point-biserial correlation, and Boruta) and prioritized features supported by multi-method consensus. Five features: child age, recent fever, household size, maternal anemia, and parasite deworming were consistently selected by all methods, while amenorrhea, ethnicity indicators, and provinces were frequently retained. We then compared eight traditional machine learning classifiers (LR, KNN, DT, RF, XGBoost, SVM, NB, LDA) with two deep learning models (DNN and TabNet) using standard evaluation metrics, emphasizing F1-score and recall due to class imbalance. Among all models, logistic regression attained the best recall (0.701) and the highest F1-score (0.649), while DNN achieved the highest accuracy (0.709), and SVM yielded the strongest discrimination with the highest AUC (0.736). Overall, the results indicate that both machine learning and deep learning models can provide competitive anemia prediction and the interpretable features such as child age, infection proxy, maternal anemia, and deworming history are central for risk stratification and public health screening in Nepal.
Read the full abstract ▾
All authors Deepak Bastola, Pitambar Acharya, Dipak Dulal, Rabina Dhakal, Yang Li
Machine Learning
ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning
Zhishen Sun, Sizhe Dang, Guang Dai et al. (+1 more)
Reinforcement learning (RL) has become a key training step for improving mathematical
reasoning in large
language models (LLMs), but it often has high GPU
memory usage, which makes it hard to use in settings with limited resources. To reduce these issues, we propose
Evolution Strategies with Sharpness-Aware Maximization (ESSAM), a full parameter fine-tuning framework that tightly combines the zero-order search in parameter space from
Evolution Strategies (ES) with the Sharpness-Aware Maximization (SAM) to improve generalization. We conduct fine-tuning experiments on the mainstream mathematica
reasoning task GSM8K. The results show that ESSAM achieves an average accuracy of 78.27% across all models and its overall performance is comparable to RL methods. It surpasses classic RL
algorithm PPO with an accuracy of 77.72% and is comparable to GRPO with an accuracy of 78.34%, and even surpassing them on some models. Further generalization experiments show that the models trained with ESSAM exhibit stronger generalization ability. Their average performance achieves the best results on 5 out of 6
datasets, indicating that ESSAM can effectively improve the generalization performance of fine-tuned models. In terms of GPU
memory usage, ESSAM reduces the average GPU
memory usage by $18\times$ compared to PPO and by $10\times$ compared to GRPO, achieving an extremely low GPU
memory usage. In addition, we design an accelerated variant of ESSAM, which achieves nearly a twofold speedup while maintaining the same GPU
memory usage as ESSAM, and attains an average accuracy of 78.02% across all models, outperforming PPO. Code: https://github.com/szs777/ESSAM
Read the full abstract ▾
All authors Zhishen Sun, Sizhe Dang, Guang Dai, Haishan Ye
Machine Learning · Cross-listed from cs.CV
CortiNet: A Physics-Perception Hybrid Cortical-Inspired Dual-Stream Network for Gallbladder Disease Diagnosis from Ultrasound
Vagish Kumar and Souvik Chakraborty
Ultrasound imaging is the primary diagnostic modality for detecting Gallbladder diseases due to its non-invasive nature, affordability, and wide accessibility. However, the low resolution and speckle noise inherent to ultrasound images hinder diagnostic reliability, prompting the use of large convolutional neural networks that are difficult to deploy in routine clinical settings. In this work, we propose CortiNet, a lightweight, cortical-inspired dual-stream neural architecture for gallbladder disease diagnosis that integrates physically interpretable multi-scale signal decomposition with perception-driven feature learning. Inspired by parallel processing pathways in the human visual cortex, CortiNet explicitly separates low-frequency structural information from high-frequency perceptual details and processes them through specialized encoding streams. By operating directly on structured, frequency-selective representations rather than raw pixel intensities, the architecture embeds strong physics-based inductive bias, enabling efficient feature learning with a significantly reduced parameter footprint. A late-stage cortical-style fusion mechanism integrates complementary structural and textural cues while preserving computational efficiency. Additionally, we propose a structure-aware explainability framework wherein gradient-weighted class activation mapping is only applied to the structural branch of the proposed CortiNet architecture. This choice allows the model to only focus on the structural features, making it robust against speckle noise. We evaluate CortiNet on 10,692 expert-annotated images spanning nine clinically relevant gallbladder disease categories. Experimental results demonstrate that CortiNet achieves high diagnostic accuracy (98.74%) with only a fraction of the parameters required by conventional deep convolutional models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Error Taxonomy-Guided Prompt Optimization
Mayank Singh, Vikas Yadav and Eduardo Blanco
Automatic
Prompt Optimization (APO) is a powerful approach for extracting performance from large
language models without modifying their weights. Many existing methods rely on trial-and-error, testing different
prompts or in-context examples until a good configuration emerges, often consuming substantial compute. Recently, natural
language feedback derived from execution logs has shown promise as a way to identify how
prompts can be improved. However, most prior approaches operate in a bottom-up manner, iteratively adjusting the
prompt based on feedback from individual problems, which can cause them to lose the global perspective. In this work, we propose Error Taxonomy-Guided
Prompt Optimization (ETGPO), a
prompt optimization
algorithm that adopts a top-down approach. ETGPO focuses on the global failure landscape by collecting model errors, categorizing them into a taxonomy, and augmenting the
prompt with guidance targeting the most frequent failure modes. Across multiple
benchmarks spanning mathematics, question answering, and logical
reasoning, ETGPO achieves accuracy that is comparable to or better than state-of-the-art methods, while requiring roughly one third of the optimization-phase
token usage and evaluation budget.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Optimal Decision-Making Based on Prediction Sets
Tao Wang and Edgar Dobriban
Prediction sets can wrap around any ML model to cover unknown test outcomes with a guaranteed probability. Yet, it remains unclear how to use them optimally for downstream decision-making. Here, we propose a decision-theoretic framework that seeks to minimize the expected loss (risk) against a worst-case distribution consistent with the prediction set's coverage guarantee. We first characterize the minimax optimal policy for a fixed prediction set, showing that it balances the worst-case loss inside the set with a penalty for potential losses outside the set. Building on this, we derive the optimal prediction set construction that minimizes the resulting robust risk subject to a coverage constraint. Finally, we introduce Risk-Optimal Conformal Prediction (ROCP), a practical
algorithm that targets these risk-minimizing sets while maintaining finite-sample distribution-free marginal coverage. Empirical evaluations on medical diagnosis and safety-critical decision-making tasks demonstrate that ROCP reduces critical mistakes compared to baselines, particularly when out-of-set errors are costly.
Read the full abstract ▾
Machine Learning
Scalable Random Wavelet Features: Efficient Non-Stationary Kernel Approximation with Convergence Guarantees
Sawan Kumar and Souvik Chakraborty
Modeling non-stationary processes, where statistical properties vary across the input domain, is a critical challenge in machine learning; yet most scalable methods rely on a simplifying assumption of stationarity. This forces a difficult trade-off: use expressive but computationally demanding models like Deep Gaussian Processes, or scalable but limited methods like Random
Fourier Features (RFF). We close this gap by introducing Random Wavelet Features (RWF), a framework that constructs scalable, non-stationary kernel approximations by sampling from wavelet families. By harnessing the inherent localization and multi-resolution structure of wavelets, RWF generates an explicit feature map that captures complex, input-dependent patterns. Our framework provides a principled way to generalize RFF to the non-stationary setting and comes with a comprehensive theoretical analysis, including positive definiteness, unbiasedness, and uniform convergence guarantees. We demonstrate empirically on a range of challenging synthetic and real-world
datasets that RWF outperforms stationary random features and offers a compelling accuracy-efficiency trade-off against more complex models, unlocking scalable and expressive kernel methods for a broad class of real-world non-stationary problems.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning
Batuhan K. Karaman, Aditya Rawal, Suhaila Shakiah et al. (+4 more)
Reinforcement learning with verifiable rewards has emerged as a promising paradigm for enhancing the
reasoning capabilities of large
language models particularly in mathematics. Current approaches in this domain present a clear trade-off: PPO-style methods (e.g., GRPO/DAPO) offer training stability but exhibit slow learning trajectories due to their trust-region constraints on policy updates, while REINFORCE-style approaches (e.g., CISPO) demonstrate improved learning efficiency but suffer from performance instability as they clip importance sampling weights while still permitting non-zero gradients outside the trust-region. To address these limitations, we introduce DISPO, a simple yet effective REINFORCE-style
algorithm that decouples the up-clipping and down-clipping of importance sampling weights for correct and incorrect responses, yielding four controllable policy update regimes. Through targeted ablations, we uncover how each regime impacts training: for correct responses, weights >1 increase the average
token entropy (i.e., exploration) while weights <1 decrease it (i.e.,
distillation) – both beneficial but causing gradual performance degradation when excessive. For incorrect responses, overly restrictive clipping triggers sudden performance collapse through repetitive outputs (when weights >1) or vanishing response lengths (when weights <1). By separately tuning these four clipping parameters, DISPO maintains the exploration-
distillation balance while preventing catastrophic failures, achieving 61.04% on AIME'24 (vs. 55.42% CISPO and 50.21% DAPO) with similar gains across various
benchmarks and models.
Read the full abstract ▾
All authors Batuhan K. Karaman, Aditya Rawal, Suhaila Shakiah, Mohammad Ghavamzadeh, Mingyi Hong, Arijit Biswas, Ruida Zhou
Machine Learning · Cross-listed from cs.CL
Trust in One Round: Confidence Estimation for Large Language Models via Structural Signals
Pengyue Yang, Jiawen Wen, Haolin Jin et al. (+3 more)
Large
language models (LLMs) are increasingly deployed in domains where errors carry high social, scientific, or safety costs. Yet standard confidence
estimators, such as
token likelihood, semantic similarity and multi-sample consistency, remain brittle under distribution shift, domain-specialised text, and compute limits. In this work, we present Structural Confidence, a single-pass, model-agnostic framework that enhances output correctness prediction based on multi-scale structural signals derived from a model's final-layer hidden-state trajectory. By combining spectral, local-variation, and global shape descriptors, our method captures internal stability patterns that are missed by probabilities and sentence
embeddings. We conduct extensive, cross-domain evaluation across four heterogeneous
benchmarks-FEVER (fact verification), SciFact (scientific claims), WikiBio-
hallucination (biographical consistency), and TruthfulQA (truthfulness-oriented QA). Our Structural Confidence framework demonstrates strong performance compared with established baselines in terms of AUROC and AUPR. More importantly, unlike sampling-based consistency methods which require multiple
stochastic generations and an auxiliary model, our approach uses a single deterministic forward pass, offering a practical basis for efficient, robust post-hoc confidence estimation in socially impactful, resource-constrained LLM applications.
Read the full abstract ▾
All authors Pengyue Yang, Jiawen Wen, Haolin Jin, Linghan Huang, Huaming Chen, Ling Chen
Machine Learning
Forest-Guided Semantic Transport for Label-Supervised Manifold Alignment
Adrien Aumon, Myriam Lizotte, Guy Wolf et al. (+2 more)
Label-supervised
manifold alignment bridges the gap between unsupervised and correspondence-based paradigms by leveraging shared label information to align
multimodal datasets. However, existing methods either rely on label-independent intra-domain geometry or incorporate supervision primarily through the
alignment objective, rather than directly constructing task-aware intra-domain affinities. To address this limitation, we introduce FoSTA (Forest-guided Semantic Transport
Alignment), which learns supervised forest geometries independently within each domain and uses their shared class-semantic structure to infer cross-domain correspondences without predefined anchors. FoSTA extends RF-GAP affinities to partially labeled data and aligns the resulting semantic representations through scalable hierarchical transport. Extensive comparisons with established baselines demonstrate strong correspondence recovery and semantic structure preservation on synthetic and real-world
benchmarks, including
multimodal data integration and single-
cell batch correction.
Read the full abstract ▾
All authors Adrien Aumon, Myriam Lizotte, Guy Wolf, Kevin R. Moon, Jake S. Rhodes
Machine Learning
On the Spectral Flattening of Quantized Embeddings
Junlin Huang, Wenyi Fang, Zhenheng Tang et al. (+5 more)
Training Large
Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic heavy-tailed spectral nature of linguistic data. By formalizing the connection between Zipfian statistics and random
matrix theory, we prove that the power-law decay in the singular value spectra of
embeddings is a fundamental requisite for semantic encoding. We derive theoretical bounds showing that uniform quantization introduces a noise floor that disproportionately truncates this spectral tail, which induces spectral flattening and a strictly provable increase in the stable rank of representations. Empirical validation across diverse architectures including GPT-2 and TinyLlama corroborates that this
geometric degradation precipitates representational collapse. This work not only quantifies the spectral sensitivity of LLMs but also establishes spectral fidelity as a necessary condition for stable low-bit optimization.
Read the full abstract ▾
All authors Junlin Huang, Wenyi Fang, Zhenheng Tang, Yuxin Wang, Xueze Kang, Yang Zheng, Bo Li, Xiaowen Chu
Machine Learning
Leonardo Ferreira Guilhoto, Akshat Kaushal and Paris Perdikaris
Scientific machine learning (SciML) increasingly requires models that capture
multimodal conditional uncertainty arising from ill-posed inverse problems, multistability, and chaotic dynamics. While recent work has favored highly expressive implicit generative models such as
diffusion and flow-based methods, these approaches are often data-hungry, computationally costly, and misaligned with the structured solution spaces frequently found in scientific problems. We demonstrate that Mixture Density Networks (MDNs) provide a principled yet largely overlooked alternative for
multimodal uncertainty quantification in SciML. As explicit parametric density
estimators, MDNs impose an inductive bias tailored to low-dimensional,
multimodal physics, enabling direct global allocation of probability mass across distinct solution branches. This structure delivers strong data efficiency, allowing reliable recovery of separated modes in regimes where scientific data is scarce. We formalize these insights through a unified probabilistic framework contrasting explicit and implicit distribution networks, and demonstrate empirically that MDNs achieve superior generalization, interpretability, and sample efficiency across a range of inverse, multistable, and chaotic scientific regression tasks.
Read the full abstract ▾
Machine Learning
Probing the Knowledge Boundary: An Interactive Agentic Framework for Deep Knowledge Extraction
Yuheng Yang, Siqi Zhu, Tao Feng et al. (+2 more)
Large
Language Models (LLMs) can be seen as compressed knowledge bases, but it remains unclear what knowledge they truly contain and how far their knowledge boundary extends. Existing
benchmarks are mostly static and provide limited support for systematic knowledge probing. In this paper, we propose an interactive agentic framework to systematically extract and quantify the knowledge of LLMs. Our method includes four adaptive exploration policies to probe knowledge at different granularity. To ensure the quality of extracted knowledge, we introduce a three-stage knowledge processing pipeline that combines vector-based filtering to remove strict duplicates, LLM-based adjudication to resolve ambiguous semantic overlap, and domain relevance
auditing to retain valid knowledge units. Through extensive experiments, we find that Recursive Taxonomy is the most effective exploration strategy. We also observe a clear knowledge scaling law, where larger models consistently recover more knowledge. In addition, we identify a Pass@1 versus Pass@k trade-off: domain-specialized models achieve higher initial accuracy but experience rapid degradation, while general-purpose models maintain stable performance over extended extraction. Finally, our results show that differences in training data composition lead to distinct and measurable knowledge profiles across model families, reflecting how pretraining shapes each model's parametric knowledge.
Read the full abstract ▾
All authors Yuheng Yang, Siqi Zhu, Tao Feng, Ge Liu, Jiaxuan You