Machine Learning
Pool-based Active Learning as Noisy Lossy Compression: Characterizing Label Complexity via Finite Blocklength Analysis
Kosuke Sugiyama and Masato Uchida
This paper proposes an information-theoretic framework for analyzing the theoretical limits of pool-based active learning (AL), in which a subset of instances is selectively labeled. The proposed framework reformulates pool-based AL as a noisy lossy compression problem by mapping pool observations to noisy symbol observations, data selection to compression, and learning to decoding. This correspondence enables a unified information-theoretic analysis of data selection and learning in pool-based AL. Applying finite blocklength analysis of noisy lossy compression, we derive information-theoretic lower bounds on label complexity and generalization error that serve as theoretical limits for a given learning
algorithm under its associated optimal data selection strategy. Specifically, our bounds include terms that reflect overfitting induced by the learning
algorithm and the discrepancy between its inductive bias and the target task, and are closely related to established information-theoretic bounds and stability theory, which have not been previously applied to the analysis of pool-based AL. These properties yield a new theoretical perspective on pool-based AL.
Read the full abstract ▾
Machine Learning
GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RL
Zifan Liu, Xinran Li, Shibo Chen et al. (+1 more)
Offline Safe Reinforcement Learning (OSRL) aims to learn a policy to achieve high performance in sequential decision-making while satisfying constraints, using only pre-collected
datasets. Recent works, inspired by the strong capabilities of Generative Models (GMs), reformulate decision-making in OSRL as a conditional generative process, where GMs generate desirable actions conditioned on predefined reward and cost values. However, GM-assisted methods face two major challenges in OSRL: (1) lacking the ability to "stitch" optimal transitions from suboptimal trajectories within the
dataset, and (2) struggling to balance reward targets with cost targets, particularly when they are conflict. To address these issues, we propose Goal-Assisted Stitching (GAS), a novel
algorithm designed to enhance stitching capabilities while effectively balancing reward maximization and constraint satisfaction. To enhance the stitching ability, GAS first augments and relabels the
dataset at the transition level, enabling the construction of high-quality trajectories from suboptimal ones. GAS also introduces novel goal functions, which estimate the optimal achievable reward and cost goals from the
dataset. These goal functions, trained using expectile regression on the relabeled and augmented
dataset, allow GAS to accommodate a broader range of reward-cost return pairs and achieve a better tradeoff between reward maximization and constraint satisfaction compared to human-specified values. The estimated goals then guide policy training, ensuring robust performance under constrained settings. Furthermore, to improve training stability and efficiency, we reshape the
dataset to achieve a more uniform reward-cost return distribution. Empirical results validate the effectiveness of GAS, demonstrating superior performance in balancing reward maximization and constraint satisfaction compared to existing methods.
Read the full abstract ▾
All authors Zifan Liu, Xinran Li, Shibo Chen, Jun Zhang
Machine Learning
Accelerated Sequential Flow Matching: A Bayesian Filtering Perspective
Yinan Huang, Hans Hao-Hsun Hsu, Junran Wang et al. (+2 more)
Sequential probabilistic inference from streaming observations requires modeling distributions over future trajectories as new observations arrive. Although
diffusion and flow-matching models are effective at capturing high-dimensional,
multimodal distributions, their deployment in real-time streaming settings typically relies on repeatedly sampling from a non-informative initial distribution. This results in substantial inference latency, particularly when multiple samples are needed to characterize the predictive distribution. In this work, we introduce Sequential
Bayesian Flow Matching, a framework inspired by
Bayesian filtering. By learning a probability flow that transports the posterior distribution from one time step to the next time step conditioned on new observations, it mirrors the recursive structure of
Bayesian belief updates. Crucially, by using the previous belief as an informative source distribution, it enables substantially faster sampling than naive resampling from scratch. Across scientific
forecasting tasks spanning accelerator beam spill dynamics, fluid dynamics, and weather
forecasting, as well as decision-making
benchmarks, our method achieves performance competitive with full-step
diffusion on distributional metrics while using far fewer sampling steps, substantially reducing inference latency. Our code is available at https://github.com/Graph-COM/Sequential_Flow_Matching.
Read the full abstract ▾
All authors Yinan Huang, Hans Hao-Hsun Hsu, Junran Wang, Bo Dai, Pan Li
Machine Learning
Formal Synthesis of Certifiably Robust Neural Lyapunov-Barrier Certificates
Chengxiao Wang, Haoze Wu and Gagandeep Singh
Neural Lyapunov and barrier certificates have recently been used as powerful tools for verifying the safety and stability properties of deep reinforcement learning (RL) controllers. However, existing methods offer guarantees only under fixed ideal unperturbed dynamics, limiting their reliability in real-world applications where dynamics may deviate due to uncertainties. In this work, we study the problem of synthesizing robust neural Lyapunov barrier certificates that maintain their guarantees under perturbations in system dynamics. We formally define a robust Lyapunov barrier function and specify sufficient conditions based on Lipschitz continuity that ensure robustness against bounded perturbations. We propose practical training objectives that enforce these conditions via
adversarial training, Lipschitz neighborhood bound, and global Lipschitz regularization. We validate our approach in two practically relevant environments, Inverted Pendulum and 2D Docking. The former is a widely studied
benchmark, while the latter is a safety-critical task in autonomous systems. We show that our methods significantly improve both certified robustness bounds (up to $4.6$ times) and empirical success rates under strong perturbations (up to $2.4$ times) compared to the baseline. Our results demonstrate effectiveness of training robust neural certificates for safe RL under perturbations in dynamics.
Read the full abstract ▾
Machine Learning
A Short and Unified Convergence Analysis of the SAG, SAGA, and IAG Algorithms
Feng Zhu, Robert W. Heath Jr. and Aritra Mitra
Stochastic variance-reduced
algorithms such as
Stochastic Average Gradient (SAG) and SAGA, and their deterministic counterparts like the Incremental Aggregated Gradient (IAG) method, have been extensively studied in large-scale machine learning. Despite their popularity, existing analyses for these
algorithms are disparate, relying on different proof techniques tailored to each method. Furthermore, the original proof of SAG is known to be notoriously involved, requiring computer-aided analysis. Focusing on finite-sum optimization with smooth and strongly convex objective functions, our main contribution is to develop a single unified convergence analysis that applies to all three
algorithms: SAG, SAGA, and IAG. Our analysis features two key steps: (i) establishing a bound on delays due to
stochastic sub-sampling using simple concentration tools, and (ii) carefully designing a novel Lyapunov function that accounts for such delays. The resulting proof is short and modular, providing the first high-probability bounds for SAG and SAGA that can be seamlessly extended to non-convex objectives and Markov sampling. As an immediate byproduct of our new analysis technique, we obtain the best known rates for the IAG
algorithm, significantly improving upon prior bounds.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Logarithmic-time Schedules for Scaling Language Models with Momentum
Damien Ferbach, Courtney Paquette, Gauthier Gidel et al. (+2 more)
In practice, the hyperparameters $(\beta_1, \beta_2)$ and weight-decay $\lambda$ in AdamW are typically kept at fixed values. Is there any reason to do otherwise? We show that for large-scale
language model training, the answer is yes: by exploiting the power-law structure of
language data, one can design time-varying schedules for $(\beta_1, \beta_2, \lambda)$ that deliver substantial performance gains. We study logarithmic-time scheduling, in which the optimizer's gradient
memory horizon grows with training time. Although naive variants of this are unstable, we show that suitable damping mechanisms restore stability while preserving the benefits of longer
memory. Based on this, we present ADANA, an AdamW-like optimizer that couples log-time schedules with explicit damping to balance stability and performance. We empirically evaluate ADANA across
transformer scalings (45M to 2.6B parameters), comparing against AdamW,
Muon, and AdEMAMix. When properly tuned, ADANA achieves up to 40% compute efficiency relative to a tuned AdamW, with gains that persist–and even improve–as model scale increases. We further show that similar benefits arise when applying logarithmic-time scheduling to AdEMAMix, and that logarithmic-time weight-decay alone can yield significant improvements. Finally, we present variants of ADANA that mitigate potential failure modes and improve robustness.
Read the full abstract ▾
All authors Damien Ferbach, Courtney Paquette, Gauthier Gidel, Katie Everett, Elliot Paquette
Machine Learning
HealthMamba: An Uncertainty-aware Spatiotemporal Graph State Space Model for Effective and Reliable Healthcare Facility Visit Prediction
Dahai Yu, Lin Jiang, Rongchao Xu et al. (+1 more)
Healthcare facility visit prediction is essential for optimizing healthcare resource allocation and informing public health policy. Despite advanced machine learning methods being employed for better prediction performance, existing works usually formulate this task as a time-series
forecasting problem without considering the intrinsic spatial dependencies of different types of healthcare facilities, and they also fail to provide reliable predictions under abnormal situations such as public emergencies. To advance existing research, we propose HealthMamba, an uncertainty-aware spatiotemporal framework for accurate and reliable healthcare facility visit prediction. HealthMamba comprises three key components: (i) a Unified Spatiotemporal Context Encoder that fuses heterogeneous static and dynamic information, (ii) a novel Graph State Space Model called GraphMamba for hierarchical spatiotemporal modeling, and (iii) a comprehensive uncertainty quantification module integrating three uncertainty quantification mechanisms for reliable prediction. We evaluate HealthMamba on four large-scale real-world
datasets from California, New York, Texas, and Florida. Results show HealthMamba achieves around 6.0% improvement in prediction accuracy and 3.5% improvement in uncertainty quantification over state-of-the-art baselines.
Read the full abstract ▾
All authors Dahai Yu, Lin Jiang, Rongchao Xu, Guang Wang
Machine Learning
Robust Inference-Time Steering of Protein Diffusion Models via Embedding Optimization
Minhuan Li, Jiequn Han, Pilar Cossio et al. (+1 more)
A core challenge in structural biophysics is generating biomolecular conformations that are both physically plausible and consistent with experimental measurements. While sequence-to-structure
diffusion models provide powerful priors, posterior sampling methods steer generation by perturbing atomic coordinates with gradients from experimental likelihoods. However, when the target lies in a low-density region of the prior, these methods require aggressive upweighting of the likelihood that can destabilize sampling and be sensitive to hyperparameters. We propose EmbedOpt, an inference-time steering framework that introduces an orthogonal optimization axis: rather than performing posterior sampling under a fixed prior, EmbedOpt directly optimizes the prior by updating the model's conditional
embedding. This
embedding space encodes rich coevolutionary signals, so optimizing it shifts the structural prior to align with experimental constraints. Empirically, EmbedOpt matches coordinate-based posterior sampling baselines on sparse distance constraints and outperforms them on cryo-electron microscopy map fitting, including real, noisy experimental ones. Furthermore, EmbedOpt's smooth optimization behavior yields robustness to hyperparameters spanning two orders of magnitude and enables comparable performance with fewer
diffusion steps. Code is available at https://github.com/rs-station/embedopt.
Read the full abstract ▾
All authors Minhuan Li, Jiequn Han, Pilar Cossio, Luhuan Wu
Machine Learning
Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
Pengyi Li, Elizaveta Goncharova, Andrey Kuznetsov et al. (+1 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an indispensable paradigm for enhancing
reasoning in Large
Language Models (LLMs). However, standard policy optimization methods, such as Group Relative Policy Optimization (GRPO), often converge to low-
entropy policies, leading to severe mode collapse and limited output diversity. We analyze this issue from the perspective of sampling probability dynamics, identifying that the standard objective disproportionately reinforces the highest-likelihood paths, thereby suppressing valid alternative
reasoning chains. To address this, we propose a novel Advantage Re-weighting Mechanism (ARM) designed to equilibrate the confidence levels across all correct responses. By incorporating
Prompt Perplexity and Answer Confidence into the advantage estimation, our method dynamically reshapes the reward signal to attenuate the gradient updates of over-confident
reasoning paths, while redistributing probability mass toward under-explored correct solutions. Empirical results demonstrate that our approach significantly enhances generative diversity and response
entropy while maintaining competitive accuracy, effectively achieving a superior trade-off between exploration and exploitation in
reasoning tasks. Empirical results on Qwen2.5 and DeepSeek models across mathematical and coding
benchmarks show that ProGRPO significantly mitigates
entropy collapse. Specifically, on Qwen2.5-7B, our method outperforms GRPO by 5.7% in Pass@1 and, notably, by 13.9% in Pass@32, highlighting its superior capability in generating diverse correct
reasoning paths.
Read the full abstract ▾
All authors Pengyi Li, Elizaveta Goncharova, Andrey Kuznetsov, Ivan Oseledets
Machine Learning
Hybrid Gated Flow (HGF): Stabilizing 1.58-bit LLMs via Selective Low-Rank Correction
David Alejandro Trejo Pizzo
The deployment of Large
Language Models (LLMs) on edge devices is fundamentally constrained by the "
Memory Wall" – a hardware limitation where
memory bandwidth, not compute, becomes the bottleneck. Recent 1.58-bit quantization techniques (e.g., BitNet b1.58) dramatically reduce
memory footprint but typically incur a perplexity degradation of 20-25% compared to FP16 baselines. In this work, we introduce Hybrid Gated Flow (HGF), a dual-stream architecture that couples a 1.58-bit ternary backbone with a learnable, low-rank FP16 correction path controlled by adaptive gates. Through extensive experiments on the TinyStories
dataset across two training regimes (2500 and 3500 steps), we demonstrate that HGF 5.4 achieves a validation loss of 0.9306 compared to BitNet's 1.0294, recovering approximately 55% of the quality gap between pure ternary quantization and the FP16 baseline (0.8490). This recovery is achieved with only ~12-15%
memory overhead beyond the ternary backbone. Furthermore, we provide empirical evidence for an emergent phenomenon: quantization as structural regularization. While a full-precision differential
attention baseline (Diff_Only) exhibited training instability with validation loss exceeding 1.68, the ternary-anchored HGF maintained robust convergence throughout training. Finally, we report preliminary results extending this architecture to 1.2B and 3B parameter models trained on SlimPajama and FineWeb-Edu. These larger-scale experiments confirm that the architectural stability and quality recovery observed in small-scale proxies scale linearly to production-grade
language modeling regimes.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
Haoran Li, Sucheng Ren, Alan Yuille et al. (+1 more)
Rotary Positional
Embedding (RoPE) is a key component of context scaling in Large
Language Models (LLMs). While various methods have been proposed to adapt RoPE to longer contexts, their guiding principles generally fall into two categories: (1) out-of-distribution (OOD) mitigation, which scales RoPE frequencies to accommodate unseen positions, and (2) Semantic Modeling, which posits that the
attention scores computed with RoPE should always prioritize semantically similar
tokens. In this work, we unify these seemingly distinct objectives through a minimalist intervention, namely CoPE: soft clipping lowfrequency components of RoPE. CoPE not only eliminates OOD outliers and refines semantic signals, but also prevents spectral leakage caused by hard clipping. Extensive experiments demonstrate that simply applying our soft clipping strategy to RoPE yields significant performance gains that scale up to 256k context length, validating our theoretical analysis and establishing CoPE as a new state-of-the-art for length generalization. Our code, data, and models are available at https://github.com/hrlics/CoPE.
Read the full abstract ▾
All authors Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang
Machine Learning
TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Training
Guanjie Cheng, Boyi Li, Lingyu Sun et al. (+4 more)
Large-scale
multimodal pre-trained models like CLIP rely heavily on high-quality training data, yet raw web-crawled
datasets are often noisy, misaligned, and redundant, leading to inefficient training and suboptimal generalization. Existing data selection methods are either heuristic-based, suffering from bias and limited diversity, or data-driven but task-agnostic, failing to optimize for multi-task scenarios. To address these gaps, we introduce TADS (Task-Aware Data Selection), a novel framework for multi-task
multimodal pre-training that integrates Intrinsic Quality, Task Relevance, and Distributional Diversity into a learnable value function. TADS employs a comprehensive quality assessment system with unimodal and cross-modal operators, quantifies task relevance via interpretable similarity vectors, and optimizes diversity through cluster-based weighting. A feedback-driven meta-learning mechanism adaptively refines the selection strategy based on proxy model performance across multiple downstream tasks. Experiments on CC12M demonstrate that TADS achieves superior zero-shot performance on
benchmarks like ImageNet, CIFAR-100, MS-COCO, and Flickr30K, using only 36% of the data while outperforming baselines by an average of 1.0%. This highlights that TADS significantly enhances data efficiency by curating a high-utility subset that yields a much higher performance ceiling within the same computational constraints.
Read the full abstract ▾
All authors Guanjie Cheng, Boyi Li, Lingyu Sun, Mengying Zhu, Yangyang Wu, Xinkui Zhao, Shuiguang Deng
Machine Learning
CORP: Closed-Form One-shot Representation-Preserving Structured Pruning
Boxiang Zhang and Baijian Yang
Transformers achieve strong accuracy but incur high compute and
memory cost. Structured
pruning reduces inference cost, but most methods rely on retraining or multi-stage optimization, which limits post-training deployment. We propose CORP, a closed-form one-shot structured
pruning method that removes MLP dimensions and
attention substructures using only unlabeled calibration data without gradients or fine-tuning. CORP formulates structured
pruning as a representation recovery problem. It models removed components as affine functions of retained components and derives closed-form ridge regression solutions that fold compensation into model weights. This minimizes a layer-local affine/logit reconstruction objective under the calibration distribution. Experiments on ImageNet with DeiT reveal strong redundancy in both MLP and
attention representations. With CORP, models retain high accuracy under aggressive sparsity. On DeiT-Huge, CORP achieves 83.24% Top-1 accuracy after
pruning 50% of both MLP and
attention structures.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SE
EGSS: Entropy-guided Stepwise Scaling for Reliable Software Engineering
Chenhui Mao, Yuanting Lei, Zhixiang Wei et al. (+6 more)
Agentic Test-Time Scaling (TTS) has delivered state-of-the-art (SOTA) performance on complex software engineering tasks such as code generation and bug fixing. However, its practical adoption remains limited due to significant computational overhead, primarily driven by two key challenges: (1) the high cost associated with deploying excessively large ensembles, and (2) the lack of a reliable mechanism for selecting the optimal candidate solution, ultimately constraining the performance gains that can be realized. To address these challenges, we propose
Entropy-Guided Stepwise Scaling (EGSS), a novel TTS framework that dynamically balances efficiency and effectiveness through
entropy-guided adaptive search and robust test-suite augmentation. Extensive experiments on SWE-Bench-Verified demonstrate that EGSS consistently boosts performance by 5-10% across all evaluated models. Specifically, it increases the resolved ratio of Kimi-K2-Intruct from 63.2% to 72.2%, and GLM-4.6 from 65.8% to 74.6%. Furthermore, when paired with GLM-4.6, EGSS achieves a new state-of-the-art among open-source large
language models. In addition to these accuracy improvements, EGSS reduces inference-time
token usage by over 28% compared to existing TTS methods, achieving simultaneous gains in both effectiveness and computational efficiency.
Read the full abstract ▾
All authors Chenhui Mao, Yuanting Lei, Zhixiang Wei, Ming Liang, Zhixiang Wang, Jingxuan Xu, Dajun Chen, Wei Jiang, Yong Li
Machine Learning · Cross-listed from cs.CV
PatchFlow: Leveraging a Flow-Based Model with Patch Features
Boxiang Zhang, Baijian Yang, Xiaoming Wang et al. (+1 more)
Die casting plays a crucial role across various industries due to its ability to craft intricate shapes with high precision and smooth surfaces. However, surface defects remain a major issue that impedes die casting quality control. Recently, computer
vision techniques have been explored to automate and improve defect detection. In this work, we combine local neighbor-aware patch features with a normalizing flow model and bridge the gap between the generic pretrained feature extractor and industrial product images by introducing an adapter module to increase the efficiency and accuracy of automated anomaly detection. Compared to state-of-the-art methods, our approach reduces the error rate by 20% on the MVTec AD
dataset, achieving an image-level AUROC of 99.28%. Our approach has also enhanced performance on the VisA
dataset , achieving an image-level AUROC of 96.48%. Compared to the state-of-the-art models, this represents a 28.2% reduction in error. Additionally, experiments on a proprietary die casting
dataset yield an accuracy of 95.77% for anomaly detection, without requiring any anomalous samples for training. Our method illustrates the potential of leveraging computer
vision and deep learning techniques to advance inspection capabilities for the die casting industry
Read the full abstract ▾
All authors Boxiang Zhang, Baijian Yang, Xiaoming Wang, Corey Vian
Machine Learning
Faithful Bi-Directional Model Steering via Distribution Matching and Distributed Interchange Interventions
Yuntai Bao, Xuhong Zhang, Jintao Chen et al. (+7 more)
Intervention-based model steering offers a lightweight and interpretable alternative to prompting and fine-tuning. However, by adapting strong optimization objectives from fine-tuning, current methods are susceptible to overfitting and often underperform, sometimes generating unnatural outputs. We hypothesize that this is because effective steering requires the faithful identification of internal model mechanisms, not the enforcement of external preferences. To this end, we build on the principles of distributed
alignment search (DAS), the standard for causal variable localization, to propose a new steering method: Concept DAS (CDAS). While we adopt the core mechanism of DAS, distributed interchange intervention (DII), we introduce a novel distribution matching objective tailored for the steering task by aligning intervened output distributions with counterfactual distributions. CDAS differs from prior work in two main ways: first, it learns interventions via weak-supervised distribution matching rather than probability maximization; second, it uses DIIs that naturally enable bi-directional steering and allow steering factors to be derived from data, reducing the effort required for hyperparameter tuning and resulting in more faithful and stable control. On AxBench, a large-scale model steering
benchmark, we show that CDAS does not always outperform preference-optimization methods but may benefit more from increased model scale. In two safety-related case studies, overriding refusal behaviors of safety-aligned models and neutralizing a chain-of-thought
backdoor, CDAS achieves systematic steering while maintaining general model utility. These results indicate that CDAS is complementary to preference-optimization approaches and conditionally constitutes a robust approach to intervention-based model steering. Our code is available at https://github.com/colored-dye/concept_das.
Read the full abstract ▾
All authors Yuntai Bao, Xuhong Zhang, Jintao Chen, Ge Su, Yuxiang Cai, Hao Peng, Bing Sun, Haiqin Weng, Liu Yan, Jianwei Yin
Machine Learning
Balanced Anomaly-guided Ego-graph Diffusion Model for Inductive Graph Anomaly Detection
Chunyu Wei, Siyuan He, Yu Wang et al. (+7 more)
Graph anomaly detection (GAD) is crucial in applications like fraud detection and cybersecurity. Despite recent advancements using graph neural networks (GNNs), two major challenges persist. At the model level, most methods adopt a transductive learning paradigm, which assumes static graph structures, making them unsuitable for dynamic, evolving networks. At the data level, the extreme class imbalance, where anomalous nodes are rare, leads to biased models that fail to generalize to unseen anomalies. These challenges are interdependent: static transductive frameworks limit effective data augmentation, while imbalance exacerbates model distortion in inductive learning settings. To address these challenges, we propose a novel data-centric framework that integrates dynamic graph modeling with balanced anomaly synthesis. Our framework features: (1) a discrete ego-graph
diffusion model, which captures the local
topology of anomalies to generate ego-graphs aligned with anomalous structural distribution, and (2) a curriculum anomaly augmentation mechanism, which dynamically adjusts synthetic data generation during training, focusing on underrepresented anomaly patterns to improve detection and generalization. Experiments on five
datasets demonstrate that the effectiveness of our framework.
Read the full abstract ▾
All authors Chunyu Wei, Siyuan He, Yu Wang, Yueguo Chen, Yunhai Wang, Bing Bai, Yidong Zhang, Yong Xie, Shunming Zhang, Fei Wang
Machine Learning
Jiecheng Lu, Xu Han, Yan Sun et al. (+4 more)
Linear
attention methods offer
Transformers $O(N)$ complexity but typically underperform standard softmax
attention. We identify two fundamental limitations affecting these approaches: the restriction to convex combinations that only permits additive information blending, and uniform accumulated weight bias that dilutes
attention in long contexts. We propose Zero-Sum Linear
Attention (ZeroS), which addresses these limitations by removing the constant zero-order term $1/t$ and reweighting the remaining zero-sum softmax residuals. This modification creates mathematically stable weights, enabling both positive and negative values and allowing a single
attention layer to perform contrastive operations. While maintaining $O(N)$ complexity, ZeroS theoretically expands the set of representable functions compared to convex combinations. Empirically, it matches or exceeds standard softmax
attention across various sequence modeling
benchmarks.
Read the full abstract ▾
All authors Jiecheng Lu, Xu Han, Yan Sun, Viresh Pati, Yubin Kim, Siddhartha Somani, Shihao Yang
Machine Learning · Cross-listed from stat.ML
Radon–Wasserstein Gradient Flows for Interacting-Particle Sampling in High Dimensions
Elias Hess-Childs, Dejan Slepčev and Lantian Xu
Gradient flows of the Kullback–Leibler (KL) divergence, such as the Fokker–Planck equation and Stein Variational Gradient Descent, evolve a distribution toward a target density known only up to a normalizing constant. We introduce new gradient flows of the KL divergence with a remarkable combination of properties: they admit accurate interacting-particle approximations in high dimensions, and the per-step cost scales linearly in both the number of particles and the dimension. These gradient flows are based on new transportation-based Riemannian geometries on the space of probability measures: the Radon–Wasserstein geometry and the related Regularized Radon–Wasserstein (RRW) geometry. We define these geometries using the Radon transform so that the gradient-flow velocities depend only on one-dimensional projections. This yields interacting-particle-based
algorithms whose per-step cost follows from efficient Fast
Fourier Transform-based evaluation of the required 1D convolutions. We additionally provide numerical experiments that study the performance of the proposed
algorithms and compare convergence behavior and quantization. Finally, we prove some theoretical results including well-posedness of the flows and long-time convergence guarantees for the RRW flow.
Read the full abstract ▾
Machine Learning
Private Prediction via Shrinkage
Chao Yan
We study differentially private prediction introduced by Dwork and Feldman (COLT 2018): an
algorithm receives one labeled sample set $S$ and then answers a stream of unlabeled queries while the output transcript remains $(\varepsilon,\delta)$-differentially private with respect to $S$. Standard composition yields a $\sqrt{T}$ dependence for $T$ queries. We show that this dependence can be reduced to polylogarithmic in $T$ in streaming settings. For an oblivious online adversary and any concept class $\mathcal{C}$, we give a private predictor that answers $T$ queries with $|S|= \tilde{O}(VC(\mathcal{C})^{3.5}\log^{3.5}T)$ labeled examples. For an adaptive online adversary and halfspaces over $\mathbb{R}^d$, we obtain $|S|=\tilde{O}\left(d^{5.5}\log T\right)$.
Read the full abstract ▾
Machine Learning
Disentangled Representation Learning via Flow Matching
Jinjin Chi, Taoping Liu, Mengtao Yin et al. (+5 more)
Disentangled representation learning aims to capture the underlying explanatory factors of observed data, enabling a principled understanding of the data-generating process. Recent advances in generative modeling have introduced new paradigms for learning such representations. However, existing
diffusion-based methods encourage factor independence via inductive biases, yet frequently lack strong semantic
alignment. In this work, we propose a flow matching-based framework for disentangled representation learning, which casts disentanglement as learning factor-conditioned flows in a compact latent space. To enforce explicit semantic
alignment, we introduce a non-overlap (orthogonality) regularizer that suppresses cross-factor interference and reduces information leakage between factors. Extensive experiments across multiple
datasets demonstrate consistent improvements over representative baselines, yielding higher disentanglement scores as well as improved controllability and sample fidelity.
Read the full abstract ▾
All authors Jinjin Chi, Taoping Liu, Mengtao Yin, Ximing Li, Yongcheng Jing, Jialie Shen, Leszek Rutkowski, Dacheng Tao
Machine Learning
Extreme Weather Nowcasting via Local Precipitation Pattern Prediction
Changhoon Song, Teng Yuan Chang and Youngjoon Hong
Accurate
forecasting of extreme weather events such as heavy rainfall or storms is critical for risk management and disaster mitigation. Although high-resolution radar observations have spurred extensive research on nowcasting models, precipitation nowcasting remains particularly challenging due to pronounced spatial locality, intricate fine-scale rainfall structures, and variability in
forecasting horizons. While recent
diffusion-based generative ensembles show promising results, they are computationally expensive and unsuitable for real-time applications. In contrast, deterministic models are computationally efficient but remain biased toward normal rainfall. Furthermore, the
benchmark datasets commonly used in prior studies are themselves skewed–either dominated by ordinary rainfall events or restricted to extreme rainfall episodes–thereby hindering general applicability in real-world settings. In this paper, we propose exPreCast, an efficient deterministic framework for generating finely detailed radar forecasts, and introduce a newly constructed balanced radar
dataset from the Korea Meteorological Administration (KMA), which encompasses both ordinary precipitation and extreme events. Our model integrates local spatiotemporal
attention, a texture-preserving cubic dual upsampling decoder, and a temporal extractor to flexibly adjust
forecasting horizons. Experiments on established
benchmarks (SEVIR and MeteoNet) as well as on the balanced KMA
dataset demonstrate that our approach achieves state-of-the-art performance, delivering accurate and reliable nowcasts across both normal and extreme rainfall regimes.
Read the full abstract ▾
Machine Learning
Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
Wentao Ni, Kangqi Zhang, Zhongming Yu et al. (+7 more)
As long-context inference becomes central to large
language models (LLMs),
attention over growing key-value caches emerges as a dominant decoding bottleneck, motivating sparse
attention for scalable inference. Fixed-budget top-k sparse
attention cannot adapt to heterogeneous
attention distributions across heads and layers, whereas top-p sparse
attention directly preserves
attention mass and provides stronger accuracy guarantees. Existing top-p methods, however, fail to jointly optimize top-p accuracy, selection overhead, and sparse
attention cost, which limits their overall efficiency. We present Double-P, a hierarchical sparse
attention framework that optimizes all three stages. Double-P first performs coarse-grained top-p estimation at the cluster level using size-weighted centroids, then adaptively refines computation through a second top-p stage that allocates
token-level
attention only when needed. Across long-context
benchmarks, Double-P consistently achieves near-zero accuracy drop, reducing
attention computation overhead by up to 1.8x and delivers up to 1.3x end-to-end decoding speedup over state-of-the-art fixed-budget sparse
attention methods.
Read the full abstract ▾
All authors Wentao Ni, Kangqi Zhang, Zhongming Yu, Oren Nelson, Mingu Lee, Hong Cai, Fatih Porikli, Jongryool Kim, Zhijian Liu, Jishen Zhao
Machine Learning · Cross-listed from cs.CL
Are Open-Weight LLMs Ready for Social Media Moderation? A Comparative Study on Bluesky
Hsuan-Yu Chou, Wajiha Naveed, Shuyan Zhou et al. (+1 more)
As internet access expands, so does exposure to harmful content, increasing the need for effective moderation. Research has demonstrated that large
language models (LLMs) can be effectively utilized for social media moderation tasks, including harmful content detection. While proprietary LLMs have been shown to zero-shot outperform traditional machine learning models, the out-of-the-box capability of open-weight LLMs remains an open question. Motivated by recent developments of
reasoning LLMs, we evaluate seven state-of-the-art models: four proprietary and three open-weight. Testing with real-world posts on Bluesky, moderation decisions by Bluesky Moderation Service, and annotations by two authors, we find a considerable degree of overlap between the sensitivity (81%–97%) and specificity (91%–100%) of the open-weight LLMs and those (72%–98%, and 93%–99%) of the proprietary ones. Additionally, our analysis reveals that specificity exceeds sensitivity for rudeness detection, but the opposite holds for intolerance and threats. Lastly, we identify inter-rater agreement across human moderators and the LLMs, highlighting considerations for deploying LLMs in both platform-scale and personalized moderation contexts. These findings show open-weight LLMs can support
privacy-preserving moderation on consumer-grade hardware and suggest new directions for designing moderation systems that balance community values with individual user preferences.
Read the full abstract ▾
All authors Hsuan-Yu Chou, Wajiha Naveed, Shuyan Zhou, Xiaowei Yang
Machine Learning
SpectraKAN: Conditioning Spectral Operators
Chun-Wun Cheng, Carola-Bibiane Schönlieb and Angelica I. Aviles-Rivero
Spectral neural operators, particularly
Fourier Neural Operators (FNO), are a powerful framework for learning solution operators of partial differential equations (PDEs) due to their efficient global mixing in the frequency domain. However, existing spectral operators rely on static
Fourier kernels applied uniformly across inputs, limiting their ability to capture multi-scale, regime-dependent, and anisotropic dynamics governed by the global state of the system. We introduce SpectraKAN, a neural operator that conditions the spectral operator on the input itself, turning static spectral convolution into an input-conditioned integral operator. This is achieved by extracting a compact global representation from spatio-temporal history and using it to modulate a multi-scale
Fourier trunk via single-query cross-
attention, enabling the operator to adapt its behaviour while retaining the efficiency of spectral mixing. We provide theoretical justification showing that this modulation converges to a resolution-independent continuous operator under mesh refinement and KAN gives smooth, Lipschitz-controlled global modulation. Across diverse PDE
benchmarks, SpectraKAN achieves state-of-the-art performance, reducing RMSE by up to 49% over strong baselines, with particularly large gains on challenging spatio-temporal prediction tasks.
Read the full abstract ▾
Machine Learning · Cross-listed from hep-th
Towards Worst-Case Guarantees with Scale-Aware Interpretability
Lauren Greenspan, David Berman, Aryeh Brill et al. (+9 more)
Neural networks organize information according to the hierarchical, multi-scale structure of natural data. Methods to interpret model internals should be similarly scale-aware, explicitly tracking how features compose across resolutions and guaranteeing bounds on the influence of fine-grained structure that is discarded as irrelevant noise. We posit that the renormalisation framework from physics can meet this need by offering technical tools that can overcome limitations of current methods. Moreover, relevant work from adjacent fields has now matured to a point where scattered research threads can be synthesized into practical, theory-informed tools. To combine these threads in an AI safety context, we propose a unifying research agenda – scale-aware interpretability – to develop formal machinery and interpretability tools that have robustness and faithfulness properties supported by statistical physics.
Read the full abstract ▾
All authors Lauren Greenspan, David Berman, Aryeh Brill, Ro Jefferson, Artemy Kolchinsky, Jennifer Lin, Andrew Mack, Anindita Maiti, Fernando E. Rosas, Alexander Stapleton, Lucas Teixeira, Dmitry Vaintrob
Machine Learning
Data-Centric Interpretability for LLM-based Multi-Agent Reinforcement Learning
John Yan, Michael Yu, Yuqi Sun et al. (+3 more)
Large
language models (LLMs) are increasingly trained in complex Reinforcement Learning, multi-
agent environments, making it difficult to understand how behavior changes over training. Sparse Autoencoders (SAEs) have recently shown to be useful for data-centric interpretability. In this work, we analyze large-scale reinforcement learning training runs from the sophisticated environment of Full-Press Diplomacy by applying pretrained SAEs, alongside LLM-summarizer methods. We introduce Meta-Autointerp, a method for grouping SAE features into interpretable hypotheses about training dynamics. We discover fine-grained behaviors including role-playing patterns, degenerate outputs,
language switching, alongside high-level strategic behaviors and environment-specific bugs. Through automated evaluation, we validate that 90% of discovered SAE Meta-Features are significant, and find a surprising reward hacking behavior. However, through two user studies, we find that even subjectively interesting and seemingly helpful SAE features may be worse than useless to humans, along with most LLM generated hypotheses. However, a subset of SAE-derived hypotheses are predictively useful for downstream tasks. We further provide validation by augmenting an untrained
agent's system
prompt, improving the score by +14.2%. Overall, we show that SAEs and LLM-summarizer provide complementary views into
agent behavior, and together our framework forms a practical starting point for future data-centric interpretability work on ensuring trustworthy LLM behavior throughout training.
Read the full abstract ▾
All authors John Yan, Michael Yu, Yuqi Sun, Alexander Duffy, Tyler Marques, Matthew Lyle Olson
Machine Learning
Benchmarking Artificial Intelligence Models for Daily Coastal Hypoxia Forecasting
Magesh Rajasekaran, Md Saiful Sajol, Chris Alvin et al. (+3 more)
Coastal hypoxia, especially in the northern part of Gulf of Mexico, presents a persistent ecological and economic concern. Seasonal models offer coarse forecasts that miss the fine-scale variability needed for daily, responsive ecosystem management. We present study that compares four deep learning architectures for daily hypoxia classification: Bidirectional Long Short-Term
Memory (BiLSTM), Medformer (Medical
Transformer), Spatio-Temporal
Transformer (ST-
Transformer), and Temporal Convolutional Network (TCN). We trained our models with twelve years of daily hindcast data from 2009-2020 Our training data consists of 2009-2020 hindcast data from a coupled hydrodynamic-biogeochemical model. Similarly, we use hindcast data from 2020 through 2024 as a test data. We constructed classification models incorporating water column stratification, sediment oxygen consumption, and temperature-dependent decomposition rates. We evaluated each architectures using the same data preprocessing, input/output formulation, and validation protocols. Each model achieved high classification accuracy and strong discriminative ability with ST-
Transformer achieving the highest performance across all metrics and tests periods (AUC-ROC: 0.982-0.992). We also employed McNemar's method to identify statistically significant differences in model predictions. Our contribution is a reproducible framework for operational real-time hypoxia prediction that can support broader efforts in the environmental and ocean modeling systems community and in ecosystem resilience. The source code is available https://github.com/rmagesh148/hypoxia-ai/
Read the full abstract ▾
All authors Magesh Rajasekaran, Md Saiful Sajol, Chris Alvin, Supratik Mukhopadhyay, Yanda Ou, Z. George Xue
Machine Learning · Cross-listed from stat.ML
Total Variation Rates for Riemannian Flow Matching
Yunrui Guan, Krishnakumar Balasubramanian and Shiqian Ma
Riemannian flow matching (RFM) extends flow-based generative modeling to data supported on
manifolds by learning a time-dependent tangent vector field whose flow-ODE transports a simple base distribution to the data law. We develop a nonasymptotic Total Variation (TV) convergence analysis for RFM samplers that use a learned vector field together with Euler discretization on
manifolds. Our key technical ingredient is a differential inequality governing the
evolution of TV between two
manifold ODE flows, which expresses the time-derivative of TV through the divergence of the vector-field mismatch and the score of the reference flow; controlling these terms requires establishing new bounds that explicitly account for parallel transport and
curvature. Under smoothness assumptions on the population flow-matching field and either uniform (compact
manifolds) or mean-square (Hadamard
manifolds) approximation guarantees for the learned field, we obtain explicit bounds of the form $\mathrm{TV}\le C_{\mathrm{Lip}}\,h + C_{\varepsilon}\,\varepsilon$ (with an additional higher-order $\varepsilon^2$ term on compact
manifolds), cleanly separating numerical discretization and learning errors. Here, $h$ is the step-size and $\varepsilon$ is the target accuracy. Instantiations yield explicit
polynomial iteration complexities on the hypersphere $S^d$, and on the SPD$(n)$
manifolds under mild moment conditions.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Finite-Particle Rates for Regularized Stein Variational Gradient Descent
Ye He, Krishnakumar Balasubramanian, Sayan Banerjee et al. (+1 more)
We derive finite-particle rates for the regularized Stein variational gradient descent (R-SVGD)
algorithm introduced by He et al. (2024) that corrects the constant-order bias of the SVGD by applying a resolvent-type preconditioner to the kernelized Wasserstein gradient. For the resulting interacting $N$-particle system, we establish explicit non-asymptotic bounds for time-averaged (annealed) empirical measures, illustrating convergence in the true (non-kernelized) Fisher information and, under a $\mathrm{W}_1\mathrm{I}$ condition on the target, corresponding $\mathrm{W}_1$ convergence for a large class of smooth kernels. Our analysis covers both continuous- and discrete-time dynamics and yields principled tuning rules for the regularization parameter, step size, and averaging horizon that quantify the trade-off between approximating the Wasserstein gradient flow and controlling finite-particle estimation error.
Read the full abstract ▾
All authors Ye He, Krishnakumar Balasubramanian, Sayan Banerjee, Promit Ghosal