Machine Learning · Cross-listed from cs.CV
A More Word-like Image Tokenization for MLLMs
Hyun Lee, Hyemin Jeong, Yejin Kim et al. (+4 more)
Modern
multimodal large
language models (MLLMs) typically keep the
language model fixed and train a visual projector that maps the pixels into a sequence of
tokens in its
embedding space, so that images can be presented in essentially the same form as text. However, the
language model has been optimized to operate on discrete, semantically meaningful
tokens, while prevailing visual projectors transform an image into a long stream of continuous and highly correlated
embeddings. This causes the visual
tokens to behave differently from the word-like units that LLMs are originally trained to understand. We propose a novel Disentangled Visual Tokenization (DiVT) that clusters patch
embeddings into coherent semantic units, so each
token corresponds to a distinct visual concept instead of a rigid grid
cell. DiVT further adapts its
token budget to image complexity, providing an explicit accuracy-compute trade-off modifying neither the
vision encoder nor the
language model. Across diverse
multimodal benchmarks, DiVT matches or surpasses baselines with significantly fewer visual
tokens, demonstrating robustness under limited
token budgets, significantly reducing
memory cost and latency while making visual inputs more compatible with LLMs. Our code is available at https://github.com/snuviplab/DiVT.
Read the full abstract ▾
All authors Hyun Lee, Hyemin Jeong, Yejin Kim, Hyungwook Choi, Hyunsoo Cho, Soo Kyung Kim, Joonseok Lee
Machine Learning · Cross-listed from cs.AI
SVFSearch: A Multimodal Knowledge-Intensive Benchmark for Short-Video Frame Search in the Gaming Vertical Domain
Lingtao Mao, Huangyu Dai, Xinyu Sun et al. (+4 more)
Multimodal large
language models are increasingly used as
agent backbones that understand
multimodal inputs, plan
retrieval actions, invoke external tools, and reason over retrieved information. Yet existing
benchmarks rarely evaluate this ability in short-
video applications, where a paused frame is often visually ambiguous and answering requires vertical, long-tail, and fast-evolving domain knowledge. We introduce SVFSearch, the first open
benchmark for short-
video frame search in the Chinese gaming domain. SVFSearch contains 5,000 four-choice test examples and 4,198 auxiliary training examples, each centered on a paused game scene from a real short-
video clip. To support fair and reproducible evaluation, SVFSearch provides a frozen offline
retrieval environment with a game-domain text corpus, a topic-linked image gallery, and text, image, and
multimodal retrieval interfaces, avoiding reliance on uncontrolled web search APIs. We evaluate representative paradigms ranging from direct QA and RAG workflow to Plan-Act-Replan
agents and learned search models. Results reveal a large gap between model-only answering, practical agentic search, and oracle knowledge: the best open-source direct-QA model reaches 66.4%, the best practical
agent achieves 79.1%, and oracle knowledge reaches 95.4%. Further analysis exposes bottlenecks in visual grounding,
retrieval quality, evidence-grounded
reasoning, and tool-use behavior, including over-search, answer-only shortcuts, and
retrieval-induced misleading.
Read the full abstract ▾
All authors Lingtao Mao, Huangyu Dai, Xinyu Sun, Zihan Liang, Ben Chen, Chenyi Lei, Wenwu Ou
Machine Learning
Training data attribution in diffusion models via mirrored unlearning and noise-consistent skew
Joan Serrà, Dipam Goswami, Fabio Morreale et al. (+2 more)
Training data attribution (TDA) should enable generative model interpretability and foster a variety of related downstream tasks. Nonetheless, current TDA approaches lack reliability and robustness, preventing their adoption in real-world setups. In this paper, we take a decisive step towards more reliable and robust TDA for
diffusion models. We propose to perform TDA with mirrored unlearning and noise-consistent skew (MUCS). The idea is to fine-tune a second model with bounded mirrored gradient ascent, and to measure the normalized skew of this model with respect to the original one using consistent noise samples. We show that, while being conceptually simple and generic, MUCS systematically outperforms existing methods on three different
datasets by a large margin. We additionally study the effect that core design choices have on final performance, and analyze novel aspects regarding the overlap of influential instances across generated items and the potential of ensembling TDA approaches. We believe that our findings may have broader implications for more general unlearning setups, as well as for tasks requiring the comparison of
diffusion losses.
Read the full abstract ▾
All authors Joan Serrà, Dipam Goswami, Fabio Morreale, Wei-Hsiang Liao, Yuki Mitsufuji
Machine Learning · Cross-listed from cs.CL
Benedict Florance Arockiaraj, Alexander Feng, Jianxiong Cai et al. (+1 more)
Recent works have illustrated that modern NLP models trained for diverse tasks ranging from sentiment analysis to
language generation succumb to universal
adversarial attacks, a class of input-agnostic attacks where a common trigger sequence is used to attack the model. Although these attacks are successful, the triggers generated by such attacks are ungrammatical and unnatural. Our work proposes a novel technique combining parts-of-
speech filtering and perplexity based loss function to generate sensible triggers that are closer to natural phrases. For the task of sentiment analysis on the SST
dataset, the method produces sensible triggers that achieve accuracies as low as 0.04 and 0.12 for flipping positive to negative predictions and vice-versa. To build robust models, we also perform
adversarial training using the generated triggers that increases the accuracy of the model from 0.12 to 0.48. We aim to illustrate that
adversarial attacks can be made difficult to detect by generating sensible triggers, and to facilitate robust model development through relevant defenses.
Read the full abstract ▾
All authors Benedict Florance Arockiaraj, Alexander Feng, Jianxiong Cai, Xiaoyu Cheng
Machine Learning
InfoFlow: A Framework for Multi-Layer Transformer Analysis
Penghao Yu, Haotian Jiang, Zeyu Bao et al. (+1 more)
While the approximation properties of single-layer
Transformer architectures have been studied in recent works, a rigorous theoretical understanding of the multi-layer setting remains limited. In this work, we establish that multi-layer
Transformers possess fundamentally different approximation capabilities from single-layer ones: for certain
retrieval tasks, any single-layer
Transformer requires least $\Omega (\varepsilon^{-k})$ parameters to achieve precision $\varepsilon$, where $k$ grows linearly with sequence length $T$, whereas a two-layer
Transformer with a single head per layer achieves the same approximation precision with at most $O (\varepsilon^{-1})$ parameters. To understand this separation, we identify two structural mechanisms underlying multi-layer approximation. Specifically, softmax
attention can only efficiently retrieve the
token attaining the maximum
attention score, incurring exponential-in-length parameter cost for $k$-th largest
retrieval with $k \geq 2$. Moreover, the parameter cost of decoding coupled information scales with the size of the retrieved
token set. Motivated by these findings, we propose InfoFlow, a framework for multi-layer
Transformers. The framework tracks an information set of accessible input positions at each
token and layer, assigning an explicit approximation rate to each mode of information propagation. This abstraction recovers known approximation bounds, remains consistent with experimental observations on trained networks, and yields concrete predictions in settings where direct theoretical analysis is currently intractable. Our results provide a principled framework for
reasoning about the approximation efficiency of multi-layer
Transformers.
Read the full abstract ▾
All authors Penghao Yu, Haotian Jiang, Zeyu Bao, Qianxiao Li
Machine Learning · Cross-listed from cs.RO
Transfer Learning for Customized Car Racing Environments
Benedict Florance Arockiaraj, Richard Chang and Wesley Yee
Transfer Learning, a technique where a model/
agent can use the knowledge/expertise that it gained from one task and exploit that to solve another closely-related task, is often used in tackling problems in deep learning. Through this project, we explore transfer learning in the purview of deep reinforcement learning. Specifically, we want to use transfer learning to achieve the fast lap times in OpenAI's Car racing environment by training the
agent on one
circuit, and racing it on other customized target environments by zero-shot transfer or by additional fine-tuning. In addition, we compare the performance of model-based and model-free approaches, and observe that model-based approaches dominate in performance and converge faster than model-free approaches in this environment. We observe that transfer learning in most setups not only boosts the performance on the target domain, but also shows high performance ability during learning.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.DC
Yucheng Guo, Yongjian Guo, Zhong Guan et al. (+6 more)
In
video generation models, particularly world models, training large-scale
video diffusion Transformers (such as DiT and MMDiT) poses significant computational challenges due to the extreme variance in sequence lengths within mixed-mode
datasets. Existing bucket-based data loading strategies typically rely on "equal
token length" constraints. This approach fails to account for the quadratic complexity of self-
attention mechanisms, leading to severe load imbalance and underutilization of GPU resources. This paper proposes AdaptiveLoad, an integrated optimization framework consisting of two core components: (1) A dual-constraint adaptive load balancing system, which eliminates long-sequence bottlenecks by simultaneously limiting
memory consumption and computational load ($B \times S^p \le M_{\text{comp}}$); (2) A fused LayerNorm-Modulate CUDA kernel, which utilizes a D-tile coalesced reduction strategy to increase throughput and alleviate
memory pressure. Experimental results on the Wan 2.1 world model demonstrate that our method reduces the computational imbalance rate from 39% to 18.9%, improves peak VRAM utilization efficiency by 22.7%, and achieves an overall training throughput increase of 27.2%.
Read the full abstract ▾
All authors Yucheng Guo, Yongjian Guo, Zhong Guan, Haoran Sun, Wen Huang, Wanting Xu, Jing Long, Shuai Di, Junwu Xiong
Machine Learning
Domain Transfer Becomes Identifiable via a Single Alignment
Sagar Shrestha, Subash Timilsina, Hoang-Son Nguyen et al. (+1 more)
Domain transfer (DT) maps source to target distributions and supports tasks such as unsupervised image-to-image translation, single-
cell analysis, and cross-platform medical imaging. However, DT is fundamentally ill-posed: push-forward mappings are generally non-identifiable, as measure-preserving
automorphisms (MPAs) preserve marginals while altering cross-domain correspondences, leading to content-misaligned translation. Recent work shows that MPAs can be eliminated by jointly transferring multiple corresponding source/target conditional distributions, but supervision signals labeling such conditionals are not always available in practice. We develop an alternative route to DT identifiability. Under a structural sparsity condition on the Jacobian support pattern, we show that distribution matching together with a single paired anchor sample suffices to identify the ground-truth transfer – requiring substantially less supervision than prior approaches. To enable practical high-dimensional learning, we further propose an efficient Jacobian sparsity regularizer based on randomized masked finite differences, yielding a scalable surrogate without explicit Jacobian evaluation. Empirical results on synthetic and real-world DT tasks validate the theory.
Read the full abstract ▾
All authors Sagar Shrestha, Subash Timilsina, Hoang-Son Nguyen, Xiao Fu
Machine Learning
DCFold: Efficient Protein Structure Generation with Single Forward Pass
Zhe Zhang, Yuanning Feng, Yuxuan Song et al. (+3 more)
AlphaFold3 introduces a
diffusion-based architecture that elevates
protein structure prediction to all-atom resolution with improved accuracy. This state-of-the-art performance has established AlphaFold3 as a foundation model for diverse generation and design tasks. However, its iterative design substantially increases inference time, limiting practical deployment in downstream settings such as virtual screening and
protein design. We propose DCFold, a single-step generative model that attains AlphaFold3-level accuracy. Our Dual Consistency training framework, which incorporates a novel Temporal Geodesic Matching (TGM) scheduler, enables DCFold to achieve a 15x acceleration in inference while maintaining predictive fidelity. We validate its effectiveness across both structure prediction and binder design
benchmarks.
Read the full abstract ▾
All authors Zhe Zhang, Yuanning Feng, Yuxuan Song, Keyue Qiu, Hao Zhou, Wei-Ying Ma
Machine Learning
Lightweight Gaussian Process Inference in C++ on Metal and CUDA
Yu-Hsueh Fang
Gaussian process (GP) inference in Python is dominated by libraries such as GPyTorch and GPflow, which are built on deep-learning frameworks and inherit their dispatch overhead and dependency footprint. We present LightGP, a dependency-free C++17 library for GP regression with Python bindings, supporting Apple Metal and NVIDIA CUDA backends alongside tuned CPU paths via Apple Accelerate and OpenBLAS. LightGP provides four inference paths – exact Cholesky,
matrix-free conjugate gradients, sparse variational free energy, and structured kernel interpolation with FFT – covering problems from $N{=}100$ to $N{=}500{,}000$. On an Apple M4, LightGP CPU is 2.6–8.7$\times$ faster than GPyTorch CPU for exact GP and ${\sim}1.5\times$ faster for sparse GP at every scale tested. On an NVIDIA RTX~3060, LightGP CUDA is 2.3–6.7$\times$ faster than GPyTorch CUDA for exact GP up to $N{=}2{,}048$, with GPyTorch closing the gap at $N{=}4{,}096$. A fused
matrix-free kernel-vector product on Metal achieves 32$\times$ over the explicit path at $N{=}20{,}000$ with $O(N)$
memory, and an FFT-accelerated SKI matvec via Accelerate vDSP runs in sub-millisecond time at $N{=}200{,}000$. LightGP compiles as a single static library with zero external dependencies and is installable via \texttt{pip install lightgp
Read the full abstract ▾
Machine Learning
CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution
Muyoung Son, Yi Chen, Seungjae Yoo et al. (+2 more)
The Mixture-of-Experts (MoE) architecture improves computational efficiency via sparse expert activation, but throughput-oriented inference faces substantial GPU
memory pressure due to a significant parameter size and intermediate data. Prior works attempt to mitigate this using expert offloading with micro-batching or by offloading computation to the CPU. However, the fragmented workload resulting from micro-batching degrades operational intensity, causing expert execution to become
memory-bound. Meanwhile, CPU offloading is constrained by slow PCIe transfers and its limited applicability to
attention computation in the decode stage. Consequently, these inefficiencies prevent effective system utilization, severely restricting the end-to-end throughput of MoE inference. To address these challenges, this paper proposes CoX-MoE, an Advanced
Matrix Extensions (AMX)-enabled CPU-GPU collaborative system that comprehensively optimizes MoE inference by combining coalesced expert execution with strategic workload orchestration for higher throughput. CoX-MoE introduces (i) a coalescing-aware orchestration policy to jointly optimize resource allocation by adopting ordinary batch, instead of micro-batch, for expert computation and selective
attention offloading, and (ii) a static expert-aware stratification scheme that pre-assigns frequently activated experts to the GPU, mitigating PCIe transfer overhead and balancing workload for the CPU and GPU during inference. Compared to state-of-the-art frameworks, CoX-MoE delivers significant gains, achieving up to 7.1x and 2.4x higher throughput than FlexGen and MoE-Lightning, respectively.
Read the full abstract ▾
All authors Muyoung Son, Yi Chen, Seungjae Yoo, Soongyu Choi, Joo-Young Kim
Machine Learning · Cross-listed from physics.flu-dyn
Long-horizon prediction of three-dimensional wall-bounded turbulence with CTA-Swin-UNet and resolvent analysis
Bo Chen, Yitong Fan, Jie Yao et al. (+1 more)
Long-horizon prediction of three-dimensional (3D) wall-bounded
turbulence with machine-learning methods remains a challenging task, due to the rapid accumulation of autoregressive errors and the substantially computational cost. To address these challenges, we present a hybrid machine-learning framework, in which a channel-time-
attention Swin-UNet (CTA-Swin-UNet) and a multi-time-scale fusion correction (MTFC) strategy are developed to predict the turbulent flow fields in a wall-parallel plane, with affordable computational cost. Then, 3D flow fields are reconstructed via a resolvent-based spectral linear
stochastic estimation (SLSE), rooting from the predicted planar flow. Results show that the CTA-Swin-UNet outperforms the baseline models (LSTM, FNO and traditional Swin-UNet) in both single-step prediction and autoregressive rollouts, indicating the effectiveness of introducing the CTA module into the Swin-UNet architecture. At the same temporal interval, the CTA-Swin-UNet remains stable for approximately 150 rollout steps, while the baseline models fail within 20 to 50 rollout steps. After introducing the MTFC strategy, a longer horizon upto 300 steps is achieved. Using the resolvent-based SLSE reconstruction further recovers the 3D flow structures and energy spectral distributions from the predicted planar inputs, which demonstrates that the proposed framework provides an effective and computationally efficient approach for long-horizon autoregressive prediction of 3D wall-bounded
turbulence.
Read the full abstract ▾
All authors Bo Chen, Yitong Fan, Jie Yao, Weipeng Li
Machine Learning
Haozheng Luo, Haoran Dai, Jingyuan Huang et al. (+10 more)
We propose OASIS, an outlier- and sink-aware method that stabilizes dual-normalized
attention-residual architectures through explicit null routing and
token-to-depth null coupling. AttnResidual introduces an additional depth-wise normalization channel that improves inter-layer routing flexibility but can also amplify
attention sinks, activation outliers, and low-bit quantization error. OASIS builds on explicit Softmax1-based null routes at both the
token and depth levels and uses
token-level null evidence to downweight depth branches exhibiting stronger null behavior. Theoretically, we characterize a conditional mechanism for sink-like
attention concentration under dual normalization, offering insight into the low-bit sensitivity observed in AttnResidual. Experimentally, we compare OASIS against five baselines on three
language-model backbones and multiple
language-modeling,
reasoning, and long-context
benchmarks and observe consistent improvements in both
attention sink mitigation and post-quantization performance. Notably, relative to Vanilla AttnResidual, OASIS reduces maximum infinity norm by 81.75% and average kurtosis by 95.90%, lowers W8A8 perplexity by 82.00%, and improves W4A4 GSM8K Pass@1 by 42.11% on average across LLaMA-3.2-1B, Qwen3-0.6B, and Phi-4. Code is available at: https://github.com/robinzixuan/OASIS.
Read the full abstract ▾
All authors Haozheng Luo, Haoran Dai, Jingyuan Huang, Shaoyang Zhang, Xi Chen, Eric Hanchen Jiang, Yijiang Li, Chenghao Qiu, Chenwei Xu, Zhenyu Pan, Haotian Zhang, Binghui Wang, Yan Chen
Machine Learning · Cross-listed from cs.DC
Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training
Guanliang Liu, Abhinandan Patni, Congzhu Lin et al. (+14 more)
Training frontier-scale foundation models involves coordinating tens of thousands of GPUs over multi-month runs, where even minor performance degradations can accumulate into substantial efficiency losses. Existing health-check mechanisms, such as NCCL tests or GPU burn-in, primarily focus on functional correctness and often fail to detect fail-slow behaviors that silently degrade system performance. In this paper, we present Guard, a scalable system for detecting stragglers and ensuring node health in large-scale training clusters. Guard combines lightweight online performance monitoring during training with an offline node-sweep mechanism that systematically evaluates and qualifies nodes before they participate in production workloads. This design enables Guard to detect both acute failures and long-running fail-slow behaviors that traditional diagnostics cannot capture. Deployed on large-scale foundation model pretraining workloads, Guard improves mean FLOPs utilization by up to 1.7x, reduces run-to-run training step variance from 20% to 1%, increases mean time to failure (MTTF), and significantly reduces operational and debugging overhead. These results demonstrate that proactive straggler detection and systematic node qualification are critical for maintaining stable and efficient large-scale training.
Read the full abstract ▾
All authors Guanliang Liu, Abhinandan Patni, Congzhu Lin, Zoe Zeng, Jack Wittmayer, Josh Wu, Ashvin Nihalani, Binxuan Huang, Yinghong Liu, Rory Na, Anthony Ko, Alexander Zhipa, Cong Cheng, Mi Sun, Vijay Rajakumar, Rejith George Joseph, Parthasarathy Govindarajen
Machine Learning
HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
Woongyeong Yeo, Yumin Choi, Taekyung Ki et al. (+1 more)
Training long-horizon LLM
agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-
distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-
distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned
distillation only to targeted action spans. Experiments on BFCL v3 and AppWorld show that our method outperforms the dense per-turn feedback baseline by up to 13.60 percentage points on average while achieving a 2.26$\times$ reduction in time per training step, suggesting that selecting where to distill is key to effective and efficient long-horizon
agent training.
Read the full abstract ▾
All authors Woongyeong Yeo, Yumin Choi, Taekyung Ki, Sung Ju Hwang
Machine Learning
DAD4TS: Data-Augmentation-Oriented Diffusion Model for Time-Series Forecasting with Small-Scale Data
Masahiro Suzuki, Bohui Xia, Hiroto Yamamoto et al. (+1 more)
Small-scale data is a critical problem in time-series
forecasting tasks. Data augmentation is an effective strategy for this task, but it has a limitation in generating meaningful data. To address this limitation, we propose DAD4TS, a
diffusion-model-based data augmentation method with reinforcement learning, designed for time-series
forecasting with small-scale data. In DAD4TS, a data generator is simultaneously trained with a time-series model and controlled by a reinforcement learning model to efficiently generate samples that improve the forecast accuracy of the time-series model. To support small-scale data, we use mathematical methods instead of conventional VAE methods to train the
diffusion model by projecting the time-series data into the
geometric space. We validated the effectiveness of DAD4TS with seven comparative methods through qualitative and quantitative experiments on six real-world
datasets and eight time-series models. As a result, DAD4TS was validated on five
datasets.
Read the full abstract ▾
All authors Masahiro Suzuki, Bohui Xia, Hiroto Yamamoto, Masanori Miyahara
Machine Learning
$\boldsymbol{f}$-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
Xianwei Chen, Shimin Zhang and Jibin Wu
Scaling on-policy
distillation (OPD) for large
language models (LLMs) confronts a fundamental tension: asynchronous execution is necessary for system efficiency, but structurally deviates from the ideal on-policy objective. To address this challenge, we theoretically decompose the objective discrepancy into rollout drift and supervision drift, capturing staleness in student rollout and teacher context, respectively. Building on this, we introduce a sample-level freshness score that quantifies the reliability of a buffered sample with respect to the on-policy objective. Guided by this signal, we further propose f-OPD, a novel framework that adaptively regulates stale-sample influence and constrains policy drift accumulated under asynchronous training. Across
reasoning, tool-use, and coding-
agent tasks of increasing interaction horizon, f-OPD consistently achieves task performance comparable to synchronous optimization while largely retaining the throughput advantages of asynchronous execution. Our results establish the first recipe for achieving a performance-efficiency trade-off in OPD, paving the way for long-horizon agentic post-training at scale.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.HC
Multi-site PPG: An In-the-Wild Physiological Dataset from Emerging Multi-site Wearables
Jiayi Shao, Jiaying Ye, Shengyao Liu et al. (+4 more)
Wearables are widely used for mobile health monitoring, and photoplethysmography (PPG) is a key sensing modality for heart rate and related physiological measurements. However, public in-the-wild PPG
datasets remain largely wrist-centric or limited to short, controlled studies, constraining research on emerging wearable form factors. We present Multi-site PPG, an in-the-wild physiological
dataset collected from four custom-developed unobtrusive wearables: a smart earring, ring, watch, and necklace. Each device records green and infrared reflective PPG, 3-axis acceleration, and temperature with timestamps for cross-device
alignment, while a Polar H10 chest strap provides reference electrocardiogram (ECG). Participants wore the devices for multiple days during daytime activities while continuing their normal routines. The
dataset contains over 350 hours of raw data and 230-290 hours of modeling-ready 8-second windows per wearable. We
benchmark heuristic, supervised, and self-supervised heart-rate estimation methods, showing substantial body-site differences: the best methods achieve mean absolute errors (MAEs) of 2.30 bpm on the earring, 5.13 bpm on the ring, 8.37 bpm on the watch, and 8.68 bpm on the necklace. We further analyze motion effects and evaluate multi-site and PPG-accelerometer fusion, demonstrating the
dataset's value for robust physiological sensing across emerging wearable form factors.
Read the full abstract ▾
All authors Jiayi Shao, Jiaying Ye, Shengyao Liu, Zachary Englhardt, Girish Narayanswamy, Vikram Iyer, Qiuyue Shirley Xue
Machine Learning
Learning over Positive and Negative Edges with Contrastive Message Passing
Peter Pao-Huang, Charilaos I. Kanatsoulis, Michael Bereket et al. (+1 more)
Conventional approaches to learning on graphs involve message passing along existing (i.e., positive) edges to update node features. However, these approaches often disregard the potentially valuable information contained in the absence (i.e., negative) of edges. Here, we theoretically analyze the value of negative edges in graph representations and prove that in settings of low label rates, high homophily, and high edge density, access to negative edges provides significant information gain over using only positive edges. Motivated by this insight, we introduce Contrastive Message Passing (CMP), a general message passing architecture that enable graph neural network layers to reason over positive and negative edges. By imposing soft positive semidefinite constraints on the learnable weights, our approach differentially applies similarity-preserving transformations to positively connected nodes and dissimilarity-inducing transformations to negatively connected nodes. Over simulated and real
datasets in varying data regimes, CMP consistently outperforms baselines in low-label settings when negative edges are informative.
Read the full abstract ▾
All authors Peter Pao-Huang, Charilaos I. Kanatsoulis, Michael Bereket, Jure Leskovec
Machine Learning · Cross-listed from stat.ML
Simple Approximation and Derivative Free Inference-Time Scaling for Diffusion Models via Sequential Monte Carlo on Path Measures
Chenyang Wang, Weizhong Wang, Yinuo Ren et al. (+2 more)
iffusion-based generative models increasingly rely on inference-time guidance, adding a drift term or reweighting mixture of experts, to improve sample quality on task-specific objectives. However, most existing techniques require repeated score or gradient evaluations, introducing bias, high computational overhead, or both. We introduce URGE, Unbiased Resampling via Girsanov Estimation, a derivative-free inference-time scaling
algorithm that performs path-wise importance reweighting via a Girsanov change of measure. Instead of computing gradient-based particle weights in previous work, URGE attaches a simple multiplicative weight to each simulated trajectory and periodically resamples. No score, no Hessian, and no PDE evaluation is required. We establish an equivalence between path-wise and particle-wise SMC: the Girsanov path weight admits a backward conditional expectation that recovers the previous particle-level weights, guaranteeing that both schemes produce the same unbiased terminal law. Empirically, URGE outperforms existing inference-time guidance baselines on synthetic tests and
diffusion-model
benchmarks, achieving better generation quality, while being significantly simpler to implement and fully gradient-free.
Read the full abstract ▾
All authors Chenyang Wang, Weizhong Wang, Yinuo Ren, Jose Blanchet, Yiping Lu
Machine Learning · Cross-listed from cs.CL
Generating Pretraining Tokens from Organic Data for Data-Bound Scaling
Zichun Yu and Chenyan Xiong
LLM pretraining is shifting from a compute-bound to a data-bound regime, where available human (organic) text falls far short of scaling demands. However, reaching the data-bound regime does not mean the model has fully utilized its organic corpus. In this paper, we introduce SynPro, a synthetic data generation framework that helps LLMs more thoroughly learn from limited organic data. SynPro applies two operations, rephrasing and reformatting, that present the same organic source in diverse forms to facilitate deeper learning without introducing external information. Both generators are optimized via reinforcement learning with quality, faithfulness, and data influence rewards, and are continuously updated as pretraining plateaus to target content the model has yet to absorb. We pretrain 400M, 1.1B, and 2B models with 10% of their Chinchilla-optimal
tokens (0.8B, 2.2B, and 4B) from DCLM-Baseline, reflecting a realistic data-bound regime in frontier pretraining. Our results reveal that organic data is significantly underutilized by standard repetition: SynPro unlocks 3.4–5.2x the effective
tokens of repetition, even surpassing the non-data-bound oracle that trains on equivalent unique data at the 1.1B and 2B scales. Analyses confirm that faithful, model-aware synthesis sustains data-bound scaling without causing distribution collapse. We open-source our code at https://github.com/cxcscmu/SynPro.
Read the full abstract ▾
Machine Learning
SNLP: Layer-Parallel Inference via Structured Newton Corrections
Ligong Han, Kai Xu, Hao Wang et al. (+1 more)
Autoregressive
language models execute
Transformer layers sequentially, creating a latency bottleneck that is not removed by conventional tensor or pipeline parallelism. We study whether this layerwise dependency can be relaxed by treating the hidden-state trace across layers as the solution of a nonlinear residual equation and solving it with parallel Newton-style updates. While this view is principled, exact Newton corrections require expensive Jacobian-vector products and naive fixed-point iterations are unstable on trained
Transformers. We introduce Structured Newton Layer Parallelism (SNLP), a training and inference framework that replaces exact layer Jacobians with cheap architecture-induced surrogate dynamics. In residual
Transformers, this yields Identity Newton (IDN), where the correction reduces to a prefix-sum-like update; in mHC-style architectures, HC Newton (HCN) uses the model's residual mixing
matrix. We also study SNLP-aware training, including pretraining regularization and direct SNLP-forward SFT. Experiments on Nanochat-scale
Transformers show that SNLP exposes a practical speed-quality frontier: on 0.5B models, it reaches up to 2.58x wall-clock speedup, and a less aggressive configuration reaches 1.40x speedup without increasing PPL. The useful tradeoff comes from the biased finite-iteration computation induced by IDN/HCN rather than exact recovery of the sequential trace. We further show that SNLP-forward SFT can preserve downstream task accuracy, and that SNLP can serve as a drafter for self-speculative decoding while a sequential verifier preserves output correctness.
Read the full abstract ▾
All authors Ligong Han, Kai Xu, Hao Wang, Akash Srivastava
Machine Learning
Balancing Knowledge Distillation for Imbalance Learning with Bilevel Optimization
Anh B.H. Nguyen, Ba Tho Phan and Viet Cuong Ta
Knowledge
distillation transfers knowledge from a high capacity teacher to a compact student using a mixture of hard and soft losses. On imbalanced data, a fixed weighting between hard and soft losses becomes brittle the learning process. Recent studies try to reweight these components in long-tailed settings. However, most of these methods do not adapt weights at the sample-wise level and do not take into account the students behavior during training. To address this, we propose BiKD – a bilevel framework that dynamically balances hard and soft losses for each sample. We employ a weight generation network that produces adaptive per-sample weights, guided by a small balanced validation set. The student is now trained with an unconstrained combination of weighted hard and soft losses, allowing the student to relax both terms. We further propose a multi-step SGD strategy to optimize the weight model more accurately and efficiently. Experiments on long-tailed CIFAR-10/100 show that our approach surpasses recent balanced
distillation methods across imbalance factors.
Read the full abstract ▾
Machine Learning
Efficient Bilevel Optimization for Meta Label Correction in Noisy Label Learning
Ba Hoang Anh Nguyen and Viet Cuong Ta
Training a deep neural network with noisy labels could reduce data annotation cost but may introduce noise into the learned model. In meta label correction approaches, an additional meta model besides the main model is trained with a small, clean
dataset to correct the large, noisy
dataset. However, the update of the meta model requires the computation of hypergradients at the inner step of the main model which signif- icantly increases the computational cost. To improve the training efficiency, we first introduce the dynamic barrier gradient descent into standard meta label correction. While this naive extenstion is able to speed up the training process to approximately first- order complexity, it lacks mechanisms to prevent the leakage of noisy signals to the main model and to stabilize the learning of the meta model. Based on this observation, we propose the EBOMLC method, which is designed with three key improvements including one-step inner loop update, mixture upper loss and
alignment- aware dynamic barrier. Empirical results on CIFAR-10 and CIFAR-100 demonstrate that EBOMLC consistently outperforms other baselines, especially under high noise rate settings, while reducing training time of the meta label correction approach.
Read the full abstract ▾
Machine Learning
Agentic Cost-Aware Query Planning with Knowledge Distillation for Big Data Analytics
Mahdi Naser-Moghadasi
Query optimization in big data analytics remains computationally expensive, particularly for resource-constrained environments where traditional optimizers fail to satisfy
memory and latency constraints. We present an agentic query planning system that combines a rule-based teacher planner, UCB1 bandit exploration, cost-aware prediction, and knowledge
distillation to a lightweight student planner. Our teacher planner generates SQL plans using six key optimization strategies, while UCB1 bandit search efficiently explores the plan space under explicit resource constraints. A Random Forest cost model predicts query latency from plan features, enabling cost-aware decisions. A distilled student planner (Logistic Regression or Gradient Boosting) learns to mimic teacher-bandit decisions for fast inference. Evaluation on NYC Taxi and IMDB
datasets demonstrates 23% latency reduction compared to default planners while maintaining 94% constraint satisfaction. The student planner achieves 89% accuracy in replicating optimal plans with 15x faster inference time. Our single-file implementation enables reproducible big-data analytics on resource-limited machines and is publicly available at https://github.com/mahdinaser/agentic-kd-planner.
Read the full abstract ▾
Machine Learning
Content-Style Identification via Differential Independence
Subash Timilsina, Hoang-Son Nguyen, Sagar Shrestha et al. (+1 more)
Generative analysis often models multi-domain observations as nonlinear mixtures of domain-invariant content variables and domain-specific style variables. Identifying both factors from unpaired domains enables tasks such as domain transfer and counterfactual data generation. Prior work establishes identifiability under (block-wise) statistical independence between content and style, or via sparse Jacobian assumptions on the nonlinear mixing function, but such conditions can be restrictive in practice. In this work, we introduce content-style differential independence (CSDI), an alternative structural condition requiring that infinitesimal variations in content and style induce orthogonal directions on the data
manifold, thereby enabling identifiability even when content and style are dependent and the Jacobian is dense. We operationalize this condition through a blockwise orthogonality constraint on the Jacobian subspaces associated with content and style. To support high-dimensional generative models, we design a
stochastic regularizer based on numerical Jacobian approximation, enabling scalable training in settings such as high-resolution image generation. Experiments across multiple
datasets corroborate the identifiability analysis and demonstrate practical benefits on counterfactual generation and domain translation.
Read the full abstract ▾
All authors Subash Timilsina, Hoang-Son Nguyen, Sagar Shrestha, Xiao Fu
Machine Learning
One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
Jucheng Shen, Barbara Su and Anastasios Kyrillidis
Can a shared-weight recurrent
Transformer—one that repeatedly applies the same
Transformer to update evolving latent states—develop distinct internal roles without being partitioned into separate modules? We study this in Asymmetric Input Recurrence (AIR), a minimal two-state
reasoning architecture in which the same
Transformer model is reused for both updates (per literature, L and H) and the only built-in difference in the update rule is that the encoded input is injected during L-updates but not H-updates. Across Sudoku-Extreme and Maze, decoded rollouts reveal a consistent split: $z_H$ behaves like a fully committed proposal state, whereas $z_L$ retains local uncertainty and shifting intermediate structure. Freeze interventions provide evidence of dynamical dependence between the two states, with responses that vary across tasks, intervention directions, and seeds. Ablations show that to induce specialization, the shared model needs to be able to tell the two update types apart, either from input injection asymmetry or from a separate level
token. Mechanistically,
attention analysis shows that L-updates are consistently more local than H-updates in both Sudoku and Maze. Together, these results show that, in a two-state recurrent setting, a clear state-identity signal can induce stable, related functional roles inside a shared-parameter recurrent
Transformer. Code is available at https://github.com/juchengshen/air.
Read the full abstract ▾
Machine Learning
A Unified Framework for Data-Free One-Step Sampling via Wasserstein Gradient Flows
Chenguang Wang and Tianshu Yu
We develop a unified theoretical framework for data-free one-step sampling from unnormalized target distributions based on Wasserstein gradient flows. For a broad class of standard f-divergence objectives, we show that the induced velocity field admits the universal form $\mathbf{V}(x)=w(r(x))\,\beta(x)$, where $\beta(x)=\nabla \log (p(x)/q(x))$ is shared across objectives and $w$ is determined solely by the choice of divergence. This decomposition shows that standard f-divergence drifts share the same asymptotic target distribution $p$ and differ primarily in how they redistribute transient repair effort across under-covered regions. To formalize this distinction, we derive a one-step regional-response theory for a soft under-coverage functional and obtain a compression–elasticity identity that links divergence choice to the geometry of mass transport into under-covered regions. We further extend the framework beyond the f-divergence family to the Log-Variance (LV) divergence, analyze how the reference distribution alters the resulting drift structure, and motivate a practical LV-inspired surrogate for data-free training. Based on this theory, we instantiate the framework with a KDE-based implementation and describe a complementary normalizing-flow route, enabling one-step inference after training. Experiments on
multimodal Gaussian-mixture
benchmarks are consistent with the theoretical predictions and demonstrate effective one-step sampling on these targets.
Read the full abstract ▾
Machine Learning
AMO: Operator-level Adaptive Muon Orthogonalization
Xinlin Zhuang, Panyi Ouyang, Yichen Li et al. (+7 more)
Muon has recently emerged as a competitive alternative to AdamW for large-scale pre-training, with orthogonalization via Newton-Schulz (NS) iteration as its core operation. Standard
Muon applies a uniform NS schedule to all parameter
matrices, overlooking possible differences in orthogonalization difficulty and its impact on performance. Through a systematic empirical study, we show that this per-
matrix heterogeneity is pervasive and strongly associated with
matrix geometry, which evolves dynamically across operator types, training stages, and network depths. Therefore, uniform NS schedules can lead to uneven orthogonalization quality across the model. Motivated by these findings, we propose Operator-level Adaptive
Muon Orthogonalization (AMO), an observe-then-commit method that measures weight geometry by operator type early in training and then uses these signals to allocate the NS budget for the remainder of training. AMO delivers consistent improvements over uniform-schedule
Muon across standard, prolonged, and continual pre-training, surpassing the strongest baseline by +0.76 on Llama3.1-1.4B and +0.51 on Qwen3-1.7B in average downstream performance of 12 evaluation tasks, with gains persisting at Llama3.1-4B scale.
Read the full abstract ▾
All authors Xinlin Zhuang, Panyi Ouyang, Yichen Li, Jiangming Shi, Yizhang Chen, Shuman Liu, Ying Qian, Weiyang Liu, Haibo Zhang, Imran Razzak
Machine Learning
GenTS: A Comprehensive Benchmark Library for Generative Time Series Models
Chenxi Wang, Xiaorong Wang, Peiyang Li et al. (+1 more)
Generative models have demonstrated remarkable potential in time series analysis tasks, like synthesis,
forecasting, imputation, etc. However, offering limited coverage for generative models, existing time series libraries are mainly engineered for discriminative models, with standardized workflows for specific tasks, such as optimizing Mean Squared Errors for time series
forecasting. This rigid structure is fundamentally incompatible with the distinct and often complex paradigms of generative models (e.g.,
adversarial training,
diffusion processes), which learn the underlying data distribution rather than a direct input-output mapping. To this end, we proposed GenTS, a comprehensive and extensible
benchmark library designed for systematic assessment on generative time series models. GenTS features a unified data preprocessing pipeline, a collection of versatile models, and panoramic evaluation metrics. Its modular design also enables the researchers to flexibly customize beyond our built-in
datasets and models. Based on GenTS, we conducted benchmarking experiments under diverse tasks, accordingly offering suggestions for model selection and identifying potential directions for future research. Our codes are open-source at https://github.com/WillWang1113/GenTS. The official tutorials and document are available at https://willwang1113.github.io/GenTS/.
Read the full abstract ▾
All authors Chenxi Wang, Xiaorong Wang, Peiyang Li, Yi Wang