Computation and Language · Cross-listed from cs.AI
GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer
Junmo Cho, Suhan Kim, Sangjune An et al. (+5 more)
Finding effective
prompts for
language models (LMs) is critical yet notoriously difficult: the
prompt space is combinatorially large, rewards are sparse due to expensive target-LM evaluation. Yet, existing RL-based
prompt optimizers often rely on on-policy updates and a meta-
prompt sampled from a fixed distribution, leading to poor sample efficiency. We propose GFlowPO, a probabilistic
prompt optimization framework that casts
prompt search as a posterior inference problem over latent
prompts regularized by a meta-prompted reference-LM prior. In the first step, we fine-tune a lightweight
prompt-LM with an off-policy Generative Flow Network (GFlowNet) objective, using a replay-based training policy that reuses past
prompt evaluations to enable sample-efficient exploration. In the second step, we introduce Dynamic
Memory Update (DMU), a training-free mechanism that updates the meta-
prompt by injecting both (i) diverse
prompts from a replay buffer and (ii) top-performing
prompts from a small priority queue, thereby progressively concentrating the search process on high-reward regions. Across few-shot text classification, instruction induction
benchmarks, and question answering tasks, GFlowPO consistently outperforms recent discrete
prompt optimization baselines.
Read the full abstract ▾
All authors Junmo Cho, Suhan Kim, Sangjune An, Minsu Kim, Dong Bok Lee, Heejun Lee, Sung Ju Hwang, Hae Beom Lee
Computation and Language
PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning
Yunzhi Shen, Hao Zhou, Xin Huang et al. (+3 more)
Reinforcement learning (RL) has shown strong promise for LLM-based machine translation, with recent methods such as GRPO demonstrating notable gains; nevertheless, translation-oriented RL remains challenged by noisy learning signals arising from Monte Carlo return estimation, as well as a large trajectory space that favors global exploration over fine-grained local optimization. We introduce PEGRL, a two-stage RL framework that uses post-editing as an auxiliary task to stabilize training and guide overall optimization. At each iteration, translation outputs are sampled to construct post-editing inputs, allowing return estimation in the post-editing stage to benefit from conditioning on the current translation behavior, while jointly supporting both global exploration and fine-grained local optimization. A task-specific weighting scheme further balances the contributions of translation and post-editing objectives, yielding a biased yet more sample-efficient
estimator. Experiments on English$\to$Finnish, English$\to$Turkish, and English$\leftrightarrow$Chinese show consistent gains over RL baselines, and for English$\to$Turkish, performance on COMET-KIWI is comparable to advanced LLM-based systems (DeepSeek-V3.2). Our code and a set of representative pretrained models are publicly available at \url{https://github.com/NJUNLP/peg-rl} and \url{https://huggingface.co/collections/DGME/pegrl}
Read the full abstract ▾
All authors Yunzhi Shen, Hao Zhou, Xin Huang, Xue Han, Junlan Feng, Shujian Huang
Computation and Language · Cross-listed from cs.LG
Robustness as an Emergent Property of Task Performance
Shir Ashury-Tahan, Ariel Gera, Elron Bandel et al. (+2 more)
Robustness is widely viewed as a key challenge for real-world applications. However, because current research focuses only on difficult tasks, it partially captures real-world readiness. In this paper, we argue and verify that robustness, defined as consistency across semantically equivalent inputs, closely follows task difficulty: once models master a task, robustness emerges naturally. Through an empirical analysis of multiple models across diverse
datasets and configurations (e.g., paraphrases, temperature changes), we observe a strong positive correlation between task performance and robustness. Furthermore, our findings indicate that robustness is driven primarily by task-specific competence rather than inherent model attributes, challenging the common view of robustness as an independent capability. This perspective implies that as tasks mature and model performance saturates, robustness on those tasks will similarly emerge. For researchers, this suggests that explicit efforts to measure robustness may deserve reduced emphasis, as robustness is likely to improve alongside performance. For practitioners, it signals that while many existing
benchmarks are still unstable, models are already reliable on earlier tasks and suitable for deployment.
Read the full abstract ▾
All authors Shir Ashury-Tahan, Ariel Gera, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen
Computation and Language
Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
Rakshith Vasudev, Melisa Russak, Dan Bikel et al. (+1 more)
Proactive interventions by LLM critic models are often assumed to improve reliability, yet their effects at deployment time are poorly understood. We show that a binary LLM critic with strong offline accuracy (AUROC 0.94) can nevertheless cause severe performance degradation, inducing a 26 percentage point (pp) collapse on one model while affecting another by near zero pp. This variability demonstrates that LLM critic accuracy alone is insufficient to determine whether intervention is safe. We identify a disruption-recovery tradeoff: interventions may recover failing trajectories but also disrupt trajectories that would have succeeded. Based on this insight, we propose a pre-deployment test that uses a small pilot of 50 tasks to estimate whether intervention is likely to help or harm, without requiring full deployment. Across
benchmarks, the test correctly anticipates outcomes: intervention degrades performance on high-success tasks (0 to -26 pp), while yielding a modest improvement on the high-failure ALFWorld
benchmark (+2.8 pp, p=0.014). The primary value of our framework is therefore identifying when not to intervene, preventing severe regressions before deployment.
Read the full abstract ▾
All authors Rakshith Vasudev, Melisa Russak, Dan Bikel, Waseem Alshikh
Computation and Language
MIRROR: A Multi-Agent Framework with Iterative Adaptive Revision and Hierarchical Retrieval for Optimization Modeling in Operations Research
Yifan Shi, Jiayi Wang, Minyi Wu et al. (+3 more)
Operations Research (OR) relies on expert-driven modeling–a slow and fragile process ill-suited to novel scenarios. While large
language models (LLMs) can automatically translate natural
language into optimization models, existing approaches either rely on costly post-training or employ multi-
agent frameworks, yet most still lack reliable collaborative error correction and task-specific
retrieval, often leading to incorrect outputs. We propose MIRROR (a Multi-
agent framework with Iterative adaptive Revision and hierarchical
Retrieval for optimization modeling in Operations Research), a fine-tuning-free, end-to-end multi-
agent framework that directly translates natural
language optimization problems into mathematical models and solver code. MIRROR integrates two core mechanisms: (1) execution-driven iterative adaptive revision for automatic error correction, and (2) hierarchical
retrieval to fetch relevant modeling and coding exemplars from a carefully curated exemplar library. Experiments show that MIRROR outperforms existing methods on standard OR
benchmarks, with notable results on complex industrial
datasets such as "IndustryOR" and "Mamo-ComplexLP". By combining precise external knowledge infusion with systematic error correction, MIRROR provides non-expert users with an efficient and reliable OR modeling solution, overcoming the fundamental limitations of general-purpose LLMs in expert optimization tasks.
Read the full abstract ▾
All authors Yifan Shi, Jiayi Wang, Minyi Wu, Ye Fan, Jialong Shi, Jianyong Sun
Computation and Language · Cross-listed from cs.LG
R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?
Jingyi Zhang, Tianyi Lin, Huanjin Yao et al. (+3 more)
In this work, we aim to develop effective data synthesis techniques that autonomously synthesize
multimodal training data for enhancing MLLMs in solving complex real-world tasks. To this end, we propose Collective
Adversarial Data Synthesis (CADS), a novel and general approach to synthesize high-quality, diverse and challenging
multimodal data for MLLMs. The core idea of CADS is to leverage collective intelligence to ensure high-quality and diverse generation, while exploring
adversarial learning to synthesize challenging samples for effectively driving model improvement. Specifically, CADS operates with two cyclic phases, i.e., Collective
Adversarial Data Generation (CAD-Generate) and Collective
Adversarial Data Judgment (CAD-Judge). CAD-Generate leverages collective knowledge to jointly generate new and diverse
multimodal data, while CAD-Judge collaboratively assesses the quality of synthesized data. In addition, CADS introduces an
Adversarial Context Optimization mechanism to optimize the generation context to encourage challenging and high-value data generation. With CADS, we construct MMSynthetic-20K and train our model R1-SyntheticVL, which demonstrates superior performance on various
benchmarks.
Read the full abstract ▾
All authors Jingyi Zhang, Tianyi Lin, Huanjin Yao, Xiang Lan, Shunyu Liu, Jiaxing Huang
Computation and Language
POP: Prefill-Only Pruning for Efficient Large Model Inference
Junhui He, Zhihui Fu, Jun Wang et al. (+1 more)
Large
Language Models (LLMs) and
Vision-
Language Models (VLMs) have demonstrated remarkable capabilities. However, their deployment is hindered by significant computational costs. Existing structured
pruning methods, while hardware-efficient, often suffer from significant accuracy degradation. In this paper, we argue that this failure stems from a stage-agnostic
pruning approach that overlooks the asymmetric roles between the prefill and decode stages. By introducing a virtual gate mechanism, our importance analysis reveals that deep layers are critical for next-
token prediction (decode) but largely redundant for context encoding (prefill). Leveraging this insight, we propose Prefill-Only
Pruning (POP), a stage-aware inference strategy that safely omits deep layers during the computationally intensive prefill stage while retaining the full model for the sensitive decode stage. To enable the transition between stages, we introduce independent Key-Value (KV) projections to maintain cache integrity, and a boundary handling strategy to ensure the accuracy of the first generated
token. Extensive experiments on Llama-3.1, Qwen3-VL, and Gemma-3 across diverse modalities demonstrate that POP achieves up to 1.37$\times$ speedup in prefill latency with minimal performance loss, effectively overcoming the accuracy-efficiency trade-off limitations of existing structured
pruning methods.
Read the full abstract ▾
All authors Junhui He, Zhihui Fu, Jun Wang, Qingan Li
Computation and Language · Cross-listed from eess.AS
Mići Princ – A Little Boy Teaching Speech Technologies the Chakavian Dialect
Nikola Ljubešić, Peter Rupnik and Tea Perinčić
This paper documents our efforts in releasing the printed and audio book of the translation of the famous novel The Little Prince into the Chakavian dialect, as a computer-readable, AI-ready
dataset, with the textual and the audio components of the two releases now aligned on the level of each written and spoken word. Our motivation for working on this release is multiple. The first one is our wish to preserve the highly valuable and specific content beyond the small editions of the printed and the audio book. With the
dataset published in the CLARIN.SI repository, this content is from now on at the fingertips of any interested individual. The second motivation is to make the data available for various artificial-intelligence-related usage scenarios, such as the one we follow upon inside this paper already – adapting the Whisper-large-v3 open automatic
speech recognition model, with decent performance on standard Croatian, to Chakavian dialectal
speech. We can happily report that with adapting the model, the word error rate on the selected test data has being reduced to a half, while we managed to remove up to two thirds of the error on character level. We envision many more usages of this
dataset beyond the set of experiments we have already performed, both on tasks of artificial intelligence research and application, as well as dialectal research. The third motivation for this release is our hope that this, now highly structured
dataset, will be transformed into a digital online edition of this work, allowing individuals beyond the research and technology communities to enjoy the beauty of the message of the little boy in the desert, told through the spectacular prism of the Chakavian dialect.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations
Yuxuan Yao, Haonan Sheng, Qingsong Lv et al. (+11 more)
The escalating scale of Large
Language Models (LLMs) necessitates efficient adaptation techniques. Model merging has gained prominence for its efficiency and controllability. However, existing merging techniques typically serve as post-hoc refinements or focus on mitigating task interference, often failing to capture the dynamic optimization benefits of supervised fine-tuning (SFT). In this work, we propose Streaming Merging, an innovative model updating paradigm that conceptualizes merging as an iterative optimization process. Central to this paradigm is ARM (Activation-guided Rotation-aware Merging), a strategy designed to approximate gradient descent dynamics. By treating merging coefficients as learning rates and deriving rotation vectors from activation subspaces, ARM effectively steers parameter updates along data-driven trajectories. Unlike conventional linear interpolation, ARM aligns semantic subspaces to preserve the
geometric structure of high-dimensional parameter
evolution. Remarkably, ARM requires only early SFT checkpoints and, through iterative merging, surpasses the fully converged SFT model. Experimental results across model scales (1.7B to 14B) and diverse domains (e.g., math, code) demonstrate that ARM can transcend converged checkpoints. Extensive experiments show that ARM provides a scalable and lightweight framework for efficient model adaptation.
Read the full abstract ▾
All authors Yuxuan Yao, Haonan Sheng, Qingsong Lv, Han Wu, Shuqi Liu, Zehua Liu, Zengyan Liu, Jiahui Gao, Haochen Tan, Xiaojin Fu, Haoli Bai, Hing Cheung So, Zhijiang Guo, Linqi Song
Computation and Language
ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs
Xuancheng Li, Haitao Li, Yujia Zhou et al. (+2 more)
Long-context inputs in large
language models (LLMs) often suffer from the "lost in the middle" problem, where critical information becomes diluted or ignored due to excessive length. Context compression methods aim to address this by reducing input size, but existing approaches struggle with balancing information preservation and compression efficiency. We propose Adaptive Task-Aware Compressor (ATACompressor), which dynamically adjusts compression based on the specific requirements of the task. ATACompressor employs a selective encoder that compresses only the task-relevant portions of long contexts, ensuring that essential information is preserved while reducing unnecessary content. Its adaptive allocation controller perceives the length of relevant content and adjusts the compression rate accordingly, optimizing resource utilization. We evaluate ATACompressor on three QA
datasets: HotpotQA, MSMARCO, and SQUAD-showing that it outperforms existing methods in terms of both compression efficiency and task performance. Our approach provides a scalable solution for long-context processing in LLMs. Furthermore, we perform a range of ablation studies and analysis experiments to gain deeper insights into the key components of ATACompressor.
Read the full abstract ▾
All authors Xuancheng Li, Haitao Li, Yujia Zhou, Qingyao Ai, Yiqun Liu
Computation and Language
Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
Dongwon Jo, Beomseok Kang, Jiwon Song et al. (+1 more)
The quadratic complexity of
attention remains the central bottleneck in long-context inference for large
language models. Prior acceleration methods either sparsify the
attention map with structured patterns or permanently evict
tokens at specific layers, which can retain irrelevant
tokens or rely on irreversible early decisions despite the layer-/head-wise dynamics of
token importance. In this paper, we propose
Token Sparse
Attention, a lightweight and dynamic
token-level sparsification mechanism that compresses per-head $Q$, $K$, $V$ to a reduced
token set during
attention and then decompresses the output back to the original sequence, enabling
token information to be reconsidered in subsequent layers. Furthermore,
Token Sparse
Attention exposes a new design point at the intersection of
token selection and sparse
attention. Our approach is fully compatible with dense
attention implementations, including Flash
Attention, and can be seamlessly composed with existing sparse
attention kernels. Experimental results show that
Token Sparse
Attention consistently improves accuracy-latency trade-off, achieving up to $\times$3.23
attention speedup at 128K context with less than 1% accuracy degradation. These results demonstrate that dynamic and interleaved
token-level sparsification is a complementary and effective strategy for scalable long-context inference.
Read the full abstract ▾
All authors Dongwon Jo, Beomseok Kang, Jiwon Song, Jae-Joon Kim
Computation and Language
ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution
Zican Dong, Peiyu Liu, Junyi Li et al. (+4 more)
Recently, large
language models (LLMs) have shown remarkable
reasoning abilities by producing long
reasoning traces. However, as the sequence length grows, the key-value (KV) cache expands linearly, incurring significant
memory and computation costs. Existing KV cache eviction methods mitigate this issue by discarding less important KV pairs, but often fail to capture complex KV dependencies, resulting in performance degradation. To better balance efficiency and performance, we introduce ForesightKV, a training-based KV cache eviction framework that learns to predict which KV pairs to evict during long-text generations. We first design the Golden Eviction
algorithm, which identifies the optimal eviction KV pairs at each step using future
attention scores. These traces and the scores at each step are then distilled via supervised training with a Pairwise Ranking Loss. Furthermore, we formulate cache eviction as a Markov Decision Process and apply the GRPO
algorithm to mitigate the significant
language modeling loss increase on low-
entropy tokens. Experiments on AIME2024 and AIME2025
benchmarks of three
reasoning models demonstrate that ForesightKV consistently outperforms prior methods under only half the cache budget, while benefiting synergistically from both supervised and reinforcement learning approaches. Code is available at https://github.com/RUCAIBox/ForesightKV.
Read the full abstract ▾
All authors Zican Dong, Peiyu Liu, Junyi Li, Zhipeng Chen, Han Peng, Shuo Wang, Wayne Xin Zhao
Computation and Language · Cross-listed from cs.LG
PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning
Wenquan Lu, Hai Huang, Enqi Liu et al. (+1 more)
Reinforcement learning
algorithms such as group-relative policy optimization (GRPO) have shown strong potential for improving the mathematical
reasoning capabilities of large
language models. While a growing body of work seeks to improve training
entropy, rollout diversity, and exploration, most existing methods still train models with a single fixed
reasoning prompt or template, which can encourage
prompt-specific overfitting and unstable training dynamics. In this work, we introduce
Prompt Augmented Policy Optimization (PrAg-PO), a simple policy optimization method that mixes
prompt templates with template-specific format rewards during training. By encouraging models to generate
reasoning traces under diverse instructions and output formats, PrAg-PO increases rollout diversity and improves robustness. Compared with GRPO and DAPO, PrAg-PO achieves significantly higher
reasoning accuracy while mitigating premature training collapse. Empirically, experiments on DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-Math-1.5B, and Qwen3-1.7B show that PrAg-PO consistently outperforms strong baselines and achieves competitive performance against recent methods on mathematics
benchmarks, using only a fixed MATH Level 3-5 training set of 8.5K problems. The code and model checkpoints are available at https://github.com/wenquanlu/PrAg-PO.
Read the full abstract ▾
All authors Wenquan Lu, Hai Huang, Enqi Liu, Randall Balestriero
Computation and Language · Cross-listed from cs.LG
DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inference
Jiancai Ye, Jun Liu, Qingchen Li et al. (+5 more)
Although Key-Value (KV) Cache is essential for efficient large
language models (LLMs) inference, its growing
memory footprint in long-context scenarios poses a significant bottleneck, making KVCache compression crucial. Current compression methods rely on rigid splitting strategies, such as fixed intervals or pre-defined delimiters. We observe that rigid splitting suffers from significant accuracy degradation (ranging from 5.5% to 55.1%) across different scenarios, owing to the scenario-dependent nature of the semantic boundaries. This highlights the necessity of dynamic semantic splitting to match semantics. To achieve this, we face two challenges. (1) Improper delimiter selection misaligns semantics with the KVCache, resulting in 28.6% accuracy loss. (2) Variable-length blocks after splitting introduce over 73.1% additional inference overhead. To address the above challenges, we propose DynSplit-KV, a KVCache compression method that dynamically identifies delimiters for splitting. We propose: (1) a dynamic importance-aware delimiter selection strategy, improving accuracy by 49.9%. (2) A uniform mapping strategy that transforms variable-length semantic blocks into a fixed-length format, reducing inference overhead by 4.9x. Experiments show that DynSplit-KV achieves the highest accuracy, 2.2x speedup compared with FlashAttention and 2.6x peak
memory reduction in long-context scenarios.
Read the full abstract ▾
All authors Jiancai Ye, Jun Liu, Qingchen Li, Tianlang Zhao, Hanbin Zhang, Jiayi Pan, Ningyi Xu, Guohao Dai
Computation and Language
Privasis: Synthesizing the Largest "Public" Private Dataset from Scratch
Hyunwoo Kim, Niloofar Mireshghallah, Michael Duan et al. (+11 more)
Research involving
privacy-sensitive data has always been constrained by data scarcity, standing in sharp contrast to other areas that have benefited from data scaling. This challenge is becoming increasingly urgent as modern AI
agents–such as OpenClaw and Gemini
Agent–are granted persistent access to highly sensitive personal information. To tackle this longstanding bottleneck and the rising risks, we present Privasis (i.e.,
privacy oasis), the first million-scale fully synthetic
dataset entirely built from scratch–an expansive reservoir of texts with rich and diverse private information–designed to broaden and accelerate research in areas where processing sensitive social data is inevitable. Compared to existing
datasets, Privasis, comprising 1.4 million records, offers orders-of-magnitude larger scale with quality, and far greater diversity across various document types, including medical history, legal documents, financial records, calendars, and text messages with a total of 55.1 million annotated attributes such as ethnicity, date of birth, workplace, etc. We leverage Privasis to construct a parallel corpus for text sanitization with our pipeline that decomposes texts and applies targeted sanitization. Our compact sanitization models (<=4B) trained on this
dataset outperform state-of-the-art large
language models, such as GPT-5 and Qwen-3 235B. We plan to release data, models, and code to accelerate future research on
privacy-sensitive domains and
agents.
Read the full abstract ▾
All authors Hyunwoo Kim, Niloofar Mireshghallah, Michael Duan, Rui Xin, Shuyue Stella Li, Jaehun Jung, David Acuna, Qi Pang, Hanshen Xiao, G. Edward Suh, Sewoong Oh, Yulia Tsvetkov, Pang Wei Koh, Yejin Choi
Computation and Language · Cross-listed from cs.AI
VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models
Woojin Kim, Sieun Hyeon, Jusang Oh et al. (+1 more)
Aligning Large
Language Models (LLMs) with the diverse spectrum of human values remains a central challenge: preference-based methods often fail to capture deeper motivational principles. Value-based approaches offer a more principled path, yet three gaps persist: extraction often ignores hierarchical structure, evaluation detects presence but not calibrated intensity, and the steerability of LLMs at controlled intensities remains insufficiently understood. To address these limitations, we introduce VALUEFLOW, the first unified framework that spans extraction, evaluation, and steering with calibrated intensity control. The framework integrates three components: (i) HIVES, a hierarchical value
embedding space that captures intra- and cross-theory value structure; (ii) the Value Intensity DataBase (VIDB), a large-scale resource of value-labeled texts with intensity estimates derived from ranking-based aggregation; and (iii) an anchor-based evaluator that produces consistent intensity scores for model outputs by ranking them against VIDB panels. Using VALUEFLOW, we conduct a comprehensive large-scale study across ten models and four value theories, identifying asymmetries in steerability and composition laws for multi-value control. This paper establishes a scalable infrastructure for evaluating and controlling value intensity, advancing pluralistic
alignment of LLMs.
Read the full abstract ▾
All authors Woojin Kim, Sieun Hyeon, Jusang Oh, Jaeyoung Do
Computation and Language
FASA: Frequency-aware Sparse Attention
Yifei Wang, Yueqi Wang, Zhenrui Yue et al. (+6 more)
The deployment of Large
Language Models (LLMs) faces a critical bottleneck when handling lengthy inputs: the prohibitive
memory footprint of the Key Value (KV) cache. To address this bottleneck, the
token pruning paradigm leverages
attention sparsity to selectively retain a small, critical subset of
tokens. However, existing approaches fall short, with static methods risking irreversible information loss and dynamic strategies employing heuristics that insufficiently capture the query-dependent nature of
token importance. We propose FASA, a novel framework that achieves query-aware
token eviction by dynamically predicting
token importance. FASA stems from a novel insight into RoPE: the discovery of functional sparsity at the frequency-chunk (FC) level. Our key finding is that a small, identifiable subset of "dominant" FCs consistently exhibits high contextual agreement with the full
attention head. This provides a robust and computationally free proxy for identifying salient
tokens. Building on this insight, FASA first identifies a critical set of
tokens using dominant FCs, and then performs focused
attention computation solely on this pruned subset. Across a spectrum of long-context tasks, from sequence modeling to complex CoT
reasoning, FASA consistently outperforms all
token-eviction baselines and achieves near-oracle accuracy, demonstrating remarkable robustness even under constraint budgets. Notably, on LongBench-V1, FASA reaches nearly 100% of full-KV performance when only keeping 256
tokens, and achieves 2.56$\times$ speedup using just 18.9% of the cache on AIME24.
Read the full abstract ▾
All authors Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, Julian McAuley
Computation and Language · Cross-listed from cs.LG
Self-Hinting Language Models Enhance Reinforcement Learning
Baohao Liao, Hanze Dong, Xinxing Xu et al. (+2 more)
Group Relative Policy Optimization (GRPO) has recently emerged as a practical recipe for aligning large
language models with verifiable objectives. However, under sparse terminal rewards, GRPO often stalls because rollouts within a group frequently receive identical rewards, causing relative advantages to collapse and updates to vanish. We propose self-hint aligned GRPO with privileged supervision (SAGE), an on-policy reinforcement learning framework that injects privileged hints during training to reshape the rollout distribution under the same terminal verifier reward. For each
prompt $x$, the model samples a compact hint $h$ (e.g., a plan or decomposition) and then generates a solution $\tau$ conditioned on $(x,h)$. Crucially, the task reward $R(x,\tau)$ is unchanged; hints only increase within-group outcome diversity under finite sampling, preventing GRPO advantages from collapsing under sparse rewards. At test time, we set $h=\varnothing$ and deploy the no-hint policy without any privileged information. Moreover, sampling diverse self-hints serves as an adaptive curriculum that tracks the learner's bottlenecks more effectively than fixed hints from an initial policy or a stronger external model. Experiments over 6
benchmarks with 3 LLMs show that SAGE consistently outperforms GRPO, on average +2.0 on Llama-3.2-3B-Instruct, +1.2 on Qwen2.5-7B-Instruct and +1.3 on Qwen3-4B-Instruct. The code is available at https://github.com/BaohaoLiao/SAGE.
Read the full abstract ▾
All authors Baohao Liao, Hanze Dong, Xinxing Xu, Christof Monz, Jiang Bian
Computation and Language
Short Chains, Deep Thoughts: Balancing Reasoning Efficiency and Intra-Segment Capability via Split-Merge Optimization
Runquan Gui, Jie Wang, Zhihai Wang et al. (+3 more)
While Large
Reasoning Models (LRMs) have demonstrated impressive capabilities in solving complex tasks through the generation of long
reasoning chains, this reliance on verbose generation results in significant latency and computational overhead. To address these challenges, we propose CoSMo (Consistency-Guided Split-Merge Optimization), a framework designed to eliminate structural redundancy rather than indiscriminately restricting
token volume. Specifically, CoSMo utilizes a split-merge
algorithm that dynamically refines
reasoning chains by merging redundant segments and splitting logical gaps to ensure coherence. We then employ structure-aligned reinforcement learning with a novel segment-level budget to supervise the model in maintaining efficient
reasoning structures throughout training. Extensive experiments across multiple
benchmarks and backbones demonstrate that CoSMo achieves superior performance, improving accuracy by 3.3 points while reducing segment usage by 28.7% on average compared to
reasoning efficiency baselines.
Read the full abstract ▾
All authors Runquan Gui, Jie Wang, Zhihai Wang, Chi Ma, Jianye Hao, Feng Wu
Computation and Language
One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence
Bowen Jiang, Taiwei Shi, Ryo Kamoi et al. (+5 more)
This paper introduces OMAR: One Model, All Roles, a reinforcement learning framework that enables AI to develop social intelligence through multi-turn, multi-
agent conversational self-play. Unlike traditional paradigms that rely on static, single-turn optimizations, OMAR allows a single model to role-play all participants in a conversation simultaneously, learning to achieve long-term goals and complex social norms directly from dynamic social interaction. To ensure training stability across long dialogues, we implement a hierarchical advantage estimation that calculates turn-level and
token-level advantages. Evaluations in the SOTOPIA social environment and Werewolf strategy games show that our trained models develop fine-grained, emergent social intelligence, such as empathy, persuasion, and compromise seeking, demonstrating the effectiveness of learning collaboration even under competitive scenarios. While we identify practical challenges like reward hacking, our results show that rich social intelligence can emerge without human supervision. We hope this work incentivizes further research on AI social intelligence in group conversations.
Read the full abstract ▾
All authors Bowen Jiang, Taiwei Shi, Ryo Kamoi, Yuan Yuan, Camillo J. Taylor, Longqi Yang, Pei Zhou, Sihao Chen
Computation and Language
ChemPro: A Progressive Chemistry Benchmark for Large Language Models
Aaditya Baranwal and Shruti Vyas
We introduce ChemPro, a progressive
benchmark with 4100 natural
language question-answer pairs in Chemistry, across 4 coherent sections of difficulty designed to assess the proficiency of Large
Language Models (LLMs) in a broad spectrum of general chemistry topics. We include Multiple Choice Questions and Numerical Questions spread across fine-grained information recall, long-horizon
reasoning, multi-concept questions, problem-solving with nuanced articulation, and straightforward questions in a balanced ratio, effectively covering Bio-Chemistry, Inorganic-Chemistry, Organic-Chemistry and Physical-Chemistry. ChemPro is carefully designed analogous to a student's academic evaluation for basic to high-school chemistry. A gradual increase in the question difficulty rigorously tests the ability of LLMs to progress from solving basic problems to solving more sophisticated challenges. We evaluate 45+7 state-of-the-art LLMs, spanning both open-source and proprietary variants, and our analysis reveals that while LLMs perform well on basic chemistry questions, their accuracy declines with different types and levels of complexity. These findings highlight the critical limitations of LLMs in general scientific
reasoning and understanding and point towards understudied dimensions of difficulty, emphasizing the need for more robust methodologies to improve LLMs.
Read the full abstract ▾
Computation and Language
The Mask of Civility: Benchmarking Chinese Mock Politeness Comprehension in Large Language Models
Yitong Zhang, Yuhan Xiang and Mingxuan Liu
From a pragmatic perspective, this study systematically evaluates the differences in performance among representative large
language models (LLMs) in recognizing politeness, impoliteness, and mock politeness phenomena in Chinese. Addressing the existing gaps in pragmatic comprehension, the research adopts the frameworks of Rapport Management Theory and the Model of Mock Politeness to construct a three-category
dataset combining authentic and simulated Chinese discourse. Six representative models, including GPT-5.1 and DeepSeek, were selected as test subjects and evaluated under four prompting conditions: zero-shot, few-shot, knowledge-enhanced, and hybrid strategies. This study serves as a meaningful attempt within the paradigm of “Great Linguistics,” offering a novel approach to applying pragmatic theory in the age of technological transformation. It also responds to the contemporary question of how technology and the humanities may coexist, representing an interdisciplinary endeavor that bridges linguistic technology and humanistic reflection.
Read the full abstract ▾
Computation and Language
Task–Specificity Score: Measuring How Much Instructions Really Matter for Supervision
Pritam Kadasi, Abhishek Upperwal and Mayank Singh
Instruction tuning is now the default way to train and adapt large
language models, but many instruction–input–output pairs are only weakly specified: for a given input, the same output can remain plausible under several alternative instructions. This raises a simple question: does the instruction uniquely determine the target output? We propose the Task–Specificity Score (TSS) to quantify how much an instruction matters for predicting its output, by contrasting the true instruction against plausible alternatives for the same input. We further introduce TSS++, which uses hard alternatives and a small quality term to mitigate easy-negative effects. Across three instruction
datasets (Alpaca, Dolly-15k, NI-20) and three open LLMs (Gemma, Llama, Qwen), we show that selecting task-specific examples improves downstream performance under tight
token budgets and complements quality-based filters such as perplexity and IFD.
Read the full abstract ▾
Computation and Language
Test-time Recursive Thinking: Self-Improvement without External Feedback
Yufan Zhuang, Chandan Singh, Liyuan Liu et al. (+5 more)
Modern Large
Language Models (LLMs) have shown rapid improvements in
reasoning capabilities, driven largely by reinforcement learning (RL) with verifiable rewards. Here, we ask whether these LLMs can self-improve without the need for additional training. We identify two core challenges for such systems: (i) efficiently generating diverse, high-quality candidate solutions, and (ii) reliably selecting correct answers in the absence of ground-truth supervision. To address these challenges, we propose Test-time Recursive Thinking (TRT), an iterative self-improvement framework that conditions generation on rollout-specific strategies, accumulated knowledge, and self-generated verification signals. Using TRT, open-source models reach 100% accuracy on AIME-25/24, and on LiveCodeBench's most difficult problems, closed-source models improve by 10.4-14.8 percentage points without external feedback.
Read the full abstract ▾
All authors Yufan Zhuang, Chandan Singh, Liyuan Liu, Yelong Shen, Dinghuai Zhang, Jingbo Shang, Jianfeng Gao, Weizhu Chen
Computation and Language
AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedback
Zhitao Gao, Jie Ma, Xuhong Li et al. (+5 more)
Large
Language Models (LLMs) have achieved significant success in complex
reasoning but remain bottlenecked by reliance on expert-annotated data and external verifiers. While existing self-
evolution paradigms aim to bypass these constraints, they often fail to identify the optimal learning zone and risk reinforcing collective
hallucinations and incorrect priors through flawed internal feedback. To address these challenges, we propose \underline{A}utonomous \underline{E}volutionary \underline{R}easoning \underline{O}ptimization (AERO), an unsupervised framework that achieves autonomous
reasoning evolution by internalizing self-questioning, answering, and criticism within a synergistic dual-loop system. Inspired by the Zone of Proximal Development (ZPD) theory, AERO utilizes
entropy-based positioning to target the “solvability gap” and employs Independent Counterfactual Correction for robust verification. Furthermore, we introduce a Staggered Training Strategy to synchronize capability growth across functional roles and prevent curriculum collapse. Extensive evaluations across nine
benchmarks spanning three domains demonstrate that AERO achieves average performance improvements of 4.57% on Qwen3-4B-Base and 5.10% on Qwen3-8B-Base, outperforming competitive baselines. Code is available at https://github.com/mira-ai-lab/AERO.
Read the full abstract ▾
All authors Zhitao Gao, Jie Ma, Xuhong Li, Pengyu Li, Ning Qu, Yaqiang Wu, Hui Liu, Jun Liu
Computation and Language
ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution
Junjie Huang, Jiarui Qin, Di Yin et al. (+4 more)
Standard training pipelines for large
language models (LLMs) are typically unidirectional, progressing from pre-training to post-training. However, the potential for a bidirectional process–where insights from post-training retroactively improve the pre-trained foundation–remains unexplored. We aim to establish a self-reinforcing flywheel: a cycle in which reinforcement learning (RL)-tuned model strengthens the base model, which in turn enhances subsequent post-training performance, requiring no specially trained teacher or reference model. To realize this, we analyze training dynamics and identify the mid-training (annealing) phase as a critical turning point for model capabilities. This phase typically occurs at the end of pre-training, utilizing high-quality corpora under a rapidly decaying learning rate. Building upon this insight, we introduce ReMiT (Reinforcement Learning-Guided Mid-Training). Specifically, ReMiT leverages the
reasoning priors of RL-tuned models to dynamically reweight
tokens during the mid-training phase, prioritizing those pivotal for
reasoning. Empirically, ReMiT achieves an average improvement of 3% on 10 pre-training
benchmarks, spanning math, code, and general
reasoning, and sustains these gains by over 2% throughout the post-training pipeline. These results validate an iterative feedback loop, enabling continuous and self-reinforcing
evolution of LLMs.
Read the full abstract ▾
All authors Junjie Huang, Jiarui Qin, Di Yin, Weiwen Liu, Yong Yu, Xing Sun, Weinan Zhang
Computation and Language · Cross-listed from cs.HC
From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality
Yoonsang Kim, Divyansh Pradhan, Devshree Jadeja et al. (+1 more)
We introduce
Speech-to-Spatial, a referent disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance. Unlike prior systems that rely on additional cues (e.g., gesture, gaze) or manual expert annotations,
Speech-to-Spatial infers the intended target solely from spoken references (
speech input). Motivated by our formative study of
speech referencing patterns, we characterize recurring ways people specify targets (Direct Attribute, Relational, Remembrance, and Chained) and ground them to our object-centric relational graph. Given an utterance, referent cues are parsed and rendered as persistent in-situ AR visual guidance, reducing iterative micro-guidance ("a bit more to the right", "now, stop.") during remote guidance. We demonstrate the use cases of our system with remote guided assistance and intent disambiguation scenarios. Our evaluation shows that Speechto-Spatial improves task efficiency, reduces cognitive load, and enhances usability compared to a conventional voice-only baseline, transforming disembodied verbal instruction into visually explainable, actionable guidance on a live shared view.
Read the full abstract ▾
All authors Yoonsang Kim, Divyansh Pradhan, Devshree Jadeja, Arie Kaufman
Computation and Language · Cross-listed from cs.AI
MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems
Vishal Venkataramani, Haizhou Shi, Zixuan Ke et al. (+6 more)
Multi-
Agent Systems (MAS) built on Large
Language Models (LLMs) often exhibit high variance in their
reasoning trajectories. Process verification, which evaluates intermediate steps in trajectories, has shown promise in general
reasoning settings, and has been suggested as a potential tool for guiding coordination of MAS; however, its actual effectiveness in MAS remains unclear. To fill this gap, we present MAS-ProVe, a systematic empirical study of process verification for multi-
agent systems (MAS). Our study spans three verification paradigms (LLM-as-a-Judge, reward models, and process reward models), evaluated across two levels of verification granularity (
agent-level and iteration-level). We further examine five representative verifiers and four context management strategies, and conduct experiments over six diverse MAS frameworks on multiple
reasoning benchmarks. We find that process-level verification does not consistently improve performance and frequently exhibits high variance, highlighting the difficulty of reliably evaluating partial multi-
agent trajectories. Among the methods studied, LLM-as-a-Judge generally outperforms reward-based approaches, with trained judges surpassing general-purpose LLMs. We further observe a small performance gap between LLMs acting as judges and as single
agents, and identify a context-length-performance trade-off in verification. Overall, our results suggest that effective and robust process verification for MAS remains an open challenge, requiring further advances beyond current paradigms. Code is available at https://github.com/Wang-ML-Lab/MAS-ProVe.
Read the full abstract ▾
All authors Vishal Venkataramani, Haizhou Shi, Zixuan Ke, Austin Xu, Xiaoxiao He, Yingbo Zhou, Semih Yavuz, Hao Wang, Shafiq Joty
Computation and Language
SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression
Xing Hu, Dawei Yang, Yuan Cheng et al. (+2 more)
The rapid growth in the parameter scale of large
language models (LLMs) has created a high demand for efficient compression techniques. As a hardware-agnostic and highly compatible technique, low-rank compression has been widely adopted. However, existing methods typically compress each layer independently by minimizing per-layer reconstruction error, overlooking a critical limitation: the reconstruction error propagates and accumulates through the network, which leads to amplified global deviations from the full-precision baseline. To address this, we propose Self-Adaptive Error Suppression SVD (SAES-SVD), a LLMs compression framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. SAES-SVD is composed of two novel components: (1) Cumulative Error-Aware Layer Compression (CEALC), which formulates the compression objective as a combination of local reconstruction and weighted cumulative error compensation. Based on it, we derive a closed-form low-rank solution relied on second-order activation statistics, which explicitly aligns each layer's output with its full-precision counterpart to compensate for accumulated errors. (2) Adaptive Collaborative Error Suppression (ACES), which automatically adjusts the weighting coefficient to enhance the low-rank structure of the compression objective in CEALC. Specifically, the coefficient is optimized to maximize the ratio between the Frobenius norm of the compressed layer's output and that of the compression objective under a fixed rank, thus ensuring that the rank budget is utilized effectively. Extensive experiments across multiple LLM architectures and tasks show that, without fine-tuning or mixed-rank strategies, SAES-SVD consistently improves post-compression performance.
Read the full abstract ▾
All authors Xing Hu, Dawei Yang, Yuan Cheng, Zhixuan Chen, Zukang Xu
Computation and Language
LatentMem: Customizing Latent Memory for Multi-Agent Systems
Muxin Fu, Xiangyuan Xue, Yafu Li et al. (+5 more)
Large
language model (LLM)-powered multi-
agent systems (MAS) demonstrate remarkable collective intelligence, wherein multi-
agent memory serves as a pivotal mechanism for continual adaptation. However, existing multi-
agent memory designs remain constrained by two fundamental bottlenecks: (i)
memory homogenization arising from the absence of role-aware customization, and (ii) information overload induced by excessively fine-grained
memory entries. To address these limitations, we propose LatentMem, a learnable multi-
agent memory framework designed to customize
agent-specific
memories in a
token-efficient manner. Specifically, LatentMem comprises an experience bank that stores raw interaction trajectories in a lightweight form, and a
memory composer that synthesizes compact latent
memories conditioned on retrieved experience and
agent-specific contexts. Further, we introduce Latent
Memory Policy Optimization (LMPO), which propagates task-level optimization signals through latent
memories to the composer, encouraging it to produce compact and high-utility representations. Extensive experiments across diverse
benchmarks and mainstream MAS frameworks show that LatentMem achieves a performance gain of up to $19.36$% over vanilla settings and consistently outperforms existing
memory architectures, without requiring any modifications to the underlying frameworks.
Read the full abstract ▾
All authors Muxin Fu, Xiangyuan Xue, Yafu Li, Zefeng He, Siyuan Huang, Xiaoye Qu, Yu Cheng, Yang Yang