Computation and Language
Quizzing the Translation: A Prover-Grounded Evaluation Metric for NL$\rightarrow$FOL
Pu Suo and Ali Emami
A standard pipeline for symbolic
reasoning over natural-
language problems translates them into first-order logic and invokes a
theorem prover. The translation step is the bottleneck: swap "every" for "some" and every inference that follows is corrupted. Yet today's metrics often score more broken translations higher than less broken ones, because BLEU, BERTScore, and Smatch++ reward surface overlap that the worst errors happen to preserve. We introduce SIV, which derives two kinds of probes from the target formula and uses a
theorem prover to verify the candidate translation against each. Positive probes are statements the candidate must entail, which detect translations that drop content; contrastive probes are statements the candidate must not entail, which detect translations that assert more than the original. On a controlled pool of perturbed FOLIO translations, the severity of the error accounts for 80% of SIV's score variance, compared with at most 17% for any prior metric. Across six error classes on a disjoint pool, SIV scores the reference above the perturbed candidate in over 99% of pairs. Because each probe is labeled with what it tests, the failure pattern also supplies a labeled error trace, recovering the perturbation class at macro-F1 0.638, nearly double the score-only baseline. On 434 expert-audited real LLM translations, SIV attains the top AUC, uniquely detects and grades expert-labeled major errors, and abstains, rather than mis-scoring, on out-of-vocabulary translations.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
Zihan Lin, Xiaohan Wang, Jie Cao et al. (+4 more)
Efficient exploration often remains a central bottleneck in reinforcement learning with verifiable rewards (RLVR). Although temperature control and test-time scaling strategies can increase rollout diversity of large
language models (LLMs), they either expand the sample budget at rollout time or leave the benefit of exploration unquantified. To this end, we propose Temperature-Grouped Reinforcement Learning (TGRL), which turns temperature-induced diversity into an explicit training signal. For each
prompt, TGRL partitions its rollout group into low- and high-temperature subsets, estimates exploration gain through their reward contrast, and allocates this group-level signal as
token-level credit using Jensen–Shannon (JS) divergence between the corresponding temperature-scaled next-
token distributions induced by the same logits. Notably, TGRL reaches equivalent accuracy up to 36% faster than strong RLVR baselines without expanding the rollout budget. Across 11
benchmarks from diverse domains, TGRL broadly improves over strong RLVR baselines: it improves the six-
benchmark math average by 1.6% at 32B, raises CodeForces rating by 196.7 points and LiveCodeBench Pass@16 by 4.4%, and improves ALFWorld/WebShop success rates by 6.3%/4.9%. Comprehensive ablations and wall-clock analysis confirm the efficacy of all proposed components. Code is available at https://github.com/1229095296/TGRL/tree/main.
Read the full abstract ▾
All authors Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Wei Lin, Guojun Yin, Ran He
Computation and Language
Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring
Gabriele Cinà
A
speech neuroprosthesis decodes attempted
speech from
brain activity and ends by rescoring the decoder's candidate sentences with a
language model of several billion parameters, the only component that needs a GPU. Replacing that model with a cheaper one is hard: general
language models asked to pick one sentence from a list answer from where a label sits in the list rather than from the sentence itself. We pose rescoring as a single typed decision, one call that returns a probability for every candidate, served by Jev, a hosted model trained for calibrated decisions, and combine it with the decoder's own score. On 978 held-out sentences from a participant with ALS, where the published decoder alone reaches 8.1% word error, Jev reaches 7.5% against 7.8% for both OPT-6.7b and Qwen2.5-7B; with the decoder's weight re-tuned, 6.9% against 7.2% and 7.4%. Jev is ahead in all four comparisons and at most 0.2 points behind at the 95% bound. It costs 0.07 USD per thousand sentences and needs no GPU; a dedicated GPU running a 7B model is cheaper per sentence only above 43% utilisation, far beyond what one user generates. End-to-end latency over the internet is 262 ms, of which 62 ms is spent at the provider, the same order as a 7B model on a local GPU (27 ms) but not faster.
Read the full abstract ▾
Computation and Language
ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective
Rui Liu, Chenheng Zhang, Haoxuan Li et al. (+1 more)
Large
language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a
manifold perspective, viewing it as a localized displacement of an edit sub-
manifold within the global knowledge
manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-
manifold, and (ii) how to preserve the remaining
manifold structure during the sub-
manifold displacement process. Based on this perspective, we propose ManiEdit, a novel
manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-
manifold.
Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space
alignment. Experiments on two base LLMs and four unstructured editing
benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit
Read the full abstract ▾
All authors Rui Liu, Chenheng Zhang, Haoxuan Li, Zhouchen Lin
Computation and Language
E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding
Khloud AL Jallad, Nada Ghneim and Ghaida Rebdawi
Natural
Language Understanding (NLU) plays a crucial role in various applications, yet its performance suffers from weaknesses in handling the complexities of human
languages, ranging from lexical ambiguity to high-level
reasoning difficulties. Analyzing errors across diverse linguistic phenomena is crucial for NLU improvement, as it will help humans get insights to comprehensively assess models' limitations and capabilities, so optimizing models' generalization. Notably, several
benchmarks contain diagnostics
datasets designed for investigation and fine-grained error analysis. When highlighting the gaps in the state-of-the-art, we noted that there is no naming convention for macro and micro categories or even a standard set of linguistic phenomena that should be covered. To overcome this gap, we propose an initial hierarchy for Cross-Lingual NLU error analysis. Moreover, we propose a methodology to create an NLI hierarchical framework and applied a case study on Arabic NLU. Moreover, this paper introduces E-CONAN diagnostics
dataset, a freely available
dataset manually-annotated with coarse-grained and fine-grained categories based on our proposed Arabic hierarchy. E-CONAN
dataset helps NLU designers better understand their models by doing error analysis and in-depth investigation. We used E-CONAN to investigate the performance of 9 pretrained
language models and 5 LLMs. Results indicate that LLMs outperform pretrained models in world knowledge and commonsense
reasoning macro-category, and underperform pretrained models in syntactic macro-category. Moreover, the hardest phenomena for all models is
Reasoning, and the easiest phenomena for all pretrained models is Syntactic, and the easiest for LLMs is Lexico-Syntactic.
Read the full abstract ▾
Computation and Language
LLMs Trust Their Own: Identity-Dependent Conformity in Multi-Agent Systems
Liron Soffer, Ravid Shwartz-Ziv and Chen Shani
Large
language models (LLMs) are increasingly deployed in multi-
agent settings, where
agents observe and influence one another, making social influence a key dimension of AI behavior and safety. We investigate whether LLMs' responses depend on the social identity of other
agents, beyond the effect of their consensus. We construct judgment tasks with a single correct answer, and place models in a multi-
agent setting where they receive incorrect answers from other
agents whose social identities (AI or human, model family, or an arbitrary minimal group) are either shared with or distinct from their own. Across 12 open-weights models and nine tasks, we find a bidirectional effect of group identity on conformity to incorrect answers: in-group consensus increases conformity (in-group favoritism), whereas out-group consensus decreases it (out-group divergence). Unlike humans, for whom one ally breaking the consensus sharply reduces conformity, models are unmoved by an ally from the majority's group. Worse, a correct ally from the opposing group intensifies this bidirectional effect. Chain-of-Thought
reasoning suppresses most of these effects, yet an in-group ally still reduces conformity to an incorrect out-group majority. Labeling peers as safety-aligned shifts overall conformity but leaves in-group favoritism and out-group divergence intact. These results show that group identity shapes how LLMs aggregate information across
agents, independently of its correctness, and identify a manipulation surface for multi-
agent AI systems.
Read the full abstract ▾
Computation and Language
DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregation
Guanzheng Chen, Viet Dac Lai, Subhojyoti Mukherjee et al. (+5 more)
While Large
Language Models (LLMs) advertise million-
token context windows,
reasoning quality often collapses as inputs grow – a phenomenon termed context rot. This failure stems from a structural
entanglement in monolithic architectures, where the massive search burden of contextual grounding exhausts the representational capacity needed for complex
reasoning. To resolve this, we propose Grounding-
Reasoning Disaggregation via DIStributed long COntext scaling (DISCO). Inspired by distributed computing frameworks like Apache Spark, DISCO partitions long context across a fleet of Worker LLMs dedicated exclusively to parallel, localized grounding. A central Driver LLM, trained via Reinforcement Learning (GRPO) to optimize planning, orchestrates execution by dynamically mapping queries into atomic extraction tasks and reducing the gathered evidence to synthesize a final answer. By isolating
reasoning from raw context noise, DISCO effectively eliminates context rot. On RULER-QA (1M
tokens), it maintains 78.4% accuracy where standard baselines collapse. Furthermore, it outperforms full-context models by up to 9.8 points on LongBench v2 and matches frontier models like Gemini-3-Pro-Preview while reducing inference costs by over 80%, establishing a highly efficient paradigm for robust long-context inference.
Read the full abstract ▾
All authors Guanzheng Chen, Viet Dac Lai, Subhojyoti Mukherjee, Branislav Kveton, Seunghyun Yoon, Franck Dernoncourt, Qizhe Xie, Trung Bui
Computation and Language · Cross-listed from cs.LG
A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards
Andreas Plesner, Curtis Northcutt, Francisco Guzmán et al. (+1 more)
When post-training large
language models on tasks with semi-verifiable rewards, there are many factors (training steps, base model size, training order, data quality, verifier accuracy, etc.) that practitioners must contend with to maximize model performance. Yet, it remains unclear how well verifier agreement predicts post-training performance on such tasks. In this paper, we explore this question with over 11k H100 GPU-hours, across HealthBench and PRBench tasks in medical, legal, and finance domains. Across the tested domains, Qwen3 trainees (1.7B-8B on HealthBench; 8B on PRBench), evaluation splits, and frontier LLM reference judges (which we call golden verifiers), higher verifier agreement does not consistently identify the best training verifier. Expensive verifiers need not outperform inexpensive ones, and open-weight Gemma verifiers produce strong training outcomes. We compare two low-cost choices retrospectively – a cost-reducing choice and a balanced choice – with estimated grading cost reductions of 98.8%-99.7% relative to the golden grading protocols and average post-training score gaps of 1-3 points from the best evaluated training verifier. These averages include larger losses in individual settings; they do not establish that verifier choices are interchangeable.
Read the full abstract ▾
All authors Andreas Plesner, Curtis Northcutt, Francisco Guzmán, Anish Athalye
Computation and Language
GSM: Efficient Language Modeling with Shared Global State
Yunao Zheng, Bin Wen, Xiaojie Wang et al. (+12 more)
Improving
language model efficiency requires not only cheaper operations but also eliminating redundant computation. We argue that effective modeling requires information accessibility and computational depth, but not repeated historical
retrieval across layers. Guided by this principle, we introduce the Global State Model (GSM), a causal encoder–decoder architecture that separates historical information acquisition from deep computation. The encoder progressively aggregates long-range information through multi-stage sparse
retrieval into a fixed-size shared state, while decoder layers perform successive transformations over this state without repeated historical access. This design makes decoder
attention and cache costs independent of history length, while dependency-aware
pruning further reduces prefill computation. Experiments across multiple model scales show that GSM achieves downstream performance comparable to matched DeepSeek-V4.1-Flash-based baselines, with improvements at some scales, while preserving long-context
retrieval capabilities. GSM also improves training and inference throughput and reduces cache overhead. These results demonstrate that eliminating architectural redundancy offers an effective path toward efficient
language modeling without sacrificing information accessibility or computational depth.
Read the full abstract ▾
All authors Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Jiankang Chen, Xu Wang, Tingting Gao, Han Li
Computation and Language
Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features
Cunchun Li, Haonan He, Yifan Gao et al. (+4 more)
Supervised fine-tuning (SFT) learns most aggressively from
tokens that the model deems least likely. This helps acquire new behaviors, but also amplifies noisy or conflicting supervision and can overwrite useful pretrained knowledge. Through a unified policy-loss view, we revisit existing
token-reweighting methods and show that they assign nonnegative coefficients to demonstrated
tokens. Consequently, they can suppress or amplify supervised updates, but cannot reverse harmful features once learned. Moreover, larger training weights do not amount to feature extrapolation, since they change the optimization trajectory rather than scale a fixed SFT direction. We argue that reversal and extrapolation require a stable reference frame defined by a fixed SFT delta. Motivated by this, we propose SCALE (Selective Control of Adaptation via Local
Entropy), an
entropy-guided adaptation-strength-control method that freezes the pretrained model and the SFT delta and learns bounded
token- and module-specific gates by minimizing predictive
entropy alone. These gates suppress, reverse, or extrapolate frozen SFT features according to their
alignment with
entropy reduction. Across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base, SCALE achieves mathematical-
reasoning averages of 37.84, 43.60, and 36.57, exceeding the strongest corresponding baselines while remaining competitive on general-retention
benchmarks. It also attains the best average code-generation performance across HumanEval, HumanEval+, and MBPP for all three models. These results suggest that effective SFT correction can benefit from controlling how already learned residuals are used, rather than only modifying how they are learned.
Read the full abstract ▾
All authors Cunchun Li, Haonan He, Yifan Gao, Minglei Li, Jingqi Ye, Qingyu Yang, Peng Ye
Computation and Language
Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends
Seonghyeon Go, Yongwoo Kim, Hyeonjin Cha et al. (+1 more)
Full-duplex spoken dialogue models can listen and speak simultaneously like the real-time dynamics of human conversation. For natural dialogue, the ability to search for external information in real-time is also an important capability. Many models remain trapped in parametric knowledge, leaving them unable to access real-time information and tool execution. Furthermore, even when Large
Language Models (LLM) retrieve information, many duplex
speech models process it within a compressed latent space rather than in its raw text form, which can lead to information loss from compression. To address this issue, we propose Context Spanning, a framework for information injection between a full-duplex
speech model and an external LLM backend via real-time chunked prefill. The injected frame is encoded in a single forward pass inside the real-time frame budget. It feeds the retrieved information to the
speech model as-is, enabling it to reason over the information independently and generate responses. With this approach, our model achieves high performance on Full-Duplex
benchmarks and strong results on Question Answering tasks, demonstrating its conversation potential. Context Spanning shows that external information can be injected directly into a duplex
speech model, introducing a new simple and powerful mechanism for duplex systems.
Read the full abstract ▾
All authors Seonghyeon Go, Yongwoo Kim, Hyeonjin Cha, Jaeho Shin
Computation and Language
MIC: Explaining Image-Claim Inconsistencies in AI-Generated Multimodal Misinformation
Ruihong Zeng, Jonathan Tonglet, Preslav Nakov et al. (+1 more)
Claims paired with AI-generated images are a rapidly growing form of misinformation. Existing automated fact-checking (AFC) methods mainly treat this as a provenance problem, detecting low-level synthesis artifacts to decide whether an image is AI-generated. However, such methods do not verify what human fact-checkers often check: whether an image's content is consistent with the context implied by its accompanying claim. To address this gap, we introduce MIC (
Multimodal Inconsistency Checking), an AFC framework that assists human fact-checkers by detecting AI-generated
multimodal misinformation and explaining inconsistencies using world knowledge. MIC first uses supervised fine-tuning (SFT) for task adaptation and then applies Group Relative Policy Optimization (GRPO) to directly optimize component-level verifiable rewards for verdict prediction, inconsistency type classification, visual evidence description, and world-knowledge explanation. We further introduce MIC-Bench, a
benchmark comprising 8,812 image-claim instances derived from 4,406 claims, where each claim is paired with an authentic image and an AI-generated counterpart that introduces a controlled contextual inconsistency. Compared with SFT alone, GRPO further improves Macro-F1 by 4.67 and 4.11 points in the in-distribution and out-of-distribution settings, respectively, while also improving the semantic similarity of visual evidence descriptions and world-knowledge explanations to reference annotations. Our code and data are available at https://github.com/UKPLab/arxiv2026-mic.
Read the full abstract ▾
All authors Ruihong Zeng, Jonathan Tonglet, Preslav Nakov, Iryna Gurevych
Computation and Language · Cross-listed from cs.AI
Raven: The Harness of Harnesses for Composable Agentic Intelligence
EverMind AI
As large
language models advance, AI
agents are moving beyond isolated, domain-specific tasks toward long-horizon, cross-domain workflows. This transition exposes two challenges: increasing harness complexity makes manual design difficult to scale, while tighter coupling to specific domains limits the generality of a single harness. The central question thus shifts from how to engineer a stronger harness for one domain to how to autonomously construct specialized harnesses, improve them through experience, and orchestrate them across domains. We introduce Raven, The Harness of Harnesses, an open-source multi-
agent ecosystem that automatically constructs and evolves modular harnesses for specific models and domains, treating each executable model–harness pair as a composable unit of intelligence. To support an All-Domain Collaboration Network, its Host
Agent decomposes goals, matches subtasks to specialized
agents, coordinates execution dependencies, and integrates results, while a host archive and EverOS preserve experience across tasks and Skill Forge makes that experience available as reusable procedures. Our theory establishes sufficient conditions for such composition to expand reliable task coverage beyond that of the available individual
agents under a shared resource budget. On complex and long-horizon tasks, Raven significantly outperforms the state-of-the-art
agent systems, pushing the frontier of composable agentic intelligence.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
SMAT: Simple and Efficient Merge-Aware Training
Yanggan Gu, Yuanyi Wang, Zhen Li et al. (+5 more)
Model merging integrates the capabilities of multiple experts without joint retraining, but standard expert training optimizes task loss alone and does not guarantee good performance after merging. Merge-aware training (MAT) aims to improve merged performance, but existing methods do not fully account for common merging operations and add training cost. We observe that, from an expert's perspective, common merging methods can be described by three operations: Scale reweights its own update, Mask removes selected coordinates, and Perturb adds updates from other experts. Based on this view, we introduce SMAT (Simple MAT), which jointly optimizes expert loss and expected loss at simulated merged parameters generated by sampling scaling coefficients, masks, and additive noise. We further introduce periodic scheduling, kernel fusion, and parameter storage switching to make SMAT efficient, with one forward and one backward pass per step. Across four
language and
vision-
language backbones, SMAT improves the mean score across five merging methods by 1.07-2.16 points over the strongest baseline for each backbone, with less than 2% training-time overhead over standard fine-tuning.
Read the full abstract ▾
All authors Yanggan Gu, Yuanyi Wang, Zhen Li, Shuo Cai, Yuhang Liu, Junzhuo Li, Zihao Wang, Hongxia Yang
Computation and Language · Cross-listed from cs.SD
CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulations
Jae Min Woo, Kyongmin Kong, Bogyung Jeong et al. (+3 more)
Text-to-Audio (T2A) retrievers are typically evaluated with caption style queries, but the same user intent can be expressed in many forms. We introduce CORA (Caption-Offset
Retrieval for Audio), a caption anchored diagnostic protocol that rewrites each source caption into five intent preserving forms (Command, Question, Indirect, Key phrase, and Statement) while fixing the target audio. By tracking the same target across query forms, CORA defines RankDrop, a metric revealing failures hidden by Recall@k. Using Pearson's correlation coefficient r, we find that RankDrop is weakly associated with raw text space movement (r=0.084), but strongly associated with Target
Alignment Loss and Target Boundary Margin Degradation (r=0.508 and r=0.615). The same pattern appears in OEA retrievers, where RankDrop is better explained by boundary degradation (r=0.472/0.478) than by query movement (r=0.084/0.046). Overall, these results suggest that robust T2A
retrieval requires preserving the target's boundary advantage over competing audio under reformulation.
Read the full abstract ▾
All authors Jae Min Woo, Kyongmin Kong, Bogyung Jeong, Minjeong Kim, HaeJun Yoo, Du-Seong Chang
Computation and Language
Zhenyu Lei, Zihan Chen, Yaochen Zhu et al. (+5 more)
Reasoning distillation from powerful teacher models to smaller students faces the Gap Curse: as teachers grow more sophisticated, their complex distributions increasingly diverge from what students can approximate, causing performance degradation. Existing mitigation strategies either filter out challenging examples through data selection or introduce weaker intermediate assistant models, inherently compromising supervision coverage or quality. We propose Teacher
Alignment, which directly adapts the teacher toward the student's distribution without discarding data or degrading
reasoning quality. However, naive
alignment through standard knowledge
distillation triggers catastrophic collapse of the teacher's
reasoning capabilities. To address this, we reformulate teacher
alignment as reinforcement learning and introduce TeacherGRPO, built on Group Relative Policy Optimization with two key innovations: (i) Curriculum Selective
Alignment applies dual
token- and distribution-level curricula to focus rewards on high-signal
reasoning gaps while filtering noise from trivial
tokens and uncertain tail distributions, and (ii) Importance-Adaptive Length Regularization selectively penalizes verbose redundancy while preserving pedagogically critical
reasoning steps. The aligned teacher then distills knowledge to students via standard pipelines. Extensive experiments show TeacherGRPO significantly outperforms baselines across diverse
reasoning benchmarks and
distillation methods. Our code is available at https://github.com/LzyFischer/TeacherGRPO.
Read the full abstract ▾
All authors Zhenyu Lei, Zihan Chen, Yaochen Zhu, Shangbin Feng, Zaiyi Zheng, Ruocheng Guo, Yushun Dong, Jundong Li
Computation and Language
Grounding Memory Summarization in Utility Intent
Zhenyu Lei, Mingjia Shi, Xingbo Fu et al. (+3 more)
Existing summarizers for
memory systems are typically optimized for human-facing criteria such as faithfulness, which misaligns with their true objective: preserving the evidence needed to support future queries. We show that conditioning summarization on query-answer pairs substantially improves answer quality, and that this utility-aware behavior is transferable across queries. Motivated by these findings, we propose MemSuit, a self-
distillation framework in which a teacher summarizer, conditioned on observed query-answer pairs, produces utility-aware
memory entries that a student learns to reproduce from the raw conversation alone. To prevent collateral erasure where conditioning on a single query-answer pair discards evidence relevant to other plausible queries, the teacher decomposes each block into multiple self-contained entries that preserve distinct query-relevant facets as independently retrievable units. To align the retriever with the compact, fact-dense style of teacher entries, we further fine-tune the
embedding model with a contrastive objective supervised by teacher entries. Across a diverse suite of conversational query types, MemSuit consistently outperforms state-of-the-art baselines, confirming the value of grounding
memory in downstream utility.
Read the full abstract ▾
All authors Zhenyu Lei, Mingjia Shi, Xingbo Fu, Haoyu He, Qi R. Wang, Jundong Li
Computation and Language · Cross-listed from cs.AI
MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses
Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu et al. (+3 more)
Rapid progress in Large
Language Models (LLMs) is saturating static
benchmarks faster than they can be designed. While existing automated
evolution frameworks attempt to generate harder questions by perturbing individual tasks, they remain constrained by rigid, hard-coded generation rules. Moving beyond the
evolution of isolated tasks, we propose to optimize the
benchmark generation workflow itself end to end with MetaBench-Harness, a dual-loop search framework. Specifically, the inner loop utilizes a
benchmark harness to generate a new
benchmark in each round, while the outer meta-harness orchestration layer iteratively refines and searches over harness implementations based on historical
evolution trajectories. By applying MetaBench-Harness to the competitive programming CodeContests and Olympiad mathematics AIME-2024
datasets, we demonstrate that the evolved
benchmarks are challenging and discriminative for frontier models. Trajectory and quality analyses verify that MetaBench-Harness enables multi-dimensional
evolution, steadily improving
evolution reasonableness,
benchmark competency, and evaluator robustness across successive rounds. Furthermore, case studies reveal its effective utilization of diverse difficulty levers to reframe problems and elevate required capabilities. Ultimately, this work provides a solution to the pressing challenge of
benchmark saturation.
Read the full abstract ▾
All authors Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu, Yinghao Ma, Jyh-Shing Roger Jang, Hung-yi Lee
Computation and Language
Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation
Yuhao Sun, Binrui Wu, Zhuoer Xu et al. (+5 more)
On-policy
distillation (OPD) transfers the capabilities of a large
language model to a smaller student by providing teacher supervision on the student's own rollouts. In long-horizon agentic tasks, however, uniform
token-level matching can allocate supervision poorly: a large local discrepancy need not improve future behavior, while consequential guidance may be beyond the current student's reach or fail to persist without privileged input. We formulate long-horizon OPD as hierarchical supervision allocation and argue that productive guidance lies at the intersection of future utility and current learnability. Crucially, this intersection evolves as the student learns. Based on this principle, we propose LENS-OPD, a coarse-to-fine framework that organizes supervision through Locate, Validate, and Refine. Locate adapts trajectory exposure to the student's evolving competence and proposes a candidate decision for intervention. Validate tests whether teacher guidance at that decision improves the same student's subsequent behavior. Refine internalizes the beneficial guided behavior into the deployable policy and concentrates
token-level supervision on decisive teacher-student conflicts within the validated turn. These stages are nested: each finer allocation is conditioned on the coarser decision, rather than being optimized as an independent importance score. Experiments across multiple long-horizon
agent benchmarks and student-teacher configurations show that LENS-OPD consistently improves task performance over vanilla OPD and strong curriculum- and selection-based baselines. Our results suggest that effective long-horizon
distillation requires teaching at the right depth, the right decision, and the right
token.
Read the full abstract ▾
All authors Yuhao Sun, Binrui Wu, Zhuoer Xu, Ming Wen, Haoxiang Xu, Bin Chen, Yan Lin, Qianzijing Zhang
Computation and Language · Cross-listed from cs.LG
Decoupling Token Roles in Autoregressive Pretraining
Suqin Yuan, Runqi Lin, Kevin Qinghong Lin et al. (+4 more)
Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual
token. The next-
token prediction objective naturally identifies a
token's contribution with its own loss. However, each
token is not only a prediction target but also context for what follows. Using controlled corruption, we decouple these two roles and find a reversal: making a noisy
token easier to predict reduces its damage as a target but increases it as context. The same decoupling helps explain text generated by
language models: generation selects each
token by its fit to the prefix, while its role as context is never tested against an independently determined continuation, because that continuation is generated to fit it. At known corrupted positions, acting through the context can reduce damage that removing the
token's own loss does not. Understanding and controlling what a model learns from a
token therefore requires decoupling its roles.
Read the full abstract ▾
All authors Suqin Yuan, Runqi Lin, Kevin Qinghong Lin, Junchi Yu, Lei Feng, Chris Russell, Tongliang Liu
Computation and Language
Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali
Kalash Shrestha and Nikhil Pradhan
A byte-level BPE vocabulary learns each inflected form of a Nepali word as a separate string, so a noun stem is spelled differently in each of its case-marked forms. We test whether splitting words into stem and affixes before BPE helps, with the corpus, vocabulary size, model and number of training steps held fixed. Our pre-tokenizer, Papaya, uses a finite-state transducer built from a published grammar of Nepali, falls back to regular expressions, and leaves the BPE trainer unchanged. On 607 words annotated by seven native speakers its segmenter reaches 0.96 boundary F1, and the resulting
tokens keep stems intact far more often than plain BPE does. In a 17M-parameter
language model it lowers bits per byte by about 1% at equal training steps; most of the larger gain seen at equal epochs comes from the extra steps that longer
token sequences buy, and an unsupervised Morfessor segmentation gives the same improvement. Downstream the effect is small: NER improves only on entities that contain words unseen in training, POS tagging and news classification do not change, and published Nepali tokenizers perform about as well. We release the annotated boundary set, a 556-affix
dataset and the code.
Read the full abstract ▾
Computation and Language
From Position Risks to Block Survival: Faster Generation for Diffusion Language Models
Siwei Chen, Yuxiang Wan, Yifan Yu et al. (+1 more)
Diffusion language models (DLMs) can accelerate generation by predicting multiple
tokens in parallel, but there is a mismatch between how these
tokens are predicted and how they ultimately contribute to generation. Parallel predictions can hardly condition on the
tokens selected earlier within the same block, even though their validity depends on this realized prefix. Under the popular proposal-verification decoding, this mismatch makes errors highly asymmetric: an early rejection prevents all subsequent proposals from contributing decoding progress. We introduce BRISK-DLM, a framework that addresses both mismatches by optimizing proposal learning and selection for verified progress. BRISK-DLM trains on self-generated sequences, using risk-reward weighting to dynamically prioritize positions by their impact on verified progress and decoding cost. During inference, a lightweight prefix-conditioned corrector reranks existing candidates using previously selected
tokens and preferences distilled from the model's own verifier. The corrector reuses the backbone's parallel representations and requires no additional backbone evaluation, while fused execution keeps its overhead small. BRISK-DLM improves end-to-end throughput by up to 37.4% while preserving task quality, establishing a new quality-throughput frontier for DLM generation.
Read the full abstract ▾
All authors Siwei Chen, Yuxiang Wan, Yifan Yu, Fan Lai
Computation and Language · Cross-listed from cs.DC
OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading
Jingyuan Xiao, Jiayue Wang, Yitao Hu et al. (+7 more)
Semi-autoregressive
diffusion large
language models (dLLMs) improve decoding parallelism through iterative block-wise denoising, but scaling them with mixture-of-experts (MoE) layers introduces a large expert parameter footprint that exceeds
memory-constrained GPU capacity. Expert offloading is a natural remedy, yet existing MoE serving systems target autoregressive decoding and rely on intra-iteration layer-wise prefetching: while computing one layer, they predict and load experts for subsequent layers. Under dLLM inference, block-wise routing expands the active expert working set within each iteration, making such prefetches difficult to complete in time and costly when mispredicted. Consequently, existing prefetch-based solutions often degenerate into on-demand expert loading with high decoding latency. We propose OLED-MoE, an expert offloading system that shifts the optimization target from intra-iteration prefetching to inter-iteration expert retention. Its key insight is that adjacent denoising iterations exhibit strong expert routing overlap, and
token confidence indicates which experts are likely to be reused. OLED-MoE uses confidence-guided inter-iteration prediction to retain high-value experts in GPU
memory without introducing extra prefetch traffic. It further compensates unavoidable cache misses through CPU-GPU cooperative execution, jointly considering dynamic expert computation load and predicted future reuse. Across diverse dLLM workloads, OLED-MoE reduces time per output
token (TPOT) by 1.23x-7.93x and improves expert cache utilization by 1.44x-4.23x over state-of-the-art offloading systems. Notably, OLED-MoE approaches full-residency performance while using only 40% of the expert GPU
memory, incurring merely 23% higher TPOT despite a 60% reduction in expert
memory footprint. OLED-MoE's source code is publicly available at https://github.com/flashserve/OLED-MoE.
Read the full abstract ▾
All authors Jingyuan Xiao, Jiayue Wang, Yitao Hu, Xinning Wang, Shi Chen, Ziqi Gong, Zhengchao Wang, Guotao Yang, Sheng Chen, Keqiu Li
Computation and Language
NLPG: Natural-Language Policy Gradients for Self-Evolving Language Agents
Xu Liu, WenZhang Wei, Jun Cao et al. (+4 more)
Large
language model
agents increasingly rely on compound programs for
retrieval, tool use,
reasoning, and verification, yet their failures often arise from local procedural decisions. Existing reinforcement-learning and
prompt-optimization approaches typically rely on scalar rewards or repeatedly modify entire
prompts, making it difficult to capture and reuse procedural improvements while preserving a frozen
agent. To address this problem, We propose Natural-
Language Policy Gradients (NLPG), an external policy-
memory method for improving a fixed
agent without changing its model parameters or program structure. NLPG diagnoses execution traces, propagates downstream feedback backward through the module graph, and converts recurring failures into route-local natural-
language corrections that are aggregated into bounded policy updates for subsequent executions. Across six
benchmarks covering
memory,
reasoning, instruction following, and evidence verification, NLPG also outperforms the strongest listed baseline for each
benchmark by 8.71 percentage points on average. These results provide evidence that evaluated procedural experience can be transformed into local and interpretable policy updates, enabling continual improvement of frozen
agents.
Read the full abstract ▾
All authors Xu Liu, WenZhang Wei, Jun Cao, Dehua Peng, Huan Chen, Zhipeng Gui, Huayi Wu
Computation and Language · Cross-listed from eess.AS
From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS
Kangxiang Xia, Xinfa Zhu, HangRui Hu et al. (+8 more)
Multi-speaker dialogue TTS requires natural
speech generation, consistent speaker identity, coherent cross-turn transitions, and fine-grained control of expressive attributes such as emotion, speaking rate, and loudness. These requirements are difficult to satisfy reliably with one-shot generation, especially in long-form dialogue. We propose a controllable multi-speaker dialogue TTS framework that formulates synthesis as critique-driven iterative refinement. Its
speech backbone, ControlEdit-TTS, unifies instruction-following synthesis and natural-
language-guided attribute editing, enabling correction of expressive errors without full regeneration. The framework further performs hierarchical utterance-level and scene-level critique, routing detected issues to editing, resynthesis, or timing adjustment. Experiments on a bilingual Chinese–English dialogue
benchmark show improved utterance-level instruction following, better dialogue-level preference than direct dialogue models and agentic baselines, and more effective refinement than regeneration-only alternatives while preserving speaker identity. Ablations further confirm the benefits of scene-level critique and edit-based correction.
Read the full abstract ▾
All authors Kangxiang Xia, Xinfa Zhu, HangRui Hu, Kexin Huang, Wenjie Tian, Ziyue Jiang, Bingshen Mu, Jingbin Hu, Ting He, Lei Xie, Jin Xu
Computation and Language
Language Discrimination Improves Linguistic Learning in Multilingual Speech Models
Maureen de Seyssel, Jie Chi and Zakaria Aldeneh
Multilingual self-supervised
speech models can benefit from sharing information across
languages, but under a matched total pretraining data budget they still fall short of monolingual models. We show that strengthening the model's ability to discriminate
languages during pretraining reduces and, on some measures, closes this multilingual gap on continuous phonetic and higher-level linguistic measures, while preserving substantial cross-
language sharing. Using a controlled English/French HuBERT setting, we test two interventions which strengthen
language discrimination: an auxiliary
language classifier and per-
language k-means targets. Across interventions, continuous-feature phone discrimination error (phone-ABX, lower is better) decreases from 11.6% in the bilingual baseline to 10.4% (monolingual: 10.8%), while lexical performance (sWUGGY, higher is better) increases from 52.1% to 56.7% (monolingual: 58.5%) and prosodic performance (ProsAudit, lexical subtask, higher is better) from 68.9% to 72.9% (monolingual: 72.6%). Across HuBERT training stages, the strongest gains on most linguistic measures occur when
language discrimination is introduced in the first iteration, whereas later or repeated interventions yield smaller improvements and are accompanied by increased
language-wise segregation. These results support a causal role for
language discrimination in reducing the additional cost of multilingual learning.
Read the full abstract ▾
Computation and Language
CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation
Praveenkumar Katwe, Rakesh Chandra Balabantaray and Kali Prasad Vittala
Faithfulness evaluation of abstractive summaries remains an open challenge, with existing metrics addressing only isolated
hallucination types: factual entity errors, relational inconsistencies, or numerical fabrications, without capturing their co-occurrence or interaction. We introduce CHI (Composite
Hallucination Index), the first unified
hallucination metric that decomposes faithfulness errors into three orthogonal dimensions: entity
hallucination (EHI), relation
hallucination (RHI*), and quantity
hallucination (QHI). Each dimension employs a shared softmax-normalized architecture over Venn diagram-derived factors representing extractiveness, positive
hallucination, over-focus, negative
hallucination, and lost focus. The novel QHI component introduces tolerance-aware numerical matching with exact, epsilon, derived, and temporal comparison modes. We fuse the three dimensions via harmonic mean to produce a single composite score that penalizes weakness in any dimension. We validate CHI on 800 source articles spanning four domains (news, medical, legal, financial) with summaries from five generation systems. Empirical results demonstrate that: (i) the three dimensions are statistically orthogonal (mean rho = 0.148), confirming they capture distinct error types; (ii) CHI achieves the highest system-level correlation with human judgments (rho = 0.66, p = 0.006) on SummEval, outperforming ROUGE (rho = 0.53), EHI (rho = 0.58), and all individual components; and (iii) ablation studies confirm that all three dimensions contribute unique variance, with the full composite outperforming any individual component while providing decomposable error diagnostics unavailable from single-score baselines. CHI provides practitioners with a decomposable, interpretable, and efficient faithfulness metric suitable for both offline evaluation and online monitoring of summarization systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
CalibHyper: Chance-Corrected Relational Hypergraphs for Few-Shot Molecular Property Prediction
Linyu Li, Zhi Jin, Yuanpeng He et al. (+7 more)
Molecular property prediction is central to drug development and materials discovery, but experiments are costly and labeled data are scarce. Context-aware methods use auxiliary assay labels to support few-shot prediction, and recent work supervises property relations with label agreement. However, label agreement is sensitive to class marginals and does not directly capture dependence between properties. We propose CalibHyper, a chance-corrected relational hypergraph method based on the joint label distribution. CalibHyper subtracts an independence baseline from the ordered four-state label distribution and shrinks the residual according to the number of joint observations. A swap-equivariant relation head estimates these residuals, which choose the auxiliary properties for each molecule and set the sign and weight of their hyperedge messages. On thirteen
datasets from five
benchmarks, in both 1-shot and 10-shot settings, CalibHyper and its ablation settings achieve ROC-AUC competitive with the strongest reported results.
Read the full abstract ▾
All authors Linyu Li, Zhi Jin, Yuanpeng He, Dongming Jin, Huanyu Liu, Huanyao Zhang, Haoran Duan, Heng Tian, Gadeng Luosang, Nyima Tashi
Computation and Language
Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Training
Xinyu Che, Hang Yan, Yanchen Liu et al. (+5 more)
Predicting how an environment will change before acting is a natural route to better decision making for
agents. Recent post-training methods therefore require
agents to predict the next observation and turn that prediction into a reward or a direct supervision signal, which is called world model. Existing next-observation training methods help the
agent to learn the environmental content. However, they additionally involve an optimization process, which may introduce several effects other than learning to predict the world. Consequently, where the performance gain comes from during the training process remains an open question. We answer this research question through replacing true next-observation targets with in-distribution mismatched observations during the training process. Across two interactive text environments, mismatched targets lower prediction accuracy by 15.3-61.6% relative to ground-truth targets, yet retain substantial task gains over the base model. Compared with the base model, trained models consider more candidate actions and exhibit less looping. We also introduce a setting that replaces prediction-based rewards with independent random signals. This training expands task coverage (pass@64) even when the reward carries no environment information. We also generalize this finding to VisualWebArena, where random-reward training raises pass@64 by 14.3% relative to the base model, without observation-matching rewards or an external
multimodal teacher for reward construction.
Read the full abstract ▾
All authors Xinyu Che, Hang Yan, Yanchen Liu, Haochen Liu, Ruifeng Li, Anran Shi, Heng Wang, Jun Liu
Computation and Language · Cross-listed from cs.LG
When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression
Haeyong Kang and Chang D. Yoo
Training-free KV-cache compression methods such as SnapKV, H2O, and PyramidKV evict
tokens at the end of prefill, aiming to preserve the
attention mass that future queries are expected to use -optimizing what to keep. We show that this objective fails in two distinct ways. (1) Compensation: restoring the evicted
attention mass can recover the
attention-level target without recovering task quality. (2) Selection: covering more of the true decode-query mass can hurt quality when the recovered mass is fragmented rather than concentrated in coherent spans. These failures share a common cause: eviction occurs before the queries that determine the answer trajectory exist. We propose Draft-Guided Eviction (DGE), which defers eviction until after drafting the first k=2 answer
tokens using the full cache - just one decode step beyond prefill. Because the draft is generated from the answer's own prefix, no cache entries are discarded before this trajectory signal becomes available. The per-head cache budget remains unchanged, and DGE can be applied directly to SnapKV, PyramidKV, H2O, and StreamingLLM without modifying their eviction scores. Unlike extra-pass methods, DGE changes when eviction occurs rather than what cache entries are selected. Extensive experiments demonstrate that DGE outperforms prior methods at every evaluated budget on five of six instruct-tuned backbones, achieving 44.2 on LongBench, nearly matching FullKV at 44.3. The timing-only control DGE-W achieves the same score, demonstrating that the gain comes from when eviction occurs rather than what is selected - an effect we term trajectory anchoring.
Read the full abstract ▾