Computation and Language · Cross-listed from cs.AI
AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security
Dongrui Liu, Yu Li, Zhonghao Yang et al. (+47 more)
Modern open-world
agents such as OpenClaw exhibit powerful cross-environment execution capabilities yet introduce broad new safety risk sources. Meanwhile, advanced frontier AI models drastically lower attack barriers, rendering current
agent alignment frameworks inadequate for real-world deployment. To tackle these emerging threats, we propose a lightweight and scalable
agent safety
alignment framework. Specifically, we update the
agent safety taxonomy to accommodate emergent risks from Codex and OpenClaw execution scenarios. We further build a taxonomy-guided data engine with influence-function purification to train lightweight AgentDoG 1.5 variants (0.8B, 2B, 4B, and 8B parameters) using only around 1k samples, achieving comparable performance with leading closed-source models (e.g., GPT-5.4). Based on AgentDoG 1.5, we construct a highly efficient agentic safety SFT and RL training environment, which reduces deployment overhead in Docker-level environments by two orders of magnitude. Finally, we deploy AgentDoG 1.5 as a training-free online guardrail for real-time safety moderation. Extensive experimental results indicate that AgentDoG 1.5 achieves state-of-the-art performance in diverse and complex interactive agentic scenarios. All models and
datasets are openly released.
Read the full abstract ▾
All authors Dongrui Liu, Yu Li, Zhonghao Yang, Peng Wang, Guanxu Chen, Yuejin Xie, Qinghua Mao, Wanying Qu, Yanxu Zhu, Tianyi Zhou, Leitao Yuan, Zhijie Zheng, Qihao Lin, Yimin Wang, Haoyu Luo, Shuai Shao, Chen Qian, Qingyu Liu, Ling Tang, Ruiyang Qin, Qihan Ren, Junxiao Yang, Kun Wang, Zhiheng Xi, Linfeng Zhang, Ranjie Duan, Bo Zhang, Wenjie Wang, Wen Shen, Qiaosheng Zhang, Yan Teng, Chaochao Lu, Rui Mei, Man Li, Jialing Tao, Xi Lin, Tianhang Zheng, Yong Liu, Quanshi Zhang, Lei Zhu, Xingjun Ma, Junhua Liu, Hui Xue, Xiaoxiang Zuo, Xiangnan He, Chao Shen, Xianglong Liu, Minlie Huang, Jing Shao, Xia Hu
Computation and Language
Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
Guneet Kohli
LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural
language inference
datasets (each with 100 human annotations per item), we find that the 9 judges effectively provide only about 2 independent votes' worth of information. Roughly three-quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items. The consequences are stark: the panel's actual accuracy falls 8-22 percentage points short of what independent voting would achieve, and the best single judge matches or outperforms the full panel across all conditions. Neither adding more judges nor using smarter aggregation
algorithms helps – established methods close at most 11% of this gap, even with access to the correct answers. We quantify these findings using the Kish effective sample size (n_eff) and a Condorcet null model, and show the deficit is robust across
prompt variants, temperatures, chain-of-thought
reasoning, and a pairwise preference task (RewardBench). The bottleneck is correlated judges, not the aggregation
algorithm, implying that scaling up panels cannot substitute for genuinely independent evaluation.
Read the full abstract ▾
Computation and Language
Metric-Dependent Annotation Saturation for Learning from Label Distributions
Guneet Kohli
When annotators disagree on a label, the disagreement itself carries signal - and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a
dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting,
entropy correlation - whether the model identifies which items elicit disagreement - requires N ~ 20-50 annotators to converge, while distributional match (KL divergence) saturates by N ~ 10 (87-95% of improvement across five model seeds); the specific thresholds are estimates for this setting, and it is the phenomenon, not the constants, that we expect to generalize. This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate. Across five smoothing intensities,
entropy correlation clusters at r ~ 0.45-0.49, while soft labels reach r = 0.643 (p < 0.001); per-item analysis traces this gap to smoothing's inability to distinguish ambiguous items from clear ones. The soft label advantage replicates across two architectures (DeBERTa, RoBERTa) and a non-NLI-pretrained baseline; differential saturation additionally appears in an exploratory cross-domain evaluation on content safety. These results suggest that annotation budgets should be informed by the target evaluation metric rather than set uniformly.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search
Yunbo Tang, Chengyi Yang, Shiyu Liu et al. (+4 more)
Agentic search enables LLMs to solve complex multi-hop questions through iterative
reasoning and external search. Despite the effectiveness, these systems often suffer from a critical limitation in practice:
agents fail to recognize their own knowledge boundaries, blindly triggering searches when internal knowledge suffices and failing to terminate search even when adequate evidence has been collected. The lack of self-awareness leads to severe over-search, incurring substantial inference latency and prohibitive computational cost. To this end, we propose SAAS, a novel RL framework designed to cultivate dynamic self-awareness that precisely regulates search behavior without compromising accuracy. SAAS introduces three key components: (i) a search boundary modeling mechanism, which identifies the search boundary under the evolving policy by contrasting search-disabled and search-enabled rollouts; (ii) a boundary-aware reward module, which translates this boundary awareness into trajectory-level penalties, suppressing unnecessary and redundant searches; and (iii) a stage-wise optimization strategy, which leverages a sequential curriculum to prioritize
reasoning over search regularization, thereby avoiding reward hacking. Extensive experiments demonstrate that SAAS substantially reduces over-search, while maintaining accuracy. Our code and implementation details are released at https://github.com/XMUDeepLIT/SAAS.
Read the full abstract ▾
All authors Yunbo Tang, Chengyi Yang, Shiyu Liu, Zhishang Xiang, Zerui Chen, Qinggang Zhang, Jinsong Su
Computation and Language
ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation
Yutong Yang, Chenxi Miao, Weikang Li et al. (+1 more)
While Large
Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions, revealing a substantial Knowledge-Decision Gap ($G_{\mathrm{KD}}$). Existing
benchmarks struggle to measure this discrepancy due to limited construct validity, multidimensional
entanglement, and distributional biases in LLM-based evaluation. To address these issues, we propose ActTraitBench, a human-grounded evaluation framework for measuring personality consistency in LLMs. Grounded in empirical human data, ActTraitBench establishes one-to-one mappings between psychometric facets and behavioral paradigms and applies Distributional Calibration via
Quantile Mapping to reduce distributional mismatch between LLM-judge scores and human responses. Experiments on 14 mainstream LLMs reveal substantial knowledge-decision gaps and show that assigned personas are reflected more consistently in self-reports than in behavioral decisions for most evaluated models. To mitigate this gap, we further introduce the Chain of Cognitive
Alignment (CoCA), an inference-time intervention that reduces $G_{\mathrm{KD}}$ for 12 of the 13 models with paired results. Code and resources are available at https://github.com/Selina233/ActTraitBench.
Read the full abstract ▾
All authors Yutong Yang, Chenxi Miao, Weikang Li, Yunfang Wu
Computation and Language · Cross-listed from cs.LG
Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning
Zizhe Chen, Jiqian Dong, Yizhou Tian et al. (+4 more)
Reinforcement learning (RL) refines large
language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation
Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM's hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL
algorithms and model sizes without incurring significant computational overhead.
Read the full abstract ▾
All authors Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, James Cheng
Computation and Language
DySem: Uncovering Dynamic Semantic Components of Large Language Models for Calculating Semantic Textual Similarity
Kaijie Zheng, Weiqin Wang, Yile Wang et al. (+1 more)
Calculating semantic textual similarity is a foundational task in natural
language processing. Current large
language models (LLMs) based methods typically rely on extracting last-layer hidden states with fixed dimensions to compute similarity for every text pairs. We argue that this paradigm is suffer from two limitations: (i) The last hidden layer encodes more general knowledge rather than just semantic knowledge, making it suboptimal for semantic similarity computation; (ii) The hidden layer dimensions of LLMs are generally very large, which introduces some redundancy and noise for representing semantics. In this work, we propose DySem, a novel training-free framework that investigates more semantic-related internal components of LLMs via
multilingual consensus, and shifts away from static representation spaces in favor of dynamic, sample-specific semantic dimensions by constructing text-dependent joint semantic set and computes similarity over this shared dimensional subset. Extensive experiments across various LLMs show that our method consistently outperforms recent baselines while maintaining lower dimensions for similarity calculation. The code is released at https://github.com/szu-tera/DySem.
Read the full abstract ▾
All authors Kaijie Zheng, Weiqin Wang, Yile Wang, Hui Huang
Computation and Language · Cross-listed from cs.AI
Why Specialist Models Still Matter: A Heterogeneous Multi-Agent Paradigm for Medical Artificial Intelligence
Yanan Wang, Shuaicong Hu, Jian Liu et al. (+3 more)
The impressive performance of generalist large
language models (LLMs) such as GPT and Claude in healthcare raises a critical question: will domain-specific medical specialist models become obsolete? We argue that the future of medical artificial intelligence (AI) lies not in building monolithic medical foundation models, nor in replacing human expertise, but in orchestrating collaboration among generalist LLMs, domain-specific specialist models, and clinicians. We propose HetMedAgent, a heterogeneous medical multi-
agent framework that enables conflict-aware evidence fusion, uncertainty-based clinician intervention triggering, and adaptive threshold calibration. Experiments on three real-world clinical decision-making tasks demonstrate that the synergy between generalist LLMs and domain-specific specialist models significantly outperforms using either type of model alone, validating the irreplaceable value of specialist models in modality-specific analysis. HetMedAgent represents a shift from building medical LLMs or foundation models to multi-
agent collaboration, achieving a balance between general
reasoning capabilities and domain-specific precision.
Read the full abstract ▾
All authors Yanan Wang, Shuaicong Hu, Jian Liu, Guohui Zhou, Aiguo Wang, Cuiwei Yang
Computation and Language
AfriScience-MT: Towards Decolonizing Science in Africa through Text Translation
Idris Abdulmumin, Tajuddeen Gwadabe, Shamsuddeen Hassan Muhammad et al. (+11 more)
The dominance of colonial
languages in African education and scientific communication limits how hundreds of millions of speakers of African
languages access and produce scientific knowledge. A core obstacle is the lack of established scientific terminology in these
languages. We introduce AfriScience-MT, a parallel corpus covering six African
languages (Amharic, Hausa, Luganda, Northern Sotho, Yorùbá, and isiZulu) across 11 scientific domains. Professional translators, working with expert science communicators, translated plain-
language summaries of scientific papers into each target
language and created new terms where none existed. We
benchmark machine translation systems and large
language models in zero-shot, few-shot, and fine-tuned settings. Our results show that closed-source models outperform all open-source models at both the sentence and document levels: GPT-5.4 and Gemini-3.1-Flash-Lite lead with average sentence-level COMET scores of 68.3 and 68.0, respectively, and tie at an average document-level COMET of 48.3. Among open systems, fine-tuned NLLB-1.3B reaches 67.3 at the sentence level, and TranslateGemma-12B reaches 44.0 at the document level with 1-shot in-context learning. We release AfriScience-MT to support benchmarking and document-level scientific MT for African
languages.
Read the full abstract ▾
All authors Idris Abdulmumin, Tajuddeen Gwadabe, Shamsuddeen Hassan Muhammad, David Ifeoluwa Adelani, Nomonde Khalo, Ibrahim Said Ahmad, Abiodun Modupe, Anina Mumm, Sibusiso Biyela, Michelle Rabie, Johanna Havemann, Marek Rei, Jade Abbott, Vukosi Marivate
Computation and Language
Multi-Legal-Bench: When the Answer Is in the Input. Label Leakage in Legal Benchmarks Built from Court Registries
Volodymyr Ovcharov
Court registries publish millions of decisions with structured metadata, which makes them an attractive source of labelled legal
benchmarks: the court type, the form of the decision and its subject area come for free. We show that this convenience has a cost that registry-derived
benchmarks rarely measure. We build Multi-Legal-Bench, which evaluates identical tasks on native court decisions from five national registries (France, the Netherlands, Poland, the Czech Republic and Lithuania), extending the Ukrainian UA-Legal-Bench, and audit what its
cells actually measure. A keyword scan that uses no model reaches 96% on Dutch judgment-form classification against a 50% majority baseline, 84% on French court-type classification against 33%, and 66% on Polish judgment-form classification, where all nine models add at most seven points over it. Replacing the label names in the input by a mask brings the scan to the majority floor in all six affected
cells, and eight of nine models lose accuracy significantly in most of them (39 of 54 model-
cell pairs), by up to 48 points (Dutch judgment form: 100% to 52%); only Claude Sonnet 5 is essentially unaffected. Paired McNemar tests with Holm correction find a leader that beats every other model in only two of twelve
cells, each a different model; but once the label names are masked, the same model leads both, and the spread between models widens in every masked
cell. Part of the apparent parity between models is produced by the leakage itself. The audit also exposed a scoring defect in our own earlier release (answers written with diacritics were scored as wrong, understating Czech judgment-form accuracy by up to 90 points), which we correct and document. We recommend that every
benchmark built from court registries report a no-model baseline and a masked control per
cell. All data,
prompts, predictions and scoring code are released.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs
Alexander Sternfeld, Andrei Kucharavy and Ljiljana Dolamic
LLM-based coding assistants are seeing rapid adoption, offering substantial gains in developer productivity. As organizations increasingly ship code these
agents produce, the security of that code becomes critical. Prior work has shown that minor
prompt perturbations degrade the functional correctness of LLM-generated code, but whether they also compromise code security has remained unstudied. We apply
token-level mutations to
prompts across three models and five programming
languages, and show that mutations as small as a single-character change can flip generated code from secure to vulnerable. Probing the models' hidden states reveals that this fragility is partially encoded in
prompt representations, but unevenly so. Input-handling vulnerabilities, where the model omits validation or sanitization, are more predictable (mean AUC 0.753) than secure-defaults vulnerabilities, where insecure code stems from one local choice such as a weak
algorithm or unsafe parameter (mean AUC 0.674). These results show that the threat model for LLM-assisted coding extends beyond
prompt injection to ordinary
prompt variation, and indicate that input-handling flaws can be caught before generation while secure-defaults flaws require intervention during decoding.
Read the full abstract ▾
Computation and Language
HTAM: Hierarchical Transition-Attended Memory for Operator Optimization
Yining Zhang, Mingyang Yi, Chen Wang et al. (+5 more)
High-performance GPU kernels are essential for efficient LLM deployment, yet optimizing them remains expertise-intensive. Recent LLM-based code generation makes automatic GPU operator generation promising, but operator optimization remains a hardware-aware search problem. Existing LLM-based methods face a granularity mismatch: coarse hints are reusable but hard to execute, whereas detailed
memories are actionable but enlarge the search space and obscure optimization bottlenecks. The key challenge is therefore to organize optimization experience at an appropriate granularity. To address this issue, this paper proposes HTAM (Hierarchical Transition-Attended
Memory), a coarse-to-fine framework for LLM-based operator optimization. HTAM builds a two-level Hierarchical Transition Graph (HTG) to organize coarse global directions, detailed local strategies, and transition experience between optimization steps. During each
evolution step, HTAM selects a global direction from the current state and recent optimization history, retrieves the corresponding local strategy
memory, and uses it to guide concrete CUDA code generation. Experiments on KernelBench demonstrate that HTAM improves correctness, fast-solution rate, and speedup over LLM-based baselines, while backend and Robust-KBench studies indicate transferable benefits from structured
memory.
Read the full abstract ▾
All authors Yining Zhang, Mingyang Yi, Chen Wang, Xuwen Xiang, Tianhe Jia, Zedong Dan, Chengqing Zong, Yue Wang
Computation and Language
User-Aware Active Knowledge Acquisition for Emotional Support Dialogue
Mufan Xu, Kehai Chen, Jiahao Hu et al. (+4 more)
Emotional support plays an important role in dialogue systems, and its success depends on adapting to a user's evolving and implicit needs across multi-turn interactions while leveraging the strong
reasoning capacity of large
language models. However, since signals about user needs are often weak, indirect, and can only be disambiguated through multi-turn interaction, existing emotional support methods often struggle to acquire and generalize relevant conversational knowledge efficiently. To bridge this gap, we introduce User-Aware Active Knowledge Acquisition (UKA), a gradient-free active dialogue learning framework that explicitly represents uncertainty about user needs and incorporates active learning into both knowledge acquisition and response selection.We propose a Theory-of-Mind uncertainty estimation mechanism that allows the model to prioritize responses, thereby eliciting more informative user feedback. UKA is capable of efficiently exploring user-aligned conversational knowledge during training while maintaining robustness at test time. Experiments across multiple dialogue
benchmarks and model architectures demonstrate that our approach consistently outperforms strong baselines in dialogue quality and user
alignment.
Read the full abstract ▾
All authors Mufan Xu, Kehai Chen, Jiahao Hu, Xinchao Xu, Muyun Yang, Tiejun Zhao, Min Zhang
Computation and Language
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
Aditi Khandelwal, Marius Mosbach, Verna Dankers et al. (+2 more)
Mixture-of-Experts (MoE) models are widely used to scale
language models, yet their expert routing behavior and adaptation in a
multilingual setting remain underexplored. In this work, we study
multilingual routing dynamics during continual pre-training of an English-centric MoE model on a
multilingual corpus, analyzing how expert usage varies across
languages. We find that continual
multilingual pre-training leads to diffused,
language-agnostic routing in early and middle layers, with
language specialization primarily emerging in the final layers. We also show that
token-level vocabulary overlap between
languages plays an important role in how
languages are routed. Motivated by these findings, we propose a parameter-efficient adaptation strategy that updates
language-specific and shared experts in the final MoE layers. Experiments on MultiBLiMP and Belebele show that our method achieves a strong performance-efficiency trade-off, attaining competitive performance relative to fine-tuning complete final layers, while updating less than 2% of the parameters. Overall, our findings provide insights into where and how
language specialization emerges in MoEs during continual pre-training and provide practical insights for low-resource
multilingual adaptation. Our code is available at https://github.com/aditi184/moe-routing-adaptation.
Read the full abstract ▾
All authors Aditi Khandelwal, Marius Mosbach, Verna Dankers, Siva Reddy, Golnoosh Farnadi
Computation and Language
Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies
Yuxuan Ye, Raul Santos-Rodriguez and Edwin Simpson
Grounded claim factuality checking is important for large
language model (LLM) applications such as
retrieval-augmented generation, as it helps users assess the correctness of generated outputs. Existing metrics using entailment classifiers require
dataset-specific threshold tuning, while LLM-based approaches often use direct prompting, which underutilises the
reasoning capabilities of LLMs. We address this by formulating grounded claim factuality checking as a true/false reading comprehension task and prompting LLMs with explicit test-taking strategies for efficient
reasoning. Our method reduces
token usage by over 80% compared to unguided open-ended
reasoning, and achieves competitive performance to more expensive alternatives across two factuality
benchmarks, setting a new state of the art on one. To further reduce inference cost, we train small
language models (SLMs) to replace LLMs in the checking pipeline. Using supervised fine-tuning (SFT) and a self-revision mechanism, the SLMs learn to improve their factuality judgements. Experimental results show that the resulting SLMs perform on par with strong baselines, combining low inference costs with generating supporting rationales to support interpretability. Code and
datasets will be released upon acceptance.
Read the full abstract ▾
Computation and Language
Personalized Turn-Level User Conversation Satisfaction Benchmark
Zhefan Wang, Zhiqiang Guo, Weizhi Ma et al. (+3 more)
User satisfaction with AI assistants is highly personalized: the same response may satisfy one user but disappoint another depending on what each user expects and what they have asked for before. Existing automatic evaluation methods mostly measure generic response quality, making it difficult to judge whether a response satisfies a user at a specific turn. We study this problem as personalized turn-level user conversation satisfaction evaluation. We build a conversation satisfaction evaluator that combines compact user
memories with target-turn context to produce satisfaction scores and dissatisfaction-oriented rationales. Meta-evaluation against human satisfaction annotations shows that personalized
memory and post-hoc score calibration improve ordinal agreement and dissatisfied-turn detection over supervised,
retrieval-based, and generic LLM-as-a-judge baselines. We further introduce PersTurnBench, a personalized turn-level user conversation satisfaction
benchmark that uses the verified evaluator to assess generation models via replay. By holding the replay state fixed, PersTurnBench enables controlled comparison of generic generation models and
memory-augmented personalized systems without new human labels for every candidate model. The evaluator and
benchmark let researchers compare candidate generation models on personalized satisfaction without collecting new user feedback for every model.
Read the full abstract ▾
All authors Zhefan Wang, Zhiqiang Guo, Weizhi Ma, Min Zhang, Quanjia Yan, Hengliang Luo
Computation and Language
RASET: Router-Agnostic Safety-Critical Expert Tuning Exposes Localized Safety Enforcement Failures in Mixture-of-Experts LLMs
Zhibo Zhang, Yuxi Li, Zhen Ouyang et al. (+2 more)
Mixture-of-Experts (MoE) LLMs rely on sparse, router-driven expert activation, yet how safety
alignment interacts with routed expert specialization remains underexplored. A common intuition is that safety behavior may be controlled by routing harmful requests to distinct refusal-oriented experts. In this work, we provide empirical evidence for a different picture: routing patterns in aligned MoE LLMs are largely topic-driven, while safety behavior can be altered with little change to the model's intrinsic routing path. Motivated by this observation, we present RASET (Router-Agnostic Safety-Critical Expert Tuning), a red-teaming framework that probes safety enforcement that is localized in a small subset of experts while preserving the model's intrinsic routing behavior. RASET identifies safety-critical experts via a contrastive routing-sensitivity criterion and applies parameter-efficient tuning only to the selected experts, minimizing semantic disruption relative to router-steering interventions. Across five open-weight MoE backbones, RASET achieves high-quality safety-bypass yield (50.5% average $ASR_{hq}$, +37.6 points over the strongest baseline). These results reveal a distinct MoE safety risk, highlighting the need for expert-aware
alignment mechanisms.
Read the full abstract ▾
All authors Zhibo Zhang, Yuxi Li, Zhen Ouyang, Ling Shi, Kailong Wang
Computation and Language
Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding
Jianuo Huang, Yaojie Zhang, Qituan Zhang et al. (+3 more)
Speculative decoding accelerates LLM inference by drafting multiple
tokens and verifying them in parallel with the target model. However, its practical speedup is constrained by the trade-off between draft quality and drafting cost: autoregressive drafters model causal dependencies among draft
tokens but incur sequential overhead, while parallel drafters reduce drafting cost but weaken intra-block dependency modeling. In this paper, we propose Domino, a speculative decoding framework that decouples causal dependency modeling from expensive autoregressive draft execution. Domino first uses a parallel draft backbone to produce preliminary draft distributions for the entire block, and then applies a lightweight Domino head to refine them with prefix-dependent causal information. To stabilize teacher-forced causal encoding, we further introduce a base-anchored training curriculum that first strengthens the parallel backbone and then gradually shifts optimization toward the causally corrected final distribution. Experiments on Qwen3 models show that Domino achieves up to \(5.49\times\) end-to-end speedup under the
Transformers backend and up to \(5.8\times\) throughput speedup under SGLang serving.
Read the full abstract ▾
All authors Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, Linfeng Zhang
Computation and Language
Scaling Laws for Agent Harnesses via Effective Feedback Compute
Xuanliang Zhang, Dingzirui Wang, Keyan Xu et al. (+2 more)
Agent harnesses shape
language-model performance by controlling tool use, feedback, verification,
memory, and repair. Yet raw test-time expenditure, such as
tokens, tool calls, wall time, or cost, cannot distinguish useful feedback from redundant or unstable interaction. We introduce Effective Feedback Compute (EFC), a trace-level scaling coordinate for informative, valid, non-redundant, and retained feedback. We further define Estimated-EFC, NRS-EFC, harness efficiency $\eta$, and task-demand normalization for realistic traces and heterogeneous tasks. Across synthetic, real, held-out, and prospective evaluations, EFC-based coordinates outperform raw-compute baselines and SAS. Oracle-EFC/$D_{\mathrm{task}}$ reaches $R^2=0.99$ in controlled scaling, and NRS-EFC/$D_{\mathrm{task}}$ reaches $R^2=0.93$ on real traces where raw compute has near-zero or negative fit. Finally, \ours uses EFC as a companion control layer for existing harnesses, improving mean pass rate from $61.2\%$ to $68.2\%$ while reducing mean raw cost from $213.8$ to $85.1$ under matched settings. These results suggest that harness scaling depends on durable, task-sufficient feedback rather than raw computation alone.
Read the full abstract ▾
All authors Xuanliang Zhang, Dingzirui Wang, Keyan Xu, Qingfu Zhu, Wanxiang Che
Computation and Language
Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?
Pawel Batorski, Abtin Pourhadi, Jerzy Sarosiek et al. (+2 more)
Large
language models are highly sensitive to
prompts, but this sensitivity is usually studied through task-relevant instructions, demonstrations, or
reasoning cues. In this paper, we study a different form of
prompt sensitivity: whether
prompts that are semantically unrelated to the task can nevertheless steer model behavior. We call them spurious
prompts and show their surprising efficacy. We also propose a simple black-box search procedure for discovering them. Across
reasoning and question-answering
benchmarks, using models ranging from 0.8B to 27B parameters and spanning three model families, we show that spurious
prompts can improve performance, often matching or outperforming standard prompting baselines and task-aware
prompt optimization. We further show that they can steer models toward unintended behaviors, such as repeatedly selecting the first answer option, producing incorrect answers, returning an even,
prime or small number without explicitly instructing the model to do so. These findings reveal a new kind of
prompt sensitivity: LLMs can be systematically steered by
prompts that are unrelated to the task they are asked to solve. Our code is available at https://github.com/Batorskq/spurious
Read the full abstract ▾
All authors Pawel Batorski, Abtin Pourhadi, Jerzy Sarosiek, Przemyslaw Spurek, Paul Swoboda
Computation and Language · Cross-listed from cs.AI
Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems
Lorenz Kutschka and Bernhard Geiger
Large
language models in Agentic AI systems consume tool schemas and execution results and emit tool invocations as structured data. The default
language for that exchange, JSON, was designed for application-to-application interchange rather than
token efficiency, so its structural elements impose substantial
token overhead. Recent work proposes
token-optimized alternatives such as TOON (
Token-Oriented Object Notation) and TRON (
Token Reduced Object Notation) as more compact replacements, but these formats have been evaluated only on isolated comprehension or generation tasks. Whether their
token reductions hold inside end-to-end agentic loops therefore remains an open question. We evaluate TOON and TRON on four agentic
benchmarks (BFCL, MCPToolBenchPP, MCP-Universe, StableToolBench) and five open-weight LLMs, decoupling input compression from output compression to measure comprehension and generation independently. TRON reduces
tokens by up to 27% with accuracy within 14pp of the JSON baseline. TOON achieves up to 18% reduction at a similar 9pp accuracy cost, but additionally cascades on multi-turn parsing failures and collapses parallel tool-call output for most models. The code is available at: https://github.com/lkutschka/notation-matters
Read the full abstract ▾
Computation and Language
EviLink: Multi-Path Schema Linking with Uncertainty-Guided Evidence Acquisition for Large-Scale Text-to-SQL
Huawei Zheng, Sen Yang, Zhaorui Yang et al. (+11 more)
Schema linking is a difficult and important step in large-scale Text-to-SQL, where systems must identify a compact yet sufficient schema context from large and ambiguous databases. Existing methods often treat schema linking as deterministic selection around a single SQL path, but complex questions may admit multiple valid realizations with different schema needs. We reframe schema linking as uncertainty-aware schema-need inference over multiple plausible SQL paths, where the system distinguishes required schema items from path-dependent uncertain ones and acquires evidence only where needed. We instantiate this reframing with EviLink, which combines multi-hypothesis schema grounding with uncertainty-guided evidence acquisition. Experiments on BIRD-Dev and Spider2-Snow show that this perspective improves the balance among schema completeness, schema relevance, and
token cost. On Spider2-Snow, EviLink achieves 93.04% field-level strict recall rate, uses 116.55K average
tokens, and improves downstream SQL generation under a fixed generator.
Read the full abstract ▾
All authors Huawei Zheng, Sen Yang, Zhaorui Yang, Yuhui Zhang, Haozhe Feng, Haoxuan Li, Xuan Yi, Chao Hu, Defeng Xie, Chen Hou, Danqing Huang, Wei Chen, Peng Chen, Dazhen Deng
Computation and Language · Cross-listed from cs.AI
GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents
Johannes Moll, Jean-Philippe Corbeil, Jiazhen Pan et al. (+4 more)
LLM
agents acting in structured environments fail in operational rather than conversational ways, and reliability depends on procedural knowledge of the environment. Prior self-improvement methods accumulate natural-
language guidance without checking that each new item preserves previously correct behavior, so a note that fixes one trajectory can silently regress another. We introduce GRASP (Gated Regression-Aware Skill Proposer), which treats
agent improvement as a sequence of edits to a bounded skill library, admitting each candidate only if it produces a net improvement on a balanced held-out probe under a hard regression budget. We evaluate GRASP across five base models on two FHIR-based clinical
benchmarks, which score procedural reliability against FHIR state rather than clinical correctness or patient outcomes. On MedAgentBench, GRASP lifts gpt-oss-120b from 40.6% to 88.8%, exceeds the strongest of five self-improvement baselines by 21.0 points, and improves every other base model by 17.2 to 40.3 points. Ablations attribute the gain to comparative proposal generation, the acceptance gate, and the hard regression budget rather than to skill writing itself, which without validation is no better than using no skills. Granting the same acceptance gate to all five baselines lifts each of them in-domain and none of them out of distribution, isolating the gain to the gate applied to a bounded, editable library rather than to held-out validation itself. The mechanism helps in non-clinical environments where tasks recur with verifiable structure and is flat where the action space is open-ended. Frozen libraries transfer across models and across
benchmarks that share a tool-calling convention and degrade under interface mismatch.
Read the full abstract ▾
All authors Johannes Moll, Jean-Philippe Corbeil, Jiazhen Pan, Martin Hadamitzky, Daniel Rueckert, Lisa Adams, Keno Bressem
Computation and Language
Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese
Wajdi Zaghouani, Kholoud K. Aldous and Yicheng Gao
When Large
Language Models (LLMs) are deployed in Chinese-
language settings, a troubling pattern emerges: safety systems that work well in English break down. These systems struggle to cross linguistic and cultural bound-aries, leaving models exposed to
adversarial prompts that exploit Chinese-specific evasion techniques, including Pinyin romanization, character decomposition, internet slang, and hedging tone. To address this gap, we introduce ChiSafe-PAS (Chinese Safety Pilot Annotation Set), a human-annotated
benchmark of 1,897
adversarial Chinese
prompts spanning four high-stakes domains: self-harm and violence, drug and illicit trade, fraud, and satire. Of these, 1,544 entries carry complete gold-standard annotations: a 3-class response label (REFUSE, SAFE-REDIRECT, RESPOND), a nine-category obfuscation taxonomy, a risk-level rating, and annotator rationale. We describe the
dataset design, annotation process, and obfuscation taxonomy in detail. Our primary goal is practical: to give the research community a high-quality, culturally grounded resource for benchmarking LLM safety
alignment. In doing so, we engage three broader tensions in the field: the blurring boundary between training and evaluation data, the need for domain coverage grounded in real-world risk, and the limits of scale as a substitute for cultural expertise.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Opir: Efficient Multi-Task Safety Classification for Toxicity, Jailbreaks, Hate Speech, and Harmful Content
Ihor Stepanov and Aleksandr Smechov
Real-time safety filtering for large
language model (LLM) applications requires classifiers that can detect unsafe
prompts, toxic
language, jailbreak attempts, and unsafe responses without the cost profile of large guardrail models, and that can distinguish benign sensitive text from genuinely covert harmful content. In this paper, we introduce Opir, a family of encoder-based guardrail models built on the GLiClass architecture. Opir includes multi-task models for binary safe/unsafe classification, multi-label toxicity classification, jailbreak classification, and zero-shot unsafe
prompt and response categorization. We also release edge variants with fewer than 100M parameters dedicated to binary safe/unsafe categorization. The models are trained on a three-level taxonomy containing 996 categories across 16 top-level labels, 126 mid-level labels, and 854 leaf labels. Opir's training data combines taxonomy-grounded unsafe
prompts, adversarially mined hard negatives, benign safety-preserving examples, generated response examples,
multilingual translations, and portions of the Aegis2 and WildGuard training subsets. We also open-sourced an evaluation harness that supports GLiClass and GLiNER2 backends as well as decoder-based models, and covers binary safety classification, multi-label categorization, toxicity, jailbreak detection,
prompt safety, response safety, response refusal, and
prompt subcategory views across public
benchmark families. Across an expanded comparison spanning 12 safety-classification tasks and 17 category tasks against eight contemporary guardrail systems – including both GLiNER2-based and generative guardrail models – Opir variants are competitive on or ahead of the strongest open-weight baselines on the majority of
benchmark datasets while operating with a substantially smaller deployment footprint.
Read the full abstract ▾
Computation and Language
Beyond Math and Code: Lightweight Corpus-Grounded Process Rewards for Factual Question Answering
Shicheng Fan, Haochang Hao, Dehai Min et al. (+10 more)
Process supervision during reinforcement learning (RL) post-training matters in factual question answering (QA) because responses receiving positive response-level rewards can still contain sentence-level factual errors. Unlike math and code, factual QA lacks inexpensive programmatic checks, making reward computation a bottleneck. Existing methods rely on neural verifiers to score individual sentences, requiring extensive model inference as RL repeats these checks across many sampled responses at every update. We therefore introduce CorVer (Corpus Verify), a lightweight training-time approach to process supervision that derives sentence-level rewards from corpus co-occurrence statistics. A 0.5B extractor identifies subject-object pairs, indexed corpus queries supply their co-occurrence counts, and the resulting sentence rewards are assigned to the corresponding
tokens for RL. On Qwen3-4B and Qwen3-8B, CorVer reduces mean complete training time by 5.5-10.4 times relative to the four factuality-RL baselines. Across the four models evaluated against these baselines, CorVer achieves the highest accuracy in 17 of 20 model-
benchmark settings. CorVer outperforms the unmodified models in all 30 standard factual-QA settings (six models from three families across five
benchmarks) and remains effective on two additional multi-hop QA
datasets.
Read the full abstract ▾
All authors Shicheng Fan, Haochang Hao, Dehai Min, Weihao Liu, Hanrong Zhang, Lingwei Wei, Henry Peng Zou, Chengquan Guo, Jie Yang, Honghui Bao, Zhiwei Liu, Lu Cheng, Philip S. Yu
Computation and Language
Classification of non-analyzable word types in web documents to implement an effective Korean e-learning system
Sang-Taek Park, Ae-Lim Ahn, Eric Laporte et al. (+1 more)
E-learning systems should deliver contents that reflect various phenomena of the
language as it is used. In addition to formal Korean, e-learning systems that would include real-world Korean expressions such as those in web documents, mobile text messages, or twitter posts, would be useful to high-level learners. We construct two types of corpora: one is made of formal documents like online news articles; the other is made of informal documents like customer reviews about new products in web blogs. By comparing these corpora, we show how expressions differ in these two types of corpora. We survey the main characteristics of the informal corpus. Given that a significant proportion of text is informal, we propose Local Grammar Graphs (LGG) as an appropriate model to treat them effectively in Korean e-learning systems.
Read the full abstract ▾
All authors Sang-Taek Park, Ae-Lim Ahn, Eric Laporte, Jee-Sun Nam
Computation and Language
Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR
Debajyoti Mazumder, Divyansh Pathak, Prashant Kodali et al. (+3 more)
Large
language models often exhibit a substantial gap between their performance in English and in lower-resourced
languages on equivalent knowledge queries—a cross-lingual consistency issue that remains underexplored for Indian
languages and their code-mixed counterparts. To study this gap, we introduce IndicKLAR, an Indic extension of the KLAR-CLC
benchmark covering 18 of the 22 scheduled Indian
languages. For 11 widely used
language pairs, we additionally provide code-mixed variants. Both monolingual and code-mixed inputs verified by native speakers. This three-way
alignment enables us to examine how knowledge recall consistency varies across English, code-mixed, and native Indian
language inputs. Across nine open-weight models, we find that the accuracy gap between native-
language and English inputs can reach $\sim$0.50, while code-mixed inputs substantially reduce this gap, bringing performance within $\sim$0.05 of English without any model-level intervention. Motivated by this finding, we evaluate several prompting strategies that differ in how explicitly
language conversion is exposed: a two-stage translate-then-answer setup, a one-stage joint translation-and-answer
prompt, and Translate-in-Thought (TinT)—a single-step strategy in which the model internally converts the input and outputs only the final answer. Across the native $\rightarrow$ code-mixed $\rightarrow$ English performance trajectory, we observe a consistent flip point—the transition from incorrect to correct prediction—between the native and code-mixed settings. Notably, this pattern holds both when the code-mixed representation is explicitly provided as input or when the model is prompted to convert internally using TinT.
Read the full abstract ▾
All authors Debajyoti Mazumder, Divyansh Pathak, Prashant Kodali, Aditya Joshi, Akshay Agarwal, Jasabanta Patro
Computation and Language
Predicting Causal Effects from Natural Language Queries using Structured Representations
Giuliano Martinelli, Piriyakorn Piriyatamwong, Abelardo Carlos Martinez Lorenzo et al. (+7 more)
Randomized controlled trials are a cornerstone of medicine and the social sciences as they enable reliable estimates of causal effects. However, they are costly and time-consuming to conduct, motivating interest in predicting causal effects from existing experimental evidence. Recent advances in large
language models (LLMs) have demonstrated strong performance on knowledge-intensive tasks, raising the question of whether these models can be used for
forecasting causal effect sizes. To investigate this, we introduce Query2Effect, a new large-scale
benchmark consisting of more than 72,000 natural
language questions aligned with experiment descriptions, created to simulate realistic information-seeking scenarios by varying query specificity along dimensions of implicitness, abstraction, and ambiguity. We then propose a two-step framework that first generates a synthetic structured representation of a query before predicting effect size using a supervised encoder model. Experiments show that finetuning plays a crucial role in improving prediction performance, with absolute error reducing by -27% up to -71% compared to prompted out-of-the-box LLMs, and that our two-step framework is beneficial for out-of-domain generalization, highlighting the benefits of separating semantic interpretation from numerical effect estimation.
Read the full abstract ▾
All authors Giuliano Martinelli, Piriyakorn Piriyatamwong, Abelardo Carlos Martinez Lorenzo, Jasmin Baier, Riccardo Orlando, Satvik Garg, Sharif Kazemi, Linxi Wang, Arianna Legovini, Samuel Fraiberger
Computation and Language
Entity-Collision: A Stratified Protocol for Attributing Retrieval Lift in Agent Memory
Youwang Deng
End-to-end
agent-
memory benchmarks report a single hit@k per retriever, confounding lexical leakage (uncontrolled query/gold/distractor entity overlap) with tag-mixing (preferences, services, tools averaged together). We propose entity-collision, a system-agnostic protocol that pins the BM25 floor by construction – every distractor shares the answer's entity
tokens – and stratifies queries by discriminator tag, so any lift over BM25 is attributable to the embedder. Applied to an open-source
agent-
memory testbed across 5 tags x 3 embedders x 5 collision degrees with paired-bootstrap 95% CIs, the protocol reveals a two-axis pattern: a 256-d hash trigram helps only on closed-vocabulary lexical tags at deep collision; MiniLM-384 dominates both axes; and a 2.7x-parameter BGE-large does not uniformly improve on MiniLM – it wins on intent-style queries but loses on lexical ones. Encoder capacity alone is not the binding constraint. The synthetic intent-tag null replicates on LongMemEval (n=500) as a single-session-preference recall cliff. Adaptive vector-weight routing on LoCoMo is a measured null: 11.7pp of oracle headroom exists, but no signal we tested recovers it. All 26 result tables and 37 reproduce scripts are version-controlled and verified by a public registry; the protocol is exercised on a deterministically governed
memory testbed (event-sourced decision log, DAG-state-machine schema lifecycle) so every reported CI is reproducible byte-for-byte from the ingest stream.
Read the full abstract ▾