Computation and Language
Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals
Akindoyin Akinrele and Shreyank N Gowda
Prompt injection poses a critical threat to the safe deployment of large
language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of
prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and
transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system
prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings.
Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating
prompt injection defences under realistic operational constraints. Code will be released.
Read the full abstract ▾
Computation and Language
PashtoTTS-Bench: automated screening for low-resource non-Latin-script text-to-speech
Hanif Rahman
Text-to-
speech (TTS) evaluation for low-resource non-Latin-script
languages can fail when it relies on a single ASR round-trip word error rate (WER). A system may produce no audio, speak a neighbouring
language, preserve target script text only in an ASR transcript, or sound unnatural to native listeners. We introduce INSV (Intelligibility, Naturalness, Script fidelity, and Verification), a reporting framework that separates these cases. This paper reports INSV-A, the automated screening subset: synthesis completion, ASR WER/CER, transcript Script Fidelity Rate, and audio
language identification. Native MOS and phonetic annotation are specified but not claimed in this release. We instantiate INSV-A as PashtoTTS-Bench, a dated
benchmark for Pashto TTS. The April-May 2026 run evaluates Edge GulNawaz, Edge Latifa, OmniVoice clone, OmniVoice auto, and an Urdu negative control on 200 FLEURS and 200 filtered Common Voice 24
prompts. Under the independent omniASR_CTC_300M_v2, OmniVoice auto has the lowest WER (24.1% FLEURS, 27.4% CV24), followed by Edge GulNawaz (32.8%, 39.5%), Edge Latifa (35.6%, 47.7%), and OmniVoice clone (45.4%, 34.8%). WER below the natural-
speech baseline reflects clean synthetic audio and should not be read as better than native
speech. Whisper Large V3 returns 0.0% Pashto labels on checked Pashto TTS audio, while MMS-LID-4017 and SpeechBrain VoxLingua107 separate Pashto outputs from the Urdu control. The release provides provider metadata, per-sentence scores, LID audits, failure logs, and scripts for adding systems.
Read the full abstract ▾
Computation and Language
Recon: Reconstruction-Guided Reasoning Synthesis for User Modeling
Alan Zhu, Mihran Miroyan, Carolyn Wang et al. (+4 more)
User modeling aims to use
language models (LMs) to mimic an individual's behavior from a corpus of past context-action pairs (e.g., conversation turns), enabling the simulation of users in settings like behavioral science, human-AI collaboration, and
market research. Recent approaches augment these corpora with synthesized
reasoning traces, typically generated by conditioning on both context and action. However, such conditioning constitutes post-hoc rationalization rather than
reasoning: the trace is guaranteed to justify the action, but may not encode the underlying latent causal decision paths. We propose Recon, which uses action reconstruction to score
reasoning traces by their predictive power: given a context and candidate
reasoning, a reconstruction model predicts the action, and reconstruction fidelity determines
reasoning quality. Across four domains, Recon achieves a 54.7% win rate over Backward Synthesis, a standard post-hoc rationalization baseline. Further, we find that training a
reasoning synthesis model with rewards derived from Recon improves downstream user modeling performance, achieving a win rate of up to 70.0% over baselines. We further show that Recon-synthesized
reasoning transfers across models, and improves user modeling beyond the reconstruction model. Our work demonstrates that post-hoc rationalization is insufficient for
reasoning synthesis, and that useful and interpretable
reasoning should naturally elicit the action from the context.
Read the full abstract ▾
All authors Alan Zhu, Mihran Miroyan, Carolyn Wang, Andrew Zhou, Lisa Dunlap, Narges Norouzi, Joseph E. Gonzalez
Computation and Language · Cross-listed from cs.LO
MerLean-Prover: A Recursive Looping Harness for Lean 4 Theorem Proving
Jinzheng Li, Zeru Zhu and Yuanjie Ren
MerLean-Prover is an end-to-end Lean4
theorem prover that replaces sorry declarations with kernel-checkable proofs. It is built from three
agent types (Planning, Check, and Lean) composed by a recursive outer loop whose unit of revision is the proof plan itself, and uses no fine-tuning, no custom RL objective, and no
theorem-specific scaffolding. On FormalQualBench, a
benchmark of 23 PhD-qualifying-exam
theorems, MerLean-Prover solves 10/23, surpassing the strongest published open-source baseline (OpenGauss, 8/23). On Putnam2025, the same harness closes 12/12 with substantially lower total wall-clock than the next-best system that closes the full set. The harness also transfers to smaller models: Sonnet closes all four tested FormalQualBench problems, and Haiku closes the two short ones. These results suggest that harness design is a central factor in end-to-end Lean4
theorem proving, alongside raw model capability, and that a relatively simple harness can already be effective.
Read the full abstract ▾
Computation and Language
Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generation
Zixuan Yang, Yiqun Chen, Wei Yang et al. (+7 more)
Reinforcement learning in open-ended long-form generation is challenging because reliable reference answers and automatic metrics are often unavailable. Existing rubric-based methods typically rely on pointwise LLM-as-a-judge scoring, but absolute scores are difficult to calibrate across complex responses, may provide weak discrimination among same-query rollouts, and can become saturated during optimization. We propose Tournament-GRPO, a group-wise reward framework that converts rubric-guided LLM judgments into relative rewards through repeated multi-round tournaments among same-query rollouts. Tournament-GRPO compares candidates within groups, accumulates tournament outcomes, and normalizes them into group-wise rewards for GRPO training. Experiments on Deep Research Bench show that Tournament-GRPO consistently outperforms existing reward-design baselines, achieving a 4.52-point overall-score improvement over the strongest baseline. Further analyses show that tournament rewards provide a favorable effectiveness–efficiency trade-off and that tournament design affects training dynamics. These results suggest that rubric-guided tournament comparison provides an effective reward signal for reinforcement learning in open-ended long-form generation.
Read the full abstract ▾
All authors Zixuan Yang, Yiqun Chen, Wei Yang, Erhan Zhang, Zihan Shen, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Jiaxin Mao
Computation and Language · Cross-listed from cs.AI
LELA: An End-to-end LLM-based Entity Linking Framework with Zero-shot Domain Adaptation
Samy Haffoudhi, Nikola Dobričić, Fabian Suchanek et al. (+1 more)
Entity linking is a key component of many downstream NLP systems, yet existing approaches are often tied to the specific target knowledge bases and domains, limiting their real world application. In this paper, we extend LELA, a modular and domain-agnostic LLM-based entity disambiguation method, into a practical Python library that integrates zero-shot Named Entity Recognition (NER) -thereby providing a complete end-toend pipeline for entity-linking in real-world usage. We provide experimental results validating LELA's performance and robustness across diverse entity linking settings. In our demo, users can play with the system on their own input texts.
Read the full abstract ▾
All authors Samy Haffoudhi, Nikola Dobričić, Fabian Suchanek, Nils Holzenberger
Computation and Language
JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
Jiho Jin, Junho Myung, Juhyun Oh et al. (+5 more)
As large
language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural contexts, from drafting personal communications to brainstorming creative ideas. These tasks are inherently cultural: they require contextual appropriateness, symbolic resonance, and tacit cultural expectations that native speakers draw on instinctively, meaning that a response can be factually plausible yet unmistakably wrong to a local reader. Existing cultural
benchmarks have treated culture as a flat set of facts via fact verification or norm entailment methods, and have adopted LLM-as-a-Judge without examining whether they can capture such thick cultural errors. To address this gap, we present JuICE (
Benchmark for LLM-Judge in Identifying Cultural Errors), a
multilingual dataset of 7,470 span-level annotations of cultural and linguistic errors in long-form LLM responses. It covers 1,050 query-response pairs from four countries (the United States, South Korea, Indonesia, and Bangladesh), in both English and their countries' main
languages. Using JuICE, we find that even the strongest LLM-judge achieves only an F1 of 0.52 in the erroneous span detection task. Furthermore, LLM-judges consistently miss thick cultural errors that local residents readily identify. Our findings suggest that robust cultural evaluation must move beyond surface-level detection toward frameworks that account for the depth and situatedness of cultural meaning.
Read the full abstract ▾
All authors Jiho Jin, Junho Myung, Juhyun Oh, Junyeong Park, Rifki Afina Putri, Sunipa Dev, Vinodkumar Prabhakaran, Alice Oh
Computation and Language
AlbanianLLMSafety: A Safety Evaluation Dataset for Large Language Models in Albanian
Wajdi Zaghouani, Kholoud K. Aldous and Isra Fejzullaj
Safety evaluation of Large
Language Models (LLMs) has largely focused on high-resource
languages, leaving low-resource
languages critically underserved. We present AlbanianLLMSafety, the first publicly available safety evaluation
dataset for LLMs in Albanian, a linguistically distinct low-resource
language with approximately 7.5 million speakers across Albania, Kosovo, North Macedonia, and the diaspora. The
dataset contains 2,951
prompts spanning 11 safety categories, including self-harm, violence, racist content, child exploitation, and radicalization, with an average of 268
prompts per category. Each
prompt is provided in Albanian with an English reference translation and a detailed category label. This resource addresses a significant gap in safety evaluation infrastruc-ture for low-resource
languages and provides an essential
benchmark for developing safer, more inclusive LLMs. The
dataset will be provided upon request to support safety evaluation, fine-tuning, red-teaming, and guardrail development for Albanian-speaking communities.
Read the full abstract ▾
Computation and Language
Efficient Agentic Reinforcement Learning with On-Policy Intrinsic Knowledge Boundary Enhancement
Dingwei Chen, Zefang Zong, Zhipeng Ma et al. (+5 more)
Agentic reinforcement learning (RL) has proven effective for training LLM-based
agents with external tool-use capabilities. However, we identify that agentic RL training induces increasing redundant tool calls and blurs the model's intrinsic knowledge boundary, where the model fails to distinguish when tools are needed versus when parametric knowledge suffices. Existing solutions based on reward shaping create coarse-grained optimization targets that tend to incentivize indiscriminate tool-call suppression, leading to reward hacking. In this paper, we propose AKBE (Agentic Knowledge Boundary Enhancement), an on-policy method that dynamically probes the model's intrinsic knowledge boundary through dual-path (with-tool and no-tool) rollouts during training. We define the knowledge boundary as the per-instance determination of whether tools are required and the minimum tool calls necessary. By comparing correctness across paths, AKBE categorizes trajectories and constructs targeted supervisory signals that guide efficient tool-use patterns for each question. These signals are integrated seamlessly into the agentic RL training loop. Experiments on seven QA
benchmarks demonstrate that AKBE improves task accuracy by +1.85 on average and reduces tool calls by 18% over standard agentic RL, yielding 25% higher tool productivity without any accuracy-efficiency trade-off. Further analysis suggests its plug-and-play compatibility across different RL
algorithms and the mechanism of each signal category. Our code is available at https://github.com/CuSO4-Chen/AKBE.
Read the full abstract ▾
All authors Dingwei Chen, Zefang Zong, Zhipeng Ma, Leo Luo, Yang Li, Chengming Li, Peng Chen, Jie Jiang
Computation and Language
KZ-SafetyPrompts: A Kazakh Safety Evaluation Prompt Dataset for Large Language Models
Wajdi Zaghouani, Shimaa Amer Ibrahim, Aruzhan Muratbek et al. (+2 more)
Kazakh is underrepresented in resources for evaluating the safety behavior of large
language models. We present KZ-SafetyPrompts, a Kazakh
prompt dataset for safety evaluation across eleven categories covering common risk areas such as self-harm, violence, child exploitation, sexual content, racist content, radicalization, and regulated goods or illegal activities. The
dataset contains 5,717
prompts written natively in Kazakh (Cyrillic), organized by category, with English translations for cross-lingual analysis.
Prompts resemble realistic user queries, often in a teen or child style, and are phrased as intent
prompts without procedural instructions. We document the writing protocol, labeling procedures (including borderline-case decision rules), and quality-control steps (schema standardization, completeness checks, and deduplication). We also align the categories with widely used safety taxonomies to support integration with existing evaluation pipelines. Baseline results with GPT-4o show an overall refusal rate of 28.2%, varying from 5.5% to 53.8% across categories, indicating that Kazakh
prompts expose category-specific safety gaps not captured by English-only evaluation.
Read the full abstract ▾
All authors Wajdi Zaghouani, Shimaa Amer Ibrahim, Aruzhan Muratbek, Olzhasbek Zhakenov, Adiya Akhmetzhanova
Computation and Language
Accountable Human-AI Deliberation with LLMs: Scaling Collective Intelligence through Symbiotic Scaffolding
Wajdi Zaghouani
Large
language models (LLMs) can support democratic deliberation at scales previously constrained by turn-taking and facilitation bandwidth. Recent work shows that LLM-generated group statements are often preferred over human-mediated outputs, while theoretical analyses argue that LLMs relax the simultaneity constraints limiting collective intelligence. Yet pure LLM mediation risks collapsing pluralism, over-optimizing for agreement, and undermining legitimacy when participants cannot contest how they are represented. We propose a symbiotic human-AI framework organized into three layers: observation and diversity amplification, facilitation with clause-level provenance, and human primacy for ratification. Our contributions include graded coverage, diversity, and erasure metrics with salience-aware weighting; a provenance pipeline combining cross-encoder similarity with causal knockout diagnostics; preference-conditioned trade-off control; equity-aware contestability workflows;
adversarial robustness tests; and an evaluation protocol with ablation designs informed by evidence of LLM-as-judge limitations. The result is a testable blueprint for deliberation technology that scales collective intelligence while preserving agency and legitimacy.
Read the full abstract ▾
Computation and Language
Beyond Questions: Evaluating the Stated Knowledge of LLMs
Luca Giordano and Simon Razniewski
Parametric knowledge in large
language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge
benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that
benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation
prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer
retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a
benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of
language models and analyze the effects of
reasoning effort, model scale,
prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at bequ.scads.ai, the
benchmark website.
Read the full abstract ▾
Computation and Language
DunbaaBERT: From Sacrifice to Semantics
Iffat Maab, Waleed Jamil and Raphael Schmitt
Large
language models have achieved strong performance across many NLP tasks, yet Urdu remains comparatively underexplored due to limited resources and fragmented evaluation settings. To address this gap, we introduce DunbaaBERT, a family of Urdu RoBERTa-base models trained from scratch with Byte-BPE vocabularies of 32k, 52k, and 96k
tokens on a deduplicated 17GB Urdu corpus. We evaluate DunbaaBERT across intrinsic and downstream Urdu NLP
benchmarks covering linguistic acceptability, news classification, offensive
language detection, and sentiment analysis while analyzing vocabulary-size effects on performance and efficiency trade-offs. Across
benchmarks, the DunbaaBERT variants achieve competitive performance against strong
multilingual baselines while consistently maintaining favorable efficiency trade-offs. Interestingly, larger vocabularies do not consistently improve downstream effectiveness, with DunbaaBERT$_{\text{32k}}$ repeatedly providing the strongest overall efficiency profile. Overall, our results demonstrate that carefully curated Urdu-specific encoder models can remain highly competitive despite comparatively compact model and training scales. All models are released under the MIT license.
Read the full abstract ▾
Computation and Language
Reasoning Depth and Environment Complexity: A Controlled Study of RLVR Data Allocation across Logical Reasoning Tasks
Yihua Zhu, Qianying Liu, Fei Cheng et al. (+4 more)
Reinforcement learning with verifiable rewards (RLVR) has become central to post-training
reasoning models, yet a key limitation of existing studies is their narrow view of the
reasoning space: difficulty is treated as
reasoning depth alone, and reward is concentrated on forward deductive state tracking. We instead characterize the
reasoning space along two dimensions. Difficulty. Beyond
reasoning depth, we study environment complexity, where models must identify the correct path amid distractors and interacting structures. Rewarded
reasoning form. We consider four abilities core to real-world
reasoning: deductive state tracking, abductive recovery of hidden events or facts, inductive rule induction, and analogical transfer. To disentangle these factors, we construct a synthetic knowledge-graph environment with controlled pre- and post-training distributions, where each instance varies along depth, complexity, and task family. Three findings emerge: joint depth-complexity coverage outperforms single-axis recipes;
reasoning families respond non-uniformly, with abductive
reasoning degrading outside the RL-covered region and task correlations clustering into deductive-abductive and inductive-analogy pairs; and uniform mixing outperforms staged curricula under a fixed budget. We also find that recent off-the-shelf models exhibit the same deductive-over-abductive asymmetry, suggesting that this gap is not merely an artifact of our controlled setup.
Read the full abstract ▾
All authors Yihua Zhu, Qianying Liu, Fei Cheng, Jiaxin Wang, Akiko Aizawa, Sadao Kurohashi, Hidetoshi Shimodaira
Computation and Language
Learning to Adapt SFT Data for Better Reasoning Generalization
Lisong Sun, Li Wang, Chen Zhang et al. (+4 more)
Large
language models (LLMs) have achieved remarkable progress, with post-training playing a crucial role in enhancing their
reasoning capabilities. Among post-training paradigms, supervised fine-tuning (SFT) is widely used: it leverages external data to provide dense supervision and enables efficient training. However, directly fine-tuning on expert data can hurt generalization when the data distribution is mismatched with the target model's own distribution. In this work, we propose Data Adaptation for
Reasoning Tuning (DART), which formulates the use of a fixed, potentially distributionally misaligned SFT
dataset as an optimization problem over demonstration transformations. DART trains a mapper model with reinforcement learning to convert original SFT data into model-adapted supervision that better matches the target model's distribution and learning preferences. The transformed data are then used for SFT, allowing the target model to better exploit external supervision. Experiments across multiple models and
datasets show that DART improves generalization, achieves higher training efficiency than direct RL, and helps models surpass standard SFT. Our code is available at https://anonymous.4open.science/r/DART525E50D.
Read the full abstract ▾
All authors Lisong Sun, Li Wang, Chen Zhang, Jinyang Wu, Kui Zhang, Tianhao Peng, Wenjun Wu
Computation and Language
Are Video Models Zero-Shot Learners and Reasoners in Education? EduVideoBench, A Knowledge-Skills-Attitude Benchmark for Educational Video Generation
Unggi Lee, Hoyoung Ahn, Yoon Choi et al. (+16 more)
Video generation models (VGMs) are rapidly entering classrooms, yet existing
benchmarks evaluate only perceptual quality, intrinsic faithfulness, generic safety, or
video as a
reasoning medium, and none assesses whether the outputs are educationally valid. In this work, we present EduVideoBench, the first balanced
benchmark in the education domain, grounded in the Knowledge-Skills-Attitude (KSA) framework so that pedagogical adequacy and educational safety are evaluated jointly rather than as ad-hoc quality dimensions. Across five frontier VGMs, our results show substantial room for improvement across knowledge, skills, and attitude before they are classroom-ready. We complement this with a qualitative analysis of expert comments, finding that educational validity is multi-component, where a single misaligned element such as pacing, legibility, or notation can invalidate an otherwise correct
video. We hope EduVideoBench will guide the development of VGMs that are pedagogically grounded and safe for the classroom.
Read the full abstract ▾
All authors Unggi Lee, Hoyoung Ahn, Yoon Choi, Seonmin Eun, Jahyun Jeong, Seonmin Jin, Harmony Jung, Hye Jin Kim, Chaerin Lee, Hyunji Lee, Jeongjin Lee, Soohwan Lee, Young-Seok Oh, Jaehyeon Park, Sun-ok Ryu, Sunyoung Shin, Yoorim Son, Haeun Park, Yeil Jeong
Computation and Language
Learning Faithful Reasoning Processes with Spatio-Temporal Latent Dynamics for Large Language Models
Weijiang Lv, Wentong Zhao, Jiayu Wang et al. (+3 more)
Chain-of-Thought (CoT)
reasoning has significantly enhanced the
reasoning capabilities of large
language models (LLMs). However, existing reinforcement learning methods primarily optimize outcome-level rewards, providing limited supervision over the
reasoning process. As a result, models may produce shortcut-prone or unstable
reasoning trajectories even when the final answer is correct. We propose GeoFaith, a spatio-temporal latent dynamics framework for learning faithful
reasoning processes without process reward models. GeoFaith combines final-state latent uncertainty, which measures whether a completed trajectory lies in a well-supported
manifold region, with predictive-
entropy dynamics, which penalize abnormal uncertainty patterns such as premature collapse, oscillation, and prolonged uncertainty. These signals are incorporated as process regularization into GRPO, jointly optimizing answer correctness and
reasoning quality without step-level process annotations. Experiments show that GeoFaith consistently improves
reasoning accuracy, produces more efficient
reasoning trajectories, and outperforms both outcome-only reinforcement learning and process-supervision baselines.
Read the full abstract ▾
All authors Weijiang Lv, Wentong Zhao, Jiayu Wang, Yuhao Wu, Jiaheng Wei, Xiaobo Xia
Computation and Language
Telenor Nordics Customer Service self-help corpus
Mike Riess
This paper presents a
multilingual customer service self-help corpus comprising 1,122 manually validated documents in Finnish, Danish, Norwegian, and Swedish, totaling 274,599 words and 1,884,833 characters. The documents have been sourced from the public self-help pages of four Nordic telecommunications operators and subsequently filtered for person-identifiable information and relevance through a combined LLM and human annotation pipeline. Domain-specific
datasets for Nordic
languages remain scarce, particularly in customer service: a domain of growing importance for
retrieval-augmented generation, cross-lingual transfer learning, and emerging
agent-based service architectures. An analysis of the corpus reveals substantial variation in document length and structure across operators, reflecting distinct editorial strategies, as well as broad topical coverage spanning network hardware, mobile services, TV and streaming, billing, and account management. The
dataset is publicly available under a CC-BY-NC-SA-4.0 license at https://zenodo.org/records/20732652, intended to support reproducible research in Nordic NLP and information
retrieval.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
Zhengyu Hu, Zheyuan Xiao, Linxin Song et al. (+10 more)
LLM training increasingly relies on teacher-generated supervision, from synthetic responses to
reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a
token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 6 tasks show that SCAS consistently improves student performance, suggesting that effective
distillation should prioritize supervision matched to the current student rather than teacher strength alone.
Read the full abstract ▾
All authors Zhengyu Hu, Zheyuan Xiao, Linxin Song, Fengqing Jiang, Yuetai Li, Zhihan Xiong, Yue Liu, Junhao Lin, Yao Su, Lijie Hu, Kaize Ding, Teng Xiao, Radha Poovendran
Computation and Language
Uncertainty-Aware Budget Allocation for Adaptive Test-Time Reasoning
Manh Nguyen, Sunil Gupta and Hung Le
Sampling multiple responses improves
language model
reasoning, but uniform compute allocation is inefficient because easy questions are over-sampled while hard questions remain under-explored. We propose Uncertainty-Aware Budget Allocation (UAB), a concave integer optimization framework that reallocates a fixed sampling budget using uncertainty estimated from the initial samples themselves. In Phase-1, every question receives a small fixed number of generations. Their answer disagreement, measured by vote
entropy, provides a difficulty signal while these generations contribute to the final vote. In Phase-2, the remaining budget is allocated by a marginal-greedy
algorithm that optimally solves a concave coverage-maximization surrogate, concentrating samples on questions whose initial answers disagree. Across five open-weight models (1.5B–27B parameters) and five
reasoning benchmarks of varying difficulty, UAB improves average accuracy by $+2.3\%$ over uniform allocation, and achieves gains of up to $+5.5\%$ on individual
benchmarks, with the largest gains in low-resource settings. Moreover, UAB meets the target budget exactly and requires no auxiliary model or additional LLM calls. Code is publicly available at https://github.com/manhitv/UAB.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
Jiacheng Li, Jianchao Tan, Hongtao Xu et al. (+5 more)
The
Muon optimizer has recently offered a promising alternative to AdamW for large
language model training, leveraging
matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods,
Muon can become trapped in sharp local minima. In this work, we present MONA, an optimizer that bridges
Muon's orthogonalization framework with
curvature-aware acceleration. MONA adds an acceleration term directly into
Muon's gradient processing pipeline. This term is calculated from the exponential moving average of gradient differences. We provide a detailed convergence analysis for MONA, showing that the acceleration term introduces
curvature-sensitive corrections while preserving
Muon's spectral-norm regularization. Empirically, MONA achieves better convergence and downstream task performance compared to both
Muon and AdamW across three scales of Mixture-of-Experts pretraining, spanning from 1B to 68B parameters, with the largest model trained on 1 trillion
tokens. Furthermore, we conduct supervised fine-tuning on the MOE-68B-A3B model and evaluate it on general capability, mathematical
reasoning, and code generation
benchmarks, where MONA achieves SOTA performance.
Read the full abstract ▾
All authors Jiacheng Li, Jianchao Tan, Hongtao Xu, Jiaqi Zhang, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai
Computation and Language
Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics
Yuxuan Ye, Raul Santos-Rodriguez and Edwin Simpson
Reinforcement learning with evaluation metrics as rewards is widely used to enhance specific capabilities of
language models. However, for tasks such as factually consistent summarisation, existing metrics remain underdeveloped, limiting their effectiveness as signals for shaping model behaviour.While individual factuality metrics are unreliable, their combination can more effectively capture diverse factual errors. We leverage this insight to introduce an automated training pipeline that improves factual consistency in summaries by aggregating scores from different weak metrics. Our approach avoids the need for complex reward shaping by mapping scores to preferences and filtering out cases with high disagreement between metrics. For each source document, we generate lexically similar summary pairs by varying decoding strategies, enabling the model to learn from factual differences caused by subtle lexical differences. This approach constructs a high-quality preference
dataset using only source documents.Experiments demonstrate consistent factuality gains across models, ranging from early encoder-decoder architectures to modern large
language models, with smaller models reaching comparable factuality to larger ones.
Read the full abstract ▾
Computation and Language
ContextGuard: Structured Self-Auditing for Context Learning in Language Models
Hongbo Jin, Chi Wang, Haoran Tang et al. (+5 more)
Recent
benchmarks reveal that despite strong
reasoning capabilities, large
language models (LLMs) still struggle to faithfully apply complex contextual knowledge. These failures are often not wholesale
reasoning collapses: in context-rich tasks, models may follow the central
reasoning path while missing peripheral, persistent, or format-sensitive requirements.
Read the full abstract ▾
All authors Hongbo Jin, Chi Wang, Haoran Tang, Zhongjing Du, Xu Jiang, Jingqi Tian, Qiaoman Zhang, Jiayu Ding
Computation and Language
Generating Logically Consistent Synthetic Supply Chain Data with LLM-Driven Knowledge Graph Reasoning
Yunbo Long, Ge Zheng, Liming Xu et al. (+1 more)
Synthetic data offers a promising solution to two persistent barriers in supply chain analytics: data scarcity and data
privacy. However, for synthetic data to support operational simulation and decision-making, it must do more than reproduce the statistical distributions of real records, and also preserve the operational logic that governs supply chain processes, including the temporal orderings, mathematical dependencies, hierarchical taxonomies, and conditional rules that make a record operationally plausible. We consider this logic as the “physics” of supply chain data. Existing tabular generative models are primarily optimized for distributional fidelity and downstream predictive utility, and therefore often generate records that appear statistically realistic but violate fundamental operational constraints. This paper introduces \textbf{TabKG}, a knowledge-graph-guided framework for logically consistent synthetic supply chain tabular data generation. TabKG constructs a \textbf{Column Relationship Knowledge Graph (CR-KG)} to represent data operational dependencies. It uses a multi-LLM ensemble with majority voting to propose candidate relationships from column metadata, validates these relationships against real data to remove hallucinated or unsupported edges, and then uses the validated CR-KG to guide generation. Specifically, TabKG compresses the original table into independent columns, generates these columns using a latent
diffusion model, and deterministically reconstructs dependent columns according to the validated relationships, enforcing logical consistency by construction with respect to the discovered operational rules.
Read the full abstract ▾
All authors Yunbo Long, Ge Zheng, Liming Xu, Alexandra Brintrup
Computation and Language
Psychological Constructs in Shared Semantic Space
Hubert Plisiecki
Psychological constructs are often measured in separate instruments,
datasets, and research traditions, which makes direct comparison difficult. This paper proposes a framework for making such constructs semantically commensurate by representing and comparing them as directions in a shared word-
embedding space. Using Supervised Semantic Differential, we estimate construct-specific semantic gradients from text-outcome associations and project them onto theoretically motivated reference axes. As an initial test case, we use Valence, Arousal, and Dominance (VAD) as an affective coordinate system. First, we recover interpretable VAD directions from English word-level affective norms. Second, we project semantic gradients for 27 GoEmotions categories into this space and recover the expected organization of
emotions, especially along valence and arousal. Third, we apply the same procedure to Big Five personality domains and facets derived from IPIP-NEO-300 item-factor associations. Domain-level placements are broadly coherent, while facet-level results are more exploratory because they rely on sparse questionnaire text. The results suggest that
embedding spaces can support construct-level comparison across otherwise incommensurable psychological measurements, provided that semantic placements are assessed for stability and interpretability.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
Zeyi Huang, Xuehai He, LiLiang Ren et al. (+8 more)
We study Latent Recurrent
Transformer (LRT), a lightweight augmentation of autoregressive
transformers that reuses a high-level source-layer hidden state from the previous
token as recurrent
memory for the next
token. Because this state is already computed during ordinary decoding, LRT introduces a cross-
token, cross-layer latent pathway while preserving the standard
attention mechanism, KV-cache interface, and one model forward per generated
token. To pretrain this recurrence without sequentially unrolling the full sequence, we introduce interleaved parallel training: one full-sequence initialization forward constructs a shared buffer, followed by sequential refinement of disjoint position subsets with parallel computation within each subset. This provides every
token with recurrent-
memory-aware supervision at approximately 2x ideal
token compute. Across 1.3B- and 2.1B-parameter nanochat-style backbones and a wide range of training budgets, LRT improves both BPB and CORE under matched effective compute. Additionally, LRT outperforms two-forward PonderLM-2 and matches a three-loop
Transformer in BPB, while retaining one-forward-per-
token decoding with 9% latency overhead over the standard
Transformer.
Read the full abstract ▾
All authors Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, Yelong Shen
Computation and Language
SeDT: Sentence-Transformer Decision-Transformer Conditioning for Multi-Turn Conversation Reliability
Ramakrishna Vamsi Setti, Jagadeesh Rachapudi, Sachin Chaudhary et al. (+2 more)
Large
language models (LLMs) achieve impressive performance when a task is fully specified in a single turn, yet the same models lose up to 39% of that performance when the identical task is revealed incrementally across multiple turns, a phenomenon documented at scale as Lost in Conversation. Crucially, this collapse is almost entirely a reliability failure; the best case, the aptitude only falls 16%, while the unreliability more than doubles (+112%). We argue that the root cause is structural, a flat conversation history assigns equal implicit weight to every prior turn, giving the model no signal to distinguish a critical constraint from incidental dialog. We present SeDT Sentence-
transformer Decision-
Transformer, a training-free inference-time method that resolves this by importing return-to-go conditioning from offline reinforcement learning. SeDT annotates each conversation shard with a cumulative relevance score derived from three complementary semantic, lexical, and positional signals and presents the full annotated history to the model at the final turn, without weight changes, without training data, and without discarding context. Evaluated on the Lost-in-Conversation
benchmark in three LLMs and three generation tasks, SeDT outperforms the sharded baseline in all nine model-task combinations, with gains up to +37.7% in mean performance P and simultaneous reductions in unreliability in seven of the nine combinations. In short, telling the model which past turns matter is sufficient to substantially recover the performance lost in conversation.
Read the full abstract ▾
All authors Ramakrishna Vamsi Setti, Jagadeesh Rachapudi, Sachin Chaudhary, Praful Hambarde, Amit Shukla
Computation and Language
Yunbo Long, Haolang Zhao, Lukas Beckenbauer et al. (+2 more)
Post-trained LLMs are often optimized to produce helpful, polite, and accommodating responses. In
adversarial negotiation, however, such behavior can become a vulnerability: emotionally framed
language may influence an
agent's bargaining decisions in ways that conflict with its user's objectives. We therefore introduce EmoDistill, an offline framework for distilling emotional negotiation skills from LLM-LLM interactions into smaller
language-model
agents. Here, an emotional negotiation skill is a state-conditioned behavior that determines which explicit
emotion to invoke in a bargaining state and how to realize that
emotion as an effective negotiation utterance. EmoDistill learns these two components separately: an Implicit Q-Learning (IQL) selector learns which
emotion to express in each bargaining state, while a LoRA-adapted 7B policy learns
emotion-conditioned expression through Supervised Fine-Tuning (SFT) and Judge Policy Optimization (JPO). Across four
emotion-sensitive negotiation domains, the full EmoDistill policy achieves competitive utility and improves over vanilla and IQL-only baselines in most settings.
Emotion-free ablations show that removing the explicit
emotion channel substantially reduces overall negotiation utility, while transfer experiments reveal partial, domain-dependent transfer and robustness to unseen LLM counterparties.
Read the full abstract ▾
All authors Yunbo Long, Haolang Zhao, Lukas Beckenbauer, Liming Xu, Alexandra Brintrup
Computation and Language
Quality Without Usefulness: LLM-Generated XAI Narratives as Trust Heuristics Rather Than Decision Aids
Fabian Lukassen, Jan Herrmann, Christoph Weisser et al. (+3 more)
Prior work shows that Large
Language Models (LLMs) can transform Explainable AI (XAI) outputs into Natural
Language Explanations (NLEs) that score highly on quality metrics such as plausibility, coherence, and comprehensibility. But does explanation quality translate to practical usefulness? We investigate this question in a time-series energy
forecasting domain through five controlled experiments (2,730 judgments across 60 test instances), each operationalising a distinct facet of usefulness studied in the XAI literature. Holding NLE quality constant at the high levels established by a prior factorial study, we find that NLEs do not improve task accuracy on any of the five tasks, while inflating self-reported confidence. A placebic control shows that this confidence boost is driven by text presence rather than content. In an out-of-distribution detection task, NLEs reduce the LLM judge's ability to flag unreliable predictions, providing false reassurance that masks model failure. We characterise these findings as the Quality-Usefulness Gap and argue that evaluation of the XAI-to-NLE pipeline must extend beyond text-quality metrics to downstream task performance.
Read the full abstract ▾
All authors Fabian Lukassen, Jan Herrmann, Christoph Weisser, Alexander Silbersdorff, Benjamin Saefken, Thomas Kneib
Computation and Language
SEEK: Semantic Evidence Extraction via Adaptive ChunKing for Multilingual Fact-Checking
Gaurav Kumar, Babu Kumar, Ayush Garg et al. (+2 more)
Multilingual fact verification requires evidence that is both relevant and sufficiently complete for reliable factuality prediction. However, existing systems often rely on search snippets, sentence-level evidence, or locally segmented passages, which can miss decisive context and produce fragmented evidence. To overcome these limitations, we propose SEEK, a Semantic Evidence Extraction with an adaptive chunKing framework that constructs coherent evidence chunks from full fact-checking articles by identifying semantic topic transitions and preserving local verification context. The constructed chunks are encoded using a
multilingual encoder and then
multilingual LLMs are finetuned using LoRA adapter for veracity prediction. Experiments on X-FACT and RU22Fact show that SEEK improves macro-f1 by up to 10% over semantic chunking, 19% over sentence chunking, and 20% over search-snippet baselines. Evidence completeness and significance analyses further show that SEEK preserves richer verification context and enables more reliable
multilingual fact-checking.
Read the full abstract ▾
All authors Gaurav Kumar, Babu Kumar, Ayush Garg, Aditya Kishore, Jasabanta Patro