Computation and Language
A Diagnostic Benchmark for Sweden-Related Factual Knowledge
Jenny Kunz
Many Swedish
benchmarks are translations of US-centric
benchmarks and are therefore not suitable for testing knowledge that is particularly relevant, or even specific, to Sweden. We therefore introduce a manually written question-answering
benchmark specifically targeted at Sweden-related personalities and events, many of which receive very limited coverage in international media. Our annotators drew inspiration from a popular
radio program featuring public figures from culture and media, as well as major sports events in Sweden. The
dataset can be used to measure factual recall across models of varying sizes and degrees of Swedish coverage, and allows probing of cross-lingual factual consistency, as it contains English translations. Using the
dataset, we find that smaller models with stronger Swedish coverage perform comparably to a
multilingual model three times larger in recalling Sweden-related facts. We also observe that continued pre-training on Swedish generally improves factual knowledge but leads to partial forgetting of previously known information. These results demonstrate the
dataset's potential as a diagnostic tool for studying
language adaptation and knowledge retention in
multilingual models during
language adaptation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Magellan: Guided MCTS for Latent Space Exploration and Novelty Generation
Lufan Chang
Large
Language Models (LLMs) often struggle with generating truly innovative ideas, typically defaulting to high-probability, familiar concepts within their training data's "
gravity wells." While advanced search-based methods like Tree of Thoughts (ToT) attempt to mitigate this, they are fundamentally limited by their reliance on unprincipled, inconsistent self-evaluation heuristics to guide exploration. To address this gap, we introduce Magellan, a novel framework that reframes creative generation as a principled, guided exploration of an LLM's latent conceptual space. At its core, Magellan employs Monte Carlo Tree Search (MCTS) governed by a hierarchical guidance system. For long-range direction, a "semantic compass" vector, formulated via orthogonal projection, steers the search towards relevant novelty. For local, step-by-step decisions, a landscape-aware value function replaces flawed self-evaluation with an explicit reward structure that balances intrinsic coherence, extrinsic novelty, and narrative progress. Extensive experiments demonstrate that Magellan significantly outperforms strong baselines, including ReAct and ToT, in generating scientific ideas with superior plausibility and innovation. Our work shows that for creative discovery, a principled, guided search is more effective than unconstrained agency, paving the way for LLMs to become more capable partners in innovation.
Read the full abstract ▾
Computation and Language
Multi-turn Training with Basic Human Feedback Helps Little on LLM Reasoning
Qiang Liu, Wuganjing Song, Zhenzhou Lin et al. (+4 more)
The
reasoning capabilities of Large
Language Models (LLMs) are typically developed through the single-turn reinforcement learning, whereas real-world applications often involve multi-turn interactions with human feedback, leading to a potential mismatch between training and deployment conditions. In this work, we study whether multi-turn training with human feedback is necessary for
reasoning tasks. We compare conventional single-turn training with three multi-turn strategies and reach contrary conclusions to previous research. We find that models trained in a single-turn setting generalize effectively to both single- and multi-turn evaluations, while models trained with multi-turn strategies exhibit a significant degradation in single-turn
reasoning performance. These results suggest that for tasks with complete information, robust single-turn training remains more effective and reliable, as multi-turn training with basic feedback provides limited benefits and can even degrade
reasoning capabilities.
Read the full abstract ▾
All authors Qiang Liu, Wuganjing Song, Zhenzhou Lin, Feifan Chen, Qiaolong Cai, Chen Li, Yongduo Sui
Computation and Language
TripTide: A Benchmark for Adaptive Travel Planning under Disruptions
Priyanshu Karmakar, Soumyabrata Chaudhuri, Shubhojit Mallick et al. (+3 more)
Recent efforts like TripCraft and TravelPlanner have advanced the use of Large
Language Models ( LLMs) for personalized, constraint aware travel itinerary generation. Yet, real travel often faces disruptions. To address this, we present TripTide, the first
benchmark evaluating LLM's ability to revise itineraries under realistic disruptions. TripTide models key dimensions such as disruption severity and traveler tolerance, enabling nuanced assessment of LLM adaptability to events like flight cancellations, weather closures, or overbooked attractions. We conduct a threefold evaluation. First, we introduce automatic metrics including Preservation of Intent (how well the revised plan maintains feasibility and goals), Responsiveness (promptness and appropriateness of disruption handling), and Adaptability (semantic, spatial, and sequential divergence between original and revised plans). Second, we apply an LLM-as-a-judge approach to automatically assess revision quality. Third, we perform manual expert evaluation to verify whether revisions preserve semantic, spatial, sequential, and responsive aspects. Our experiments show that LLMs maintain strong sequential consistency and semantic stability, while spatial deviations are larger for shorter trips but decrease with longer ones, indicating that extended plans encourage better geographic coherence. However, disruption-handling ability declines as plan length increases, highlighting limits in LLM robustness. TripTide establishes a
benchmark for evaluating adaptability, personalization, and resilience in LLM-based travel planning under real-world uncertainty.
Read the full abstract ▾
All authors Priyanshu Karmakar, Soumyabrata Chaudhuri, Shubhojit Mallick, Manish Gupta, Abhik Jana, Shreya Ghosh
Computation and Language
Typoglycemia under the Hood: Investigating Language Models' Understanding of Scrambled Words
Gianluca Sperduti and Alejandro Moreo
Research in linguistics has shown that humans can read words with internally scrambled letters, a phenomenon recently dubbed typoglycemia. Some specific NLP models have recently been proposed that similarly demonstrate robustness to such distortions by ignoring the internal order of characters by design. This raises a fundamental question: how can models perform well when many distinct words (e.g., form and from) collapse into identical representations under typoglycemia? Our work, focusing exclusively on the English
language, seeks to shed light on the underlying aspects responsible for this robustness. We hypothesize that the main reasons have to do with the fact that (i) relatively few English words collapse under typoglycemia, and that (ii) collapsed words tend to occur in contexts so distinct that disambiguation becomes trivial. In our analysis, we (i) analyze the British National Corpus to quantify word collapse and ambiguity under typoglycemia, (ii) evaluate BERT's ability to disambiguate collapsing forms, and (iii) conduct a probing experiment by comparing variants of BERT trained from scratch on clean versus typoglycemic Wikipedia text; our results reveal that the performance degradation caused by scrambling is smaller than expected.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Leverage Unlearning to Sanitize LLMs
Antoine Boutet and Lucas Magnana
Pre-trained large
language models (LLMs) are becoming useful for various tasks. To improve their performance on certain tasks, it is necessary to fine-tune them on specific data corpora (e.g., medical reports, business data). These specialized data corpora may contain sensitive data (e.g., personal or confidential data) that will be memorized by the model and likely to be regurgitated during its subsequent use. This memorization of sensitive information by the model poses a significant
privacy or confidentiality issue. To remove this memorization and sanitize the model without requiring costly additional fine-tuning on a secured data corpus, we propose SANI. SANI is an unlearning approach to sanitize
language models. It relies on both an erasure and repair phases that 1) reset certain
neurons in the last layers of the model to disrupt the memorization of fine-grained information, and then 2) fine-tune the model while avoiding memorizing sensitive information. We comprehensively evaluate SANI to sanitize both a model fine-tuned and specialized with medical data by removing directly and indirectly identifiers from the memorization of the model, and a standard pre-trained model by removing specific terms defined as confidential information from the model. Results show that with only few additional epochs of unlearning, the model is sanitized and the number of regurgitations is drastically reduced. This approach can be particularly useful for hospitals or other industries that have already spent significant resources training models on large
datasets and wish to sanitize them before sharing.
Read the full abstract ▾
Computation and Language
Efficient semantic uncertainty quantification in language models via diversity-steered sampling
Ji Won Park and Kyunghyun Cho
Accurately estimating semantic aleatoric and epistemic uncertainties in large
language models (LLMs) is particularly challenging in free-form question answering (QA), where obtaining stable estimates often requires many expensive generations. We introduce a diversity-steered sampler that discourages semantically redundant outputs during decoding, covers both autoregressive and masked
diffusion paradigms, and yields substantial sample-efficiency gains. The key idea is to inject a continuous semantic-similarity penalty into the model's proposal distribution using a natural
language inference (NLI) model lightly finetuned on partial prefixes or intermediate
diffusion states. We debias downstream uncertainty estimates with importance reweighting and shrink their variance with control variates. Across four QA
benchmarks, our method matches or surpasses baselines while covering more semantic clusters with the same number of samples. Being modular and requiring no gradient access to the base LLM, the framework promises to serve as a drop-in enhancement for uncertainty estimation in risk-sensitive model deployments.
Read the full abstract ▾
Computation and Language
PARL: Prompt-based Agents for Reinforcement Learning
Yarik Menchaca Resendiz and Roman Klinger
Large
language models (LLMs) have demonstrated high performance on tasks expressed in natural
language, particularly in zero- or few-shot settings. These are typically framed as supervised (e.g., classification) or unsupervised (e.g., clustering) problems. However, limited work evaluates LLMs as
agents in reinforcement learning (RL) tasks (e.g., playing games), where learning occurs through interaction with an environment and a reward system. While prior work focused on representing tasks that rely on a
language representation, we study structured, non-linguistic
reasoning - such as interpreting positions in a grid world. We therefore introduce PARL (
Prompt-based
Agent for Reinforcement Learning), a method that uses LLMs as RL
agents through prompting, without any fine-tuning. PARL encodes actions, states, and rewards in the
prompt, enabling the model to learn through trial-and-error interaction. We evaluate PARL on three standard RL tasks that do not entirely rely on natural
language. We show that it can match or outperform traditional RL
agents in simple environments by leveraging pretrained knowledge. However, we identify performance limitations in tasks that require complex mathematical operations or decoding states and actions.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models
Yingzhi Mao, Chunkang Zhang, Junxiang Wang et al. (+6 more)
Large
Reasoning Models (LRMs) achieve strong performance on complex multi-step
reasoning, yet they still exhibit severe safety failures such as harmful content generation. Existing methods often apply coarse-grained constraints over the entire
reasoning trajectories, which can undermine
reasoning capability while failing to address the root causes of unsafe behavior. In this work, we uncover a previously underexplored failure mode in LRMs, termed Self-Jailbreak, where models initially recognize the harmful intent of a query, but override this judgment during subsequent
reasoning steps, ultimately generating unsafe outputs. Such a phenomenon reveals that LRMs are capable of recognizing harm, while safety failures primarily arise from
reasoning steps. Motivated by this finding, we propose Chain-of-Guardrail(CoG), a trajectory-level training framework that mitigates Self-Jailbreak via targeted, step-level interventions while maintaining
reasoning ability. Experiments across multiple safety and
reasoning benchmarks indicate that CoG achieves a favorable balance between safety and
reasoning performance compared with existing approaches.
Read the full abstract ▾
All authors Yingzhi Mao, Chunkang Zhang, Junxiang Wang, Xinyan Guan, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
Computation and Language · Cross-listed from cs.IR
Pctx: Tokenizing Personalized Context for Generative Recommendation
Qiyong Zhong, Jiajie Su, Yunshan Ma et al. (+2 more)
Generative recommendation (GR) models tokenize each action into a few discrete
tokens (called semantic IDs) and autoregressively generate the next
tokens as predictions, showing advantages such as
memory efficiency, scalability, and the potential to unify
retrieval and ranking. Despite these benefits, existing tokenization methods are static and non-personalized. They typically derive semantic IDs solely from item features, assuming a universal item similarity that overlooks user-specific perspectives. However, under the autoregressive paradigm, semantic IDs with the same prefixes always receive similar probabilities, so a single fixed mapping implicitly enforces a universal item similarity standard across all users. In practice, the same item may be interpreted differently depending on user intentions and preferences. To address this issue, we propose a personalized context-aware tokenizer that incorporates a user's historical interactions when generating semantic IDs. This design allows the same item to be tokenized into different semantic IDs under different user contexts, enabling GR models to capture multiple interpretive standards and produce more personalized predictions. Experiments on three public
datasets demonstrate up to 11.44% improvement in NDCG@10 over non-personalized action tokenization baselines. Our code is available at https://github.com/YoungZ365/Pctx.
Read the full abstract ▾
All authors Qiyong Zhong, Jiajie Su, Yunshan Ma, Julian McAuley, Yupeng Hou
Computation and Language
Sparser Block-Sparse Attention via Token Permutation
Xinghao Wang, Pengyu Wang, Dong Zhang et al. (+7 more)
Scaling the context length of large
language models (LLMs) offers significant benefits but is computationally expensive. This expense stems primarily from the self-
attention mechanism, whose $O(N^2)$ complexity with respect to sequence length presents a major bottleneck for both
memory and latency. Fortunately, the
attention matrix is often sparse, particularly for long sequences, suggesting an opportunity for optimization. Block-sparse
attention has emerged as a promising solution that partitions sequences into blocks and skips computation for a subset of these blocks. However, the effectiveness of this method is highly dependent on the underlying
attention patterns, which can lead to sub-optimal block-level sparsity. For instance, important key
tokens for queries within a single block may be scattered across numerous other blocks, leading to computational redundancy. In this work, we propose Permuted Block-Sparse
Attention (PBS-Attn), a plug-and-play method that leverages the permutation properties of
attention to increase block-level sparsity and enhance the computational efficiency of LLM prefilling. We conduct comprehensive experiments on challenging real-world long-context
datasets, demonstrating that PBS-Attn consistently outperforms existing block-sparse
attention methods in model accuracy and closely matches the full
attention baseline. Powered by our custom permuted-FlashAttention kernels, PBS-Attn achieves an end-to-end speedup of up to $2.75\times$ in long-context prefilling, confirming its practical viability. Code available at https://github.com/xinghaow99/pbs-attn
Read the full abstract ▾
All authors Xinghao Wang, Pengyu Wang, Dong Zhang, Chenkun Tan, Shaojun Zhou, Zhaoxiang Liu, Shiguo Lian, Fangxu Liu, Kai Song, Xipeng Qiu
Computation and Language
Correlation Dimension of Auto-Regressive Large Language Models
Xin Du and Kumiko Tanaka-Ishii
Large
language models (LLMs) have achieved remarkable progress in natural
language generation, yet they continue to display puzzling behaviors – such as repetition and incoherence – even when exhibiting low perplexity. This highlights a key limitation of conventional evaluation metrics, which emphasize local prediction accuracy while overlooking long-range structural complexity. We introduce correlation dimension, a fractal-
geometric measure of self-similarity, to quantify the epistemological complexity of text as perceived by a
language model. This measure captures the hierarchical recurrence structure of
language, bridging local and global properties in a unified framework. Through extensive experiments, we show that correlation dimension (1) reveals three distinct phases during pretraining, (2) reflects context-dependent complexity, (3) indicates a model's tendency toward
hallucination, and (4) reliably detects multiple forms of degeneration in generated text. The method is computationally efficient, robust to model quantization (down to 4-bit precision), broadly applicable across autoregressive architectures (e.g.,
Transformer and Mamba), and provides fresh insight into the generative dynamics of LLMs.
Read the full abstract ▾
Computation and Language
DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services
Xiang Li, Huizi Yu, Wenkong Wang et al. (+17 more)
Objective: Emergency medical dispatch (EMD) is a high-stakes process challenged by caller distress, ambiguity, and cognitive load. Large
Language Models (LLMs) and Multi-
Agent Systems (MAS) offer opportunities to augment dispatchers. This study aimed to develop and evaluate a taxonomy-grounded, LLM-powered multi-
agent system for simulating realistic EMD scenarios. Methods: We constructed a clinical taxonomy (32 chief complaints, 6 caller identities from MIMIC-III) and a six-phase call protocol. Using this framework, we developed an AutoGen-based MAS with Caller and Dispatcher
Agents. The system grounds interactions in a fact commons to ensure clinical plausibility and mitigate misinformation. We used a hybrid evaluation framework: four physicians assessed 100 simulated cases for "Guidance Efficacy" and "Dispatch Effectiveness," supplemented by automated linguistic analysis (sentiment, readability, politeness). Results: Human evaluation, with substantial inter-rater agreement (Gwe's AC1 > 0.70), confirmed the system's high performance. It demonstrated excellent Dispatch Effectiveness (e.g., 94 % contacting the correct potential other
agents) and Guidance Efficacy (advice provided in 91 % of cases), both rated highly by physicians. Algorithmic metrics corroborated these findings, indicating a predominantly neutral affective profile (73.7 % neutral sentiment; 90.4 % neutral
emotion), high readability (Flesch 80.9), and a consistently polite style (60.0 % polite; 0 % impolite). Conclusion: Our taxonomy-grounded MAS simulates diverse, clinically plausible dispatch scenarios with high fidelity. Findings support its use for dispatcher training, protocol evaluation, and as a foundation for real-time decision support. This work outlines a pathway for safely integrating advanced AI
agents into emergency response workflows.
Read the full abstract ▾
All authors Xiang Li, Huizi Yu, Wenkong Wang, Yiran Wu, Jiayan Zhou, Wenyue Hua, Xinxin Lin, Wenjia Tan, Lexuan Zhu, Bingyi Chen, Guang Chen, Ming-Li Chen, Yang Zhou, Zhao Li, Themistocles L. Assimes, Yongfeng Zhang, Qingyun Wu, Xin Ma, Lingyao Li, Lizhou Fan
Computation and Language
The "Right" Discourse on Migration: Analysing Migration-Related Tweets in Right and Far-Right Political Movements
Nishan Chatterjee, Veronika Bajt, Ana Zwitter Vitez et al. (+1 more)
The rise of right-wing populism in Europe has brought to the forefront the significance of analysing social media discourse to understand the dissemination of extremist ideologies and their impact on political outcomes. Twitter, as a platform for interaction and mobilisation, provides a unique window into the everyday communication of far-right supporters. In this paper, we propose a methodology that uses state-of-the-art natural
language processing techniques with sociological insights to analyse the MIGR-TWIT corpus of far-right tweets in English and French. We aim to uncover patterns of discourse surrounding migration, hate
speech, and persuasion techniques employed by right and far-right actors. By integrating linguistic, sociological, and computational approaches, we seek to offer cross-disciplinary insights into societal dynamics and contribute to a better understanding of contemporary challenges posed by right-wing extremism on social media platforms.
Read the full abstract ▾
All authors Nishan Chatterjee, Veronika Bajt, Ana Zwitter Vitez, Senja Pollak
Computation and Language
Helena Grete Lillepalu and Tanel Alumäe
The availability of LLM
benchmarks for the Estonian
language is limited, and a comprehensive evaluation comparing the performance of different LLMs on Estonian tasks has yet to be conducted. We introduce a new
benchmark for evaluating LLMs in Estonian, based on seven diverse
datasets. These
datasets assess general and domain-specific knowledge, understanding of Estonian grammar and vocabulary, summarization abilities, contextual comprehension, and more. The
datasets are all generated from native Estonian sources without using machine translation. We compare the performance of base models, instruction-tuned open-source models, and commercial models. Our evaluation includes 6 base models and 26 instruction-tuned models. To assess the results, we employ both human evaluation and LLM-as-a-judge methods. Human evaluation scores showed moderate to high correlation with
benchmark evaluations, depending on the
dataset. Claude 3.7 Sonnet, used as an LLM judge, demonstrated strong
alignment with human ratings, indicating that top-performing LLMs can effectively support the evaluation of Estonian-
language models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference
Stephen Zhao, Aidan Li, Rob Brekelmans et al. (+1 more)
Reinforcement learning (RL) has become a predominant technique to align
language models (LMs) with human preferences or promote outputs which are deemed to be desirable by a given reward function. Standard RL approaches optimize average reward, while methods explicitly focused on reducing the probability of undesired outputs typically come at a cost to average-case performance. To improve this tradeoff, we introduce RePULSe, a new training method that augments the standard RL loss with an additional loss that uses learned proposals to guide sampling low-reward outputs, and then reduces those outputs' probability. We run experiments demonstrating that RePULSe produces a better tradeoff of expected reward versus the probability of undesired outputs and is more adversarially robust, compared to standard RL
alignment approaches and alternatives.
Read the full abstract ▾
All authors Stephen Zhao, Aidan Li, Rob Brekelmans, Roger Grosse
Computation and Language · Cross-listed from cs.CV
Junzhe Zhang, Huixuan Zhang and Xiaojun Wan
The rapid progress of
multimodal large
language models (MLLMs) calls for more reliable evaluation protocols. Existing static
benchmarks suffer from the potential risk of data contamination and saturation, leading to inflated or misleading performance evaluations. To address these issues, we first apply Graph formulation to represent a static or dynamic VQA sample. With the formulation, we propose Knowledge-enhanced
Benchmark Evolution(KBE), a dynamic
multimodal evaluation framework. KBE first analyzes the original static
benchmark, then expands it by integrating
multimodal knowledge, transforming the static
benchmark into a controllable, dynamic evolving version. Crucially, KBE can both reconstruct questions by Re-selecting visual information in the original image and expand existing questions with external textual knowledge. It enables difficulty-controllable evaluation by adjusting the degree of question exploration. Extensive experiments demonstrate that KBE alleviates the risk of data contamination, data saturation, and provides a more comprehensive assessment of MLLM capabilities.
Read the full abstract ▾
Computation and Language
Social Simulations with Large Language Model Risk Utopian Illusion
Ning Bian, Xianpei Han, Hongyu Lin et al. (+2 more)
Reliable simulation of human behavior is essential for explaining, predicting, and intervening in our society. Recent advances in large
language models (LLMs) have shown promise in emulating human behaviors, interactions, and decision-making, offering a powerful new lens for social science studies. However, the extent to which LLMs diverge from authentic human behavior in social contexts remains underexplored, posing risks of misinterpretation in scientific studies and unintended consequences in real-world applications. Here, we introduce a systematic framework for analyzing LLMs' behavior in social simulation. Our approach simulates multi-
agent interactions through chatroom-style conversations and analyzes them across five linguistic dimensions, providing a simple yet effective method to examine emergent social cognitive biases. We conduct extensive experiments involving eight representative LLMs across three families. Our findings reveal that LLMs do not faithfully reproduce genuine human behavior but instead reflect overly idealized versions of it, shaped by the social desirability bias. In particular, LLMs show social role bias, primacy effect, and positivity bias, resulting in "Utopian" societies that lack the complexity and variability of real human interactions. These findings call for more socially grounded LLMs that capture the diversity of human social behavior.
Read the full abstract ▾
All authors Ning Bian, Xianpei Han, Hongyu Lin, Baolei Wu, Jun Wang
Computation and Language
Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
Guangxin Su, Hanchen Wang, Jianwei Wang et al. (+3 more)
Large
Language Models (LLMs) have achieved remarkable success in natural
language processing through strong semantic understanding and generation. However, their black-box nature limits structured and multi-hop
reasoning. In contrast, Text-Attributed Graphs (TAGs) provide explicit relational structures enriched with textual context, yet often lack semantic depth. Recent research shows that combining LLMs and TAGs yields complementary benefits: enhancing TAG representation learning and improving the
reasoning and interpretability of LLMs. This survey provides the first systematic review of LLM–TAG integration from an orchestration perspective. We introduce a novel taxonomy covering two fundamental directions: LLM for TAG, where LLMs enrich graph-based tasks, and TAG for LLM, where structured graphs improve LLM
reasoning. We categorize orchestration strategies into sequential, parallel, and multi-module frameworks, and discuss advances in TAG-specific pretraining, prompting, and parameter-efficient fine-tuning. Beyond methodology, we summarize empirical insights, curate available
datasets, and highlight diverse applications across recommendation systems, biomedical analysis, and knowledge-intensive question answering. Finally, we outline open challenges and promising research directions, aiming to guide future work at the intersection of
language and graph learning.
Read the full abstract ▾
All authors Guangxin Su, Hanchen Wang, Jianwei Wang, Wenjie Zhang, Ying Zhang, Jian Pei
Computation and Language
The Gray Zone of Faithfulness: Taming Ambiguity in Unfaithfulness Detection
Qiang Ding, Lvzhou Luo, Yixuan Cao et al. (+1 more)
Ensuring that Large
Language Models (LLMs) generate summaries faithful to a given source document is essential for real-world applications. While prior research has explored LLM faithfulness, existing
benchmarks suffer from annotation ambiguity, primarily due to the ill-defined boundary of permissible external knowledge in generated outputs. For instance, common sense is often incorporated into responses and labeled as "faithful", yet the acceptable extent of such knowledge remains unspecified, leading to inconsistent annotations. To address this issue, we propose a novel faithfulness annotation framework, which introduces an intermediate category, Out-Dependent, to classify cases where external knowledge is required for verification. Using this framework, we construct VeriGray (Verification with the Gray Zone) – a new unfaithfulness detection
benchmark in summarization. Statistics reveal that even SOTA LLMs, such as GPT-5, exhibit
hallucinations ($\sim 6\%$ of sentences) in summarization tasks. Moreover, a substantial proportion ($\sim 9\%$ on average of models) of generated sentences fall into the Out-Dependent category, underscoring the importance of resolving annotation ambiguity in unfaithfulness detection
benchmarks. Experiments demonstrate that our
benchmark poses significant challenges to multiple baseline methods, indicating considerable room for future improvement.
Read the full abstract ▾
All authors Qiang Ding, Lvzhou Luo, Yixuan Cao, Ping Luo
Computation and Language
Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
Qingru Zhang, Liang Qiu, Ilgee Hong et al. (+11 more)
Supervised fine-tuning (SFT) has emerged as a crucial method for aligning large
language models (LLMs) with human-annotated demonstrations. However, SFT, being an off-policy approach similar to behavior cloning, often struggles with overfitting and poor out-of-domain generalization, especially in limited-data scenarios. To address these limitations, we propose Self-Rewarding PPO, a novel fine-tuning method that leverages on-policy techniques to enhance generalization performance. Our approach combines the strengths of SFT and proximal policy optimization (PPO) to achieve more effective
alignment from demonstration data. At its core is a reward function designed as the log policy ratio between the SFT model and the pretrained base model. This function serves as an implicit reward signal, using the pretrained policy as a baseline and the SFT policy as a target. By doing so, it enables on-policy fine-tuning without relying on human preference annotations. The integration of this self-rewarding mechanism with PPO addresses key limitations of SFT, improving generalization, data efficiency, and robustness. Our empirical evaluation across a range of natural
language processing tasks demonstrates that Self-Rewarding PPO consistently outperforms traditional SFT methods. The results highlight the effectiveness of our approach in aligning LLMs using demonstration data, particularly in scenarios where high-quality annotated data is scarce.
Read the full abstract ▾
All authors Qingru Zhang, Liang Qiu, Ilgee Hong, Zhenghao Xu, Tianyi Liu, Shiyang Li, Rongzhi Zhang, Zheng Li, Lihong Li, Bing Yin, Chao Zhang, Jianshu Chen, Haoming Jiang, Tuo Zhao
Computation and Language · Cross-listed from cs.HC
Designing and Evaluating Chain-of-Hints for Scientific Question Answering
Anubhav Jangra and Smaranda Muresan
LLMs are reshaping education, with students increasingly relying on them for learning. Implemented using general-purpose models, these systems are likely to give away the answers, potentially undermining conceptual understanding and critical thinking. Prior work shows that hints can effectively promote cognitive engagement. Building on this insight, we evaluate 18 open-source LLMs on chain-of-hints generation that scaffold users toward the correct answer. We compare two distinct hinting strategies: static hints, pre-generated for each problem, and dynamic hints, adapted to a learners' progress. We evaluate these systems on five pedagogically grounded automatic metrics for hint quality. Using the best performing LLM as the backbone of a quantitative study with 41 participants, we uncover distinct user preferences across hinting strategies, and identify the limitations of automatic evaluation metrics to capture them. Our findings highlight key design considerations for future research on tutoring systems and contribute toward the development of more learner-centered educational technologies.
Read the full abstract ▾
Computation and Language
MediRec: Enhancing Chinese Medication Recommendation with Explainable Clinical Reasoning
Juntao Li, Haobin Yuan, Ling Luo et al. (+3 more)
Large
language models (LLMs) have shown strong potential for clinical decision support through their advanced
language understanding and
reasoning capabilities. However, their application to Chinese clinical medication recommendation remains largely unexplored. Existing approaches are primarily developed on English electronic health record
datasets and focus on coarse-grained medication code prediction, offering limited support for interpretable clinical decision-making. In this work, we propose MediRec, an explainable LLM-based framework for Chinese medication recommendation from electronic health records. MediRec combines clinically grounded
reasoning-chain
distillation with reinforcement learning to improve both recommendation accuracy and interpretability. Comprehensive experiments on a Chinese medication recommendation
benchmark show that MediRec achieves strong performance, with an F1 score of 0.5813 and a Jaccard score of 0.4626. Further analyses indicate that MediRec generates clinically plausible recommendations with transparent
reasoning, demonstrating its effectiveness for explainable medication decision support in Chinese healthcare settings.
Read the full abstract ▾
All authors Juntao Li, Haobin Yuan, Ling Luo, Yuanyuan Sun, Jian Wang, Hongfei Lin
Computation and Language
Bridging Language Gaps with Adaptive RAG: Improving Indonesian Language Question Answering
William Christian, Daniel Adamlu, Adrian Yu et al. (+1 more)
Question Answering (QA) has seen significant improvements with the advancement of machine learning models, further studies enhanced this question answering system by retrieving external information, called
Retrieval-Augmented Generation (RAG) to produce more accurate and informative answers. However, these state-of-the-art-performance is predominantly in English
language. To address this gap we made an effort of bridging
language gaps by incorporating Adaptive RAG system to Indonesian
language. Adaptive RAG system integrates a classifier whose task is to distinguish the question complexity, which in turn determines the strategy for answering the question. To overcome the limited availability of Indonesian
language dataset, our study employs machine translation as data augmentation approach. Experiments show reliable question complexity classifier; however, we observed significant inconsistencies in multi-
retrieval answering strategy which negatively impacted the overall evaluation when this strategy was applied. These findings highlight both the promise and challenges of question answering in low-resource
language suggesting directions for future improvement.
Read the full abstract ▾
All authors William Christian, Daniel Adamlu, Adrian Yu, Derwin Suhartono
Computation and Language
Dynamic Retriever for In-Context Knowledge Editing via Policy Optimization
Mahmud Wasif Nafee, Maiqi Jiang, Haipeng Chen et al. (+1 more)
Large
language models (LLMs) excel at factual recall yet still propagate stale or incorrect knowledge. In-context knowledge editing offers a gradient-free remedy suitable for black-box APIs, but current editors rely on static demonstration sets chosen by surface-level similarity, leading to two persistent obstacles: (i) a quantity-quality trade-off, and (ii) lack of adaptivity to task difficulty. We address these issues by dynamically selecting supporting demonstrations according to their utility for the edit. We propose Dynamic Retriever for In-Context Knowledge Editing (DR-IKE), a lightweight framework that (1) trains a BERT retriever with REINFORCE to rank demonstrations by editing reward, and (2) employs a learnable threshold to prune low-value examples, shortening the
prompt when the edit is easy and expanding it when the task is hard. DR-IKE performs editing without modifying model weights, relying solely on forward passes for compatibility with black-box LLMs. On the COUNTERFACT
benchmark, it improves edit success by up to 17.1%, reduces latency by 41.6%, and preserves accuracy on unrelated queries, demonstrating scalable and adaptive knowledge editing. The code is available at https://github.com/mwnafee/DR-IKE .
Read the full abstract ▾
All authors Mahmud Wasif Nafee, Maiqi Jiang, Haipeng Chen, Yanfu Zhang