Computation and Language
LLM-Specific Utility for Retrieval-Augmented Generation
Hengran Zhang, Keping Bi, Jiafeng Guo et al. (+4 more)
Retrieval-augmented generation (RAG) is typically optimized for topical relevance, yet its success ultimately depends on whether retrieved passages are useful for a large
language model (LLM) to generate correct and complete answers. We argue that such utility is often LLM-specific rather than universal, due to differences in models' knowledge,
reasoning, and ability to leverage evidence. We formalize LLM-specific utility as the performance improvement of a target LLM when a passage is provided, compared to answering without evidence. To systematically study LLM-specific utility, we construct a
benchmark of LLM-specific gold utilitarian passages for four LLMs (Qwen3-8B/14B/32B and Llama 3.1-8B) on three QA
datasets (Natural Questions, TriviaQA, and MS MARCO-FQA). Our analysis shows that utilitarian passages are model-dependent and non-transferable: each LLM performs best with its own utilitarian evidence, while evidence optimized for other LLMs is consistently suboptimal. Human-annotated evidence remains a strong general baseline but does not fully match individual LLM utility needs. We further introduce the LLM-specific utility judgment task and construct the corresponding
benchmark, i.e., SpecUBench (LLM-Specific Utility
Benchmark). Experiments show that existing utility-aware selection and scoring methods largely capture model-agnostic usefulness and struggle to reliably estimate LLM-specific utility. Overall, our findings highlight the limitations of current utility-aware
retrieval and motivate generator-tailored evidence selection for improving RAG. Our code and
datasets can be found at https://github.com/Trustworthy-Information-Access/LLM_specific_utility.
Read the full abstract ▾
All authors Hengran Zhang, Keping Bi, Jiafeng Guo, Jiaming Zhang, Shuaiqiang Wang, Dawei Yin, Xueqi Cheng
Computation and Language · Cross-listed from cs.SD
Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gap
KiHyun Nam, Jongmin Choi, Hyeongkeun Lee et al. (+2 more)
Contrastive audio-
language pretraining yields powerful joint representations, yet a persistent audio-text modality gap limits the benefits of coupling
multimodal encoders with large
language models (LLMs). We present
Diffusion-Link, a
diffusion-based modality-bridging module that generatively maps audio
embeddings into the text-
embedding distribution. The module is trained at the output
embedding from the frozen
multimodal encoder and implemented as a lightweight network with three residual MLP blocks. To assess the effect of
Diffusion-Link on
multimodal encoder-LLM coupling, we evaluate on Automatic Audio Captioning (AAC); to our knowledge, this is the first application of
diffusion-based modality bridging to AAC. We report two results. (1) Modality-gap analysis: on similarity and
geometric criteria,
Diffusion-Link reduces the modality gap the most among prior
diffusion-based methods and shows a collective migration of audio
embeddings toward the text distribution. (2) Downstream AAC: attaching
Diffusion-Link to the same
multimodal LLM baseline achieves state-of-the-art on AudioCaps in both zero-shot and fully supervised captioning without external knowledge, with relative gains up to 52.5% and 7.5%, respectively. These findings show that closing the modality gap is pivotal for effective coupling between
multimodal encoders and LLMs, and
diffusion-based modality bridging offers a promising direction beyond knowledge-
retrieval-centric designs. Code will be released upon acceptance https://github.com/DevKiHyun/
Diffusion-Link
Read the full abstract ▾
All authors KiHyun Nam, Jongmin Choi, Hyeongkeun Lee, Jungwoo Heo, Joon Son Chung
Computation and Language
Do LLMs "Feel"? Emotion Circuits Discovery and Control
Chenxi Wang, Yixuan Zhang, Ruiji Yu et al. (+8 more)
As the demand for emotional intelligence in large
language models (LLMs) grows, a key challenge lies in understanding the internal mechanisms that give rise to emotional expression and in controlling
emotions in generated text. This study addresses three core questions: (1) Do LLMs contain context-agnostic mechanisms shaping emotional expression? (2) What form do these mechanisms take? (3) Can they be harnessed for universal
emotion control? We first construct a controlled
dataset, SEV (Scenario-Event with Valence), to elicit comparable internal states across
emotions. Subsequently, we extract context-agnostic
emotion directions that reveal consistent, cross-context encoding of
emotion (Q1). We identify
neurons and
attention heads that locally implement emotional computation through analytical decomposition and causal analysis, and validate their causal roles via ablation and enhancement interventions. Next, we quantify each sublayer's causal influence on the model's final
emotion representation and integrate the identified local components into coherent global
emotion circuits that drive emotional expression (Q2). Directly modulating these
circuits achieves 99.65%
emotion-expression accuracy on the test set, surpassing prompting- and steering-based methods (Q3). To our knowledge, this is the first systematic study to uncover and validate
emotion circuits in LLMs, offering new insights into interpretability and controllable emotional intelligence.
Read the full abstract ▾
All authors Chenxi Wang, Yixuan Zhang, Ruiji Yu, Yufei Zheng, Lang Gao, Zirui Song, Zixiang Xu, Gus Xia, Huishuai Zhang, Dongyan Zhao, Xiuying Chen
Computation and Language
Template-Based Text-to-Image Alignment for Language Accessibility: A Study on Visualizing Text Simplifications
Belkiss Souayed, Sarah Ebling and Yingqiang Gao
Individuals with intellectual disabilities often have difficulties in comprehending complex texts. While many text-to-image models prioritize aesthetics over accessibility, it is not clear how visual illustrations relate to text simplifications (TS) generated from them. This paper presents a structured
vision-
language model (VLM) prompting framework for generating accessible images from simplified texts. We designed five
prompt templates, i.e., Basic Object Focus, Contextual Scene, Educational Layout, Multi-Level Detail, and Grid Layout, each following distinct spatial arrangements while adhering to accessibility constraints such as object count limits, spatial separation, and content restrictions. Using 400 sentence-level simplifications from four established TS
datasets (OneStopEnglish, SimPA, Wikipedia, and ASSET), we conducted a two-phase evaluation: Phase 1 assessed
prompt template effectiveness with CLIPScores, and Phase 2 involved human annotation of generated images across ten visual styles by four accessibility experts. Results show that the Basic Object Focus
prompt template achieved the highest semantic
alignment, indicating that visual minimalism enhances
language accessibility. Expert evaluation further identified Retro style as the most accessible and Wikipedia as the most effective data source. Inter-annotator agreement varied across dimensions, with Text Simplicity showing strong reliability and Image Quality proving more subjective. Overall, our framework offers practical guidelines for accessible content generation and underscores the importance of structured prompting in AI-generated visual accessibility tools.
Read the full abstract ▾
Computation and Language
FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks
Sabrina McCallum, Amit Parekh and Alessandro Suglia
Current approaches to
embodied AI tend to learn policies from expert demonstrations. However, without a mechanism to evaluate the quality of demonstrated actions, they are limited to learning from optimal behaviour, or they risk replicating errors and inefficiencies. While reinforcement learning offers one alternative, the associated exploration typically results in sacrificing data efficiency. This work explores how
agents trained with imitation learning can learn robust representations from both optimal and suboptimal demonstrations when given access to constructive
language feedback as a means to contextualise different modes of behaviour. We directly provide
language feedback
embeddings as part of the input sequence into a
Transformer-based policy, and optionally complement the traditional next action prediction objective with auxiliary self-supervised learning objectives for feedback prediction. We test our approach on a range of
embodied Vision-and-
Language tasks in our custom BabyAI-XGen environment and show significant improvements in
agents' compositional generalisation abilities and robustness, suggesting that our data-efficient method allows models to successfully convert suboptimal behaviour into learning opportunities. Overall, our results suggest that
language feedback is a competitive and intuitive alternative to intermediate scalar rewards for
language-specified
embodied tasks.
Read the full abstract ▾
Computation and Language
Are Large Language Models Effective Knowledge Graph Constructors?
Ruirui Chen, Weifeng Jiang, Chengwei Qin et al. (+10 more)
Knowledge graphs (KGs) are widely used in knowledge-intensive applications, yet it remains unclear how effectively current large
language models (LLMs) can construct document-grounded KGs in a zero-shot, schema-free setting without relying on complex task-specific frameworks. We introduce Detail-to-Abstract Hierarchical Knowledge Graph (D2A-HKG) construction framework, which decomposes KG construction into three stages: initial extraction, splitting, and abstraction, and evaluates the resulting graphs from both semantic and structural perspectives. Using seven frontier LLMs, we
benchmark zero-shot KG construction on CMW-Lit, a
dataset derived from published paediatric research articles on children's mental well-being. CMW-Lit provides a challenging test bed due to its heterogeneous evidence, interconnected factors, and complex, statistically qualified relationships. Our results show that state-of-the-art LLMs can generally produce relevant and document-faithful triples with limited
hallucination, while exhibiting substantially different extraction behaviors across the construction stages. These findings provide empirical insight into the strengths and limitations of frontier LLMs for direct knowledge graph construction. We further release CMW-Lit and the resulting knowledge graphs as resources for future research, with the generated graphs providing a strong foundation for expert refinement and downstream knowledge-intensive applications.
Read the full abstract ▾
All authors Ruirui Chen, Weifeng Jiang, Chengwei Qin, Bo Xiong, Kaiwen Wei, Fiona Liausvia, Pei Fang Tan, Ker Yung Chua, Dongkyu Choi, Mukkesh Kumar, Evelyn C. Law, Dennis Wang, Boon Kiat Quek
Computation and Language
Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs
Nikita Afonin, Nikita Andriianov, Vahagn Hovhannisyan et al. (+9 more)
Recent work has shown that narrow finetuning can produce broadly misaligned LLMs, a phenomenon termed emergent misalignment (EM). While concerning, these findings were limited to finetuning and activation steering, leaving out in-context learning (ICL). We therefore ask: does EM emerge in ICL? We find that it does: across four model families (Gemini, Kimi-K2, Grok, and Qwen), narrow in-context examples cause models to produce misaligned responses to benign, unrelated queries. With 16 in-context examples, EM rates range from 1% to 24% depending on model and domain, appearing with as few as 2 examples. Neither larger model scale nor explicit
reasoning provides reliable protection, and larger models are typically even more susceptible. Next, we formulate and test a hypothesis, which explains in-context EM as conflict between safety objectives and context-following behavior. Consistent with this, instructing models to prioritize safety reduces EM while prioritizing context-following increases it. These findings establish ICL as a previously underappreciated vector for emergent misalignment that resists simple scaling-based solutions.
Read the full abstract ▾
All authors Nikita Afonin, Nikita Andriianov, Vahagn Hovhannisyan, Nikhil Bageshpura, Kyle Liu, Kevin Zhu, Sunishchal Dev, Ashwinee Panda, Oleg Rogov, Elena Tutubalina, Alexander Panchenko, Mikhail Seleznyov
Computation and Language · Cross-listed from cs.LG
Gareth Seneque, Lap-Hang Ho, Nafise Erfanian Saeedi et al. (+3 more)
We present Entropic Mutual-Information Geometry Large-
Language Model
Alignment (ENIGMA), a novel approach to Large-
Language Model (LLM) training that jointly improves
reasoning,
alignment and robustness by treating an organisation's policies/principles as directions to move on a model's information
manifold. Our single-loop trainer combines Group-Relative Policy Optimisation (GRPO), an on-policy, critic-free RL method with Chain-of-Thought (CoT)-format only rewards; a Self-Supervised
Alignment with Mutual Information (SAMI)-style symmetric InfoNCE auxiliary; and an entropic Sinkhorn optimal-transport regulariser on hidden-state distributions to bound geometry drift. We also introduce infoNCE metrics that specialise to a standard MI lower bound under matched negatives to measure how strongly a model's CoT encodes these policies. These metrics include a Sufficiency Index (SI) that enables the selection and creation of principles that maximise downstream performance prior to training. In our experiments using small (1B) LLMs, high-SI principles predict steadier training dynamics and improved
benchmark performance over GRPO ablations. Our information-geometry analysis of trained models validates desirable structural change in the
manifold. These results support our hypothesis that
reasoning,
alignment, and robustness are projections of a single information-
geometric objective, and that models trained using ENIGMA demonstrate principled
reasoning without the use of a reward model, offering a path to trusted capability
Read the full abstract ▾
All authors Gareth Seneque, Lap-Hang Ho, Nafise Erfanian Saeedi, Jeffrey Molendijk, Ariel Kuperman, Tim Elson
Computation and Language
Towards Real-Time Fake News Detection under Evidence Scarcity
Guangyu Wei, Ke Han, Yueming Lyu et al. (+4 more)
Fake news detection becomes particularly challenging in real-time scenarios, where emerging events often lack sufficient supporting evidence. Existing approaches often rely heavily on external evidence and therefore struggle to generalize under evidence scarcity. To address this issue, we propose Evaluation-Aware Selection of Experts (EASE), a novel framework for real-time fake news detection that dynamically adapts its decision-making process according to the assessed sufficiency of available evidence. EASE introduces a sequential evaluation mechanism comprising three independent perspectives: (1) Evidence-based evaluation, which assesses evidence and incorporates it into decision-making only when the evidence is sufficiently supportive; (2)
Reasoning-based evaluation, which leverages the world knowledge of large
language models (LLMs) and applies them only when their reliability is adequately established; and (3) Sentiment-based fallback, which integrates sentiment cues when neither evidence nor
reasoning is reliable. To enhance the accuracy of evaluation processes, EASE employs instruction tuning with pseudo labels to guide each evaluator in justifying its perspective-specific knowledge through interpretable
reasoning. Furthermore, the expert modules integrate the evaluators' justified assessments with the news content to enable evaluation-aware decision-making, thereby enhancing overall detection accuracy. Moreover, we introduce RealTimeNews-25, a new
benchmark comprising recent news for evaluating model generalization on emerging news with limited evidence. Extensive experiments demonstrate that EASE not only achieves state-of-the-art performance across multiple
benchmarks, but also significantly improves generalization to real-time news. The code and
dataset are available: https://github.com/wgyhhhh/EASE.
Read the full abstract ▾
All authors Guangyu Wei, Ke Han, Yueming Lyu, Yu Luo, Yue Jiang, Caifeng Shan, Nicu Sebe
Computation and Language
Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Morality
Jana Jung, Marlene Lutz, Indira Sen et al. (+1 more)
Psychometric tests are increasingly used to assess psychological constructs in large
language models (LLMs). However, it remains unclear whether these tests – originally developed for humans – yield meaningful results when applied to LLMs. In this study, we systematically evaluate the reliability and validity of human psychometric tests on 17 LLMs for three constructs: sexism, racism, and morality. We find moderate reliability across multiple item and
prompt variations. Validity is evaluated through both convergent (i.e., testing theory-based inter-test correlations) and ecological approaches (i.e., testing the
alignment between tests scores and behavior in real-world downstream tasks). Crucially, we find that psychometric test scores do not align, and in some cases even negatively correlate with, model behavior in downstream tasks, indicating low ecological validity. Our results highlight that systematic evaluations of psychometric tests on LLMs are essential before interpreting their scores. Our findings also suggest that psychometric tests designed for humans cannot be applied directly to LLMs without adaptation.
Read the full abstract ▾
All authors Jana Jung, Marlene Lutz, Indira Sen, Markus Strohmaier
Computation and Language
Attacks by Content: Automated Fact-checking is an AI Security Issue
Michael Schlichtkrull
When AI
agents retrieve and reason over external documents, adversaries can manipulate the data they receive to subvert their behaviour. Previous research has studied indirect
prompt injection, where the attacker injects malicious instructions. We argue that injection of instructions is not necessary to manipulate
agents - attackers could instead supply biased, misleading, or false information. We term this an attack by content. Existing defenses, which focus on detecting hidden commands, are ineffective against attacks by content. To defend themselves and their users,
agents must critically evaluate retrieved information, corroborating claims with external evidence and evaluating source trustworthiness. We argue that this is analogous to an existing NLP task, automated fact-checking, which we propose to repurpose as a cognitive self-defense tool for
agents.
Read the full abstract ▾
Computation and Language
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
Haoqi Yang, Yao Yao, Zuchao Li et al. (+3 more)
Large
Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural
language processing tasks. However, their extensive
memory requirements, particularly due to KV cache growth during long-text understanding and generation, present significant challenges for deployment in resource-constrained environments. Quantization has emerged as a promising solution to reduce
memory consumption while preserving historical information. We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization. XQuant introduces two key innovations: a computationally negligible data-free calibration method and cross-layer KV cache compression, enabling quantization to sub-1.4 bits. Extensive experiments on TruthfulQA and LongBench demonstrate that XQuant outperforms state-of-the-art methods (e.g., KIVI-2bit and AsymKV-1.5bit) by achieving lower bit-width while maintaining superior performance, establishing a better trade-off between
memory efficiency and model accuracy.
Read the full abstract ▾
All authors Haoqi Yang, Yao Yao, Zuchao Li, Baoyuan Qi, Guoming Liu, Hai Zhao
Computation and Language
CNSocialDepress: A Chinese Social Media Dataset for Depression Risk Detection and Structured Analysis
Jinyuan Xu, Tian Lan, Xintao Yu et al. (+6 more)
Depression is a pressing global public health issue, yet publicly available Chinese-
language resources for depression risk detection remain scarce and largely focus on binary classification. To address this limitation, we release CNSocialDepress, a
benchmark dataset for depression risk detection on Chinese social media. The
dataset contains 44,178 posts from 233 users; psychological experts annotated 10,306 depression-related segments. CNSocialDepress provides binary risk labels along with structured, multidimensional psychological attributes, enabling interpretable and fine-grained analyses of depressive signals. Experimental results demonstrate the
dataset's utility across a range of NLP tasks, including structured psychological profiling and fine-tuning large
language models for depression detection. Comprehensive evaluations highlight the
dataset's effectiveness and practical value for depression risk identification and psychological analysis, thereby providing insights for mental health applications tailored to Chinese-speaking populations.
Read the full abstract ▾
All authors Jinyuan Xu, Tian Lan, Xintao Yu, Xue He, Hezhi Zhang, Ying Wang, Pierre Magistry, Mathieu Valette, Lei Li
Computation and Language
A Theorem-Proving-Based Evaluation of Neural Semantic Parsing
Hayate Funakura, Hyunsoo Kim and Koji Mineshima
Graph-matching metrics such as Smatch are the de facto standard for evaluating neural semantic parsers, yet they capture surface overlap rather than logical equivalence. We reassess evaluation by pairing graph-matching with automated
theorem proving. We compare two approaches to building parsers: supervised fine-tuning (T5-Small/Base) and few-shot in-context learning (GPT-4o/4.1/5), under normalized and unnormalized targets. We evaluate outputs using graph-matching, bidirectional entailment between source and target formulas with a first-order logic
theorem prover, and well-formedness. Across settings, we find that models performing well on graph-matching often fail to produce logically equivalent formulas. Normalization reduces incidental target variability, improves well-formedness, and strengthens logical adequacy. Error analysis shows performance degrades with increasing formula complexity and with coordination, prepositional phrases, and passive voice; the dominant failures involve variable binding and indexing, and predicate naming. These findings highlight limits of graph-based metrics for
reasoning-oriented applications and motivate logic-sensitive evaluation and training objectives together with simplified, normalized target representations. All code and data for our experiments are publicly available.
Read the full abstract ▾
Computation and Language
Fairness Metric Design Exploration in Multi-Domain Moral Sentiment Classification using Transformer-Based Models
Battemuulen Naranbat, Seyed Sahand Mohammadi Ziabari, Yousuf Nasser Al Husaini et al. (+1 more)
Ensuring
fairness in natural
language processing for moral sentiment classification is challenging, particularly under cross-domain shifts where
transformer models are increasingly deployed. Using the Moral Foundations Twitter Corpus (MFTC) and Moral Foundations Reddit Corpus (MFRC), this work evaluates BERT and DistilBERT in a multi-label setting with in-domain and cross-domain protocols. Aggregate performance can mask disparities: we observe pronounced asymmetry in transfer, with Twitter->Reddit degrading micro-F1 by 14.9% versus only 1.5% for Reddit->Twitter. Per-label analysis reveals
fairness violations hidden by overall scores; notably, the authority label exhibits Demographic Parity Differences of 0.22-0.23 and Equalized Odds Differences of 0.40-0.41. To address this gap, we introduce the Moral
Fairness Consistency (MFC) metric, which quantifies the cross-domain stability of moral foundation detection. MFC shows strong empirical validity, achieving a perfect negative correlation with Demographic Parity Difference (rho = -1.000, p < 0.001) while remaining independent of standard performance metrics. Across labels, loyalty demonstrates the highest consistency (MFC = 0.96) and authority the lowest (MFC = 0.78). These findings establish MFC as a complementary, diagnosis-oriented metric for
fairness-aware evaluation of moral
reasoning models, enabling more reliable deployment across heterogeneous linguistic contexts. .
Read the full abstract ▾
All authors Battemuulen Naranbat, Seyed Sahand Mohammadi Ziabari, Yousuf Nasser Al Husaini, Ali Mohammed Mansoor Alsahag
Computation and Language
WebRouter: Query-specific Router via Variational Information Bottleneck for Cost-sensitive Web Agent
Tao Li, Jinlong Hu, Yang Wang et al. (+2 more)
LLM-brained web
agents offer powerful capabilities for web automation but face a critical cost-performance trade-off. The challenge is amplified by web
agents' inherently complex
prompts that include goals, action histories, and environmental states, leading to degraded LLM ensemble performance. To address this, we introduce WebRouter, a novel query-specific router trained from an information-theoretic perspective. Our core contribution is a cost-aware Variational Information Bottleneck (ca-VIB) objective, which learns a compressed representation of the input
prompt while explicitly penalizing the expected operational cost. Experiments on five real-world websites from the WebVoyager
benchmark show that WebRouter reduces operational costs by a striking 87.8% compared to a GPT-4o baseline, while incurring only a 3.8% accuracy drop.
Read the full abstract ▾
All authors Tao Li, Jinlong Hu, Yang Wang, Junfeng Liu, Xuejun Liu
Computation and Language
The Curious Case of Factual (Mis)Alignment between LLMs' Short- and Long-Form Answers
Saad Obaid ul Islam, Anne Lauscher and Goran Glavaš
Large
language models (LLMs) can correctly answer "When was Einstein born?" yet fail to provide the same date when writing about Einstein's life revealing a fundamental inconsistency in how models access factual knowledge across task complexities. While models display impressive accuracy on factual question-answering
benchmarks, the reliability gap between simple and complex queries remains poorly understood, eroding their trustworthiness. In this work, we introduce Short-Long Form
Alignment for Factual Question Answering (SLAQ), a controlled evaluation framework that compares LLMs' answers to the same factual questions asked (a) in isolation (short) vs. (b) integrated into complex queries (long). Looking at 16 LLMs across 600 queries, we find a systematic misalignment of answers to the corresponding short and long queries. We further uncover position-dependent accuracy loss and momentum effects where consecutive correct or incorrect answers create self-reinforcing patterns. Through mechanistic analysis, we find that aligned facts activate overlapping model internals, and that metrics based on mechanistic similarity can predict short-long answer
alignment with up to 78% accuracy. Our work establishes factual consistency over query complexity as an important aspect of LLMs' trustworthiness and challenges current evaluation practices, which implicitly assume that good performance for simple factual queries implies reliability in more complex knowledge-seeking tasks too.
Read the full abstract ▾
Computation and Language
Domain-Specific Data Generation Framework for RAG Adaptation
Chris Xing Tian, Weihao Xie, Zhen Chen et al. (+5 more)
Retrieval-Augmented Generation (RAG) combines the
language understanding and
reasoning power of large
language models (LLMs) with external
retrieval to enable domain-grounded responses. Effectively adapting RAG systems to domain-specific settings requires specialized, context-rich training data beyond general-purpose question-answering. Here, we propose RAGen, a scalable and modular framework for generating domain-grounded question-answer-context (QAC) triples tailored to diverse RAG adaptation approaches. RAGen produces these QAC triples by identifying key concepts in documents, generating diverse questions guided by Bloom's Taxonomy-inspired principles, and pairing them with precise answers extracted from relevant contexts. RAGen supports multiple RAG adaptation strategies, including the optimization of key components such as the LLM, retriever, and
embedding model, etc. Its modular pipeline features semantic chunking, hierarchical concept extraction, and multi-chunk
retrieval, along with the introduction of curated distractor contexts to promote robust
reasoning. Designed for scalability, RAGen efficiently handles large and evolving document corpora without redundant processing, making it especially suitable for dynamic evolving domains such as scientific research and enterprise knowledge bases.
Read the full abstract ▾
All authors Chris Xing Tian, Weihao Xie, Zhen Chen, Zhengyuan Yi, Hui Liu, Haoliang Li, Shiqi Wang, Siwei Ma
Computation and Language
Discursive Circuits: How Do Language Models Understand Discourse Relations?
Yisong Miao and Min-Yen Kan
Which components in
transformer language models are responsible for discourse understanding? We hypothesize that sparse computational graphs, termed as discursive
circuits, control how models process discourse relations. Unlike simpler tasks, discourse relations involve longer spans and complex
reasoning. To make
circuit discovery feasible, we introduce a task called Completion under Discourse Relation (CuDR), where a model completes a discourse given a specified relation. To support this task, we construct a corpus of minimal contrastive pairs tailored for activation patching in
circuit discovery. Experiments show that sparse
circuits ($\approx 0.2\%$ of a full GPT-2 model) recover discourse understanding in the English PDTB-based CuDR task. These
circuits generalize well to unseen discourse frameworks such as RST and SDRT. Further analysis shows lower layers capture linguistic features such as lexical semantics and coreference, while upper layers encode discourse-level abstractions. Feature utility is consistent across frameworks (e.g., coreference supports Expansion-like relations).
Read the full abstract ▾
Computation and Language
Johannes Moll, Markus Graf, Tristan Lemke et al. (+7 more)
Vision-
language models (VLMs) often produce chain-of-thought (CoT) explanations that sound plausible yet fail to reflect the underlying decision process, undermining trust in high-stakes clinical use. Existing evaluations rarely catch this misalignment, prioritizing answer accuracy or adherence to formats. We present a clinically grounded framework for chest X-ray visual question answering (VQA) that probes CoT faithfulness via controlled text and image modifications across three axes: clinical fidelity, causal attribution, and confidence calibration. In a reader study (n=4), evaluator-radiologist correlations fall within the observed inter-radiologist range for all axes, with strong
alignment for attribution (Kendall's $\tau_b=0.670$), moderate
alignment for fidelity ($\tau_b=0.387$), and weak
alignment for confidence tone ($\tau_b=0.091$), which we report with caution. Benchmarking six VLMs shows that answer accuracy and explanation quality can be decoupled, acknowledging injected cues does not ensure grounding, and text cues shift explanations more than visual cues. While some open-source models match final answer accuracy, proprietary models score higher on attribution (25.0% vs. 1.4%) and often on fidelity (36.1% vs. 31.7%), highlighting deployment risks and the need to evaluate beyond final answer accuracy.
Read the full abstract ▾
All authors Johannes Moll, Markus Graf, Tristan Lemke, Nicolas Lenhart, Daniel Truhn, Jean-Benoit Delbrouck, Jiazhen Pan, Daniel Rueckert, Lisa C. Adams, Keno K. Bressem
Computation and Language · Cross-listed from cs.LG
Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalization
Zhengyu Chen, Jinluan Yang, Teng Xiao et al. (+6 more)
Recent advances in large
language models (LLMs) have demonstrated remarkable capabilities in
reasoning and tool utilization. However, the generalization of tool-augmented reinforcement learning (RL) across diverse domains remains a significant challenge. Standard paradigms often treat tool usage as a linear or isolated event, which becomes brittle when transferring skills from restricted domains (e.g., mathematics) to open-ended tasks. In this work, we investigate the cross-domain generalization of an LLM
agent trained exclusively on mathematical problem-solving. To facilitate robust skill transfer, we propose a {Reinforcement Learning for Interleaved Tool Execution (RITE)}. Unlike traditional methods, RITE enforces a continuous “Plan-Action-Reflection” cycle, allowing the model to ground its
reasoning in intermediate tool outputs and self-correct during long-horizon tasks. To effectively train this complex interleaved policy, we introduce {Dr. GRPO}, a robust optimization objective that utilizes
token-level loss aggregation with importance sampling to mitigate reward sparsity and high-variance credit assignment. Furthermore, we employ a dual-component reward system and dynamic curriculum via online rollout filtering to ensure structural integrity and sample efficiency. Extensive experiments reveal that our approach, despite being trained solely on math tasks, achieves state-of-the-art performance across diverse
reasoning domains, demonstrating high
token efficiency and strong generalization capabilities.
Read the full abstract ▾
All authors Zhengyu Chen, Jinluan Yang, Teng Xiao, Ruochen Zhou, Luan Zhang, Xiangyu Xi, Xiaowei Shi, Wei Wang, Jinggang Wang
Computation and Language · Cross-listed from cs.LG
EAGer: Entropy-Aware GEneRation for Adaptive Inference-Time Scaling
Daniel Scalena, Leonidas Zotos, Elisabetta Fersini et al. (+2 more)
With the rise of
reasoning language models and test-time scaling methods as a paradigm for improving model performance, substantial computation is often required to generate multiple candidate sequences from the same
prompt. This enables exploration of different
reasoning paths toward the correct solution, however, allocates the same compute budget for each
prompt. Grounded on the assumption that different
prompts carry different degrees of complexity, and thus different computation needs, we propose EAGer, a training-free generation method that leverages model uncertainty through
token-wise
entropy distribution to reduce redundant computation and concurrently improve overall performance. EAGer allows branching to multiple
reasoning paths only in the presence of high-
entropy tokens, and reallocates the saved compute budget to instances where exploration of alternative paths is most needed. We validate EAGer across multiple open-source models on complex
reasoning benchmarks, with gains specifically demonstrated on AIME 2025. When target labels are accessible – as in RLVR training pipelines – EAGer achieves up to +37% in Pass@k and 59% fewer
tokens; in test-time settings it still yields +12% in Pass@k and 64% fewer
tokens compared to Full Parallel Sampling.
Read the full abstract ▾
All authors Daniel Scalena, Leonidas Zotos, Elisabetta Fersini, Malvina Nissim, Ahmet Üstün
Computation and Language · Cross-listed from cs.LG
ELMO: Efficiency via Low-precision and Peak Memory Optimization in Large Output Spaces
Jinbin Zhang, Nasib Ullah, Erik Schultheis et al. (+1 more)
Large output spaces, also referred to as Extreme multilabel classification (XMC), is a setting that arises, e.g., in large-scale tagging and product-to-product recommendation, and is characterized by the number of labels ranging from hundreds of thousands to millions. This means that the linear classification head, usually only a tiny fraction of the overall model, turns into the main driver for compute and
memory demand. Current state-of-the-art XMC methods predominantly rely on FP16-FP32 mixed-precision training, which we show can be unstable, and inefficient in terms of
memory usage and computational overhead. Meanwhile, existing low-precision methods typically retain higher precision for the classification layer. In this work, we propose ELMO, a pure low-precision training framework for XMC models using BFloat16 and Float8 data types. By leveraging Kahan summation and
stochastic rounding, we demonstrate that XMC models can be effectively trained entirely in Float8, without relying on single-precision master weights or tensor scaling. Low-precision training, combined with our proposed
memory optimizations – gradient fusion and chunking – enables significant reductions in GPU
memory usage. For example, we train a 3-million-label XMC model with only 6.6 GiB of GPU
memory, compared to the 39.7 GiB required by the optimized SOTA method, Renee without compromising accuracy.
Read the full abstract ▾
All authors Jinbin Zhang, Nasib Ullah, Erik Schultheis, Rohit Babbar
Computation and Language
Bridging Gaps in Hate Speech Detection: Meta-Collections and Benchmarks for Low-Resource Iberian Languages
Paloma Piot, José Ramom Pichel Campos and Javier Parapar
Hate
speech poses a serious threat to social cohesion and individual well-being, particularly on social media, where it spreads rapidly. While research on hate
speech detection has progressed, it remains largely focused on English, resulting in limited resources and
benchmarks for low-resource
languages. Moreover, many of these
languages have multiple linguistic varieties, a factor often overlooked in current approaches. At the same time, large
language models require substantial amounts of data to perform reliably, a requirement that low-resource
languages often cannot meet. In this work, we address these gaps by compiling a meta-collection of hate
speech datasets for European Spanish, standardised with unified labels and metadata. This collection is based on a systematic analysis and integration of existing resources, aiming to bridge the data gap and support more consistent and scalable hate
speech detection. We extended this collection by translating it into European Portuguese and into a Galician standard that is more convergent with Spanish and another Galician variant that is more convergent with Portuguese, creating aligned
multilingual corpora. Using these resources, we establish new
benchmarks for hate
speech detection in Iberian
languages. We evaluate state-of-the-art large
language models in zero-shot, few-shot, and fine-tuning settings, providing baseline results for future research. Moreover, we perform a cross-lingual analysis with our target
languages. Our findings underscore the importance of
multilingual and variety-aware approaches in hate
speech detection and offer a foundation for improved benchmarking in underrepresented European
languages.
Read the full abstract ▾
Computation and Language
One Size Does Not Fit All: Exploring Variable Thresholds for Distance-Based Multi-Label Text Classification
Jens Van Nooten, Andriy Kosar, Guy De Pauw et al. (+1 more)
Distance-based unsupervised text classification is a method within text classification that leverages the semantic similarity between a label and a text to determine label relevance. This method provides numerous benefits, including fast inference and adaptability to expanding label sets, as opposed to zero-shot, few-shot, and fine-tuned neural networks that require re-training in such cases. In multi-label distance-based classification and information
retrieval algorithms, thresholds are required to determine whether a text instance is "similar" to a label or query. Similarity between a text and label is determined in a dense
embedding space, usually generated by state-of-the-art sentence encoders. Multi-label classification complicates matters, as a text instance can have multiple true labels, unlike in multi-class or binary classification, where each instance is assigned only one label. We expand upon previous literature on this underexplored topic by thoroughly examining and evaluating the ability of sentence encoders to perform distance-based classification. First, we perform an exploratory study to verify whether the semantic relationships between texts and labels vary across models,
datasets, and label sets by conducting experiments on a diverse collection of realistic multi-label text classification (MLTC)
datasets. We find that similarity distributions show statistically significant differences across models,
datasets and even label sets. We propose a novel method for optimizing label-specific thresholds using a validation set. Our label-specific thresholding method achieves an average improvement of 46% over normalized 0.5 thresholding and outperforms uniform thresholding approaches from previous work by an average of 14%. Additionally, the method demonstrates strong performance even with limited labeled examples.
Read the full abstract ▾
All authors Jens Van Nooten, Andriy Kosar, Guy De Pauw, Walter Daelemans
Computation and Language
TypePilot: Leveraging the Scala Type System for Secure LLM-generated Code
Alexander Sternfeld, Andrei Kucharavy and Ljiljana Dolamic
Large
language Models (LLMs) have shown remarkable proficiency in code generation tasks across various programming
languages. However, their outputs often contain subtle but critical vulnerabilities, posing significant risks when deployed in security-sensitive or mission-critical systems. This paper introduces TypePilot, an agentic AI framework designed to enhance the security and robustness of LLM-generated code by leveraging strongly typed and verifiable
languages, using Scala as a representative example. We evaluate the effectiveness of our approach in two settings: formal verification with the Stainless framework and general-purpose secure code generation. Our experiments with leading open-source LLMs reveal that while direct code generation often fails to enforce safety constraints, just as naive prompting for more secure code, our type-focused agentic pipeline substantially mitigates input validation and injection vulnerabilities. The results demonstrate the potential of structured, type-guided LLM workflows to improve the SotA of the trustworthiness of automated code generation in high-assurance domains.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
$How^{2}$: How to learn from procedural How-to questions
Gautier Dagan, Frank Keller and Alex Lascarides
An
agent facing a planning problem can use answers to how-to questions to reduce uncertainty and fill knowledge gaps, helping it solve both current and future tasks. However, their open ended nature, where valid answers to "How do I X?" range from executable actions to high-level descriptions of X's sub-goals, makes them challenging for AI
agents to ask, and for AI experts to answer, in ways that support efficient planning. We introduce $How^{2}$, a
memory agent framework that enables
agents to ask how-to questions, store the answers, and reuse them for lifelong learning in interactive environments. We evaluate our approach in Plancraft, a Minecraft crafting environment, where
agents must complete an assembly task by manipulating inventory items. Using teacher models that answer at varying levels of abstraction, from executable action sequences to high-level subgoal descriptions, we show that lifelong learning
agents benefit most from answers that are abstracted and decoupled from the current state. $How^{2}$ offers a way for LLM-based
agents to improve their planning capabilities over time by asking questions in interactive environments.
Read the full abstract ▾
Computation and Language
Enhancing LLM Reasoning via Non-Human-Like Reasoning Path Preference Optimization
Junjie Lu, Yuliang Liu, Chaofeng Qu et al. (+4 more)
Current approaches for strengthening LLM
reasoning tend to introduce a training bias toward human-like
reasoning trajectories. In step-wise preference optimization, in particular, dependence on human or higher-capacity model annotations for intermediate steps limits exploration of alternative, non-human-like
reasoning paths and thus constrains achievable performance. Furthermore, through a small-scale pilot study, we observed that in approximately 75% of cases, the model's first erroneous step occurs after the lowest-confidence point. This suggests that guiding the model at its lowest-confidence point before an error provides more accurate supervision than locating the first explicit error. In this paper, we propose Confidence-Guided
Reasoning Path Preference Optimization (CGPO), a method that leverages a confidence signal to identify points of maximal uncertainty in the model's
reasoning process and applies self-generated, non-human-like
reasoning-path guidance to mitigate trajectory drift. Our experiments span diverse models applied to both code and mathematical
reasoning tasks. The results show that, with the same amount of training data, our method using data generated by a small model can achieve better performance in most cases compared with approaches using data generated by a strong model or human-annotated.
Read the full abstract ▾
All authors Junjie Lu, Yuliang Liu, Chaofeng Qu, Wei Shen, Zhouhan Lin, Chuheng Zhang, Min Xu
Computation and Language · Cross-listed from cs.SD
VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents
Jiliang Hu, Wenfu Wang, Zuchao Li et al. (+6 more)
Recent advances in large audio
language models (LALMs) have greatly enhanced
multimodal conversational systems. However, existing
benchmarks remain limited – they are mainly English-centric, rely on synthetic
speech, and lack comprehensive, discriminative evaluation across multiple dimensions. To address these gaps, we present Voice Chat Bot Bench (VCB Bench) – a high-quality Chinese
benchmark built entirely on real human
speech. VCB Bench evaluates LALMs from three complementary perspectives: instruction following (including
speech-level control beyond text commands), knowledge understanding (general knowledge,
reasoning, and daily dialogue), and robustness (stability under perturbations in content, environment, and speaker traits). Experiments on representative LALMs reveal notable performance gaps and highlight future directions for improvement. VCB Bench provides a reproducible and fine-grained evaluation framework, offering standardized methodology and practical insights for advancing Chinese voice conversational models.
Read the full abstract ▾
All authors Jiliang Hu, Wenfu Wang, Zuchao Li, Chenxing Li, Yiyang Zhao, Hanzhao Li, Liqiang Zhang, Meng Yu, Dong Yu
Computation and Language
Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States
Qinglin Zhu, Yizhen Yao, Runcong Zhao et al. (+8 more)
Autoregressive (AR) models remain the dominant paradigm for natural
language generation, but their strictly sequential decoding process leads to high inference latency. Recent
diffusion-inspired
language models, such as LLaDA and Dream, alleviate this issue through parallel generation, yet they still face two key limitations: information loss, since predictive distributions over non-finalised
tokens are discarded at each step, and unstable commitment dynamics, where local
token decisions are not sufficiently coordinated at the global level. We propose Latent Refinement Decoding (LRD), a two-stage decoding framework consisting of Latent Refinement and a Predictive Feedback Loop. In the first stage, LRD keeps masked positions as distributional mixtures of predicted
tokens and the mask
embedding, enabling the model to form more globally consistent beliefs before committing
tokens. In the second stage, LRD progressively finalises confident
tokens while preserving uncertain ones for further iterative refinement, with KL-divergence dynamics serving as a stable criterion for convergence and early stopping. Experiments show that LRD improves performance on both coding tasks, including HumanEval by 6.3 and MBPP by 2.6, and
reasoning tasks, including GSM8K by 2.9 and MATH500 by 3.8, while achieving up to 10.6x overall speedup. Moreover, LRD is compatible with single-
token decoding, multi-
token threshold commitment, and system-level accelerators; when combined with Fast-dLLM, it reaches up to 18.4x speedup over vanilla decoding while further improving accuracy.
Read the full abstract ▾
All authors Qinglin Zhu, Yizhen Yao, Runcong Zhao, Yanzheng Xiang, Siya Qi, Amrutha Saseendran, Chen Jin, Philip Teare, Bin Liang, Yulan He, Lin Gui