Computation and Language · Cross-listed from cs.PL
LLMs Lean on Priors, Not Programming Language Semantics
Aditya Thimmaiah, Jiyang Zhang, Jayanth Srinivasa et al. (+2 more)
Recent work asks whether large
language models (LLMs) condition their
reasoning on explicit rules rather than statistical regularities from pretraining. Program execution provides a canonical instance: formal semantics define behavior through symbolic transition rules that can be systematically altered under distribution shift. We investigate whether LLMs can condition their
reasoning on formal semantics through program execution and introduce PLSemanticsBench, pairing featherweight C programs with two semantic systems – small-step operational semantics and K semantics – and probing four capabilities: composing rules for final states, selecting rules when state is unmutated, sustaining such conditioning over long traces, and following supplied rules under novel semantics. To decouple semantic
reasoning from syntactic familiarity, we redefine familiar operators to induce symbol-meaning conflict and introduce novel symbols defined only through the supplied rules, and stress-test models on Human-Written, LLM-Translated, and Fuzzer-Generated splits with increasing structural complexity. Across 11 frontier LLMs, strong final-state accuracy under standard semantics (up to 90%) drops sharply – by as much as 40–60% points – under semantic mutations and increasing structural complexity. Only a handful of models achieve non-zero long-horizon conditioning accuracy, and even the best systems reach just 35%. Together, these results suggest that contemporary LLMs often rely on pretrained lexical associations rather than systematically conditioning on supplied formal rules. PLSemanticsBench is publicly available at https://EngineeringSoftware.github.io/PLSemanticsBench.
Read the full abstract ▾
All authors Aditya Thimmaiah, Jiyang Zhang, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric
Computation and Language · Cross-listed from cs.AI
Know Thyself? On the Incapability and Implications of AI Self-Recognition
Xiaoyan Bai, Aryan Shrivastava, Ari Holtzman et al. (+1 more)
Self-recognition is a crucial metacognitive capability for AI systems, relevant not only for psychological analysis but also for safety, particularly in evaluative scenarios. Motivated by contradictory interpretations of whether models possess self-recognition (Panickssery et al., 2024; Davidson et al., 2024), we introduce a systematic evaluation framework that can be easily applied and updated. Specifically, we measure how well 10 contemporary larger
language models (LLMs) can identify their own generated text versus text from other models through two tasks: binary self-recognition and exact model prediction. Different from prior claims, our results reveal a consistent failure in self-recognition. Only 4 out of 10 models predict themselves as generators, and the performance is rarely above random chance. Additionally, models exhibit a strong bias toward predicting GPT and Claude families. We also provide the first evaluation of model awareness of their own and others' existence, as well as the
reasoning behind their choices in self-recognition. We find that the model demonstrates some knowledge of its own existence and other models, but their
reasoning reveals a hierarchical bias. They appear to assume that GPT, Claude, and occasionally Gemini are the top-tier models, often associating high-quality text with them. We conclude by discussing the implications of our findings on AI safety and future directions to develop appropriate AI self-awareness.
Read the full abstract ▾
All authors Xiaoyan Bai, Aryan Shrivastava, Ari Holtzman, Chenhao Tan
Computation and Language · Cross-listed from cs.LG
Studying the Korean Word-Chain Game with RLVR: Mitigating Reward Conflicts via Curriculum Learning
Donghwan Rho
Reinforcement learning with verifiable rewards (RLVR) is a promising approach for training large
language models (LLMs) with stronger
reasoning abilities. It has also been applied to a variety of logic puzzles. In this work, we study the Korean word-chain game using RLVR. We show that rule-derived rewards can naturally conflict, and demonstrate through experiments that a curriculum-learning scheme mitigates these conflicts. Our findings motivate further studies of puzzle tasks in diverse
languages.
Read the full abstract ▾
Computation and Language
Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks
Arjun Arunasalam, Madison Pickering, Z. Berkay Celik et al. (+1 more)
Large
language models (LLMs) can underpin AI assistants that help users with everyday tasks, such as by making recommendations or performing basic computation. Despite AI assistants' promise, little is known about the implicit values these assistants display while completing subjective everyday tasks. Humans may consider values like environmentalism, charity, and diversity. To what extent do LLMs exhibit these values in completing everyday tasks? How do they compare with humans? We answer these questions by
auditing how six popular LLMs complete 30 everyday tasks, comparing LLMs to each other and to 100 human crowdworkers from the US. We find LLMs often do not align with humans, nor with other LLMs, in the implicit values exhibited.
Read the full abstract ▾
All authors Arjun Arunasalam, Madison Pickering, Z. Berkay Celik, Blase Ur
Computation and Language · Cross-listed from cs.CY
Lightweight Prompt Engineering for Cognitive Alignment in Educational AI: A OneClickQuiz Case Study
Antoun Yaacoub, Zainab Assaghir and Jérôme Da-Rugna
The rapid integration of Artificial Intelligence (AI) into educational technology promises to revolutionize content creation and assessment. However, the quality and pedagogical
alignment of AI-generated content remain critical challenges. This paper investigates the impact of lightweight
prompt engineering strategies on the cognitive
alignment of AI-generated questions within OneClickQuiz, a Moodle plugin leveraging generative AI. We evaluate three
prompt variants-a detailed baseline, a simpler version, and a persona-based approach-across Knowledge, Application, and Analysis levels of Bloom's Taxonomy. Utilizing an automated classification model (from prior work) and human review, our findings demonstrate that explicit, detailed
prompts are crucial for precise cognitive
alignment. While simpler and persona-based
prompts yield clear and relevant questions, they frequently misalign with intended Bloom's levels, generating outputs that are either too complex or deviate from the desired cognitive objective. This study underscores the importance of strategic
prompt engineering in fostering pedagogically sound AI-driven educational solutions and advises on optimizing AI for quality content generation in learning analytics and smart learning environments.
Read the full abstract ▾
Computation and Language
Reward Models are Metrics in a Trench Coat
Sebastian Gehrmann
The emergence of reinforcement learning in post-training of large
language models has sparked significant interest in reward models. Reward models assess the quality of sampled model outputs to generate training signals. This task is also performed by evaluation metrics that monitor the performance of an AI model. We find that the two research areas are mostly separate, leading to redundant terminology and repeated pitfalls. Common challenges include susceptibility to spurious correlations, impact on downstream reward hacking, methods to improve data quality, and approaches to meta-evaluation. Our position paper argues that a closer collaboration between the fields can help overcome these issues. To that end, we show how metrics outperform reward models on specific tasks and provide an extensive survey of the two areas. Grounded in this survey, we point to multiple research topics in which closer
alignment can improve reward models and metrics in areas such as preference elicitation methods, avoidance of spurious correlations and reward hacking, and calibration-aware meta-evaluation.
Read the full abstract ▾
Computation and Language
Hongxiang Zhang, Yuan Tian and Tianyi Zhang
Large
Language Models (LLMs) tend to generate a long
reasoning chain when solving complex tasks. However, as the
reasoning chain extends, critical intermediate steps and the original
prompt will be buried in the context, receiving insufficient
attention and leading to errors. In this work, we present ATAR, a novel
reasoning method that leverages the inherent
reasoning structure to steer LLM
attention. Our experiments show that ATAR outperforms SOTA methods across six
benchmarks, achieving up to 15.39% absolute improvement. Furthermore, with ATAR, "non-
reasoning" models achieve comparable or even better performance compared to
reasoning models of the same size in most
benchmarks. Finally, our ablation studies show that the
attention alignment component contributes significantly, and that these improvements are persist under different attentionsteering backends.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
Guanhua Huang, Tingqiang Xu, Mingze Wang et al. (+7 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has propelled Large
Language Models in complex
reasoning, yet its scalability is often hindered by a training bottleneck where performance plateaus as policy
entropy collapses, signaling a loss of exploration. Previous methods typically address this by maintaining high policy
entropy, yet the precise mechanisms that govern meaningful exploration have remained underexplored. Our analysis suggests that an unselective focus on
entropy risks amplifying irrelevant
tokens and destabilizing training. This paper investigates the exploration dynamics within RLVR and identifies a key issue: the gradual elimination of valuable low-probability exploratory
tokens, which we term \textbf{
reasoning sparks}. We find that while abundant in pre-trained models, these sparks are systematically extinguished during RLVR due to over-penalization, leading to a degeneracy in exploration. To address this, we introduce Low-probability Regularization (Lp-Reg). Its core mechanism regularizes the policy towards a heuristic proxy distribution. This proxy is constructed by filtering out presumed noise
tokens and re-normalizing the distribution over the remaining candidates. The result is a less-noisy proxy where the probability of
reasoning sparks is amplified, which then serves as a soft regularization target to shield these valuable
tokens from elimination via KL divergence. Experiments show that Lp-Reg enables stable on-policy RL, sustaining continuous scaling across $3,000$ training steps and $81,204$ GPU-hours, where baseline
entropy-control methods collapse. This sustained exploration leads to state-of-the-art performance, achieving a $60.17\%$ average accuracy on five math
benchmarks, an improvement of $2.66\%$ over prior methods. Code is available at https://github.com/CarlanLark/Lp-Reg.
Read the full abstract ▾
All authors Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, Xue Gong, Siheng Li, Ruibin Xiong, Kejiao Li, Yuhao Jiang, Bo Zhou
Computation and Language
Cache-to-Cache: Direct Semantic Communication Between Large Language Models
Tianyu Fu, Zihan Min, Hanling Zhang et al. (+4 more)
Multi-LLM systems harness the complementary strengths of diverse Large
Language Models, achieving performance and efficiency gains that are not attainable by a single model. In existing designs, LLMs communicate through text, forcing internal representations to be transformed into output
token sequences. This process both loses rich semantic information and incurs
token-by-
token generation latency. Motivated by these limitations, we ask: Can LLMs communicate beyond text? Oracle experiments show that enriching the KV-Cache semantics can improve response quality without increasing cache size, supporting KV-Cache as an effective medium for inter-model communication. Thus, we propose Cache-to-Cache (C2C), a new paradigm for direct semantic communication between LLMs. C2C uses a neural network to project and fuse the source model's KV-cache with that of the target model to enable direct semantic transfer. A learnable gating mechanism selects the target layers that benefit from cache communication. Compared with text communication, C2C utilizes the deep, specialized semantics from both models, while avoiding explicit intermediate text generation. Experiments show that C2C achieves 6.4-14.2% higher average accuracy than individual models. It further outperforms the text communication paradigm by approximately 3.1-5.4%, while delivering an average 2.5x speedup in latency. Our code is available at https://github.com/thu-nics/C2C.
Read the full abstract ▾
All authors Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, Yu Wang
Computation and Language · Cross-listed from cs.AI
Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner
Cai Zhou, Chenxiao Yang, Yi Hu et al. (+7 more)
Diffusion language models, especially masked discrete
diffusion models, have achieved great success recently. While there are some theoretical and primary empirical results showing the advantages of latent
reasoning with looped
transformers or continuous chain-of-thoughts, continuous
diffusion models typically underperform their discrete counterparts. In this paper, we argue that
diffusion language models do not necessarily need to be in the discrete space. In particular, we prove that continuous
diffusion models have stronger expressivity than discrete
diffusions and looped
transformers. We attribute the contradiction between the theoretical expressiveness and empirical performance to their practical trainability: while continuous
diffusion provides intermediate supervision that looped
transformers lack, they introduce additional difficulty decoding
tokens into the discrete
token space from the continuous representation space. We therefore propose Coevolutionary Continuous Discrete
Diffusion (CCDD), which defines a joint
multimodal diffusion process on the union of a continuous representation space and a discrete
token space, leveraging a single model to simultaneously denoise in the joint space. By combining two modalities, CCDD is expressive with rich semantics in the latent space, as well as good trainability and sample quality with the help of explicit discrete
tokens. We also propose effective architectures and advanced training/sampling techniques for CCDD, which reveals strong empirical performance in extensive
language modeling experiments on real-world tasks.
Read the full abstract ▾
All authors Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, Dinghuai Zhang
Computation and Language
FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agents
Imene Kerboua, Sahar Omidi Shayegan, Megh Thakkar et al. (+7 more)
Web
agents powered by large
language models (LLMs) must process lengthy web page observations to complete user goals; these pages often exceed tens of thousands of
tokens. This saturates context limits and increases computational cost processing; moreover, processing full pages exposes
agents to security risks such as
prompt injection. Existing
pruning strategies either discard relevant content or retain irrelevant context, leading to suboptimal action prediction. We introduce FocusAgent, a simple yet effective approach that leverages a lightweight LLM retriever to extract the most relevant lines from accessibility tree (AxTree) observations, guided by task goals. By
pruning noisy and irrelevant content, FocusAgent enables efficient
reasoning while reducing vulnerability to injection attacks. Experiments on WorkArena and WebArena
benchmarks show that FocusAgent matches the performance of strong baselines, while reducing observation size by over 50%. Furthermore, a variant of FocusAgent significantly reduces the success rate of
prompt-injection attacks, including banner and pop-up attacks, while maintaining task success performance in attack-free settings. Our results highlight that targeted LLM-based
retrieval is a practical and robust strategy for building web
agents that are efficient, effective, and secure.
Read the full abstract ▾
All authors Imene Kerboua, Sahar Omidi Shayegan, Megh Thakkar, Xing Han Lù, Léo Boisvert, Massimo Caccia, Jérémy Espinas, Alexandre Aussem, Véronique Eglin, Alexandre Lacoste
Computation and Language
Model-Based Ranking of Source Languages for Zero-Shot Cross-Lingual Transfer
Abteen Ebrahimi, Adam Wiemerslage and Katharina von der Wense
We present NN-Rank, an
algorithm for ranking source
languages for cross-lingual transfer, which leverages hidden representations from
multilingual models and unlabeled target-
language data. We experiment with two pretrained
multilingual models and two tasks: part-of-
speech tagging (POS) and named entity recognition (NER). We consider 51 source
languages and evaluate on 56 and 72 target
languages for POS and NER, respectively. When using in-domain data, NN-Rank beats state-of-the-art baselines that leverage lexical and linguistic features, with average improvements of up to 35.56 NDCG for POS and 18.14 NDCG for NER. As prior approaches can fall back to
language-level features if target
language data is not available, we show that NN-Rank remains competitive using only the Bible, an out-of-domain corpus available for a large number of
languages. Ablations on the amount of unlabeled target data show that, for subsets consisting of as few as 25 examples, NN-Rank produces high-quality rankings which achieve 92.8% of the NDCG achieved using all available target data for ranking.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.RO
Simulation to Rules: A Dual-VLM Framework for Formal Visual Planning
Yilun Hao, Yongchao Chen, Chuchu Fan et al. (+1 more)
Vision Language Models (VLMs) show strong potential for visual planning but struggle with precise spatial and long-horizon
reasoning, while Planning Domain Definition
Language (PDDL) planners excel at formal long-horizon planning but cannot interpret visual inputs. Recent works combine these complementary advantages by translating visual problems into PDDL. However, while VLMs can generate PDDL problem files satisfactorily, accurately generating PDDL domain files, which encode planning rules, remains challenging and typically requires human expertise or environment interaction. We propose VLMFP, a Dual-VLM-guided framework that autonomously generates both PDDL problem and domain files for formal visual planning. VLMFP combines a SimVLM that simulates action consequences with a GenVLM that generates and iteratively refines PDDL files by aligning symbolic execution with simulated outcomes, enabling multiple levels of generalization across unseen instances, visual appearances, and game rules. We evaluate VLMFP on 6 grid-world domains and demonstrate its generalization capability. On average, SimVLM achieves 87.3% and 86.0% scenario understanding and action simulation for seen and unseen appearances, respectively. With the guidance of SimVLM, VLMFP attains 70.0%, 54.1% planning success on unseen instances in seen and unseen appearances, respectively. We further demonstrate that VLMFP scales to complex long-horizon 3D planning tasks, including multi-
robot collaboration and assembly scenarios with partial observability and diverse visual variations. Project page: https://sites.google.com/view/vlmfp.
Read the full abstract ▾
All authors Yilun Hao, Yongchao Chen, Chuchu Fan, Yang Zhang
Computation and Language · Cross-listed from cs.SE
When Names Disappear: Revealing What LLMs Actually Understand About Code
Cuong Chi Le, Minh V.T. Pham, Cuong Duc Van et al. (+3 more)
Large
Language Models (LLMs) achieve strong results on code tasks, but how they derive program meaning remains unclear. We argue that code communicates through two channels: structural semantics, which define formal behavior, and human-interpretable naming, which conveys intent. Removing the naming channel severely degrades intent-level tasks such as summarization, where models regress to line-by-line descriptions. Surprisingly, we also observe consistent reductions on execution tasks that should depend only on structure, revealing that current
benchmarks reward memorization of naming patterns rather than genuine semantic
reasoning. To disentangle these effects, we introduce a suite of semantics-preserving obfuscations and show that they expose identifier leakage across both summarization and execution. Building on these insights, we release ClassEval-Obf, an obfuscation-enhanced
benchmark that systematically suppresses naming cues while preserving behavior. Our results demonstrate that ClassEval-Obf reduces inflated performance gaps, weakens memorization shortcuts, and provides a more reliable basis for assessing LLMs' code understanding and generalization.
Read the full abstract ▾
All authors Cuong Chi Le, Minh V.T. Pham, Cuong Duc Van, Hoang N. Phan, Huy N. Phan, Tien N. Nguyen
Computation and Language
LLM as Attention-Informed NTM and Topic Modeling as long-input Generation: Interpretability and long-Context Capability
Xuan Xu, Zhongliang Yang, Haolun Li et al. (+5 more)
Topic modeling aims to produce interpretable topic representations and topic–document correspondences from corpora, but classical neural topic models (NTMs) remain constrained by limited representation assumptions and semantic abstraction ability. We study LLM-based topic modeling from both white-box and black-box perspectives. For white-box LLMs, we propose an
attention-informed framework that recovers interpretable structures analogous to those in NTMs, including document-topic and topic-word distributions. This validates the view that LLM can serve as an
attention-informed NTM. For black-box LLMs, we reformulate topic modeling as a structured long-input task and introduce a post-generation signal compensation method based on diversified topic cues and hybrid
retrieval. Experiments show that recovered
attention structures support effective topic assignment and keyword extraction, while black-box long-context LLMs achieve competitive or stronger performance than other baselines. These findings suggest a connection between LLMs and NTMs and highlight the promise of long-context LLMs for topic modeling.
Read the full abstract ▾
All authors Xuan Xu, Zhongliang Yang, Haolun Li, Beilin Chu, Rui Tian, Yu Li, Shaolin Tan, Linna Zhou
Computation and Language
Neural Correlates of Language Models Are Specific to Human Language
Iñigo Parra
Previous work has shown correlations between the hidden states of large
language models and fMRI
brain responses, on
language tasks. These correlations have been taken as evidence of the representational similarity of these models and
brain states. This study tests whether these previous results are robust to several possible concerns. Specifically this study shows: (i) that the previous results are still found after dimensionality reduction, and thus are not attributable to the curse of dimensionality; (ii) that previous results are confirmed when using new measures of similarity; (iii) that correlations between
brain representations and those from models are specific to models trained on human
language; and (iv) that the results are dependent on the presence of positional encoding in the models. These results confirm and strengthen the results of previous research and contribute to the debate on the biological plausibility and interpretability of state-of-the-art large
language models.
Read the full abstract ▾
Computation and Language
EditLens: Quantifying the Extent of AI Editing in Text
Katherine Thai, Bradley Emi, Elyas Masrour et al. (+1 more)
A significant proportion of queries to large
language models ask them to edit user-provided text, rather than generate new text from scratch. While previous work focuses on detecting fully AI-generated text, we demonstrate that AI-edited text is distinguishable from human-written and AI-generated text. First, we propose using lightweight similarity metrics to quantify the magnitude of AI editing present in a text given the original human-written text and validate these metrics with human annotators. Using these similarity metrics as intermediate supervision, we then train EditLens, a regression model that predicts the amount of AI editing present within a text. Our model achieves state-of-the-art performance on both binary (F1=94.7%) and ternary (F1=90.4%) classification tasks in distinguishing human, AI, and mixed writing. Not only do we show that AI-edited text can be detected, but also that the degree of change made by AI to human writing can be detected, which has implications for authorship attribution, education, and policy. Finally, as a case study, we use our model to analyze the effects of AI-edits applied by Grammarly, a popular writing assistance tool. To encourage further research, we commit to publicly releasing our models and
dataset.
Read the full abstract ▾
All authors Katherine Thai, Bradley Emi, Elyas Masrour, Mohit Iyyer
Computation and Language
Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
Ej Zhou, Caiqi Zhang, Tiancheng Hu et al. (+4 more)
Confidence calibration, the
alignment of a model's predicted confidence with its actual accuracy, is crucial for the reliable deployment of Large
Language Models (LLMs). However, this critical property remains largely under-explored in
multilingual contexts. In this work, we conduct the first large-scale, systematic studies of
multilingual calibration across six model families and over 100
languages, revealing that non-English
languages suffer from systematically worse calibration. To diagnose this, we investigate the model's internal representations and find that the final layer, biased by English-centric training, provides a poor signal for
multilingual confidence. In contrast, our layer-wise analysis uncovers a key insight that late-intermediate layers consistently offer a more reliable and better-calibrated signal. Building on this, we introduce a suite of training-free methods, including
Language-Aware Confidence Ensemble (LACE), which adaptively selects an optimal ensemble of layers for each specific
language. Our study highlights the hidden costs of English-centric
alignment and offer a new path toward building more globally equitable and trustworthy LLMs by looking beyond the final layer.
Read the full abstract ▾
All authors Ej Zhou, Caiqi Zhang, Tiancheng Hu, Chengzu Li, Nigel Collier, Ivan Vulić, Anna Korhonen
Computation and Language
SurveyBench: Can LLM(-Agents) Write Academic Surveys that Align with Reader Needs?
Zhaojun Sun, Xuzhou Zhu, Xuanhe Zhou et al. (+6 more)
Academic survey writing, which distills vast literature into a coherent and insightful narrative, remains a labor-intensive and intellectually demanding task. While recent approaches, such as general DeepResearch
agents and survey-specialized methods, can generate surveys automatically (a.k.a. LLM4Survey), their outputs often fall short of human standards and there lacks a rigorous, reader-aligned
benchmark for thoroughly revealing their deficiencies. To fill the gap, we propose a fine-grained, quiz-driven evaluation framework SurveyBench, featuring (1) typical survey topics source from recent 11,343 arXiv papers and corresponding 4,947 high-quality surveys; (2) a multifaceted metric hierarchy that assesses the outline quality (e.g., coverage breadth, logical coherence), content quality (e.g., synthesis granularity, clarity of insights), and non-textual richness; and (3) a dual-mode evaluation protocol that includes content-based and quiz-based answerability tests, explicitly aligned with readers' informational needs. Results show SurveyBench effectively challenges existing LLM4Survey approaches (e.g., on average 21% lower than human in content-based evaluation).
Read the full abstract ▾
All authors Zhaojun Sun, Xuzhou Zhu, Xuanhe Zhou, Xin Tong, Shuo Wang, Jie Fu, Guoliang Li, Zhiyuan Liu, Fan Wu
Computation and Language
Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation
Jacobo Romero-Díaz, Gerard I. Gállego, Oriol Pareras et al. (+3 more)
Speech-to-Text Translation (S2TT) systems built from Automatic
Speech Recognition (ASR) and Text-to-Text Translation (T2TT) modules face two major limitations: error propagation and the inability to exploit prosodic or other acoustic cues. Chain-of-Thought (CoT) prompting has recently been introduced, with the expectation that jointly accessing
speech and transcription will overcome these issues. Analyzing CoT through attribution methods, robustness evaluations with corrupted transcripts, and prosody-awareness, we find that it largely mirrors cascaded behavior, relying mainly on transcripts while barely leveraging
speech. Simple training interventions, such as adding Direct S2TT data or noisy transcript injection, enhance robustness and increase
speech attribution. These findings challenge the assumed advantages of CoT and highlight the need for architectures that explicitly integrate acoustic information into translation.
Read the full abstract ▾
All authors Jacobo Romero-Díaz, Gerard I. Gállego, Oriol Pareras, Federico Costa, Javier Hernando, Cristina España-Bonet
Computation and Language
Semantic Similarity in Radiology Reports via LLMs and NER
Beth Pearson, Ahmed Adnan and Zahraa S. Abdallah
Radiology report evaluation is a crucial part of radiologists' training and plays a key role in ensuring diagnostic accuracy. As part of the standard reporting workflow, a junior radiologist typically prepares a preliminary report, which is then reviewed and edited by a senior radiologist to produce the final report. Identifying semantic differences between preliminary and final reports is essential for junior doctors, both as a training tool and to help uncover gaps in clinical knowledge. While AI in radiology is a rapidly growing field, the application of large
language models (LLMs) remains challenging due to the need for specialised domain knowledge. In this paper, we explore the ability of LLMs to provide explainable and accurate comparisons of reports in the radiology domain. We begin by comparing the performance of several LLMs in comparing radiology reports. We then assess a more traditional approach based on Named-Entity-Recognition (NER). However, both approaches exhibit limitations in delivering accurate feedback on semantic similarity. To address this, we propose Llama-EntScore, a semantic similarity scoring method using a combination of Llama 3.1 and NER with tunable weights to emphasise or de-emphasise specific types of differences. Our approach generates a quantitative similarity score for tracking progress and also gives an interpretation of the score that aims to offer valuable guidance in reviewing and refining their reporting. We find our method achieves 67% exact-match accuracy and 93% accuracy within +/- 1 when compared to radiologist-provided ground truth scores - outperforming both LLMs and NER used independently. Code is available at: https://github.com/otmive/llama_reports
Read the full abstract ▾
Computation and Language
Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?
Oriol Pareras, Gerard I. Gállego, Federico Costa et al. (+2 more)
Recent work on
Speech-to-Text Translation (S2TT) has focused on LLM-based models, introducing the increasingly adopted Chain-of-Thought (CoT) prompting, where the model is guided to first transcribe the
speech and then translate it. CoT typically outperforms direct prompting primarily because it can exploit abundant Automatic
Speech Recognition (ASR) and Text-to-Text Translation (T2TT)
datasets to explicitly model its steps. In this paper, we systematically compare CoT and Direct prompting under increasing amounts of S2TT data. To this end, we pseudo-label an ASR corpus by translating its transcriptions into six European
languages, and train LLM-based S2TT systems with both prompting strategies at different data scales. Our results show that Direct improves more consistently as the amount of data increases, suggesting that it may become a more effective approach as larger S2TT resources are created.
Read the full abstract ▾
All authors Oriol Pareras, Gerard I. Gállego, Federico Costa, Cristina España-Bonet, Javier Hernando
Computation and Language
Semantic Differentiation in Speech Emotion Recognition: Insights from Descriptive and Expressive Speech Roles
Rongchen Guo, Vincent Francoeur, Isar Nejadgholi et al. (+2 more)
Speech Emotion Recognition (SER) is essential for improving human-computer interaction, yet its accuracy remains constrained by the complexity of emotional nuances in
speech. In this study, we distinguish between descriptive semantics, which represents the contextual content of
speech, and expressive semantics, which reflects the speaker's emotional state. After watching emotionally charged movie segments, we recorded audio clips of participants describing their experiences, along with the intended
emotion tags for each clip, participants' self-rated emotional responses, and their valence/arousal scores. Through experiments, we show that descriptive semantics align with intended
emotions, while expressive semantics correlate with evoked
emotions. Our findings inform SER applications in human-AI interaction and pave the way for more context-aware AI systems.
Read the full abstract ▾
All authors Rongchen Guo, Vincent Francoeur, Isar Nejadgholi, Sylvain Gagnon, Miodrag Bolic
Computation and Language
Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines
Matthew Lewis, Samuel Thio, Amy Roberts et al. (+8 more)
This paper presents the development and evaluation of a
Retrieval-Augmented Generation (RAG) system for querying the United Kingdom's National Institute for Health and Care Excellence (NICE) clinical guidelines using Large
Language Models (LLMs). The extensive length and volume of these guidelines can impede their utilisation within a time-constrained healthcare system, a challenge this project addresses through the creation of a system capable of providing users with precisely matched information in response to natural
language queries. The system's
retrieval architecture, composed of a hybrid
embedding mechanism, was evaluated against a corpus of 10,195 text chunks derived from three hundred guidelines. It demonstrates high performance, with a Mean Reciprocal Rank (MRR) of 0.814, a Recall of 81% at the first chunk and of 99.1% within the top ten retrieved chunks, when evaluated on 7901 queries. The most significant impact of the RAG system was observed during the generation phase. When evaluated on a manually curated
dataset of seventy question-answer pairs, RAG-enhanced models showed substantial gains in performance. Faithfulness, the measure of whether an answer is supported by the source text, was increased by 64.7 percentage points to 99.5% for the RAG-enhanced O4-Mini model and significantly outperformed the medical-focused Meditron3-8B LLM, which scored 43%. Clinical evaluation by seven Subject Matter Experts (SMEs) further validated these findings, with GPT-4.1 achieving 98.7% accuracy while reducing unsafe responses by 67% compared to O4-Mini (from 3.0 to 1.0 per evaluator). This study thus establishes RAG as an effective, reliable, and scalable approach for applying generative AI in healthcare, enabling cost-effective access to medical guidelines.
Read the full abstract ▾
All authors Matthew Lewis, Samuel Thio, Amy Roberts, Catherine Siju, Whoasif Mukit, Rebecca Kuruvilla, Zhangshu Joshua Jiang, Niko Möller-Grell, Aditya Borakati, Richard JB Dobson, Spiros Denaxas
Computation and Language
Leave No TRACE: Black-box Detection of Copyrighted Dataset Usage in Large Language Models via Watermarking
Jingqi Zhang, Ruibo Chen, Yingqing Yang et al. (+3 more)
Large
Language Models (LLMs) are increasingly fine-tuned on smaller, domain-specific
datasets to improve downstream performance. These
datasets often contain proprietary or copyrighted material, raising the need for reliable safeguards against unauthorized use. Existing membership inference attacks (MIAs) and
dataset-inference methods typically require access to internal signals such as logits, while current black-box approaches often rely on handcrafted
prompts or a clean reference
dataset for calibration, both of which limit practical applicability. Watermarking is a promising alternative, but prior techniques can degrade text quality or reduce task performance. We propose TRACE, a practical framework for fully black-box detection of copyrighted
dataset usage in LLM fine-tuning. TRACE rewrites
datasets with distortion-free watermarks guided by a private key, ensuring both text quality and downstream utility. At detection time, we exploit the radioactivity effect of fine-tuning on watermarked data and introduce an
entropy-gated procedure that selectively scores high-uncertainty
tokens, substantially amplifying detection power. Across diverse
datasets and model families, TRACE consistently achieves significant detections (p<0.05), often with extremely strong statistical evidence. Furthermore, it supports multi-
dataset attribution and remains robust even after continued pretraining on large non-watermarked corpora. These results establish TRACE as a practical route to reliable black-box verification of copyrighted
dataset usage. We will make our code available at: https://github.com/NusIoraPrivacy/TRACE.
Read the full abstract ▾
All authors Jingqi Zhang, Ruibo Chen, Yingqing Yang, Peihua Mai, Heng Huang, Yan Pang
Computation and Language
Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval
Yohan Lee, Yongwoo Song and Sangyeop Kim
We present the Conversational Data
Retrieval (CDR)
benchmark, the first comprehensive test set for evaluating systems that retrieve conversation data for product insights. With 1.6k queries across five analytical tasks and 9.1k conversations, our
benchmark provides a reliable standard for measuring conversational data
retrieval performance. Our evaluation of 16 popular
embedding models shows that even the best models reach only around NDCG@10 of 0.51, revealing a substantial gap between document and conversational data
retrieval capabilities. Our work identifies unique challenges in conversational data
retrieval (implicit state recognition, turn dynamics, contextual references) while providing practical query templates and detailed error analysis across different task categories. The
benchmark dataset and code are available at https://github.com/l-yohai/CDR-
Benchmark.
Read the full abstract ▾
Computation and Language
Self-Reflective Generation at Test Time
Jian Mu, Qixin Zhang, Zhiyong Wang et al. (+5 more)
Large
language models (LLMs) increasingly solve complex
reasoning tasks via long chain-of-thought, but their forward-only autoregressive generation process is fragile; early
token errors can cascade, which creates a clear need for self-reflection mechanisms. However, existing self-reflection either performs revisions over full drafts or learns self-correction via expensive training, both fundamentally reactive and inefficient. To address this, we propose Self-Reflective Generation at Test Time (SRGen), a lightweight test-time framework that reflects before generating at uncertain points. During
token generation, SRGen utilizes dynamic
entropy thresholding to identify high-uncertainty
tokens. For each identified
token, it trains a specific corrective vector, which fully exploits the already generated context for a self-reflective generation to correct the
token probability distribution. By retrospectively analyzing the partial output, this self-reflection enables more trustworthy decisions, thereby significantly reducing the probability of errors at highly uncertain points. Evaluated on challenging mathematical
reasoning benchmarks and a diverse set of LLMs, SRGen can significantly strengthen model
reasoning. Moreover, our findings position SRGen as a plug-and-play method that integrates reflection into the generation process for reliable LLM
reasoning, achieving consistent gains with bounded overhead and can be combined with other training-time (e.g., RLHF) and test-time (e.g., SLOT) techniques.
Read the full abstract ▾
All authors Jian Mu, Qixin Zhang, Zhiyong Wang, Menglin Yang, Shuang Qiu, Chengwei Qin, Zhongxiang Dai, Yao Shu
Computation and Language
Constraint Satisfaction Approaches to Wordle: Novel Heuristics and Cross-Lexicon Validation
Jahidul Arafat, Fariha Tasmin and Sanjaya Poudel
Wordle presents an algorithmically rich testbed for constraint satisfaction problem (CSP) solving. While existing solvers rely on information-theoretic
entropy maximization or frequency-based heuristics without formal constraint treatment, we present the first comprehensive CSP formulation of Wordle with novel constraint-aware solving strategies. We introduce CSP-Aware
Entropy, computing information gain after constraint propagation rather than on raw candidate sets, and a Probabilistic CSP framework integrating
Bayesian word-frequency priors with logical constraints. Through evaluation on 2,315 English words, CSP-Aware
Entropy achieves 3.54 average guesses with 99.9% success rate, a statistically significant 1.7% improvement over Forward Checking (t=-4.82, p<0.001, Cohen's d=0.07) with 46% faster runtime (12.9ms versus 23.7ms per guess). Under 10% noise, CSP-aware approaches maintain 5.3 percentage point advantages (29.0% versus 23.7%, p=0.041), while Probabilistic CSP achieves 100% success across all noise levels (0-20%) through constraint recovery mechanisms. Cross-lexicon validation on 500 Spanish words demonstrates 88% success with zero
language-specific tuning, validating that core CSP principles transfer across
languages despite an 11.2 percentage point gap from linguistic differences (p<0.001, Fisher's exact test). Our open-source implementation with 34 unit tests achieving 91% code coverage provides reproducible infrastructure for CSP research. The combination of formal CSP treatment, constraint-aware heuristics, probabilistic-logical integration, robustness analysis, and cross-lexicon validation establishes new performance
benchmarks demonstrating that principled constraint satisfaction techniques outperform classical information-theoretic and learning-based approaches for structured puzzle-solving domains.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents
Wonjoong Kim, Sangwu Park, Yeonjun In et al. (+3 more)
Although recent tool-augmented
benchmarks involve complex requests, evaluation remains limited to answer matching, neglecting critical trajectory aspects like efficiency,
hallucination, and adaptivity. The most straightforward method for evaluation is to compare an
agent's trajectory with the ground-truth, but annotating all valid ground-truth trajectories is prohibitively expensive. In this manner, we introduce TRACE, a reference-free framework for the multi-dimensional evaluation of tool-augmented LLMs. By incorporating an evidence bank which accumulates knowledge from preceding steps, TRACE assesses an
agent's
reasoning trajectory effectively. To validate our framework, we develop a new meta-evaluation
dataset with diverse and flawed trajectories, each labeled with multi-faceted performance scores. Our results confirm that TRACE accurately evaluates complex trajectories even with small open-source LLMs. Furthermore, we apply our method to evaluate the trajectories that
agents produce while solving tool-augmented tasks, presenting previously unreported observations and their corresponding insights.
Read the full abstract ▾
All authors Wonjoong Kim, Sangwu Park, Yeonjun In, Sein Kim, Dongha Lee, Chanyoung Park
Computation and Language
Evaluating Large Language Models for IUCN Red List Species Information
Shinya Uryu
Large
Language Models (LLMs) are rapidly being adopted in conservation to address the biodiversity crisis, yet their reliability for species evaluation is uncertain. This study systematically validates five leading models on 21,955 species across four core IUCN Red List assessment components: taxonomy, conservation status, distribution, and threats. A critical paradox was revealed: models excelled at taxonomic classification (94.9%) but consistently failed at conservation
reasoning (27.2% for status assessment). This knowledge-
reasoning gap, evident across all models, suggests inherent architectural constraints, not just data limitations. Furthermore, models exhibited systematic biases favoring charismatic vertebrates, potentially amplifying existing conservation inequities. These findings delineate clear boundaries for responsible LLM deployment: they are powerful tools for information
retrieval but require human oversight for judgment-based decisions. A hybrid approach is recommended, where LLMs augment expert capacity while human experts retain sole authority over risk assessment and policy.
Read the full abstract ▾