Computation and Language · Cross-listed from cs.AI
Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning
Zhaoyang Wang, Canwen Xu, Boyi Liu et al. (+5 more)
Recent advances in large
language model (LLM) have empowered autonomous
agents to perform multi-turn interactions with tools and environments. However, scaling such
agent training is limited by the lack of diverse and reliable environments. In this paper, we propose
Agent World Model (AWM), a fully synthetic environment generation pipeline. Using this pipeline, we scale to 1,000 environments covering everyday scenarios, in which
agents can interact with rich toolsets and obtain high-quality observations. Notably, these environments are code-driven and backed by databases, providing more reliable and consistent state transitions than environments simulated by LLMs. Moreover, they enable more efficient
agent interaction compared with collecting trajectories from realistic environments. To demonstrate the effectiveness of this resource, we perform large-scale reinforcement learning for multi-turn tool-use
agents. Thanks to the fully executable environments and accessible database states, we can also design reliable reward functions. Experiments on three
benchmarks show that training exclusively in synthetic environments, rather than
benchmark-specific ones, yields strong out-of-distribution generalization. The code is available at https://github.com/Snowflake-Labs/
agent-world-model.
Read the full abstract ▾
All authors Zhaoyang Wang, Canwen Xu, Boyi Liu, Yite Wang, Siwei Han, Zhewei Yao, Huaxiu Yao, Yuxiong He
Computation and Language
Anagent For Enhancing Scientific Table & Figure Analysis
Xuehang Guo, Zhiyong Lu, Tom Hope et al. (+1 more)
In scientific research, analysis requires accurately interpreting complex
multimodal knowledge, integrating evidence from different sources, and drawing inferences grounded in domain-specific knowledge. However, current artificial intelligence (AI) systems struggle to consistently demonstrate such capabilities. The complexity and variability of scientific tables and figures, combined with heterogeneous structures and long-context requirements, pose fundamental obstacles to scientific table & figure analysis. To quantify these challenges, we introduce AnaBench, a large-scale
benchmark featuring $63,178$ instances from nine scientific domains, systematically categorized along seven complexity dimensions. To tackle these challenges, we propose Anagent, a multi-
agent framework for enhanced scientific table & figure analysis through four specialized
agents: Planner decomposes tasks into actionable subtasks, Expert retrieves task-specific information through targeted tool execution, Solver synthesizes information to generate coherent analysis, and Critic performs iterative refinement through five-dimensional quality assessment. We further develop modular training strategies that leverage supervised finetuning and specialized reinforcement learning to optimize individual capabilities while maintaining effective collaboration. Comprehensive evaluation across 170 subdomains demonstrates that Anagent achieves substantial improvements, up to $\uparrow 13.43\%$ in training-free settings and $\uparrow 42.12\%$ with finetuning, while revealing that task-oriented
reasoning and context-aware problem-solving are essential for high-quality scientific table & figure analysis. Our project page: https://xhguo7.github.io/Anagent/.
Read the full abstract ▾
All authors Xuehang Guo, Zhiyong Lu, Tom Hope, Qingyun Wang
Computation and Language · Cross-listed from cs.CR
CAPID: Context-Aware PII Detection for Question-Answering Systems
Mariia Ponomarenko, Sepideh Abedini, Masoumeh Shafieinejad et al. (+3 more)
Detecting personally identifiable information (PII) in user queries is critical for ensuring
privacy in question-answering systems. Current approaches mainly redact all PII, disregarding the fact that some of them may be contextually relevant to the user's question, resulting in a degradation of response quality. Large
language models (LLMs) might be able to help determine which PII are relevant, but due to their closed source nature and lack of
privacy guarantees, they are unsuitable for sensitive data processing. To achieve
privacy-preserving PII detection, we propose CAPID, a practical approach that fine-tunes a locally owned small
language model (SLM) that filters sensitive information before it is passed to LLMs for QA. However, existing
datasets do not capture the context-dependent relevance of PII needed to train such a model effectively. To fill this gap, we propose a synthetic data generation pipeline that leverages LLMs to produce a diverse, domain-rich
dataset spanning multiple PII types and relevance levels. Using this
dataset, we fine-tune an SLM to detect PII spans, classify their types, and estimate contextual relevance. Our experiments show that relevance-aware PII detection with a fine-tuned SLM substantially outperforms existing baselines in span, relevance and type accuracy while preserving significantly higher downstream utility under anonymization.
Read the full abstract ▾
All authors Mariia Ponomarenko, Sepideh Abedini, Masoumeh Shafieinejad, D.B.Emerson, Shubhankar Mohapatra, Xi He
Computation and Language · Cross-listed from cs.IR
Overview of the TREC 2025 RAGTIME Track
Dawn Lawrie, Sean MacAvaney, James Mayfield et al. (+3 more)
The principal goal of the RAG TREC Instrument for
Multilingual Evaluation (RAGTIME) track at TREC is to study report generation from
multilingual source documents. The track has created a document collection containing Arabic, Chinese, English, and Russian news stories. RAGTIME includes three task types:
Multilingual Report Generation, English Report Generation, and
Multilingual Information
Retrieval (MLIR). A total of 125 runs were submitted by 13 participating teams (and as baselines by the track coordinators) for three tasks. This overview describes these three tasks and presents the available results.
Read the full abstract ▾
All authors Dawn Lawrie, Sean MacAvaney, James Mayfield, Luca Soldaini, Eugene Yang, Andrew Yates
Computation and Language
MEVER: Multi-Modal and Explainable Claim Verification with Graph-based Evidence Retrieval
Delvin Ce Zhang, Suhan Cui, Zhelin Chu et al. (+2 more)
Verifying the truthfulness of claims usually requires joint multi-modal
reasoning over both textual and visual evidence, such as analyzing both textual caption and chart image for claim verification. In addition, to make the
reasoning process transparent, a textual explanation is necessary to justify the verification result. However, most claim verification works mainly focus on the
reasoning over textual evidence only or ignore the explainability, resulting in inaccurate and unconvincing verification. To address this problem, we propose a novel model that jointly achieves evidence
retrieval, multi-modal claim verification, and explanation generation. For evidence
retrieval, we construct a two-layer multi-modal graph for claims and evidence, where we design image-to-text and text-to-image
reasoning for multi-modal
retrieval. For claim verification, we propose
token- and evidence-level fusion to integrate claim and evidence
embeddings for multi-modal verification. For explanation generation, we introduce multi-modal Fusion-in-Decoder for explainability. Finally, since almost all the
datasets are in general domain, we create a scientific
dataset, AIChartClaim, in AI domain to complement claim verification community. Experiments show the strength of our model.
Read the full abstract ▾
All authors Delvin Ce Zhang, Suhan Cui, Zhelin Chu, Xianren Zhang, Dongwon Lee
Computation and Language
Decoupled Reasoning with Implicit Fact Tokens (DRIFT): A Dual-Model Framework for Efficient Long-Context Inference
Wenxuan Xie, Yujia Wang, Xin Tan et al. (+3 more)
The integration of extensive, dynamic knowledge into Large
Language Models (LLMs) remains a significant challenge due to the inherent
entanglement of factual data and
reasoning patterns. Existing solutions, ranging from non-parametric
Retrieval-Augmented Generation (RAG) to parametric knowledge editing, are often constrained in practice by finite context windows, retriever noise, or the risk of catastrophic forgetting. In this paper, we propose DRIFT, a novel dual-model architecture designed to explicitly decouple knowledge extraction from the
reasoning process. Unlike static
prompt compression, DRIFT employs a lightweight knowledge model to dynamically compress document chunks into implicit fact
tokens conditioned on the query. These dense representations are projected into the
reasoning model's
embedding space, replacing raw, redundant text while maintaining inference accuracy. Extensive experiments show that DRIFT significantly improves performance on long-context tasks, outperforming strong baselines among comparably sized models. Our approach provides a scalable and efficient paradigm for extending the effective context window and
reasoning capabilities of LLMs. Our code is available at https://github.com/Lancelot-Xie/DRIFT.
Read the full abstract ▾
All authors Wenxuan Xie, Yujia Wang, Xin Tan, Chaochao Lu, Xia Hu, Xuhong Wang
Computation and Language
SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation
Homaira Huda Shomee, Rochana Chaturvedi, Yangxinyu Xie et al. (+1 more)
Large
language models (LLMs) are increasingly used to support question answering and decision-making in high-stakes, domain-specific settings such as natural hazard response and infrastructure planning, where effective answers must convey fine-grained, decision-critical details. However, existing evaluation frameworks for
retrieval-augmented generation (RAG) and open-ended question answering primarily rely on surface-level similarity, factual consistency, or semantic relevance, and often fail to assess whether responses provide the specific information required for domain-sensitive decisions. To address this gap, we propose a multi-dimensional, reference-free evaluation framework that assesses LLM outputs along four complementary dimensions: specificity, robustness to paraphrasing and semantic perturbations, answer relevance, and context utilization. We introduce a curated
dataset of 1,412 domain-specific question-answer pairs spanning 40 professional roles and seven natural hazard types to support systematic evaluation. We further conduct human evaluation to assess inter-annotator agreement and
alignment between model outputs and human judgments, which highlights the inherent subjectivity of open-ended, domain-specific evaluation. Our results show that no single metric sufficiently captures answer quality in isolation and demonstrate the need for structured, multi-metric evaluation frameworks when deploying LLMs in high-stakes applications.
Read the full abstract ▾
All authors Homaira Huda Shomee, Rochana Chaturvedi, Yangxinyu Xie, Tanwi Mallick
Computation and Language
ViSpeechFormer: A Phonemic Approach for Vietnamese Automatic Speech Recognition
Khoa Anh Nguyen, Long Minh Hoang, Nghia Hieu Nguyen et al. (+2 more)
Vietnamese has a phonetic orthography, where each grapheme corresponds to at most one phoneme and vice versa. Exploiting this high grapheme-phoneme transparency, we propose ViSpeechFormer (Vietnamese
Speech TransFormer), a phoneme-based approach for Vietnamese Automatic
Speech Recognition (ASR). To the best of our knowledge, this is the first Vietnamese ASR framework that explicitly models phonemic representations. Experiments on two publicly available Vietnamese ASR
datasets show that ViSpeechFormer achieves strong performance, generalizes better to out-of-vocabulary words, and is less affected by training bias. This phoneme-based paradigm is also promising for other
languages with phonetic orthographies. The code will be released upon acceptance of this paper.
Read the full abstract ▾
All authors Khoa Anh Nguyen, Long Minh Hoang, Nghia Hieu Nguyen, Luan Thanh Nguyen, Ngan Luu-Thuy Nguyen
Computation and Language
A Unified Assessment of the Poverty of the Stimulus Argument for Neural Language Models
Xiulin Yang, Arianna Bisazza, Nathan Schneider et al. (+1 more)
Several recent contributions have evaluated the Poverty of the Stimulus Hypothesis (PoSH) using Artificial Neural Networks (ANNs). The results suggest that ANN-based
language models can acquire certain structure-dependent generalizations from limited input without structural inductive biases of the type traditionally hypothesized by linguists. However, existing studies have largely focused on individual phenomena and adopted different evaluation protocols, leaving it unclear whether previous findings generalize across phenomena and learning conditions. We introduce \poshbench, a unified
benchmark covering four canonical PoS phenomena. Training
Transformer, LSTM, and n-gram models, we find that ANN-based models can achieve above-chance generalization from surprisingly limited input (10M words), but they show less efficient learning than children as input scale grows. Moreover, cognitively motivated inductive biases substantially improve broad syntactic competence but do not consistently translate to PoS-relevant generalization. Our findings provide systematic evidence challenging the claim that innate syntax is the only possible route to generalization, while suggesting that human-like learning efficiency requires inductive biases beyond those implemented here.
Read the full abstract ▾
All authors Xiulin Yang, Arianna Bisazza, Nathan Schneider, Ethan Gotlieb Wilcox
Computation and Language
ViMultiChoice: Toward a Method That Gives Explanation for Multiple-Choice Reading Comprehension in Vietnamese
Trung Tien Cao, Lam Minh Thai, Nghia Hieu Nguyen et al. (+2 more)
Multiple-choice Reading Comprehension (MCRC) models aim to select the correct answer from a set of candidate options for a given question. However, they typically lack the ability to explain the
reasoning behind their choices. In this paper, we introduce a novel Vietnamese
dataset designed to train and evaluate MCRC models with explanation generation capabilities. Furthermore, we propose ViMultiChoice, a new method specifically designed for modeling Vietnamese reading comprehension that jointly predicts the correct answer and generates a corresponding explanation. Experimental results demonstrate that ViMultiChoice outperforms existing MCRC baselines, achieving state-of-the-art (SotA) performance on both the ViMMRC 2.0
benchmark and the newly introduced
dataset. Additionally, we show that jointly training option decision and explanation generation leads to significant improvements in multiple-choice accuracy.
Read the full abstract ▾
All authors Trung Tien Cao, Lam Minh Thai, Nghia Hieu Nguyen, Duc-Vu Nguyen, Ngan Luu-Thuy Nguyen
Computation and Language
ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning
Shuaiyi Nie, Siyu Ding, Wenyuan Zhang et al. (+7 more)
Large
reasoning models trained with reinforcement learning and verifiable rewards (RLVR) achieve strong performance on complex
reasoning tasks, yet often overthink, generating redundant
reasoning without performance gains. Existing trajectory-level length penalties often fail to effectively shorten
reasoning length and degrade accuracy, as they uniformly treat all
reasoning steps and lack fine-grained signals to distinguish redundancy from necessity. Meanwhile, process-supervised methods are typically resource-intensive and suffer from inaccurate credit assignment. To address these issues, we propose ATTNPO, a low-overhead process-supervised RL framework that leverages the model's intrinsic
attention signals for step-level credit assignment. We first identify a set of special
attention heads that naturally focus on essential steps while suppressing redundant ones. By leveraging the
attention scores of these heads, We then employ two sub-strategies to mitigate overthinking by discouraging redundant steps while preserving accuracy by reducing penalties on essential steps. Experimental results show that ATTNPO substantially reduces
reasoning length while significantly improving performance across 9
benchmarks.
Read the full abstract ▾
All authors Shuaiyi Nie, Siyu Ding, Wenyuan Zhang, Linhao Yu, Tianmeng Yang, Yao Chen, Weichong Yin, Yu Sun, Hua Wu, Tingwen Liu
Computation and Language
LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations
William Lugoloobi, Thomas Foster, William Bankes et al. (+1 more)
Running LLMs with extended
reasoning on every problem is expensive, but determining which inputs actually require additional compute remains challenging. We investigate whether their own likelihood of success is recoverable from their internal representations before generation, and if this signal can guide more efficient inference. We train linear probes on pre-generation activations to predict policy-specific success on math and coding tasks, substantially outperforming surface features such as question length and TF-IDF. Using E2H-AMC, which provides both human and model performance on identical problems, we show that models encode a model-specific notion of difficulty that is distinct from human difficulty, and that this distinction increases with extended
reasoning. Leveraging these probes, we demonstrate that routing queries across a pool of models can exceed the best-performing model whilst reducing inference cost by up to 70% on MATH, showing that internal representations enable practical efficiency gains even when they diverge from human intuitions about difficulty. Our code is available at: https://github.com/KabakaWilliam/llms_know_difficulty
Read the full abstract ▾
All authors William Lugoloobi, Thomas Foster, William Bankes, Chris Russell
Computation and Language
AmharicIR+Instr: A Two-Dataset Resource for Neural Retrieval and Instruction Tuning
Tilahun Yeshambel, Moncef Garouani and Josiane Mothe
Neural
retrieval and GPT-style generative models rely on large, high-quality supervised data, which is still scarce for low-resource
languages such as Amharic. We release an Amharic data resource consisting of two
datasets that supports research on (i) neural
retrieval-ranking and (ii) instruction-following text generation. The
retrieval-ranking
dataset contains 1,091 manually verified query-positive-negative document triplets drawn from diverse Amharic sources and constructed to support contrastive training and benchmarking of neural retrievers (e.g., DPR, ColBERT-style late interaction and SPLADE-style sparse neural
retrieval). Triplets are created through a combination of expert-curated queries, web-derived queries, and LLM-assisted generation, with positive/negative documents selected from the web or synthesized by LLMs and then validated by native speakers. The instruction
prompt-response
dataset comprises 6,285 Amharic
prompt-response pairs spanning multiple domains and instruction types, generated with several LLMs and refined through manual review and correction for grammaticality, relevance, fluency, and factual plausibility. We release both
datasets with standardized splits and formats (CSV,JSON,JSONL) to enable reproducible work on Amharic
retrieval, ranking, and generative modelling. These
datasets also come with a methodology that can be generalized to other low-resource
languages.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
QP-OneModel: A Unified Generative LLM for Multi-Task Query Understanding in Xiaohongshu Search
Jianzhao Huang, Xiaorui Huang, Fei Zhao et al. (+9 more)
Query Processing (QP) bridges user intent and content supply in large-scale Social Network Service (SNS) search engines. Traditional QP systems rely on pipelines of isolated discriminative models (e.g., BERT), suffering from limited semantic understanding and high maintenance overhead. While Large
Language Models (LLMs) offer a potential solution, existing approaches often optimize sub-tasks in isolation, neglecting intrinsic semantic synergy and necessitating independent iterations. Moreover, standard generative methods often lack grounding in SNS scenarios, failing to bridge the gap between open-domain corpora and informal SNS linguistic patterns, while struggling to adhere to rigorous business definitions. We present QP-OneModel, a Unified Generative LLM for Multi-Task Query Understanding in the SNS domain. We reformulate heterogeneous sub-tasks into a unified sequence generation paradigm, adopting a progressive three-stage
alignment strategy culminating in multi-reward Reinforcement Learning. Furthermore, QP-OneModel generates intent descriptions as a novel high-fidelity semantic signal, effectively augmenting downstream tasks such as query rewriting and ranking. Offline evaluations show QP-OneModel achieves a 7.35% overall gain over discriminative baselines, with significant F1 boosts in NER (+9.01%) and Term Weighting (+9.31%). It also exhibits superior generalization, surpassing a 32B model by 7.60% accuracy on unseen tasks. Fully deployed at Xiaohongshu, online A/B tests confirm its industrial value, optimizing
retrieval relevance (DCG) by 0.21% and lifting user retention by 0.044%.
Read the full abstract ▾
All authors Jianzhao Huang, Xiaorui Huang, Fei Zhao, Yunpeng Liu, Hui Zhang, Fangcheng Shi, Congfeng Li, Zechen Sun, Yi Wu, Yao Hu, Yunhan Bai, Shaosheng Cao
Computation and Language
The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies
Chenxu Wang, Chaozhuo Li, Songyang Liu et al. (+10 more)
The emergence of multi-
agent systems built from large
language models (LLMs) offers a promising paradigm for scalable collective intelligence and self-
evolution. Ideally, such systems would achieve continuous self-improvement in a fully closed loop while maintaining robust safety
alignment–a combination we term the self-
evolution trilemma. However, we demonstrate both theoretically and empirically that an
agent society satisfying continuous self-
evolution, complete isolation, and safety invariance is impossible. Drawing on an information-theoretic framework, we formalize safety as the divergence degree from anthropic value distributions. We theoretically demonstrate that isolated self-
evolution induces statistical blind spots, leading to the irreversible degradation of the system's safety
alignment. Empirical and qualitative results from an open-ended
agent community (Moltbook) and two closed self-evolving systems reveal phenomena that align with our theoretical prediction of inevitable safety erosion. We further propose several solution directions to alleviate the identified safety concern. Our work establishes a fundamental limit on the self-evolving AI societies and shifts the discourse from symptom-driven safety patches to a principled understanding of intrinsic dynamical risks, highlighting the need for external oversight or novel safety-preserving mechanisms.
Read the full abstract ▾
All authors Chenxu Wang, Chaozhuo Li, Songyang Liu, Zejian Chen, Jinyu Hou, Ji Qi, Rui Li, Litian Zhang, Qiwei Ye, Zheng Liu, Xu Chen, Xi Zhang, Philip S. Yu
Computation and Language
Steer2Edit: From Activation Steering to Component-Level Editing
Chung-En Sun, Ge Yan, Zimo Wang et al. (+1 more)
Steering methods influence Large
Language Model behavior by identifying semantic directions in hidden representations, but are typically realized through inference-time activation interventions that apply a fixed, global modification to the model's internal states. While effective, such interventions often induce unfavorable attribute-utility trade-offs under strong control, as they ignore the fact that many behaviors are governed by a small and heterogeneous subset of model components. We propose Steer2Edit, a theoretically grounded, training-free framework that transforms steering vectors from inference-time control signals into diagnostic signals for component-level rank-1 weight editing. Instead of uniformly injecting a steering direction during generation, Steer2Edit selectively redistributes behavioral influence across individual
attention heads and MLP
neurons, yielding interpretable edits that preserve the standard forward pass and remain compatible with optimized parallel inference. Across safety
alignment,
hallucination mitigation, and
reasoning efficiency, Steer2Edit consistently achieves more favorable attribute-utility trade-offs: at matched downstream performance, it improves safety by up to 17.2%, increases truthfulness by 9.8%, and reduces
reasoning length by 12.2% on average. Overall, Steer2Edit provides a principled bridge between representation steering and weight editing by translating steering signals into interpretable, training-free parameter updates. Our code is available at https://github.com/Trustworthy-ML-Lab/Steer2Edit
Read the full abstract ▾
All authors Chung-En Sun, Ge Yan, Zimo Wang, Tsui-Wei Weng
Computation and Language · Cross-listed from cs.CV
Code2World: A GUI World Model via Renderable Code Generation
Yuhao Zheng, Li'an Zhong, Yi Wang et al. (+6 more)
Autonomous GUI
agents interact with environments by perceiving interfaces and executing actions. As a virtual sandbox, the GUI World model empowers
agents with human-like foresight by enabling action-conditioned prediction. However, existing text- and pixel-based approaches struggle to simultaneously achieve high visual fidelity and fine-grained structural controllability. To this end, we propose Code2World, a
vision-
language coder that simulates the next visual state via renderable code generation. Specifically, to address the data scarcity problem, we construct AndroidCode by translating GUI trajectories into high-fidelity HTML and refining synthesized code through a visual-feedback revision mechanism, yielding a corpus of over 80K high-quality screen-action pairs. To adapt existing VLMs into code prediction, we first perform SFT as a cold start for format layout following, then further apply Render-Aware Reinforcement Learning which uses rendered outcome as the reward signal by enforcing visual semantic fidelity and action consistency. Extensive experiments demonstrate that Code2World-8B achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image. Notably, Code2World significantly enhances downstream
navigation success rates in a flexible manner, boosting Gemini-2.5-Flash by +9.5% on AndroidWorld
navigation. The code is available at https://github.com/AMAP-ML/Code2World.
Read the full abstract ▾
All authors Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin
Computation and Language
How Do People Quantify Naturally: Evidence from Mandarin Picture Description
Yayun Zhang, Guanyi Chen, Fahime Same et al. (+2 more)
Quantification is a fundamental component of everyday
language use, yet little is known about how speakers decide whether and how to quantify in naturalistic production. We investigate quantification in Mandarin Chinese using a picture-based elicited description task in which speakers freely described scenes containing multiple objects, without explicit instructions to count or quantify. Across both spoken and written modalities, we examine three aspects of quantification: whether speakers choose to quantify at all, how precise their quantification is, and which quantificational strategies they adopt. Results show that object numerosity, animacy, and production modality systematically shape quantificational behaviour. In particular, increasing numerosity reduces both the likelihood and the precision of quantification, while animate referents and modality selectively modulate strategy choice. This study demonstrates how quantification can be examined under unconstrained production conditions and provides a naturalistic
dataset for further analyses of quantity expression in
language production.
Read the full abstract ▾
All authors Yayun Zhang, Guanyi Chen, Fahime Same, Saad Mahamood, Tingting He
Computation and Language
LLM Reasoning Predicts When Models Are Right: Evidence from Coding Classroom Discourse
Bakhtawar Ahtisham, Kirk Vanacore, Zhuqian Zhou et al. (+2 more)
Large
Language Models (LLMs) are increasingly deployed to automatically label and analyze educational dialogue at scale, yet current pipelines lack reliable ways to detect when models are wrong. We investigate whether
reasoning generated by LLMs can be used to predict the correctness of a model's own predictions. We analyze 30,300 teacher utterances from classroom dialogue, each labeled by multiple state-of-the-art LLMs with an instructional move construct and an accompanying
reasoning. Using human-verified ground-truth labels, we frame the task as predicting whether a model's assigned label for a given utterance is correct. We encode LLM
reasoning using Term Frequency-Inverse Document Frequency (TF-IDF) and evaluate five supervised classifiers. A Random Forest classifier achieves an F1 score of 0.83 (Recall = 0.854), successfully identifying most incorrect predictions and outperforming baselines. Training specialist detectors for specific instructional move constructs further improves performance on difficult constructs, indicating that error detection benefits from construct-specific linguistic cues. Using the Linguistic Inquiry and Word Count (LIWC) framework, we examine four linguistic markers of correctness: Causation, Differentiation, Tentativeness, and Insight. Correct predictions exhibit grounded causal
language (e.g., because, therefore), while incorrect
reasoning is substantially more likely to rely on epistemic hedging (e.g., might, could) and performative metacognition (e.g., think, realize). Syntactic complexity does not distinguish correct from incorrect
reasoning, and longer
reasoning is not more reliable. These findings demonstrate that
reasoning-based error detection offers a practical and scalable approach to quality control in automated educational dialogue analysis.
Read the full abstract ▾
All authors Bakhtawar Ahtisham, Kirk Vanacore, Zhuqian Zhou, Jinsook Lee, Rene F. Kizilcec
Computation and Language
From FusHa to Folk: Exploring Cross-Lingual Transfer in Arabic Language Models
Abdulmuizz Khalak, Abderrahmane Issam and Gerasimos Spanakis
Arabic
Language Models (LMs) are pretrained predominately on Modern Standard Arabic (MSA) and are expected to transfer to its dialects. While MSA as the standard written variety is commonly used in formal settings, people speak and write online in various dialects that are spread across the Arab region. This poses limitations for Arabic LMs, since its dialects vary in their similarity to MSA. In this work we study cross-lingual transfer of Arabic models using probing on 3 Natural
Language Processing (NLP) Tasks, and representational similarity. Our results indicate that transfer is possible but disproportionate across dialects, which we find to be partially explained by their geographic proximity. Furthermore, we find evidence for negative interference in models trained to support all Arabic dialects. This questions their degree of similarity, and raises concerns for cross-lingual transfer in Arabic models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Covo-Audio Technical Report
Wenfu Wang, Chenxing Li, Liqiang Zhang et al. (+23 more)
In this work, we present Covo-Audio, a 7B-parameter end-to-end LALM that directly processes continuous audio inputs and generates audio outputs within a single unified architecture. Through large-scale curated pretraining and targeted post-training, Covo-Audio achieves state-of-the-art or competitive performance among models of comparable scale across a broad spectrum of tasks, including
speech-text modeling, spoken dialogue,
speech understanding, audio understanding, and full-duplex voice interaction. Extensive evaluations demonstrate that the pretrained foundation model exhibits strong
speech-text comprehension and semantic
reasoning capabilities on multiple
benchmarks, outperforming representative open-source models of comparable scale. Furthermore, Covo-Audio-Chat, the dialogue-oriented variant, demonstrates strong spoken conversational abilities, including understanding, contextual
reasoning, instruction following, and generating contextually appropriate and empathetic responses, validating its applicability to real-world conversational assistant scenarios. Covo-Audio-Chat-FD, the evolved full-duplex model, achieves substantially superior performance on both spoken dialogue capabilities and full-duplex interaction behaviors, demonstrating its competence in practical robustness. To mitigate the high cost of deploying end-to-end LALMs for natural conversational systems, we propose an intelligence-speaker decoupling strategy that separates dialogue intelligence from voice rendering, enabling flexible voice customization with minimal text-to-
speech (TTS) data while preserving dialogue performance. Overall, our results highlight the strong potential of 7B-scale models to integrate sophisticated audio intelligence with high-level semantic
reasoning, and suggest a scalable path toward more capable and versatile LALMs.
Read the full abstract ▾
All authors Wenfu Wang, Chenxing Li, Liqiang Zhang, Yiyang Zhao, Yuxiang Zou, Hanzhao Li, Mingyu Cui, Hao Zhang, Kun Wei, Le Xu, Zikang Huang, Jiajun Xu, Jiliang Hu, Xiang He, Zeyu Xie, Jiawen Kang, Youjun Chen, Meng Yu, Dong Yu, Rilin Chen, Linlin Di, Shulin Feng, Na Hu, Yang Liu, Bang Wang, Shan Yang
Computation and Language
Text summarization via global structure awareness
Jiaquan Zhang, Chaoning Zhang, Shuxu Chen et al. (+9 more)
Text summarization is a fundamental task in natural
language processing (NLP), and the information explosion has made long-document processing increasingly demanding, making summarization essential. Existing research mainly focuses on model improvements and sentence-level
pruning, but often overlooks global structure, leading to disrupted coherence and weakened downstream performance. Some studies employ large
language models (LLMs), which achieve higher accuracy but incur substantial resource and time costs. To address these issues, we introduce GloSA-sum, the first summarization approach that achieves global structure awareness via topological data analysis (TDA). GloSA-sum summarizes text efficiently while preserving semantic cores and logical dependencies. Specifically, we construct a semantic-weighted graph from sentence
embeddings, where persistent homology identifies core semantics and logical structures, preserved in a “protection pool” as the backbone for summarization. We design a
topology-guided iterative strategy, where lightweight proxy metrics approximate sentence importance to avoid repeated high-cost computations, thus preserving structural integrity while improving efficiency. To further enhance long-text processing, we propose a hierarchical strategy that integrates segment-level and global summarization. Experiments on multiple
datasets demonstrate that GloSA-sum reduces redundancy while preserving semantic and logical integrity, striking a balance between accuracy and efficiency, and further benefits LLM downstream tasks by shortening contexts while retaining essential
reasoning chains.
Read the full abstract ▾
All authors Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Yibei Liu, Chenghao Li, Qigan Sun, Shuai Yuan, Fachrina Dewi Puspitasari, Dongshen Han, Guoqing Wang, Sung-Ho Bae, Yang Yang
Computation and Language
AnalyticsGPT: An LLM Workflow for Scientometric Question Answering
Khang Ly, Georgios Cheirmpos, Adrian Raudaschl et al. (+2 more)
This paper introduces AnalyticsGPT, an intuitive and efficient large
language model (LLM)-powered workflow for scientometric question answering. This underrepresented downstream task addresses the subcategory of meta-scientific questions concerning the "science of science." When compared to traditional scientific question answering based on papers, the task poses unique challenges in the planning phase. Namely, the need for named-entity recognition of academic entities within questions and multi-faceted data
retrieval involving scientometric indices, e.g. impact factors. Beyond their exceptional capacity for treating traditional natural
language processing tasks, LLMs have shown great potential in more complex applications, such as task decomposition and planning and
reasoning. In this paper, we explore the application of LLMs to scientometric question answering, and describe an end-to-end system implementing a sequential workflow with
retrieval-augmented generation and agentic concepts. We also address the secondary task of effectively synthesizing the data into presentable and well-structured high-level analyses. As a database for
retrieval-augmented generation, we leverage a proprietary research performance assessment platform. For evaluation, we consult experienced subject matter experts and leverage LLMs-as-judges. In doing so, we provide valuable insights on the efficacy of LLMs towards a niche downstream task. Our (skeleton) code and
prompts are available at: https://github.com/lyvykhang/llm-
agents-scientometric-qa/tree/acl.
Read the full abstract ▾
All authors Khang Ly, Georgios Cheirmpos, Adrian Raudaschl, Christopher James, Seyed Amin Tabatabaei
Computation and Language
Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs
Daniel Kaiser, Arnoldo Frigessi, Ali Ramezani-Kebrya et al. (+1 more)
As
reasoning LLMs increasingly trade
tokens for accuracy through deliberation, search, and self-correction, a single accuracy score can no longer tell whether those
tokens buy useful
reasoning, recovery from hard instances, or unnecessary verbosity. We introduce a trace-optional evaluation protocol that exactly decomposes
token efficiency using three observables available even for closed models: completion rate, conditional correctness given completion, and generated length. When instance-level workload metadata is available, we further normalize generated length by declared task-implied work and separate mean verbalization overhead from workload-dependent scaling. When such metadata is absent, we define an auditable solver-derived workload scale and evaluate its stability under leave-self-out, leave-top-k, and held-out-reference-pool perturbations. We evaluate 14 shared open-weight models on CogniLoad, GSM8K, ProofWriter, and ZebraLogic. We further evaluate 11 additional models on CogniLoad, enabling a fine-grained analysis of
reasoning-task difficulty factors: task length, intrinsic difficulty, and distractor density. Efficiency and overhead rankings remain stable across all
benchmark pairs, more robustly than accuracy rankings, while the decomposition separates logic-limited, context-limited (truncation-driven), and verbosity-limited failure modes that look identical under accuracy-per-
token. We release an evaluation artifact and reporting template, which elaborates on why an LLM is inefficient at
reasoning.
Read the full abstract ▾
All authors Daniel Kaiser, Arnoldo Frigessi, Ali Ramezani-Kebrya, Benjamin Ricaud
Computation and Language · Cross-listed from cs.AI
Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices
Manon Reusens, Sofie Goethals, Toon Calders et al. (+1 more)
As Large
Language Models (LLMs) are increasingly deployed in applications such as travel assistance and purchasing support, they are often required to make subjective choices on behalf of users in settings where no objectively correct answer exists. We study LLM decision-making in a travel-assistant context by presenting models with choice dilemmas and analyzing their responses using multinomial logit models to derive implied willingness to pay (WTP) estimates. These WTP values are subsequently compared to human
benchmark values from the economics literature. In addition to a baseline setting, we examine how model behavior changes under more realistic conditions, including the provision of information about users' past choices and persona-based prompting. Our results show that while meaningful WTP values can be derived for larger LLMs, they also display systematic deviations at the attribute level. Additionally, they tend to overestimate human WTP overall, particularly when expensive options or business-oriented personas are introduced. Conditioning models on prior preferences for cheaper options yields valuations that are closer to human
benchmarks. Overall, our findings highlight both the potential and the limitations of using LLMs for subjective decision support and underscore the importance of careful model selection,
prompt design, and user representation when deploying such systems in practice.
Read the full abstract ▾
All authors Manon Reusens, Sofie Goethals, Toon Calders, David Martens
Computation and Language
Where Are We At with Automatic Speech Recognition for the Bambara Language?
Seydou Diallo, Yacouba Diarra, Mamadou K. Keita et al. (+3 more)
This paper introduces the first standardized
benchmark for evaluating Automatic
Speech Recognition (ASR) in the Bambara
language, utilizing one hour of professionally recorded Malian constitutional text. Designed as a controlled reference set under near-optimal acoustic and linguistic conditions, the
benchmark was used to evaluate 37 models, ranging from Bambara-trained systems to large-scale commercial models. Our findings reveal that current ASR performance remains significantly below deployment standards in a narrow formal domain; the top-performing system in terms of Word Error Rate (WER) achieved 46.76% and the best Character Error Rate (CER) of 13.00% was set by another model, while several prominent
multilingual models exceeded 100% WER. These results suggest that
multilingual pre-training and model scaling alone are insufficient for underrepresented
languages. Furthermore, because this
dataset represents a best-case scenario of the most simplified and formal form of spoken Bambara, these figures are yet to be tested against practical, real-world settings. We provide the
benchmark and an accompanying public leaderboard to facilitate transparent evaluation and future research in Bambara
speech technology.
Read the full abstract ▾
All authors Seydou Diallo, Yacouba Diarra, Mamadou K. Keita, Panga Azazia Kamaté, Adam Bouno Kampo, Aboubacar Ouattara
Computation and Language · Cross-listed from cs.LG
Circuit Fingerprints: How Answer Tokens Encode Their Geometrical Path
Andres Saurez, Neha Sengar and Dongsoo Har
Circuit discovery and activation steering in
transformers have developed as separate research threads, yet both operate on the same representational space. Are they two views of the same underlying structure? We show they follow a single
geometric principle: answer
tokens, processed in isolation, encode the directions that would produce them. This
Circuit Fingerprint hypothesis enables
circuit discovery without gradients or causal intervention – recovering comparable structure to gradient-based methods through
geometric alignment alone. We validate this on standard
benchmarks (IOI, SVA, MCQA) across four model families, achieving
circuit discovery performance comparable to gradient-based methods. The same directions that identify
circuit components also enable controlled steering – achieving 69.8%
emotion classification accuracy versus 53.1% for instruction prompting while preserving factual accuracy. Beyond method development, this read-write duality reveals that
transformer circuits are fundamentally
geometric structures: interpretability and controllability are two facets of the same object.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraints
Andres Saurez, Yousung Lee and Dongsoo Har
Linear probes and sparse autoencoders consistently recover meaningful structure from
transformer representations – yet why should such simple methods succeed in deep, nonlinear systems? We show this is not merely an empirical regularity but a consequence of architectural necessity:
transformers communicate information through linear interfaces (
attention OV
circuits, unembedding
matrices), and any semantic feature decoded through such an interface must occupy a context-invariant linear subspace. We formalize this as the Invariant Subspace Necessity
theorem and derive the Self-Reference Property:
tokens directly provide the
geometric direction for their associated features, enabling zero-shot identification of semantic structure without labeled data or learned probes. Empirical validation in eight classification tasks and four model families confirms the
alignment between class
tokens and semantically related instances. Our framework provides a principled architectural explanation for why linear interpretability methods work, unifying linear probes and sparse autoencoders.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective
Kun Chen, Peng Shi, Fanfan Liu et al. (+4 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a critical method for enhancing the
reasoning capabilities of Large
Language Models (LLMs). However, continuous training often leads to policy
entropy collapse, characterized by a rapid decay in
entropy that results in premature overconfidence, reduced output diversity, and vanishing gradient norms that inhibit learning. Gradient-Preserving Clipping is a primary factor influencing these dynamics, but existing mitigation strategies are largely static and lack a framework connecting clipping mechanisms to precise
entropy control. This paper proposes reshaping
entropy control in RL from the perspective of Gradient-Preserving Clipping. We first theoretically and empirically verify the contributions of specific importance sampling ratio regions to
entropy growth and reduction. Leveraging these findings, we introduce a novel regulation mechanism using dynamic clipping thresholds to precisely manage
entropy. Furthermore, we design and evaluate dynamic
entropy control strategies, including increase-then-decrease, decrease-increase-decrease, and oscillatory decay. Experimental results demonstrate that these strategies effectively mitigate
entropy collapse and achieve superior performance across multiple
benchmarks.
Read the full abstract ▾
All authors Kun Chen, Peng Shi, Fanfan Liu, Haibo Qiu, Zhixiong Zeng, Siqi Yang, Wenji Mao
Computation and Language
Improving Interpretability of Lexical Semantic Change with Neurobiological Features
Kohei Oda, Hiroya Takamura, Kiyoaki Shirai et al. (+1 more)
Lexical Semantic Change (LSC) is the phenomenon in which the meaning of a word change over time. Most studies on LSC focus on improving the performance of estimating the degree of LSC, however, it is often difficult to interpret how the meaning of a word change. Enhancing the interpretability of LSC is a significant challenge as it could lead to novel insights in this field. To tackle this challenge, we propose a method to map the semantic space of contextualized
embeddings of words obtained by a pre-trained
language model to a neurobiological feature space. In the neurobiological feature space, each dimension corresponds to a primitive feature of words, and its value represents the intensity of that feature. This enables humans to interpret LSC systematically. When employed for the estimation of the degree of LSC, our method demonstrates superior performance in comparison to the majority of the previous methods. In addition, given the high interpretability of the proposed method, several analyses on LSC are carried out. The results demonstrate that our method not only discovers interesting types of LSC that have been overlooked in previous studies but also effectively searches for words with specific types of LSC.
Read the full abstract ▾
All authors Kohei Oda, Hiroya Takamura, Kiyoaki Shirai, Natthawut Kertkeidkachorn