Computation and Language · Cross-listed from cs.AI
Finding RELIEF: Shaping Reasoning Behavior without Reasoning Supervision via Belief Engineering
Chak Tou Leong, Dingwei Chen, Heming Xia et al. (+4 more)
Large
reasoning models (LRMs) have achieved remarkable success in complex problem-solving, yet they often suffer from computational redundancy or
reasoning unfaithfulness. Current methods for shaping LRM behavior typically rely on reinforcement learning or fine-tuning with gold-standard
reasoning traces, a paradigm that is both computationally expensive and difficult to scale. In this paper, we reveal that LRMs possess latent
reasoning beliefs that internally track their own
reasoning traits, which can be captured through simple logit probing. Building upon this insight, we propose
Reasoning Belief Engineering (RELIEF), a simple yet effective framework that shapes LRM behavior by aligning the model's self-concept with a target belief blueprint. Crucially, RELIEF completely bypasses the need for
reasoning-trace supervision. It internalizes desired traits by fine-tuning on synthesized, self-reflective question-answering pairs that affirm the target belief. Extensive experiments on efficiency and faithfulness tasks demonstrate that RELIEF matches or outperforms behavior-supervised and preference-based baselines while requiring lower training costs. Further analysis validates that shifting a model's
reasoning belief effectively shapes its actual behavior.
Read the full abstract ▾
All authors Chak Tou Leong, Dingwei Chen, Heming Xia, Qingyu Yin, Sunbowen Lee, Jian Wang, Wenjie Li
Computation and Language
Pro-AI Bias in Large Language Models
Benaya Trabelsi, Jonathan Shaki and Sarit Kraus
Large
language models (LLMs) are increasingly employed for decision-support across multiple domains. We ask whether these models favor artificial intelligence (AI) itself. We examine this question through three experiments. First, in open-ended advice across investment, study, career, and startup domains, AI is prominent in most recommendation lists and is usually ranked first when it appears. Second, in salary estimations, models overestimate AI-related jobs more than they do for closely matched non-AI jobs. In a controlled follow-up, changing only the job title from the matched non-AI title to the AI-related title raises predicted salary by 10%, while all other supplied job details remain fixed. Finally, we show that the number of degrees and other credentials awarded by AI programs were 28% more overestimated than corresponding awards for matched computing programs. Overall, we show that AI is not only prominent in everyday LLM advice, but also receives systematically inflated quantitative estimates compared to non-AI cases, even against its closest disciplinary neighbors, raising concerns for high-stakes decision support.
Read the full abstract ▾
Computation and Language
Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluation
Arjun Chandra, Kevin Miller, Venkatesh Ravichandran et al. (+2 more)
Large
Language Model (LLM) judges exhibit strong
reasoning capabilities but are limited to textual content. This leaves current automatic
Speech-to-
Speech (S2S) evaluation methods reliant on opaque and expensive Audio
Language Models (ALMs). In this work, we propose TRACE (Textual
Reasoning over Audio Cues for Evaluation), a novel framework that enables LLM judges to reason over audio cues to achieve cost-efficient and human-aligned S2S evaluation. To demonstrate the strength of the framework, we first introduce a Human Chain-of-Thought (HCoT) annotation protocol to improve the diagnostic capability of existing judge
benchmarks by separating evaluation into explicit dimensions: content (C), voice quality (VQ), and paralinguistics (P). Using this data, TRACE constructs a textual blueprint of inexpensive audio signals and
prompts an LLM to render dimension-wise judgments, fusing them into an overall rating via a deterministic policy. TRACE achieves higher agreement with human raters than ALMs and transcript-only LLM judges while being significantly more cost-effective. We will release the HCoT annotations and the TRACE framework to enable scalable and human-aligned S2S evaluation.
Read the full abstract ▾
All authors Arjun Chandra, Kevin Miller, Venkatesh Ravichandran, Constantinos Papayiannis, Venkatesh Saligrama
Computation and Language
Towards robust long-context understanding of large language model via active recap learning
Chenyu Hui
In this paper, we propose active recap learning (ARL), a framework for enhancing large
language model (LLM) in understanding long contexts. ARL enables models to revisit and summarize earlier content through targeted sequence construction during continued pretraining and retrospective summarization at inference. First, we identify key
tokens in prepared long context based on loss gaps between long and short forward contexts and find the most relevant preceding paragraphs, then summarize them using an LLM. Second, ARL equips models with the ability to autonomously generate and utilize these retrospective summaries during inference, thereby establishing a recursive
memory mechanism across paragraphs. Experimental results show substantial gains, with ARL achieving an improvement of up to 129.7% on RULER and 11–13% on LongBench. Overall, ARL offers a simple yet effective continued pretraining-based approach to strengthen long-context understanding, advancing scalable
memory augmentation in LLM. Code is available at \href{https://github.com/nanfangxiansheng/TOWARDS-ROBUST-LONG-CONTEXT-UNDERSTANDING-OF-LARGE-
LANGUAGE-MODEL-VIA-ACTIVE-RECAP-LEARNING}{CODE}
Read the full abstract ▾
Computation and Language
On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation
Weichuan Wang, Mingyang Liu, Linqi Song et al. (+1 more)
In recent years, the non-deterministic properties of
language models have garnered considerable
attention and have shown a significant influence on real-world applications. However, such properties remain under-explored in machine translation (MT), a complex, non-deterministic NLP task. In this study, we systematically evaluate modern MT systems and identify temperature-constrained Non-Deterministic MT (ND-MT) as a distinct phenomenon. Additionally, we demonstrate that ND-MT exhibits significant potential in addressing the multimodality issue that has long challenged MT research and provides higher-quality candidates than Deterministic MT (D-MT) under temperature constraints. However, ND-MT introduces new challenges in evaluating system performance. Specifically, the evaluation framework designed for D-MT fails to yield consistent evaluation results when applied to ND-MT. We further investigate this emerging challenge by evaluating state-of-the-art ND-MT systems using both lexical-based and semantic-based metrics at varying sampling sizes. The results reveal a Buckets Effect across these systems: the ranking of ND-MT systems is dominated by the worst-quality candidate translation, as shown by automatic evaluation metrics. To mitigate this issue, we propose ExpectoSample, a strategy that first identifies reliable metrics and then enables robust ND-MT system selection for real-world.
Read the full abstract ▾
All authors Weichuan Wang, Mingyang Liu, Linqi Song, Chen Ma
Computation and Language
OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents
Yulin Hu, Zimo Long, Jiahe Guo et al. (+5 more)
Memory-augmented conversational
agents enable personalized interactions using long-term user
memory and have gained substantial traction. However, existing
benchmarks primarily focus on whether
agents can recall and apply user information, while overlooking whether such personalization is used appropriately. In fact,
agents may overuse personal information, producing responses that feel forced, intrusive, or socially inappropriate to users. We refer to this issue as over-personalization. In this work, we formalize over-personalization into three types: Irrelevance, Repetition, and Sycophancy, and introduce OP-Bench a
benchmark of 1,700 verified instances constructed from long-horizon dialogue histories. Using OP-Bench, we evaluate multiple large
language models and
memory-augmentation methods, and find that over-personalization is widespread when
memory is introduced. Further analysis reveals that
agents tend to retrieve and over-attend to user
memories even when unnecessary. To address this issue, we propose Self-ReCheck, a lightweight, model-agnostic
memory filtering mechanism that mitigates over-personalization while preserving personalization performance. Our work takes an initial step toward more controllable and appropriate personalization in
memory-augmented dialogue systems.
Read the full abstract ▾
All authors Yulin Hu, Zimo Long, Jiahe Guo, Xingyu Sui, Xing Fu, Weixiang Zhao, Yanyan Zhao, Bing Qin
Computation and Language
Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff
Zehan Li, Yuxuan Wang, Ali El Lahib et al. (+2 more)
Evaluating LLM
forecasting capabilities is constrained by a fundamental tension: prospective evaluation offers methodological rigor but prohibitive latency, while retrospective
forecasting (RF) – evaluating on already-resolved events – faces rapidly shrinking clean evaluation data as SOTA models possess increasingly recent knowledge cutoffs. Simulated Ignorance (SI), prompting models to suppress pre-cutoff knowledge, has emerged as a potential solution. We provide the first systematic test of whether SI can approximate True Ignorance (TI). Across 477 competition-level questions and 9 models, we find that SI fails systematically: (1) cutoff instructions leave a 52% performance gap between SI and TI; (2) chain-of-thought
reasoning fails to suppress prior knowledge, even when
reasoning traces contain no explicit post-cutoff references; (3)
reasoning-optimized models exhibit worse SI fidelity despite superior
reasoning trace quality. These findings demonstrate that
prompts cannot reliably "rewind" model knowledge. We conclude that RF on pre-cutoff events is methodologically flawed; we recommend against using SI-based retrospective setups to
benchmark forecasting capabilities.
Read the full abstract ▾
All authors Zehan Li, Yuxuan Wang, Ali El Lahib, Ying-Jieh Xia, Xinyu Pi
Computation and Language
GerAV: Towards New Heights in German Authorship Verification using Fine-Tuned LLMs on a New Benchmark
Lotta Kiefer, Christoph Leiter, Sotaro Takeshita et al. (+2 more)
Authorship verification (AV) is the task of determining whether two texts were written by the same author and has been studied extensively, predominantly for English data. In contrast, large-scale
benchmarks and systematic evaluations for other
languages remain scarce. We address this gap by introducing GerAV, a comprehensive
benchmark for German AV comprising over 400k labeled text pairs. GerAV is built from Twitter and Reddit data, with the Reddit part further divided into in-domain and cross-domain message-based subsets, as well as a profile-based subset. This design enables controlled analysis of the effects of data source, topical domain, and text length. Using the provided training splits, we conduct a systematic evaluation of strong baselines and state-of-the-art models and find that our best approach, a fine-tuned large
language model, outperforms recent baselines by up to 0.09 absolute F1 score and surpasses GPT-5 in a zero-shot setting by 0.08. We further observe a trade-off between specialization and generalization: models trained on specific data types perform best under matching conditions but generalize less well across data regimes, a limitation that can be mitigated by combining training sources. Overall, GerAV provides a challenging and versatile
benchmark for advancing research on German and cross-domain AV. Our code and information about data access are available on GitHub.
Read the full abstract ▾
All authors Lotta Kiefer, Christoph Leiter, Sotaro Takeshita, Elena Schmidt, Steffen Eger
Computation and Language · Cross-listed from cs.LG
Who Benefits From Sinus Surgery? Comparing Generative AI and Supervised Machine Learning for Predicting Surgical Outcomes in Chronic Rhinosinusitis
Sayeed Shafayet Chowdhury, Snehasis Mukhopadhyay, Shiaofen Fang et al. (+1 more)
Artificial intelligence has reshaped medical imaging, yet the use of AI on clinical data for prospective decision support remains limited. We study pre-operative prediction of clinically meaningful improvement in chronic rhinosinusitis (CRS), defining success as a more than 8.9-point reduction in SNOT-22 at 6 months (MCID). In a prospectively collected cohort where all patients underwent surgery, we ask whether models using only pre-operative clinical data could have identified those who would have poor outcomes, i.e. those who should have avoided surgery. We
benchmark supervised ML (logistic regression, tree ensembles, and an in-house MLP) against generative AI (ChatGPT, Claude, Gemini, Perplexity), giving each the same structured inputs and constraining outputs to binary recommendations with confidence. Our best ML model (MLP) achieves 85 % accuracy with superior calibration and decision-curve net benefit. GenAI models underperform on discrimination and calibration across zero-shot setting. Notably, GenAI justifications align with clinician heuristics and the MLP's feature importance, repeatedly highlighting baseline SNOT-22, CT/endoscopy severity, polyp phenotype, and physchology/pain comorbidities. We provide a reproducible tabular-to-GenAI evaluation protocol and subgroup analyses. Findings support an ML-first, GenAI- augmented workflow: deploy calibrated ML for primary triage of
surgical candidacy, with GenAI as an explainer to enhance transparency and shared decision-making.
Read the full abstract ▾
All authors Sayeed Shafayet Chowdhury, Snehasis Mukhopadhyay, Shiaofen Fang, Vijay R. Ramakrishnan
Computation and Language · Cross-listed from cs.AI
Hidden in Plain Text: Measuring LLM Deception Quality Against Human Baselines Using Social Deduction Games
Christopher Kao, Vanshika Vats and James Davis
Large
Language Model (LLM)
agents are increasingly used in many applications, raising concerns about their safety. While previous work has shown that LLMs can deceive in controlled tasks, less is known about their ability to deceive using natural
language in social contexts. In this paper, we study deception in the Social Deduction Game (SDG) Mafia, where success is dependent on deceiving others through conversation. Unlike previous SDG studies, we use an asynchronous multi-
agent framework which better simulates realistic social contexts. We simulate 35 Mafia games with GPT-4o LLM
agents. We then create a Mafia Detector using GPT-4-Turbo to analyze game transcripts without player role information to predict the mafia players. We use prediction accuracy as a surrogate marker for deception quality. We compare this prediction accuracy to that of 28 human games and a random baseline. Results show that the Mafia Detector's mafia prediction accuracy is lower on LLM games than on human games. The result is consistent regardless of the game days and the number of mafias detected. This indicates that LLMs blend in better and thus deceive more effectively. We also release a
dataset of LLM Mafia transcripts to support future research. Our findings underscore both the sophistication and risks of LLM deception in social contexts.
Read the full abstract ▾
Computation and Language
Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning
Zhihang Yuan, Chengyu Yue, Long Huang et al. (+2 more)
Instruction tuning is a standard paradigm for adapting large
language models (LLMs), but modern instruction
datasets are large, noisy, and redundant, making full-data fine-tuning costly and often unnecessary. Existing data selection methods either build expensive gradient datastores or assign static scores from a weak proxy, largely ignoring evolving uncertainty, and thus missing a key source of LLM interpretability. We propose GRADFILTERING, an objective-agnostic, uncertainty-aware data selection framework that utilizes a small GPT-2 proxy with a LoRA ensemble and aggregates per-example gradients into a Gradient Signal-to-Noise Ratio (G-SNR) utility. Our method matches or surpasses random subsets and strong baselines in most LLM-as-a-judge evaluations as well as in human assessment. Moreover, GRADFILTERING-selected subsets converge faster than competitive filters under the same compute budget, reflecting the benefit of uncertainty-aware scoring.
Read the full abstract ▾
All authors Zhihang Yuan, Chengyu Yue, Long Huang, Litu Ou, Lei Shi
Computation and Language
OptiSQL: Executable SQL Generation from Optical Tokens
Sifan Li, Hongkai Chen, Yujun Cai et al. (+3 more)
Executable SQL generation is typically studied in text-to-SQL settings, where tables are provided as fully linearized textual schemas and contents. While effective, this formulation assumes access to structured text and incurs substantial
token overhead, which is misaligned with many real-world scenarios where tables appear as visual artifacts in documents or webpages. We investigate whether compact optical representations can serve as an efficient interface for executable semantic parsing. We present OptiSQL, a
vision-driven framework that generates executable SQL directly from table images and natural
language questions using compact optical
tokens. OptiSQL leverages an OCR-oriented visual encoder to compress table structure and content into a small set of optical
tokens and fine-tunes a pretrained decoder for SQL generation while freezing the encoder to isolate representation sufficiency. Experiments on a visualized version of Spider 2.0-Snow show that OptiSQL retains strong execution accuracy while reducing table input
tokens by an order of magnitude. Robustness analyses further demonstrate that optical
tokens preserve essential structural information under visual perturbations.
Read the full abstract ▾
All authors Sifan Li, Hongkai Chen, Yujun Cai, Liyang Chen, Qingwen Ye, Yiwei Wang
Computation and Language
Dr. Assistant: Enhancing Clinical Diagnostic Inquiry via Structured Diagnostic Reasoning Data and Reinforcement Learning
Yue Guo, Fanfu Wang, Jianwei Lv et al. (+9 more)
Clinical Decision Support Systems (CDSSs) provide
reasoning and inquiry guidance for physicians, yet they face notable challenges, including high maintenance costs and low generalization capability. Recently, Large
Language Models (LLMs) have been widely adopted in healthcare due to their extensive knowledge reserves,
retrieval, and communication capabilities. While LLMs show promise and excel at medical
benchmarks, their diagnostic
reasoning and inquiry skills are constrained. To mitigate this issue, we propose (1) Clinical Diagnostic
Reasoning Data (CDRD) structure to capture abstract clinical
reasoning logic, and a pipeline for its construction, and (2) the Dr. Assistant, a clinical diagnostic model equipped with clinical
reasoning and inquiry skills. Its training involves a two-stage process: SFT, followed by RL with a tailored reward function. We also introduce a
benchmark to evaluate both diagnostic
reasoning and inquiry. Our experiments demonstrate that the Dr. Assistant outperforms open-source models and achieves competitive performance to closed-source models, providing an effective solution for clinical diagnostic inquiry guidance. Project information can be found at: https://github.com/YGswu/Dr.-Assistant .
Read the full abstract ▾
All authors Yue Guo, Fanfu Wang, Jianwei Lv, Xincheng Shi, Yuchen Li, Youya Wang, Yunsheng Zeng, Yujing Liu, Yunhao Qiao, Gen Li, Junfeng Wang, Bo Yuan
Computation and Language
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
Zhiyuan Shi, Qibo Qiu, Feng Xue et al. (+5 more)
The linear
memory growth of the KV cache poses a significant bottleneck for LLM inference in long-context tasks. Existing static compression methods often fail to preserve globally important information. Although recent dynamic
retrieval approaches attempt to address this issue, they typically suffer from coarse-grained caching strategies and incur high I/O overhead. To overcome these limitations, we propose HeteroCache, a training-free dynamic compression framework. Our method is built on two key insights:
attention heads exhibit diverse temporal heterogeneity, and there is significant spatial redundancy among heads within the same layer. Guided by these insights, HeteroCache categorizes heads based on stability and similarity, applying a fine-grained weighting strategy that allocates larger cache budgets to heads with rapidly shifting
attention to capture context changes. Furthermore, it features a hierarchical storage mechanism where representative heads monitor
attention drift to trigger asynchronous, on-demand context
retrieval, thereby hiding I/O latency. Experiments demonstrate that HeteroCache achieves state-of-the-art performance on long-context
benchmarks and accelerates decoding by up to $3\times$ compared to the original model with a 224K context. Our code is available at https://github.com/ponytaill/HeteroCache.
Read the full abstract ▾
All authors Zhiyuan Shi, Qibo Qiu, Feng Xue, Zhonglin Jiang, Li Yu, Jian Jiang, Xiaofei He, Wenxiao Wang
Computation and Language
CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks
Jiayu Lin and Zhongyu Wei
Large
language models (LLMs)
alignment ensures model behaviors reflect human value. Existing
alignment strategies primarily follow two paths: one assumes a universal value set for a unified goal (i.e., one-size-fits-all), while the other treats every individual as unique to customize models (i.e., individual-level). However, assuming a monolithic value space marginalizes minority norms, while tailoring individual models is prohibitively expensive. Recognizing that human society is organized into social clusters with high intra-group value
alignment, we propose community-level
alignment as a "middle ground". Practically, we introduce CommunityBench, the first large-scale
benchmark for community-level
alignment evaluation, featuring four tasks grounded in Common Identity and Common Bond theory. With CommunityBench, we conduct a comprehensive evaluation of various foundation models on CommunityBench, revealing that current LLMs exhibit limited capacity to model community-specific preferences. Furthermore, we investigate the potential of community-level
alignment in facilitating individual modeling, providing a promising direction for scalable and pluralistic
alignment.
Read the full abstract ▾
Computation and Language
Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis
Chunlei Meng, Ziyang Zhou, Lucas He et al. (+3 more)
Multimodal Sentiment Analysis integrates Linguistic, Visual, and Acoustic. Mainstream approaches based on modality-invariant and modality-specific factorization or on complex fusion still rely on spatiotemporal mixed modeling. This ignores spatiotemporal heterogeneity, leading to spatiotemporal information asymmetry and thus limited performance. Hence, we propose TSDA, Temporal-Spatial Decouple before Act, which explicitly decouples each modality into temporal dynamics and spatial structural context before any interaction. For every modality, a temporal encoder and a spatial encoder project signals into separate temporal and spatial body. Factor-Consistent Cross-Modal
Alignment then aligns temporal features only with their temporal counterparts across modalities, and spatial features only with their spatial counterparts. Factor specific supervision and decorrelation regularization reduce cross factor leakage while preserving complementarity. A Gated Recouple module subsequently recouples the aligned streams for task. Extensive experiments show that TSDA outperforms baselines. Ablation analysis studies confirm the necessity and interpretability of the design.
Read the full abstract ▾
All authors Chunlei Meng, Ziyang Zhou, Lucas He, Xiaojing Du, Chun Ouyang, Zhongxue Gan
Computation and Language
Beyond Known Facts: Generating Unseen Temporal Knowledge to Address Data Contamination in LLM Evaluation
Arthur Amalvy and Hen-Hsen Huang
The automatic extraction of information is important for populating large web knowledge bases such as Wikidata. The temporal version of that task, temporal knowledge graph extraction (TKGE), involves extracting temporally grounded facts from text, represented as semantic quadruples (subject, relation, object, timestamp). Many recent systems take advantage of large
language models (LLMs), which are becoming a new cornerstone of the web due to their performance on many tasks across the natural
language processing (NLP) field. Despite the importance of TKGE, existing
datasets for training and evaluation remain scarce, and contamination of evaluation data is an unaddressed issue, potentially inflating LLMs' perceived performance due to overlaps between training and evaluation sets. To mitigate these challenges, we propose a novel synthetic evaluation
dataset constructed from predicted future, previously unseen temporal facts, thereby eliminating contamination and enabling robust and unbiased benchmarking. Our
dataset creation involves a two-step approach: (1) Temporal Knowledge Graph
Forecasting (TKGF) generates plausible future quadruples, which are subsequently filtered to adhere to the original knowledge base schema; (2) LLMs perform quadruple-to-text generation, creating semantically aligned textual descriptions. We
benchmark Extract, Define and Canonicalize (EDC), a state-of-the-art LLM-based extraction framework, demonstrating that LLM performance decreases when evaluated on our
dataset compared to a
dataset of known facts. We publicly release our
dataset consisting of 4.2K future quadruples and corresponding textual descriptions, along with the generation methodology, enabling continuous creation of unlimited future temporal
datasets to serve as long-term, contamination-free
benchmarks for TKGE.
Read the full abstract ▾
Computation and Language
Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge
Xiaolin Zhou, Zheng Luo, Yicheng Gao et al. (+4 more)
Recent advances in Large
Language Models (LLMs) have incentivized the development of LLM-as-a-judge, an application of LLMs where they are used as judges to decide the quality of a certain piece of text given a certain context. However, previous studies have demonstrated that LLM-as-a-judge can be biased towards different aspects of the judged texts, which often do not align with human preference. One of the identified biases is
language bias, which indicates that the decision of LLM-as-a-judge can differ based on the
language of the judged texts. In this paper, we study two types of
language bias in pairwise LLM-as-a-judge: (1) performance disparity between
languages when the judge is prompted to compare options from the same
language, and (2) bias towards options written in major
languages when the judge is prompted to compare options of two different
languages. We find that for same-
language judging, there exist significant performance disparities across
language families, with European
languages consistently outperforming African
languages, and this bias is more pronounced in culturally-related subjects. For inter-
language judging, we observe that most models favor English answers, and that this preference is influenced more by answer
language than question
language. Finally, we investigate whether
language bias is in fact caused by low-perplexity bias, a previously identified bias of LLM-as-a-judge, and we find that while perplexity is slightly correlated with
language bias,
language bias cannot be fully explained by perplexity only.
Read the full abstract ▾
All authors Xiaolin Zhou, Zheng Luo, Yicheng Gao, Qixuan Chen, Xiyang Hu, Yue Zhao, Ruishan Liu
Computation and Language
Towards Token-Level Text Anomaly Detection
Yang Cao, Bicheng Yu, Sikun Yang et al. (+2 more)
Despite significant progress in text anomaly detection for web applications such as spam filtering and fake news detection, existing methods are fundamentally limited to document-level analysis, unable to identify which specific parts of a text are anomalous. We introduce
token-level anomaly detection, a novel paradigm that enables fine-grained localization of anomalies within text. We formally define text anomalies at both document and
token-levels, and propose a unified detection framework that operates across multiple levels. To facilitate research in this direction, we collect and annotate three
benchmark datasets spanning spam, reviews and grammar errors with
token-level labels. Experimental results demonstrate that our framework get better performance than other 6 baselines, opening new possibilities for precise anomaly localization in text. All the codes and data are publicly available on https://github.com/charles-cao/TokenCore.
Read the full abstract ▾
All authors Yang Cao, Bicheng Yu, Sikun Yang, Ming Liu, Yujiu Yang
Computation and Language
Activation-Space Anchored Access Control for Multi-Class Permission Reasoning in Large Language Models
Zhaopeng Zhang, Pengcheng Sun, Lan Zhang et al. (+4 more)
Large
language models (LLMs) are increasingly deployed over knowledge bases for efficient knowledge
retrieval and question answering. However, LLMs can inadvertently answer beyond a user's permission scope, leaking sensitive content, thus making it difficult to deploy knowledge-base QA under fine-grained access control requirements. In this work, we identify a
geometric regularity in intermediate activations: for the same query, representations induced by different permission scopes cluster distinctly and are readily separable. Building on this separability, we propose Activation-space Anchored Access Control (AAAC), a training-free framework for multi-class permission control. AAAC constructs an anchor bank, with one permission anchor per class, from a small offline sample set and requires no fine-tuning. At inference time, a multi-anchor steering mechanism redirects each query's activations toward the anchor-defined authorized region associated with the current user, thereby suppressing over-privileged generations by design. Finally, extensive experiments across three LLM families demonstrate that AAAC reduces permission violation rates by up to 86.5% and
prompt-based attack success rates by 90.7%, while improving response usability with minor inference overhead compared to baselines.
Read the full abstract ▾
All authors Zhaopeng Zhang, Pengcheng Sun, Lan Zhang, Chen Tang, Jiewei Lai, Yunhao Wang, Hui Jin
Computation and Language
CauScientist: Teaching LLMs to Respect Data for Causal Discovery
Bo Peng, Sirui Chen, Lei Xu et al. (+1 more)
Causal discovery is fundamental to scientific understanding and reliable decision-making. Existing approaches face critical limitations: purely data-driven methods suffer from statistical indistinguishability and modeling assumptions, while recent LLM-based methods either ignore statistical evidence or incorporate unverified priors that can mislead result. To this end, we propose CauScientist, a collaborative framework that synergizes LLMs as hypothesis-generating "data scientists" with probabilistic statistics as rigorous "verifiers". CauScientist employs hybrid initialization to select superior starting graphs, iteratively refines structures through LLM-proposed modifications validated by statistical criteria, and maintains error
memory to guide efficient search space. Experiments demonstrate that CauScientist substantially outperforms purely data-driven baselines, achieving up to 53.8% F1 score improvement and enhancing recall from 35.0% to 100.0%. Notably, while standalone LLM performance degrades with graph complexity, CauScientist reduces structural hamming distance (SHD) by 44.0% compared to Qwen3-32B on 37-node graphs. Our project page is at https://github.com/OpenCausaLab/CauScientist.
Read the full abstract ▾
All authors Bo Peng, Sirui Chen, Lei Xu, Chaochao Lu
Computation and Language · Cross-listed from cs.AI
DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems
Maojun Sun, Yifei Xie, Yue Wu et al. (+5 more)
Recent LLM-based data
agents aim to automate data science tasks ranging from data analysis to deep learning. However, the open-ended nature of real-world data science problems, which often span multiple taxonomies and lack standard answers, poses a significant challenge for evaluation. To address this, we introduce DSAEval, a
benchmark comprising 641 real-world data science problems grounded in 285 diverse
datasets, covering both structured and unstructured data (e.g., image and text). DSAEval incorporates three distinctive features: (1)
Multimodal Environment Perception, which enables
agents to interpret observations from multiple modalities, including text and
vision; (2) Multi-Query Interactions, which mirror the iterative and cumulative nature of real-world data science projects; and (3) Multi-Dimensional Evaluation, which provides a holistic assessment across
reasoning, code, and results. We systematically evaluate 13 recent advanced agentic LLMs using DSAEval. Our results show that Claude-Sonnet-4.5 achieves the strongest overall performance,
MiMo-V2-Pro and GPT-5.2 lead in duration and step efficiency, respectively, and
MiMo-V2-Flash is the most cost-effective. We further demonstrate that
multimodal perception consistently improves performance on
vision-related tasks, with gains ranging from 2.04% to 11.30%. Overall, while current data science
agents perform well on structured data and routine data analysis workflows, substantial challenges remain in unstructured domains. Finally, we offer critical insights and outline future research directions.
Read the full abstract ▾
All authors Maojun Sun, Yifei Xie, Yue Wu, Ruijian Han, Binyan Jiang, Defeng Sun, Yancheng Yuan, Jian Huang
Computation and Language
Vulnerability of LLMs' Stated Beliefs? LLMs Belief Resistance Check Through Strategic Persuasive Conversation Interventions
Fan Huang, Haewoon Kwak and Jisun An
Large
Language Models (LLMs) are increasingly employed in various question-answering tasks. However, recent studies showcase that LLMs are susceptible to persuasion and could adopt counterfactual beliefs. We present a systematic evaluation of LLM susceptibility to persuasion under the Source–Message–Channel–Receiver (SMCR) communication framework. Across six mainstream Large
Language Models (LLMs) and three domains (factual knowledge, medical QA, and social bias), we analyze how different persuasive strategies influence stated belief stability over multiple interaction turns. We further examine whether verbalized confidence prompting (i.e., eliciting self-reported confidence scores) affects resistance to persuasion. Results show that the smallest model (Llama 3.2-3B) exhibits extreme compliance, with 82.5% of belief changes occurring at the first persuasive turn (average end turn of 1.1–1.4). Contrary to expectations, verbalized confidence prompting increases vulnerability by accelerating belief erosion rather than enhancing robustness. Finally, an exploratory study of
adversarial fine-tuning reveals highly model-dependent effectiveness: GPT-4o-mini achieves near-complete robustness (98.6%), and Mistral~7B improves substantially (35.7% $\rightarrow$ 79.3%), but Llama models remain highly susceptible ($<$14% RQ1) even when fine-tuned on their own failure cases. Together, these findings highlight substantial model-dependent limits of current robustness interventions and offer guidance for developing more trustworthy LLMs.
Read the full abstract ▾
Computation and Language
TREX: Tokenizer Regression for Optimal Data Mixture
Inho Won, Hangyeol Yoo, Minkyung Cho et al. (+3 more)
Building effective tokenizers for
multilingual Large
Language Models (LLMs) requires careful control over
language-specific data mixtures. While a tokenizer's compression performance critically affects the efficiency of LLM training and inference, existing approaches rely on heuristics or costly large-scale searches to determine optimal
language ratios. We introduce Tokenizer Regression for Optimal Data MiXture (TREX), a regression-based framework that efficiently predicts the optimal data mixture for tokenizer training. TREX trains small-scale proxy tokenizers on random mixtures, gathers their compression statistics, and learns to predict compression performance from data mixtures. This learned model enables scalable mixture search before large-scale tokenizer training, mitigating the accuracy-cost trade-off in
multilingual tokenizer design. Tokenizers trained with TReX's predicted mixtures outperform mixtures based on LLaMA3 and uniform distributions by up to 12% in both inand out-of-distribution compression efficiency, demonstrating strong scalability, robustness, and practical effectiveness.
Read the full abstract ▾
All authors Inho Won, Hangyeol Yoo, Minkyung Cho, Jungyeul Park, Hoyun Song, KyungTae Lim
Computation and Language
Comparing Without Saying: A Dataset and Benchmark for Implicit Comparative Opinion Mining from Same-User Reviews
Thanh-Lam T. Nguyen, Ngoc-Quang Le, Quoc-Trung Phu et al. (+4 more)
Existing studies on comparative opinion mining have mainly focused on explicit comparative expressions, which are uncommon in real-world reviews. This leaves implicit comparisons - here users express preferences across separate reviews - largely underexplored. We introduce SUDO, a novel
dataset for implicit comparative opinion mining from same-user reviews, allowing reliable inference of user preferences even without explicit comparative cues. SUDO comprises 4,150 annotated review pairs (15,191 sentences) with a bi-level structure capturing aspect-level mentions and review-level preferences. We
benchmark this task using two baseline architectures: traditional machine learning- and
language model-based baselines. Experimental results show that while the latter outperforms the former, overall performance remains moderate, revealing the inherent difficulty of the task and establishing SUDO as a challenging and valuable
benchmark for future research.
Read the full abstract ▾
All authors Thanh-Lam T. Nguyen, Ngoc-Quang Le, Quoc-Trung Phu, Thi-Phuong Le, Ngoc-Huyen Pham, Phuong-Nguyen Nguyen, Hoang-Quynh Le
Computation and Language · Cross-listed from cs.LG
Self-Improvement as Coherence Optimization: A Theoretical Account
Tianyi Qiu, Ahmed Hani Ismail, Zhonghao He et al. (+1 more)
Can
language models improve their accuracy without external supervision? Methods such as debate, bootstrap, and internal coherence maximization achieve this surprising feat, even matching golden finetuning performance. Yet why they work remains theoretically unclear. We show that they can all be understood as coherence optimization, the search for a context-to-behavior mapping that is most compressible and jointly predictable, with debate an exact instance and bootstrap and internal coherence maximization closely related to it. We prove that coherence optimization is equivalent to description-length regularization, and that among all such regularization schemes, coherence regularization with a prior derived from a pretrained model optimizes a lower bound of worst-case accuracy for semi-supervised learning. Our theory, supported by preliminary experiments, explains why feedback-free self-improvement works and predicts when it should succeed or fail.
Read the full abstract ▾
All authors Tianyi Qiu, Ahmed Hani Ismail, Zhonghao He, Shi Feng
Computation and Language · Cross-listed from cs.AI
Leveraging ChatGPT and Other NLP Methods for Identifying Risk and Protective Behaviors in MSM: Social Media and Dating apps Text Analysis
Mehrab Beikzadeh, Chenglin Hong, Cory J Cascalheira et al. (+3 more)
Men who have sex with men (MSM) are at elevated risk for sexually transmitted infections and harmful drinking compared to heterosexual men. Text data collected from social media and dating applications may provide new opportunities for personalized public health interventions by enabling automatic identification of risk and protective behaviors. In this study, we evaluated whether text from social media and dating apps can be used to predict sexual risk behaviors, alcohol use, and pre-exposure prophylaxis (PrEP) uptake among MSM. With participant consent, we collected textual data and trained machine learning models using features derived from ChatGPT
embeddings, BERT
embeddings, LIWC, and a dictionary-based risk term approach. The models achieved strong performance in predicting monthly binge drinking and having more than five sexual partners, with F1 scores of 0.78, and moderate performance in predicting PrEP use and heavy drinking, with F1 scores of 0.64 and 0.63. These findings demonstrate that social media and dating app text data can provide valuable insights into risk and protective behaviors and highlight the potential of large
language model-based methods to support scalable and personalized public health interventions for MSM.
Read the full abstract ▾
All authors Mehrab Beikzadeh, Chenglin Hong, Cory J Cascalheira, Callisto Boka, Majid Sarrafzadeh, Ian W Holloway
Computation and Language
HateXScore: A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations
Yujia Hu and Roy Ka-Wei Lee
Hateful
speech detection is a key component of content moderation, yet current evaluation frameworks rarely assess why a text is deemed hateful. We introduce HateXScore, a four-component metric suite designed to evaluate the
reasoning quality of model explanations. It assesses (i) conclusion explicitness, (ii) faithfulness and causal grounding of quoted spans, (iii) protected group identification (policy-configurable), and (iv) logical consistency among these elements. Evaluated on six diverse hate
speech datasets, HateXScore is intended as a diagnostic complement to reveal interpretability failures and annotation inconsistencies that are invisible to standard metrics like Accuracy or F1. Moreover, human evaluation shows strong agreement with HateXScore, validating it as a practical tool for trustworthy and transparent moderation. \textcolor{red}{Disclaimer: This paper contains sensitive content that may be disturbing to some readers.}
Read the full abstract ▾
Computation and Language
When Wording Steers the Evaluation: Framing Bias in LLM judges
Yerin Hwang, Dongryeol Lee, Taegwan Kang et al. (+2 more)
Large
language models (LLMs) are known to produce varying responses depending on
prompt phrasing, indicating that subtle guidance in phrasing can steer their answers. However, the impact of this framing bias on LLM-based evaluation, where models are expected to make stable and impartial judgments, remains largely underexplored. Drawing inspiration from the framing effect in psychology, we systematically investigate how deliberate
prompt framing skews model judgments across four high-stakes evaluation tasks. We design symmetric
prompts using predicate-positive and predicate-negative constructions and demonstrate that such framing induces significant discrepancies in model outputs. Across 14 LLM judges, we observe clear susceptibility to framing, with model families showing distinct tendencies toward agreement or rejection. These findings suggest that framing bias is a structural property of current LLM-based evaluation systems, underscoring the need for framing-aware protocols.
Read the full abstract ▾
All authors Yerin Hwang, Dongryeol Lee, Taegwan Kang, Minwoo Lee, Kyomin Jung
Computation and Language · Cross-listed from cs.CR
Eliciting Harmful Capabilities by Fine-Tuning On Safeguarded Outputs
Jackson Kaunismaa, Avery Griffin, John Hughes et al. (+3 more)
Model developers implement safeguards in frontier models to prevent misuse, for example, by employing classifiers to filter dangerous outputs. In this work, we demonstrate that even robustly safeguarded models can be used to elicit harmful capabilities in open-source models through elicitation attacks. Our elicitation attacks consist of three stages: (i) constructing
prompts in adjacent domains to a target harmful task that do not request dangerous information; (ii) obtaining responses to these
prompts from safeguarded frontier models; (iii) fine-tuning open-source models on these
prompt-output pairs. Since the requested
prompts cannot be used to directly cause harm, they are not refused by frontier model safeguards. We evaluate these elicitation attacks within the domain of hazardous chemical synthesis and processing, and demonstrate that our attacks recover approximately 40% of the capability gap between the base open-source model and an unrestricted frontier model. We then show that the efficacy of elicitation attacks scales with the capability of the frontier model and the amount of generated fine-tuning data. Our work demonstrates the challenge of mitigating ecosystem level risks with output-level safeguards.
Read the full abstract ▾
All authors Jackson Kaunismaa, Avery Griffin, John Hughes, Christina Q. Knight, Mrinank Sharma, Erik Jones