Computation and Language
Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth
Yoav Gur-Arieh, Ana Marasović and Mor Geva
Chains of thought (CoTs) have become central in interpreting and
auditing behaviors of large
language models. Yet growing evidence suggests that these traces often fail to faithfully represent the computations behind a model's predictions. Several faithfulness metrics have been proposed, but whether they indeed measure faithfulness remains unknown. Answering this requires ground-truth labels, which are hard to obtain since internal computations are not directly observable. Consequently, most works proposing metrics report only absolute scores or comparisons to prior metrics, and the few existing
benchmarks rely on proxies like plausibility or importance, properties orthogonal to faithfulness that can mislead about whether a CoT can be trusted. We address this challenge by constructing tasks whose outputs reveal which intermediate computations must have produced them, and developing an automated labeling pipeline that yields ground-truth faithfulness labels at both the step and CoT level. Building on this methodology, we present BonaFide, a
benchmark of 3,066 labeled CoTs across 13 tasks and 10 models, and use it to conduct the first systematic evaluation of prominent faithfulness metrics. Our experiments show that most metrics perform near chance, exhibit strong prediction biases and degrade on longer CoTs. The best metric reaches only 0.70 AUROC at the CoT level while another reaches 0.59 at the step level, with neither transferring across settings, while entailing prohibitively high computational cost. Our results expose fundamental gaps in current faithfulness evaluation and call for the development of more reliable and efficient metrics.
Read the full abstract ▾
Computation and Language
TRACE: A taxonomy-grounded synthetic dataset for teaching-program generation and session interpretation in Applied Behavior Analysis
Festus Kahunla
Applied Behavior Analysis (ABA) is a clinical discipline whose documentation, teaching programs and multi-session behavioral logs, is formulaic and high-volume, yet real session data is HIPAA-protected and bound by professional confidentiality rules, blocking the release of a training corpus. We present TRACE (Taxonomy-Referenced ABA Clinical Examples), a 2,999-example synthetic instruction-tuning
dataset covering two ABA tasks: teaching-program generation across Discrete Trial Training, Natural Environment Teaching, and Task Analysis; and multi-session behavioral interpretation across twelve trajectory patterns and thirteen target behaviors. Every example is produced by a deterministic taxonomy-driven generator grounded in the canonical ABA literature, and every example carries complete sampling provenance, the exact taxonomy
cells that produced it. The
dataset is released under CC BY-NC 4.0 for data and MIT for code, with stratified train (2,549), validation (149), test (281), and sanity (20) splits. TRACE is a research artifact and has not been clinically validated.
Read the full abstract ▾
Computation and Language
Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
Yangneng Chen and Jing Li
Large
Vision-
Language Models (LVLMs) extend large
language models with visual understanding, but remain vulnerable to
hallucination, where outputs are fluent yet inconsistent with images. Recent studies link this issue to
language bias-the tendency of LVLMs to over-rely on text while neglecting visual inputs. Yet most analyses remain empirical without uncovering its underlying cause. In this paper, we provide a systematic study of
language bias and identify its root in modality misalignment during training. Our analysis shows that both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) often prioritize textual improvements, which may cause LVLMs to overly lean toward
language modeling rather than balanced
multimodal understanding. To address this, we propose two simple yet effective methods:
Language Bias Regularization (LBR) which mitigates
language bias through regularization during instruction tuning, and
Language Bias Penalty (LBP), which penalizes
language bias in the DPO training process. Extensive experiments across diverse models and
benchmarks demonstrate the effectiveness of our approach. LBR consistently improves performance on over ten general
benchmarks, while LBP significantly reduces
hallucination and improves trustworthiness. Together, these methods not only mitigate
language bias but also advance the overall
alignment of LVLMs, all without introducing any additional data or auxiliary models. Our code is publicly available at https://github.com/lab-klc/LVLM-
Language-Bias.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Privacy-Preserving Local Language Models for Longitudinal Data Retrieval in Chronic Dermatologic Disease: Implementation in Pemphigus Patients
Abdurrahim Yilmaz, Ayşe Esra Koku Aksu, Duygu Yamen et al. (+5 more)
Chronic dermatologic diseases such as pemphigus require long-term follow-up, generating extensive longitudinal clinical documentation that is difficult to review comprehensively during routine visits and increasing clinician workload as well as the risk of missing critical historical information. We evaluated whether a locally deployed,
privacy-preserving small
language model (SLM) could retrieve structured clinical features and generate longitudinal summaries from long-term dermatology follow-up records. In this retrospective case series, thirty pemphigus patients contributed 541 visit notes that were aggregated into full longitudinal records (89,336 words); 56 clinically relevant features were annotated by two expert dermatologists. The locally deployed SLM (Qwen3 4B Thinking 2507) was queried with each complete record to retrieve 56 features and generate one final report summaries. Across 1,680 feature
retrieval tasks, mean accuracy was 82.25%. Dermatologists' ratings of AI-generated summaries were high for overall quality (8.23-8.47), clinical accuracy (7.93-8.20), and usefulness (8.47-8.50), with no significant inter-evaluator differences and an overall preference for AI summaries in 53.3% of evaluations. These findings suggest that
privacy-preserving, locally deployed SLMs can outperform medical experts and reliably generate clinically meaningful longitudinal summaries. SLMs may support clinical decision-making when integrated with appropriate oversight.
Read the full abstract ▾
All authors Abdurrahim Yilmaz, Ayşe Esra Koku Aksu, Duygu Yamen, Vefa Asli Erdemir, Mehmet Salih Gurel, Gulsum Gencoglan, Joram M. Posma, Burak Temelkuran
Computation and Language
Better, Faster: Harnessing Self-Improvement in Large Reasoning Models
Qihuang Zhong, Liang Ding, Juhua Liu et al. (+3 more)
Self-improvement training enables the large
reasoning models (LRMs) to improve themselves by self-generating
reasoning trajectories as training data without external supervision. However, we find that this method often falls short in complex
reasoning tasks and even leads to model collapse. Through a series of preliminary analyses, we reveal two problems: (1) data imbalance, where most training samples are simple, but the challenging yet crucial samples are scarce; (2) overthinking, where many undesired samples with redundant
reasoning steps are used for self-training. To this end, we propose HSIR, which effectively Harnesses Self-Improvement in large
Reasoning models via two simple-yet-effective approaches. Specifically, HSIR introduces a verify-then-exit sampling strategy to mitigate data imbalance by efficiently collecting more accurate solutions for difficult queries, and designs an Intrinsic Diversity score to quantify overthinking and filter out the undesired solutions. We apply HSIR to various post-training paradigms, among which we further propose H-GRPO, an enhanced GRPO
algorithm that leverages the intrinsic diversity as an external reward to encourage concise and diverse
reasoning via reinforcement learning. Extensive results show that HSIR not only effectively enhances the
reasoning performance, i.e., bringing up to +10.9% average performance gains, but also significantly improves the
reasoning efficiency by reducing up to 42.4% relative inference overhead.
Read the full abstract ▾
All authors Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Leszek Rutkowski, Dacheng Tao
Computation and Language
Exploring Profiles of Cognitive Distortions Associated with Mental Health Disorders
Alina Anikejeva and Kairit Sirts
Cognitive distortions, distorted patterns of thinking, have been increasingly studied in computational mental health research. Although they are related to many, if not all, mental health disorders, most existing studies focus primarily on depression. In this work, we explore distortion profiles across multiple mental health conditions. We analyzed a large Reddit-based
dataset containing posts from nine self-reported mental health groups as well as a control group using both an n-gram-based method and a fine-tuned
transformer model for detecting cognitive distortions. Mental health groups, both when pooled together and when examined individually, showed higher prevalence of cognitive distortions compared to the control group, with the effect sizes ranging from small to moderate. When comparing distortion profiles across conditions, we observed largely similar patterns, although some groups exhibited overall higher levels of distortions than others. These findings suggest that relatively simple lexical approaches can be useful for exploratory analyses of group-level trends in large-scale mental health text data.
Read the full abstract ▾
Computation and Language
Large Language Model Selection with Limited Annotations
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch et al. (+2 more)
Choosing a Large
Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23
datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Read the full abstract ▾
All authors Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel
Computation and Language · Cross-listed from cs.CV
Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models
Farhad Nooralahzadeh, Benjamin Gundersen, Nicolas Deperrois et al. (+7 more)
Medical
vision-
language models (VLMs) often hallucinate findings when generating chest X-ray reports: they fabricate findings that are not present in the image, miss important ones, or locate them incorrectly. We mitigate this without weight updates by decoding-time residual steering on a per-
token sparse autoencoder (SAE) basis: Top-$K$ SAEs on late layers, causal steering against clinical errors, then combined suppress/boost intervention at inference time. On the MIMIC-CXR test split, our inference-only method improves the quality of generated reports for three radiology VLMs (RadVLM, LLaVA-Rad, and CheXOne), with relative improvements of +5.4%, +7.2%, and +17.0% in the clinical composite metric, and statistically significant GREEN gains on all backbones. A cross-model feature
alignment shows that the quality-promoting (boost) directions overlap strongly across architectures, whereas
hallucination-linked (suppress) directions are model-specific. Therefore, transferable steering must treat suppression per-backbone, rather than sharing a universal suppress list. The same recipe transfers zero-shot to IU-Xray (Green $+7.7\%$ rel.) without retraining, confirming that the identified features are properties of the model, not of the training corpus. We release causal feature sets and an interactive feature dashboard: https://cxr-sparse-feature-dashboard.netlify.app/.
Read the full abstract ▾
All authors Farhad Nooralahzadeh, Benjamin Gundersen, Nicolas Deperrois, Hidetoshi Matsuom, Mizuho Nishio, Thomas Frauenfelder, Ahmed Allam, Christian Blüthgen, Michael Moor, Michael Krauthammer
Computation and Language · Cross-listed from cs.CV
MinerU-Popo: Universal Post-Processing Model for Structured Document Parsing
Bangrui Xu, Ziyang Miao, Xuanhe Zhou et al. (+8 more)
VLM-based OCR models have become the de facto choice for document parsing, as they can accurately extract page-level elements (e.g., paragraphs within individual pages) together with their bounding boxes and textual content. However, downstream applications such as RAG require coherent document-level information, whereas these models often break cross-page continuity and fail to recover disrupted structures, such as paragraphs and tables truncated by page boundaries. Such relationships are not confined to a single page; instead, they require joint analysis of titles, paragraphs, tables, and images spanning multiple pages. A natural solution is therefore to reuse existing OCR outputs and reconstruct document-level logical structures through post-processing. To this end, we propose MinerU-Popo, a lightweight and universal framework for POst-Processing OCR outputs, which converts page-level results from diverse parsers into coherent document-level structures. MinerU-Popo decomposes the problem into four focused subtasks: text truncation recovery, table truncation recovery, title hierarchy reconstruction, and image-text association. To address these effectively, we build a task-oriented data engine with task-specific input filtering, and use the generated data (30K) to fine-tune a lightweight post-processing model (Qwen3-VL-4B). To support long documents, we introduce dynamic chunking with overlap-based synchronization, which aligns chunk-level outputs from the fine-tuned model and preserves global consistency. Finally, we assemble the aligned outputs into a tree-structured document representation, further enriched with node chunking and summaries for downstream
retrieval and analysis. Empirical results show MinerU-Popo improves title-hierarchy TEDS by at least 20% across all five tested OCR models, improves RAG accuracy and reduces per-query latency.
Read the full abstract ▾
All authors Bangrui Xu, Ziyang Miao, Xuanhe Zhou, Yiming Lin, Zirui Tang, Xiaomeng Zhao, Fan Wu, Cheng Tan, Fan Wu, Bin Wang, Conghui He
Computation and Language
Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization
Jingyi Sun, Qianli Wang, Pepa Atanasova et al. (+2 more)
Chain-of-Thought (CoT) faithfulness, i.e., whether CoTs genuinely reflect large
language models' (LLM) underlying behavior, is typically evaluated with metrics under two disjoint paradigms: contextual faithfulness, measured by perturbing the input or CoT trace, and parametric faithfulness, assessed by intervening on a model's parametric knowledge. Yet prior work compares them only descriptively. We fill this gap by proposing FaithMATE, a unified preference-
alignment interface for optimizing models towards either faithfulness paradigm. It enables us to investigate the interplay between the two paradigms, examining whether and to what extent faithfulness gains generalize within and across paradigms. Across three models, two
datasets, and six faithfulness metrics, we find that the two paradigms are positively coupled, yet asymmetric: optimizing towards parametric faithfulness yields consistent gains across both paradigms, whereas the contextual counterpart delivers more variable gains. Within the contextual paradigm, faithfulness gains on one metric do not consistently transfer to others, implying that existing contextual metrics capture disjoint facets of faithfulness and exposing inherent trade-offs. These findings imply that CoT faithfulness is not a monolithic objective and therefore requires multifaceted optimization and evaluation.
Read the full abstract ▾
All authors Jingyi Sun, Qianli Wang, Pepa Atanasova, Nils Feldhus, Isabelle Augenstein
Computation and Language
SEP-Attack: A Simple and Effective Paradigm for Transfer-Based Textual Adversarial Attack
Han Liu, Zhi Xu, Xiaotong Zhang et al. (+5 more)
Despite the strong performance of deep neural networks in modern Web and
language applications, they remain vulnerable to
adversarial attacks, especially transferable attacks that generate
adversarial examples using surrogate models without accessing the victim model. Transferable attacks in the text domain are still under-explored, with only a few studies addressing this challenging issue, often with suboptimal results due to equal treatment of submodels or inaccurate estimation of importance scores. To address these challenges, we propose a simple yet effective paradigm for transfer-based textual
adversarial attack, named SEP-Attack. Specifically, we employ the Determinantal Point Process (DPP) to generate diverse surrogate ensemble weights, representing the transferability of submodels. Using these weights, we introduce a new metric to evaluate prediction confidence scores, which in turn are used to calculate word importance scores and generate
adversarial candidates. Finally, we quantify the transferability score for each candidate and select the top ones as the final transferable
adversarial examples. Experiments conducted on four
datasets and two real-world APIs validate the efficacy of SEP-Attack, significantly outperforming state-of-the-art baselines.
Read the full abstract ▾
All authors Han Liu, Zhi Xu, Xiaotong Zhang, Feng Zhang, Xiaoming Xu, Wei Wang, Fenglong Ma, Hong Yu
Computation and Language
NITP: Next Implicit Token Prediction for LLM Pre-training
Xiangdong Zhang, Debing Zhang, Shaofeng Zhang et al. (+3 more)
Standard next-
token prediction (NTP) supervises
language models solely through discrete labels in the output logit space. We argue that this sparse one-hot supervision leaves the latent representation space under-constrained, allowing hidden states to drift into degenerate and anisotropic configurations that can limit generalization. To address this issue, we propose Next Implicit
Token Prediction (NITP), which augments discrete prediction with dense continuous supervision directly in the representation space. NITP trains the model to predict the implicit semantic content of the next
token, using shallow-layer representations from the same model as stable self-supervised targets. We provide theoretical analysis showing that NITP regularizes the optimization landscape by mitigating under-constrained degrees of freedom and encouraging a compact, structured representation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B parameters, NITP consistently improves downstream performance with negligible computational overhead. On a 9B MoE model, NITP achieves a 5.7% absolute improvement on MMLU-Pro, along with gains of 6.4% on C3 and 4.3% on CommonsenseQA, with approximately 2% additional training FLOPs and no additional inference cost. Our implementation is available at https://github.com/aHapBean/NITP.
Read the full abstract ▾
All authors Xiangdong Zhang, Debing Zhang, Shaofeng Zhang, Xiaohan Qin, Yu Cheng, Junchi Yan
Computation and Language
H$^{2}$MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer
Maryam Haghifam, Zifan He, Jason Cong et al. (+1 more)
Transformer-based LLMs achieve strong results on many
language tasks; however, long inputs remain challenging because context windows are finite, and prefill latency and
memory grow rapidly with
prompt length. Flat
token-stream processing and chunk-based
retrieval can therefore spend substantial computation and context budget on text unrelated to the query. Offline-indexed RAG additionally introduces external storage and index management overhead, and typically appends retrieved evidence as raw text, increasing prefill cost and latency. H^{2}MT makes long-context inference structure-aware: it builds a semantic hierarchy offline, computes a
memory embedding for each node via bottom-up post-order aggregation, and routes queries coarse-to-fine at inference to prune irrelevant branches early. On LongBench QA (NarrativeQA, HotpotQA, QASPER) and two structured technical-document settings, H MT achieves favorable quality efficiency trade-offs, delivering competitive ROUGE-L and F1 (where applicable) with lower peak GPU
memory and time-to-first-
token (TTFT) than
prompt compression,
memory-
token methods, and
retrieval-augmented generation baselines.
Read the full abstract ▾
All authors Maryam Haghifam, Zifan He, Jason Cong, Yizhou Sun
Computation and Language
MultiHaluDet: Multilingual Hallucination Detection via LLM Hidden State Probing
Riasad Alvi, Nurul Labib Sayeedi and Md. Faiyaz Abdullah Sayeedi
Hallucinations in Large
Language Models (LLMs) represent a critical barrier to their reliable deployment, a vulnerability heavily exacerbated in non-English and resource-constrained contexts. Existing detection approaches that rely on output confidence heuristics or single-layer internal representations frequently fail to capture deep, complex factual inconsistencies across diverse
languages. To address this, we introduce MultiHaluDet, a novel three-stage stacking framework that detects
multilingual hallucinations by probing the full hidden state trajectories of frozen LLMs without requiring
language-specific fine-tuning. Our method extracts sequential features across multiple layers and processes them via a hybrid architecture using multi-scale
attention and self-
attention pooling. By generating out-of-fold
embeddings that feed into a calibrated classical classifier ensemble, MultiHaluDet captures both fine-grained and coarse-grained patterns of factual inconsistency. Extensive experiments demonstrate that our framework achieves state-of-the-art detection performance, reaching up to 98.55% AUROC on the English HaluEval and TriviaQA
benchmarks using Mistral-7B and LLaMA2-7B architectures. Crucially, we rigorously evaluate our framework's cross-lingual generalization across high (French), medium (Bangla), and low-resource (Amharic)
languages. MultiHaluDet demonstrates exceptional representational robustness, consistently outperforming baselines and successfully transferring
hallucination detection capabilities across typologically diverse linguistic tiers.
Read the full abstract ▾
Computation and Language
Overview of the PsyDefDetect Shared Task at BioNLP 2026: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations
Hongbin Na, Zimu Wang, Zhaoming Chen et al. (+8 more)
We present an overview of PsyDefDetect, the shared task on detecting levels of psychological defense mechanisms in emotional support dialogues, co-located with BioNLP@ACL 2026. Grounded in the clinically validated Defense Mechanism Rating Scales (DMRS) framework, the task asks systems to classify a target seeker utterance, given its preceding dialogue context, into one of nine categories: seven hierarchical DMRS levels plus two auxiliary labels. Participants worked on PsyDefConv, a newly released corpus of 200 dialogues and 2336 help-seeker utterances annotated under DMRS with substantial inter-annotator agreement. The task attracted 172 participants on CodaBench who produced 563 submissions, with 21 teams officially registering their results for the final ranking. The best system achieved a macro F1-score of 0.420, surpassing the strongest fine-tuned baseline reported in the
dataset paper by a notable margin, yet leaving clear headroom. Our analysis highlights (i) a persistent tendency to over-predict the majority High-Adaptive class, (ii) a widening gap between accuracy and macro-F1 that reveals class-imbalance sensitivity, and (iii) the value of theory-aware and LLM-based approaches for fine-grained defensive-function classification. We release all task materials and invite the community to continue work on this novel intersection of clinical psychology and NLP.
Read the full abstract ▾
All authors Hongbin Na, Zimu Wang, Zhaoming Chen, Yining Hua, Rena Gao, Kailai Yang, Ling Chen, Wei Wang, Shaoxiong Ji, John Torous, Sophia Ananiadou
Computation and Language
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber et al. (+1 more)
Machine-translated
benchmarks are widely used to assess the
multilingual capabilities of large
language models (LLMs), yet translation errors in these
benchmarks remain underexplored, raising concerns about the reliability and comparability of
multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on
benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated
benchmarks. We find that span agreement is non-trivial on naturally occurring
benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.
Read the full abstract ▾
All authors Klaudia-Doris Thellmann, Bernhard Stadler, Michael Färber, Jens Lehmann
Computation and Language
When Reasoning Hurts: Source-Aware Evaluation of Frontier LLMs for Clinical SOAP Note Generation
Faizan Faisal
Reasoning-enabled LLMs perform strongly on medical
reasoning benchmarks, but it remains unclear whether these gains transfer to structured clinical documentation; we investigate this question using SOAP note generation from clinical dialogue in a source-aware
benchmark spanning OMI Health, ACI-Bench, and PriMock57. We evaluate GPT-5.4, DeepSeek-V4-Flash, and Gemma-4-E4B in a controlled 2x2 design that independently toggles provider-native
reasoning and same-source
retrieval-augmented generation (RAG). Outputs are assessed using seven automatic metrics alongside two reference-aware LLM judges. Both evaluation approaches agree that a non-
reasoning GPT-5.4 configuration achieves the highest overall quality, while DeepSeek-V4-Flash performs best among
reasoning-enabled configurations. Enabling
reasoning significantly degrades GPT-5.4 performance across all three
datasets, whereas same-source RAG yields smaller, model-dependent improvements. Overall, the findings indicate that stronger
reasoning capability should not be assumed to improve fidelity-sensitive SOAP note generation without dedicated, task-specific evaluation.
Read the full abstract ▾
Computation and Language
DTO: a Differentiable Training Objective for Effective Counterfactual Story Rewriting
Amelia Girard and Massimo Piccardi
Counterfactual story rewriting is a natural
language processing task that requires updating an existing story to reflect a chosen alternative event, yet preserving all the unaffected storyline elements and overall coherence. While large
language models have recently made remarkable progress on this task, it still remains challenging since the required modifications are typically very small in size and highly localized. As a consequence, models trained in a conventional manner with the maximum-likelihood training objective tend to overlook these nuances. At the same time, more sophisticated training approaches based on reinforcement learning are notoriously slow and difficult to set up. For these reasons, our paper proposes a novel, differentiable training objective (DTO) that directly optimizes for the requisite counterfactual improvements. In our approach, a
transformer model is fine-tuned via end-to-end backpropagation against a fully differentiable loss function that jointly rewards (i) fidelity to the reference rewrite and (ii) semantic consistency with the source narrative. The empirical evaluation on the TimeTravel and ART
datasets shows that the proposed DTO approach has been able to surpass a maximum-likelihood baseline and a preference-based approach, and perform competitively against two contemporary large
language models in all evaluation metrics. These findings substantiate the effectiveness of task-specific differentiable objectives for nuanced, controlled text-generation tasks.
Read the full abstract ▾
Computation and Language
Towards a Universal Causal Reasoner
Qirun Dai, Xiao Liu, Jiawei Zhang et al. (+3 more)
Despite the importance of causal
reasoning, training LLMs to reason causally remains underexplored. Existing data efforts mostly focus on benchmarking LLMs on specific aspects of causality, making them less suitable for training generalizable causal reasoners. To address this, we propose UniCo, a data generation framework that both (1) addresses 18 causal query types across Pearl's Causal Ladder and (2) translates natively symbolic examples into code and natural
language forms to simulate real-world use cases where causal terms are not explicitly specified. To ensure data quality, UniCo grounds answers with exact causal inference and filters cases with
reasoning shortcuts. Upon supervised finetuning with 66.6K UniCo-generated instances, Qwen3-4B, Qwen3-8B and Olmo-3-7B-Instruct achieve an average of 22.9% improvements across all 18 in-distribution query types, and 8.1% over state-of-the-art causal data generation frameworks on 7 established causal
benchmarks outside the training distribution. More importantly, in real-world medical understanding, legal decision, and tabular
reasoning, UniCo-trained models consistently display more faithful
reasoning traces, outperforming the base models by an average of 20.2% in faithfulness metrics. These suggest that causality-centered training not only strengthens causal
reasoning, but also equips LLMs with a causal mindset in general
reasoning tasks.
Read the full abstract ▾
All authors Qirun Dai, Xiao Liu, Jiawei Zhang, Dylan Zhang, Hao Peng, Chenhao Tan
Computation and Language
Lngram: N-gram Conditional Memory in Latent Space
Yunao Zheng, Guoyang Xia, Xiaojie Wang et al. (+1 more)
Sequence modeling requires both compositional
reasoning and local static knowledge
retrieval, yet standard
Transformers handle both through dense computation. Engram partially decouples
retrieval from the backbone, but its
token-based keys remain tied to text tokenization and hash compression. We propose Lngram, a latent-space conditional
memory module that learns discrete symbols directly from hidden states and performs N-gram lookup over these symbols. This design removes the dependence on tokenizer IDs and naturally extends to non-text modalities. In our evaluated settings, Lngram outperforms
Transformer and Engram baselines, consistently reduces perplexity in long-context
language modeling, and effectively injects domain knowledge when added post hoc to pretrained models. Joint training with the backbone further surpasses full fine-tuning, while experiments on
vision-
language and
vision-
language-action tasks show overall gains. Analyses with LogitLens and CKA suggest that Lngram enables prediction-relevant information to emerge earlier, increasing effective depth with limited inference and
memory overhead. Code is available at https://github.com/zyaaa-ux/Lngram.
Read the full abstract ▾
All authors Yunao Zheng, Guoyang Xia, Xiaojie Wang, Lei Ren
Computation and Language · Cross-listed from cs.AI
Clustering as Reasoning: A $k$-Means Interpretation of Chain-of-Thought Graph Learning
Xuanting Xie, Zhaochen Guo, Bingheng Li et al. (+4 more)
Chain-of-Thought (CoT) prompting has shown promise in enhancing the
reasoning capabilities of large
language models (LLMs) on text-attributed graphs (TAGs). This work reframes CoT-based graph learning through the principle of clustering as
reasoning, offering a $k$-means interpretation of how iterative
reasoning operates over graph-structured data. We observe that existing graph CoT methods rely on disjoint architectures and fixed graph representations, limiting step-by-step semantic-topological interaction and interpretability. To overcome this limitation, we propose a unified framework named KCoT that integrates CoT
reasoning with graph representation learning. Our key theoretical result reveals a formal mathematical correspondence between a
Transformer block and the $k$-means
algorithm, allowing
reasoning to be interpreted as iterative assignment and update steps. Based on this insight, we introduce a Semantic Discriminating
Prompt that explicitly formulates these steps as structured CoT
reasoning, together with a structure-grounded
alignment strategy to fuse topological priors with evolving thought-conditioned representations. Experiments on standard
benchmarks demonstrate consistent improvements over state-of-the-art methods, validating clustering as a principled mechanism for CoT-based graph learning.
Read the full abstract ▾
All authors Xuanting Xie, Zhaochen Guo, Bingheng Li, Xingtong Yu, Zhifei Liao, Zhao Kang, Yuan Fang
Computation and Language
Repeated Sequences Reveal Gaps between Large Language Models and Natural Language
Kumiko Tanaka-Ishii
Evaluating whether large
language models (LLMs) capture the structure of natural
language beyond local fluency remains an open challenge. Existing evaluation methods, largely based on task performance or short-context behavior, provide limited insight into the long-range statistical organization of generated text. We propose a complementary evaluation framework based on repeated subsequences. By analyzing their distribution across scales and relating it to higher-order Rényi
entropies, we probe how texts reuse previously established structure under finite-length conditions. Experiments on human-written texts and length-matched GPT-generated texts show that, while power-law models can describe restricted ranges of block length, the observed
entropy growth is often equally or better characterized by logarithmic–power forms. Across
datasets, natural
language exhibits stable
entropy-growth patterns over accessible ranges, with consistent average behavior despite variability across individual texts. In contrast, GPT-generated texts show systematic and statistically significant shifts in estimated exponents with model size. These results demonstrate that repeated-subsequence
entropy provides a quantitative structural diagnostic that reveals systematic differences in long-range organization, distinguishing natural
language from state-of-the-art LLM outputs beyond surface-level fluency.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning
Chenyou Guo, Zongqi Liu, Yizhou Zhang et al. (+2 more)
While general-purpose Large
Language Models (LLMs) applied to Geology often hallucinate when
reasoning about subsurface structures and deep-time
evolution, current AI in Earth sciences predominantly targets surface remote sensing and GIS. To bridge this gap, we introduce Geo-Expert, a family of parameter-efficient geological LLMs fine-tuned on a custom-curated, high-quality instruction
dataset processed using our custom instruction synthesis pipeline. We investigate the impact of model scaling and architecture by fine-tuning three base models: Qwen3-8B, Qwen3-32B, and Gemma-3-27B, with Low-Rank Adaptation (LoRA) method. Our extensive evaluation on a novel domain-specific
benchmark, Geo-Eval, reveals that a domain-aligned 8B model can outperform open-weight 70B generalists and proprietary GPT-4o on specialized geological
reasoning, while a 32B variant approaches frontier
reasoning models. The optimized 8B model further offers a competitive cost-performance ratio for deployment. This work provides a reproducible recipe for democratizing scientific LLMs and establishes a baseline for geological artificial intelligence.
Read the full abstract ▾
All authors Chenyou Guo, Zongqi Liu, Yizhou Zhang, Zhaorui Jiang, Ze Liu
Computation and Language
Translators as Invisible Teachers of AI: Copyright, Translation Memory, and the Political Economy of Linguistic Data
Masaru Yamada
This paper examines how the labour of translators has been transformed into foundational data capital for the age of artificial intelligence (AI). Translation
memories (TM) and parallel corpora preserve a one-to-one correspondence between source and target text and therefore constitute extraordinarily valuable supervised training data for machine translation. The development of statistical machine translation (SMT), neural machine translation (NMT), the
Transformer architecture, and
multilingual large
language models (LLMs) cannot be disentangled from the accumulation of such translation data. And yet, translators' renditions have been bought as deliverables under contract, segmented as technical objects, and processed as "information analysis" data under copyright law – losing their moral, creative, and economic attribution to the translators who produced them. The paper develops two concepts to capture this process. The first is appropriation without consumption: a mode of use in which works are not read, viewed, or listened to, but only mined for statistical features – a use that is legitimated under Article 30-4 of the Japanese Copyright Act. The second is the invisible teacherisation of translators: the process by which translators, through the construction of translation
memories, post-editing, and quality assessment, have functioned as teachers of AI without recognition as such. Drawing on the data supply chain that runs from translators through
language service providers (LSPs) and platforms to model developers, on a comparative reading of Japanese, European, and United States legal frameworks, on the distinction between open and proprietary AI models, and on the premium status that human-generated data has acquired in the era of model collapse, the paper asks what translators are actually afraid of, and points toward concrete directions for redistributive design.
Read the full abstract ▾
Computation and Language · Cross-listed from stat.ME
Correcting test set contamination by spiking the training data
Johnny Tian-Zheng Wei, Jerry Li, Ameya Godbole et al. (+1 more)
The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our core proposal is to spike the training data by intentionally contaminating some test examples at known rates. The spiked examples can then be used to calibrate predictors of model memorization which enable principled statistical correction of inflated test scores. To evaluate different correction
estimators, we first present a simulation framework based on the Hubble models. Hubble models come in minimal pairs, where the perturbed model was deliberately contaminated with several test sets, while the standard model was not, serving as the counterfactual and correction target. We consider
estimators that use information from a memorization predictor, correctness predictor, or both. In simulation, we establish basic statistical intuitions and show that
estimators leveraging memorization and correctness information are better than naive estimation which makes no correction at all. We then instantiate several memorization and correctness predictors, and find that simple predictors such as Platt-scaled membership inference metrics provide good signal for correction. Finally, we examine the practical considerations of spiking. Simple memorization predictors need no more than 10 examples for calibration and often transfer from one
dataset to another. Taken together, spiking is a promising solution for test set contamination.
Read the full abstract ▾
All authors Johnny Tian-Zheng Wei, Jerry Li, Ameya Godbole, Robin Jia
Computation and Language · Cross-listed from cs.CR
RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry
Bo Lv, Zhiheng Xu, KeDong Xiu et al. (+4 more)
As Mixture-of-Experts (MoE) architectures are increasingly adopted for scaling Large
Language Models (LLMs), safety
auditing becomes necessary to verify whether these models produce or facilitate harmful behaviors during operation. However, existing content-based
auditing methods typically require access to user
prompts, model internals, or outputs, potentially exposing sensitive user information and creating a tension between LLM safety and user
privacy. On the other hand, we observe that, in MoE models, different inputs induce different sparse expert-routing patterns, which produce measurable footprints in low-level GPU execution telemetry. We refer to these hardware-observable signals induced by expert-routing decisions as expert routing telemetry; they are derived from GPU execution rather than from router logits or
token-level routing assignments. Inspired by this observation, we propose RouteScan, a non-intrusive
auditing framework for detecting harmful behaviors through such routing-induced GPU telemetry. Specifically, RouteScan utilizes the number of active GPU threads allocated to expert modules during the prefilling phase as a discriminative micro-architectural fingerprint, and builds a lightweight detection pipeline that isolates cross-domain invariant risk indicators for the precise identification of malicious
prompts. Comprehensive evaluations on four open-source MoE LLMs with distinct routing designs demonstrate that RouteScan achieves strong generalization, with an AUROC exceeding 0.91 on unseen harmful domains. Moreover,
privacy stress tests show that, although aggregated execution telemetry retains input-related attribute information, full
prompts and exact sensitive fields cannot be reliably recovered under the evaluated attacks.
Read the full abstract ▾
All authors Bo Lv, Zhiheng Xu, KeDong Xiu, Ruyi Ding, Tianhang Zheng, Zhibo Wang, Kui Ren
Computation and Language · Cross-listed from cs.CV
Lin Qiu, Hanqing Zeng, Yao Liu et al. (+3 more)
Reinforcement learning (RL) has emerged as an effective paradigm for improving the
reasoning capability of
vision-
language models (VLMs). However, RL-based optimization typically depends on costly high-quality annotations that are difficult to scale. Existing unsupervised alternatives may drift toward biased solutions due to weak visual grounding and the lack of reliable verification signals. We propose a self-evolving post-training framework, DUEL, where supervision emerges from
adversarial interactions between two policies initialized from the same pretrained VLM. A Challenger generates an image-grounded true claim together with a minimally perturbed hard-negative counterpart, while a Solver verifies both claims against the image, encouraging fine-grained visual discrimination under near-neighbor semantics. To stabilize optimization, we introduce a length-normalized log-likelihood reward that preserves informative optimization signals beyond binary outcome supervision and improves learning stability under sparse feedback. Experiments show that DUEL consistently improves visual
reasoning and robust discrimination without additional human annotations, external reward models, or image editing tools.
Read the full abstract ▾
All authors Lin Qiu, Hanqing Zeng, Yao Liu, Bingjun Sun, Guangdeng Liao, Ji Liu
Computation and Language
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
Jinze Li, Yixing Xu, Guanchen Li et al. (+7 more)
Speculative decoding (SPD) accelerates large
language model (LLM) inference by letting a smaller draft model propose multiple future
tokens that are verified in parallel by a larger target model. The dominant SPD paradigm treats the target model as the sole reliable teacher, accepting a draft
token only when it exactly matches the target prediction. This design implicitly assumes that the target is always the better choice at every position. In practice, this assumption does not hold. Although the draft is the weaker model overall, it is not uniformly inferior at the
token level. In a meaningful fraction of cases where draft and target disagree, the draft's choice is the one that leads to the correct final answer. Inspired by this, we introduce Collaborative Speculative Decoding (CoSpec), a generalization of SPD that no longer treats the target model as the sole
token-level authority. CoSpec trains an arbitration policy via reinforcement learning to decide whether to accept
tokens from the draft or target model, selectively accepting draft
tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance. By shifting the emphasis from imitation to collaboration, CoSpec suggests a new perspective on speculative decoding.
Read the full abstract ▾
All authors Jinze Li, Yixing Xu, Guanchen Li, Jinfeng Xu, Shuo Yang, Yang Zhang, Xuanwu Yin, Dong Li, Edith C.H. Ngai, Emad Barsoum