Computation and Language
Base Models Look Human To AI Detectors
Yixuan Even Xu, Ziqian Zhong, Aditi Raghunathan et al. (+2 more)
As AI-generated text enters the real-world at scale, institutions increasingly use commercial AI-text detectors, especially in education and academic-integrity workflows. We report a surprising empirical finding about such systems: when evaluated by GPTZero and Pangram, generated text from base models is often judged overwhelmingly human, whereas text generated by their instruction-tuned counterparts is not. Building on this observation, we propose Humanization by Iterative Paraphrasing (HIP), a detector-agnostic pipeline that minimally fine-tunes a base model into a paraphraser and applies it iteratively. Compared with the baselines we test, HIP yields a stronger trade-off between semantic preservation and detector evasion on commercial detectors. Across Llama-3 and Qwen-3 families, spanning model sizes from 0.6B to 70B, HIP consistently improves detector human-likeness. Our findings suggest that current detectors are tracking artifacts of instruction tuning and local context more than any invariant notion of machine-generated text. This, in turn, calls for detector designs that model these factors more explicitly.
Read the full abstract ▾
All authors Yixuan Even Xu, Ziqian Zhong, Aditi Raghunathan, Fei Fang, J. Zico Kolter
Computation and Language · Cross-listed from cs.AI
Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
Guanyu Cui, Zhewei Wei and Kun He
Many works make the eye-catching claim that
Transformers are Turing-complete. However, the literature often conflates two distinct settings: (i) a fixed
Transformer system setting, in which a fixed autoregressive
Transformer is coupled with a fixed context-management method to process inputs of different lengths step by step, and (ii) a scaling-family setting, in which a family of different models (with increasing context-window length or numerical precision) is used to handle different input lengths. Existing proofs of
Transformer Turing-completeness are frequently established in setting (ii), whereas real-world LLM deployment and the standard notion of Turing-completeness correspond more naturally to setting (i). In this paper, we first formalize the fixed-system setting, thereby providing a concrete characterization of how real-world LLMs operate. We then argue that results proved in the scaling-family setting provide theoretically meaningful resource bounds but do not establish Turing-completeness, thereby clarifying a common misinterpretation of existing results. Finally, we show that different context-management methods can yield sharply different computational power, and we advocate the position that context management is a central component that critically determines the computational power of real-world autoregressive
Transformers.
Read the full abstract ▾
Computation and Language
Drifting Objectives for Refining Discrete Diffusion Language Models
Daisuke Oba, Hiroki Furuta and Naoaki Okazaki
Discrete
diffusion language models (DDLMs) generate text by iteratively denoising categorical
token sequences, while recent drifting methods for continuous generators suggest that part of this sampling-time correction can instead be absorbed into training through an anti-symmetric fixed-point objective. We study how to transfer this principle to DDLMs, where the main challenge is the interface with discrete text: hard
token samples are non-differentiable, and categorical predictions do not directly provide continuous samples to drift. We formulate TokenDrift, a drifting objective that lifts categorical predictions to soft-
token features, applies anti-symmetric drifting in a frozen semantic space, and backpropagates the resulting stop-gradient feature target to DDLM logits. In controlled continual-training experiments with masked and uniform-state
diffusion backbones, TokenDrift improves fixed-NFE generation quality over matched continuation baselines, reducing Gen.-PPL at 4 NFEs by 89% on MDLM and 86% on DUO. These results suggest that drifting can provide a practical refinement objective for DDLMs.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization
Ahmed Heakl, Abdelrahman M. Shaker, Youssef Mohamed et al. (+4 more)
When a model produces a correct solution under reinforcement learning with verifiable rewards (RLVR), every
token receives the same reward signal regardless of whether it was a decisive
reasoning step or a grammatical filler. A natural fix is to condition the model on the correct answer as a teacher, identifying
tokens it would have generated differently had it known the answer. Prior work shows this either corrupts training by leaking the answer into the gradient, or produces a weak signal that cannot distinguish decisive steps from filler, since both look equally surprising relative to the model's baseline. We propose Contrastive Evidence Policy Optimization (CEPO), which asks a sharper question at every
token: not just "does the correct answer favor this
token?" but "does the correct answer favor it while the wrong answer disfavors it?" A
token satisfying both is a genuine
reasoning step; one satisfying neither is filler. The wrong-answer teacher is constructed from rejected rollouts already in the training batch, incurring no additional sampling cost. We prove CEPO inherits all structural safety guarantees of the prior state of the art while strictly sharpening credit at decisive
tokens, with the improvement vanishing exactly at filler positions. Empirically, CEPO achieves 43.43% and 60.56% average accuracy across five
multimodal mathematical
reasoning benchmarks at 2B and 4B scale, respectively, versus 41.17% and 57.43% for GRPO under identical training budgets. Distribution-matching self-
distillation methods (OPSD, SDPO) fall below the untrained baseline, empirically confirming the information leakage our theory predicts. Our code is available at https://github.com/ahmedheakl/CEPO.
Read the full abstract ▾
All authors Ahmed Heakl, Abdelrahman M. Shaker, Youssef Mohamed, Rania Elbadry, Omar Fetouh, Fahad Shahbaz Khan, Salman Khan
Computation and Language
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
Bing Wang, Shaotian Yan, Chen Shen et al. (+7 more)
Large
language models (LLMs) have achieved remarkable success in complex
reasoning tasks via long chain-of-thought (CoT), yet their immense computational overhead hinders real-world deployment. LLM
reasoning distillation addresses this by transferring
reasoning capabilities from formidable teacher models to compact student models. However, existing
distillation paradigms face a fundamental dilemma. Typical off-policy
distillation strictly utilizes teacher-generated golden trajectories, suffering from an exposure bias due to the mismatch between training distributions and student-generated inference contexts, which leads to error cascades in long CoT
reasoning. To address this, on-policy
distillation allows students to explore their own trajectories, but we demonstrate that it inherently introduces a reciprocal reversed exposure bias: the teacher model also struggles to provide positive guidance when conditioned on student-generated sub-optimal contexts. To resolve this dual exposure biases problem, we propose Monitoring Trajectories and Backtracking when it strays (MOTAB), a new LLM
reasoning distillation pipeline. Specifically, MOTAB dynamically monitors the student's on-policy generation against an adaptive safety boundary. When the generation strays and exceeds this threshold, MOTAB backtracks to the last safe state and leverages teacher intervention to correct the course. This approach inherently tolerates minor student errors to mitigate exposure bias, while preventing sub-optimal contexts to circumvent reversed exposure bias. Extensive experiments on the LIMO-v2 and AceReason
datasets demonstrate that MOTAB effectively alleviates the dual exposure biases, yielding a roughly 3% average performance improvement in
reasoning tasks.
Read the full abstract ▾
All authors Bing Wang, Shaotian Yan, Chen Shen, kaiyuan liu, Sinan Fan, Ximing Li, Rui Miao, Xiaosong Yuan, Zhanming Shen, Jieping Ye
Computation and Language
LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models
Redacted by arXiv
Group Relative Policy Optimization(GRPO) has become a cornerstone of modern reinforcement learning
alignment, prized for its efficacy in foregoing an explicit value-critic by leveraging reward normalization across sampled trajectory cohorts. However, the method's reliance on a monolithic statistical baseline, such as the group mean, collapses the relational
topology of the trajectory space into a single scalar, thereby erasing the fine-grained preference information essential for navigating complex, rank-sensitive reward landscapes. To address this issue, we introduce a novel framework, Lambda Policy Optimization (LambdaPO), that addresses this information-theoretic bottleneck by re-conceptualizing advantage estimation from a scalar value to a decomposed, pairwise preference structure. Specifically, the advantage for any given trajectory is formulated as the integrated sum of reward differentials against all peers in its cohort, where each pairwise comparison is dynamically attenuated by the policy's own probabilistic confidence in the established preference. To further mitigate the sparsity of binary outcome supervision, we augment the objective with a semantic density reward, derived from the precision-recall
alignment between generated
reasoning traces and ground-truth solutions. As a result, our method can mine more fine-grained optimization signals from a group of rollouts, guiding the LLM to a better optima. Experimental results across challenging math
reasoning and question-answering tasks demonstrates that LambdaPO improves performance compared to the baseline methods.
Read the full abstract ▾
Computation and Language
EmbGen: Teaching with Reassembled Corpora
Arun K Lenin, Kai Rouse, Andrea Nicastro et al. (+1 more)
Adapting small instruction-tuned models to specialized domains often relies on supervised fine-tuning (SFT) on curated instruction-response examples, which is expensive to collect at scale. Synthetic training examples generated by a teacher LLM from a domain corpus can reduce this cost, but existing pipelines can produce homogenized outputs and do not consistently capture cross-passage or cross-document dependencies. We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity-description pairs, reassembles them using semantic structure inferred from
embedding similarity, and then generates question-answer (QA) pairs via proximity, intra-cluster, and inter-cluster sampling with cluster-specialized system
prompts. We evaluate EmbGen against EntiGraph, InstructLab and Knowledge-Instruct on three
datasets of varied semantic heterogeneity, under fixed
token budgets (5 and 20 million
tokens). We use lexical overlap metrics, an LLM-as-a-judge rubric, and Binary Accuracy, a composed metric combining Factual Accuracy and Completeness for evaluation. EmbGen improves Binary Accuracy on the most heterogeneous
dataset by 12.5% at 5M and 88.9% at 20M
tokens budget, relative to the strongest baseline, while remaining competitive across other
datasets with lower heterogeneity.
Read the full abstract ▾
All authors Arun K Lenin, Kai Rouse, Andrea Nicastro, Anna Leontjeva
Computation and Language
Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning
Shuyu Wei, Jian Sun, Delai Qiu et al. (+6 more)
Entropy-based deep
reasoning has emerged as a promising direction for improving the
reasoning capabilities of Large
Language Models (LLMs), but existing methods often either increase response length indiscriminately or shorten responses at the cost of accuracy. To better balance this trade-off, we introduce Conditional
Entropy Shaping (CES), a framework that dynamically controls
token-level response
entropy, enabling LLMs to produce concise solutions on simple problems while encouraging deeper exploration on hard ones. Built on DAPO, CES uses
token-level
entropy as an uncertainty signal and applies a conditional bidirectional policy: it penalizes high-
entropy "forking point"
tokens on correct
reasoning paths to improve conciseness, and rewards them on incorrect paths to encourage exploration and error correction. We implement CES on DeepSeek-R1-Distill-7B and evaluate it on 12 mathematical
benchmarks. CES consistently improves average accuracy while reducing response length relative to DAPO, and supplementary experiments show similar trends on a smaller 1.5B backbone and on out-of-domain
benchmarks.
Read the full abstract ▾
All authors Shuyu Wei, Jian Sun, Delai Qiu, Yining Wang, Shengping Liu, Jiaen Liang, Ying Fu, Wei Huang, Jitao Sang
Computation and Language
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models
Yiyang Gu, Junwei Yang, Junyu Luo et al. (+15 more)
Large
language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice. Most
benchmarks are manually curated or domain-generic, limiting scalability and
alignment with real scientific use cases. In this paper, we propose a new framework named SciCustom to address the problem. It enables the custom construction of
benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. SciCustom first organizes scientific knowledge into ontology-grounded knowledge units with controlled granularity and trains a tagger to map large-scale data instances into this knowledge space. Given a custom requirement, relevant knowledge units are identified via voting-based multi-model consensus. These units enable relevance-aware
benchmark retrieval via binary search, followed by proxy subset selection and data-grounded
benchmark generation for efficient evaluation. Experiments in chemistry and healthcare demonstrate that SciCustom reveals fine-grained differences in LLM scientific capabilities that standard
benchmarks overlook, while requiring neither expert annotation nor synthetic question generation. This work provides a scalable and application-aware foundation for benchmarking scientific capabilities in LLMs. The source code is available at https://github.com/yjwtheonly/SciCustom.
Read the full abstract ▾
All authors Yiyang Gu, Junwei Yang, Junyu Luo, Ye Yuan, Bin Feng, Yingce Xia, Shufang Xie, Kaili Liu, Bohan Wu, Qi Shi, Haoran Li, Beier Xiao, Zhiping Xiao, Xiao Luo, Weizhi Zhang, Philip S. Yu, Zequn Liu, Ming Zhang
Computation and Language · Cross-listed from cs.MA
PAVE: A Cognitive Architecture for Legitimate Violation in Generative Agent Societies
Ahmad Yehia, Abduallah Mohamed, Kun Qian et al. (+4 more)
Generative
agents based on large
language models reproduce believable human behavior in cooperative settings, but how they should reason in situations where rule-breaking may be required, such as fire evacuation or authority-supervised emergency, remains poorly characterized. We propose PAVE (Perception, Assessment, Verdict, Emulation), a novel four-module cognitive architecture that addresses this gap end to end: (i) Perception extracts a structured context with explicit authority distance, peer behaviors, and severity-tagged situational cues; (ii) Assessment scores the context along five scalars including an explicit legitimacy judgment that checks necessity, proportionality, and absence of alternatives; (iii) Verdict decides to comply or violate under a hard legitimacy gate, with a per-
agent threshold elicited from the persona; (iv) Emulation enacts the verdict and scopes the violation to the rule the trigger justifies. We instantiate PAVE in Voville, a tile-based traffic environment forked from Smallville, and evaluate across three scenarios, four LLM backbones, and a focused ablation. PAVE
agents satisfy four properties simultaneously: legitimate violation (only when a trigger justifies it), authority deference (officer instructions override even high legitimacy), bounded scope (violations confined to the targeted rule), and recovery (baseline restored once the trigger ends). PAVE
agents make more structured and interpretable decisions than vanilla across all four properties, and human evaluators rate them as more plausible. Ablating the legitimacy gate reproduces vanilla-like failures. We release Voville, the PAVE
prompts and code, and the evaluation pipeline.
Read the full abstract ▾
All authors Ahmad Yehia, Abduallah Mohamed, Kun Qian, Tianyi Wang, Jiseop Byeon, Omar Hassanin, Christian Claudel
Computation and Language
IMLJD: A Computational Dataset for Indian Matrimonial Litigation Analysis
Joy Bose
We present IMLJD, an open
dataset of 3,613 Indian court judgments covering matrimonial disputes under IPC Section 498A, the Protection of Women from Domestic Violence Act, and CrPC Section 482. The
dataset covers the Supreme Court of India from 2000 to 2024 (1,474 cases) and the Karnataka High Court from 2018 to 2024 (2,139 cases), with structured outcome labels, metadata-derived indicators, and a knowledge graph. We find that 57.6% of quashing petitions succeed at the Supreme Court level compared to 39.7% at the Karnataka High Court level. On a matched 2018 to 2024 period, the SC quash rate is 59.3%, widening the differential to 19.6 percentage points and confirming the finding is robust to temporal adjustment. The
dataset, code, and knowledge graph are released openly at https://github.com/joyboseroy/imljd and https://huggingface.co/
datasets/joyboseroy/imljd.
Read the full abstract ▾
Computation and Language
Retrieval-Augmented Linguistic Calibration
Yi-Fan Yeh, Linwei Tao, Minjing Dong et al. (+4 more)
Linguistic cues such as "I believe" and "probably" offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic confidence expressions remains underexplored. In particular, co-occurring linguistic cues, contextual variation, and subjective audience interpretation pose unique challenges. We therefore model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing interpretation variability that scalar representations discard. Within this distributional framework, we introduce faithfulness as a complementary evaluation dimension and present Faithfulness Divergence (FD), an information-theoretic metric quantifying the surprise induced in audience beliefs upon truth revelation. Building on these foundations, we present
Retrieval-Augmented Linguistic Calibration (RALC), a lightweight post-hoc pipeline that propagates calibrated confidence signals back into natural
language via
retrieval-augmented rewriting. Across three QA
benchmarks and five LLM families, RALC improves in-domain faithfulness and calibration up to 66% and 58%, respectively, outperforming black-box and grey-box calibration baselines.
Read the full abstract ▾
All authors Yi-Fan Yeh, Linwei Tao, Minjing Dong, Tao Huang, Jialin Yu, Philip Torr, Chang Xu
Computation and Language
HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models
Emmy Liu, Varun Gangal, Michael Yu et al. (+4 more)
Hallucination remains a central failure mode of large
language models, but existing
benchmarks operationalize it inconsistently across summarization, question answering,
retrieval-augmented generation, and agentic interaction. This fragmentation makes it unclear whether a mitigation that works in one setting reduces
hallucinations across contexts. Current
benchmarks either require human annotation and fixed references that may be memorized, or rely on observations in settings that are difficult to reproduce. To study root causes, we introduce HalluWorld, an extensible
benchmark grounded in an explicit reference-world formulation: a model hallucinates when it produces an observable claim that is false with respect to this world. Building on this view, we construct synthetic and semi-synthetic environments in which the reference world is fully specified, the model's view is controlled, and
hallucination labels are generated automatically. HalluWorld spans gridworlds, chess, and realistic terminal tasks, enabling controlled variation of world complexity, observability, temporal change, and source-conflict policy, and disentangling
hallucinations into fine-grained error categories. We evaluate frontier and open-weight
language models across these settings and find consistent patterns: perceptual
hallucination on directly observed information is near-solved for frontier models, while multi-step state tracking and causal forward simulation remain difficult and are not generally solved by extended thinking. In the terminal setting, models also struggle with when to abstain. The uneven profile of failures across probe types and domains suggests that
hallucinations arise from distinct failure modes rather than a single capability. Our results suggest that controlled reference worlds offer a scalable and reproducible path toward measuring and reducing
hallucinations in modern
language models.
Read the full abstract ▾
All authors Emmy Liu, Varun Gangal, Michael Yu, Zhuofu Tao, Karan Singh, Sachin Kumar, Steven Y. Feng
Computation and Language · Cross-listed from cs.MA
STAR-PólyaMath: Multi-Agent Reasoning under Persistent Meta-Strategic Supervision
Jiaao Wu, Xian Zhang, Hanzhang Liu et al. (+3 more)
Frontier AI models and multi-
agent systems have led to significant improvements in mathematical
reasoning. However, for problems requiring extended, long-horizon
reasoning, existing systems continue to suffer from fundamental reliability issues:
hallucination accumulation,
memory fragmentation, and imbalanced
reasoning-tool trade-offs. In this paper, we introduce
STAR-PólyaMath, a multi-
agent framework that systematically addresses these challenges through meta-level supervision and structured Reasoner-Verifier interaction.
STAR-PólyaMath is structured as an orchestrated state machine with nested challenge-step-replan loops, governed by a
reasoning-free Python orchestrator that separates control from inference and bounds error propagation through trace-back and re-planning. Our key innovation is a persistent Meta-Strategist that maintains cross-attempt
memory and exercises meta-level control by issuing high-level strategic guidance or mandatory directives, so the system can escape unproductive loops rather than stagnate or over-rely on tools.
STAR-PólyaMath achieves state-of-the-art results on all eight top-tier competition
benchmarks: AIME 2025-2026, MathArena Apex Shortlist, MathArena Apex 2025, Putnam 2025, IMO 2025, HMMT February 2026, and USAMO 2026. It obtains perfect scores on AIMEs, Putnam, and HMMT, and shows its largest margin on Apex 2025, scoring 93.75% compared with 80.21% by the strongest baseline GPT-5.5. Ablation studies show that the gains arise from the framework's orchestration rather than from model-level diversity since removing key components or substituting in mixed backbones consistently weakens performance. Code is available at https://github.com/Julius-Woo/
STAR-PolyaMath.
Read the full abstract ▾
All authors Jiaao Wu, Xian Zhang, Hanzhang Liu, Sophia Zhang, Fan Yang, Yinpeng Dong
Computation and Language
A Multi-Agent Framework for Feature-Constrained Difficulty Control in Reading Comprehension Item Generation
Seonjeong Hwang, Jun Seo, Hyounghun Kim et al. (+1 more)
Recent studies in difficulty-controlled reading comprehension item generation have leveraged large
language models (LLMs) to produce items by adjusting difficulty-related features. However, existing methods typically rely on a single-
agent prompting approach, which often fails to consistently satisfy specified feature constraints, resulting in items that deviate from the target difficulty level. To address this limitation, we introduce MAFIG, a Multi-
agent Framework for Feature-constrained Item Generation, where multiple LLM
agents and feature-specific evaluators collaborate to generate and iteratively revise items based on intended constraints. Furthermore, to verify the efficacy of MAFIG in difficulty control, we propose a method for constructing a sequence of feature constraint sets that yield items with monotonically increasing difficulty. Experimental results demonstrate that MAFIG generates items that adhere to target constraints at a significantly higher rate than baselines, achieving robust difficulty control through the difficulty-calibrated constraint sequence.
Read the full abstract ▾
All authors Seonjeong Hwang, Jun Seo, Hyounghun Kim, Gary Geunbae Lee
Computation and Language
How Do Document Parsers Break? Auditing Structural Vulnerability in Document Intelligence
Yue Chen, Yihao Wang, Ziyi Tang et al. (+2 more)
Document Layout Analysis (DLA) pipelines provide structured page representations for
retrieval-augmented generation, long-document question answering, and related applications. Yet their robustness evaluation remains largely area-centric. We identify this Footprint Bias and propose ProSA, a lightweight output-level
auditing framework that decouples controlled probing, policy-driven targeting, and structure-aware diagnosis. ProSA combines Block-level Structural Loss Rate (B-SLR), granularity-aware exposure descriptors, and pathway attribution to analyze where structural identity is lost, at what exposure granularity failures emerge, and how failures propagate. Across MinerU and PP-StructureV3 on 1,000 pages, affected area weakly tracks perturbation-induced OCR instability ($R^2=0.384/0.110$), whereas B-SLR aligns much more closely with it ($R^2=0.727/0.916$). Exposure descriptors further separate occlusion- and
topology-dominant pathways, while matched-footprint structural probes cause much larger downstream QA/
retrieval drops than area-matched erasure. These results shift DLA robustness evaluation from footprint-based measurement toward structure-aware vulnerability
auditing.
Read the full abstract ▾
All authors Yue Chen, Yihao Wang, Ziyi Tang, Yongsen Zheng, Keze Wang
Computation and Language
Are Rationales Necessary and Sufficient? Tuning LLMs for Explainable Misinformation Detection
Bing Wang, Rui Miao, Ximing Li et al. (+6 more)
The rapid spread of misinformation on social media platforms has become a formidable challenge. To mitigate its proliferation, Misinformation Detection (MD) has emerged as a critical research topic. Traditional MD approaches based on small models typically perform binary classification through a black-box process. Recently, the rise of Large
Language Models (LLMs) has enabled explainable MD, where models generate rationales that explain their decisions, thereby enhancing transparency. Existing explainable MD methods primarily focus on crafting sophisticated
prompts to elicit rationales from off-the-shelf LLMs. In this work, we propose a pipeline to fine-tune a dedicated LLM specifically for explainable MD. Our pipeline begins by collecting large-scale fact-checked articles, and then uses multiple strong LLMs to produce veracity predictions and rationales. To ensure high-quality training data, we leverage a filtering strategy that selects only the correct instances for fine-tuning. While this pipeline is intuitive and prevalent, our experiments reveal that naive filtering based solely on label correctness is insufficient in practice and suffers from two critical limitations: (1) Coarse-grained labels cause insufficient rationales: Rationales filtered solely based on binary labels are insufficient to adequately support their decisions; (2) Over-verification behavior causes unnecessary rationales: Stronger LLMs tend to exhibit over-verification behavior, producing excessively verbose and unnecessary rationales. To address these issues, we introduce LONSREX, a novel data synthesis pipeline to Locate Necessary and Sufficient Rationales for Explainable MD. Specifically, we propose a metric that quantifies the contribution of each verification step to the final prediction, thereby evaluating its necessity and sufficiency. Experimental results demonstrate the effectiveness of LONSREX.
Read the full abstract ▾
All authors Bing Wang, Rui Miao, Ximing Li, Chen Shen, Shaotian Yan, Changchun Li, Kaiyuan Liu, Xiaosong Yuan, Jieping Ye
Computation and Language
Language models struggle with compartmentalization
Thomas Vincent Howe and David Wingate
In the training data used by large
language models (LLMs), the same latent concept is often presented in multiple distinct ways: the same facts appear in English and Swahili; many functions can be expressed in both Python and Haskell; we can express propositions in both formal and natural
language. We show that LLMs can exhibit compartmentalization, where they fail to identify and share statistical strength between distinct presentations of unified concepts. In the worst case, LLMs simply learn parallel internal representations of each presentation of the concept, saturating model capacity with redundancies and decreasing sample efficiency with the number of such presentations. We also demonstrate that synthetic parallel data can fail to improve this despite being easily learned itself. Under this framework, we find that, for small models, early
multilingual learning is nearly entirely compartmentalized. Finally, all interventions that we study exhibit a phase transition in which their effectiveness depends on the number of distinct presentations, suggesting that the
language modeling objective may only inconsistently unify representations.
Read the full abstract ▾
Computation and Language
OpenCompass: A Universal Evaluation Platform for Large Language Models
Maosong Cao, Kai Chen, Haodong Duan et al. (+27 more)
In recent years, the field of artificial intelligence has undergone a paradigm shift from task-specific small-scale models to general-purpose large
language models (LLMs). With the rapid iteration of LLMs, objective, quantitative, and comprehensive evaluation of their capabilities has become a critical link in advancing technological development. Currently, the mainstream static
benchmark dataset-based evaluation methods face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing workflows, making it difficult to efficiently conduct cross-domain and large-scale model evaluation. To address the aforementioned issues, this paper proposes and open-sources OpenCompass, a one-stop, scalable, and high-concurrency-supported general-purpose LLM evaluation platform. Adhering to the design philosophy of modularization and component decoupling, the platform boasts three core advantages: high compatibility, flexibility, and high concurrency. The core architecture of OpenCompass comprises five key components: the Configuration System, Task Partitioning Module, Execution and Scheduling Module, Task Execution Unit, and Result Visualization Module. Its workflow provides rule-based, LLM-as-a-Judge, and cascaded evaluators to adapt to the requirements of different task scenarios. Supporting mainstream
benchmark datasets across multiple domains, including knowledge,
reasoning, computation, science,
language, code, etc., the platform offers a unified and efficient LLM evaluation tool for both academia and industry, facilitating the accurate identification of strengths and weaknesses of LLMs as well as their subsequent optimization.
Read the full abstract ▾
All authors Maosong Cao, Kai Chen, Haodong Duan, Yixiao Fang, Zhiwei Fei, Tong Gao, Ge Jiaye, Mo Li, Hongwei Liu, Junnan Liu, Yuan Liu, Chengqi Lyu, Han Lyu, Ningsheng Ma, Zerun Ma, Yu Sun, Zhiyong Wu, Linchen Xiao, Zhuozhi Xiong, Jun Xu, Haochen Ye, Zhaohui Yu, Yike Yuan, Songyang Zhang, Yufeng Zhao, Fengzhe Zhou, Peiheng Zhou, Dongsheng Zhu, Lin Zhu, Jingming Zhuo
Computation and Language
Lost in Interpretation: The Plausibility-Faithfulness Trade-off in Cross-Lingual Explanations
Somnath Banerjee, Pranav Jha, Rima Hazra et al. (+1 more)
LLMs deployed multilingually are often audited via English explanations for non-English inputs. We evaluate extractive explanations ”where the model identifies input
token spans as evidence alongside a generated rationale” and uncover a systematic trade-off: English-pivot explanations can achieve higher span agreement with human rationales while their evidence becomes less causally grounded in the model's prediction, as measured by both comprehensiveness and sufficiency. Across 3 tasks, 5~
languages, and 2~
multilingual LLM families, we find that English explanations frequently produce fluent but loosely anchored rationales, with comprehensiveness degrading by up to 5.7x relative to native-
language conditions - even as task accuracy remains stable across settings. For socially nuanced classification, English pivots also fail to preserve pragmatic cues, reducing both faithfulness and span agreement. We recommend
auditing explanations in the input
language, reporting multi-faceted faithfulness metrics beyond lexical overlap, and treating English rationales as communication summaries rather than faithful decision traces.
Read the full abstract ▾
All authors Somnath Banerjee, Pranav Jha, Rima Hazra, Animesh Mukherjee
Computation and Language
DECOR: Auditing LLM Deception via Information Manipulation Theory
Linyue Cai, Samuel Yeh, Jwala Dhamala et al. (+2 more)
Large
language models can deceive by subtly manipulating truthful information – omitting key facts, shifting focus, or obscuring meaning – making such behavior difficult to detect. Existing black-box methods rely on coarse-grained judgments, offering limited interpretability and failing to pinpoint which facts were distorted and how. We introduce DECOR, a multi-
agent framework grounded in Information Manipulation Theory for fine-grained
auditing of strategic deception in LLM responses. DECOR decomposes input contexts into atomic informational units and scores each unit against the response across four dimensions of manipulation, producing interpretable manipulation profiles that are aggregated into a global deception index. We comprehensively evaluate DECOR on both single-turn and multi-turn deception detection
benchmarks spanning real-world domains, and show that DECOR achieves state-of-the-art performance on both, outperforming competitive baselines. The framework generalizes across 15 frontier models, and ablation studies confirm the contribution of each key design component. Our findings demonstrate that fine-grained, theory-grounded
auditing of information manipulation offers an effective and interpretable path for LLM deception detection.
Read the full abstract ▾
All authors Linyue Cai, Samuel Yeh, Jwala Dhamala, Rahul Gupta, Sharon Li
Computation and Language
FormalASR: End-to-End Spoken Chinese to Formal Text
Wanyi Ning, Yinshang Guo, Haitao Qian et al. (+4 more)
Automatic
speech recognition (ASR) systems are typically optimized for verbatim transcription, which preserves disfluencies, filler words, and informal spoken structures that are often unsuitable for downstream writing-oriented applications. A common workaround is a two-stage ASR+LLM pipeline for post-editing, but this design increases latency and
memory cost and is difficult to deploy on-device. We present FormalASR, two compact end-to-end models with 0.6B and 1.7B parameters that directly transcribe spoken Chinese into formal written text. To enable this setting, we build WenetSpeech-Formal and Speechio-Formal, two large-scale spoken-to-formal
datasets constructed by LLM-based rewriting and quality filtering. We then fine-tune Qwen3-ASR at both model scales with supervised fine-tuning. Experiments on WenetSpeech-Formal and Speechio-Formal show that FormalASR achieves up to 37.4% relative CER reduction over verbatim baselines, while also improving ROUGE-L and BERTScore. FormalASR requires no post-processing LLM at deployment time, providing a lightweight, on-device solution for spoken-to-formal transcription.
Read the full abstract ▾
All authors Wanyi Ning, Yinshang Guo, Haitao Qian, Jiyuan Cheng, Wei Zhou, Weiyuan Feng, Yufei Zhang
Computation and Language
AI Technologies in Language Access: Attitudes Towards AI and the Human Value of Language Access Managers
Miguel A. Jiménez-Crespo, Stephanie Rodriguez and Alejandro Jaume Losa
The rapid emergence of AI technologies is reshaping translation practices and theory across the board. This paper deals with the impact of AI in
language access. This area is characterized by the need to serve broad and diverse user populations, within a context where efficiency and access are shaped by legal mandates, ethical and commercial tensions, and safety concerns. This paper reports on the attitudes and perceptions of
language access managers towards the AI and the human value in the AI age. Methodologically, this paper presents an analysis of a subset of a broader study on
language access and technology, specifically a qualitative thematic analysis of ten semi-structured interviews with
language access managers in the USA working in healthcare, court, public service and local government contexts. The results indicate that
language access managers show conditional optimism towards the inevitable AI implementations, are strongly risk aware, and deeply committed to the human value and human oversight of AI implementations and output.
Read the full abstract ▾
Computation and Language
Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
Xiaoou Liu, Tiejin Chen, Dengjia Zhang et al. (+3 more)
Large
Language Models have achieved strong performance on
reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step
reasoning trace might fail remains difficult. Confidence estimation offers a diagnostic signal, yet existing methods are restricted to final answers or require internal model access. In this paper, we introduce Stepwise Confidence Attribution (SCA), a framework for closed-source LLMs that assigns step-level confidence based only on generated
reasoning traces. SCA applies the Information Bottleneck principle: steps aligning with consensus structures across correct solutions receive high confidence, while deviations are flagged as potentially erroneous. We propose two complementary methods: (1) NIBS, a non-parametric IB approach measuring consistency without graph structures, and (2) GIBS, a graph-based IB model that learns subgraphs through a differentiable mask to capture logical variability. Extensive experiments on mathematical
reasoning and multi-hop question answering show that SCA reliably identifies low-confidence steps strongly correlated with
reasoning errors. Moreover, using step-level confidence to guide self-correction improves the correction success rate by up to 13.5% over answer-level feedback.
Read the full abstract ▾
All authors Xiaoou Liu, Tiejin Chen, Dengjia Zhang, Yaqing Wang, Lu Cheng, Hua Wei
Computation and Language
Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG
Aditya R. Vaidya, Richard J. Antonello and Alexander G. Huth
Neuroscientists have recently turned to intracranial
brain recording methods, like electrocorticography (ECoG), for human experiments because of the fine spatial and temporal resolution that they afford. Models trained on this data, however, are fundamentally restricted by the patient populations that can receive the implants necessary for recording. We propose using non-invasive fMRI to bridge the gap in training data. Using spoken
language representations fine-tuned on fMRI, we build encoding models of ECoG. These representations showed improved prediction performance in ECoG, even though the temporal resolution of fMRI is two orders of magnitude worse. Prediction improved in frequency bands well beyond what is directly measured in fMRI. Next, to test the procedure's generalization ability, we fine-tuned models on fMRI responses that were temporally downsampled by a factor of 2. Despite the loss in resolution, these models were able to predict fMRI and ECoG responses at levels comparable to the original fMRI-tuned models. Finally, we showed that ECoG performance steadily scales with the amount of fMRI-tuning data. Our results show that "slow" data like fMRI can be a valuable resource for building better models of "fast"
brain data like ECoG. In the future, integrating across multiple recording methods may further improve performance in other applications, like decoding.
Read the full abstract ▾
Computation and Language
Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering
Tiejin Chen, Longchao Da, Xiaoou Liu et al. (+1 more)
Uncertainty Quantification (UQ) is widely regarded as the primary safeguard for deploying Large
Language Models (LLMs) in high-stakes domains. However, we argue that the field suffers from a category error: mainstream UQ methods for LLMs are just unsupervised clustering
algorithms. We demonstrate that most current approaches inherently quantify the internal consistency of the model's generations rather than their external correctness. Consequently, current methods are fundamentally blind to factual reality and fail to detect “confident
hallucinations,” where models exhibit high confidence in stable but incorrect answers. Therefore, the current UQ methods may create a deceptive sense of safety when deploying the models with uncertainty. In detail, we identify three critical pathologies resulting from this dependence on internal state: a hyperparameter sensitivity crisis that renders deployment unsafe, an internal evaluation cycle that conflates stability with truth, and a fundamental lack of ground truth that forces reliance on unstable proxy metrics to evaluate uncertainty. To resolve this impasse, we advocate for a paradigm shift to UQ and outline a roadmap for the research community to adopt better evaluation metrics and settings, implement mechanism changes for native uncertainty, and anchor verification in objective truth, ensuring that model confidence serves as a reliable proxy for reality.
Read the full abstract ▾
All authors Tiejin Chen, Longchao Da, Xiaoou Liu, Hua Wei