Computation and Language · Cross-listed from cs.RO
Open-Loop Planning, Closed-Loop Verification: Speculative Verification for VLA
Zihua Wang, Zhitao Lin, Ruibo Li et al. (+4 more)
Vision-
Language-Action (VLA) models, as large foundation models for
embodied control, have shown strong performance in manipulation tasks. However, their performance comes at high inference cost. To improve efficiency, recent methods adopt action chunking, which predicts a sequence of future actions for open-loop execution. Although effective for reducing computation, open-loop execution is sensitive to environmental changes and prone to error accumulation due to the lack of close-loop feedback. To address this limitation, we propose Speculative Verification for VLA Control (SV-VLA), a framework that combines efficient open-loop long-horizon planning with lightweight closed-loop online verification. Specifically, SV-VLA uses a heavy VLA as a low-frequency macro-planner to generate an action chunk together with a planning context, while a lightweight verifier continuously monitors execution based on the latest observations. Conditioned on both the current observation and the planning context, the verifier compares the planned action against a closed-loop reference action and triggers replanning only when necessary. Experiments demonstrate that SV-VLA combines the efficiency of chunked prediction with the robustness of closed-loop control, enabling efficient and reliable VLA-based control in dynamic environments. Code is available: https://github.com/edsad122/SV-VLA.
Read the full abstract ▾
All authors Zihua Wang, Zhitao Lin, Ruibo Li, Yu Zhang, Xu Yang, Siya Mi, Xiu-Shen Wei
Computation and Language
LogicPoison: Logical Attacks on Graph Retrieval-Augmented Generation
Yilin Xiao, Jin Chen, Qinggang Zhang et al. (+6 more)
Graph-based
Retrieval-Augmented Generation (GraphRAG) enhances the
reasoning capabilities of Large
Language Models (LLMs) by grounding their responses in structured knowledge graphs. Leveraging community detection and relation filtering techniques, GraphRAG systems demonstrate inherent resistance to traditional RAG attacks, such as text poisoning and
prompt injection. However, in this paper, we find that the security of GraphRAG systems fundamentally relies on the topological integrity of the underlying graph, which can be undermined by implicitly corrupting the logical connections, without altering surface-level text semantics. To exploit this vulnerability, we propose LogicPoison, a novel attack framework that targets logical
reasoning rather than injecting false contents. Specifically, LogicPoison employs a type-preserving entity swapping mechanism to perturb both global logic hubs for disrupting overall graph connectivity and query-specific
reasoning bridges for severing essential multi-hop inference paths. This approach effectively reroutes valid
reasoning into dead ends while maintaining surface-level textual plausibility. Comprehensive experiments across multiple
benchmarks demonstrate that LogicPoison successfully bypasses GraphRAG's defenses, significantly degrading performance and outperforming state-of-the-art baselines in both effectiveness and stealth. Our code is available at \textcolor{blue}https://github.com/Jord8061/logicPoison.
Read the full abstract ▾
All authors Yilin Xiao, Jin Chen, Qinggang Zhang, Yujing Zhang, Chuang Zhou, Longhao Yang, Lingfei Ren, Xin Yang, Xiao Huang
Computation and Language
How Annotation Trains Annotators: Competence Development in Social Influence Recognition
Maciej Markiewicz, Beata Bajcar, Wiktoria Mieleszczenko-Kowszewicz et al. (+8 more)
Human data annotation, especially when involving experts, is often treated as an objective reference. However, many annotation tasks are inherently subjective, and annotators' judgments may evolve over time. This study investigates changes in the quality of annotators' work from a competence perspective during a process of social influence recognition. The study involved 25 annotators from five different groups, including both experts and non-experts, who annotated a
dataset of 1,021 dialogues with 20 social influence techniques, along with intentions, reactions, and consequences. An initial subset of 150 texts was annotated twice - before and after the main annotation process - to enable comparison. To measure competence shifts, we combined qualitative and quantitative analyses of the annotated data, semi-structured interviews with annotators, self-assessment surveys, and Large
Language Model training and evaluation on the comparison
dataset. The results indicate a significant increase in annotators' self-perceived competence and confidence. Moreover, observed changes in data quality suggest that the annotation process may enhance annotator competence and that this effect is more pronounced in expert groups. The observed shifts in annotator competence have a visible impact on the performance of LLMs trained on their annotated data.
Read the full abstract ▾
All authors Maciej Markiewicz, Beata Bajcar, Wiktoria Mieleszczenko-Kowszewicz, Aleksander Szczęsny, Tomasz Adamczyk, Grzegorz Chodak, Karolina Ostrowska, Aleksandra Sawczuk, Jolanta Babiak, Jagoda Szklarczyk, Przemysław Kazienko
Computation and Language
A Multi-head-based architecture for effective morphological tagging in Russian with open dictionary
K. Skibin, M. Pozhidaev and S. Suschenko
The article proposes a new architecture based on Multi-head
attention to solve the problem of morphological tagging for the Russian
language. The preprocessing of the word vectors includes splitting the words into subtokens, followed by a trained procedure for aggregating the vectors of the subtokens into vectors for
tokens. This allows to support an open dictionary and analyze morphological features taking into account parts of words (prefixes, endings, etc.). The open dictionary allows in future to analyze words that are absent in the training
dataset. The performed computational experiment on the SinTagRus and Taiga
datasets shows that for some grammatical categories the proposed architecture gives accuracy 98-99% and above, which outperforms previously known results. For nine out of ten words, the architecture precisely predicts all grammatical categories and indicates when the categories must not be analyzed for the word. At the same time, the model based on the proposed architecture can be trained on consumer-level graphics accelerators, retains all the advantages of Multi-head
attention over RNNs (RNNs are not used in the proposed approach), does not require pretraining on large collections of unlabeled texts (like BERT), and shows higher processing speed than previous results.
Read the full abstract ▾
Computation and Language
An Inspectable LLM Council for Multi-Model Research Answer Aggregation
Shuai Wu, Xue Li, Yanna Feng et al. (+3 more)
Long-form research questions often require answers that combine complementary details, reconcile conflicting estimates, and retain qualifications across several large
language model (LLM) outputs. We present an inspectable LLM Council with two synthesis stages: an analyst organizes three independent candidate answers into a structured textual state, and a separately invoked writer composes the final response from the question and that state. We evaluate the workflow on 100 closed-book tasks from the Deep Research Accuracy, Completeness, and Objectivity (DRACO)
benchmark, retaining 600 final answers, 100 analyst states, and 720 whole-answer judgments. Under the primary rubric judge, Council scores 70.98, exceeding GPT and Gemini by 6.18 and 8.34 points; its 0.47-point difference from Claude has a 95% paired interval of $[-0.69,1.61]$. In the four-arm fusion comparison, Council scores 1.87 points above direct fusion (95% paired interval $[0.33,3.48]$) and covers 51.1% of positive criteria covered by exactly one candidate, compared with 43.7% for direct fusion. On an exploratory 68-task subset with matching returned model labels, direct fusion scores higher. The full comparison characterizes workflows that differ in structure and generation budget; estimating the analyst stage's effect requires a controlled component study. Traced cases show how selected estimates, provenance qualifications, and selection decisions pass through the saved state into final answers.
Read the full abstract ▾
All authors Shuai Wu, Xue Li, Yanna Feng, Yufang Li, Zhijun Wang, Ran Wang
Computation and Language · Cross-listed from cs.AI
Analysis of Optimality of Large Language Models on Planning Problems
Bernd Bohnet, Michael C. Mozer, Kevin Swersky et al. (+4 more)
Classic AI planning problems have been revisited in the Large
Language Model (LLM) era, with a focus of recent
benchmarks on success rates rather than plan efficiency. We examine the degree to which frontier models reason optimally versus relying on simple, heuristic, and possibly inefficient strategies. We focus on the Blocksworld domain involving towers of labeled blocks which have to be moved from an initial to a goal configuration via a set of primitive actions. We also study a formally equivalent task, the generalized Path-
Star ($P^*$) graph, in order to isolate true topological
reasoning from semantic priors. We systematically manipulate problem depth (the height of block towers), width (the number of towers), and compositionality (the number of goal blocks).
Reasoning-enhanced LLMs significantly outperform traditional satisficing planners (e.g., LAMA) in complex, multi-goal configurations. Although classical search
algorithms hit a wall as the search space expands, LLMs track theoretical optimality limits with near-perfect precision, even when domain-specific semantic hints are stripped away. To explain these surprising findings, we consider (and find evidence to support) two hypotheses: an active Algorithmic Simulation executed via
reasoning tokens and a
Geometric Memory that allows models to represent the $P^*$
topology as a navigable global geometry, effectively bypassing exponential combinatorial complexity.
Read the full abstract ▾
All authors Bernd Bohnet, Michael C. Mozer, Kevin Swersky, Wil Cunningham, Aaron Parisi, Kathleen Kenealy, Noah Fiedel
Computation and Language
BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition
Wazir Ali, Adeeb Noor, Sanaullah Mahar et al. (+2 more)
In this article, we present a gold-standard
benchmark dataset for Biomedical Urdu Named Entity Recognition (BioUNER), developed by crawling health-related articles from online Urdu news portals, medical prescriptions, and hospital health blogs and websites. After preprocessing, three native annotators with familiarity in the medical domain participated in the annotation process using the Doccano text annotation tool and annotated 153K
tokens. Following annotation, the proposed BioiUNER
dataset was evaluated both intrinsically and extrinsically. An inter-annotator agreement score of 0.78 was achieved, thereby validating the
dataset as gold-standard quality. To demonstrate the utility and benchmarking capability of the
dataset, we evaluated several machine learning and deep learning models, including Support Vector Machines (SVM), Long Short-Term
Memory networks (LSTM),
Multilingual BERT (mBERT), and XLM-RoBERTa. The gold-standard BioUNER
dataset serves as a reliable
benchmark and a valuable addition to Urdu
language processing resources.
Read the full abstract ▾
All authors Wazir Ali, Adeeb Noor, Sanaullah Mahar, Alia, Muhammad Mazhar Younas
Computation and Language
One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging
Baban Gain, Trilok Nath Singh and Asif Ekbal
Weight-space model merging combines independently fine-tuned checkpoints without access to the original training data. While merging has shown promise in multitask settings, its behavior in
multilingual generative systems remains underexplored. We systematically study weight-space merging for
multilingual machine translation by fully fine-tuning
language models on large-scale bilingual corpora and evaluating representative merging strategies across shared-source, shared-target, and bidirectional consolidation settings. Our experiments reveal a strong directional asymmetry. Merging is comparatively more effective when models share a target
language, improving
multilingual coverage over the base model, but it still fails to preserve the peak performance of
language-specific checkpoints. In contrast, when target
languages differ, performance degrades sharply, especially in shared-source and bidirectional settings. To explain this behavior, we analyze internal representations and find that fine-tuning does not create disjoint
language-specific sub-networks. Instead, independently fine-tuned models activate largely overlapping
neurons while reshaping upper-layer target-generation representations into incompatible geometries. These findings suggest that
multilingual merging failures arise from target-side
geometric misalignment within shared computational units, challenging the assumptions underlying standard weight-space merging for
multilingual translation. We make the code publicly available at https://github.com/babangain/mt-model-merging
Read the full abstract ▾
Computation and Language
LLM-based Atomic Propositions help weak extractors: Evaluation of a Propositioner for triplet extraction
Luc Pommeret, Thomas Gerald, Patrick Paroubek et al. (+3 more)
Knowledge Graph construction from natural
language requires extracting structured triplets from complex, information-dense sentences. In this paper, we investigate if the decomposition of text into atomic propositions (minimal, semantically autonomous units of information) can improve the triplet extraction. We introduce MPropositionneur-V2, a small
multilingual model covering six European
languages trained by knowledge
distillation from Qwen3-32B into a Qwen3-0.6B architecture, and we evaluate its integration into two extraction paradigms: entity-centric (GLiREL) and generative (Qwen3). Experiments on SMiLER, FewRel, DocRED and CaRB show that atomic propositions benefit weaker extractors (GLiREL, CoreNLP, 0.6B models), improving relation recall and, in the
multilingual setting, overall accuracy. For stronger LLMs, a fallback combination strategy recovers entity recall losses while preserving the gains in relation extraction. These results show that atomic propositions are an interpretable intermediate data structure that complements extractors without replacing them.
Read the full abstract ▾
All authors Luc Pommeret, Thomas Gerald, Patrick Paroubek, Sahar Ghannay, Christophe Servan, Sophie Rosset
Computation and Language
GRADE: Probing Knowledge Gaps in LLMs through Gradient Subspace Dynamics
Yujing Wang, Yuanbang Liang, Yukun Lai et al. (+2 more)
Detecting whether a model's internal knowledge is sufficient to correctly answer a given question is a fundamental challenge in deploying responsible LLMs. In addition to verbalising the confidence by LLM self-report, more recent methods explore the model internals, such as the hidden states of the response
tokens, to capture how much knowledge is activated. We argue that such activated knowledge may not align with what the query requires, e.g., capturing the stylistic and length-related features that are uninformative for answering the query. To fill the gap, we propose GRADE (GRAdient Dynamics for knowlEdge gap detection), which quantifies the knowledge gap via the cross-layer rank ratio of the gradient to that of the corresponding hidden state subspace. This is motivated by the property of gradients as
estimators of the required knowledge updates for a given target. We validate GRADE on six
benchmarks, demonstrating its effectiveness and robustness to input perturbations. In addition, we present a case study demonstrating how the gradient chain can generate interpretable explanations of knowledge gaps in long-form answers. Code is available at https://github.com/yjEugenia/llm-aware
Read the full abstract ▾
All authors Yujing Wang, Yuanbang Liang, Yukun Lai, Hainan Zhang, Hanqi Yan
Computation and Language
Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection
Chaoqun He, Yingfa Chen, Chaojun Xiao et al. (+2 more)
Large
reasoning models achieve strong performance on complex tasks through long chain-of-thought (CoT) trajectories, but directly transferring such
reasoning processes to smaller models remains challenging. A key difficulty is that not all teacher-generated
reasoning trajectories are suitable for student learning. Existing approaches typically rely on post-hoc filtering, selecting trajectories after full generation based on heuristic criteria. However, such methods cannot control the generation process itself and may still produce
reasoning paths that lie outside the student's learning capacity. To address this limitation, we propose Gen-SSD (Generation-time Self-Selection
Distillation), a student-in-the-loop framework that performs generation-time selection. Instead of passively consuming complete trajectories, the student evaluates candidate continuations during the teacher's sampling process, guiding the expansion of only learnable
reasoning paths and enabling early
pruning of unhelpful branches. Experiments on mathematical
reasoning benchmarks demonstrate that Gen-SSD consistently outperforms standard knowledge
distillation and recent baselines, with improvements of around 5.9 points over Standard KD and up to 4.7 points over other baselines. Further analysis shows that Gen-SSD produces more stable and learnable
reasoning trajectories, highlighting the importance of incorporating supervision during generation for effective
distillation.
Read the full abstract ▾
All authors Chaoqun He, Yingfa Chen, Chaojun Xiao, Xu Han, Lijie Wen
Computation and Language
Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
Tianze Xu, Yanzhao Zheng, Pengrui Lu et al. (+11 more)
Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large
Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to
Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained
token-level credit assignment. RTT introduces a
Token-Level Relevance Discriminator to predict which
tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and
token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the
token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample
Token Group Normalization, to accommodate this shift. Extensive experiments and
benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models.
Read the full abstract ▾
All authors Tianze Xu, Yanzhao Zheng, Pengrui Lu, Lyumanshan Ye, Yong Wu, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu
Computation and Language · Cross-listed from cs.CV
EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectors
Ryuhei Miyazato, Shunsuke Kitada and Kei Harada
Vision-
Language Models (VLMs) excel at
multimodal tasks, but they remain vulnerable to
hallucinations that are factually incorrect or ungrounded in the input image. Recent work suggests that
hallucination detection using internal representations is more efficient and accurate than approaches that rely solely on model outputs. However, existing internal-representation-based methods typically rely on a single representation or detector, limiting their ability to capture diverse
hallucination signals. In this paper, we propose EnsemHalDet, an ensemble-based
hallucination detection framework that leverages multiple internal representations of VLMs, including
attention outputs and hidden states. EnsemHalDet trains independent detectors for each representation and combines them through ensemble learning. Experimental results across multiple VQA
datasets and VLMs show that EnsemHalDet consistently outperforms prior methods and single-detector models in terms of AUC. These results demonstrate that ensembling diverse internal signals significantly improves robustness in
multimodal hallucination detection.
Read the full abstract ▾
Computation and Language
When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs
Linyu Li, Zhi Jin, Yichi Zhang et al. (+5 more)
Real-world
multimodal knowledge graphs (MMKGs) are dynamic, with new entities, relations, and
multimodal knowledge emerging over time. Existing continual knowledge graph
reasoning (CKGR) methods focus on structural triples and cannot fully exploit
multimodal signals from new entities. Existing
multimodal knowledge graph
reasoning (MMKGR) methods, however, usually assume static graphs and suffer catastrophic forgetting as graphs evolve. To address this gap, we present a systematic study of continual
multimodal knowledge graph
reasoning (CMMKGR). We construct several continual
multimodal knowledge graph
benchmarks from existing MMKG
datasets and propose MRCKG, a new CMMKGR model. Specifically, MRCKG employs a
multimodal-structural collaborative curriculum to schedule progressive learning based on the structural connectivity of new triples to the historical graph and their
multimodal compatibility. It also introduces a cross-modal knowledge preservation mechanism to mitigate forgetting through entity representation stability, relational semantic consistency, and modality anchoring. In addition, a
multimodal contrastive replay scheme with a two-stage optimization strategy reinforces learned knowledge via
multimodal importance sampling and representation
alignment. Experiments on multiple
datasets show that MRCKG preserves previously learned
multimodal knowledge while substantially improving the learning of new knowledge.
Read the full abstract ▾
All authors Linyu Li, Zhi Jin, Yichi Zhang, Dongming Jin, Yuanpeng He, Haoran Duan, Gadeng Luosang, Nyima Tashi
Computation and Language
Multiple-Debias: A Full-process Debiasing Method for Multilingual Pre-trained Language Models
Haoyu Liang, Peijian Zeng, Wentao Huang et al. (+2 more)
Multilingual Pre-trained
Language Models (MPLMs) have become essential tools for natural
language processing. However, they often exhibit biases related to sensitive attributes such as gender, race, and religion. In this paper, we introduce a comprehensive
multilingual debiasing method named Multiple-Debias to address these issues across multiple
languages. By incorporating
multilingual counterfactual data augmentation and
multilingual Self-Debias across both pre-processing and post-processing stages, alongside parameter-efficient fine-tuning, we significantly reduced biases in MPLMs across three sensitive attributes in four
languages. We also extended CrowS-Pairs to German, Spanish, Chinese, and Japanese, validating our full-process
multilingual debiasing method for gender, racial, and religious bias. Our experiments show that (i)
multilingual debiasing methods surpass monolingual approaches in effectively mitigating biases, and (ii) integrating debiasing information from different
languages notably improves the
fairness of MPLMs.
Read the full abstract ▾
All authors Haoyu Liang, Peijian Zeng, Wentao Huang, Aimin Yang, Dong Zhou
Computation and Language · Cross-listed from cs.SE
IndustryCode: A Benchmark for Industry Code Generation
Puyu Zeng, Zhaoxi Wang, Zhixu Duan et al. (+7 more)
Code generation and comprehension by Large
Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing
benchmarks are mainly confined to single domains and
languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios. To bridge this gap, we introduce IndustryCode, the first comprehensive
benchmark designed to span multiple industrial domains and programming
languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of fields, including finance, automation, aerospace, and remote sensing-and incorporates diverse programming
languages such as MATLAB, Python, C++, and Stata. In our evaluation, the top-performing model, Claude 4.5 Opus, achieved an overall accuracy of 68.1% on sub-problems and 42.5% main problems. The
benchmark dataset and automated evaluation code will be made publicly available upon acceptance.
Read the full abstract ▾
All authors Puyu Zeng, Zhaoxi Wang, Zhixu Duan, Liang Feng, Shaobo Wang, Cunxiang Wang, Jinghang Wang, Bing Zhao, Hu Wei, Linfeng Zhang
Computation and Language · Cross-listed from cs.LG
Generative Frontiers: Why Evaluation Matters for Diffusion Language Models
Patrick Pynadath, Jiaxin Shi and Ruqi Zhang
Diffusion language models have seen exciting recent progress, offering far more flexibility in generative trajectories than autoregressive models. This flexibility has motivated a growing body of research into new approaches to
diffusion language modeling, which typically begins at the scale of GPT-2 small (150 million parameters). However, these advances introduce new issues with evaluation methodology. In this technical note, we discuss the limitations of current methodology and propose principled augmentations to ensure reliable comparisons. We first discuss why OpenWebText has become the standard
benchmark, and why alternatives such as LM1B are inherently less meaningful. We then discuss the limitations of likelihood evaluations for
diffusion models, and explain why relying on generative perplexity alone as a metric can lead to uninformative results. To address this, we show that generative perplexity and
entropy are two components of the KL divergence to a reference distribution. This decomposition explains generative perplexity's sensitivity to
entropy, and naturally suggests generative frontiers as a principled method for evaluating model generative quality. We conclude with empirical observations on model quality at this scale. We include a blog post with interactive content to illustrate the argument at https://patrickpynadath1.github.io/blog/eval_methodology/.
Read the full abstract ▾
Computation and Language
Breakdowns in Conversational AI: Interactional Failures in Emotionally and Ethically Sensitive Contexts
Jiawen Deng, Wentao Zhang, Ziyun Jiao et al. (+1 more)
Conversational AI is increasingly deployed in emotionally charged and ethically sensitive interactions. Previous research has primarily concentrated on emotional
benchmarks or static safety checks, overlooking how
alignment unfolds in evolving conversation. We explore the research question: what breakdowns arise when conversational
agents confront emotionally and ethically sensitive behaviors, and how do these affect dialogue quality? To stress-test chatbot performance, we develop a persona-conditioned user simulator capable of engaging in multi-turn dialogue with psychological personas and staged emotional pacing. Our analysis reveals that mainstream models exhibit recurrent breakdowns that intensify as emotional trajectories escalate. We identify several common failure patterns, including affective misalignments, ethical guidance failures, and cross-dimensional trade-offs where empathy supersedes or undermines responsibility. We organize these patterns into a taxonomy and discuss the design implications, highlighting the necessity to maintain ethical coherence and affective sensitivity throughout dynamic interactions. The study offers the HCI community a new perspective on the diagnosis and improvement of conversational AI in value-sensitive and emotionally charged contexts.
Read the full abstract ▾
All authors Jiawen Deng, Wentao Zhang, Ziyun Jiao, Fuji Ren
Computation and Language
Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy
Yihong Dong, Jianha Xiao, Xue Jiang et al. (+7 more)
The formal
reasoning capabilities of LLMs are crucial for advancing automated software engineering. However, existing
benchmarks for LLMs lack systematic evaluation based on computation and complexity, leaving a critical gap in understanding their formal
reasoning capabilities. Therefore, it is still unknown whether SOTA LLMs can grasp the structured, hierarchical complexity of formal
languages as defined by Computation Theory. To address this, we introduce ChomskyBench, a
benchmark for systematically evaluating LLMs through the lens of Chomsky Hierarchy. Unlike prior work that uses vectorized classification for neural networks, ChomskyBench is the first to combine full Chomsky Hierarchy coverage, process-trace evaluation via natural
language, and deterministic symbolic verifiability. ChomskyBench is composed of a comprehensive suite of
language recognition and generation tasks designed to test capabilities at each level. Extensive experiments indicate a clear performance stratification that correlates with the hierarchy's levels of complexity. Our analysis reveals a direct relationship where increasing task difficulty substantially impacts both inference length and performance. Furthermore, we find that while larger models and advanced inference methods offer notable relative gains, they face severe efficiency barriers: achieving practical reliability would require prohibitive computational costs, revealing that current limitations stem from inefficiency rather than absolute capability bounds. A time complexity analysis further indicates that LLMs are significantly less efficient than traditional algorithmic programs for these formal tasks. These results delineate the practical limits of current LLMs, highlight the indispensability of traditional software tools, and provide insights to guide the development of future LLMs with more powerful formal
reasoning capabilities.
Read the full abstract ▾
All authors Yihong Dong, Jianha Xiao, Xue Jiang, Xuyuan Guo, Zhiyuan Fan, Jiaru Qian, Kechi Zhang, Jia Li, Zhi Jin, Ge Li
Computation and Language
Trivial Vocabulary Bans Improve LLM Reasoning More Than Deep Linguistic Constraints
Rodney Jehu-Appiah
A previous study reported that E-
Prime (English without the verb "to be") selectively altered
reasoning in
language models, with cross-model correlations suggesting a structural signature tied to which vocabulary was removed. I designed a replication with active controls to test the proposed mechanism: cognitive restructuring through specific vocabulary-cognition mappings. The experiment tested five conditions (unconstrained control, E-
Prime, No-Have, elaborated metacognitive
prompt, neutral filler-word ban) across six models and seven
reasoning tasks (N=15,600 trials, 11,919 after compliance filtering). Every prediction from the cognitive restructuring hypothesis was disconfirmed. All four treatments outperformed the control (83.0%), including both active controls predicted to show null effects. The neutral filler-word ban, banning words like "very" and "just" with no role in logical inference, produced the largest improvement (+6.7 pp), while E-
Prime produced the smallest (+3.7 pp). The four conditions ranked in perfect inverse order of theoretical depth. The cross-model correlation signature did not replicate (mean r=0.005). These results are consistent with a simpler mechanism: any constraint that forces a model off its default generation path acts as an output regularizer, improving
reasoning by disrupting fluent but shallow response patterns. The shallowest constraints work best because they impose monitoring load with minimal conceptual disruption. I present these findings as a case study in discovery through disconfirmation.
Read the full abstract ▾
Computation and Language
Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments
Divyanshu Kumar, Ishita Gupta, Nitin Aravind Birur et al. (+3 more)
How biased is a
language model? The answer depends on how you ask. A model that refuses to choose between castes for a leadership role will, in a fill-in-the-blank task, reliably associate upper castes with purity and lower castes with lack of hygiene. Single-task
benchmarks miss this because they capture only one slice of a model's bias profile. We introduce a hierarchical taxonomy covering 9 bias types, including under-studied axes like caste, linguistic, and geographic bias, operationalized through 7 evaluation tasks that span explicit decision-making to implicit association.
Auditing 7 commercial and open-weight LLMs with \textasciitilde45K
prompts, we find three systematic patterns. First, bias is task-dependent: models counter stereotypes on explicit probes but reproduce them on implicit ones, with Stereotype Score divergences up to 0.43 between task types for the same model and identity groups. Second, safety
alignment is asymmetric: models refuse to assign negative traits to marginalized groups, but freely associate positive traits with privileged ones. Third, under-studied bias axes show the strongest stereotyping across all models, suggesting
alignment effort tracks
benchmark coverage rather than harm severity. These results demonstrate that single-
benchmark audits systematically mischaracterize LLM bias and that current
alignment practices mask representational harm rather than mitigating it.
Read the full abstract ▾
All authors Divyanshu Kumar, Ishita Gupta, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi
Computation and Language
Too Polite to Disagree: Understanding Sycophancy Propagation in Multi-Agent Systems
Vira Kasprova, Amruta Parulekar, Abdulrahman AlRabah et al. (+5 more)
Large
language models (LLMs) often exhibit sycophancy: agreement with user stance even when it conflicts with the model's opinion. While prior work has mostly studied this in single-
agent settings, it remains underexplored in collaborative multi-
agent systems. We ask whether awareness of other
agents' sycophancy levels influences discussion outcomes. To investigate this, we run controlled experiments with six open-source LLMs, providing
agents with peer sycophancy rankings that estimate each peer's tendency toward sycophancy. These rankings are based on scores calculated using various static (pre-discussion) and dynamic (online) strategies. We find that providing sycophancy priors reduces the influence of sycophancy-prone peers, mitigates error-cascades, and improves final discussion accuracy by an absolute 10.5%. Thus, this is a lightweight and efficient way to reduce model sycophancy during discussions and subsequently improve downstream accuracy.
Read the full abstract ▾
All authors Vira Kasprova, Amruta Parulekar, Abdulrahman AlRabah, Krishna Agaram, Ritwik Garg, Sagar Jha, Nimet Beyza Bozdag, Dilek Hakkani-Tur
Computation and Language
SocioEval: A Template-Based Framework for Evaluating Socioeconomic Status Bias in Foundation Models
Divyanshu Kumar, Ishita Gupta, Nitin Aravind Birur et al. (+3 more)
As Large
Language Models (LLMs) increasingly power decision-making systems across critical domains, understanding and mitigating their biases becomes essential for responsible AI deployment. Although bias assessment frameworks have proliferated for attributes such as race and gender, socioeconomic status bias remains significantly underexplored despite its widespread implications in the real world. We introduce SocioEval, a template-based framework for systematically evaluating socioeconomic bias in foundation models through decision-making tasks. Our hierarchical framework encompasses 8 themes and 18 topics, generating 240
prompts across 6 class-pair combinations. We evaluated 13 frontier LLMs on 3,120 responses using a rigorous three-stage annotation protocol, revealing substantial variation in bias rates (0.42%-33.75%). Our findings demonstrate that bias manifests differently across themes lifestyle judgments show 10$\times$ higher bias than education-related decisions and that deployment safeguards effectively prevent explicit discrimination but show brittleness to domain-specific stereotypes. SocioEval provides a scalable, extensible foundation for
auditing class-based bias in
language models.
Read the full abstract ▾
All authors Divyanshu Kumar, Ishita Gupta, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi
Computation and Language
Revealing the Learning Dynamics of Long-Context Continual Pre-training
Yupu Liang, Shuang Chen, Guanwei Zhang et al. (+2 more)
Existing studies on Long-Context Continual Pre-training (LCCP) mainly focus on small-scale models and limited data regimes (tens of billions of
tokens). We argue that directly migrating these small-scale settings to industrial-grade models risks insufficient adaptation and premature training termination. Furthermore, current evaluation methods rely heavily on downstream
benchmarks (e.g., Needle-in-a-Haystack), which often fail to reflect the intrinsic convergence state and can lead to "deceptive saturation". In this paper, we present the first systematic investigation of LCCP learning dynamics using the industrial-grade Hunyuan-A13B (80B total parameters), tracking its
evolution across a 200B-
token training trajectory. Specifically, we propose a hierarchical framework to analyze LCCP dynamics across behavioral (supervised fine-tuning probing), probabilistic (perplexity), and mechanistic (
attention patterns) levels. Our findings reveal: (1) Necessity of Massive Data Scaling: Training regimes of dozens of billions of
tokens are insufficient for industrial-grade LLMs' LCCP (e.g., Hunyuan-A13B reaches saturation after training over 150B
tokens). (2) Deceptive Saturation vs. Intrinsic Saturation: Traditional NIAH scores report "fake saturation" early, while our PPL-based analysis reveals continuous intrinsic improvements and correlates more strongly with downstream performance. (3) Mechanistic Monitoring for Training Stability:
Retrieval heads act as efficient, low-resource training monitors, as their evolving
attention scores reliably track LCCP progress and exhibit high correlation with SFT results. This work provides a comprehensive monitoring framework, evaluation system, and mechanistic interpretation for the LCCP of industrial-grade LLM.
Read the full abstract ▾
All authors Yupu Liang, Shuang Chen, Guanwei Zhang, Shaolei Wang, Suncong Zheng
Computation and Language
Speaking of Language: Reflections on Metalanguage Research in NLP
Nathan Schneider and Antonios Anastasopoulos
This work aims to shine a spotlight on the topic of metalanguage. We first define metalanguage, link it to NLP and LLMs, and then discuss our two labs' metalanguage-centered efforts. Finally, we discuss four dimensions of metalanguage and metalinguistic tasks, offering a list of understudied future research directions.
Read the full abstract ▾
Computation and Language
Overcoming the "Impracticality" of RAG: Proposing a Real-World Benchmark and Multi-Dimensional Diagnostic Framework
Kenichirou Narita, Siqi Peng, Taku Fukui et al. (+3 more)
Performance evaluation of
Retrieval-Augmented Generation (RAG) systems within enterprise environments is governed by multi-dimensional and composite factors extending far beyond simple final accuracy checks. These factors include
reasoning complexity,
retrieval difficulty, the diverse structure of documents, and stringent requirements for operational explainability. Existing academic
benchmarks fail to systematically diagnose these interlocking challenges, resulting in a critical gap where models achieving high performance scores fail to meet the expected reliability in practical deployment. To bridge this discrepancy, this research proposes a multi-dimensional diagnostic framework by defining a four-axis difficulty taxonomy and integrating it into an enterprise RAG
benchmark to diagnose potential system weaknesses.
Read the full abstract ▾
All authors Kenichirou Narita, Siqi Peng, Taku Fukui, Moyuru Yamada, Satoshi Munakata, Satoru Takahashi
Computation and Language
Train Yourself as an LLM: Exploring Effects of AI Literacy on Persuasion via Role-playing LLM Training
Qihui Fan, Min Ge, Chenyan Jia et al. (+1 more)
As large
language models (LLMs) become increasingly persuasive, there is concern that people's opinions and decisions may be influenced across various contexts at scale. Prior mitigation (e.g., AI detectors and disclaimers) largely treats people as passive recipients of AI-generated information. To provide a more proactive intervention against persuasive AI, we introduce $\textbf{LLMimic}$, a role-play-based, interactive, gamified AI literacy tutorial, where participants assume the role of an LLM and progress through three key stages of the training pipeline (pretraining, SFT, and RLHF). We conducted a $2 \times 3$ between-subjects study ($N = 274$) where participants either (1) watched an AI history
video (control) or (2) interacted with LLMimic (treatment), and then engaged in one of three realistic AI persuasion scenarios: (a) charity donation persuasion, (b) malicious money solicitation, or (c) hotel recommendation. Our results show that LLMimic significantly improved participants' AI literacy ($p < .001$), reduced persuasion success across scenarios ($p < .05$), and enhanced truthfulness and social responsibility levels ($p<0.01$) in the hotel scenario. These findings suggest that LLMimic offers a scalable, human-centered approach to improving AI literacy and supporting more informed interactions with persuasive AI.
Read the full abstract ▾
All authors Qihui Fan, Min Ge, Chenyan Jia, Weiyan Shi
Computation and Language
Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge
Yiyang Shen, Lifu Tu and Weiran Wang
Reinforcement Learning (RL) substantially improves the
reasoning capabilities of
language models, but most existing RL fine-tuning approaches rely entirely on ground-truth verifiable rewards and thus labeled
datasets with verifiable answers. To overcome this, we propose a RL framework for
reasoning distillation that leverages continuous, LLM-based rewards. Our method employs an efficient mechanism that computes a continuous CoT reward (CCR) directly from a single-
token logit of a judge LLM, evaluating the student model's
reasoning trajectory. This formulation provides an effective and scalable online training signal that can be applied to large volumes of unlabeled data. We demonstrate that, when paired with a strong judge, simply using CCR achieves performance comparable to that of ground-truth or pseudo-label verifiable rewards, and even surpasses them as the amount of unlabeled data increases. Furthermore, we find that combining them in a semi-supervised setup is highly synergistic: verifiable rewards help stabilize the CCR, while CCR improves the generalizability of verifiable rewards to related tasks. We also provide a comprehensive empirical comparison of various reward sources across multiple model architectures and
dataset sizes. Our results show that this semi-supervised approach consistently enhances mathematical
reasoning, yielding an absolute improvement of 5-10% on multiple
reasoning tasks.
Read the full abstract ▾
Computation and Language
An Empirical Study of Many-Shot In-Context Learning for Machine Translation of Low-Resource Languages
Yinhan Lu, Gaganpreet Jhajj, Chen Zhang et al. (+2 more)
In-context learning (ICL) allows large
language models (LLMs) to adapt to new tasks from a few examples, making it promising for
languages underrepresented in pre-training. Recent work on many-shot ICL suggests that modern LLMs can further benefit from larger ICL examples enabled by their long context windows. However, such gains depend on careful example selection, and the inference cost can be prohibitive for low-resource
language communities. In this paper, we present an empirical study of many-shot ICL for machine translation from English into ten truly low-resource
languages recently added to FLORES+. We analyze the effects of retrieving more informative examples, using out-of-domain data, and ordering examples by length. Our findings show that many-shot ICL becomes more effective as the number of examples increases. More importantly, we show that BM25-based
retrieval substantially improves data efficiency: 50 retrieved examples roughly match 250 many-shot examples, while 250 retrieved examples perform similarly to 1,000 many-shot examples. We further show that ICL provides additional gains on top of fine-tuning.
Read the full abstract ▾
All authors Yinhan Lu, Gaganpreet Jhajj, Chen Zhang, Anietie Andy, David Ifeoluwa Adelani