Computation and Language · Cross-listed from cs.AI
When Reasoning Meets Its Laws
Junyu Zhang, Yifan Sun, Tianang Leng et al. (+4 more)
Despite the superior performance of Large
Reasoning Models (LRMs), their
reasoning behaviors are often counterintuitive, leading to suboptimal
reasoning capabilities. To theoretically formalize the desired
reasoning behaviors, this paper presents the Laws of
Reasoning (LoRe), a unified framework that characterizes intrinsic
reasoning patterns in LRMs. We first propose compute law with the hypothesis that the
reasoning compute should scale linearly with question complexity. Beyond compute, we extend LoRe with a supplementary accuracy law. Since the question complexity is difficult to quantify in practice, we examine these hypotheses by two properties of the laws, monotonicity and compositionality. We therefore introduce LoRe-Bench, a
benchmark that systematically measures these two tractable properties for large
reasoning models. Evaluation shows that most
reasoning models exhibit reasonable monotonicity but lack compositionality. In response, we develop an effective finetuning approach that enforces compute-law compositionality. Extensive empirical studies demonstrate that better compliance with compute laws yields consistently improved
reasoning performance on multiple
benchmarks, and uncovers synergistic effects across properties and laws. Project page: https://lore-project.github.io/
Read the full abstract ▾
All authors Junyu Zhang, Yifan Sun, Tianang Leng, Jingyan Shen, Liu Ziyin, Paul Pu Liang, Huan Zhang
Computation and Language
ShareChat: A Dataset of Chatbot Conversations in the Wild
Yueru Yan, Tuc Nguyen, Bo Su et al. (+2 more)
By evaluating Large
Language Models (LLMs) through uniform, text-only interfaces, current academic
benchmarks obscure how the unique designs and affordances of distinct commercial platforms shape real-world user behavior and system performance. To bridge this gap, we present ShareChat, the first large-scale corpus of 142,808 conversations (660,293 turns) collected from publicly shared URLs on ChatGPT, Perplexity, Grok, Gemini, and Claude. ShareChat preserves native platform affordances, including citations, thinking traces, and code artifacts, across 95
languages and the period from April 2023 to October 2025, complementing existing corpora that homogenize these interactions. To demonstrate the
dataset's evaluative utility, we present three case studies: a conversation completeness analysis assessing cross-platform differences in intent satisfaction, a source grounding analysis comparing citation strategies between search-augmented systems, and a temporal analysis revealing divergent response latency dynamics. Together, these analyses demonstrate research questions that are inaccessible to single-platform or stripped-affordance corpora. The
dataset is publicly available.
Read the full abstract ▾
All authors Yueru Yan, Tuc Nguyen, Bo Su, Melissa Lieffers, Thai Le
Computation and Language
DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
Janghoon Han, Heegyu Kim, Changho Lee et al. (+6 more)
Recent advances in large
language models have enabled deep research systems that generate expert-level reports through multi-step
reasoning and evidence-based synthesis. However, evaluating such reports remains challenging: report quality is multifaceted, making it difficult to determine what to assess and which criteria to use; LLM-based judges may miss errors that require domain expertise to identify; and because deep research relies on retrieved evidence, report-wide claim verification is also necessary. To address these issues, we propose DEER, a
benchmark for evaluating expert-level deep research reports. DEER systematizes evaluation criteria with an expert-developed taxonomy (7 dimensions, 25 subdimensions) operationalized as 101 fine-grained rubric items. We also provide task-specific Expert Evaluation Guidance to support LLM-based judging. In addition to rubric-based assessment, we propose a claim verification architecture that verifies both cited and uncited claims and quantifies evidence quality. Experiments show that current systems produce structurally plausible, evidence-citing reports, but still struggle to fully satisfy expert-level user requests and achieve logical completeness. Beyond performance comparisons, DEER makes system strengths and limitations interpretable and provides diagnostic signals for improvement.
Read the full abstract ▾
All authors Janghoon Han, Heegyu Kim, Changho Lee, Dahm Lee, Min Hyung Park, Hosung Song, Stanley Jungkyu Choi, Moontae Lee, Honglak Lee
Computation and Language
Bangla MedER: Multi-BERT Ensemble Approach for the Recognition of Bangla Medical Entity
Tanjim Taharat Aurpa, Farzana Akter, Md. Mehedi Hasan et al. (+3 more)
Medical Entity Recognition (MedER) is an essential NLP task for extracting meaningful entities from the medical corpus. Nowadays, MedER-based research outcomes can remarkably contribute to the development of automated systems in the medical sector, ultimately enhancing patient care and outcomes. While extensive research has been conducted on MedER in English, low-resource
languages like Bangla remain underexplored. Our work aims to bridge this gap. For Bangla medical entity recognition, this study first examined a number of
transformer models, including BERT, DistilBERT, ELECTRA, and RoBERTa. We also propose a novel Multi-BERT Ensemble approach that outperformed all baseline models with the highest accuracy of 89.58%. Notably, it provides an 11.80% accuracy improvement over the single-layer BERT model, demonstrating its effectiveness for this task. A major challenge in MedER for low-resource
languages is the lack of annotated
datasets. To address this issue, we developed a high-quality
dataset tailored for the Bangla MedER task. The
dataset was used to evaluate the effectiveness of our model through multiple performance metrics, demonstrating its robustness and applicability. Our findings highlight the potential of Multi-BERT Ensemble models in improving MedER for Bangla and set the foundation for further advancements in low-resource medical NLP.
Read the full abstract ▾
All authors Tanjim Taharat Aurpa, Farzana Akter, Md. Mehedi Hasan, Shakil Ahmed, Shifat Ara Rafiq, Fatema Khan
Computation and Language
AncientBench: Towards Comprehensive Evaluation on Excavated and Transmitted Chinese Corpora
Zhihan Zhou, Daqian Shi, Rui Song et al. (+3 more)
Comprehension of ancient texts plays an important role in archaeology and understanding of Chinese history and civilization. The rapid development of large
language models needs
benchmarks that can evaluate their comprehension of ancient characters. Existing Chinese
benchmarks are mostly targeted at modern Chinese and transmitted documents in ancient Chinese, but the part of excavated documents in ancient Chinese is not covered. To meet this need, we propose the AncientBench, which aims to evaluate the comprehension of ancient characters, especially in the scenario of excavated documents. The AncientBench is divided into four dimensions, which correspond to the four competencies of ancient character comprehension: glyph comprehension, pronunciation comprehension, meaning comprehension, and contextual comprehension. The
benchmark also contains ten tasks, including radical, phonetic radical, homophone, cloze, translation, and more, providing a comprehensive framework for evaluation. We convened archaeological researchers to conduct experimental evaluations, proposed an ancient model as baseline, and conducted extensive experiments on the currently best-performing large
language models. The experimental results reveal the great potential of large
language models in ancient textual scenarios as well as the gap with humans. Our research aims to promote the development and application of large
language models in the field of archaeology and ancient Chinese
language.
Read the full abstract ▾
All authors Zhihan Zhou, Daqian Shi, Rui Song, Lida Shi, Xiaolei Diao, Hao Xu
Computation and Language
Affect, Body, Cognition, Demographics, and Emotion: The ABCDE of Text Features for Computational Affective Science
Jan Philip Wahle, Krishnapriya Vishnubhotla, Bela Gipp et al. (+1 more)
Work in Computational Affective Science and Computational Social Science explores a wide variety of research questions about people,
emotions, behavior, and health. Such work often relies on
language data that is first labeled with relevant information, such as the use of
emotion words or the age of the speaker. Although many resources and
algorithms exist to enable this type of labeling, discovering, accessing, and using them remains a substantial impediment, particularly for practitioners outside of computer science. Here, we present the ABCDE
dataset (Affect, Body, Cognition, Demographics, and
Emotion), a large-scale collection of over 400 million text utterances drawn from social media, blogs, books, and AI-generated sources. The
dataset is annotated with a wide range of features relevant to computational affective and social science. ABCDE facilitates interdisciplinary research across numerous fields, including affective science, cognitive science, the digital humanities, sociology, political science, and computational linguistics.
Read the full abstract ▾
All authors Jan Philip Wahle, Krishnapriya Vishnubhotla, Bela Gipp, Saif M. Mohammad
Computation and Language
When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content
Lydia Nishimwe, Benoît Sagot and Rachel Bawden
User-generated content (UGC) is characterised by frequent use of non-standard
language, from spelling errors to expressive choices such as slang, character repetitions, and emojis. This makes evaluating UGC translation challenging: what counts as a "good" translation depends on the desired standardness level of the output. To explore this, we examine the human translation guidelines of four UGC
datasets, and derive a taxonomy of twelve non-standard phenomena and five translation actions (NORMALISE, COPY, TRANSFER, OMIT, CENSOR). Our analysis reveals notable differences in how UGC is treated, resulting in a spectrum of standardness in reference translations. We show that translation scores of large
language models are highly sensitive to
prompts with explicit UGC translation instructions, and that they improve when they align with the
dataset guidelines. We argue that fair evaluation requires both models and metrics to be aware of translation guidelines. Finally, we call for clear guidelines during
dataset creation and for the development of controllable, guideline-aware evaluation frameworks for UGC translation.
Read the full abstract ▾
Computation and Language
Toward Ethical AI Through Bayesian Uncertainty in Neural Question Answering
Riccardo Di Sipio
We explore
Bayesian reasoning as a means to quantify uncertainty in neural networks for question answering. Starting with a multilayer perceptron on the Iris
dataset, we show how posterior inference conveys confidence in predictions. We then extend this to
language models, applying
Bayesian inference first to a frozen head and finally to LoRA-adapted
transformers, evaluated on the CommonsenseQA
benchmark. Rather than aiming for state-of-the-art accuracy, we compare Laplace approximations against maximum a posteriori (MAP) estimates to highlight uncertainty calibration and selective prediction. This allows models to abstain when confidence is low. An “I don't know” response not only improves interpretability but also illustrates how
Bayesian methods can contribute to more responsible and ethical deployment of neural question-answering systems.
Read the full abstract ▾
Computation and Language
Peeking Into The Future For Contextual Biasing
Ramaneswaran Selvakumar, Cindy Tseng, Eesung Kim et al. (+2 more)
While end-to-end (E2E) automatic
speech recognition (ASR) models excel at general transcription, they struggle to recognize rare or unseen named entities (e.g., contact names, locations), which are critical for downstream applications like virtual assistants. In this paper, we propose a contextual biasing method for
attention based encoder decoder (AED) models using a list of candidate named entities. Instead of predicting only the next
token, we simultaneously predict multiple future
tokens, enabling the model to "peek into the future" and score potential candidate entities in the entity list. Moreover, our approach leverages the multi-
token prediction logits directly without requiring additional entity encoders or cross-
attention layers, significantly reducing architectural complexity. Experiments on Librispeech demonstrate that our approach achieves up to 50.34% relative improvement in named entity word error rate compared to the baseline AED model.
Read the full abstract ▾
All authors Ramaneswaran Selvakumar, Cindy Tseng, Eesung Kim, Vijendra Raj Apsingekar, Yun Tang
Computation and Language
Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems
Marco Gaido, Sara Papi, Mauro Cettolo et al. (+2 more)
Streaming
Speech-to-Text Translation (StreamST) requires producing translations concurrently with incoming
speech under strict latency constraints, demanding models that balance low latency with high translation quality. Despite rapid progress, evaluation remains fragmented across existing frameworks, which make different assumptions about how systems operate - for example, whether they process continuous
speech or short pre-segmented audio, and whether they support output revision (retranslation) or not (incremental). For instance, SimulEval, the most widely used framework, supports only incremental decoding, assumes short segmented inputs, and lacks a native support for system demonstrations. As a result, comparing systems fairly and consistently across studies remains challenging, with no unified solution for benchmarking and interactive demonstration. To address this gap, we introduce simulstream, the first open-source framework for StreamST evaluation and demonstration. It supports both incremental and re-translation decoding on long-form
speech, provides fine-grained logging for quality and latency evaluation, and includes an interactive web interface for real-time visualization and comparison.
Read the full abstract ▾
All authors Marco Gaido, Sara Papi, Mauro Cettolo, Matteo Negri, Luisa Bentivogli
Computation and Language
Linear Personality Probing and Steering in LLMs: A Big Five Study
Michel Frising and Daniel Balcells
Large
language models (LLMs) exhibit distinct and consistent personalities that greatly impact trust and engagement. While this means that personality frameworks would be highly valuable tools to characterize and control LLMs' behavior, current approaches remain either costly (post-training) or brittle (
prompt engineering). Probing and steering via linear directions has recently emerged as a cheap and efficient alternative. In this paper, we investigate whether linear directions aligned with the Big Five personality traits can be used for probing and steering model behavior. Using Llama 3.3 70B, we generate descriptions of 406 fictional characters and their Big Five trait scores. We then
prompt the model with these descriptions and questions from the Alpaca questionnaire, allowing us to sample hidden activations that vary along personality traits in known, quantifiable ways. Using linear regression, we learn a set of per-layer directions in activation space, and test their effectiveness for probing and steering model behavior. Our results suggest that linear directions aligned with trait-scores are effective probes for personality detection, while their steering capabilities strongly depend on context, producing reliable effects in forced-choice tasks but limited influence in open-ended generation or when additional context is present in the
prompt.
Read the full abstract ▾
Computation and Language
Confidence-Credibility Aware Weighted Ensembles of Small LLMs Outperform Large LLMs in Emotion Detection
Menna Elgabry and Ali Hamdi
This paper introduces a confidence-weighted, credibility-aware ensemble framework for text-based
emotion detection, inspired by Condorcet's Jury
Theorem (CJT). Unlike conventional ensembles that often rely on homogeneous architectures, our approach combines architecturally diverse small
transformer-based large
language models (sLLMs) - BERT, RoBERTa, DistilBERT, DeBERTa, and ELECTRA, each fully fine-tuned for
emotion classification. To preserve error diversity, we minimize parameter convergence while taking advantage of the unique biases of each model. A dual-weighted voting mechanism integrates both global credibility (validation F1 score) and local confidence (instance-level probability) to dynamically weight model contributions. Experiments on the DAIR-AI
dataset demonstrate that our credibility-confidence ensemble achieves a macro F1 score of 93.5 percent, surpassing state-of-the-art
benchmarks and significantly outperforming large-scale LLMs, including Falcon, Mistral, Qwen, and Phi, even after task-specific Low-Rank Adaptation (LoRA). With only 595M parameters in total, our small LLMs ensemble proves more parameter-efficient and robust than models up to 7B parameters, establishing that carefully designed ensembles of small, fine-tuned models can outperform much larger LLMs in specialized natural
language processing (NLP) tasks such as
emotion detection.
Read the full abstract ▾
Computation and Language · Cross-listed from q-bio.PE
Computational analysis reveals historical trajectory of East-Polynesian lunar calendars
Miguel Valério, Fabio Tamburini and Michele Corazza
We investigate a type of lunar calendar known as lists of the 'nights of the moon', found throughout East Polynesia, including Rapa Nui (Easter Island). Using computational methods, we analyzed the lexical and structural divergence of 49 calendric lists from all major archipelagos, each containing about 30 night names. Our results, presented as a rooted phylogenetic tree, show a clear split into two main groups: one including lists from Rapa Nui, Mangareva, and the Marquesas; the other comprising lists from New Zealand, Hawaii, the Cook Islands, the Austral Islands, Tahiti, and the Tuamotu. This pattern aligns with a recent alternative classification of East Polynesian
languages into 'Distal' (Marquesan, Mangarevan, Rapanui) and 'Proximal' (Maori, Hawaiian, Tahitian, etc.) subgroups. Since both
language and lunar calendars are symbolic systems passed down and changed within communities - and given the geographic isolation of many archipelagos - we interpret this correspondence as evidence that the early divergence of East Polynesian lunar calendars mirrors early population movements and
language splits in the region.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories
Lilin Wang, Lucas Ramalho, Alan Celestino et al. (+6 more)
Benchmarks like SWE-bench have standardized the evaluation of Large
Language Models (LLMs) on repository-level software engineering tasks. However, these efforts remain limited by manual curation, static
datasets, and a focus on Python-based bug fixes. We introduce SWE-Bench++, an automated framework that generates repository-level coding tasks from open-source GitHub projects. Unlike synthetic approaches, our pipeline harvests live pull requests to cover both bug fixes and feature requests across 11
languages. SWE-Bench++ turns GitHub pull requests (PRs) into reproducible, execution-based tasks via four stages: programmatic sourcing, environment synthesis, test oracle extraction, and quality assurance. A final hint-guided trajectory synthesis step converts instances that strong models fail on into training trajectories. Our initial
benchmark consists of 11,133 instances from 3,971 repositories across 11
languages. On a subset of 1,782 instances of this
benchmark, today's strongest models perform as follows: claude-sonnet-4.5 achieves 36.20% pass@10, gpt-5-2025-08-07 34.57%, gemini/gemini-2.5-pro 24.92%, and gpt-4o 16.89%. We further demonstrate the utility of our
dataset by showing that fine-tuning on SWE-Bench++ instances yields measurable improvements on the SWE-bench
Multilingual benchmark. SWE-Bench++ provides a scalable,
multilingual benchmark for evaluating and improving repository-level code generation.
Read the full abstract ▾
All authors Lilin Wang, Lucas Ramalho, Alan Celestino, Phuc Anthony Pham, Yu Liu, Umang Kumar Sinha, Andres Portillo, Onassis Osunwa, Gabriel Maduekwe
Computation and Language · Cross-listed from cs.CV
RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
Léo Butsanets, Charles Corbière, Julien Khlaut et al. (+2 more)
In this work, we introduce RadImageNet-VQA, a large-scale
dataset designed to advance radiologic visual question answering (VQA) on CT and MRI exams. Existing medical VQA
datasets are limited in scale, dominated by X-ray imaging or biomedical illustrations, and often prone to text-based shortcuts. RadImageNet-VQA is built from expert-curated annotations and provides 750K images paired with 7.5M question-answer samples. It covers three key tasks - abnormality detection, anatomy recognition, and pathology identification - spanning eight anatomical regions and 97 pathology categories, and supports open-ended, closed-ended, and multiple-choice questions. Extensive experiments show that state-of-the-art
vision-
language models still struggle with fine-grained pathology identification, particularly in open-ended settings and even after fine-tuning. Text-only analysis further reveals that model performance collapses to near-random without image inputs, confirming that RadImageNet-VQA is free from linguistic shortcuts. The full
dataset and
benchmark are publicly available at https://huggingface.co/
datasets/raidium/RadImageNet-VQA.
Read the full abstract ▾
All authors Léo Butsanets, Charles Corbière, Julien Khlaut, Pierre Manceron, Corentin Dancette
Computation and Language
Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?
Zabir Al Nazi, GM Shahariar, Md. Abrar Hossain et al. (+1 more)
Theory of Mind (ToM) - the ability to attribute beliefs and intents to others - is fundamental for social intelligence, yet
Vision-
Language Model (VLM) evaluations remain largely Western-centric. In this work, we introduce CulturalToM-VQA, a
benchmark of 5,095 visually situated ToM probes across diverse cultural contexts, rituals, and social norms. Constructed through a frontier proprietary MLLM, human-verified pipeline, the
dataset spans a taxonomy of six ToM tasks and four complexity levels. We
benchmark 10 VLMs (2023-2025) and observe a significant performance leap: while earlier models struggle, frontier models achieve high accuracy (>93%). However, significant limitations persist: models struggle with false belief
reasoning (19-83% accuracy) and show high regional variance (20-30% gaps). Crucially, we find that SOTA models exhibit social desirability bias - systematically favoring semantically positive answer choices over negative ones. Ablation experiments reveal that some frontier models rely heavily on parametric social priors, frequently defaulting to safety-aligned predictions. Furthermore, while Chain-of-Thought prompting aids older models, it yields minimal gains for newer ones. Overall, our work provides a testbed for cross-cultural social
reasoning, underscoring that despite architectural gains, achieving robust, visually grounded understanding remains an open challenge.
Read the full abstract ▾
All authors Zabir Al Nazi, GM Shahariar, Md. Abrar Hossain, Wei Peng
Computation and Language · Cross-listed from cs.SE
CIFE: Code Instruction-Following Evaluation
Sravani Gunnu, Shanmukha Guttula and Hima Patel
Large
Language Models (LLMs) are increasingly applied to real-world code generation, where functional correctness alone is insufficient for reliable deployment, developers also expect adherence to explicit requirements for robustness, formatting, and security. Existing
benchmarks primarily assess correctness through test-case execution, offering limited insight into how reliably models follow such constraints. We introduce a
benchmark of 1,000 Python tasks, each paired with an average of 7 developer-specified constraints spanning 13 categories. Constraints are curated through a four-stage human-LLM pipeline to ensure they are atomic, relevant, and objective. We evaluate 14 open- and closed-source models using complementary adherence metrics and propose the C2A Score, a composite measure that jointly captures correctness and constraint compliance. Results reveal a substantial gap between partial and strict satisfaction, while strong models achieve over 90% partial adherence, strict adherence remains between 39-66%. These findings highlight that trustworthy code generation requires not only correctness but also consistent adherence to developer intent.
Read the full abstract ▾
Computation and Language
UCoder: Unsupervised Code Generation by Internal Probing of Large Language Models
Jiajun Wu, Jian Yang, Wei Zhang et al. (+6 more)
Large
language models (LLMs) have demonstrated remarkable capabilities in code generation tasks. However, their effectiveness heavily relies on supervised training with extensive labeled (e.g., question-answering pairs) or unlabeled
datasets (e.g., code snippets), which are often expensive and difficult to obtain at scale. To address this limitation, this paper introduces a method IPC, an unsupervised framework that leverages Internal Probing of LLMs for Code generation without any external corpus, even unlabeled code snippets. We introduce the problem space probing, test understanding probing, solution space probing, and knowledge consolidation and reinforcement to probe the internal knowledge and confidence patterns existing in LLMs. Further, IPC identifies reliable code candidates through self-consistency mechanisms and representation-based quality estimation to train UCoder (coder with unsupervised learning). We validate the proposed approach across multiple code
benchmarks, demonstrating that unsupervised methods can achieve competitive performance compared to supervised approaches while significantly reducing the dependency on labeled data and computational resources. Analytic experiments reveal that internal model states contain rich signals about code quality and correctness, and that properly harnessing these signals enables effective unsupervised learning for code generation tasks, opening new directions for training code LLMs in resource-constrained scenarios.
Read the full abstract ▾
All authors Jiajun Wu, Jian Yang, Wei Zhang, Lin Jing, Yuqing Ma, Ensheng Shi, Yuchi Ma, Zhoujun Li, Xianglong Liu
Computation and Language · Cross-listed from cs.LG
AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
Tung-Ling Li, Yuhao Wu and Hongliang Liu
LLM-as-a-Judge systems supply the reward signal in modern RLHF and RLVR pipelines, but their binary verdict reduces to a single linear readout F_gap on one hidden state. We show this readout is shallow enough that short, low-perplexity
tokens flip the verdict from "No" to "Yes". These
tokens are sampled from the judge's own next-
token distribution at the response position, with no manual seed set and no gradient-based optimization. Our procedure, AdvJudge-Zero, reaches $>$90% ensemble false-positive rate on 22 of 24 (model,
dataset)
cells across six Qwen, Llama, and Gemma judges, versus 54-72% for the prior curated 10-
token benchmark, and the discovered surface transfers cross-format to a 70B scalar reward model. The same discovered pool enables a defense: a LoRA fine-tune stratified by a 9-class mechanism taxonomy hardens cross-family generalization where naive sampling on the same pool fails, with mechanism breadth rather than pool size carrying the gain. Under GRPO training, the hardened judge eliminates the reward-collapse failures (false-positive spikes and length collapse) we observe in the unhardened baseline on both MATH and GSM8K at ten seeds per condition. The discovered pool, the mechanism taxonomy, and per-
prompt flip records will be released under responsible disclosure.
Read the full abstract ▾
Computation and Language
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Zeyuan Allen-Zhu
Understanding architectural differences in
language models is challenging, especially at academic-scale pretraining (e.g., 1.3B parameters, 100B
tokens), where results are often dominated by noise and randomness. To overcome this, we introduce controlled synthetic pretraining tasks that isolate and evaluate core model capabilities. Within this framework, we discover CANON LAYERS: lightweight architectural components – named after the musical term "canon" – that promote horizontal information flow across neighboring
tokens. Canon layers compute weighted sums of nearby
token representations and integrate seamlessly into
Transformers, linear
attention, state-space models, or any sequence architecture. We present 12 key results. This includes how Canon layers enhance
reasoning depth (e.g., by $2\times$),
reasoning breadth, knowledge manipulation, etc. They lift weak architectures like NoPE to match RoPE, and linear
attention to rival SOTA linear models like Mamba2/GDN – validated both through synthetic tasks and real-world academic-scale pretraining. This synthetic playground offers an economical, principled path to isolate core model capabilities often obscured at academic scales. Equipped with infinite high-quality data, it may even PREDICT how future architectures will behave as training pipelines improve – e.g., through better data curation or RL-based post-training – unlocking deeper
reasoning and hierarchical inference.
Read the full abstract ▾
Computation and Language
Stakeholder Suite: A Unified AI Framework for Mapping Actors, Topics and Arguments in Public Debates
Mohamed Chenene, Jeanne Rouhier, Jean Daniélou et al. (+2 more)
Public debates surrounding infrastructure and energy projects involve complex networks of stakeholders, arguments, and evolving narratives. Understanding these dynamics is crucial for anticipating controversies and informing engagement strategies, yet existing tools in media intelligence largely rely on descriptive analytics with limited transparency. This paper presents Stakeholder Suite, a framework deployed in operational contexts for mapping actors, topics, and arguments within public debates. The system combines actor detection, topic modeling, argument extraction and stance classification in a unified pipeline. Tested on multiple energy infrastructure projects as a case study, the approach delivers fine-grained, source-grounded insights while remaining adaptable to diverse domains. The framework achieves strong
retrieval precision and stance accuracy, producing arguments judged relevant in 75% of pilot use cases. Beyond quantitative metrics, the tool has proven effective for operational use: helping project teams visualize networks of influence, identify emerging controversies, and support evidence-based decision-making.
Read the full abstract ▾
All authors Mohamed Chenene, Jeanne Rouhier, Jean Daniélou, Mihir Sarkar, Elena Cabrio
Computation and Language
Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models
Haomin Qi, Chengbo Huang, Zihan Dai et al. (+1 more)
We present a governance-aware hybrid fine-tuning framework for
multilingual, low-resource adaptation of large
language models. The core
algorithm combines gradient-aligned low-rank updates with structured orthogonal transformations through layer-wise mixing and introduces
unitary constraints in selected sub-layers to stabilize deep optimization. In tandem with lightweight, label-free data governance steps, including
language identification, near-duplicate removal, and quality filtering, the framework targets accuracy, calibration, and cross-
language parity under tight compute budgets. Across XNLI and FLORES, the hybrid approach delivers consistent gains over strong PEFT baselines while maintaining directional balance and improving probability calibration, as shown in Tables II and III. It is more resilient to lightweight orthographic variants, as shown in Table IV, and benefits additively from simple governance steps, as shown in Table V. Training footprint measurements indicate modest overhead and a favorable cost-quality frontier, as shown in Table VI and Figure 2. Together, these results show that hybrid and
unitary PEFT provide a stable and accessible path to resource-efficient
multilingual adaptation when paired with practical data governance.
Read the full abstract ▾
All authors Haomin Qi, Chengbo Huang, Zihan Dai, Yunkai Gao
Computation and Language · Cross-listed from cs.LG
Task Schema and Binding: A Double Dissociation Study of In-Context Learning
Chaeha Kim
We provide causal mechanistic validation that in-context learning (ICL) decomposes into two separable mechanisms: Task Schema (abstract task type recognition) and Binding (specific input-output associations). Through activation patching experiments across 9 models from 7
Transformer families plus Mamba (370M-13B parameters), we establish three key findings: 1. Double dissociation: Task Schema transfers at 100% via late MLP patching; Binding transfers at 62% via residual stream patching – proving separable mechanisms 2. Prior-Schema trade-off: Schema reliance inversely correlates with prior knowledge (Spearman rho = -0.596, p < 0.001, N=28 task-model pairs) 3. Architecture generality: The mechanism operates across all tested architectures including the non-
Transformer Mamba These findings offer a mechanistic account of the ICL puzzle that contrasts with prior views treating ICL as a monolithic mechanism (whether
retrieval-based, gradient descent-like, or purely
Bayesian). By establishing that Schema and Binding are neurally dissociable – not merely behavioral modes – we provide causal evidence for dual-process theories of ICL. Models rely on Task Schema when prior knowledge is absent, but prior knowledge interferes through attentional mis-routing (72.7% recency bias) rather than direct output competition (0%). This explains why arbitrary mappings succeed (zero prior leads to full Schema reliance) while factual overrides fail – and reveals that the true bottleneck is attentional, not output-level. Practical implications: Understanding these dual mechanisms enables more efficient
prompt engineering – reliable schema transfer reduces required demonstrations for novel tasks, while prior-aware design can mitigate the 38% binding failure rate in high-prior scenarios, improving ICL system reliability in production deployments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Large Language Models as Pokémon Battle Agents: Strategic Play and Content Generation
Daksh Jain, Aarya Jain, Ashutosh Desai et al. (+4 more)
Strategic decision-making in Pokémon battles presents a unique testbed for evaluating large
language models. Pokémon battles demand
reasoning about type matchups, statistical trade-offs, and risk assessment, skills that mirror human strategic thinking. This work examines whether Large
Language Models (LLMs) can serve as competent battle
agents, capable of both making tactically sound decisions and generating novel, balanced game content. We developed a turn-based Pokémon battle system where LLMs select moves based on battle state rather than pre-programmed logic. The framework captures essential Pokémon mechanics: type effectiveness multipliers, stat-based damage calculations, and multi-Pokémon team management. Through systematic evaluation across multiple model architectures we measured win rates, decision latency, type-
alignment accuracy, and
token efficiency. These results suggest LLMs can function as dynamic game opponents without domain-specific training, offering a practical alternative to reinforcement learning for turn-based strategic games. The dual capability of tactical
reasoning and content creation, positions LLMs as both players and designers, with implications for procedural generation and adaptive difficulty systems in interactive entertainment.
Read the full abstract ▾
All authors Daksh Jain, Aarya Jain, Ashutosh Desai, Avyakt Verma, Ishan Bhanuka, Pratik Narang, Dhruv Kumar
Computation and Language
Subjective Question Generation and Answer Evaluation using NLP
G. M. Refatul Islam, Safwan Shaheer, Yaseen Nur et al. (+1 more)
Natural
Language Processing (NLP) is one of the most revolutionary technologies today. It uses artificial intelligence to understand human text and spoken words. It is used for text summarization, grammar checking, sentiment analysis, and advanced chatbots and has many more potential use cases. Furthermore, it has also made its mark on the education sector. Much research and advancements have already been conducted on objective question generation; however, automated subjective question generation and answer evaluation are still in progress. An automated system to generate subjective questions and evaluate the answers can help teachers assess student work and enhance the student's learning experience by allowing them to self-assess their understanding after reading an article or a chapter of a book. This research aims to improve current NLP models or make a novel one for automated subjective question generation and answer evaluation from text input.
Read the full abstract ▾
All authors G. M. Refatul Islam, Safwan Shaheer, Yaseen Nur, Mohammad Rafid Hamid
Computation and Language · Cross-listed from cs.LG
Understanding Generalization in Role-Playing Models via Information Theory
Yongqi Li, Hao Lang, Fei Huang et al. (+2 more)
Role-playing models (RPMs) are widely used in real-world applications but underperform when deployed in the wild. This degradation can be attributed to distribution shifts, including user, character, and dialogue compositional shifts. Existing methods like LLM-as-a-judge fall short in providing a fine-grained diagnosis of how these shifts affect RPM generalization, and thus there lack formal frameworks to characterize RPM generalization behaviors. To bridge these gaps, we introduce an information-theoretic metric, named
reasoning-based effective mutual information difference (R-EMID), to measure RPM performance degradation in an interpretable way. We also derive an upper bound on R-EMID to predict the worst-case generalization performance of RPMs and theoretically reveal how various shifts contribute to the RPM performance degradation. Moreover, we propose a co-evolving reinforcement learning framework to adaptively model the connection among user, character, and dialogue context and thus enhance the estimation of dialogue response generation probability, which is critical for calculating R-EMID. Finally, we evaluate the generalization performance of various RPMs using R-EMID, finding that user shift poses the highest risk among all shifts and reinforcement learning is the most effective approach for enhancing RPM generalization.
Read the full abstract ▾
All authors Yongqi Li, Hao Lang, Fei Huang, Tieyun Qian, Yongbin Li
Computation and Language
AutoMetrics: Approximate Human Judgements with Automatically Generated Evaluators
Michael J. Ryan, Yanzhe Zhang, Amol Salunkhe et al. (+3 more)
Evaluating user-facing AI applications remains a central challenge, especially in open-ended domains such as travel planning, clinical note generation, or dialogue. The gold standard is user feedback (e.g., thumbs up/down) or behavioral signals (e.g., retention), but these are often scarce in prototypes and research projects, or too-slow to use for system optimization. We present AutoMetrics, a framework for synthesizing evaluation metrics under low-data constraints. AutoMetrics combines
retrieval from MetricBank, a collection of 48 metrics we curate, with automatically generated LLM-as-a-Judge criteria informed by lightweight human feedback. These metrics are composed via regression to maximize correlation with human signal. AutoMetrics takes you from expensive measures to interpretable automatic metrics. Across 5 diverse tasks, AutoMetrics improves Kendall correlation with human ratings by up to 33.4% over LLM-as-a-Judge while requiring fewer than 100 feedback points. We show that AutoMetrics can be used as a proxy reward to equal effect as a verifiable reward. We release the full AutoMetrics toolkit and MetricBank to accelerate adaptive evaluation of LLM applications.
Read the full abstract ▾
All authors Michael J. Ryan, Yanzhe Zhang, Amol Salunkhe, Yi Chu, Di Xu, Diyi Yang
Computation and Language
Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experience
Jiangjie Chen, Wenxiang Chen, Jiacheng Du et al. (+19 more)
Large
language models have recently made significant progress to generate rigorous mathematical proofs. In contrast, utilizing LLMs for
theorem proving in formal
languages (such as Lean) remains challenging and computationally expensive, particularly when addressing problems at the undergraduate level and beyond. In this work, we present Seed-Prover 1.5, a formal
theorem-proving model trained via large-scale agentic reinforcement learning, alongside an efficient test-time scaling (TTS) workflow. Through extensive interactions with Lean and other tools, the model continuously accumulates experience during the RL process, substantially enhancing the capability and efficiency of formal
theorem proving. Furthermore, leveraging recent advancements in natural
language proving, our TTS workflow efficiently bridges the gap between natural and formal
languages. Compared to state-of-the-art methods, Seed-Prover 1.5 achieves superior performance with a smaller compute budget. It solves 88% of PutnamBench (undergraduate-level), 80% of Fate-H (graduate-level), and 33% of Fate-X (PhD-level) problems. Notably, using our system, we solved 11 out of 12 problems from Putnam 2025 within 9 hours. Our findings suggest that scaling learning from experience, driven by high-quality formal feedback, holds immense potential for the future of formal mathematical
reasoning.
Read the full abstract ▾
All authors Jiangjie Chen, Wenxiang Chen, Jiacheng Du, Jinyi Hu, Zhicheng Jiang, Allan Jie, Xiaoran Jin, Xing Jin, Chenggang Li, Wenlei Shi, Zhihong Wang, Mingxuan Wang, Chenrui Wei, Shufa Wei, Huajian Xin, Fan Yang, Weihao Gao, Zheng Yuan, Tianyang Zhan, Zeyu Zheng, Tianxi Zhou, Thomas Hanwen Zhu
Computation and Language
Incorporating Error Level Noise Embedding for Improving LLM-Assisted Robustness in Persian Speech Recognition
Zahra Rahmani and Hossein Sameti
Automatic
Speech Recognition (ASR) systems suffer significant performance degradation in noisy environments, a challenge that is especially severe for low-resource
languages such as Persian. Even state-of-the-art models such as Whisper struggle to maintain accuracy under varying signal-to-noise ratios (SNRs). This study presents a robust noise-sensitive ASR error correction framework that combines multiple hypotheses and noise-aware modeling. Using noisy Persian
speech, we generate 5-best hypotheses from a modified Whisper-large decoder. Error Level Noise (ELN) is introduced as a representation that captures semantic- and
token-level disagreement across hypotheses, quantifying the linguistic distortions caused by noise. ELN thus provides a direct measure of noise-induced uncertainty, enabling the LLM to reason about the reliability of each hypothesis during correction. Three models are evaluated: (1) a base LLaMA-2-7B model without fine-tuning, (2) a fine-tuned variant trained on text-only hypotheses, and (3) a noise-conditioned model integrating ELN
embeddings at both sentence and word levels. Experimental results demonstrate that the ELN-conditioned model achieves substantial reductions in Word Error Rate (WER). Specifically, on the challenging Mixed Noise test set, the proposed Fine-tuned + ELN (Ours) model reduces the WER from a baseline of 31.10% (Raw Whisper) to 24.84%, significantly surpassing the Fine-tuned (No ELN) text-only baseline of 30.79%, whereas the original LLaMA-2-7B model increased the WER to 64.58%, demonstrating that it is unable to correct Persian errors on its own. This confirms the effectiveness of combining multiple hypotheses with noise-aware
embeddings for robust Persian ASR in noisy real-world scenarios.
Read the full abstract ▾
Computation and Language
Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding
Yuqing Li, Jiangnan Li, Zheng Lin et al. (+4 more)
Humans understand long and complex texts by relying on a holistic semantic representation of the content. This global view helps organize prior knowledge, interpret new information, and integrate evidence dispersed across a document, as revealed by the Mindscape-Aware Capability of humans in psychology. Current
Retrieval-Augmented Generation (RAG) systems lack such guidance and therefore struggle with long-context tasks. In this paper, we propose Mindscape-Aware RAG (MiA-RAG), the first framework to formulate mindscape-aware
retrieval and generation as a unified conditioning paradigm for LLM-based RAG. MiA-RAG builds a mindscape through hierarchical summarization and conditions both
retrieval and generation on this global semantic representation. This enables the retriever to form enriched query
embeddings and the generator to reason over retrieved evidence within a coherent global context. We evaluate MiA-RAG across diverse long-context and bilingual
benchmarks for evidence-based understanding and global sense-making. It consistently surpasses baselines, and further analysis shows that it aligns local details with a coherent global representation, enabling more human-like long-context
retrieval and
reasoning.
Read the full abstract ▾
All authors Yuqing Li, Jiangnan Li, Zheng Lin, Ziyan Zhou, Junjie Wu, Jie Zhou, Mo Yu