Computation and Language
Targum – A Multilingual New Testament Translation Corpus
Maciej Rapacz and Aleksander Smywiński-Pohl
Many European
languages possess rich biblical translation histories, yet existing corpora - in prioritizing linguistic breadth - often fail to capture this depth. To address this gap, we introduce a
multilingual corpus of 651 New Testament translations, of which 334 are unique, spanning five
languages with 2.4-5.0x more translations per
language than any prior corpus: English (194 unique versions from 390 total), French (41 from 78), Italian (17 from 33), Polish (29 from 48), and Spanish (53 from 102). Aggregated from 12 online biblical libraries and one preexisting corpus, each translation is annotated with metadata that maps the text to a standardized identifier for the work, its specific edition, and its year of revision. This canonicalization allows researchers to define "uniqueness" for their own needs: they can perform micro-level analyses on translation families, such as the KJV lineage, or conduct macro-level studies by deduplicating closely related texts. By providing the first
multilingual resource with sufficient depth per
language for flexible, multilevel analysis, the corpus fills a gap in the quantitative study of translation history.
Read the full abstract ▾
Computation and Language
AI-Assisted Scientific Assessment: A Case Study on Climate Change
Christian Buck, Levke Caesar, Michelle Chen Huebscher et al. (+13 more)
The emerging paradigm of AI co-scientists focuses on tasks characterized by repeatable verification, where
agents explore search spaces in 'guess and check' loops. This paradigm does not extend to problems where repeated evaluation is impossible and ground truth is established by the consensus synthesis of theory and existing evidence. We evaluate a Gemini-based AI environment designed to support collaborative scientific assessment, integrated into a standard scientific workflow. In collaboration with a diverse group of 13 scientists working in the field of
climate science, we tested the system on a complex topic: the stability of the Atlantic Meridional Overturning Circulation (AMOC). Our results show that AI can accelerate the scientific workflow. The group produced a comprehensive synthesis of 79 papers through 104 revision cycles in just over 46 person-hours. AI contribution was significant: most AI-generated content was retained in the report. AI also helped maintain logical consistency and presentation quality. However, expert additions were crucial to ensure its acceptability: less than half of the report was produced by AI. Furthermore, substantial oversight was required to expand and elevate the content to rigorous scientific standards.
Read the full abstract ▾
All authors Christian Buck, Levke Caesar, Michelle Chen Huebscher, Massimiliano Ciaramita, Erich M. Fischer, Zeke Hausfather, Özge Kart Tokmak, Reto Knutti, Markus Leippold, Joseph Ludescher, Katharine J. Mach, Sofia Palazzo Corner, Kasra Rafiezadeh Shahi, Johan Rockström, Joeri Rogelj, Boris Sakschewski
Computation and Language
Unsupervised Layer-Wise Dynamic Test Time Adaptation for LLMs
Longhuan Xu, Cunjian Chen and Feng Yin
Test-time adaptation (TTA) for large
language models (LLMs) updates model parameters at inference time using signals available at deployment. This paper focuses on a common yet under-explored regime: unsupervised, sample-specific TTA, where the model adapts independently for each
prompt using only the
prompt itself, without gold answers or external supervision. Although appealing, naive unsupervised TTA with a fixed, handcrafted learning rate can be unstable: updates may overfit to
prompt-specific statistics, drift from the desired answer distribution, and ultimately degrade generation quality. This failure mode is not surprising, as in this case TTA must adapt to a single
prompt within only a few gradient steps, unlike standard training that averages updates over large
datasets and long optimization horizons. Therefore, we propose layer-wise dynamic test-time adaptation, a framework which explicitly modulates TTA strength as a function of
prompt representation, LLM structure and adaptation step. In our setting, TTA updates only LoRA parameters, and a lightweight hypernetwork predicts per-layer, per-step learning-rate multipliers, enabling fine-grained control. Experiments across various
datasets and LLMs consistently show that our method substantially strengthens TTA by learning effective scaling patterns over adaptation steps and
transformer layer projections, improving stability while delivering better performance.
Read the full abstract ▾
Computation and Language
TraceMem: Weaving Narrative Memory Schemata from User Conversational Traces
Yiming Shu, Pei Liu, Tiange Zhang et al. (+3 more)
Sustaining long-term interactions remains a bottleneck for Large
Language Models (LLMs), as their limited context windows struggle to manage dialogue histories that extend over time. Existing
memory systems often treat interactions as disjointed snippets, failing to capture the underlying narrative coherence of the dialogue stream. We propose TraceMem, a cognitively-inspired framework that weaves structured, narrative
memory schemata from user conversational traces through a three-stage pipeline: (1) Short-term
Memory Processing, which employs a deductive topic
segmentation approach to demarcate episode boundaries and extract semantic representation; (2) Synaptic
Memory Consolidation, a process that summarizes episodes into episodic
memories before distilling them alongside semantics into user-specific traces; and (3) Systems
Memory Consolidation, which utilizes two-stage hierarchical clustering to organize these traces into coherent, time-evolving narrative threads under unifying themes. These threads are encapsulated into structured user
memory cards, forming narrative
memory schemata. For
memory utilization, we provide an agentic search mechanism to enhance
reasoning process. Evaluation on the LoCoMo
benchmark shows that TraceMem achieves state-of-the-art performance with a
brain-inspired architecture. Analysis shows that by constructing coherent narratives, it surpasses baselines in multi-hop and temporal
reasoning, underscoring its essential role in deep narrative comprehension. Additionally, we provide an open discussion on
memory systems, offering our perspectives and future outlook on the field. Our code implementation is available at: https://github.com/YimingShu-teay/TraceMem
Read the full abstract ▾
All authors Yiming Shu, Pei Liu, Tiange Zhang, Ruiyang Gao, Jun Ma, Chen Sun
Computation and Language
Maastricht University at AMIYA: Adapting LLMs for Dialectal Arabic using Fine-tuning and MBR Decoding
Abdulhai Alali and Abderrahmane Issam
Large
Language Models (LLMs) are becoming increasingly
multilingual, supporting hundreds of
languages, especially high resource ones. Unfortunately, Dialect variations are still underrepresented due to limited data and linguistic variation. In this work, we adapt a pre-trained LLM to improve dialectal performance. Specifically, we use Low Rank Adaptation (LoRA) fine-tuning on monolingual and English Dialect parallel data, adapter merging and dialect-aware MBR decoding to improve dialectal fidelity generation and translation. Experiments on Syrian, Moroccan, and Saudi Arabic show that merging and MBR improve dialectal fidelity while preserving semantic accuracy. This combination provides a compact and effective framework for robust dialectal Arabic generation.
Read the full abstract ▾
Computation and Language
Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation
Joseph Attieh, Timothee Mickus, Anne-Laure Ligozat et al. (+2 more)
Knowledge
distillation (KD) is a technique to compress a larger teacher system into a smaller student. In machine translation, KD is commonly evaluated through translation quality and inference efficiency, without jointly accounting for the environmental costs of producing and deploying the distilled system. We evaluate representative KD methods both on bespoke MT models and LLMs, by considering both translation quality and computational cost, using the Machine Learning Life Cycle Assessment tool, which accounts for costs throughout the KD model life cycle. Our key finding is that the deployment volume required to amortize KD is serving-dependent and can shift by several orders of magnitude under batching. We include actionable guidance for selecting, developing, and evaluating KD methods under quality and compute-induced constraints.
Read the full abstract ▾
All authors Joseph Attieh, Timothee Mickus, Anne-Laure Ligozat, Aurélie Névéol, Jörg Tiedemann
Computation and Language
MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answering
Sieun Hyeon, Jusang Oh, Sunghwan Steve Cho et al. (+1 more)
Recent advances in Large
Language Models (LLMs) have significantly improved table understanding tasks such as Table Question Answering (TableQA), yet challenges remain in ensuring reliability, scalability, and efficiency, especially in resource-constrained or
privacy-sensitive environments. In this paper, we introduce MATA, a multi-
agent TableQA framework that leverages multiple complementary
reasoning paths and a set of tools built with small
language models. MATA generates candidate answers through diverse
reasoning styles for a given table and question, then refines or selects the optimal answer with the help of these tools. Furthermore, it incorporates an
algorithm designed to minimize expensive LLM
agent calls, enhancing overall efficiency. MATA maintains strong performance with small, open-source models and adapts easily across various LLM types. Extensive experiments on two
benchmarks of varying difficulty with ten different LLMs demonstrate that MATA achieves state-of-the-art accuracy and highly efficient
reasoning while avoiding excessive LLM inference. Our results highlight that careful orchestration of multiple
reasoning pathways yields scalable and reliable TableQA. The code is available at https://github.com/AIDASLab/MATA.
Read the full abstract ▾
All authors Sieun Hyeon, Jusang Oh, Sunghwan Steve Cho, Jaeyoung Do
Computation and Language
MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluation
Nalin Srun, Parisa Rastin, Guénaël Cabanes et al. (+1 more)
We introduce MILE-RefHumEval, a reference-free framework for evaluating Large
Language Models (LLMs) without ground-truth annotations or evaluator coordination. It leverages an ensemble of independently prompted evaluators guided by a human-aligned schema, supporting both discrete and continuous scoring judgement. With task-specific
prompts from best candidate selection, summarization and image captioning to dialogue, MILE-RefHumEval provides flexible, interpretable, and scalable assessments. Experiments show it aligns closely with human judgments, outperforms prior methods, and reduces computational overhead, offering an efficient, robust, and human-aligned solution for real-world LLM evaluation.
Read the full abstract ▾
All authors Nalin Srun, Parisa Rastin, Guénaël Cabanes, Lydia Boudjeloud Assala
Computation and Language
AlignTune: Modular Toolkit for Post-Training Alignment of Large Language Models
R E Zera Marveen Lyngkhoi, Chirag Chawla, Pratinav Seth et al. (+5 more)
Post-training
alignment is central to deploying large
language models (LLMs), yet practical workflows remain split across backend-specific tools and ad-hoc glue code, making experiments hard to reproduce. We identify backend interference, reward fragmentation, and irreproducible pipelines as key obstacles in
alignment research. We introduce AlignTune, a modular toolkit exposing a unified interface for supervised fine-tuning (SFT) and RLHF-style optimization with interchangeable TRL and Unsloth backends. AlignTune standardizes configuration, provides an extensible reward layer (rule-based and learned), and integrates evaluation over standard
benchmarks and custom tasks. By isolating backend-specific logic behind a single factory boundary, AlignTune enables controlled comparisons and reproducible
alignment experiments.
Read the full abstract ▾
All authors R E Zera Marveen Lyngkhoi, Chirag Chawla, Pratinav Seth, Utsav Avaiya, Soham Bhattacharjee, Mykola Khandoga, Rui Yuan, Vinay Kumar Sankarapu
Computation and Language
Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning
Qiao Liang, Yuke Zhu, Chao Ge et al. (+4 more)
Tool-integrated
reasoning (TIR) enables LLM
agents to solve tasks through planning, tool use, and iterative revision, but outcome-only reinforcement learning in this setting suffers from sparse, delayed rewards and weak step-level credit assignment. In long-horizon TIR trajectories, an early irrecoverable mistake can determine success or failure, making it crucial to localize the first irrecoverable step and leverage it for fine-grained credit assignment. We propose Error-Localized Policy Optimization (ELPO), which localizes the first irrecoverable step via binary-search rollout trees under a fixed rollout budget, converts the resulting tree into stable learning signals through hierarchical advantage attribution, and applies error-localized adaptive clipping to strengthen corrective updates on the critical step and its suffix. Across TIR
benchmarks in math, science QA, and code execution, ELPO consistently outperforms strong Agentic RL baselines under comparable sampling budgets, with additional gains in Pass@K and Major@K scaling, rollout ranking quality, and tool-call efficiency. Our code will be publicly released soon.
Read the full abstract ▾
All authors Qiao Liang, Yuke Zhu, Chao Ge, Lei Yang, Ying Shen, Bo Zheng, Sheng Guo
Computation and Language
On the Optimal Reasoning Length for RL-Trained Language Models
Daisuke Nohara, Taishi Nakamura and Rio Yokota
Reinforcement learning substantially improves
reasoning in large
language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although length-control methods have been proposed, the length-accuracy relationship they induce remains unclear. We train policies with several length-control methods on multiple base models in a controlled setup and find that, across both mathematical
reasoning and code generation, accuracy is non-monotonic in output length, peaking at an intermediate value. Mode accuracy, however, continues to improve with length even in settings where sample accuracy plateaus or declines, indicating that the non-monotonic length-accuracy relationship is driven by dispersion around an increasingly correct center.
Read the full abstract ▾
Computation and Language
Context-Aware Counterfactual Data Augmentation for Gender Bias Mitigation in Language Models
Shweta Parihar, Liu Guangliang, Natalie Parde et al. (+1 more)
A challenge in mitigating social bias in fine-tuned
language models (LMs) is the potential reduction in
language modeling capability, which can harm downstream performance. Counterfactual data augmentation (CDA), a widely used method for fine-tuning, highlights this issue by generating synthetic data that may align poorly with real-world distributions or creating overly simplistic counterfactuals that ignore the social context of altered sensitive attributes (e.g., gender) in the pretraining corpus. To address these limitations, we propose a simple yet effective context-augmented CDA method, Context-CDA, which uses large LMs to enhance the diversity and contextual relevance of the debiasing corpus. By minimizing discrepancies between the debiasing corpus and pretraining data through augmented context, this approach ensures better
alignment, enhancing
language modeling capability. We then employ uncertainty-based filtering to exclude generated counterfactuals considered low-quality by the target smaller LMs (i.e., LMs to be debiased), further improving the fine-tuning corpus quality. Experimental results on gender bias
benchmarks demonstrate that Context-CDA effectively mitigates bias without sacrificing
language modeling performance while offering insights into social biases by analyzing distribution shifts in next-
token generation probabilities.
Read the full abstract ▾
All authors Shweta Parihar, Liu Guangliang, Natalie Parde, Lu Cheng
Computation and Language
Aligning Tree-Search Policies with Fixed Token Budgets in Test-Time Scaling of LLMs
Sora Miyamoto, Daisuke Oba and Naoaki Okazaki
Tree-search decoding is an effective form of test-time scaling for large
language models (LLMs), but real-world deployment often imposes a fixed per-query
token budget that varies across settings. Existing tree-search policies are largely budget-agnostic, treating the budget merely as a termination condition, thereby risking late-stage over-branching or premature termination. We propose Budget-Guided MCTS (BG-MCTS), a tree-search decoding
algorithm that aligns its search policy with the remaining
token budget: it starts with broad exploration, then prioritizes refinement and answer completion as the remaining budget decreases while reducing late-stage branching from shallow nodes. BG-MCTS consistently outperforms budget-agnostic tree-search baselines across inference budgets on mathematical
reasoning benchmarks and an additional physics
reasoning benchmark with open-weight LLMs.
Read the full abstract ▾
Computation and Language
Narges Baba Ahmadi, Jan Strich, Martin Semmann et al. (+1 more)
Large
language models (LLMs) are increasingly used to access legal information. Yet, their deployment in
multilingual legal settings is constrained by unreliable
retrieval and the lack of domain-adapted, open-
embedding models. In particular, existing
multilingual legal corpora are not designed for semantic
retrieval, and PDF-based legislative sources introduce substantial noise due to imperfect text extraction. To address these challenges, we introduce LEMUR, a large-scale
multilingual corpus of EU environmental legislation constructed from 24,953 official EUR-Lex PDF documents covering 25
languages. We quantify the fidelity of PDF-to-text conversion by measuring lexical consistency against authoritative HTML versions using the Lexical Content Score (LCS). Building on LEMUR, we fine-tune three state-of-the-art
multilingual embedding models using contrastive objectives in both monolingual and bilingual settings, reflecting realistic legal-
retrieval scenarios. Experiments across low- and high-resource
languages demonstrate that legal-domain fine-tuning consistently improves Top-k
retrieval accuracy relative to strong baselines, with particularly pronounced gains for low-resource
languages. Cross-lingual evaluations show that these improvements transfer to unseen
languages, indicating that fine-tuning primarily enhances
language-independent, content-level legal representations rather than
language-specific cues. We publish code\footnote{\href{https://github.com/nargesbh/eur_lex}{GitHub Repository}} and data\footnote{\href{https://huggingface.co/
datasets/G4KMU/LEMUR}{Hugging Face
Dataset}}.
Read the full abstract ▾
All authors Narges Baba Ahmadi, Jan Strich, Martin Semmann, Chris Biemann
Computation and Language
Adaptive Test-Time Compute Allocation for Block Diffusion Language Models in Complex Reasoning
Yi Lu, Deyang Kong, Jianing Wang et al. (+8 more)
Recent advances in block
diffusion language models have demonstrated competitive performance and strong scalability on
reasoning tasks. However, their test-time compute allocation remains largely unexplored, leaving a critical speed-effectiveness trade-off unresolved in long Chain-of-Thought
reasoning. To address this, we propose a unified test-time compute allocation framework that introduces adaptivity in both step-wise decoding and blockwise generation. At the decoding level, we propose Bounded Adaptive Confidence Decoding (BACD), a difficulty-aware sampling strategy that dynamically adjusts denoising based on model confidence, accelerating inference while controlling error accumulation. Beyond step-wise adaptivity, we introduce the Think Coarse, Critic Fine (TCCF) paradigm that allocates large block sizes for exploratory thinking and smaller block sizes for precise refinement. To stabilize training under varying block configurations, we adopt Progressive Block Size Extension, which mitigates quality degradation when scaling up block sizes. Extensive evaluations on six
benchmarks show that our TDAR-8B model with BACD and TCCF achieves a 2.38$\times$ speedup and +3.4% average accuracy over the strong TraDo-8B baseline, unlocking the potential of block
diffusion in complex
reasoning.
Read the full abstract ▾
All authors Yi Lu, Deyang Kong, Jianing Wang, Linsen Guo, Xue Wang, Qi Guo, Tao Gui, Xuanjing Huang, Wei Ye, Shikun Zhang, Wei Wang
Computation and Language
Comprehensive Comparison of RAG Methods Across Multi-Domain Conversational QA
Klejda Alushi, Jan Strich, Chris Biemann et al. (+1 more)
Conversational question answering increasingly relies on
retrieval-augmented generation (RAG) to ground large
language models (LLMs) in external knowledge. Yet, most existing studies evaluate RAG methods in isolation and primarily focus on single-turn settings. This paper addresses the lack of a systematic comparison of RAG methods for multi-turn conversational QA, where dialogue history, coreference, and shifting user intent substantially complicate
retrieval. We present a comprehensive empirical study of vanilla and advanced RAG methods across eight diverse conversational QA
datasets spanning multiple domains. Using a unified experimental setup, we evaluate
retrieval quality and answer generation using generator and
retrieval metrics, and analyze how performance evolves across conversation turns. Our results show that robust yet straightforward methods, such as reranking, hybrid BM25, and HyDE, consistently outperform vanilla RAG. In contrast, several advanced techniques fail to yield gains and can even degrade performance below the No-RAG baseline. We further demonstrate that
dataset characteristics and dialogue length strongly influence
retrieval effectiveness, explaining why no single RAG strategy dominates across settings. Overall, our findings indicate that effective conversational RAG depends less on method complexity than on
alignment between the
retrieval strategy and the
dataset structure. We publish the code used.\footnote{\href{https://github.com/Klejda-A/exp-rag.git}{GitHub Repository}}
Read the full abstract ▾
All authors Klejda Alushi, Jan Strich, Chris Biemann, Martin Semmann
Computation and Language
UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignment
Hongyan Xie, Yikun Ban, Ruiyu Fang et al. (+6 more)
Multi-objective
alignment aims to align LLM responses with multiple human preference objectives. Among existing methods, guiding the generation of frozen LLMs through autoregressive reward models (ARMs) to accomplish multi-objective test-time
alignment is a low-cost solution. However, these methods typically rely on independent parameters for each preference objective, either by training ARMs independently across preference dimensions, which neglects interactions among preference features, or by training a single ARM with separate feature extraction modules for each preference, which can cause feature
entanglement. Both strategies can result in misalignment between generated outputs and user preferences. To address this limitation, we propose Preference-Modulated & Shared Low-Rank Adaptation (MoSLoRA) for ARM training, which first extracts shared features via a preference-agnostic module and then applies affine transformations to shared features via a preference modulation module conditioned on mixed preference vectors. This design mitigates feature
entanglement and enables precise control over preference trade-offs during inference. Building on this, we introduce the Unified Autoregressive Reward Model (UniARM), a novel framework for multi-objective test-time
alignment. UniARM jointly models all preference dimensions in a single parameter space, eliminating the need for independent parameters for each preference objective. es on larger-scale LLMs, enhancing its practical usability.
Read the full abstract ▾
All authors Hongyan Xie, Yikun Ban, Ruiyu Fang, Zixuan Huang, Deqing Wang, Jianxin Li, Yitong Yao, Chao Wang, Shuangyong Song
Computation and Language
Knowledge Integration Decay in Search-Augmented Reasoning of Large Language Models
Sangwon Yu, Ik-hwan Kim, Donghun Kang et al. (+6 more)
Modern Large
Language Models (LLMs) have demonstrated remarkable capabilities in complex tasks by employing search-augmented
reasoning to incorporate external knowledge into long chains of thought. However, we identify a critical yet underexplored bottleneck in this paradigm, termed Knowledge Integration Decay (KID). Specifically, we observe that as the length of
reasoning generated before search grows, models increasingly fail to integrate retrieved evidence into subsequent
reasoning steps, limiting performance even when relevant information is available. To address this, we propose Self-Anchored Knowledge Encoding (SAKE), a training-free inference-time strategy designed to stabilize knowledge utilization. By anchoring retrieved knowledge at both the beginning and end of the
reasoning process, SAKE prevents it from being overshadowed by prior context, thereby preserving its semantic integrity. Extensive experiments on multi-hop QA and complex
reasoning benchmarks demonstrate that SAKE significantly mitigates KID and improves performance, offering a lightweight yet effective solution for knowledge integration in agentic LLMs.
Read the full abstract ▾
All authors Sangwon Yu, Ik-hwan Kim, Donghun Kang, Bongkyu Hwang, Junhwa Choi, Suk-hoon Jung, Seungki Hong, Taehee Lee, Sungroh Yoon
Computation and Language
The CLEF-2026 CheckThat! Lab: Advancing Multilingual Fact-Checking
Julia Maria Struß, Sebastian Schellhammer, Stefan Dietze et al. (+9 more)
The CheckThat! lab aims to advance the development of innovative technologies combating disinformation and manipulation efforts in online communication across a multitude of
languages and platforms. While in early editions the focus has been on core tasks of the verification pipeline (check-worthiness, evidence
retrieval, and verification), in the past three editions, the lab added additional tasks linked to the verification process. In this year's edition, the verification pipeline is at the center again with the following tasks: Task 1 on source
retrieval for scientific web claims (a follow-up of the 2025 edition), Task 2 on fact-checking numerical and temporal claims, which adds a
reasoning component to the 2025 edition, and Task 3, which expands the verification pipeline with generation of full-fact-checking articles. These tasks represent challenging classification and
retrieval problems as well as generation challenges at the document and span level, including
multilingual settings.
Read the full abstract ▾
All authors Julia Maria Struß, Sebastian Schellhammer, Stefan Dietze, Venktesh V, Vinay Setty, Tanmoy Chakraborty, Preslav Nakov, Avishek Anand, Primakov Chungkham, Salim Hafid, Dhruv Sahnan, Konstantin Todorov
Computation and Language
EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies
Xavier Hu, Jinxiang Xia, Shengze Xu et al. (+13 more)
Long-horizon planning is widely recognized as a core capability of autonomous LLM-based
agents; however, current evaluation frameworks suffer from being largely episodic, domain-specific, or insufficiently grounded in persistent economic dynamics. We introduce EcoGym, a generalizable
benchmark for continuous plan-and-execute decision making in interactive economies. EcoGym comprises three diverse environments: Vending (adapted from the closed-source Vending-Bench, with full open-source release), Freelance (new), and Operation (new), implemented in a unified decision-making process with standardized interfaces, and budgeted actions over an effectively unbounded horizon (1000+ steps if 365 day-loops for evaluation). The evaluation of EcoGym is based on business-relevant outcomes (e.g., net worth, income, and DAU), targeting long-term strategic coherence and robustness under partial observability and stochasticity. Experiments across eleven leading LLMs expose a systematic tension: no single model dominates across all three scenarios. Critically, we find that models exhibit significant suboptimality in either high-level strategies or efficient actions executions. EcoGym is released as an open, extensible testbed for transparent long-horizon
agent evaluation and for studying controllability utility trade-offs in economic settings.
Read the full abstract ▾
All authors Xavier Hu, Jinxiang Xia, Shengze Xu, Kangqi Song, Yishuo Yuan, Guibin Zhang, JinCheng Ren, Boyu Feng, Li Lu, Tieyong Zeng, Jiaheng Liu, Minghao Liu, He Zhu, Yuchen Eleanor Jiang, Wei Wang, Wangchunshu Zhou
Computation and Language
Where-to-Unmask: Ground-Truth-Guided Unmasking Order Learning for Masked Diffusion Language Models
Hikaru Asano, Tadashi Kozuno, Kuniaki Saito et al. (+1 more)
Masked
Diffusion Language Models (MDLMs) generate text by iteratively filling masked
tokens, requiring two coupled decisions at each step: which positions to unmask (where-to-unmask) and which
tokens to place (what-to-unmask). While standard MDLM training directly optimizes
token prediction (what-to-unmask), inference-time unmasking orders (where-to-unmask) are typically determined by heuristic confidence measures or trained through reinforcement learning with costly on-policy rollouts. To address this, we introduce Gt-Margin, a position-wise score derived from ground-truth
tokens, defined as the probability margin between the correct
token and its strongest alternative. Gt-Margin yields an oracle unmasking order that prioritizes easier positions first under each partially masked state. We demonstrate that leveraging this oracle unmasking order significantly enhances final generation quality, particularly on logical
reasoning benchmarks. Building on this insight, we train a supervised unmasking planner via learning-to-rank to imitate the oracle ordering from masked contexts. The resulting planner integrates into standard MDLM sampling to select where-to-unmask, improving
reasoning accuracy without modifying the
token prediction model.
Read the full abstract ▾
All authors Hikaru Asano, Tadashi Kozuno, Kuniaki Saito, Yukino Baba
Computation and Language
Listen to the Layers: Mitigating Hallucinations with Inter-Layer Disagreement
Koduvayur Subbalakshmi, Sabbir Hossain Ujjal, Venkata Krishna Teja Mangichetty et al. (+1 more)
Pretrained Large
Language Models (LLMs) are prone to generating fluent yet factually incorrect text-a phenomenon known as
hallucinations, undermining their reliability and utility in downstream tasks. We hypothesize that a generated text span's factuality is correlated with its representational instability across the model's internal layers. Based on this, we propose the CoCoA (Confusion and Consistency Aware) decoder, a novel, training-free decoding
algorithm that mitigates
hallucinations at inference time by listening to these signals in the middle layers. We propose two metrics to quantify this instability in the middle layers and use it to penalize outputs that exhibit high internal confusion, thereby steering the model towards more internally consistent and factually grounded outputs. We further propose a self-information gated variant, CoCoA-SIG, that dynamically modulates this penalty to selectively target high-surprise, unstable generations. Extensive experiments on diverse tasks, including question-answering, summarization, mathematical
reasoning and code generation, demonstrate that CoCoA significantly improves factual correctness across multiple model families (e.g., Llama-3, Qwen-2.5, Mistral). By leveraging model-intrinsic signals, CoCoA offers an effective and broadly applicable method for enhancing the trustworthiness of LLMs at inference time, without requiring any model retraining.
Read the full abstract ▾
All authors Koduvayur Subbalakshmi, Sabbir Hossain Ujjal, Venkata Krishna Teja Mangichetty, Nastaran Jamalipour Soofi
Computation and Language
NOWJ @BioCreative IX ToxHabits: An Ensemble Deep Learning Approach for Detecting Substance Use and Contextual Information in Clinical Texts
Huu-Huy-Hoang Tran, Gia-Bao Duong, Quoc-Viet-Anh Tran et al. (+2 more)
Extracting drug use information from unstructured Electronic Health Records remains a major challenge in clinical Natural
Language Processing. While Large
Language Models demonstrate advancements, their use in clinical NLP is limited by concerns over trust, control, and efficiency. To address this, we present NOWJ submission to the ToxHabits Shared Task at BioCreative IX. This task targets the detection of toxic substance use and contextual attributes in Spanish clinical texts, a domain-specific, low-resource setting. We propose a multi-output ensemble system tackling both Subtask 1 - ToxNER and Subtask 2 - ToxUse. Our system integrates BETO with a CRF layer for sequence labeling, employs diverse training strategies, and uses sentence filtering to boost precision. Our top run achieved 0.94 F1 and 0.97 precision for Trigger Detection, and 0.91 F1 for Argument Detection.
Read the full abstract ▾
All authors Huu-Huy-Hoang Tran, Gia-Bao Duong, Quoc-Viet-Anh Tran, Thi-Hai-Yen Vuong, Hoang-Quynh Le
Computation and Language · Cross-listed from cs.SE
AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms
Haoyu Zhao, Ziran Yang, Jiawei Li et al. (+6 more)
Vericoding refers to the generation of formally verified code from rigorous specifications. Recent AI models show promise in vericoding, but a unified methodology for cross-paradigm evaluation is lacking. Existing
benchmarks test only individual
languages/tools (e.g., Dafny, Verus, and Lean) and each covers very different tasks, so the performance numbers are not directly comparable. We address this gap with AlgoVeri, a
benchmark that evaluates vericoding of $77$ classical
algorithms in Dafny, Verus, and Lean. By enforcing identical functional contracts, AlgoVeri reveals critical capability gaps in verification systems. While frontier models achieve tractable success in Dafny ($40.3$% for Gemini-3 Flash), where high-level abstractions and SMT automation simplify the workflow, performance collapses under the systems-level
memory constraints of Verus ($24.7$%) and the explicit proof construction required by Lean (7.8%). Beyond aggregate metrics, we uncover a sharp divergence in test-time compute dynamics: Gemini-3 effectively utilizes iterative repair to boost performance (e.g., tripling pass rates in Dafny), whereas GPT-OSS saturates early. Finally, our error analysis shows that
language design affects the refinement trajectory: while Dafny allows models to focus on logical correctness, Verus and Lean trap models in persistent syntactic and semantic barriers. All data and evaluation code can be found at https://github.com/haoyuzhao123/algoveri.
Read the full abstract ▾
All authors Haoyu Zhao, Ziran Yang, Jiawei Li, Deyuan He, Zenan Li, Chi Jin, Venugopal V. Veeravalli, Aarti Gupta, Sanjeev Arora
Computation and Language · Cross-listed from cs.SE
SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents
Zhirui Zhang, Hongbo Zhang, Haoxiang Fei et al. (+11 more)
Although large
language models (LLMs) have demonstrated impressive coding capabilities, their ability to autonomously build production-scale software from explicit specifications remains an open question. We introduce SWE-AGI, an open-source
benchmark for evaluating end-to-end, specification-driven construction of software systems written in MoonBit. SWE-AGI tasks require LLM-based
agents to implement parsers, interpreters, binary decoders, and SAT solvers strictly from authoritative standards and RFCs under a fixed API scaffold. Each task involves implementing 1,000-10,000 lines of core logic, corresponding to weeks or months of engineering effort for an experienced human developer. By leveraging the nascent MoonBit ecosystem, SWE-AGI minimizes data leakage, forcing
agents to rely on long-horizon architectural
reasoning rather than code
retrieval. Across frontier models, gpt-5.3-codex achieves the best overall performance (solving 19/22 tasks, 86.4%), outperforming claude-opus-4.6 (15/22, 68.2%), and kimi-2.5 exhibits the strongest performance among open-source models. Performance degrades sharply with increasing task difficulty, particularly on hard, specification-intensive systems. Behavioral analysis further reveals that as codebases scale, code reading, rather than writing, becomes the dominant bottleneck in AI-assisted development. Overall, while specification-driven autonomous software engineering is increasingly viable, substantial challenges remain before it can reliably support production-scale development.
Read the full abstract ▾
All authors Zhirui Zhang, Hongbo Zhang, Haoxiang Fei, Zhiyuan Bao, Yubin Chen, Zhengyu Lei, Ziyue Liu, Yixuan Sun, Mingkun Xiao, Zihang Ye, Yu Zhang, Hongcheng Zhu, Yuxiang Wen, Heung-Yeung Shum
Computation and Language
Conceptual Cultural Index: A Metric for Cultural Specificity via Relative Generality
Takumi Ohashi and Hitoshi Iyatomi
Large
language models (LLMs) are increasingly deployed in multicultural settings; however, systematic evaluation of cultural specificity at the sentence level remains underexplored. We propose the Conceptual Cultural Index (CCI), which estimates cultural specificity at the sentence level. CCI is defined as the difference between the generality estimate within the target culture and the average generality estimate across other cultures. This formulation enables users to operationally control the scope of culture via comparison settings and provides interpretability, since the score derives from the underlying generality estimates. We validate CCI on 400 sentences (200 culture-specific and 200 general), and the resulting score distribution exhibits the anticipated pattern: higher for culture-specific sentences and lower for general ones. For binary separability, CCI outperforms direct LLM scoring, yielding more than a 10-point improvement in AUC for models specialized to the target culture. Our code is available at https://github.com/IyatomiLab/CCI .
Read the full abstract ▾
Computation and Language
Evaluating Social Bias in RAG Systems: When External Context Helps and Reasoning Hurts
Shweta Parihar and Lu Cheng
Social biases inherent in large
language models (LLMs) raise significant
fairness concerns.
Retrieval-Augmented Generation (RAG) architectures, which retrieve external knowledge sources to enhance the generative capabilities of LLMs, remain susceptible to the same bias-related challenges. This work focuses on evaluating and understanding the social bias implications of RAG. Through extensive experiments across various
retrieval corpora, LLMs, and bias evaluation
datasets, encompassing more than 13 different bias types, we surprisingly observe a reduction in bias in RAG. This suggests that the inclusion of external context can help counteract stereotype-driven predictions, potentially improving
fairness by diversifying the contextual grounding of the model's outputs. To better understand this phenomenon, we then explore the model's
reasoning process by integrating Chain-of-Thought (CoT) prompting into RAG while assessing the faithfulness of the model's CoT. Our experiments reveal that the model's bias inclinations shift between stereotype and anti-stereotype responses as more contextual information is incorporated from the retrieved documents. Interestingly, we find that while CoT enhances accuracy, contrary to the bias reduction observed with RAG, it increases overall bias across
datasets, highlighting the need for bias-aware
reasoning frameworks that can mitigate this trade-off.
Read the full abstract ▾
Computation and Language
Breaking the Pre-Sampling Barrier: Activation-Informed Difficulty-Aware Self-Consistency
Taewoong Yoon, Geunyeong Jeong, Geon Park et al. (+2 more)
Self-Consistency (SC) is an effective decoding strategy that improves the
reasoning performance of Large
Language Models (LLMs) by generating multiple chain-of-thought
reasoning paths and selecting the final answer via majority voting. However, it suffers from substantial inference costs because it requires a large number of samples. To mitigate this issue, Difficulty-Adaptive Self-Consistency (DSC) was proposed to reduce unnecessary
token usage for easy problems by adjusting the number of samples according to problem difficulty. However, DSC requires additional model calls and pre-sampling to estimate difficulty, and this process is repeated when applying to each
dataset, leading to significant computational overhead. In this work, we propose Activation-Informed Difficulty-Aware Self-Consistency (ACTSC) to address these limitations. ACTSC leverages internal difficulty signals reflected in the feed-forward network
neuron activations to construct a lightweight difficulty estimation probe, without any additional
token generation or model calls. The probe dynamically adjusts the number of samples for SC and can be applied to new
datasets without requiring pre-sampling for difficulty estimation. To validate its effectiveness, we conduct experiments on five
benchmarks. Experimental results show that ACTSC effectively reduces inference costs while maintaining accuracy relative to existing methods.
Read the full abstract ▾
All authors Taewoong Yoon, Geunyeong Jeong, Geon Park, Sihyeong Yeom, Harksoo Kim
Computation and Language
Are Language Models Sensitive to Morally Irrelevant Distractors?
Andrew Shaw, Christina Hahn, Catherine Rasgaitis et al. (+5 more)
With the rapid uptake of large
language models (LLMs) across high-stakes settings, it is becoming increasingly important to ensure that LLMs behave in ways that align with human values. Existing moral
benchmarks for this purpose often
prompt LLMs with value statements, moral scenarios, or psychological questionnaires, with the implicit underlying assumption that LLMs report somewhat stable moral preferences. However, moral psychology research has shown that even human moral judgements are sensitive to morally irrelevant situational factors such as the smell of cinnamon rolls or the level of ambient noise, thereby challenging moral theories which assume that human moral judgements are stable. Here we draw inspiration from this "situationist" view of moral psychology to evaluate whether LLMs exhibit similar cognitive moral biases. We curate a novel
multimodal dataset of 60 "moral distractors" from existing psychological
datasets of emotionally-valenced images and narratives, which have no moral relevance to the situation presented. After injecting these distractors into existing moral
benchmarks, we find that moral distractors can shift the moral judgements of LLMs by over 30% even in unambiguous scenarios, highlighting the instability of LLMs' moral judgements and the need for more contextual approaches to AI
alignment.
Read the full abstract ▾
All authors Andrew Shaw, Christina Hahn, Catherine Rasgaitis, Yash Mishra, Alisa Liu, Natasha Jaques, Yulia Tsvetkov, Amy X. Zhang
Computation and Language · Cross-listed from eess.AS
TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization
Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah et al. (+1 more)
Real-time voice conversion and speaker anonymization require causal, low-latency synthesis without sacrificing intelligibility or naturalness. Current systems have a core representational mismatch: content is time-varying, while speaker identity is injected as a static global
embedding. We introduce a streamable
speech synthesizer that aligns the temporal granularity of identity and content via a content-synchronous, time-varying timbre (TVT) representation. A Global Timbre
Memory expands a global timbre instance into multiple compact facets; frame-level content attends to this
memory, a gate regulates variation, and spherical interpolation preserves identity geometry while enabling smooth local changes. In addition, a factorized vector-quantized bottleneck regularizes content to reduce residual speaker leakage. The resulting system is streamable end-to-end, with <80 ms GPU latency. Experiments show improvements in naturalness, speaker transfer, and anonymization compared to SOTA streaming baselines, establishing TVT as a scalable approach for
privacy-preserving and expressive
speech synthesis under strict latency budgets.
Read the full abstract ▾
All authors Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah, Ricardo Gutierrez-Osuna