Computation and Language
The Effect of Scripts and Formats on LLM Numeracy
Varshini Reddy, Craig W. Schmidt, Seth Ebner et al. (+3 more)
Large
language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little
attention has been given to how these models perform when numerical expressions deviate from the prevailing conventions present in their training corpora. In this work, we investigate numerical
reasoning across a wide range of numeral scripts and formats. We show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats, despite the underlying mathematical
reasoning being identical. We further demonstrate that targeted prompting strategies, such as few-shot prompting and explicit numeral mapping, can greatly narrow this gap. Our findings highlight an overlooked challenge in
multilingual numerical
reasoning and provide actionable insights for working with LLMs to reliably interpret, manipulate, and generate numbers across diverse numeral scripts and formatting styles.
Read the full abstract ▾
All authors Varshini Reddy, Craig W. Schmidt, Seth Ebner, Adam Wiemerslage, Yuval Pinter, Chris Tanner
Computation and Language
Taxonomy-Aligned Risk Extraction from 10-K Filings with Autonomous Improvement Using LLMs
Rian Dolphin, Joe Dursun, Jarrett Blankenship et al. (+2 more)
We present a methodology for extracting structured risk factors from corporate 10-K filings while maintaining adherence to a predefined hierarchical taxonomy. Our three-stage pipeline combines LLM extraction with supporting quotes,
embedding-based semantic mapping to taxonomy categories, and LLM-as-a-judge validation that filters spurious assignments. To evaluate our approach, we extract 10,688 risk factors from S&P 500 companies and examine risk profile similarity across industry clusters. Beyond extraction, we introduce autonomous taxonomy maintenance where an AI
agent analyzes evaluation feedback to identify problematic categories, diagnose failure patterns, and propose refinements, achieving 104.7% improvement in
embedding separation in a case study. External validation confirms the taxonomy captures economically meaningful structure: same-industry companies exhibit 63% higher risk profile similarity than cross-industry pairs (Cohen's d=1.06, AUC 0.82, p<0.001). The methodology generalizes to any domain requiring taxonomy-aligned extraction from unstructured text, with autonomous improvement enabling continuous quality maintenance and enhancement as systems process more documents.
Read the full abstract ▾
All authors Rian Dolphin, Joe Dursun, Jarrett Blankenship, Katie Adams, Quinton Pike
Computation and Language
MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answering
Anjishnu Mukherjee, Ziwei Zhu and Antonios Anastasopoulos
Large
language models can memorize competing locale-specific facts yet fail to select among them when the locale changes, defaulting instead to a single globally dominant answer. We formalize this as localized knowledge disambiguation and introduce LocalNewsQA, an 18,700-item English-news
benchmark that pairs the same question across two locales and scores whether a model actually switches its answer when the locale changes. We also introduce MAPLE, a controlled family of decoder-only models pretrained with document-level geographic metadata (source URL, country, and continent) already present in the training corpus, and compare it to metadata-free controls trained on identical data with the same
token budget, architecture, and optimization. In controlled experiments at 1B and 3B, with inference-time metadata fixed, pretraining with metadata in MAPLE produces measurable switching and improves accuracy on questions whose correct answer depends on locale. Ablations and external-
benchmark evaluations further suggest that locale-conditioned prediction benefits from geographic provenance learned during pretraining and that these benefits strengthen at larger model sizes.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Jianshu Zhang, Chengxuan Qian, Haosen Sun et al. (+4 more)
Estimating task progress requires
reasoning over long-horizon dynamics rather than recognizing static visual content. While modern
Vision-
Language Models (VLMs) excel at describing what is visible, it remains unclear whether they can infer how far a task has progressed from partial observations. To this end, we introduce Progress-Bench, a
benchmark for systematically evaluating progress
reasoning in VLMs. Beyond benchmarking, we further explore a human-inspired two-stage progress
reasoning paradigm through both training-free prompting and training-based approach based on curated
dataset ProgressLM-45K. Experiments on 14 VLMs show that most models are not yet ready for task progress estimation, exhibiting sensitivity to demonstration modality and viewpoint changes, as well as poor handling of unanswerable cases. While training-free prompting that enforces structured progress
reasoning yields limited and model-dependent gains, the training-based ProgressLM-3B achieves consistent improvements even at a small model scale, despite being trained on a task set fully disjoint from the evaluation tasks. Further analyses reveal characteristic error patterns and clarify when and why progress
reasoning succeeds or fails. Website: https://progresslm.github.io/ProgressLM/
Read the full abstract ▾
All authors Jianshu Zhang, Chengxuan Qian, Haosen Sun, Haoran Lu, Dingcheng Wang, Letian Xue, Han Liu
Computation and Language
Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models
Anmol Goel, Cornelius Emde, Sangdoo Yun et al. (+2 more)
We identify a novel phenomenon in
language models: benign fine-tuning of frontier models can lead to
privacy collapse. We find that diverse, subtle patterns in training data can degrade contextual
privacy, including optimisation for helpfulness, exposure to user information, emotional and subjective dialogue, and debugging code printing internal variables, among others. Fine-tuned models lose their ability to reason about contextual
privacy norms, share information inappropriately with tools, and violate
memory boundaries across contexts.
Privacy collapse is a “silent failure” because models maintain high performance on standard safety and utility
benchmarks whilst exhibiting severe
privacy vulnerabilities. Our experiments show evidence of
privacy collapse across six models (closed and open weight), five fine-tuning
datasets (real-world and controlled data), and two task categories (agentic and
memory-based). Our mechanistic analysis reveals that
privacy representations are uniquely fragile to fine-tuning, compared to task-relevant features which are preserved. Our results reveal a critical gap in current safety evaluations, in particular for the deployment of specialised
agents.
Read the full abstract ▾
All authors Anmol Goel, Cornelius Emde, Sangdoo Yun, Seong Joon Oh, Martin Gubri
Computation and Language · Cross-listed from cs.AI
LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries
Shijie Lian, Bin Yu, Xiaopeng Lin et al. (+6 more)
Vision-
Language-Action (VLA) models have shown promise in
robot manipulation but often struggle to generalize to new instructions or complex multi-task scenarios. We identify a critical pathology in current training paradigms where goal-driven data collection creates a
dataset bias. In such
datasets,
language instructions are highly predictable from visual observations alone, causing the conditional mutual information between instructions and actions to vanish, a phenomenon we term Information Collapse. Consequently, models degenerate into
vision-only policies that ignore
language constraints and fail in out-of-distribution (OOD) settings. To address this, we propose LangForce, a novel framework that enforces instruction following via
Bayesian decomposition. By introducing learnable Latent Action Queries, we construct a dual-branch architecture to estimate both a
vision-only prior $p(a \mid v)$ and a
language-conditioned posterior $\pi(a \mid v, \ell)$. We then optimize the policy to maximize the conditional Pointwise Mutual Information (PMI) between actions and instructions. This objective effectively penalizes the
vision shortcut and rewards actions that explicitly explain the
language command. Without requiring new data, LangForce significantly improves generalization. Extensive experiments across on SimplerEnv and RoboCasa demonstrate substantial gains, including an 11.3% improvement on the challenging OOD SimplerEnv
benchmark, validating the ability of our approach to robustly ground
language in action.
Read the full abstract ▾
All authors Shijie Lian, Bin Yu, Xiaopeng Lin, Laurence T. Yang, Zhaolong Shen, Changti Wu, Yuzhuo Miao, Cong Huang, Kai Chen
Computation and Language
Supporting Humans in Evaluating AI Summaries of Legal Depositions
Naghmeh Farzi, Laura Dietz and Dave D. Lewis
While large
language models (LLMs) are increasingly used to summarize long documents, this trend poses significant challenges in the legal domain, where the factual accuracy of deposition summaries is crucial. Nugget-based methods have been shown to be extremely helpful for the automated evaluation of summarization approaches. In this work, we translate these methods to the user side and explore how nuggets could directly assist end users. Although prior systems have demonstrated the promise of nugget-based evaluation, its potential to support end users remains underexplored. Focusing on the legal domain, we present a prototype that leverages a factual nugget-based approach to support legal professionals in two concrete scenarios: (1) determining which of two summaries is better, and (2) manually improving an automatically generated summary.
Read the full abstract ▾
Computation and Language
Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time
Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya et al. (+1 more)
Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing
evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its
evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.
Read the full abstract ▾
All authors Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych
Computation and Language
The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
Zanlin Ni, Shenzhi Wang, Yang Yue et al. (+8 more)
Diffusion Large
Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs, enabling
token generation in arbitrary orders. Intuitively, this flexibility implies a solution space that strictly supersets the fixed autoregressive trajectory, theoretically unlocking superior
reasoning potential. However, in this paper, we find that for general
reasoning tasks (e.g., mathematics and coding), arbitrary order generation may in fact limit the
reasoning potential of dLLMs. We observe that dLLMs tend to exploit this order flexibility to bypass high-uncertainty
tokens that are crucial for exploration, which can lead to a premature collapse of solution coverage. This observation motivates a rethink of RL approaches for dLLMs, where considerable complexities, such as handling combinatorial trajectories and intractable likelihoods, are often devoted to preserving this flexibility. We show that effective
reasoning can be elicited by simply forgoing arbitrary order and applying standard Group Relative Policy Optimization (GRPO) instead. Our approach, JustGRPO, is minimalist yet surprisingly effective (e.g., 89.1% accuracy on GSM8K) while fully retaining the parallel decoding ability of dLLMs. Project page: https://nzl-thu.github.io/the-flexibility-trap
Read the full abstract ▾
All authors Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, Gao Huang
Computation and Language
Retrieval-Augmented Agentic Rubric Generation for Reliable Medical Response Evaluation
Yinzhu Chen, Abdine Maiga, Hossein A. Rahmani et al. (+1 more)
Large
Language Models (LLMs) are increasingly used for clinical decision support, where
hallucinations and unsafe suggestions may pose direct risks to patient safety. These risks are hard to assess: subtle clinical errors are often missed by generic metrics and LLM judges using general criteria, while expert-authored fine-grained rubrics are expensive and difficult to scale. In this paper, we propose a
retrieval-augmented multi-
agent framework for automatically generating instance-specific evaluation rubrics. Our approach grounds evaluation in authoritative medical evidence by decomposing retrieved content into atomic facts and synthesizing them with user interaction constraints to form fine-grained evaluation criteria. Evaluated on HealthBench and LLMEval-Med, our framework achieves Clinical Intent
Alignment (CIA) scores of 50.20% and 31.90%, significantly outperforming the GPT-4o baseline and showing consistent improvements across English and Chinese medical
benchmarks. In discriminative tests on HealthBench, our rubrics achieve a 7.8% point higher win rate than GPT-4o and increase the mean score difference from 4.972 to 8.658. Ablation studies further show that individual components contribute differently across
datasets, with interaction-intent modeling providing the most consistent contribution to clinical-criterion coverage. Beyond evaluation, our rubrics guide response refinement, improving response quality by 9.2%. These results suggest that automated, knowledge-grounded rubric generation provides a scalable foundation for evaluating and improving medical LLMs. The code is available at https://anonymous.4open.science/r/Automated-Rubric-Generation-E716/.
Read the full abstract ▾
All authors Yinzhu Chen, Abdine Maiga, Hossein A. Rahmani, Emine Yilmaz
Computation and Language · Cross-listed from cs.AI
Knowledge Graphs are Implicit Reward Models: Path-Derived Signals Enable Compositional Reasoning
Yuval Kansal and Niraj K. Jha
Large
language models have achieved near-expert performance in structured
reasoning domains like mathematics and programming, yet their ability to perform compositional multi-hop
reasoning in specialized scientific fields remains limited. We propose a bottom-up learning paradigm in which models are grounded in axiomatic domain facts and compose them to solve complex, unseen tasks. To this end, we present a post-training pipeline, based on a combination of supervised fine-tuning and reinforcement learning (RL), in which knowledge graphs act as implicit reward models. By deriving novel reward signals from knowledge graph paths, we provide verifiable, scalable, and grounded supervision that encourages models to compose intermediate axioms rather than optimize only final answers during RL. We validate this approach in the medical domain, training a 14B model on short-hop
reasoning paths (1-3 hops) and evaluating its zero-shot generalization to complex multi-hop queries (4-5 hops). Our experiments show that path-derived rewards act as a "compositional bridge", enabling our model to significantly outperform much larger models and frontier systems like GPT-5.2 and Gemini 3 Pro, on the most difficult
reasoning tasks. Furthermore, we demonstrate the robustness of our approach to
adversarial perturbations against option-shuffling stress tests. This work suggests that grounding the
reasoning process in structured knowledge is a scalable and efficient path toward intelligent
reasoning. Our code is publicly available at: https://github.com/jha-lab/kg-implicit-reward-compositional-rl/.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The Plausibility Trap: Using Probabilistic Engines for Deterministic Tasks
Ivan Carrera and Daniel Maldonado-Ruiz
The ubiquity of Large
Language Models (LLMs) is driving a paradigm shift where user convenience supersedes computational efficiency. This article defines the "Plausibility Trap": a phenomenon where individuals with access to Artificial Intelligence (AI) models deploy expensive probabilistic engines for simple deterministic tasks-such as Optical Character Recognition (OCR) or basic verification-resulting in significant resource waste. Through micro-
benchmarks and case studies on OCR and fact-checking, we quantify the "efficiency tax"-demonstrating a ~6.5x latency penalty-and the risks of algorithmic sycophancy. To counter this, we introduce Tool Selection Engineering and the Deterministic-Probabilistic Decision
Matrix, a framework to help developers determine when to use Generative AI and, crucially, when to avoid it. We argue for a curriculum shift, emphasizing that true digital literacy relies not only in knowing how to use Generative AI, but also on knowing when not to use it.
Read the full abstract ▾
Computation and Language
RSNA Large Language Model Benchmark Dataset for Chest Radiographs of Cardiothoracic Disease: Radiologist Evaluation and Validation Enhanced by AI Labels (REVEAL-CXR)
Yishu Wei, Adam E. Flanders, Errol Colak et al. (+35 more)
Multimodal large
language models have demonstrated comparable performance to that of radiology trainees on multiple-choice board-style exams. However, to develop clinically useful
multimodal LLM tools, high-quality
benchmarks curated by domain experts are essential. To curate released and holdout
datasets of 100 chest radiographic studies each and propose an artificial intelligence (AI)-assisted expert labeling procedure to allow radiologists to label studies more efficiently. A total of 13,735 deidentified chest radiographs and their corresponding reports from the MIDRC were used. GPT-4o extracted abnormal findings from the reports, which were then mapped to 12
benchmark labels with a locally hosted LLM (Phi-4-
Reasoning). From these studies, 1,000 were sampled on the basis of the AI-suggested
benchmark labels for expert review; the sampling
algorithm ensured that the selected studies were clinically relevant and captured a range of difficulty levels. Seventeen chest radiologists participated, and they marked "Agree all", "Agree mostly" or "Disagree" to indicate their assessment of the correctness of the LLM suggested labels. Each chest radiograph was evaluated by three experts. Of these, at least two radiologists selected "Agree All" for 381 radiographs. From this set, 200 were selected, prioritizing those with less common or multiple finding labels, and divided into 100 released radiographs and 100 reserved as the holdout
dataset. The holdout
dataset is used exclusively by RSNA to independently evaluate different models. A
benchmark of 200 chest radiographic studies with 12
benchmark labels was created and made publicly available https://imaging.rsna.org, with each chest radiograph verified by three radiologists. In addition, an AI-assisted labeling procedure was developed to help radiologists label at scale, minimize unnecessary omissions, and support a semicollaborative environment.
Read the full abstract ▾
All authors Yishu Wei, Adam E. Flanders, Errol Colak, John Mongan, Luciano M Prevedello, Po-Hao Chen, Henrique Min Ho Lee, Gilberto Szarf, Hamilton Shoji, Jason Sho, Katherine Andriole, Tessa Cook, Lisa C. Adams, Linda C. Chu, Maggie Chung, Geraldine Brusca-Augello, Djeven P. Deva, Navneet Singh, Felipe Sanchez Tijmes, Jeffrey B. Alpert, Elsie T. Nguyen, Drew A. Torigian, Kate Hanneman, Lauren K Groner, Alexander Phan, Ali Islam, Matias F.Callejas, Gustavo Borges da Silva Teles, Faisal Jamal, Maryam Vazirabad, Ali Tejani, Hari Trivedi, Paulo Kuriki, Rajesh Bhayana, Elana T. Benishay, Yi Lin, Yifan Peng, George Shih
Computation and Language · Cross-listed from cs.SD
WavLink: Compact Audio-Text Embeddings with a Global Whisper Token
Gokul Karthik Kumar, Ludovick Lepauloux and Hakim Hacid
Whisper has become the de-facto encoder for extracting general-purpose audio features in large audio-
language models, where a 30-second clip is typically represented by 1500 frame features projected into an LLM. In contrast, audio-text
embedding models like CLAP-based models have largely relied on alternative audio encoders (e.g., HTS-AT, PaSST), and have not leveraged Whisper effectively. We present WavLink, a compact audio-text
embedding model that augments Whisper encoder with a learnable global
token, trained jointly with a text encoder. Through a systematic study of design choices, including pretrained text encoders, loss functions, training modes, and data mixtures, we identify configurations that yield state-of-the-art
retrieval performance. Our two-stage training recipe across three model sizes, combined with Matryoshka-style supervision, improves scalability, enabling 8x smaller
embeddings with minimal performance drop. WavLink also demonstrates competitive performance on AIR-Bench with MCQs and zero-shot classification.
Read the full abstract ▾
Computation and Language
Circadian Modulation of Semantic Exploration in Social Media Language
Vuong Hung Truong, Mariana Gabrielle Cangco Reyes, Masatoshi Koizumi et al. (+1 more)
Human cognition exhibits strong circadian modulation, yet its influence on high-dimensional semantic behavior remains poorly understood. Using large-scale Reddit data, we quantify time-of-day variation in
language use by
embedding text into a pretrained
transformer model and measuring semantic
entropy as an index of linguistic exploration-exploitation, for which we show a robust circadian rhythmicity that could be entrained by seasonal light cues. Distinguishing between local and global semantic
entropy reveals a systematic temporal dissociation: local semantic exploration peaks in the morning, reflecting broader exploration of semantic space, whereas global semantic diversity peaks later in the day as submissions accumulate around already established topics, consistent with "rich-get-richer" dynamics. These patterns are not explained by sentiment or affective valence, indicating that semantic exploration captures a cognitive dimension distinct from mood. The observed temporal structure aligns with known diurnal patterns in neuromodulatory systems, suggesting that biological circadian rhythms extend to the semantic domain.
Read the full abstract ▾
All authors Vuong Hung Truong, Mariana Gabrielle Cangco Reyes, Masatoshi Koizumi, Jihwan Myung
Computation and Language
Multi-Agent Constraint Factorization Reveals Latent Invariant Solution Structure
Christopher Scofield
Multi-
agent systems (MAS) composed of large
language models often exhibit improved problem-solving performance despite operating on identical information. In this work, we provide a formal explanation for this phenomenon grounded in operator theory and constrained optimization. We model each
agent as enforcing a distinct family of validity constraints on a shared solution state, and show that a MAS implements a factorized composition of constraint-enforcement operators. Under mild conditions, these dynamics converge to invariant solution sets defined by the intersection of
agent constraint sets. Such invariant structures are generally not dynamically accessible to a single
agent applying all constraints simultaneously, even when expressive capacity and information are identical. We extend this result from exact constraint enforcement to soft constraints via proximal operators, and apply the formalism to contemporary text-based dialog systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution
Chen Qian, Peng Wang, Dongrui Liu et al. (+10 more)
Large
Language Model (LLM)-based
agents are widely used in real-world applications such as customer service, web
navigation, and software engineering. As these systems become more autonomous and are deployed at scale, understanding why an
agent takes a particular action becomes increasingly important for accountability and governance. However, existing research predominantly focuses on failure attribution to localize explicit errors in unsuccessful trajectories, which is insufficient for explaining the reason behind
agent behaviors. To bridge this gap, we propose a novel framework for general agentic attribution, designed to identify the internal factors driving
agent actions regardless of the task outcome. Our framework operates hierarchically to manage the complexity of
agent interactions. Specifically, at the component level, we employ temporal likelihood dynamics to identify critical interaction steps; then at the sentence level, we refine this localization using perturbation-based analysis to isolate the specific textual evidence. We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like
memory-induced bias. Experimental results demonstrate that the proposed framework reliably pinpoints pivotal historical events and sentences behind the
agent behavior, offering a critical step toward safer and more accountable agentic systems. Codes are available at https://github.com/AI45Lab/AgentDoG.
Read the full abstract ▾
All authors Chen Qian, Peng Wang, Dongrui Liu, Junyao Yang, Dadi Guo, Ling Tang, Jilin Mei, Qihan Ren, Shuai Shao, Yong Liu, Jie Fu, Jing Shao, Xia Hu
Computation and Language
Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScore
Zhichao Yan, Yunxiao Zhao, Jiapu Wang et al. (+4 more)
Current evaluation methods for
Retrieval Augmented Generation (RAG) suffer from factual myopia: they relentlessly emphasize factual accuracy yet neglect global logical integrity in long-form answer generation. This drives models to force unnatural connections, producing factually grounded yet logically incoherent responses with unaddressed gaps, ambiguous links, or redundant premises. To mitigate this, we present LogicScore, shifting from local, fact-by-fact assessment to rigorous global
reasoning scrutiny. Grounded in Horn Rules, our approach integrates a backward verification mechanism to systematically evaluate three key
reasoning dimensions: Completeness (logically sound deduction), Essentiality (non-redundancy), and Determinateness (consistent answer entailment). Extensive experiments across three multi-hop QA
datasets (HotpotQA, MusiQue, and 2WikiMultiHopQA) and over 20 LLMs (including GPT-5, Gemini-3-Pro, LLaMA3, and task-specific tuned models) reveal a critical capability gap: leading models often achieve high factual accuracy (e.g., 92.85% precision for Gemini-3 Pro) but struggle with global
reasoning quality (e.g., 35.11% Essentiality for Gemini-3 Pro). Our work establishes a robust standard for logical evaluation, highlighting the need to prioritize
reasoning coherence alongside factual grounding in LLM development.
Read the full abstract ▾
All authors Zhichao Yan, Yunxiao Zhao, Jiapu Wang, Jiaoyan Chen, Xiaoli Li, Ru Li, Jeff Z. Pan
Computation and Language
Knowledge Restoration-driven Prompt Optimization: Unlocking LLM Potential for Open-Domain Relational Triplet Extraction
Xiaonan Jing, Gongqing Wu, Xingrui Zhuo et al. (+2 more)
Open-domain Relational Triplet Extraction (ORTE) aims to mine structured knowledge without predefined relation schemas. Large
Language Models (LLMs) have advanced ORTE toward a
prompt-driven paradigm through powerful in-context learning. However, adapting their extraction behavior to varying open-domain contexts remains challenging. Existing methods typically rely on manually crafted
prompts that remain fixed across inputs, despite substantial variation in linguistic expressions and contextual structures. This mismatch may lead to unsupported triplets, while the absence of ground-truth annotations makes such deficiencies difficult to identify and correct. Moreover, free-form relation generation produces non-canonical relation surface forms, undermining knowledge graph consistency. To address these challenges, we propose Knowledge Restoration-driven
Prompt Optimization (KRPO), a framework for label-free target-corpus adaptation. KRPO restores extracted triplets into textual statements and evaluates their semantic consistency with the source inputs, deriving intrinsic feedback without gold annotations. This feedback is transformed into natural-
language optimization guidance for batch-wise
prompt optimization and adaptation. KRPO further introduces a
Memory-augmented Relation Canonicalizer that aligns free-form relations with a dynamically updated schema
memory, improving relation consistency. Experiments on three ORTE
benchmarks with multiple LLM backbones demonstrate strong overall performance, with KRPO achieving the best average F1 score across the evaluated settings.
Read the full abstract ▾
All authors Xiaonan Jing, Gongqing Wu, Xingrui Zhuo, Lang Sun, Jiapu Wang
Computation and Language
Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora
Chaymaa Abbas, Nour Shammaa and Mariette Awad
Data contamination can invalidate
benchmark evaluation when a model benefits from memorized evaluation content rather than genuine generalization. Yet contamination is difficult to audit when the exposed content differs in
language from the evaluation
benchmark. We study this failure mode by deliberately exposing four open-weight instruction-tuned LLMs to Arabic translations of MMLU and XQuAD evaluation items at increasing exposure levels, then evaluating them on the original English tasks. This controlled setup is a proxy for contamination rather than a reconstruction of real-world pretraining leakage. We first test two English-centric post-hoc probes, TS-Guessing and Min-K%++, and find that their signals largely disappear under translated exposure: TS-Guessing remains weak except for model-specific positional recall on MMLU, while Min-K%++ stays at or below chance. At the same time, English MMLU performance increases with Arabic exposure, showing that the absence of an English contamination signal does not imply the absence of an exposure effect. We then introduce Translation-Aware Contamination Detection (TACD), a training-data-free diagnostic based on cross-lingual prediction consistency and choice reordering. Cross-lingual consistency is substantially higher than an independence baseline and generally increases relative to the clean condition, although its magnitude is model-dependent and not strictly monotonic. These results show that translation can conceal contamination-related effects from English-only probes and motivate
multilingual diagnostics that are explicitly framed as evidence of contamination-consistent behavior rather than definitive membership tests.
Read the full abstract ▾
Computation and Language
Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala
Minuri Rajapakse and Ruvan Weerasinghe
The performance of
Language Models (LMs) on low-resource, morphologically rich
languages like Sinhala remains largely unexplored, particularly regarding script variation in digital communication. Sinhala exhibits script duality, with Unicode used in formal contexts and Romanized text dominating social media, while mixed-script usage is common in practice. This paper
benchmarks 24 open-source LMs on Unicode, Romanized and mixed-script Sinhala using perplexity evaluation across diverse text sources. Results reveal substantial script sensitivity, with median performance degradation exceeding 300 times from Unicode to Romanized text. Critically, model size shows no correlation with script-handling competence, as smaller models often outperform architectures 28 times larger. Unicode performance strongly predicts mixed-script robustness but not Romanized capability, demonstrating that single-script evaluation substantially underestimates real-world deployment challenges. These findings establish baseline LM capabilities for Sinhala and provide practical guidance for model selection in multi-script low-resource environments.
Read the full abstract ▾
Computation and Language
CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning
Zhiyuan Lu, Chenliang Li, Yingcheng Shi et al. (+3 more)
While large
language models now handle million-
token contexts, their capacity for
reasoning across entire document repositories remains largely untested. Existing
benchmarks are inadequate, as they are mostly limited to single long texts or rely on a "sparse
retrieval" assumption-that answers can be derived from a few relevant chunks. This assumption fails for true corpus-level analysis, where evidence is highly dispersed across hundreds of documents and answers require global integration, comparison, and statistical aggregation. To address this critical gap, we introduce CorpusQA, a new
benchmark scaling up to 10 million
tokens, generated via a novel data synthesis framework. By decoupling
reasoning from textual representation, this framework creates complex, computation-intensive queries with programmatically guaranteed ground-truth answers, challenging systems to perform holistic
reasoning over vast, unstructured text without relying on fallible human annotation. We further demonstrate the utility of our framework beyond evaluation, showing that fine-tuning on our synthesized data effectively enhances an LLM's general long-context
reasoning capabilities. Extensive experiments reveal that even state-of-the-art long-context LLMs struggle as input length increases, and standard
retrieval-augmented generation systems collapse entirely. Our findings indicate that
memory-augmented agentic architectures offer a more robust alternative, suggesting a critical shift is needed from simply extending context windows to developing advanced architectures for global information synthesis.
Read the full abstract ▾
All authors Zhiyuan Lu, Chenliang Li, Yingcheng Shi, Weizhou Shen, Ming Yan, Fei Huang
Computation and Language · Cross-listed from cs.CV
TempViz: On the Evaluation of Temporal Knowledge in Text-to-Image Models
Carolin Holtermann, Nina Krebs and Anne Lauscher
Time alters the visual appearance of entities in our world, like objects, places, and animals. Thus, for accurately generating contextually-relevant images, knowledge and
reasoning about time can be crucial (e.g., for generating a landscape in spring vs. in winter). Yet, although substantial work exists on understanding and improving temporal knowledge in natural
language processing, research on how temporal phenomena appear and are handled in text-to-image (T2I) models remains scarce. We address this gap with TempViz, the first data set to holistically evaluate temporal knowledge in image generation, consisting of 7.9k
prompts and more than 600 reference images. Using TempViz, we study the capabilities of five T2I models across five temporal knowledge categories. Human evaluation shows that temporal competence is generally weak, with no model exceeding 75% accuracy across categories. Towards larger-scale studies, we also examine automated evaluation methods, comparing several established approaches against human judgments. However, none of these approaches provides a reliable assessment of temporal cues - further indicating the pressing need for future research on temporal knowledge in T2I.
Read the full abstract ▾
Computation and Language
The GDN-CC Dataset: Automatic Corpus Clarification for AI-enhanced Democratic Citizen Consultations
Pierre-Antoine Lequeu, Léo Labat, Laurène Cave et al. (+3 more)
LLMs are ubiquitous in modern NLP, and while their applicability extends to texts produced for democratic activities such as online deliberations or large-scale citizen consultations, ethical questions have been raised for their usage as analysis tools. We continue this line of research with two main goals: (a) to develop resources that can help standardize citizen contributions in public forums at the pragmatic level, and make them easier to use in topic modeling and political analysis; (b) to study how well this standardization can reliably be performed by small, open-weights LLMs, i.e. models that can be run locally and transparently with limited resources. Accordingly, we introduce Corpus Clarification as a preprocessing framework for large-scale consultation data that transforms noisy, multi-topic contributions into structured, self-contained argumentative units ready for downstream analysis. We present GDN-CC, a manually-curated
dataset of 1,231 contributions to the French Grand Débat National, comprising 2,285 argumentative units annotated for argumentative structure and manually clarified. We then show that finetuned Small
Language Models match or outperform LLMs on reproducing these annotations, and measure their usability for an opinion clustering task. We finally release GDN-CC-large, an automatically annotated corpus of 240k contributions, the largest annotated democratic consultation
dataset to date.
Read the full abstract ▾
All authors Pierre-Antoine Lequeu, Léo Labat, Laurène Cave, Gaël Lejeune, François Yvon, Benjamin Piwowarski
Computation and Language · Cross-listed from cs.SD
Generative Artificial Intelligence, Musical Heritage and the Construction of Peace Narratives: A Case Study in Mali
Nouhoum Coulibaly, Ousmane Ly, Michael Leventhal et al. (+1 more)
This study explores the capacity of generative artificial intelligence (Gen AI) to contribute to the construction of peace narratives and the revitalization of musical heritage in Mali. The study has been made in a political and social context where inter-community tensions and social fractures motivate a search for new symbolic frameworks for reconciliation. The study empirically explores three questions: (1) how Gen AI can be used as a tool for musical creation rooted in national
languages and traditions; (2) to what extent Gen AI systems enable a balanced hybridization between technological innovation and cultural authenticity; and (3) how AI-assisted musical co-creation can strengthen social cohesion and cultural sovereignty. The experimental results suggest that Gen AI, embedded in a culturally conscious participatory framework, can act as a catalyst for symbolic diplomacy, amplifying local voices instead of standardizing them. However, challenges persist regarding the availability of linguistic corpora, algorithmic censorship, and the ethics of generating compositions derived from copyrighted sources.
Read the full abstract ▾
All authors Nouhoum Coulibaly, Ousmane Ly, Michael Leventhal, Ousmane Goro
Computation and Language
CodeDelegator: Mitigating Context Pollution via Role Separation in Code-as-Action Agents
Tianxiang Fei, Cheng Chen, Yue Pan et al. (+2 more)
Recent advances in large
language models (LLMs) allow
agents to represent actions as executable code, offering greater expressivity than traditional tool-calling. However, real-world tasks often demand both strategic planning and detailed implementation. Using a single
agent for both leads to context pollution from debugging traces and intermediate failures, impairing long-horizon performance. We propose CodeDelegator, a multi-
agent framework that separates planning from implementation via role specialization. A persistent Delegator maintains strategic oversight by decomposing tasks, writing specifications, and monitoring progress without executing code. For each sub-task, a new Coder
agent is instantiated with a clean context containing only its specification, shielding it from prior failures. To coordinate between
agents, we introduce Ephemeral-Persistent State Separation (EPSS), which isolates each Coder's execution state while preserving global coherence, preventing debugging traces from polluting the Delegator's context. Experiments on various
benchmarks demonstrate the effectiveness of CodeDelegator across diverse scenarios.
Read the full abstract ▾
All authors Tianxiang Fei, Cheng Chen, Yue Pan, Mao Zheng, Mingyang Song
Computation and Language
PodBench: A Comprehensive Benchmark for Instruction-Aware Audio-Oriented Podcast Script Generation
Chenning Xu, Mao Zheng, Mingyu Zheng et al. (+1 more)
Podcast script generation requires LLMs to synthesize structured, context-grounded dialogue from diverse inputs, yet systematic evaluation resources for this task remain limited. To bridge this gap, we introduce PodBench, a
benchmark comprising 800 samples with inputs up to 21K
tokens and complex multi-speaker instructions. We propose a multifaceted evaluation framework that integrates quantitative constraints with LLM-based quality assessment. Extensive experiments reveal that while proprietary models generally excel, open-source models equipped with explicit
reasoning demonstrate superior robustness in handling long contexts and multi-speaker coordination compared to standard baselines. However, our analysis uncovers a persistent divergence where high instruction following does not guarantee high content substance. PodBench offers a reproducible testbed to address these challenges in long-form, audio-centric generation.
Read the full abstract ▾
All authors Chenning Xu, Mao Zheng, Mingyu Zheng, Mingyang Song
Computation and Language
Rui Qi, Fengran Mo, Yufeng Chen et al. (+6 more)
Multilingual retrieval-augmented generation (MRAG) requires models to effectively acquire and integrate beneficial external knowledge from
multilingual collections. However, most existing studies employ a unitive process where queries of equivalent semantics across different
languages are processed through a single-turn
retrieval and subsequent optimization. Such a “one-size-fits-all” strategy is often suboptimal in
multilingual settings, as the models occur to knowledge bias and conflict during the interaction with the search engine. To alleviate the issues, we propose LcRL, a
multilingual search-augmented reinforcement learning framework that integrates a
language-coupled Group Relative Policy Optimization into the policy and reward models. We adopt the
language-coupled group sampling in the rollout module to reduce knowledge bias, and regularize an auxiliary anti-consistency penalty in the reward models to mitigate the knowledge conflict. Experimental results demonstrate that LcRL not only achieves competitive performance but is also appropriate for various practical scenarios such as constrained training data and
retrieval over collections encompassing a large number of
languages. Our code is available at https://github.com/Cherry-qwq/LcRL-Open.
Read the full abstract ▾
All authors Rui Qi, Fengran Mo, Yufeng Chen, Xue Zhang, Shuo Wang, Hongliang Li, Jinan Xu, Jian-Yun Nie, Kaiyu Huang
Computation and Language · Cross-listed from cs.LG
What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study
Keyu Lv, Manyi Zhang, Xiaobo Xia et al. (+6 more)
Reasoning models excel at complex tasks such as coding and mathematics, yet their inference is often slow and
token-inefficient. To improve the inference efficiency, post-training quantization (PTQ) usually comes with the cost of large accuracy drops, especially for
reasoning tasks under low-bit settings. In this study, we present a systematic empirical study of quantization-aware training (QAT) for
reasoning models. Our key findings include: (1) Knowledge
distillation is a robust objective for
reasoning models trained via either supervised fine-tuning or reinforcement learning; (2) PTQ provides a strong initialization for QAT, improving accuracy while reducing training cost; (3) Reinforcement learning remains feasible for quantized models given a viable cold start and yields additional gains; and (4) Aligning the PTQ calibration domain with the QAT training domain accelerates convergence and often improves the final accuracy. Finally, we consolidate these findings into an optimized workflow (
Reasoning-QAT), and show that it consistently outperforms state-of-the-art PTQ methods across multiple LLM backbones and
reasoning datasets. For instance, on Qwen3-0.6B, it surpasses GPTQ by 44.53% on MATH-500 and consistently recovers performance in the 2-bit regime.
Read the full abstract ▾
All authors Keyu Lv, Manyi Zhang, Xiaobo Xia, Jingchen Ni, Shannan Yan, Xianzhi Yu, Lu Hou, Chun Yuan, Haoli Bai
Computation and Language · Cross-listed from cs.LG
Strategic Doctrine Language Models (sdLM): A Learning-System Framework for Doctrinal Consistency and Geopolitical Forecasting
Olaf Yunus Laitinen Imanov, Taner Yilmaz and Derya Umut Kulali
We introduce Strategic Doctrine
Language Models (sdLM), a learning-system framework for multi-document strategic
reasoning with doctrinal consistency constraints and calibrated uncertainty. The approach combines multi-document
attention, temporal encoding, and a doctrine-consistency layer to improve long-horizon
forecasting and plan plausibility while reducing severe doctrinal violations. We evaluate sdLM using (i) expert-panel scoring of strategic scenarios (N=47), (ii) doctrine consistency on 336 doctrine publications (12,847 statements), and (iii) geopolitical
forecasting on 127 historical counterfactuals (1945-2020) across 12-60 month horizons. Across these
benchmarks, sdLM achieves higher strategic quality and better calibration than strong general-purpose LLM baselines, and remains competitive with human experts on long-horizon judgments. We further report ablations, scaling trends, and deployment-oriented performance/latency characteristics to clarify which components drive improvements and how they translate to operational settings.
Read the full abstract ▾