Computation and Language
GLM-OCR Technical Report
Shuaiqi Duan, Yadong Xue, Weihan Wang et al. (+20 more)
GLM-OCR is an efficient 0.9B-parameter compact
multimodal model designed for real-world document understanding. It combines a 0.4B-parameter CogViT visual encoder with a 0.5B-parameter GLM
language decoder, achieving a strong balance between computational efficiency and recognition performance. To address the inefficiency of standard autoregressive decoding in deterministic OCR tasks, GLM-OCR introduces a Multi-
Token Prediction (MTP) mechanism that predicts multiple
tokens per step, significantly improving decoding throughput while keeping
memory overhead low through shared parameters. At the system level, a two-stage pipeline is adopted: PP-DocLayout-V3 first performs layout analysis, followed by parallel region-level recognition. Extensive evaluations on public
benchmarks and industrial scenarios show that GLM-OCR achieves competitive or state-of-the-art performance in document parsing, text and formula transcription, table structure recovery, and key information extraction. Its compact architecture and structured generation make it suitable for both resource-constrained edge deployment and large-scale production systems.
Read the full abstract ▾
All authors Shuaiqi Duan, Yadong Xue, Weihan Wang, Zhe Su, Huan Liu, Sheng Yang, Guobing Gan, Guo Wang, Zihan Wang, Shengdong Yan, Dexin Jin, Yuxuan Zhang, Guohong Wen, Yanfeng Wang, Yutao Zhang, Xiaohan Zhang, Wenyi Hong, Yukuo Cen, Da Yin, Bin Chen, Wenmeng Yu, Xiaotao Gu, Jie Tang
Computation and Language
From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers
Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar et al. (+1 more)
Knowledge
distillation (KD) methods are pivotal in compressing large pre-trained
language models into smaller models, ensuring computational efficiency without significantly dropping performance. Traditional KD techniques assume homogeneity in modalities between the teacher (source) and the student (target) models. On the other hand, existing
multimodal knowledge
distillation methods require modality-specific pre-training of the teacher model, which is computationally infeasible in most cases. In this paper, we introduce ARMADA, an efficient cross-modal knowledge
distillation framework designed to transfer knowledge from large
vision-
language models, including black-box models, to
language-only models. Unlike existing KD techniques that rely on the internal structures of
multimodal teachers or require computationally expensive pre-training, ARMADA leverages novel
alignment techniques to distil knowledge without altering the teacher model, ensuring efficiency and scalability. We empirically validate ARMADA on twelve natural
language understanding, eight complex generative
reasoning and five instruction-tuning tasks, demonstrating consistent performance improvements in large models such as DeBERTa-v2-1.4B, OPT-1.3B, LLaMA-{3B, 7B, 8B}. ARMADA achieves up to 3.4% improvement on
language understanding tasks and 2.6% boost in generative
reasoning, all without requiring expensive
multimodal pre-training or fine-tuning of the teacher model. Our findings challenge conventional knowledge
distillation paradigms by demonstrating that even
vision-
language models, despite lacking direct textual understanding, can significantly enhance
language models when distilled appropriately.
Read the full abstract ▾
All authors Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar, Tanmoy Chakraborty
Computation and Language
An Extreme Multi-label Text Classification (XMTC) Library Dataset: What if we took "Use of Practical AI in Digital Libraries" seriously?
Jennifer D'Souza, Sameer Sadruddin, Maximilian Kähler et al. (+5 more)
Subject indexing is vital for discovery but hard to sustain at scale and across
languages. We release a large bilingual (English/German) corpus of catalog records annotated with the Integrated Authority File (GND), plus a machine-actionable GND taxonomy. The resource enables ontology-aware multi-label classification, mapping text to authority terms, and
agent-assisted cataloging with reproducible, authority-grounded evaluation. We provide a brief statistical profile and qualitative error analyses of three systems. We invite the community to assess not only accuracy but usefulness and transparency, toward authority-anchored AI co-pilots that amplify catalogers' work.
Read the full abstract ▾
All authors Jennifer D'Souza, Sameer Sadruddin, Maximilian Kähler, Andrea Salfinger, Luca Zaccagna, Francesca Incitti, Lauro Snidaro, Osma Suominen
Computation and Language
SiDiaC-v.2.0: Sinhala Diachronic Corpus Version 2.0
Nevidu Jayatilleke, Nisansa de Silva, Uthpala Nimanthi et al. (+3 more)
SiDiaC-v.2.0 is the largest comprehensive Sinhala Diachronic Corpus to date, covering a period from 1800 CE to 1955 CE in terms of publication dates, and a historical span from the 5th to the 20th century CE in terms of written dates. The corpus consists of 229k words across 185 literary works that underwent thorough filtering, preprocessing, and copyright compliance checks, followed by extensive post-processing. Additionally, a subset of 59 documents totalling 65k words was annotated based on their written dates. Texts from the National Library of Sri Lanka were selected from the SiDiaC-v.1.0 non-filtered list, which was digitised using Google Document AI OCR. This was followed by post-processing to correct formatting issues, address code-mixing, include special
tokens, and fix malformed
tokens. The construction of SiDiaC-v.2.0 was informed by practices from other corpora, such as FarPaHC, SiDiaC-v.1.0, and CCOHA. This was particularly relevant for syntactic annotation and text normalisation strategies, given the shared characteristics of low-resource
language status between Faroese and the similar cleaning strategies utilised in CCOHA. This corpus is categorised into two layers based on genres: primary and secondary. The primary categorisation is binary, assigning each book to either Non-Fiction or Fiction. The secondary categorisation is more detailed, grouping texts under specific genres such as Religious, History, Poetry,
Language, and Medical. Despite facing challenges due to limited resources, SiDiaC-v.2.0 serves as a comprehensive resource for Sinhala NLP, building upon the work previously done in SiDiaC-v.1.0.
Read the full abstract ▾
All authors Nevidu Jayatilleke, Nisansa de Silva, Uthpala Nimanthi, Gagani Kulathilaka, Azra Safrullah, Johan Sofalas
Computation and Language · Cross-listed from cs.LG
$V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts
Yi-Kai Zhang, Yueqing Sun, Hongyan Hao et al. (+4 more)
In Reinforcement Learning with Verifiable Rewards (RLVR), constructing a robust advantage baseline is critical for policy gradients, effectively guiding the policy model to reinforce desired behaviors. Recent research has introduced Generalist Value Models (such as $V_0$), which achieve pre-trained value estimation by explicitly encoding model capabilities in-context, eliminating the need to synchronously update the value model alongside the policy model. In this paper, we propose $V_{0.5}$, which adaptively fuses the baseline predicted by such value model (acting as a prior) with the empirical mean derived from sparse rollouts. This constructs a robust baseline that balances computational efficiency with extremely low variance. Specifically, we introduce a real-time statistical testing and dynamic budget allocation. This balances the high variance caused by sparse sampling against the systematic bias (or
hallucinations) inherent in the value model's prior. By constructing a hypothesis test to evaluate the prior's reliability in real-time, the system dynamically allocates additional rollout budget on demand. This mechanism minimizes the baseline
estimator's Mean Squared Error (MSE), guaranteeing stable policy gradients, even under extreme sparsity with a group size of 4. Extensive evaluations across six mathematical
reasoning benchmarks demonstrate that $V_{0.5}$ significantly outperforms GRPO and DAPO, achieving faster convergence and over some 10% performance improvement.
Read the full abstract ▾
All authors Yi-Kai Zhang, Yueqing Sun, Hongyan Hao, Qi Gu, Xunliang Cai, De-Chuan Zhan, Han-Jia Ye
Computation and Language · Cross-listed from cs.LG
Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis
Yujie Zheng, Zhuo Li, Shengtao Zhang et al. (+8 more)
Deploying Large
Language Models to data-scarce programming domains poses significant challenges, particularly for kernel synthesis on emerging Domain-Specific Architectures where a "Data Wall" limits available training data. While models excel on data-rich platforms like CUDA, they suffer catastrophic performance drops on data-scarce ecosystems such as NPU programming. To overcome this cold-start barrier without expensive fine-tuning, we introduce EvoKernel, a self-evolving agentic framework that automates the lifecycle of kernel synthesis from initial drafting to continual refining. EvoKernel addresses this by formulating the synthesis process as a
memory-based reinforcement learning task. Through a novel value-driven
retrieval mechanism, it learns stage-specific Q-values that prioritize experiences based on their contribution to the current objective, whether bootstrapping a feasible draft or iteratively refining latency. Furthermore, by enabling cross-task
memory sharing, the
agent generalizes insights from simple to complex operators. By building an NPU variant of KernelBench and evaluating on it, EvoKernel improves frontier models' correctness from 11.0% to 83.0% and achieves a median speedup of 3.60x over initial drafts through iterative refinement. This demonstrates that value-guided experience accumulation allows general-purpose models to master the kernel synthesis task on niche hardware ecosystems. Our official page is available at https://evokernel.zhuo.li.
Read the full abstract ▾
All authors Yujie Zheng, Zhuo Li, Shengtao Zhang, Hanjing Wang, Junjie Sheng, Jiaqian Wang, Junchi Yan, Weinan Zhang, Ying Wen, Bo Tang, Muning Wen
Computation and Language
PivotAttack: Rethinking the Search Trajectory in Hard-Label Text Attacks via Pivot Words
Yuzhi Liang, Shiliang Xiao, Jingsong Wei et al. (+2 more)
Existing hard-label text attacks often rely on inefficient "outside-in" strategies that traverse vast search spaces. We propose PivotAttack, a query-efficient "inside-out" framework. It employs a Multi-Armed Bandit
algorithm to identify Pivot Sets-combinatorial
token groups acting as prediction anchors-and strategically perturbs them to induce label flips. This approach captures inter-word dependencies and minimizes query costs. Extensive experiments across traditional models and Large
Language Models demonstrate that PivotAttack consistently outperforms state-of-the-art baselines in both Attack Success Rate and query efficiency.
Read the full abstract ▾
All authors Yuzhi Liang, Shiliang Xiao, Jingsong Wei, Qiliang Lin, Xia Li
Computation and Language
Konstantin Dobler, Simon Lehnerer, Federico Scozzafava et al. (+2 more)
We present the
Multilingual Reasoning Gym, an extension of
Reasoning Gym (Stojanovski et al., 2025), that procedurally generates verifiable
reasoning problems across 14
languages. We translate templates for 94 tasks with native-speaker validation in 10
languages and targeted code or template adaptations to ensure linguistic naturalness. The
Multilingual Reasoning Gym preserves the core benefits of the procedural generation approach used in the original
Reasoning Gym, such as virtually unlimited problem instance generation and adjustable difficulty, and remains directly usable for Reinforcement Learning from Verifiable Rewards and evaluation settings. Problems in the
Multilingual Reasoning Gym are parallel across
languages, enabling crosslingually parallel data generation at massive scale due to the procedural nature of the environments. We release our implementation to support research into
multilingual reasoning models.
Read the full abstract ▾
All authors Konstantin Dobler, Simon Lehnerer, Federico Scozzafava, Jonathan Janke, Mohamed Ali
Computation and Language
LuxBorrow: From Pompier to Pompjee, Tracing Borrowing in Luxembourgish
Nina Hosseini-Kivanani and Fred Philippy
We present LuxBorrow, a borrowing-first analysis of Luxembourgish (LU) news spanning 27 years (1999-2025), covering 259,305 RTL articles and 43.7M
tokens. Our pipeline combines sentence-level
language identification (LU/DE/FR/EN) with a
token-level borrowing resolver restricted to LU sentences, using lemmatization, a collected loanword registry, and compiled morphological and orthographic rules. Empirically, LU remains the
matrix language across all documents, while
multilingual practice is pervasive: 77.1% of articles include at least one donor
language and 65.4% use three or four. Breadth does not imply intensity: median code-mixing index (CMI) increases from 3.90 (LU+1) to only 7.00 (LU+3), indicating localized insertions rather than balanced bilingual text. Domain and period summaries show moderate but persistent mixing, with CMI rising from 6.1 (1999-2007) to a peak of 8.4 in 2020.
Token-level adaptations total 25,444 instances and exhibit a mixed profile: morphological 63.8%, orthographic 35.9%, lexical 0.3%. The most frequent individual rules are orthographic, such as on->oun and eur->er, while morphology is collectively dominant. Diachronically, code-switching intensifies, and morphologically adapted borrowings grow from a small base. French overwhelmingly supplies adapted items, with modest growth for German and negligible English. We advocate borrowing-centric evaluation, including borrowed
token and type rates, donor
entropy over borrowed items, and assimilation ratios, rather than relying only on document-level mixing indices.
Read the full abstract ▾
Computation and Language
Interpretable Chinese Metaphor Identification via LLM-Assisted MIPVU Rule Script Generation: A Comparative Protocol Study
Weihang Huang and Mengna Liu
Metaphor identification is a foundational task in figurative
language processing, yet most computational approaches operate as opaque classifiers offering no insight into why an expression is judged metaphorical. This interpretability gap is especially acute for Chinese, where rich figurative traditions, absent morphological cues, and limited annotated resources compound the challenge. We present an LLM-assisted pipeline that operationalises four metaphor identification protocols–MIP/MIPVU lexical analysis, CMDAG conceptual-mapping annotation,
emotion-based detection, and simile-oriented identification–as executable, human-auditable rule scripts. Each protocol is a modular chain of deterministic steps interleaved with controlled LLM calls, producing structured rationales alongside every classification decision. We evaluate on seven Chinese metaphor
datasets spanning
token-, sentence-, and span-level annotation, establishing the first cross-protocol comparison for Chinese metaphor identification. Within-protocol evaluation shows Protocol A (MIP) achieves an F1 of 0.472 on
token-level identification, while cross-protocol analysis reveals striking divergence: pairwise Cohen's kappa between Protocols A and D is merely 0.001, whereas Protocols B and C exhibit near-perfect agreement (kappa = 0.986). An interpretability audit shows all protocols achieve 100% deterministic reproducibility, with rationale correctness from 0.40 to 0.87 and editability from 0.80 to 1.00. Error analysis identifies conceptual-domain mismatch and register sensitivity as dominant failure modes. Our results demonstrate that protocol choice is the single largest source of variation in metaphor identification, exceeding model-level variation, and that rule-script architectures achieve competitive performance while maintaining full transparency.
Read the full abstract ▾
Computation and Language
Large Language Models as Annotators for Machine Translation Quality Estimation
Sidi Wang, Sophie Arnoult and Amir Kamran
Large
Language Models (LLMs) have demonstrated excellent performance on Machine Translation Quality Estimation (MTQE), yet their high inference costs make them impractical for direct application. In this work, we propose applying LLMs to generate MQM-style annotations for training a COMET model: following Fernandes et al. (2023), we reckon that segment-level annotations provide a strong rationale for LLMs and are key to good segment-level QE. We propose a simplified MQM scheme, mostly restricted to top-level categories, to guide LLM selection. We present a systematic approach for the development of a GPT-4o-based
prompt, called PPbMQM (
Prompt-Pattern-based-MQM). We show that the resulting annotations correlate well with human annotations and that training COMET on them leads to competitive performance on segment-level QE for Chinese-English and English-German.
Read the full abstract ▾
Computation and Language
Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
Zhipeng Yang, Shu Yang, Lijie Hu et al. (+1 more)
Large
language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level
token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained
attention analysis and show that in-group
attention among characters belonging to the same canonical
token is critical for word recovery: masking such
attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.
Read the full abstract ▾
All authors Zhipeng Yang, Shu Yang, Lijie Hu, Di Wang
Computation and Language
mAceReason-Math: A Dataset of High-Quality Multilingual Math Problems Ready For RLVR
Konstantin Dobler, Simon Lehnerer, Federico Scozzafava et al. (+2 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has been successfully applied to significantly boost the capabilities of pretrained large
language models, especially in the math and logic problem domains. However, current research and available training
datasets remain English-centric. While
multilingual training data and
benchmarks have been created in the past, they were not created with RLVR and current model capability in mind, and their level of difficulty is often too low to provide appropriate training signals for current models. To address this gap, we provide mAceReason-Math, a
dataset of high-quality translations of challenging math problems sourced from a corpus specifically curated for RLVR (AceReason-Math). We further take specific care to clean and improve our translations, resulting in a coverage of 14
languages with more than 10,000 samples per
language. We release the
dataset to facilitate
multilingual RLVR research and benchmarking in the research community.
Read the full abstract ▾
All authors Konstantin Dobler, Simon Lehnerer, Federico Scozzafava, Jonathan Janke, Mohamed Ali
Computation and Language
HeartAgent: An Autonomous Agent System for Explainable Differential Diagnosis in Cardiology
Shuang Zhou, Kai Yu, Song Wang et al. (+11 more)
Heart diseases remain a leading cause of morbidity and mortality worldwide, necessitating accurate and trustworthy differential diagnosis. However, existing artificial intelligence-based diagnostic methods are often limited by insufficient cardiology knowledge, inadequate support for complex
reasoning, and poor interpretability. Here we present HeartAgent, a cardiology-specific
agent system designed to support a reliable and explainable differential diagnosis. HeartAgent integrates customized tools and curated data resources and orchestrates multiple specialized sub-
agents to perform complex
reasoning while generating transparent
reasoning trajectories and verifiable supporting references. Evaluated on the MIMIC
dataset and a private electronic health records cohort, HeartAgent achieved over 36% and 20% improvements over established comparative methods, in top-3 diagnostic accuracy, respectively. Additionally, clinicians assisted by HeartAgent demonstrated gains of 26.9% in diagnostic accuracy and 22.7% in explanatory quality compared with unaided experts. These results demonstrate that HeartAgent provides reliable, explainable, and clinically actionable decision support for cardiovascular care.
Read the full abstract ▾
All authors Shuang Zhou, Kai Yu, Song Wang, Wenya Xie, Zaifu Zhan, Meng-Han Tsai, Yuen-Hei Chung, Shutong Hou, Huixue Zhou, Min Zeng, Bhavadharini Ramu, Lin Yee Chen, Feng Xie, Rui Zhang
Computation and Language
PRISM-$\Delta$: Differential Subspace Steering for Prompt Highlighting in Large Language Models
Yuyao Ge, Shenghua Liu, Yiwei Wang et al. (+5 more)
Prompt highlighting steers a large
language model to prioritize user-specified text spans during generation. A key challenge of existing Key-editing approaches is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-$\Delta$ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance
matrices to maximize discriminative energy while eliminating shared directions. Each
attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four
benchmarks and five models, our methods match or exceed the best existing method on 19 of 20 configurations, with relative gains up to +13.0%, while halving the fluency cost of steering. Our methods also scale to long-context
retrieval, outperforming the best existing method by up to +5.1% relative gain. PRISM-$\Delta$ is compatible with FlashAttention and adds negligible
memory overhead.
Read the full abstract ▾
All authors Yuyao Ge, Shenghua Liu, Yiwei Wang, Baolong Bi, Lingrui Mei, Jiayu Yao, Jiafeng Guo, Xueqi Cheng
Computation and Language · Cross-listed from cs.DB
EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution
Tianshu Zhang, Kun Qian, Siddhartha Sahai et al. (+4 more)
Neural text-to-SQL models, which translate natural
language questions (NLQs) into SQL queries given a database schema, have achieved remarkable performance. However, database schemas frequently evolve to meet new requirements. Such schema
evolution often leads to performance degradation for models trained on static schemas. Existing work either mainly focuses on simply paraphrasing some syntactic or semantic mappings among NLQ, DB and SQL, or lacks a comprehensive and controllable way to investigate the model robustness issue under the schema
evolution, which is insufficient when facing the increasingly complex and rich database schema changes in reality, especially in the LLM era. To address the challenges posed by schema
evolution, we present EvoSchema, a comprehensive
benchmark designed to assess and enhance the robustness of text-to-SQL systems under real-world schema changes. EvoSchema introduces a novel schema
evolution taxonomy, encompassing ten perturbation types across columnlevel and table-level modifications, systematically simulating the dynamic nature of database schemas. Through EvoSchema, we conduct an in-depth evaluation spanning different open source and closed-source LLMs, revealing that table-level perturbations have a significantly greater impact on model performance compared to column-level changes. Furthermore, EvoSchema inspires the development of more resilient text-to-SQL systems, in terms of both model training and database design. The models trained on EvoSchema's diverse schema designs can force the model to distinguish the schema difference for the same questions to avoid learning spurious patterns, which demonstrate remarkable robustness compared to those trained on unperturbed data on average. This
benchmark offers valuable insights into model behavior and a path forward for designing systems capable of thriving in dynamic, real-world environments.
Read the full abstract ▾
All authors Tianshu Zhang, Kun Qian, Siddhartha Sahai, Yuan Tian, Shaddy Garg, Huan Sun, Yunyao Li
Computation and Language · Cross-listed from cs.AI
Emulating Clinician Cognition via Self-Evolving Deep Clinical Research
Ruiyang Ren, Yuhao Wang, Yunsen Liang et al. (+8 more)
Clinical diagnosis is a complex cognitive process, grounded in dynamic cue acquisition and continuous expertise accumulation. Yet most current artificial intelligence (AI) systems are misaligned with this reality, treating diagnosis as single-pass retrospective prediction while lacking auditable mechanisms for governed improvement. We developed DxEvolve, a self-evolving diagnostic
agent that bridges these gaps through an interactive deep clinical research workflow. The framework autonomously requisitions examinations and continually externalizes clinical experience from increasing encounter exposure as diagnostic cognition primitives. On the MIMIC-CDM
benchmark, DxEvolve improved diagnostic accuracy by 11.2% on average over backbone models and reached 90.4% on a reader-study subset, comparable to the clinician reference (88.8%). DxEvolve improved accuracy on an independent external cohort by 10.2% (categories covered by the source cohort) and 17.1% (uncovered categories) compared to the competitive method. By transforming experience into a governable learning asset, DxEvolve supports an accountable pathway for the continual
evolution of clinical AI.
Read the full abstract ▾
All authors Ruiyang Ren, Yuhao Wang, Yunsen Liang, Lan Luo, Jing Liu, Haifeng Wang, Cong Feng, Yinan Zhang, Chunyan Miao, Ji-Rong Wen, Wayne Xin Zhao
Computation and Language
Making Bielik LLM Reason (Better): A Field Report
Adam Trybus, Bartosz Bartnicki and Remigiusz Kinas
This paper presents a research program dedicated to evaluating and advancing the
reasoning capabilities of Bielik, a Polish large
language model. The study describes a number of stages of work: initial benchmarking and creation of evaluation methodology, analyzing of comparative results with other LLMs and outlining of future prospects that take into account the limitations of the analyses conducted so far and aims to keep Bielik in the race give the ever-changing – and competitive – AI landscape.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Reinforcement Learning with Conditional Expectation Reward
Changyi Xiao, Caijun Xu and Yixin Cao
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the
reasoning capabilities of large
language models, particularly in domains such as mathematics where reliable rule-based verifiers can be constructed. However, the reliance on handcrafted, domain-specific verification rules substantially limits the applicability of RLVR to general
reasoning domains with free-form answers, where valid answers often exhibit significant variability, making it difficult to establish complete and accurate rules. To address this limitation, we propose Conditional Expectation Reward (CER), which leverages the large
language model itself as an implicit verifier, and is therefore applicable to general domains and eliminates the need for external verifiers or auxiliary models. CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer. In contrast to rule-based verifiers that yield binary feedback, CER provides a soft, graded reward signal that reflects varying degrees of correctness, making it better suited to tasks where answers vary in correctness. Experimental results demonstrate that CER is effective across a wide range of
reasoning tasks, spanning both mathematical and general domains, indicating that CER serves as a flexible and general verification mechanism. The code is available at https://github.com/changyi7231/CER.
Read the full abstract ▾
Computation and Language
Disentangling Similarity and Relatedness in Topic Models
Hanlin Xiao, Yang Wang, Mauricio A. Álvarez et al. (+1 more)
The recent success of large pre-trained
language models (PLMs) has motivated their integration into topic modeling. However, PLM-augmented topic models differ from classical co-occurrence models such as Latent Dirichlet Allocation (LDA) not only in performance, but also in the type of semantic structure they capture. We formalize this distinction along two psycholinguistic axes: thematic relatedness (dog/bone) and taxonomic similarity (dog/wolf). To measure both axes over topic words, we construct a large synthetic
benchmark of word pairs using LLM-based annotation and train a neural scorer on it. Across multiple corpora and model families, the scorer places different topic-model families at distinct positions within the joint similarity-relatedness space. The two scores further predict downstream task performance: tasks requiring similarity benefit from similarity-rich topics, whereas tasks requiring relatedness benefit from the converse, and excessive emphasis on either axis degrades performance on tasks aligned with the opposing semantic structure. Neither axis is uniformly beneficial. Measuring both therefore provides a practical, model-agnostic diagnostic for evaluating the semantic structure captured by topic models.
Read the full abstract ▾
All authors Hanlin Xiao, Yang Wang, Mauricio A. Álvarez, Rainer Breitling
Computation and Language
Yuji Chen, Alistair Plum, Hansi Hettiarachchi et al. (+4 more)
The goal of news image captioning is to generate captions by integrating news article content with corresponding images, highlighting the relationship between textual context and visual elements. The majority of research on news image captioning focuses on English, primarily because
datasets in other
languages are scarce. To address this limitation, we create the first
multilingual news image captioning
benchmark, MUNIChus, comprising 9
languages, including several low-resource
languages such as Sinhala and Urdu. We evaluate various state-of-the-art neural news image captioning models on MUNIChus and find that news image captioning remains challenging. We also make MUNIChus publicly available with over 20 models already benchmarked. MUNIChus opens new avenues for further advancements in developing and evaluating
multilingual news image captioning models.
Read the full abstract ▾
All authors Yuji Chen, Alistair Plum, Hansi Hettiarachchi, Diptesh Kanojia, Saroj Basnet, Marcos Zampieri, Tharindu Ranasinghe
Computation and Language · Cross-listed from cs.AI
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Zhaowei Zhang, Xiaohan Liu, Xuekai Zhu et al. (+6 more)
Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable success in logical
reasoning tasks, yet whether large
language model (LLM)
alignment requires fundamentally different approaches remains unclear. Given the apparent tolerance for multiple valid responses in moral
reasoning, a natural hypothesis is that
alignment tasks inherently require diversity-seeking distribution-matching
algorithms rather than reward-maximizing policy-based methods. We conduct the first comprehensive empirical study comparing both paradigms on MoReBench. To enable stable RLVR training, we build a rubric-grounded reward pipeline by training a Qwen3-1.7B judge model. Contrary to our hypothesis, we find that distribution-matching approaches do not demonstrate significant advantages over reward-maximizing methods as expected on
alignment tasks. Through semantic visualization mapping high-reward responses to semantic space, we demonstrate that moral
reasoning exhibits more concentrated high-reward distributions than mathematical
reasoning, where diverse solution strategies yield similarly high rewards. This counter-intuitive finding explains why mode-seeking optimization proves equally or more effective for
alignment tasks. Our results suggest that
alignment tasks do not inherently require diversity-preserving
algorithms, and standard reward-maximizing RLVR methods can effectively transfer to moral
reasoning without explicit diversity mechanisms.
Read the full abstract ▾
All authors Zhaowei Zhang, Xiaohan Liu, Xuekai Zhu, Junchao Huang, Ceyao Zhang, Zhiyuan Feng, Yaodong Yang, Xiaoyuan Yi, Xing Xie
Computation and Language
End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering
Nhi Dang, Tung Le and Huy Tien Nguyen
Large
language models (LLMs) combined with
retrieval augmented generation have enabled the deployment of domain-specific chatbots, but these systems remain prone to generating unsupported or incorrect answers. Reliable evaluation is therefore critical, yet manual review is costly and existing frameworks often depend on curated test sets and static metrics, limiting scalability. We propose an end-to-end automatic evaluator designed to substantially reduce human effort. Our system generates Q&A pairs directly from the underlying knowledge base, uses LLMs to judge chatbot responses against reference answers, and applies confidence-based filtering to highlight uncertain cases. Applied to a Vietnamese news
dataset, the evaluator achieves high agreement with human judgments while significantly lowering review overhead. The framework is modular and
language-agnostic, making it readily adaptable to diverse domains. This work introduces a practical, scalable solution for evaluating chatbots with minimal reliance on manual intervention.
Read the full abstract ▾
Computation and Language
Automatic End-to-End Data Integration using Large Language Models
Aaron Steiner and Christian Bizer
Designing data integration pipelines typically requires substantial manual effort from data engineers to configure pipeline components and label training data. While LLMs have shown promise in handling individual steps of the integration process, their potential to replace all human input across end-to-end data integration pipelines has not been investigated. As a step toward exploring this potential, we present an automatic data integration pipeline that uses GPT-5.2 to generate all artifacts required to adapt the pipeline to specific use cases. These artifacts are schema mappings, value mappings for data normalization, training data for entity matching, and validation data for selecting conflict resolution heuristics in data fusion. We compare the performance of this LLM-based pipeline to the performance of human-designed pipelines along three case studies requiring the integration of
video game, music, and company related data. Our experiments show that the LLM-based pipeline is able to produce similar results, for some tasks even better results, as the human-designed pipelines. End-to-end, the human and the LLM pipelines produce integrated
datasets of comparable size and density. Having the LLM configure the pipelines costs approximately \$10 per case study, which represents only a small fraction of the cost of having human data engineers perform the same tasks.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
Zichao Li, Jie Lou, Fangchen Dong et al. (+8 more)
Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient
reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR$^3$), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR$^3$~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines.
Read the full abstract ▾
All authors Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang, Le Sun, Yaojie Lu, Xing Yu
Computation and Language
AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations
Dimosthenis Athanasiou, Maria Lymperaiou, Giorgos Filandrianos et al. (+2 more)
We present the AILS-NTUA system for SemEval-2026 Task 8 (MTRAGEval), addressing all three subtasks of multi-turn
retrieval-augmented generation: passage
retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C). Our unified architecture is built on two principles: (i) a query-diversity-over-retriever-diversity strategy, where five complementary LLM-based query reformulations are issued to a single corpus-aligned sparse retriever and fused via variance-aware nested Reciprocal Rank Fusion; and (ii) a multistage generation pipeline that decomposes grounded generation into evidence span extraction, dual-candidate drafting, and calibrated multi-judge selection. Our system ranks 1st in Task A (nDCG@5: 0.5776, +20.5% over the strongest baseline) and 2nd in Task B (HM: 0.7698). Empirical analysis shows that query diversity over a well-aligned retriever outperforms heterogeneous retriever ensembling, and that answerability calibration-rather than
retrieval coverage-is the primary bottleneck in end-to-end performance.
Read the full abstract ▾
All authors Dimosthenis Athanasiou, Maria Lymperaiou, Giorgos Filandrianos, Athanasios Voulodimos, Giorgos Stamou
Computation and Language · Cross-listed from cs.AI
IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs
Chuan Guo, Juan Felipe Ceron Uribe, Sicheng Zhu et al. (+10 more)
Instruction hierarchy (IH) defines how LLMs prioritize system, developer, user, and tool instructions under conflict, providing a concrete, trust-ordered policy for resolving instruction conflicts. IH is key to defending against jailbreaks, system
prompt extractions, and agentic
prompt injections. However, robust IH behavior is difficult to train: IH failures can be confounded with instruction-following failures, conflicts can be nuanced, and models can learn shortcuts such as overrefusing. We introduce IH-Challenge, a reinforcement learning training
dataset, to address these difficulties. Fine-tuning GPT-5-Mini on IH-Challenge with online
adversarial example generation improves IH robustness by +10.0% on average across 16 in-distribution, out-of-distribution, and human red-teaming
benchmarks (84.1% to 94.1%), reduces unsafe behavior from 6.6% to 0.7% while improving helpfulness on general safety evaluations, and saturates an internal static agentic
prompt injection evaluation, with minimal capability regression. We release the IH-Challenge
dataset (https://huggingface.co/
datasets/openai/ih-challenge) to support future research on robust instruction hierarchy.
Read the full abstract ▾
All authors Chuan Guo, Juan Felipe Ceron Uribe, Sicheng Zhu, Christopher A. Choquette-Choo, Steph Lin, Nikhil Kandpal, Milad Nasr, Rai, Sam Toyer, Miles Wang, Yaodong Yu, Alex Beutel, Kai Xiao
Computation and Language
Safe and Scalable Web Agent Learning via Recreated Websites
Hyungjoo Chae, Jungsoo Park and Alan Ritter
Training autonomous web
agents is fundamentally limited by the environments they learn from: real-world websites are unsafe to explore, hard to reset, and rarely provide verifiable feedback. We propose VeriEnv, a framework that treats
language models as environment creators, automatically cloning real-world websites into fully executable, verifiable synthetic environments. By exposing controlled internal access via a Python SDK, VeriEnv enables
agents to self-generate tasks with deterministic, programmatically verifiable rewards, eliminating reliance on heuristic or LLM-based judges. This design decouples
agent learning from unsafe real-world interaction while enabling scalable self-
evolution through environment expansion. Through experiments on web
agent benchmarks, we show that
agents trained with VeriEnv generalize to unseen websites, achieve site-specific mastery through self-evolving training, and benefit from scaling the number of training environments. Code and resources will be released at https://github.com/kyle8581/VeriEnv upon acceptance.
Read the full abstract ▾
Computation and Language
Coverage-Controlled Preference Mining from Noisy Claim Verification for Evidence-Grounded Generation
Weixin Liu, Congning Ni, Qingyuan Song et al. (+4 more)
Evidence-grounded generation produces summaries whose claims should be supported by supplied evidence, but claim-level verifiers provide noisy feedback and can reward models that simply say less. We study this problem in clinical Brief Hospital Course summarization, where outputs must remain grounded in patient-specific EHR evidence. We introduce VERI-DPO, a preference-mining framework that converts noisy claim verification into coverage-controlled summary-level preferences. For each evidence-window
prompt, VERI-DPO samples multiple candidate summaries, decomposes them into claims, verifies each claim against patient evidence, and forms a preference pair only when the chosen summary has better aggregate verifier-estimated support while retaining comparable verifiable content. Standard Direct Preference Optimization then distills these pairs into a single-sample policy, avoiding inference-time reranking. On patient-disjoint MIMIC-III-Ext-VeriFact-BHC test data, VERI-DPO reduces Not Supported rates from 10.7% to 1.9% under the mining verifier and from 11.6% to 6.4% under a separately prompted GPT-4o judge. In 100 blinded pairwise assessments by two domain researchers, VERI-DPO is preferred over the base model 56 times versus 18 times for factual faithfulness. In a locked zero-shot MIMIC-IV-Ext-BHC transfer test with 1,000 patients and no model adaptation, VERI-DPO lowers Not Supported rates with nearly unchanged scored-claim counts. Multi-seed ablations show that verifier-guided pair construction drives the gains, while coverage and anti-degeneration controls prevent apparent factuality improvements from coming from shorter or less checkable outputs.
Read the full abstract ▾
All authors Weixin Liu, Congning Ni, Qingyuan Song, Susannah L. Rose, Murat Kantarcioglu, Bradley A. Malin, Zhijun Yin
Computation and Language
Human-AI Co-reasoning for Clinical Diagnosis with Evidence-Integrated Language Agent
Zhongzhen Huang, Yan Ling, Hong Chen et al. (+7 more)
We present PULSE, a medical
reasoning agent that combines a domain-tuned large
language model with scientific literature
retrieval to support diagnostic decision-making in complex real-world cases. To evaluate its capabilities, we curated a
benchmark of 82 authentic endocrinology case reports encompassing a broad spectrum of disease types and incidence levels. In controlled experiments, we compared PULSE's performance against physicians with varying levels of expertise-from residents to senior specialists-and examined how AI assistance influenced human diagnostic
reasoning. PULSE attained expert-competitive accuracy, outperforming residents and junior specialists while matching senior specialist performance at both Top@1 and Top@4 thresholds. Unlike physicians, whose accuracy declined with disease rarity, PULSE maintained stable performance across incidence tiers. The
agent also exhibited adaptive
reasoning, increasing output length with case difficulty in a manner analogous to the longer deliberation observed among expert clinicians. When used collaboratively, PULSE enabled physicians to correct initial errors and broaden diagnostic hypotheses, but also introduced risks of automation bias. The study explores both serial and concurrent collaboration workflows, revealing that PULSE offers robust support across common and rare presentations. These findings underscore both the promise and the limitations of
language model-based
agents in clinical diagnosis, and offer a framework for evaluating their role in real-world decision-making.
Read the full abstract ▾
All authors Zhongzhen Huang, Yan Ling, Hong Chen, Ye Feng, Li Wu, Linjie Mu, Shaoting Zhang, Xiaofan Zhang, Kun Qian, Xiaomu Li