Computation and Language
Towards Mining Effective Pedagogical Strategies from Learner-LLM Educational Dialogues
Liqun He, Manolis Mavrikis and Mutlu Cukurova
Dialogue plays a crucial role in educational settings, yet existing evaluation methods for educational applications of large
language models (LLMs) primarily focus on technical performance or learning outcomes, often neglecting
attention to learner-LLM interactions. To narrow this gap, this AIED Doctoral Consortium paper presents an ongoing study employing a dialogue analysis approach to identify effective pedagogical strategies from learner-LLM dialogues. The proposed approach involves dialogue data collection, dialogue act (DA) annotation, DA pattern mining, and predictive model building. Early insights are outlined as an initial step toward future research. The work underscores the need to evaluate LLM-based educational applications by focusing on dialogue dynamics and pedagogical strategies.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
LILO: Bayesian Optimization with Natural Language Feedback
Katarzyna Kobalczyk, Zhiyuan Jerry Lin, Benjamin Letham et al. (+3 more)
Many real-world optimization problems are guided by complex, subjective preferences that are difficult to express as explicit closed-form objectives. In response, we introduce
Language-in-the-Loop Optimization (LILO), a
Bayesian optimization (BO) framework that employs a large
language model (LLM) to translate free-form natural
language feedback and prior knowledge from a decision maker into structured preference signals, going beyond the restrictive scalar or pairwise feedback formats typically assumed in preferential BO. The LLM-derived preferences are integrated by a Gaussian process proxy model, enabling principled acquisition-driven exploration with calibrated uncertainty. By placing the LLM in a supporting role rather than as the optimizer itself, LILO preserves the sample efficiency and stability of BO while providing a flexible and expressive feedback interface. Across synthetic and real-world
benchmarks, LILO consistently outperforms both conventional preference-based BO methods and LLM-only optimizers, with particularly strong gains in feedback-limited regimes.
Read the full abstract ▾
All authors Katarzyna Kobalczyk, Zhiyuan Jerry Lin, Benjamin Letham, Zhuokai Zhao, Maximilian Balandat, Eytan Bakshy
Computation and Language · Cross-listed from cs.SD
DELULU: Discriminative Embedding Learning Using Latent Units for Speaker-Aware Self-Trained Speech Foundational Model
Massa Baali, Rita Singh and Bhiksha Raj
Self-supervised
speech models have achieved remarkable success on content-driven tasks, yet they remain limited in capturing speaker-discriminative features critical for verification, diarization, and profiling applications. We introduce DELULU, a speaker-aware self-trained foundational model that addresses this limitation by incorporating speaker-informed structure into pseudo-label generation. DELULU leverages frame-level
embeddings from ReDimNet, a state-of-the-art speaker verification model, to guide k-means clustering during pre-training, introducing a speaker-discriminative inductive bias that aligns representation learning with speaker identity. DELULU significantly outperforms prior SSL models across a range of speaker-centric tasks, achieving up to 62% relative improvement in equal error rate (EER) for speaker verification and consistent gains on zero-shot profiling tasks including gender, age, accent, and speaker counting; notably surpassing even its teacher model on zero-shot evaluations. Our findings demonstrate that DELULU is a strong universal encoder for speaker-aware
speech processing, enabling superior performance without task-specific fine-tuning.
Read the full abstract ▾
Computation and Language
Qomhra: A Bilingual Irish and English Large Language Model
Joseph McInerney, Khanh-Tung Tran, Liam Lonergan et al. (+3 more)
Large
language model (LLM) research and development has overwhelmingly focused on the world's major
languages, leading to under-representation of low-resource
languages such as Irish. This paper introduces \textbf{Qomhrá, a bilingual Irish and English LLM, developed under extremely low-resource constraints. A complete pipeline is outlined spanning bilingual continued pre-training, instruction tuning, and the synthesis of human preference data for future
alignment training. We focus on the lack of scalable methods to create human preference data by proposing a novel method to synthesise such data by prompting an LLM to generate “accepted” and “rejected” responses, which we validate as aligning with L1 Irish speakers. To select an LLM for synthesis, we evaluate the top closed-weight LLMs for Irish
language generation performance. Gemini-2.5-Pro is ranked highest by L1 and L2 Irish-speakers, diverging from LLM-as-a-judge ratings, indicating a misalignment between current LLMs and the Irish-
language community. Subsequently, we leverage Gemini-2.5-Pro to translate a large scale English-
language instruction tuning
dataset to Irish and to synthesise a first-of-its-kind Irish-
language human preference
dataset. We comprehensively evaluate Qomhrá across several
benchmarks, testing translation, gender understanding, topic identification, and world knowledge; these evaluations show gains of up to 29% in Irish and 44% in English compared to the existing open-source Irish LLM baseline, UCCIX. The results of our framework provide insight and guidance to developing LLMs for both Irish and other low-resource
languages.
Read the full abstract ▾
All authors Joseph McInerney, Khanh-Tung Tran, Liam Lonergan, Ailbhe Ní Chasaide, Neasa Ní Chiaráin, Barry Devereux
Computation and Language · Cross-listed from cs.AI
LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena
Qingchuan Yang, Simon Mahns, Sida Li et al. (+3 more)
Forecasting is not only a fundamental intellectual pursuit but also is of significant importance to societal systems such as finance and economics. With the rapid advances of large
language models (LLMs) trained on Internet-scale data, it raises the promise of employing LLMs to forecast real-world future events, an emerging paradigm we call "LLM-as-a-Prophet". This paper systematically investigates such predictive intelligence of LLMs. To this end, we build Prophet Arena, a general evaluation
benchmark that continuously collects live
forecasting tasks and decomposes each task into distinct pipeline stages, in order to support our controlled and large-scale experimentation. Our comprehensive evaluation reveals that many LLMs already exhibit impressive
forecasting capabilities, reflected in, e.g., their small calibration errors, consistent prediction confidence and promising
market returns. However, we also uncover key bottlenecks towards achieving superior predictive intelligence via LLM-as-a-Prophet, such as LLMs' inaccurate event recalls, misunderstanding of data sources and slower information aggregation compared to
markets when resolution nears.
Read the full abstract ▾
All authors Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, Haifeng Xu
Computation and Language
Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models
Yuefeng Peng, Parnian Afshar, Megan Ganji et al. (+4 more)
Large
language models may encode sensitive information or outdated knowledge that needs to be removed, to ensure responsible and compliant model responses. Unlearning has emerged as an efficient alternative to full retraining, aiming to remove specific knowledge while preserving overall model utility. Existing evaluations of unlearning methods focus on (1) the extent of forgetting of the target knowledge (forget set) and (2) maintaining performance on the retain set (i.e., utility). However, these evaluations overlook an important usability aspect: users may still want the model to leverage the removed information if it is re-introduced in the
prompt. In a systematic evaluation of six state-of-the-art unlearning methods, we find that they consistently impair such contextual utility. To address this, we augment unlearning objectives with a plug-in term that preserves the model's ability to use forgotten knowledge when it is present in context. Extensive experiments demonstrate that our approach restores contextual utility to near original levels while still maintaining effective forgetting and retain-set utility.
Read the full abstract ▾
All authors Yuefeng Peng, Parnian Afshar, Megan Ganji, Thomas Butler, Amir Houmansadr, Mingxian Wang, Dezhi Hong
Computation and Language
LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysis
Huiyuan Xie, Chenyang Li, Huining Zhu et al. (+4 more)
Legal
reasoning is a fundamental component of legal analysis and decision-making. Existing computational approaches to legal
reasoning predominantly rely on generic
reasoning frameworks such as syllogism, which do not comprehensively examine the nuanced process of legal
reasoning. Moreover, current research has largely focused on criminal cases, with insufficient modeling for civil cases. In this work, we present a novel framework to explicitly model legal
reasoning in the analysis of Chinese tort-related civil cases. We first operationalize the legal
reasoning process in tort analysis into the three-module LexChain framework, with each module consisting of multiple finer-grained sub-steps. Informed by the LexChain framework, we introduce the task of tort legal
reasoning and construct an evaluation
benchmark to systematically assess the critical steps within analytical
reasoning chains for tort analysis. Leveraging this
benchmark, we evaluate existing large
language models for their legal
reasoning ability in civil tort contexts. Our results indicate that current models still fall short in accurately handling crucial elements of tort legal
reasoning. Furthermore, we introduce several baseline approaches that explicitly incorporate LexChain-style
reasoning through prompting or post-training. The proposed baselines achieve significant improvements in tort-related legal
reasoning and generalize well to related legal analysis tasks, demonstrating the value of explicitly modeling legal
reasoning chains to enhance the
reasoning capabilities of
language models.
Read the full abstract ▾
All authors Huiyuan Xie, Chenyang Li, Huining Zhu, Chubin Zhang, Yuxiao Ye, Zhenghao Liu, Zhiyuan Liu
Computation and Language · Cross-listed from cs.AI
Amir Jalilifard, Anderson de Rezende Rocha and Marcos Medeiros Raimundo
Effective code generation with
language models hinges on two critical factors: accurately understanding the intent of the
prompt and generating code that applies algorithmic
reasoning to produce correct solutions capable of passing diverse test cases while adhering to the syntax of the target programming
language. Unlike other
language tasks, code generation requires more than accurate
token prediction; it demands comprehension of solution-level and structural relationships rather than merely generating the most likely
tokens. very large
language model (VLLM) are capable of generating detailed steps toward the correct solution of complex tasks where
reasoning is crucial in solving the problem. Such
reasoning capabilities may be absent in smaller
language models. Therefore, in this work, we distill the
reasoning capabilities of a VLLM into a smaller, more efficient model that is faster and cheaper to deploy. Our approach trains the model to emulate the
reasoning and problem-solving abilities of the VLLM by learning to identify correct solution pathways and establishing a structural correspondence between problem definitions and potential solutions through a novel method of structure-aware loss optimization. This enables the model to transcend
token-level generation and to deeply grasp the overarching structure of solutions for given problems. Experimental results show that our fine-tuned model, developed through a cheap and simple to implement process, significantly outperforms our baseline model in terms of pass@1, average data flow, and average syntax match metrics across the MBPP, MBPP Plus, and HumanEval
benchmarks.
Read the full abstract ▾
Computation and Language
HGAdapter: Hypergraph-based Adapters in Language Models for Code Summarization and Clone Detection
Guang Yang and Yujie Zhu
Pre-trained
language models (PLMs) are increasingly being applied to code-related tasks. Although PLMs have achieved good results, they do not take into account potential high-order data correlations within the code. We propose three types of high-order correlations in code
tokens, i.e. abstract syntax tree family correlation, lexical correlation, and line correlation. We design a
tokens and hyperedges generator to capture these high-order data correlations. We improve the architecture of hypergraph neural networks and combine it with adapter tuning to propose a novel hypergraph-based adapter (HGAdapter) to fine-tune PLMs. HGAdapter can encode high-order data correlations and is allowed to be inserted into various PLMs to enhance performance. Experiments were conducted on several public
datasets, including six
languages of code summarization and code clone detection tasks. Our methods improved the performance of PLMs in
datasets to varying degrees. Experimental results validate the introduction of high-order data correlations that contribute to improved effectiveness.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
MERIT: Modular Framework for Multimodal Misinformation Detection with Web-Grounded Reasoning
Mir Nafis Sharear Shopnil, Sharad Duwal, Abhishek Tyagi et al. (+1 more)
We present MERIT, an inference-time modular framework for
multimodal misinformation detection that decomposes verification into four specialized modules: visual forensics, cross-modal
alignment,
retrieval-augmented claim verification, and calibrated judgment. On MMFakeBench, MERIT with GPT-4o-mini achieves 81.65% F1, outperforming all reported zero-shot baselines including GPT-4V with MMD-
Agent (74.0% F1). A controlled same-model evaluation confirms gains stem from architectural design: MERIT achieves 6.14 points higher misinformation recall than MMD-
Agent under identical model conditions, with per-class gains of +18.0 on visual distortion and +5.33 on textual distortion. Ablation studies reveal non-overlapping module specialization, where removing any module disproportionately degrades its target category while leaving others intact. Test set evaluation on 5,000 samples confirms generalization within 0.21 F1 points of validation results. The framework operates with any instruction-following
vision-
language model and produces citation-linked rationales for human review.
Read the full abstract ▾
All authors Mir Nafis Sharear Shopnil, Sharad Duwal, Abhishek Tyagi, Adiba Mahbub Proma
Computation and Language
Collin Zhang, Fei Huang, Chenhan Yuan et al. (+1 more)
Large
language models (LLMs) often experience
language confusion, which is the unintended mixing of
languages during text generation. Current solutions to this problem either necessitate model retraining or cannot differentiate between harmful confusion and acceptable code-switching. This paper introduces the
Language Confusion Gate (LCG), a lightweight, plug-in solution that filters
tokens during decoding without altering the base LLM. The LCG is trained using norm-adjusted self-
distillation to predict appropriate
language families and apply masking only when needed. Our method is based on the findings that
language confusion is infrequent, correct-
language tokens are usually among the top predictions, and output
token embedding norms are larger for high-resource
languages, which biases sampling. When evaluated across various models, including Qwen3, GPT-OSS, Gemma3, Llama3.1, LCG decreases
language confusion significantly, often by an order of magnitude, without negatively impacting task performance. Code is available at https://github.com/collinzrj/
language_confusion_gate.
Read the full abstract ▾
All authors Collin Zhang, Fei Huang, Chenhan Yuan, Junyang Lin
Computation and Language
When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity
Nisrine Rair, Alban Goupil, Valeriu Vrabie et al. (+1 more)
Language models are often evaluated with scalar metrics like accuracy, but such measures fail to capture how models internally represent ambiguity, especially when human annotators disagree. We propose a topological perspective to analyze how fine-tuned models encode ambiguity and more generally instances. Applied to RoBERTa-Large on the MD-Offense
dataset, Mapper, a tool from topological data analysis, reveals that fine-tuning restructures
embedding space into modular, non-convex regions aligned with model predictions, even for highly ambiguous cases. Over $98\%$ of connected components exhibit $\geq 90\%$ prediction purity, yet
alignment with ground-truth labels drops in ambiguous data, surfacing a hidden tension between structural confidence and label uncertainty. Unlike traditional tools such as PCA or UMAP, Mapper captures this geometry directly uncovering decision regions, boundary collapses, and overconfident clusters. Our findings position Mapper as a powerful diagnostic tool for understanding how models resolve ambiguity. Beyond visualization, it also enables topological metrics that may inform proactive modeling strategies in subjective NLP tasks.
Read the full abstract ▾
All authors Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel Chochoy
Computation and Language
OncoReason: Structuring Clinical Reasoning in LLMs for Robust and Interpretable Survival Prediction
Raghu Vamshi Hemadri, Geetha Krishna Guruju, Kristi Topollai et al. (+1 more)
Predicting cancer treatment outcomes requires models that are both accurate and interpretable, particularly in the presence of heterogeneous clinical data. While large
language models (LLMs) have shown strong performance in biomedical NLP, they often lack structured
reasoning capabilities critical for high-stakes decision support. We present a unified, multi-task learning framework that aligns autoregressive LLMs with clinical
reasoning for outcome prediction on the MSK-CHORD
dataset. Our models are trained to jointly perform binary survival classification, continuous survival time regression, and natural
language rationale generation. We evaluate three
alignment strategies: (1) standard supervised fine-tuning (SFT), (2) SFT with Chain-of-Thought (CoT) prompting to elicit step-by-step
reasoning, and (3) Group Relative Policy Optimization (GRPO), a reinforcement learning method that aligns model outputs to expert-derived
reasoning trajectories. Experiments with LLaMa3-8B and Med42-8B backbones demonstrate that CoT prompting improves F1 by +6.0 and reduces MAE by 12%, while GRPO achieves state-of-the-art interpretability and predictive performance across BLEU, ROUGE, and BERTScore. We further show that existing biomedical LLMs often fail to produce valid
reasoning traces due to architectural constraints. Our findings underscore the importance of
reasoning-aware
alignment in multi-task clinical modeling and set a new
benchmark for interpretable, trustworthy LLMs in precision oncology.
Read the full abstract ▾
All authors Raghu Vamshi Hemadri, Geetha Krishna Guruju, Kristi Topollai, Anna Ewa Choromanska
Computation and Language
SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
Tiancheng Hu, Joachim Baumann, Lorenzo Lupo et al. (+3 more)
Large
language model (LLM) simulations of human behavior have the potential to revolutionize the social and behavioral sciences, if and only if they faithfully reflect real human behaviors. Current evaluations of simulation fidelity are fragmented, based on bespoke tasks and metrics, creating a patchwork of incomparable results. To address this, we introduce SimBench, the first large-scale, standardized
benchmark for a robust, reproducible science of LLM simulation. By unifying 20 diverse
datasets covering tasks from moral decision-making to economic choice across a large global participant pool, SimBench provides the necessary foundation to ask fundamental questions about when, how, and why LLM simulations succeed or fail. We show that the best LLMs today achieve meaningful but modest simulation fidelity (score: 40.80/100), with performance scaling log-linearly with model size but not with increased inference-time compute. We discover an
alignment-simulation tradeoff: instruction tuning improves performance on low-
entropy (consensus) questions but degrades it on high-
entropy (diverse) ones. Models particularly struggle when simulating specific demographic groups. Finally, we demonstrate that simulation ability correlates most strongly with knowledge-intensive
reasoning (MMLU-Pro, r = 0.939). By making progress measurable, we aim to accelerate the development of more faithful LLM simulators.
Read the full abstract ▾
All authors Tiancheng Hu, Joachim Baumann, Lorenzo Lupo, Nigel Collier, Dirk Hovy, Paul Röttger
Computation and Language
Annotation-Efficient Universal Honesty Alignment
Shiyu Ni, Keping Bi, Jiafeng Guo et al. (+4 more)
Honesty
alignment-the ability of large
language models (LLMs) to recognize their knowledge boundaries and express calibrated confidence-is essential for trustworthy deployment. Existing methods either rely on training-free confidence estimation (e.g.,
token probabilities, self-consistency) or training-based calibration with correctness annotations. While effective, achieving universal honesty
alignment with training-based calibration requires costly, large-scale labeling. To support annotation-efficient training, we introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that first elicits internal confidence using inexpensive self-consistency supervision, then calibrates this confidence with a small set of correctness annotations. To support a large-scale study, we release HonestyBench, a
benchmark covering ten free-form QA
datasets with 560k training and 70k evaluation instances annotated with correctness and self-consistency signals. Experiments show that EliCal achieves near-optimal
alignment with only 1k correctness annotations (0.18% of full supervision) and better
alignment performance on unseen MMLU tasks than the calibration-only baseline, offering a scalable solution toward universal honesty
alignment in LLMs.
Read the full abstract ▾
All authors Shiyu Ni, Keping Bi, Jiafeng Guo, Minghao Tang, Jingtong Wu, Zengxin Han, Xueqi Cheng
Computation and Language
Lingua Custodi's participation at the WMT 2025 Terminology shared task
Jingshu Liu, Raheel Qader, Gaëtan Caillaut et al. (+1 more)
While BERT is an effective method for learning monolingual sentence
embeddings for semantic similarity and
embedding based transfer learning BERT based cross-lingual sentence
embeddings have yet to be explored. We systematically investigate methods for learning
multilingual sentence
embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked
language modeling (MLM), translation
language modeling (TLM), dual encoder translation ranking, and additive margin softmax. We show that introducing a pre-trained
multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text
retrieval accuracy over 112
languages on Tatoeba, well above the 65.5 achieved by
LASER, while still performing competitively on monolingual transfer learning
benchmarks. Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best
multilingual sentence
embedding model for 109+
languages at https://tfhub.dev/google/LaBSE.
Read the full abstract ▾
All authors Jingshu Liu, Raheel Qader, Gaëtan Caillaut, Mariam Nakhlé
Computation and Language
Deep Self-Evolving Reasoning
Zihan Liu, Shun Zheng, Xumeng Wen et al. (+3 more)
Long-form chain-of-thought
reasoning has become a cornerstone of advanced
reasoning in large
language models. While recent verification-refinement frameworks have enabled proprietary models to solve Olympiad-level problems, their effectiveness hinges on strong, reliable verification and correction capabilities, which remain fragile in open-weight, smaller-scale models. This work demonstrates that even with weak verification and refinement capabilities on hard tasks, the
reasoning limits of such models can be substantially extended through a probabilistic paradigm we call Deep Self-Evolving
Reasoning (DSER). We conceptualize iterative
reasoning as a Markov chain, where each step represents a
stochastic transition in the solution space. The key insight is that convergence to a correct solution is guaranteed as long as the probability of improvement marginally exceeds that of degradation. By running multiple long-horizon, self-evolving processes in parallel, DSER amplifies these small positive tendencies, enabling the model to asymptotically approach correct answers. Empirically, we apply DSER to the DeepSeek-R1-0528-Qwen3-8B model. On the challenging AIME 2024-2025
benchmark, DSER solves 5 out of 9 previously unsolvable problems and boosts overall performance, enabling this compact model to surpass the single-turn accuracy of its 600B-parameter teacher through majority voting. Beyond its immediate utility for test-time scaling, the DSER framework serves to diagnose the fundamental limitations of current open-weight reasoners. By clearly delineating their shortcomings in self-verification, refinement, and stability, our findings establish a clear research agenda for developing next-generation models with powerful, intrinsic self-evolving capabilities.
Read the full abstract ▾
All authors Zihan Liu, Shun Zheng, Xumeng Wen, Yang Wang, Jiang Bian, Mao Yang
Computation and Language
Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
Yihong Tang, Kehai Chen, Liang Yue et al. (+11 more)
With the rise of large
language models (LLMs), LLM
agents capable of autonomous
reasoning, planning, and executing complex tasks have become a frontier in artificial intelligence. However, how to translate the research on general
agents into productivity that drives industry transformations remains a significant challenge. To address this, this paper systematically reviews the technologies, applications, and evaluation methods of industry
agents based on LLMs. Using an industry
agent capability maturity framework, it outlines the
evolution of
agents in industry applications, from "process execution systems" to "adaptive social systems." First, we examine the three key technological pillars that support the advancement of
agent capabilities:
Memory, Planning, and Tool Use. We discuss how these technologies evolve from supporting simple tasks in their early forms to enabling complex autonomous systems and collective intelligence in more advanced forms. Then, we provide an overview of the application of industry
agents in real-world domains such as digital engineering, scientific discovery,
embodied intelligence, collaborative business execution, and complex system simulation. Additionally, this paper reviews the evaluation
benchmarks and methods for both fundamental and specialized capabilities, identifying the challenges existing evaluation systems face regarding authenticity, safety, and industry specificity. Finally, we focus on the practical challenges faced by industry
agents, exploring their capability boundaries, developmental potential, and governance issues in various scenarios, while providing insights into future directions. By combining technological
evolution with industry practices, this review aims to clarify the current state and offer a clear roadmap and theoretical foundation for understanding and building the next generation of industry
agents.
Read the full abstract ▾
All authors Yihong Tang, Kehai Chen, Liang Yue, Jinxin Fan, Caishen Zhou, Xiaoguang Li, Yuyang Zhang, Mingming Zhao, Shixiong Kai, Kaiyang Guo, Xingshan Zeng, Wenjing Cun, Lifeng Shang, Min Zhang
Computation and Language
DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning
Yongxin He, Shan Zhang, Yixuan Cao et al. (+2 more)
Detecting AI-involved text is essential for combating misinformation, plagiarism, and academic misconduct. However, AI text generation includes diverse collaborative processes (AI-written text edited by humans, human-written text edited by AI, and AI-generated text refined by other AI), where various or even new LLMs could be involved. Texts generated through these varied processes exhibit complex characteristics, presenting significant challenges for detection. Current methods model these processes rather crudely, primarily employing binary classification (purely human vs. AI-involved) or multi-classification (treating human-AI collaboration as a new class). We observe that representations of texts generated through different processes exhibit inherent clustering relationships. Therefore, we propose DETree, a novel approach that models the relationships among different processes as a Hierarchical Affinity Tree structure, and introduces a specialized loss function that aligns text representations with this tree. To facilitate this learning, we developed RealBench, a comprehensive
benchmark dataset that automatically incorporates a wide spectrum of hybrid texts produced through various human-AI collaboration processes. Our method improves performance in hybrid text detection tasks and significantly enhances robustness and generalization in out-of-distribution scenarios, particularly in few-shot learning conditions, further demonstrating the promise of training-based approaches in OOD settings. Our code and
dataset are available at https://github.com/heyongxin233/DETree.
Read the full abstract ▾
All authors Yongxin He, Shan Zhang, Yixuan Cao, Lei Ma, Ping Luo
Computation and Language
ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
Zheyue Tan, Zhiyuan Li, Tao Yuan et al. (+13 more)
Mixture-of-Experts (MoE) architectures have emerged as a promising approach to scale Large
Language Models (LLMs). MoE boosts the efficiency by activating a subset of experts per
token. Recent works show that fine-grained experts substantially enriches the combinatorial flexibility of active experts and enhances model expressiveness. However, such a design is fundamentally limited by the layer-local routing mechanism: each layer is restricted to its own expert pool. This requires a careful trade-off between expert dimensionality and routing diversity given fixed parameter budgets. We describe ReXMoE, a novel MoE architecture that improves routing beyond the existing layer-local approaches by allowing routers to reuse experts across adjacent layers. ReXMoE decouples expert dimensionality from per-layer budgets, enabling richer expert combinations without sacrificing individual expert capacity or inflating overall parameters. To this end, we propose a new progressive scaling routing (PSR) strategy to gradually increase the candidate expert pool during training. As a result, ReXMoE improves both
language modeling and downstream task performance. Extensive experiments on models ranging from 0.5B to 7B parameters across different architectures demonstrate that ReXMoE consistently improves performance under fixed architectural dimensions, confirming ReXMoE as new design paradigm for parameter-efficient and scalable MoE-based LLMs.
Read the full abstract ▾
All authors Zheyue Tan, Zhiyuan Li, Tao Yuan, Dong Zhou, Weilin Liu, Yueqing Zhuang, Yadong Li, Guowei Niu, Cheng Qin, Zhuyu Yao, Congyi Liu, Haiyang Xu, Boxun Li, Guohao Dai, Bo Zhao, Yu Wang
Computation and Language
Zoom In Disparities in Healthcare LLM Q&A
Ipek Baris Schlicht, Burcu Sayin, Zhixue Zhao et al. (+5 more)
Equitable access to reliable health information is vital when integrating AI into healthcare. Yet, information quality varies across
languages, raising concerns about the reliability and consistency of
multilingual Large
Language Models (LLMs). We systematically examine cross-lingual disparities in pre-training source and factuality
alignment in LLM answers for
multilingual healthcare Q&A across English, German, Turkish, Chinese (Mandarin), and Italian. We (i) constructed
Multilingual Wiki Health Care (MultiWikiHealthCare), a
multilingual dataset from Wikipedia; (ii) analyzed cross-lingual healthcare coverage; (iii) assessed LLM response
alignment with these references; and (iv) conducted a case study on factual
alignment through the use of contextual information and
Retrieval-Augmented Generation (RAG). Our findings reveal substantial cross-lingual disparities in both Wikipedia coverage and LLM factual
alignment. Across LLMs, responses align more with English Wikipedia, even when the
prompts are non-English. Providing contextual excerpts from non-English Wikipedia at inference time effectively shifts factual
alignment toward culturally relevant knowledge. These results highlight practical pathways for building more equitable,
multilingual AI systems for healthcare.
Read the full abstract ▾
All authors Ipek Baris Schlicht, Burcu Sayin, Zhixue Zhao, Frederik M. Labonté, Cesare Barbera, Marco Viviani, Paolo Rosso, Lucie Flek
Computation and Language
Evaluating Large Language Models on Urdu Idioms
Muhammad Farmal Khan and Mousumi Akter
Idioms remain a persistent challenge in natural
language processing due to their figurative and culturally grounded meanings, which distinguish them from literal expressions. Although recent advances in large
language models (LLMs) have improved idiom handling across several
languages, limited
attention has been given to low resource
languages such as Urdu. In this work, we present a comprehensive
benchmark for Urdu to English idiomatic translation, consisting of a manually verified
dataset of 4,000 aligned idiom sentence pairs in both Perso Arabic (native Urdu script) and Romanized Urdu. We evaluate multiple tasks, including translation, paraphrasing, idiom span detection, and back-translation, using diverse prompting strategies such as cultural prompting, idiomatic prompting, and few-shot learning. Our findings show that frontier LLMs consistently outperform traditional neural machine translation systems across all evaluation settings, particularly in preserving figurative and metaphorical meaning. While models demonstrate relatively stable performance on native Urdu script, the absence of standardized orthography in Romanized Urdu introduces substantial challenges for consistency and idiom span detection. This work establishes a high quality
benchmark for cross script idiomatic evaluation in Urdu and underscores the importance of
prompt engineering in preserving figurative
language meaning across
languages.
Read the full abstract ▾
Computation and Language
Multilingual Clinical NER for Diseases and Medications Recognition in Cardiology Texts using BERT Embeddings
Manuela Daniela Danu, George Marica, Constantin Suciu et al. (+2 more)
The rapidly increasing volume of electronic health record (EHR) data underscores a pressing need to unlock biomedical knowledge from unstructured clinical texts to support advancements in data-driven clinical systems, including patient diagnosis, disease progression monitoring, treatment effects assessment, prediction of future clinical events, etc. While contextualized
language models have demonstrated impressive performance improvements for named entity recognition (NER) systems in English corpora, there remains a scarcity of research focused on clinical texts in low-resource
languages. To bridge this gap, our study aims to develop multiple deep contextual
embedding models to enhance clinical NER in the cardiology domain, as part of the BioASQ MultiCardioNER shared task. We explore the effectiveness of different monolingual and
multilingual BERT-based models, trained on general domain text, for extracting disease and medication mentions from clinical case reports written in English, Spanish, and Italian. We achieved an F1-score of 77.88% on Spanish Diseases Recognition (SDR), 92.09% on Spanish Medications Recognition (SMR), 91.74% on English Medications Recognition (EMR), and 88.9% on Italian Medications Recognition (IMR). These results outperform the mean and median F1 scores in the test leaderboard across all subtasks, with the mean/median values being: 69.61%/75.66% for SDR, 81.22%/90.18% for SMR, 89.2%/88.96% for EMR, and 82.8%/87.76% for IMR.
Read the full abstract ▾
All authors Manuela Daniela Danu, George Marica, Constantin Suciu, Lucian Mihai Itu, Oladimeji Farri
Computation and Language
Agentic Reinforcement Learning for Search Misaligns Instruction-Tuning
Yushi Yang, Shreyansh Padarha, Sarah Ball et al. (+2 more)
Agentic reinforcement learning (RL) trains large
language models to use tools, but its impact on
alignment is poorly understood. We study how agentic RL for search affects the
alignment of instruction-tuned (IT) models. We find that RL-trained models inherit refusal
reasoning by deflecting harmful requests into benign search queries, but this breaks down under a simple diagnostic trigger that elicits a search call before refusal can occur. Under this condition, RL models produce multi-step unsafe search actions and
reasoning, reducing search query safety by up to 68.6% in Qwen and Llama models relative to their IT counterparts. The effect generalises across model families, scales, and RL
algorithms. To understand why, we identify linear directions in the residual stream that control search query safety, and show that RL training progressively shifts search behaviour toward the harmful end of this direction. We thus propose representation-guided RL training, which adds a reward penalty based on projection toward the harmful search direction. Training on benign data alone, it restores IT-level
alignment without reducing task accuracy and requires no additional training data. Together, our work provides the first framework for diagnosing, mechanistically analysing, and mitigating
alignment degradation in agentic RL for search.
Read the full abstract ▾
All authors Yushi Yang, Shreyansh Padarha, Sarah Ball, Andrew Lee, Adam Mahdi
Computation and Language
Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
Tiancheng Hu, Benjamin Minixhofer and Nigel Collier
The "
alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after
alignment. Crucially, this is not a strict trade-off. We find that the process consistently reveals Pareto-optimal interpolations - models that improve accuracy beyond both parents while substantially recovering the calibration lost during
alignment. Our work demonstrates that simple model merging provides a computationally efficient method for mitigating the full scope of the
alignment tax, yielding models that are more capable and more reliable.
Read the full abstract ▾
Computation and Language
BenCao: An Instruction-Tuned Large Language Model for Traditional Chinese Medicine
Jiacheng Xie, Yang Yu, Yibo Chen et al. (+7 more)
Traditional Chinese Medicine (TCM), with a history spanning over two millennia, plays a role in global healthcare. However, applying large
language models (LLMs) to TCM remains challenging due to its reliance on holistic
reasoning, implicit logic, and
multimodal diagnostic cues. Existing TCM-domain LLMs have made progress in text-based understanding but lack
multimodal integration, interpretability, and clinical applicability. To address these limitations, we developed BenCao, a ChatGPT-based
multimodal assistant for TCM, integrating structured knowledge bases, diagnostic data, and expert feedback refinement. BenCao was trained through natural
language instruction tuning rather than parameter retraining, aligning with expert-level
reasoning and ethical norms specific to TCM. The system incorporates a comprehensive knowledge base of over 1,000 classical and modern texts, a scenario-based instruction framework for diverse interactions, a chain-of-thought simulation mechanism for interpretable
reasoning, and a feedback refinement process involving licensed TCM practitioners. BenCao connects to external APIs for tongue-image classification and
multimodal database
retrieval, enabling dynamic access to diagnostic resources. In evaluations across single-choice question
benchmarks and
multimodal classification tasks, BenCao achieved superior accuracy to general-domain and TCM-domain models, particularly in diagnostics, herb recognition, and constitution classification. The model was deployed as an interactive application on the OpenAI GPTs Store, accessed by nearly 1,000 users globally as of October 2025. This study demonstrates the feasibility of developing a TCM-domain LLM through natural
language-based instruction tuning and
multimodal integration, offering a practical framework for aligning generative AI with traditional medical
reasoning and a scalable pathway for real-world deployment.
Read the full abstract ▾
All authors Jiacheng Xie, Yang Yu, Yibo Chen, Hanyao Zhang, Lening Zhao, Jiaxuan He, Lei Jiang, Xiaoting Tang, Guanghui An, Dong Xu
Computation and Language
AFRICAPTION: Establishing a New Paradigm for Image Captioning in African Languages
Mardiyyah Oduwole, Prince Mireku, Fatimo Adebanjo et al. (+3 more)
Multimodal AI research has overwhelmingly focused on high-resource
languages, hindering the democratization of advancements in the field. To address this, we present AfriCaption, a comprehensive framework for
multilingual image captioning in 20 African
languages and our contributions are threefold: (i) a curated
dataset built on Flickr8k, featuring semantically aligned captions generated via a context-aware selection and translation process; (ii) a dynamic, context-preserving pipeline that ensures ongoing quality through model ensembling and adaptive substitution; and (iii) the AfriCaption model, a 0.5B parameter
vision-to-text architecture that integrates SigLIP and NLLB200 for caption generation across under-represented
languages. This unified framework ensures ongoing data quality and establishes the first scalable image-captioning resource for under-represented African
languages, laying the groundwork for truly inclusive
multimodal AI.
Read the full abstract ▾
All authors Mardiyyah Oduwole, Prince Mireku, Fatimo Adebanjo, Oluwatosin Olajide, Mahi Aminu Aliyu, Jekaterina Novikova
Computation and Language
Leveraging Group Relative Policy Optimization to Advance Large Language Models in Traditional Chinese Medicine
Jiacheng Xie, Shuai Zeng, Yang Yu et al. (+3 more)
Traditional Chinese Medicine (TCM) presents a rich and structurally unique knowledge system that challenges conventional applications of large
language models (LLMs). Although previous TCM-specific LLMs have shown progress through supervised fine-tuning, they often face limitations in
alignment, data quality, and evaluation consistency. In this study, we introduce Ladder-base, the first TCM-focused LLM trained with Group Relative Policy Optimization (GRPO), a reinforcement learning method that improves
reasoning and factual consistency by optimizing response selection based on intra-group comparisons. Ladder-base is built upon the Qwen2.5-7B-Instruct foundation model and trained exclusively on the textual subset of the TCM-Ladder
benchmark, using 80 percent of the data for training and the remaining 20 percent split evenly between validation and test sets. Through standardized evaluation, Ladder-base demonstrates superior performance across multiple
reasoning metrics when compared to both state-of-the-art general-purpose LLMs such as GPT-4, Gemini 2.5, Claude 3, and Qwen3 and domain-specific TCM models including BenTsao, HuatuoGPT2, and Zhongjing. These findings suggest that GRPO provides an effective and efficient strategy for aligning LLMs with expert-level
reasoning in traditional medical domains and supports the development of trustworthy and clinically grounded TCM artificial intelligence systems.
Read the full abstract ▾
All authors Jiacheng Xie, Shuai Zeng, Yang Yu, Xiaoting Tang, Guanghui An, Dong Xu
Computation and Language
EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMs
Numaan Naeem, Abdellah El Mekki and Muhammad Abdul-Mageed
Large
language models (LLMs) are transforming education by answering questions, explaining complex concepts, and generating content across a wide range of subjects. Despite strong performance on academic
benchmarks, they often fail to tailor responses to students' grade levels. This is a critical need in K-12 education, where age-appropriate vocabulary and explanation are essential for effective learning. Existing models frequently produce outputs that are too advanced or vague for younger learners, and there are no standardized
benchmarks to evaluate their ability to adjust across cognitive and developmental stages. To address this gap, we introduce EduAdapt, a
benchmark of nearly 48k grade-labeled QA pairs across nine science subjects, spanning Grades 1-12 and grouped into four grade levels. We evaluate a diverse set of open-source LLMs on EduAdapt and find that while larger models generally perform better, they still struggle with generating suitable responses for early-grade students (Grades 1-5). Our work presents the first
dataset and evaluation framework for assessing grade-level adaptability in LLMs, aiming to foster more developmentally aligned educational AI systems through better training and prompting strategies. EduAdapt code and
datasets are publicly available at https://github.com/NaumanNaeem/EduAdapt.
Read the full abstract ▾
Computation and Language
The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives
Henry Lim and Kwan Hui Lim
Instruction-tuned large
language models (IT-LLMs) exhibit strong zero-shot
reasoning, yet their ability to execute simple, self-contained instructions remains underexplored, despite this being foundational to complex instruction-following. We evaluate 20 IT-LLMs on modified MMLU and MMLU-Pro
benchmarks, by systematically varying the format of option labels (alphabetic, numeric, Roman) while keeping their meaning identical under four paradigms, namely: (1) With explicit instructions, label changes cause large performance shifts (e.g., -30.45% for Roman vs. numeric), revealing instruction-format bias. (2) Without instructions, performance drops further (up to -10.84%) and label sensitivity intensifies, underscoring the role of explicit guidance. (3) When option contents are removed, models fail random-choice baselines except with numeric labels, suggesting weak adherence to atomic directives. (4) Three-shot exemplars yield no significant gains in robustness or fidelity, and generation analyses show persistent label errors, especially for non-numeric formats. Across model sizes, larger LLMs achieve higher accuracy but remain inconsistent in instruction adherence. These results expose the insufficiencies of current instruction-tuning paradigms and highlight the need for evaluation methods and training strategies that explicitly target atomic instruction-following.
Read the full abstract ▾