Computation and Language
Subword Tokenization Strategies for Kurdish Word Embeddings
Ali Salehi and Cassandra L. Jacobs
We investigate tokenization strategies for Kurdish word
embeddings by comparing word-level, morpheme-based, and BPE approaches on morphological similarity preservation tasks. We develop a BiLSTM-CRF morphological segmenter using bootstrapped training from minimal manual annotation and evaluate Word2Vec
embeddings across comprehensive metrics including similarity preservation, clustering quality, and semantic organization. Our analysis reveals critical evaluation biases in tokenization comparison. While BPE initially appears superior in morphological similarity, it evaluates only 28.6% of test cases compared to 68.7% for morpheme model, creating artificial performance inflation. When assessed comprehensively, morpheme-based tokenization demonstrates superior
embedding space organization, better semantic neighborhood structure, and more balanced coverage across morphological complexity levels. These findings highlight the importance of coverage-aware evaluation in low-resource
language processing and offers different tokenization methods for low-resourced
language processing.
Read the full abstract ▾
Computation and Language
Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances
Rishu Kumar Singh, Navneet Shreya, Sarmistha Das et al. (+2 more)
Existing approaches to complaint analysis largely rely on unimodal, short-form content such as tweets or product reviews. This work advances the field by leveraging
multimodal, multi-turn customer support dialogues, where users often share both textual complaints and visual evidence (e.g., screenshots, product photos) to enable fine-grained classification of complaint aspects and severity. We introduce VALOR, a Validation-Aware Learner with Expert Routing, tailored for this
multimodal setting. It employs a multi-expert
reasoning setup using large-scale generative models with Chain-of-Thought (CoT) prompting for nuanced decision-making. To ensure coherence between modalities, a semantic
alignment score is computed and integrated into the final classification through a meta-fusion strategy. In
alignment with the United Nations Sustainable Development Goals (UN SDGs), the proposed framework supports SDG 9 (Industry, Innovation and Infrastructure) by advancing AI-driven tools for robust, scalable, and context-aware service infrastructure. Further, by enabling structured analysis of complaint narratives and visual context, it contributes to SDG 12 (Responsible Consumption and Production) by promoting more responsive product design and improved accountability in consumer services. We evaluate VALOR on a curated
multimodal complaint
dataset annotated with fine-grained aspect and severity labels, showing that it consistently outperforms baseline models, especially in complex complaint scenarios where information is distributed across text and images. This study underscores the value of
multimodal interaction and expert validation in practical complaint understanding systems. Resources related to data and codes are available here: https://github.com/sarmistha-D/VALOR
Read the full abstract ▾
All authors Rishu Kumar Singh, Navneet Shreya, Sarmistha Das, Apoorva Singh, Sriparna Saha
Computation and Language
Ground Truth Generation for Multilingual Historical NLP using LLMs
Clovis Gladstone, Zhao Fang and Spencer Dean Stewart
Historical and low-resource NLP remains challenging due to limited annotated data and domain mismatches with modern, web-sourced corpora. This paper outlines our work in using large
language models (LLMs) to create ground-truth annotations for historical French (16th-20th centuries) and Chinese (1900-1950) texts. By leveraging LLM-generated ground truth on a subset of our corpus, we were able to fine-tune spaCy to achieve significant gains on period-specific tests for part-of-
speech (POS) annotations, lemmatization, and named entity recognition (NER). Our results underscore the importance of domain-specific models and demonstrate that even relatively limited amounts of synthetic data can improve NLP tools for under-resourced corpora in computational humanities research.
Read the full abstract ▾
Computation and Language
Encoding and Understanding Astrophysical Information in Large Language Model-Generated Summaries
Kiera McCormick and Rafael Martínez-Galarza
Large
Language Models have demonstrated the ability to generalize well at many levels across domains, modalities, and even shown in-context learning capabilities. This enables research questions regarding how they can be used to encode physical information that is usually only available from scientific measurements, and loosely encoded in textual descriptions. Using astrophysics as a test bed, we investigate if LLM
embeddings can codify physical summary statistics that are obtained from scientific measurements through two main questions: 1) Does prompting play a role on how those quantities are codified by the LLM? and 2) What aspects of
language are most important in encoding the physics represented by the measurement? We investigate this using sparse autoencoders that extract interpretable features from the text.
Read the full abstract ▾
Computation and Language
SMRC: Aligning Large Language Models with Student Reasoning for Mathematical Error Correction
Biaojie Zeng, Min Zhang, Juan Zhou et al. (+4 more)
Large
language models (LLMs) often make
reasoning errors when solving mathematical problems, and how to automatically detect and correct these errors has become an important research direction. However, existing approaches mainly focus on self-correction within the model, which falls short of the "teacher-style" correction required in educational settings, i.e., systematically guiding and revising a student' s problem-solving process. To address this gap, we propose SMRC (\textit{\underline{S}tudent \underline{M}athematical \underline{R}easoning \underline{C}orrection}), a novel method that aligns LLMs with student
reasoning. Specifically, SMRC formulates student
reasoning as a multi-step sequential decision problem and introduces Monte Carlo Tree Search (MCTS) to explore optimal correction paths. To reduce the cost of the annotating process-level rewards, we leverage breadth-first search (BFS) guided by LLMs and final-answer evaluation to generate reward signals, which are then distributed across intermediate
reasoning steps via a back-propagation mechanism, enabling fine-grained process supervision. Additionally, we construct a
benchmark for high school mathematics, MSEB (Multi-Solution Error
Benchmark), consisting of 158 instances that include problem statements, student solutions, and correct
reasoning steps. We further propose a dual evaluation protocol centered on solution accuracy and correct-step retention, offering a comprehensive measure of educational applicability. Experiments demonstrate that SMRC significantly outperforms existing methods on two public
datasets (ProcessBench and MR-GSM8K) and our MSEB in terms of effectiveness and overall performance. The code are available at https://github.com/ECNU-RAIL/SMRC-EMNLP2026.
Read the full abstract ▾
All authors Biaojie Zeng, Min Zhang, Juan Zhou, Fengrui Liu, Ruiyang Huang, Yu Song, Xin Lin
Computation and Language
Quadratic Term Correction on Heaps' Law
Oscar Fontanelli and Wentian Li
Heaps' or Herdan's law characterizes the word-type vs. word-
token relation by a power-law function, which is concave in linear-linear scale but a straight line in log-log scale. However, it has been observed that even in log-log scale, the type-
token curve is still slightly concave, invalidating the power-law relation. At the next-order approximation, we have shown, by twenty English novels or writings (some are translated from another
language to English), that quadratic functions in log-log scale fit the type-
token data perfectly. Regression analyses of log(type)-log(
token) data with both a linear and quadratic term consistently lead to a linear coefficient of slightly larger than 1, and a quadratic coefficient around -0.02. Using the “random drawing colored ball from the bag with replacement" model, we have shown that the
curvature of the log-log scale is identical to a “pseudo-variance" which is negative. Although a pseudo-variance calculation may encounter numeric instability when the number of
tokens is large, due to the large values of pseudo-weights, this formalism provides a rough estimation of the
curvature when the number of
tokens is small.
Read the full abstract ▾
Computation and Language
Streamlining Industrial Contract Management with Retrieval-Augmented LLMs
Kristi Topollai, Tolga Dimlioglu, Anna Choromanska et al. (+2 more)
Contract management involves reviewing and negotiating provisions, individual clauses that define rights, obligations, and terms of agreement. During this process, revisions to provisions are proposed and iteratively refined, some of which may be problematic or unacceptable. Automating this workflow is challenging due to the scarcity of labeled data and the abundance of unstructured legacy contracts. In this paper, we present a modular framework designed to streamline contract management through a
retrieval-augmented generation (RAG) pipeline. Our system integrates synthetic data generation, semantic clause
retrieval, acceptability classification, and reward-based
alignment to flag problematic revisions and generate improved alternatives. Developed and evaluated in collaboration with an industry partner, our system achieves over 80% accuracy in both identifying and optimizing problematic revisions, demonstrating strong performance under real-world, low-resource conditions and offering a practical means of accelerating contract revision workflows.
Read the full abstract ▾
All authors Kristi Topollai, Tolga Dimlioglu, Anna Choromanska, Simon Odie, Reginald Hui
Computation and Language
Bias in, Bias out: Annotation Bias in Multilingual Large Language Models
Xia Cui, Ziyi Huang and Naeemeh Adel
Annotation bias in NLP
datasets remains a major challenge for developing
multilingual Large
Language Models (LLMs), particularly in culturally diverse settings. Bias from task framing, annotator subjectivity, and cultural mismatches can distort model outputs and exacerbate social harms. We propose a comprehensive framework for understanding annotation bias, distinguishing among instruction bias, annotator bias, and contextual and cultural bias. We review detection methods (including inter-annotator agreement, model disagreement, and metadata analysis) and highlight emerging techniques such as
multilingual model divergence and cultural inference. We further outline proactive and reactive mitigation strategies, including diverse annotator recruitment, iterative guideline refinement, and post-hoc model adjustments. Our contributions include: (1) a typology of annotation bias; (2) a synthesis of detection metrics; (3) an ensemble-based bias mitigation approach adapted for
multilingual settings, and (4) an ethical analysis of annotation processes. Together, these insights aim to inform more equitable and culturally grounded annotation pipelines for LLMs.
Read the full abstract ▾
Computation and Language
Graded strength of comparative illusions is explained by Bayesian inference
Yuhan Zhang, Erxiao Wang and Cory Shain
Like visual processing,
language processing is susceptible to illusions in which people systematically misperceive stimuli. In one such case–the comparative illusion (CI), e.g., More students have been to Russia than I have–comprehenders tend to judge the sentence as acceptable despite its underlying nonsensical comparison. Prior research has argued that this phenomenon can be explained as
Bayesian inference over a noisy channel: the posterior probability of an interpretation of a sentence is proportional to both the prior probability of that interpretation and the likelihood of corruption into the observed (CI) sentence. Initial behavioral work has supported this claim by evaluating a narrow set of alternative interpretations of CI sentences and showing that comprehenders favor interpretations that are more likely to have been corrupted into the illusory sentence. In this study, we replicate and go substantially beyond this earlier work by directly predicting the strength of illusion with a quantitative model of the posterior probability of plausible interpretations, which we derive through a novel synthesis of statistical
language models with human behavioral data. Our model explains not only the fine gradations in the strength of CI effects, but also a previously unexplained effect caused by pronominal vs. full noun phrase than-clause subjects. These findings support a noisy-channel theory of sentence comprehension by demonstrating that the theory makes novel predictions about the comparative illusion that bear out empirically. This outcome joins related evidence of noisy channel processing in both illusory and non-illusory contexts to support noisy channel inference as a unified computational-level theory of diverse
language processing phenomena.
Read the full abstract ▾
Computation and Language
A Specialized Large Language Model for Clinical Reasoning and Diagnosis in Rare Diseases
Tao Yang, Dandan Huang, Yunting Lin et al. (+25 more)
Rare diseases affect hundreds of millions worldwide, yet diagnosis often spans years. Convectional pipelines decouple noisy evidence extraction from downstream inferential diagnosis, and general/medical large
language models (LLMs) face scarce real world electronic health records (EHRs), stale domain knowledge, and
hallucinations. We assemble a large, domain specialized clinical corpus and a clinician validated
reasoning set, and develop RareSeek R1 via staged instruction tuning, chain of thought learning, and graph grounded
retrieval. Across multicenter EHR narratives and public
benchmarks, RareSeek R1 attains state of the art accuracy, robust generalization, and stability under noisy or overlapping phenotypes. Augmented
retrieval yields the largest gains when narratives pair with prioritized variants by resolving ambiguity and aligning candidates to mechanisms. Human studies show performance on par with experienced physicians and consistent gains in assistive use. Notably, transparent
reasoning highlights decisive non phenotypic evidence (median 23.1%, such as imaging, interventions, functional tests) underpinning many correct diagnoses. This work advances a narrative first, knowledge integrated
reasoning paradigm that shortens the diagnostic odyssey and enables auditable, clinically translatable decision support.
Read the full abstract ▾
All authors Tao Yang, Dandan Huang, Yunting Lin, Pengfei Wu, Zhikun Wu, Gangyuan Ma, Yulan Lu, Xinran Dong, Dingpeng Li, Junshuang Ge, Zhiyan Zhang, Xuanzhao Huang, Wenyan Nong, Yao Zhou, Hui Tang, Hongxi Yang, Shijie Zhang, Juan Li, Xiaojun Cao, Lin Yang, Xia Gao, Kaishou Xu, Xiaoqiong Gu, Wen Zhang, Huimin Xia, Li Liu, Wenhao Zhou, Mulin Jun Li
Computation and Language
Enhancing Agentic Autonomous Scientific Discovery with Vision-Language Model Capabilities
Kahaan Gandhi, Boris Bolliet and Inigo Zubeldia
We show that multi-
agent systems guided by
vision-
language models (VLMs) improve end-to-end autonomous scientific discovery. By treating plots as verifiable checkpoints, a VLM-as-a-judge evaluates figures against dynamically generated domain-specific rubrics, enabling
agents to correct their own errors and steer exploratory data analysis in real-time. Case studies in cosmology and astrochemistry demonstrate recovery from faulty
reasoning paths and adaptation to new
datasets without human intervention. On a 10-task
benchmark for data-driven discovery, VLM-augmented systems achieve pass at 1 scores of 0.7-0.8, compared to 0.2-0.3 for code-only and 0.4-0.5 for code-and-text baselines, while also providing auditable
reasoning traces that improve interpretability. Code available here: https://github.com/CMBAgents/cmbagent
Read the full abstract ▾
Computation and Language
Bridging Human and Model Perspectives: A Comparative Analysis of Political Bias Detection in News Media Using Large Language Models
Shreya Adrita Banik, Niaz Nafi Rahman, Tahsina Moiukh et al. (+1 more)
Detecting political bias in news media is a complex task that requires interpreting subtle linguistic and contextual cues. Although recent advances in Natural
Language Processing (NLP) have enabled automatic bias classification, the extent to which large
language models (LLMs) align with human judgment still remains relatively underexplored and not yet well understood. This study aims to present a comparative framework for evaluating the detection of political bias across human annotations and multiple LLMs, including GPT, BERT, RoBERTa, and FLAN. We construct a manually annotated
dataset of news articles and assess annotation consistency, bias polarity, and inter-model agreement to quantify divergence between human and model perceptions of bias. Experimental results show that among traditional
transformer-based models, RoBERTa achieves the highest
alignment with human labels, whereas generative models such as GPT demonstrate the strongest overall agreement with human annotations in a zero-shot setting. Among all
transformer-based baselines, our fine-tuned RoBERTa model acquired the highest accuracy and the strongest
alignment with human-annotated labels. Our findings highlight systematic differences in how humans and LLMs perceive political slant, underscoring the need for hybrid evaluation frameworks that combine human interpretability with model scalability in automated media bias detection.
Read the full abstract ▾
All authors Shreya Adrita Banik, Niaz Nafi Rahman, Tahsina Moiukh, Farig Sadeque
Computation and Language
A Method for Characterizing Disease Progression from Acute Kidney Injury to Chronic Kidney Disease
Yilu Fang, Jordan G. Nestor, Casey N. Ta et al. (+2 more)
Patients with acute kidney injury (AKI) are at high risk of developing chronic kidney disease (CKD), but identifying those at greatest risk remains challenging. We used electronic health record (EHR) data to dynamically track AKI patients' clinical
evolution and characterize AKI-to-CKD progression. Post-AKI clinical states were identified by clustering patient vectors derived from longitudinal medical codes and creatinine measurements. Transition probabilities between states and progression to CKD were estimated using multi-state modeling. After identifying common post-AKI trajectories, CKD risk factors in AKI subpopulations were identified through survival analysis. Of 20,699 patients with AKI at admission, 3,491 (17%) developed CKD. We identified fifteen distinct post-AKI states, each with different probabilities of CKD development. Most patients (75%, n=15,607) remained in a single state or made only one transition during the study period. Both established (e.g., AKI severity, diabetes, hypertension, heart failure, liver disease) and novel CKD risk factors, with their impact varying across these clinical states. This study demonstrates a data-driven approach for identifying high-risk AKI patients, supporting the development of decision-support tools for early CKD detection and intervention.
Read the full abstract ▾
All authors Yilu Fang, Jordan G. Nestor, Casey N. Ta, Jerard Z. Kneifati-Hayek, Chunhua Weng
Computation and Language
Leveraging Digitized Newspapers to Collect Summarization Data in Low-Resource Languages
Noam Dahan, Omer Kidron and Gabriel Stanovsky
High quality summarization data remains scarce in under-represented
languages. However, historical newspapers, made available through recent digitization efforts, offer an abundant source of untapped, naturally annotated data. In this work, we present a novel method for collecting naturally occurring summaries via Front-Page Teasers, where editors summarize full length articles. We show that this phenomenon is common across seven diverse
languages and supports multi-document summarization. To scale data collection, we develop an automatic process, suited to varying linguistic resource levels. Finally, we apply this process to a Hebrew newspaper title, producing HEBTEASESUM, the first dedicated multi-document summarization
dataset in Hebrew.
Read the full abstract ▾
Computation and Language
Examining the Metrics for Document-Level Claim Extraction in Czech and Slovak
Lucia Makaiova, Martin Fajcik and Antonin Jarolim
Document-level claim extraction remains an open challenge in the field of fact-checking, and subsequently, methods for evaluating extracted claims have received limited
attention. In this work, we explore approaches to aligning two sets of claims pertaining to the same source document and computing their similarity through an
alignment score. We investigate techniques to identify the best possible
alignment and evaluation method between claim sets, with the aim of providing a reliable evaluation framework. Our approach enables comparison between model-extracted and human-annotated claim sets, serving as a metric for assessing the extraction performance of models and also as a possible measure of inter-annotator agreement. We conduct experiments on newly collected
dataset-claims extracted from comments under Czech and Slovak news articles-domains that pose additional challenges due to the informal
language, strong local context, and subtleties of these closely related
languages. The results draw
attention to the limitations of current evaluation approaches when applied to document-level claim extraction and highlight the need for more advanced methods-ones able to correctly capture semantic similarity and evaluate essential claim properties such as atomicity, checkworthiness, and decontextualization.
Read the full abstract ▾
Computation and Language
LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation
David Carmel, Simone Filice, Guy Horowitz et al. (+4 more)
With
Retrieval Augmented Generation (RAG) becoming more and more prominent in generative AI solutions, there is an emerging need for systematically evaluating their effectiveness. We introduce the LiveRAG
benchmark, a publicly available
dataset of 895 synthetic questions and answers designed to support systematic evaluation of RAG-based Q&A systems. This synthetic
benchmark is derived from the one used during the SIGIR'2025 LiveRAG Challenge, where competitors were evaluated under strict time constraints. It is augmented with information that was not made available to competitors during the Challenge, such as the ground-truth answers, together with their associated supporting claims which were used for evaluating competitors' answers. In addition, each question is associated with estimated difficulty and discriminability scores, derived from applying an Item Response Theory model to competitors' responses. Our analysis highlights the
benchmark's questions diversity, the wide range of their difficulty levels, and their usefulness in differentiating between system capabilities. The LiveRAG
benchmark will hopefully help the community advance RAG research, conduct systematic evaluation, and develop more robust Q&A systems.
Read the full abstract ▾
All authors David Carmel, Simone Filice, Guy Horowitz, Yoelle Maarek, Alex Shtoff, Oren Somekh, Ran Tavory
Computation and Language
Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning
Mingyue Cheng, Shuo Yu, Daoyu Wang et al. (+7 more)
Large
language models (LLMs) have rapidly evolved from single-turn text generators into the foundation of increasingly capable
agents. As these
agents take on more complex
reasoning, decision making, tool use, and long-horizon tasks, reinforcement learning (RL) is becoming increasingly important for shaping their behavior. This shift is especially visible in agentic RL, where models must interact with tools and environments across multiple rounds rather than produce a single standalone response. In this regime, the usual view of a trajectory as one ever-growing
token sequence becomes increasingly inadequate: it makes context
evolution rigid and creates representation mismatches between rollout and training. This paper presents
Agent-R1, a unified and modular framework for agentic RL built around step-level trajectory representation, flexible context management, and layered interfaces for workflows, environments and optimization. The key idea is to treat each interaction step as the basic reinforcement-learning transition, while keeping the optimization layer flexible: once the interaction is modeled at the step level, the framework can support
token-level credit assignment, step-level credit assignment, or other compatible designs. These design choices make the framework compatible with a range of optimization strategies rather than tying it to a single
algorithm. Together, these components provide a principled, extensible, and reusable substrate for agentic RL.
Read the full abstract ▾
All authors Mingyue Cheng, Shuo Yu, Daoyu Wang, Qingchuan Li, Xiaoyu Tao, Jie Ouyang, Yucong Luo, Yitong Zhou, Qi Liu, Enhong Chen
Computation and Language
Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning
Trishala Jayesh Ahalpara
We present Tell Me, a mental well-being system that leverages advances in large
language models to provide accessible, context-aware support for users and researchers. The system integrates three components: (i) a
retrieval-augmented generation (RAG) assistant for personalized, knowledge-grounded dialogue; (ii) a synthetic client-therapist dialogue generator conditioned on client profiles to facilitate research on therapeutic
language and data augmentation; and (iii) a Well-being AI crew, implemented with CrewAI, that produces weekly self-care plans and guided meditation audio. The system is designed as a reflective space for emotional processing rather than a substitute for professional therapy. It illustrates how conversational assistants can lower barriers to support, complement existing care, and broaden access to mental health resources. To address the shortage of confidential therapeutic data, we introduce synthetic client-therapist dialogue generation conditioned on client profiles. Finally, the planner demonstrates an innovative agentic workflow for dynamically adaptive, personalized self-care, bridging the limitations of static well-being tools. We describe the architecture, demonstrate its functionalities, and report evaluation of the RAG assistant in curated well-being scenarios using both automatic LLM-based judgments and a human-user study. This work highlights opportunities for interdisciplinary collaboration between NLP researchers and mental health professionals to advance responsible innovation in human-AI interaction for well-being.
Read the full abstract ▾
Computation and Language
MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
Jinru Ding, Lu Lu, Chao Ding et al. (+15 more)
Recent advances in medical large
language models (LLMs),
multimodal models, and
agents demand evaluation frameworks that reflect real clinical workflows and safety constraints. We present MedBench v4, a nationwide, cloud-based benchmarking infrastructure comprising over 700,000 expert-curated tasks spanning 24 primary and 91 secondary specialties, with dedicated tracks for LLMs,
multimodal models, and
agents. Items undergo multi-stage refinement and multi-round review by clinicians from more than 500 institutions, and open-ended responses are scored by an LLM-as-a-judge calibrated to human ratings. We evaluate 15 frontier models. Base LLMs reach a mean overall score of 54.1/100 (best: Claude Sonnet 4.5, 62.5/100), but safety and ethics remain low (18.4/100).
Multimodal models perform worse overall (mean 47.5/100; best: GPT-5, 54.9/100), with solid perception yet weaker cross-modal
reasoning.
Agents built on the same backbones substantially improve end-to-end performance (mean 79.8/100), with Claude Sonnet 4.5-based
agents achieving up to 85.3/100 overall and 88.9/100 on safety tasks. MedBench v4 thus reveals persisting gaps in
multimodal reasoning and safety for base models, while showing that governance-aware agentic orchestration can markedly enhance benchmarked clinical readiness without sacrificing capability. By aligning tasks with Chinese clinical guidelines and regulatory priorities, the platform offers a practical reference for hospitals, developers, and policymakers
auditing medical AI.
Read the full abstract ▾
All authors Jinru Ding, Lu Lu, Chao Ding, Mouxiao Bian, Jiayuan Chen, Wenrao Pang, Ruiyao Chen, Xinwei Peng, Renjie Lu, Sijie Ren, Guanxu Zhu, Xiaoqin Wu, Zhiqiang Liu, Rongzhao Zhang, Luyi Jiang, Bing Han, Yunqiu Wang, Jie Xu
Computation and Language
Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Education
Xin Yi, Yue Li, Dongsheng Shi et al. (+3 more)
Large
Language Models (LLMs) are increasingly integrated into educational applications. However, they remain vulnerable to jailbreak and fine-tuning attacks, which can compromise safety
alignment and lead to harmful outputs. Existing studies mainly focus on general safety evaluations, with limited
attention to the unique safety requirements of educational scenarios. To address this gap, we construct EduHarm, a
benchmark containing safe-unsafe instruction pairs across five representative educational scenarios, enabling systematic safety evaluation of educational LLMs. Furthermore, we propose a three-stage shield framework (TSSF) for educational LLMs that simultaneously mitigates both jailbreak and fine-tuning attacks. First, safety-aware
attention realignment redirects
attention toward critical unsafe
tokens, thereby restoring the harmfulness feature that discriminates between unsafe and safe inputs. Second, layer-wise safety judgment identifies harmfulness features by aggregating safety cues across multiple layers to detect unsafe instructions. Finally, defense-driven dual routing separates safe and unsafe queries, ensuring normal processing for benign inputs and guarded responses for harmful ones. Extensive experiments across eight jailbreak attack strategies demonstrate that TSSF effectively strengthens safety while preventing over-refusal of benign queries. Evaluations on three fine-tuning attack
datasets further show that it consistently achieves robust defense against harmful queries while maintaining preserving utility gains from benign fine-tuning.
Read the full abstract ▾
All authors Xin Yi, Yue Li, Dongsheng Shi, Linlin Wang, Xiaoling Wang, Liang He
Computation and Language
Mitigating Label Length Bias in Large Language Models
Mario Sanz-Guerrero and Katharina von der Wense
Large
language models (LLMs) are powerful zero- and few-shot learners. However, when predicting over a set of candidate options, LLMs suffer from label biases, and existing calibration methods overlook biases arising from multi-
token class labels. We tackle an issue we call label length bias, where labels of different lengths are treated inconsistently, even after standard length normalization. To mitigate it, we propose normalized contextual calibration (NCC), an effective method that normalizes and calibrates predictions at the full-label level. NCC achieves statistically significant improvements over prior approaches across multiple
datasets and models, with gains of up to 10% F1. Moreover, NCC extends bias mitigation to broader tasks such as multiple-choice question answering. Our analysis shows that, when combined with in-context learning, NCC is less sensitive to few-shot example selection, requires fewer examples for competitive performance, and produces more reliable confidence estimates. These findings highlight the importance of mitigating full-label biases to improve the performance and robustness of LLM-based methods, particularly in real-world applications where class labels naturally consist of multiple
tokens.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model
Rishi Gupta, Mukilan Karuppasamy, Shyam Marjit et al. (+2 more)
While Large
Vision Language Models (LVLMs) are increasingly deployed in real-world applications, their ability to interpret abstract visual inputs remains limited. Specifically, they struggle to comprehend hand-drawn sketches, a modality that offers an intuitive means of expressing concepts that are difficult to describe textually. We identify the primary bottleneck as the absence of a large-scale
dataset that jointly models sketches, photorealistic images, and corresponding natural
language instructions. To address this, we present two key contributions: (1) a new, large-scale
dataset of image-sketch-instruction triplets designed to facilitate both pretraining and instruction tuning, and (2) O3SLM, an LVLM trained on this
dataset. Comprehensive evaluations on multiple sketch-based tasks: (a) object localization, (b) counting, (c) image
retrieval i.e., (SBIR and fine-grained SBIR), and (d) visual question answering (VQA); while incorporating the three existing sketch
datasets, namely QuickDraw!, Sketchy, and Tu Berlin, along with our generated SketchVCL
dataset, show that O3SLM achieves state-of-the-art performance, substantially outperforming existing LVLMs in sketch comprehension and
reasoning.
Read the full abstract ▾
All authors Rishi Gupta, Mukilan Karuppasamy, Shyam Marjit, Aditay Tripathi, Anirban Chakraborty
Computation and Language
ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning
Hongwei Liu, Junnan Liu, Shudong Liu et al. (+33 more)
The rapid advancement of Large
Language Models (LLMs) has led to performance saturation on many established
benchmarks, questioning their ability to distinguish frontier models. Concurrently, existing high-difficulty
benchmarks often suffer from narrow disciplinary focus, oversimplified answer formats, and vulnerability to data contamination, creating a fidelity gap with real-world scientific inquiry. To address these challenges, we introduce ATLAS (AGI-Oriented Testbed for Logical Application in Science), a large-scale, high-difficulty, and cross-disciplinary evaluation suite composed of approximately 800 original problems. Developed by domain experts (PhD-level and above), ATLAS spans seven core scientific fields: mathematics, physics, chemistry, biology, computer science, earth science, and materials science. Its key features include: (1) High Originality and Contamination Resistance, with all questions newly created or substantially adapted to prevent test data leakage; (2) Cross-Disciplinary Focus, designed to assess models' ability to integrate knowledge and reason across scientific domains; (3) High-Fidelity Answers, prioritizing complex, open-ended answers involving multi-step
reasoning and LaTeX-formatted expressions over simple multiple-choice questions; and (4) Rigorous Quality Control, employing a multi-stage process of expert peer review and
adversarial testing to ensure question difficulty, scientific value, and correctness. We also propose a robust evaluation paradigm using a panel of LLM judges for automated, nuanced assessment of complex answers. Preliminary results on leading models demonstrate ATLAS's effectiveness in differentiating their advanced scientific
reasoning capabilities. We plan to develop ATLAS into a long-term, open, community-driven platform to provide a reliable "ruler" for progress toward Artificial General Intelligence.
Read the full abstract ▾
All authors Hongwei Liu, Junnan Liu, Shudong Liu, Haodong Duan, Yuqiang Li, Mao Su, Xiaohong Liu, Guangtao Zhai, Xinyu Fang, Qianhong Ma, Taolin Zhang, Zihan Ma, Yufeng Zhao, Peiheng Zhou, Linchen Xiao, Wenlong Zhang, Shijie Zhou, Xingjian Ma, Siqi Sun, Jiaye Ge, Meng Li, Yuhong Liu, Jianxin Dong, Jiaying Li, Hui Wu, Hanwen Liang, Jintai Lin, Yanting Wang, Jie Dong, Tong Zhu, Tianfan Fu, Conghui He, Qi Zhang, Songyang Zhang, Lei Bai, Kai Chen
Computation and Language
The Tokenization Bottleneck: How Vocabulary Extension Improves Chemistry Representation Learning in Pretrained Language Models
Prathamesh Kalamkar, Ned Letcher, Meissane Chami et al. (+3 more)
The application of large
language models (LLMs) to chemistry is frequently hampered by a "tokenization bottleneck", where tokenizers tuned on general-domain text tend to fragment chemical representations such as SMILES into semantically uninformative sub-
tokens. This paper introduces a principled methodology to resolve this bottleneck by unifying the representation of natural
language and molecular structures within a single model. Our approach involves targeted vocabulary extension-augmenting a pretrained LLM's vocabulary with chemically salient
tokens, followed by continued pretraining on chemistry-domain text to integrate this new knowledge. We provide an empirical demonstration of the effectiveness of this strategy, showing that our methodology leads to superior performance on a range of downstream chemical tasks.
Read the full abstract ▾
All authors Prathamesh Kalamkar, Ned Letcher, Meissane Chami, Sahger Lad, Shayan Mohanty, Prasanna Pendse
Computation and Language · Cross-listed from cs.DL
SciRAG: Adaptive, Citation-Aware, and Outline-Guided Retrieval and Synthesis for Scientific Literature
Hang Ding, Yilun Zhao, Tiansheng Hu et al. (+2 more)
The accelerating growth of scientific publications has intensified the need for scalable, trustworthy systems to synthesize knowledge across diverse literature. While recent
retrieval-augmented generation (RAG) methods have improved access to scientific information, they often overlook citation graph structure, adapt poorly to complex queries, and yield fragmented, hard-to-verify syntheses. We introduce SciRAG, an open-source framework for scientific literature exploration that addresses these gaps through three key innovations: (1) adaptive
retrieval that flexibly alternates between sequential and parallel evidence gathering; (2) citation-aware symbolic
reasoning that leverages citation graphs to organize and filter supporting documents; and (3) outline-guided synthesis that plans, critiques, and refines answers to ensure coherence and transparent attribution. Extensive experiments across multiple
benchmarks such as QASA and ScholarQA demonstrate that SciRAG outperforms prior systems in factual accuracy and synthesis quality, establishing a new foundation for reliable, large-scale scientific knowledge aggregation.
Read the full abstract ▾
All authors Hang Ding, Yilun Zhao, Tiansheng Hu, Manasi Patwardhan, Arman Cohan
Computation and Language
ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions
Xingwei He, Qianru Zhang, Pengfei Chen et al. (+4 more)
Instruction-following is a critical capability of Large
Language Models (LLMs). While existing works primarily focus on assessing how well LLMs adhere to user instructions, they often overlook scenarios where instructions contain conflicting constraints-a common occurrence in complex
prompts. The behavior of LLMs under such conditions remains under-explored. To bridge this gap, we introduce ConInstruct, a
benchmark specifically designed to assess LLMs' ability to detect and resolve conflicts within user instructions. Using this
dataset, we evaluate LLMs' conflict detection performance and analyze their conflict resolution behavior. Our experiments reveal two key findings: (1) Most proprietary LLMs exhibit strong conflict detection capabilities, whereas among open-source models, only DeepSeek-R1 demonstrates similarly strong performance. DeepSeek-R1 and Claude-4.5-Sonnet achieve the highest average F1-scores at 91.5% and 87.3%, respectively, ranking first and second overall. (2) Despite their strong conflict detection abilities, LLMs rarely explicitly notify users about the conflicts or request clarification when faced with conflicting constraints. These results underscore a critical shortcoming in current LLMs and highlight an important area for future improvement when designing instruction-following LLMs.
Read the full abstract ▾
All authors Xingwei He, Qianru Zhang, Pengfei Chen, Guanhua Chen, Linlin Yu, Yuan Yuan, Siu-Ming Yiu
Computation and Language · Cross-listed from cs.CR
SteganoBackdoor: Evading Data-Poisoning Defenses via Steganographic Backdoors
Eric Xue, Ruiyi Zhang and Pengtao Xie
Transformer-based models are highly susceptible to
backdoor attacks via supervised fine-tuning (SFT). To red-team existing data-poisoning defenses, prior work has increasingly focused on stylized triggers, synthetic artifacts, and
token-level perturbations designed to evade detection. However, this trend has shifted threat models away from naturally occurring semantic triggers and realistic low-budget poisoning settings. Addressing this gap, we introduce SteganoBackdoor, an optimization-based framework that transforms semantic-trigger seeds through autoregressive
token replacement, sequentially minimizing
embedding overlap with the inference-time trigger while preserving a strong per-sample training-time payload. The resulting SteganoPoisons maintain linguistic fluency and encode the payload across ordinary
tokens, such that no individual
token carries a concentrated signal and the full payload instead emerges from their exact combination and ordering. Across 18 encoder-based and decoder-only models spanning 120M to 14B parameters, SteganoBackdoor achieves high attack success under sub-percent poisoning budgets and exposes limitations in existing data-poisoning defenses.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
DataSage: Multi-agent Collaboration for Insight Discovery with External Knowledge Retrieval, Multi-role Debating, and Multi-path Reasoning
Xiaochuan Liu, Yuanfeng Song, Xiaoming Yin et al. (+1 more)
In today's data-driven era, fully automated end-to-end data analytics, particularly insight discovery, is critical for discovering actionable insights that assist organizations in making effective decisions. With the rapid advancement of large
language models (LLMs), LLM-driven
agents have emerged as a promising paradigm for automating data analysis and insight discovery. However, existing data insight
agents remain limited in several key aspects, often failing to deliver satisfactory results due to: (1) insufficient utilization of domain knowledge, (2) shallow analytical depth, and (3) error-prone code generation during insight generation. To address these issues, we propose DataSage, a novel multi-
agent framework that incorporates three innovative features including external knowledge
retrieval to enrich the analytical context, a multi-role debating mechanism to simulate diverse analytical perspectives and deepen analytical depth, and multi-path
reasoning to improve the accuracy of the generated code and insights. Extensive experiments on InsightBench demonstrate that DataSage consistently outperforms existing data insight
agents across all difficulty levels, offering an effective solution for automated data insight discovery.
Read the full abstract ▾
All authors Xiaochuan Liu, Yuanfeng Song, Xiaoming Yin, Xing Chen
Computation and Language
AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models
Mohammad Zbeeb, Hasan Abed Al Kader Hammoud, Sina Mukalled et al. (+5 more)
We present AraLingBench: a fully human annotated
benchmark for evaluating the Arabic linguistic competence of large
language models (LLMs). The
benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 expert-designed multiple choice questions that directly assess structural
language understanding. Evaluating 35 Arabic and bilingual LLMs reveals that current models demonstrate strong surface level proficiency but struggle with deeper grammatical and syntactic
reasoning. AraLingBench highlights a persistent gap between high scores on knowledge-based
benchmarks and true linguistic mastery, showing that many models succeed through memorization or pattern recognition rather than authentic comprehension. By isolating and measuring fundamental linguistic skills, AraLingBench provides a diagnostic framework for developing Arabic LLMs. The full evaluation code is publicly available on GitHub.
Read the full abstract ▾
All authors Mohammad Zbeeb, Hasan Abed Al Kader Hammoud, Sina Mukalled, Nadine Rizk, Fatima Karnib, Issam Lakkis, Ammar Mohanna, Bernard Ghanem
Computation and Language
Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distribution
Ante Wang, Weizhi Ma and Yang Liu
Knowing the reliability of a model's response is essential in practical applications. Given the strong generation capabilities of large
language models (LLMs), research has focused on generating verbalized confidence. This approach is further enhanced by integrating chain-of-thought
reasoning, which provides logical and transparent estimates. However, how
reasoning strategies affect the estimated confidence remains under-explored. In this work, we demonstrate that predicting a verbalized probability distribution effectively promotes
reasoning for confidence estimation. It requires an LLM to consider all possible answers rather than relying on a single guess, and the requirement of producing a distribution elicits more careful confidence assignment. We conduct systematic experiments comparing different verbalization-based methods across multiple LLMs and tasks. Our method consistently shows advantages, whether in the simple prompting setup or after optimization via reinforcement learning (RL). Notably, it achieves higher
reasoning efficacy during inference-time scaling, saving nearly 6$\times$ the computation to reach the best Brier score of the strongest baseline on MMLU-Pro. Additionally, we reveal its limitations on specific tasks and discuss possible solutions for broader applicability.
Read the full abstract ▾