Computation and Language
Shanghao Li, Jinda Han, Yibo Wang et al. (+5 more)
Graph-based
Retrieval-Augmented Generation (GraphRAG) enhances Large
Language Models (LLMs) by incorporating external knowledge from linearized subgraphs retrieved from knowledge graphs. However, LLMs struggle to interpret the relational and topological information in these inputs, resulting in
hallucinations that are inconsistent with the retrieved knowledge. To analyze how LLMs attend to and retain structured knowledge during generation, we propose two lightweight interpretability metrics: Path Reliance Degree (PRD), which measures over-reliance on shortest-path triples, and Semantic
Alignment Score (SAS), which assesses how well the model's internal representations align with the retrieved knowledge. Through empirical analysis on a knowledge-based QA task, we identify failure patterns associated with over-reliance on salient paths and weak semantic grounding, as indicated by high PRD and low SAS scores. We further develop a lightweight post-hoc
hallucination detector, Graph Grounding and
Alignment (GGA), which outperforms strong semantic and confidence-based baselines across AUC and F1. By grounding
hallucination analysis in mechanistic interpretability, our work offers insights into how structural limitations in LLMs contribute to
hallucinations, informing the design of more reliable GraphRAG systems in the future.
Read the full abstract ▾
All authors Shanghao Li, Jinda Han, Yibo Wang, Yuanjie Zhu, Zihe Song, Langzhou He, Kenan Kamel A Alghythee, Philip S. Yu
Computation and Language
Knowledge-Guided Large Language Model for Automatic Pediatric Dental Record Understanding and Safe Antibiotic Recommendation
Zihan Han, Junyan Ge and Caifeng Li
Accurate interpretation of pediatric dental clinical records and safe antibiotic prescribing remain persistent challenges in dental informatics. Traditional rule-based clinical decision support systems struggle with unstructured dental narratives, incomplete radiographic descriptions, and complex safety constraints. To address these limitations, this study proposes a Knowledge-Guided Large
Language Model (KG-LLM) that integrates a pediatric dental knowledge graph,
retrieval-augmented generation (RAG), and a multi-stage safety validation pipeline for evidence-grounded antibiotic recommendation. The framework first employs a clinical NER/RE module to extract structured entities and relations from dental notes and radiology reports. Relevant guidelines, drug-safety rules, and analogous historical cases are subsequently retrieved from the knowledge graph and supplied to the LLM for diagnostic summarization and dose-drug-duration prediction. Safety assurance is achieved through a dual-layer validation mechanism combining deterministic rule checking with a learned classifier for detecting allergies, contraindications, and dosing errors. Experiments on 32,000 de-identified pediatric dental visit records demonstrate the effectiveness of the proposed approach. Compared with a domain-adapted Llama-2 clinical baseline, KG-LLM improves record-understanding performance (F1: 0.914 vs. 0.867), drug-dose-duration accuracy (Top-1: 0.782 vs. 0.716), and reduces unsafe antibiotic suggestions by 50%. Additional evaluation across summary quality, recommendation accuracy, and global safety scores further confirms the robustness of the system. Ablation analyses indicate that the knowledge graph, RAG, and safety modules each contribute substantially to clinical reliability and interpretability.
Read the full abstract ▾
Computation and Language
Luxical: High-Speed Lexical-Dense Text Embeddings
DatologyAI: Luke Merrick, Alex Fang, Aldo Carranza et al. (+29 more)
Frontier
language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of
transformer text
embedding models flexibly support numerous workflows (e.g., clustering, classification, and
retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text
embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF–IDF features, a small ReLU network, and a knowledge
distillation training regimen to approximate large
transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical architecture and training objective and evaluate a concrete Luxical model in two disparate applications: a targeted webcrawl document
retrieval test and an end-to-end
language model data curation task grounded in text classification. In these tasks we demonstrate speedups ranging from 3x to 100x over varying-sized neural baselines, and comparable to FastText model inference during the data curation task. On these evaluations, the tested Luxical model illustrates favorable compute/quality trade-offs for large-scale text organization, matching the quality of neural baselines. Luxical is available as open-source software at https://github.com/datologyai/luxical.
Read the full abstract ▾
All authors DatologyAI: Luke Merrick, Alex Fang, Aldo Carranza, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Darren Teh, David Schwab, Fan Pan, Haakon Mongstad, Haoli Yin, Jack Urbanek, Jason Lee, Jason Telanoff, Josh Wills, Kaleigh Mentzer, Paul Burstein, Parth Doshi, Paul Burnstein, Pratyush Maini, Ricardo Monti, Rishabh Adiga, Scott Loftin, Siddharth Joshi, Spandan Das, Tony Jiang, Vineeth Dorna, Zhengping Wang, Bogdan Gaza, Ari Morcos, Matthew Leavitt
Computation and Language · Cross-listed from cs.LG
Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
Jakub Krajewski, Amitis Shidani, Dan Busbridge et al. (+2 more)
While scaling laws for Large
Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliable. This paper challenges that view by proposing a direct framework to model the scaling of
benchmark performance from the training budget. We find that for a fixed
token-to-parameter ratio, a simple power law can accurately describe the scaling behavior of log accuracy on multiple popular downstream tasks. Our results show that the direct approach extrapolates better than the previously proposed two-stage procedure, which is prone to compounding errors. Furthermore, we introduce functional forms that predict accuracy across
token-to-parameter ratios and account for inference compute under repeated sampling. We validate our findings on models with up to 17B parameters trained on up to 350B
tokens across two
dataset mixtures. To support reproducibility and encourage future research, we release the complete set of pretraining losses and downstream evaluation results.
Read the full abstract ▾
All authors Jakub Krajewski, Amitis Shidani, Dan Busbridge, Sam Wiseman, Jason Ramapuram
Computation and Language
Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoders
Guangzhi Xiong, Zhenghao He, Bohan Liu et al. (+2 more)
Retrieval-Augmented Generation (RAG) improves the factuality of large
language models (LLMs) by grounding outputs in retrieved evidence, but faithfulness failures, where generations contradict or extend beyond the provided sources, remain a critical challenge. Existing
hallucination detection methods for RAG often rely either on large-scale detector training, which requires substantial annotated data, or on querying external LLM judges, which leads to high inference costs. Although some approaches attempt to leverage internal representations of LLMs for
hallucination detection, their accuracy remains limited. Motivated by recent advances in mechanistic interpretability, we employ sparse autoencoders (SAEs) to disentangle internal activations, successfully identifying features that are specifically triggered during RAG
hallucinations. Building on a systematic pipeline of information-based feature selection and additive feature modeling, we introduce RAGLens, a lightweight
hallucination detector that accurately flags unfaithful RAG outputs using LLM internal representations. RAGLens not only achieves superior detection performance compared to existing methods, but also provides interpretable rationales for its decisions, enabling effective post-hoc mitigation of unfaithful RAG. Finally, we justify our design choices and reveal new insights into the distribution of
hallucination-related signals within LLMs. The code is available at https://github.com/Teddy-XiongGZ/RAGLens.
Read the full abstract ▾
All authors Guangzhi Xiong, Zhenghao He, Bohan Liu, Sanchit Sinha, Aidong Zhang
Computation and Language
Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis
Ferdinand Kapl, Emmanouil Angelis, Tobias Höppe et al. (+4 more)
Gradually growing the depth of
Transformers during training can not only reduce training cost but also lead to improved
reasoning performance, as shown by MIDAS (Saunshi et al., 2024). Thus far, however, a mechanistic understanding of these gains has been missing. In this work, we establish a connection to recent work showing that layers in the second half of non-grown, pre-layernorm
Transformers contribute much less to the final output distribution than those in the first half - also known as the Curse of Depth (Sun et al., 2025, Csordás et al., 2025). Using depth-wise analyses, we demonstrate that growth via gradual middle stacking yields more effective utilization of model depth, alters the residual stream structure, and facilitates the formation of permutable computational blocks. In addition, we propose a lightweight modification of MIDAS that yields further improvements in downstream
reasoning benchmarks. Overall, this work highlights how the gradual growth of model depth can lead to the formation of distinct computational
circuits and overcome the limited depth utilization seen in standard non-grown models.
Read the full abstract ▾
All authors Ferdinand Kapl, Emmanouil Angelis, Tobias Höppe, Kaitlin Maile, Johannes von Oswald, Nino Scherrer, Stefan Bauer
Computation and Language
Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts
Yifan Lyu and Liang Zhang
Understanding human personality is important for personalized recommendation and mental health assessment. Existing personality detection methods typically encode social media posts into user representations and predict personality labels. Despite advances in large
language models (LLMs), these methods face limited supervision from scarce labels and an unclear mapping between user
language and psychological constructs. We propose ROME, a framework that incorporates psychological knowledge into personality detection. Inspired by standardized self-assessment tests, ROME uses LLM role-playing to simulate users' responses to validated psychometric questionnaires. These question-level answers provide interpretable, questionnaire-grounded evidence and intermediate supervision. A question-conditioned Mixture-of-Experts module jointly routes post and question representations to predict questionnaire answers under explicit supervision. The predicted answers are aggregated into an interpretable answer vector and fused with user representations for final personality prediction in a multi-task learning framework. Experiments on two real-world
datasets show that ROME outperforms state-of-the-art baselines, with relative improvements of 13.60% on Kaggle and 19.72% on Pandora. Code is available at https://github.com/JasmineLyf/ROME.
Read the full abstract ▾
Computation and Language
A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs
Mahmoud Srewa, Tianyu Zhao and Salma Elmalaki
This paper addresses the challenge of aligning large
language models (LLMs) with diverse human preferences within
federated learning (FL) environments, where standard methods often fail to adequately represent diverse viewpoints. We introduce a comprehensive evaluation framework that systematically assesses the trade-off between
alignment quality and
fairness when using different aggregation strategies for human preferences. In our
federated setting, each group locally evaluates rollouts and produces reward signals, and the server aggregates these group-level rewards without accessing any raw data. Specifically, we evaluate standard reward aggregation techniques (min, max, and average) and introduce a novel adaptive scheme that dynamically adjusts preference weights based on a group's historical
alignment performance. Our experiments on question-answering (Q/A) tasks using a PPO-based RLHF pipeline demonstrate that our adaptive approach consistently achieves superior
fairness while maintaining competitive
alignment scores. This work offers a robust methodology for evaluating LLM behavior across diverse populations and provides a practical solution for developing truly pluralistic and fairly aligned models.
Read the full abstract ▾
Computation and Language
Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages
David Samuel, Lilja Øvrelid, Erik Velldal et al. (+1 more)
We propose a post-training method for lower-resource
languages that preserves the fluency of
language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource
languages lack both
datasets written by native speakers and instruction-tuned
language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligned
language model without any instruction-tuning data in the target
language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine-translated data and
multilingual finetuning. We conduct a case study on Norwegian Bokmål and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.
Read the full abstract ▾
All authors David Samuel, Lilja Øvrelid, Erik Velldal, Andrey Kutuzov
Computation and Language · Cross-listed from cs.CV
Pose-Based Sign Language Spotting via an End-to-End Encoder Architecture
Samuel Ebimobowei Johnny, Blessed Guda, Emmanuel Enejo Aaron et al. (+1 more)
Automatic Sign
Language Recognition (ASLR) has emerged as a vital field for bridging the gap between deaf and hearing communities. However, the problem of sign-to-sign
retrieval or detecting a specific sign within a sequence of continuous signs remains largely unexplored. We define this novel task as Sign
Language Spotting. In this paper, we present a first step toward sign
language retrieval by addressing the challenge of detecting the presence or absence of a query sign
video within a sentence-level gloss or sign
video. Unlike conventional approaches that rely on intermediate gloss recognition or text-based matching, we propose an end-to-end model that directly operates on pose keypoints extracted from sign
videos. Our architecture employs an encoder-only backbone with a binary classification head to determine whether the query sign appears within the target sequence. By focusing on pose representations instead of raw RGB frames, our method significantly reduces computational cost and mitigates visual noise. We evaluate our approach on the Word Presence Prediction
dataset from the WSLP 2025 shared task, achieving 61.88% accuracy and 60.00% F1-score. These results demonstrate the effectiveness of our pose-based framework for Sign
Language Spotting, establishing a strong foundation for future research in automatic sign
language retrieval and verification. Code is available at https://github.com/EbimoJohnny/Pose-Based-Sign-
Language-Spotting
Read the full abstract ▾
All authors Samuel Ebimobowei Johnny, Blessed Guda, Emmanuel Enejo Aaron, Assane Gueye
Computation and Language
Automatic Essay Scoring and Feedback Generation in Basque Language Learning
Ekhi Azurmendi, Xabier Arregi and Oier Lopez de Lacalle
This paper introduces the first publicly available
dataset for Automatic Essay Scoring (AES) and feedback generation in Basque, targeting the CEFR C1 proficiency level. The
dataset comprises 3,200 essays from HABE, each annotated by expert evaluators with criterion specific scores covering correctness, richness, coherence, cohesion, and task
alignment enriched with detailed feedback and error examples. We fine-tune open-source models, including RoBERTa-EusCrawl and Latxa 8B/70B, for both scoring and explanation generation. Our experiments show that encoder models remain highly reliable for AES, while supervised fine-tuning (SFT) of Latxa significantly enhances performance, surpassing state-of-the-art (SoTA) closed-source systems such as GPT-5 and Claude Sonnet 4.5 in scoring consistency and feedback quality. We also propose a novel evaluation methodology for assessing feedback generation, combining automatic consistency metrics with expert-based validation of extracted learner errors. Results demonstrate that the fine-tuned Latxa model produces criterion-aligned, pedagogically meaningful feedback and identifies a wider range of error types than proprietary models. This resource and
benchmark establish a foundation for transparent, reproducible, and educationally grounded NLP research in low-resource
languages such as Basque.
Read the full abstract ▾
Computation and Language
An Agentic AI System for Multi-Framework Communication Coding
Bohao Yang, Rui Yang, Joshua M. Biro et al. (+14 more)
Clinical communication is central to patient outcomes, yet large-scale human annotation of patient-provider conversation remains labor-intensive, inconsistent, and difficult to scale. Existing approaches based on large
language models typically rely on single-task models that lack adaptability, interpretability, and reliability, especially when applied across various communication frameworks and clinical domains. In this study, we developed a Multi-framework Structured Agentic AI system for Clinical Communication (MOSAIC), built on a LangGraph-based architecture that orchestrates four core
agents, including a Plan
Agent for codebook selection and workflow planning, an Update
Agent for maintaining up-to-date
retrieval databases, a set of Annotation
Agents that applies codebook-guided
retrieval-augmented generation (RAG) with dynamic few-shot prompting, and a Verification
Agent that provides consistency checks and feedback. To evaluate performance, we compared MOSAIC outputs against gold-standard annotations created by trained human coders. We developed and evaluated MOSAIC using 26 gold standard annotated transcripts for training and 50 transcripts for testing, spanning rheumatology and OB/GYN domains. On the test set, MOSAIC achieved an overall F1 score of 0.928. Performance was highest in the Rheumatology subset (F1 = 0.962) and strongest for Patient Behavior (e.g., patients asking questions, expressing preferences, or showing assertiveness). Ablations revealed that MOSAIC outperforms baseline benchmarking.
Read the full abstract ▾
All authors Bohao Yang, Rui Yang, Joshua M. Biro, Haoyuan Wang, Jessica L. Handley, Brianna N. Richardson, Sophia Bessias, Nicoleta Economou-Zavlanos, Armando D. Bedoya, Monica Agrawal, Michael M. Zavlanos, Anand Chowdhury, Raj M. Ratwani, Kai Sun, Kathryn I. Pollak, Michael J. Pencina, Chuan Hong
Computation and Language
QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models
Maximilian Kreutner, Jens Rupprecht, Georg Ahnert et al. (+2 more)
We introduce QSTN, an open-source Python framework for systematically generating responses from questionnaire-style
prompts to support in-silico surveys and annotation tasks with large
language models (LLMs). QSTN enables robust evaluation of questionnaire presentation,
prompt perturbations, and response generation methods. Our extensive evaluation (>40 million survey responses) shows that question structure and response generation methods have a significant impact on the
alignment of generated survey responses with human answers. We also find that answers can be obtained for a fraction of the compute cost, by changing the presentation method. In addition, we offer a no-code user interface that allows researchers to set up robust experiments with LLMs without coding knowledge. We hope that QSTN will support the reproducibility and reliability of LLM-based research in the future.
Read the full abstract ▾
All authors Maximilian Kreutner, Jens Rupprecht, Georg Ahnert, Ahmed Salem, Markus Strohmaier
Computation and Language
HealthcareNLP: where are we and what is next?
Lifeng Han, Paul Rayson, Andrew Moore et al. (+2 more)
This tutorial focused on Healthcare Domain Applications of NLP, what we have achieved around HealthcareNLP, and the challenges that lie ahead for the future. Existing reviews in this domain either overlook some important tasks, such as synthetic data generation for addressing
privacy concerns, or explainable clinical NLP for improved integration and implementation, or fail to mention important methodologies, including
retrieval augmented generation and the neural symbolic integration of LLMs and KGs. In light of this, the goal of this tutorial is to provide an introductory overview of the most important sub-areas of a patient- and resource-oriented HealthcareNLP, with three layers of hierarchy: data/resource layer: annotation guidelines, ethical approvals, governance, synthetic data; NLP-Eval layer: NLP tasks such as NER, RE, sentiment analysis, and linking/coding with categorised methods, leading to explainable HealthAI; patients layer: Patient Public Involvement and Engagement (PPIE), health literacy, translation, simplification, and summarisation (also NLP tasks), and shared decision-making support. A hands-on session will be included in the tutorial for the audience to use HealthcareNLP applications. The target audience includes NLP practitioners in the healthcare application domain, NLP researchers who are interested in domain applications, healthcare researchers, and students from NLP fields. The type of tutorial is "Introductory to CL/NLP topics (HealthcareNLP)" and the audience does not need prior knowledge to attend this. Tutorial materials: https://github.com/4dpicture/HealthNLP
Read the full abstract ▾
All authors Lifeng Han, Paul Rayson, Andrew Moore, Goran Nenadic, Suzan Verberne
Computation and Language
Curriculum Guided Massive Multi Agent System Solving For Robust Long Horizon Tasks
Indrajit Kar and Kalathur Chenchu Kishore Kumar
Large
Language Models and multi-
agent systems have shown promise in decomposing complex tasks, yet they struggle with long-horizon
reasoning tasks and escalating computation cost. This work introduces a hierarchical multi-
agent architecture that distributes
reasoning across a 64*64 grid of lightweight
agents, supported by a selective oracle. A spatial curriculum progressively expands the operational region of the grid, ensuring that
agents master easier central tasks before tackling harder peripheral ones. To improve reliability, the system integrates Negative Log-Likelihood as a measure of confidence, allowing the curriculum to prioritize regions where
agents are both accurate and well calibrated. A Thompson Sampling curriculum manager adaptively chooses training zones based on competence and NLL-driven reward signals. We evaluate the approach on a spatially grounded Tower of Hanoi
benchmark, which mirrors the long-horizon structure of many robotic manipulation and planning tasks. Results demonstrate improved stability, reduced oracle usage, and stronger long-range
reasoning from distributed
agent cooperation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Beyond Real Weights: Hypercomplex Representations for Stable Quantization
Jawad Ibn Ahad, Maisha Rahman, Amrijit Biswas et al. (+5 more)
Multimodal language models (MLLMs) require large parameter capacity to align high-dimensional visual features with linguistic representations, making them computationally heavy and difficult to deploy efficiently. We introduce a progressive reparameterization strategy that compresses these models by gradually replacing dense feed-forward network blocks with compact Parameterized Hypercomplex Multiplication (PHM) layers. A residual interpolation schedule, together with lightweight reconstruction and knowledge
distillation losses, ensures that the PHM modules inherit the functional behavior of their dense counterparts during training. This transition yields substantial parameter and FLOP reductions while preserving strong
multimodal alignment, enabling faster inference without degrading output quality. We evaluate the approach on multiple
vision-
language models (VLMs). Our method maintains performance comparable to the base models while delivering significant reductions in model size and inference latency. Progressive PHM substitution thus offers an architecture-compatible path toward more efficient
multimodal reasoning and complements existing low-bit quantization techniques.
Read the full abstract ▾
All authors Jawad Ibn Ahad, Maisha Rahman, Amrijit Biswas, Muhammad Rafsan Kabir, Robin Krambroeckers, Sifat Momen, Nabeel Mohammed, Shafin Rahman
Computation and Language
Soft Inductive Bias Approach via Explicit Reasoning Perspectives in Inappropriate Utterance Detection Using Large Language Models
Ju-Young Kim, Ji-Hong Park, Se-Yeon Lee et al. (+2 more)
Recent incidents in certain online games and communities, where anonymity is guaranteed, show that unchecked inappropriate remarks frequently escalate into verbal abuse and even criminal behavior, raising significant social concerns. Consequently, there is a growing need for research on techniques that can detect inappropriate utterances within conversational texts to help build a safer communication environment. Although large-scale
language models trained on Korean corpora and chain-of-thought
reasoning have recently gained
attention, research applying these approaches to inappropriate utterance detection remains limited. In this study, we propose a soft inductive bias approach that explicitly defines
reasoning perspectives to guide the inference process, thereby promoting rational decision-making and preventing errors that may arise during
reasoning. We fine-tune a Korean large
language model using the proposed method and conduct both quantitative performance comparisons and qualitative evaluations across different training strategies. Experimental results show that the Kanana-1.5 model achieves an average accuracy of 87.0046, improving by approximately 3.89 percent over standard supervised learning. These findings indicate that the proposed method goes beyond simple knowledge imitation by large
language models and enables more precise and consistent judgments through constrained
reasoning perspectives, demonstrating its effectiveness for inappropriate utterance detection.
Read the full abstract ▾
All authors Ju-Young Kim, Ji-Hong Park, Se-Yeon Lee, Sujin Park, Gun-Woo Kim
Computation and Language
What Triggers my Model? Contrastive Explanations Inform Gender Choices by Translation Models
Janiça Hackenbuchner, Arda Tezcan and Joke Daems
Interpretability can be implemented to understand decisions taken by (black box) models, such as neural machine translation (NMT) or large
language models (LLMs). Yet, research in this area has been limited in relation to a manifested problem in these models: gender bias. In this work, we aim to move away from simply measuring bias to exploring its origins. Working with gender-ambiguous natural source data, this exploratory study examines which context, in the form of input
tokens in the source sentence (EN), influences (or triggers) the NMT model's choice of a certain gender inflection in the target
languages (DE/ES). To analyse this, we compute saliency attribution based on contrastive translations. We first address the challenge of the lack of a scoring threshold and specifically examine different attribution levels of source words on the model's gender decisions in the translation. We compare salient source words with human perceptions of gender and demonstrate a noticeable overlap between human perceptions and model attribution. Additionally, we provide a linguistic analysis of salient words. Our work showcases the relevance of understanding model translation decisions in terms of gender, how this compares to human decisions and that this information should be leveraged to mitigate gender bias.
Read the full abstract ▾
Computation and Language
Are generative AI text annotations systematically biased?
Sjoerd B. Stolwijk, Mark Boukes and Damian Trilling
This paper investigates bias in GLLM annotations by conceptually replicating manual annotations of Boukes (2024). Using various GLLMs (Llama3.1:8b, Llama3.3:70b, GPT4o, Qwen2.5:72b) in combination with five different
prompts for five concepts (political content, interactivity, rationality, incivility, and ideology). We find GLLMs perform adequate in terms of F1 scores, but differ from manual annotations in terms of prevalence, yield substantively different downstream results, and display systematic bias in that they overlap more with each other than with manual annotations. Differences in F1 scores fail to account for the degree of bias.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Ontology-Based Knowledge Graph Framework for Industrial Standard Documents via Hierarchical and Propositional Structuring
Jiin Park, Hyuna Jeon, Yoonseo Lee et al. (+2 more)
Ontology-based knowledge graph (KG) construction is a core technology that enables multidimensional understanding and advanced
reasoning over domain knowledge. Industrial standards, in particular, contain extensive technical information and complex rules presented in highly structured formats that combine tables, scopes of application, constraints, exceptions, and numerical calculations, making KG construction especially challenging. In this study, we propose a method that organizes such documents into a hierarchical semantic structure, decomposes sentences and tables into atomic propositions derived from conditional and numerical rules, and integrates them into an ontology-knowledge graph through LLM-based triple extraction. Our approach captures both the hierarchical and logical structures of documents, effectively representing domain-specific semantics that conventional methods fail to reflect. To verify its effectiveness, we constructed rule, table, and multi-hop QA
datasets, as well as a toxic clause detection
dataset, from industrial standards, and implemented an ontology-aware KG-RAG framework for comparative evaluation. Experimental results show that our method achieves significant performance improvements across all QA types compared to existing KG-RAG approaches. This study demonstrates that reliable and scalable knowledge representation is feasible even for industrial documents with intertwined conditions, constraints, and scopes, contributing to future domain-specific RAG development and intelligent document management.
Read the full abstract ▾
All authors Jiin Park, Hyuna Jeon, Yoonseo Lee, Jisu Hong, Misuk Kim
Computation and Language · Cross-listed from cs.AI
The High Cost of Incivility: Quantifying Interaction Inefficiency via Multi-Agent Monte Carlo Simulations
Benedikt Mangold
Workplace toxicity is widely recognized as detrimental to organizational culture, yet quantifying its direct impact on operational efficiency remains methodologically challenging due to the ethical and practical difficulties of reproducing conflict in human subjects. This study leverages Large
Language Model (LLM) based Multi-
Agent Systems to simulate 1-on-1
adversarial debates, creating a controlled "sociological sandbox". We employ a Monte Carlo method to simulate hundrets of discussions, measuring the convergence time (defined as the number of arguments required to reach a conclusion) between a baseline control group and treatment groups involving
agents with "toxic" system
prompts. Our results demonstrate a statistically significant increase of approximately 25% in the duration of conversations involving toxic participants. We propose that this "latency of toxicity" serves as a proxy for financial damage in corporate and academic settings. Furthermore, we demonstrate that
agent-based modeling provides a reproducible, ethical alternative to human-subject research for measuring the mechanics of social friction.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Reasoning Models Ace the CFA Exams
Jaisal Patel, Yunzhe Chen, Kaiwen He et al. (+4 more)
Previous research has reported that large
language models (LLMs) demonstrate poor performance on the Chartered Financial Analyst (CFA) exams. However, recent
reasoning models have achieved strong results on graduate-level academic and professional examinations across various disciplines. In this paper, we evaluate state-of-the-art
reasoning models on a set of mock CFA exams consisting of 980 questions across three Level I exams, two Level II exams, and three Level III exams. Using the same pass/fail criteria from prior studies, we find that most models clear all three levels. The models that pass, ordered by overall performance, are Gemini 3.0 Pro, Gemini 2.5 Pro, GPT-5, Grok 4, Claude Opus 4.1, and DeepSeek-V3.1. Specifically, Gemini 3.0 Pro achieves a record score of 97.6% on Level I. Performance is also strong on Level II, led by GPT-5 at 94.3%. On Level III, Gemini 2.5 Pro attains the highest score with 86.4% on multiple-choice questions while Gemini 3.0 Pro achieves 92.0% on constructed-response questions.
Read the full abstract ▾
All authors Jaisal Patel, Yunzhe Chen, Kaiwen He, Keyi Wang, David Li, Kairong Xiao, Xiao-Yang Liu
Computation and Language
ClinicalTrialsHub: Bridging Registries and Literature for Comprehensive Clinical Trial Access
Jiwoo Park, Ruoqi Liu, Avani Jagdale et al. (+5 more)
We present ClinicalTrialsHub, an interactive search-focused platform that consolidates all data from ClinicalTrials.gov and augments it by automatically extracting and structuring trial-relevant information from PubMed research articles. Our system effectively increases access to structured clinical trial data by 83.8% compared to relying on ClinicalTrials.gov alone, with potential to make access easier for patients, clinicians, researchers, and policymakers, advancing evidence-based medicine. ClinicalTrialsHub uses large
language models such as GPT-5.1 and Gemini-3-Pro to enhance accessibility. The platform automatically parses full-text research articles to extract structured trial information, translates user queries into structured database searches, and provides an attributed question-answering system that generates evidence-grounded answers linked to specific source sentences. We demonstrate its utility through a user study involving clinicians, clinical researchers, and PhD students of pharmaceutical sciences and nursing, and a systematic automatic evaluation of its information extraction and question answering capabilities.
Read the full abstract ▾
All authors Jiwoo Park, Ruoqi Liu, Avani Jagdale, Andrew Srisuwananukorn, Jing Zhao, Lang Li, Ping Zhang, Sachin Kumar
Computation and Language
Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward
Sampriti Soor, Suklav Ghosh and Arijit Sur
Language models are vulnerable to short
adversarial suffixes that can reliably alter predictions. Previous works usually find such suffixes with gradient search or rule-based methods, but these are brittle and often tied to a single task or model. In this paper, a reinforcement learning framework is used where the suffix is treated as a policy and trained with Proximal Policy Optimization against a frozen model as a reward oracle. Rewards are shaped using calibrated cross-
entropy, removing label bias and aggregating across surface forms to improve transferability. The proposed method is evaluated on five diverse NLP
benchmark datasets, covering sentiment, natural
language inference, paraphrase, and commonsense
reasoning, using three distinct
language models: Qwen2-1.5B Instruct, TinyLlama-1.1B Chat, and Phi-1.5. Results show that RL-trained suffixes consistently degrade accuracy and transfer more effectively across tasks and models than previous
adversarial triggers of similar genres.
Read the full abstract ▾
Computation and Language
Universal Adversarial Suffixes Using Calibrated Gumbel-Softmax Relaxation
Sampriti Soor, Suklav Ghosh and Arijit Sur
Language models (LMs) are often used as zero-shot or few-shot classifiers by scoring label words, but they remain fragile to
adversarial prompts. Prior work typically optimizes task- or model-specific triggers, making results difficult to compare and limiting transferability. We study universal
adversarial suffixes: short
token sequences (4-10
tokens) that, when appended to any input, broadly reduce accuracy across tasks and models. Our approach learns the suffix in a differentiable "soft" form using Gumbel-Softmax relaxation and then discretizes it for inference. Training maximizes calibrated cross-
entropy on the label region while masking gold
tokens to prevent trivial leakage, with
entropy regularization to avoid collapse. A single suffix trained on one model transfers effectively to others, consistently lowering both accuracy and calibrated confidence. Experiments on sentiment analysis, natural
language inference, paraphrase detection, commonsense QA, and physical
reasoning with Qwen2-1.5B, Phi-1.5, and TinyLlama-1.1B demonstrate consistent attack effectiveness and transfer across tasks and model families.
Read the full abstract ▾