Computation and Language
Categorical Emotions or Appraisals - Which Emotion Model Explains Argument Convincingness Better?
Lynn Greschner, Meike Bauer, Sabine Weber et al. (+1 more)
The convincingness of an argument does not only depend on its structure (logos), the person who makes the argument (ethos), but also on the
emotion that it causes in the recipient (pathos). While the overall intensity and categorical values of
emotions in arguments have received considerable
attention in the research community, we argue that the
emotion an argument evokes in a recipient is subjective. It depends on the recipient's goals, standards, prior knowledge, and stance. Appraisal theories lend themselves as a link between the subjective cognitive assessment of events and
emotions. They have been used in event-centric
emotion analysis, but their suitability for assessing argument convincingness remains unexplored. In this paper, we evaluate whether appraisal theories are suitable for
emotion analysis in arguments by considering subjective cognitive evaluations of the importance and impact of an argument on its receiver. Based on the annotations in the recently published ContArgA corpus, we perform zero-shot prompting experiments to evaluate the importance of gold-annotated and predicted
emotions and appraisals for the assessment of the subjective convincingness labels. We find that, while categorical
emotion information does improve convincingness prediction, the improvement is more pronounced with appraisals. This work presents the first systematic comparison between
emotion models for convincingness prediction, demonstrating the advantage of appraisals, providing insights for theoretical and practical applications in computational argumentation.
Read the full abstract ▾
All authors Lynn Greschner, Meike Bauer, Sabine Weber, Roman Klinger
Computation and Language
TCM-Eval: An Expert-Level Dynamic and Extensible Benchmark for Traditional Chinese Medicine
Zihao Cheng, Yuheng Lu, Huaiqian Ye et al. (+10 more)
Large
Language Models (LLMs) have demonstrated remarkable capabilities in modern medicine, yet their application in Traditional Chinese Medicine (TCM) remains severely limited by the absence of standardized
benchmarks and the scarcity of high-quality training data. To address these challenges, we introduce TCM-Eval, the first dynamic and extensible
benchmark for TCM, meticulously curated from national medical licensing examinations and validated by TCM experts. Furthermore, we construct a large-scale training corpus and propose Self-Iterative Chain-of-Thought Enhancement (SI-CoTE) to autonomously enrich question-answer pairs with validated
reasoning chains through rejection sampling, establishing a virtuous cycle of data and model co-
evolution. Using this enriched training data, we develop ZhiMingTang (ZMT), a state-of-the-art LLM specifically designed for TCM, which significantly exceeds the passing threshold for human practitioners. To encourage future research and development, we release a public leaderboard, fostering community engagement and continuous improvement.
Read the full abstract ▾
All authors Zihao Cheng, Yuheng Lu, Huaiqian Ye, Zeming Liu, Minqi Wang, Jingjing Liu, Zihan Li, Wei Fan, Yuanfang Guo, Ruiji Fu, Shifeng She, Gang Wang, Yunhong Wang
Computation and Language
LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
Seungeon Lee, Soumi Das, Manish Gupta et al. (+1 more)
Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for fine-tuning large
language models. However, conventional LoRA adapters are typically trained for a single task, limiting their applicability in real-world settings where inputs may span diverse and unpredictable domains. At inference time, existing approaches combine multiple LoRAs for improving performance on diverse tasks, while usually requiring labeled data or additional task-specific training, which is expensive at scale. In this work, we introduce LoRA on the Go (LoGo), a training-free framework that dynamically selects and merges adapters at the instance level without any additional requirements. LoGo leverages signals extracted from a single forward pass through LoRA adapters, to identify the most relevant adapters and determine their contributions on-the-fly. Across 5 NLP
benchmarks, 27
datasets, and 3 model families, LoGo outperforms training-based baselines on some tasks upto a margin of 3.6% while remaining competitive on other tasks and maintaining inference throughput, highlighting its effectiveness and practicality.
Read the full abstract ▾
All authors Seungeon Lee, Soumi Das, Manish Gupta, Krishna P. Gummadi
Computation and Language
Think Consistently, Reason Efficiently: Energy-Based Calibration for Implicit Chain-of-Thought
Zhikang Chen, Sen Cui, Deheng Ye et al. (+3 more)
Large
Language Models (LLMs) have demonstrated strong
reasoning capabilities through Chain-of-Thought (CoT) prompting, which enables step-by-step intermediate
reasoning. However, explicit CoT methods rely on discrete
token-level
reasoning processes that are prone to error propagation and limited by vocabulary expressiveness, often resulting in rigid and inconsistent
reasoning trajectories. Recent research has explored implicit or continuous
reasoning in latent spaces, allowing models to perform internal
reasoning before generating explicit output. Although such approaches alleviate some limitations of discrete CoT, they generally lack explicit mechanisms to enforce consistency among
reasoning steps, leading to divergent
reasoning paths and unstable outcomes. To address this issue, we propose EBM-CoT, an Energy-Based Chain-of-Thought Calibration framework that refines latent thought representations through an energy-based model (EBM). Our method dynamically adjusts latent
reasoning trajectories toward lower-energy, high-consistency regions in the
embedding space, improving both
reasoning accuracy and consistency without modifying the base
language model. Extensive experiments across mathematical, commonsense, and symbolic
reasoning benchmarks demonstrate that the proposed framework significantly enhances the consistency and efficiency of multi-step
reasoning in LLMs.
Read the full abstract ▾
All authors Zhikang Chen, Sen Cui, Deheng Ye, Yu Zhang, Yatao Bian, Tingting Zhu
Computation and Language
More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists
Khashayar Alavi, Zhastay Yeltay, Lucie Flek et al. (+1 more)
When LLM
agents work together, they seem to be more powerful than a single LLM in mathematical question answering. However, are they also more robust to
adversarial inputs? We investigate this question using adversarially perturbed math questions. These perturbations include punctuation noise with three intensities (10%, 30%, 50%), plus real-world and human-like typos (WikiTypo, R2ATA). Using a unified sampling-and-voting framework (
Agent Forest), we evaluate six open-source models (Qwen3-4B/14B, Llama3.1-8B, Mistral-7B, Gemma3-4B/12B) across four
benchmarks (GSM8K, MATH, MMLU-Math, MultiArith), with various numbers of
agents n = {1,2,5,10,15,20,25}. Our findings show that 1) Noise type matters: punctuation noise harm scales with its severity, and the human typos remain the dominant bottleneck, yielding the largest gaps to Clean accuracy and the highest attack success rate (ASR) even with a large number of
agents; 2) Collaboration reliably improves accuracy as the number of
agents, n, increases, with the largest gains from n=1 to n=5 and diminishing returns beyond n$\approx$10. However, the
adversarial robustness gap persists regardless of the
agent count.
Read the full abstract ▾
All authors Khashayar Alavi, Zhastay Yeltay, Lucie Flek, Akbar Karimi
Computation and Language · Cross-listed from cs.AI
MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs
Liang Shan, Kaicheng Shen, Wen Wu et al. (+8 more)
Ensuring the safety of Large
Language Models (LLMs) is critical for real-world deployment. However, current safety measures often fail to address implicit, domain-specific risks. To investigate this gap, we introduce a
dataset of 3,000 annotated queries spanning education, finance, and management. Evaluations across 14 leading LLMs reveal a concerning vulnerability: an average jailbreak success rate of 57.8%. In response, we propose MENTOR, a metacognition-driven self-
evolution framework. MENTOR performs metacognitive self-assessment, using strategies such as perspective-taking and consequential
reasoning to uncover latent model misalignments. MENTOR couples single-pass rule-guided inference for routine requests with a selectively invoked metacognitive
evolution cycle that revises residual unsafe responses, distills successful corrections into a dynamic rule graph, and compiles validated rules into activation-level steering signals for future inference. Experiments demonstrate that MENTOR substantially reduces attack success rates across all tested domains and outperforms existing safety
alignment methods. The code and
dataset for MENTOR are available at: https://anonymous.4open.science/r/MENTOR-Evo.
Read the full abstract ▾
All authors Liang Shan, Kaicheng Shen, Wen Wu, Zhenyu Ying, Chaochao Lu, Yan Teng, Jingqi Huang, Qingshan Liu, Guoqing Wang, Jie Zhou, Liang He
Computation and Language
Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora
Khalil Hennara, Ahmad Bastati, Muhammad Hreden et al. (+4 more)
The performance of large
language models (LLMs) and large
multimodal models (LMMs) depends heavily on the quality and scale of their pre-training
datasets. Recent research shows that large
multimodal models trained on natural documents where images and text are interleaved outperform those trained only on image-text pairs across a wide range of
benchmarks, leveraging advanced pre-trained models to enforce semantic
alignment, image-sequence consistency, and textual coherence. For Arabic, however, the lack of high-quality
multimodal datasets that preserve document structure has limited progress. In this paper, we present our pipeline Wasm for processing the Common Crawl
dataset to create a new Arabic
multimodal dataset that uniquely provides markdown output. Unlike existing Arabic corpora that focus solely on text extraction, our approach preserves the structural integrity of web content while maintaining flexibility for both text-only and
multimodal pre-training scenarios. We provide a comprehensive comparative analysis of our data processing pipeline against those used for major existing
datasets, highlighting the convergences in filtering strategies and justifying our specific design choices. To support future research, we publicly release a representative
dataset dump along with the
multimodal processing pipeline for Arabic.
Read the full abstract ▾
All authors Khalil Hennara, Ahmad Bastati, Muhammad Hreden, Mohamed Motasim Hamed, Zeina Aldallal, Sara Chrouf, Safwan AlModhayan
Computation and Language
EmoBang: Detecting Emotion From Bengali Texts
Abdullah Al Maruf, Aditi Golder, Zakaria Masud Jiyad et al. (+2 more)
Emotion detection from text seeks to identify an individual's emotional or mental state - positive, negative, or neutral - based on linguistic cues. While significant progress has been made for English and other high-resource
languages, Bengali remains underexplored despite being the world's fourth most spoken
language. The lack of large, standardized
datasets classifies Bengali as a low-resource
language for
emotion detection. Existing studies mainly employ classical machine learning models with traditional feature engineering, yielding limited performance. In this paper, we introduce a new Bengali
emotion dataset annotated across eight
emotion categories and propose two models for automatic
emotion detection: (i) a hybrid Convolutional Recurrent Neural Network (CRNN) model (EmoBangHybrid) and (ii) an AdaBoost-Bidirectional Encoder Representations from
Transformers (BERT) ensemble model (EmoBangEnsemble). Additionally, we evaluate six baseline models with five feature engineering techniques and assess zero-shot and few-shot large
language models (LLMs) on the
dataset. To the best of our knowledge, this is the first comprehensive
benchmark for Bengali
emotion detection. Experimental results show that EmoBangH and EmoBangE achieve accuracies of 92.86% and 93.69%, respectively, outperforming existing methods and establishing strong baselines for future research.
Read the full abstract ▾
All authors Abdullah Al Maruf, Aditi Golder, Zakaria Masud Jiyad, Abdullah Al Numan, Tarannum Shaila Zaman
Computation and Language
Importance-Aware Data Selection for Efficient LLM Instruction Tuning
Tingyu Jiang, Shen Li, Yiyao Song et al. (+6 more)
Instruction tuning plays a critical role in enhancing the performance and efficiency of Large
Language Models (LLMs). Its success depends not only on the quality of the instruction data but also on the inherent capabilities of the LLM itself. Some studies suggest that even a small amount of high-quality data can achieve instruction fine-tuning results that are on par with, or even exceed, those from using a full-scale
dataset. However, rather than focusing solely on calculating data quality scores to evaluate instruction data, there is a growing need to select high-quality data that maximally enhances the performance of instruction tuning for a given LLM. In this paper, we propose the Model Instruction Weakness Value (MIWV) as a novel metric to quantify the importance of instruction data in enhancing model's capabilities. The MIWV metric is derived from the discrepancies in the model's responses when using In-Context Learning (ICL), helping identify the most beneficial data for enhancing instruction tuning performance. Our experimental results demonstrate that selecting only the top 1% of data based on MIWV can outperform training on the full
dataset. Furthermore, this approach extends beyond existing research that focuses on data quality scoring for data selection, offering strong empirical evidence supporting the effectiveness of our proposed method.
Read the full abstract ▾
All authors Tingyu Jiang, Shen Li, Yiyao Song, Lan Zhang, Hualei Zhu, Yuan Zhao, Xiaohang Xu, Kenjiro Taura, Hao Henry Wang
Computation and Language
Aligning Attention with Human Rationales for Self-Explaining Hate Speech Detection
Brage Eilertsen, Røskva Bjørgfinsdóttir, Francielle Vargas et al. (+1 more)
The opaque nature of deep learning models presents significant challenges for the ethical deployment of hate
speech detection systems. To address this limitation, we introduce Supervised Rational
Attention (SRA), a framework that explicitly aligns model
attention with human rationales, improving both interpretability and
fairness in hate
speech classification. SRA integrates a supervised
attention mechanism into
transformer-based classifiers, optimizing a joint objective that combines standard classification loss with an
alignment loss term that minimizes the discrepancy between
attention weights and human-annotated rationales. We evaluated SRA on hate
speech benchmarks in English (HateXplain) and Portuguese (HateBRXplain) with rationale annotations. Empirically, SRA achieves 2.4x better explainability compared to current baselines, and produces
token-level explanations that are more faithful and human-aligned. In terms of
fairness, SRA achieves competitive
fairness across all measures, with second-best performance in detecting toxic posts targeting identity groups, while maintaining comparable results on other metrics. These findings demonstrate that incorporating human rationales into
attention mechanisms can enhance interpretability and faithfulness without compromising
fairness.
Read the full abstract ▾
All authors Brage Eilertsen, Røskva Bjørgfinsdóttir, Francielle Vargas, Ali Ramezani-Kebrya
Computation and Language
Complete Evidence Extraction with Model Ensembles: A Case Study on Medical Coding
Katharina Beckh, Sven Heuser and Stefan Rüping
High-stakes decisions informed by decision support systems require explicit evidence. While prior work focuses on short sufficient evidence, regulatory compliance and medical billing call for complete evidence: all relevant input
tokens that support a decision. We formulate complete evidence extraction as a task and study it in a medical coding setting. Motivated by the Rashomon effect, we aggregate
token-level evidence from multiple
language models to increase evidence completeness. We perform a case study using existing equally-performing models, feature attributions, and a
dataset with human-annotated evidence. Our results show that Rashomon ensembles significantly increase evidence recall while incurring only a small
token overhead over individual models. Ensembles of only three models already outperform the best single model and recover information that individual models miss.
Read the full abstract ▾
Computation and Language
Evaluating Large Language Models for Anxiety, Depression, and Stress Detection: Insights into Prompting Strategies and Synthetic Data
Mihael Arcan and David-Paul Niland
Mental health disorders affect over one-fifth of adults globally, yet detecting such conditions from text remains challenging due to the subtle and varied nature of symptom expression. This study evaluates multiple approaches for mental health detection, comparing Large
Language Models (LLMs) such as Llama and GPT with classical machine learning and
transformer-based architectures including BERT, XLNet, and Distil-RoBERTa. Using the DAIC-WOZ
dataset of clinical interviews, we fine-tuned models for anxiety, depression, and stress classification and applied synthetic data generation to mitigate class imbalance. Results show that Distil-RoBERTa achieved the highest F1 score (0.883) for GAD-2, while XLNet outperformed others on PHQ tasks (F1 up to 0.891). For stress detection, a zero-shot synthetic approach (SD+Zero-Shot-Basic) reached an F1 of 0.884 and ROC AUC of 0.886. Findings demonstrate the effectiveness of
transformer-based models and highlight the value of synthetic data in improving recall and generalization. However, careful calibration is required to prevent precision loss. Overall, this work emphasizes the potential of combining advanced
language models and data augmentation to enhance automated mental health assessment from text.
Read the full abstract ▾
Computation and Language
Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks
Yauhen Babakhin, Radek Osmulski, Ronay Ak et al. (+5 more)
We introduce llama-embed-nemotron-8b, an open-weights text
embedding model that achieves state-of-the-art performance on the
Multilingual Massive Text
Embedding Benchmark (MMTEB) leaderboard as of October 21, 2025. While recent models show strong performance, their training data or methodologies are often not fully disclosed. We aim to address this by developing a fully open-source model, publicly releasing its weights and detailed ablation studies, and planning to share the curated training
datasets. Our model demonstrates superior performance across all major
embedding tasks – including
retrieval, classification and semantic textual similarity (STS) – and excels in challenging
multilingual scenarios, such as low-resource
languages and cross-lingual setups. This state-of-the-art performance is driven by a novel data mix of 16.1 million query-document pairs, split between 7.7 million samples from public
datasets and 8.4 million synthetically generated examples from various open-weight LLMs. One of our key contributions is a detailed ablation study analyzing core design choices, including a comparison of contrastive loss implementations, an evaluation of synthetic data generation (SDG) strategies, and the impact of model merging. The llama-embed-nemotron-8b is an instruction-aware model, supporting user-defined instructions to enhance performance for specific use-cases. This combination of top-tier performance, broad applicability, and user-driven flexibility enables it to serve as a universal text
embedding solution.
Read the full abstract ▾
All authors Yauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, Bo Liu, Even Oldridge
Computation and Language
Multilingual Lexical Feature Analysis of Spoken Language for Predicting Major Depression Symptom Severity
Anastasiia Tokareva, Judith Dineley, Zoe Firth et al. (+19 more)
Background: Remotely captured spoken
language could provide objective, regular indicators of depression symptom severity. However, research to date has largely used non-clinical, cross-sectional written
language and complex machine learning (ML) approaches with limited interpretability. Methods: We used linear mixed-effect models to identify interpretable lexical features associated with symptom severity in data from the RADAR-MDD study that comprised 5,846 smartphone recordings and Patient Health Questionnaire (PHQ-8) scores from 467 participants in the UK, Netherlands and Spain. We then developed ML models and systematically assessed via nested cross-validation whether interpretable lexical features or high-dimensional vector
embeddings improved the accuracy of PHQ-8 prediction over sociodemographic and confounding features. Results: Depression symptom severity was associated with five lexical features, including reductions in word count measures, use of first-person plural pronouns and positive word frequency. Associations were stable across countries, except for positive word frequency. Lexical features and vector
embeddings did improve prediction accuracy beyond baseline models. Limitations: Our cohort was skewed in age (median = 53, IQR 35 to 62) and majority female (n=357), potentially affecting the generalizability of our results. A lack of natural
language processing tools for non-English
languages restricted our feature choices. Conclusion: Further research is required to realise the value of spoken lexical markers in clinical research and practice including larger and more diverse samples, elicitation protocol development and analytical methods that account for within- and between-individual variations.
Read the full abstract ▾
All authors Anastasiia Tokareva, Judith Dineley, Zoe Firth, Pauline Conde, Faith Matcham, Sara Siddi, Femke Lamers, Ewan Carr, Carolin Oetzmann, Daniel Leightley, Yuezhou Zhang, Amos A. Folarin, Josep Maria Haro, Brenda W.J.H. Penninx, Raquel Bailon, Srinivasan Vairavan, Til Wykes, Richard J.B. Dobson, Vaibhav A. Narayan, Matthew Hotopf, Nicholas Cummins, The RADAR-CNS Consortium
Computation and Language
A Picture is Worth a Thousand (Correct) Captions: A Vision-Guided Judge-Corrector System for Multimodal Machine Translation
Siddharth Betala, Kushan Raj, Vipul Betala et al. (+1 more)
In this paper, we describe our system under the team name BLEU Monday for the English-to-Indic
Multimodal Translation Task at WAT 2025. We participate in the text-only translation tasks for English-Hindi, English-Bengali, English-Malayalam, and English-Odia
language pairs. We present a two-stage approach that addresses quality issues in the training data through automated error detection and correction, followed by parameter-efficient model fine-tuning. Our methodology introduces a
vision-augmented judge-corrector pipeline that leverages
multimodal language models to systematically identify and correct translation errors in the training data. The judge component classifies translations into three categories: correct, visually ambiguous (requiring image context), or mistranslated (poor translation quality). Identified errors are routed to specialized correctors: GPT-4o-mini regenerates captions requiring visual disambiguation, while IndicTrans2 retranslates cases with pure translation quality issues. This automated pipeline processes 28,928 training examples across four
languages, correcting an average of 17.1% of captions per
language. We then apply Low-Rank Adaptation (LoRA) to fine-tune the IndicTrans2 en-indic 200M distilled model on both original and corrected
datasets. Training on corrected data yields consistent improvements, with BLEU score gains of +1.30 for English-Bengali on the evaluation set (42.00 -> 43.30) and +0.70 on the challenge set (44.90 -> 45.60), +0.60 for English-Odia on the evaluation set (41.00 -> 41.60), and +0.10 for English-Hindi on the challenge set (53.90 -> 54.00).
Read the full abstract ▾
All authors Siddharth Betala, Kushan Raj, Vipul Betala, Rohan Saswade
Computation and Language
NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs
Yingfeng Luo, Ziqiang Xu, Yuxuan Ouyang et al. (+9 more)
Large
language models have significantly advanced
Multilingual Machine Translation (MMT), yet scaling to many
languages while keeping quality robust across directions remains challenging. In this paper, we identify a failure mode of
multilingual supervised fine-tuning (SFT) on multi-way parallel data: when such data are reused symmetrically around a pivot
language (e.g., English), performance on reverse directions (X $\to$ pivot) can drop substantially. We term this phenomenon Directional Degeneration and attribute it to excessive many-to-one mappings, which encourage shortcut learning. We propose Strategic Downsampling (SD), a simple yet effective method to mitigate this degeneration. In addition, we introduce Parallel
Multilingual Prompting (PMP), which augments translation instructions with an auxiliary parallel sentence to promote cross-lingual transfer during training and enables optional test-time enhancement when auxiliary translations are available. We further develop NiuTrans.LMT (Large-scale
Multilingual Translation, abbreviated as LMT), a Chinese-English-centric suite of
multilingual translation models spanning four sizes (0.6B/1.7B/4B/8B) and covering 60
languages and 234 directions. Comprehensive evaluations show that LMT is competitive among open-source MMT systems, and that our 4B LMT model performs on par with or better than substantially larger baselines. We release our models and project resources to support inclusive and scalable MMT.
Read the full abstract ▾
All authors Yingfeng Luo, Ziqiang Xu, Yuxuan Ouyang, Murun Yang, Dingyang Lin, Kaiyan Chang, Tong Zheng, Bei Li, Peinan Feng, Quan Du, Tong Xiao, Jingbo Zhu
Computation and Language
Automated Attribution Graph Interpretation via Probe Prompting
Giuseppe Birardi and Gonçalo Paulo
Even though we know the precise computations that lead from a large
language model (LLM) input to its output this computation remains very hard to interpret. One way to make it easier to understand this process is by creating a sparse computational graph that captures most of the model behavior with smallest number of computational nodes. Cross-layer transcoders (CLT) decompose the dense computations of the MLP but the resulting
circuits still contain thousands of nodes even for short
prompts. Existing automated interpretation methods label individual features from corpus activations, and it often happens that these labels are not validated by causal intervention. We introduce probe prompting, a transparent rule-based pipeline that groups the features of an attribution graph into concept-aligned supernodes from their responses on a small set of concept-targeted probe
prompts, summarized as Cross-
Prompt Activation Signatures (CPAS). Across four factual domains, on Gemma-2-2B with a public CLT dictionary and 45,596 entity-swap interventions, we find that the labeled supernodes have the predicted steering behavior in every one of them. Code,
datasets, and an interactive demo are released anonymously as a reusable harness for calibrating supernode labels against causal interventions.
Read the full abstract ▾
Computation and Language
SCOPE: Intrinsic Semantic Space Control for Mitigating Copyright Infringement in LLMs
Zhenliang Zhang, Xinyu Hu and Xiaojun Wan
Large
language models sometimes inadvertently reproduce passages that are copyrighted, exposing downstream applications to legal risk. Most existing studies for inference-time defences focus on surface-level
token matching and rely on external blocklists or filters, which add deployment complexity and may overlook semantically paraphrased leakage. In this work, we reframe copyright infringement mitigation as intrinsic semantic-space control and introduce SCOPE, an inference-time method that requires no parameter updates or auxiliary filters. Specifically, the sparse autoencoder (SAE) projects hidden states into a high-dimensional, near-monosemantic space; benefiting from this representation, we identify a copyright-sensitive subspace and clamp its activations during decoding. Experiments on widely recognized
benchmarks show that SCOPE mitigates copyright infringement without degrading general utility. Further interpretability analyses confirm that the isolated subspace captures high-level semantics.
Read the full abstract ▾
Computation and Language
HLPD: Aligning LLMs to Human Language Preference for Machine-Revised Text Detection
Fangqi Dai, Xingjian Jiang and Zizhuang Deng
To prevent misinformation and social issues arising from trustworthy-looking content generated by LLMs, it is crucial to develop efficient and reliable methods for identifying the source of texts. Previous approaches have demonstrated exceptional performance in detecting texts fully generated by LLMs. However, these methods struggle when confronting more advanced LLM output or text with
adversarial multi-task machine revision, especially in the black-box setting, where the generating model is unknown. To address this challenge, grounded in the hypothesis that human writing possesses distinctive stylistic patterns, we propose Human
Language Preference Detection (HLPD). HLPD employs a reward-based
alignment process, Human
Language Preference Optimization (HLPO), to shift the scoring model's
token distribution toward human-like writing, making the model more sensitive to human writing, therefore enhancing the identification of machine-revised text. We test HLPD in an
adversarial multi-task evaluation framework that leverages a five-dimensional
prompt generator and multiple advanced LLMs to create diverse revision scenarios. When detecting texts revised by GPT-series models, HLPD achieves a 15.11% relative improvement in AUROC over ImBD, surpassing Fast-DetectGPT by 45.56%. When evaluated on texts generated by advanced LLMs, HLPD achieves the highest average AUROC, exceeding ImBD by 5.53% and Fast-DetectGPT by 34.14%. Code will be made available at https://github.com/dfq2021/HLPD.
Read the full abstract ▾
Computation and Language
RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation
Haofeng Wang and Yu Zhang
Large
Vision-
Language Models (LVLMs) excel in
multimodal reasoning and have shown impressive performance on various
multimodal benchmarks. However, most of these
benchmarks evaluate models primarily through multiple-choice or short-answer formats, which do not take the
reasoning process into account. Although some
benchmarks assess the
reasoning process, their methods are often overly simplistic and only examine
reasoning when answers are incorrect. This approach overlooks scenarios where flawed
reasoning leads to correct answers. In addition, these
benchmarks do not consider the impact of intermodal relationships on
reasoning. To address this issue, we propose the
Reasoning Process Tree Score (RPTS), a tree structure-based metric to assess
reasoning processes. Specifically, we organize the
reasoning steps into a
reasoning tree and leverage its hierarchical information to assign weighted faithfulness scores to each
reasoning step. By dynamically adjusting these weights, RPTS not only evaluates the overall correctness of the
reasoning, but also pinpoints where the model fails in the
reasoning. To validate RPTS in real-world
multimodal scenarios, we construct a new
benchmark, RPTS-Eval, comprising 374 images and 390
reasoning instances. Each instance includes reliable visual-textual clues that serve as leaf nodes of the
reasoning tree. Furthermore, we define three types of intermodal relationships to investigate how intermodal interactions influence the
reasoning process. We evaluated representative LVLMs (e.g., GPT4o, Llava-Next), uncovering their limitations in
multimodal reasoning and highlighting the differences between open-source and closed-source commercial LVLMs. We believe that this
benchmark will contribute to the advancement of research in the field of
multimodal reasoning.
Read the full abstract ▾
Computation and Language
EduGuardBench: A Holistic Benchmark for Evaluating the Pedagogical Fidelity and Adversarial Safety of LLMs as Simulated Teachers
Yilin Jiang, Mingzi Zhang, Xuanyu Yin et al. (+5 more)
Large
Language Models for Simulating Professions (SP-LLMs), particularly as teachers, are pivotal for personalized education. However, ensuring their professional competence and ethical safety is a critical challenge, as existing
benchmarks fail to measure role-playing fidelity or address the unique teaching harms inherent in educational scenarios. To address this, we propose EduGuardBench, a dual-component
benchmark. It assesses professional fidelity using a Role-playing Fidelity Score (RFS) while diagnosing harms specific to the teaching profession. It also probes safety vulnerabilities using persona-based
adversarial prompts targeting both general harms and, particularly, academic misconduct, evaluated with metrics including Attack Success Rate (ASR) and a three-tier Refusal Quality assessment. Our extensive experiments on 14 leading models reveal a stark polarization in performance. While
reasoning-oriented models generally show superior fidelity, incompetence remains the dominant failure mode across most models. The
adversarial tests uncovered a counterintuitive scaling paradox, where mid-sized models can be the most vulnerable, challenging monotonic safety assumptions. Critically, we identified a powerful Educational Transformation Effect: the safest models excel at converting harmful requests into teachable moments by providing ideal Educational Refusals. This capacity is strongly negatively correlated with ASR, revealing a new dimension of advanced AI safety. EduGuardBench thus provides a reproducible framework that moves beyond siloed knowledge tests toward a holistic assessment of professional, ethical, and pedagogical
alignment, uncovering complex dynamics essential for deploying trustworthy AI in education. See https://github.com/YL1N/EduGuardBench for Materials.
Read the full abstract ▾
All authors Yilin Jiang, Mingzi Zhang, Xuanyu Yin, Sheng Jin, Suyu Lu, Zuocan Ying, Zengyi Yu, Xiangjie Kong
Computation and Language
Inclusion of Role into Named Entity Recognition and Ranking
Neelesh Kumar Shukla and Sanasam Ranbir Singh
Most of the Natural
Language Processing systems are involved in entity-based processing for several tasks like Information Extraction, Question-Answering, Text-Summarization and so on. A new challenge comes when entities play roles according to their act or attributes in certain context. Entity Role Detection is the task of assigning such roles to the entities. Usually real-world entities are of types: person, location and organization etc. Roles could be considered as domain-dependent subtypes of these types. In the cases, where retrieving a subset of entities based on their roles is needed, poses the problem of defining the role and entities having those roles. This paper presents the study of study of solving Entity Role Detection problem by modeling it as Named Entity Recognition (NER) and Entity
Retrieval/Ranking task. In NER, these roles could be considered as mutually exclusive classes and standard NER methods like sequence tagging could be used. For Entity
Retrieval, Roles could be formulated as Query and entities as Collection on which the query needs to be executed. The aspect of Entity
Retrieval task, which is different than document
retrieval task is that the entities and roles against which they need to be retrieved are indirectly described. We have formulated automated ways of learning representative words and phrases and building representations of roles and entities using them. We have also explored different contexts like sentence and document. Since the roles depend upon context, so it is not always possible to have large domain-specific
dataset or knowledge bases for learning purposes, so we have tried to exploit the information from small
dataset in domain-agnostic way.
Read the full abstract ▾
Computation and Language
CLiFT-ASR: A Cross-Lingual Fine-Tuning Framework for Low-Resource Taiwanese Hokkien Speech Recognition
Hung-Yang Sung, Chien-Chun Wang, Kuan-Tang Huang et al. (+4 more)
Automatic
speech recognition (ASR) for low-resource
languages such as Taiwanese Hokkien is difficult due to the scarcity of annotated data. However, direct fine-tuning on Han-character transcriptions often fails to capture detailed phonetic and tonal cues, while training only on romanization lacks lexical and syntactic coverage. In addition, prior studies have rarely explored staged strategies that integrate both annotation types. To address this gap, we present CLiFT-ASR, a cross-lingual fine-tuning framework that builds on Mandarin HuBERT models and progressively adapts them to Taiwanese Hokkien. The framework employs a two-stage process in which it first learns acoustic and tonal representations from phonetic Tai-lo annotations and then captures vocabulary and syntax from Han-character transcriptions. This progressive adaptation enables effective
alignment between
speech sounds and orthographic structures. Experiments on the TAT-MOE corpus demonstrate that CLiFT-ASR achieves a 24.88% relative reduction in character error rate (CER) compared with strong baselines. The results indicate that CLiFT-ASR provides an effective and parameter-efficient solution for Taiwanese Hokkien ASR and that it has potential to benefit other low-resource
language scenarios.
Read the full abstract ▾
All authors Hung-Yang Sung, Chien-Chun Wang, Kuan-Tang Huang, Tien-Hong Lo, Yu-Sheng Tsao, Yung-Chang Hsu, Berlin Chen
Computation and Language
Beyond Plain Demos: A Demo-centric Anchoring Paradigm for In-Context Learning in Alzheimer's Disease Detection
Puzhen Su, Haoran Yin, Yongzhu Miao et al. (+3 more)
Detecting Alzheimer's disease (AD) from narrative transcripts challenges large
language models (LLMs): pre-training rarely covers this out-of-distribution task, and all transcript demos describe the same scene, producing highly homogeneous contexts. These factors cripple both the model's built-in task knowledge (task cognition) and its ability to surface subtle, class-discriminative cues (contextual perception). Because cognition is fixed after pre-training, improving in-context learning (ICL) for AD detection hinges on enriching perception through better demonstration (demo) sets. We demonstrate that standard ICL quickly saturates, its demos lack diversity (context width) and fail to convey fine-grained signals (context depth), and that recent task vector (TV) approaches improve broad task adaptation by injecting TV into the LLMs' hidden states (HSs), they are ill-suited for AD detection due to the mismatch of injection granularity, strength and position. To address these bottlenecks, we introduce DA4ICL, a demo-centric anchoring framework that jointly expands context width via \emph{Diverse and Contrastive
Retrieval} (DCR) and deepens each demo's signal via \emph{Projected Vector Anchoring} (PVA) at every
Transformer layer. Across three AD
benchmarks, DA4ICL achieves large, stable gains over both ICL and TV baselines, charting a new paradigm for fine-grained, OOD and low-resource LLM adaptation.
Read the full abstract ▾
All authors Puzhen Su, Haoran Yin, Yongzhu Miao, Jintao Tang, Shasha Li, Ting Wang
Computation and Language
Learning to Focus: Focal Attention for Selective and Scalable Transformers
Dhananjay Ram, Wei Xia and Stefano Soatto
Attention is a core component of
transformer architecture, whether encoder-only, decoder-only, or encoder-decoder model. However, the standard softmax
attention often produces noisy probability distribution, which can impair effective feature selection at every layer of these models, particularly for long contexts. We propose Focal
Attention, a simple yet effective modification that sharpens the
attention distribution by controlling the softmax temperature, either as a fixed hyperparameter or as a learnable parameter during training. This sharpening enables the model to concentrate on the most relevant
tokens while suppressing irrelevant ones. Empirically, Focal
Attention scales more favorably than standard
transformer with respect to model size, training data, and context length. Across diverse
benchmarks, it achieves the same accuracy with up to 42% fewer parameters or 33% less training data. On long-context tasks, it delivers substantial relative improvements ranging from 17% to 82%, demonstrating its effectiveness in real world applications.
Read the full abstract ▾
Computation and Language
SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces
Ruiheng Liu, XiaoBing Chen, Jinyu Zhang et al. (+3 more)
The rapid advancement of Large
Language Models (LLMs) has driven significant progress in Natural
Language Interface to Database (NLIDB). However, the widespread adoption of LLMs has raised critical
privacy and security concerns. During interactions, LLMs may unintentionally expose confidential database contents or be manipulated by attackers to exfiltrate data through seemingly benign queries. While current efforts typically rely on rule-based heuristics or LLM
agents to mitigate this leakage risk, these methods still struggle with complex inference-based attacks, suffer from high false positive rates, and often compromise the reliability of SQL queries. To address these challenges, we propose SafeNlidb, a novel
privacy-security
alignment framework for LLM-based NLIDB. The framework features an automated pipeline that generates hybrid chain-of-thought interaction data from scratch, seamlessly combining implicit security
reasoning with SQL generation. Additionally, we introduce
reasoning warm-up and alternating preference optimization to overcome the multi-preference oscillations of Direct Preference Optimization (DPO), enabling LLMs to produce security-aware SQL through fine-grained
reasoning without the need for human-annotated preference data. Extensive experiments demonstrate that our method outperforms both larger-scale LLMs and ideal-setting baselines, achieving significant security improvements while preserving high utility. WARNING: This work may contain content that is offensive and harmful!
Read the full abstract ▾
All authors Ruiheng Liu, XiaoBing Chen, Jinyu Zhang, Qiongwen Zhang, Yu Zhang, Bailong Yang
Computation and Language
Sensitivity of Small Language Models to Fine-tuning Data Contamination
Nicy Scaria, Silvester John Joseph Kennedy and Deepak Subramani
Small
Language Models (SLMs) are increasingly being deployed in resource-constrained environments, yet their behavioral robustness to data contamination during instruction tuning remains poorly understood. We systematically investigate the contamination sensitivity of 23 SLMs (270M to 4B parameters) across multiple model families by measuring susceptibility to syntactic and semantic transformation types during instruction tuning: syntactic transformations (character and word reversal) and semantic transformations (irrelevant and counterfactual responses), each applied at contamination levels of 25%, 50%, 75%, and 100%. Our results reveal fundamental asymmetries in vulnerability patterns: syntactic transformations cause catastrophic performance degradation, with character reversal producing near-complete failure across all models regardless of size or family, while semantic transformations demonstrate distinct threshold behaviors and greater resilience in core linguistic capabilities. Critically, we discover a “capability curse" where larger, more capable models become more susceptible to learning semantic corruptions, effectively following harmful instructions more readily, while our analysis of base versus instruction-tuned variants reveals that
alignment provides inconsistent robustness benefits, sometimes even reducing resilience. Our work establishes three core contributions: (1) empirical evidence of SLMs' disproportionate vulnerability to syntactic pattern contamination, (2) identification of asymmetric sensitivity patterns between syntactic and semantic transformations, and (3) systematic evaluation protocols for contamination robustness assessment. These findings have immediate deployment implications, suggesting that current robustness assumptions may not hold for smaller models and highlighting the need for contamination-aware training protocols.
Read the full abstract ▾
Computation and Language
Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
Hyunjae Kim, Jiwoong Sohn, Aidan Gilson et al. (+24 more)
Large
language models (LLMs) are transforming the landscape of medicine, yet two fundamental challenges persist: keeping up with rapidly evolving medical knowledge and providing verifiable, evidence-grounded
reasoning.
Retrieval-augmented generation (RAG) has been widely adopted to address these limitations by supplementing model outputs with retrieved evidence. However, whether RAG reliably achieves these goals remains unclear. Here, we present the most comprehensive expert evaluation of RAG in medicine to date. Eighteen medical experts contributed a total of 80,502 annotations, assessing 800 model outputs generated by GPT-4o and Llama-3.1-8B across 200 real-world patient and USMLE-style queries. We systematically decomposed the RAG pipeline into three components: (i) evidence
retrieval (relevance of retrieved passages), (ii) evidence selection (accuracy of evidence usage), and (iii) response generation (factuality and completeness of outputs). Contrary to expectation, standard RAG often degraded performance: only 22% of top-16 passages were relevant, evidence selection remained weak (precision 41-43%, recall 27-49%), and factuality and completeness dropped by up to 6% and 5%, respectively, compared with non-RAG variants.
Retrieval and evidence selection remain key failure points for the model, contributing to the overall performance drop. We further show that simple yet effective strategies, including evidence filtering and query reformulation, substantially mitigate these issues, improving performance on MedMCQA and MedXpertQA by up to 12% and 8.2%, respectively. These findings call for re-examining RAG's role in medicine and highlight the importance of stage-aware evaluation and deliberate system design for reliable medical LLM applications.
Read the full abstract ▾
All authors Hyunjae Kim, Jiwoong Sohn, Aidan Gilson, Nicholas Cochran-Caggiano, Serina Applebaum, Heeju Jin, Seihee Park, Yujin Park, Jiyeong Park, Seoyoung Choi, Brittany Alexandra Herrera Contreras, Thomas Huang, Jaehoon Yun, Ethan F. Wei, Roy Jiang, Leah Colucci, Eric Lai, Amisha Dave, Tuo Guo, Maxwell B. Singer, Yonghoe Koo, Ron A. Adelman, James Zou, Andrew Taylor, Arman Cohan, Hua Xu, Qingyu Chen
Computation and Language · Cross-listed from cs.CV
Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View
Jianyu Qi, Ding Zou, Wenrui Yan et al. (+5 more)
Recent advances in
Multimodal Large
Language Models (MLLMs) have spurred significant progress in Chain-of-Thought (CoT)
reasoning. Building on the success of Deepseek-R1, researchers extended
multimodal reasoning to post-training paradigms based on reinforcement learning (RL), focusing predominantly on mathematical
datasets. However, existing post-training paradigms tend to neglect two critical aspects: (1) The lack of quantifiable difficulty metrics capable of strategically screening samples for post-training optimization. (2) Suboptimal post-training paradigms that fail to jointly optimize perception and
reasoning capabilities. To address this gap, we propose two novel difficulty-aware sampling strategies: Progressive Image Semantic Masking (PISM) quantifies sample hardness through systematic image degradation, while Cross-Modality
Attention Balance (CMAB) assesses cross-modal interaction complexity via
attention distribution analysis. Leveraging these metrics, we design a hierarchical training framework that incorporates both GRPO-only and SFT+GRPO hybrid training paradigms, and evaluate them across six
benchmark datasets. Experiments demonstrate consistent superiority of GRPO applied to difficulty-stratified samples compared to conventional SFT+GRPO pipelines, indicating that strategic data sampling can obviate the need for supervised fine-tuning while improving model accuracy. Our code will be released at https://github.com/qijianyu277/DifficultySampling.
Read the full abstract ▾
All authors Jianyu Qi, Ding Zou, Wenrui Yan, Rui Ma, Jiaxu Li, Zhijie Zheng, Zhiguo Yang, Rongchang Zhao
Computation and Language
Sentiment Analysis On YouTube Comments Using Machine Learning Techniques Based On Video Games Content
Adi Danish Bin Muhammad Amin, Mohaiminul Islam Bhuiyan, Nur Shazwani Kamarudin et al. (+2 more)
The rapid
evolution of the gaming industry, driven by technological advancements and a burgeoning community, necessitates a deeper understanding of user sentiments, especially as expressed on popular social media platforms like YouTube. This study presents a sentiment analysis on
video games based on YouTube comments, aiming to understand user sentiments within the gaming community. Utilizing YouTube API, comments related to various
video games were collected and analyzed using the TextBlob sentiment analysis tool. The pre-processed data underwent classification using machine learning
algorithms, including Naïve Bayes, Logistic Regression, and Support Vector Machine (SVM). Among these, SVM demonstrated superior performance, achieving the highest classification accuracy across different
datasets. The analysis spanned multiple popular gaming
videos, revealing trends and insights into user preferences and critiques. The findings underscore the importance of advanced sentiment analysis in capturing the nuanced
emotions expressed in user comments, providing valuable feedback for game developers to enhance game design and user experience. Future research will focus on integrating more sophisticated natural
language processing techniques and exploring additional data sources to further refine sentiment analysis in the gaming domain.
Read the full abstract ▾
All authors Adi Danish Bin Muhammad Amin, Mohaiminul Islam Bhuiyan, Nur Shazwani Kamarudin, Zulfahmi Toh, Nur Syafiqah Nafis