Computation and Language · Cross-listed from cs.AI
FactGuard: Event-Centric and Commonsense-Guided Fake News Detection
Jing He, Han Zhang, Yuanhui Xiao et al. (+3 more)
Fake news detection methods based on writing style have achieved remarkable progress. However, as adversaries increasingly imitate the style of authentic news, the effectiveness of such approaches is gradually diminishing. Recent research has explored incorporating large
language models (LLMs) to enhance fake news detection. Yet, despite their transformative potential, LLMs remain an untapped goldmine for fake news detection, with their real-world adoption hampered by shallow functionality exploration, ambiguous usability, and prohibitive inference costs. In this paper, we propose a novel fake news detection framework, dubbed FactGuard, that leverages LLMs to extract event-centric content, thereby reducing the impact of writing style on detection performance. Furthermore, our approach introduces a dynamic usability mechanism that identifies contradictions and ambiguous cases in factual
reasoning, adaptively incorporating LLM advice to improve decision reliability. To ensure efficiency and practical deployment, we employ knowledge
distillation to derive FactGuard-D, enabling the framework to operate effectively in cold-start and resource-constrained scenarios. Comprehensive experiments on two
benchmark datasets demonstrate that our approach consistently outperforms existing methods in both robustness and accuracy, effectively addressing the challenges of style sensitivity and LLM usability in fake news detection.
Read the full abstract ▾
All authors Jing He, Han Zhang, Yuanhui Xiao, Wei Guo, Shaowen Yao, Renyang Liu
Computation and Language
MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
He Zhang, Wenqian Cui, Haoning Xu et al. (+5 more)
Full-Duplex
Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing
benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Also, existing
benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel
benchmark designed for a comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our
benchmark. Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench
Read the full abstract ▾
All authors He Zhang, Wenqian Cui, Haoning Xu, Xiaohui Li, Lei Zhu, Haoli Bai, Shaohua Ma, Irwin King
Computation and Language · Cross-listed from cs.AI
Minbae Park, Hyemin Yang, Jeonghyun Kim et al. (+2 more)
Large
Language Models (LLMs) demonstrate strong
reasoning capabilities but struggle with
hallucinations and limited transparency. Recently, KG-enhanced LLMs that integrate knowledge graphs (KGs) have been shown to improve
reasoning performance, particularly for complex, knowledge-intensive tasks. However, these methods still face significant challenges, including inaccurate
retrieval and
reasoning failures, often exacerbated by long input contexts that obscure relevant information or by context constructions that struggle to capture the richer logical directions required by different question types. Furthermore, many of these approaches rely on LLMs to directly retrieve evidence from KGs, and to self-assess the sufficiency of this evidence, which often results in premature or incorrect
reasoning. To address the
retrieval and
reasoning failures, we propose ProgRAG, a multi-hop knowledge graph question answering (KGQA) framework that decomposes complex questions into sub-questions, and progressively extends partial
reasoning paths by answering each sub-question. At each step, external retrievers gather candidate evidence, which is then refined through uncertainty-aware
pruning by the LLM. Finally, the context for LLM
reasoning is optimized by organizing and rearranging the partial
reasoning paths obtained from the sub-question answers. Experiments on three well-known
datasets demonstrate that ProgRAG outperforms existing baselines in multi-hop KGQA, offering improved reliability and
reasoning quality.
Read the full abstract ▾
All authors Minbae Park, Hyemin Yang, Jeonghyun Kim, Kunsoo Park, Hyunjoon Kim
Computation and Language
VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction
Yuhao Wang, Ziyang Cheng, Heyang Liu et al. (+4 more)
Current end-to-end spoken
language models (SLMs) have made notable progress, yet they still encounter considerable response latency. This delay primarily arises from the autoregressive generation of
speech tokens and the reliance on complex flow-matching models for
speech synthesis. To overcome this, we introduce VocalNet-M2, a novel low-latency SLM that integrates a multi-codebook tokenizer and a multi-
token prediction (MTP) strategy. Our model directly generates multi-codebook
speech tokens, thus eliminating the need for a latency-inducing flow-matching model. Furthermore, our MTP strategy enhances generation efficiency and improves overall performance. Extensive experiments demonstrate that VocalNet-M2 achieves a substantial reduction in first chunk latency (from approximately 725ms to 350ms) while maintaining competitive performance across mainstream SLMs. This work also provides a comprehensive comparison of single-codebook and multi-codebook strategies, offering valuable insights for developing efficient and high-performance SLMs for real-time interactive applications.
Read the full abstract ▾
All authors Yuhao Wang, Ziyang Cheng, Heyang Liu, Ronghua Wu, Qunshan Gu, Yanfeng Wang, Yu Wang
Computation and Language
LangGPS: Language Separability Guided Data Pre-Selection for Joint Multilingual Instruction Tuning
Yangfan Ye, Xiaocheng Feng, Xiachong Feng et al. (+7 more)
Joint
multilingual instruction tuning is a widely adopted approach to improve the
multilingual instruction-following ability and downstream performance of large
language models (LLMs), but the resulting
multilingual capability remains highly sensitive to the composition and selection of the training data. Existing selection methods, often based on features like text quality, diversity, or task relevance, typically overlook the intrinsic linguistic structure of
multilingual data. In this paper, we propose LangGPS, a lightweight two-stage pre-selection framework guided by
language separability which quantifies how well samples in different
languages can be distinguished in the model's representation space. LangGPS first filters training data based on separability scores and then refines the subset using existing selection methods. Extensive experiments across six
benchmarks and 22
languages demonstrate that applying LangGPS on top of existing selection methods improves their effectiveness and generalizability in
multilingual training, especially for understanding tasks and low-resource
languages. Further analysis reveals that highly separable samples facilitate the formation of clearer
language boundaries and support faster adaptation, while low-separability samples tend to function as bridges for cross-lingual
alignment. Besides, we also find that
language separability can serve as an effective signal for
multilingual curriculum learning, where interleaving samples with diverse separability levels yields stable and generalizable gains. Together, we hope our work offers a new perspective on data utility in
multilingual contexts and support the development of more linguistically informed LLMs.
Read the full abstract ▾
All authors Yangfan Ye, Xiaocheng Feng, Xiachong Feng, Lei Huang, Weitao Ma, Qichen Hong, Yunfei Lu, Duyu Tang, Dandan Tu, Bing Qin
Computation and Language
Persona-Aware Alignment Framework for Personalized Dialogue Generation
Guanrong Li, Xinyu Liu, Zhen Wu et al. (+1 more)
Personalized dialogue generation aims to leverage persona profiles and dialogue history to generate persona-relevant and consistent responses. Mainstream models typically rely on
token-level
language model training with persona dialogue data, such as Next
Token Prediction, to implicitly achieve personalization, making these methods tend to neglect the given personas and generate generic responses. To address this issue, we propose a novel Persona-Aware
Alignment Framework (PAL), which directly treats persona
alignment as the training objective of dialogue generation. Specifically, PAL employs a two-stage training method including Persona-aware Learning and Persona
Alignment, equipped with an easy-to-use inference strategy Select then Generate, to improve persona sensitivity and generate more persona-relevant responses at the semantics level. Through extensive experiments, we demonstrate that our framework outperforms many state-of-the-art personalized dialogue methods and large
language models.
Read the full abstract ▾
All authors Guanrong Li, Xinyu Liu, Zhen Wu, Xinyu Dai
Computation and Language
EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Models
Junquan Huang, Haotian Wu, Yubo Gao et al. (+7 more)
Large
language models (LLMs) with Chain-of-Thought (CoT) prompting achieve strong
reasoning but often produce unnecessarily long explanations, increasing cost and sometimes reducing accuracy. Fair comparison of efficiency-oriented approaches is hindered by fragmented evaluation practices. We introduce EffiReason-Bench, a unified
benchmark for rigorous cross-paradigm evaluation of efficient
reasoning methods across three categories:
Reasoning Blueprints, Dynamic Execution, and Post-hoc Refinement. To enable step-by-step evaluation, we construct verified CoT annotations for CommonsenseQA and LogiQA via a pipeline that enforces standardized
reasoning structures, comprehensive option-wise analysis, and human verification. We evaluate 7 methods across 6 open-source LLMs (1B-70B) on 4
datasets spanning mathematics, commonsense, and logic, and propose the E3-Score, a principled metric inspired by economic trade-off modeling that provides smooth, stable evaluation without discontinuities or heavy reliance on heuristics. Experiments show that no single method universally dominates; optimal strategies depend on backbone scale, task complexity, and architecture.
Read the full abstract ▾
All authors Junquan Huang, Haotian Wu, Yubo Gao, Yibo Yan, Junyan Zhang, Yonghua Hei, Song Dai, Jie Zhang, Puay Siew Tan, Xuming Hu
Computation and Language
Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQL
Qifeng Cai, Hao Liang, Chang Xu et al. (+3 more)
The data-centric paradigm has emerged as a pivotal direction in artificial intelligence (AI), emphasizing the role of high-quality training data. This shift is especially critical in the Text-to-SQL task, where the scarcity, limited diversity, and structural simplicity of existing
datasets constrain model performance. To address these challenges, we propose Text2SQL-Flow, a SQL-aware data augmentation framework that systematically generates large-scale, semantically valid, and structurally diverse Text-to-SQL pairs from limited seed data. Our framework spans six augmentation dimensions and integrates an end-to-end pipeline with auxiliary database selection, SQL executability verification, natural
language (NL) question generation, NL-SQL correspondence verification, and chain-of-thought (CoT)
reasoning trace generation. Leveraging this framework, we construct SQLFlow, a high-quality
dataset comprising 75,386 annotated examples. We demonstrate the utility of SQLFlow in both fine-tuning and
prompt-based settings. (1) For open-source large
language models (LLMs), fine-tuning with SQLFlow improves problem-solving ability, delivering competitive gains across multiple
benchmarks under the same data budget. (2) For closed-source LLMs, we propose a masked
alignment retrieval method that uses SQLFlow as both a knowledge base and training data for the
retrieval model, enabling structure-aware example matching via fine-grained NL-SQL
alignments. Experiments show that our
retrieval strategy outperforms existing example
retrieval methods, highlighting the combined value of SQLFlow's data quality and our
retrieval technique. Overall, our work provides a scalable, data-centric foundation for advancing Text-to-SQL systems and underscores the importance of structured, high-fidelity data in modern AI development. Our code is available at https://github.com/TechNomad-ds/Text2SQL-Flow.
Read the full abstract ▾
All authors Qifeng Cai, Hao Liang, Chang Xu, Tao Xie, Wentao Zhang, Bin Cui
Computation and Language
Beyond the Black Box: Demystifying Multi-Turn LLM Reasoning with VISTA
Yiran Zhang, Mingyang Lin, Mark Dras et al. (+1 more)
Recent research has increasingly focused on the
reasoning capabilities of Large
Language Models (LLMs) in multi-turn interactions, as these scenarios more closely mirror real-world problem-solving. However, analyzing the intricate
reasoning processes within these interactions presents a significant challenge due to complex contextual dependencies and a lack of specialized visualization tools, leading to a high cognitive load for researchers. To address this gap, we present VISTA, an web-based Visual Interactive System for Textual Analytics in multi-turn
reasoning tasks. VISTA allows users to visualize the influence of context on model decisions and interactively modify conversation histories to conduct "what-if" analyses across different models. Furthermore, the platform can automatically parse a session and generate a
reasoning dependency tree, offering a transparent view of the model's step-by-step logical path. By providing a unified and interactive framework, VISTA significantly reduces the complexity of analyzing
reasoning chains, thereby facilitating a deeper understanding of the capabilities and limitations of current LLMs. The platform is open-source and supports easy integration of custom
benchmarks and local models.
Read the full abstract ▾
All authors Yiran Zhang, Mingyang Lin, Mark Dras, Usman Naseem
Computation and Language
Generalizing to Unseen Disaster Events: A Causal View
Philipp Seeberger, Steffen Freisinger, Tobias Bocklet et al. (+1 more)
Due to the rapid growth of social media platforms, these tools have become essential for monitoring information during ongoing disaster events. However, extracting valuable insights requires real-time processing of vast amounts of data. A major challenge in existing systems is their exposure to event-related biases, which negatively affects their ability to generalize to emerging events. While recent advancements in debiasing and causal learning offer promising solutions, they remain underexplored in the disaster event domain. In this work, we approach bias mitigation through a causal lens and propose a method to reduce event- and domain-related biases, enhancing generalization to future events. Our approach outperforms multiple baselines by up to +1.9% F1 and significantly improves a PLM-based classifier across three disaster classification tasks.
Read the full abstract ▾
All authors Philipp Seeberger, Steffen Freisinger, Tobias Bocklet, Korbinian Riedhammer
Computation and Language
On the Military Applications of Large Language Models
Satu Johansson and Taneli Riihonen
In this paper, military use cases or applications and implementation thereof are considered for natural
language processing and large
language models, which have broken into fame with the invention of the generative pre-trained
transformer (GPT) and the extensive foundation model pretraining done by OpenAI for ChatGPT and others. First, we interrogate a GPT-based
language model (viz. Microsoft Copilot) to make it reveal its own knowledge about their potential military applications and then critically assess the information. Second, we study how commercial cloud services (viz. Microsoft Azure) could be used readily to build such applications and assess which of them are feasible. We conclude that the summarization and generative properties of
language models directly facilitate many applications at large and other features may find particular uses.
Read the full abstract ▾
Computation and Language
ELYADATA & LIA at NADI 2025: ASR and ADI Subtasks
Haroun Elleuch, Youssef Saidi, Salima Mdhaffar et al. (+2 more)
This paper describes Elyadata & LIA's joint submission to the NADI multi-dialectal Arabic
Speech Processing 2025. We participated in the Spoken Arabic Dialect Identification (ADI) and multi-dialectal Arabic ASR subtasks. Our submission ranked first for the ADI subtask and second for the multi-dialectal Arabic ASR subtask among all participants. Our ADI system is a fine-tuned Whisper-large-v3 encoder with data augmentation. This system obtained the highest ADI accuracy score of 79.83% on the official test set. For multi-dialectal Arabic ASR, we fine-tuned SeamlessM4T-v2 Large (Egyptian variant) separately for each of the eight considered dialects. Overall, we obtained an average WER and CER of 38.54% and 14.53%, respectively, on the test set. Our results demonstrate the effectiveness of large pre-trained
speech models with targeted fine-tuning for Arabic
speech processing.
Read the full abstract ▾
All authors Haroun Elleuch, Youssef Saidi, Salima Mdhaffar, Yannick Estève, Fethi Bougares
Computation and Language
Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts
Xanh Ho, Yun-Ang Wu, Sunisth Kumar et al. (+3 more)
With the growing number of submitted scientific papers, there is an increasing demand for systems that can assist reviewers in evaluating research claims. Experimental results are a core component of scientific work, often presented in varying formats such as tables or charts. Understanding how robust current
multimodal large
language models (
multimodal LLMs) are at verifying scientific claims across different evidence formats remains an important and underexplored challenge. In this paper, we design and conduct a series of experiments to assess the ability of
multimodal LLMs to verify scientific claims using both tables and charts as evidence. To enable this evaluation, we adapt two existing
datasets of scientific papers by incorporating annotations and structures necessary for a
multimodal claim verification task. Using this adapted
dataset, we evaluate 12
multimodal LLMs and find that current models perform better with table-based evidence while struggling with chart-based evidence. We further conduct human evaluations and observe that humans maintain strong performance across both formats, unlike the models. Our analysis also reveals that smaller
multimodal LLMs (under 8B) show weak correlation in performance between table-based and chart-based tasks, indicating limited cross-modal generalization. These findings highlight a critical gap in current models'
multimodal reasoning capabilities. We suggest that future
multimodal LLMs should place greater emphasis on improving chart understanding to better support scientific claim verification.
Read the full abstract ▾
All authors Xanh Ho, Yun-Ang Wu, Sunisth Kumar, Florian Boudin, Atsuhiro Takasu, Akiko Aizawa
Computation and Language
ADI-20: Arabic Dialect Identification dataset and models
Haroun Elleuch, Salima Mdhaffar, Yannick Estève et al. (+1 more)
We present ADI-20, an extension of the previously published ADI-17 Arabic Dialect Identification (ADI)
dataset. ADI-20 covers all Arabic-speaking countries' dialects. It comprises 3,556 hours from 19 Arabic dialects in addition to Modern Standard Arabic (MSA). We used this
dataset to train and evaluate various state-of-the-art ADI systems. We explored fine-tuning pre-trained ECAPA-TDNN-based models, as well as Whisper encoder blocks coupled with an
attention pooling layer and a classification dense layer. We investigated the effect of (i) training data size and (ii) the model's number of parameters on identification performance. Our results show a small decrease in F1 score while using only 30% of the original training data. We open-source our collected data and trained models to enable the reproduction of our work, as well as support further research in ADI.
Read the full abstract ▾
All authors Haroun Elleuch, Salima Mdhaffar, Yannick Estève, Fethi Bougares
Computation and Language · Cross-listed from cs.AI
Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning
Yuxuan Zhou, Yubin Wang, Bin Wang et al. (+4 more)
Large
language models (LLMs) have shown great promise in the medical domain, achieving strong performance on several
benchmarks. However, they continue to underperform in real-world medical scenarios, which often demand stronger context-awareness, i.e., the ability to recognize missing or critical details (e.g., user identity, medical history, risk factors) and provide safe, helpful, and contextually appropriate responses. To address this issue, we propose Multifaceted Self-Refinement (MuSeR), a data-driven approach that enhances LLMs' context-awareness along three key facets (decision-making, communication, and safety) through self-evaluation and refinement. Specifically, we first design a attribute-conditioned query generator that simulates diverse real-world user contexts by varying attributes such as role, geographic region, intent, and degree of information ambiguity. An LLM then responds to these queries, self-evaluates its answers along three key facets, and refines its responses to better align with the requirements of each facet. Finally, the queries and refined responses are used for supervised fine-tuning to reinforce the model's context-awareness ability. Evaluation results on the latest HealthBench
dataset demonstrate that our method significantly improves LLM performance across multiple aspects, with particularly notable gains in the context-awareness axis. Furthermore, by incorporating knowledge
distillation with the proposed method, the performance of a smaller backbone LLM (e.g., Qwen3-32B) surpasses its teacher model, achieving a new SOTA across all open-source LLMs on HealthBench (63.8%) and its hard subset (43.1%). Code and
dataset will be released at https://muser-llm.github.io.
Read the full abstract ▾
All authors Yuxuan Zhou, Yubin Wang, Bin Wang, Chen Ning, Xien Liu, Ji Wu, Jianye Hao
Computation and Language
GraphIF: Enhancing Multi-Turn Instruction Following for Large Language Models with Relation Graph Prompt
Zhenhe Li, Can Lin, Ling Zheng et al. (+3 more)
Multi-turn instruction following is essential for building intelligent conversational systems that can consistently adhere to instructions across dialogue turns. However, existing approaches to enhancing multi-turn instruction following primarily rely on collecting or generating large-scale multi-turn dialogue
datasets to fine-tune large
language models (LLMs), which treat each response generation as an isolated task and fail to explicitly incorporate multi-turn instruction following into the optimization objectives. As a result, instruction-tuned LLMs often struggle with complex long-distance constraints. In multi-turn dialogues, relational constraints across turns can be naturally modeled as labeled directed edges, making graph structures particularly suitable for modeling multi-turn instruction following. Despite this potential, leveraging graph structures to enhance the multi-turn instruction following capabilities of LLMs remains unexplored. To bridge this gap, we propose GraphIF, a plug-and-play framework that models multi-turn dialogues as directed relation graphs and leverages graph
prompts to enhance the instruction following capabilities of LLMs. GraphIF comprises three key components: (1) an
agent-based relation extraction module that captures inter-turn semantic relations via action-triggered mechanisms to construct structured graphs; (2) a relation graph
prompt generation module that converts structured graph information into natural
language prompts; and (3) a response rewriting module that refines initial LLM outputs using the generated graph
prompts. Extensive experiments on two long multi-turn dialogue
datasets demonstrate that GraphIF can be seamlessly integrated into instruction-tuned LLMs and leads to significant improvements across all four multi-turn instruction-following evaluation metrics.
Read the full abstract ▾
All authors Zhenhe Li, Can Lin, Ling Zheng, Wen-Da Wei, Junli Liang, Qi Song
Computation and Language
Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism
Jinhong Jeong, Sunghyun Lee, Jaeyoung Lee et al. (+2 more)
Sound symbolism is a linguistic concept that refers to non-arbitrary associations between phonetic forms and their meanings. We suggest that this can be a compelling probe into how
Multimodal Large
Language Models (MLLMs) interpret auditory information in human
languages. We investigate MLLMs' performance on phonetic iconicity across textual (orthographic and IPA) and auditory forms of inputs with up to 25 semantic dimensions (e.g., sharp vs. round), observing models' layer-wise information processing by measuring phoneme-level
attention fraction scores. To this end, we present LEX-ICON, an extensive mimetic word
dataset consisting of 8,052 words from four natural
languages (English, French, Japanese, and Korean) and 2,930 systematically constructed pseudo-words, annotated with semantic features applied across both text and audio modalities. Our key findings demonstrate (1) MLLMs' phonetic intuitions that align with existing linguistic research across multiple semantic dimensions and (2) phonosemantic
attention patterns that highlight models' focus on iconic phonemes. These results bridge domains of artificial intelligence and cognitive linguistics, providing the first large-scale, quantitative analyses of phonetic iconicity in terms of MLLMs' interpretability.
Read the full abstract ▾
All authors Jinhong Jeong, Sunghyun Lee, Jaeyoung Lee, Seonah Han, Youngjae Yu
Computation and Language
ScaleFormer: Span Representation Cumulation for Long-Context Transformer
Jiangshu Du, Wenpeng Yin and Philip Yu
The quadratic complexity of standard self-
attention severely limits the application of
Transformer-based models to long-context tasks. While efficient
Transformer variants exist, they often require architectural changes and costly pre-training from scratch. To circumvent this, we propose ScaleFormer(Span Representation Cumulation for Long-Context
Transformer) - a simple and effective plug-and-play framework that adapts off-the-shelf pre-trained encoder-decoder models to process long sequences without requiring architectural modifications. Our approach segments long inputs into overlapping chunks and generates a compressed, context-aware representation for the decoder. The core of our method is a novel, parameter-free fusion mechanism that endows each chunk's representation with structural awareness of its position within the document. It achieves this by enriching each chunk's boundary representations with cumulative context vectors from all preceding and succeeding chunks. This strategy provides the model with a strong signal of the document's narrative flow, achieves linear complexity, and enables pre-trained models to reason effectively over long-form text. Experiments on long-document summarization show that our method is highly competitive with and often outperforms state-of-the-art approaches without requiring architectural modifications or external
retrieval mechanisms.
Read the full abstract ▾
Computation and Language
PustakAI: Curriculum-Aligned and Interactive Textbooks Using Large Language Models
Shivam Sharma, Riya Naik, Tejas Gawas et al. (+2 more)
Large
Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating human-like content. This has revolutionized various sectors such as healthcare, software development, and education. In education, LLMs offer potential for personalized and interactive learning experiences, especially in regions with limited teaching resources. However, adapting these models effectively to curriculum-specific content, such as the National Council of Educational Research and Training (NCERT) syllabus in India, presents unique challenges in terms of accuracy,
alignment, and pedagogical relevance. In this paper, we present the framework "PustakAI"\footnote{Pustak means `book' in many Indian
languages.} for the design and evaluation of a novel question-answering
dataset "NCERT-QA" aligned with the NCERT curriculum for English and Science subjects of grades 6 to 8. We classify the curated QA pairs as Factoid, Inferential, and Others (evaluative and
reasoning). We evaluate the
dataset with various prompting techniques, such as meta-
prompt, few-shot, and CoT-style prompting, using diverse evaluation metrics to understand which approach aligns more efficiently with the structure and demands of the curriculum. Along with the usability of the
dataset, we analyze the strengths and limitations of current open-source LLMs (Gemma3:1b, Llama3.2:3b, and Nemotron-mini:4b) and high-end LLMs (Llama-4-Scout-17B and Deepseek-r1-70B) as AI-based learning tools in formal education systems.
Read the full abstract ▾
All authors Shivam Sharma, Riya Naik, Tejas Gawas, Heramb Patil, Kunal Korgaonkar
Computation and Language
FinNuE: Exposing the Risks of Using BERTScore for Numerical Semantic Evaluation in Finance
Yu-Shiang Huang, Yun-Yu Lee, Tzu-Hsin Chou et al. (+2 more)
BERTScore has become a widely adopted metric for evaluating semantic similarity between natural
language sentences. However, we identify a critical limitation: BERTScore exhibits low sensitivity to numerical variation, a significant weakness in finance where numerical precision directly affects meaning (e.g., distinguishing a 2% gain from a 20% loss). We introduce FinNuE, a diagnostic
dataset constructed with controlled numerical perturbations across earnings calls, regulatory filings, social media, and news articles. Using FinNuE, demonstrate that BERTScore fails to distinguish semantically critical numerical differences, often assigning high similarity scores to financially divergent text pairs. Our findings reveal fundamental limitations of
embedding-based metrics for finance and motivate numerically-aware evaluation frameworks for financial NLP.
Read the full abstract ▾
All authors Yu-Shiang Huang, Yun-Yu Lee, Tzu-Hsin Chou, Che Lin, Chuan-Ju Wang
Computation and Language
Language Drift in Multilingual Retrieval-Augmented Generation: Characterization and Decoding-Time Mitigation
Bo Li, Zhenghua Xu and Rui Xie
Multilingual Retrieval-Augmented Generation (RAG) enables large
language models (LLMs) to perform knowledge-intensive tasks in
multilingual settings by leveraging retrieved documents as external evidence. However, when the retrieved evidence differs in
language from the user query and in-context exemplars, the model often exhibits
language drift by generating responses in an unintended
language. This phenomenon is especially pronounced during
reasoning-intensive decoding, such as Chain-of-Thought (CoT) generation, where intermediate steps introduce further
language instability. In this paper, we systematically study output
language drift in
multilingual RAG across multiple
datasets,
languages, and LLM backbones. Our controlled experiments reveal that the drift results not from comprehension failure but from decoder-level collapse, where dominant
token distributions and high-frequency English patterns dominate the intended generation
language. We further observe that English serves as a semantic attractor under cross-lingual conditions, emerging as both the strongest interference source and the most frequent fallback
language. To mitigate this, we propose Soft Constrained Decoding (SCD), a lightweight, training-free decoding strategy that gently steers generation toward the target
language by penalizing non-target-
language tokens. SCD is model-agnostic and can be applied to any generation
algorithm without modifying the architecture or requiring additional data. Experiments across three
multilingual datasets and multiple typologically diverse
languages show that SCD consistently improves
language alignment and task performance, providing an effective and generalizable solution in
multilingual RAG.
Read the full abstract ▾
Computation and Language
Modeling Uncertainty Trends for Timely Retrieval in Dynamic RAG
Bo Li, Tian Tian, Zhenghua Xu et al. (+3 more)
Dynamic
retrieval-augmented generation (RAG) allows large
language models (LLMs) to fetch external knowledge on demand, offering greater adaptability than static RAG. A central challenge in this setting lies in determining the optimal timing for
retrieval. Existing methods often trigger
retrieval based on low
token-level confidence, which may lead to delayed intervention after errors have already propagated. We introduce
Entropy-Trend Constraint (ETC), a training-free method that determines optimal
retrieval timing by modeling the dynamics of
token-level uncertainty. Specifically, ETC utilizes first- and second-order differences of the
entropy sequence to detect emerging uncertainty trends, enabling earlier and more precise
retrieval. Experiments on six QA
benchmarks with three LLM backbones demonstrate that ETC consistently outperforms strong baselines while reducing
retrieval frequency. ETC is particularly effective in domain-specific scenarios, exhibiting robust generalization capabilities. Ablation studies and qualitative analyses further confirm that trend-aware uncertainty modeling yields more effective
retrieval timing. The method is plug-and-play, model-agnostic, and readily integrable into existing decoding pipelines. Implementation code is included in the supplementary materials.
Read the full abstract ▾
All authors Bo Li, Tian Tian, Zhenghua Xu, Hao Cheng, Shikun Zhang, Wei Ye
Computation and Language
NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction
Peter Røysland Aarnes and Vinay Setty
Large
language models show strong performance on knowledge intensive tasks such as fact-checking and question answering, yet they often struggle with numerical
reasoning. We present a systematic evaluation of state-of-the-art models for veracity prediction on numerical claims and evidence pairs using controlled perturbations, including label-flipping probes, to test robustness. Our results indicate that even leading proprietary systems experience accuracy drops of up to 62% under certain perturbations. No model proves to be robust across all conditions. We further find that increasing context length generally reduces accuracy, but when extended context is enriched with perturbed demonstrations, most models substantially recover. These findings highlight critical limitations in numerical fact-checking and suggest that robustness remains an open challenge for current
language models.
Read the full abstract ▾
Computation and Language
REAP: Enhancing RAG with Recursive Evaluation and Adaptive Planning for Multi-Hop Question Answering
Yijie Zhu, Haojie Zhou, Wanting Hong et al. (+2 more)
Retrieval-augmented generation (RAG) has been extensively employed to mitigate
hallucinations in large
language models (LLMs). However, existing methods for multi-hop
reasoning tasks often lack global planning, increasing the risk of falling into local
reasoning impasses. Insufficient exploitation of retrieved content and the neglect of latent clues fail to ensure the accuracy of
reasoning outcomes. To overcome these limitations, we propose Recursive Evaluation and Adaptive Planning (REAP), whose core idea is to explicitly maintain structured sub-tasks and facts related to the current task through the Sub-task Planner (SP) and Fact Extractor (FE) modules. SP maintains a global perspective, guiding the overall
reasoning direction and evaluating the task state based on the outcomes of FE, enabling dynamic optimization of the task-solving trajectory. FE performs fine-grained analysis over retrieved content to extract reliable answers and clues. These two modules incrementally enrich a logically coherent representation of global knowledge, enhancing the reliability and the traceability of the
reasoning process. Furthermore, we propose a unified task paradigm design that enables effective multi-task fine-tuning, significantly enhancing SP's performance on complex, data-scarce tasks. We conduct extensive experiments on multiple public multi-hop
datasets, and the results demonstrate that our method significantly outperforms existing RAG methods in both in-domain and out-of-domain settings, validating its effectiveness in complex multi-hop
reasoning tasks.
Read the full abstract ▾
All authors Yijie Zhu, Haojie Zhou, Wanting Hong, Tailin Liu, Ning Wang
Computation and Language
Leveraging Large Language Models for Identifying Knowledge Components
Canwen Wang, Jionghao Lin and Kenneth R. Koedinger
Knowledge Components (KCs) are foundational to adaptive learning systems, but their manual identification by domain experts is a significant bottleneck. While Large
Language Models (LLMs) offer a promising avenue for automating this process, prior research has been limited to small
datasets and has been shown to produce superfluous, redundant KC labels. This study addresses these limitations by first scaling a "simulated textbook" LLM prompting strategy (using GPT-4o-mini) to a larger
dataset of 646 multiple-choice questions. We found that this initial automated approach performed significantly worse than an expert-designed KC model (RMSE 0.4285 vs. 0.4206) and generated an excessive number of KCs (569 vs. 101). To address the issue of redundancy, we proposed and evaluated a novel method for merging semantically similar KC labels based on their cosine similarity. This merging strategy significantly improved the model's performance; a model using a cosine similarity threshold of 0.8 achieved the best result, reducing the KC count to 428 and improving the RMSE to 0.4259. This demonstrates that while scaled LLM generation alone is insufficient, combining it with a semantic merging technique offers a viable path toward automating and refining KC identification.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Compensating Distribution Drifts in Class-incremental Learning of Pre-trained Vision Transformers
Xuan Rao, Simian Xu, Zheng Li et al. (+4 more)
Recent advances have shown that sequential fine-tuning (SeqFT) of pre-trained
vision transformers (ViTs), followed by classifier refinement using approximate distributions of class features, can be an effective strategy for class-incremental learning (CIL). However, this approach is susceptible to distribution drift, caused by the sequential optimization of shared backbone parameters. This results in a mismatch between the distributions of the previously learned classes and that of the updater model, ultimately degrading the effectiveness of classifier performance over time. To address this issue, we introduce a latent space transition operator and propose Sequential Learning with Drift Compensation (SLDC). SLDC aims to align feature distributions across tasks to mitigate the impact of drift. First, we present a linear variant of SLDC, which learns a linear operator by solving a regularized least-squares problem that maps features before and after fine-tuning. Next, we extend this with a weakly nonlinear SLDC variant, which assumes that the ideal transition operator lies between purely linear and fully nonlinear transformations. This is implemented using learnable, weakly nonlinear mappings that balance flexibility and generalization. To further reduce representation drift, we apply knowledge
distillation (KD) in both algorithmic variants. Extensive experiments on standard CIL
benchmarks demonstrate that SLDC significantly improves the performance of SeqFT. Notably, by combining KD to address representation drift with SLDC to compensate distribution drift, SeqFT achieves performance comparable to joint training across all evaluated
datasets. Code: https://github.com/raoxuan98-hash/sldc.git.
Read the full abstract ▾
All authors Xuan Rao, Simian Xu, Zheng Li, Bo Zhao, Derong Liu, Mingming Ha, Cesare Alippi
Computation and Language
MINDS: A Cross-cultural Dialogue Corpus for Social Norm Classification and Adherence Detection
Pritish Sahu, Anirudh Som, Dimitra Vergyri et al. (+1 more)
Social norms are implicit, culturally grounded expectations that guide interpersonal communication. Unlike factual commonsense, norm
reasoning is subjective, context-dependent, and varies across cultures, posing challenges for computational models. Prior works provide valuable normative annotations but mostly target isolated utterances or synthetic dialogues, limiting their ability to capture the fluid, multi-turn nature of real-world conversations. In this work, we present Norm-RAG, a
retrieval-augmented, agentic framework for nuanced social norm inference in multi-turn dialogues. Norm-RAG models utterance-level attributes including communicative intent, speaker roles, interpersonal framing, and linguistic cues and grounds them in structured normative documentation retrieved via a novel Semantic Chunking approach. This enables interpretable and context-aware
reasoning about norm adherence and violation across
multilingual dialogues. We further introduce MINDS (
Multilingual Interactions with Norm-Driven
Speech), a bilingual
dataset comprising 31 multi-turn Mandarin-English and Spanish-English conversations. Each turn is annotated for norm category and adherence status using multi-annotator consensus, reflecting cross-cultural and realistic norm expression. Our experiments show that Norm-RAG improves norm detection and generalization, demonstrates improved performance for culturally adaptive and socially intelligent dialogue systems.
Read the full abstract ▾
All authors Pritish Sahu, Anirudh Som, Dimitra Vergyri, Ajay Divakaran
Computation and Language
HI-TransPA: Hearing Impairments Translation Personal Assistant
Zhiming Ma, Shiyu Gan, Junhao Zhao et al. (+10 more)
Hearing-impaired individuals often face significant barriers in daily communication due to the inherent challenges of producing clear
speech. To address this, we introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant. The model fuses indistinct
speech with lip dynamics, enabling both translation and dialogue within a single
multimodal framework. To address the distinctive pronunciation patterns of hearing-impaired
speech and the limited adaptability of existing models, we develop a
multimodal preprocessing and curation pipeline that detects facial landmarks, stabilizes the lip region, and quantitatively evaluates sample quality. These quality scores guide a curriculum learning strategy that first trains on clean, high-confidence samples and progressively incorporates harder cases to strengthen model robustness. Architecturally, we employs a novel unified 3D-Resampler to efficiently encode the lip dynamics, which is critical for accurate interpretation. Experiments on purpose-built HI-Dialogue
dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity. Our work establishes a foundation for applying Omni-Models to assistive communication technology, providing an end-to-end modeling framework and essential processing tools for future research.
Read the full abstract ▾
All authors Zhiming Ma, Shiyu Gan, Junhao Zhao, Xianming Li, Qingyun Pan, Peidong Wang, Mingjun Pan, Yuhao Mo, Jiajie Cheng, Chengxin Chen, Zhonglun Cao, Chonghan Liu, Shi Cheng
Computation and Language · Cross-listed from cs.CV
Regional Attention-Enhanced Swin Transformer for Clinically Relevant Medical Image Captioning
Zubia Naz, Farhan Asghar, Muhammad Ishfaq Hussain et al. (+4 more)
Automated medical image captioning translates complex radiological images into diagnostic narratives that can support reporting workflows. We present a Swin-BART encoder-decoder system with a lightweight regional
attention module that amplifies diagnostically salient regions before cross-
attention. Trained and evaluated on ROCO, our model achieves state-of-the-art semantic fidelity while remaining compact and interpretable. We report results as mean$\pm$std over three seeds and include $95\%$ confidence intervals. Compared with baselines, our approach improves ROUGE (proposed 0.603, ResNet-CNN 0.356, BLIP2-OPT 0.255) and BERTScore (proposed 0.807, BLIP2-OPT 0.645, ResNet-CNN 0.623), with competitive BLEU, CIDEr, and METEOR. We further provide ablations (regional
attention on/off and
token-count sweep), per-modality analysis (CT/MRI/X-ray), paired significance tests, and qualitative heatmaps that visualize the regions driving each description. Decoding uses beam search (beam size $=4$), length penalty $=1.1$, $no\_repeat\_ngram\_size$ $=3$, and max length $=128$. The proposed design yields accurate, clinically phrased captions and transparent regional attributions, supporting safe research use with a human in the loop.
Read the full abstract ▾
All authors Zubia Naz, Farhan Asghar, Muhammad Ishfaq Hussain, Yahya Hadadi, Muhammad Aasim Rafique, Wookjin Choi, Moongu Jeon
Computation and Language
EnchTable: Unified Safety Alignment Transfer in Fine-tuned Large Language Models
Jialin Wu, Kecen Li, Zhicong Huang et al. (+3 more)
Many machine learning models are fine-tuned from large
language models (LLMs) to achieve high performance in specialized domains like code generation, biomedical analysis, and mathematical problem solving. However, this fine-tuning process often introduces a critical vulnerability: the systematic degradation of safety
alignment, undermining ethical guidelines and increasing the risk of harmful outputs. Addressing this challenge, we introduce EnchTable, a novel framework designed to transfer and maintain safety
alignment in downstream LLMs without requiring extensive retraining. EnchTable leverages a Neural Tangent Kernel (NTK)-based safety vector
distillation method to decouple safety constraints from task-specific
reasoning, ensuring compatibility across diverse model architectures and sizes. Additionally, our interference-aware merging technique effectively balances safety and utility, minimizing performance compromises across various task domains. We implemented a fully functional prototype of EnchTable on three different task domains and three distinct LLM architectures, and evaluated its performance through extensive experiments on eleven diverse
datasets, assessing both utility and model safety. Our evaluations include LLMs from different vendors, demonstrating EnchTable's generalization capability. Furthermore, EnchTable exhibits robust resistance to static and dynamic jailbreaking attacks, outperforming vendor-released safety models in mitigating
adversarial prompts. Comparative analyses with six parameter modification methods and two inference-time
alignment baselines reveal that EnchTable achieves a significantly lower unsafe rate, higher utility score, and universal applicability across different task domains. Additionally, we validate EnchTable can be seamlessly integrated into various deployment pipelines without significant overhead.
Read the full abstract ▾
All authors Jialin Wu, Kecen Li, Zhicong Huang, Xinfeng Li, Xiaofeng Wang, Cheng Hong