Computation and Language
HybridRAG: A Practical LLM-based ChatBot Framework based on Pre-Generated Q&A over Raw Unstructured Documents
Sungmoon Kim, Hyuna Jeon, Dahye Kim et al. (+3 more)
Retrieval-Augmented Generation (RAG) has emerged as a powerful approach for grounding Large
Language Model (LLM)-based chatbot responses on external knowledge. However, existing RAG studies typically assume well-structured textual sources (e.g. Wikipedia or curated
datasets) and perform
retrieval and generation at query time, which can limit their applicability in real-world chatbot scenarios. In this paper, we present HybridRAG, a novel and practical RAG framework towards more accurate and faster chatbot responses. First, HybridRAG ingests raw, unstructured PDF documents containing complex layouts (text, tables, figures) via Optical Character Recognition (OCR) and layout analysis, and convert them into hierarchical text chunks. Then, it pre-generates a plausible question-answer (QA) knowledge base from the organized chunks using an LLM. At query time, user questions are matched against this QA bank to retrieve immediate answers when possible, and only if no suitable QA match is found does our framework fall back to an on-the-fly response generation. Experiments on OHRBench demonstrate that our HybridRAG provides higher answer quality and lower latency compared to a standard RAG baseline. We believe that HybridRAG could be a practical solution for real-world chatbot applications that must handle large volumes of unstructured documents and lots of users under limited computational resources.
Read the full abstract ▾
All authors Sungmoon Kim, Hyuna Jeon, Dahye Kim, Mingyu Kim, Dong-Kyu Chae, Jiwoong Kim
Computation and Language · Cross-listed from cs.LG
Zero-Training Temporal Drift Detection for Transformer Sentiment Models: A Comprehensive Analysis on Authentic Social Media Streams
Aayam Bansal and Ishaan Gangwani
We present a comprehensive zero-training temporal drift analysis of
transformer-based sentiment models validated on authentic social media data from major real-world events. Through systematic evaluation across three
transformer architectures and rigorous statistical validation on 12,279 authentic social media posts, we demonstrate significant model instability with accuracy drops reaching 23.4% during event-driven periods. Our analysis reveals maximum confidence drops of 13.0% (Bootstrap 95% CI: [9.1%, 16.5%]) with strong correlation to actual performance degradation. We introduce four novel drift metrics that outperform
embedding-based baselines while maintaining computational efficiency suitable for production deployment. Statistical validation across multiple events confirms robust detection capabilities with practical significance exceeding industry monitoring thresholds. This zero-training methodology enables immediate deployment for real-time sentiment monitoring systems and provides new insights into
transformer model behavior during dynamic content periods.
Read the full abstract ▾
Computation and Language
Benchmarking Automatic Speech Recognition Models for African Languages
Alvin Nahabwe, Sulaiman Kagumire, Denis Musinguzi et al. (+5 more)
Automatic
speech recognition (ASR) for African
languages remains constrained by limited labeled data and the lack of systematic guidance on model selection, data scaling, and decoding strategies. Large pre-trained systems such as Whisper, XLS-R, MMS, and W2v-BERT have expanded access to ASR technology, but their comparative behavior in African low-resource contexts has not been studied in a unified and systematic way. In this work, we
benchmark four state-of-the-art ASR models across 13 African
languages, fine-tuning them on progressively larger subsets of transcribed data ranging from 1 to 400 hours. Beyond reporting error rates, we provide new insights into why models behave differently under varying conditions. We show that MMS and W2v-BERT are more data efficient in very low-resource regimes, XLS-R scales more effectively as additional data becomes available, and Whisper demonstrates advantages in mid-resource conditions. We also analyze where external
language model decoding yields improvements and identify cases where it plateaus or introduces additional errors, depending on the
alignment between acoustic and text resources. By highlighting the interaction between pre-training coverage, model architecture,
dataset domain, and resource availability, this study offers practical and insights into the design of ASR systems for underrepresented
languages.
Read the full abstract ▾
All authors Alvin Nahabwe, Sulaiman Kagumire, Denis Musinguzi, Bruno Beijuka, Jonah Mubuuke Kyagaba, Peter Nabende, Andrew Katumba, Joyce Nakatumba-Nabende
Computation and Language
ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages
Subham Kumar, Prakrithi Shivaprakash, Abhishek Manoharan et al. (+6 more)
Automatic
Speech Recognition (ASR) is increasingly used to document clinical encounters, yet its reliability in
multilingual and demographically diverse Indian healthcare contexts remains largely unknown. In this study, we conduct the first systematic audit of ASR performance on real world clinical interview data spanning Kannada, Hindi, and Indian English, comparing leading models including Indic Whisper, Whisper, Sarvam, Google
speech to text, Gemma3n, Omnilingual, Vaani, and Gemini. We evaluate transcription accuracy across
languages, speakers, and demographic subgroups, with a particular focus on error patterns affecting patients vs. clinicians and gender based or intersectional disparities. Our results reveal substantial variability across models and
languages, with some systems performing competitively on Indian English but failing on code mixed or vernacular
speech. We also uncover systematic performance gaps tied to speaker role and gender, raising concerns about equitable deployment in clinical settings. By providing a comprehensive
multilingual benchmark and
fairness analysis, our work highlights the need for culturally and demographically inclusive ASR development for healthcare ecosystem in India.
Read the full abstract ▾
All authors Subham Kumar, Prakrithi Shivaprakash, Abhishek Manoharan, Astut Kurariya, Diptadhi Mukherjee, Lekhansh Shukla, Animesh Mukherjee, Prabhat Chand, Pratima Murthy
Computation and Language
Dialect Identification Using Resource-Efficient Fine-Tuning Approaches
Zirui Lin, Haris Gulzar, Monnika Roslianna Busto et al. (+3 more)
Dialect Identification (DI) is a task to recognize different dialects within the same
language from a
speech signal. DI can help to improve the downstream
speech related tasks even when speakers have a strong dialect. However, fine-tuning a
speech model for tasks like DI is expensive in terms of computation cost and
memory requirement. Recent studies have explored fine-tuning pre-trained
speech models for tasks like DI using Parameter-Efficient Fine-Tuning (PEFT) methods, which offer parameter efficiency but limited improvement in
memory efficiency and training speed. To address these challenges, we explore
Memory-Efficient Fine-Tuning (MEFT) methods, originally proposed for
language processing, and apply them to the general-purpose pre-trained
speech model. We then comprehensively analyze the GPU
memory usage and fine-tuning speed based on various MEFT methods. As a case study, we fine-tune the Whisper model to identify six Mandarin subdialects from the KeSpeech
dataset, reducing GPU
memory usage by up to 73.25% and accelerating training speed by a factor of 2.1, while maintaining accuracy comparable to vanilla fine-tuning and PEFT methods.
Read the full abstract ▾
All authors Zirui Lin, Haris Gulzar, Monnika Roslianna Busto, Akiko Masaki, Takeharu Eda, Kazuhiro Nakadai
Computation and Language · Cross-listed from cs.LG
Mode-Conditioning Unlocks Superior Test-Time Scaling
Chen Henry Wu, Sachin Goyal and Aditi Raghunathan
Parallel sampling promises substantial gains in test-time scaling, but its effectiveness is sharply limited by diversity collapse, where models concentrate on a few modes and repeated samples produce the same mistakes. We propose the mode-conditioning (ModC) framework, which explicitly allocates test-time compute across
reasoning modes using either specialist models or mode-specific prefixes. ModC consistently improves scaling across controlled graph-search tasks and large-scale
reasoning benchmarks, spanning model families and sizes from 0.5B to 7B. On OpenThoughts, fine-tuning Qwen2.5-7B with ModC achieves a 4x efficiency gain over standard training while also improving the maximum attainable Pass@k. We further show that gradient clustering enables ModC without explicit mode labels, yielding up to 10% gains on
datasets such as NuminaMath. Finally, we show that ModC improves reinforcement learning (RL) and can further boost diversity-inducing RL methods. These results demonstrate that standard training underutilizes the diversity in data, and that ModC provides a simple, effective remedy for unlocking the full benefits of diversity in test-time scaling.
Read the full abstract ▾
Computation and Language
How do we measure privacy in text? A survey of text anonymization metrics
Yaxuan Ren, Krithika Ramesh, Yaxing Yao et al. (+1 more)
In this work, we aim to clarify and reconcile metrics for evaluating
privacy protection in text through a systematic survey. Although text anonymization is essential for enabling NLP research and model development in domains with sensitive data, evaluating whether anonymization methods sufficiently protect
privacy remains an open challenge. In manually reviewing 47 papers that report
privacy metrics, we identify and compare six distinct
privacy notions, and analyze how the associated metrics capture different aspects of
privacy risk. We then assess how well these notions align with legal
privacy standards (HIPAA and GDPR), as well as user-centered expectations grounded in HCI studies. Our analysis offers practical guidance on navigating the landscape of
privacy evaluation approaches further and highlights gaps in current practices. Ultimately, we aim to facilitate more robust, comparable, and legally aware
privacy evaluations in text anonymization.
Read the full abstract ▾
All authors Yaxuan Ren, Krithika Ramesh, Yaxing Yao, Anjalie Field
Computation and Language · Cross-listed from cs.CV
Generalised Medical Phrase Grounding
Wenjun Zhang, Shekhar S. Chandra and Aaron Nicolson
Medical phrase grounding (MPG) maps textual descriptions of radiological findings to corresponding image regions. These grounded reports are easier to interpret, especially for non-experts. Existing MPG systems mostly follow the referring expression comprehension (REC) paradigm and return exactly one bounding box per phrase. Real reports often violate this assumption. They contain multi-region findings, non-diagnostic text, and non-groundable phrases, such as negations or descriptions of normal anatomy. Motivated by this, we reformulate the task as generalised medical phrase grounding (GMPG), where each sentence is mapped to zero, one, or multiple scored regions. To realise this formulation, we introduce the first GMPG model: MedGrounder. We adopted a two-stage training regime: pre-training on report sentence–anatomy box
alignment datasets and fine-tuning on report sentence–human annotated box
datasets. Experiments on PadChest-GR and MS-CXR show that MedGrounder achieves strong zero-shot transfer and outperforms REC-style and grounded report generation baselines on multi-region and non-groundable phrases, while using far fewer human box annotations. Finally, we show that MedGrounder can be composed with existing report generators to produce grounded reports without retraining the generator.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Testing the Machine Consciousness Hypothesis
Stephen Fitz
The Machine Consciousness Hypothesis states that consciousness is a substrate-free functional property of computational systems capable of second-order perception. I propose a research program to investigate this idea in silico by studying how collective self-models (coherent, self-referential representations) emerge from distributed learning systems embedded within universal self-organizing environments. The theory outlined here starts from the supposition that consciousness is an emergent property of collective intelligence systems undergoing synchronization of prediction through communication. It is not an epiphenomenon of individual modeling but a property of the
language that a system evolves to internally describe itself. For a model of base reality, I begin with a minimal but general computational world: a cellular automaton, which exhibits both computational irreducibility and local reducibility. On top of this computational substrate, I introduce a network of local, predictive, representational (neural) models capable of communication and adaptation. I use this layered model to study how collective intelligence gives rise to self-representation as a direct consequence of inter-
agent alignment. I suggest that consciousness does not emerge from modeling per se, but from communication. It arises from the noisy, lossy exchange of predictive messages between groups of local observers describing persistent patterns in the underlying computational substrate (base reality). It is through this representational dialogue that a shared model arises, aligning many partial views of the world. The broader goal is to develop empirically testable theories of machine consciousness, by studying how internal self-models may form in distributed systems without centralized control.
Read the full abstract ▾
Computation and Language
ELR-1000: A Community-Generated Dataset for Endangered Indic Indigenous Languages
Neha Joshi, Pamir Gogoi, Aasim Mirza et al. (+7 more)
We present a culturally-grounded
multimodal dataset of 1,060 traditional recipes crowdsourced from rural communities across remote regions of Eastern India, spanning 10 endangered
languages. These recipes, rich in linguistic and cultural nuance, were collected using a mobile interface designed for contributors with low digital literacy. Endangered
Language Recipes (ELR)-1000 – captures not only culinary practices but also the socio-cultural context embedded in indigenous food traditions. We evaluate the performance of several state-of-the-art large
language models (LLMs) on translating these recipes into English and find the following: despite the models' capabilities, they struggle with low-resource, culturally-specific
language. However, we observe that providing targeted context – including background information about the
languages, translation examples, and guidelines for cultural preservation – leads to significant improvements in translation quality. Our results underscore the need for
benchmarks that cater to underrepresented
languages and domains to advance equitable and culturally-aware
language technologies. As part of this work, we release the ELR-1000
dataset to the NLP community, hoping it motivates the development of
language technologies for endangered
languages.
Read the full abstract ▾
All authors Neha Joshi, Pamir Gogoi, Aasim Mirza, Aayush Jansari, Aditya Yadavalli, Ayushi Pandey, Arunima Shukla, Deepthi Sudharsan, Kalika Bali, Vivek Seshadri
Computation and Language
How Semantically Stable Are LLM Refusals? Measuring Confusion in Local Safety Boundaries
Riad Ahmed Anonto, Md Labid Al Nahiyan and Md Tanvir Hassan
As safety
alignment becomes standard in large
language models, refusal behavior has become an important part of model reliability. However, models may still reject benign
prompts, especially when the wording resembles risky content. Existing evaluations usually report global scores, such as false rejection rate or compliance rate. These scores are useful, but they treat each
prompt independently. As a result, they miss local inconsistency, where a model accepts one phrasing of an intent but rejects a close paraphrase. This makes it difficult to understand whether refusals are only frequent or also semantically unstable. We address this gap by introducing Semantic Confusion, a failure mode that captures contradictory refusal decisions across meaning-preserving paraphrases. We build ParaGuard, a 10k-
prompt corpus of controlled paraphrase clusters that keep intent fixed while varying surface form. We also propose three model-agnostic
token-level metrics: Confusion Index, Confusion Rate, and Confusion Depth. These metrics compare each rejected
prompt with its nearest accepted neighbors using
token embeddings, next-
token probabilities, and perplexity signals. Experiments across diverse model families show that global false rejection rate can hide important structure in the refusal boundary. Our metrics reveal cases where confusion is spread broadly, cases where it appears only in specific semantic regions, and cases where stricter refusal does not lead to more local inconsistency. These findings show that refusal evaluation should measure not only how often a model refuses, but also how consistently it refuses across nearby paraphrases. This gives developers a practical signal for reducing false refusals while preserving safety.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Associative Syntax and Maximal Repetitions reveal context-dependent complexity in fruit bat communication
Luigi Assom
This study presents an unsupervised method to infer discreteness, syntax and temporal structures of fruit-bats vocalizations, as a case study of graded vocal systems, and evaluates the complexity of communication patterns in relation with behavioral context. The method improved the baseline for unsupervised labeling of vocal units (i.e. syllables) through
manifold learning, by investigating how dimensionality reduction on mel-spectrograms affects labeling, and comparing it with unsupervised labels based on acoustic similarity. We then encoded vocalizations as syllabic sequences to analyze the type of syntax, and extracted the Maximal Repetitions (MRs) to evaluate syntactical structures. We found evidence for: i) associative syntax, rather than combinatorial (context classification is unaffected by permutation of sequences, F 1 > 0.9); ii) context-dependent use of syllables (Wilcoxon rank-sum tests, p-value < 0.05); iii) heavy-tail distribution of MRs (truncated power-law, exponent {\alpha} < 2), indicative of mechanism encoding combinatorial complexity. Analysis of MRs and syllabic transition networks revealed that mother-pupil interactions were characterized by repetitions, while communication in conflict-contexts exhibited higher complexity (longer MRs and more interconnected vocal sequences) than non-agonistic contexts. We propose that communicative complexity is higher in scenarios of disagreement, reflecting lower compressibility of information.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics
Jinu Lee, Kyoung-Woon On, Simeng Han et al. (+2 more)
Evaluating the quality of LLM-generated
reasoning traces in expert domains (e.g., law) is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such
reasoning tasks. We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal
reasoning dataset with an emphasis on
reasoning trace evaluation. We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the
reasoning traces. We verify the reliability of these rubrics via human expert annotations and comparison with coarse, less informative rubrics. Using the LEGIT
dataset, we show that (1) LLMs' legal
reasoning ability is seriously affected by both legal issue coverage and correctness, and that (2)
retrieval-augmented generation (RAG) and RL with rubrics bring complementary benefits for legal
reasoning abilities, where RAG improves overall
reasoning capability, whereas RL improves correctness albeit with reduced coverage.
Read the full abstract ▾
All authors Jinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier
Computation and Language
Advancing Academic Chatbots: Evaluation of Non Traditional Outputs
Nicole Favero, Francesca Salute and Daniel Hardt
Most evaluations of large
language models focus on standard tasks such as factual question answering or short summarization. This research expands that scope in two directions: first, by comparing two
retrieval strategies, Graph RAG, structured knowledge-graph based, and Advanced RAG, hybrid keyword-semantic search, for QA; and second, by evaluating whether LLMs can generate high quality non-traditional academic outputs, specifically slide decks and podcast scripts. We implemented a prototype combining Meta's LLaMA 3 70B open weight and OpenAI's GPT 4o mini API based. QA performance was evaluated using both human ratings across eleven quality dimensions and large
language model judges for scalable cross validation. GPT 4o mini with Advanced RAG produced the most accurate responses. Graph RAG offered limited improvements and led to more
hallucinations, partly due to its structural complexity and manual setup. Slide and podcast generation was tested with document grounded
retrieval. GPT 4o mini again performed best, though LLaMA 3 showed promise in narrative coherence. Human reviewers were crucial for detecting layout and stylistic flaws, highlighting the need for combined human LLM evaluation in assessing emerging academic outputs.
Read the full abstract ▾
Computation and Language
ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents
Zhihan Guo, Feiyang Xu, Yifan Li et al. (+7 more)
A surge in academic publications calls for automated deep research (DR) systems, but accurately evaluating them is still an open problem. First, existing
benchmarks often focus narrowly on
retrieval while neglecting high-level planning and
reasoning. Second, existing
benchmarks favor general domains over the academic domains that are the core application for DR
agents. To address these gaps, we introduce ADRA-Bank, a modular
benchmark for Academic DR
Agents. Grounded in academic literature, our
benchmark is a human-annotated
dataset of 200 instances across 10 academic domains, including both research and review papers. Furthermore, we propose a modular Evaluation Paradigm for Academic DR
Agents (ADRA-Eval), which leverages the rich structure of academic papers to assess the core capabilities of planning,
retrieval, and
reasoning. It employs two complementary modes: an end-to-end evaluation for \task
agents and an isolated evaluation for foundational LLMs as potential backbones. Results reveal uneven capabilities: while
agents show specialized strengths, they struggle with multi-source
retrieval and cross-field consistency. Moreover, improving high-level planning capability is the crucial factor for unlocking the
reasoning potential of foundational LLMs as backbones. By exposing these actionable failure modes, ADRA-Bank provides a diagnostic tool to guide the development of more reliable automatic academic research assistants.
Read the full abstract ▾
All authors Zhihan Guo, Feiyang Xu, Yifan Li, Muzhi Li, Shuai Zou, Jiele Wu, Han Shi, Haoli Bai, Ho-fung Leung, Irwin King
Computation and Language · Cross-listed from cs.LG
WUSH: Near-Optimal Adaptive Transforms for LLM Quantization
Jiale Chen, Vage Egiazarian, Roberto L. Castro et al. (+2 more)
Quantizing LLM weights and activations is a standard approach for efficient deployment, but a few extreme outliers can stretch the dynamic range and amplify low-bit quantization errors. Prior transform-based mitigations (e.g., Hadamard rotations) are fixed and data-agnostic, and their optimality for quantization has remained unclear. We derive closed-form optimal linear blockwise transforms for joint weight-activation quantization under standard RTN AbsMax-scaled block quantizers, covering both integer and floating-point formats. The resulting construction, WUSH, combines a Hadamard backbone with a data-dependent second-moment component to form a non-orthogonal transform that is provably near-optimal for FP and INT quantizers under mild assumptions while admitting an efficient fused GPU implementation. Empirically, WUSH improves W4A4 accuracy over the strongest Hadamard-based baselines (e.g., on Llama-3.1-8B-Instruct in MXFP4, it gains +2.8 average points with RTN and +0.7 with GPTQ) while delivering up to 5.8$\times$ per-layer throughput over BF16 via FP4 MatMul. Source code is available at https://github.com/IST-DASLab/WUSH.
Read the full abstract ▾
All authors Jiale Chen, Vage Egiazarian, Roberto L. Castro, Torsten Hoefler, Dan Alistarh
Computation and Language
Table as a Modality for Large Language Models
Liyao Li, Chao Ye, Wentao Ye et al. (+9 more)
To migrate the remarkable successes of Large
Language Models (LLMs), the community has made numerous efforts to generalize them to the table
reasoning tasks for the widely deployed tabular data. Despite that, in this work, by showing a probing experiment on our proposed StructQA
benchmark, we postulate that even the most advanced LLMs (such as GPTs) may still fall short of coping with tabular data. More specifically, the current scheme often simply relies on serializing the tabular data, together with the meta information, then inputting them through the LLMs. We argue that the loss of structural information is the root of this shortcoming. In this work, we further propose TAMO, which bears an ideology to treat the tables as an independent modality integrated with the text
tokens. The resulting model in TAMO is a
multimodal framework consisting of a hypergraph neural network as the global table encoder seamlessly integrated with the mainstream LLM. Empirical results on various benchmarking
datasets, including HiTab, WikiTQ, WikiSQL, FeTaQA, and StructQA, have demonstrated significant improvements on generalization with an average relative gain of 42.65%.
Read the full abstract ▾
All authors Liyao Li, Chao Ye, Wentao Ye, Yifei Sun, Zhe Jiang, Haobo Wang, Jiaming Tian, Yiming Zhang, Ningtao Wang, Xing Fu, Gang Chen, Junbo Zhao
Computation and Language
Fine-tuning of lightweight large language models for sentiment classification on heterogeneous financial textual data
Alvaro Paredes Amorin, Andre Python and Christoph Weisser
Large
language models (LLMs) play an increasingly important role in financial
markets analysis by capturing signals from complex and heterogeneous textual data sources, such as tweets, news articles, reports, and microblogs. However, their performance is dependent on large computational resources and proprietary
datasets, which are costly, restricted, and therefore inaccessible to many researchers and practitioners. To reflect realistic situations we investigate the ability of lightweight open-source LLMs – smaller and publicly available models designed to operate with limited computational resources – to generalize sentiment understanding from financial
datasets of varying sizes, sources, formats, and
languages. We compare the
benchmark finance natural
language processing (NLP) model, FinBERT, and three open-source lightweight LLMs, DeepSeek-LLM 7B, Llama3 8B Instruct, and Qwen3 8B on five publicly available
datasets: FinancialPhraseBank, Financial Question Answering, Gold News Sentiment, Twitter Sentiment and Chinese Finance Sentiment. We find that LLMs, specially Qwen3 8B and Llama3 8B, perform best in most scenarios, even from using only 5% of the available training data. These results hold in zero-shot and few-shot learning scenarios. Our findings indicate that lightweight, open-source large
language models (LLMs) constitute a cost-effective option, as they can achieve competitive performance on heterogeneous textual data even when trained on only a limited subset of the extensive annotated corpora that are typically deemed necessary.
Read the full abstract ▾
Computation and Language
DeformAr: Rethinking NER Evaluation through Component Analysis and Visual Analytics
Ahmed Mustafa Younes
Transformer models have significantly advanced Natural
Language Processing (NLP), demonstrating strong performance in English. However, their effectiveness in Arabic, particularly for Named Entity Recognition (NER), remains limited, even with larger pre-trained models. This performance gap stems from multiple factors, including tokenisation,
dataset quality, and annotation inconsistencies. Existing studies often analyze these issues in isolation, failing to capture their joint effect on system behaviour and performance. We introduce DeformAr (Debugging and Evaluation Framework for
Transformer-based NER Systems), a novel framework designed to investigate and explain the performance discrepancy between Arabic and English NER systems. DeformAr integrates a data extraction library and an interactive dashboard, supporting two modes of evaluation: cross-component analysis and behavioural analysis. The framework divides each
language into
dataset and model components to examine their interactions. The analysis proceeds in two stages. First, cross-component analysis provides systematic diagnostic measures across data and model subcomponents, addressing the "what," "how," and "why" behind observed discrepancies. The second stage applies behavioural analysis by combining interpretability techniques with
token-level metrics, interactive visualisations, and representation space analysis. These stages enable a component-aware diagnostic process that detects model behaviours and explains them by linking them to underlying representational patterns and data factors. DeformAr is the first Arabic-specific, component-based interpretability tool, offering a crucial resource for advancing model analysis in under-resourced
languages.
Read the full abstract ▾
Computation and Language
Mitigating Hallucinations in Zero-Shot Scientific Summarisation: A Pilot Study
Imane Jaaouine and Ross D. King
Large
language models (LLMs) produce context inconsistency
hallucinations, which are LLM generated outputs that are misaligned with the user
prompt. This research project investigates whether
prompt engineering (PE) methods can mitigate context inconsistency
hallucinations in zero-shot LLM summarisation of scientific texts, where zero-shot indicates that the LLM relies purely on its pre-training data. Across eight yeast biotechnology research paper abstracts, six instruction-tuned LLMs were prompted with seven methods: a baseline
prompt, two levels of increasing instruction complexity (PE-1 and PE-2), two levels of context repetition (CR-K1 and CR-K2), and two levels of random addition (RA-K1 and RA-K2). Context repetition involved the identification and repetition of K key sentences from the abstract, whereas random addition involved the repetition of K randomly selected sentences from the abstract, where K is 1 or 2. A total of 336 LLM-generated summaries were evaluated using six metrics: ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, METEOR, and cosine similarity, which were used to compute the lexical and semantic
alignment between the summaries and the abstracts. Four hypotheses on the effects of
prompt methods on summary
alignment with the reference text were tested. Statistical analysis on 3744 collected datapoints was performed using bias-corrected and accelerated (BCa) bootstrap confidence intervals and Wilcoxon signed-rank tests with Bonferroni-Holm correction. The results demonstrated that CR and RA significantly improve the lexical
alignment of LLM-generated summaries with the abstracts. These findings indicate that
prompt engineering has the potential to impact
hallucinations in zero-shot scientific summarisation tasks.
Read the full abstract ▾
Computation and Language
Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios
Jianxiang Zang, Yongda Wei, Ruxue Bai et al. (+5 more)
Reliable reward models (RMs) are critical for ensuring the safe
alignment of large
language models (LLMs). However, current RM evaluation methods focus solely on preference perception accuracies in given specific scenarios, obscuring the critical vulnerabilities of RMs in real-world scenarios. We identify the true challenge lies in assessing a novel dimension: Suitability, defined as conditional reliability under specific real-world perturbations. To this end, we introduce Reward Auditor, a hypothesis-testing framework specifically designed for RM suitability inference. Rather than answering "How accurate is the RM's preference perception for given samples?", it employs scientific
auditing to answer: "Can we infer RMs exhibit systematic vulnerabilities in specific real-world scenarios?". Under real-world perturbed scenarios, Reward Auditor quantifies statistical significance and effect size by
auditing distribution degradation of RM preference perception confidence. This enables inference of both the certainty and severity of RM vulnerabilities across diverse real-world scenarios. This lays a solid foundation for building next-generation LLM
alignment systems that are verifiably safe, more robust, and trustworthy.
Read the full abstract ▾
All authors Jianxiang Zang, Yongda Wei, Ruxue Bai, Shiyu Jiang, Nijia Mo, Binhong Li, Qiang Sun, Hui Liu
Computation and Language · Cross-listed from cs.LG
Towards Active Synthetic Data Generation for Finetuning Language Models
Samuel Kessler, Menglin Xia, Daniel Madrigal Diaz et al. (+5 more)
A common and effective means for improving
language model capabilities involves finetuning a “student”
language model's parameters on generations from a more proficient “teacher” model. Termed “synthetic data”, these generations are often produced before any student finetuning, but some work has considered generating new synthetic samples as training progresses. This paper studies and advocates for the latter case, where data are generated in an iterative, closed-loop fashion that is guided by the current state of the student model. For a fixed budget of generated samples, or a budget in terms of compute spent querying a teacher, we show that this curation of finetuning data affords improved student performance over static generation. Further, while there have been several LLM-specific methods proposed that operate in this regime, we find that simple, inexpensive selection criteria from the active learning literature tend to be most performant. We validate these claims across four mathematical and logical
reasoning datasets using four different small
language models.
Read the full abstract ▾
All authors Samuel Kessler, Menglin Xia, Daniel Madrigal Diaz, Dongge Han, Helia Heshemi, Saravan Rajmohan, Victor Ruehle, Jordan T. Ash
Computation and Language
Less is More: Resource-Efficient Low-Rank Adaptation
Chunlin Tian, Xuyang Wei, Huanrong Liu et al. (+2 more)
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method for Large
Language Models (LLMs), but it still incurs notable overhead and suffers from parameter interference in complex
datasets. While recent works decouple LoRA update
matrices to exploit
matrix-wise asymmetry, training costs remain high. We revisit LoRA from the perspective of inter-
matrix and intra-layer parameter redundancy and propose Resource-Efficient Low-Rank Adaptation, EffiLoRA, a lightweight and generalizable approach for
language,
multimodal, and
diffusion models. EffiLoRA employs a unified A
matrix across all
transformer layers and introduces a runtime selective B
matrices update to dynamically trade-off the system resource budget and model performance. EffiLoRA consistently outperforms LoRA across diverse modalities, including commonsense
reasoning, visual instruction tuning, and image generation, demonstrating improved efficiency and robustness.
Read the full abstract ▾
All authors Chunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo, Li Li
Computation and Language · Cross-listed from cs.AI
One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces
Yandong Sun, Qiang Huang, Ziwei Xu et al. (+3 more)
Embedding spaces are fundamental to modern AI, translating raw data into high-dimensional vectors that encode rich semantic relationships. Yet, their internal structures remain opaque, with existing approaches often sacrificing semantic coherence for structural regularity or incurring high computational overhead to improve interpretability. To address these challenges, we introduce the Semantic Field Subspace (SFS), a geometry-preserving, context-aware representation that captures local semantic neighborhoods within the
embedding space. We also propose SAFARI (SemAntic Field subspAce deteRmInation), an unsupervised, modality-agnostic
algorithm that uncovers hierarchical semantic structures using a novel metric called Semantic Shift, which quantifies how semantics evolve as SFSes evolve. To ensure scalability, we develop an efficient approximation of Semantic Shift that replaces costly SVD computations, achieving a 15~30x speedup with average errors below 0.01. Extensive evaluations across six real-world text and image
datasets show that SFSes outperform standard classifiers not only in classification but also in nuanced tasks such as political bias detection, while SAFARI consistently reveals interpretable and generalizable semantic hierarchies. This work presents a unified framework for structuring, analyzing, and scaling semantic understanding in
embedding spaces.
Read the full abstract ▾
All authors Yandong Sun, Qiang Huang, Ziwei Xu, Yiqun Sun, Yixuan Tang, Anthony K. H. Tung
Computation and Language
WaterSearch: Exploring Seed Pooling for Improving the Quality-Detectability Trade-off in LLM Watermarking
Yukang Lin, Jiahao Shao, Shuoran Jiang et al. (+5 more)
Watermarking acts as a critical safeguard in text generated by Large
Language Models (LLMs). By
embedding identifiable signals into model outputs, watermarking enables reliable attribution and enhances the security of machine-generated content. Existing approaches typically embed signals by manipulating
token generation probabilities. Despite their effectiveness, these methods inherently face a trade-off between detectability and text quality: the signal strength and randomness required for robust watermarking tend to degrade the performance of downstream tasks. In this paper, we design a novel
embedding scheme that controls seed pools to facilitate diverse parallel generation of watermarked text. Based on that scheme, we propose WaterSearch, a sentence-level, search-based watermarking framework adaptable to a wide range of existing methods. WaterSearch enhances text quality by jointly optimizing two key aspects: 1) distribution fidelity and 2) watermark signal characteristics. Furthermore, WaterSearch is complemented by a sentence-level detection method with strong attack robustness. We evaluate our method on three popular LLMs across ten diverse tasks. Extensive experiments demonstrate that our method achieves an average performance improvement of 51.01% over state-of-the-art baselines at a watermark detectability strength of 95%. In challenging scenarios such as short text generation and low-
entropy output generation, our method yields performance gains of 47.78% and 36.47%, respectively. Moreover, under different attack senarios including insertion, synonym substitution and paraphrase attasks, WaterSearch maintains high detectability, further validating its robust anti-attack capabilities. Our code is available at \href{https://github.com/Yukang-Lin/WaterSearch}{https://github.com/Yukang-Lin/WaterSearch}.
Read the full abstract ▾
All authors Yukang Lin, Jiahao Shao, Shuoran Jiang, Wentao Zhu, Bingjie Lu, Xiangping Wu, Joanna Siebert, Qingcai Chen
Computation and Language
Accelerating Bangla NLP Tasks with Automatic Mixed Precision: Resource-Efficient Training Preserving Model Efficacy
Md Mehrab Hossain Opi, Sumaiya Khan and Moshammad Farzana Rahman
Training models for Natural
Language Processing (NLP) requires substantial computational resources and time, posing significant challenges, especially for NLP development in Bangla, where access to high-end hardware is often limited. In this work, we explore automatic mixed precision (AMP) training as a means to improve computational efficiency without sacrificing model performance. By leveraging a dynamic mix of 16-bit and 32-bit floating-point computations, AMP lowers GPU
memory requirements and speeds up training without degrading model performance. We evaluate AMP across four standard Bangla NLP tasks, namely sentiment analysis, named entity recognition, error classification, and question answering, using four
transformer-based models: BanglaBERT, BanglishBERT, XLM-R, and mBERT. Our results demonstrate that AMP accelerates training by 44.5% and reduces
memory consumption by 17.6%, while maintaining F-1 score within 99.7% of the full-precision baselines. This empirical study highlights AMP's potential to democratize access to state-of-the-art NLP capabilities in hardware-constrained settings by lowering computational barriers.
Read the full abstract ▾