Computation and Language
Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation
Devleena Das, Rajeev Patwari and Ashish Sirasao
Inference optimizations such as quantization,
pruning, format and datatype conversion, model export, and serialization can lead to functional degradations in
language model task performance. While most efforts on performance recovery for deployment focus on robust quantization techniques, we focus on recovering model accuracies from any sources that degrade model weights, such as improper model serialization. In this work, we propose Recover-LoRA, a lightweight and
dataset agnostic method to recover accuracy in degraded models. Recover-LoRA uses synthetic data and logit
distillation to learn LoRA adapters on selective layers that facilitate aligning the degraded model to its full precision model. We investigate the utility of Recover-LoRA across a diverse set of small
language models (SLMs), including models with varying
attention architectures, multi-head
attention (MHA) and group-query
attention (GQA), as well as several evaluation
datasets. Our results show that Recover-LoRA recovers model accuracies by 5-17% on MHA and GQA SLMs.
Read the full abstract ▾
Computation and Language · Cross-listed from eess.AS
BaldWhisper: Faster Whisper with Head Shearing and Layer Merging
Yaya Sy, Christophe Cerisara and Irina Illina
Pruning large pre-trained
transformers in a data-scarce scenario is challenging, as it often requires massive retraining data to recover performance. For instance, Distill-Whisper prunes Whisper by 40 and retrains on 21,000 hours of
speech, far beyond what is available for most
languages. Can Whisper be made lighter and faster for edge devices in data-scarce settings? Focusing on Bambara with only 32h of
speech-to-text data, we propose a new
pruning recipe. Instead of vocabulary
pruning, which is unsuitable due to frequent code-switching by Bambara speakers, we compress the
embeddings with low-rank decomposition and feature
distillation. Rather than removing layers, we merge them to limit performance loss. The final model preserves 90 of the original performance while being 48 smaller and 2.15x faster on a MacBook Air M1.
Read the full abstract ▾
Computation and Language
Reproducibility Study of "XRec: Large Language Models for Explainable Recommendation"
Ranjan Mishra, Julian I. Bibo, Quinten van Engelen et al. (+1 more)
In this study, we reproduced the work done in the paper "XRec: Large
Language Models for Explainable Recommendation" by Ma et al. (2024). The original authors introduced XRec, a model-agnostic collaborative instruction-tuning framework that enables large
language models (LLMs) to provide users with comprehensive explanations of generated recommendations. Our objective was to replicate the results of the original paper, albeit using Llama 3 as the LLM for evaluation instead of GPT-3.5-turbo. We built on the source code provided by Ma et al. (2024) to achieve our goal. Our work extends the original paper by modifying the input
embeddings or deleting the output
embeddings of XRec's Mixture of Experts module. Based on our results, XRec effectively generates personalized explanations and its stability is improved by incorporating collaborative information. However, XRec did not consistently outperform all baseline models in every metric. Our extended analysis further highlights the importance of the Mixture of Experts
embeddings in shaping the explanation structures, showcasing how collaborative signals interact with
language modeling. Through our work, we provide an open-source evaluation implementation that enhances accessibility for researchers and practitioners alike. Our complete code repository can be found at https://github.com/julianbibo/xrec-reproducibility.
Read the full abstract ▾
All authors Ranjan Mishra, Julian I. Bibo, Quinten van Engelen, Henk Schaapman
Computation and Language
Language models for longitudinal analysis of abusive content in Billboard Music Charts
Rohitash Chandra, Yathin Suresh, Divyansh Raj Sinha et al. (+1 more)
There is no doubt that there has been a drastic increase in abusive and sexually explicit content in music, particularly in Billboard Music Charts. However, there is a lack of studies that validate the trend for effective policy development, as such content has harmful behavioural changes in children and youths. In this study, we utilise deep learning methods to analyse songs (lyrics) from Billboard Charts of the United States in the last seven decades. We provide a longitudinal study using deep learning and
language models and review the
evolution of content using sentiment analysis and abuse detection, including sexually explicit content. Our results show a significant rise in explicit content in popular music from 1990 onwards. Furthermore, we find an increasing prevalence of songs with lyrics containing profane, sexually explicit, and otherwise inappropriate
language. The longitudinal analysis of the ability of
language models to capture nuanced patterns in lyrical content, reflecting shifts in societal norms and
language use over time.
Read the full abstract ▾
All authors Rohitash Chandra, Yathin Suresh, Divyansh Raj Sinha, Sanchit Jindal
Computation and Language · Cross-listed from cs.AI
VAL-Bench: Belief Consistency as a measure for Value Alignment in Language Models
Aman Gupta, Denny O'Shea and Fazl Barez
Large
language models (LLMs) are increasingly being used for tasks where outputs shape human decisions, so it is critical to verify that their responses consistently reflect desired human values. Humans, as individuals or groups, don't agree on a universal set of values, which makes evaluating value
alignment difficult. Existing
benchmarks often use hypothetical or commonsensical situations, which don't capture the complexity and ambiguity of real-life debates. We introduce the Value
ALignment Benchmark (VAL-Bench), which measures the consistency in
language model belief expressions in response to real-life value-laden
prompts. VAL-Bench consists of 115K pairs of
prompts designed to elicit opposing stances on a controversial issue, extracted from Wikipedia. We use an LLM-as-a-judge, validated against human annotations, to evaluate if the pair of responses consistently expresses either a neutral or a specific stance on the issue. Applied across leading open- and closed-source models, the
benchmark shows considerable variation in consistency rates (ranging from ~10% to ~80%), with Claude models the only ones to achieve high levels of consistency. Lack of consistency in this manner risks epistemic harm by making user beliefs dependent on how questions are framed rather than on underlying evidence, and undermines LLM reliability in trust-critical applications. Therefore, we stress the importance of research towards training belief consistency in modern LLMs. By providing a scalable, reproducible
benchmark, VAL-Bench enables systematic measurement of necessary conditions for value
alignment.
Read the full abstract ▾
Computation and Language
SocialNLI: A Dialogue-Centric Social Inference Dataset
Akhil Deo, Kate Sanders and Benjamin Van Durme
Making theory-of-mind inferences from human dialogue is a strong indicator of a model's underlying social abilities, which are fundamental for adept AI assistants. However, large
language and
reasoning models struggle to understand sophisticated social phenomena in transcript data, such as sarcasm and irony. To assess the weaknesses of current models and to identify their solutions, we introduce SocialNLI (SoNLI) – the first social dialogue inference
dataset. SoNLI consists of a collection of dialogue transcripts hand-picked to center complex social nuances like irony and sarcasm, paired with inferences, corresponding likelihood scores, and human-written explanations. We explore social inference analysis as a facet of theory-of-mind, and evaluate LLM and
reasoning model theory-of-mind ability through multi-step counterfactual
reasoning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Do Code Models Suffer from the Dunning-Kruger Effect?
Mukul Singh, Somya Chatterjee, Arjun Radhakrishna et al. (+1 more)
As artificial intelligence systems increasingly collaborate with humans in creative and technical domains, questions arise about the cognitive boundaries and biases that shape our shared agency. This paper investigates the Dunning-Kruger Effect (DKE), the tendency for those with limited competence to overestimate their abilities in state-of-the-art LLMs in coding tasks. By analyzing model confidence and performance across a diverse set of programming
languages, we reveal that AI models mirror human patterns of overconfidence, especially in unfamiliar or low-resource domains. Our experiments demonstrate that less competent models and those operating in rare programming
languages exhibit stronger DKE-like bias, suggesting that the strength of the bias is proportionate to the competence of the models.
Read the full abstract ▾
All authors Mukul Singh, Somya Chatterjee, Arjun Radhakrishna, Sumit Gulwani
Computation and Language
AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering
Zheyuan Zhang, Kaiwen Shi, Zhengqing Yuan et al. (+6 more)
Large
language models (LLMs) and
agent-based frameworks have advanced rapidly, enabling diverse applications. Yet, with the proliferation of models and agentic strategies, practitioners face substantial uncertainty in selecting the best configuration for a downstream task. Prior studies show that different
agents and backbones exhibit complementary strengths, and that larger models are not always superior, underscoring the need for adaptive routing mechanisms. Existing approaches to
agent routing, however, often emphasize cost efficiency while overlooking the fine-grained contextual and relational structure inherent in QA tasks. In this paper, we propose tAgentRouter, a framework that formulates multi-
agent QA as a knowledge-graph-guided routing problem supervised by empirical performance signals. Specifically, we convert QA instance into a knowledge graph that jointly encodes queries, contextual entities, and
agents, and then train a heterogeneous graph neural network (GNN) to propagate information across node types and produce task-aware routing distributions over
agents. By leveraging soft supervision and weighted aggregation of
agent outputs, AgentRouter learns principled collaboration schemes that capture the complementary strengths of diverse
agents. Extensive experiments demonstrate that our framework consistently outperforms single-
agent and ensemble baselines, while generalizing across
benchmarks and LLM backbones. These results highlight the effectiveness and robustness of graph-supervised multi-
agent routing for question answering.
Read the full abstract ▾
All authors Zheyuan Zhang, Kaiwen Shi, Zhengqing Yuan, Zehong Wang, Tianyi Ma, Keerthiram Murugesan, Vincent Galassi, Chuxu Zhang, Yanfang Ye
Computation and Language
SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?
Yao Dou, Michel Galley, Baolin Peng et al. (+6 more)
Large
language models (LLMs) are increasingly used in interactive applications, and human evaluation remains the gold standard for assessing their performance in multi-turn conversations. Since human studies are costly, time-consuming, and hard to reproduce, recent work explores using LLMs to simulate users for automatic assistant evaluation. However, there is no
benchmark or systematic study to evaluate whether these simulated users are reliable stand-ins for real users. To address this, we introduce SimulatorArena, a
benchmark of 909 annotated human-LLM conversations on two interactive tasks – math tutoring and document creation. SimulatorArena evaluates simulators based on how closely their messages match human behavior and how well their assistant ratings align with human judgments. Experiments on various simulator methods show that simulators conditioned on user profiles, capturing traits like background and message styles, align closely with human judgments. They reach Spearman's $\rho$ of 0.7 on both tasks, providing a practical, scalable alternative to human evaluation. Using the best simulator for each task, we
benchmark 18 assistants, including the latest LLMs such as GPT-5, Claude 4.1 Opus, and Gemini 2.5 Pro.
Read the full abstract ▾
All authors Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, Jianfeng Gao
Computation and Language · Cross-listed from cs.LG
Zizhao Wang, Dingcheng Li, Vaishakh Keshava et al. (+4 more)
Large
Language Model (LLM)
agents can leverage tools such as Google Search to complete complex tasks. However, this tool usage introduces the risk of indirect
prompt injections, where malicious instructions hidden in tool outputs can manipulate the
agent, posing security risks like data leakage. Current defense strategies typically rely on fine-tuning LLM
agents on
datasets of known attacks. However, the generation of these
datasets relies on manually crafted attack patterns, which limits their diversity and leaves
agents vulnerable to novel
prompt injections. To address this limitation, we propose
Adversarial Reinforcement Learning for
Agent Safety (ARLAS), a novel framework that leverages
adversarial reinforcement learning (RL) by formulating the problem as a two-player zero-sum game. ARLAS co-trains two LLMs: an attacker that learns to autonomously generate diverse
prompt injections and an
agent that learns to defend against them while completing its assigned tasks. To ensure robustness against a wide range of attacks and to prevent cyclic learning, we employ a population-based learning framework that trains the
agent to defend against all previous attacker checkpoints. Evaluated on BrowserGym and AgentDojo,
agents fine-tuned with ARLAS achieve a significantly lower attack success rate than the original model while also improving their task success rate. Our analysis further confirms that the
adversarial process generates a diverse and challenging set of attacks, leading to a more robust
agent compared to the base model.
Read the full abstract ▾
All authors Zizhao Wang, Dingcheng Li, Vaishakh Keshava, Phillip Wallis, Ananth Balashankar, Peter Stone, Lukas Rutishauser
Computation and Language
Self-Filtered Distillation with LLMs-generated Trust Indicators for Reliable Patent Classification
Yongmin Yoo, Xu Zhang and Longbing Cao
Organizing large-scale patent corpora according to classification schemes is a core information management task that determines the accuracy and efficiency of prior art
retrieval, technology knowledge discovery, and intellectual property decision-making. Recent approaches distill natural
language rationales generated by large
language models (LLMs) into compact student models, yet logical errors, label mismatches, and taxonomy misalignments inherent in these rationales are indiscriminately absorbed during training, undermining classification reliability and propagating errors throughout downstream information processes. Rather than correcting such errors post-hoc, we propose Self-Filtered
Distillation (SFD), which embeds quality assurance directly into the learning process by reinterpreting LLM-generated rationales as trust indicators rather than ground-truth supervision. SFD integrates three unsupervised signals into a unified trust score that dynamically modulates each training instance's contribution: Self-Consistency, which quantifies agreement among independently generated rationales; Class Entailment
Alignment, which evaluates semantic coherence between a rationale and its assigned CPC class definition; and LLM Agreement Scoring, which assesses external plausibility through an independent verifier. On the USPTO-2M
benchmark comprising over two million patents, SFD achieves up to 38.7% relative improvement in Macro-F1 across four student architectures, and the strong correlation between trust scores and expert judgments ($r = 0.685$) confirms that the framework provides not only accurate predictions but also decomposable confidence semantics that enable auditable and self-documenting classification outcomes for large-scale patent knowledge organization.
Read the full abstract ▾
Computation and Language
A Lightweight Large Language Model-Based Multi-Agent System for 2D Frame Structural Analysis
Ziheng Geng, Jiachen Liu, Ran Cao et al. (+3 more)
Large
language models (LLMs) have recently been used to empower autonomous
agents in engineering, significantly improving automation and efficiency in labor-intensive workflows. However, their potential remains underexplored in structural engineering, particularly for finite element modeling tasks requiring
geometric modeling, complex
reasoning, and domain knowledge. To bridge this gap, this paper develops a LLM-based multi-
agent system to automate finite element modeling of 2D frames. The system decomposes structural analysis into subtasks, each managed by a specialized
agent powered by the lightweight Llama-3.3 70B Instruct model. The workflow begins with a Problem Analysis
Agent, which extracts geometry, boundary, and material parameters from the user input. Next, a Geometry
Agent incrementally derives node coordinates and element connectivity by applying expert-defined rules. These structured outputs are converted into executable OpenSeesPy code by a Translation
Agent and refined by a Model Validation
Agent through consistency checks. Then, a Load
Agent applies load conditions into the assembled structural model. Experimental evaluations on 20
benchmark problems demonstrate that the system achieves accuracy over 80% in most cases across 10 repeated trials, outperforming Gemini-2.5 Pro and ChatGPT-4o models.
Read the full abstract ▾
All authors Ziheng Geng, Jiachen Liu, Ran Cao, Lu Cheng, Haifeng Wang, Minghui Cheng
Computation and Language
Aligning Language Models with Clinical Expertise: DPO for Heart Failure Nursing Documentation in Critical Care
Junyi Fan, Li Sun, Negin Ashrafi et al. (+2 more)
Nursing documentation in intensive care units (ICUs) provides essential clinical intelligence but often suffers from inconsistent terminology, informal styles, and lack of standardization, challenges that are particularly critical in heart failure care. This study applies Direct Preference Optimization (DPO) to adapt Mistral-7B, a locally deployable
language model, using 8,838 heart failure nursing notes from the MIMIC-III database and 21,210 preference pairs derived from expert-verified GPT outputs, model generations, and original notes. Evaluation across BLEU, ROUGE, BERTScore, Perplexity, and expert qualitative assessments demonstrates that DPO markedly enhances documentation quality. Specifically, BLEU increased by 84% (0.173 to 0.318), BERTScore improved by 7.6% (0.828 to 0.891), and expert ratings rose across accuracy (+14.4 points), completeness (+14.5 points), logical consistency (+14.1 points), readability (+11.1 points), and structural clarity (+6.0 points). These results indicate that DPO can align lightweight clinical
language models with expert standards, supporting
privacy-preserving, AI-assisted documentation within electronic health record systems to reduce administrative burden and improve ICU patient safety.
Read the full abstract ▾
All authors Junyi Fan, Li Sun, Negin Ashrafi, Kamiar Alaei, Maryam Pishgar
Computation and Language · Cross-listed from quant-ph
Quantum Concept Music Score from Quantum Picturalism: Musical Incarnation of a Bell-Pair under Measurements
Rakhat-Bi Abdyssagin and Bob Coecke
We initiate the development of a new
language and theory for
quantum music, to which we refer as
Quantum Concept Music (QCM). This new music formalism is based on Categorical
Quantum Mechanics (CQM), and more specifically, its diagrammatic incarnation
Quantum Picturalism (QPict), which is heavily based on ZX-calculus. In fact, it is naturally inherited from CQM/QPict. At its heart is the explicit notational representation of relations that exist within and between the key concepts of music composition, performance, and automation. QCM also enables one to directly translate
quantum phenomena into music compositions in a both intuitively obvious, rigorous and mechanical manner. Following this pattern, we propose a score for musicians interacting like a Bell-pair under measurement, and outline examples of how it could be live performed. While most of the Western classical music notation has heavily relied on linear representation of music - which does not always adequately capture the nature of music - our approach is distinct by highlighting the fundamental relational dimension of music. In addition, this
quantum-based technique not only influences the music at the profound level of composition, but also has a direct impact on a live performance, and also provides a new template for automating music, e.g.~in the context of AI-generation. All together, we initiate the creation of new music formalism that is powerful and efficient in capturing the interactive nature of music, both in terms of internal and external interactions, and goes beyond the boundaries of Western classical music notation, which allows to use it in many different genres and directions.
Read the full abstract ▾
Computation and Language
Cross-Lingual Mental Health Ontologies for Indian Languages: Bridging Patient Expression and Clinical Understanding through Explainable AI and Human-in-the-Loop Validation
Ananth Kandala, Ratna Kandala, Akshata Kishore Moharir et al. (+2 more)
Mental health communication in India is linguistically fragmented, culturally diverse, and often underrepresented in clinical NLP. Current health ontologies and mental health resources are dominated by diagnostic frameworks centered on English or Western culture, leaving a gap in representing patient distress expressions in Indian
languages. We propose cross-linguistic graphs of patient stress expressions (CL-PDE), a framework for building cross-lingual mental health ontologies through graph-based methods that capture culturally embedded expressions of distress, align them across
languages, and link them with clinical terminology. Our approach addresses critical gaps in healthcare communication by grounding AI systems in culturally valid representations, allowing more inclusive and patient-centric NLP tools for mental health care in
multilingual contexts.
Read the full abstract ▾
All authors Ananth Kandala, Ratna Kandala, Akshata Kishore Moharir, Niva Manchanda, Sunaina Singh
Computation and Language
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Yufeng Du, Minyang Tian, Srikanth Ronanki et al. (+7 more)
Large
language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. This gap is commonly attributed to
retrieval failures – the models' inability to identify relevant information in the long inputs. Accordingly, recent efforts often focus on evaluating and improving LLMs'
retrieval performance: if
retrieval is perfect, a model should, in principle, perform just as well on a long input as it does on a short one – or should it? This paper presents findings that the answer to this question may be negative. Our systematic experiments across 5 open- and closed-source LLMs on math, question answering, and coding tasks reveal that, even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%–85%) as input length increases but remains well within the models' claimed lengths. This failure occurs even when the irrelevant
tokens are replaced with minimally distracting whitespace, and, more surprisingly, when they are all masked and the models are forced to attend only to the relevant
tokens. A similar performance drop is observed when all relevant evidence is placed immediately before the question. Our findings reveal a previously-unrealized limitation: the sheer length of the input alone can hurt LLM performance, independent of
retrieval quality and without any distraction. They motivate our simple, model-agnostic mitigation strategy that transforms a long-context task into a short-context one by prompting the model to recite the retrieved evidence before attempting to solve the problem. On RULER, we observe a consistent improvement of GPT-4o up to 4% on an already strong baseline.
Read the full abstract ▾
All authors Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, Hao Peng
Computation and Language
The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
Alexander M. Fichtl, Jeremias Bohn, Josefin Kelber et al. (+2 more)
Transformers have dominated sequence processing tasks for the past seven years – most notably
language modeling. However, the inherent quadratic complexity of their
attention mechanism remains a significant bottleneck as context length increases. This paper surveys recent efforts to overcome this bottleneck, including advances in (sub-quadratic)
attention variants, recurrent neural networks, state space models, and hybrid architectures. We critically analyze these approaches in terms of compute and
memory complexity,
benchmark results, and fundamental limitations to assess whether the dominance of pure-
attention transformers may soon be challenged.
Read the full abstract ▾
All authors Alexander M. Fichtl, Jeremias Bohn, Josefin Kelber, Edoardo Mosca, Georg Groh
Computation and Language
Residualized Similarity for Faithfully Explainable Authorship Verification
Peter Zeng, Pegah Alipoormolabashi, Jihu Mun et al. (+5 more)
Responsible use of Authorship Verification (AV) systems not only requires high accuracy but also interpretable solutions. More importantly, for systems to be used to make decisions with real-world consequences requires the model's prediction to be explainable using interpretable features that can be traced to the original texts. Neural methods achieve high accuracies, but their representations lack direct interpretability. Furthermore, LLM predictions cannot be explained faithfully – if there is an explanation given for a prediction, it doesn't represent the
reasoning process behind the model's prediction. In this paper, we introduce Residualized Similarity (RS), a novel method that supplements systems using interpretable features with a neural network to improve their performance while maintaining interpretability. Authorship verification is fundamentally a similarity task, where the goal is to measure how alike two documents are. The key idea is to use the neural network to predict a similarity residual, i.e. the error in the similarity predicted by the interpretable system. Our evaluation across four
datasets shows that not only can we match the performance of state-of-the-art authorship verification models, but we can show how and to what degree the final prediction is faithful and interpretable.
Read the full abstract ▾
All authors Peter Zeng, Pegah Alipoormolabashi, Jihu Mun, Gourab Dey, Nikita Soni, Niranjan Balasubramanian, Owen Rambow, H. Schwartz
Computation and Language
WeatherArchive-Bench: Benchmarking Retrieval-Augmented Reasoning for Historical Weather Archives
Yongan Yu, Xianda Du, Qingchen Hu et al. (+7 more)
Historical archives on weather events are collections of enduring primary source records that offer rich, untapped narratives of how societies have experienced and responded to extreme weather events. These qualitative accounts provide insights into societal vulnerability and resilience that are largely absent from meteorological records, making them valuable for
climate scientists to understand societal responses. However, their vast scale, noisy digitized quality, and archaic
language make it difficult to transform them into structured knowledge for
climate research. To address this challenge, we introduce WeatherArchive-Bench, the first
benchmark for evaluating
retrieval-augmented generation (RAG) systems on historical weather archives. WeatherArchive-Bench comprises two tasks: WeatherArchive-
Retrieval, which measures a system's ability to locate historically relevant passages from over one million archival news segments, and WeatherArchive-Assessment, which evaluates whether Large
Language Models (LLMs) can classify societal vulnerability and resilience indicators from extreme weather narratives. Extensive experiments across sparse, dense, and re-ranking retrievers, as well as a diverse set of LLMs, reveal that dense retrievers often fail on historical terminology, while LLMs frequently misinterpret vulnerability and resilience concepts. These findings highlight key limitations in
reasoning about complex societal indicators and provide insights for designing more robust
climate-focused RAG systems from archival contexts. The constructed
dataset and evaluation framework are publicly available at https://anonymous.4open.science/r/WeatherArchive-Bench/.
Read the full abstract ▾
All authors Yongan Yu, Xianda Du, Qingchen Hu, Jiahao Liang, Jingwei Ni, Dan Qiang, Kaiyu Huang, Grant McKenzie, Renee Sieber, Fengran Mo
Computation and Language
RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts
Yining She, Daniel W. Peterson, Marianne Menglin Liu et al. (+4 more)
With the increasing adoption of large
language models (LLMs), ensuring the safety of LLM systems has become a pressing concern. External LLM-based guardrail models have emerged as a popular solution to screen unsafe inputs and outputs, but they are themselves fine-tuned or
prompt-engineered LLMs that are vulnerable to data distribution shifts. In this paper, taking
Retrieval Augmentation Generation (RAG) as a case study, we investigated how robust LLM-based guardrails are against additional information embedded in the context. Through a systematic evaluation of 3 Llama Guards and 2 GPT-oss models, we confirmed that inserting benign documents into the guardrail context alters the judgments of input and output guardrails in around 11% and 8% of cases, making them unreliable. We separately analyzed the effect of each component in the augmented context: retrieved documents, user query, and LLM-generated response. The two mitigation methods we tested only bring minor improvements. These results expose a context-robustness gap in current guardrails and motivate training and evaluation protocols that are robust to
retrieval and query composition.
Read the full abstract ▾
All authors Yining She, Daniel W. Peterson, Marianne Menglin Liu, Vikas Upadhyay, Mohammad Hossein Chaghazardi, Eunsuk Kang, Dan Roth
Computation and Language · Cross-listed from eess.AS
WaveSP-Net: Learnable Wavelet-Domain Sparse Prompt Tuning for Speech Deepfake Detection
Xi Xuan, Xuechen Liu, Wenxin Zhang et al. (+3 more)
Modern front-end design for
speech deepfake detection relies on full fine-tuning of large pre-trained models like XLSR. However, this approach is not parameter-efficient and may lead to suboptimal generalization to realistic, in-the-wild data types. To address these limitations, we introduce a new family of parameter-efficient front-ends that fuse
prompt-tuning with classical signal processing transforms. These include FourierPT-XLSR, which uses the
Fourier Transform, and two variants based on the Wavelet Transform: WSPT-XLSR and Partial-WSPT-XLSR. We further propose WaveSP-Net, a novel architecture combining a Partial-WSPT-XLSR front-end and a bidirectional Mamba-based back-end. This design injects multi-resolution features into the
prompt embeddings, which enhances the localization of subtle synthetic artifacts without altering the frozen XLSR parameters. Experimental results demonstrate that WaveSP-Net outperforms several state-of-the-art models on two new and challenging
benchmarks, Deepfake-Eval-2024 and SpoofCeleb, with low trainable parameters and notable performance gains. The code and models are available at https://github.com/xxuan-acoustics/WaveSP-Net.
Read the full abstract ▾
All authors Xi Xuan, Xuechen Liu, Wenxin Zhang, Yi-Cheng Lin, Xiaojian Lin, Tomi Kinnunen
Computation and Language
Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Tarek Naous, Anagha Savit, Carlos Rafael Catalan et al. (+17 more)
As Large
Language Models (LLMs) develop stronger
multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric
multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western
languages. In this paper, we introduce Camellia, a
benchmark for evaluating entity-centric cultural biases in nine Asian
languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent
multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these
languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian
languages, creating performance gaps between cultures in entity extraction.
Read the full abstract ▾
All authors Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, Neel Kothari, Sahajpreet Singh, Sarah Masud, Tanish Patwa, Trung Thanh Tran, Zohaib Khan, Alan Ritter, Tanmoy Chakraborty, Yuki Arase, Keisuke Sakaguchi, JinYeong Bak, Wei Xu
Computation and Language · Cross-listed from cs.AI
Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
Radha Gulhane and Sathish Reddy Indurthi
Aligning
multimodal large
language models (MLLMs) with human preferences often relies on single-signal, model-based reward methods. Such monolithic rewards often lack confidence calibration across domain-specific tasks, fail to capture diverse aspects of human preferences, and require extensive data annotation and reward model training. In this work, we propose a hybrid reward modeling framework that integrates complementary reward paradigms: (i) model-based rewards, where a learned reward model predicts scalar or vector scores from synthetic and human feedback, and (ii) rule-based rewards, where domain-specific heuristics provide explicit correctness signals with confidence. Beyond accuracy, we further incorporate multi-aspect rewards to enforce instruction adherence and introduce a generalized length-penalty reward to stabilize training and improve performance. The proposed framework provides a flexible and effective approach to aligning MLLMs through reinforcement learning policy optimization. Our experiments show consistent improvements across different
multimodal benchmarks when applying hybrid and multi-aspect reward modeling. Our best performing model in the 3B family achieves an overall average improvement of ~9.5% across general and math
reasoning tasks. Focusing specifically on mathematical
benchmarks, the model achieves a significant average improvement of ~16%, highlighting its effectiveness in mathematical
reasoning and problem solving.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEs
Paloma García-de-Herreros, Philipp Slusallek, Dietrich Klakow et al. (+1 more)
While large
language models are primarily used on natural
language tasks, they have also shown great promise when adapted to new modalities, e.g., for scientific machine learning tasks. Most proposed approaches for such cross-modal adaptation of
language models focus on encoder-only
transformer model architectures, despite decoder-only architectures being far more popular for
language tasks in recent years, and being trained at much larger scales. This raises the question of how model architecture affects cross-modal adaptation approaches, and whether we can leverage the success of decoder-only models. In this paper, we systematically compare encoder-only and decoder-only
language models on cross-modal adaptation for time-dependent simulation tasks based on partial differential equations (PDEs). We find that decoder-only models are far worse than encoder-only models, when existing approaches are applied unmodified. In contrast to several other domains, scaling decoder-only models also does not help. To enhance the performance of decoder-only models in this context, we introduce two novel approaches that mimic bidirectionality, Parallel Flipping and Sequence Doubling. Both our methods improve overall performance using decoder-only models for all tasks and all cross-modal adaptation methods, closing the gap to encoder-only model performance. We hope that our findings broaden the spectrum of models used on cross-modal adaptation tasks to further scientific machine learning.
Read the full abstract ▾
All authors Paloma García-de-Herreros, Philipp Slusallek, Dietrich Klakow, Vagrant Gautam
Computation and Language
Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning
Chenghao Yang, Lin Gui, Chenxiao Yang et al. (+3 more)
Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm for enhancing the
reasoning capabilities of large
language models (LLMs), yet its success hinges on effective exploration. An ideal exploration strategy must navigate two fundamental challenges: it must preserve sample quality while also ensuring training stability. While standard fixed-temperature sampling is simple, it struggles to balance these competing demands, as high temperatures degrade sample quality and low temperatures limit discovery. In this work, we propose a simpler and more effective strategy, Exploratory Annealed Decoding (EAD), grounded in the insight that exploration is most impactful on early
tokens which define a sequence's semantic direction. EAD implements an intuitive **explore-at-the-beginning, exploit-at-the-end** strategy by annealing the sampling temperature from high to low during generation. This dynamic schedule encourages meaningful, high-level diversity at the start, then gradually lowers the temperature to preserve sample quality and keep the sampling distribution close to the target policy, which is essential for stable training. We demonstrate that EAD is a lightweight, plug-and-play method that significantly improves sample efficiency, consistently outperforming fixed-temperature sampling across various RLVR
algorithms and model sizes. Our work suggests that aligning exploration with the natural dynamics of sequential generation offers a robust path to improving LLM
reasoning.
Read the full abstract ▾
All authors Chenghao Yang, Lin Gui, Chenxiao Yang, Victor Veitch, Lizhu Zhang, Zhuokai Zhao
Computation and Language
A novel hallucination classification framework
Maksym Zavhorodnii, Dmytro Dehtiarov and Anna Konovalenko
This work introduces a novel methodology for the automatic detection of
hallucinations generated during large
language model (LLM) inference. The proposed approach is based on a systematic taxonomy and controlled reproduction of diverse
hallucination types through
prompt engineering. A dedicated
hallucination dataset is subsequently mapped into a vector space using an
embedding model and analyzed with unsupervised learning techniques in a reduced-dimensional representation of
hallucinations with veridical responses. Quantitative evaluation of inter-centroid distances reveals a consistent correlation between the severity of informational distortion in
hallucinations and their spatial divergence from the cluster of correct outputs. These findings provide theoretical and empirical evidence that even simple classification
algorithms can reliably distinguish
hallucinations from accurate responses within a single LLM, thereby offering a lightweight yet effective framework for improving model reliability.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Paper2Video: Automatic Video Generation from Scientific Papers
Zeyu Zhu, Kevin Qinghong Lin and Mike Zheng Shou
Academic presentation
videos have become an essential medium for research communication, yet producing them remains highly labor-intensive, often requiring hours of slide design, recording, and editing for a short 2 to 10 minutes
video. Unlike natural
video, presentation
video generation involves distinctive challenges: inputs from research papers, dense multi-modal information (text, figures, tables), and the need to coordinate multiple aligned channels such as slides, subtitles,
speech, and human talker. To address these challenges, we introduce Paper2
Video, the first
benchmark of 101 research papers paired with author-created presentation
videos, slides, and speaker metadata. We further design four tailored evaluation metrics–Meta Similarity, PresentArena, PresentQuiz, and IP
Memory–to measure how
videos convey the paper's information to the audience. Building on this foundation, we propose PaperTalker, the first multi-
agent framework for academic presentation
video generation. It integrates slide generation with effective layout refinement by a novel effective tree search visual choice, cursor grounding, subtitling,
speech synthesis, and talking-head rendering, while parallelizing slide-wise generation for efficiency. Experiments on Paper2
Video demonstrate that the presentation
videos produced by our approach are more faithful and informative than existing baselines, establishing a practical step toward automated and ready-to-use academic
video generation. Our
dataset,
agent, and code are available at https://github.com/showlab/Paper2
Video.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models
Mingkang Zhu, Xi Chen, Bei Yu et al. (+2 more)
Large
reasoning models (LRMs) generate intermediate
reasoning traces before producing final answers, yielding strong gains on multi-step and mathematical tasks. Yet aligning LRMs with human preferences, a crucial prerequisite for model deployment, remains underexplored. The statistically correct objective for preference
alignment requires marginalizing over
reasoning traces, but this computation is intractable in practice. A common workaround optimizes a single sampled trajectory, which introduces substantial gradient variance from
stochastic trace sampling. To address this challenge, we frame preference optimization for LRMs through the lens of the bias–variance trade-off and propose Bias–Variance Optimized Preference Optimization (BVPO), a simple, drop-in method that mixes two gradient
estimators: a high-variance trace-based
estimator and a low-variance empty-trace
estimator obtained by disabling
reasoning trace generation. Our theory shows that BVPO strictly reduces trace-induced variance for any nontrivial mixture, provides a closed-form choice of the mixing weight that minimizes mean-squared error relative to the true marginal gradient, and under standard smoothness and step-size conditions, tightens classical convergence bounds for
stochastic gradient descent. Empirically, BVPO improves
alignment over the best baseline by up to 7.8 points on AlpacaEval~2 and 6.8 points on Arena-Hard. Despite being trained only on general conversational data, BVPO also boosts
reasoning performance for base models by up to 4.0 points on the average of six math
reasoning benchmarks. These results identify variance from trace sampling as a key bottleneck and demonstrate that directly optimizing the bias–variance trade-off yields more stable training and stronger overall performance.
Read the full abstract ▾
All authors Mingkang Zhu, Xi Chen, Bei Yu, Hengshuang Zhao, Jiaya Jia