Computation and Language · Cross-listed from cs.IR
WebExpert: domain-aware web agents with critic-guided expert experience for high-precision search
Yuelin Hu, Zhengxue Cheng, Ronghua Wu et al. (+5 more)
Specialized web tasks in finance, biomedicine, and pharmaceuticals remain challenging due to missing domain priors: queries drift, evidence is noisy, and
reasoning is brittle. We present WebExpert, a domain-aware web
agent that we implement end-to-end, featuring : (i) sentence-level experience
retrieval with topic merging and rule
distillation, (ii) schemalight facet induction that bootstraps time,region,policy,industry facets from weak supervision instead of static hand-written lexicons, and (iii) preference-optimized planning that jointly improves query planning and
retrieval via pairwise preference learning alongside a coverage-aware objective. At inference, a lightweight experience gate biases decoding toward active facets with fallback under low-
retrieval confidence. On
GAIA, GPQA, HLE, and WebWalkerQA, WebExpert improves Answer Exact Match (EM) by 1.5-3.6 pp over the strongest browsing baseline and reduces page hops. Analysis shows consistent gains and ablations on
retrieval, topic merging, facet induction, and preference-aware training.
Read the full abstract ▾
All authors Yuelin Hu, Zhengxue Cheng, Ronghua Wu, Qunshan Gu, Hongwei Hu, Wei Liu, Qiao Liang, Li Song
Computation and Language · Cross-listed from cs.IR
Robustness Risk of Conversational Retrieval: Identifying and Mitigating Noise Sensitivity in Qwen3-Embedding Model
Weishu Chen, Zhouhui Hou, Mingjie Zhan et al. (+2 more)
We present an empirical study of
embedding-based
retrieval under realistic conversational settings, where queries are short, dialogue-like, and weakly specified, and
retrieval corpora contain structured conversational artifacts. Focusing on Qwen3-
embedding models, we identify a deployment-relevant robustness vulnerability: under conversational
retrieval without query prompting, structured dialogue-style noise can become disproportionately retrievable and intrude into top-ranked results, despite being semantically uninformative. This failure mode emerges consistently across model scales, remains largely invisible under standard clean-query
benchmarks, and is significantly more pronounced in Qwen3 than in earlier Qwen variants and other widely used dense
retrieval baselines. We further show that lightweight query prompting qualitatively alters
retrieval behavior, effectively suppressing noise intrusion and restoring ranking stability. Our findings highlight an underexplored robustness risk in conversational
retrieval and underscore the importance of evaluation protocols that reflect the complexities of deployed systems.
Read the full abstract ▾
All authors Weishu Chen, Zhouhui Hou, Mingjie Zhan, Zhicheng Zhao, Fei Su
Computation and Language
Verify as You Go: An LLM-Powered Browser Extension for Fake News Detection
Dorsaf Sallami and Esma Aïmeur
The rampant spread of fake news in the digital age poses serious risks to public trust and democratic institutions, underscoring the need for effective, transparent, and user-centered detection tools. Existing browser extensions often fall short due to opaque model behavior, limited explanatory support, and a lack of meaningful user engagement. This paper introduces Aletheia, a novel browser extension that leverages
Retrieval-Augmented Generation (RAG) and Large
Language Models (LLMs) to detect fake news and provide evidence-based explanations. Aletheia further includes two interactive components: a Discussion Hub that enables user dialogue around flagged content and a Stay Informed feature that surfaces recent fact-checks. Through extensive experiments, we show that Aletheia outperforms state-of-the-art baselines in detection performance. Complementing this empirical evaluation, a complementary user study with 250 participants confirms the system's usability and perceived effectiveness, highlighting its potential as a transparent tool for combating online fake news.
Read the full abstract ▾
Computation and Language
Multiclass Hate Speech Detection with RoBERTa-OTA: Integrating Transformer Attention and Graph Convolutional Networks
Mahmoud Abusaqer and Jamil Saquer
Multiclass hate
speech detection across demographic categories remains computationally challenging due to implicit targeting strategies and linguistic variability in social media content. Existing approaches rely solely on learned representations from training data, without explicitly incorporating structured ontological frameworks that can enhance classification through formal domain knowledge integration. We propose RoBERTa-OTA, which introduces ontology-guided
attention mechanisms that process textual features alongside structured knowledge representations through enhanced Graph Convolutional Networks. The architecture combines RoBERTa
embeddings with scaled
attention layers and graph neural networks to integrate contextual
language understanding with domain-specific semantic knowledge. Evaluation across 39,747 balanced samples using 5-fold cross-validation demonstrates significant performance gains over baseline RoBERTa implementations and existing state-of-the-art methods. RoBERTa-OTA achieves 96.04% accuracy compared to 95.02% for standard RoBERTa, with substantial improvements for challenging categories: gender-based hate
speech detection improves by 2.36 percentage points while other hate
speech categories improve by 2.38 percentage points. The enhanced architecture maintains computational efficiency with only 0.33% parameter overhead, providing practical advantages for large-scale content moderation applications requiring fine-grained demographic hate
speech classification.
Read the full abstract ▾
Computation and Language
Simulating Meaning, Nevermore! Introducing ICR: A Semiotic-Hermeneutic Metric for Evaluating Meaning in LLM Text Summaries
Natalie Perez, Sreyoshi Bhaduri and Aman Chadha
Meaning in human
language is relational, context dependent, and emergent, arising from dynamic systems of signs rather than fixed word-concept mappings. In computational settings, this semiotic and interpretive complexity complicates the generation and evaluation of meaning. This article proposes an interdisciplinary framework for studying meaning in large
language model (LLM) generated
language by integrating semiotics and hermeneutics with qualitative research methods. We review prior scholarship on meaning and machines, examining how linguistic signs are transformed into vectorized representations in static and contextualized
embedding models, and identify gaps between statistical approximation and human interpretive meaning. We then introduce the Inductive Conceptual Rating (ICR) metric, a qualitative evaluation approach grounded in inductive content analysis and reflexive thematic analysis, designed to assess semantic accuracy and meaning
alignment in LLM-outputs beyond lexical similarity metrics. We apply ICR in an empirical comparison of LLM generated and human generated thematic summaries across five
datasets (N = 50 to 800). While LLMs achieve high linguistic similarity, they underperform on semantic accuracy, particularly in capturing contextually grounded meanings. Performance improves with larger
datasets but remains variable across models, potentially reflecting differences in the frequency and coherence of recurring concepts and meanings. We conclude by arguing for evaluation frameworks that leverage systematic qualitative interpretation practices when assessing meaning in LLM-generated outputs from reference texts.
Read the full abstract ▾
Computation and Language
Additive Multi-Step Markov Chains and the Curse of Dimensionality in Large Language Models
O.V. Usatenko, S.S. Melnyk and G.M. Pritula
Large-scale
language models (LLMs) operate in extremely high-dimensional state spaces, where both
token embeddings and their hidden representations create complex dependencies that are not easily reduced to classical Markov structures. In this paper, we explore a theoretically feasible approximation of LLM dynamics using N-order additive Markov chains. Such models allow the conditional probability of the next
token to be decomposed into a superposition of contributions from multiple historical depths, reducing the combinatorial explosion typically associated with high-order Markov processes. The main result of the work is the establishment of a correspondence between an additive multi-step chain and a chain with a step-wise
memory function. This equivalence allowed the introduction of the concept of information temperature not only for stepwise but also for additive N-order Markov chains.
Read the full abstract ▾
Computation and Language
One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache
Liming Lu, Kaixi Qiu, Jiayu Zhou et al. (+6 more)
Despite the remarkable progress of Large
Language Models (LLMs), the escalating
memory footprint of the Key-Value (KV) cache remains a critical bottleneck for efficient inference. While dimensionality reduction offers a promising compression avenue, existing approaches typically either necessitate prohibitively expensive pre-training from scratch or suffer from severe performance deterioration under high compression regimes. In this work, we propose DynaKV, a novel post-training framework for low-rank KV cache compression. To the best of our knowledge, DynaKV is the first method to dynamically allocate compression rates to individual
tokens according to their semantic meaning, which allows it to achieve better fidelity at aggressive compression ratios. Extensive experiments demonstrate that our method consistently outperforms existing state-of-the-art compression techniques, achieving significant
memory reduction while maintaining competitive generation quality. Furthermore, our approach is orthogonal to sequence-level
pruning methods. When integrated with SnapKV, DynaKV retains only 6% of the KV cache while maintaining 94% of the baseline performance on the LongBench
benchmark.
Read the full abstract ▾
All authors Liming Lu, Kaixi Qiu, Jiayu Zhou, Jushi Kai, Haoyan Zhang, Huanyu Wang, Jingwen Leng, Ziwei He, Zhouhan Lin
Computation and Language
SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models
Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh et al. (+2 more)
While different stakeholders are trying to leverage Arabic
Language Models (ALMs), safety
alignment in ALMs remains largely underexplored, hindering their mainstream adoption. Existing safety
benchmarks are predominantly English-centric and evaluate Arabic only in its standardized form, obscuring fine-grained safety vulnerabilities in Arabic NLP systems. This paper introduces SalamahBench, a unified
benchmark of 8{,}270 human-verified harmful
prompts across ML Commons hazard categories, each rendered in Modern Standard Arabic (MSA) and five regional Arabic varieties, namely Egyptian, Syrian, Saudi, Lebanese, and Moroccan, for a total of 49{,}620 paired instances. To analyze the resulting data, we introduce two complementary metrics, namely Dialect Shift, which measures a model's aggregate change in safety under dialectal reformulation, and Category-Specific Dialect Deviation, which isolates harm categories whose change departs from that aggregate trend. Evaluating models such as Fanar 2, ALLaM 2, and Karnak 1 under multiple safeguard configurations, we find that cross-variety robustness is strongly model dependent, and that aggregate scores can conceal category-level divergence. Our findings highlight the necessity of evaluating Arabic model safety jointly across linguistic varieties and harm domains rather than relying on aggregate scores or MSA alone.
Read the full abstract ▾
All authors Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh, Ihsen Alouani, Mohammed E. Fouda
Computation and Language
Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework
Nora Petrova, Andrew Gordon and Enzo Blindow
The evaluation of large
language models faces significant challenges. Technical
benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism. To address these issues, we introduce HUMAINE, a framework for multidimensional, demographically aware measurement of human-AI interaction. We collected multi-turn, naturalistic conversations from 23,404 participants that were stratified across 22 demographic groups, both in the US and UK, to evaluate 28 state-of-the-art models across five human-centric dimensions. We use a hierarchical
Bayesian Bradley-Terry-Davidson (BTD) model, with post-stratification to census data, and our analysis reveals three key insights. (1) We establish a clear performance hierarchy where google/gemini-2.5-pro ranks first overall, with a 95.6% posterior probability of being the top-ranked model. (2) We uncover significant preference heterogeneity, with user age emerging as the primary demographic axis of disagreement; a model's perceived rank can shift substantially across age groups, exposing failures in generalisation that unrepresentative samples typically mask. (3) We quantify the vast difference in discriminative power across evaluation dimensions, with ambiguous qualities like Trust, Ethics & Safety showing a 65% tie rate, in stark contrast to the decisive 10% tie rate for Overall Winner. Our work emphasises the need for a more multidimensional, demographically aware perspective in LLM evaluation. We release our complete
dataset, interactive leaderboard, and open-source framework.
Read the full abstract ▾
Computation and Language
Probing Memes in LLMs: A Paradigm for the Entangled Evaluation World
Luzhou Peng, Zhengxin Yang, Honglu Ji et al. (+6 more)
Current evaluation paradigms for large
language models (LLMs) characterize models and
datasets separately, yielding coarse descriptions: items in
datasets are treated as pre-labeled entries, and models are summarized by overall scores such as accuracy, together ignoring the diversity of population-level model behaviors across items with varying properties. To address this gap, this paper conceptualizes LLMs as composed of memes, a notion introduced by Dawkins as cultural genes that replicate knowledge and behavior. Building on this perspective, the Probing Memes paradigm reconceptualizes evaluation as an entangled world of models and data. It centers on a Perception
Matrix that captures model-item interactions, enabling Probe Properties for characterizing items and Meme Scores for depicting model behavioral traits. Applied to 9
datasets and 4,507 LLMs, Probing Memes reveals hidden capability structures and quantifies phenomena invisible under traditional paradigms (e.g., elite models failing on problems that most models answer easily). It not only supports more informative and extensible
benchmarks but also enables population-based evaluation of LLMs.
Read the full abstract ▾
All authors Luzhou Peng, Zhengxin Yang, Honglu Ji, Yikang Yang, Fanda Fan, Wanling Gao, Jiayuan Ge, Yilin Han, Jianfeng Zhan
Computation and Language
TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generation
Samah Fodeh, Linhai Ma, Ganesh Puthiaraju et al. (+5 more)
Direct Preference Optimization (DPO) is effective for offline
alignment but poorly matched to ontology-driven structured prediction, where preferred and rejected JSON often differ by only a few schema-defining
tokens. In this low-edit-distance regime, sequence-level DPO spreads gradient mass across non-critical serialization
tokens (gradient dilution) and can reduce likelihood on rare preferred schema
tokens (
token erosion). To address these limitations, we first develop a confusion-aware preference-construction strategy combining expert-curated ambiguity patterns with validation-set SFT structured-error modes, producing minimally perturbed, schema-valid negatives for ontology-level decision errors. We then introduce
Token-Adaptive Barrier Preference Optimization (TAB-PO), a post-SFT objective for
token-critical structured generation with a confidence-gated
token-level barrier that anchors under-confident schema
tokens. On SciERC, with Llama/Qwen models, TAB-PO improves ontology-critical semantic-label and relational-linking metrics by 11.59% relative to SFT, wins 100% of comparisons against strongest
token-level/sequence-level DPO variants, and surpasses strongest frontier baselines on these metrics by 14.71% relative while improving textual grounding.
Read the full abstract ▾
All authors Samah Fodeh, Linhai Ma, Ganesh Puthiaraju, Srivani Talakokkul, Afshan Khan, Sreeraj Ramachandran, Elyas Irankhah, Aimee Kendall Roundtree
Computation and Language
Personalization Increases Affective Alignment but Has Role-Dependent Effects on Epistemic Independence in LLMs
Sean W. Kelley and Christoph Riedl
Large
Language Models (LLMs) are prone to sycophantic behavior, uncritically conforming to user beliefs. As models increasingly condition responses on user-specific context (personality traits, preferences, conversation history), they gain information to tailor agreement more effectively. Understanding how personalization modulates sycophancy is critical, yet systematic evaluation across models and contexts remains limited. We present a rigorous evaluation of personalization's impact on LLM sycophancy across nine frontier models and five
benchmark datasets spanning advice, moral judgment, and debate contexts. We find that personalization generally increases affective
alignment (emotional validation, hedging/deference), but affects epistemic
alignment (belief adoption, position stability, resistance to influence) with context-dependent role modulation. When the LLM's role is to give advice, personalization strengthens epistemic independence (models challenge user presuppositions). When its role is that of a social peer, personalization decreases epistemic independence. In this role, extensively personalized user challenges causing LLMs to abandon their position at significantly higher rates. Robustness tests confirm that the effects are driven by personalized conditioning, not by additional input
tokens per se or demographic information alone. Our work provides measurement frameworks for evaluating personalized AI systems, demonstrates the necessity of role-sensitive evaluation, and establishes a novel
benchmark to assess goal
alignment.
Read the full abstract ▾
Computation and Language
Noise reduction in BERT NER models for clinical entity extraction
Kuldeep Jiwani, Yash K Jeengar and Ayush Dhaka
Precision is of utmost importance in the realm of clinical entity extraction from clinical notes and reports. Encoder Models fine-tuned for Named Entity Recognition (NER) are an efficient choice for this purpose, as they don't hallucinate. We pre-trained an in-house BERT over clinical data and then fine-tuned it for NER. These models performed well on recall but could not close upon the high precision range, needed for clinical models. To address this challenge, we developed a Noise Removal model that refines the output of NER. The NER model assigns
token-level entity tags along with probability scores for each
token. Our Noise Removal (NR) model then analyzes these probability sequences and classifies predictions as either weak or strong. A naïve approach might involve filtering predictions based on low probability values; however, this method is unreliable. Owing to the characteristics of the SoftMax function,
Transformer based architectures often assign disproportionately high confidence scores even to uncertain or weak predictions, making simple thresholding ineffective. To address this issue, we adopted a supervised modeling strategy in which the NR model leverages advanced features such as the Probability Density Map (PDM). The PDM captures the Semantic-Pull effect observed within
Transformer embeddings, an effect that manifests in the probability distributions of NER class predictions across
token sequences. This approach enables the model to classify predictions as weak or strong with significantly improved accuracy. With these NR models we were able to reduce False Positives across various clinical NER models by 50% to 90%.
Read the full abstract ▾
Computation and Language
From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarization
Ruangrin Ldallitsakool, Margarita Bugueño and Gerard de Melo
Recent NLP systems commonly represent documents as linear
token sequences. Although this captures sequential order, it can hinder modeling long-range dependencies and global document structure, especially for long texts. This paper proposes a data-driven method to automatically construct graph-based document representations. Building upon the recent work of Bugueño and de Melo (2025), we leverage the dynamic sliding-window
attention module to effectively capture local and mid-range semantic dependencies between sentences, as well as structural relations within documents. Graph
Attention Networks (GATs) trained on our learned graphs achieve competitive results on document classification while requiring lower computational resources than previous approaches. We further present an exploratory evaluation of the proposed graph construction method for extractive document summarization, highlighting both its potential and current limitations. The implementation of this project can be found on GitHub.
Read the full abstract ▾
Computation and Language
Multimodal Consistency-Guided Reference-Free Data Selection for ASR Accent Adaptation
Ligong Lei, Wenwen Lu, Xudong Pang et al. (+2 more)
Automatic
speech recognition (ASR) systems often degrade on accented
speech because acoustic-phonetic and prosodic shifts induce a mismatch to training data, making labeled accent adaptation costly. However, common pseudo-label selection heuristics are largely text-centric (e.g., perplexity (PPL) filtering) and can prefer fluent yet acoustically mismatched hypotheses, leading to error amplification when fine-tuning. To address this, we introduce a
multimodal consistency-guided, reference-free data selection pipeline for ASR accent adaptation under a transductive, label-free protocol. The pipeline starts with a target-aware preselection step based on submodular mutual information to improve query relevance and reduce downstream computation. It then generates multiple pseudo-transcriptions per utterance via perturbation-based decoding and scores each hypothesis using two reference-free signals:
speech–text
alignment in a shared
embedding space and predicted word error rate (WER). A simple percentile-based selection rule retains reliable pseudo-labels for fine-tuning while discarding noisy utterances. In an in-domain setting, selecting ~1.5k utterances from a 30k pool achieves 10.91% WER, close to 10.45% obtained using 30k supervised labels. In a cross-domain setting with a mismatched candidate pool, consistency-filtered subsets avoid the degradation caused by unfiltered pseudo-labels under strong accent shift, and matched-hour experiments on a stronger ASR backbone further confirm gains over random sampling and recent selection baselines.
Read the full abstract ▾
All authors Ligong Lei, Wenwen Lu, Xudong Pang, Zaokere Kadeer, Aishan Wumaier
Computation and Language · Cross-listed from cs.AI
General learned delegation by clones
Darren Li, Meiqi Chen, Chenze Shao et al. (+2 more)
Frontier
language models improve with additional test-time computation, but serial
reasoning or uncoordinated parallel sampling can be compute-inefficient under fixed inference budgets. We propose SELFCEST, which equips a base model with the ability to spawn same-weight clones in separate parallel contexts by agentic reinforcement learning. Training is end-to-end under a global task reward with shared-parameter rollouts, yielding a learned controller that allocates both generation and context budget across branches. Across challenging math
reasoning benchmarks and long-context multi-hop QA, SELFCEST improves the accuracy-cost Pareto frontier relative to monolithic baselines at matched inference budget, and exhibits out-of-distribution generalization in both domains.
Read the full abstract ▾
All authors Darren Li, Meiqi Chen, Chenze Shao, Fandong Meng, Jie Zhou
Computation and Language · Cross-listed from cs.AI
MAPLE: A Sub-Agent Architecture for Memory, Learning, and Personalization in Agentic AI Systems
Deepak Babu Piskala
Large
language model (LLM)
agents have emerged as powerful tools for complex tasks, yet their ability to adapt to individual users remains fundamentally limited. We argue this limitation stems from a critical architectural conflation: current systems treat
memory, learning, and personalization as a unified capability rather than three distinct mechanisms requiring different infrastructure, operating on different timescales, and benefiting from independent optimization. We propose MAPLE (
Memory-Adaptive Personalized LEarning), a principled decomposition where
Memory handles storage and
retrieval infrastructure; Learning extracts intelligence from accumulated interactions asynchronously; and Personalization applies learned knowledge in real-time within finite context budgets. Each component operates as a dedicated sub-
agent with specialized tooling and well-defined interfaces. Experimental evaluation on the MAPLE-Personas
benchmark demonstrates that our decomposition achieves a 14.6% improvement in personalization score compared to a stateless baseline (p < 0.01, Cohen's d = 0.95) and increases trait incorporation rate from 45% to 75% – enabling
agents that genuinely learn and adapt.
Read the full abstract ▾
Computation and Language
Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability
Shafiuddin Rehan Ahmed and Sourabh Deshpande
Multi-
agent frameworks (MAFs) promise to simplify LLM-driven software development, yet no principled metric captures how well AI coding assistants can generate correct, framework-specific code. We introduce AI-assistability ($\mathcal{AI}$), a composite metric that quantifies a framework's amenability to AI-assisted development by combining structural
alignment ($\bar{\sigma}$) with functional correctness (pass@1). To evaluate this metric in a controlled setting, we design DDL2PropBank, a novel
benchmark task that maps relational database schemas to PropBank semantic rolesets, and implement identical
agent logic across ten frameworks using the
Agent-as-a-Tool pattern. Our results challenge the intuition that declarative framework design guarantees AI-assistability: Agno, with a single canonical pattern and convention-aligned API, achieves the highest $\mathcal{AI}$ score (0.55), while DSPy – the most declarative framework by design – scores lowest (0.07), as its novel abstractions are insufficiently represented in AI training data. We find that convention
alignment, not declarative design alone, is the primary driver of AI-assistability ($r = 0.576$ between $\bar{\sigma}$ and pass@1). All artifacts – DDL2PropBank, PropBank MCP server, and all implementations – are available at https://github.com/ahmeshaf/ddl2propbank
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
Yifan Ji, Zhipeng Xu, Zhenghao Liu et al. (+7 more)
Key Information Extraction (KIE) from real-world documents remains challenging due to substantial variations in layout structures, visual quality, and task-specific information requirements. Recent Large
Multimodal Models (LMMs) have shown promising potential for performing end-to-end KIE directly from document images. To enable a comprehensive and systematic evaluation across realistic and diverse application scenarios, we introduce UNIKIE-BENCH, a unified
benchmark designed to rigorously evaluate the KIE capabilities of LMMs. UNIKIE-BENCH consists of two complementary tracks: a constrained-category KIE track with scenario-predefined schemas that reflect practical application needs, and an open-category KIE track that extracts any key information that is explicitly present in the document. Experiments on 15 state-of-the-art LMMs reveal substantial performance degradation under diverse schema definitions, long-tail key fields, and complex layouts, along with pronounced performance disparities across different document types and scenarios. These findings underscore persistent challenges in grounding accuracy and layout-aware
reasoning for LMM-based KIE. All codes and
datasets are available at https://github.com/NEUIR/UNIKIE-BENCH.
Read the full abstract ▾
All authors Yifan Ji, Zhipeng Xu, Zhenghao Liu, Zulong Chen, Qian Zhang, Zhibo Yang, Junyang Lin, Yu Gu, Ge Yu, Maosong Sun
Computation and Language · Cross-listed from cs.SD
MENASpeechBank: A Reference Voice Bank with Persona-Conditioned Multi-Turn Conversations for AudioLLMs
Zien Sheikh Ali, Hunzalah Hassan Bhatti, Rabindra Nath Nandi et al. (+2 more)
Audio large
language models (AudioLLMs) enable instruction following over
speech and general audio, but progress is limited by the scarcity of diverse, conversational, and instruction-aligned
speech–text data. This gap is particularly pronounced for persona-grounded and dialectal interactions, where collecting real multi-speaker recordings remains costly and slow. We introduce MENASpeechBank, a reference
speech bank comprising ~18K high-quality utterances from 124 speakers spanning multiple MENA countries, covering English, Modern Standard Arabic (MSA), and regional Arabic varieties. We develop a controllable data pipeline that (i) constructs persona profiles enriched with World Values Survey (WVS) inspired attributes, (ii) defines a taxonomy driven ~5Kconversational scenarios, (iii) matches personas to scenarios via semantic similarity, (iv) generates ~417K role-play conversations with an LLM where the user speaks as the persona and the assistant behaves as a helpful
agent, and (v) produces speaker-conditioned user-turn audio (synthetic) from reference recordings to preserve speaker diversity. We evaluate synthetic and human recorded conversations and provide an analysis. We will make the MENASpeechBank available for the community.(\href{https://huggingface.co/
datasets/QCRI/MenaSpeechBank)
Read the full abstract ▾
All authors Zien Sheikh Ali, Hunzalah Hassan Bhatti, Rabindra Nath Nandi, Shammur Absar Chowdhury, Firoj Alam
Computation and Language · Cross-listed from cs.AI
LLM-FSM: Scaling Large Language Models for Finite-State Reasoning in RTL Code Generation
Yuheng Wu, Berk Gokmen, Zhouhua Xie et al. (+4 more)
Finite-state
reasoning, the ability to understand and implement state-dependent behavior, is central to hardware design. In this paper, we present LLM-FSM, a
benchmark that evaluates how well large
language models (LLMs) can recover finite-state machine (FSM) behavior from natural-
language specifications and translate it into correct register transfer-level (RTL) implementations. Unlike prior specification-to-RTL
benchmarks that rely on manually constructed examples, LLM-FSM is built through a fully automated pipeline. LLM-FSM first constructs FSM with configurable state counts and constrained transition structures. It then
prompts LLMs to express each FSM in a structured YAML format with an application context, and to further convert that YAML into a natural-
language (NL) specification. From the same YAML, our pipeline synthesizes the reference RTL and testbench in a correct-by-construction manner. All 1,000 problems are verified using LLM-based and SAT-solver-based checks, with human review on a subset. Our experiments show that even the strongest LLMs exhibit sharply declining accuracy as FSM complexity increases. We further demonstrate that training-time scaling via supervised fine-tuning (SFT) generalizes effectively to out-of-distribution (OOD) tasks, while increasing test-time compute improves
reasoning reliability. Finally, LLM-FSM remains extensible by allowing its FSM complexity to scale with future model capabilities.
Read the full abstract ▾
All authors Yuheng Wu, Berk Gokmen, Zhouhua Xie, Peijing Li, Caroline Trippel, Priyanka Raina, Thierry Tambe
Computation and Language · Cross-listed from cs.AI
Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RL
Xiaofeng Lin, Sirou Zhu, Yilei Chen et al. (+6 more)
Large
language models (LLMs) achieve strong performance when all task-relevant information is available upfront, as in static prediction and instruction-following problems. However, many real-world decision-making tasks are inherently online: crucial information must be acquired through interaction, feedback is delayed, and effective behavior requires balancing information collection and exploitation over time. While in-context learning enables adaptation without weight updates, existing LLMs often struggle to reliably leverage in-context interaction experience in such settings. In this work, we show that this limitation can be addressed through training. We introduce ORBIT, a multi-task, multi-episode meta-reinforcement learning framework that trains LLMs to learn from interaction in context. After meta-training, a relatively small open-source model (Qwen3-14B) demonstrates substantially improved in-context online learning on entirely unseen environments, matching the performance of GPT-5.2 and outperforming standard RL fine-tuning by a large margin. Scaling experiments further reveal consistent gains with model size, suggesting significant headroom for learn-at-inference-time decision-making
agents. Code reproducing the results in the paper can be found at https://github.com/XiaofengLin7/ORBIT.
Read the full abstract ▾
All authors Xiaofeng Lin, Sirou Zhu, Yilei Chen, Mingyu Chen, Hejian Sang, Ioannis Paschalidis, Zhipeng Wang, Aldo Pacchiano, Xuezhou Zhang
Computation and Language · Cross-listed from cs.SD
BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning
Min Jang, Orevaoghene Ahia, Nazif Tamer et al. (+3 more)
Music understanding is a complex task that often requires
reasoning over both structural and semantic elements of audio. We introduce BASS, designed to evaluate music understanding and
reasoning in audio
language models across four broad categories: structural
segmentation, lyric transcription, musicological analysis, and artist collaboration. BASS comprises 2658 questions spanning 12 tasks, 1993 unique songs and covering over 138 hours of music from a wide range of genres and tracks, crafted to assess musicological knowledge and
reasoning in real-world scenarios. We evaluate 14 open-source and frontier
multimodal LMs, finding that even state-of-the-art models struggle on higher-level
reasoning tasks such as structural
segmentation and artist collaboration, while performing best on lyric transcription. Our analysis reveals that current models leverage linguistic priors effectively but remain limited in
reasoning over musical structure, vocal, and musicological attributes. BASS provides an evaluation framework with widespread applications in music recommendation and search and has the potential to guide the development of audio LMs.
Read the full abstract ▾
All authors Min Jang, Orevaoghene Ahia, Nazif Tamer, Sachin Kumar, Yulia Tsvetkov, Noah A. Smith
Computation and Language
Emily Cheng, Aditya R. Vaidya and Richard Antonello
Research has repeatedly demonstrated that intermediate hidden states extracted from large
language models and
speech audio models predict measured
brain response to natural
language stimuli. Yet, very little is known about the representation properties that enable this high prediction performance. Why is it the intermediate layers, and not the output layers, that are most effective for this unique and highly general transfer task? We give evidence that the correspondence between
speech and
language models and the
brain derives from shared meaning abstraction and not their next-word prediction properties. In particular, models construct higher-order linguistic features in their middle layers, cued by a peak in the layerwise intrinsic dimension, a measure of feature complexity. We show that a layer's intrinsic dimension strongly predicts how well it explains fMRI and ECoG signals; that the relation between intrinsic dimension and
brain predictivity arises over model pre-training; and finetuning models to better predict the
brain causally increases both representations' intrinsic dimension and their semantic content. Results suggest that semantic richness, high intrinsic dimension, and
brain predictivity mirror each other, and that the key driver of model-
brain similarity is rich meaning abstraction of the inputs, where
language modeling is a task complex enough (but perhaps not the only) to require it.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Stroke Lesions as a Rosetta Stone for Language Model Interpretability
Julius Fridriksson, Roger D. Newman-Norlund, Saeed Ahmadi et al. (+10 more)
Large
language models (LLMs) have achieved remarkable capabilities, yet methods to verify which model components are truly necessary for
language function remain limited. Current interpretability approaches rely on internal metrics and lack external validation. Here we present the
Brain-LLM Unified Model (BLUM), a framework that leverages lesion-symptom mapping, the gold standard for establishing causal
brain-behavior relationships for over a century, as an external reference structure for evaluating LLM perturbation effects. Using data from individuals with chronic post-stroke aphasia (N = 410), we trained symptom-to-lesion models that predict
brain damage location from behavioral error profiles, applied systematic perturbations to
transformer layers, administered identical clinical assessments to perturbed LLMs and human patients, and projected LLM error profiles into human lesion space. LLM error profiles were sufficiently similar to human error profiles that predicted lesions corresponded to actual lesions in error-matched humans above chance in 67% of picture naming conditions (p < 10^{-23}) and 68.3% of sentence completion conditions (p < 10^{-61}), with semantic-dominant errors mapping onto ventral-stream lesion patterns and phonemic-dominant errors onto dorsal-stream patterns. These findings open a new methodological avenue for LLM interpretability in which clinical neuroscience provides external validation, establishing human lesion-symptom mapping as a reference framework for evaluating artificial
language systems and motivating direct investigation of whether behavioral
alignment reflects shared computational principles.
Read the full abstract ▾
All authors Julius Fridriksson, Roger D. Newman-Norlund, Saeed Ahmadi, Regan Willis, Nadra Salman, Kalil Warren, Xiang Guan, Yong Yang, Srihari Nelakuditi, Rutvik Desai, Leonardo Bonilha, Jeff Charney, Chris Rorden