Computation and Language · Cross-listed from cs.LG
SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
Shikhar Shukla
Speculative decoding accelerates large
language model (LLM) inference by using a small draft model to propose candidate
tokens that a larger target model verifies. A critical hyperparameter in this process is the speculation length $\gamma$, which determines how many
tokens the draft model proposes per step. Nearly all existing systems use a fixed $\gamma$ (typically 4), yet empirical evidence suggests that the optimal value varies across task types and, crucially, depends on the compression level applied to the target model. In this paper, we present SpecKV, a lightweight adaptive controller that selects $\gamma$ per speculation step using signals extracted from the draft model itself. We profile speculative decoding across 4 task categories, 4 speculation lengths, and 3 compression levels (FP16, INT8, NF4), collecting 5,112 step-level records with per-step acceptance rates, draft
entropy, and draft confidence. We demonstrate that the optimal $\gamma$ shifts across compression regimes and that draft model confidence and
entropy are strong predictors of acceptance rate (correlation $\approx$ 0.56). SpecKV uses a small MLP trained on these signals to maximize expected
tokens per speculation step, achieving a 56.0% improvement over the fixed-$\gamma=4$ baseline with only 0.34 ms overhead per decision (<0.5% of step time). The improvement is statistically significant (p < 0.001, paired bootstrap test). We release all profiling data, trained models, and notebooks as open-source artifacts.
Read the full abstract ▾
Computation and Language
FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents
Quang Hieu Pham, Yang He, Ping Nie et al. (+5 more)
Text-to-SQL over large analytical databases requires navigating complex schemas, resolving ambiguous queries, and grounding decisions in actual data. Most current systems follow a fixed pipeline where schema elements are retrieved once upfront and the database is only revisited for post-hoc repair, limiting recovery from early mistakes. We present FlexSQL, a text-to-SQL
agent whose core design principle is flexible database interaction: the
agent can explore schema structure, inspect data values, and run verification queries at any point during
reasoning. FlexSQL generates diverse execution plans to cover multiple query interpretations, implements each plan in either SQL or Python depending on the task, and uses a two-tiered repair mechanism that can backtrack from code-level errors to plan-level revisions. On Spider2-Snow, using gpt-oss-120b, FlexSQL achieves a 65.4% score, outperforming strong open-source baselines that use stronger, larger models such as gpt-o3 and DeepSeek-R1. When integrated into a general-purpose coding
agent (as skills in Claude Code), our approach yields over 10% relative improvement on Spider2-Snow. Further analysis shows that flexible exploration and flexible execution jointly contribute to the effectiveness of our approach, highlighting flexibility as a key design principle. Our code is available at: https://github.com/StringNLPLAB/FlexSQL
Read the full abstract ▾
All authors Quang Hieu Pham, Yang He, Ping Nie, Canwen Xu, Davood Rafiei, Yuepeng Wang, Xi Ye, Jocelyn Qiaochu Chen
Computation and Language
Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces
Chenchen Zhang
As large
language model (LLM)
agents evolve from isolated tool users into coordinated teams, reinforcement learning (RL) must optimize not only individual actions but also how work is spawned, delegated, communicated, aggregated, and stopped. This paper studies RL for LLM-based multi-
agent systems through orchestration traces: temporal interaction graphs whose events include sub-
agent spawning, delegation, communication, tool use, return, aggregation, and stopping decisions. Using this lens, we identify three technical axes. First, reward design spans eight families, including orchestration rewards for parallelism speedup, split correctness, and aggregation quality. Second, reward and credit signals attach to eight credit- or signal-bearing units from
token to team; explicit counterfactual message-level credit remains especially sparse in our curated pool. Third, orchestration learning decomposes into five sub-decisions: when to spawn, whom to delegate to, how to communicate, how to aggregate, and when to stop. In our curated pool as of May 4, 2026, we found no explicit RL training method for the stopping decision. We connect academic methods to public industrial evidence from Kimi
Agent Swarm, OpenAI Codex, and Anthropic Claude Code. The resulting scale gap is a gap between publicly reported deployment envelopes and open academic evaluation regimes, not independent verification of industrial training traces. We release the artifact at https://github.com/xxzcc/awesome-llm-mas-rl, including an 84-entry tagged paper pool, a 32-record exclusion log, scripted corpus statistics, and a minimal JSON schema for replayable orchestration traces.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
FunFuzz: An LLM-Powered Evolutionary Fuzzing Framework
Mario Rodríguez Béjar, B. Romera-Paredes and Jose L. Hernández-Ramos
Modern fuzzers increasingly use Large
Language Models (LLMs) to generate structured inputs, but LLM-driven fuzzing is sensitive to
prompt initialization and sampling variance, which can reduce exploration efficiency and lead to redundant inputs. We present FunFuzz, a multi-island evolutionary fuzzing framework that runs several isolated searches in parallel and periodically migrates high-value candidates to maintain diversity. FunFuzz derives initial generation
prompts from documentation and initializes islands with topic-specific instructions, then continuously adapts
prompts using feedback-guided selection. During fuzzing, candidates are prioritized by incremental compiler coverage, while compiler-internal failure signals are used to identify crash-inducing inputs. We evaluate FunFuzz on compiler fuzzing, where inputs are source programs and success is measured by compiler coverage and unique compiler-internal failures. Across repeated 24-hour campaigns on GCC and Clang, FunFuzz achieves higher compiler coverage than previous LLM-driven baselines and discovers more unique failure-triggering inputs.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition
Pehuén Moure, Niclas Pokel, Bilal Bounajma et al. (+4 more)
Automatic
speech recognition (ASR) systems remain brittle on dysarthric and other atypical
speech. Recent audio-
language models raise the possibility of improving performance by conditioning on additional clinical context at inference time, but it is unclear whether these models can make use of such information. We introduce a
benchmark built on the
Speech Accessibility Project (SAP)
dataset that tests whether diagnosis labels, clinician-derived
speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric
speech. Across matched comparisons on nine models, we find that current models do not meaningfully use this context: diagnosis-informed and clinically detailed
prompts yield negligible improvements and often degrade word error rate. We complement the prompting analysis with context-dependent fine-tuning, showing that LoRA adaptation with a mixture of clinical
prompt formats achieves a WER of 0.066, a 52% relative reduction over the frozen baseline, while preserving performance when context is unavailable. Subgroup analyses reveal significant gains for Down syndrome and mild-severity speakers. These results clarify where current models fall short and provide a testbed for measuring progress toward more inclusive ASR.
Read the full abstract ▾
All authors Pehuén Moure, Niclas Pokel, Bilal Bounajma, Yingqiang Gao, Roman Boehringer, Longbiao Cheng, Shih-Chii Liu
Computation and Language · Cross-listed from cs.AI
Mitigating Misalignment Contagion by Steering with Implicit Traits
Maria Chang, Ronny Luss, Miao Liu et al. (+3 more)
Language models (LMs) are increasingly used in high-stakes, multi-
agent settings, where following instructions and maintaining value
alignment are critical. Most
alignment research focuses on interactions between a single LM and a single user, failing to address the risk of misaligned behavior spreading between multiple LMs in multi-turn interactions. We find evidence of this phenomenon, which we call misalignment contagion, across multiple LMs as they engage multi-turn conversational social dilemma games. Specifically, we find that LMs become more anti-social after gameplay and that this effect is intensified when other players are steered to act maliciously. We explore different steering techniques to mitigate such misalignment contagion and find that reinforcing an LM's system
prompt is insufficient and often harmful. Instead, we propose steering with implicit traits: a technique that intermittently injects system
prompts with statements that reinforce an LMs initial traits and is more effective than system
prompt repetition at keeping models in line with their initial pro-social behaviors. Importantly, this method does not require access to model parameters or internal model states, making it suitable for increasingly common use cases where complex multi-
agent workflows are being designed with black box models.
Read the full abstract ▾
All authors Maria Chang, Ronny Luss, Miao Liu, Keerthiram Murugesan, Karthikeyan Ramamurthy, Djallel Bouneffouf
Computation and Language · Cross-listed from cs.AI
Foundation Models to Unlock Real-World Evidence from Nationwide Medical Claims
Fan Ma, Yuntian Liu, Xiang Lan et al. (+23 more)
Real-world data increasingly inform regulatory and clinical decisions. Administrative claims, among their major sources, continuously track care and its cost across providers throughout enrollment, yet remain largely unexplored for healthcare foundation models. Here we present ReClaim, a family of generative foundation models built on 43.8 billion claims records from more than 200 million enrollees in the MarketScan Research Databases. ReClaim learns diagnoses, procedures, medications and expenditure jointly through pre-training from scratch, then specializes for downstream prediction tasks through post-training, with a training cutoff at the end of 2022. In predicting more than 1,000 diseases, ReClaim achieved a mean ROC-AUC of 0.769, outperforming disease-specific gradient-boosted models (0.663) and a
transformer-based baseline (0.694), and retained this advantage when generalizing to two independent cross-domain electronic health record
datasets. ReClaim also outperformed gradient-boosted models in
forecasting next-year expenditure and in identifying high-need, high-cost enrollees. ReClaim maintained its advantage in disease prediction and expenditure
forecasting on post-cutoff data (2023-2024). Beyond prediction, ReClaim representations used as
covariates resulted in improved effect estimates compared with
algorithm-derived
covariates, when benchmarked against results expected from randomized controlled trials. Together, these results establish nationwide insurance claims as a scalable substrate for healthcare foundation models supporting disease prediction, expenditure
forecasting and confounding adjustment in real-world evidence studies. Code is available at https://github.com/BIDS-Xu-Lab/ReClaim.
Read the full abstract ▾
All authors Fan Ma, Yuntian Liu, Xiang Lan, Weipeng Zhou, Jun Ni, Mauro Giuffrè, Lingfei Qian, Xueqing Peng, Yujia Zhou, Ruey-Ling Weng, Huan He, Lu Li, Huiyuan Wang, Qingyu Chen, Andrew Loza, Laila Rasmy, Degui Zhi, Yuan Lu, Chenjie Zeng, Joshua C Denny, Lee Schwamm, Daniella Meeker, Brian Ondov, Lucila Ohno-Machado, Yong Chen, Hua Xu
Computation and Language · Cross-listed from cs.CV
Scientific Domain Knowledge Improves Vision-Language Fundus Models
Verena Jasmin Hallitschke, Carsten Eickhoff and Philipp Berens
Vision-
language models hold considerable promise for ophthalmology, but it remains unclear which training data source best conveys expert domain knowledge. Existing ophthalmic models are trained on fixed text templates, medical reports, or general biomedical literature, sources that have never been compared under matched conditions. To include domain-specific literature in this comparison, we present PubMed-Ophtha, a hierarchical
dataset with high domain density of 102,023 panels with their subcaptions from 15,842 open-access articles in PubMed Central. We then finetuned identical CLIP models on each source, using a general biomedical literature model as baseline, and found that domain-specific literature achieved the best average performance across 110 clinical tasks, reaching a mean linear probing AUROC of 88.63% ahead of medical reports (85.68%). Restricting the
dataset to fundus images, to the image count of the medical report
dataset, or to articles unrelated to the evaluation
datasets did not reduce performance, indicating that the gains likely stem from domain density. We release the
dataset, the finetuned models, and the full generation pipeline.
Read the full abstract ▾
Computation and Language
mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection
Dominik Macko
SemEval-2026 Task 10 is focused on conspiracy detection. Specifically, the goal is to detect whether a Reddit comment expresses a conspiracy belief. Our submitted mdok-style system utilizes data augmentation and self-training (to cope with a rather small amount of training data) to finetune the Qwen3-32B model for a binary text-classification task. The submitted system is very competitive, ranking in the 85th percentile (8th out of 52 submissions). The results shown that our approach, which originated in machine-generated text detection, can be used for conspiracy detection as well.
Read the full abstract ▾
Computation and Language
mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection
Dominik Macko, Alok Debnath and Jakub Simko
SemEval-2026 Task 9 is focused on
multilingual polarization detection. Specifically, it covers the identification of
multilingual, multicultural and multievent polarization along three axes (in subtasks), namely detection, type, and manifestation. Online polarization presents a concern, because it is often followed by hate
speech, offensive discourse, and social fragmentation. Therefore, its detection before it escalates is crucial for a safer and more inclusive online space. We have coped with this SemEval task by finetuning mid-size LLMs for the sequence-classification task using the QLoRA parameter-efficient finetuning technique. The training data augmented the
multilingual (22
languages) training sets by anonymized, lower-cased, upper-cased, and homoglyphied counterparts, making the detection more robust.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The 2026 ACII Dyadic Conversations (DaiKon) Workshop & Challenge
Panagiotis Tzirakis, Alice Baird, Jeffrey Brooks et al. (+6 more)
The 2026 ACII Dyadic Conversations (ACII-DaiKon) Workshop & Challenge introduces a
benchmark for modeling interpersonal affect and social dynamics in dyadic conversations. Although conversational affect modeling has advanced rapidly, most
benchmarks remain speaker-centric and underrepresent coupled, time-evolving processes between partners, including directional influence, conversational timing coordination, and rapport development. To address this gap, ACII-DaiKon presents three coordinated sub-challenges built on a shared
dataset: (1) directional interpersonal influence prediction, (2) turn-taking prediction (next-speaker and time-to-next-
speech), and (3) rapport trajectory prediction across full interactions. The challenge is built on the Hume-DaiKon
dataset, comprising 945 dyadic conversations (743.4 hours of audiovisual data) collected under naturalistic conditions across five
languages. The
benchmark supports
multimodal modeling, temporal
reasoning, and cross-context generalization through fixed train/validation/test splits, standardized metrics, and released baseline systems. Evaluation uses Concordance Correlation Coefficient (CCC), Pearson correlation, Macro-F1, and Mean Absolute Error (MAE) depending on the sub-challenge. Baseline experiments establish initial reference performance, with best test results of 0.40 CCC and 0.50 Pearson for influence prediction, 0.66 Macro-F1 and 1.50~s MAE for turn-taking, and 0.68 CCC and 0.70 Pearson for rapport trajectory modeling. These results indicate that while current methods capture coarse dyadic patterns, robust modeling of directional dependence and long-horizon interpersonal dynamics remains challenging. The workshop provides a shared platform for rigorous comparison and cross-disciplinary discussion on data validity, evaluation protocols, and culturally aware modeling for dyadic interaction.
Read the full abstract ▾
All authors Panagiotis Tzirakis, Alice Baird, Jeffrey Brooks, Emilia Parada-Cabaleiro, Lukas Stappen, Sharath Rao, Theo Lebryk, Jakub Piotr Clapa, Jens Madsen
Computation and Language
Fuzzy Fingerprinting Encoder Pre-trained Language Models for Emotion Recognition in Conversations: Human Assessment and Validity Study
Patrícia Pereira, Helena Moniz and Joao Paulo Carvalho
In
Emotion Recognition in Conversations (ERC), model decisions should align with nuanced human perception and ideally provide insights on the classification process. Standard encoder pre-trained
language models (PLMs) are the state-of-the-art at these tasks but offer little insight into why a certain prediction is made. This is especially problematic in imbalanced
datasets, where most utterances are labeled as neutral, making these models frequently misclassify minority
emotions as the majority neutral class. To tackle this issue, we introduced a novel, interpretable approach to ERC by combining PLMs with Fuzzy Fingerprints (FFPs). FFP provide class-specific prototypes that reflect the characteristic class activation patterns in the PLM's latent space. They are derived by ranking and fuzzifying the activations of the pooled conversational context-dependent
embeddings across training instances for each
emotion. At inference time, each input utterance is similarly fuzzy fingerprinted and matched to the
emotion prototypes using a fuzzy similarity function based on the aggregation of the intersection of the fuzzy sets that define each FFP. Experimental results show that FFP integration reduces overclassification into the neutral class and human evaluation further supports the adequacy of FFP predictions. Our proposed method thus bridges the gap between deep neural inference and human perception, performing at state-of-the-art level while simultaneously offering valuable insights into the classification procedure.
Read the full abstract ▾
Computation and Language
ContextualJailbreak: Evolutionary Red-Teaming via Simulated Conversational Priming
Mario Rodríguez Béjar, Francisco J. Cortés-Delgado, S. Braghin et al. (+1 more)
Large
language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety
alignment and elicit harmful responses. A growing body of work shows that contextual priming, where earlier turns covertly bias later replies, constitutes a powerful attack surface, with hand-crafted multi-turn scaffolds consistently outperforming single-turn manipulations on capable models. However, automated optimization-based red-teaming has remained largely limited to the single-turn setting, iterating over static
prompts and lacking the ability to reason about which forms of conversational priming induce compliance. While recent multi-turn, search-based approaches have begun to bridge this gap, the mutator design space underlying effective primed dialogues remains largely unexplored. We present ContextualJailbreak, a black-box red-teaming strategy that performs evolutionary search over a simulated multi-turn primed dialogue. The strategy leverages a graded 0-5 harm score from a two-level judge as an in-loop signal, enabling partially harmful responses to guide the search process rather than being discarded. Search is driven by five semantically defined mutation operators: roleplay, scenario, expand, troubleshooting, and mechanistic, of which the last two are novel contributions of this work. Across 50 representative HarmBench behaviors, ContextualJailbreak achieves an ASR of 100% on gpt-oss:20B, 100% on qwen3-8B, 100% on llama3.1:70B, and 90% on gpt-oss:120B, outperforming four single- and multi-turn baselines by 31-96 percentage points on average. The 40 maximally harmful attacks discovered against gpt-oss:120B transfer without adaptation to closed frontier models, achieving 90.0% on gpt-4o-mini, 70.0% on gpt-5, and 70.0% on gemini-3-flash, but only 17.5% on claude-opus-4-7 and 15.0% on claude-sonnet-4-6, revealing a pronounced provider-level asymmetry in
alignment robustness.
Read the full abstract ▾
All authors Mario Rodríguez Béjar, Francisco J. Cortés-Delgado, S. Braghin, Jose L. Hernández-Ramos
Computation and Language
Mapping Discourse Reframing: A Multi-Layer Network Approach to Italian HPV Vaccine Discourse on X (2010-2024)
Lorella Viola
Understanding how online narratives travel through coalitions is critical for identifying information disorder, yet computational analyses often rely on conservative network constructions that erase initially sparse but salient signals. This paper proposes a novel multi-layer framework that captures low-frequency signals of emerging information disorder allowing for locating where online discourse is reframed and amplified over time. The use case is 14 years of Italian discourse on X regarding the Human Papillomavirus (HPV) vaccine across three pivotal epochs (2010-2024). Utilizing hashtag co-occurrence networks, we introduce a dual-layer approach. We first identify robust core discourse coalitions through conservative community detection, revealing a stable prevention-oriented backbone contrasted with increasingly separable skepticism coalitions. We then introduce a coverage layer and project fringe hashtags into core coalitions based on weighted connectivity. Using a manually labelled set of skeptical and conspiratorial seed tweets, we demonstrate that this core-coverage projection significantly improves the recovery of long-tail, problematic hashtags while preserving an interpretable coalition structure. Our findings characterize the structural maturation of polarized narratives and provide a methodology for mapping how discourse is reframed and amplified by information disorder over time.
Read the full abstract ▾
Computation and Language
Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversations
Yu Lu Liu, Hyokun Yun, Tanya Roosta et al. (+1 more)
There is growing interest in exploring user simulation as an alternative to gathering and scoring real user-chatbot interactions for AI chatbot evaluation. For this purpose, it is important to ensure the realism of the simulation, i.e., the extent to which simulated dialogues reflect real dialogues users have with chatbots. Most existing methods evaluating simulation realism produce coarse quality signal and remain solely at the level of individual dialogues. To support more rigorous evaluation in this area, we propose realsim, an evaluation framework that enables practitioners to take a distributional view of real vs. simulated dialogues along 8 dimensions, covering attributes related to the communicative functions of the interaction, user states, and the surface form of user messages. We then instantiate the framework with a curated
dataset of 1K multi-turn task-focused real user-chatbot dialogues that cover 16 domains of chatbot applications. Overall, we find that simulated users tend to struggle at capturing communication frictions that real users introduce to interactions, which could make evaluations based on such simulations overly optimistic. We also observe variability in performance across different domains, which may indicate a need for domain-specific user simulators.
Read the full abstract ▾
All authors Yu Lu Liu, Hyokun Yun, Tanya Roosta, Ziang Xiao
Computation and Language
Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race
Andreas Maier, Moritz Zaiss and Siming Bayer
Reproducing an empirical NLP study used to take weeks. Given the released data and a modern agentic-research harness, we redo every experiment of a recent ACL\,2026 study on personal-style post-editing of LLM drafts – and add three new ones – with the human investigator acting only as a reviewer-in-the-loop. We reproduce all seven preregistered hypotheses and recover the paper's headline correlation between perceived self-similarity and
embedding-measured self-similarity to three decimal places ($r{=}{+}0.244$, $p{<}10^{-8}$, $n{=}648$). Under a leakage-free held-out protocol, GPT-5.5 and Claude\,Opus\,4.7 close $71$–$75\,\%$ of the style gap to the same-author ceiling on $324$ paired tasks, against $24\,\%$ for the human post-edit, and beat the human post-edit on $\sim$$80\,\%$ of tasks. We then frame the same data as an AI-text detection arms race. A leave-authors-out linear SVM on LUAR-MUD
embeddings reaches AUC $0.93$–$1.00$ across approaches; six diagnostics show that GPT-5.5 detection is mostly a length confound while Opus detection is a genuine stylistic signature. Given $T{=}20$ feedback iterations against the frozen detector, an Opus
agent flips two of five held-out test mimics to the human half-space and shrinks every margin by an order of magnitude. With moderate effort against a known detector, a frontier LLM can already efficiently lower its own AI-detection probability. All code, $648$ mimic drafts, trained detectors, diagnostics, and
adversarial trajectories are released.
Read the full abstract ▾
Computation and Language
Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages
Kevin Guan, Happy Buzaaba and Christiane Fellbaum
Transformer-based models achieve state-of-the-art dependency parsing for high-resource
languages, yet their advantage over simpler architectures in low-resource settings remains poorly understood. We evaluate four parsers—the Biaffine LSTM, Stack-Pointer Network, AfroXLMR-large, and RemBERT—across twelve typologically diverse
languages, with a focus on low-resource African
languages. We find that the Biaffine LSTM consistently outperforms
transformer models in low-resource regimes, with
transformers recovering their advantage as training data increases. The crossover falls within a resource range typical of treebanks for under-resourced
languages. Morphological complexity (measured via MATTR) emerges as a significant secondary predictor of
transformers' relative disadvantage after controlling for corpus size. These results indicate that the Biaffine LSTM may be better suited for syntactic tool development in low-resource regimes until sufficient annotated data is available to leverage the representational capacity of pre-trained
transformers.
Read the full abstract ▾
Computation and Language
SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures
Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros et al. (+27 more)
We present our shared task on evaluating the adaptability of LLMs and NLP systems across multiple
languages and cultures. The task data consist of an extended version of our manually constructed BLEnD
benchmark (Myung et al. 2024), covering more than 30
language-culture pairs, predominantly representing low-resource
languages spoken across multiple continents. As the task is designed strictly for evaluation, participants were not permitted to use the data for training, fine-tuning, few-shot learning, or any other form of model modification. Our task includes two tracks: (a) Short-Answer Questions (SAQ) and (b) Multiple-Choice Questions (MCQ). Participants were required to predict labels and were allowed to submit any NLP system and adopt diverse modelling strategies, provided that the
benchmark was used solely for evaluation. The task attracted more than 140 registered participants, and we received final submissions from 62 teams, along with 19 system description papers. We report the results and present an analysis of the best-performing systems and the most commonly adopted approaches. Furthermore, we discuss shared insights into open questions and challenges related to evaluation, misalignment, and methodological perspectives on model behaviour in low-resource
languages and for under-represented cultures.
Read the full abstract ▾
All authors Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros, Jiho Jin, Amr Keleg, Meriem Beloucif, Yi Zhou, Rodrigo Agerri, Vladimir Araujo, Naomi Baes, James Barry, Joanne Boisson, Nancy F. Chen, Christine de Kock, Aleksandra Edwards, Joseba Fernandez de Landa, Mohamed Fazli Imam, Huda Hakami, Shu-Kai Hsieh, Joseph Marvin Imperial, Roy Ka-Wei Lee, Zhengyuan Liu, Chenyang Lyu, Younes Samih, Johan Sjons, Bryan Tan, Asahi Ushio, Weihua Zheng, Alice Oh, Jose Camacho-Collados
Computation and Language
Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
Devi Prasad Bal and Subhashree Puhan
Retrieval-Augmented Generation (RAG) offers a well-established path to grounding large
language model (LLM) outputs in external knowledge, yet the question of which
retrieval strategy works best in a high-stakes domain such as biomedicine has not received the controlled, multi-metric treatment it deserves. This paper presents a systematic empirical comparison of five
retrieval strategies – Dense Vector Search, Hybrid BM25 + Dense
retrieval, Cross-Encoder Reranking, Multi-Query Expansion, and Maximal Marginal Relevance (MMR) – within a biomedical question-answering RAG pipeline. All strategies share a fixed generation model (GPT-4o-mini), a common vector store (ChromaDB), and OpenAI's text-
embedding-3-small
embeddings, ensuring that observed differences are attributable to
retrieval alone. Evaluation is conducted on 250 question-answer pairs drawn from a preprocessed subset of the BioASQ
benchmark (rag-mini-bioasq) using four DeepEval metrics: contextual precision, contextual recall, faithfulness, and answer relevancy, each reported with 95% confidence intervals. A no-context ablation is included as a lower bound. Cross-Encoder Reranking achieves the best composite score (0.827) and highest contextual precision (0.852), confirming that query-document interaction yields measurable
retrieval gains. Multi-Query Expansion, despite its recall-oriented design, produces the weakest contextual precision (0.671), suggesting naive query diversification introduces
retrieval noise. MMR sacrifices answer relevancy for diversity, while the Dense baseline (composite 0.822) falls within 0.005 points of the top strategy. All RAG conditions dramatically outperform the no-context ablation on answer relevancy (0.658-0.701 vs. 0.287), confirming the practical value of
retrieval. The full pipeline, hyperparameters, and evaluation code are publicly available.
Read the full abstract ▾
Computation and Language
Revisiting Semantic Role Labeling: Efficient Structured Inference with Dependency-Informed Analysis
Sangpil Youm, Leah Jones and Bonnie J. Dorr
Semantic Role Labeling (SRL) provides an explicit representation of predicate-argument structure, capturing linguistically grounded relations such as who did what to whom. While recent NLP progress has been dominated by large
language models (LLMs), these systems often rely on implicit semantic representations, often lacking explicit structural constraints and systematic explanatory mechanisms. Traditionally, SRL systems have often relied on AllenNLP; however, the framework entered maintenance mode in December 2022, limiting compatibility with evolving encoder architectures and modern inference requirements. We revisit structured SRL modeling, introducing a modernized encoder-based framework that preserves explicit predicate-argument structure while enabling inference 10 times faster. Using BERT-base, the model attains comparable predictive performance, and RoBERTa and DeBERTa further improve F1 performance within the same framework. We adopt a dependency-informed diagnostic methodology to characterize span-level inconsistencies and conduct a representation-level analysis of LLM behavior under dependency-informed structural signals. Results indicate that dependency cues primarily improve structural stability. Finally, we illustrate how the framework's explicit predicate-argument structure can support
multilingual SRL projection as a downstream application.
Read the full abstract ▾
Computation and Language
Freja Thoresen and Dan Saattrup Smart
Most
hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource
languages. We investigate faithfulness
hallucinations, defined as model-generated content that is fluent and plausible but diverges from the provided input or is internally inconsistent. Leveraging the
multilingual MultiWikiQA
dataset, we utilize the LettuceDetect framework to create synthetic
hallucination datasets for 306
languages, from which we train
token-level
hallucination classifiers for 30 European
languages. In this work, we present evaluations of model
hallucinations on a selection of
languages: English, Danish, German, and Icelandic. Using these classifiers, we evaluate the
hallucination rates for Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B, and cogito-v1-preview-llama-70B. Our classifiers reveal notably higher
hallucination rates for Qwen3-0.6B (up to 60% of answers containing at least one
hallucination, peaking in Icelandic) and generally lower rates for larger models, with cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B performing best on most
languages.
Hallucination rates are consistently higher for lower-resource
languages, particularly Icelandic.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Tibetan-TTS:Low-Resource Tibetan Speech Synthesis with Large Model Adaptation
Jiaxu He, Chao Wang, Jie Lian et al. (+4 more)
Tibetan text-to-
speech (TTS) has long been challenged by scarce
speech resources, significant dialectal variation, and the complex mapping between written text and spoken pronunciation. To address these issues, this work presents, to the best of our knowledge, the first large-model-based Tibetan TTS system in the industry, built upon a large
speech synthesis model developed by Xingchen AGI Lab. The proposed system integrates data quality enhancement, Tibetan-oriented text representation and tokenizer adaptation, and cross-lingual adaptive training for low-resource Tibetan
speech synthesis. Experimental results show that the system can generate stable, natural, and intelligible Tibetan
speech under low-resource conditions. In subjective evaluation, the MOS scores of the syllable-level and BPE-based systems reach 4.28 and 4.35, while their pronunciation accuracies reach 97.6% and 96.6%, respectively, outperforming an external commercial Tibetan TTS interface. These results demonstrate that combining a large-model backbone with Tibetan-oriented text representation adaptation and cross-lingual adaptive training enables highly usable low-resource Tibetan
speech synthesis, and also provides a technical foundation for future unified multi-dialect Tibetan
speech synthesis.
Read the full abstract ▾
All authors Jiaxu He, Chao Wang, Jie Lian, Yuqing Cai, Yongxiang Li, Renzeg Duojie, Jie Li
Computation and Language · Cross-listed from cs.AI
GRAIL: A Deep-Granularity Hybrid Resonance Framework for Real-Time Agent Discovery via SLM-Enhanced Indexing
Jinliang Xu
As the ecosystem of Large
Language Model (LLM)-based
agents expands rapidly, efficient and accurate
Agent Discovery becomes a critical bottleneck for large-scale multi-
agent collaboration. Existing approaches typically face a dichotomy: either relying on heavy-weight LLMs for intent parsing, leading to prohibitive latency (often exceeding 30 seconds), or using monolithic vector
retrieval that sacrifices semantic precision for speed. To bridge this gap, we propose GRAIL (Granular Resonance-based
Agent/AI Link), a novel framework achieving sub-400ms discovery latency without compromising accuracy. GRAIL introduces three key innovations: (1) SLM-Enhanced Prediction, replacing the generalized LLM parser with a specialized, fine-tuned Small
Language Model (SLM) for millisecond-level capability tag prediction; (2) Pseudo-Document Expansion, augmenting
agent descriptions with synthetic queries to enhance semantic density for robust dense
retrieval; and (3) MaxSim Resonance, a fine-grained matching mechanism computing maximum similarity between user queries and discrete
agent usage examples, effectively mitigating semantic dilution. Validated on AgentTaxo-9K, our new large-scale
dataset of 9,240
agents, GRAIL reduces end-to-end discovery latency by over \textbf{79$\times$} compared to LLM-parsing baselines, while significantly outperforming traditional vector search in Recall@10. This framework offers a scalable, industrial-grade solution for the real-time “Internet of
Agents."
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Shadow-Loom: Causal Reasoning over Graphical World Models of Narratives
David Wilmot
Stories hold a reader's
attention because they have causes, secrets, and consequences. Shadow-Loom is an experimental open-source framework that turns a narrative into a versioned graphical world model and lets two engines act on it: a causal physics grounded in Pearl's ladder of causation and a recently proposed counterfactual calculus over Ancestral Multi-World Networks; and a narrative physics that scores the same graph against four structural reader-states – mystery, dramatic irony, suspense, and surprise – in the tradition of Sternberg's curiosity/suspense/surprise triad, with suspense formalised in the structural-affect line of work on story comprehension and computational suspense. Large
language models are used only at the boundary: extraction, rendering, and audit; identification, intervention, and counterfactual
reasoning are carried out in typed code over the graph. The system is offered as a research artefact rather than as a benchmarked NLP model; code, fixtures, and pipeline are released open source.
Read the full abstract ▾
Computation and Language
Accurate Legal Reasoning at Scale: Neuro-Symbolic Offloading and Structural Auditability for Robust Legal Adjudication
Stanisław Sójka and Witold Kowalczyk
Legal texts often contain computational legal clauses–provisions whose understanding requires complex logic. While frontier Large
Reasoning Models (LRMs) can describe such clauses, building production-ready systems is limited by
reasoning errors and the high cost of inference. We propose Amortized Intelligence, a neuro-symbolic approach where we use an LLM once to translate a legal text into Deterministic Autonomous Contract
Language (DACL): a typed graph intermediate representation. Adjudication then relies on deterministic graph executions with a visually auditable trace. In comparison against runtime LRM baselines (including GPT-5.2 and Gemini 3 Pro), our DACL-based
Agent achieves near-perfect consistency and mitigates the "
reasoning cliff" observed in probabilistic models. The system reduces compute costs by over 90% in high-volume workflows while satisfying the strict auditability requirements of legal adjudication.
Read the full abstract ▾
Computation and Language
ATLAS: Article Tracking, Linking, and Analysis of Swedish Encyclopedias
Albin Andersson, Salam Jonasson, Fredrik Wastring et al. (+1 more)
The digitization of old encyclopedias represents an important step to improve access to historically structured knowledge. Often, however, this process does not go beyond an optical character recognition, leaving all the underlying structure unexploited. In addition, many encyclopedias had multiple editions reflecting the
evolution of knowledge. The lack of structure in the raw text makes it difficult to track changes across these editions. In this work, we built a pipeline to restore the text structure, where we extract the headwords and identify entries; categorize the entities; match entries across editions; and link entries to a Wikidata item. We applied this pipeline to the four major editions of Nordisk familjebok, an authoritative Swedish encyclopedia published between 1876 and 1951. We could extract the headwords with an F1 score of 97.8% and we obtained an F1 score of 93.4% on the headword classification. On a small-scale evaluation, we reached a 93% precision on the cross-edition matching, 85% precision and 16.5% recall on the Wikidata linking. This shows that an automated approach to digitized historical knowledge is possible. This should facilitate the preservation of general knowledge and the understanding of knowledge transmission. The
datasets and programs are available online.
Read the full abstract ▾
All authors Albin Andersson, Salam Jonasson, Fredrik Wastring, Pierre Nugues
Computation and Language
Leveraging Argument Structure to Predict Content Hatefulness
Nicolás Benjamín Ocampo and Davide Ceolin
Information disorder is a challenging phenomenon that affects society at large. This phenomenon entails the
diffusion of misleading, misinforming, and hateful content online. In different contexts, one aspect of the problem may prevail, but overall, this is a broad problem that requires comprehensive solutions. While each dimension of the problem (hate
speech, disinformation, misinformation, etc.) requires in-depth analysis, in this paper, we look into the possibility of argument structure to provide relevant information to link these different areas of the problem. In particular, we focus on the WSF-ARG+
dataset, which consists of white supremacy forum messages annotated in terms of argument structure (premises and conclusion). There, we leverage the checkworthiness and hatefulness annotations of the argument components to obtain insights into the hatefulness of the whole message. Our results show promising insights (up to 96% F1), indicating the possibility of extending this direction in the future to tackle hateful content identification and information disorder countering.
Read the full abstract ▾
Computation and Language
PC-MNet: Dual-Level Congruity Modeling for Multimodal Sarcasm Detection via Polarity-Modulated Attention
Maoheng Li, Ling Zhou, Xiaohua Huang et al. (+3 more)
Multimodal sarcasm detection, which aims to precisely identify pragmatic incongruities between literal text and nonverbal cues, has gained substantial
attention in
multimodal understanding. Recent advancements have predominantly relied on naïve similarity-based
attention mechanisms and uniform late fusion strategies.Furthermore, given that functional
entanglement restricts traditional late fusions, we incorporate a scalar congruity routing mechanism and a prior-guided contextual graph. This mechanism anchors a generalized incongruity
manifold through a two-stage asymmetric optimization driven by inconsistency-aware contrastive learning, selectively fusing only the most discriminative multi-granularity evidence. Extensive experiments on the MUStARD
benchmark and its spurious-correlation-mitigated balanced
datasets demonstrate that our approach achieves new state-of-the-art performance, surpassing the strongest
multimodal baseline by a substantial 3.14% improvement in Macro-F1. By architecturally isolating atomic, composition, and contextual conflicts. This work provides a robust, decoupled paradigm for modeling subtle pragmatic incongruities in human communication.
Read the full abstract ▾
All authors Maoheng Li, Ling Zhou, Xiaohua Huang, Rubing Huang, Wenming Zheng, Guoying Zhao
Computation and Language
HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs
Ahmed Cherif
Large
Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural
language processing tasks, yet they remain susceptible to
hallucinations – generating content that is factually incorrect, unfaithful to provided context, or misaligned with user instructions. We present HalluScan, a comprehensive
benchmark framework that systematically evaluates
hallucination detection and mitigation across 72 configurations spanning 6 detection methods, 4 open-weight model families, and 3 diverse domains. We introduce three key contributions: (1) HalluScore, a novel composite metric that achieves a Pearson correlation of r = 0.41 with human expert judgments; (2) Adaptive Detection Routing (ADR), an intelligent routing
algorithm achieving 2.0x cost reduction with only 0.1% AUROC degradation; and (3) systematic error cascade decomposition revealing substantial variation in
hallucination error types across domains. Our experiments reveal that NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Measuring AI Reasoning: A Guide for Researchers
Munachiso Samuel Nwadike, Zangir Iklassov, Kareem Ali et al. (+2 more)
In this paper, we offer a guide for researchers on evaluating
reasoning in
language models, building the case that
reasoning should be assessed through evidence of adaptive, multi-step search rather than final-answer accuracy alone. Under an evaluation-oriented definition,
reasoning requires selecting intermediate steps and halting according to input-dependent conditions, which we formalize as a search-like procedure. We show that single forward passes in scalable architectures are structurally limited in their ability to realize such variable-depth computation, motivating intermediate decoding and externalized
reasoning traces as appropriate evaluation interfaces. Central to our argument is that final-answer accuracy alone is an insufficient measure of
reasoning, because it provides little ability to diagnose or debug the underlying processes that produce individual solutions in frontier models. We therefore argue for a shift toward process-based evaluation, in which
reasoning is assessed through the faithfulness and validity of intermediate
reasoning traces as first-class evaluation targets.
Read the full abstract ▾
All authors Munachiso Samuel Nwadike, Zangir Iklassov, Kareem Ali, Rifo Genadi, Kentaro Inui