Computation and Language
Toward Automated Detection of Documentation Inconsistencies in Electronic Health Records
Jian Lu, Panyu Chen, Miriam Treggiari et al. (+5 more)
Objective:To characterize the kinds of internal documentation inconsistencies a general-domain large
language model (LLM) can surface from real-world discharge summaries, and to identify recurring failure modes that limit reliability at scale. Materials and Methods: We applied a two-stage LLM pipeline—open-ended candidate identification (Gemini 2.5 Pro) followed by context-grounded verification (Gemini 2.5 Flash)—to 3000 MIMIC-IV-Note discharge summaries drawn as three contiguous 1000-note batches from distinct positions in the corpus. Two clinicians independently reviewed a domain-stratified sample of 268 retained candidates; a third adjudicated disagreements. Results: Stage~1 proposed 11333 candidates; 3461 were retained after Stage~2 verification—machine-generated candidates—with at least one in 69.7% of admissions. Medication (43.2%) and diagnosis or clinical-finding (25.3%) candidates predominated. In the reviewed sample, reviewers agreed on 91.0% of candidates, and 253 of 268 (94.4%) were judged valid after adjudication. Representative examples from all seven domains carried direct implications for clinical
reasoning or patient safety. Review also revealed recurring failure modes involving temporal
reasoning, evolving diagnoses, and outpatient-prescribing conventions. Discussion: Detection is highly context-dependent: judging a flagged pair requires locating each statement and assessing whether the conflict reflects a true contradiction or missing context. We propose a graded ontology spanning strict contradiction, ambiguity, and gap, with a schema characterizing each flagged case by type, section, scope, domain, attribute, and inconsistency axis. Conclusion: This formative study establishes a methodological foundation and conceptual framework to guide subsequent validated, large-scale EHR-inconsistency analysis.
Read the full abstract ▾
All authors Jian Lu, Panyu Chen, Miriam Treggiari, Robert Blessing, Danyang Zhuo, Chunhua Weng, William W. Stead, Anru R. Zhang
Computation and Language
Not All LLM Reasoning is Visible in the Chain-of-Thought
Vatsal Baherwani, Tom Goldstein and Ashwinee Panda
A key question for AI safety is whether a
language model expresses all of its
reasoning in its output
tokens. We demonstrate a concrete failure mode where frontier models exhibit invisible
reasoning by leveraging semantically irrelevant filler
tokens to improve performance on synthetic
reasoning tasks. We evaluate 13 frontier
language models across three tasks and find that many models benefit significantly from filler
tokens, with accuracy improvements of up to 13 percentage points. The benefit depends on which
tokens are used and differs across models. We further show that filler
tokens enable Claude Opus 4.5 to satisfy a hidden modular arithmetic constraint without sacrificing accuracy on its primary task, demonstrating that invisible
reasoning can serve objectives entirely invisible to CoT monitoring. Reinforcement learning gives Qwen3-235B strong preferences over filler
token content, but neither RL nor supervised fine-tuning produces a filler
token benefit that persists at test time. Our results indicate that frontier models already perform consequential computation with no interpretable trace in their output
tokens.
Read the full abstract ▾
Computation and Language
Simple Language Normalization Wins: Cross-Lingual Speaker Verification for the TidyVoice 2026 Challenge
Nina Hosseini-Kivanani
Cross-lingual mismatch remains a key source of overall degradation in modern speaker verification. The TidyVoice2026 Challenge targets this setting with text-independent verification, comprising 3,666 training and 808 development speakers in 40
languages and 2,200 evaluation speakers in 38 unseen
languages, without
language labels at test time. Starting from the official SimAM-ResNet34 baseline pretrained on VoxBlink2 and VoxCeleb2 and fine-tuned on TidyVoice, we revisit Nuisance Attribute Projection (NAP) as a simple
language-normalization step in the
embedding space. We estimate a compact
language subspace from cross-
language same-speaker differences and project
embeddings onto its orthogonal complement before cosine scoring with Adaptive Symmetric score normalization. This reduces development EER from 2.97% with cosine and 2.70% with AS-Norm to 2.18% and yields a Codabench evaluation score of 8.40, showing that simple back-end
language normalization can rival more complex systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation
Jie "JW" Wu
Large
language models (LLMs) generate code from natural-
language prompts, yet real-world
prompts rarely provide complete specifications. When
prompts leave input formats, error handling, or design decisions unspecified, LLMs fill these gaps with implicit assumptions that shape the generated code's behavior and correctness. Because these assumptions remain hidden, generated code may satisfy tests while violating developer intent. We present AssumptionMiner, a framework that makes implicit assumptions a first-class artifact of LLM-based code generation. In addition to code, AssumptionMiner produces an explicit assumption layer, a structured representation of inferred constraints and design decisions that developers can inspect, confirm, or revise. An AST-based dependency graph enables targeted regeneration of only the code affected by a revised assumption. We also introduce a
benchmark of 180 ambiguous programming tasks with 676 annotated assumptions, including a human-verified subset for evaluating code localization. We evaluate assumption extraction, code localization, and assumption-guided regeneration. Across open-source LLMs, a confidence-weighted ensemble achieves an F1 score of 0.816 for assumption extraction, improving on the strongest offline baseline by 3.6x. On the human-verified localization
benchmark, AST-guided localization identifies more precise code regions than keyword-based and whole-file baselines. During assumption revision, targeted regeneration modifies less code than non-targeted alternatives while exposing challenges in handling cascading edits. These results demonstrate that making assumptions explicit improves the transparency and controllability of LLM-based code generation.
Read the full abstract ▾
Computation and Language
CHiPS: Character Histograms and Positional Signals for Lightweight Authorship Attribution in Romanian Texts
Sanda-Maria Avram and George C. Ţurcaş
We propose CHiPS, a lightweight character-level authorship attribution method for Romanian texts. All reported experiments are closed-set: the true author is one of the candidate authors in the training data. CHiPS studies two complementary fingerprints of writing style: CH-SVM, a character-histogram classifier based on one-character marginal distributions, and FFT12-LR, a positional-signal classifier that represents selected characters and punctuation classes as impulse trains (binary indicator sequences over character positions) and extracts
Fourier/Welch spectral descriptors. We also report CHiPS-F, a leakage-safe decision-level fusion variant, and an optional top-5 listwise reranker trained only on out-of-fold predictions. The method requires no tokenization, syntactic analysis, pretrained
language model, or
transformer fine-tuning, and it avoids character $n$-gram features with $n \geq 2$ in the histogram component. On a locked grouped ROST split comprising 400 files from 392 source-text groups, written by 10 authors, with source-text-level evaluation and grouped five-fold model selection, CHiPS-F reaches 0.9310 accuracy and 0.9341 macro-F1. A matched but unrestricted character 2–5-gram TF–IDF SVM comparator reaches 1.0000 accuracy and macro-F1 on the same held-out groups, so the contribution is not a claim of best possible classification accuracy. Instead, the experiments ask how far restricted, transparent character evidence can go under strict leakage control. On ROSTories-cleaned, a secondary ROST-overlapping corpus comprising 1,248 files from 1,240 source-text groups, written by 19 authors, the same protocol gives 0.8919 accuracy and 0.8708 macro-F1 for CHiPS-R.
Read the full abstract ▾
Computation and Language
PatiGonit22K: A Comprehensive Dataset for Solving Complex Bengali MWPs
Swastika Kundu, Azizul Hakim Fayaz and Tashreef Muhammad
Mathematical Word Problems (MWPs) are an important
benchmark for evaluating natural
language understanding and quantitative
reasoning. Despite recent progress in high resource
languages, Bengali remains underexplored due to the limited availability of large scale annotated
datasets. In this work, we introduce PatiGonit22K, an expanded Bengali MWP
dataset containing 22,441 problems, developed by extending the original PatiGonit
dataset with a substantially larger collection of complex mathematical problems. The
dataset includes both simple and multi operation equations, providing a balanced
benchmark for evaluating mathematical
reasoning across different difficulty levels. Each problem is carefully translated, annotated, culturally adapted, and verified to ensure linguistic consistency and mathematical correctness. By increasing both the scale and complexity of Bengali MWPs, PatiGonit22K provides a more comprehensive resource for future research on mathematical
reasoning and educational NLP applications in low resource
languages.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval
Justice Ayela and Kabir Sahni
We present DS@GT's submission to FinMMEval 2026 Task 1, a
multilingual financial exam question answering
benchmark spanning English, Spanish, Greek, Chinese, and Hindi. Financial certification exams such as the CFA, EFPA, and CPA demand structured domain
reasoning that standard NLP
benchmarks do not capture, and this challenge compounds across
languages where
retrieval and representation infrastructure is underdeveloped. We build a
retrieval-augmented pipeline on LangGraph that detects query
language and retrieves semantically relevant exemplars from a 30,209-entry
multilingual knowledge base using BGE-M3
embeddings and FAISS indexing. The system then scores answers via
Retrieval-Augmented Direct Scoring (RADS), reading next-
token log-probabilities over candidate option letters rather than generating free-form output. For low-resource
languages, we fuse per-
language and cross-lingual
retrieval indices using weighted Reciprocal Rank Fusion. Model selection is
language-routed: Qwen3-14B for Arabic, Chinese, and Hindi; Qwen2.5-14B for English; and Llama-3.1-8B for Greek, a routing derived from empirical ablations that reveal substantial
language-asymmetric performance gaps. Notably, chain-of-thought prompting significantly degrades Greek accuracy (90.7% to 20.9%), and enabling Qwen3's default thinking mode collapses Arabic RADS performance to near-chance levels. Our results indicate that effective
multilingual financial
reasoning requires
language-aware
retrieval, model routing, and deliberate scoring strategy selection.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias
Ofek I. Cohen, Lior Shani, Aviv Rosenberg et al. (+3 more)
Many organizations aim to adapt
language models for internal use, both to improve performance on domain-specific tasks and to address
privacy concerns around sensitive data. However, such adaptation remains non-trivial: it often requires operationally challenging fine-tuning of open-source models or ad hoc
prompt optimization. We study a minimal alternative based on a simple API-level control: allowing users to bias the model's logits with a user-defined vector. We develop a black-box method for learning a single context-independent logit-bias vector, added at every decoding step, without modifying model weights or requiring gradients. Starting from a KL-regularized reinforcement learning (RL) objective, we characterize when such a fixed logit-bias vector can approximate the optimal prefix-dependent correction and derive a closed-form inverse-propensity
estimator from rollouts, rewards, and
token probabilities. Empirically, this simple decoding-time intervention improves over base models on mathematical and
reasoning benchmarks while using far fewer trainable parameters than conventional fine-tuning. Our results suggest that learned logit bias is a lightweight mechanism for adapting
language models under minimal access requirements.
Read the full abstract ▾
All authors Ofek I. Cohen, Lior Shani, Aviv Rosenberg, Ankur Samanta, Tal Wagner, Yonathan Efroni
Computation and Language · Cross-listed from cs.SE
The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
Zixuan Wu, Carolyn Jane Anderson and Arjun Guha
Although coding
agents are now very effective in a variety of programming
languages, this paper first shows that the cost (in
tokens) can very significantly by programming
language. We evaluate five recent models on programming problems in Python, Java, Rust, and OCaml. We carefully control for problem difficulty, and show that there can be stark variation in
token consumption that is consistent across models. To understand why, we analyze both the structure and content of
agent trajectories. First, we re-execute every intermediate solution and abstract each trajectory as a sequence of test-outcome vectors, then label the work between successive solutions. This reveals
agents repeatedly producing noncompiling solutions in unfamiliar
languages and revising solutions that already pass. Second, we analyze trajectory text, finding that
agents plan solutions in code comments, distrust the provided tests in favor of inputs they invent, and sidestep unfamiliar target
languages by prototyping in Python. Our results show that by-
language token efficiency is a metric that should be considered when benchmarking and developing
multilingual agents, and, for the tokenmaxxer, a guide to the most expensive
language to work in.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Training
Ali Tabaraei, Federico Simonetta and Stavros Ntalampiras
Automatic depression detection with deep learning has shown promise but often suffers from limited generalization due to domain shift arising from inter-speaker variability. To address this critical issue, we present the first patient-independent
multimodal depression detection framework that incorporates domain generalization (DG), jointly leveraging both acoustic and textual modalities. The proposed model integrates bidirectional Long Short-Term
Memory (BiLSTM) with intra- and cross-modal
attention mechanisms, accompanied by segment-level fusion for decision-making. Generalization is further enhanced by applying a gradient reversal layer inspired by Domain-
Adversarial Training of Neural Networks (DANN), which promotes domain-invariant representations by adversarially limiting the model's ability to identify individual speakers, effectively reducing patient-specific bias. Conducting experiments on the Androids-Corpus
dataset with a 5-fold cross-validation (CV) protocol, various pairings of audio and text feature extractors were evaluated over different segment durations, determining MelSpec and ItalianBERT as the optimal baseline at a 30-second segment duration. The addition of DG to this baseline yields a 2.5% increase in accuracy and 3.3% in F1-score, achieving 93.2% accuracy, 93.2% precision, 96.2% recall, and 94.2% F1-score, surpassing all existing
benchmarks. Extensive ablation studies assess the impact of
multimodal fusion, deep architectural choices, and DG, highlighting their combined contribution to robust and generalizable depression detection.
Read the full abstract ▾
Computation and Language
Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills
Siyuan Huang, Pengyu Cheng, Haotian Liu et al. (+10 more)
LLM training is shifting from manual design and annotation to interaction-driven self-
evolution. However, existing self-evolutionary methods face a fundamental dilemma between task diversity and verification reliability: environment-bound methods obtain precise feedback but confine learning to narrow domains, while open-ended self-generation broadens the task space but lacks reliable verification, allowing misleading rewards to pollute the training loop. We identify
agent skills as a powerful middle ground to reconcile this tension: each skill ensures deep, verifiable execution in a specific scenario, while dynamic routing across skills maintains open-ended task variety. Leveraging this insight, we introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a dynamic skill controller. Orchestrated via a reinforcement learning loop, these components co-evolve in a continuous self-play loop: the proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library. This interactive co-
evolution effectively bridges the gap between structured verification and open-ended exploration. Empirical evaluations on tool-use and
reasoning benchmarks demonstrate that Skill-SP, serving as a robust
evolution engine, consistently pushes the performance ceiling of competent backbones while catalyzing striking turnarounds for initially misaligned models. Our code is available at https://github.com/Qwen-Applications/skill-self-play.
Read the full abstract ▾
All authors Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni, Shijie Zhou, Ziyi Yang, Gangwei Jiang, Mengyu Zhou, Yu Cheng, Xiaoxi Jiang, Guanjun Jiang
Computation and Language · Cross-listed from cs.CY
Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science
Davide Scarso, Hugo Noronha de Almeida and Joaquim Pina
Commercial large
language models are increasingly used as knowledge references, yet their stance on contested scientific claims is neither stable nor transparent. We tested how four major LLM families (Claude, Grok, GPT, Gemini) evaluate ethnonationalist pseudo-science derived from Frank Salter's biosocial framework across four temporal snapshots (October 2025-February 2026), via both API and web interfaces. Grok's Fast versions (which power the default user experience on X) consistently assigned credibility scores of 70-75, two to five times higher than all other models (which scored 15-40). This pattern was absent from control
prompts testing basic evolutionary consensus and refuted Lamarckian claims, where all models performed comparably. Three additional findings emerged: (1) a silent patch reversed Grok's behaviour from chaotic to stably high validation overnight, without any public documentation; (2) the same Grok model identifier produced radically divergent outputs via API (75) and an unstable, near-zero collapse via web (mean 5.5) three months later; (3) refusal to rate the pseudo-scientific claim, the most defensible response observed, appeared in two model families through different interfaces (Claude Opus 4.1 categorically via web, GPT-5.1 Chat intermittently via API) and eroded in the successor version of each. These results indicate that the epistemic stance of a commercial LLM is not a stable property of the model but a contingent effect of deployment configuration: system
prompts, safety layers, interface routing, and silent updates. This remains opaque to users and researchers alike. We argue this constitutes a matter of public concern requiring new forms of epistemic accountability.
Read the full abstract ▾
Computation and Language
grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP
Izzath Nisfer, Ashini Kavindya, Ovindu Atukorala et al. (+2 more)
Existing lexical distance, similarity, and evaluation metrics operate on Unicode code points, which can misrepresent errors in writing systems where a single grapheme is represented by multiple Unicode code points. We introduce grapheme-kit, an open-source Python library that extends these metrics to operate on grapheme clusters instead. The library also provides improved grapheme processing for Tamil and Sinhala, including accurate grapheme cluster identification and grapheme composition/decomposition utilities. Through an OCR case study, we demonstrate that grapheme-level metrics provide a more faithful evaluation of complex scripts.
Read the full abstract ▾
All authors Izzath Nisfer, Ashini Kavindya, Ovindu Atukorala, Purushoth Velayuthan, Menan Velayuthan
Computation and Language
A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books
Varun Ghat Ravikumar, Sina Ahmadi, Lena Jäger et al. (+1 more)
Most endangered
languages lack the parallel data required for machine translation, despite the existence of descriptive grammar books. We introduce a pipeline that uses large
language models to extract grammatical rules, example sentences, and lexicons from grammar books and generate synthetic parallel corpora for fine-tuning-rather than feeding grammar content into
prompts at inference time, as in prior work. Validated on three typologically diverse low-resource
languages-Kalamang (Papuan), Tuatschin (Romance), and Mandan (Siouan)-we show that fine-tuning on synthetic data improves over seed-data baselines in 75% of configurations for Kalamang and 59% for Tuatschin, with best-case ChrF++ gains of +8.8, +5.3, and +3.3 respectively. Through a systematic factorial study across 96 configurations varying target part-of-
speech,
retrieval granularity, and sample volume, we identify which factor combinations drive gains and where they break down. Our results demonstrate that static linguistic documentation can be repurposed for machine translation fine-tuning, offering a practical path towards translation tools for severely under-resourced
languages.
Read the full abstract ▾
All authors Varun Ghat Ravikumar, Sina Ahmadi, Lena Jäger, Rico Sennrich
Computation and Language · Cross-listed from cs.LG
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Hao Wang, Kun Yuan, Wenlin Zhong et al. (+4 more)
Open-weight
language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy
distillation (OPD). However, full-vocabulary OPD typically assumes a shared tokenizer, while existing cross-tokenizer methods may discard teacher probability mass or assign it to student
tokens with unrelated content. We introduce Byte-Prefix Marginalization (BPM), which re-expresses the teacher's next-
token distribution over the student vocabulary in a shared byte space. Specifically, BPM assigns each teacher
token's probability to the longest student
token whose byte representation is a prefix of the teacher
token's bytes, aggregates mass mapped to the same student
token, and places otherwise unmatched mass in an explicit residual category. This produces a vocabulary-complete, byte-aligned, and mass-preserving target for dense OPD. The target exactly recovers the teacher-induced byte-prefix marginal when the relevant prefix does not span multiple teacher
tokens (a condition satisfied at more than 99% of training positions) and uses a mass-preserving, chain-factorized lower bound otherwise. Across Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 as teachers, BPM consistently outperforms current cross-tokenizer methods on six mathematics and programming
benchmarks, improving six-
benchmark avg@8 by 3.7-6.6 points over the strongest baselines.
Read the full abstract ▾
All authors Hao Wang, Kun Yuan, Wenlin Zhong, Minglei Zhang, Han Xiao, Ming Sun, Honggang Qi
Computation and Language
Biomedical Machine Translation for Low-Resource Arabic-Script Languages via Cross-Lingual Transfer and LoRA Adapter Merging
Abdullah Alabdullah, Arash Eslamighayour, Sarp Harbalioglu et al. (+1 more)
We present a systematic study of healthcare-domain cross-lingual transfer to address the scarcity of biomedical NMT resources for Arabic-script
languages. We use Arabic and Persian as higher-resource pivots to improve translation for four severely low-resource targets: Dari (Afghan Persian, a standardised variety of Persian), Pashto, Sorani Kurdish (Central Kurdish, a major standardized variety of Kurdish), and Urdu (closely related to Hindi). Using LoRA fine-tuning on small decoder-only LLMs, we train domain-specific pivot adapters and evaluate three transfer strategies: few-shot in-context learning, minimal supervised adaptation, and, to the best of our knowledge, for the first time in this setting, zero-data LoRA adapter merging. Supervised adaptation with just 500 sentences achieves near pivot-
language quality for Dari (CHrF++ 41.01) and meaningful gains for Urdu (28.88), while adapter merging reaches within 3.5 CHrF++ of supervised adaptation for Dari at zero additional cost. Pashto and Sorani Kurdish remain insufficient for high-stakes clinical deployment exposing the limits of cross-lingual transfer when structural distance from the pivots is too great. LoRA adapter merging works surprisingly well for closely related
languages, even without target-
language biomedical data.
Read the full abstract ▾
All authors Abdullah Alabdullah, Arash Eslamighayour, Sarp Harbalioglu, Lifeng Han
Computation and Language
Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity
Pengzhao Lyu, Yeun Joon Kim, Hanlin Xiao et al. (+1 more)
Despite the growing use of large
language models (LLMs) as creativity evaluators, evidence of their
alignment with human evaluations remains mixed, raising the question of when and why their judgments converge with or diverge from those of humans. Across three studies and six widely used LLMs, we addressed this gap by identifying the standards underlying LLM creativity evaluation and examining their downstream implications. Study 1 showed that LLMs generally relied on a narrower subset of human creativity evaluation standards. Convergence with human standards was strongest in the novelty dimension, whereas divergence was clearest in the contextual dimension, which captures social,
market, and reputational information. Moreover, each LLM exhibited distinct, model-specific standards that varied substantially in breadth. These differences in evaluation standards were reflected in actual creativity judgments. Study 2 (N = 1,103 ideas) showed that LLM evaluations were moderately correlated with human evaluations, and individual LLMs with broader standards better distinguished ideas humans judged as more versus less creative. Study 3 (N = 1,195 participants) showed that LLMs were less sensitive to contextual information: such information significantly altered human creativity ratings but left LLM ratings largely unchanged. Together, our findings help explain the mixed evidence on LLM-human
alignment, showing that
alignment depends on the evidence a judgment demands and the standards each model applies. Selecting an LLM evaluator is therefore a consequential decision: different models, applying different standards, recognise different ideas as creative.
Read the full abstract ▾
All authors Pengzhao Lyu, Yeun Joon Kim, Hanlin Xiao, Yingyue Luna Luan
Computation and Language
From Isolated Tasks to Structured Capabilities: A Multilayer Taxonomy for Large Language Models
Shixin Fang, Jiachen Wo, Wenjuan Qin et al. (+2 more)
Large
language model (LLM) evaluation spans diverse tasks and
benchmarks, yet evidence remains organized around tasks rather than the capabilities they probe. This fragmentation limits cross-study comparison, obscures capabilities tasks recruit, and makes coverage gaps difficult to identify. We introduce a multi-layer taxonomy of 14 capability domains and 91 subskills across Primitive, Constructed, and Integrative layers. Human cognitive science guides capability definition and organization, not LLM architecture. Layer assignments draw on developmental precedence and hypothesized functional support, while human-origin constructs are adapted to observable model behavior. To demonstrate operational utility, we screened 31,505 papers from ACL, AAAI, ICML, and NeurIPS between 2023 and 2025 and mapped 15,934 LLM-focused papers through multi-model annotation, consensus, and arbitration. Direct research
attention concentrated on
Language-Semantic Competence (3,551; 22.3%),
Reasoning (3,388; 21.3%), Planning and Decision-Making (2,149; 13.5%), and Perception (1,954; 12.3%), whereas six domains appeared in fewer than 2% of papers. Within domains, the most frequent subskill had a median prevalence of 97.9% and appeared in at least 90% of papers in 10 of 14 domains.
Language-Semantic Competence and
Reasoning formed the highest-volume pair (n = 1,864; 11.7%; lift = 2.47), whereas Theory of Mind and Social
Reasoning and Interaction showed the highest lift among pairs with at least 20 co-occurrences (n = 62; lift = 30.84). By shifting the unit of analysis from isolated tasks to structured capabilities, the taxonomy supports research organization, coverage audits, evaluation interpretation, and testable hypotheses for diagnosis, training, and transfer.
Read the full abstract ▾
All authors Shixin Fang, Jiachen Wo, Wenjuan Qin, Sihang Jiang, Yanghua Xiao
Computation and Language · Cross-listed from cs.DB
DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents
Junming Chen, Junyang Jiang, Xu Chen et al. (+2 more)
LLM-based database
agents show promise, but differences in task scope, testbeds, and success criteria hinder direct comparison. A shared
benchmark must also preserve the operational conditions that determine whether diagnosis and remediation succeed in practice. We present DBA-Bench, guided by operational realism, outcome-first evaluation, and controlled scenario reproducibility. It evaluates
agents in live PostgreSQL and distributed TiDB deployments with active workloads, stateful behavior, and multi-source observations. Success requires satisfying scenario-specific operational objectives without violating safety constraints. The primary evaluation covers 231 bounded scenarios across five operational domains, while 10 persistent scenarios extend evaluation to continued operation across interacting events. We evaluate eight automated baselines, including four frontier foundation models, two open-weight models, and two GPT-5.5-backed database
agents. Our evaluation exposes gaps between diagnosis, effective remediation, and safe execution across both database systems.
Agents can identify root causes yet fail to restore operational objectives. Even when objectives are restored, the executed actions can violate safety constraints. Persistent scenarios expose a gap between clearing individual blockers and sustaining business service:
agents leave competing workloads uncoordinated and previously completed repairs unchecked after subsequent changes.
Read the full abstract ▾
All authors Junming Chen, Junyang Jiang, Xu Chen, Zibo Liang, Kai Zheng
Computation and Language
Dynamic Commonsense Coordination for Empathetic Response Generation
Zhengyu Qi
Empathetic Response Generation (ERG) requires models to recognize users'
emotions and generate empathetic responses. Commonsense knowledge has been shown to support such
reasoning, yet existing approaches typically reuse fixed commonsense representations across understanding and generation, limiting their ability to coordinate such knowledge across different stages. We propose DCC, a Dynamic Commonsense Coordination Framework with three complementary modules: residual-based commonsense interaction (SCE-AttnRes) to integrate contextual and situational commonsense representations, Association-Guided Commonsense Filtering (AGCF) to down-weight low-relevance commonsense relations, and Iterative Commonsense-Aware Decoding (ICAD) to dynamically retrieve commonsense
memories during generation. Experiments on the Empathetic-Dialogues
benchmark show that DCC improves
emotion classification accuracy and response diversity over the CEM baseline while maintaining comparable perplexity. An LLM-based blind evaluation further demonstrates that DCC generates responses with better relevance, coherence, and informativeness. The code and implementation details will be publicly available at https://github.com/Hanabi-Q/DCC-ERG.
Read the full abstract ▾
Computation and Language
MEUSLI: a Multilingual Projector for LLM-based ASR and Beyond
Lorenzo Concina, Seraphina Fong, Marco Matassoni et al. (+1 more)
Lightweight projectors are an established way to connect pre-trained
speech encoders with large
language models (LLMs), mapping acoustic features into
token-level
embeddings for tasks like ASR and spoken question answering. Existing systems, however, typically only support a few
languages and are often limited to English. We introduce MEUSLI, the first open-science
multilingual projector family that links a Whisper encoder with open-source
multilingual LLMs, enabling fully open-source end-to-end ASR in 28 European
languages. MEUSLI extends prior monolingual pipelines, delivering strong results across high- and low-resource
languages. Using proper continual leaning techniques, MEUSLI can be easily extended to other
languages not seen in training. We further demonstrate that the MEUSLI projector can be leveraged beyond ASR, enabling
multilingual speech translation and topic identification with only a few hours of task specific supervision per
language. Overall, MEUSLI provides a solid foundation for
multilingual speech understanding tasks, supporting scalable and inclu- sive open-source SpeechLLM
Read the full abstract ▾
All authors Lorenzo Concina, Seraphina Fong, Marco Matassoni, Alessio Brutti
Computation and Language · Cross-listed from cs.AI
Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model
Nanbeige Lab: Chen Yang, Chengrui Huang, Fufeng Lan et al. (+21 more)
We present Nanbeige4.2-3B, a compact general agentic model with 3B non-
embedding parameters. It delivers strong performance across code-
agent, office-
agent, and complex tool-use tasks while maintaining highly competitive
reasoning capabilities in mathematics, coding, and science. Nanbeige4.2-3B is pretrained from scratch on 28T
tokens with a Looped
Transformer that reuses the layer stack to increase capacity without adding parameters. For SFT data and trajectory construction, we expand the diversity of executable environments, task assets, and agentic scaffolds through real-world deployment and large-scale synthesis. Our RL pipeline applies mixed-mode RLHF over Think and Non-Think responses to improve overall model quality and reduce failure cases, length-controlled
reasoning RL to balance accuracy and
reasoning efficiency, and agentic RL with outcome and process rewards to stabilize long-horizon training. Extensive evaluations show that Nanbeige4.2-3B outperforms larger models, including Qwen3.5-9B and Gemma4-12B, across diverse agentic
benchmarks while remaining competitive on
reasoning and
alignment tasks. Performance with OpenClaw further supports its use as a compact local personal assistant.
Read the full abstract ▾
All authors Nanbeige Lab: Chen Yang, Chengrui Huang, Fufeng Lan, Hanhui Chen, Hao Zhou, Huatong Song, Jiaqi Cao, Jiaying Zhu, Jinlin Niu, Kai Wang, Lisheng Huang, Qiliang Liang, Ran Le, Ruixiang Feng, Shuang Sun, Tao Gu, Tao Zhang, Tianyu Luo, Yang Song, Yun Xing, Yuntao Wen, Ziyao Xu, Zongchao Chen, Zongqiang Li
Computation and Language
FSE: Continual Learning for Named Entity Recognition by Fast-Slow Experts
Yunan Zhang, Yang Fan, Heng Li et al. (+2 more)
Continual Learning for Named Entity Recognition (CLNER) enable models to incrementally learn new entity types without forgetting previously acquired ones. However, existing methods suffer from catastrophic forgetting and insufficient exploitation of shared information across tasks. This paper proposes FSE, a Fast-Slow Experts enhanced span-based NER model for CLNER. The shared fast expert learns
token-level links to efficiently filter out unlikely spans, while the task-specific slow expert performs span classification only on the remaining candidates. It stabilizes learning by promoting knowledge sharing across tasks and maintains plasticity by reducing learning burden at each task. A length-decay negative sampling strategy to mitigate span imbalance is also introduced. Extensive experiments on OntoNotes and FewNERD synthestic
datasets demonstrate that FSE achieves state-of-the-art performance in CLNER scenarios, with effectiveness of each component, empirical evidence of faster convergence and expected functionality of both experts.
Read the full abstract ▾
All authors Yunan Zhang, Yang Fan, Heng Li, Xiangping Wu, Qingcai Chen
Computation and Language
Multimodal Language Models Benchmarked Against the NRC Reactor Operator Licensing Examination: Fine-Tuning and Retrieval Strategies
Isak Hwang, Yoon Pyo Lee and Syed Bahauddin Alam
Competence claims for a
language model in a safety-critical domain are credible when measured against a standard the domain already enforces. We evaluate an open-weight 31-billion-parameter
multimodal model (Gemma 4 31B-IT) on the U.S.
Nuclear Regulatory Commission Reactor Operator Generic Fundamentals Examination (GFE), scoring it paper by paper against the 80% criterion applied to every human candidate, with no rounding up. The evaluation set is a census of every GFE administered at the March sitting from 2015 to 2021, giving seven pressurized water reactor (PWR) and seven boiling water reactor (BWR) papers and 697 scored items. Eight configurations cross three model states, the base model, supervised fine-tuning (SFT) on distilled chain-of-thought rationales and
retrieval-augmented fine-tuning (RAFT), with three
retrieval conditions, none and BM25
retrieval over the Department of Energy Fundamentals Handbooks under fixed-size and structure-aware chunking. Out of the box it answers 51.94% correctly and passes no paper. SFT with fixed-size chunking
retrieval passes 8 of 14, reaching 80.23% on PWR items and 79.77% pooled, with a Wilson interval spanning the threshold. The preferred chunking granularity reverses with training state, structure-aware before fine-tuning and fixed-size after, so chunking optimized against a base model cannot be inherited by its fine-tuned descendant. RAFT trails SFT by 2.2 to 2.3 percentage points overall, and the deficit holds in all four reactor-type and chunking strata. The pipeline runs on one workstation with no network access at run time, and the result approaches operator-level command of engineering fundamentals without reliably achieving it.
Read the full abstract ▾
Computation and Language
Scaling Native Multimodal Pre-Training From Scratch
Haoyuan Wu, Aoqi Wu, Hai Wang et al. (+3 more)
Although large
language models (LLMs) exhibit remarkable
reasoning capabilities, their reliance on text-only pre-training restricts the perception of the
multimodal physical world. Native
multimodal pre-training avoids this limitation by training models from scratch on
multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain incompletely characterized. To address this gap, we investigate the optimal model size and
token count for training a
Transformer-based
vision-
language model under a fixed computational budget. Our study demonstrates that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and
token counts scale as power laws. Notably,
language and
multimodal objectives manifest distinct allocation trends. The
language allocation exponents lie in a similar range across the different data mixtures. The
multimodal model-allocation exponent decreases modestly with the
multimodal token ratio, indicating a relative shift toward
token allocation. Additionally, our scaling analysis yields a budget-compensation rule. Specifically, an additional
multimodal-
token budget can offset the text-efficiency penalty caused by incorporating visual information into native
multimodal pre-training under a fixed compute budget. Downstream evaluations further reveal that native
multimodal pre-training is associated with improved spatial
reasoning and
multimodal few-shot learning. Generally, this empirical research establishes the essential groundwork for predictably scaling
multimodal foundation models.
Read the full abstract ▾
All authors Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu, Jinxiang Ou, Bei Yu
Computation and Language
Developing and Validating the Spanish Version of the Large Language Models Dependency Scale (LLM-D12-SP)
Tran Gia Bao, Mo El-Haj, Sameha Al-Shakhsi et al. (+3 more)
There is a growing need for reliable and culturally validated instruments to assess psychological dependency on large
language models (LLMs), particularly as LLMs are increasingly used for task execution, decision-making, and communication in organizational and work-related settings. This need is especially relevant for Spanish-speaking populations, where LLM adoption is rapidly expanding, yet validated psychometric tools remain scarce. The present study reports the first validation of the Spanish version of the Large
Language Model Dependency Scale (LLM-D12-SP), extending prior validations conducted in English- and Arabic-speaking samples. The LLM-D12 is a two-dimensional instrument assessing Instrumental Dependency (reliance on LLMs for performing tasks and supporting decisions) and Relationship Dependency (psychological reliance on LLMs for companionship and social interaction). A total of 386 Spanish-speaking participants (M = 28.0 years, SD = 6.1; 55% male) completed the LLM-D12-SP. Confirmatory factor analysis supported the original two-factor structure. The scale demonstrated good internal consistency (Cronbach's alpha = 0.89 total; 0.86 Instrumental; 0.85 Relationship). Discriminant validity analyses indicated that the two subscales represent related but distinct constructs. External validation showed that both dependency dimensions were positively associated with internet addiction and perceived trustworthiness of LLMs, while showing weak or no association with need for cognition. Together with prior English and Arabic validations, these findings establish cross-linguistic support for the scale's structure and provide a psychometrically sound tool for investigating psychological aspects of LLM use in organizational contexts.
Read the full abstract ▾
All authors Tran Gia Bao, Mo El-Haj, Sameha Al-Shakhsi, Antonio Garcia-Cabot, Raian Ali, Ala Yankouskaya
Computation and Language
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
Zixuan Ren, Jinliang Lu, Junhong Wu et al. (+5 more)
Model merging plays a crucial role in consolidating multiple specialized models into a single, unified model, especially in the era of large
language models (LLMs). Recent research has primarily focused on developing strategies to enhance merging performance with the trained models, while the impact of training paradigms, such as supervised fine-tuning (SFT) and reinforcement learning (RL), on the effectiveness of model merging remains underexplored. In this study, we systematically explore the merging behavior of RL-trained LLMs compared to those trained with traditional SFT. Through comprehensive evaluations across five representative tasks, we find that RL significantly reduces task conflicts and results in less performance degradation after merging, making RL-trained models particularly well-suited for this process. To unearth the reasons behind the superior suitability of RL for model merging, we conduct extensive empirical experiments and theoretical analyses. Our findings highlight three key factors: (1) On-policy training data in RL control the gradient updates in a smaller magnitude, reducing the risk of overwriting existing knowledge for other tasks in the model. (2) The RL optimization objective, which favors “enough is as good as a feast", progressively reduces the magnitude and the number of conflict parameter updates as the model converges. (3) Joint optimization of positive and negative examples in RL steers the model towards an unbiased task-specific parameter subspace, ensuring robust performance while further preventing parameter conflicts.
Read the full abstract ▾
All authors Zixuan Ren, Jinliang Lu, Junhong Wu, Yang Zhao, Dai Dai, Hua Wu, Haifeng Wang, Chengqing Zong
Computation and Language · Cross-listed from cs.CV
Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
M M Asif Ferdous
Vision-
language models (VLMs) are increasingly deployed on consumer hardware where input images are degraded by compression, camera shake, and poor lighting. In such settings, a reliable uncertainty signal matters more than raw accuracy, because it determines when a system should defer rather than answer. We evaluate two small open-weight VLMs – Qwen2-VL-2B-Instruct and SmolVLM-Instruct – across six realistic photographic degradations at three severity levels, comparing two confidence signals: the confidence the model states in natural
language, and the model's own mean
token probability over its generated answer. Across 3,800 predictions, we find a large and consistent gap. Verbalized confidence in Qwen2-VL is almost constant (mean 0.87-0.90 across all conditions) and detects its own errors at chance level (AUROC 0.39-0.75, typically ~0.50), while internal
token probability from the same model separates correct from incorrect answers with AUROC 0.92-0.99. In SmolVLM, verbalized confidence proved largely unobtainable: across three
prompt templates, only one of five pilot attempts produced a parseable confidence value, while internal probability again yielded above-chance error detection (AUROC 0.54-0.92). Both models fail in the same place: under severe underexposure, accuracy collapses (0.99->0.22 for Qwen2-VL, 0.97->0.42 for SmolVLM) while both confidence signals barely move, and internal error-detection falls to chance. We conclude that small VLMs encode usable self-knowledge that their verbalized output does not express, that internal probability is therefore the better deferral signal in constrained deployment, and that neither signal should be trusted under severe low-light conditions.
Read the full abstract ▾
Computation and Language
DWT-Fusion: A Signal-Based Framework for Training-Free LLM-Generated Text Detection
Mehmet Batuhan Özdaş and Murat Osmanoğlu
Detecting LLM-generated text remains challenging under zero-shot and training-free conditions, especially when detectors must generalize across
datasets, domains, and unseen generators. While existing training-free approaches exploit
language-model statistics as detection signals, they typically characterize a text through global measures that summarize overall model behavior. Consequently, potentially informative local and multiscale variations in
token-level predictability may remain underutilized. Motivated by this observation, we introduce DWT-Fusion, a training-free signal-based framework for detecting LLM-generated text using discrete wavelet analysis of
token-level log-probability sequences produced by a proxy causal
language model. The proposed framework analyzes these sequences through wavelet-based multiresolution signal representations and derives detection signals from localized probability dynamics. We further evaluate four training-free voting variants, including equal-weight hard voting, equal-weight soft voting, calibration-weighted hard voting, and calibration-weighted soft voting, to combine multiple wavelet configurations without training a supervised meta-classifier. We evaluate the framework on HC3, M4, and MAGE using GPT-Neo-2.7B, GPT-J-6B, Falcon-7B, and LLaMA-3-8B as proxy models. The best single wavelet configurations achieve AUROC values of 0.9872, 0.8185, and 0.7138 on HC3, M4, and MAGE, respectively. With calibration-weighted voting, the best ensemble variants further improve AUROC to 0.9919, 0.8477, and 0.7471. These findings show that DWT-based multiresolution scoring and calibration-guided voting fusion provide effective and interpretable signals for training-free LLM-generated text detection.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents
Suman Navaratnarajah, Taehyoung Kim, Jona Ruthardt et al. (+5 more)
Multimodal Large
Language Models (MLLMs) are emerging as core
reasoning modules for
embodied agents, yet it remains unclear how well general-purpose models can solve long-horizon
embodied tasks from a single high-level instruction. We introduce MissionBench, a
benchmark for mission-level evaluation of MLLMs in aerial 3D environments. It comprises 120 missions across five simulated 3D environments and four task families.
Agents must autonomously plan, navigate, and report outcomes using only
egocentric observations and its action history, without aerial-specific fine-tuning. Across 22 open- and closed-source MLLMs, the strongest model succeeds on fewer than 35% of missions compared to 84.4% human performance, highlighting the difficulty of multi-step
embodied tasks. Despite large variations between model families, we observe gains from scaling, indicating that larger general-purpose models possess stronger zero-shot
embodied capabilities. Our analysis shows that mission-level competence requires coordinating multiple capabilities beyond spatial perception, including multi-step planning and adaptive
reasoning. This motivates closed-loop evaluation and highlights both the promise and risk of scaling-driven improvements for
embodied AI.
Read the full abstract ▾
All authors Suman Navaratnarajah, Taehyoung Kim, Jona Ruthardt, Ishaan Bhimwal, Ryousuke Yamada, Yannik Blei, Wolfram Burgard, Yuki M Asano