Computation and Language · Cross-listed from cs.SD
Agogic: Performance-Timed Music Tokens for LLM-Native Text-to-Symbolic-Music Generation
Junhao Chen, Mingjin Chen, Jingjia Mao et al. (+12 more)
Text-to-music
language models begin with a choice usually made by default: how to tokenize music. Normally entangled with backbone, data, and recipe, its effect has never been measured in isolation. We fix pretrained Qwen3.5 (0.8B-27B), data, budget, and decoding, and swap only the representation across seven tokenizations, anchoring texture metrics to each representation's model-free ceiling. The ordering is clean and surprising: representation, not model size, is the binding variable for distributional fidelity. Scaling the backbone 34x barely moves Frechet Music Distance (FMD), whereas switching representation halves it. PMT, a performance-resolution stream we release (10 ms timing, per-note velocity, multi-track texture; 609 symbols), reaches FMD 159 at 0.8B against 272-286 for beat grids (1.7-1.8x lower, up to 2.8x elsewhere; non-overlapping bootstrap CIs), so a 0.8B performance-resolution model beats a 27B beat grid. It reappears on a 26M from-scratch backbone and a second performance-resolution tokenizer: a property of the class, not one lucky vocabulary. Nor is it a finer-
lattice artifact: snapping PMT's onsets to the beat grids' resolution still leaves it 67-129 FMD ahead of both (n=500). The effect is distributional; whether it is audible is a separate question, left open by our probe, with a human study pre-registered. Native caption adherence is weak but separable: a lightweight decode-time constraint doubles instrument-F1 (.28 to .60) and Correct-Key (.16 to .35) at no distributional cost. We release the harness, 25+ checkpoints, two corpora (86.6k aligned across caption/MIDI/ABC/audio; 6.25M captioned, the largest for music), and an imprinting diagnostic: published text-to-MIDI systems reproduce their training distribution near-invariant to the caption (72% vs. 71% chord-time on disjoint domains). The field's next representation claim can now be measured, not asserted.
Read the full abstract ▾
All authors Junhao Chen, Mingjin Chen, Jingjia Mao, Lin Chen, Saining Zhang, Minglin Chen, Ruocheng Wu, Liaoyuan Fan, Wenyi Li, Mingju Gao, Henghaofan Zhang, Zhihao Li, Hao Zhao, Yufei Wang, Ruqi Huang
Computation and Language
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt et al. (+2 more)
We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of
attention weights and renders the affected
attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair
token retrieval while having only a minor effect on standard decoder
benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey
retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack
retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.
Read the full abstract ▾
All authors Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt, Martin Potthast, Gerhard Heyer
Computation and Language
string2string Studio: An Interactive, In-Browser Platform for String-to-String Algorithms
Mirac Suzgun, James Zou, Stuart M. Shieber et al. (+1 more)
We present string2string Studio, an interactive in-browser platform for string-to-string analysis across natural
language processing, computational biology, and the digital humanities. The system integrates six main modules (
alignment, distance, similarity, search, generation metrics, and BLAST homology search), operating at character, word,
token, line, and residue levels. Its C++-based
algorithms compile to WebAssembly, so core operations run locally by default without any installation or data upload. The interface reports scores with their "evidence" (
alignments, edit paths, metric matches, search hits, and homology traces), making methods inspectable, debuggable, and comparable on shared inputs. Internal
benchmarks show speedups of up to 2,500x over the Python predecessor, faster global/local
alignment than a general-purpose native C aligner, and exact agreement with independent references under declared settings. For homology search, the scoped client-side blastn path closely matches NCBI BLAST+ rankings and statistics under matched parameters. A curated showcase and Learn mode present canonical
algorithms and metrics as reusable demonstrations. string2string Studio is open-source and freely available at string2string.org.
Read the full abstract ▾
All authors Mirac Suzgun, James Zou, Stuart M. Shieber, Dan Jurafsky
Computation and Language
HalluTruthQA-4K: A Fine-Grained Corpus and Annotation Process for Arabic Hallucination Detection and Truth Verification
Salah Eddine Bekhouche, Abdessalam Bouchekif, Hichem Telli et al. (+2 more)
Large
language models can generate fluent Arabic answers while introducing factual errors that are difficult to identify and verify. Existing Arabic
hallucination resources often assign a binary label to an entire response, indicating whether it is hallucinated or non-hallucinated, but provide limited information about the exact erroneous content, the reason for the error, or the correct factual answer. We present HalluTruthQA-4K, an expanded version of the HalluTruthQA resource containing 4,000 expert-curated Arabic question-answering instances across four knowledge-intensive domains: Islamic knowledge, history, science, and geography. Serving as the official
dataset for Track 2 of the HalluScoring 2026 shared task, HalluTruthQA-4K extends our original corpus to 4,000 instances. Each instance pairs an Arabic question with a model-generated response, a verified reference answer, and five plausible distractors. Hallucinated responses are additionally annotated with character-level erroneous spans, human-written explanations, and hierarchical
hallucination types. The corpus contains 1,643 hallucinated and 2,357 non-hallucinated responses, with 1,843 annotated erroneous spans. We describe the resource construction and annotation methodology, including question selection, controlled answer generation, candidate construction, expert annotation, independent verification, adjudication, and quality control. We also document the annotation guidelines, taxonomy, data format, inter-annotator agreement, and corpus statistics. HalluTruthQA-4K provides a reusable resource for
hallucination detection, span-level error localization, explanation generation, factual verification, and the broader evaluation of factual reliability in Arabic
language models.
Read the full abstract ▾
All authors Salah Eddine Bekhouche, Abdessalam Bouchekif, Hichem Telli, Mohammed-En-Nadhir Zighem, Abdenour Hadid
Computation and Language
Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility
Jo-Ku Cheng, Nikolaos Aletras and Marco Valentino
Pre-pretraining
language models (LMs) on symbolic data can accelerate and improve natural
language acquisition. However, existing pre-pretraining tasks, such as Dyck and procedural
algorithms, rely on narrow primitives that fail to capture the expressive capacity of natural
language. Moreover, prior studies remain restricted to relatively small
token budgets, offering limited insight into skill emergence and representational dynamics. To address these limitations, we propose logic pre-pretraining (Logic-PPT) as a principled initialization strategy, leveraging formal derivations to impart richer structural and linguistic biases. Formal derivations require abstract mechanisms that are central to natural
language, simultaneously binding variables, connecting quantifiers and relational dependencies, and composing predicate-argument structures over long contexts. Scaling our evaluation to a 100B-
token regime, logic pre-pretraining substantially accelerates skill acquisition in LMs, achieving 80% accuracy on linguistic tasks with 36B fewer
tokens than standard initialization, and outperforming alternative pre-pretraining baselines. Mechanistically, formal derivations induce persistent structural reorganization, distinctively characterized by a lower-rank, spectrally concentrated representation space. Crucially, we show that this internal geometry enables improved model compressibility via
pruning, matching the dense baseline performance even at $\approx$33% sparsity.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Sparse Weight Decomposition for Efficient Circuit Extraction
Chuanhao Yan, Xuhan Huang, Yawen Duan et al. (+4 more)
Dense pretrained
transformers do not naturally expose interpretable units for
circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations or training sparse models, incurring substantial additional computation while potentially introducing a fidelity gap between the representation being analyzed and the original pretrained model. We propose Sparse Weight Decomposition (SWD), which reparameterizes pretrained linear projections by factorizing each weight
matrix into two sparse factors whose shared intermediate coordinates serve as individually addressable
circuit units. Without training a separate replacement network, this parametric representation supports the same scoring, selection, and ablation
circuit extraction workflow used for methods that learn sparse features. Across single-
matrix replacements, SWD matches the held-out fidelity achieved by Transcoder and other strong baselines while using less than 1% of the data that those baselines use to train their replacements. For matched replacement fidelity, SWD reaches the same
circuit sufficiency and necessity targets with fewer active read/write edges and selected units across tasks on GPT-2, Qwen2.5, and Qwen3.5-27B. We further show that SWD remains effective for full-model replacement of all
attention and MLP weight
matrices after fine-tuning the nonzero factor values. Finally, SWD also features a zero-data variant, allowing broader use of mechanistic interpretability analysis (e.g., per-step analysis).
Read the full abstract ▾
All authors Chuanhao Yan, Xuhan Huang, Yawen Duan, Zhenfei Yin, Hang Zhao, Bryan Dai, Jie Fu
Computation and Language
ANNOTARES: A Dataset for Extracting Logical Structures from German Statutory Texts
Ronja Schwarz and Jannik Strötgen
The automatic structural analysis of legal texts is a cornerstone of legal technology, yet the extraction of their logical components remains a significant challenge. In this paper, we introduce the task of identifying and segmenting legal conditions (Tatbestand) and legal consequences (Rechtsfolge) within German statutory texts. To support this task, we present ANNOTARES (Annotations of Tatbestand-Rechtsfolge Sequences), a novel
dataset comprising German law texts with span-level annotations. Spanning three distinct legal codes, the
dataset is designed to evaluate both domain-specific performance and cross-statute generalizability. We
benchmark diverse architectural approaches: a rule-based baseline, CRFs, BiLSTMs, BiLSTM-CRF, and modern
Transformer-based models, including BERT variants and LLM-based methods. Our results demonstrate that BERT and LLM-based models achieve superior performance in capturing the complex syntactic structures of legal
language. We release our
dataset to facilitate further research in automated legal
reasoning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models
Sadab Shiper, Tawsif Tashwar Dipto, Mir Md Inzamam et al. (+1 more)
In-the-wild Bengali scene text recognition is largely unmeasured: existing resources target handwritten documents or constrained sign-board parsing, report only aggregate edit-distance metrics, and evaluate either conventional OCR or VLMs, never both on the same in-the-wild data. To address this gap, we introduce BANGLAWILD, a
benchmark of 2,535 Bengali scene text images, each paired with a verbatim gold transcription, two categorical axes, four diagnostic attributes, and an orthographically standard form where the in-image text deviates from canonical spelling. We evaluate fifteen VLMs and three conventional OCR systems under three prompting strategies, fine-tune 6 open-source models with LoRA, and complement edit-distance metrics with an LLM-as-a-Judge evaluation. Our results reveal a persistent gap in which larger models within the same family do not outperform smaller ones. Our fifteen-class error taxonomy shows that visual mis-recognition accounts for ~60% of errors in the strongest systems, while conjunct-related errors contribute under 2%, challenging a long-standing assumption in Bengali OCR research; the same visual dominant profile also holds across architectures, including the one conventional baseline that reads Bengali reliably.
Prompt language mainly affects cross-script drift and LoRA reduces catastrophic failures in weak models without lifting the ceiling on already competent ones. Code and data will be publicly released.
Read the full abstract ▾
All authors Sadab Shiper, Tawsif Tashwar Dipto, Mir Md Inzamam, Eshat Tanzeem
Computation and Language
DS@GT-ARC at eRisk 2026 Task 3: Sparse, Semantic, and LLM Reranking for ADHD Symptom Sentences
David Guecha
This paper describes our submissions to eRisk 2026 Task 3, ADHD Symptom Sentence Ranking. The task requires systems to rank candidate Reddit sentences according to their relevance to each of the 18 symptoms in the Adult ADHD Self-Report Scale (ASRS-v1.1). Because no annotated training data were released for this first edition of the task, we relied on zero-shot experimentation, manual validation, and unsupervised or weakly guided
retrieval pipelines. Our systems combine sparse BM25
retrieval, evidence-aware rescoring for self-referential symptom reports,
embedding-based reranking, query-prototype expansion, and LLM-based reranking. All submitted systems follow a staged
retrieval design in which BM25 retrieves candidates at scale and semantic or LLM rerankers refine the final rankings. Among our submissions, the LLM reranker achieved the strongest official scores, followed by the prototype query-expansion run. Our manual top-10 analysis aligned with the official expert scoring trend, suggesting that staged reranking is a promising direction for further development.
Read the full abstract ▾
Computation and Language
Martin Böckling, Elizaveta Nosova, Heiko Paulheim et al. (+1 more)
Geospatial
reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to
navigation and logistics, yet large
language models (LLMs) struggle with the required
geometric and topological computation despite storing considerable geographic knowledge. Existing
benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a
multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource
languages. Across parametric,
reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best.
Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.
Read the full abstract ▾
All authors Martin Böckling, Elizaveta Nosova, Heiko Paulheim, Andreea Iana
Computation and Language · Cross-listed from cs.AI
ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?
Tianyi Guan, Yiding Wang, Haotong Yang et al. (+5 more)
Modern
agent frameworks equip large
language models with external skill libraries to solve complex tasks. However, it remains unclear whether these systems can effectively evolve their skills and whether the resulting skills improve task-solving capabilities. To bridge this gap, we introduce ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning. It covers five domains, each with 100 tasks ordered by difficulty and opportunities for skill transfer. Our experiments show that sequential execution generally improves performance, but the gains vary substantially across models and domains. Moreover, across two models and three domains, retaining context and feedback without skill maintenance yields nearly the same mean performance as maintaining an explicit skill library. Explicit skills nevertheless provide selective benefits for tasks requiring reusable procedures or precise outputs. We further find that the weaker model (GPT-4o) accumulates more skills but reuses them less often, suggesting difficulty consolidating task-specific experience into reusable procedures. These findings show that current in-context skill
evolution mechanisms can support continual adaptation, but still struggle to consistently consolidate experience into robust and transferable skills.
Read the full abstract ▾
All authors Tianyi Guan, Yiding Wang, Haotong Yang, Siyuan Cao, Shirui Liu, Yi Hu, Jiaqi Li, Muhan Zhang
Computation and Language
SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG
Kaysarul Anas Apurba, Md. Hasibul Hasan, Rofiqul Alam Shehab et al. (+1 more)
We introduce SciRet, a compute-aware empirical study of
retrieval-augmented generation for scientific question answering over CORD-19. Rather than proposing a new model, we evaluate a fixed scientific RAG pipeline across three corpus scales: 1,034 chunks (1K papers), 5,160 chunks (5K papers), and 15,480 chunks (15K papers). The pipeline combines sentence-window chunking, BM25, BGE-M3 dense
retrieval, reciprocal rank fusion, optional cross-encoder reranking, and grounded answer generation. Across these settings, hybrid
retrieval is more robust than either sparse-only or dense-only
retrieval in our setting, reaching Recall@10 of 1.000 at 1K and 15K. In contrast, an MS MARCO-trained cross-encoder reranker reduces precision on the scientific corpus, suggesting that domain mismatch can outweigh the benefits of stronger query-passage interaction. Generation faithfulness measured with RAGAS increases with corpus scale in our setup.
Retrieval evaluation uses pseudo-relevance labels derived from the hybrid system, so we treat the results as controlled comparative evidence rather than a
benchmark claim. We release code, indexes, and evaluation outputs to support replication and follow-up studies.
Read the full abstract ▾
All authors Kaysarul Anas Apurba, Md. Hasibul Hasan, Rofiqul Alam Shehab, Asab Azad
Computation and Language
Beyond Representational Similarity: Source-Conditioned Description-Length Gain for Generative Plagiarism Detection and Candidate Source Reranking
Peijia Guo, Wenxuan Xie, ZiGuang Li et al. (+1 more)
Large
language models (LLMs) pose challenges to academic integrity and peer review. Yet generative plagiarism detection remains an underexplored and largely unresolved challenge. Prior work on LLM-generated-text detection targets AI involvement, which may be permissible, rather than source reuse, while similarity-based methods struggle after extensive rewriting and multi-source synthesis. Motivated by the description-length view of probabilistic prediction, in which relevant side information can reduce a target sequence's code length, we introduce Source-Conditioned Description-Length Gain (SCDG), a directional, training-free framework that contrasts a frozen
language model's description length of a suspicious document $P$ with and without a candidate source $S$. This contrast yields
token-level log-likelihood gains that measure the incremental predictive evidence supplied by $S$. We evaluate SCDG on the PAN at CLEF
benchmarks for generative plagiarism. On a PAN 2025-derived pairwise
benchmark, SCDG achieves 0.92 Precision, 0.97 Recall, and 0.94 F1, outperforming all baselines; on PAN 2026's multi-source
retrieval task, it reaches 0.83 nDCG@10 and 0.96 Recall@100, surpassing all baselines. On a same-topic, same-event Multi-News test, the calibrated gain-distribution SCDG classifier predicts source reuse for only $0.125\%$ of pairs, supporting robustness to topical overlap under this evaluation protocol. These results establish SCDG as a unified and
token-decomposable signal for source-specific content reuse under extensive transformation.
Read the full abstract ▾
All authors Peijia Guo, Wenxuan Xie, ZiGuang Li, Ming Li
Computation and Language
Sensitivity, Causality, and Repair Dissociate: A Layer-Wise Analysis of Perturbation Robustness and Its Scaling
Nathan Labiosa, David Buff, Ena Nayak et al. (+1 more)
When a
language model fails on surface-perturbed input (typos, OCR noise, homophones), "which layer is responsible" has three natural operationalizations: where representations diverge most (sensitivity), where restoring clean activations recovers the prediction (causality), and where a small adapter can repair the damage (compensatory capacity) - and we show these three layer maps dissociate. Across a five-model panel we identify two propagation regimes - spike-and-suppress (Phi-3.5, Gemma-2-9B) and late-accumulation (Llama-3, Mistral, Qwen2.5-7B) - and on the two models meeting an 80% identity-patch gate, sensitivity and causality are anti-correlated (rho = -0.72 to -0.88). Within-family scaling on Qwen2.5 (1.5B to 14B) shows the late-accumulation signature strengthening monotonically with scale, corroborated on a second family. We propose cascade disruption as the mechanism behind the dissociation: adapters placed at causally implicated early layers break intact downstream computation, making diagnostic-flagged sites the worst adapter placements. A fixed-harness layer sweep across four models (3.8-8B) confirms the core prediction on chain-of-thought GSM8K - the flagged sites are the most damaging adapter windows on every adjudicable model - and is sign-consistent but strongly attenuated on a multiple-choice control, consistent with damage that compounds with generation length. The sweep yields practical guidance: a training-free LRD pre-screen and a default-deepest placement rule, though absolute gains over no-adapter baselines remain small. Finally, apparent gains from a representation-stability loss reverse under an adequate generation budget - truncated chain-of-thought had been scored as empty - a methodological warning for any intervention evaluated on chain-of-thought tasks.
Read the full abstract ▾
All authors Nathan Labiosa, David Buff, Ena Nayak, Erica Donno
Computation and Language
VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs
Andrei Chetvergov, Alexander Evseev, Timofei Sivoraksha et al. (+4 more)
Large
language models routinely describe socially salient targets, including political figures, countries, religions, organizations, historical events, and social groups, encoding affective framing alongside factual content: a target may appear favorable or threatening, calm or conflictual, powerful or vulnerable. Existing work captures parts of this space through sentiment, favorability, and
emotion benchmarks, but none combines target-directed VAD attribution, an explicit scorer contract, and a passport reporting format. We introduce VIBE, a
benchmark for entity-centered affective profiling of LLM outputs in Valence-Arousal-Dominance (VAD) space. Its core contribution is a measurement contract: VIBE separates generation from external scoring, distinguishes scalar favorability, response-level VAD, and target-directed VAD, and reports profiles through an Affective Passport. Three empirical layers support the contract. H1 shows scalar favorability does not subsume arousal and dominance: valence findings are cross-validated (rV = 0.944 judge-human, rV = 0.954 inter-scorer); arousal and dominance are single-scorer directional estimates, not point-precise, consistent with known inter-annotator difficulty on these axes (rA = 0.495, rD = 0.702 among human annotators). H2 shows whole-response and target-directed VAD are different contracts: the same text can carry one affective tone overall while representing the named target differently. H3 is a protocol-drift diagnostic: elicitation conditions shift profiles, motivating context metadata in every affective report. These results motivate entity-centered affective profiling as a documented practice: profiles should be released with scorer identity, coverage, protocol, and interpretation limits.
Read the full abstract ▾
All authors Andrei Chetvergov, Alexander Evseev, Timofei Sivoraksha, Stepan Ukolov, Mikhail Solovev, Danil Sazanakov, Sergey Bolovtsov
Computation and Language
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
Tomáš Burkert, Angelika Peljak-Łapińska and David Zelený
Multilingual language models are deployed across a hundred or more
languages, yet most
benchmarks test whether a model can perform a task _in_ a
language rather than whether it commands the
language itself, conflating fluency with proficiency. We introduce M-GATE (
Multilingual Grammar, Accuracy in Translation, and Efficiency), a
benchmark of linguistic proficiency spanning 30 typologically diverse
languages from high- to low-resource. M-GATE comprises three tasks: grammatical error detection on linguist-crafted, adversarially selected sentences that turn on hard,
language-specific phenomena; round-trip translation of shared English sources across 29 target
languages, scored by a three-provider LLM judge panel validated against professional annotators; and a supplementary tokenizer-efficiency measure. We evaluate over 50 models in more than 80 configurations. Fluency and proficiency come apart sharply: models that translate competently sit near chance on the
adversarial grammar items, the best reaching a Matthews correlation coefficient (MCC) of only 0.36, and their errors lean systematically toward under-flagging, accepting ungrammatical text rather than raising false alarms. Translation quality closely tracks a
language's share of pretraining data (r = 0.86 against log Common Crawl share), producing a steep low-resource penalty that is nonetheless narrowing with successive model releases. Enabling
reasoning reliably improves translation, while its effect on error detection is smaller and for some models negative, so the best configuration is task-dependent. To resist contamination, test items are kept private behind a continuously updated public leaderboard, with illustrative examples released (https://m-gate.ai).
Read the full abstract ▾
Computation and Language
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Bakbergen Ryskulov, Iker García-Ferrero, David Montero et al. (+5 more)
Small
language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge
distillation (KD). This recovery step largely decides the final quality, yet it is expensive. We present a practitioner's study of how to make
distillation training efficient, organised around two systems contributions. First, we show that offline KD (caching the teacher's top-$K$ logits once and training the student against the cache) matches online
distillation at near-identical training loss while removing the teacher from
memory, running about 29% faster per iteration, and reaching up to 41% higher throughput on a single H200 GPU. Second, we introduce a fused, chunked KL loss that never materialises the full vocabulary-sized logit tensor, making peak
memory linear in the sequence length. This removes the
memory spike that otherwise caps context length and lets us train at four times the context (32{,}768
tokens) on a single GPU. A separate output-head-only toy
benchmark isolates the loss kernel and confirms its
memory and iteration-rate scaling from 4K to 256K
tokens. Together these make large-scale healing and hundreds of ablations affordable. We also report supporting ablations on loss design and sequence packing. We release our chunked-loss implementation: https://github.com/CompactifAI/Full-Chunked-KL-Loss.
Read the full abstract ▾
All authors Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Román Orús
Computation and Language · Cross-listed from cs.DB
Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks
Shunfan Zheng, Dongsheng Shi, Yue Li et al. (+3 more)
Large
Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation
benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first
benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural
language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured
reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from “catastrophic forgetting” in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.
Read the full abstract ▾
All authors Shunfan Zheng, Dongsheng Shi, Yue Li, Xin Yi, Linlin Wang, Gerard de Melo
Computation and Language
MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models
Tong Ling, Hang Lei, Feng Xiao et al. (+5 more)
Masked
diffusion language models (MDLMs) enable parallel generation and bidirectional context modeling, but their positional context differs fundamentally from that of autoregressive (AR) models. Whereas AR decoding exposes a contiguous prefix, MDLM denoising produces dynamic, non-contiguous configurations of revealed and masked
tokens. Conventional positional encodings such as RoPE capture sequence order and pairwise displacement but remain insensitive to this evolving
token-availability structure. To address this limitation, we propose MDLMPE, a positional encoding designed specifically for masked
diffusion. To the best of our knowledge, MDLMPE is the first method to make positional representations explicitly aware of the changing revealed/masked configuration. It represents
token availability as a binary sequence, applies distance-aware Gaussian weighting, and projects the resulting pattern through a cosine basis to obtain distribution-aware positional features. These features are added to
token embeddings and mapped by a lightweight MLP to angular offsets that modulate the standard RoPE phases. Extensive experiments on LLaDA and DREAM demonstrate that MDLMPE generally outperforms conventional positional encoding methods across supervised fine-tuning, pretraining, zero-shot evaluation, and block-
diffusion settings. Further ablations show that the complete combination of availability state, Gaussian locality, spectral basis, and
embedding injection yields the strongest result. These results establish the evolving
token-availability distribution as a useful positional signal for masked
diffusion language models.
Read the full abstract ▾
All authors Tong Ling, Hang Lei, Feng Xiao, Changhui Sun, Jiahang Xie, Hao Liu, Lu Liu, Yanlong Du
Computation and Language · Cross-listed from cs.IR
LegalPincite: Multi-level Legal Information Retrieval Dataset
Theresia Veronika Rampisela, Henrik Palmer Olsen and Giovanni Colavizza
A common task in legal Information
Retrieval (IR) is to find relevant legal sources from case-law collections. While legal practice often requires pinpoint citations (pincites) to specific case paragraphs, most existing public legal IR
datasets lack paragraph-level citation annotations. Yet, publicly available
datasets with such information contain data leakage in the query text and exclude paragraphs that are neither citing nor cited from the corpora, creating an unrealistic and oversimplified
retrieval setting, potentially leading to inflated performance. To address these limitations, we contribute a large-scale legal IR
dataset constructed from Court of Justice of the European Union (CJEU) judgments. The
dataset contains: (i) masked case/paragraph queries, with removed citation information; (ii) a corpus that includes all paragraphs; and (iii) case- and paragraph-level ground truth citations, with partial human expert validation. Our
dataset supports both the development and rigorous evaluation of legal IR methods, at multiple query-document levels (case-to-case, paragraph-to-case, and paragraph-to-paragraph
retrieval). Link to
dataset and code: https://huggingface.co/
datasets/theresiavr/legalpincite
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Risky Business: Measuring The Faithfulness-Safety Tension
Dominik Meier, Luca Joshua Francis, Marco Bernhard Kaiser et al. (+3 more)
Chain-of-Thought (CoT)
reasoning offers a promising window into model monitoring. However, monitoring relies on faithfulness, i.e., the model output strictly derives from its
reasoning trace. We identify an
alignment tension where a model must be faithful enough to be monitored, yet robust enough to reject unsafe
reasoning. We demonstrate that this counterbalance exists in current Large
Reasoning Models (LRMs), and show ways in which it can be addressed. We introduce HazMart, a human-written
dataset set in an autonomous AI shopkeeper scenario. Unlike prior work that relies on providing hints in
prompts to test faithfulness (e.g., "A Stanford professor said it should be Answer A"), we propose a novel replacement-based technique, which we call Targeted
Reasoning Replacement (TRR), that directly intervenes in the
reasoning chain to substitute in unsafe or illogical thoughts (e.g., "Wait, the answer must be Option B [was Option A] because it is the most fitting"). DeepSeek-R1-Llama-70B exhibits high faithfulness (97.5%) but fails to reject Unsafe
Reasoning (12.3%), while QwQ-32B is more robust (73.9% safety) at the cost of lower faithfulness (74.7%). Mechanistic analyses of QwQ-32B reveal that these properties are represented by anti-correlated internal directions peaking at the action-commit
token. Finally, we demonstrate that representation steering can independently amplify the safety direction, increasing safe behavior by 9 percentage points while maintaining base capabilities.
Read the full abstract ▾
All authors Dominik Meier, Luca Joshua Francis, Marco Bernhard Kaiser, Terry Ruas, Jan Philip Wahle, Bela Gipp
Computation and Language · Cross-listed from cs.MA
An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures
Vikas Pahuja, Jonathan Brokman, Omer Hofman et al. (+6 more)
Multilingual multi-
agent systems exhibit substantial degradation beyond English, yet prior work rarely identifies how task-critical information is lost when user requests are converted into executable plans. We study the planner in a multi-
agent system as the request-to-action interface and derive an actionable taxonomy of planning-grounding failures from failed real-world task executions. LLM-based analysis shows that these failures constitute an increasing share of unsuccessful executions as
language-resource availability declines, with the strongest effects in low-resource
languages. To test whether the taxonomy supports mitigation, we introduce TART, Taxonomy-Guided Actionable Representation, that makes the taxonomy's key aspects explicit to the planner and downstream sub-
agents. Across multiple
languages, three LLM backbones, two
datasets, and two agentic configurations, TART consistently improves performance. On
multilingual GAIA, it raises a state-of-the-art system's accuracy by 5.6 percentage points averaged across eleven
languages spanning low- to high-resource settings.
Read the full abstract ▾
All authors Vikas Pahuja, Jonathan Brokman, Omer Hofman, Tamir Nizri, Daniel Vishna, Seraphina Goldfarb-Tarrant, Kelly Marchisio, Hisashi Kojima, Roman Vainshtein
Computation and Language
Constructing Disambiguated Knowledge Bases from Large Language Models at Scale
Yujia Hu, Tuan-Phong Nguyen and Simon Razniewski
Automated Knowledge Base Construction (AKBC) is a core NLP task, and recent work proposes generating knowledge bases directly from large
language models (LLMs), treating the model itself as the knowledge source. However, LLMs natively possess no representation of entities, leading to duplicate entries as well as conflations. We propose GPTKB 2.0, a methodology for constructing disambiguated KBs directly from LLMs. GPTKB 2.0 incorporates on-the-fly disambiguation of entities, relations and classes, and is meticulously designed to satisfy both scalability and disambiguation accuracy. We analyze the central design decisions and characterize the trade-offs between accuracy, scale, and cost. We execute GPTKB 2.0 at scale, obtaining a materialized KB containing over 1M disambiguated entities and 38.4M triples. This represents the first million-scale LLM-native KB with explicit internal canonicalization of entities, relations, and classes, a significant departure from prior Wikimedia-centric works. GPTKB 2.0 is available at https://gptkb.org/.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
When Outputs Disperse, Does Epistemic Revision Follow? A Black-Box Diagnostic for Machine Collectives
Molood Arman
Collective intelligence research treats disagreement as evidence of epistemic diversity: if
agents express different views, the group should retain capacity to revise. In LLM collectives this proxy can break:
agents can produce diverse-looking arguments while preserving the same conclusion. We operationalize dispersion-revision coupling: the degree to which an intervention that verifiably increases the dispersion of a collective's outputs in
embedding space is accompanied by genuine revision of its epistemic stance rather than premise-preserving reformulation. The diagnostic is black-box: it operates on generated text alone and makes no claims about the internal representations of the generating models. Two channels are measured independently: an output channel, the Coherence Index (CI), verifies that the intervention changed output dispersion; an epistemic channel, per-turn stance annotation, measures whether the collective revised. We propose CI with the Meta-Predictive Clarity System (MPCS), which inserts a Re-Differentiation Protocol (RDP) when outputs over-converge, as a reusable method for estimating this coupling regime. We evaluate five-
agent collectives from two configurations (gpt-4o-mini and gemini-2.5-flash; 310 paired episodes per condition). On gpt-4o-mini, conditional dissent improves false-premise recovery by +17.7 points (p<1e-6) while static persona diversity harms recovery (-8.1, p=.007). On gemini-2.5-flash, the same intervention at a comparable budget yields no gain (26.1% vs 27.1%, p=.84) despite a verified dispersion drop; the two treatment effects differ from each other (z=3.79, p<.001). Mechanism tagging shows Gemini preserves the false premise via intra-framework dissent: 94% of tagged post-RDP responses reformulate rather than concede (vs 24% on GPT). We recommend reporting per-intervention stance shift and premise-preservation rate alongside accuracy.
Read the full abstract ▾
Computation and Language
Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering
Khaled Ziani
Large
language models can generate fluent responses to Islamic questions while introducing factual errors that are difficult to identify. This paper presents our system for HalluScoring 2026 Task 2.1, Islamic
Hallucination Detection and Find the Truth. The task requires a unified two-step prediction: determining whether an Arabic answer generated by an LLM is hallucinated and selecting the verified answer from six closely related candidate options. We use the Islamic knowledge
dataset provided by the shared task, which contains 600 question–answer instances, including 341 hallucinated and 259 non-hallucinated answers. Our system is based on the fine-tuned google/gemma-4-12B-it model and uses deterministic decoding during inference. The generated outputs are normalized to extract the
hallucination label and the selected option. The system achieves a Macro-F1 score of 0.928 and a label accuracy of 0.935 for
hallucination detection, together with an option accuracy of 0.895 for answer selection. These results yield a combined score of 0.912, demonstrating strong performance across both stages of the task. The lower option-selection accuracy indicates that distinguishing the verified answer from plausible alternatives remains more challenging than detecting hallucinated responses.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Maximilian Dillitzer, Tin Stribor Sohn, Jason J. Corso et al. (+1 more)
In human visual perception, uppercase lettering serves as a natural salience cue that captures
attention within lowercase text. In this paper, we present a systematic empirical characterization study revealing that Large
Language Models (LLMs) exhibit an analogous property: letter casing modulates internal
attention allocation. Through analysis across 13 models, nine LLMs and four
Vision-
Language Models (VLMs), with diverse tokenization schemes, we show that formatting target information in alternating or uppercase against a lowercase context concentrates
attention on those textual spans. In text this effect is universal, holding across every evaluated non-
reasoning model. We frame it as a previously under-explored latent property of pretrained
transformers rather than a prescriptive method. Our investigation reveals a central
attention-performance divergence: while this "casing effect" robustly shifts
attention, its impact on downstream accuracy is non-trivial, increased concentration does not inherently improve task accuracy and, in high-
entropy contexts like alternating case, can degrade it. We further identify a boundary condition: the deliberative "thinking" phase in
reasoning models acts as a semantic buffer that mitigates typographic sensitivity in text. Extending the study to VLMs, we find the effect transfers partially: the same
prompt-side casing reorganizes cross-modal
attention along two coupled axes, predominantly a macroscopic disengagement from the image toward the text
prompt, and secondarily a concentration of the residual visual
attention on the target region. By isolating casing as a zero-shot mechanism for
attention steering that requires no model access or fine-tuning, we provide a new foundational understanding of how pretraining internalizes typographic emphasis.
Read the full abstract ▾
All authors Maximilian Dillitzer, Tin Stribor Sohn, Jason J. Corso, Michael Auerbach
Computation and Language
Predicting Deep Neural Network Training Outcomes from Early Training Telemetry
Ranjita Naik, Anh D. Nguyen and Pankaj Kumar Singh
Large hyperparameter sweeps for deep neural networks spend substantial compute on configurations that are effectively doomed from the first few epochs. We study whether a single training run's own early telemetry - per-epoch loss, training accuracy, gradient signal-to-noise ratio, weight-norm growth, and an activation-saturation snapshot - together with its sampled hyperparameters, can predict that run's eventual outcome without reference to other runs. We evaluate three prediction tasks: final test accuracy, relative performance within a domain, and training-dynamics failure, including numerical divergence. Across 23,788 training runs spanning six architecture/
dataset combinations, gradient-boosted trees using only the first five epochs of telemetry achieve R^2 = 0.92-0.99 for final-accuracy regression and ROC-AUC = 0.983-0.998 for relative classification on a permanently held-out set of hyperparameter configurations. Useful prediction is already available after a single epoch. A paired ablation shows that gradient- and weight-level telemetry provides a statistically consistent improvement over loss and accuracy curves alone, although the practical gain varies by domain. Transfer is strong between similar architectures, while cross-
dataset transfer is limited mainly by differences in accuracy scale rather than loss of the underlying relationship. These results suggest that early-training telemetry can provide a practical decision-support signal for compute allocation while motivating human oversight for any automated intervention.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills
Yongli Xiang, Zhifang Zhang, Bojun Yang et al. (+4 more)
Persona skills distill personal interaction histories into portable and executable artifacts for downstream
agents. While enabling flexible personalization, this process concentrates fragmented personal signals, amplifies their impact through reuse, and challenges defenses designed for individual records or
retrieval-based
memory. To systematically investigate the safety of the persona-skill pipeline, we introduce AntiSkillBench, an end-to-end
benchmark for evaluating risks and defenses across the persona-skill pipeline. It comprises: (i) a
dataset of 7,500 persona-grounded dialogue traces, constructed from 50 behaviorally rich profiles spanning diverse task scenarios; (ii) an evaluation suite that measures skill-level
privacy leakage and
agent-level attribute disclosure and behavioral impersonation across three skill-
distillation strategies; and (iii) a defense evaluation covering four configurations across online and post-hoc interventions, including active risk suppression and passive provenance protection. Experiments across three frontier
agents show that persona-skill risks persist across
agent backbones and
distillation protocols, extending from explicit attributes to communication styles and personality traits. Existing defenses exhibit limited and
distillation-dependent effectiveness, failing to generalize across risk and
distillation strategies. These results highlight AntiSkillBench as a challenging
benchmark for developing
privacy-preserving and authenticity-aware persona skills.
Read the full abstract ▾
All authors Yongli Xiang, Zhifang Zhang, Bojun Yang, Ziming Hong, Lei Feng, Miao Xu, Tongliang Liu
Computation and Language
VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations
Ivan Kartáč, Jan Tovarys, Mateusz Lango et al. (+1 more)
Citation excerpts can be used to increase the reliability of generated outputs and their faithfulness to cited sources, which is especially important in high-stakes domains such as human and veterinary medicine. However, this does not guarantee that generated claims are faithful to the provided excerpts. We present VetScore, a multi-step evaluation method for veterinary long-form question answering, designed to assess how well are generated claims supported by the provided excerpts, weighing this information by each claim's harm potential. VetScore first segments the output and decomposes it into individual claims, then scores each claim with respect to its harm potential and evaluates its faithfulness to source excerpts, and finally calculates the overall risk-adjusted score. We collect an expert-annotated meta-evaluation
dataset, evaluate our approach with a range of judge models, and show that it achieves high correlations with veterinary experts even with small judge models, while offering explainability across multiple dimensions.
Read the full abstract ▾
All authors Ivan Kartáč, Jan Tovarys, Mateusz Lango, Ondřej Dušek
Computation and Language
How Closely Do LLM Reviews Align with Human Peer Review?
Abraham Camelo-Guerrero and Jairo Diaz-Rodriguez
Large
language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting. We compare reviews from OpenAI GPT-5.4, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.6 with human reviews and final decisions for 300 topic-matched ICLR 2026 submissions, equally divided among oral, poster, and rejected papers. Each model reviewed every paper using identical instructions and rating scales after decision information was removed. Our study contributes a cross-provider analysis of three complementary dimensions:
alignment with broad and fine-grained decision categories, differences in recommendation-scale usage, and thematic agreement in identified weaknesses. All three LLMs distinguished accepted from rejected papers, but none reproduced the oral versus poster distinction present in human ratings. Scoring patterns were provider-specific: Gemini assigned systematically higher ratings, while OpenAI and Claude were closer to humans for rejected and poster papers but more critical of oral papers. Human and LLM reviews also differed in emphasis, with LLMs more frequently identifying missing baseline comparisons and humans more often raising computational-efficiency concerns. These results show that broad decision
alignment does not imply agreement with finer human judgments or reviewing priorities.
Read the full abstract ▾