Computation and Language
How Retrieved Context Shapes Internal Representations in RAG
Samuel Yeh and Sharon Li
Retrieval-augmented generation (RAG) enhances large
language models (LLMs) by conditioning generation on retrieved external documents, but the effect of retrieved context is often non-trivial. In realistic
retrieval settings, the retrieved document set often contains a mixture of documents that vary in relevance and usefulness. While prior work has largely examined these phenomena through output behavior, little is known about how retrieved context shapes the internal representations that mediate information integration in RAG. In this work, we study RAG through the lens of latent representations. We systematically analyze how different types of retrieved documents affect the hidden states of LLMs, and how these internal representation shifts relate to downstream generation behavior. Across four question-answering
datasets and three LLMs, we analyze internal representations under controlled single- and multi-document settings. Our results reveal how context relevancy and layer-wise processing influence internal representations, providing explanations of LLMs' output behaviors and insights for RAG system design.
Read the full abstract ▾
Computation and Language
Multilingual Large Language Models do not comprehend all natural languages to equal degrees
Natalia Moskvina, Raquel Montero, Masaya Yoshida et al. (+8 more)
Large
Language Models (LLMs) play a critical role in how humans access information. While their core use relies on comprehending written requests, our understanding of this ability is currently limited, because most
benchmarks evaluate LLMs in high-resource
languages predominantly spoken by Western, Educated, Industrialised, Rich, and Democratic (WEIRD) communities. The default assumption is that English is the best-performing
language for LLMs, while smaller, low-resource
languages are linked to less reliable outputs, even in
multilingual, state-of-the-art models. To track variation in the comprehension abilities of LLMs, we
prompt 3 popular models on a
language comprehension task across 12
languages, representing the Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, and Japonic
language families. Our results suggest that the models exhibit remarkable linguistic accuracy across typologically diverse
languages, yet they fall behind human baselines in all of them, albeit to different degrees. Contrary to what was expected, English is not the best-performing
language, as it was systematically outperformed by several Romance
languages, even lower-resource ones. We frame the results by discussing the role of several factors that drive LLM performance, such as tokenization,
language distance from Spanish and English, size of training data, and data origin in high- vs. low-resource
languages and WEIRD vs. non-WEIRD communities.
Read the full abstract ▾
All authors Natalia Moskvina, Raquel Montero, Masaya Yoshida, Ferdy Hubers, Paolo Morosi, Walid Irhaymi, Jin Yan, Tamara Serrano, Elena Pagliarini, Fritz Günther, Evelina Leivada
Computation and Language
Marco Scharringhausen
In this study, the output of large
language models (LLM) is considered an information source generating an unlimited sequence of symbols drawn from a finite alphabet. Given the probabilistic nature of modern LLMs, we assume a probabilistic model for these LLMs, following a constant random distribution and the source itself thus being stationary. We compare this source
entropy (per word) to that of natural
language (written or spoken) as represented by the Open American National Corpus (OANC). Our results indicate that the word
entropy of such LLMs is lower than the word
entropy of natural
speech both in written or spoken form. The long-term goal of such studies is to formalize the intuitions of information and uncertainty in large
language training to assess the impact of training an LLM from LLM generated training data. This refers to texts from the world wide web in particular.
Read the full abstract ▾
Computation and Language
AI Alignment Breaks at the Edge
Han Bao, Yue Huang, Xiaoda Wang et al. (+5 more)
General
Alignment has improved average-case helpfulness and safety, but current
alignment practice still rewards confident, single-turn responses. The problem is not only that models fail on edge cases; it is that current evaluation makes many of these failures hard to see. We take the position that
alignment must move beyond average-case evaluation by making failures under value conflict, plural stakeholder disagreement, and epistemic ambiguity visible and actionable. Scalar rewards compress diverse values into a single number; data and evaluation regimes collapse, filter, or fail to elicit the cases where
alignment is hardest; and governance often lacks mechanisms for adjudicating contested cases. These blind spots produce value flattening, representation loss, and uncertainty blindness. We use Edge
alignment to name a detection, evaluation, and governance agenda for surfacing these failures and connecting them to appropriate interventions. Rather than a single training objective, Edge
alignment defines the conditions under which standard
alignment should yield to mechanisms that preserve multidimensional value structure, represent plural perspectives, and support uncertainty-aware interaction. A pilot diagnostic set of 91 edge cases and four contemporary models illustrates that ordinary helpfulness and safety readings can miss process failures that edge-aware evaluation exposes. We outline operational edge signals, process-aware evaluation criteria, and a three-phase process stack that reframes
alignment as a lifecycle problem of dynamic normative governance.
Read the full abstract ▾
All authors Han Bao, Yue Huang, Xiaoda Wang, Zheyuan Zhang, Yujun Zhou, Carl Yang, Xiangliang Zhang, Yanfang Ye
Computation and Language
AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization
Fahmida Liza Piya and Rahmatollah Beheshti
Large
language models (LLMs) offer substantial promise for automating clinical text summarization, yet maintaining factual consistency remains challenging due to the length, noise, and heterogeneity of clinical documentation. We present AgenticSum, an inference-time, agentic framework that separates context selection, generation, verification, and targeted correction to reduce hallucinated content. The framework decomposes summarization into coordinated stages that compress task-relevant context, generate an initial draft, identify weakly supported spans using internal
attention grounding signals, and selectively revise flagged content under supervisory control. We evaluate AgenticSum on two public
datasets, using reference-based metrics, LLM-as-a-judge assessment, and human evaluation. Across various measures, AgenticSum demonstrates consistent improvements compared to vanilla LLMs and other strong baselines. Our results indicate that structured, agentic design with targeted correction offers an effective inference time solution to improve clinical note summarization using LLMs.
Read the full abstract ▾
Computation and Language
CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems
Wanyong Feng, Alexander Scarlatos, Ruochen Sun et al. (+1 more)
Existing Computerized Adaptive Testing (CAT) frameworks typically select questions based on the predicted likelihood that the student will answer correctly. This design ignores information contained in students' open-ended responses, especially in domains such as programming education, where code structures and bugs contain rich information on student knowledge. In this work, we propose Code GENerative CAT (CodeGENCAT), a generative CAT framework that selects questions using predicted student code responses. First, we develop a Generative Item Response Theory (GIRT) model that generates code responses conditioned on estimated student knowledge, trained with supervised fine-tuning followed by direct preference optimization for knowledge-response
alignment. Second, we introduce three question-selection
algorithms that measure uncertainty, coding style diversity, and information from predicted student code responses. Experiments on two real-world programming education
datasets show that CodeGENCAT outperforms all CAT baselines, achieving an AUC improvement of up to 4.32% over the strongest baseline in the early stages of adaptive testing.
Read the full abstract ▾
All authors Wanyong Feng, Alexander Scarlatos, Ruochen Sun, Andrew Lan
Computation and Language
QUIETT: Query-Independent Table Transformation for Robust Reasoning
Gaurav Najpande, Tampu Ravi Kumar, Manan Roy Choudhury et al. (+3 more)
Real-world tables often contain schema inconsistencies, heterogeneous value formats, and implicit relational structures that degrade table
reasoning and question answering. Existing approaches address these issues at query time, repeating normalization and restructuring for every new question and producing representations that do not generalize to unseen queries. We introduce QuIeTT, a transform-first, query-later framework that converts each raw table into a single SQL-ready canonical representation before any evaluation query is observed, under three constraints: no evaluation query access, information preservation, and single-table reuse. QuIeTT operates through a three-stage pipeline: issue probing, where synthetic queries surface structural deficiencies such as ambiguous schemas and heterogeneous formats; plan generation; and structured execution. Experiments on four
benchmarks across five model families show consistent improvements over strong baselines in both direct prompting and agentic
reasoning paradigms, with particularly strong gains on a manually annotated challenge set of structurally diverse unseen questions.
Read the full abstract ▾
All authors Gaurav Najpande, Tampu Ravi Kumar, Manan Roy Choudhury, Neha Valeti, Yanjie Fu, Vivek Gupta
Computation and Language
Cross-lingual Matryoshka Representation Learning across Speech and Text
Yaya Sy, Dioula Doucouré, Christophe Cerisara et al. (+1 more)
Speakers of under-represented
languages face both a
language barrier, as most online knowledge is in a few dominant
languages, and a modality barrier, since information is largely text-based while many
languages are primarily oral. We address this for French-Wolof by training the first bilingual
speech-text Matryoshka
embedding model, enabling efficient
retrieval of French text from Wolof
speech queries without relying on a costly ASR-translation pipelines. We introduce large-scale data curation pipelines and new
benchmarks, compare modeling strategies, and show that modality fusion within a frozen text Matryoshka model performs best. Although trained only for
retrieval, the model generalizes well to other tasks, such as
speech intent detection, indicating the learning of general semantic representations. Finally, we analyze cost-accuracy trade-offs across Matryoshka dimensions and ranks, showing that information is concentrated only in a few components, suggesting potential for efficiency improvements.
Read the full abstract ▾
All authors Yaya Sy, Dioula Doucouré, Christophe Cerisara, Irina Illina
Computation and Language
ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting
Yuxing Tian, Fengran Mo, Weixu Zhang et al. (+2 more)
The strong capabilities of recent Large
Language Models (LLMs) have made them highly effective for zero-shot re-ranking task.
Attention-based re-ranking methods, which derive relevance scores directly from
attention weights, offer an efficient and interpretable alternative to generation-based re-ranking methods. However, they still face two major limitations. First,
attention signals are highly concentrated a small subset of
tokens within a few documents, making others indistinguishable. Second,
attention often overemphasizes phrases lexically similar to the query, yielding biased rankings that irrelevant documents with mere lexical resemblance are regarded as relevant. In this paper, we propose ReAttn, a post-hoc re-weighting strategy for
attention-based re-ranking methods. It first compute the cross-document IDF weighting to down-weight
attention on query-overlapping
tokens that frequently appear across the candidate documents, reducing lexical bias and emphasizing distinctive terms. It then employs
entropy-based regularization to mitigate over-concentrated
attention, encouraging a more balanced distribution across informative
tokens. Both adjustments operate directly on existing
attention weights without additional training or supervision. Extensive experiments demonstrate the effectiveness of our method.
Read the full abstract ▾
All authors Yuxing Tian, Fengran Mo, Weixu Zhang, Yiyan Qi, Jian-Yun Nie
Computation and Language
Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval
Yibo Yan, Jiahao Huo, Guanbo Feng et al. (+12 more)
With the rapid proliferation of
multimodal information, Visual Document
Retrieval (VDR) has emerged as a critical frontier in bridging the gap between unstructured visually rich data and precise information acquisition. Unlike traditional natural image
retrieval, visual documents exhibit unique characteristics defined by dense textual content, intricate layouts, and fine-grained semantic dependencies. This paper surveys the VDR landscape as a
retrieval problem in its own right, rather than as the front end of a generation pipeline, and does so specifically through the lens of the
Multimodal Large
Language Model (MLLM) era. We begin by examining the
benchmark landscape, including the recent turn toward
reasoning-intensive evaluation, and then dive into the methodological
evolution along two orthogonal axes: what a retriever is, spanning
multimodal embedding models and reranker models, and how it is deployed, from single-stage
retrieval through
Retrieval-Augmented Generation (RAG) to Agentic systems. Cutting across both, we analyse the representation–efficiency trade-off that late interaction imposes. We ground these categories in leaderboard evidence on where the empirical frontier actually lies, and close by identifying persistent challenges and outlining promising future directions for
multimodal document intelligence.
Read the full abstract ▾
All authors Yibo Yan, Jiahao Huo, Guanbo Feng, Mingdong Ou, Yi Cao, Xin Zou, Shuliang Liu, Yuanhuiyi Lyu, Yu Huang, Jungang Li, Kening Zheng, Xu Zheng, Philip S. Yu, James Kwok, Xuming Hu
Computation and Language
Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming
Ian Steenstra, Paola Pedrelli, Weiyan Shi et al. (+2 more)
Large
Language Models (LLMs) are increasingly utilized for mental health support; however, current safety
benchmarks often fail to detect the complex, longitudinal risks inherent in therapeutic dialogue. We introduce an evaluation framework that pairs AI psychotherapists with simulated patient
agents equipped with dynamic cognitive-affective models and assesses therapy session simulations against a comprehensive quality of care and risk ontology. We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI
agents (including ChatGPT, Gemini, and Character AI) against a clinically-validated cohort of 15 patient personas representing diverse clinical phenotypes. Our large-scale simulation (N=369 sessions) reveals critical safety gaps in the use of AI for mental health support. We identify specific iatrogenic risks, including the validation of patient delusions ("AI Psychosis") and failure to de-escalate suicide risk. Finally, we validate an interactive data visualization dashboard with diverse stakeholders, including AI engineers and red teamers, mental health professionals, and policy experts (N=9), demonstrating that this framework effectively enables stakeholders to audit the "black box" of AI psychotherapy. These findings underscore the critical safety risks of AI-provided mental health support and the necessity of simulation-based clinical red teaming before deployment.
Read the full abstract ▾
All authors Ian Steenstra, Paola Pedrelli, Weiyan Shi, Stacy Marsella, Timothy W. Bickmore
Computation and Language
Janus-Q: End-to-End Event-Driven Trading via Hierarchical-Gated Reward Modeling
Xiang Li, Zikai Wei, Yiyan Qi et al. (+6 more)
Financial
market movements are often driven by discrete financial events conveyed through news, whose impacts are heterogeneous, abrupt, and difficult to capture under purely numerical prediction objectives. These limitations have motivated growing interest in using textual information as the primary source of trading signals in learning-based systems. Two key challenges hinder existing approaches: (1) the absence of large-scale, event-centric
datasets that jointly model news semantics and statistically grounded
market reactions, and (2) the misalignment between
language model
reasoning and financially valid trading behavior under dynamic
market conditions. To address these challenges, we propose Janus-Q, an end-to-end event-driven trading framework that elevates financial news events from auxiliary signals to primary decision units. Janus-Q unifies event-centric data construction and model optimization under a two-stage paradigm. Stage I focuses on event-centric data construction, building a large-scale financial news event
dataset comprising 62,400 articles annotated with 10 fine-grained event types, associated stocks, sentiment labels, and event-driven cumulative abnormal return (CAR). Stage II performs decision-oriented fine-tuning, combining supervised learning with reinforcement learning guided by a Hierarchical Gated Reward Model (HGRM), which explicitly captures trade-offs among multiple trading objectives. Extensive experiments demonstrate that Janus-Q achieves more consistent, interpretable, and profitable trading decisions than
market indices and LLM baselines, improving the Sharpe Ratio by up to 102.0% while increasing direction accuracy by over 17.5% compared to the strongest competing strategies.
Read the full abstract ▾
All authors Xiang Li, Zikai Wei, Yiyan Qi, Wanyun Zhou, Xiang Liu, Penglei Sun, Jian Guo, Yongqi Zhang, Xiaowen Chu
Computation and Language · Cross-listed from cs.LG
DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning
Zhongwei Wan, Yun Shen, Zhihao Dou et al. (+9 more)
Reinforcement learning with verifiers (RLVR) is a central paradigm for improving large
language model (LLM)
reasoning, yet existing methods often suffer from limited exploration. Policies tend to collapse onto a few
reasoning patterns and prematurely stop deep exploration, while conventional
entropy regularization introduces only local stochasticity and fails to induce meaningful path-level diversity, leading to weak and unstable learning signals in group-based policy optimization. We propose DSDR, a Dual-Scale Diversity Regularization reinforcement learning framework that decomposes diversity in LLM
reasoning into global and coupling components. Globally, DSDR promotes diversity among correct
reasoning trajectories to explore distinct solution modes. Locally, it applies a length-invariant,
token-level
entropy regularization restricted to correct trajectories, preventing
entropy collapse within each mode while preserving correctness. The two scales are coupled through a global-to-local allocation mechanism that emphasizes local regularization for more distinctive correct trajectories. We provide theoretical support showing that DSDR preserves optimal correctness under bounded regularization, sustains informative learning signals in group-based optimization, and yields a principled global-to-local coupling rule. Experiments on multiple
reasoning benchmarks demonstrate consistent improvements in accuracy and pass@k, highlighting the importance of dual-scale diversity for deep exploration in RLVR. Code is available at https://github.com/SUSTechBruce/DSDR.
Read the full abstract ▾
All authors Zhongwei Wan, Yun Shen, Zhihao Dou, Donghao Zhou, Yu Zhang, Xin Wang, Hui Shen, Jing Xiong, Chaofan Tao, Zixuan Zhong, Peizhou Huang, Mi Zhang
Computation and Language
Denotational Semantics for ODRL: Knowledge-Based Constraint Conflict Detection
Daham Mustafa, Diego Collarana, Yixin Peng et al. (+4 more)
ODRL's six set-based operators – isA, isPartOf, hasPart, isAnyOf, isAllOf, isNoneOf – depend on external domain knowledge that the W3C specification leaves unspecified. Without it, every cross-dataspace policy comparison defaults to Unknown. We present a denotational semantics that maps each ODRL constraint to the set of knowledge-base concepts satisfying it. Conflict detection reduces to denotation intersection under a three-valued verdict – Conflict, Compatible, or Unknown – that is sound under incomplete knowledge. The framework covers all three ODRL composition modes (and, or, xone) and all three semantic domains arising in practice: taxonomic (class subsumption), mereological (part-whole containment), and nominal (identity). For cross-dataspace interoperability, we define order-preserving
alignments between knowledge bases and prove two guarantees: conflicts are preserved across different KB standards, and unmapped concepts degrade gracefully to Unknown – never to false conflicts. A runtime soundness
theorem ensures that design-time verdicts hold for all execution contexts. The encoding stays within the decidable EPR fragment of first-order logic. We validate it with 154
benchmarks across six knowledge base families (GeoNames, ISO 3166, W3C DPV, a GDPR-derived taxonomy, BCP 47, and ISO 639-3) and four structural KBs targeting
adversarial edge cases. Both the Vampire
theorem prover and the Z3 SMT solver agree on all 154 verdicts. A key finding is that exclusive composition (xone) requires strictly stronger KB axioms than conjunction or disjunction: open-world semantics blocks exclusivity even when positive evidence appears to satisfy exactly one branch.
Read the full abstract ▾
All authors Daham Mustafa, Diego Collarana, Yixin Peng, Rafiqul Haque, Christoph Lange-Bever, Christoph Quix, Stephan Decker
Computation and Language
Axis-Aligned Semantics for ODRL: Resolving Dimensional Ambiguity in Policy Constraints
Daham Mustafa, Diego Collarana, Sabrina Kirrane et al. (+5 more)
The Open Digital Rights
Language (ODRL) represents policy constraints as triples of a left operand, an operator, and a value. Several spatial operands, however, range over multi-axis domains such as width, height, and depth, while the constraint syntax provides no explicit axis identity. As a result, policy engines cannot determine whether multiple constraints apply to the same axis or different ones, making conflict detection unsound or incomplete. We resolve this ambiguity by axis decomposition, replacing multi-axis operands with axis-specific scalar operands over totally ordered domains. Each constraint then denotes an interval per axis and each policy an axis-aligned box, reducing conflict detection to box comparison. We define a three-valued semantics (Conflict, Compatible, Unknown), prove the decomposition sound and backward compatible with ODRL, instantiate it as ODRL Axis-Aligned Profile (OAAP), and validate it on a
benchmark of 256 ODRL policy problems, each expressed in Turtle and compiled to first-order (TPTP) and SMT-LIB form, using Vampire, E, Z3, and cvc5.
Read the full abstract ▾
All authors Daham Mustafa, Diego Collarana, Sabrina Kirrane, Christoph Lange, Christoph Quix, Rafiqul Haque, Yixin Peng, Stefan Decker
Computation and Language
SHIELD: Semantic Heterogeneity Integrated Embedding for Latent Discovery in Clinical Trial Safety Signals
Francois Vandenhende, Anna Georgiou, Theodoros Psaras et al. (+1 more)
We present SHIELD, a novel methodology for automated and integrated safety signal detection in clinical trials. SHIELD combines disproportionality analysis with semantic clustering of adverse event (AE) terms applied to MedDRA term
embeddings. For each AE, the pipeline computes an information-theoretic disproportionality measure (Information Component) with effect size derived via empirical
Bayesian shrinkage. A utility
matrix is constructed by weighting semantic term-term similarities by signal magnitude, followed by spectral
embedding and clustering to identify groups of related AEs. Resulting clusters are annotated with syndrome-level summary labels using large
language models, yielding a coherent, data-driven representation of treatment-associated safety profiles in the form of a network graph and hierarchical tree. We implement the SHIELD framework in the context of a single-arm incidence summary, to compare two treatment arms or for the detection of any treatment effect in a multi-arm trial. We illustrate its ability to recover known safety signals and generate interpretable, cluster-based summaries in a real clinical trial example. This work bridges statistical signal detection with modern natural
language processing to enhance safety assessment and causal interpretation in clinical trials.
Read the full abstract ▾
All authors Francois Vandenhende, Anna Georgiou, Theodoros Psaras, Ellie Karekla
Computation and Language
SAMAS: A Spectrum-Guided Multi-Agent System for Achieving Style Fidelity in Literary Translation
Jingzhuo Wu, Jiajun Zhang, Keyan Jin et al. (+2 more)
Modern large
language models (LLMs) excel at generating fluent and faithful translations. However, they struggle to preserve an author's unique literary style, often producing semantically correct but generic outputs. This limitation stems from the inability of current single-model and static multi-
agent systems to perceive and adapt to stylistic variations. To address this, we introduce the Style-Adaptive Multi-
Agent System (SAMAS), a novel framework that treats style preservation as a signal processing task. Specifically, our method quantifies literary style into a Stylistic Feature Spectrum (SFS) using the wavelet packet transform. This SFS serves as a control signal to dynamically assemble a tailored workflow of specialized translation
agents based on the source text's structural patterns. Extensive experiments on translation
benchmarks show that SAMAS achieves competitive semantic accuracy against strong baselines, primarily by leveraging its statistically significant advantage in style fidelity.
Read the full abstract ▾
All authors Jingzhuo Wu, Jiajun Zhang, Keyan Jin, Dehua Ma, Junbo Wang
Computation and Language
Keyboards for the Endangered Idu Mishmi Language
Akhilesh Kakolu Ramarao
We present a mobile and desktop keyboard suite for Idu Mishmi, an endangered Trans-Himalayan
language spoken by approximately 11,000 people in Arunachal Pradesh, India. Although a Latin-based orthography was developed in 2018, no digital input tools existed to use it, forcing speakers into ad-hoc romanizations that cannot represent the full writing system. Our keyboards comprise two tools: (1) an Android mobile keyboard, published on the Google Play Store and actively used in teacher training programs, and (2) a Windows desktop keyboard currently undergoing community testing. Both tools support the complete Idu Mishmi character inventory, including schwa, retracted schwa, nasalized vowels, and accented forms. Both operate fully offline with zero network permissions, addressing connectivity constraints and data sovereignty concerns. We describe the design, implementation, and deployment as a replicable model for other endangered
language communities.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.FL
NILE: Formalizing Natural-Language Descriptions of Formal Languages
Tristan Kneisel, Marko Schmellenkamp, Fabian Vehlken et al. (+1 more)
This paper explores how natural-
language descriptions of formal
languages can be compared to their formal representations and how semantic differences can be explained. This is motivated from educational scenarios where learners describe a formal
language (presented, e.g., by a finite state automaton, regular expression, pushdown automaton, context-free grammar or in set notation) in natural
language, and an educational support system has to (1) judge whether the natural-
language description accurately describes the formal
language, and to (2) provide explanations why descriptions are not accurate. To address this question, we introduce a representation
language for formal
languages, Nile, which is designed so that Nile expressions can mirror the syntactic structure of natural-
language descriptions of formal
languages. Nile is sufficiently expressive to cover a broad variety of formal
languages, including all regular
languages and fragments of context-free
languages typically used in educational contexts. Generating Nile expressions that are syntactically close to natural-
language descriptions then allows to provide explanations for inaccuracies in the descriptions algorithmically. In experiments on an educational data set, we show that LLMs can translate natural-
language descriptions into equivalent, syntactically close Nile expressions with high accuracy - allowing to algorithmically provide explanations for incorrect natural-
language descriptions. Our experiments also show that while natural-
language descriptions can also be translated into regular expressions (but not context-free grammars), the expressions are often not syntactically close and thus not suitable for providing explanations.
Read the full abstract ▾
All authors Tristan Kneisel, Marko Schmellenkamp, Fabian Vehlken, Thomas Zeume
Computation and Language
KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge
Alex Robertson, Huizhi Liang, Mahbub Gani et al. (+2 more)
Large
Language Models (LLMs) possess a remarkable capacity to generate persuasive and intelligible
language. However, coherence does not equate to truthfulness, as the responses often contain subtle
hallucinations. Existing
benchmarks are limited by static and narrow questions, leading to limited coverage and misleading evaluations. We present KGHaluBench, a Knowledge Graph-based
hallucination benchmark that assesses LLMs across the breadth and depth of their knowledge, providing a fairer and more comprehensive insight into LLM truthfulness. Our framework utilises the KG to dynamically construct challenging, multifaceted questions, whose difficulty is then statistically estimated to address popularity bias. Our automated verification pipeline detects abstentions and verifies the LLM's response at both conceptual and correctness levels to identify different types of
hallucinations. We evaluate 25 frontier models, using novel accuracy and
hallucination metrics. The results provide a more interpretable insight into the knowledge factors that cause
hallucinations across different model sizes. KGHaluBench is publicly available to support future developments in
hallucination mitigation.
Read the full abstract ▾
All authors Alex Robertson, Huizhi Liang, Mahbub Gani, Rohit Kumar, Srijith Rajamohan
Computation and Language · Cross-listed from cs.IT
Nacrith: Neural Lossless Compression via Ensemble Context Modeling and High-Precision CDF Coding
Roberto Tacconelli
We present Nacrith, a lossless compression system that combines a 135M-parameter
transformer language model (SmolLM2-135M) with an ensemble of lightweight online predictors and a 32-bit arithmetic coder, achieving the best compression results among the systems evaluated in this study on natural
language text. Beyond the base LLM-plus-arithmetic-coding paradigm, Nacrith introduces several contributions: (1) a CDF precision upgrade from 2^16 to 2^24 that eliminates ~75% of quantization overhead caused by minimum-probability floors in large vocabularies; (2) a
token-level N-gram model for fast local predictions; (3) an adaptive log-space bias head correcting per-document LLM errors via online gradient descent; (4) confidence-based LLM skip for accelerating highly predictable
tokens; (5) a hybrid binary format (NC06) extending neural compression to arbitrary binary files–to our knowledge a first among LLM-based compressors; (6) a llama cpp inference backend achieving ~7x faster single-
token decode than PyTorch; (7) parallel multi-GPU compression across up to 8 workers; and (8) native KV cache sliding window reducing per-slide cost by ~37x. The system requires only ~500 MB of GGUF weights and ~1.2 GB VRAM per worker, running on consumer GPUs. On alice29 (Canterbury Corpus, 152 KB), Nacrith achieves 0.918 bits per byte (bpb)–outperforming gzip by 3.1x, bzip2 by 2.5x, CMIX v21 by 44%, and ts_zip by 20%, while compressing below the 0th-, 1st-, and 2nd-order byte-level Shannon
entropy bounds. On enwik8 (100 MB), Nacrith achieves 0.9389 bpb (11.74%), surpassing ts_zip (~1.11 bpb) by 15% and FineZip (1.024 bpb) by 8% despite using a 60x smaller model with no fine-tuning. An out-of-distribution (OOD) evaluation on a document published after the model's training cutoff confirms these gains are not memorization artifacts, achieving 0.723 bpb on unseen text.
Read the full abstract ▾
Computation and Language
Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning
Anna Borisiuk, Andrey Savchenko, Alexander Panchenko et al. (+1 more)
Machine Unlearning (MU) enables Large
Language Models (LLMs) to remove unsafe or outdated information. However, existing work assumes that all facts are equally forgettable and largely ignores whether the forgotten knowledge originates from pretraining or supervised fine-tuning (SFT). In this paper, we introduce DUET (Dual Unlearning Evaluation across Training Stages), a
benchmark of 28.6k Wikidata-derived triplets annotated with fact popularity using Wikipedia link counts and LLM-based salience scores. Our experiments show that pretrained and SFT models respond differently to unlearning. An SFT step on the forget data yields smoother forgetting, more stable tuning, and 10-50% higher retention, while direct unlearning on pretrained models remains unstable and prone to relearning or catastrophic forgetting.
Read the full abstract ▾
All authors Anna Borisiuk, Andrey Savchenko, Alexander Panchenko, Elena Tutubalina
Computation and Language
Eye-Tracking-while-Reading: A Living Survey of Datasets with Open Library Support
Deborah N. Jakobi, David R. Reich, Paul Prasse et al. (+3 more)
Eye-tracking-while-reading corpora are a valuable resource for many different disciplines and use cases. Use cases range from studying the cognitive processes underlying reading to machine-learning-based applications, such as gaze-based assessments of reading comprehension. The past decades have seen an increase in the number and size of eye-tracking-while-reading
datasets as well as increasing diversity with regard to the stimulus
languages covered, the linguistic background of the participants, or accompanying psychometric or demographic data. The spread of data across different disciplines and the lack of data sharing standards across the communities lead to many existing
datasets that cannot be easily reused due to a lack of interoperability. In this work, we aim at creating more transparency and clarity with regards to existing
datasets and their features across different disciplines by i) presenting an extensive overview of existing
datasets, ii) simplifying the sharing of newly created
datasets by publishing a living overview online, https://t.uzh.ch/1Yh, presenting over 55 features for each
dataset, and iii) integrating all publicly available
datasets into the Python package pymovements which offers an eye-tracking
datasets library. By doing so, we aim to strengthen the FAIR principles in eye-tracking-while-reading research and promote good scientific practices, such as reproducing and replicating studies.
Read the full abstract ▾
All authors Deborah N. Jakobi, David R. Reich, Paul Prasse, Jana M. Hofmann, Lena S. Bolliger, Lena A. Jäger
Computation and Language
DEEP: Docker-based Execution and Evaluation Platform
Sergio Gómez González, Miguel Domingo and Francisco Casacuberta
Comparative evaluation of several systems is a recurrent task in researching. It is a key step before deciding which system to use for our work, or, once our research has been conducted, to demonstrate the potential of the resulting model. Furthermore, it is the main task of competitive, public challenges evaluation. Our proposed software (DEEP) automates both the execution and scoring of machine translation and optical character recognition models. Furthermore, it is easily extensible to other tasks. DEEP is prepared to receive dockerized systems, run them (extracting information at that same time), and assess hypothesis against some references. With this approach, evaluators can achieve a better understanding of the performance of each model. Moreover, the software uses a clustering
algorithm based on a statistical analysis of the significance of the results yielded by each model, according to the evaluation metrics. As a result, evaluators are able to identify clusters of performance among the swarm of proposals and have a better understanding of the significance of their differences. Additionally, we offer a visualization web-app to ensure that the results can be adequately understood and interpreted. Finally, we present an exemplary case of use of DEEP.
Read the full abstract ▾
Computation and Language
Temporal-Aware Heterogeneous Graph Reasoning with Multi-View Fusion for Temporal Question Answering
Wuzhenghong Wen, Bowen Zhou, Jinwen Huang et al. (+5 more)
Question Answering over Temporal Knowledge Graphs (TKGQA) has attracted growing interest for handling time-sensitive queries. However, existing methods still struggle with: 1) weak incorporation of temporal constraints in question representation, causing biased
reasoning; 2) limited ability to perform explicit multi-hop
reasoning; and 3) suboptimal fusion of
language and graph representations. We propose a novel framework with temporal-aware question encoding, multi-hop graph
reasoning, and multi-view heterogeneous information fusion. Specifically, our approach introduces: 1) a constraint-aware question representation that combines semantic cues from
language models with temporal entity dynamics; 2) a temporal-aware graph neural network for explicit multi-hop
reasoning via time-aware message passing; and 3) a multi-view
attention mechanism for more effective fusion of question context and temporal graph knowledge. Experiments on multiple TKGQA
benchmarks demonstrate consistent improvements over multiple baselines.
Read the full abstract ▾
All authors Wuzhenghong Wen, Bowen Zhou, Jinwen Huang, Xianjie Wu, Yuwei Sun, Su Pan, Liang Li, Jianting Liu
Computation and Language
Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework
Yibo Yan, Mingdong Ou, Yi Cao et al. (+5 more)
Visual Document
Retrieval (VDR), which aims to retrieve relevant pages within vast corpora of visually-rich documents, is of significance in current
multimodal retrieval applications. The state-of-the-art multi-vector paradigm excels in performance but suffers from prohibitive overhead, a problem that current efficiency methods like
pruning and merging address imperfectly, creating a difficult trade-off between compression rate and feature fidelity. To overcome this dilemma, we introduce Prune-then-Merge, a novel two-stage framework that synergizes these complementary approaches. Our method first employs an adaptive
pruning stage to filter out low-information patches, creating a refined, high-signal set of
embeddings. Subsequently, a hierarchical merging stage compresses this pre-filtered set, effectively summarizing semantic content without the noise-induced feature dilution seen in single-stage methods. Extensive experiments on 29 VDR
datasets demonstrate that our framework consistently outperforms existing methods, significantly extending the near-lossless compression range and providing robust performance at high compression ratios.
Read the full abstract ▾
All authors Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Jiahao Huo, Shuliang Liu, James Kwok, Xuming Hu
Computation and Language
Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
Jeffrey Li, Josh Gardner, Doug Kang et al. (+10 more)
One of the first pre-processing steps for constructing web-scale LLM pretraining
datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source
datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard
language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple intervention: by taking a Union over different extractors, we can increase the
token yield of DCLM-Baseline by up to 71% while maintaining
benchmark performance. We further show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance, with differences of up to 10 percentage points (p.p.) on WikiTQ and 3 p.p. on HumanEval.
Read the full abstract ▾
All authors Jeffrey Li, Josh Gardner, Doug Kang, Fangping Shi, Karanjeet Singh, Chun-Liang Li, Herumb Shandilya, David Hall, Oncel Tuzel, Percy Liang, Ludwig Schmidt, Hadi Pour Ansari, Fartash Faghri
Computation and Language
Hyper-KGGen: A Skill-Driven Knowledge Extractor for High-Quality Knowledge Hypergraph Generation
Rizhuo Huang, Yifan Feng, Rundong Xue et al. (+5 more)
Knowledge hypergraphs surpass traditional binary knowledge graphs by encapsulating complex n-ary atomic facts, providing a more comprehensive paradigm for semantic representation. However, constructing high-quality hypergraphs remains challenging due to the scenario gap: generic extractors struggle to generalize across diverse domains with specific jargon, while existing methods often fail to balance structural skeletons with fine-grained details. To bridge this gap, we propose Hyper-KGGen, a skill-driven framework that reformulates extraction as a dynamic skill-evolving process. First, Hyper-KGGen employs a coarse-to-fine mechanism to systematically decompose documents, ensuring full-dimensional coverage from binary links to complex hyperedges. Crucially, it incorporates an adaptive skill acquisition module that actively distills domain expertise into a Global Skill Library. This is achieved via a stability-based feedback loop, where extraction stability serves as a relative reward signal to induce high-quality skills from unstable traces and missed predictions. Additionally, we present HyperDocRED, a rigorously annotated
benchmark for document-level knowledge hypergraph extraction. Experiments demonstrate that Hyper-KGGen significantly outperforms strong baselines, validating that evolved skills provide substantially richer guidance than static few-shot examples in multi-scenario settings.
Read the full abstract ▾
All authors Rizhuo Huang, Yifan Feng, Rundong Xue, Shihui Ying, Jun-Hai Yong, Chuan Shi, Shaoyi Du, Yue Gao
Computation and Language
How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1
Yinuo Xu, Shuo Lu, Jianjie Cheng et al. (+5 more)
Deep Research
agents tackle knowledge-intensive tasks through multi-round
retrieval and decision-oriented generation. While reinforcement learning (RL) has been shown to improve performance in this paradigm, its contributions remain underexplored. To fully understand the role of RL, we conduct a systematic study along three decoupled dimensions:
prompt template, reward function, and policy optimization. Our study reveals that: 1) the Fast Thinking template yields greater stability and better performance than the Slow Thinking template used in prior work; 2) the F1-based reward underperforms the EM due to training collapse driven by answer avoidance; this can be mitigated by incorporating action-level penalties, ultimately surpassing EM; 3) REINFORCE outperforms PPO while requiring fewer search actions, whereas GRPO shows the poorest stability among policy optimization methods. Building on these insights, we then introduce Search-R1++, a strong baseline that improves the performance of Search-R1 from 0.403 to 0.442 (Qwen2.5-7B) and 0.289 to 0.331 (Qwen2.5-3B). We hope that our findings can pave the way for more principled and reliable RL training strategies in Deep Research systems.
Read the full abstract ▾
All authors Yinuo Xu, Shuo Lu, Jianjie Cheng, Meng Wang, Qianlong Xie, Xingxing Wang, Ran He, Jian Liang
Computation and Language · Cross-listed from cs.AI
Classroom Final Exam: An Instructor-Tested Reasoning Benchmark
Chongyang Gao, Diji Yang, Shuyan Zhou et al. (+4 more)
We introduce CFE-Bench (Classroom Final Exam), a
multimodal benchmark for evaluating the
reasoning capabilities of large
language models across more than 20 STEM domains. CFE-Bench is curated from repeatedly used, authentic university homework and exam problems, paired with reference solutions provided by course instructors. CFE-Bench remains challenging for frontier models: the newly released Gemini-3.1-pro-preview achieves 59.69% overall accuracy, while the second-best model, Gemini-3-flash-preview, reaches 55.46%, leaving substantial room for improvement. Beyond aggregate scores, we conduct a diagnostic analysis by decomposing instructor reference solutions into structured
reasoning flows. We find that while frontier models often answer intermediate sub-questions correctly, they struggle to reliably derive and maintain correct intermediate states throughout multi-step solutions. We further observe that model-generated solutions typically contain more
reasoning steps than instructor solutions, indicating lower step efficiency and a higher risk of error accumulation. Data and code are available at https://github.com/Analogy-AI/CFE_Bench.
Read the full abstract ▾
All authors Chongyang Gao, Diji Yang, Shuyan Zhou, Xichen Yan, Luchuan Song, Shuo Li, Kezhen Chen