Computation and Language
Efficient and Reliable Estimation of Named Entity Linking Quality: A Case Study on GutBrainIE
Marco Martinelli, Stefano Marchesin and Gianmaria Silvello
Named Entity Linking (NEL) is a core component of biomedical Information Extraction (IE) pipelines, yet assessing its quality at scale is challenging due to the high cost of expert annotations and the large size of corpora. In this paper, we present a sampling-based framework to estimate the NEL accuracy of large-scale IE corpora under statistical guarantees and constrained annotation budgets. We frame NEL accuracy estimation as a constrained optimization problem, where the objective is to minimize expected annotation cost subject to a target Margin of Error (MoE) for the corpus-level accuracy estimate. Building on recent works on knowledge graph accuracy estimation, we adapt Stratified Two-Stage Cluster Sampling (STWCS) to the NEL setting, defining label-based strata and global surface-form clusters in a way that is independent of NEL annotations. Applied to 11,184 NEL annotations in GutBrainIE – a new biomedical corpus openly released in fall 2025 – our framework reaches a MoE $\leq 0.05$ by manually annotating only 2,749 triples (24.6%), leading to an overall accuracy estimate of $0.915 \pm 0.0473$. A time-based cost model and simulations against a Simple Random Sampling (SRS) baseline show that our design reduces expert annotation time by about 29% at fixed sample size. The framework is generic and can be applied to other NEL
benchmarks and IE pipelines that require scalable and statistically robust accuracy assessment.
Read the full abstract ▾
Computation and Language
Pragya: An AI-Based Semantic Recommendation System for Sanskrit Subhasitas
Tanisha Raorane and Prasenjit Kole
Sanskrit Subhasitas encapsulate centuries of cultural and philosophical wisdom, yet remain underutilized in the digital age due to linguistic and contextual barriers. In this work, we present Pragya, a
retrieval-augmented generation (RAG) framework for semantic recommendation of Subhasitas. We curate a
dataset of 200 verses annotated with thematic tags such as motivation, friendship, and compassion. Using sentence
embeddings (IndicBERT), the system retrieves top-k verses relevant to user queries. The retrieved results are then passed to a generative model (Mistral LLM) to produce transliterations, translations, and contextual explanations. Experimental evaluation demonstrates that semantic
retrieval significantly outperforms keyword matching in precision and relevance, while user studies highlight improved accessibility through generated summaries. To our knowledge, this is the first attempt at integrating
retrieval and generation for Sanskrit Subhasitas, bridging cultural heritage with modern applied AI.
Read the full abstract ▾
Computation and Language
N2N-GQA: Noise-to-Narrative for Graph-Based Table-Text Question Answering Using LLMs
Mohamed Sharafath, Aravindh Annamalai, Ganesh Murugan et al. (+1 more)
Multi-hop question answering over hybrid table-text data requires retrieving and
reasoning across multiple evidence pieces from large corpora, but standard
Retrieval-Augmented Generation (RAG) pipelines process documents as flat ranked lists, causing
retrieval noise to obscure
reasoning chains. We introduce N2N-GQA. To our knowledge, it is the first zeroshot framework for open-domain hybrid table-text QA that constructs dynamic evidence graphs from noisy
retrieval outputs. Our key insight is that multi-hop
reasoning requires understanding relationships between evidence pieces: by modeling documents as graph nodes with semantic relationships as edges, we identify bridge documents connecting
reasoning steps, a capability absent in list-based
retrieval. On OTT-QA, graph-based evidence curation provides a 19.9-point EM improvement over strong baselines, demonstrating that organizing
retrieval results as structured graphs is critical for multihop
reasoning. N2N-GQA achieves 48.80 EM, matching finetuned
retrieval models (CORE: 49.0 EM) and approaching heavily optimized systems (COS: 56.9 EM) without any task specific training. This establishes graph-structured evidence organization as essential for scalable, zero-shot multi-hop QA systems and demonstrates that simple, interpretable graph construction can rival sophisticated fine-tuned approaches.
Read the full abstract ▾
All authors Mohamed Sharafath, Aravindh Annamalai, Ganesh Murugan, Aravindakumar Venugopalan
Computation and Language
Probing Multimodal Large Language Models on Cognitive Biases in Chinese Short-Video Misinformation
Jen-tse Huang, Chang Chen, Shiyang Lai et al. (+3 more)
Short-
video platforms have become major channels for misinformation, where deceptive claims frequently leverage visual experiments and social cues. While
Multimodal Large
Language Models (MLLMs) have demonstrated impressive
reasoning capabilities, their robustness against misinformation entangled with cognitive biases remains under-explored. In this paper, we introduce a comprehensive evaluation framework using a high-quality, manually annotated
dataset of 200 short
videos spanning four health domains. This
dataset provides fine-grained annotations for three deceptive patterns-experimental errors, logical fallacies, and fabricated claims-each verified by evidence such as national standards and academic literature. We evaluate eight frontier MLLMs across five modality settings. Experimental results demonstrate that Gemini-2.5-Pro achieves the highest performance in the
multimodal setting with a belief score of 71.5/100, while o3 performs the worst at 35.2. Furthermore, we investigate social cues that induce false beliefs in
videos and find that models are susceptible to biases like authoritative channel IDs.
Read the full abstract ▾
All authors Jen-tse Huang, Chang Chen, Shiyang Lai, Wenxuan Wang, Michelle R. Kaufman, Mark Dredze
Computation and Language
How Context Shapes Truth: Geometric Transformations of Statement-level Truth Representations in LLMs
Shivam Adarsh, Maria Maistro and Christina Lioma
Large
Language Models (LLMs) often encode whether a statement is true as a vector in their residual stream activations. These vectors, also known as truth vectors, have been studied in prior work, however how they change when context is introduced remains unexplored. We study this question by measuring (1) the directional change ($\theta$) between the truth vectors with and without context and (2) the relative magnitude of the truth vectors upon adding context. Across four LLMs and four
datasets, we find that (1) truth vectors are roughly orthogonal in early layers, converge in middle layers, and may stabilize or continue increasing in later layers; (2) adding context generally increases the truth vector magnitude, i.e., the separation between true and false representations in the activation space is amplified; (3) larger models distinguish relevant from irrelevant context mainly through directional change ($\theta$), while smaller models show this distinction through magnitude differences. We also find that context conflicting with parametric knowledge produces larger
geometric changes than parametrically aligned context. Collectively, these findings provide a
geometric characterization of how context transforms the truth vector in the activation space of LLMs.
Read the full abstract ▾
Computation and Language
Detecting LLM-Generated Text with Performance Guarantees
Hongyi Zhou, Jin Zhu, Ying Yang et al. (+1 more)
Large
language models (LLMs) such as GPT, Claude, Gemini, and Grok have been deeply integrated into our daily life. They now support a wide range of tasks – from dialogue and email drafting to assisting with teaching and coding, serving as search engines, and much more. However, their ability to produce highly human-like text raises serious concerns, including the spread of fake news, the generation of misleading governmental reports, and academic misconduct. To address this practical problem, we train a classifier to determine whether a piece of text is authored by an LLM or a human. Our detector is deployed on an online CPU-based platform https://huggingface.co/spaces/stats-powered-ai/StatDetectLLM, and contains three novelties over existing detectors: (i) it does not rely on auxiliary information, such as watermarks or knowledge of the specific LLM used to generate the text; (ii) it more effectively distinguishes between human- and LLM-authored text; and (iii) it enables statistical inference, which is largely absent in the current literature. Empirically, our classifier achieves higher classification accuracy compared to existing detectors, while maintaining type-I error control, high statistical power, and computational efficiency.
Read the full abstract ▾
All authors Hongyi Zhou, Jin Zhu, Ying Yang, Chengchun Shi
Computation and Language
Stylistic Evolution and LLM Neutrality in Singlish Language
Linus Tze En Foo, Weihan Angela Ng, Wenkai Li et al. (+1 more)
Singlish is a creole rooted in Singapore's
multilingual environment that continues to evolve alongside social and technological change. We examine diachronic stylistic change across a decade of informal digital messages and ask whether Large
Language Models (LLMs) can generate temporally neutral outputs approximating the stable essence of the variety. Using lexical, pragmatic, psycholinguistic, and encoder-based features, we find that stylistic separability increases with temporal distance, driven primarily by structural features such as length and complexity. Evaluated against a null distribution baseline, most LLMs fail to achieve both authenticity and temporal neutrality simultaneously, revealing a structural trade-off: models generating realistic Singlish inherit its temporal biases, while temporally neutral models produce inauthentic outputs. These findings position temporal neutrality as a diagnostic metric for assessing sociolectal grounding in LLMs.
Read the full abstract ▾
All authors Linus Tze En Foo, Weihan Angela Ng, Wenkai Li, Lynnette Hui Xian Ng
Computation and Language
Are Emotions Arranged in a Circle? Geometric Analysis of Emotion Representations via Hyperspherical Contrastive Learning
Yusuke Yamauchi and Akiko Aizawa
Psychological research has long utilized circumplex models to structure
emotions, placing similar
emotions adjacently and opposing ones diagonally. Although frequently used to interpret deep learning representations, these models are rarely directly incorporated into the representation learning of
language models, leaving their
geometric validity unexplored. This paper proposes a method to induce circular
emotion representations within
language model
embeddings via contrastive learning on a hypersphere. We show that while this circular
alignment offers superior interpretability and robustness against dimensionality reduction, it underperforms compared to conventional designs in high-dimensional settings and fine-grained classification. Our findings elucidate the trade-offs involved in applying psychological circumplex models to deep learning architectures.
Read the full abstract ▾
Computation and Language
EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation
Pei Yang, Wanyi Chen, Ke Wang et al. (+3 more)
Large
language models are increasingly applied to various development scenarios. However, in on-chain transaction scenarios, even a minor error can cause irreversible loss for users. Existing evaluations often overlook execution accuracy and safety. We introduce EVM-QuestBench, an execution-grounded
benchmark for natural-
language transaction-script generation on EVM-compatible chains. The
benchmark employs dynamic evaluation: instructions are sampled from template pools, numeric parameters are drawn from predefined intervals, and validators verify outcomes against these instantiated values. EVM-QuestBench contains 107 tasks (62 atomic, 45 composite). Its modular architecture enables rapid task development. The runner executes scripts on a forked EVM chain with snapshot isolation; composite tasks apply step-efficiency decay. We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion. Code: https://anonymous.4open.science/r/bsc_quest_bench-A9CF/.
Read the full abstract ▾
All authors Pei Yang, Wanyi Chen, Ke Wang, Lynn Ai, Eric Yang, Tianyu Shi
Computation and Language
CSR-RAG: An Efficient Retrieval System for Text-to-SQL on the Enterprise Scale
Rajpreet Singh, Novak Boškov, Lawrence Drabeck et al. (+2 more)
Natural
language to SQL translation (Text-to-SQL) is one of the long-standing problems that has recently benefited from advances in Large
Language Models (LLMs). While most academic Text-to-SQL
benchmarks request schema description as a part of natural
language input, enterprise-scale applications often require table
retrieval before SQL query generation. To address this need, we propose a novel hybrid
Retrieval Augmented Generation (RAG) system consisting of contextual, structural, and relational
retrieval (CSR-RAG) to achieve computationally efficient yet sufficiently accurate
retrieval for enterprise-scale databases. Through extensive enterprise
benchmarks, we demonstrate that CSR-RAG achieves up to 40% precision and over 80% recall while incurring a negligible average query generation latency of only 30ms on commodity data center hardware, which makes it appropriate for modern LLM-based enterprise-scale systems.
Read the full abstract ▾
All authors Rajpreet Singh, Novak Boškov, Lawrence Drabeck, Aditya Gudal, Manzoor A. Khan
Computation and Language · Cross-listed from cs.IR
L-RAG: Balancing Context and Retrieval with Entropy-Based Lazy Loading
Sergii Voloshyn
Retrieval-Augmented Generation (RAG) has emerged as the predominant paradigm for grounding Large
Language Model outputs in factual knowledge, effectively mitigating
hallucinations. However, conventional RAG systems operate under a "retrieve-always" assumption, querying vector databases for every input regardless of query complexity. This static approach incurs substantial computational overhead and inference latency, particularly problematic for high-throughput production deployments. We introduce L-RAG (Lazy
Retrieval-Augmented Generation), an adaptive framework that implements hierarchical context management through
entropy-based gating. L-RAG employs a two-tier architecture: queries are first processed with a compact document summary, and expensive chunk
retrieval is triggered only when the model's predictive
entropy exceeds a calibrated threshold, signaling genuine uncertainty. Through experiments on SQuAD 2.0 (N=500) using the Phi-2 model, we demonstrate that L-RAG provides a tunable accuracy-efficiency trade-off: at a conservative threshold (tau=0.5), L-RAG achieves 78.2% accuracy, matching Standard RAG (77.8%), with 8%
retrieval reduction; at a balanced threshold (tau=1.0),
retrieval reduction increases to 26% with modest accuracy trade-off (76.0%). Latency analysis shows that L-RAG saves 80-210ms per query when
retrieval latency exceeds 500ms. Analysis of
entropy distributions reveals statistically significant separation (p < 0.001) between correct predictions (H=1.72) and errors (H=2.20), validating
entropy as a reliable uncertainty signal. L-RAG offers a practical, training-free approach toward more efficient RAG deployment, providing system architects with a configurable knob to balance accuracy and throughput requirements.
Read the full abstract ▾
Computation and Language
Mechanism-Faithful Queueing Simulation Model Translation with Large Language Model Support
Jun-Qi Chen, Kun Zhang, Rui Zheng et al. (+1 more)
Queueing simulation studies often require substantial manual effort to translate conceptual system descriptions into executable programs and to verify that the implemented mechanisms match the intended queueing logic. Although large
language models (LLMs) may produce executable scripts, executability alone is insufficient when arrival, routing, interruption, or reporting logic is wrong. This study presents a simulation-oriented support framework for SimPy-based queueing model translation. We propose a category-template framework for mechanism coverage with a staged adaptation workflow that targets structured event logic and common simulation-specific failure modes. On held-out task instances, the adapted models improve executability, output-format compliance, and instruction-mechanism consistency across basic, behavioral, and networked queueing settings, so the generated scripts are more reliable as queueing simulation scripts. Error analysis shows better preservation of routing semantics and interruption-resume logic, while also exposing remaining weaknesses in multi-node transfer and residual-service updates. Overall, the results suggest that the proposed framework can act as a simulation-faithful generator for more standardized and reproducible queueing model construction.
Read the full abstract ▾
All authors Jun-Qi Chen, Kun Zhang, Rui Zheng, Ying Zhong
Computation and Language
Exposía: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback
Dennis Zyska, Alla Rozovskaya, Ilia Kuznetsov et al. (+1 more)
We present Exposía, the first public
dataset that connects writing and feedback in higher education, enabling research on educationally grounded computational approaches to teaching and evaluating academic writing. Exposía includes student research project proposals and peer and instructor feedback consisting of comments and free-text reviews. The
dataset was collected in the "Introduction to Scientific Work" course of the Computer Science. Exposía reflects the multi-stage nature of the academic writing process that includes drafting, receiving feedback, and revising the writing based on the feedback received. Both the project proposals and peer feedback are accompanied by human assessment scores based on a fine-grained, pedagogically-grounded schema for writing and feedback assessment that we develop. We use Exposía to
benchmark state-of-the-art large
language models (LLMs) on two tasks: automated scoring of (1) the proposals and (2) the student reviews. We find that the two tasks are best served by different LLMs. Furthermore, closed-source models consistently outperform open-weight models, motivating further research on improving the performance of open-weight models preferred in classroom settings. Finally, we establish that a prompting strategy that scores multiple aspects of the writing together is the most effective, paving the way for more effective classroom deployment of modern LLMs.
Read the full abstract ▾
All authors Dennis Zyska, Alla Rozovskaya, Ilia Kuznetsov, Iryna Gurevych
Computation and Language
Atomic-SNLI: Fine-Grained Natural Language Inference through Atomic Fact Decomposition
Minghui Huang
Current Natural
Language Inference (NLI) systems primarily operate at the sentence level, providing black-box decisions that lack explanatory power. While atomic-level NLI offers a promising alternative by decomposing hypotheses into individual facts, we demonstrate that the conventional assumption that a hypothesis is entailed only when all its atomic facts are entailed fails in practice due to models' poor performance on fine-grained
reasoning. Our analysis reveals that existing models perform substantially worse on atomic level inference compared to sentence level tasks. To address this limitation, we introduce Atomic-SNLI, a novel
dataset constructed by decomposing SNLI and enriching it with carefully curated atomic level examples through linguistically informed generation strategies. Experimental results demonstrate that models fine-tuned on Atomic-SNLI achieve significant improvements in atomic
reasoning capabilities while maintaining strong sentence level performance, enabling both accurate judgements and transparent, explainable results at the fact level.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Liang Chen, Weichu Xie, Yiyan Liang et al. (+27 more)
While humans develop core visual skills long before acquiring
language, contemporary
Multimodal LLMs (MLLMs) still rely heavily on linguistic priors to compensate for their fragile visual understanding. We uncovered a crucial fact: state-of-the-art MLLMs consistently fail on basic visual tasks that humans, even 3-year-olds, can solve effortlessly. To systematically investigate this gap, we introduce BabyVision, a
benchmark designed to assess core visual abilities independent of linguistic knowledge for MLLMs. BabyVision spans a wide range of tasks, with 388 items divided into 22 subclasses across four key categories. Empirical results and human evaluation reveal that leading MLLMs perform significantly below human baselines. Gemini3-Pro-Preview scores 49.7, lagging behind 6-year-old humans and falling well behind the average adult score of 94.1. These results show despite excelling in knowledge-heavy evaluations, current MLLMs still lack fundamental visual primitives. Progress in BabyVision represents a step toward human-level visual perception and
reasoning capabilities. We also explore solving visual
reasoning with generation models by proposing BabyVision-Gen and automatic evaluation toolkit. Our code and
benchmark data are released at https://github.com/UniPat-AI/BabyVision for reproduction.
Read the full abstract ▾
All authors Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, Zhibo Yang, Zhiqi Huang, Haoning Wu, Haoyu Lu, Y. charles, Yiping Bao, Yuantao Fan, Guopeng Li, Haiyang Shen, Xuanzhong Chen, Wendong Xu, Shuzheng Si, Zefan Cai, Wenhao Chai, Ziqi Huang, Fangfu Liu, Tianyu Liu, Baobao Chang, Ming Wu, Xiaobo Hu, Kaiyuan Chen, Yixin Ren, Yang Liu, Yuan Gong, Kuan Li
Computation and Language
MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generation
Yuelyu Ji, Min Gu Kwak, Hang Zhang et al. (+3 more)
Biomedical
retrieval-augmented generation (RAG) can ground LLM answers in medical literature, yet long-form outputs often contain isolated unsupported or contradictory claims with safety implications. We introduce MedRAGChecker, a claim-level verification and diagnostic framework for biomedical RAG. Given a question, retrieved evidence, and a generated answer, MedRAGChecker decomposes the answer into atomic claims and estimates claim support by combining evidence-grounded natural
language inference (NLI) with biomedical knowledge-graph (KG) consistency signals. Aggregating claim decisions yields answer-level diagnostics that help disentangle
retrieval and generation failures, including faithfulness, under-evidence, contradiction, and safety-critical error rates. To enable scalable evaluation, we distill the pipeline into compact biomedical models and use an ensemble verifier with class-specific reliability weighting. Experiments on four biomedical QA
benchmarks show that MedRAGChecker reliably flags unsupported and contradicted claims and reveals distinct risk profiles across generators, particularly on safety-critical biomedical relations.
Read the full abstract ▾
All authors Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wang
Computation and Language
Spec-o3: A Tool-Augmented Vision-Language Agent for Rare Celestial Object Candidate Vetting via Automated Spectral Inspection
Minghui Jia, Qichao Zhang, Ali Luo et al. (+5 more)
Due to the limited generalization and interpretability of deep learning classifiers, The final vetting of rare celestial object candidates still relies on expert visual inspection–a manually intensive process. In this process, astronomers leverage specialized tools to analyze spectra and construct reliable catalogs. However, this practice has become the primary bottleneck, as it is fundamentally incapable of scaling with the data deluge from modern spectroscopic surveys. To bridge this gap, we propose Spec-o3, a tool-augmented
vision-
language agent that performs astronomer-aligned spectral inspection via interleaved
multimodal chain-of-thought
reasoning. Spec-o3 is trained with a two-stage post-training recipe: cold-start supervised fine-tuning on expert inspection trajectories followed by outcome-based reinforcement learning on rare-type verification tasks. Evaluated on five rare-object identification tasks from LAMOST, Spec-o3 establishes a new State-of-the-Art, boosting the macro-F1 score from 28.3 to 76.5 with a 7B parameter base model and outperforming both proprietary VLMs and specialized deep models. Crucially, the
agent demonstrates strong generalization to unseen inspection tasks across survey shifts (from LAMOST to SDSS/DESI). Expert evaluations confirm that its
reasoning traces are coherent and physically consistent, supporting transparent and trustworthy decision-making. Code, data, and models are available at https://github.com/Maxwell-Jia/spec-o3.
Read the full abstract ▾
All authors Minghui Jia, Qichao Zhang, Ali Luo, Linjing Li, Shuo Ye, Hailing Lu, Wen Hou, Dongbin Zhao
Computation and Language
IndRegBias: A Dataset for Studying Indian Regional Biases in English and Code-Mixed Social Media Comments
Debasmita Panda, Akash Anil and Neelesh Kumar Shukla
Warning: This paper consists of examples representing regional biases in Indian regions that might be offensive towards a particular region. While social biases corresponding to gender, race, socio-economic conditions, etc., have been extensively studied in the major applications of Natural
Language Processing (NLP), biases corresponding to regions have garnered less
attention. This is mainly because of (i) difficulty in the extraction of regional bias
datasets, (ii) disagreements in annotation due to inherent human biases, and (iii) regional biases being studied in combination with other types of social biases and often being under-represented. This paper focuses on creating a
dataset IndRegBias, consisting of regional biases in an Indian context reflected in users' comments on popular social media platforms, namely Reddit and YouTube. We carefully selected 25,000 comments appearing on various threads in Reddit and
videos on YouTube discussing trending topics on regional issues in India. Furthermore, we propose a multilevel annotation strategy to annotate the comments describing the severity of regional biased statements. To detect the presence of regional bias and its severity in IndRegBias, we evaluate open-source Large
Language Models (LLMs) and Indic
Language Models (ILMs) using zero-shot, few-shot, and fine-tuning strategies. We observe that zero-shot and few-shot approaches show lower accuracy in detecting regional biases and severity in the majority of the LLMs and ILMs. However, the fine-tuning approach significantly enhances the performance of the LLM in detecting Indian regional bias along with its severity.
Read the full abstract ▾
Computation and Language
PRISP: Privacy-Safe Few-Shot Personalization via Lightweight Adaptation
Junho Park, Dohoon Kim and Taesup Moon
Large
language model (LLM) personalization aims to adapt general-purpose models to individual users. Most existing methods, however, are developed under data-rich and resource-abundant settings, often incurring
privacy risks. In contrast, realistic personalization typically occurs after deployment under (i) extremely limited user data, (ii) constrained computational resources, and (iii) strict
privacy requirements. We propose PRISP, a lightweight and
privacy-safe personalization framework tailored to these constraints. PRISP leverages a Text-to-LoRA hypernetwork to generate task-aware LoRA parameters from task descriptions, and enables efficient user personalization by optimizing a small subset of task-aware LoRA parameters together with minimal additional modules using few-shot user data. Experiments on a few-shot variant of the LaMP
benchmark demonstrate that PRISP achieves strong overall performance compared to prior approaches, while reducing computational overhead and eliminating
privacy risks.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Gecko: An Efficient Neural Architecture Inherently Processing Sequences with Arbitrary Lengths
Xuezhe Ma, Shicheng Wen, Linghao Jin et al. (+11 more)
Designing a unified neural network to efficiently and inherently process sequential data with arbitrary lengths is a central and challenging problem in sequence modeling. The design choices in
Transformer, including quadratic complexity and weak length extrapolation, have limited their ability to scale to long sequences. In this work, we propose Gecko, a neural architecture that inherits the design of Mega and Megalodon (exponential moving average with gated
attention), and further introduces multiple technical components to improve its capability to capture long range dependencies, including timestep decay normalization, sliding chunk
attention mechanism, and adaptive working
memory. In a controlled pretraining comparison with Llama2 and Megalodon in the scale of 7 billion parameters and 2 trillion training
tokens, Gecko achieves better efficiency and long-context scalability. Gecko reaches a training loss of 1.68, significantly outperforming Llama2-7B (1.75) and Megalodon-7B (1.70), and landing close to Llama2-13B (1.67). Notably, without relying on any context-extension techniques, Gecko exhibits inherent long-context processing and
retrieval capabilities, stably handling sequences of up to 4 million
tokens and retrieving information from contexts up to $4\times$ longer than its
attention window. Code: https://github.com/XuezheMax/gecko-llm
Read the full abstract ▾
All authors Xuezhe Ma, Shicheng Wen, Linghao Jin, Bilge Acun, Ruihang Lai, Bohan Hou, Will Lin, Hao Zhang, Songlin Yang, Ryan Lee, Mengxi Wu, Jonathan May, Luke Zettlemoyer, Carole-Jean Wu
Computation and Language · Cross-listed from cs.CV
Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMs
Weihao Hong, Zhiyuan Jiang, Bingyu Shen et al. (+4 more)
Vision-
Language Models (VLMs) are increasingly used in safety-critical applications that require reliable visual grounding. However, these models often hallucinate details that are not present in the image to satisfy user
prompts. While recent
datasets and
benchmarks have been introduced to evaluate systematic
hallucinations in VLMs, many
hallucination behaviors remain insufficiently characterized. In particular, prior work primarily focuses on object presence or absence, leaving it unclear how
prompt phrasing and structural constraints can systematically induce
hallucinations. In this paper, we investigate how different forms of
prompt pressure influence
hallucination behavior. We introduce Ghost-100, a procedurally generated
dataset of synthetic scenes in which key visual details are deliberately removed, enabling controlled analysis of absence-based
hallucinations. Using a structured 5-Level
Prompt Intensity Framework, we vary
prompts from neutral queries to toxic demands and rigid formatting constraints. We evaluate three representative open-weight VLMs: MiniCPM-V 2.6-8B, Qwen2-VL-7B, and Qwen3-VL-8B. Across all three models,
hallucination rates do not increase monotonically with
prompt intensity. All models exhibit reductions at higher intensity levels at different thresholds, though not all show sustained reduction under maximum coercion. These results suggest that current safety
alignment is more effective at detecting semantic hostility than structural coercion, revealing model-specific limitations in handling compliance pressure. Our
dataset is available at: https://github.com/bli1/tone-matters
Read the full abstract ▾
All authors Weihao Hong, Zhiyuan Jiang, Bingyu Shen, Xinlei Guan, Yangyi Feng, Meng Xu, Boyang Li
Computation and Language
LitVISTA: A Benchmark for Narrative Orchestration in Literary Text
Mingzhe Lu, Yiwen Wang, Yanbing Liu et al. (+8 more)
Computational narrative analysis aims to capture rhythm, tension, and emotional dynamics in literary texts. Existing large
language models can generate long stories but overly focus on causal coherence, neglecting the complex story arcs and orchestration inherent in human narratives. This suggests a structural misalignment between model- and human-generated narratives. We therefore position narrative analysis as a diagnostic proxy for generation and propose VISTA Space, a high-dimensional framework for narrative orchestration that unifies human and model perspectives while jointly characterizing narrative function and structure in a common space. We further introduce LitVISTA, a structurally annotated
benchmark grounded in literary texts, which operationalizes VISTA Space for systematic evaluation of models' narrative orchestration capabilities. Under an oracle setting with gold event anchors, we evaluate frontier LLMs including GPT, Claude, Grok, and Gemini. Results reveal systematic deficiencies, as current models struggle to jointly capture narrative function and structure and fail to form an integrated global view of literary narrative orchestration. End-to-end analysis further shows that failures are dominated by anchor identification and localization errors. Even advanced thinking modes yield mixed and often limited gains for literary narrative understanding.
Read the full abstract ▾
All authors Mingzhe Lu, Yiwen Wang, Yanbing Liu, Qi You, Chong Liu, Ruize Qin, Haoyu Dong, Wenyu Zhang, Jiarui Zhang, Yue Hu, Yunpeng Li
Computation and Language
Time Travel Engine: A Shared Latent Chronological Manifold Enables Historical Navigation in Large Language Models
Jingmin An, Wei Liu, Qian Wang et al. (+1 more)
Time functions as a fundamental dimension of human cognition, yet the mechanisms by which Large
Language Models (LLMs) encode chronological progression remain opaque. We demonstrate that temporal information in their latent space is organized not as discrete clusters but as a continuous, traversable geometry. We introduce the Time Travel Engine (TTE), an interpretability-driven framework that projects diachronic linguistic patterns onto a shared chronological
manifold. Unlike surface-level prompting, TTE directly modulates latent representations to induce coherent stylistic, lexical, and conceptual shifts aligned with target eras. By parameterizing diachronic
evolution as a continuous
manifold within the residual stream, TTE enables fluid
navigation through period-specific "zeitgeists" while restricting access to future knowledge. Furthermore, experiments across diverse architectures reveal topological isomorphism between the temporal subspaces of Chinese and English-indicating that distinct
languages share a universal
geometric logic of historical
evolution. These findings bridge historical linguistics with mechanistic interpretability, offering a novel paradigm for controlling temporal
reasoning in neural networks.
Read the full abstract ▾
All authors Jingmin An, Wei Liu, Qian Wang, Fang Fang
Computation and Language
NC-Bench: An LLM Benchmark for Evaluating Conversational Competence
Robert J. Moore, Sungeun An, Farhan Ahmed et al. (+1 more)
Existing LLM
benchmarks evaluate what models say, such as whether answers are correct, faithful, or helpful, but they do not test whether models produce the right type of conversational action at the right point in an interaction. The Natural Conversation
Benchmark (NC-Bench) fills this gap by evaluating conversational competence: the ability to perform structurally appropriate actions such as repairing, closing, or refusing, as defined by conversation science. Grounded in the Natural Conversation Framework (NCF), NC-Bench comprises three sets: (1) the basic set evaluates fundamental sequence management practices, such as answering inquiries, repairing responses, and closing conversational pairs; (2) the
retrieval-augmented generation (RAG) set applies the same patterns but incorporates information-seeking via RAG; (3) the complex request set extends to requests involving more intricate sequence management. Each set tests a model's ability to produce contextually appropriate conversational actions in response to characteristic interaction patterns. Evaluations across six open-source models and one closed-source model on 14 interaction patterns reveal quantifiable shortcomings in conversational competence present in current models. By operationalizing fundamental principles of human conversation, NC-Bench provides a lightweight, extensible, and theory-grounded framework for identifying specific conversational action gaps in LLMs beyond topical or task-specific
benchmarks.
Read the full abstract ▾
All authors Robert J. Moore, Sungeun An, Farhan Ahmed, Jay Pankaj Gala
Computation and Language
Sazia Tabasum Mim, Jack Morris, Manish Dhakal et al. (+3 more)
To explore a more scalable path for adding
multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason about its own informational needs and provide effective feedback to optimize a
multimodal model? To answer this, we propose a method that enables a
language agent to give feedback to a
vision-
language model (VLM) to adapt text generation to the
agent's preferences. Our results from different experiments affirm this hypothesis, showing that LLM preference feedback significantly enhances VLM descriptions. Using our proposed method, we find that the VLM can generate
multimodal scene descriptions to help the LLM better understand
multimodal context, leading to improvements of maximum 13% in absolute accuracy compared to the baseline
multimodal approach. Furthermore, a human study validated our AI-driven feedback, showing a 64.6% preference
alignment rate between the LLM's choices and human judgments. Extensive experiments provide insights on how and why the method works and its limitations.
Read the full abstract ▾
All authors Sazia Tabasum Mim, Jack Morris, Manish Dhakal, Yanming Xiu, Maria Gorlatova, Yi Ding
Computation and Language
Structured Episodic Event Memory
Zhengxuan Lu, Dongfang Li, Yukun Shi et al. (+3 more)
Current approaches to
memory in Large
Language Models (LLMs) predominantly rely on static
Retrieval-Augmented Generation (RAG), which often results in scattered
retrieval and fails to capture the structural dependencies required for complex
reasoning. For autonomous
agents, these passive and flat architectures lack the cognitive organization necessary to model the dynamic and associative nature of long-term interaction. To address this, we propose Structured Episodic Event
Memory (SEEM), a hierarchical framework that synergizes a graph
memory layer for relational facts with a dynamic episodic
memory layer for narrative progression. Grounded in cognitive frame theory, SEEM transforms interaction streams into structured Episodic Event Frames (EEFs) anchored by precise provenance pointers. Furthermore, we introduce an agentic associative fusion and Reverse Provenance Expansion (RPE) mechanism to reconstruct coherent narrative contexts from fragmented evidence. Experimental results on the LoCoMo and LongMemEval
benchmarks demonstrate that SEEM significantly outperforms baselines, enabling
agents to maintain superior narrative coherence and logical consistency.
Read the full abstract ▾
All authors Zhengxuan Lu, Dongfang Li, Yukun Shi, Beilun Wang, Longyue Wang, Baotian Hu
Computation and Language
Value of Information: A Framework for Human-Agent Communication
Yijiang River Dong, Tiancheng Hu, Zheng Hui et al. (+4 more)
Large
Language Model (LLM)
agents deployed for real-world tasks face a fundamental dilemma: user requests are underspecified, yet
agents must decide whether to act on incomplete information or interrupt users for clarification. Existing approaches either rely on brittle confidence thresholds that require task-specific tuning, or fail to account for the varying stakes of different decisions. We introduce a decision-theoretic framework that resolves this trade-off through the Value of Information (VoI), enabling
agents to dynamically weigh the expected utility gain from asking questions against the cognitive cost imposed on users. Our inference-time method requires no hyperparameter tuning and adapts seamlessly across contexts-from casual games to medical diagnosis. Experiments across four diverse domains (20 Questions, medical diagnosis, flight booking, and e-commerce) show that VoI consistently matches or exceeds the best manually-tuned baselines, achieving up to 1.36 utility points higher in high-cost settings. This work provides a parameter-free framework for adaptive
agent communication that explicitly balances task risk, query ambiguity, and user effort.
Read the full abstract ▾
All authors Yijiang River Dong, Tiancheng Hu, Zheng Hui, Caiqi Zhang, Ivan Vulić, Andreea Bobu, Nigel Collier
Computation and Language
Steer Model beyond Assistant: Controlling System Prompt Strength via Contrastive Decoding
Yijiang River Dong, Tiancheng Hu, Zheng Hui et al. (+1 more)
Large
language models excel at complex instructions yet struggle to deviate from their helpful assistant persona, as post-training instills strong priors that resist conflicting instructions. We introduce system
prompt strength, a training-free method that treats
prompt adherence as a continuous control. By contrasting logits from target and default system
prompts, we isolate and amplify the behavioral signal unique to the target persona by a scalar factor alpha. Across five diverse
benchmarks spanning constraint satisfaction, behavioral control, pluralistic
alignment, capability modulation, and stylistic control, our method yields substantial improvements: up to +8.5 strict accuracy on IFEval, +45pp refusal rate on OffTopicEval, and +13% steerability on
Prompt-Steering. Our approach enables practitioners to modulate system
prompt strength, providing dynamic control over model behavior without retraining.
Read the full abstract ▾
All authors Yijiang River Dong, Tiancheng Hu, Zheng Hui, Nigel Collier
Computation and Language · Cross-listed from cs.AI
BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation
Xin Guo, Rongjunchen Zhang, Guilong Lu et al. (+4 more)
Large
language models are becoming increasingly significant in financial applications. Nevertheless, prevailing
benchmarks are largely dependent on simulated or generic data, which leads to a significant gap between reported performance and actual efficacy in real-world scenarios. To tackle this challenge, we present BizFinBench.v2, the first integrated offline and online
benchmark built upon authentic user query-response data from both Chinese and U.S. equity
markets. It comprises 28,860 questions across eight offline and two online tasks. Experimental results show that GPT-5 achieves a mere 61.5% accuracy, still failing to meet the practical business requirement (84.8%). Among the evaluated commercial models, DeepSeek-R1 exhibits superior investment efficacy. Error analysis grounded in real financial practice reveals persistent limitations in existing models. By overcoming the constraints of prior
benchmarks, BizFinBench.v2 provides a substantiated foundation for advancing LLM deployment in the financial sector. Our data and code are available at https://github.com/HiThink-Research/BizFinBench.v2.
Read the full abstract ▾
All authors Xin Guo, Rongjunchen Zhang, Guilong Lu, Xuntao Guo, Shuai Jia, Zhi Yang, Liwen Zhang
Computation and Language
MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan
Sebastian Nehrdich and Kurt Keutzer
Ancient Buddhist literature features frequent, yet often unannotated, textual parallels spread across diverse
languages: Sanskrit, Pāli, Buddhist Chinese, Tibetan, and more. The scale of this material makes manual examination prohibitive. We present the MITRA framework, which consists of a novel pipeline for
multilingual parallel passage mining, MITRA-parallel, a large-scale corpus of 1.74 million parallel sentence pairs between Sanskrit, Chinese, and Tibetan, and the development of the domain-specific pretrained
language model Gemma 2 MITRA. We present Gemma 2 MITRA-MT, a version of this base model fine-tuned on machine translation tasks, reaching state-of-the-art performance for machine translation of these
languages into English and outperforming even much larger open-source models. We also present Gemma 2 MITRA-E, a semantic
embedding model that shows state-of-the-art performance on a novel, detailed semantic
embedding benchmark. We make the parallel
dataset, model weights, and semantic similarity
benchmark openly available to aid both NLP research and philological studies in Buddhist and classical Asian literature.
Read the full abstract ▾