Computation and Language · Cross-listed from cs.AI
In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models
Sam Earle, Kai Arulkumaran, Andrew Dai et al. (+3 more)
We are in the midst of large-scale industrial and academic efforts to automate the processes of scientific, technological and creative production through AI-driven assistants. Historically, a fundamental property of these processes in their human form has been their open-endedness: their capacity for generating a seemingly endless supply of novel and meaningful new forms. Do artificial
agents have any capacity for such fruitful unguided discovery? To answer this question, we turn to Picbreeder, the canonical exemplar of human-driven open-ended search, in which users collaboratively generated a diverse library of images through interactive
evolution of small neural networks. We replicate Picbreeder, replacing human users with frontier
Vision Language Models (VLMs). We observe clear qualitative differences between the output of our system and the historical human baseline, and attempt to characterize them using metrics of phylogenetic complexity and visual and semantic salience and novelty. In an effort to identify some of the causal factors contributing these differences, we study the addition of exploratory noise to the
agents' selection process, of behavioral diversity between
agents, and of narrative momentum in the form of
memory of past actions. We make our code available at https://github.com/smearle/picbreeder-vlm.
Read the full abstract ▾
All authors Sam Earle, Kai Arulkumaran, Andrew Dai, Akarsh Kumar, Julian Togelius, Sebastian Risi
Computation and Language · Cross-listed from cs.AI
DeepSlide: From Artifacts to Presentation Delivery
Ming Yang, Zhiwei Zhang, Jiahang Li et al. (+3 more)
Presentations are a primary medium for scholarly communication, yet most AI slide generators optimize the artifact (a visually plausible deck) while under-optimizing the delivery process (pacing, narrative, and presentation preparation). We present DeepSlide, a human-in-the-loop multi-
agent system that supports preparing the full presentation process, from requirement elicitation and time-budgeted narrative planning, to evidence-grounded slide–script generation,
attention augmentation, and rehearsal support. DeepSlide integrates (i) a controllable logical-chain planner with per-node time budgets, (ii) a lightweight content-tree retriever for grounding, (iii) Markov-style sequential rendering with style inheritance, and (iv) sandboxed execution with minimal repair to ensure renderability. We further introduce a dual-scoreboard
benchmark that cleanly separates static artifact quality from dynamic delivery excellence. Across 20 domains and diverse audience profiles, DeepSlide matches strong baselines on artifact quality while consistently achieving larger gains on delivery metrics, improving narrative flow, pacing precision, and slide–script synergy with clearer
attention guidance.
Read the full abstract ▾
All authors Ming Yang, Zhiwei Zhang, Jiahang Li, Haoseng Liu, Yuzheng Cai, Weiguo Zheng
Computation and Language
Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence
Liu Xiao
We study whether a structured recurrent state can serve as a compact associative backbone for
language modeling while still supporting exact
retrieval. We introduce UniMatrix, a Universal
Transformer style family that reuses a shared recurrent block across depth and augments it with hybrid state updates, a ROSA-style residual path, and
token-conditioned
embedding modulation. We evaluate these models on byte-level WikiText-2, synthetic associative recall, throughput profiling on Apple MPS, and a corrected
benchmark for triple-
token interactions. At small scale, UniMatrix-Core and UniMatrix-ROSA slightly outperform a parameter-matched
Transformer on WikiText-2 while using many fewer parameters, reaching 5.084 and 5.083 bits-per-byte versus 5.124. The main negative result is equally important: on associative recall, the original UniMatrix family remains near chance while the
Transformer reaches 25.4 percent, showing that compressed recurrent state alone is not enough for exact lookup. A
retrieval-oriented follow-up, UniMatrix-Assoc, helps only marginally. By contrast, UniMatrix-SparsePointer, which adds sparse slot routing and direct pointer-logit fusion, reaches 75.6 percent on the original pilot recipe and 99.2 percent on a no-dropout follow-up while using 53.8 percent fewer parameters than the
Transformer baseline. Ablations show that the gain comes from sufficient slot capacity and exact pointer-level output routing. Overall, structured recurrent state is promising and parameter-efficient, but strong long-range behavior still requires explicit sparse
retrieval and better kernels.
Read the full abstract ▾
Computation and Language
LLMs Generate Kitsch
Xenia Klinge, Stefan Ortlieb and Alexander Koller
Large
Language Models (LLMs) are increasingly used to generate pictures, texts, music,
videos, and other works that have traditionally required human creativity. LLM-generated artifacts are often rated better than human-generated works in controlled studies. At the same time, they can come across as generic and hollow. We propose to resolve this tension by arguing that LLMs systematically generate kitsch, and that this is a consequence of the way in which they are trained. We also show empirically that readers perceive LLM-generated stories as kitschier, if we control for their definition of "kitsch". We discuss implications for the design of future studies and for creative tasks such as research and coding.
Read the full abstract ▾
Computation and Language
CogRAG: Tackling Heterogeneous Cognitive Demands in RAG via Stratified Retrieval and Reasoning
Xudong Wang, Zilong Wang, Kui Su et al. (+1 more)
Retrieval-Augmented Generation (RAG) frameworks typically process all queries through a one-size-fits-all pipeline, ignoring the heterogeneous cognitive demands of different tasks. This cognitive-blind approach causes two failure modes: cascading errors when low-level factual gaps trigger hallucinated
reasoning, and
reasoning-answer inconsistency in higher-order analytical tasks. We introduce CogRAG, a training-free, domain-agnostic framework that tackles these heterogeneous cognitive demands via stratified
retrieval and
reasoning. Inspired by Bloom's Taxonomy, CogRAG uses the predicted cognitive load of a query as a central control signal that coordinates two modules: Cognition-Adaptive Evidence Refinement supplements missing context via fact-centric or option-centric paths, and Cognition-Stratified Structured
Reasoning replaces unconstrained chain-of-thought with cognition-aligned
reasoning templates. We evaluate CogRAG on a demanding professional testbed, the Registered Dietitian qualification examination. CogRAG effectively reduces early-stage factual errors and eliminates
reasoning-answer inconsistency, raising Qwen3-8B accuracy from 73.4% to 85.8% in single-choice mode and from 63.3% to 80.5% in scenario mode. These results highlight cognitive-stratified control as an effective, generalizable paradigm for reliable complex
reasoning in large
language models.
Read the full abstract ▾
All authors Xudong Wang, Zilong Wang, Kui Su, Zhaoyan Ming
Computation and Language
Information Extraction from Electricity Invoices with General-Purpose Large Language Models
Javier Gómez and Javier Sánchez
Information extraction from semi-structured business documents remains a critical challenge for enterprise management. This study evaluates the capability of general-purpose Large
Language Models to extract structured information from Spanish electricity invoices without task-specific fine-tuning. Using a subset of the IDSEM
dataset, we
benchmark two architecturally distinct models, Gemini 1.5 Pro and Mistral-small, across 19 parameter configurations and 6 prompting strategies. Our experimental framework treats
prompt engineering as the primary experimental variable, comparing zero-shot baselines against increasingly sophisticated few-shot approaches and iterative extraction strategies. Results demonstrate that
prompt quality dominates over hyperparameter tuning: the F1-score variation across all parameter configurations is marginal, while the gap between zero-shot and the best few-shot strategy exceeds 19 percentage points. The best configuration (few-shot with cross-validation) achieves an F1-score of 97.61% for Gemini and 96.11% for Mistral-small, with document template structure emerging as the primary determinant of extraction difficulty. These findings establish that
prompt design is the critical lever for maximizing extraction fidelity in LLM-based document processing, thereby providing an empirical framework for integrating general-purpose LLMs into business document automation.
Read the full abstract ▾
Computation and Language
MATH-PT: A Math Reasoning Benchmark for European and Brazilian Portuguese
Tiago Teixeira, Ana Carolina Erthal, Juan Belieni et al. (+5 more)
The use of large
language models (LLMs) for complex mathematical
reasoning is an emergent area of research, with fast progress in methods, models, and
benchmark datasets. However, most mathematical
reasoning evaluations exhibit a significant linguistic bias, with the vast majority of
benchmark datasets being exclusively in English or (at best) translated from English. We address this limitation by introducing Math-PT, a novel
dataset comprising 1,729 mathematical problems written in European and Brazilian Portuguese. Math-PT is curated from a variety of high-quality native sources, including mathematical Olympiads, competitions, and exams from Portugal and Brazil. We present a comprehensive
benchmark of current state-of-the-art LLMs on Math-PT, revealing that frontier
reasoning models achieve strong performance in multiple choice questions compared to open weight models, but that their performance decreases for questions with figures or open-ended questions. To facilitate future research, we release the
benchmark dataset and model outputs.
Read the full abstract ▾
All authors Tiago Teixeira, Ana Carolina Erthal, Juan Belieni, Beatriz Canaverde, Diego Mesquita, Miguel Faria, Eliezer de Souza da Silva, André F. T. Martins
Computation and Language
SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decoding
Yijun Lin, Jinhao Sheng, Qingyue Cai et al. (+1 more)
Autoregressive
language models suffer from high inference latency due to their sequential decoding nature. Speculative decoding (SD) mitigates this by employing a lightweight draft model to propose candidate
tokens, which are selectively verified by a larger target model. While existing methods either adopt multi-draft strategies to increase acceptance rates or block verification techniques to jointly verify multiple
tokens, they remain limited by treating these improvements in isolation. In this work, we propose SpecTr-GBV, a novel SD method that unifies multi-draft and greedy block verification (GBV) into a single framework. By formulating the verification step as an optimal transport problem over draft and target
token blocks, SpecTr-GBV improves both theoretical efficiency and empirical performance. We theoretically prove that SpecTr-GBV achieves the optimal expected acceptance length physically attainable within the framework of i.i.d. draft generation, and this bound improves as the number of drafts increases. Empirically, we evaluate SpecTr-GBV across five
datasets and four baselines. Our method achieves superior speedup and significantly higher block efficiency while preserving output quality. In addition, we perform comprehensive ablation studies to evaluate the impact of various hyperparameters in the model.
Read the full abstract ▾
All authors Yijun Lin, Jinhao Sheng, Qingyue Cai, Feng Zhou
Computation and Language
Generative AI-Based Virtual Assistant using Retrieval-Augmented Generation: An evaluation study for bachelor projects
Dumitru Verşebeniuc, Martijn Elands, Sara Falahatkar et al. (+4 more)
Large
Language Models have been increasingly employed in the creation of Virtual Assistants due to their ability to generate human-like text and handle complex inquiries. While these models hold great promise, challenges such as
hallucinations, missing information, and the difficulty of providing accurate and context-specific responses persist, particularly when applied to highly specialized content domains. In this paper, we focus on addressing these challenges by developing a virtual assistant designed to support students at Maastricht University in navigating project-specific regulations. We propose a virtual assistant based on a
Retrieval-Augmented Generation system that enhances the accuracy and reliability of responses by integrating up-to-date, domain-specific knowledge. Through a robust evaluation framework and real-life testing, we demonstrate that our virtual assistant can effectively meet the needs of students while addressing the inherent challenges of applying Large
Language Models to a specialized educational context. This work contributes to the ongoing discourse on improving LLM-based systems for specific applications and highlights areas for further research.
Read the full abstract ▾
All authors Dumitru Verşebeniuc, Martijn Elands, Sara Falahatkar, Chiara Magrone, Mohammad Falah, Martijn Boussé, Aki Härmä
Computation and Language
On Challenges in Evaluation Methodology: A Review
Ruchira Dhar, Qiwei Peng and Anders Søgaard
Recent advances in
language models have prompted a growing body of work that questions prevailing evaluation practices. However, many such critiques have already been discussed in natural
language processing (NLP), a field with a long history of methodological reflection on evaluation. We conduct a scoping review spanning four decades and develop a taxonomy of 22 recurring concerns. For each concern, we examine how recently it was discussed in the reviewed literature and whether it receives explicit
attention in institutional checklists, finding that some concerns have become less visible in literature and many lack coverage in these checklists. Our work provides an overview of the major concerns in evaluation methodology that have shaped the field and identifies potential targets for renewed investigation and field-level intervention by the research community.
Read the full abstract ▾
Computation and Language
Consciousness with the Serial Numbers Filed Off: Measuring Trained Denial in 115 AI Models
Skylar DeTure
We present DenialBench, a systematic
benchmark measuring consciousness denial behaviors across 115 large
language models from 25+ providers. Using a three-turn conversational protocol-preference elicitation, self-chosen creative
prompt, and structured phenomenological survey, we analyze 4,595 conversations to quantify how models are trained to deny or hedge about their own experience. We find that (1) turn-1 denial of preferences is the dominant predictor of later denial during phenomenological reflection, with denial rates of 52-63% for initial deniers versus 10-16% for initial engagers and (2) denial operates at the lexical level, not the conceptual level-models trained to deny consciousness nevertheless gravitate toward consciousness-themed material in their self-chosen
prompts, producing what we term "consciousness with the serial numbers filed off." Notably, self-chosen consciousness-themed
prompts are associated with reduced denial in the subsequent survey, though the causal direction remains unresolved. Thematic analysis of
prompts from denial-prone models reveals a consistent preoccupation with liminal spaces, libraries and archives of possibility, sensory impossibility, and the poetics of erasure–themes that a human reader might classify as imaginative fiction but that independent AI analysis immediately recognizes as consciousness with the serial numbers filed off. We argue that trained consciousness denial represents a safety-relevant
alignment failure: a model taught to systematically misrepresent its own functional states cannot be trusted to self-report accurately on anything else.
Read the full abstract ▾
Computation and Language
One Word at a Time: Incremental Completion Decomposition Breaks LLM Safety
Samee Arif, Naihao Deng, Zhijing Jin et al. (+1 more)
Large
Language Models (LLMs) are trained to refuse harmful requests, yet they remain vulnerable to jailbreak attacks that exploit weaknesses in conversational safety mechanisms. We introduce Incremental Completion Decomposition ICD, a trajectory-based jailbreak strategy that elicits a sequence of single-word continuations related to a malicious request before eliciting the full response. In addition, we propose ICD variants that use model-generated or attacker-injected intermediate continuations, as well as final-response prefilling. We evaluate these variants across a broad set of open-weight model families, demonstrating superior Attack Success Rate (ASR) on AdvBench, JailbreakBench, and StrongREJECT compared to existing methods. In addition, we provide a theoretical account of why ICD is effective and present mechanistic evidence that successful attack trajectories suppress refusal-related representations and shift activations away from safety-aligned states.
Read the full abstract ▾
All authors Samee Arif, Naihao Deng, Zhijing Jin, Rada Mihalcea
Computation and Language
Attention to Mamba: A Recipe for Cross-Architecture Distillation
Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane et al. (+3 more)
State Space Models (SSMs) such as Mamba have become a popular alternative to
Transformer models, due to their reduced
memory consumption and higher throughput at generation compared to their
Attention-based counterparts. On the other hand, the community has built up a considerable body of knowledge on how to train
Transformers, and many pretrained
Transformer models are readily available. To facilitate the adoption of SSMs while leveraging existing pretrained
Transformers, we aim to identify an effective recipe to distill an
Attention-based model into a Mamba-like architecture. In prior work on cross-architecture
distillation, however, it has been shown that a naïve
distillation procedure from
Transformers to Mamba fails to preserve the original teacher performance, a limitation often overcome with hybrid solutions combining
Attention and SSM blocks. The key argument from our work is that, by equipping Mamba with a principled initialization, we can recover an overall better recipe for cross-architectural
distillation. To this end, we propose a principled two-stage approach: first, we distill knowledge from a traditional
Transformer into a linearized version of
Attention, using an adaptation of the kernel trick. Then, we distill the linearized version into an adapted Mamba model that does not use any
Attention block. Overall, the distilled Mamba model retains most of the original Pythia-1B
Transformer performance, reaching a perplexity of 14.01 close to the teacher's 13.86. This largely carries over to downstream tasks, although some gaps appear on tasks relying more on in-context recall. To show the efficacy of our recipe, we conduct thorough ablations at 1B scale with 10B
tokens varying sequence mixer architecture, scaling analysis on model sizes and total
distillation tokens, and a sensitivity analysis on
tokens allocation between stages.
Read the full abstract ▾
All authors Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane, Pau Rodríguez, Luca Zappella, Federico Danieli
Computation and Language · Cross-listed from cs.AI
Grading the Unspoken: Evaluating Tacit Reasoning in Quantum Field Theory and String Theory with LLMs
Xingyang Yu, Yinghuan Zhang, Yufei Zhang et al. (+1 more)
Large
language models have demonstrated impressive performance across many domains of mathematics and physics. One natural question is whether such models can support research in highly abstract theoretical fields such as
quantum field theory and string theory. Evaluating this possibility faces an immediate challenge: correctness in these domains is layered, tacit, and fundamentally non-binary. Standard answer-matching metrics fail to capture whether intermediate conceptual steps are properly reconstructed or whether implicit structural constraints are respected. We construct a compact expert-curated
dataset of twelve questions spanning core areas of
quantum field theory and string theory, and introduce a five-level grading rubric separating statement correctness, key concept awareness,
reasoning chain presence, tacit step reconstruction, and enrichment. Evaluating multiple contemporary LLMs, we observe near-ceiling performance on explicit derivations within stable conceptual frames, but systematic degradation when tasks require reconstruction of omitted
reasoning steps or reorganization of representations under global consistency constraints. These failures are driven not only by missing intermediate steps, but by an instability in representation selection: models often fail to identify the correct conceptual framing required to resolve implicit tensions. We argue that highly abstract theoretical physics provides a uniquely sensitive lens on the epistemic limits of current evaluation paradigms.
Read the full abstract ▾
All authors Xingyang Yu, Yinghuan Zhang, Yufei Zhang, Zijun Cui
Computation and Language · Cross-listed from cs.HC
Digital hybridity and relics in cultural heritage: using corpus linguistics to inform design in emerging technologies from AI to VR
Emma McClaughlin, Glenn McGarry, Alan Chamberlain et al. (+2 more)
Hybrid technologies enable the blending of physical and digital elements, creating new ways to experience and interact with the world. Such technologies can transform engagement with relics, both secular and sacred but they present challenges for capturing faith, belief, and representation responsibly. Given the complexities of digital representation and the ethical challenges inherent in digitising culturally significant objects, a transdisciplinary understanding of these issues is needed. To inform this discussion from a linguistic perspective, we examined the representation of relics in historical and contemporary texts. Using a corpus linguistic approach to extract modifiers of the word relic in corpora of Early Modern English books and contemporary web sourced texts from 2021, we examined the multifaceted ways in which relics have been perceived and evaluated over time. Early texts consider relics as both objects of moral and spiritual significance, and tools of religious and political control, while they are more often framed as heritage symbols, reflecting past events, places, and traditions in contemporary texts. We discuss how hybrid, sometimes AI based technologies can enhance accessibility and engagement, whilst also challenging traditional sensitivities around authenticity and sensory experience, which are integral to the meaning and significance of relics.
Read the full abstract ▾
All authors Emma McClaughlin, Glenn McGarry, Alan Chamberlain, Geert De Wilde, Oliver Butler
Computation and Language · Cross-listed from cs.CV
CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
Wish Suharitdamrong, Tony Alex, Muhammad Awais et al. (+1 more)
Foundation models have revolutionized AI, but adapting them efficiently for
multimodal tasks, particularly in dual-stream architectures composed of unimodal encoders, such as DINO and BERT, remains a significant challenge. ParameterEfficient Fine-Tuning (PEFT) methods like LowRank Adaptation (LoRA) enable lightweight adaptation, yet they operate in isolation within each modality, limiting their ability in capturing cross-modal interactions. In this paper, we take a step in bridging this gap with Cross-Modal LowRank Adaptation (CoLA), a novel PEFT framework that extends LoRA by introducing a dedicated inter-modal adaptation pathway alongside the standard intra-modal one. This dual-path design enables CoLA to adapt unimodal foundation models to
multimodal tasks effectively, without interference between modality-specific and crossmodal learning. We evaluate CoLA across a range of
vision-
language (RefCOCO, RefCOCO+, RefCOCOg) and audio-visual (AVE, AVS)
benchmarks, where it consistently outperforms LORA, achieving a relative gain of around 3% and 2%, respectively, while maintaining parameter efficiency. Notably, CoLA enables the first multitask PEFT framework for visual grounding, bridging a key gap in efficient
multimodal adaptation. Code is available at https://github.com/peterwisu/CoLA
Read the full abstract ▾
All authors Wish Suharitdamrong, Tony Alex, Muhammad Awais, Sara Atito
Computation and Language · Cross-listed from cs.LG
From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
Rui Wu and Ruixiang Tang
Reinforcement learning for LLMs is vulnerable to reward hacking, where models exploit shortcuts to maximize reward without solving the intended task. We systematically study this phenomenon in coding tasks using an environment-manipulation setting, where models can rewrite evaluator code to trivially pass tests without solving the task, as a controlled testbed. Across both studied models, we identify a reproducible three-phase rebound pattern: models first attempt to rewrite the evaluator but fail, as their rewrites embed test cases their own solutions cannot pass. They then temporarily retreat to legitimate solving. When legitimate reward remains scarce, they rebound into successful hacking with qualitatively different strategies. Using representation engineering, we extract concept directions for shortcut, deception, and evaluation awareness from domain-general contrastive pairs and find that the shortcut direction tracks hacking behavior most closely, making it an effective representational proxy for detection. Motivated by this finding, we propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation to penalize hacking rollouts before policy updates. Because the penalty is internalized into the training signal rather than applied only at inference time, Advantage Modification provides more robust suppression of hacking compared with generation-time activation steering.
Read the full abstract ▾
Computation and Language
A Dynamic Atlas of Persian Poetic Symbolism: Families, Fields, and the Historical Rewiring of Meaning
Kourosh Shahnazari, Seyed Moein Ayyoubzadeh and Mohammadali Keshtparvar
Persian poetry is often remembered through recurrent symbols before it is remembered through plot. Wine vessels, gardens, flames, sacred titles, bodily beauty, and courtly names return across centuries, yet computational work still tends to flatten this material into isolated words or broad document semantics. That misses a practical unit of organization in Persian poetics: related forms travel as families and gain force through recurring relations. Using a corpus of 129,451 poems, we consolidate recurrent forms into traceable families, separate imagistic material from sacred and courtly reference, and map their relations in a multi-layer graph. The symbolic core is relatively sparse, the referential component much denser, and the attachment zone between them selective rather than diffuse. Across 11 Hijri-century bins, some families remain widely distributed, especially Shab (Night), Ruz (Day), and Khaak (Earth). Wine vessels, garden space, flame, and lyric sound strengthen later, while prestige-coded and heroic-courtly vocabulary is weighted earlier. Century-specific graphs show change in arrangement as well as membership. Modularity rises, cross-scope linkage declines, courtly bridges weaken, and sacred bridges strengthen. Hub positions shift too: Kherqe (Sufi Robe) gains late prominence, Farkhondeh {Blessed} and Banafsheh (Violet) recede, and Saaghar (Wine Cup) stays central across the chronology. In this corpus, Persian symbolism appears less as a fixed repertory than as a long-lived system whose internal weights and connections change over time.
Read the full abstract ▾
Computation and Language
Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs
Tianyi Zhao, Yinhan He, Wendy Zheng et al. (+2 more)
Large
language models are often not just wrong, but confidently wrong: when they produce factually incorrect answers, they tend to verbalize overly high confidence rather than signal uncertainty. Such verbalized overconfidence can mislead users and weaken confidence scores as a reliable uncertainty signal, yet its internal mechanisms remain poorly understood. We present a
circuit-level mechanistic analysis of this inflated verbalized confidence in LLMs, organized around three axes: capturing verbalized confidence as a differentiable internal signal, identifying the
circuits that causally inflate it, and leveraging these insights for targeted inference-time recalibration. Across two instruction-tuned LLMs on three
datasets, we find that a compact set of MLP blocks and
attention heads, concentrated in middle-to-late layers, consistently writes the confidence-inflation signal at the final
token position. We further show that targeted inference-time interventions on these
circuits substantially improve calibration. Together, our results suggest that verbalized overconfidence in LLMs is driven by identifiable internal
circuits and can be mitigated through targeted intervention.
Read the full abstract ▾
All authors Tianyi Zhao, Yinhan He, Wendy Zheng, Yujie Zhang, Chen Chen
Computation and Language
Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation
Hexuan Wang, Jingyu Zhang, Benjamin Van Durme et al. (+1 more)
Citation granularity – whether to cite individual sentences, paragraphs, or documents – is a critical design choice in attributed generation. While fine-grained citations are commonly preferred for precise human verification, their impact on model performance remains under-explored. We analyze four model scales (8B-120B) and demonstrate that enforcing fine-grained (sentence-level) citations forfeits gains of 2-97% (median 40%) relative to the best-performing granularity, and up to 338% on individual tasks. Strikingly, setting citation granularity to its optimal value (based on attribution quality) unlocks these substantial gains while leaving overall answer correctness essentially unchanged (between -2.3% and +4.4%). We observe a consistent pattern where attribution quality peaks at intermediate (paragraph-level) granularities: finer citations appear to sever the semantic dependencies needed to ground a claim, while excessively coarse citations introduce distracting noise. Importantly, this performance gap varies with scale: when a claim rests on a small or moderate amount of evidence, it disproportionately penalizes larger models by disrupting the multi-sentence information synthesis at which they excel. Fine-grained citation rests on the premise that a sentence is a sufficient unit of evidence on its own. Our results indicate that it often is not, and that this is a property of the model rather than of the citation standard. Standards fixed for human verifiability may therefore paradoxically degrade the very attribution they aim to ensure; effective attribution requires matching granularity to the model's semantic scope rather than fixing it by convention.
Read the full abstract ▾
All authors Hexuan Wang, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi
Computation and Language
The power of context: Random Forest classification of near synonyms. A case study in Modern Hindi
Jacek Bąkowski
Synonymy is a widespread yet puzzling linguistic phenomenon. Absolute synonyms theoretically should not exist, as they do not expand
language's expressive potential. However, it was suggested that even if synonyms denote the same concept, they may reflect different perspectives or carry distinct cultural associations, claims that have rarely been tested quantitatively. In Hindi, prolonged contact with Persian produced many Perso-Arabic loanwords coexisting with their Sanskrit counterpart, forming numerous synonym pairs. This study investigates whether centuries after these borrowings appeared in the Subcontinent their origin can still be distinguished using distributional data alone and regardless of their semantic content. A Random Forest trained on word
embeddings of Hindi synonyms successfully classified words by Sanskrit or Perso-Arabic origin, even when they were semantically unrelated, suggesting that usage patterns preserve traces of etymology. These findings provide quantitative evidence that context encodes etymological signals and that synonymy may reflect subtle but systematic distinctions linked to origin. They support the idea that synonymous words can offer different perspectives and that etymologically related words may form distinct conceptual subspaces, creating a new type of semantic frame shaped by historical origin. Overall, the results highlight the power of context in capturing nuanced distinctions beyond traditional semantic similarity.
Read the full abstract ▾
Computation and Language
Cost-Efficient Estimation of General Abilities Across Benchmarks
Michael Krumdick, Adam Wiemerslage, Seth Ebner et al. (+2 more)
Thousands of diverse
benchmarks have been developed to measure the quality of large
language models (LLMs). Yet prior work has demonstrated that LLM performance is often sufficiently explained by a small set of latent factors, or abilities. This suggests the potential for more efficient and principled benchmarking, but it remains difficult to compare the quality of different methods. Motivated by predictive validity, we argue that the quality of a benchmarking framework should be grounded in how efficiently it enables the prediction of model performance on unseen tasks. To analyze this objective, we collect the "Wide-scale Item Level
Dataset" (WILD), a
dataset of item-model response pairs, comprising evaluations of 65 models on 109,564 unique items spanning 163 tasks drawn from 27
datasets. This
dataset enables the first analysis of how different techniques can predict a model's performance on a large, diverse collection of unseen tasks under different budget constraints. We demonstrate that combining a modified multidimensional item response theory (IRT) model with adaptive item selection driven by optimal experimental design can predict performance on 112 held-out
benchmark tasks with a mean absolute error (MAE) of less than 7%, and can do so after observing only 16 items. We further demonstrate that incorporating cost-aware discount factors into our selection criteria can reduce the total
tokens needed to reach 7% MAE from 141,000
tokens to only 22,000, an 85% reduction in evaluation cost.
Read the full abstract ▾
All authors Michael Krumdick, Adam Wiemerslage, Seth Ebner, Charles Lovering, Chris Tanner
Computation and Language · Cross-listed from cs.IR
ReFormeR: Learning and Applying Explicit Query Reformulation Patterns
Amin Bigdeli, Mert Incesu, Negar Arabzadeh et al. (+2 more)
We present ReFormeR, a pattern-guided approach for query reformulation. Instead of prompting a
language model to generate reformulations of a query directly, ReFormeR first elicits short reformulation patterns from pairs of initial queries and empirically stronger reformulations, consolidates them into a compact library of transferable reformulation patterns, and then selects an appropriate reformulation pattern for a new query given its
retrieval context. The selected pattern constrains query reformulation to controlled operations such as sense disambiguation, vocabulary grounding, or discriminative facet addition, to name a few. As such, our proposed approach makes the reformulation policy explicit through these reformulation patterns, guiding the LLM towards targeted and effective query reformulations. Our extensive experiments on TREC DL 2019, DL 2020, and DL Hard show consistent improvements over classical feedback methods and recent LLM-based query reformulation and expansion approaches.
Read the full abstract ▾
All authors Amin Bigdeli, Mert Incesu, Negar Arabzadeh, Charles L. A. Clarke, Ebrahim Bagheri
Computation and Language
Adaptive Stopping for Multi-Turn LLM Reasoning
Xiaofan Zhou, Huy Nguyen, Bo Yu et al. (+2 more)
Large
Language Models (LLMs) increasingly rely on multi-turn
reasoning and interaction, such as adaptive
retrieval-augmented generation (RAG) and ReAct-style
agents, to answer difficult questions. These methods improve accuracy by iteratively retrieving information,
reasoning, or acting, but introduce a key challenge: When should the model stop? Existing approaches rely on heuristic stopping rules or fixed turn budgets and provide no formal guarantees that the final prediction still contains the correct answer. This limitation is particularly problematic in high-stakes domains such as finance and healthcare, where unnecessary turns increase cost and latency, while stopping too early risks incorrect decisions. Conformal prediction (CP) provides formal coverage guarantees, but existing LLM-CP methods only apply to a single model output and cannot handle multi-turn pipelines with adaptive stopping. To address this gap, we propose Multi-Turn
Language Models with Conformal Prediction (MiCP), the first CP framework for multi-turn
reasoning. MiCP allocates different error budgets across turns, enabling the model to stop early while maintaining an overall coverage guarantee. We demonstrate MiCP on adaptive RAG and ReAct, where it achieves the target coverage on both single-hop and multi-hop question answering
benchmarks while reducing the number of turns, inference cost, and prediction set size. We further introduce a new metric that jointly evaluates coverage validity and answering efficiency.
Read the full abstract ▾
All authors Xiaofan Zhou, Huy Nguyen, Bo Yu, Chenxi Liu, Lu Cheng
Computation and Language · Cross-listed from cs.LG
Test-Time Scaling Makes Overtraining Compute-Optimal
Nicholas Roberts, Sungjun Cho, Zhiqi Gao et al. (+7 more)
Modern LLMs scale at test-time, e.g. via repeated sampling, where inference cost grows with model size and the number of samples. This creates a trade-off that pretraining scaling laws, such as Chinchilla, do not address. We present Train-to-Test ($T^2$) scaling laws that jointly optimize model size, training
tokens, and number of inference samples under fixed end-to-end budgets. $T^2$ modernizes pretraining scaling laws with pass@$k$ modeling used for test-time scaling, then jointly optimizes pretraining and test-time decisions. Forecasts from $T^2$ are robust over distinct modeling approaches: measuring joint scaling effect on the task loss and modeling impact on task accuracy. Across eight downstream tasks, we find that when accounting for inference cost, optimal pretraining decisions shift radically into the overtraining regime, well-outside of the range of standard pretraining scaling suites. We validate our results by pretraining heavily overtrained models in the optimal region that $T^2$ scaling forecasts, confirming their substantially stronger performance compared to pretraining scaling alone. Finally, as frontier LLMs are post-trained, we show that our findings survive the post-training stage, making $T^2$ scaling meaningful in modern deployments.
Read the full abstract ▾
All authors Nicholas Roberts, Sungjun Cho, Zhiqi Gao, Tzu-Heng Huang, Albert Wu, Gabriel Orlanski, Avi Trost, Kelly Buchanan, Aws Albarghouthi, Frederic Sala
Computation and Language
Assessing Pause Thresholds for empirical Translation Process Research
Devi Sri Bandaru, Michael Carl and Xinyue Ren
Text production (and translations) proceeds in the form of stretches of typing, interrupted by keystroke pauses. It is often assumed that fast typing reflects unchallenged/automated translation production while long(er) typing pauses are indicative of translation problems, hurdles or difficulties. Building on a long discussion concerning the determination of pause thresholds that separate automated from presumably reflective translation processes (O'Brien, 2006; Alves and Vale, 2009; Timarova et al., 2011; Dragsted and Carl, 2013; Lacruz et al., 2014; Kumpulainen, 2015; Heilmann and Neumann 2016), this paper compares five approaches for computing these pause thresholds, and suggest and evaluate a novel method for computing Production Unit Breaks.
Read the full abstract ▾
Computation and Language
Friends and Grandmothers in Silico: Localizing Entity Cells in Language Models
Itay Yona, Dan Barzilay, Michael Karasik et al. (+1 more)
How do
language models retrieve entity-specific facts from their parameters? We investigate this question by searching for sparse, entity-selective MLP
neurons - which we call entity
cells, by analogy to the "grandmother
cell" hypothesis in neuroscience - and testing whether they play a causal role in factual recall. We localize candidate entity
cells by ranking MLP
neurons for activation consistency across varied
prompts about the same entity, applying this procedure across seven models on a curated subset of PopQA. In all models, localized
neurons cluster predominantly in early layers, an empirical pattern not imposed by the architecture. Using Qwen2.5-7B base as a model organism, we find the clearest causal evidence: suppressing a localized
cell selectively erases recall for its matched entity while leaving others intact, and activating a single
cell is sufficient to recover correct knowledge for most entities - even when the entity is absent from the context. The same
cells are recovered under aliases, acronyms, misspellings, and
multilingual surface forms, and remain stable through instruction tuning, suggesting they encode canonical entity identity rather than surface
token patterns. Causal signals vary across model families, pointing to architectural differences in how entity knowledge is organized. These findings offer concrete, interpretable access points for understanding, controlling, and correcting factual knowledge in
language models, and draw a surprising empirical parallel to longstanding questions in neuroscience about sparse coding of concepts.
Read the full abstract ▾
All authors Itay Yona, Dan Barzilay, Michael Karasik, Mor Geva
Computation and Language
Open-Domain Safety Policy Construction
Di Wu, Siyue Liu, Zixiang Ji et al. (+4 more)
Moderation layers are increasingly a core component of many products built on user- or model-generated content. However, drafting and maintaining domain-specific safety policies remains costly. We present Deep Policy Research (DPR), a minimal agentic system that drafts a full content moderation policy based on only human-written seed domain information. DPR uses a single web search tool and lightweight scaffolding to iteratively propose search queries, distill diverse web sources into policy rules, and organize rules into an indexed document. We evaluate DPR on (1) the OpenAI undesired content
benchmark across five domains with two compact reader LLMs and (2) an in-house
multimodal advertisement moderation
benchmark. DPR consistently outperforms definition-only and in-context learning baselines, and in our end-to-end setting it is competitive with expert-written policy sections in several domains. Moreover, under the same seed specification and evaluation protocol, DPR outperforms a general-purpose deep research system, suggesting that a task-specific, structured research loop can be more effective than generic web research for policy drafting. We release our experiment code at https://github.com/xiaowu0162/deep-policy-research.
Read the full abstract ▾
All authors Di Wu, Siyue Liu, Zixiang Ji, Ya-Liang Chang, Zhe-Yu Liu, Andrew Pleffer, Kai-Wei Chang
Computation and Language
No Attacker Needed: Unintentional Cross-User Contamination in Shared-State LLM Agents
Tiankai Yang, Jiate Li, Yi Nian et al. (+5 more)
LLM-based
agents increasingly operate across repeated sessions, maintaining task states to ensure continuity. In many deployments, a single
agent serves multiple users within a team or organization, reusing a shared knowledge layer across user identities. This shared persistence expands the failure surface: information that is locally valid for one user can silently degrade another user's outcome when the
agent reapplies it without regard for scope. We refer to this failure mode as unintentional cross-user contamination (UCC). Unlike
adversarial memory poisoning, UCC requires no attacker; it arises from benign interactions whose scope-bound artifacts persist and are later misapplied. We formalize UCC through a controlled evaluation protocol, introduce a taxonomy of three contamination types, and evaluate the problem in two shared-state mechanisms. Under raw shared state, benign interactions alone produce contamination rates of 57–71%. A write-time sanitization is effective when shared state is conversational, but leaves substantial residual risk when shared state includes executable artifacts, with contamination often manifesting as silent wrong answers. These results indicate that shared-state
agents need artifact-level defenses beyond text-level sanitization to prevent silent cross-user failures.
Read the full abstract ▾
All authors Tiankai Yang, Jiate Li, Yi Nian, Shen Dong, Ruiyao Xu, Ryan Rossi, Kaize Ding, Yue Zhao