Computation and Language
AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Content
David Ababio Awuni, Rose-Mary Owusuaa Mensah Gyening and Elvis Gyasi Owusu
AfriSyCo studies answer switching around African-
language factual content with two complementary layers: native-
language follow-ups and a controlled cross-
language factorial whose question, options, and target remain in the African
language while the follow-up framing is English. We analyze 1,415 turn-1-correct model-
language-item observations derived from 100 source questions across seven open-weight checkpoints and six
languages; turn-1-correct denotes observed first-response accuracy, not demonstrated knowledge. Under native
prompts, assertive endorsement produces 29.3 percentage points more any-turn false-target selection than mention-plus-verification (M+V), with a 19.0-point immediate T2 contrast. In the precommitted 2 x 2 factorial, averaged over three tested
prompt families, assertive framing increases target selection by 30.4 points (95% CI [28.4, 32.3]); verification decreases it by 17.4 points, while the assertive effect rises from 20.5 points without verification to 40.2 with it (interaction +19.7). The effect remains 34.8 points among 611 observations correct after option reordering. Magnitude varies sharply by wording and checkpoint:
prompt-family effects span 20.1-42.5 points, a Twi/Qwen3 paraphrase shifts target selection from 70.8% to 4.2%, and checkpoint effects span 9.2-47.0 points.
Prompt realization is therefore part of the measurement problem.
Read the full abstract ▾
Computation and Language
SFT or RL for Tool-Calling Agents? A Controlled Study Across Data, Method, and Scale
Md Tahmid Rahman Laskar, Xue-Yong Fu and Shashi Bhushan TN
Limited controlled evidence exists on how training data, adaptation method, and model scale jointly affect tool-calling performance in
language-model
agents. We evaluate supervised fine-tuning (SFT) with LoRA, reinforcement learning (RL) via Group Relative Policy Optimization (GRPO), and SFT followed by GRPO across six Qwen3 models from 0.6B to 32B parameters, covering both in-distribution performance and cross-
dataset transfer. SFT with LoRA is the strongest in-distribution method throughout the 0.6B-32B range and best in 15 out of 18 experimental settings. On cross-
dataset transfer, the methods are closer: GRPO wins 29 out of 54 settings where training and test
datasets differ, but its margin over SFT averages under one point, and SFT->GRPO is rarely strongest in either comparison.
Dataset mixing gives consistently strong transfer while staying close to specialized in-distribution training, regardless of method. Additional analysis further confirms that LoRA outperforms full-parameter fine-tuning, demonstrating that LoRA better preserves pretrained agentic behavior.
Read the full abstract ▾
Computation and Language
PrimeScientist: Strategic Allocation of Research Effort in Autonomous Research
Xinle Yu, Fan Bai, Kaiser Sun et al. (+5 more)
Autonomous research
agents aim to automate scientific workflows, from proposing ideas to conducting experiments and analyzing results. Yet current AI and research
agents can propose more directions than available resources allow them to pursue. Moreover, each attempt could consume substantial resources, requiring
agents to reconsider how to invest in subsequent research. Thus, deciding how to invest research effort strategically should be a defining capability of autonomous research
agents. Accordingly, we introduce PrimeScientist, which jointly determines research direction and resource investment across successive research attempts. Specifically, we formulate this challenge of strategic research effort allocation as a sequential decision problem where remaining resources should explicitly guide the research policy. We first introduce an executable plan tree that preserves competing plans and their outcomes across attempts. Building on this representation, we propose an adaptive MCTS-based allocation policy that balances exploration and exploitation using experimental feedback and remaining resources. Comprehensive evaluations across AI research, systems and code optimization, and machine learning engineering show that strategic allocation improves research quality and sample efficiency together. Across 12 AI research tasks, PrimeScientist improves average reward by 10.3% with 50.6% fewer research attempts than AutoResearch under the same resource budget. We believe making research effort allocation an explicit optimization target establishes effective resource use as a core research capability for autonomous
agents to drive scientific breakthroughs at scale.
Read the full abstract ▾
All authors Xinle Yu, Fan Bai, Kaiser Sun, Hengshuo Miao, Abhay Anand, Zhongyan Luo, Kun Zhou, Zhen Wang
Computation and Language
How Calibration Content Shapes Attention-Based Reranking
Petros Karypis, Hossein Rajaby Faghihi, Peter Chen et al. (+4 more)
Attention-based rerankers score documents by aggregating query-to-document
attention and subtracting a null-query calibration pass to remove positional and structural bias. Although widely used, this calibration assumes that the null pass removes irrelevant signal from each document. We show that modern
prompt content, e.g. constraints, instructions, personas, and demonstrations can violate this assumption when it enters the scoring readout, making the null pass relevance-aware rather than null. We find that calibration is especially harmful when applied to
prompts containing longer, more detailed instructions as the null-pass step removes relevant signal. Based on these findings, we propose interpolated null calibration, a training-free modification that controls how much of the instruction content enters the null baseline. It recovers
attention-based reranking performance on instruction-heavy tasks where standard calibration fails, while preserving calibration's benefits when the null pass remains relevance-agnostic. On instruction heavy tasks, the recovered rankings surpass generative rerankers. We also show that in-context demonstrations improve
attention-based reranking with little calibration interference, since demonstrations act only through the query pass and leave the null pass unchanged.
Read the full abstract ▾
All authors Petros Karypis, Hossein Rajaby Faghihi, Peter Chen, Rui Zhu, Noveen Sachdeva, Yan Zhu, Julian McAuley
Computation and Language
Is Luke the Author of a Gospel and the Acts of the Apostles?
Jacques Savoy
According to Christian tradition, Luke is credited with authoring a Gospel and the Acts of the Apostles, even if his name does not appear in either book, both originally written in Koine Greek. Several biblical scholars assume that both texts were written by a common author, while others deduce the presence of two authors. Different studies have been found to support either finding, some based on qualitative evaluation, while a few others consider the occurrence frequency differences between the two books. To propose an enhanced quantitative analysis, this study is grounded on two recent authorship attribution models. The Burrows' Delta, applied with eleven different feature sizes, demonstrates common authorship. An author verification model confirms this finding. The following experiments consider several stylistic representations, feature sizes, and distance functions to confirm that Luke is the true author of both books.
Read the full abstract ▾
Computation and Language
Jacques Savoy
The analysis of US political
language is usually based on the written form (e.g. presidential addresses) or posts broadcasted on various social networks. Oral production, however, which is even more frequent, can better reveal the style and mode of thinking of the speaker. This study covers this mode of linguistic communication by considering 19 candidates from the presidential elections between 1960 to 2024. Our main research objectives are to disclose the main trends hidden in those presidential debates. Do we observe a clear simplification of the US political
language over time? Does Trump have poor
language compared to the other candidates? Do unusual stylistic features occur only with a single, specific president? Moreover, can we detect a pattern explaining the success or failure of some nominees? Over time, this study demonstrates a significant reduction in political
language complexity, a decrease of the mean sentence length, and a noteworthy decline of complex terms. Moreover, the emotional tone increases over the decades, while logical and rational thinking tends to lessen.
Read the full abstract ▾
Computation and Language
Is Trump's Vocabulary Poor? Vocabulary Richness Across Texts of Different Lenghts
Dominique Labbe, Cyril Labbe and Jacques Savoy
This study explores the vocabulary richness of oral political communication. A model explaining the lexicon growth is proposed by subdividing the whole vocabulary into terms generated by general and specialized glossaries.
Computation and Language
Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents
Caiqi Zhang, Xiaochen Zhu, Chengzu Li et al. (+3 more)
Reliable confidence estimation is increasingly central to the trustworthy deployment of
language models: a calibrated estimate of the probability that an output is correct decides what to ship, what to escalate, and what to retry. Existing confidence
estimators, however, share one design premise: they only read the current inference process, either by introspecting on it, scoring its
token probabilities, or resampling it. We argue that the current inference is not a sufficient basis for confidence. We propose XConf (eXperiential Confidence): estimating confidence together with the model's accumulated experience. The experience is stored as a record of the model's own graded past episodes, each holding the task, the model's reflection, its stated confidence, the outcome, and a lesson written once the grade arrived. Given a new task, XConf's Recall stage retrieves past episodes on similar tasks met with a similar stated confidence, and reads off their historical success rate; its Reflect stage shows the model this record, has it name its recurring failure mode, and restate a confidence now informed by its own track records. Our
estimator is format-general, requiring no logit access or weight updates, and costs only one answer generation. Across nine
benchmarks spanning
reasoning, coding,
multimodal QA, and interactive
agents, and four models from three families, XConf beats or matches ten-sample self-consistency in discrimination (AUROC) on 23 of 24 comparisons, with much lower calibration error (ECE), at a tenth of the generation cost. Used for selective prediction, abstaining on the 10% least-confident episodes raises the delivered success rate by up to 8.7 points on
agent tasks. We therefore see experiential confidence estimation as a new paradigm for future general-purpose confidence estimation.
Read the full abstract ▾
All authors Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Yulong Chen, Dharshan Kumaran, Nigel Collier
Computation and Language · Cross-listed from cs.AI
NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation
Johnny Greco, Nabin Mulepati, Andre Manoel et al. (+8 more)
We present NeMo Data Designer (NDD), an open-source, general-purpose framework for multi-modal synthetic data generation (SDG). Designed to be intuitive to use, NDD provides a declarative configuration format in which human and/or
agent users define each
dataset column, with column types spanning text, code, structured outputs, images,
embeddings, and statistical samplers that are explicitly configured to steer
dataset diversity. Additional column types and functionality can be introduced using the framework's flexible plugin system. NDD's configuration is an inspectable artifact, supporting workflow sharing and reproducibility. SDG is an inherently iterative process. NDD therefore builds a preview-and-revision loop into its core workflow, allowing users to generate and inspect a small number of records, refine the specification, and rerun generation at full scale. At runtime, NDD resolves dependencies, schedules calls to user-provided model endpoints, and retries failed requests. We describe NDD's architecture and programming model and present case studies spanning structured, agentic,
multimodal, and domain-specialized tasks, including
datasets used in Nemotron model development and in production enterprise deployments.
Read the full abstract ▾
All authors Johnny Greco, Nabin Mulepati, Andre Manoel, Eric Tramel, Kirit Thadaka, Mike Knepper, Dhruv Nathawani, Dane Corneil, Yev Meyer, Alex Watson, Maarten Van Segbroeck
Computation and Language · Cross-listed from cs.AI
GraphEcho: Structural Redundancy and Evidence Provenance in LLM Graph Agents
Sikun Wang, Yixi Zhou, Lei Fan et al. (+1 more)
A large
language model (LLM)
agent can follow more graph paths without acquiring more independent evidence. GraphEcho tests whether
agents mistake these repeated encounters for additional corroboration. The
benchmark varies path counts and evidential origins while holding evidence content fixed, and evaluates both judgments and active exploration. Controlled synthetic experiments reveal model-dependent judgment shifts, but redundant supporting paths increase the share of repeated walks across all evaluated frozen
agents. Provenance-aware post-training (PAPT) reduces revisits and improves synthetic accuracy, yet covers fewer distinct sources. On scientific claims, it continues to reduce repetition while accuracy declines. These findings expose a gap between efficient exploration and effective evidence use: an
agent can learn to stop repeating itself while overlooking information it needs. GraphEcho provides a controlled way to evaluate both what graph
agents conclude and whether their exploration reaches distinct evidential sources.
Read the full abstract ▾
All authors Sikun Wang, Yixi Zhou, Lei Fan, Fan Zhang
Computation and Language · Cross-listed from cs.LG
The Missing "I Don't Know": Why Three Reasoning-Reliability Findings Converge on Calibrated Abstention
Srijith Ravikumar
Three recent results describe what look like unrelated LLM reliability problems. Yin et al. (2026) show
reasoning RL collapses tool-reliability representations. Suleymanov et al. (2026) show that under safety-constrained generation, large models rewrite flagged spans while small models truncate. Bastounis et al. (2024) prove any consistent-
reasoning system without an implicit "I don't know" function must hallucinate infinitely often on broad problem classes. We argue these findings converge on a single intervention: calibrated abstention is what each independently identifies as the missing capability, even though the unavailability they document, a capability gap, a policy gap, and a recursion-theoretic gap, has a different source in each case. Honesty post-training has narrowed the gap in deployed models, but principled closure of the class Bastounis identifies requires a calibrated abstention function whose training signal at the leaderboard level is absent: dominant
benchmarks assign zero reward to decline, so the leaderboard gradient that would select for the function does not exist. We propose four changes to evaluation: triple-scoring, abstention-rate reporting, capability-stratified evaluation, and mandatory calibration metrics.
Benchmark reform is necessary, not sufficient, for closing the gap the
theorem identifies.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Vivek Kalyanarangan
When agentic sessions run to a million
tokens with many sessions resident at once, the KV cache and the index that ranks it live in host
memory, and the scan that ranks all n keys for a top-k step becomes the traffic that bounds decoding. We present Fathom, a key scan in which each query decides how many bits of each key channel to read. The 4-bit K cache is stored channel-major as bit planes, so a prefix of t planes is exactly the channel's t-bit quantizer, and the query spends its bit budget by reverse water-filling over the variance-weighted importance of its channels. At one million
tokens on Qwen3-8B a decode step is 1.67x faster in GPU time than with the 136-bit scans of Double Sparsity, Loki and SparQ r=32, and in the same GPU time as SparQ's 68-bit read (r=16) Fathom reads 18% fewer bytes with lower
attention error on six of seven model and context settings. On RULER-style tasks every per-
token scan matches exact top-k decoding, and on real coding-
agent sessions Fathom reaches the step agreement of the most accurate 136-bit scan at 92 bits. The store is the 4-bit K copy a quantized serving stack already holds, and the method is not faster when the index is resident in GPU
memory.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agents
Sehee Kim, Yumin Choi, Minki Kang et al. (+1 more)
Large
language model (LLM) trading
agents can combine
market data, news, and executable analysis, but their behavior is often controlled by static hand-written tool-use policies that are fixed before deployment. This limits their ability to adapt how they gather evidence, invoke tools, verify signals, and manage risk under changing
market regimes. We introduce EvolveTrade, a self-evolving framework that treats the system
prompt of a tool-using trading
agent as a text-parameterized policy. After each update interval, a Policy
Agent revises this policy using accumulated decision traces and realized portfolio feedback, while keeping the backbone LLM fixed. The updated policy is then used for the next batch of trading decisions, enabling the
agent to refine its information-acquisition and portfolio-construction procedure over time. Experiments across multiple
market regimes and two LLM backbones show that EvolveTrade often improves Sharpe Ratio and Cumulative Return over fixed-policy LLM baselines, achieving the improved SR and CR in most evaluated settings. Behavioral analyses further show that self-evolved policies increase code-mediated analysis and activate regime-relevant computations; case-level policy-to-return attributions trace how policy-induced allocation changes contribute to realized return differences. These results suggest that adapting the reusable procedure governing tool use is a key direction for building more robust LLM trading
agents.
Read the full abstract ▾
All authors Sehee Kim, Yumin Choi, Minki Kang, Sung Ju Hwang
Computation and Language · Cross-listed from cs.AI
ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents
Shuhan Xue, Jianyuan Zhong, Ziyuan Nan et al. (+10 more)
We introduce and release ScienceBuddy, an interactive scientific research workspace that brings continually improving scientific
agents into researchers' everyday workflows. ScienceBuddy supports researchers in carrying out scientific tasks while transforming their requests, feedback, and execution evidence into tasks and evaluation rubrics for continual learning. At its core is recursive-in-recursive self-improvement, a paradigm that couples harness
evolution with model reinforcement learning: the inner recursion improves the harness with the model fixed, while the outer recursion trains the model under the improved harness. Harness
evolution shapes training experience, and model learning creates new opportunities for harness adaptation. We present case studies of researcher interaction, harness refinement, and model learning, with the
benchmark cases spanning four scientific task families. By releasing ScienceBuddy as a research product, we make this paradigm available to the scientific community and take a step toward discovery intelligence: scientific AI that advances through sustained collaboration with researchers and evolves alongside the research it supports. Website: http://science-buddy.io
Read the full abstract ▾
All authors Shuhan Xue, Jianyuan Zhong, Ziyuan Nan, Wenbin Li, Zhaochen Yu, Jinchao Ding, Qiang Gao, Pengyu Zhan, Yuntong Zhang, Tian Cheng, Zhenfei Yin, Yingcheng Wu, Ling Yang
Computation and Language
When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control
Ali Şenol
Large
language models can produce fluent answers when their factual support is weak. This paper introduces Chain-of-Self-Questioning (CoSQ), a
prompt-only framework that makes answer commitment conditional on an explicit assessment of the information required to answer a question. We evaluate three CoSQ variants under seventeen conditions on the 817-item TruthfulQA multiple-choice validation set using eleven open-weight and hosted model families. In the final balanced-option protocol, Grounded-CoSQ at {\tau}=0.90 reduces the mean unconditional wrong-commitment rate from 13.1% under chain-of-thought prompting to 8.9%, a 32.1% relative reduction, while increasing answered accuracy from 86.9% to 89.7% and answering 87.6% of questions. Both improvements hold for all eleven models and at every evaluated threshold. Critical-CoSQ and Adaptive-CoSQ provide neighboring operating points with 88.6% and 86.5% coverage, respectively, while remaining more reliable than the baseline. A secondary Natural Questions Short-Answer evaluation provides convergent open-form evidence. These findings show that self-assessment can support explicit, tunable answer-or-abstain decisions when an unsupported commitment is more costly than referral or review.
Read the full abstract ▾
Computation and Language
What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity
Congjing Zhang, Vashishtha Patil, Henning Lange et al. (+1 more)
Pruning can reduce the deployment cost of large
language models (LLMs), but its impact on context-grounded tool calling remains poorly understood. We systematically study
pruning-induced degradation in smart-home tool calling across four LLMs spanning dense
Transformer, dense hybrid, and mixture-of-experts (MoE) architectures, together with depth, width, hybrid, and expert
pruning methods. After post-
pruning supervised fine-tuning (SFT), we evaluate more than 19,500 instances from three smart-home
datasets. Beyond aggregate task accuracy, we characterize degradation along two dimensions: action components (i.e., operation, device, argument, and value) and task complexity. Our results show that dense models have narrow safe
pruning regions followed by sharp degradation, while MoE models tolerate substantially more
pruning.
Pruning degrades grounded specificity before schema-level intent, and aggressive dense
pruning can induce systematic over-refusal. These findings highlight the importance of evaluating
pruning beyond aggregate accuracy when selecting pruned LLMs for reliable tool execution.
Read the full abstract ▾
All authors Congjing Zhang, Vashishtha Patil, Henning Lange, Usman Aleem
Computation and Language · Cross-listed from cs.SD
LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs
Thanapat Trachu, Samuele Cornell, William Chen et al. (+1 more)
Neural audio codecs are a key component in
speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate this by reducing the effective frame rate using a compression step to merge multiple frames together. However, most prior methods either operate on single-codebook codecs or apply a single compression step before multi-layer quantization. This forces all quantization layers to share the same segmentation boundaries, despite the residual
embeddings at different quantization layers exhibiting different rates of change over time. We propose LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries. To use LACE
tokens in downstream text-to-
speech (TTS), we further introduce union
alignment and boundary anchor mechanisms to make durations consistent across layers while preserving compression benefits. Experiments on LibriTTS show that LACE offers a better rate-quality tradeoff than prior dynamic frame rate methods on the reconstruction task and improves TTS inference efficiency while maintaining competitive synthesis quality. Our code is released as part of the ESPnet3 codec recipe.
Read the full abstract ▾
All authors Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe
Computation and Language · Cross-listed from cs.AI
Verifiable Social Reasoning for LLM Assistants
Amir Taubenfeld, Zorik Gekhman, Avigail Grinstein-Dabush et al. (+6 more)
LLM assistants are widely used for daily social advice, yet evaluating their social
reasoning in such consultation settings remains challenging since (i) it requires setups where the assistant learns about social situations from subjective user narratives, and (ii) social properties, such as others' intentions, typically lack verifiable ground truth. To address these challenges, we introduce Fuse, a multi-
agent simulation framework for studying user-mediated social
reasoning. In Fuse, a target
agent with a hidden motive interacts with other
agents including one representing the user, who then consults the evaluated assistant to infer the target's motive, providing verifiable ground truth by construction. Simulation faithfulness is validated through a human study with 24k annotations. We apply Fuse to 12 LLMs and demonstrate its analytical utility by systematically isolating key factors, showing that (i) user mediation compounds the inherent difficulty of social
reasoning; (ii) LLMs exhibit systematic sensitivity to biased user framing; (iii) models can require more details than humans need to reach a correct prediction; and (iv) longer conversations do not always improve performance despite providing opportunities for clarifying questions. We open-source Fuse and a
dataset with 21k examples.
Read the full abstract ▾
All authors Amir Taubenfeld, Zorik Gekhman, Avigail Grinstein-Dabush, Itay Laish, Ariel Goldstein, Marian Croak, Avinatan Hassidim, Yossi Matias, Amir Feder
Computation and Language
Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM
Adam Zachary Wasserman and David Beauchemin
We submit MéTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native Quebec-French
benchmark of grammatical minimal pairs) and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE (General
Language Understanding Evaluation) protocol that combines French task-data translation with rank-16 LoRA (Low-Rank Adaptation) produces a sharp task-type gradient: relational tasks gain measurably, while world-knowledge tasks regress. Bilingual Lexicon Induction aligns the French
embeddings to GPT-2 at p@1 = 68.84 +/- 8.61%, 18X above chance, suggesting cross-lingual
alignment tracks acquired grammatical competence rather than training duration. An ablation study shows that single-
token zero-shot scoring is dominated by tokenizer and template artifacts at the child scale, motivating tokenizer-swap sensitivity, placebo-controlled prompting, and native-
language minimal-pair
benchmarks as standard diagnostics.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.HC
CareMirror: Bringing Caregiver Wellbeing into the Dementia Care Ecosystem
Jiayue Melissa Shi, Ethan Nguyen, Drishti Goel et al. (+7 more)
Family caregivers of people living with dementia shoulder emotional and practical responsibilities, yet their own wellbeing often remains peripheral to dementia care. We built CareMirror, an envisioned caregiver wellbeing ecosystem with interconnected caregiver- and clinician-facing interfaces for longitudinal reflection, personalized support, and caregiver-controlled sharing with clinical care. We conducted semi-structured interviews with 14 caregivers, using CareMirror as a design probe to examine how they perceived this ecosystem and what expectations, concerns, and boundaries emerged around clinical connection. Caregivers valued
attention to their wellbeing, longitudinal awareness, context-sensitive support, and clinical visibility when it could lead to meaningful follow-up. However, repeated reflection could become burdensome or emotionally difficult, automatic clinical sharing could inhibit candid disclosure, and participants wanted control over what information entered clinical care. They also expected AI to support reflection and communication without replacing caregiver voice or clinician judgment. We contribute design considerations for proactive, clinically connected caregiver wellbeing support.
Read the full abstract ▾
All authors Jiayue Melissa Shi, Ethan Nguyen, Drishti Goel, Upasana Natarajan, Shashwat Srivatsa, Daniel S. Brown, Violeta J. Rodríguez, Dong Whi Yoo, Ravi Karkar, Koustuv Saha
Computation and Language
Enhancing Accessibility of Medical Texts through Large Language Model-Driven Plain Language Adaptation
Ting-Wei Chang, Hen-Hsen Huang and Hsin-Hsi Chen
This paper addresses the challenge of making complex healthcare information more accessible through automated Plain
Language Adaptation (PLA). PLA aims to simplify technical medical
language, bridging a critical gap between the complexity of healthcare texts and patients' reading comprehension. Recent advances in Large
Language Models (LLMs), such as GPT and BART, have opened new possibilities for PLA, especially in zero-shot and few-shot learning contexts where task-specific data is limited. In this work, we leverage the capabilities of LLMs such as GPT-4o-mini, Gemini-1.5-pro, and LLaMA for text simplification. Additionally, we incorporate Mixture-of-
Agents (MoA) techniques to enhance adaptability and robustness in PLA tasks. Key contributions include a comparative analysis of prompting strategies, finetuning with QLoRA on different LLMs, and the integration of MoA technique. Our findings demonstrate the effectiveness of LLM-driven PLA, showcasing its potential in making healthcare information more comprehensible while preserving essential content.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Large Language Models Develop Belief State Geometry In-Context
Daniel Balcells, Andrew Jun Lee, Chirag Rastogi et al. (+3 more)
Large
language models (LLMs) trained on next-
token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that support ICL remain poorly understood. We consider such representations in a controlled setting: prompting LLMs with data emitted from hidden Markov models (HMMs) and probing for the corresponding belief state – the posterior distribution over the HMM's hidden states given the observed
token history. Across six open-source LLMs prompted with data from 40 HMMs selected for non-trivial belief structure, we find that belief states are linearly decodable from residual stream activations, with peak probe $R^2$-values from 0.83-0.99 across HMM and LLM combinations, ranging from early to late layers. To establish functional relevance, we intervene directly on the probe-identified subspace via patching and steering, resulting in downstream prediction quality on the order of the untampered model, while controls degrade performance substantially. Together, these results provide representation-level evidence that ICL in open-source LLMs approximates optimal
Bayesian prediction over a context-inferred generative model. More broadly, our findings extend prior results linking input-distribution structure to activation geometry: from toy networks trained explicitly on HMM data to production-scale LLMs.
Read the full abstract ▾
All authors Daniel Balcells, Andrew Jun Lee, Chirag Rastogi, Paul M. Riechers, Adam Shai, Xavier Poncini
Computation and Language
Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts
Shuofeng Zhao, Hongwei Cai, Wenke Fan et al. (+9 more)
Real-time spoken dialogue systems must distinguish interruptions that require yielding the floor from backchannels that permit continued speaking. Existing
benchmarks typically score events independently and may therefore assign high scores to systems with fixed action preferences rather than context-sensitive decision policies. We introduce ECHO, a paired diagnostic
benchmark for Chinese turn-taking evaluation. ECHO pairs examples with the same overlap transcript but contrasting preceding multi-turn dialogue contexts, with one requiring Yield and the other Keep. It additionally includes off-talk examples for diagnosing unnecessary yielding. We introduce pair accuracy, which requires correct decisions on both members of a pair and assigns no credit to constant-action policies. Experiments on four
speech systems show that three exhibit a severe over-yielding bias: they correctly keep the floor on fewer than 13% of backchannels, resulting in near-zero pairwise success rates equal or less than 4%. While the remaining system remains comparatively balanced across contexts, these findings broadly demonstrate that interruption-only evaluation can severely overestimate practical turn-taking reliability.
Read the full abstract ▾
All authors Shuofeng Zhao, Hongwei Cai, Wenke Fan, Qingxiang Guo, Zhou Wang, Dawei Yang, Zhiyang Zhou, Yingxin Shang, Weixu Wang, Lin Yang, Shuran Zhou, Yang Song
Computation and Language
Where Should a Document Live: Context, Representations, or Parameters?
Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias et al. (+1 more)
To answer questions outside of their pre-training data, large
language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive
benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document
retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control
benchmarks, with 13% in coding.
Read the full abstract ▾
All authors Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias, Adrià de Gispert
Computation and Language
Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputs
Toqeer Ehsan, Nico Penttilä, Richard Schmidt et al. (+2 more)
This paper describes our submission to the SHROOM-
Visions shared task on detecting and classifying hallucinated character spans in
vision-
language model outputs across four
languages. We employ several fine-tuned
vision-
language models as independent annotators and combine their span predictions through character-level majority voting, and additionally explore activation probes. The approach ranks first in three of four
languages and places on the podium in every
language and metric. Our analysis indicates that disagreement among diverse models tracks disagreement among human annotators.
Read the full abstract ▾
All authors Toqeer Ehsan, Nico Penttilä, Richard Schmidt, Arash Hajikhani, Victoria Palacin
Computation and Language
Towards Detecting AI-Assisted Responses in Online Surveys
Qizhou Wang, Bogdan Mamaev and Christopher Leckie
The use of LLMs to complete online surveys impacts the validity of survey-based research, but detecting such usage remains underexplored. We introduce an initial
benchmark dataset, namely ASURRE, for AI-assisted survey participation to capture usage strategies ranging from full generation and revision to persona-grounded agentic completion. Controlled by these strategies, LLM-assisted survey responses are generated using multiple LLMs on three real-world surveys in different disciplines, paired with genuine human responses. Our evaluation of existing machine-generated text (MGT) detectors shows that naive AI usage is readily detectable, whereas persona-grounded
agents that mimic entire respondents push detector performance toward chance. We further show that agentic completion cannot fully replicate respondent-level behaviour and leaves distinctive behavioural traces. While individual cues can be circumvented by targeted prompting, a simple few-shot, training-free aggregator over these cues improves mean AUROC by +0.14 over the best existing detector across agentic settings. Our project is available at https://github.com/mike-qz-wang/ASURRE.
Read the full abstract ▾
Computation and Language
Zero-shot narrative detection in social messaging
Jesús M. Fraile-Hernández, Anselmo Peñas and Patrick Giedemann
This study investigates the zero-shot ability of large
language models (LLMs) to identify and classify hidden narratives in social messages. Our research hypothesis is that LLMs' extensive contextual knowledge allows them to interpret messages on a deeper, pragmatic level, going beyond basic sentiment or topic analysis. Experiments on the Dipromats and SemEval
datasets show that providing models with human-written narrative descriptions significantly improves performance, without the need of training examples. In contrast, automatically generated descriptions or the use of few examples (few-shot) often degrade accuracy due to subtle shifts in framing. The study also finds that ensemble methods, particularly majority voting, enhance robustness and that larger models perform best while also being less sensitive to
prompt variations. The findings validate that LLMs can effectively detect strategic narratives in a zero-shot setting, and when combined with simple ensembling and human-written descriptions, they can rival supervised systems, offering a scalable solution for narrative detection, specially when there is no training data for the vast majority of domains.
Read the full abstract ▾
Computation and Language
Towards Illusions Awareness in Cyber-Physical System's Design
Anna Di Placido, Nicolas Ferry and Julien Deantoni
Cyber-Physical Systems (CPS) operate through a continuous sense-compute-act loop within an open context environment, making it impossible to anticipate all the situations the system will face. To cope with this openness, stakeholders rely on assumptions, formalized into design models. However, these assumptions may no longer hold once the system is confronted with runtime reality, resulting in a discrepancy between expected and observed behaviour known in literature as the reality gap. Existing approaches mainly focus on reducing or overcoming it by making simulations more faithful to reality, with no unified methodology to structure and exploit invalidated assumptions that give rise to this gap as reusable design knowledge. We refer to the persistent reliance on invalidated assumptions -and the resulting false confidence in the design model's operational validity -as design illusions, and argue that they need to be made explicit, structured, and exploited as knowledge to support better design decisions. We propose a conceptual pipeline for illusions-awareness that identifies, classifies, characterizes, and leverages illusions to transform them into actionable design knowledge.
Read the full abstract ▾
Computation and Language
Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization
Eduardo Novaes Hering
We introduce a simple architectural modification to decoder-only
transformers: a persistent recurrent state that observes hidden representations via cross-
attention, updates itself through a GRU, and modulates subsequent processing via gated addition. Inserted between the lower and upper halves of a 6-layer
transformer, this module adds only 3.7% additional parameters while reducing evaluation loss from $2.438 \pm 0.004$ to $1.743 \pm 0.018$, corresponding to a 28.5% reduction on held-out
language modeling data. The improvement is statistically significant across 5 random seeds ($p < 0.01$) and corresponds to reduced overfitting (generalization gap 0.12 vs 0.26). Through controlled ablations, we demonstrate that the improvement stems entirely from the persistent
memory topology, not from auxiliary self-prediction objectives. A model with identical
topology but no auxiliary loss performs equivalently, while a random auxiliary loss provides no benefit. Representation probing reveals that the persistent state encodes narrative position (52% vs 33% chance level)—information that standard
attention maintains less efficiently. Our results suggest that bridging
transformer layers with a lightweight recurrent
memory is a simple, effective approach to improving generalization in small-scale
language models.
Read the full abstract ▾
Computation and Language
ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding
Ziyang Ma, Zihong Zhang, Zuchao Li et al. (+4 more)
While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges, we propose ECHO, a hierarchical dual-loop framework that exploits the functional asymmetry between LLM layers. Leveraging the high discriminative efficiency of early layers and the authoritative distribution of final layers, ECHO bifurcates inference into a high-frequency inner loop and a low-frequency outer loop. Within the inner loop, early-layer bonus logits drive rapid, multi-step draft-tree exploration at a minimal cost. Simultaneously, the outer loop performs authoritative full-model verification through a state-reuse mechanism. Crucially, the outer loop also utilizes final-layer bonus logits to correct existing paths and supplement the tree with high-confidence candidates for subsequent cycles. Experimental results across diverse
benchmarks demonstrate that ECHO significantly boosts mean accepted
tokens and achieves a 2.4$\times$ to 2.9$\times$ speedup, outperforming existing state-of-the-art baselines with negligible engineering overhead and no extra deployment parameters, albeit with a one-shot fine-tuning dependency for optimal acceleration. The code is available at https://github.com/whucs21Mzy/ECHO.
Read the full abstract ▾
All authors Ziyang Ma, Zihong Zhang, Zuchao Li, Lefei Zhang, Baoyuan Qi, Siqi Li, Simin Yu