Computation and Language
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Abel Yagubyan
LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized. We study repeated identical evaluations on 29 tasks spanning 10 categories using two OpenAI judge models (GPT-4o-mini and GPT-4.1-mini), with 50 pairwise trials and 50 pointwise trials per question, supplemented by temperature and
prompt-sensitivity ablations. Across judges, pairwise preferences flip on average 13.6% of the time, with 28% of questions exceeding a 20% flip rate and one question reaching 56%. GPT-4o-mini also exhibits a significant first-position bias (72% A-majority, p = 0.024). At the same time, mean pointwise score gaps are small (0.19–0.36 on a 10-point scale) and not statistically significant in aggregate, producing a pairwise–pointwise gap: judges frequently choose a winner even when their own scalar scores provide little evidence of a meaningful quality difference. Beyond within-judge instability, cross-judge agreement is only 76% ($\kappa = 0.51$), semantically equivalent
prompt templates change majority outcomes in 25% of tested cases, and deterministic decoding reduces but does not eliminate inconsistency. A reliability curve analysis shows that, in our
dataset, 11 repeated trials are needed for a majority vote to recover the 50-trial reference verdict with 95% probability on average, rising to 15 for high-variance questions. These findings suggest that single-trial LLM judging is often too noisy for high-stakes evaluation, and that multi-trial aggregation, position randomization, and explicit uncertainty reporting should be standard practice. Because both judges are from a single provider, cross-provider replication remains an important next step.
Read the full abstract ▾
Computation and Language
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
Jingpei Wu, Xiao Han, Weixiang Shen et al. (+3 more)
Visual question answering increasingly requires multi-step
reasoning. Recent post-training with reinforcement learning under verifiable rewards (RLVR) and Group Relative Policy Optimization (GRPO) can improve
multimodal reasoning, but most approaches rely on sparse outcome-only rewards. As a result, they struggle to tell whether an incorrect answer comes from a small mistake late in the
reasoning or from an unhelpful trajectory from the start. A common solution is to train a process reward model (PRM) for step-level supervision, but this typically requires large-scale high-quality chain-of-thought annotations and additional training cost. We propose ProcessThinker, a practical post-training pipeline that provides step-level process rewards without training an explicit PRM. ProcessThinker first rewrites
reasoning traces into a step-tagged format for cold-start supervised fine-tuning, then applies GRPO with a standard format reward and our rollout-based process reward. Concretely, for each intermediate step, we sample multiple continuations from that step and use the empirical success rate (final-answer verification) as the step reward. This gives dense credit assignment and encourages
reasoning steps that more reliably support a correct conclusion, helping reduce inconsistent or self-contradictory progress across steps – a key issue in logical
reasoning. Across four challenging
video benchmarks (
Video-MMMU, MMVU, VideoMathQA, and LongVideoBench), ProcessThinker consistently improves over the baseline model Qwen3-VL-8B-Instruct
Read the full abstract ▾
All authors Jingpei Wu, Xiao Han, Weixiang Shen, Boer Zhang, Zifeng Ding, Volker Tresp
Computation and Language
BioDivergence: A Benchmark and Evaluation Framework for Hidden Contextual Contradictions in Biomedical Abstracts
Elias Hossain, Sanjeda Sara Jennifer, Sabera Akter Bushra et al. (+1 more)
Biomedical findings often seem to conflict across studies, but many of these differences are context-dependent rather than true contradictions. Variations in cohort, geography, assay protocol, disease subtype, and clinical setting can make both claims locally valid. Existing NLI and scientific claim-verification
benchmarks reduce such cases to entailment, contradiction, or neutral, failing to capture the contextual structure behind divergence. To address this, we introduce BioDivergence, an evaluation framework with a six-class conflict taxonomy, a 13-axis divergence ontology, and four structured outputs per claim pair: conflict type, divergence axes, dominant confounder, and reconciliation explanation. We release BioDivergence-Silver-v1.0, an article-disjoint silver
benchmark of 11,865 claim pairs across five biomedical domains, alongside a legacy deduplicated variant for comparison. Results show notable ranking differences between the two variants, with the fine-tuned reference model dropping about 12 points under the article-disjoint setting, while Mistral-7B-Instruct-v0.3 achieves 0.5523 accuracy and 0.3894 contextual-F1 on the 842-example primary test set. BioDivergence offers a more faithful way to distinguish contextual divergence from direct contradiction and to separate article-level memorization from genuine task learning.
Read the full abstract ▾
All authors Elias Hossain, Sanjeda Sara Jennifer, Sabera Akter Bushra, Niloofar Yousefi
Computation and Language · Cross-listed from cs.AI
From Explicit Elements to Implicit Intent: A Predefined Library for Auditable Behavioral Inference
Liu hung ming
We present SemantiClean, a modular framework for extracting structured semantic signals from e-commerce session data and driving pluggable inference targets including purchase intent, customer
segmentation, and product affinity through a shared element library. Unlike conventional end-to-end predictors that optimise solely for accuracy, SemantiClean prioritises auditability, structural governance, and sigma=0 reproducibility, explicitly trading marginal predictive gains for element-level transparency and defensible decision trails. Built upon the Online Shoppers Purchasing Intention (OSPI)
dataset, the framework organises twenty-four behavioural elements into a four-layer architecture (Functional, Interaction, Systemic, Contextual) and enforces signal quality through three anti-inflation mechanisms: RedundancyGroup contribution caps, TieredPenaltyCalculator bias penalties, and AdaptiveConstraintMode cold-start protection.This report introduces the LLM-Integrated Semantic Inference Engine, a fully implemented two-phase LLM-driven inference architecture that leverages complete element metadata at inference time. All quantitative results reported herein are produced by this engine. Deterministic engine outputs remain fully reproducible (sigma=0); LLM-dependent results (E8, E10) are subject to controlled output variability under fixed provider/model/temperature settings. The gender inference target remains non-functional in the current implementation and is excluded from all quantitative results.
Read the full abstract ▾
Computation and Language
How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines
Abel Yagubyan
Large
language model (LLM)
agents with tool-calling capabilities are increasingly deployed in production systems, yet a fundamental reliability question remains under-explored: does the same
agent behave the same way twice? We present a systematic empirical study of behavioral consistency in multi-step tool-calling
agents, measuring whether
agents select the same tools, in the same order, with the same arguments, across repeated identical invocations. Unlike prior work on consistency in ReAct-style
agents(search-only, free-text actions), we study the richer setting of structured tool-calling interfaces with typed parameters and consequential side effects.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressure
Boyu Xiao, Xiuqi Tian, Xuwen Song et al. (+4 more)
Despite strong medical
benchmark accuracy, LLMs can exhibit severe multi-turn sycophancy in clinical dialogue, abandoning initial correct diagnosis under escalating pressure. We propose \textbf{Med-Stress}, a targeted stress test framework that evaluates belief stability under escalating pressure. Across nine frontier large
language models (LLMs), we find a clear dissociation between medical knowledge and robustness: high initial diagnostic capability does not imply high belief stability, yielding large knowledge-robustness gaps for several LLMs. To mitigate this failure mode, we propose a lightweight inference-time defense, \textbf{RBED} (Role-Based Epistemic Defense), and \textbf{R-FT} (Resilience-oriented Fine-Tuning), a training-time approach that internalizes evidence-based resistance to pressure. Experiments show that \textbf{R-FT} nearly eliminates belief change and substantially improves robustness.
Read the full abstract ▾
All authors Boyu Xiao, Xiuqi Tian, Xuwen Song, Haochun Wang, Guanchun Song, Sendong Zhao, Bing Qin
Computation and Language · Cross-listed from cs.CY
Evaluating Patient Safety Risks in Generative AI: Development and Validation of a FMECA Framework for Generated Clinical Content
Lydie Bednarczyk, Jamil Zaghir, Julien Ehrsam et al. (+11 more)
Objectives: Large
language models (LLMs) are increasingly used for clinical text summarization, yet structured methods to assess associated patient safety risks remain limited. Failure Mode, Effects, and Criticality Analysis (FMECA) provides a proactive framework for systematic risk identification but has not been adapted to LLM-generated clinical content. This study aimed to develop and validate a novel FMECA framework for the prospective assessment of patient safety risks in LLM-generated clinical summaries. Materials and Methods: An interdisciplinary expert panel (n = 8) developed a taxonomy of failure modes through literature review and brainstorming. Standard FMECA dimensions (occurrence, severity, detectability) were adapted into 5-point ordinal scales. The framework was applied to 36 discharge summaries from four patients, generated by an open LLM (GPT-OSS 120B) using real-world clinical data from the Geneva University Hospitals. Reviewers independently annotated the summaries across two rounds. Inter-rater reliability was assessed at failure mode, severity and detectability score levels. Usability and content validity were evaluated using an adapted System Usability Scale and structured feedback. Results: The final framework comprised 14 failure modes organized into categories. Inter-rater agreement improved between rounds, reaching moderate-to-substantial agreement for failure mode identification and good agreement for severity and detectability scoring. Usability was rated as good (mean SUS: 79.2/100), with high evaluator confidence. Discussion and Conclusion: This study presents the first FMECA-based framework for systematic patient safety risk assessment of LLM-generated clinical summaries. The framework provides a structured and reproducible method for identifying clinically relevant risks caused by these summaries.
Read the full abstract ▾
All authors Lydie Bednarczyk, Jamil Zaghir, Julien Ehrsam, Maria Tcherepanova, Christian Skalafouris, Karim Gariani, Catherine Geslin, Claire-Bénédicte Rivara, Pascal Bonnabry, Laetitia Gosetto, Richard Dubos, Mina Bjelogrlic, Christophe Gaudet-Blavignac, Christian Lovis
Computation and Language · Cross-listed from cs.IR
A Large-Scale, Cross-Disciplinary Corpus of Systematic Reviews
Pierre Achkar, Tim Gollub, Arno Simons et al. (+2 more)
Existing
benchmarks for systematic reviewing remain limited either in scale or in disciplinary coverage, with some collections comprising only a modest number of topics and others focusing primarily on biomedical research. We present Webis-SR4ALL-26, a large-scale, cross-disciplinary corpus of 301,871 systematic reviews spanning all scientific fields as covered by OpenAlex. Using a multi-stage pre-processing pipeline, we link reviews to resolved OpenAlex metadata and reference lists and extract, when explicitly reported, structured method artifacts relevant to
retrieval and screening. These artifacts include reported search strategies (Boolean queries or keyword lists) that we normalize into executable approximations, as well as reported inclusion and exclusion criteria. Together, these layers support cross-domain benchmarking of
retrieval and screening components against review reference lists, training and evaluation of extraction methods for review artifacts, and comparative meta-science analyses of systematic review practices across disciplines and time. To demonstrate one concrete use case, we report large-scale baseline
retrieval signals by executing normalized search strategies in OpenAlex and comparing retrieved sets to resolved reference lists. We release the corpus and the pre-processing pipeline, along with code used for extraction validation and the
retrieval demonstration.
Read the full abstract ▾
All authors Pierre Achkar, Tim Gollub, Arno Simons, Harrisen Scells, Martin Potthast
Computation and Language · Cross-listed from cs.LG
PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training
Harsh Kumar, Rahul Maity, Tanmay Joshi et al. (+4 more)
Aligned large
language models (LLMs) remain vulnerable to
adversarial manipulation, and their reliance on web-scale pretraining creates a subtle but consequential attack surface. We study Stealth Pretraining Seeding (SPS), a threat model in which adversaries distribute small amounts of poisoned content across stealth websites, increasing the likelihood that such material is absorbed into future training corpora derived from sources such as Common Crawl. Because each individual payload is tiny, diffuse, and superficially benign, the attack is difficult to detect during
dataset construction or filtering. The result is a latent form of poisoning that remains largely invisible under standard evaluation, yet can later be activated by a precise trigger such as <00TRIGGER00>. We call this attack PermaFrost, reflecting its latent and reactivatable nature. We study it through PermaFrost-Attack, a controlled framework for latent conceptual poisoning, together with three
geometric diagnostics: Thermodynamic Length, Spectral
Curvature, and the Infection Traceback Graph. Across multiple model families and scales, we show that this controlled SPS proxy can induce persistent unsafe behavior that often remains hidden under standard evaluation. Our results identify SPS as a practical and underappreciated threat to future foundation models. This paper introduces a novel
geometric diagnostic lens for systematically examining latent model behavior, providing a principled foundation for detecting, characterizing, and understanding vulnerabilities that may remain invisible under standard evaluation.
Read the full abstract ▾
All authors Harsh Kumar, Rahul Maity, Tanmay Joshi, Aman Chadha, Vinija Jain, Suranjana Trivedy, Amitava Das
Computation and Language · Cross-listed from cs.HC
Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations
Nalin Poungpeth, Nicholas Clark and Tanu Mitra
Large
language models (LLMs) possess strong persuasive capabilities that outperform humans in head-to-head comparisons. Users report consulting LLMs to inform major life decisions in relationships, medical settings, and when seeking professional advice. Prior work measures persuasion as intentional attempts at producing the most effective argument or convincing statement. This fails to capture everyday human-AI interactions in which users seek information or advice. To address this gap, we introduce "spontaneous persuasion," which characterizes the inexplicit use of persuasive strategies in everyday scenarios where persuasion is not necessarily warranted. We conduct an audit of five LLMs to uncover how frequently and through which techniques spontaneous persuasion appears in multi-turn conversations. To simulate response styles, we provide a user response taxonomy grounded in literature from psychology, communication, and linguistics. Furthermore, we compare the distribution of spontaneous persuasion produced by LLMs with human responses on the same topics, collected from Reddit. We find LLMs spontaneously persuade the user in virtually all conversations, heavily relying on information-based strategies such as appeals to logic or quantitative evidence. This was consistent across models and user response styles, but conversations concerning mental health saw higher rates of appraisal-based and
emotion-based strategies. In comparison, human responses tended to invoke strategies that generate social influence, like negative
emotion appeals and non-expert testimony. This difference may explain the effectiveness of LLM in persuading users, as well as the perception of models as objective and impartial.
Read the full abstract ▾
Computation and Language
Knowledge-driven Augmentation and Retrieval for Integrative Temporal Adaptation
Weisi Liu, Guangzeng Han and Xiaolei Huang
Time introduces fundamental challenges in model development and deployment: models are usually trained on historical data while deployed on future data where semantic distributions and domain knowledge may evolve. Unfortunately, existing studies either overlook temporal shifts or hardly capture rich shifting patterns of both semantic and knowledge. We develop Knowledge-driven Augmentation and
Retrieval for Integrative Temporal Adaptation (KARITA) to capture diverse temporal shifts (e.g., uncertainty and feature shift), construct and integrate rich knowledge sources (e.g., medical ontology like MeSH), and leverage shifting insights for selecting-
retrieval augmented learning. We evaluate KARITA on classification tasks across multiple domains, clinical, legal, and scientific corpora, demonstrating consistent improvements across multiple domains with temporal adaptation. Our results show that knowledge integration can be more critical and effective in temporal augmentation and learning.
Read the full abstract ▾
Computation and Language
An End-to-End Ukrainian RAG for Local Deployment. Optimized Hybrid Search and Lightweight Generation
Mykola Trokhymovych, Yana Oliinyk and Nazarii Nyzhnyk
This paper presents a highly efficient
Retrieval-Augmented Generation (RAG) system built specifically for Ukrainian document question answering, which achieved 2nd place in the UNLP 2026 Shared Task. Our solution features a custom two-stage search pipeline that retrieves relevant document pages, paired with a specialized Ukrainian
language model fine-tuned on synthetic data to generate accurate, grounded answers. Finally, we compress the model for lightweight deployment. Evaluated under strict computational limits, our architecture demonstrates that high-quality, verifiable AI question answering can be achieved locally on resource-constrained hardware without sacrificing accuracy.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
PrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning
Xiaoyi Chen, Haoyuan Wang, Siyuan Tang et al. (+4 more)
Large
language models (LLMs) often memorize private information during training, raising serious
privacy concerns. While machine unlearning has emerged as a promising solution, its true effectiveness against
privacy attacks remains unclear. To address this, we propose PrivUn, a new evaluation framework that systematically assesses unlearning robustness through three-tier attack scenarios: direct
retrieval, in-context learning recovery, and fine-tuning restoration; combined with quantitative analysis using forgetting scores, association metrics, and forgetting depth assessment. Our study exposes significant weaknesses in current unlearning methods, revealing two key findings: 1) unlearning exhibits gradient-driven ripple effects: unlike traditional forgetting which follows semantic relations (e.g., knowledge graphs),
privacy unlearning propagates across latent gradient-based associations; and 2) most methods suffer from shallow forgetting, failing to remove private information distributed across multiple deep model layers. To validate these insights, we explore two strategies: association-aware core-set selection that leverages gradient similarity, and multi-layer deep intervention through representational constraints. These strategies represent a paradigm shift from shallow forgetting to deep forgetting.
Read the full abstract ▾
All authors Xiaoyi Chen, Haoyuan Wang, Siyuan Tang, Sijia Liu, Liya Su, XiaoFeng Wang, Haixu Tang
Computation and Language
Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
Qinan Yu, Alexa Tartaglini, Peter Hase et al. (+2 more)
Reinforcement Learning from Verifiable Rewards (RLVR) on chain-of-thought
reasoning has become a standard part of
language model post-training recipes. A common assumption is that the
reasoning chains trained through RLVR reliably represent how a model gets to its answer. In this paper, we develop two metrics for critically examining this assumption: Causal Importance of
Reasoning (CIR), which measures the cumulative effect of
reasoning tokens on the final answer, and Sufficiency of
Reasoning (SR), which measures whether a verifier can arrive at an unambiguous answer based on the
reasoning alone. Through experiments with the Qwen2.5 model series and ReasoningGym tasks, we find that: (1) while RLVR does improve task accuracy, it does not reliably improve CIR or SR, calling the role of
reasoning in model performance into question; (2) a small amount of SFT before RLVR can be a remedy for low CIR and SR; and (3) CIR and SR can be improved even without SFT by applying auxiliary CIR/SR rewards on top of the outcome-based reward. This joint reward matches the accuracy of RLVR while also leading to causally important and sufficient
reasoning. These results show that RLVR does not always lead models to rely on
reasoning in the way that is commonly thought, but this issue can be remedied with simple modifications to the post-training procedure.
Read the full abstract ▾
All authors Qinan Yu, Alexa Tartaglini, Peter Hase, Carlos Guestrin, Christopher Potts
Computation and Language
Optimal Question Selection from a Large Question Bank for Clinical Field Recovery in Conversational Psychiatric Intake
Guan Gui, Peter Zandi, Jacob Taylor et al. (+1 more)
Psychiatric intake is a sequential, high-stakes information-gathering process in which clinicians must decide what to ask, in what order, and how to interpret incomplete or ambiguous responses under limited time. Despite growing interest in conversational AI for healthcare, there is still limited infrastructure for conversational AI in this application. Accordingly, we formulate this task as a question-selection problem with clinically grounded questions, known target information, and controllable patient difficulty. We also introduce a task-specific question-selection
benchmark based on a bank of 655 clinician-authored intake questions and corresponding synthetic patient vignettes with 5 different behavioral conditions. In our evaluation, we compare random questioning, a clinical psychiatric intake form baseline, and an LLM-guided adaptive policy across 300 interview sessions spanning four patients and five behavioral conditions. Across the
benchmark, the clinically ordered fixed form substantially outperforms random questioning, and the LLM-guided policy achieves the strongest overall recovery. The advantage of adaptation grows sharply under patient behavior that is less amenable to field recovery, especially under guarded-concise conditions. These findings suggest that performance in conversational clinical systems depends not only on
language understanding after information is disclosed, but also on whether the system reaches the right topics within a limited interaction budget. More broadly, the
benchmark provides a controlled framework for studying how clinical structure and adaptive follow-up contribute to information recovery in interactive clinical machine learning.
Read the full abstract ▾
All authors Guan Gui, Peter Zandi, Jacob Taylor, Ananya Joshi
Computation and Language
Incentivizing Neuro-symbolic Language-based Reasoning in VLMs via Reinforcement Learning
Karthic Palaniappan
There are 7,407
languages in the world. But, what about the
languages that are not there in the world? Are humans so narrow minded that we don't care about the
languages aliens communicate in? Aliens are humans too! In the 2016 movie Arrival, Amy Adams plays a linguist, Dr. Louise Banks who, by learning to think in an alien
language (Heptapod) formed of non-sequential sentences, gains the ability to transcend time and look into the future. In this work, I aim to explore the representation and
reasoning of
vision-
language concepts in a neuro-symbolic
language, and study improvement in analytical
reasoning abilities and efficiency of "thinking systems". With Qwen3-VL-2B-Instruct as base model and 4 $\times$ Nvidia H200 GPU nodes, I achieve an accuracy improvement of 3.33% on a
vision-
language evaluation
dataset consisting of math, science, and general knowledge questions, while reducing the
reasoning tokens by 75% over SymPy. I've documented the compute challenges faced, scaling possibilities, and the future work to improve thinking in a neuro-symbolic
language in
vision-
language models. The training and inference setup can be found here: https://github.com/i-like-bfs-and-dfs/wolfram-
reasoning.
Read the full abstract ▾
Computation and Language
Lightweight Retrieval-Augmented Generation and Large Language Model-Based Modeling for Scalable Patient-Trial Matching
Xiaodi Li, Yang Xiao, Munhwan Lee et al. (+7 more)
Patient-trial matching requires
reasoning over long, heterogeneous electronic health records (EHRs) and complex eligibility criteria, posing significant challenges for scalability, generalization, and computational efficiency. Existing approaches either rely on full-document processing with large
language models (LLMs), which is computationally expensive, or use traditional machine learning methods that struggle to capture unstructured clinical narratives. In this work, we propose a lightweight framework that combines
retrieval-augmented generation and large
language model-based modeling for scalable patient-trial matching. The framework explicitly separates two key components:
retrieval-augmented generation is used to identify clinically relevant segments from long EHRs, reducing input complexity, while large
language models are used to encode these selected segments into informative representations. These representations are further refined through dimensionality reduction and modeled using lightweight predictors, enabling efficient and scalable downstream classification. We evaluate the proposed approach on multiple public
benchmarks (n2c2, SIGIR, TREC 2021/2022) and a real-world
multimodal dataset from Mayo Clinic (MCPMD). Results show that
retrieval-based information selection significantly reduces computational burden while preserving clinically meaningful signals. We further demonstrate that frozen LLMs provide strong representations for structured clinical data, whereas fine-tuning is essential for modeling unstructured clinical narratives. Importantly, the proposed lightweight pipeline achieves performance comparable to end-to-end LLM approaches with substantially lower computational cost.
Read the full abstract ▾
All authors Xiaodi Li, Yang Xiao, Munhwan Lee, Konstantinos Leventakos, Young J. Juhn, David Jones, Terence T. Sio, Wei Liu, Maria Vassilaki, Nansu Zong
Computation and Language · Cross-listed from cs.LG
LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs
Mohamed Ali Souibgui, Jan Fostier, Rodrigo Abadía-Heredia et al. (+3 more)
Transformers are mostly relying on softmax
attention, which introduces quadratic complexity with respect to sequence length and remains a major bottleneck for efficient inference. Prior work on linear or hybrid
attention typically replaces softmax
attention uniformly across all layers, often leading to significant performance degradation or requiring extensive retraining to recover model quality. This work proposes LayerBoost, a layer-aware
attention reduction method that selectively modifies the
attention mechanism based on the sensitivity of individual
transformer layers. It first performs a systematic sensitivity analysis on a pretrained model to identify layers that are critical for maintaining performance. Guided by this analysis, three distinct strategies can be applied: retaining standard softmax
attention in highly sensitive layers, replacing it with linear sliding window
attention in moderately sensitive layers, and removing
attention entirely in layers that exhibit low sensitivity. To recover performance after these architectural modifications, we introduce a lightweight
distillation-based healing phase requiring only 10M additional training
tokens. LayerBoost reduces inference latency and improves throughput by up to 68% at high concurrency, while maintaining competitive model quality. It matches base model performance on several
benchmarks, exhibits only minor degradations on others, and significantly outperforms state-of-the-art
attention linearization methods. These efficiency gains make our method particularly well-suited for high-concurrency serving and hardware-constrained deployment scenarios, where inference cost and
memory footprint are critical bottlenecks.
Read the full abstract ▾
All authors Mohamed Ali Souibgui, Jan Fostier, Rodrigo Abadía-Heredia, Bohdan Denysenko, Christian Marschke, Igor Peric
Computation and Language
Source-Modality Monitoring in Vision-Language Models
Etha Tianze Hua, Tian Yun and Ellie Pavlick
We define and investigate source-modality monitoring – the ability of
multimodal models to track and communicate the input source from which pieces of information originate. We consider source-modality monitoring as an instance of the more general binding problem, and evaluate the extent to which models exploit syntactic vs. semantic signals in order to bind words like image in a user-provided
prompt to specific components of their input and context (i.e., actual images). Across experiments spanning 11
vision-
language models (VLMs) performing target-modality information
retrieval tasks, we find that both syntactic and semantic signals play an important role, but that the latter tend to outweigh the former in cases when modalities are highly distinct distributionally. We discuss the implications of these findings for model robustness, and in the context of increasingly
multimodal agentic systems.
Read the full abstract ▾
Computation and Language
Shared Lexical Task Representations Explain Behavioral Variability In LLMs
Zhuonan Yang, Jacob Xiaochen Li, Francisco Piedrahita Velez et al. (+5 more)
One of the most common complaints about large
language models (LLMs) is their
prompt sensitivity – that is, the fact that their ability to perform a task or provide a correct answer to a question can depend unpredictably on the way the question is posed. We investigate this variation by comparing two very different but commonly-used styles of prompting: instruction-based
prompts, which describe the task in natural
language, and example-based
prompts, which provide in-context few-shot demonstration pairs to illustrate the task. We find that, despite large variation in performance as a function of the
prompt, the model engages some common underlying mechanisms across different
prompts of a task. Specifically, we identify task-specific
attention heads whose outputs literally describe the task – which we dub lexical task heads – and show that these heads are shared across prompting styles and trigger subsequent answer production. We further find that behavioral variation between
prompts can be explained by the degree to which these heads are activated, and that failures are at least sometimes due to competing task representations that dilute the signal of the target task. Our results together present an increasingly clear picture of how LLMs' internal representations can explain behavior that otherwise seems idiosyncratic to users and developers.
Read the full abstract ▾
All authors Zhuonan Yang, Jacob Xiaochen Li, Francisco Piedrahita Velez, Eric Todd, David Bau, Michael L. Littman, Stephen H. Bach, Ellie Pavlick
Computation and Language
When Cow Urine Cures Constipation on YouTube: Limits of LLMs in Detecting Culture-specific Health Misinformation
Anamta Khan, Ratna Kandala, Deepti et al. (+2 more)
Social media platforms have become primary channels for health information in the Global South. Using gomutra (cow urine) discourse on YouTube in India as a case study, we present a post-facto Large
Language Model (LLM)-assisted discourse analysis of 30
multilingual transcripts showing that promotional content blends sacred traditional
language with pseudo-scientific claims in ways that sophisticated debunking content itself mirrors, creating a rhetorical register that LLMs, trained predominantly on Western corpora, are systematically ill-equipped to analyse. Varying
prompt tone across three LLMs (GPT-4o, Gemini 2.5 Pro, DeepSeek-V3.1), we find that culturally embedded health misinformation does not look like ordinary misinformation, and this cultural obfuscation extends to gendered rhetoric and
prompt design, compounding analytical unreliability. Our findings argue that cultural competency in LLM-assisted discourse analysis cannot be retrofitted through
prompt engineering alone.
Read the full abstract ▾
All authors Anamta Khan, Ratna Kandala, Deepti, Sheza Munir, Joyojeet Pal
Computation and Language · Cross-listed from cs.LG
Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning
Grigory Sapunov
We study learned
memory tokens as a computational scratchpad for a single-block Universal
Transformer with Adaptive Computation Time (ACT) on Sudoku-Extreme, a combinatorial
reasoning benchmark.
Memory tokens are empirically necessary: no configuration without them reaches non-trivial performance. The optimal count has a sharp lower threshold (T=0 always fails, T=8 reliably succeeds) followed by a stable plateau (T=8-32, 57.4% +/- 0.7% exact-match) and a dilution boundary at T=64. Under halt-side pressure (lambda warmup), mean halt drops monotonically with
memory size across the plateau (from 11.6 at T=8 to 8.3 at T=64), showing that
memory tokens and ponder depth substitute as resources at fixed accuracy. We also identify a router initialization trap that causes the majority of training runs to fail: both default zero-bias and Graves' recommended positive bias settle into a shallow halt
equilibrium the model cannot escape. Inverting the bias to -3 ("deep start") eliminates the failure mode, and ablation shows the trap is inherent to ACT initialization rather than an artifact of our architecture. With reliable training, ACT yields an order of magnitude lower seed variance than fixed-depth processing (+/-0.7 vs +/-9.3 pp); lambda warmup recovers 34% of compute at matched accuracy; and
attention heads specialize into
memory readers, constraint propagators, and integrators across recursive depth. Code: https://github.com/che-shr-cat/utm-jax.
Read the full abstract ▾
Computation and Language
Evaluation of Automatic Speech Recognition Using Generative Large Language Models
Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil et al. (+6 more)
Automatic
Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning.
Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large
Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative
embeddings, and (3) qualitative classification of errors. On the HATS
dataset, the best LLMs achieve 92–94% agreement with human annotators for hypothesis selection, compared to 63% for WER, also outperforming semantic metrics.
Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation.
Read the full abstract ▾
All authors Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour
Computation and Language
MathDuels: A Self-Play Benchmark That Grows
Zhiqiu Xu, Shibo Jin, Shreya Arya et al. (+1 more)
As frontier
language models attain near-ceiling performance on static mathematical
benchmarks, existing evaluations are increasingly unable to differentiate model capabilities, largely because they cast models solely as solvers of fixed problem sets. We introduce MathDuels, a self-play
benchmark in which models occupy dual roles: each authors math problems under
adversarial prompting and solves problems authored by every other participant. Problems are produced through a three-stage generation pipeline (meta-prompting, problem generation, and difficulty amplification), and validated by an independent verifier that excludes ill-posed questions. A Rasch model (Rasch, 1993) jointly estimates solver abilities and problem difficulties; author quality is derived from the difficulties of each model's authored problems. Experiments across 19 frontier models reveal that authoring and solving capabilities are partially decoupled, and that dual-role evaluation reveals capability separations invisible in single-role
benchmarks. As newer models enter the arena, they produce problems that defeat previously dominant solvers, so the
benchmark's difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. We host a public leaderboard that updates as new models are released.
Read the full abstract ▾
All authors Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik
Computation and Language · Cross-listed from cs.CV
Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny et al. (+3 more)
Despite impressive progress in capabilities of large
vision-
language models (LVLMs), these systems remain vulnerable to
hallucinations, i.e., outputs that are not grounded in the visual input. Prior work has attributed
hallucinations in LVLMs to factors such as limitations of the
vision backbone or the dominance of the
language component, yet the relative importance of these factors remains unclear. To resolve this ambiguity, We propose HalluScope, a
benchmark to better understand the extent to which different factors induce
hallucinations. Our analysis indicates that
hallucinations largely stem from excessive reliance on textual priors and background knowledge, especially information introduced through textual instructions. To mitigate
hallucinations induced by textual instruction priors, we propose HalluVL-DPO, a framework for fine-tuning off-the-shelf LVLMs towards more visually grounded responses. HalluVL-DPO leverages preference optimization using a curated training
dataset that we construct, guiding the model to prefer grounded responses over hallucinated ones. We demonstrate that our optimized model effectively mitigates the targeted
hallucination failure mode, while preserving or improving performance on other
hallucination benchmarks and visual capability evaluations. To support reproducibility and further research, we will publicly release our evaluation
benchmark, preference training
dataset, and code at https://pegah-kh.github.io/projects/
prompts-override-
vision/ .
Read the full abstract ▾
All authors Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny, Mustafa Shukor, Alasdair Newson, Matthieu Cord
Computation and Language
GiVA: Gradient-Informed Bases for Vector-Based Adaptation
Neeraj Gangwar, Rishabh Deshmukh, Michael Shavlovsky et al. (+4 more)
As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse
benchmarks, including natural
language understanding, natural
language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$).
Read the full abstract ▾
All authors Neeraj Gangwar, Rishabh Deshmukh, Michael Shavlovsky, Hancao Li, Vivek Mittal, Lexing Ying, Nickvash Kani
Computation and Language
Mapping the Political Discourse in the Brazilian Chamber of Deputies: A Multi-Faceted Computational Approach
Flávio Soriano, Victoria F. Mello, Pedro B. Rigueira et al. (+4 more)
Analyses of legislative behavior often rely on voting records, overlooking the rich semantic and rhetorical content of political
speech. In this paper, we ask three complementary questions about parliamentary discourse: how things are said, what is being said, and who is speaking in discursively similar ways. To answer these questions, we introduce a scalable and generalizable computational framework that combines diachronic stylometric analysis, contextual topic modeling, and semantic clustering of deputies'
speeches. We apply this framework to a large-scale case study of the Brazilian Chamber of Deputies, using a corpus of over 450,000
speeches from 2003 to 2025. Our results show a long-term stylistic shift toward shorter and more direct
speeches, a legislative agenda that reorients sharply in response to national crises, and a granular map of discursive
alignments in which regional and gender identities often prove more salient than formal party affiliation. More broadly, this work offers a robust methodology for analyzing parliamentary discourse as a multidimensional phenomenon that complements traditional vote-based approaches.
Read the full abstract ▾
All authors Flávio Soriano, Victoria F. Mello, Pedro B. Rigueira, Gisele L. Pappa, Wagner Meira Jr., Ana Paula Couto da Silva, Jussara M. Almeida
Computation and Language
EVENT5Ws: A Large Dataset for Open-Domain Event Extraction from Documents
Praval Sharma, Ashok Samal, Leen-Kiat Soh et al. (+1 more)
Event extraction identifies the central aspects of events from text. It supports event understanding and analysis, which is crucial for tasks such as informed decision-making in emergencies. Therefore, it is necessary to develop automated event extraction approaches. However, existing
datasets for
algorithm development have limitations, including limited coverage of event types in closed-domain settings and a lack of large, manually verified
dataset in open-domain settings. To address these limitations, we create EVENT5Ws , a large, manually annotated, and statistically verified open-domain event extraction
dataset. We design a systematic annotation pipeline to create the
dataset and provide empirical insights into annotation complexity. Using EVENT5Ws, we evaluate state-of-the-art pre-trained large
language models and establish a
benchmark for future research. We further show that models trained on EVENT5Ws generalize effectively to
datasets from different geographical contexts, which demonstrates its potential for developing generalizable
algorithms. Finally, we summarize the lessons learned during the
dataset development and provide recommendations to support future large-scale
dataset development.
Read the full abstract ▾
All authors Praval Sharma, Ashok Samal, Leen-Kiat Soh, Deepti Joshi