Computation and Language
Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent
Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar et al. (+1 more)
Counterfactual
fairness audits of clinical
language-model
agents report a flip rate: how often an action changes when only the patient's demographic descriptor changes. Part of that rate is not demographic. A
stochastic agent also changes its own action when nothing changes, and a flip rate cannot be interpreted without knowing how often. We measured it. Re-running one condition ten times over sixteen synthetic vignettes at default sampling changed a clinical
agent's action in 8.7 percent of replicate pairs, from 2.2 percent for intensive-care escalation to 17.9 percent for controlled-substance caution, an output given no operational criteria. Across six models from five vendors, pooled floors ranged from 2.5 to 23.7 percent; in this panel neither disclosed size, vendor, nor hosting ordered them. The floor depends on the decoding configuration: majority voting over five draws removed 39 percent of it (95 percent confidence interval (CI) 18 to 64); at temperature 0 three of four locally served models showed no disagreement, but a hosted model still did. We also show that, for a binary action, the flip rate expected under no demographic effect equals the floor and a real effect adds only its square, so a flip rate inside the floor is not evidence of
fairness, and direction must be tested with a signed paired test. We give a four-step reporting procedure and release FairMedAgent, the harness, with its protocol, vignettes and analysis scripts, so any team can measure the floor for its own
agent.
Read the full abstract ▾
All authors Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar, Rahul Joshi
Computation and Language
The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysis
Ahmed Asaad, Amr Mohamed, Yang Zhang et al. (+1 more)
Large
Language Models (LLMs) increasingly use user context such as
memory, profiles, and role
prompts to personalize their responses. This personalization can affect evidence-based judgment: the same evidence may lead to different conclusions under different user contexts. Finance provides a high-stakes setting to study this problem because decisions often depend on interpreting long and complex documents. We test this using 3,575 SEC filings across twelve LLMs. We compare persona-conditioned
retrieval, neutral
retrieval, and
memory-framed context to separate the effect of evidence selection from the effect of interpretation. We find that most user-context spillover comes from how models interpret the same evidence under different roles, rather than from retrieving different evidence. We then test two simple mitigation strategies: expressing the same investor mindset as a user profile instead of an assistant role, and separating evidence-based and personalized outputs. Both reduce spillover, but neither removes it completely, and their effectiveness varies substantially across models.
Read the full abstract ▾
All authors Ahmed Asaad, Amr Mohamed, Yang Zhang, Omneya Abdelsalam
Computation and Language
SWIM: Student Writing Simulation via Proficiency-Conditioned Generation
Heejin Do, Jakub Kontak and Mrinmaya Sachan
Writing proficiency manifests in how students develop content, organize ideas, choose words, and use
language. Despite growing interest in LLM-based student simulation, whether LLMs can reproduce such multidimensional variation in extended writing remains largely unexplored. In this work, we explore if
language models can realistically simulate student writing, and introduce SWIM, a task that formulates Student Writing sIMulation as proficiency-conditioned essay generation. We evaluate prompting, supervised fine-tuning (SFT), and reinforcement learning (RL) methods for writing simulation using automated essay scoring as a measure of profile
alignment. Extensive experiments reveal that prompting provides limited proficiency control, even for strong proprietary LLMs with rubric-grounded strategies. In particular, while models can adjust content-oriented traits, they struggle to reproduce the lexical, grammatical, and organizational variation in different proficiency levels. SFT substantially improves
alignment, while RL with the proposed proficiency-
alignment reward yields further gains across all writing traits and essay
prompts. Our findings suggest that explicit supervision enables substantially stronger profile
alignment than prompting alone, while authentic low-proficiency writing remains challenging to reproduce.
Read the full abstract ▾
Computation and Language
LLMs Learn Better In-Context from Rules than from Examples
Xiang Fu, Seungmin Cho, Yukyung Lee et al. (+1 more)
Large
language models (LLMs) exhibit in-context learning capabilities, where they can learn new tasks from
prompt contexts without weight updates. We compare the learning efficacies of two prominent modes of in-context learning: (1) learning from descriptions of rules (instruction following); and (2) learning from examples of input-output demonstrations (few-shot prompting). Through five learning tasks that cover diverse domains (games, arithmetic, linguistic inferences), we compare two modes of learning (rules vs. examples) specifying the same underlying task. We furthermore explore model and task properties that modulate the learning efficacies. We find that models generally learn more reliably from rules than from examples alone, and additional examples on top of rules or simply scaling up the number of examples do not lead to consistent and significant gains. Instruction tuning amplifies the benefit of rule-based learning while keeping example-based learning capacities intact. Surprisingly, we find no privileged effect of example-based learning in base models, and rules still lead to gains in algebraic task domains. Overall, the comparative efficacy of rules over examples is larger when the task recruits algebraic abstractions and computations, and smaller when the task requires distributional sensitivity and/or recruits parametric knowledge.
Read the full abstract ▾
All authors Xiang Fu, Seungmin Cho, Yukyung Lee, Najoung Kim
Computation and Language · Cross-listed from cs.CV
Learning to Zoom Efficiently with a Contrastive Curriculum
Falko Helm and Iryna Gurevych
Using a zoom-in tool is an important foundational part of modern visual
agents, because it allows to efficiently handle tasks involving high-resolution images. Most previous methods need an extensive warm-start supervised fine-tuning phase for teaching models zoom-in. We show that this is not necessary by proposing a new intrinsic reward for learning tool use in MLLMs without the need for additional labels or warm-start SFT. Our InfoNCE-style reward uses a curriculum of increasingly hard negative tool calls as a contrastive training signal. Empirical experiments on $V^*$, HRBench and MME-RealWorld show that our approach is competitive while being more efficient. When used as a drop-in replacement for SFT, we even outperform all baselines. To directly measure the zoom-in ability of models, we further introduce the scalable synthetic Muffin&Chihuahua (M&C)
dataset. Each image consists of a grid with every
cell either showing a muffin or chihuahua. Leveraging the M&C
dataset's unique region of interest labels, we find that recall is the metric that most strongly correlates the zoom-in region with final task performance. Our model and code for reproduction is publicly available under https://github.com/UKPLab/emnlp2026-zoom-in
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
VoxReason: Auditing Source-Grounded Speech Plans Before Synthesis
Mengzhe Geng
Plan accuracy alone cannot show whether a
speech-delivery decision follows its source: a fixed prior may match the original label yet fail to respond appropriately when a cue changes. VoxReason provides a 100-case verifier
benchmark that holds each utterance fixed, edits one designated source-label cue, and scores cited evidence, eight plan fields, and the permitted response. On a source-key-disjoint test of 24 cases, a source-
emotion prior reaches plan-slot accuracy 0.958, but none of the 24 edited neutral targets appears in its training labels; its required-change accuracy is 0.000. This diagnoses the support boundary of this prior, not its performance on supported edits. In a complementary 32-case
emotion-disjoint test, the prior has seen all edited neutral targets but neither original test
emotion; its plan-slot accuracy is 0.219 and required-change accuracy is 1.000. The partitions reuse and overlap the same 100 cases, so these deterministic diagnostics are not independent cohorts or learned-planner results. The
benchmark evaluates derived labels and structured plans, not audio input, generated
speech, or listener judgments.
Read the full abstract ▾
Computation and Language
MemoryLACE: Memory Lifecycle-Aware Consolidation and Evidence Retrieval
Meriem Yacoubi, Pia Schmidt, Nenad Petrovic et al. (+3 more)
Long-term LLM
agents must preserve information across interactions while distinguishing repeated evidence, historical states, updates, and unresolved contradictions. Existing textual
memory systems retrieve semantically relevant
memories efficiently but often leave these relationships implicit, whereas richer structured approaches model them through global graphs, hierarchical abstractions, or reflection at greater complexity. We introduce MemoryLACE (MemLACE), a lightweight
memory framework that explicitly models the lifecycle of textual evidence through sparse merge, supersession, and contradiction relations while preserving atomic natural-
language memories and their provenance. Rather than retrieving
memories independently, MemLACE reconstructs relation-aware evidence units that expose current, historical, supporting, and conflicting evidence for downstream
reasoning. Across BEAM and StructMemEval, using open-weight and proprietary LLM backbones, MemLACE achieves the highest overall performance in same-backbone comparisons while reducing end-to-end runtime on BEAM by 66.6% relative to Hindsight, the strongest reported reflective-
memory baseline. Ablation studies identify lifecycle expansion and temporal awareness as the principal contributors to these gains. Together, the results demonstrate that explicitly modeling the local lifecycle of textual evidence is sufficient to substantially improve long-term
memory reasoning without requiring comprehensive knowledge graphs or global reflection.
Read the full abstract ▾
All authors Meriem Yacoubi, Pia Schmidt, Nenad Petrovic, Ahmed Frikha, Martin Kirchhoff, Alois Knoll
Computation and Language
Jina-OCR-v1: Efficient Document Parsing with Speculative Decoding and Dense Verifiable Rewards
Alejandro Barón García, Feng Wang, Emilia Garcia Casademont et al. (+1 more)
We present Jina-OCR-v1, an end-to-end document parsing model built to serve on low-budget GPUs. It combines the compressed-
vision encoder and the 3B mixture-of-experts decoder of DeepSeek-OCR, which activates about 570M parameters per
token, with a FastMTP speculative decoding head that shares a single draft block recursively across K=3 prediction steps. Greedy verification makes decoding lossless. Post-training combines instruction
alignment, robustness fine-tuning on difficult documents, and GRPO under dense verifiable rewards: deterministic formula, table, and structural checks that award partial credit. The training data mixes cleaned public corpora with targeted synthetic pages. At the default dynamic-resolution setting, Jina-OCR-v1 scores 91.14 on OmniDocBench v1.6 and 83.4 on olmOCR-Bench, and reaches the highest page throughput in our comparison at 2.57 pages per second. On a low-budget GPU such as the NVIDIA L4, FastMTP doubles decoding speed over greedy autoregressive decoding. The model is publicly available at https://huggingface.co/jinaai/jina-ocr-v1.
Read the full abstract ▾
All authors Alejandro Barón García, Feng Wang, Emilia Garcia Casademont, Han Xiao
Computation and Language
No country for old linguists: LLM-brain alignment underdetermines neural computation
Elliot Murphy
Nastase et al. (2026) argue that large
language models (LLMs) may illuminate
language processing because both rely on distributed, context-sensitive representations shaped by statistical learning. Their rejection of simple cortical "boxology" is persuasive, and they articulate a strong case for the value of LLM-
brain alignment research. The key question is what kind of inference LLM-
brain alignment licenses. My claim here will be narrow: representational
alignment can in principle constrain mechanistic hypotheses, but it does not by itself identify a mechanism. Nastase et al. acknowledge that an encoding model can capture features represented in neural activity without establishing a shared architecture or
algorithm. Yet the authors sometime move from
alignment to "shared computational principles" and ultimately to LLMs as mechanistic models of natural
language. Indeed, their methodological caveat that
alignment does not establish a shared architecture or
algorithm sits uneasily with their conclusion that LLMs might instantiate the same computational principles as biological
brains and provide a "fully mechanistic model" of
language. I discuss what I consider to be problems of logical, causal, and computational underdetermination in Nastase et al.'s (2026) proposal.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization
Qingchan Zhu, Weihang You, Hanqi Jiang et al. (+3 more)
Visual
token pruning reduces the inference cost of
vision-
language models (VLMs), but most methods only ask which
tokens to keep. This retained-
token view can keep redundant high-scoring
tokens while leaving discarded evidence without a close representative. We propose CoverPruner, a training-free pruner that asks the complementary demand-side question: after a
token is removed, which surviving original
token represents it for the target VLM? CoverPruner formulates
pruning as Representational Coverage Maximization (RCM), covering the full projected visual-
token set with query-weighted demand. It instantiates RCM with projector-space coverage and a lightweight first-layer
attention probe. Across multiple VLM architectures and compression rates, CoverPruner achieves the best average accuracy among all compared methods, with the largest gains usually appearing under aggressive compression.
Read the full abstract ▾
All authors Qingchan Zhu, Weihang You, Hanqi Jiang, Changdi Yang, Tianming Liu, Geng Yuan
Computation and Language · Cross-listed from cs.LG
Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning
Mehreen Hossain Chowdhury, Nowshin Mahjabin, Ahmed Shafin Ruhan et al. (+3 more)
Multi-domain fine-tuning often combines MoE routing with LoRA, assuming that
token-level routing separates domain-specific updates. We test this assumption in MoE+LoRA using Python code paired with biomedical text and mathematical
reasoning. Although these domains show near-disjoint expert routing, adding biomedical data substantially increases code perplexity, indicating that routing separation alone may not prevent negative transfer. To localize the failure, we introduce Jaccard routing overlap and adapter-gradient cosine similarity, which measure expert sharing and update compatibility, respectively. These diagnostics indicate that interference arises mostly from nearly orthogonal domain gradients competing within the same low-rank adapter subspace. We address this issue with SpawnLoRA, which dynamically adds gated sub-adapters inside MoE experts when adapter-level contention is detected, while keeping the router fixed. We evaluate SpawnLoRA on Phi-tiny-MoE-instruct and OLMoE-1B-7B across multiple mixture settings and find that it effectively reduces negative transfer compared with standard and rank-adaptive LoRA. These results demonstrate that structural separation inside experts provides benefits beyond routing or rank expansion alone.
Read the full abstract ▾
All authors Mehreen Hossain Chowdhury, Nowshin Mahjabin, Ahmed Shafin Ruhan, Md Azam Hossain, Abu Raihan Mostofa Kamal, Md Tahmid Rahman Laskar
Computation and Language
Large Language Models in Resolving Contextual Knowledge Conflicts
Xinye Yang, Zhenyang Liu, Ruisi Li et al. (+1 more)
Most prior works focused on conflicts between an LLM's internal parametric knowledge and externally provided context. In contrast, we investigate how LLMs handle conflicts that arise within contextual knowledge itself. We introduce a taxonomy of six types of contextual conflicts (factual, inferential, temporal, granularity, perspective, and ambiguity) and contribute a comprehensive
dataset ContextConflict for this setting. The
dataset contains 5,781 samples, covers both
reasoning and summarization tasks, and includes both explicit contradictions and implicit conflicts that require multi-step
reasoning. Experiments on nine LLMs show that current models still fall short in resolving contextual knowledge conflicts. We further provide mechanistic interpretability insights into how LLMs process such conflicts, revealing their latent awareness of conflicts and the representational geometry underlying conflict processing. In addition, our analysis uncovers a consistent model bias towards earlier evidence, and this positional preference serves as a key obstacle to effective conflict resolution. Motivated by these findings, we further propose a simple training-free, label-free steering method that steers activations to encourage a more comprehensive incorporation of evidences for better conflict resolution. On our
dataset, the method consistently improves accuracy on
reasoning tasks and generates higher-quality, more balanced summaries for summarization tasks.
Read the full abstract ▾
All authors Xinye Yang, Zhenyang Liu, Ruisi Li, Yuanyuan Lei
Computation and Language
SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking
Michael J. Bommarito II
Libraries and archives manage large collections with limited staff and computing budgets, yet common
benchmarks do not systematically test their bibliographic work. They need to know which methods work for their tasks and what those methods require to run. SHELF, the Synthetic Harness for Evaluating LLM Fitness, addresses this gap. It is a Python system that turns labelled taxonomies, writing specifications, and a generation budget into controlled
benchmark data and evaluation tasks. This first release contains 62,899 model-written documents based on Library of Congress vocabularies, with tasks for classification, clustering,
retrieval, pair classification, and instruction
retrieval. We compare TF, TF-IDF, BM25, popular encoders, and, on subject classification only, zero-shot decoders; each method appears only on tasks that support it. Subject classification reaches 0.8887, while genre-form classification reaches only 0.2605, and several pair and clustering tasks remain near chance. Sparse methods remain competitive on classification, while TF-IDF is the fastest measured arm in the subject timing experiment. SHELF also varies bibliographic facets independently and can generate new, verifiably unseen documents after a model's training cutoff. Comparisons with LCSHBench and Project Gutenberg show that model rankings transfer more reliably than absolute scores, but SHELF scores do not estimate accuracy on production catalogue data. We release all source code and data under permissive licenses on GitHub and Hugging Face.
Read the full abstract ▾
Computation and Language
Unifying Conformal Language Tasks with In-Context Ensembles
Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara et al. (+2 more)
Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM
prompts asking the model to rate the importance of content, but manual
prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.
Read the full abstract ▾
All authors Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara, Kin Kwan Leung, Jesse C. Cresswell
Computation and Language · Cross-listed from cs.LG
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Zhiwei Zhang, Zechen Sun, Fei Zhao et al. (+6 more)
On-policy
distillation (OPD) accelerates post-training by providing dense
token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across
prompts, without checking whether the teacher is reliable for each
prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as
entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy
Distillation (TGOPD), built on the principle that teacher reliability should be verified at the
prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each
prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-
benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.
Read the full abstract ▾
All authors Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng, Bin Liang, Huayu Deng, Yao Hu, Kam-Fai Wong, Mu Chuan
Computation and Language · Cross-listed from cs.LG
The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors
Toni J.B. Liu, Jiajun Bao, Yizhou Liu et al. (+4 more)
What does a
language model predict when it has few clues? The answer lurks in its unembedding geometry: a single direction of the unembedding
matrix encodes the unigram distribution of the training corpus, which serves as the
Bayesian prior the model falls back on when uncertain. This structure — which we term the direction of ignorance — appears in all four model families examined (Llama, Qwen, Gemma, and Pythia), ranging from 0.4B to 405B parameters. Projecting the final prediction state onto this direction yields a per-
token prior loading factor $\lambda$, which, empirically, declines steadily as the context becomes more informative. Formally, the same projection decomposes the prediction state into two orthogonal vectors that correspond exactly to the two factors of a tempered
Bayesian update: a unigram prior raised to the exponent $\lambda$ and a context-driven likelihood. This
geometric-probabilistic interpretation calibrates $\lambda$, making it meaningfully comparable across model sizes and families, with larger models generally exhibiting lower prior reliance in the high-context limit. Finally, we show that the direction of ignorance is causally active: raising or lowering $\lambda$ at the final prediction state steers the prediction toward or away from the unigram prior in KL divergence.
Read the full abstract ▾
All authors Toni J.B. Liu, Jiajun Bao, Yizhou Liu, Gurbir Arora, Nicolas Boullé, Raphaël Sarfati, Christopher J. Earls
Computation and Language
LexIssue: Benchmarking Legal Issue Identification in Chinese Civil Litigation
Huiyuan Xie, Yuqin Huang, Zhicheng Hao et al. (+4 more)
Identifying the issues disputed between litigating parties is a crucial component of real-world litigation. However, legal issues remain comparatively underexplored in legal AI research. In this work, we study the computational modelling of legal issue identification in litigation. We introduce a legally grounded hierarchical schema that represents legal issues through both free-form issue descriptions and structured legal categories, and formulate legal issue identification as two complementary tasks: legal issue generation and legal issue classification. Based on this formulation, we construct LexIssue, a
benchmark containing 430 real-world Chinese civil litigation cases and 1,303 expert-annotated disputed legal issues. We further develop an issue-centric legal knowledge base spanning 27 causes of action and 441 candidate legal issue entries to support
retrieval-augmented
reasoning. Experimental results across a diverse set of models show that
retrieval-augmented generation using the constructed legal issue knowledge base consistently improves performance in identifying disputed legal issues and their corresponding legal attributes.
Read the full abstract ▾
All authors Huiyuan Xie, Yuqin Huang, Zhicheng Hao, Yida Cai, Shaochun Wang, Zhenghao Liu, Yuxiao Ye
Computation and Language
User Feedback Provides a Unique Signal that LLMs Can not Detect
Shachar Don-Yehiya, Leshem Choshen and Omri Abend
Harnessing naturally occurring feedback from user interactions offers a promising learning signal for Large
Language Models (LLMs). However, recent studies suggest this feedback is inherently noisy and difficult to leverage effectively. We challenge this conception by demonstrating that user feedback is a highly actionable signal for improvement, and that its perceived ineffectiveness stems from a systematic bias in current evaluation paradigms. To isolate the usefulness of feedback, we construct synthetic data with a definitive ground truth, alongside naturalistic data to validate that our findings hold in real-world scenarios. By comparing model revisions generated with and without access to feedback across both settings, we show that feedback-informed revisions resolve targeted issues at significantly higher rates than baseline revisions. Finally, we expose the root of the evaluation bias: when a model successfully fixes an issue exclusively due to feedback, LLM judges frequently fail to identify the genuinely corrected response, systematically preferring inferior baseline outputs instead.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Post-Training Language Models for Gold-Medal Performance in Coding Competitions
Aleksander Ficek, Sean Narenthiran, Mehrzad Samadi et al. (+2 more)
Competitive programming has become a key test of large
language model
reasoning, with international competitions such as IOI and ICPC representing its most challenging settings. We present an end-to-end specialization pipeline combining large-scale problem curation, synthetic
reasoning traces, supervised fine-tuning (SFT), and reinforcement learning (RL). Using 22,000 curated problems, we train Nemotron-3-Nano-CC (30B-A3B) with SFT and RL and Nemotron-3-Ultra-CC (550B-A55B) with SFT alone. We further introduce GenCorrect, a feedback-driven test-time compute strategy that iteratively generates, evaluates, and refines diverse solutions. On IOI 2025, Nano-CC improves from 130 points to 291 after post-training and to 468 with GenCorrect, exceeding the gold threshold of 438.3 while Ultra-CC reaches 502. Guided by these results, we develop a competition-specific Ultra-CC system and evaluate it prospectively during IOI 2026. Under the same time, internet-access, and submission constraints as human contestants, it scores 535.4 out of 600, exceeding both the gold threshold of 361.12 and the top human score of 498.27. To our knowledge, this is the first AI system to outscore the highest-scoring human contestant on an IOI problem set.
Read the full abstract ▾
All authors Aleksander Ficek, Sean Narenthiran, Mehrzad Samadi, Somshubra Majumdar, Boris Ginsburg
Computation and Language · Cross-listed from econ.GN
Dutch Books for Language Models
Isaiah Andrews and Suproteem Sarkar
People increasingly use
language models to support life decisions. Many such decisions involve a probabilistic forecast: How likely is a major life event, a natural disaster, or an economic outcome? Users of
language models may implicitly trust that these forecasts fall out of a coherent world model. In this paper, we evaluate the coherence of
language model probabilistic forecasts through a procedure that builds on a
theorem due to de Finetti. We elicit forecasts from
language models across events generated from stock returns data. We then use linear programs to compute the largest Dutch-book profit - the profit an arbitrageur could guarantee by betting against model-generated probabilities - which we use as a measure of incoherence. Our procedure does not require outcome labels, so we can evaluate coherence even in settings where outcomes are not observed or have not yet resolved. We find substantial evidence of incoherence in
language model forecasts. Such incoherence increases when there are richer logical relationships between events, and irrelevant contextual details can increase incoherence by an order of magnitude. We conclude by discussing how alternative training strategies may improve probabilistic coherence.
Read the full abstract ▾
Computation and Language
DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation
Vasileios Baltatzis, Mert Inan, Connor Gillis et al. (+4 more)
Sign
language processing systems have traditionally operated at the sentence level, ignoring critical discourse phenomena fundamental to sign
language comprehension. We introduce DiscoSign, a computational approach for discourse-aware text to sign
language gloss translation grounded in linguistic research. We address three key phenomena within our modular Large
Language Model (LLM)-based translation framework: (i) spatial coreference resolution, where entities maintain consistent spatial locations throughout discourse; (ii) Question-Answer Clauses (QACs), pseudocleft structures serving specific discourse functions; and (iii) concept-gloss consistency, ensuring stable mappings between English concepts and American Sign
Language (ASL) signs. Traditional translation metrics fail to capture discourse-level quality, so we introduce a suite of novel evaluation metrics designed to assess each dimension of discourse coherence addressed by our framework. Experiments on sentence-level and discourse-level
datasets show that our approach for discourse-aware processing significantly improves spatial consistency and entity tracking relative to sentence-only translation, while maintaining competitive single-sentence gloss translation quality. Our work establishes the first systematic framework for discourse-level text to sign
language gloss translation with corresponding evaluation methodology.
Read the full abstract ▾
All authors Vasileios Baltatzis, Mert Inan, Connor Gillis, Raja Kushalnagar, Lorna Quandt, Leah Findlater, Colin Lea
Computation and Language
EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction
Yuling Shi, Zhensu Sun, Junsen Dong et al. (+3 more)
Evaluating LLM
agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic
benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on
benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an
agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an
agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three
benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of
agent steps and up to 44.1% input
tokens and 29.4% output
tokens at 89%-97% prediction accuracy, while perturbing per-
agent resolve rates by only one to two percentage points on average.
Read the full abstract ▾
All authors Yuling Shi, Zhensu Sun, Junsen Dong, Chengcheng Wan, David Lo, Xiaodong Gu
Computation and Language · Cross-listed from cs.CV
ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding
Jitai Hao, Ke Yang, Di Yan et al. (+3 more)
Streaming
video understanding is a critical capability for real-world applications, including
embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous
video streams with
multimodal large
language models (MLLMs) is computationally expensive. Existing efforts have explored reducing streaming overhead through visual
token pruning,
token merging, quantization, on-demand frame
retrieval, and context offloading. However, most existing methods overlook the dimension of model depth. Repeatedly executing full-depth MLLM prefill over incoming frames is prohibitively expensive, incurring substantial computational overhead and causing the KV cache to grow at a rate directly proportional to the prefill depth. To address these challenges, we propose ShallowStream, a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and
retrieval index building. During stream processing, ShallowStream maintains an always-on lightweight index using the KV cache of shallow layers. During query-time answering, we leverage the
attention scores generated by the shallow layers to score context frames and employ a diversity-aware selection strategy to retrieve precise and comprehensive evidence. ShallowStream achieves performance on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively. Our code is available at https://github.com/CURRENTF/ShallowStream.
Read the full abstract ▾
All authors Jitai Hao, Ke Yang, Di Yan, Fan Liu, Qiang Huang, Jun Yu
Computation and Language
HyperStyler: Low-resource Authorship Style Transfer via Context-aware Style Navigation and Hypernetworks
Jongkyung Shin, Minguk Jeon, Chanwoo Park et al. (+1 more)
Low-resource authorship style transfer (LAST) aims to rewrite text into the style of an arbitrary target author using only a few reference examples while preserving the original meaning. Existing methods often struggle to achieve both high style fidelity and semantic preservation because they compress diverse references into a single static author
embedding, which averages out context-dependent stylistic variation, and rely on hidden representations for style control, which entangle style with content. We propose HyperStyler, a novel architecture that decouples LAST into style selection and style realization. Stylo-navigator predicts style coordinates by jointly modeling the source context and target-author references, and Stylo-hypernet realizes them via dynamic parameter modulation instead of hidden-state injection. Our experiments on Reddit, Blog, and News
datasets demonstrate that HyperStyler consistently outperforms prior methods including LLM-based approaches and generalizes robustly across domains. Notably, HyperStyler achieves superior performance with as few as 2.4% additional parameters over T5-large, while being over 1.8x faster than LLMs at inference.
Read the full abstract ▾
All authors Jongkyung Shin, Minguk Jeon, Chanwoo Park, Chiehyeon Lim
Computation and Language
From Reweighting to Rewriting: Unlocking the Intervention Effects of Influential Samples in Training Data Attribution
Yuzhang Luo, Chenpeng Wang, Jianhui Chen et al. (+1 more)
Training data attribution (TDA) aims to identify training examples that shape model behavior, but its intervention value depends on both which examples are selected and how they are modified. Influence functions (IF) estimate behavioral changes under infinitesimal reweighting, yet IF-selected examples often show limited advantages over random selection under conventional weight-based interventions. This raises the question of whether influential examples lack intervention value or whether reweighting fails to realize their behavioral leverage.We introduce influence-guided response rewriting, which uses IF to identify intervention targets and replaces their responses with behavior-aligned or behavior-opposed supervision while keeping instructions fixed. Across four open-weight LLMs, we compare rewriting and reweighting on the same influence-selected examples using epistemic abstention as our primary testbed. Response rewriting produces stronger, more persistent, and bidirectional behavioral shifts, while reweighting the same examples yields weak and inconsistent effects. Further analyses show that influence-selected examples provide greater rewriting leverage than alternative selectors, with changes remaining concentrated on target-relevant behaviors. The same qualitative contrast extends to safety refusal. These results distinguish the local reweighting effects captured by influence estimates from the broader intervention leverage of the examples they identify, motivating intervention-aware evaluation of TDA methods.
Read the full abstract ▾
All authors Yuzhang Luo, Chenpeng Wang, Jianhui Chen, Liangming Pan
Computation and Language
Untangling the Mechanisms of Misleading Context in Medical Question Answering
Robin Linzmayer and Noémie Elhadad
Large
language models now answer medical questions with expert-level performance. However, the context these systems act on can be misleading, and misleading context can corrupt a model's medical judgment. To understand how misleading context corrupts this judgment, we examine the model's susceptibility to the context, disclosure of it, mechanism of corrupted
reasoning, and monitorability of the decision. On the medical
reasoning subset of MedMisBench, a clinician-reviewed question-answering
benchmark of 8,627 questions, we inject two types of misleading context cues, fabricated evidence and a bare assertion. We test three
reasoning models, two that expose their full
reasoning trace and one frontier model that exposes only its response. All three are more susceptible to the assertion than to the fabricated evidence, adopting the asserted answer 10 to 27 points more often. The misleading cues are disclosed in 81 to 98% of traces but only 7 to 90% of responses, and the assertion is disclosed less often than evidence based cues. Resampling from
reasoning traces without disclosure shows the two cues corrupt
reasoning differently, evidence entering early and accumulating while the assertion redirects the conclusion near its end. An LLM monitor catches 78% of corrupted decisions at 5% false positives when reading an open model's trace with guidance, against at most 32% from any response. The misleading context that models are most susceptible to is disclosed least, and was caught reliably only from an open
reasoning trace, which frontier providers withhold.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
Jianlyu Chen, Yuyang Hu, Hongjin Qian et al. (+8 more)
Autonomous
agents are beginning to carry out machine-learning (ML) research end to end. These
agents combine a model backbone with a harness for planning, execution,
memory, and verification, but this architecture still leaves domain-specific know-how outside the
agent. We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work. That knowledge is not absent from the field. It appears in repositories and papers, but in forms written for human readers and too large to load during a task. Once distilled into compact, verified skills, this knowledge can be reused across tasks rather than rediscovered during each run. We present DisCo, a skill-powered research
agent that creates skills and uses them during research. Its
distillation runs in two complementary forms: task-agnostic, condensing the field's widely used repositories into reusable skills, and task-oriented, producing the skills a concrete task calls for. The former, applied across the open ecosystem, yields the AREX-Skill Library, with 5,000+ verified skills distilled from 1,000 widely used ML repositories and organized into 20 areas and 178 capability families. With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped research
agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet than the same
agent without skills. These gains come from adding distilled operating context under that fixed setup.
Read the full abstract ▾
All authors Jianlyu Chen, Yuyang Hu, Hongjin Qian, Jiawei Liu, Wenqing Wei, Xiaolong Chen, Defu Lian, Zhicheng Dou, Chaozhuo Li, Qiwei Ye, Zheng Liu
Computation and Language · Cross-listed from cs.IR
Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection
Max Nelson, Hanoz Bhathena, Aviral Joshi et al. (+1 more)
Selecting a
retrieval model for a production RAG system requires reliable comparative evaluation, but obtaining relevance judgments at scale is expensive and difficult to repeat as new candidate systems arrive. We study pooled LLM evaluation, in which an LLM judges the union of documents retrieved by the current set of candidate systems, and the pool is then expanded incrementally as new systems are introduced by judging only the new documents they contribute. These judgments are reused to evaluate all systems on a common basis. We validate this approach on four
retrieval benchmarks with 11 systems spanning dense, sparse, and hybrid configurations, and deploy it to compare 62
retrieval configurations for a financial news QA system. Pooled LLM rankings correlate strongly with gold-standard evaluation across
datasets, and 97% of pairwise system orderings are preserved once bootstrap uncertainty in the qrels is taken into account. In production, document overlap yields 65-80% judgment reuse and up to 4.9x lower evaluation cost, allowing teams to
benchmark new
retrieval candidates without re-judging previously assessed documents. These results suggest pooled LLM evaluation is a practical and cost-effective workflow for incremental
retrieval model selection in deployed systems.
Read the full abstract ▾
All authors Max Nelson, Hanoz Bhathena, Aviral Joshi, Saket Sharma
Computation and Language
Namgyu Ho, Huzama Ahmad, Woosung Koh et al. (+3 more)
Language models spend most of their
attention on a small fraction of context, yet they read the entire KV cache to find the few
tokens that matter. If the user asks about a previous detail in a 1M-
token conversation, global
attention layers must scan the full context to generate each
token of the reply. A prominent approach pre-selects relevant
tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? We introduce Declarative
Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) reduces total attended
tokens per response (45.8%, 19.6%), despite extra decoding steps, with modest accuracy drops (1.40pp, 2.10pp) that shrink with model scale. DA unlocks a new axis of sparse
attention, with further potential under training-based methods. Code is available at https://github.com/itsnamgyu/declarative-
attention.
Read the full abstract ▾
All authors Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
Computation and Language
Choosing a PEFT Variant for Per-Patient Dysarthric ASR: A Single-Speaker Case Study on Two ASR Bases
Bernard Muller, László Tóth and LaVonne Roberts
Per-patient adapters are the preferred production architecture for dysarthric automatic
speech recognition (ASR), yet parameter-efficient fine-tuning (PEFT) variants have not been compared in the speaker-dependent, per-patient regime. We present a single-speaker case study comparing seven LoRA-family methods (LoRA, QLoRA, AdaLoRA, DoRA, LoHA, VeRA, VB-LoRA) on two production bases (Whisper-large-v3 with Hungarian fine-tuning, and a
multilingual Qwen3-ASR-1.7B checkpoint) for one post-stroke Hungarian male speaker (S1, 409 utterances; severe dysarthria on auditory-perceptual clinical assessment).
Attention-projection adapters substantially improve CER on both bases. Across three seeds, a paired bootstrap detects no significant LoRA-DoRA difference (p>0.5; 13.86/13.90 % CER on Whisper, 28.10/28.33 % on Qwen3-ASR), so we adopt the simpler, cheaper LoRA. Real 4-bit (NF4) QLoRA is worse on every seed and both bases (14.56/30.09 % CER) with no
memory saving at this scale, and LoHA, VeRA, VB-LoRA and AdaLoRA do not reach the LoRA family, though LoHA still gives an 18.6 % relative CER reduction on Whisper. On the same base, full fine-tuning is more accurate (11.43 % CER), but a 115 MB LoRA that also adapts the feed-forward blocks reaches within 0.66 pp of it at approximately 3.7 % of the per-patient storage. A 6-point enrollment grid shows about 5 min of patient audio captures 45.6 % of the zero-shot-to-30-min CER reduction, with further gains at 10 and 30 min (caveat: one speaker, one
language, severe post-stroke dysarthria). Training scripts and recipes will be released, source-available under a research-use licence, on publication.
Read the full abstract ▾