Computation and Language · Cross-listed from cs.AI
Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting
Liang Twist Shan, Tianyu Hu, Hao Yan et al. (+1 more)
Large
language models (LLMs) have been rapidly improving in long-context tasks, powered by Chain-of-Thought (CoT)
reasoning. However, the internal mechanisms underlying this improvement remain unclear. We investigate these mechanisms through a needle-in-a-haystack (NIAH) counting task, where an LLM is asked to count the number of records dispersed in a long text. Across twelve model comparison groups, Thinking (or
reasoning) improves counting accuracy over Non-thinking, with pronounced gains at larger counts. This motivates our mechanistic analysis, which identifies two contrasting mechanisms: (i) broad
retrieval, where Non-thinking models broadly attend to multiple needles; (ii) targeted
retrieval, where Thinking models use enumeration in CoT traces to successively retrieve needles. Targeted
retrieval concentrates
attention on individual needles and is accompanied by more compact internal representations. Moreover, causal intervention analysis suggests that Thinking models use the CoT trace to maintain and update an internal counter as needles are successively retrieved, even without explicit numbering. In small controlled experiments, both
retrieval mechanisms and counter states emerge under standard autoregressive training. Together, our results connect long-context
retrieval with representation geometry of counting, supporting a state-tracking account of CoT
reasoning.
Read the full abstract ▾
All authors Liang Twist Shan, Tianyu Hu, Hao Yan, Yiqiao Zhong
Computation and Language
GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis
Qisheng Su, Hanchen Wang, Guanru Zhu et al. (+10 more)
Working
agents need to read diverse files, coordinate tools, and produce deliverables. Training such
agents requires tasks built on many real files with verifiable results, but few pipelines exist to synthesize this kind of data. Existing pipelines either generate files with models, which lack realism and diversity, or build tasks on real files without task-specific verifiers, leaving result quality unchecked. We introduce GraphForge, an evidence-graph based framework that grounds both the task and its verification in real files. Starting from occupation-grounded seeds for controlled diversity, GraphForge assembles a workspace of real files for each seed and builds an evidence graph over their relations. Since the task statement and rubrics are both derived from this graph, task requirements are backed by the workspace files and each criterion is anchored to the files needed to verify it. An initial rollout further tests executability, and a revision
agent repairs the task and rubrics against the original files before trajectories are collected. Fine-tuning Qwen3.6-27B on 2,169 GraphForge trajectories brings GDPVal to 1445.7 (+65.7) under OpenHands, and Workspace-Bench-Lite and SpreadsheetBench II to 63.7 (+7.7) and 24.0 (+13.7) under Claude Code. Rejection fine-tuning on the SFT model's own rollouts, with candidates selected by the evidence-anchored rubrics, yields further improvements on all three
benchmarks, suggesting that the rubrics provide a useful selection signal.
Read the full abstract ▾
All authors Qisheng Su, Hanchen Wang, Guanru Zhu, Huicheng Jiang, Qiuyinzhe Zhang, Kou Shi, Zhen Fang, Ziao Zhang, Qingnan Ren, Honglin Guo, Zehui Chen, Tao Gui, Feng Zhao
Computation and Language · Cross-listed from cs.LG
Yingchuan Zhang, Haoran Lu, Wenxuan Zhong et al. (+1 more)
Knowledge
distillation can transfer
reasoning from stronger teachers to frozen students through reusable
prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unverified, and agreement with the teacher can reward shared mistakes. We introduce Knowledge-to-
Prompt (K2P) for label-free knowledge
distillation to
prompts. K2P synthesizes reusable instructions from teacher solutions, refines them using paired teacher and student responses, and guides search and selection with answer agreement. It retains candidates that adaptive search may undervalue and selects on reserved questions. Deployment uses only the frozen student and selected
prompt. Our theory separates generation and selection gaps and gives conditions under which agreement-guided construction yields accuracy guarantees despite imperfect teacher references. Across
reasoning tasks and students, K2P outperforms label-free alternatives overall and remains competitive with supervised
prompt optimization. Ablations and archive diagnostics assess the contributions of teacher solutions and refinement, while revealing the limits of agreement-guided selection.
Read the full abstract ▾
All authors Yingchuan Zhang, Haoran Lu, Wenxuan Zhong, Ping Ma
Computation and Language · Cross-listed from eess.AS
VOSSA: Voiceprint Optimization for Streaming Speech Architectures
Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah et al. (+1 more)
Real-time voice conversion (VC) systems commonly rely on pretrained speaker
embeddings from automatic speaker verification (ASV) models. While effective for speaker discrimination, these
embeddings are trained to remain stable across phonetic and prosodic variations within-speaker, which may conflict with frame-level acoustic generation in streaming constraints. To address this issue, we propose VOSSA (Voiceprint Optimization for Streaming
Speech Architectures), a speaker representation framework that extracts speaker information from intermediate content encoder layers and aggregates using attentive statistics pooling. The
embedding is trained jointly with VC objectives, removing the need for a separate speaker encoder. Across six
datasets, VOSSA improves F0 dynamics and vowel-discriminative acoustic cues while maintaining comparable NISQA-MOS, WER, and speaker similarity. Perceptual tests further indicate improvements in naturalness, speaker similarity, intelligibility, and vibrancy.
Read the full abstract ▾
All authors Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah, Ricardo Gutierrez-Osuna
Computation and Language · Cross-listed from cs.SD
Audio Token Attention Is Predictable Before the Language Model Runs
Kyoungjun Park, Yunzhe Li and Lili Qiu
A large audio
language model (LALM) turns a minute of
speech into 750-1,500
tokens and prefills every one. Image-
token pruning often cuts after the
language model's first layers, where image
tokens draw little
attention. Audio
tokens draw much more
attention there, and their ranking is still far from final, so audio needs a ranking before the
language model runs. Surprisingly, the
attention an audio
token will receive across the
language model is already linearly predictable from its encoder output, before the
language model runs. A linear map, fitted in closed form without labels, predicts this all-layer
attention ranking at $\rho \geq .69$ on eleven of thirteen LALMs. Our method, Triage, cuts audio
tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the
attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the
language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io
Read the full abstract ▾
Computation and Language
TRACE: Target-Aware Retrieval, Attributed Evidence, and Contract-Constrained Extraction for LitTraceQA
Sachin Gupta and Divya Godara
Finding a relevant paper is not the same as producing a verifiable answer from it. LitTraceQA requires canonical paper identifiers, exact evidence at the page or object level, and typed answers that match the evaluator. We call the separation between source access and scorer-visible correctness the grounding contract gap. TRACE - Target-Aware
Retrieval, Attributed Evidence, and Contract-Constrained Extraction - addresses this gap with target-grouped
retrieval, independent typed evidence localization,
multimodal table extraction, schema-driven table construction, and fail-closed validation. It indexes 27,487 papers through passage, object, alias, citation, and dense representations while retaining the question target behind each signal. For tables, TRACE predicts the observation unit before extracting values and assembles rows with evaluator-compatible key normalization. Our audited selected clean-track artifact scores 0.760613 on the official 71-question test set, including 0.9728 paper F1, 0.6847 evidence F1, 0.9800 multiple-choice accuracy, 0.5423 table-row F1, and 0.3508 macro
cell accuracy. On 11 public-development table records, a clean baseline and coordinate-aware visual fill obtain row F1 of 0.291 and 0.411, respectively; this diagnostic comparison includes fallback outputs and is not an official-test claim. Remaining errors chiefly concern locator, observation-unit, row-key, and source-value identity.
Read the full abstract ▾
Computation and Language
Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis
Xia Hu, Brian Potetz, Chun-Ta Lu et al. (+6 more)
End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic
benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84% of the gain from supplying all prerequisites.
Read the full abstract ▾
All authors Xia Hu, Brian Potetz, Chun-Ta Lu, Huanfen Yao, Leonidas Guibas, Zhicheng Wang, Howard Zhou, Pengfei Xing, Andrew Gallagher
Computation and Language · Cross-listed from cs.AI
OpenJev-RLCD: A Working RLCD Implementation
Zhimin Gao and Pichao Wang
Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes
reasoning models overconfident. We present a working implementation of reinforcement learning for calibrated decisions (RLCD) for
reasoning models: the model samples a rationale, and we score the answer distribution it commits to afterwards with a strictly proper scoring rule. A variance identity shows that scoring the mixture of several samples rewards disagreeing rationales, and that RLVR is exactly this mixture objective without its diversity term. Optimized naively, the per-rationale objective either switches
reasoning off or is drowned out by policy-gradient noise, which leads to a two-stage recipe: calibrate, then reinforce. With Qwen3-1.7B on two
reasoning tasks (3 seeds, paired tests), RLCD matches or beats SFT, RFT/
STaR and GRPO (each temperature-scaled) in accuracy and beats all of them in selective prediction; on GSM8K answer verification a single query decides \gvTwoCovFive% of the items at $\le$5% error, versus \gvGrpoCovFive% for GRPO. When uncertainty comes from annotator disagreement, RLCD provably cannot beat cross-
entropy. Code and results: https://github.com/ZimmyGao/openjev-rlcd.
Read the full abstract ▾
Computation and Language
Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Head
Zizhuo Fu, Runsheng Wang and Meng Li
Scaling
attention parameters can improve
language model quality, but retaining full
token histories makes additional heads costly at long contexts. Furthermore, since
attention retrieves and combines contextual information, parameter scaling should also support longer contexts. We therefore ask whether
attention parameter scaling can directly enable efficient and effective context scaling. We introduce NAMOH, an architecture-native sparse
attention mechanism that activates $K$ of $H$ heads per
token. Each head retains only its assigned
tokens and performs causal
attention within this subsequence. Head selection thus jointly determines active parameters and available context without scanning the full history. Under balanced assignments, increasing $H$ at fixed $K$ shortens head histories and reduces per-
token key-value (KV) access without increasing total KV storage. We further support head-relative rotary position
embeddings to shorten positional spans within routed subsequences, aiming to mitigate position-induced
attention noise. Experiments show that NAMOH can outperform fully activated models with the same total parameters, while enabling more efficient long-context inference than smaller dense models with matched active parameter counts. It remains compatible with GQA and existing sparse
attention mechanisms. We hope this work offers a new path for scaling
attention, with parameter scaling directly enabling context scaling.
Read the full abstract ▾
Computation and Language
Forging LLM Authorship Fingerprints with Targeted Rewriting
Haohan Yuan, Simin Chen, Xi Niu et al. (+3 more)
Model-attribution classifiers can often identify which
language model produced a text, making model-specific writing patterns a signal of provenance. Accurate attribution on unmodified text, however, does not show whether the prediction still identifies the original source after deliberate rewriting. We formulate this problem as targeted fingerprint transfer: rewriting one model's output so that attribution classifiers assign it to a chosen target model. We study summarization, where different models receive the same document and express the same underlying content, providing a controlled setting for conditional generation. We introduce ForgePrint, a search-then-distil framework that first searches for rewrites that move attribution toward a target fingerprint, then distils the selected rewrites into a one-pass 4B Student model. On CNN/DM, the Student reaches 70.2% target success rate, outperforming both its Teacher (54.1%) and the strongest of six published rewriting baselines (39.3%), against held-out classifiers that are never queried by the attack. It also reaches 68.3% target success when transferring summaries from an open model toward chosen commercial models. These results show that fingerprint detectability should not be conflated with source authenticity, and that text-only attribution can provide misleading evidence of model identity under targeted rewriting, even when it is accurate on unmodified text.
Read the full abstract ▾
All authors Haohan Yuan, Simin Chen, Xi Niu, Hanqing Guo, Depeng Xu, Haopeng Zhang
Computation and Language
BARRAC: Adaptation of an English Aspect-based Sentiment Analysis Approach for Classification Tasks in Arabic Dialects
Ali Almutairi, Gelareh Mohammadi, Imran Razzak et al. (+1 more)
With the rapid growth of Arabic NLP, several models,
datasets and
benchmarks have been reported. This paper asks whether approaches developed for majority
languages like English can be adapted to Arabic tasks. We adapt an English aspect-based sentiment analysis framework to Arabic classification tasks and present the adaptation as BARRAC: Brainstorming
Alignment and Replaced Representation learning for ArabiC tasks. BARRAC replaces consumer-review attribute pools with Arabic linguistic devices and markers for dialectal sentiment, sarcasm, and dialect identification, and replaces noisy self-training with two-stage training. Evaluated on five Arabic dialect
datasets, BARRAC achieves a mean macro-F1 of 63.93%, outperforming the best few-label SOTA by 3%, and outperforming GPT-4o on four out of five tasks. Error analysis provides insights into remaining challenges. These results demonstrate that adapting task-specific approaches is a promising direction for Arabic NLP alongside adapting models,
datasets and
benchmarks.
Read the full abstract ▾
All authors Ali Almutairi, Gelareh Mohammadi, Imran Razzak, Aditya Joshi
Computation and Language · Cross-listed from cs.AI
Whose Voice Survives the Summary? A Voice-Retention Audit of LLM Employee Listening
Thilo Tamme, Anton Hantel and Bijan Khosrawi-Rad
Organizations increasingly route employee feedback to leaders through large
language model (LLM) summaries, an unaudited layer that silences already-spoken voice. We introduce a Voice Retention / Representation Ratio metric for representational bias in summarization and apply it to a bilingual (English/German) corpus of 2,586 free-text responses from a global professional service company. First, employees supply criticism more reliably than praise (withholding praise is 82 times more common). Second, across 45 leader-summaries the pipeline filters by popularity, not sentiment: criticism survives, yet a concern voiced once is dropped 86% of the time, with short and German-only content lost on the same axis (theme retention 0.14 vs 0.74; German directional). Controlling for frequency, sentiment has no independent effect; the harm is prevalence-driven, which sentiment-only audits miss. A targeted
prompt recovers only named themes. We contribute the metric, field evidence, and a disaggregated voice-retention card.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Yuanhe Zhang, Ziwei Wang, Jie Ren et al. (+6 more)
Large
reasoning models (LRMs) improve performance on complex tasks through extended
reasoning, yet the same process can degenerate into redundant verification and persistent generation loops. Such uncontrolled
reasoning increases inference cost and creates risks of resource exhaustion and service degradation. However, existing mitigations largely truncate long outputs or react to surface repetition, and thus fail to distinguish normal thinking from uncontrolled
reasoning or explain how benign
reasoning degenerates into harmful behavior. In this paper, we operationalize LRM generation as four states and further introduce
Reasoning-state Analysis via Dynamic
Attention Responses (RADAR), which identifies the current
reasoning state in real time and characterizes how effective reflection can develop into uncontrolled generation. Guided by RADAR's analysis, we further realign abnormal
attention distributions toward patterns observed in normal requests and examine how this correction affects excessive reflection and persistent looping. Temporal analyses show that uncontrolled
reasoning is characterized by
attention distributions that deviate from normal generation, with abnormal trends becoming detectable before repetition begins. Correcting these deviations through
Attention Realignment consistently reduces looping while largely preserving benign performance. Together, RADAR provide a mechanistic account of how
reasoning becomes uncontrolled, offering actionable guidance for identifying critical failure stages and designing targeted runtime interventions.
Read the full abstract ▾
All authors Yuanhe Zhang, Ziwei Wang, Jie Ren, Haoran Gao, Zhenhong Zhou, Fanyu Meng, Cong Wu, Li Sun, Sen Su
Computation and Language
You're Hired: Strategic Model Selection for LLM Collaboration
Zongwan Cao, Ziyuan Yang, Shangbin Feng et al. (+5 more)
While multi-
agent and model collaboration
algorithms gain traction to combine the strengths of diverse Large
Language Models (LLMs), existing systems remain bottlenecked on pre-defined and hand-crafted model pools. In this work, we investigate the problem of model selection in multi-LLM systems. We propose and systematically evaluate a taxonomy of 9 selection
algorithms ranging from diversity of model descriptions, capability-aware behavioral diversity, and LLM-based recruiters. We conduct extensive experiments across two candidate pools of 10 and 32 models, deployed in four model collaboration
algorithms, and evaluated across tasks spanning math, coding, QA, and
reasoning. Results demonstrate that successful selection
algorithms greatly outperform random or heuristics-based teams such as merely selecting the models with top individual performance, by up to 36.1% across settings. Specifically, capability- and training-based selection strategies alleviate selection variance and achieve the best performance, which we recommend to employ before deploying real-world multi-LLM systems. Further analysis reveals that larger candidate pools pose greater challenges to shallow selection heuristics, while
algorithms grounded in interacting with candidate models and understanding model capability robustly filter out misaligned, unsafe models, as well as generalizing to novel, out-of-distribution tasks. Together, we establish that principled and informed team selection is critical and present strong model selection
algorithms for assembling effective multi-LLM systems.
Read the full abstract ▾
All authors Zongwan Cao, Ziyuan Yang, Shangbin Feng, Michael Duan, Skyler Hallinan, Bingbing Wen, Lucy Lu Wang, Yulia Tsvetkov
Computation and Language
Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification
Yingfeng Luo, Shaowei Wei, Daixin Wang et al. (+7 more)
Terminal
agents rely on self-verification to assess and correct their solutions as they solve tasks through interaction with command-line environments. Yet how trustworthy such self-verification is remains poorly understood. To investigate this question systematically, we introduce a diagnostic framework that identifies the first complete solution in each trajectory, determines whether it is objectively correct, and uses this ground truth to quantify the
agent's subsequent verification and recovery behavior. Applying it to ten terminal
agents on TerminalBench2.1, we find that verification is nearly universal after a complete candidate is formed, yet only 61.43% of incorrect candidates are detected and only 49.36% of detected errors are successfully repaired. These results show that the main weakness in self-verification lies not in initiating verification, but in detecting and repairing errors. Motivated by these findings, we propose Student-Conditioned Verification
Distillation (SCVD), which lets the student first produce a candidate solution and distills a stronger teacher's subsequent verification and recovery from the same interaction context. Across three Qwen3.5 backbones, SCVD improves Pass@1 on TerminalBench2.1 by 9.74–16.85 percentage points over the corresponding base models and by 4.49–8.61 points over the standard full-trajectory
distillation, while avoiding the pronounced out-of-distribution degradation of full-trajectory
distillation on SWE-bench Verified.
Read the full abstract ▾
All authors Yingfeng Luo, Shaowei Wei, Daixin Wang, Dingyang Lin, Kaiyan Chang, Weiqiao Shan, Tong Zheng, Zhiqiang Zhang, Jingbo Zhu, Tong Xiao
Computation and Language
StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning
Naen Xu, Wanqing Cui, Yibo Hu et al. (+5 more)
Large
language models deployed as personalized assistants must reason over long, evolving interaction histories. However, in long-term dialogue
reasoning, relevant evidence is scattered across sessions, preferences may be revised over time, and standard long-context training fails to address these challenges under data scarcity and prohibitive computational costs. We propose StateTree, a data-driven RL method that constructs a challenging auxiliary task from scarce dialogues with verifiable ground truth. StateTree augments multi-session dialogues with a tree-structured path-tracing task: key-value records are embedded across sessions to form a binary tree. Solving the task requires the model to traverse from root to leaf by retrieving records across sessions and comparing timestamps to resolve branches, then recover the hidden target question among distractor leaves. We apply curriculum RL training progressively increasing tree depth and introduce a compositional variant whose edges carry step-level
reasoning fragments, training the model to compose partial cues into coherent queries. Trained on 10K-
token contexts, StateTree generalizes to 128K
tokens without full-length RL costs and exhibits capabilities including cross-session
retrieval, temporal
reasoning, knowledge update, and compositional multi-hop
reasoning. StateTree outperforms both SFT and RL-based baselines while preserving short-context general
reasoning. StateTree-7B achieves gains up to +23.60% on LongMemEval (128k), and StateTree-14B reaches 59.00% accuracy on LongMemEval, surpassing QwenLong-L1-32B (45.20%).
Read the full abstract ▾
All authors Naen Xu, Wanqing Cui, Yibo Hu, Shixin Hong, Hengyu An, Meiguang Jin, Junfeng Ma, Tianyu Du
Computation and Language · Cross-listed from cs.LG
Blackboard Intelligence Can Surpass Autoregressive on Globally Constrained Problems
Woosang Jeon, Jaeyeon Kim, Sham Kakade et al. (+6 more)
Next-
token prediction has driven remarkable progress in large
language models, yet a growing body of evidence suggests that they can struggle on problems governed by complex global constraints. In this work, we focus on this regime and ask whether some of these limitations arise from the inference interface induced by next-
token prediction itself. We study this question through blackboard intelligence: an inference-time perspective in which a model works on a fixed, revisable canvas and searches over candidate solution states rather than committing to a causal, left-to-right trajectory. We instantiate this idea with
diffusion language models, whose any-order prediction interface naturally exposes predictions over partially filled solution states. Our key observation is that mean confidence, a simple model-internal quantity available from the standard masked
diffusion objective, provides a useful proxy for global coherence and can guide inference-time search and revision. Empirically, across ZebraLogic, Nurse Rostering, and Job-Shop Scheduling, Blackboard consistently improves inference while holding the fine-tuned LLaDA-8B-Instruct checkpoint fixed and substantially outperforms same-scale autoregressive baselines, reaching 90.4% accuracy on ZebraLogic-Hard, 76.4% exact feasibility on Nurse Rostering, and 80.2% optimality on JSSP. Stronger autoregressive search and refinement also fail to close the gap on ZebraLogic-Hard, while Blackboard surpasses tested frontier LLMs there and on JSSP despite their substantially greater scale and strong test-time
reasoning. We open-source our codebase at https://github.com/jwoosang1/blackboard-intelligence.
Read the full abstract ▾
All authors Woosang Jeon, Jaeyeon Kim, Sham Kakade, Yilun Du, Amrit Singh Bedi, Arun Kumar Chithanar, Chul Lee, Taehyeong Kim, Sitan Chen
Computation and Language
Uncovering Uncontrolled Repetition through Residual Stream Dynamics
Yuanhe Zhang, Xinyao Zhou, Haoran Gao et al. (+5 more)
Uncontrolled repetition can prolong autoregressive generation in large
language models (LLMs) and enable resource consumption attacks. Prior analyses of repetitive generation have identified strongly activated features in intermediate and late layers. However, how uncontrolled repetition activity emerges and develops before becoming prominent in these layers remains insufficiently understood. In this paper, we investigate this question primarily in large
vision-
language models (LVLMs), which support a richer set of uncontrolled repetitions through both visual and textual inputs. We propose Tokenwise Residual Comparison (TRC), a method that identifies and localizes anomalies associated with repetition from residual dynamics during generation. TRC compares
attention and multilayer perceptron writes to the residual stream across generated
tokens to identify patterns associated with repetition. It then selectively suppresses coordinates in the residual stream at the identified layer. Experiments show that TRC effectively mitigates uncontrolled repetition, reducing loop rates by 57% on average. Our analysis further shows that repetition semantics emerge in shallow layers and propagate through the residual stream, disrupting normal representations. TRC also generalizes to large
language models (LLMs) and large
reasoning models (LRMs), where it consistently captures analogous repetition dynamics and achieves effective mitigation. Our work broadens the study of repetitive generation from its prominent internal representations to earlier opportunities for intervention, providing insights for mitigating resource consumption attacks.
Read the full abstract ▾
All authors Yuanhe Zhang, Xinyao Zhou, Haoran Gao, Yuyao Zhang, Zhenhong Zhou, Fanyu Meng, Li Sun, Sen Su
Computation and Language
Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?
Eftekhar Hossain and Santu Karmaker
Understanding multi-perspective alternative narratives requires identifying how their information agrees, conflicts, or differs across sources. Existing work on cross-text relations largely focuses on categorizing relations between predefined text pairs, such as entailment or contradiction, rather than directly extracting such information from full narratives. To address this gap, we introduce Overlap-Unique-Conflict (OUC) extraction, a cross-narrative task that extracts all overlapping, conflicting, and unique clauses from two narratives. To support this study, we construct a
benchmark of approximately 22K narrative pairs and 140K OUC instances spanning factual, argumentative, and political discourse. Evaluating 14 open-source LLMs (0.6B-35B), we find that unique information is far easier to extract than overlap and conflict: the strongest model, Gemma-4-31B, reaches only 61.13% F1-score on overlap and 48.58% on conflict, against more than 75% on unique. Further diagnostic analysis reveals that this difficulty does not stem from relation recognition alone, but rather from a failure to pair and extract the corresponding clauses from full narratives, especially in smaller models. Nevertheless, learning these extractions with task-specific supervision narrows the gap considerably: a fine-tuned Qwen-3-8B gains 15-28% absolute over its baseline and surpasses models roughly four times its size (e.g., Qwen-3.6-35B) on several tasks. Even so, overlap and conflict remain well below satisfactory, leaving cross-narrative clause extraction an open challenge.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Evaluating Persistent Calibration under Evolving Model Knowledge
Victor Wang, Thomas Hofweber, Mohit Bansal et al. (+1 more)
As AI systems move from static repositories to
agents that are capable of continual adaptation and learning, maintaining their trustworthiness means equipping the models backing them with the ability to produce confidence estimates that dynamically reflect their changing skills and knowledge. We introduce the problem of persistent calibration, which requires a confidence
estimator to faithfully reflect the knowledge contained in a model as that knowledge changes, without recurring supervision. We operationalize this by examining persistent calibration across checkpoints of open models, asking whether confidence
estimators trained on earlier checkpoints can generalize to later ones. Specifically, we aim to shed light on whether confidence is dependent on knowledge, a question with implications for the reliability of confidence estimates. To measure this relationship, we define and evaluate calibration on knowledge contrast sets: subsets containing questions that one checkpoint answers correctly and another checkpoint answers incorrectly, reflecting a change in knowledge. We show that both inference-time and fine-tuning methods fall short on contrast-set calibration compared to oracle methods trained on future checkpoints, even for methods that are well-calibrated on the full
dataset. We provide evidence for the hypothesis that persistent calibration is challenging because there is a vast space of possible confidence functions that are well-calibrated on a given checkpoint, out of which only some rely on meta-knowledge features that would generalize to other checkpoints. Towards improving contrast-set calibration, we show that multi-checkpoint training helps, suggesting an avenue for identifying confidence features that remain robust across changing knowledge.
Read the full abstract ▾
All authors Victor Wang, Thomas Hofweber, Mohit Bansal, Elias Stengel-Eskin
Computation and Language
Recovering Off-Policy Supervision for Speculative Decoding
Jungseob Lee, Chanjun Park, Sugyeong Eo et al. (+1 more)
Block drafters for speculative decoding are commonly trained on corpora written by external models, where a single off-policy
token invalidates supervision for all subsequent slots in a block. Existing approaches discard these divergent slots, resulting in severe supervision loss. To resolve this problem while preserving the training corpus, we propose a rollout-based training framework that recovers full supervision through two complementary components. The first component, Anchor-Label Relabelling (ALR), replaces corpus labels with distributions from greedy target rollouts, restoring valid supervision across all predicted slots. The second component, In-Rollout Anchors (IRA), places draft blocks directly inside these rollouts to expose the drafter to target-generated context, reusing precomputed rollout features at no additional target cost. Across fixed
vision-
language and text corpora, our framework increases greedy accepted length by up to 36.5% over DFlash and consistently outperforms erasing baselines. Notably, a single epoch of our method surpasses the best erase schedules. After three epochs, it matches the acceptance length of training on target-regenerated responses. These results show that our framework provides an effective and compute-efficient approach for training speculative drafters on fixed corpora without modifying the original text. Code is available at https://github.com/js-lee-AI/ALR-IRA.
Read the full abstract ▾
All authors Jungseob Lee, Chanjun Park, Sugyeong Eo, Hyeonseok Moon
Computation and Language
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Ilgee Hong, Changlong Yu, Zhenghao Xu et al. (+5 more)
We study training LLM judges from natural
language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation criteria the judge invokes and how it weighs them. The dominant approach, outcome-supervised RL (e.g., GRPO), credits every
token in the rollout with a single scalar determined only by the accuracy of the final verdict, providing no separate credit at the criterion-choice
tokens and ignoring the rich
language feedback (e.g., preference rationales) that naturally accompanies preference labels. Self-
Distillation (SD) is one natural way to use this
language feedback: the same model, conditioned on this feedback, acts as a teacher providing dense, position-level supervision. However, not all positions carry equally useful signal. Using the per-position
entropy shift between teacher and student, we identify two regimes: context sharpening, where the teacher concentrates probability on a particular feedback-aligned criterion expression, and context spreading, where the teacher distributes probability across multiple feedback-aligned alternatives. We interpret these patterns as follows: sharpening encourages memorization of a particular criterion expression, whereas spreading promotes semantic understanding by preserving these alternatives. Motivated by this asymmetry, we introduce position masking based on the
entropy shift that retains the lower tail of the
entropy-shift distribution. Experiments show that masking higher-
entropy-shift positions improves out-of-distribution generalization over naive SD. The resulting self-distilled judges outperform judges trained with outcome-supervised RL by 2-9 percentage points on the evaluated subjective subcategories, while remaining competitive on objective ones.
Read the full abstract ▾
All authors Ilgee Hong, Changlong Yu, Zhenghao Xu, Xin Liu, Yuwei Zhang, Qin Lu, Bing Yin, Tuo Zhao
Computation and Language · Cross-listed from cs.CR
Anchor-ECC: Local Integrity Checking for Watermarked LLM Outputs via Error-Correcting Codes
Zewei Deng, Muhammad Siddeek, Liyan Xie et al. (+4 more)
LLM watermarking has become an effective approach to distinguishing AI-generated text from human-written text by
embedding detectable patterns during generation. However, a small post-generation edit may change the meaning of the text without removing its overall watermark signal, creating a risk that the modified content is still attributed to the original model. We propose Anchor-ECC, which incorporates the error-correcting code (ECC) constraints and explicit boundary anchors into the watermark structure and pairs them with a dynamic-programming decoder to detect and localize post-generation edits. Across Qwen3-8B, Mistral-7B-Instruct-v0.3, and OPT-125M, the approximate-hard setting achieves about 99.7% block-level true positive rate (TPR) with at most 7.6% false alarm rate (FAR) for edit detection under mixed insertions, deletions, and substitutions, while preserving the distinction between watermarked outputs and unwatermarked text. Additional quality experiments identify lower-perplexity configurations that retain strong edit-detection performance. Together, these results extend LLM watermarking from source identification to local integrity verification while supporting configurable trade-offs between detection reliability and generation quality.
Read the full abstract ▾
All authors Zewei Deng, Muhammad Siddeek, Liyan Xie, Mohamed Seif, Mengdi Wang, H. Vincent Poor, Andrea Goldsmith
Computation and Language
MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptation
Mehdi Jafari, Hao Xue and Flora Salim
Steering large
language models typically relies on linear, context-independent interventions in activation space, an assumption that recent work has challenged and that can induce an information bottleneck when a fixed representation must encode many behavioral distinctions. We introduce MetaSteer, a method that learns nonlinear interventions with context-dependent effects and applies them to
attention projection
matrices, producing activation effects that vary with the input context by construction and requiring no linear concept-geometry assumption. Framed as preference-based optimization, MetaSteer is trained once on a pooled preference corpus and transferred zero-shot to unseen concepts and out-of-distribution contexts. We find that, despite using low-rank adapters, MetaSteer induces structured, context-dependent changes in hidden-state trajectories while partially preserving aspects of their local trajectory dynamics, including velocity and
curvature. We evaluate MetaSteer on three controlled text-generation
benchmarks and three agentic settings across multiple model families and scales. MetaSteer matches or outperforms strong task-specific steering baselines on most aggregate comparisons in the zero-shot regime. Across the evaluated settings, stronger text-generation steering is associated with stronger agentic steering performance. We further discuss
geometric trajectory effects, capability retention, and safety considerations raised by transferable steering.
Read the full abstract ▾