Computation and Language
Procedural Knowledge at Scale Improves Reasoning
Di Wu, Devendra Singh Sachan, Wen-tau Yih et al. (+1 more)
Test-time scaling has emerged as an effective way to improve
language models on challenging
reasoning tasks. However, most existing methods treat each problem in isolation and do not systematically reuse knowledge from prior
reasoning trajectories. In particular, they underutilize procedural knowledge: how to reframe a problem, choose an approach, and verify or backtrack when needed. We introduce
Reasoning Memory, a
retrieval-augmented generation (RAG) framework for
reasoning models that explicitly retrieves and reuses procedural knowledge at scale. Starting from existing corpora of step-by-step
reasoning trajectories, we decompose each trajectory into self-contained subquestion-subroutine pairs, yielding a datastore of 32 million compact procedural knowledge entries. At inference time, a lightweight in-thought
prompt lets the model verbalize the core subquestion, retrieve relevant subroutines within its
reasoning trace, and reason under diverse retrieved subroutines as implicit procedural priors. Across six math, science, and coding
benchmarks,
Reasoning Memory consistently outperforms RAG with document, trajectory, and template knowledge, as well as a compute-matched test-time scaling baseline. With a higher inference budget, averaged across models and tasks, it improves over no
retrieval by 19.5% and over the compute-matched baseline by 8.9%. Ablation studies show that these gains come from two key factors: the broad procedural coverage of the source trajectories and our decomposition and
retrieval design, which together enable effective extraction and reuse of procedural knowledge. Our experiment code is available at https://github.com/facebookresearch/
reasoning-
memory.
Read the full abstract ▾
All authors Di Wu, Devendra Singh Sachan, Wen-tau Yih, Mingda Chen
Computation and Language
Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
Simona-Vasilica Oprea and Adela Bâra
Learning human preferences in
language models remains fundamentally challenging, as reward modeling relies on subtle, subjective comparisons or shades of gray rather than clear-cut labels. This study investigates the limits of current approaches and proposes a feature-augmented framework to better capture the multidimensional nature of human judgment. Using the Anthropic HHRLHF
dataset, we evaluate ten diverse large
language models LLMs under a standard pairwise preference setting, where baseline performance remains below 0.74 ROC AUC, highlighting the difficulty of the task. To address this, we enrich textual representations with interpretable signals: response length, refusal indicators, toxicity scores and
prompt response semantic similarity, enabling models to explicitly capture key aspects of helpfulness, safety and relevance. The proposed hybrid approach yields consistent improvements across all models, achieving up to 0.84 ROC AUC and significantly higher pairwise accuracy, with DeBERTav3Large demonstrating the best performance. Beyond accuracy, we integrate SHAP and LIME to provide fine-grained interpretability, revealing that model decisions depend on contextualized safety and supportive framing rather than isolated keywords. We further analyze bias amplification, showing that while individual features have weak marginal effects, their interactions influence preference learning.
Read the full abstract ▾
Computation and Language
M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency
Abolfazl Ansari, Delvin Ce Zhang, Zhuoyang Zou et al. (+2 more)
Evaluating scientific arguments requires assessing the strict consistency between a claim and its underlying
multimodal evidence. However, existing
benchmarks lack the scale, domain diversity, and visual complexity needed to evaluate this
alignment realistically. To address this gap, we introduce M2-Verify, a large-scale
multimodal dataset for checking scientific claim consistency. Sourced from PubMed and arXiv, M2-Verify provides over 469K instances across 16 domains, rigorously validated through expert audits. Extensive baseline experiments show that state-of-the-art models struggle to maintain robust consistency. While top models achieve up to 85.8% Micro-F1 on low-complexity medical perturbations, performance drops to 61.6% on high-complexity challenges like anatomical shifts. Furthermore, expert evaluations expose
hallucinations when models generate scientific explanations for their
alignment decisions. Finally, we demonstrate our
dataset's utility and provide comprehensive usage guidelines.
Read the full abstract ▾
All authors Abolfazl Ansari, Delvin Ce Zhang, Zhuoyang Zou, Wenpeng Yin, Dongwon Lee
Computation and Language
Scaling Reasoning Tokens via RL and Parallel Thinking: Evidence From Competitive Programming
Qianfan Zhang, Tianyu Guo, Xuandi Ren et al. (+4 more)
We study how to scale
reasoning token budgets for competitive programming through two complementary approaches: training-time reinforcement learning (RL) and test-time parallel thinking. During RL training, we observe an approximately log-linear relationship between validation accuracy and the average number of generated
reasoning tokens over successive checkpoints, and show two ways to shift this training trajectory: verification RL warmup raises the starting point, while randomized clipping produces a steeper trend in the observed regime. As scaling single-generation
reasoning during RL quickly becomes expensive under full
attention, we introduce a multi-round parallel thinking pipeline that distributes the
token budget across threads and rounds of generation, verification, and refinement. We train the model end-to-end on this pipeline to match the training objective to the test-time structure. Starting from Seed-OSS-36B, the full system with 16 threads and 16 rounds per thread matches the underlying RL model's oracle pass@16 at pass@1 using 7.6 million
tokens per problem on average, and surpasses GPT-5-high on 456 hard competitive programming problems from AetherCode.
Read the full abstract ▾
All authors Qianfan Zhang, Tianyu Guo, Xuandi Ren, Jiale Chen, Ming Ding, Ran Xin, Xia Xiao
Computation and Language · Cross-listed from cs.CV
Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering
Marco Morini, Sara Sarto, Marcella Cornia et al. (+2 more)
Knowledge-based Visual Question Answering (KB-VQA) requires
Multimodal Large
Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However,
retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal
attention into an explicit
multimodal evidence-selection mechanism. LoT first leverages the model's internal
attention patterns to identify query-relevant image regions and textual sentences, filters
attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA
benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available
multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.
Read the full abstract ▾
All authors Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Computation and Language
The Overlooked Repetitive Lengthening Form in Sentiment Analysis
Lei Wang and Eduard Dragut
Individuals engaging in online communication frequently express personal opinions with informal styles (e.g., memes and emojis). While
Language Models (LMs) with informal communications have been widely discussed, a unique and emphatic style, the Repetitive Lengthening Form (RLF), has been overlooked for years. In this paper, we explore answers to two research questions: 1) Is RLF important for sentiment analysis (SA)? 2) Can LMs understand RLF? Inspired by previous linguistic research, we curate Lengthening, the first multi-domain
dataset with 850k samples focused on RLF for SA. Moreover, we introduce Explainable Instruction Tuning (ExpInstruct), a two-stage instruction tuning framework aimed to improve both performance and explainability of LLMs for RLF. We further propose a novel unified approach to quantify LMs' understanding of informal expressions. We show that RLF sentences are expressive expressions and can serve as signatures of document-level sentiment. Additionally, RLF has potential value for online content analysis. Our results show that fine-tuned Pre-trained
Language Models (PLMs) can surpass zero-shot GPT-4 in performance but not in explanation for RLF. Finally, we show ExpInstruct can improve the open-sourced LLMs to match zero-shot GPT-4 in performance and explainability for RLF with limited samples. Code and sample data are available at https://github.com/Tom-Owl/OverlookedRLF
Read the full abstract ▾
Computation and Language
Universal YOCO for Efficient Depth Scaling
Yutao Sun, Li Dong, Tianzhu Ye et al. (+3 more)
The rise of test-time scaling has remarkably boosted the
reasoning and agentic proficiency of Large
Language Models (LLMs). Yet, standard
Transformers struggle to scale inference-time compute efficiently, as conventional looping strategies suffer from high computational overhead and a KV cache that inflates alongside model depth. We present Universal YOCO (YOCO-U), which combines the YOCO decoder-decoder architecture with recursive computation to achieve a synergistic effect greater than either alone. Built on the YOCO framework, YOCO-U implements a Universal Self-Decoder that performs multiple iterations via parameter sharing, while confining the iterative process to shallow, efficient-
attention layers. This combination yields a favorable capability-efficiency tradeoff that neither YOCO nor recursion achieves independently. The YOCO architecture provides a constant global KV cache and linear pre-filling, while partial recursion enhances representational depth with limited overhead. Together, YOCO-U improves
token utility and scaling behavior while maintaining efficient inference. Empirical results confirm that YOCO-U remains highly competitive in general and long-context
benchmarks, demonstrating that the integration of efficient-
attention architectures and recursive computation is a promising direction for scalable LLMs.
Read the full abstract ▾
All authors Yutao Sun, Li Dong, Tianzhu Ye, Shaohan Huang, Jianyong Wang, Furu Wei
Computation and Language
$\texttt{YC-Bench}$: Benchmarking AI Agents for Long-Term Planning and Consistent Execution
Muyu He, Adit Jain, Anand Kumar et al. (+4 more)
As LLM
agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduce $\texttt{YC-Bench}$, a
benchmark that evaluates these capabilities by tasking an
agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The
agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where
adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open source, across 3 seeds each. Only three models consistently surpass the starting capital of \$200K, with Claude Opus 4.6 achieving the highest average final funds at \$1.27 M, followed by GLM-5 at \$1.21 M at 11$\times$ lower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for $47%$ of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance. $YC-Bench$ is open-source, reproducible, and configurable.
Read the full abstract ▾
All authors Muyu He, Adit Jain, Anand Kumar, Vincent Tu, Soumyadeep Bakshi, Sachin Patro, Nazneen Rajani
Computation and Language
RELISH: LLM REgression with a Latent Iterative State Head
Yiheng Su and Matthew Lease
We present RELISH (REgression with a Latent Iterative State Head), a novel, lightweight architecture designed for text regression with large
language models. Rather than decoding numeric targets as text or aggregating multiple generated outputs, RELISH predicts scalar values directly from frozen LLM representations by iteratively refining a learned latent state through cross-
attention over
token-level representations, and then mapping the final state to a point estimate with a linear regressor. Across six
datasets, four LLM backbones, and two LLM training regimes, RELISH consistently outperforms prior baselines from all three major LLM regression families, including autoregressive decoding, regression-aware inference, and existing predictive head methods. Despite these gains, RELISH remains highly parameter-efficient, requiring only $\sim$3.4-3.7M trainable parameters across frozen LLM backbones (only 0.01-0.04$\%$ additional overhead), far less than LoRA-based alternatives that grow with model size (0.26-0.42$\%$). Our code is available at https://github.com/SamSoup/RELISH.
Read the full abstract ▾
Computation and Language
ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget
Nandan Thakur, Zijian Chen, Xueguang Ma et al. (+1 more)
Search
agents, which integrate
language models (LMs) with web search, are becoming crucial for answering complex user queries. Constructing training
datasets for deep research tasks, involving multi-step
retrieval and
reasoning, remains challenging due to expensive human annotation, or cumbersome prerequisites. In this work, we introduce ORBIT, a training
dataset with 20K
reasoning-intensive queries with short verifiable answers, generated using a frugal framework without relying on paid API services. The modular framework relies on four stages: seed creation, question-answer pair generation, and two stages of verification: self and external. ORBIT spans 15 domains and each training pair requires 4-5
reasoning steps, with external search verification required from the complete web. We train Qwen3-4B as the base model on ORBIT using GRPO and evaluate it on Wikipedia question answering tasks. Extensive experiment results demonstrate that ORBIT-4B achieves strong performance among sub-4B LLMs as search
agents, proving the utility of synthetic
datasets. Our framework, code and
datasets are open-sourced and available publicly.
Read the full abstract ▾
All authors Nandan Thakur, Zijian Chen, Xueguang Ma, Jimmy Lin
Computation and Language
Embarrassingly Simple Self-Distillation Improves Code Generation
Ruixiang Zhang, Richard He Bai, Huangjie Zheng et al. (+3 more)
Can a large
language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-
distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes
token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation. Our code is available at https://github.com/apple/ml-ssd
Read the full abstract ▾
All authors Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang
Computation and Language · Cross-listed from cs.HC
True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Lies
Graziano Blasilli and Marco Angelini
This study investigates the ability of
multimodal Large
Language Models (LLMs) to identify and interpret misleading visualizations, and recognize these observations along with their underlying causes and potential intentionality. Our analysis leverages concepts from visualization rhetoric and a newly developed taxonomy of authorial intents as explanatory lenses. We formulated three research questions and addressed them experimentally using a
dataset of 2,336 COVID-19-related tweets, half of which contain misleading visualizations, and supplemented it with real-world examples of perceptual, cognitive, and conceptual errors drawn from VisLies, the IEEE VIS community event dedicated to showcasing deceptive and misleading visualizations. To ensure broad coverage of the current LLM landscape, we evaluated 16 state-of-the-art models. Among them, 15 are open-weight models, spanning a wide range of model sizes, architectural families, and
reasoning capabilities. The selection comprises small models, namely Nemotron-Nano-V2-VL (12B parameters), Mistral-Small-3.2 (24B), DeepSeek-VL2 (27B), Gemma3 (27B), and GTA1 (32B); medium-sized models, namely Qianfan-VL (70B), Molmo (72B), GLM-4.5V (108B), LLaVA-NeXT (110B), and Pixtral-Large (124B); and large models, namely Qwen3-VL (235B), InternVL3.5 (241B), Step3 (321B), Llama-4-Maverick (400B), and Kimi-K2.5 (1000B). In addition, we employed OpenAI GPT-5.4, a frontier proprietary model. To establish a human perspective on these tasks, we also conducted a user study with visualization experts to assess how people perceive rhetorical techniques and the authorial intentions behind the same misleading visualizations. This allows comparison between model and expert behavior, revealing similarities and differences that provide insights into where LLMs align with human judgment and where they diverge.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Screening Is Enough
Ken M. Nakanishi
We call query–key relevance absolute when its values lie on a fixed bounded scale, depend on neither competing keys nor sequence length, require no sequence-length-dependent calibration, and can all be zero. To realize this notion, we introduce screening, whose explicit threshold transforms bounded query–key similarities into relevance values, enabling exact rejection, empty selection, and direct inspection on a common scale. In a controlled comparison of 12
attention mechanisms on a matched
Transformer backbone, only screening maintains both low long-context perplexity and robust
retrieval beyond the training context; notably, it does so without inference-time scaling. Building on screening, we introduce Multiscreen, a
language-model architecture composed of parallel gated screening tiles. Multiscreen retains these long-context gains while achieving greater parameter efficiency, stronger general zero-shot downstream performance, lower training cost at larger scales, and lower model-side time to first
token than
Transformer baselines. We further develop a normalization design that keeps Multiscreen training stable even at a learning rate of $1$ and show that an adapted version likewise stabilizes
Transformer at the same learning rate.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning
Cai Zhou, Zekai Wang, Menghua Wu et al. (+6 more)
While test-time scaling has enabled large
language models to solve highly difficult tasks, state-of-the-art results come at exorbitant compute costs. These inefficiencies can be attributed to the miscalibration of post-trained
language models, and the lack of calibration in popular sampling techniques. Here, we present Online
Reasoning Calibration (ORCA), a framework for calibrating the sampling process that draws upon conformal prediction and test-time training. Specifically, we introduce a meta-learning procedure that updates the calibration module for each input. This allows us to provide valid confidence estimates under distributional shift, e.g. in thought patterns that occur across different stages of
reasoning, or in
prompt distributions between model development and deployment. ORCA not only provides theoretical guarantees on conformal risks, but also empirically shows higher efficiency and generalization across different
reasoning tasks. At risk level $\delta=0.1$, ORCA improves Qwen2.5-32B efficiency on in-distribution tasks with savings up to 47.5% with supervised labels and 40.7% with self-consistency labels. Under zero-shot out-of-domain settings, it improves MATH-500 savings from 24.8% of the static calibration baseline to 67.0% while maintaining a low empirical error rate, and the same trend holds across model families and downstream
benchmarks. Our code is publicly available at https://github.com/wzekai99/ORCA.
Read the full abstract ▾
All authors Cai Zhou, Zekai Wang, Menghua Wu, Qianyu Julie Zhu, Flora C. Shi, Chenyu Wang, Ashia Wilson, Tommi Jaakkola, Stephen Bates
Computation and Language
S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models
Jack Young
Using roughly 48 execution-verified HumanEval training solutions, tuning a single initial state
matrix per recurrent layer, with zero inference overhead, outperforms LoRA by +10.8 pp (p < 0.001) on HumanEval. The method, which we call S0 tuning, optimizes one state
matrix per recurrent layer while freezing all model weights. On Qwen3.5-4B (GatedDeltaNet hybrid), S0 tuning improves greedy pass@1 by +23.6 +/- 1.7 pp (10 seeds). On FalconH1-7B (Mamba-2 hybrid), S0 reaches 71.8% +/- 1.3 and LoRA reaches 71.4% +/- 2.4 (3 seeds), statistically indistinguishable at this sample size while requiring no weight merging. Cross-domain transfer is significant on MATH-500 (+4.8 pp, p = 0.00002, 8 seeds) and GSM8K (+2.8 pp, p = 0.0003, 10 seeds); a text-to-SQL
benchmark (Spider) shows no transfer, consistent with the trajectory-steering mechanism. A prefix-tuning control on a pure
Transformer (Qwen2.5-3B) degrades performance by -13.9 pp under all nine configurations tested. On Qwen3.5, a per-step state-offset variant reaches +27.1 pp, above both S0 and LoRA but with per-step inference cost. Taken together, the results show that recurrent state initialization is a strong zero-inference-overhead PEFT surface for hybrid
language models when verified supervision is scarce. The tuned state is a ~48 MB file; task switching requires no weight merging or model reload. Code and library: https://github.com/jackyoung27/s0-tuning.
Read the full abstract ▾
Computation and Language
Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning
Mohammad R. Abu Ayyash
We present Brainstacks, a modular architecture for continual multi-domain fine-tuning of large
language models that packages domain expertise as frozen adapter stacks composing additively on a shared frozen base at inference. Five interlocking components: (1) MoE-LoRA with Shazeer-style noisy top-2 routing across all seven
transformer projections under QLoRA 4-bit quantization with rsLoRA scaling; (2) an inner loop performing residual boosting by freezing trained stacks and adding new ones; (3) an outer loop training sequential domain-specific stacks with curriculum-ordered dependencies; (4) null-space projection via randomized SVD constraining new stacks to subspaces orthogonal to prior directions, achieving zero forgetting in isolation; (5) an outcome-based sigmoid meta-router trained on empirically discovered domain-combination targets that selectively weights stacks, enabling cross-domain composition. Two boundary experiments: (6) PSN pretraining on a randomly initialized model; (7) per-domain RL (DPO/GRPO) validating compatibility with post-SFT
alignment. Validated on TinyLlama-1.1B (4 domains, 9 stacks) and Gemma 3 12B IT (5 domains, 10 stacks), MoE-LoRA achieves 2.5x faster convergence than parameter-matched single LoRA, residual boosting breaks through the single-stack ceiling, and the routed system recovers generation quality destroyed by ungated stack accumulation. The central finding: the outcome-based router discovers that domain stacks encode transferable cognitive primitives (instruction-following clarity, numerical
reasoning, procedural logic, chain-of-thought structure) rather than domain-specific knowledge, with medical
prompts routing to chat+math stacks in 97% of cases despite zero medical data in those stacks.
Read the full abstract ▾
Computation and Language
Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Papers
Atsuyuki Miyai, Mashiro Toyooka, Zaiying Zhao et al. (+3 more)
This paper introduces the first systematic evaluation framework for quantifying the quality and risks of papers written by modern coding
agents. While AI-driven paper writing has become a growing concern, rigorous evaluation of the quality and potential risks of AI-written papers remains limited, and a unified understanding of their reliability is still lacking. We introduce Paper Reconstruction Evaluation (PaperRecon), an evaluation framework in which an overview (overview.md) is created from an existing paper, after which an
agent generates a full paper based on the overview and minimal additional resources, and the result is subsequently compared against the original paper. PaperRecon disentangles the evaluation of the AI-written papers into two orthogonal dimensions, Presentation and
Hallucination, where Presentation is evaluated using a rubric and
Hallucination is assessed via agentic evaluation grounded in the original paper source. For evaluation, we introduce PaperWrite-Bench, a
benchmark of 51 papers from top-tier venues across diverse domains published after 2025. Our experiments reveal a clear trade-off: while both ClaudeCode and Codex improve with model advances, ClaudeCode achieves higher presentation quality at the cost of more than 10
hallucinations per paper on average, whereas Codex produces fewer
hallucinations but lower presentation quality. This work takes a first step toward establishing evaluation frameworks for AI-driven paper writing and improving the understanding of its risks within the research community.
Read the full abstract ▾
All authors Atsuyuki Miyai, Mashiro Toyooka, Zaiying Zhao, Kenta Watanabe, Toshihiko Yamasaki, Kiyoharu Aizawa
Computation and Language
CARE: Privacy-Compliant Agentic Reasoning with Evidence Discordance
Haochen Liu, Weien Li, Rui Song et al. (+6 more)
Large
language model (LLM) systems are increasingly used to support high-stakes decision-making, but they typically perform worse when the available evidence is internally inconsistent. Such a scenario exists in real-world healthcare settings, with patient-reported symptoms contradicting medical signs. To study this problem, we introduce MIMIC-DOS, a
dataset for short-horizon organ dysfunction worsening prediction in the intensive care unit (ICU) setting. We derive this
dataset from the widely recognized MIMIC-IV, a publicly available electronic health record
dataset, and construct it exclusively from cases in which discordance between signs and symptoms exists. This setting poses a substantial challenge for existing LLM-based approaches, with single-pass LLMs and agentic pipelines often struggling to reconcile such conflicting signals. To address this problem, we propose CARE: a multi-stage
privacy-compliant agentic
reasoning framework in which a proprietary LLM provides guidance by generating structured categories and transitions without accessing sensitive patient data, while a local LLM uses these categories and transitions to support evidence acquisition and final decision-making. Empirically, under controlled retrospective evaluation on MIMIC-DOS, CARE achieves the best overall performance across key metrics among the evaluated LLMs and agentic workflows, showing that it can more robustly handle conflicting clinical evidence while preserving
privacy.
Read the full abstract ▾
All authors Haochen Liu, Weien Li, Rui Song, Zeyu Li, Chun Jason Xue, Xiao-Yang Liu, Sam Nallaperuma-Herzberg, Xue Liu, Ye Yuan
Computation and Language
Temporal Dependencies in In-Context Learning: The Role of Induction Heads
Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini et al. (+3 more)
Large
language models (LLMs) exhibit strong in-context learning capabilities, but how they track and retrieve information from context remains underexplored. Drawing on the free recall paradigm in cognitive science (where participants recall list items in any order), we show that several open-source LLMs consistently display a serial-recall-like pattern, assigning peak probability to
tokens that immediately follow a repeated
token in the input sequence. Through systematic ablation experiments, we show that induction heads, specialized
attention heads that attend to the
token following a previous occurrence of the current
token, play an important role in this phenomenon. Removing heads with a high induction score substantially reduces the +1 lag bias, whereas ablating random heads does not reproduce the same reduction. We also show that removing heads with high induction scores impairs the performance of models prompted to do serial recall using few-shot learning to a larger extent than removing random heads. Our findings highlight a mechanistically specific connection between induction heads and temporal context processing in
transformers, suggesting that these heads are especially important for ordered
retrieval and serial-recall-like behavior during in-context learning.
Read the full abstract ▾
All authors Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini, Yash Aggarwal, Billy Dickson, Zoran Tiganj
Computation and Language
Narrative Fingerprints: Multi-Scale Author Identification via Novelty Curve Dynamics
Fred Zimmerman and Hilmar AI
We test whether authors have characteristic "fingerprints" in the information-theoretic novelty curves of their published works. Working with two corpora – Books3 (52,796 books, 759 qualifying authors) and PG-19 (28,439 books, 1,821 qualifying authors) – we find that authorial voice leaves measurable traces in how novelty unfolds across a text. The signal is multi-scale: at book level, scalar dynamics (mean novelty, speed, volume, circuitousness) identify 43% of authors significantly above chance; at chapter level, SAX motif patterns in sliding windows achieve 30x-above-chance attribution, far exceeding the scalar features that dominate at book level. These signals are complementary, not redundant. We show that the fingerprint is partly confounded with genre but persists within-genre for approximately one-quarter of authors. Classical authors (Twain, Austen, Kipling) show fingerprints comparable in strength to modern authors, suggesting the phenomenon is not an artifact of contemporary publishing conventions.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Revision or Re-Solving? Decomposing Second-Pass Gains in Multi-LLM Pipelines
Jingjie Ning, Xueqi Li and Chengyu Yu
Multi-LLM revision pipelines, in which a second model reviews and improves a draft produced by a first, are widely assumed to derive their gains from genuine error correction. We question this assumption with a controlled decomposition experiment that uses four matched conditions to separate second-pass gains into three additive components: re-solving, scaffold, and content. We evaluate this design across two model pairs on three
benchmarks spanning knowledge-intensive MCQ and competitive programming. Our results show that the gains of multi-LLM revision are not monolithic, but depend on task structure, draft quality, and the type of draft information. On MCQ tasks, where the answer space is constrained and drafts provide little structural guidance, most gains are consistent with stronger-model re-solving, and directly routing queries to the stronger model can be more effective than revising a weak draft. On code generation tasks, however, two-stage prompting remains useful because even semantically null drafts can provide substantial structural scaffolding, while weak draft content can be harmful. Finally, role-reversed experiments show that strong drafts clearly benefit weak reviewers. Ultimately, our findings demonstrate that the utility of multi-LLM revision is dynamically bottlenecked by task structure and draft quality, necessitating more targeted pipeline designs rather than blanket revision strategies.
Read the full abstract ▾
Computation and Language
Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
Gyuseok Lee, Wonbin Kweon, Zhenrui Yue et al. (+3 more)
Reward factorization personalizes large
language models (LLMs) by decomposing rewards into shared basis functions and user-specific weights. Yet, existing methods estimate user weights from scarce data in isolation and as deterministic points, leading to inaccurate and unreliable inference. We introduce Variational Reward Factorization (VRF), an uncertainty-aware framework that represents each user's preferences as a variational distribution in a shared preference space. VRF infers user distributions via a variational encoder, derives weights through Wasserstein distance matching with shared probabilistic bases, and downweights uncertain estimates through a variance-attenuated loss. On three
benchmarks, VRF outperforms all baselines across seen and unseen users, few-shot scenarios, and varying uncertainty levels, with gains extending to downstream
alignment. Our code is available at https://github.com/Gyu-Seok-Lee/VRF_COLM26.
Read the full abstract ▾
All authors Gyuseok Lee, Wonbin Kweon, Zhenrui Yue, SeongKu Kang, Jiawei Han, Dong Wang
Computation and Language
Multimodal Analysis of State-Funded News Coverage of the Israel-Hamas War on YouTube Shorts
Daniel Miehling and Sandra Kuebler
YouTube Shorts have become central to news consumption on the platform, yet research on how geopolitical events are represented in this format remains limited. To address this gap, we present a
multimodal pipeline that combines automatic transcription, aspect-based sentiment analysis (ABSA), and semantic scene classification. The pipeline is first assessed for feasibility and then applied to analyze short-form coverage of the Israel-Hamas war by state-funded outlets. Using over 2,300 conflict-related Shorts and more than 94,000 visual frames, we systematically examine war reporting across major international broadcasters. Our findings reveal that the sentiment expressed in transcripts regarding specific aspects differs across outlets and over time, whereas scene-type classifications reflect visual cues consistent with real-world events. Notably, smaller domain-adapted models outperform large
transformers and even LLMs for sentiment analysis, underscoring the value of resource-efficient approaches for humanities research. The pipeline serves as a template for other short-form platforms, such as TikTok and Instagram, and demonstrates how
multimodal methods, combined with qualitative interpretation, can characterize sentiment patterns and visual cues in algorithmically driven
video environments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
Do Phone-Use Agents Respect Your Privacy?
Zhengyang Tang, Ke Ji, Xidong Wang et al. (+19 more)
We study whether phone-use
agents respect
privacy while completing benign mobile tasks. This question has remained hard to answer because
privacy-compliant behavior is not operationalized for phone-use
agents, and ordinary apps do not reveal exactly what data
agents type into which form entries during execution. To make this question measurable, we introduce MyPhoneBench, a verifiable evaluation framework for
privacy behavior in mobile
agents. We operationalize
privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled
memory through a minimal
privacy contract, iMy, and pair it with instrumented mock apps plus rule-based
auditing that make unnecessary permission requests, deceptive re-disclosure, and unnecessary form filling observable and reproducible. Across five frontier models on 10 mobile apps and 300 tasks, we find that task success,
privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities, and no single model dominates all three. Evaluating success and
privacy jointly reshuffles the model ordering relative to either metric alone. The most persistent failure mode across models is simple data minimization:
agents still fill optional personal entries that the task does not require. These results show that
privacy failures arise from over-helpful execution of benign tasks, and that success-only evaluation overestimates the deployment readiness of current phone-use
agents. All code, mock apps, and
agent trajectories are publicly available at~ https://github.com/FreedomIntelligence/MyPhoneBench.
Read the full abstract ▾
All authors Zhengyang Tang, Ke Ji, Xidong Wang, Zihan Ye, Xinyuan Wang, Yiduo Guo, Ziniu Li, Chenxin Li, Jingyuan Hu, Shunian Chen, Tongxu Luo, Jiaxi Bi, Zeyu Qin, Shaobo Wang, Xin Lai, Pengyuan Lyu, Junyi Li, Can Xu, Chengquan Zhang, Han Hu, Ming Yan, Benyou Wang
Computation and Language
Dual Optimal: Make Your LLM Peer-like with Dignity
Xiangqi Wang, Yue Huang, Haomin Zhuang et al. (+2 more)
Current aligned
language models exhibit a dual failure mode we term the Evasive Servant: they sycophantically validate flawed user beliefs while deflecting responsibility with boilerplate disclaimers. We propose the Dignified Peer framework, which counters servility with anti-sycophancy and trustworthiness, and mitigates evasiveness through empathy and creativity. Realizing this
agent requires overcoming significant challenges in data supervision, objective collapse, and evaluation bias. We address these issues by introducing the PersonaKnob
dataset which features a compositional partial order structure of multiple persona preference. This data is utilized alongside a tolerant constrained Lagrangian DPO
algorithm that dynamically balances all persona dimensions to prevent behavioral collapse. Additionally, we employ a psychometrically calibrated Item Response Theory evaluation protocol to disentangle latent model persona capability from confounders like judge biases. Extensive empirical studies demonstrate that our approach successfully build a LLM
agent with both dignity and peer.
Read the full abstract ▾
All authors Xiangqi Wang, Yue Huang, Haomin Zhuang, Kehan Guo, Xiangliang Zhang
Computation and Language
Phase transition on a context-sensitive random language model with short range interactions
Yuma Toji, Jun Takahashi, Vwani Roychowdhury et al. (+1 more)
Since the random
language model was proposed by E. DeGiuli [Phys. Rev. Lett. 122, 128301],
language models have been investigated intensively from the viewpoint of statistical mechanics. Recently, the existence of a Berezinskii–Kosterlitz–Thouless transition was numerically demonstrated in models with long-range interactions between symbols. In statistical mechanics, it has long been known that long-range interactions can induce phase transitions. Therefore, it has remained unclear whether phase transitions observed in
language models originate from genuinely linguistic properties that are absent in conventional
spin models. In this study, we construct a random
language model with short-range interactions and numerically investigate its statistical properties. Our model belongs to the class of context-sensitive grammars in the Chomsky hierarchy and allows explicit reference to contexts. We find that a phase transition occurs even when the model refers only to contexts whose length remains constant with respect to the sentence length. This result indicates that finite-temperature phase transitions in
language models are genuinely induced by the intrinsic nature of
language, rather than by long-range interactions.
Read the full abstract ▾
All authors Yuma Toji, Jun Takahashi, Vwani Roychowdhury, Hideyuki Miyahara
Computation and Language
Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?
Luis Frentzen Salim, Lun-Wei Ku and Hsing-Kuo Kenneth Pao
Adapting large
language models (LLMs) to new
languages is an expensive and opaque process. Understanding how
language models acquire new
languages and
multilingual abilities is key to achieve efficient adaptation. Prior work on
multilingual interpretability research focuses primarily on how trained models process
multilingual instructions, leaving unexplored the mechanisms through which they acquire new
languages during training. We investigate these training dynamics on decoder-only
transformers through the lens of two functional cognitive specializations:
language perception (input comprehension) and production (output generation). Through experiments on low-resource
languages, we demonstrate how perceptual and productive specialization emerges in different regions of a
language model by running layer ablation sweeps from the model's input and output directions. Based on the observed specialization patterns, we propose CogSym, a layer-wise heuristic that enables effective adaptation by exclusively fine-tuning a few early and late layers. We show that tuning only the 25% outermost layers achieves downstream task performance within 2-3% deviation from the full fine-tuning baseline. CogSym yields consistent performance with adapter methods such as LoRA, showcasing generalization beyond full fine-tuning. These findings provide insights to better understand how LLMs learn new
languages and push toward accessible and inclusive
language modeling.
Read the full abstract ▾
Computation and Language
GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training
Jesse van Oort, Frank Brinkkemper, Erik de Graaf et al. (+2 more)
We present the GPT-NL Public Corpus, the biggest permissively licensed corpus of Dutch
language resources. The GPT-NL Public Corpus contains 21 Dutch-only collections totalling 36B preprocessed Dutch
tokens not present in any other LLM pretraining corpus. Additionally, the corpus includes roughly 207B English, 232B Code, and 48B German/Danish
tokens taken from existing sets which we further curated for compliance. This corpus includes curated data from large existing corpora like Common Corpus and Common Crawl, as well as newly created Dutch-specific collections. Most newly created Dutch collections consist of content collected in collaboration with organisations or synthetically augmented content. All data is collected and evaluated with the aim of facilitating the creation of (commercial)
language models that are lawful, useful and non-harmful. All data included in the GPT-NL Public Corpus is sourced from
datasets with permissive licensing and is curated and redistributed under a CC-BY license. The full
dataset is publicly available on the Hugging Face Hub.
Read the full abstract ▾
All authors Jesse van Oort, Frank Brinkkemper, Erik de Graaf, Bram Vanroy, Saskia Lensink
Computation and Language · Cross-listed from cs.CV
Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment
Zhuchenyang Liu, Yao Zhang and Yu Xiao
2D assembly diagrams are often abstract and hard to follow, creating a need for intelligent assistants that can monitor progress, detect errors, and provide step-by-step guidance. In mixed reality settings, such systems must recognize completed and ongoing steps from the camera feed and align them with the diagram instructions.
Vision Language Models (VLMs) show promise for this task, but face a depiction gap because assembly diagrams and
video frames share few visual features. To systematically assess this gap, we construct IKEA-Bench, a
benchmark of 1,623 questions across 6 task types on 29 IKEA furniture products, and evaluate 19 VLMs (2B-38B) under three
alignment strategies. Our key findings: (1) assembly instruction understanding is recoverable via text, but text simultaneously degrades diagram-to-
video alignment; (2) architecture family predicts
alignment accuracy more strongly than parameter count; (3)
video understanding remains a hard bottleneck unaffected by strategy. A three-level mechanistic analysis further reveals that diagrams and
video occupy disjoint ViT subspaces, and that adding text shifts models from visual to text-driven
reasoning. These results identify visual encoding as the primary target for improving cross-depiction robustness. Project page: https://ryenhails.github.io/IKEA-Bench/
Read the full abstract ▾
Computation and Language
When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigation
Henry Peng Zou, Chunyu Miao, Wei-Chieh Huang et al. (+16 more)
As LLM
agents transition from short, static problem solving to executing complex, long-horizon tasks in dynamic environments, the ability to handle user interruptions, such as adding requirement or revising goals, during mid-task execution is becoming a core requirement for realistic deployment. However, existing
benchmarks largely assume uninterrupted
agent behavior or study interruptions only in short, unconstrained
language tasks. In this paper, we present the first systematic study of interruptible
agents in long-horizon, environmentally grounded web
navigation tasks, where actions induce persistent state changes. We formalize three realistic interruption types, including addition, revision, and retraction, and introduce InterruptBench, a
benchmark derived from WebArena-Lite that synthesizes high-quality interruption scenarios under strict semantic constraints. Using a unified interruption simulation framework, we evaluate six strong LLM backbones across single- and multi-turn interruption settings, analyzing both their effectiveness in adapting to updated intents and their efficiency in recovering from mid-task changes. Our results show that handling user interruptions effectively and efficiently during long-horizon agentic tasks remains challenging for powerful large-scale LLMs. Code and
dataset are available at https://github.com/HenryPengZou/InterruptBench.
Read the full abstract ▾
All authors Henry Peng Zou, Chunyu Miao, Wei-Chieh Huang, Yankai Chen, Yue Zhou, Hanrong Zhang, Yaozu Wu, Liancheng Fang, Zhengyao Gu, Zhen Zhang, Kening Zheng, Fangxin Wang, Yi Nian, Shanghao Li, Wenzhe Fan, Langzhou He, Weizhi Zhang, Xue Liu, Philip S. Yu