Computation and Language
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
Rabin Adhikari
Mechanistic interpretability aims to reverse-engineer large
language models (LLMs) into human-understandable computational
circuits. However, the complexity of pretrained models often obscures the minimal mechanisms required for specific
reasoning tasks. In this work, we train small,
attention-only
transformers from scratch on a symbolic version of the Indirect Object Identification (IOI) task, a
benchmark for studying coreference-like
reasoning in
transformers. Surprisingly, a single-layer model with only two
attention heads achieves perfect IOI accuracy, despite lacking MLPs and normalization layers. Through residual stream decomposition, spectral analysis, and
embedding interventions, we find that the two heads specialize into additive and contrastive subcircuits that jointly implement IOI resolution. Furthermore, we show that a two-layer, one-head model composes information from the previous layer primarily through query-key interactions. These results demonstrate that task-specific training induces highly interpretable, minimal
circuits, offering a controlled testbed for probing the computational foundations of
transformer reasoning.
Read the full abstract ▾
Computation and Language
POWSM: A Phonetic Open Whisper-Style Speech Foundation Model
Chin-Jou Li, Kalvin Chang, Shikhar Bharadwaj et al. (+5 more)
Recent advances in spoken
language processing have led to substantial progress in phonetic tasks such as automatic
speech recognition (ASR), phone recognition (PR), grapheme-to-phoneme conversion (G2P), and phoneme-to-grapheme conversion (P2G). Despite their conceptual similarity, these tasks have largely been studied in isolation, each relying on task-specific architectures and
datasets. In this paper, we introduce POWSM (Phonetic Open Whisper-style
Speech Model), the first unified framework capable of jointly performing multiple phone-related tasks. POWSM enables seamless conversion between audio, text (graphemes), and phones, opening up new possibilities for universal and low-resource
speech processing. Our model outperforms or matches specialized PR models of similar size (Wav2Vec2Phoneme and ZIPA) while jointly supporting G2P, P2G, and ASR. Our training data, code and models are released to foster open science.
Read the full abstract ▾
All authors Chin-Jou Li, Kalvin Chang, Shikhar Bharadwaj, Eunjung Yeo, Kwanghee Choi, Jian Zhu, David Mortensen, Shinji Watanabe
Computation and Language · Cross-listed from cs.LG
Sequences of Logits Reveal the Low Rank Structure of Language Models
Noah Golowich, Allen Liu and Abhishek Shetty
A major problem in the study of large
language models is to understand their inherent low-dimensional structure. We introduce an approach to study the low-dimensional structure of
language models at a model-agnostic level: as sequential probabilistic models. We first empirically demonstrate that a wide range of modern
language models exhibit low-rank structure: in particular,
matrices built from the model's logits for varying sets of
prompts and responses have low approximate rank. We then show that this low-rank structure can be leveraged for generation – in particular, we can generate a response to a target
prompt using a linear combination of the model's outputs on unrelated, or even nonsensical
prompts. On the theoretical front, we observe that studying the approximate rank of
language models in the sense discussed above yields a simple universal abstraction whose theoretical predictions parallel our experiments. We then analyze the representation power of the abstraction and give provable learning guarantees.
Read the full abstract ▾
Computation and Language
Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
James A. Michaelov, Roger P. Levy and Benjamin K. Bergen
We show that across architecture (
Transformer vs. Mamba vs. RWKV), training
dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregressive
language models exhibit highly consistent patterns of change in their behavior over the course of pretraining. Based on our analysis of over 1,400
language model checkpoints on over 110,000
tokens of English, we find that up to 98% of the variance in
language model behavior at the word level can be explained by three simple heuristics: the unigram probability (frequency) of a given word, the $n$-gram probability of the word, and the semantic similarity between the word and its context. Furthermore, we see consistent behavioral phases in all
language models, with their predicted probabilities for words overfitting to those words' $n$-gram probabilities for increasing $n$ over the course of training. Taken together, these results suggest that learning in neural
language models may follow a similar trajectory irrespective of model details.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Xiutian Zhao, Rochelle Choenni, Rohit Saxena et al. (+1 more)
Despite their impressive performance,
vision-
language models (VLMs) still struggle on culturally situated inputs. To understand how VLMs process culturally grounded information, we study the presence of culture-sensitive
neurons, i.e.,
neurons whose activations show preferential sensitivity to inputs associated with particular cultural contexts. We examine whether such
neurons are important for culturally diverse visual question answering and where they are located. Using the CVQA
benchmark, we identify
neurons of culture selectivity and perform diagnostic tests by deactivating the
neurons flagged by various identification methods. Experiments on three VLMs across 25 cultural groups demonstrate the existence of
neurons whose ablation disproportionately harms performance on questions about the corresponding cultures, while having limited effects on others. Moreover, we introduce a new margin-based selector Contrastive Activation Margin (ConAct) and show that it outperforms probability- and
entropy-based methods in identifying
neurons associated with cultural selectivity. Finally, our layer-wise analyses reveal that such
neurons are not uniformly distributed: they cluster in specific decoder layers in a model-dependent way.
Read the full abstract ▾
All authors Xiutian Zhao, Rochelle Choenni, Rohit Saxena, Ivan Titov
Computation and Language
SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens
Yinhan He, Wendy Zheng, Yaochen Zhu et al. (+6 more)
The verbosity of Chain-of-Thought (CoT)
reasoning hinders its mass deployment in efficiency-critical applications. Recently, implicit CoT approaches have emerged, which encode
reasoning steps within LLM's hidden
embeddings (termed “implicit
reasoning”) rather than explicit
tokens. This approach accelerates CoT by reducing the
reasoning length and bypassing some LLM components. However, existing implicit CoT methods face two significant challenges: (1) they fail to preserve the semantic
alignment between the implicit
reasoning (when transformed to natural
language) and the ground-truth
reasoning, resulting in a significant CoT performance degradation, and (2) they focus on reducing the length of the implicit
reasoning; however, they neglect the considerable time cost for an LLM to generate one individual implicit
reasoning token. To tackle these challenges, we propose a novel semantically-aligned implicit CoT framework termed SemCoT. In particular, for the first challenge, we design a contrastively trained sentence
transformer that evaluates semantic
alignment between implicit and explicit
reasoning, which is used to enforce semantic preservation during implicit
reasoning optimization. To address the second challenge, we introduce an efficient implicit
reasoning generator by finetuning a lightweight
language model using knowledge
distillation. This generator is guided by our sentence
transformer to distill ground-truth
reasoning into semantically aligned implicit
reasoning, while also optimizing for accuracy. SemCoT is the first approach that enhances CoT efficiency by jointly optimizing
token-level generation speed and preserving semantic
alignment with ground-truth
reasoning. Extensive experiments demonstrate the superior performance of SemCoT compared to state-of-the-art methods in both efficiency and effectiveness. Our code can be found at https://github.com/YinhanHe123/SemCoT/.
Read the full abstract ▾
All authors Yinhan He, Wendy Zheng, Yaochen Zhu, Zaiyi Zheng, Lin Su, Sriram Vasudevan, Qi Guo, Liangjie Hong, Jundong Li
Computation and Language
Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
James A. Michaelov and Catherine Arnett
Language models generally produce grammatical text, but they are more likely to make errors in certain contexts. Drawing on paradigms from psycholinguistics, we carry out a fine-grained analysis of those errors in different syntactic contexts. We demonstrate that by disaggregating over the conditions of carefully constructed
datasets and comparing model performance on each over the course of training, it is possible to better understand the intermediate stages of grammatical learning in
language models. Specifically, we identify distinct phases of training where
language model behavior aligns with specific heuristics such as word frequency and local context rather than generalized grammatical rules. We argue that taking this approach to analyzing
language model behavior more generally can serve as a powerful tool for understanding the intermediate learning phases, overall training dynamics, and the specific generalizations learned by
language models.
Read the full abstract ▾
Computation and Language
RiddleBench: A New Generative Reasoning Benchmark for LLMs
Deepon Halder, Alan Saji, Thanmay Jayakumar et al. (+3 more)
Large
Language Models have demonstrated strong performance on many established
reasoning benchmarks. However, these
benchmarks primarily evaluate structured skills like quantitative problem-solving, leaving a gap in assessing flexible, multifaceted
reasoning abilities that are central to human intelligence. These abilities require integrating logical deduction with spatial awareness and constraint satisfaction, which current evaluations do not measure well. To address this, we introduce RiddleBench, a
benchmark of 1,737 challenging puzzles in English designed to probe these core
reasoning capabilities. Evaluation of state-of-the-art models on RiddleBench shows fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3, and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and 63.16%). Analysis further reveals deep failures, including
hallucination cascades (accepting flawed
reasoning from other models) and poor self-correction due to a strong self-confirmation bias. Their
reasoning is also fragile, with performance degrading significantly when constraints are reordered or irrelevant information is introduced. RiddleBench functions as a diagnostic tool for these issues and as a resource for guiding the development of more robust and reliable
language models.
Read the full abstract ▾
All authors Deepon Halder, Alan Saji, Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan, Raj Dabre
Computation and Language
Idea2Plan: Exploring AI-Powered Research Planning
Jin Huang, Silviu Cucerzan, Sujay Kumar Jauhar et al. (+1 more)
Large
language models (LLMs) have demonstrated significant potential to accelerate scientific discovery as valuable tools for analyzing data, generating hypotheses, and supporting innovative approaches in various scientific fields. In this work, we investigate how LLMs can handle the transition from conceptual research ideas to well-structured research plans. Effective research planning not only supports scientists in advancing their research but also represents a crucial capability for the development of autonomous research
agents. Despite its importance, the field lacks a systematic understanding of LLMs' research planning capability. To rigorously measure this capability, we introduce the Idea2Plan task and Idea2Plan Bench, a set of
benchmarks built from ICML 2025 and Nature Mental Health papers released after major LLM training cutoffs. Each
benchmark instance includes a research idea and a grading rubric capturing the key components of valid plans. We further propose Idea2Plan JudgeEval, a complementary
benchmark to assess the reliability of LLM-based judges against expert annotations. Experimental results show that GPT-5 achieves the strongest performance on the
benchmark, though substantial headroom remains for improvement. Our study provides new insights into LLMs' capability for research planning and lays the groundwork for future progress.
Read the full abstract ▾
All authors Jin Huang, Silviu Cucerzan, Sujay Kumar Jauhar, Ryen W. White
Computation and Language
Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation
Alexander Martin, William Walden, Reno Kriz et al. (+5 more)
We introduce MiRAGE, an evaluation framework for
retrieval-augmented generation (RAG) from
multimodal sources. As audiovisual media becomes a more prevalent source of information online, RAG systems must integrate such media into generation. Yet, existing evaluation methods for RAG are largely text-centric and do not readily transfer to
multimodal settings. MiRAGE is a claim-centric approach to
multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE and compare it to
multimodal variants of three prominent text-centric RAG metrics—ALCE, ARGUE, and RAGAS—finding that MiRAGE outperforms all three on text while being the only one to generalize to
multimodal sources. We release open-source implementations and outline evaluation methods for
multimodal RAG.
Read the full abstract ▾
All authors Alexander Martin, William Walden, Reno Kriz, Dengjia Zhang, Kate Sanders, Eugene Yang, Chihsheng Jin, Benjamin Van Durme
Computation and Language
Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish
Lujun Li, Yewei Song, Lama Sleem et al. (+7 more)
Grammar refers to the system of rules that governs the structural organization and the semantic relations among linguistic units such as sentences, phrases, and words within a given
language. In natural
language processing, there remains a notable scarcity of grammar focused evaluation protocols, a gap that is even more pronounced for low-resource
languages. Moreover, the extent to which large
language models genuinely comprehend grammatical structure, especially the mapping between syntactic structures and meanings, remains under debate. To investigate this issue, we propose a Grammar Book Guided evaluation pipeline intended to provide a systematic and generalizable framework for grammar evaluation consisting of four key stages, and in this work we take Luxembourgish as a case study. The results show a weak positive correlation between translation performance and grammatical understanding, indicating that strong translations do not necessarily imply deep grammatical competence. Larger models perform well overall due to their semantic strength but remain weak in morphology and syntax, struggling particularly with Minimal Pair tasks, while strong
reasoning ability offers a promising way to enhance their grammatical understanding.
Read the full abstract ▾
All authors Lujun Li, Yewei Song, Lama Sleem, Yiqun Wang, Yangjie Xu, Cedric Lothritz, Niccolo Gentile, Radu State, Tegawende F. Bissyande, Jacques Klein
Computation and Language
Parallel Loop Transformer for Efficient Test-Time Computation Scaling
Bohong Wu, Mengzhao Chen, Xiang Luo et al. (+9 more)
Large
Language Models (LLMs) are powerful but often too slow and costly for real-world use during inference. Looped
transformers save on parameters by reusing the same weights for multiple computational steps, or "loops." However, this approach has a major flaw: the loops run one after another, causing inference latency and
memory requirements to increase with each added loop. This makes them impractical for fast applications. To solve this problem, we introduce the Parallel Loop
Transformer (PLT). PLT is a new architecture that delivers the performance benefits of a deep, looped model but with the low latency of a standard, non-looped model. PLT works using two key techniques. First, Cross-Loop Parallelism (CLP) breaks the sequential dependency by computing different loops for different
tokens at the same time, all within a single pass. Second, to prevent
memory costs from growing, we use an Efficient Representation Enhancement strategy. This method shares the
memory (KV cache) from the first loop with all other loops. It then uses a Gated Sliding-Window
Attention (G-SWA) to combine this shared global information with local information, maintaining high accuracy. Our experiments show that PLT achieves the high accuracy of a traditional looped model but with almost no extra latency or
memory cost compared to a standard
transformer.
Read the full abstract ▾
All authors Bohong Wu, Mengzhao Chen, Xiang Luo, Shen Yan, Qifan Yu, Fan Xia, Tianqi Zhang, Hongrui Zhan, Zheng Zhong, Xun Zhou, Siyuan Qiao, Xingyan Bin
Computation and Language
Towards a Method for Synthetic Generation of Persons with Aphasia Transcripts
Jason M. Pittman, Anton Phillips Jr., Yesenia Medina-Santos et al. (+1 more)
In aphasia research,
Speech-
Language Pathologists (SLPs) devote extensive time to manually coding
speech samples using Correct Information Units (CIUs), a measure of how informative an individual sample of
speech is. Developing automated systems to recognize aphasic
language is limited by data scarcity. For example, only about 600 transcripts are available in AphasiaBank yet billions of
tokens are used to train large
language models (LLMs). In the broader field of machine learning (ML), researchers increasingly turn to synthetic data when such are sparse. Therefore, this study constructs and validates two methods to generate synthetic transcripts of the AphasiaBank Cat Rescue picture description task. One method leverages a procedural programming approach while the second uses Mistral 7b Instruct and Llama 3.1 8b Instruct LLMs. The methods generate transcripts across four severity levels (Mild, Moderate, Severe, Very Severe) through word dropping, filler insertion, and paraphasia substitution. Overall, we found, compared to human-elicited transcripts, Mistral 7b Instruct best captures key aspects of linguistic degradation observed in aphasia, showing realistic directional changes in NDW, word count, and word length amongst the synthetic generation methods. Based on the results, future work should plan to create a larger
dataset, fine-tune models for better aphasic representation, and have SLPs assess the realism and usefulness of the synthetic transcripts.
Read the full abstract ▾
All authors Jason M. Pittman, Anton Phillips Jr., Yesenia Medina-Santos, Brielle C. Stark
Computation and Language
ProofSketch: Efficient Verified Reasoning for Large Language Models
Disha Sheshanarayana and Tanishka Magar
Reasoning methods such as chain-of-thought prompting and self-consistency have shown immense potential to improve the accuracy of large
language models across various
reasoning tasks. However such methods involve generation of lengthy
reasoning chains, which substantially increases
token consumption, computational cost, and latency. To address this inefficiency, we propose ProofSketch, a verification-guided
reasoning framework that integrates symbolic closure computation, lexicographic verification and adaptive sketch generation. Our experiments show that ProofSketch consistently reduces
token usage while improving accuracy, demonstrating that this approach offers a promising path for efficient and trustworthy
reasoning.
Read the full abstract ▾
Computation and Language
COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations
Rui Xing, Preslav Nakov, Timothy Baldwin et al. (+1 more)
Fact-checking on major platforms, such as X, Meta, and TikTok, is shifting from expert-driven verification to a community-based setup, where users contribute explanatory notes to clarify why a post might be misleading. An important challenge here is determining whether an explanation is helpful for understanding real-world claims and the reasons why, which remains largely underexplored in prior research. In practice, most community notes remain unpublished due to slow community annotation, and the reasons for helpfulness lack clear definitions. To bridge these gaps, we introduce the task of predicting both the helpfulness of explanatory notes and the reason for this. We present COMMUNITYNOTES, a large-scale
multilingual dataset of 104k posts with user-provided notes and helpfulness labels. We further propose a framework that automatically generates and improves reason definitions via automatic
prompt optimization, and integrate them into prediction. Our experiments show that the optimized definitions can improve both helpfulness and reason prediction. Finally, we show that the helpfulness information is beneficial for existing fact-checking systems.
Read the full abstract ▾
All authors Rui Xing, Preslav Nakov, Timothy Baldwin, Jey Han Lau
Computation and Language · Cross-listed from cs.CV
Conflict Adaptation in Vision-Language Models
Xiaoyang Hu
A signature of human cognitive control is conflict adaptation: improved performance on a high-conflict trial following another high-conflict trial. This phenomenon offers an account for how cognitive control, a scarce resource, is recruited. Using a sequential Stroop task, we find that 12 of 13
vision-
language models (VLMs) tested exhibit behavior consistent with conflict adaptation, with the lone exception likely reflecting a ceiling effect. To understand the representational basis of this behavior, we use sparse autoencoders (SAEs) to identify task-relevant supernodes in InternVL 3.5 4B. Partially overlapping supernodes emerge for text and color in both early and late layers, and their relative sizes mirror the automaticity asymmetry between reading and color naming in humans. We further isolate a conflict-modulated supernode in layers 24-25 whose ablation significantly increases Stroop errors while minimally affecting congruent trials.
Read the full abstract ▾
Computation and Language
MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task
Juraj Juraska, Tobias Domhan, Mara Finkelstein et al. (+5 more)
In this paper, we present our submissions to the unified WMT25 Translation Evaluation Shared Task. For the Quality Score Prediction subtask, we create a new generation of MetricX with improvements in the input format and the training protocol, while for the Error Span Detection subtask we develop a new model, GemSpanEval, trained to predict error spans along with their severities and categories. Both systems are based on the state-of-the-art
multilingual open-weights model Gemma 3, fine-tuned on publicly available WMT data. We demonstrate that MetricX-25, adapting Gemma 3 to an encoder-only architecture with a regression head on top, can be trained to effectively predict both MQM and ESA quality scores, and significantly outperforms its predecessor. Our decoder-only GemSpanEval model, on the other hand, we show to be competitive in error span detection with xCOMET, a strong encoder-only sequence-tagging baseline. With error span detection formulated as a generative task, we instruct the model to also output the context for each predicted error span, thus ensuring that error spans are identified unambiguously.
Read the full abstract ▾
All authors Juraj Juraska, Tobias Domhan, Mara Finkelstein, Tetsuji Nakagawa, Geza Kovacs, Daniel Deutsch, Pidong Wang, Markus Freitag
Computation and Language
ComboBench: Can LLMs Manipulate Physical Devices to Play Virtual Reality Games?
Shuqing Li, Jiayi Yan, Chenyu Niu et al. (+5 more)
Virtual Reality (VR) games require players to translate high-level semantic actions into precise device manipulations using controllers and head-mounted displays (HMDs). While humans intuitively perform this translation based on common sense and
embodied understanding, whether Large
Language Models (LLMs) can effectively replicate this ability remains underexplored. This paper introduces a
benchmark, ComboBench, evaluating LLMs' capability to translate semantic actions into VR device manipulation sequences across 262 scenarios from four popular VR games: Half-Life: Alyx, Into the Radius, Moss: Book II, and Vivecraft. We evaluate seven LLMs, including GPT-3.5, GPT-4, GPT-4o, Gemini-1.5-Pro, LLaMA-3-8B, Mixtral-8x7B, and GLM-4-Flash, compared against annotated ground truth and human performance. Our results reveal that while top-performing models like Gemini-1.5-Pro demonstrate strong task decomposition capabilities, they still struggle with procedural
reasoning and spatial understanding compared to humans. Performance varies significantly across games, suggesting sensitivity to interaction complexity. Few-shot examples substantially improve performance, indicating potential for targeted enhancement of LLMs' VR manipulation capabilities. We release all materials at https://sites.google.com/view/combobench.
Read the full abstract ▾
All authors Shuqing Li, Jiayi Yan, Chenyu Niu, Jen-tse Huang, Yun Peng, Wenxuan Wang, Yepang Liu, Michael R. Lyu
Computation and Language
Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents
Yueqi Song, Ketan Ramaneti, Zaid Sheikh et al. (+18 more)
Public research results on large-scale supervised finetuning of AI
agents remain relatively rare, since the collection of
agent training data presents unique challenges. In this work, we argue that the bottleneck is not a lack of underlying data sources, but that a large variety of data is fragmented across heterogeneous formats, tools, and interfaces. To this end, we introduce the
agent data protocol (ADP), a light-weight representation
language that serves as an "interlingua" between
agent datasets in diverse formats and unified
agent training pipelines downstream. The design of ADP is expressive enough to capture a large variety of tasks, including API/tool use, browsing, coding, software engineering, and general agentic workflows, while remaining simple to parse and train on without engineering at a per-
dataset level. In experiments, we unified a broad collection of 13 existing
agent training
datasets into ADP format, and converted the standardized ADP data into training-ready formats for multiple
agent frameworks. We performed SFT on these data, and demonstrated an average performance gain of ~20% over corresponding base models, and delivers state-of-the-art or near-SOTA performance on standard coding, browsing, tool use, and research
benchmarks, without domain-specific tuning. All code and data are released publicly, in the hope that ADP could help lower the barrier to standardized, scalable, and reproducible
agent training.
Read the full abstract ▾
All authors Yueqi Song, Ketan Ramaneti, Zaid Sheikh, Ziru Chen, Boyu Gou, Tianbao Xie, Yiheng Xu, Danyang Zhang, Apurva Gandhi, Fan Yang, Joseph Liu, Tianyue Ou, Zhihao Yuan, Frank Xu, Shuyan Zhou, Xingyao Wang, Xiang Yue, Tao Yu, Huan Sun, Yu Su, Graham Neubig
Computation and Language
Tongyi DeepResearch Technical Report
Tongyi DeepResearch Team: Baixuan Li, Bo Zhang, Dingchu Zhang et al. (+53 more)
We present Tongyi DeepResearch, an agentic large
language model, which is specifically designed for long-horizon, deep information-seeking research tasks. To incentivize autonomous deep research agency, Tongyi DeepResearch is developed through an end-to-end training framework that combines agentic mid-training and agentic post-training, enabling scalable
reasoning and information seeking across complex tasks. We design a highly scalable data synthesis pipeline that is fully automatic, without relying on costly human annotation, and empowers all training stages. By constructing customized environments for each stage, our system enables stable and consistent interactions throughout. Tongyi DeepResearch, featuring 30.5 billion total parameters, with only 3.3 billion activated per
token, achieves state-of-the-art performance across a range of agentic deep research
benchmarks, including Humanity's Last Exam, BrowseComp, BrowseComp-ZH, WebWalkerQA, xbench-DeepSearch, FRAMES and xbench-DeepSearch-2510. We open-source the model, framework, and complete solutions to empower the community.
Read the full abstract ▾
All authors Tongyi DeepResearch Team: Baixuan Li, Bo Zhang, Dingchu Zhang, Fei Huang, Guangyu Li, Guoxin Chen, Huifeng Yin, Jialong Wu, Jingren Zhou, Kuan Li, Liangcai Su, Litu Ou, Liwen Zhang, Pengjun Xie, Rui Ye, Wenbiao Yin, Xinmiao Yu, Xinyu Wang, Xixi Wu, Xuanzhong Chen, Yida Zhao, Zhen Zhang, Zhengwei Tao, Zhongwang Zhang, Zile Qiao, Chenxi Wang, Donglei Yu, Gang Fu, Haiyang Shen, Jiayin Yang, Jun Lin, Junkai Zhang, Kui Zeng, Li Yang, Hailong Yin, Maojia Song, Ming Yan, Minpeng Liao, Peng Xia, Qian Xiao, Rui Min, Ruixue Ding, Runnan Fang, Shaowei Chen, Shen Huang, Shihang Wang, Shihao Cai, Weizhou Shen, Xiaobin Wang, Xin Guan, Xinyu Geng, Yingcheng Shi, Yuning Wu, Zhuo Chen, Zijian Li, Yong Jiang
Computation and Language
AgentFold: Long-Horizon Web Agents with Proactive Context Management
Rui Ye, Zhongwang Zhang, Kuan Li et al. (+12 more)
LLM-based web
agents show immense promise for information seeking, yet their effectiveness on long-horizon tasks is hindered by a fundamental trade-off in context management. Prevailing ReAct-based
agents suffer from context saturation as they accumulate noisy, raw histories, while methods that fixedly summarize the full history at each step risk the irreversible loss of critical details. Addressing these, we introduce AgentFold, a novel
agent paradigm centered on proactive context management, inspired by the human cognitive process of retrospective consolidation. AgentFold treats its context as a dynamic cognitive workspace to be actively sculpted, rather than a passive log to be filled. At each step, it learns to execute a `folding' operation, which manages its historical trajectory at multiple scales: it can perform granular condensations to preserve vital, fine-grained details, or deep consolidations to abstract away entire multi-step sub-tasks. The results on prominent
benchmarks are striking: with simple supervised fine-tuning (without continual pre-training or RL), our AgentFold-30B-A3B
agent achieves 36.2% on BrowseComp and 47.3% on BrowseComp-ZH. Notably, this performance not only surpasses or matches open-source models of a dramatically larger scale, such as the DeepSeek-V3.1-671B-A37B, but also surpasses leading proprietary
agents like OpenAI's o4-mini.
Read the full abstract ▾
All authors Rui Ye, Zhongwang Zhang, Kuan Li, Huifeng Yin, Zhengwei Tao, Yida Zhao, Liangcai Su, Liwen Zhang, Zile Qiao, Xinyu Wang, Pengjun Xie, Fei Huang, Siheng Chen, Jingren Zhou, Yong Jiang
Computation and Language
ParallelMuse: Agentic Parallel Thinking for Deep Information Seeking
Baixuan Li, Dingchu Zhang, Jialong Wu et al. (+9 more)
Parallel thinking expands exploration breadth, complementing the deep exploration of information-seeking (IS)
agents to further enhance problem-solving capability. However, conventional parallel thinking faces two key challenges in this setting: inefficiency from repeatedly rolling out from scratch, and difficulty in integrating long-horizon
reasoning trajectories during answer generation, as limited context capacity prevents full consideration of the
reasoning process. To address these issues, we propose ParallelMuse, a two-stage paradigm designed for deep IS
agents. The first stage, Functionality-Specified Partial Rollout, partitions generated sequences into functional regions and performs uncertainty-guided path reuse and branching to enhance exploration efficiency. The second stage, Compressed
Reasoning Aggregation, exploits
reasoning redundancy to losslessly compress information relevant to answer derivation and synthesize a coherent final answer. Experiments across multiple open-source
agents and
benchmarks demonstrate up to 62% performance improvement with a 10–30% reduction in exploratory
token consumption.
Read the full abstract ▾
All authors Baixuan Li, Dingchu Zhang, Jialong Wu, Wenbiao Yin, Zhengwei Tao, Yida Zhao, Liwen Zhang, Haiyang Shen, Runnan Fang, Pengjun Xie, Jingren Zhou, Yong Jiang
Computation and Language
WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking
Zhengwei Tao, Haiyang Shen, Baixuan Li et al. (+11 more)
Large
Language Model (LLM)-based
agents have emerged as a transformative approach for open-ended problem solving, with information seeking (IS) being a core capability that enables autonomous
reasoning and decision-making. While prior research has largely focused on improving
retrieval depth, we observe that current IS
agents often suffer from low search efficiency, which in turn constrains overall performance. A key factor underlying this inefficiency is the sparsity of target entities in training tasks, which limits opportunities for
agents to learn and generalize efficient search behaviors. To address these challenges, we propose WebLeaper, a framework for constructing high-coverage IS tasks and generating efficient solution trajectories. We formulate IS as a tree-structured
reasoning problem, enabling a substantially larger set of target entities to be embedded within a constrained context. Leveraging curated Wikipedia tables, we propose three variants for synthesizing IS tasks, Basic, Union, and Reverse-Union, to systematically increase both IS efficiency and efficacy. Finally, we curate training trajectories by retaining only those that are simultaneously accurate and efficient, ensuring that the model is optimized for both correctness and search performance. Extensive experiments on both basic and comprehensive settings, conducted on five IS
benchmarks, BrowserComp,
GAIA, xbench-DeepSearch, WideSearch, and Seal-0, demonstrate that our method consistently achieves improvements in both effectiveness and efficiency over strong baselines.
Read the full abstract ▾
All authors Zhengwei Tao, Haiyang Shen, Baixuan Li, Wenbiao Yin, Jialong Wu, Kuan Li, Zhongwang Zhang, Huifeng Yin, Rui Ye, Liwen Zhang, Xinyu Wang, Pengjun Xie, Jingren Zhou, Yong Jiang
Computation and Language
AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis
Xuanzhong Chen, Zile Qiao, Guoxin Chen et al. (+7 more)
Training large
language model
agents on tasks at the frontier of their capabilities is key to unlocking advanced
reasoning. We introduce a data synthesis approach inspired by the educational theory of the Zone of Proximal Development (ZPD), which defines this frontier as tasks an LLM cannot solve alone but can master with guidance. To operationalize this, we present the AgentFrontier Engine, an automated pipeline that synthesizes high-quality, multidisciplinary data situated precisely within the LLM's ZPD. This engine supports both continued pre-training with knowledge-intensive data and targeted post-training on complex
reasoning tasks. From the same framework, we derive the ZPD Exam, a dynamic and automated
benchmark designed to evaluate
agent capabilities on these frontier tasks. We train AgentFrontier-30B-A3B model on our synthesized data, which achieves state-of-the-art results on demanding
benchmarks like Humanity's Last Exam, even surpassing some leading proprietary
agents. Our work demonstrates that a ZPD-guided approach to data synthesis offers a scalable and effective path toward building more capable LLM
agents.
Read the full abstract ▾
All authors Xuanzhong Chen, Zile Qiao, Guoxin Chen, Liangcai Su, Zhen Zhang, Xinyu Wang, Pengjun Xie, Fei Huang, Jingren Zhou, Yong Jiang
Computation and Language
Repurposing Synthetic Data for Fine-grained Search Agent Supervision
Yida Zhao, Kuan Li, Xixi Wu et al. (+11 more)
LLM-based search
agents are increasingly trained on entity-centric synthetic data to solve complex, knowledge-intensive tasks. However, prevailing training methods like Group Relative Policy Optimization (GRPO) discard this rich entity information, relying instead on sparse, outcome-based rewards. This critical limitation renders them unable to distinguish informative "near-miss" samples-those with substantially correct
reasoning but a flawed final answer-from complete failures, thus discarding valuable learning signals. We address this by leveraging the very entities discarded during training. Our empirical analysis reveals a strong positive correlation between the number of ground-truth entities identified during an
agent's
reasoning process and final answer accuracy. Building on this insight, we introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity-aware reward function. E-GRPO assigns partial rewards to incorrect samples proportional to their entity match rate, enabling the model to effectively learn from these "near-misses". Experiments on diverse question-answering (QA) and deep research
benchmarks show that E-GRPO consistently and significantly outperforms the GRPO baseline. Furthermore, our analysis reveals that E-GRPO not only achieves superior accuracy but also induces more efficient
reasoning policies that require fewer tool calls, demonstrating a more effective and sample-efficient approach to aligning search
agents.
Read the full abstract ▾
All authors Yida Zhao, Kuan Li, Xixi Wu, Liwen Zhang, Dingchu Zhang, Baixuan Li, Maojia Song, Zhuo Chen, Chenxi Wang, Xinyu Wang, Kewei Tu, Pengjun Xie, Jingren Zhou, Yong Jiang
Computation and Language · Cross-listed from cs.SD
STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence
Zihan Liu, Zhikang Niu, Qiuyang Xiao et al. (+10 more)
Despite rapid progress in Multi-modal Large
Language Models and Large Audio-
Language Models, existing audio
benchmarks largely test semantics that can be recovered from text captions, masking deficits in fine-grained perceptual
reasoning. We formalize audio 4D intelligence that is defined as
reasoning over sound dynamics in time and 3D space, and introduce
STAR-Bench to measure it.
STAR-Bench combines a Foundational Acoustic Perception setting (six attributes under absolute and relative regimes) with a Holistic Spatio-Temporal
Reasoning setting that includes segment reordering for continuous and discrete processes and spatial tasks spanning static localization, multi-source relations, and dynamic trajectories. Our data curation pipeline uses two methods to ensure high-quality samples. For foundational tasks, we use procedurally synthesized and physics-simulated audio. For holistic data, we follow a four-stage process that includes human annotation and final selection based on human performance. Unlike prior
benchmarks where caption-only answering reduces accuracy slightly,
STAR-Bench induces far larger drops (-31.5% temporal, -35.2% spatial), evidencing its focus on linguistically hard-to-describe cues. Evaluating 19 models reveals substantial gaps compared with humans and a capability hierarchy: closed-source models are bottlenecked by fine-grained perception, while open-source models lag across perception, knowledge, and
reasoning. Our
STAR-Bench provides critical insights and a clear path forward for developing future models with a more robust understanding of the physical world.
Read the full abstract ▾
All authors Zihan Liu, Zhikang Niu, Qiuyang Xiao, Zhisheng Zheng, Ruoqi Yuan, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Jianze Liang, Xie Chen, Leilei Sun, Dahua Lin, Jiaqi Wang
Computation and Language
SPICE: Self-Play In Corpus Environments Improves Reasoning
Bo Liu, Chuanyang Jin, Seungone Kim et al. (+7 more)
Self-improving systems require environmental interaction for continuous adaptation. We introduce SPICE (Self-Play In Corpus Environments), a reinforcement learning framework where a single model acts in two roles: a Challenger that mines documents from a large corpus to generate diverse
reasoning tasks, and a Reasoner that solves them. Through
adversarial dynamics, the Challenger creates an automatic curriculum at the frontier of the Reasoner's capability, while corpus grounding provides the rich, near-inexhaustible external signal necessary for sustained improvement. Unlike existing ungrounded self-play methods that offer more limited benefits, SPICE achieves consistent gains across mathematical (+8.9%) and general
reasoning (+9.8%)
benchmarks on multiple model families. Our analysis reveals how document grounding is a key ingredient in SPICE to continuously generate its own increasingly challenging goals and achieve them, enabling sustained self-improvement.
Read the full abstract ▾
All authors Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, Jason Weston
Computation and Language
Dissecting Role Cognition in Medical LLMs via Neuronal Ablation
Xun Liang, Huayi Lai, Hanyu Wang et al. (+5 more)
Large
language models (LLMs) have gained significant traction in medical decision support systems, particularly in the context of medical question answering and role-playing simulations. A common practice,
Prompt-Based Role Playing (PBRP), instructs models to adopt different clinical roles (e.g., medical students, residents, attending physicians) to simulate varied professional behaviors. However, the impact of such role
prompts on model
reasoning capabilities remains unclear. This study introduces the RP-
Neuron-Activated Evaluation Framework(RPNA) to evaluate whether role
prompts induce distinct, role-specific cognitive processes in LLMs or merely modify linguistic style. We test this framework on three medical QA
datasets, employing
neuron ablation and representation analysis techniques to assess changes in
reasoning pathways. Our results demonstrate that role
prompts do not significantly enhance the medical
reasoning abilities of LLMs. Instead, they primarily affect surface-level linguistic features, with no evidence of distinct
reasoning pathways or cognitive differentiation across clinical roles. Despite superficial stylistic changes, the core decision-making mechanisms of LLMs remain uniform across roles, indicating that current PBRP methods fail to replicate the cognitive complexity found in real-world medical practice. This highlights the limitations of role-playing in medical AI and emphasizes the need for models that simulate genuine cognitive processes rather than linguistic imitation.We have released the related code in the following repository:https: //github.com/IAAR-Shanghai/RolePlay_LLMDoctor
Read the full abstract ▾
All authors Xun Liang, Huayi Lai, Hanyu Wang, Wentao Zhang, Linfeng Zhang, Yanfang Chen, Feiyu Xiong, Zhiyu Li
Computation and Language
InteractComp: Evaluating Search Agents With Ambiguous Queries
Mingyi Deng, Lijun Huang, Yani Fan et al. (+23 more)
Language agents have demonstrated remarkable potential in web search and information
retrieval. However, many search-
agent benchmarks assume that user queries are complete and unambiguous. This assumption leaves under-tested a practical failure mode:
agents may face ambiguous requests where the intended target cannot be identified without clarification. Yet most
agents lack interactive mechanisms during the search process, and existing
benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a
benchmark designed to evaluate whether search
agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates controlled ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than
reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search
agents. The code is available at https://github.com/FoundationAgents/InteractComp.
Read the full abstract ▾
All authors Mingyi Deng, Lijun Huang, Yani Fan, Fanqi Kong, Jiayi Zhang, Fashen Ren, Jinyi Bai, Fuzhen Yang, Dayi Miao, Zhaoyang Yu, Yifan Wu, Yanfei Zhang, Fengwei Teng, Yingjia Wan, Song Hu, Yude Li, Xin Jin, Conghao Hu, Haoyu Li, Qirui Fu, Tai Zhong, Xinyu Wang, Xiangru Tang, Nan Tang, Chenglin Wu, Yuyu Luo
Computation and Language
MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation
Parker Riley, Daniel Deutsch, Mara Finkelstein et al. (+3 more)
Human evaluation of machine translation is in an arms race with translation model quality: as our models get better, our evaluation methods need to be improved to ensure that quality gains are not lost in evaluation noise. To this end, we experiment with a two-stage version of the current state-of-the-art translation evaluation paradigm (MQM), which we call MQM re-annotation. In this setup, an MQM annotator reviews and edits a set of pre-existing MQM annotations, that may have come from themselves, another human annotator, or an automatic MQM annotation system. We demonstrate that rater behavior in re-annotation aligns with our goals, and that re-annotation results in higher-quality annotations, mostly due to finding errors that were missed during the first pass.
Read the full abstract ▾
All authors Parker Riley, Daniel Deutsch, Mara Finkelstein, Colten DiIanni, Juraj Juraska, Markus Freitag