Computation and Language
ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering
Marianne Menglin Liu, Daniel Garcia, Fjona Parllaku et al. (+3 more)
Large
language model (LLM)
agents rely on external tools to solve complex tasks, but real-world toolsets often contain redundant tools with overlapping names and descriptions, introducing ambiguity and reducing selection accuracy. LLMs also face strict input context limits, preventing efficient consideration of large toolsets. To address these challenges, we propose ToolScope, which includes: (1) ToolScopeMerger with Auto-Correction to automatically audit and fix tool merges, reducing redundancy, and (2) ToolScopeRetriever to rank and select only the most relevant tools for each query, compressing toolsets to fit within context limits without sacrificing accuracy. Evaluations on three state-of-the-art LLMs and three open-source tool-use
benchmarks show gains of 8.38% to 38.6% in tool selection accuracy, demonstrating ToolScope's effectiveness in enhancing LLM tool use.
Read the full abstract ▾
All authors Marianne Menglin Liu, Daniel Garcia, Fjona Parllaku, Vikas Upadhyay, Syed Fahad Allam Shah, Dan Roth
Computation and Language
Improving Transfer Learning for Sequence Labeling Tasks by Adapting Pre-trained Neural Language Models
David Dukić
This doctoral thesis improves the transfer learning for sequence labeling tasks by adapting pre-trained neural
language models. The proposed improvements in transfer learning involve introducing a multi-task model that incorporates an additional signal, a method based on architectural modifications in autoregressive large
language models, and a sequence labeling framework for autoregressive large
language models utilizing supervised in-context fine-tuning combined with response-oriented adaptation strategies. The first improvement is given in the context of domain transfer for the event trigger detection task. The domain transfer of the event trigger detection task can be improved by incorporating an additional signal obtained from a domain-independent text processing system into a multi-task model. The second improvement involves modifying the model's architecture. For that purpose, a method is proposed to enable bidirectional information flow across layers of autoregressive large
language models. The third improvement utilizes autoregressive large
language models as text generators through a generative supervised in-context fine-tuning framework. The proposed model, method, and framework demonstrate that pre-trained neural
language models achieve their best performance on sequence labeling tasks when adapted through targeted transfer learning paradigms.
Read the full abstract ▾
Computation and Language
Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation
Alexandra Apostolopoulou, Konstantinos Kanaris, Athanasios Koursaris et al. (+3 more)
The advancement of natural
language processing for morphologically rich and moderately-resourced
languages like Modern Greek has been hindered by architectural stagnation, data scarcity, and limited context processing capabilities, particularly in specialized domains such as law. In this work, we propose the Greek
Embedding Models (GEMs), a new family of
transformer-based
language models, specifically developed to address these limitations through architectural diversity and enhanced data curation. The proposed family of models are trained on several large-scale, meticulously curated corpora, encompassing both comprehensive general-domain
datasets and specialized legal collections, addressing the persistent data scarcity that has impeded Greek
language modeling advancement. The proposed quality-based corpus curation methodology incorporates extensive preprocessing pipelines, sophisticated deduplication strategies and targeted repetition of high-quality legal sub-corpora to enhance domain adaptation. The GEMs family comprises both established architectures (RoBERTa and Longformer) and advanced models not previously applied to Greek (ELECTRA, ConvBERT, and ModernBERT), providing comprehensive coverage of modern
transformer designs. Additionally, we introduce the first bilingual Greek-English
embedding models tailored for cross-lingual legal applications. Comprehensive evaluation across three core natural
language understanding
benchmarks demonstrates that the proposed GEM-RoBERTa and GEM-ConvBERT achieve statistically significant performance improvements over established state-of-the-art models, with accuracy gains of up to 3.6% while conducted statistical analysis using Friedman Aligned-Ranks and Finner post-hoc tests confirms the superiority of our approach across multiple evaluation metrics.
Read the full abstract ▾
All authors Alexandra Apostolopoulou, Konstantinos Kanaris, Athanasios Koursaris, Dimitris Tsakalidis, George Domalis, Ioannis E. Livieris
Computation and Language
Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs
Yunpeng Xiao, Carl Yang, Mark Mai et al. (+2 more)
Large
language models (LLMs) show promise for clinical use. They are often evaluated using
datasets such as MedQA. However, Many medical
datasets, such as MedQA, rely on simplified Question-Answering (Q\A) that underrepresents real-world clinical decision-making. Based on this, we propose a unifying paradigm that characterizes clinical decision-making tasks along two dimensions: Clinical Backgrounds and Clinical Questions. As the background and questions approach the real clinical environment, the difficulty increases. We summarize the settings of existing
datasets and
benchmarks along two dimensions. Then we review methods to address clinical decision-making, including training-time and test-time techniques, and summarize when they help. Next, we extend evaluation beyond accuracy to include efficiency, explainability. Finally, we highlight open challenges. Our paradigm clarifies assumptions, standardizes comparisons, and guides the development of clinically meaningful LLMs.
Read the full abstract ▾
All authors Yunpeng Xiao, Carl Yang, Mark Mai, Xiao Hu, Kai Shu
Computation and Language
A Fundamental Algorithm for Dependency Parsing (With Corrections)
Michael A. Covington
This paper presents a fundamental
algorithm for parsing natural
language sentences into dependency trees. Unlike phrase-structure (constituency) parsers, this
algorithm operates one word at a time, attaching each word as soon as it can be attached, corresponding to properties claimed for the parser in the human
brain. Like phrase-structure parsing, its worst-case complexity is $O(n^3)$, but in human
language, the worst case occurs only for small $n$.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.MA
Communication to Completion: Modeling Collaborative Workflows with Intelligent Multi-Agent Communication
Yiming Lu, Xun Wang, Simin Ma et al. (+6 more)
Multi-
agent LLM systems have demonstrated impressive capabilities in complex collaborative tasks, yet most frameworks treat communication as instantaneous and free, overlooking a fundamental constraint in real world teamwork, collaboration cost. We propose a scalable framework implemented via Communication to Completion (C2C), which explicitly models communication as a constrained resource with realistic temporal costs. We introduce the
Alignment Factor (AF), a dynamic metric inspired by Shared Mental Models, to quantify the link between task understanding and work efficiency. Through experiments on 15 software engineering workflows spanning three complexity tiers and team sizes from 5 to 17
agents, we demonstrate that cost-aware strategies achieve over 40% higher efficiency compared to unconstrained interaction. Our analysis reveals emergent coordination patterns:
agents naturally adopt manager centric hub-and-spoke
topologies, strategically escalate from asynchronous to synchronous channels based on complexity, and prioritize high value help requests. These patterns remain consistent across multiple frontier models (GPT-5.2, Claude Sonnet 4.5, Gemini 2.5 Pro). This study moves beyond simple
agent construction, offering a theoretical foundation for quantifying and optimizing the dynamics of collaboration in future digital workplaces.
Read the full abstract ▾
All authors Yiming Lu, Xun Wang, Simin Ma, Shujian Liu, Sathish Reddy Indurthi, Song Wang, Haoyun Deng, Fei Liu, Kaiqiang Song
Computation and Language
LLM-Augmented Symbolic NLU System for More Reliable Continuous Causal Statement Interpretation
Xin Lian and Kenneth D. Forbus
Despite the broad applicability of large
language models (LLMs), their reliance on probabilistic inference makes them vulnerable to errors such as
hallucination in generated facts and inconsistent output structure in natural
language understanding (NLU) tasks. By contrast, symbolic NLU systems provide interpretable understanding grounded in curated lexicons, semantic resources, and syntactic & semantic interpretation rules. They produce relational representations that can be used for accurate
reasoning and planning, as well as incremental debuggable learning. However, symbolic NLU systems tend to be more limited in coverage than LLMs and require scarce knowledge representation and linguistics skills to extend and maintain. This paper explores a hybrid approach that integrates the broad-coverage
language processing of LLMs with the symbolic NLU capabilities of producing structured relational representations to hopefully get the best of both approaches. We use LLMs for rephrasing and text simplification, to provide broad coverage, and as a source of information to fill in knowledge gaps more automatically. We use symbolic NLU to produce representations that can be used for
reasoning and for incremental learning. We evaluate this approach on the task of extracting and interpreting quantities and causal laws from commonsense science texts, along with symbolic- and LLM-only pipelines. Our results suggest that our hybrid method works significantly better than the symbolic-only pipeline.
Read the full abstract ▾
Computation and Language
LyriCAR: A Difficulty-Aware Curriculum Reinforcement Learning Framework For Controllable Lyric Translation
Le Ren, Xiangjian Zeng, Qingqiang Wu et al. (+1 more)
Lyric translation is a challenging task that requires balancing multiple musical constraints. Existing methods often rely on hand-crafted rules and sentence-level modeling, which restrict their ability to internalize musical-linguistic patterns and to generalize effectively at the paragraph level, where cross-line coherence and global rhyme are crucial. In this work, we propose LyriCAR, a novel framework for controllable lyric translation that operates in a fully unsupervised manner. LyriCAR introduces a difficulty-aware curriculum designer and an adaptive curriculum strategy, ensuring efficient allocation of training resources, accelerating convergence, and improving overall translation quality by guiding the model with increasingly complex challenges. Extensive experiments on the EN-ZH lyric translation task show that LyriCAR achieves state-of-the-art results across both standard translation metrics and multi-dimensional reward scores, surpassing strong baselines. Notably, the adaptive curriculum strategy reduces training steps by nearly 40% while maintaining superior performance. Code, data and model can be accessed at https://github.com/rle27/LyriCAR.
Read the full abstract ▾
All authors Le Ren, Xiangjian Zeng, Qingqiang Wu, Ruoxuan Liang
Computation and Language · Cross-listed from cs.SE
BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills
Atharv Sonwane, Isadora White, Hyunji Lee et al. (+8 more)
High quality bugs are key to training the next generation of
language model based software engineering (SWE)
agents. We introduce a novel method for synthetic generation of difficult and diverse bugs. Our method instructs SWE
Agents to introduce a feature into the codebase whereby they may unintentionally break tests, resulting in bugs. Prior approaches often induce an out-of-distribution effect by generating bugs intentionally (e.g. by introducing local perturbation to existing code), which does not reflect realistic development processes. We perform qualitative analysis to demonstrate that our approach for generating bugs more closely reflects the patterns found in human-authored edits. Through extensive experiments, we demonstrate that our bugs provide more efficient training data for supervised fine-tuning, outperforming other bug
datasets by 2% with half the training data (1.2k vs. 3k bugs). We train on our newly generated bugs in addition to existing bug
datasets to get FrogBoss a state-of-the-art 32B parameter model on SWE-bench Verified with a pass@1 of 54.6% and FrogMini a state-of-the-art 14B model on SWE-bench Verified with a pass@1 of 45.3% on SWE-bench Verified averaged over three seeds.
Read the full abstract ▾
All authors Atharv Sonwane, Isadora White, Hyunji Lee, Matheus Pereira, Lucas Caccia, Minseon Kim, Zhengyan Shi, Chinmay Singh, Alessandro Sordoni, Marc-Alexandre Côté, Xingdi Yuan
Computation and Language
Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation
Jackson Hassell, Dan Zhang, Hannah Kim et al. (+2 more)
We investigate how
agents built on pretrained large
language models (LLMs) can learn target classification functions from labeled examples without parameter updates. While conventional approaches like fine-tuning are often costly, inflexible, and opaque, we propose a
memory-augmented framework that leverages LLM-generated critiques grounded in labeled data. Our framework uses episodic
memory to store instance-level critiques - capturing specific past experiences - and semantic
memory to distill these into reusable, task-level guidance. Across a diverse set of tasks and models, our best performing self-critique strategy (utilizing both
memory types) yields an average improvement of 8.1 percentage points over the zero shot baseline, and 4.6pp over a RAG-based baseline that relies only on labels. However, improvements vary substantially across models and domains. To explain this variation, we introduce suggestibility - a novel metric capturing how receptive a model is to external
reasoning provided in context. We use suggestibility to illuminate when and why
memory augmentation succeeds or falls short. Beyond accuracy gains, we find pre-computed critiques substantially reduce inference-time computation for
reasoning models, cutting thinking
tokens by an average of 31.95% across all
datasets by substituting for
reasoning that the model would otherwise perform independently. Our findings highlight the conditions under which
memory-driven, reflective learning can serve as a lightweight, interpretable, and efficient strategy for improving LLM adaptability.
Read the full abstract ▾
All authors Jackson Hassell, Dan Zhang, Hannah Kim, Tom Mitchell, Estevam Hruschka
Computation and Language
Large Language Model enabled Mathematical Modeling
Guoyun Zhang
The integration of Large
Language Models (LLMs) with optimization modeling offers a promising avenue for advancing decision-making in operations research (OR). Traditional optimization methods,such as linear programming, mixed integer programming, and simulation depend heavily on domain expertise to translate real-world problems into solvable mathematical models. While solvers like Gurobi and COPT are powerful, expert input remains essential for defining objectives, constraints, and variables. This research investigates the potential of LLMs, specifically the DeepSeek-R1 model, to bridge this formulation gap using natural
language understanding and code generation. Although prior models like GPT-4, Claude, and Bard have shown strong performance in NLP and
reasoning tasks, their high
token costs and tendency toward
hallucinations limit real-world applicability in supply chain contexts. In contrast, DeepSeek-R1, a cost-efficient and high-performing model trained with reinforcement learning, presents a viable alternative. Despite its success in
benchmarks such as LiveCodeBench and Math-500, its effectiveness in applied OR scenarios remains under explored. This study systematically evaluates DeepSeek-R1 across four key OR
benchmarks: NL4OPT, IndustryOR, EasyLP, and ComplexOR. Our methodology includes baseline assessments, the development of a
hallucination taxonomy, and the application of mitigation strategies like LLM-as-a-Judge, Few-shot Learning (FSL), Tool Calling, and a Multi-
agent Framework. These techniques aim to reduce
hallucinations, enhance formulation accuracy, and better align model outputs with user intent.
Read the full abstract ▾
Computation and Language
Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilities
Nishant Balepur, Dang Nguyen and Dayeon Ki
Multi-modal large
language models (MLMs) are often assessed on static, individual
benchmarks – which cannot jointly assess MLM capabilities in a single task – or rely on human or model pairwise comparisons – which is highly subjective, expensive, and allows models to exploit superficial shortcuts (e.g., verbosity) to inflate their win-rates. To overcome these issues, we propose game-based evaluations to holistically assess MLM capabilities. Games require multiple abilities for players to win, are inherently competitive, and are governed by fix, objective rules, and makes evaluation more engaging, providing a robust framework to address the aforementioned challenges. We manifest this evaluation specifically through Dixit, a fantasy card game where players must generate captions for a card that trick some, but not all players, into selecting the played card. Our quantitative experiments with five MLMs show Dixit win-rate rankings are perfectly correlated with those on popular MLM
benchmarks, while games between human and MLM players in Dixit reveal several differences between
agent strategies and areas of improvement for MLM
reasoning.
Read the full abstract ▾
Computation and Language
An Expert-grounded benchmark of General Purpose LLMs in LCA
Artur Donaldson, Bharathan Balaji, Cajetan Oriekezie et al. (+2 more)
Purpose: Artificial intelligence (AI), and in particular large
language models (LLMs), are increasingly being explored as tools to support life cycle assessment (LCA). While demonstrations exist across environmental and social domains, systematic evidence on their reliability, robustness, and usability remains limited. This study provides the first expert-grounded
benchmark of LLMs in LCA, addressing the absence of standardized evaluation frameworks in a field where no clear ground truth or consensus protocols exist. Methods: We evaluated eleven general-purpose LLMs, spanning both commercial and open-source families, across 22 LCA-related tasks. Seventeen experienced practitioners reviewed model outputs against criteria directly relevant to LCA practice, including scientific accuracy, explanation quality, robustness, verifiability, and adherence to instructions. We collected 168 expert reviews. Results: Experts judged 37% of responses to contain inaccurate or misleading information. Ratings of accuracy and quality of explanation were generally rated average or good on many models even smaller models, and format adherence was generally rated favourably.
Hallucination rates varied significantly, with some models producing hallucinated citations at rates of up to 40%. There was no clear-cut distinction between ratings on open-weight versus closed-weight LLMs, with open-weight models outperforming or competing on par with closed-weight models on criteria such as accuracy and quality of explanation. Conclusion: These findings highlight the risks of applying LLMs naïvely in LCA, such as when LLMs are treated as free-form oracles, while also showing benefits especially around quality of explanation and alleviating labour intensiveness of simple tasks. The use of general-purpose LLMs without grounding mechanisms presents ...
Read the full abstract ▾
All authors Artur Donaldson, Bharathan Balaji, Cajetan Oriekezie, Manish Kumar, Laure Patouillard
Computation and Language
Automated HIV Screening on Dutch Electronic Health Records with Large Language Models
Lang Zhou, Amrish Jhingoer, Yinghao Luo et al. (+7 more)
Efficient screening and early diagnosis of HIV are critical for reducing onward transmission. Although large scale laboratory testing is not feasible, the widespread adoption of Electronic Health Records (EHRs) offers new opportunities to address this challenge. Existing research primarily focuses on applying machine learning methods to structured data, such as patient demographics, for improving HIV diagnosis. However, these approaches often overlook unstructured text data such as clinical notes, which potentially contain valuable information relevant to HIV risk. In this study, we propose a novel pipeline that leverages a Large
Language Model (LLM) to analyze unstructured EHR text and determine a patient's eligibility for further HIV testing. Experimental results on clinical data from Erasmus University Medical Center Rotterdam demonstrate that our pipeline achieved high accuracy while maintaining a low false negative rate.
Read the full abstract ▾
All authors Lang Zhou, Amrish Jhingoer, Yinghao Luo, Klaske Vliegenthart–Jongbloed, Carlijn Jordans, Ben Werkhoven, Tom Seinen, Erik van Mulligen, Casper Rokx, Yunlei Li
Computation and Language
Stream: Scaling up Mechanistic Interpretability to Long Context in LLMs via Sparse Attention
J Rosser, José Luis Redondo García, Gustavo Penha et al. (+2 more)
As Large
Language Models (LLMs) scale to million-
token contexts, traditional Mechanistic Interpretability techniques for analyzing
attention scale quadratically with context length, demanding terabytes of
memory beyond 100,000
tokens. We introduce Sparse Tracing, a novel technique that leverages dynamic sparse
attention to efficiently analyze long context
attention patterns. We present Stream, a compilable hierarchical
pruning algorithm that estimates per-head sparse
attention masks in near-linear time $O(T \log T)$ and linear space $O(T)$, enabling one-pass interpretability at scale. Stream performs a binary-search-style refinement to retain only the top-$k$ key blocks per query while preserving the model's next-
token behavior. We apply Stream to long chain-of-thought
reasoning traces and identify thought anchors while
pruning 97-99% of
token interactions. On the RULER
benchmark, Stream preserves critical
retrieval paths while discarding 90-96% of interactions and exposes layer-wise routes from the needle to output. Our method offers a practical drop-in tool for analyzing
attention patterns and tracing information flow without terabytes of caches. By making long context interpretability feasible on consumer GPUs, Sparse Tracing helps democratize chain-of-thought monitoring. Code is available at https://anonymous.4open.science/r/stream-03B8/.
Read the full abstract ▾
All authors J Rosser, José Luis Redondo García, Gustavo Penha, Konstantina Palla, Hugues Bouchard
Computation and Language
From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model
Yatai Ji, Teng Wang, Yuying Ge et al. (+4 more)
Discrete
diffusion models have emerged as a promising direction for
vision-
language tasks, offering bidirectional context modeling and theoretical parallelization. However, their practical application is severely hindered by a train-inference discrepancy, which leads to catastrophic error cascades: initial
token errors during parallel decoding pollute the generation context, triggering a chain reaction of compounding errors and leading to syntactic errors and semantic
hallucinations. To address this fundamental challenge, we reframe the generation process from passive denoising to active refining. We introduce ReDiff, a refining-enhanced
diffusion framework that teaches the model to identify and correct its own errors. Our approach features a two-stage training process: first, we instill a foundational revision capability by training the model to revise synthetic errors; second, we implement a novel online self-correction loop where the model is explicitly trained to revise its own flawed drafts by learning from an expert's corrections. This mistake-driven learning endows the model with the crucial ability to revisit and refine its already generated output, effectively breaking the error cascade. Extensive experiments demonstrate that ReDiff significantly improves the coherence and factual accuracy of generated content, enabling stable and efficient parallel generation far superior to traditional denoising methods. Our codes and models are available at https://rediff-hku.github.io/.
Read the full abstract ▾
All authors Yatai Ji, Teng Wang, Yuying Ge, Zhiheng Liu, Sidi Yang, Ying Shan, Ping Luo
Computation and Language
An Evaluation of the Pedagogical Soundness and Usability of AI-Generated Lesson Plans Across Different Models and Prompt Frameworks in High-School Physics
Xincheng Liu
This study evaluates the pedagogical soundness and usability of AI-generated lesson plans across five leading large
language models: ChatGPT (GPT-5), Claude Sonnet 4.5, Gemini 2.5 Flash, DeepSeek V3.2, and Grok 4. Beyond model choice, three structured
prompt frameworks were tested: TAG (Task, Audience, Goal), RACE (Role, Audience, Context, Execution), and COSTAR (Context, Objective, Style, Tone, Audience, Response Format). Fifteen lesson plans were generated for a single high-school physics topic, The Electromagnetic Spectrum. The lesson plans were analyzed through four automated computational metrics: (1) readability and linguistic complexity, (2) factual accuracy and
hallucination detection, (3) standards and curriculum
alignment, and (4) cognitive demand of learning objectives. Results indicate that model selection exerted the strongest influence on linguistic accessibility, with DeepSeek producing the most readable teaching plan (FKGL = 8.64) and Claude generating the densest
language (FKGL = 19.89). The
prompt framework structure most strongly affected the factual accuracy and pedagogical completeness, with the RACE framework yielding the lowest
hallucination index and the highest incidental
alignment with NGSS curriculum standards. Across all models, the learning objectives in the fifteen lesson plans clustered at the Remember and Understand tiers of Bloom's taxonomy. There were limited higher-order verbs in the learning objectives extracted. Overall, the findings suggest that readability is significantly governed by model design, while instructional reliability and curricular
alignment depend more on the
prompt framework. The most effective configuration for lesson plans identified in the results was to combine a readability-optimized model with the RACE framework and an explicit checklist of physics concepts, curriculum standards, and higher-order objectives.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
SODBench: A Large Language Model Approach to Documenting Spreadsheet Operations
Amila Indika and Igor Molybog
Numerous knowledge workers utilize spreadsheets in business, accounting, and finance. However, a lack of systematic documentation methods for spreadsheets hinders automation, collaboration, and knowledge transfer, which risks the loss of crucial institutional knowledge. This paper introduces Spreadsheet Operations Documentation (SOD), an AI task that involves generating human-readable explanations from spreadsheet operations. Many previous studies have utilized Large
Language Models (LLMs) for generating spreadsheet manipulation code; however, translating that code into natural
language for SOD is a less-explored area. To address this, we present a
benchmark of 111 spreadsheet manipulation code snippets, each paired with a corresponding natural
language summary. We evaluate five LLMs, GPT-4o, GPT-4o-mini, LLaMA-3.3-70B, Mixtral-8x7B, and Gemma2-9B, using BLEU, GLEU, ROUGE-L, and METEOR metrics. Our findings suggest that LLMs can generate accurate spreadsheet documentation, making SOD a feasible prerequisite step toward enhancing reproducibility, maintainability, and collaborative workflows in spreadsheets, although there are challenges that need to be addressed.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
olmOCR 2: Unit Test Rewards for Document OCR
Jake Poznanski, Luca Soldaini and Kyle Lo
We present olmOCR 2, the latest in our family of powerful OCR systems for converting digitized print documents, like PDFs, into clean, naturally ordered plain text. olmOCR 2 is powered by olmOCR-2-7B-1025, a specialized, 7B
vision language model (VLM) trained using reinforcement learning with verifiable rewards (RLVR), where our rewards are a diverse set of binary unit tests. To scale unit test creation, we develop a pipeline for generating synthetic documents with diverse and challenging layouts, known ground-truth HTML source code, and extracted test cases. We show that RL training on these test cases results in state-of-the-art performance on olmOCR-Bench, our English-
language OCR
benchmark, with the largest improvements in math formula conversion, table parsing, and multi-column layouts compared to previous versions. We release our model, data and code under permissive open licenses.
Read the full abstract ▾
Computation and Language
Hubble: a Model Suite to Advance the Study of LLM Memorization
Johnny Tian-Zheng Wei, Ameya Godbole, Mohammad Aflah Khan et al. (+7 more)
We present Hubble, a suite of fully open-source large
language models (LLMs) for the scientific study of LLM memorization. Hubble models come in standard and perturbed variants: standard models are pretrained on a large English corpus, and perturbed models are trained in the same way but with controlled insertion of text (e.g., book passages, biographies, and test sets) designed to emulate key memorization risks. Our core release includes 8 models – standard and perturbed models with 1B or 8B parameters, pretrained on 100B or 500B
tokens – establishing that memorization risks are determined by the frequency of sensitive data relative to size of the training corpus (i.e., a password appearing once in a smaller corpus is memorized better than the same password in a larger corpus). Our release also includes 6 perturbed models with text inserted at different pretraining phases, showing that sensitive data without continued exposure can be forgotten. These findings suggest two best practices for addressing memorization risks: to dilute sensitive data by increasing the size of the training corpus, and to order sensitive data to appear earlier in training. Beyond these general empirical findings, Hubble enables a broad range of memorization research; for example, analyzing the biographies reveals how readily different types of private information are memorized. We also demonstrate that the randomized insertions in Hubble make it an ideal testbed for membership inference and machine unlearning, and invite the community to further explore,
benchmark, and build upon our work.
Read the full abstract ▾
All authors Johnny Tian-Zheng Wei, Ameya Godbole, Mohammad Aflah Khan, Ryan Wang, Xiaoyuan Zhu, James Flemings, Nitya Kashyap, Krishna P. Gummadi, Willie Neiswanger, Robin Jia
Computation and Language · Cross-listed from cs.CV
Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing
Yusu Qian, Eli Bocek-Rivele, Liangchen Song et al. (+5 more)
Recent advances in
multimodal models have demonstrated remarkable text-guided image editing capabilities, with systems like GPT-4o and Nano-Banana setting new
benchmarks. However, the research community's progress remains constrained by the absence of large-scale, high-quality, and openly accessible
datasets built from real images. We introduce Pico-Banana-400K, a comprehensive 400K-image
dataset for instruction-based image editing. Our
dataset is constructed by leveraging Nano-Banana to generate diverse edit pairs from real photographs in the OpenImages collection. What distinguishes Pico-Banana-400K from previous synthetic
datasets is our systematic approach to quality and diversity. We employ a fine-grained image editing taxonomy to ensure comprehensive coverage of edit types while maintaining precise content preservation and instruction faithfulness through MLLM-based quality scoring and careful curation. Beyond single turn editing, Pico-Banana-400K enables research into complex editing scenarios. The
dataset includes three specialized subsets: (1) a 72K-example multi-turn collection for studying sequential editing,
reasoning, and planning across consecutive modifications; (2) a 56K-example preference subset for
alignment research and reward model training; and (3) paired long-short editing instructions for developing instruction rewriting and summarization capabilities. By providing this large-scale, high-quality, and task-rich resource, Pico-Banana-400K establishes a robust foundation for training and benchmarking the next generation of text-guided image editing models.
Read the full abstract ▾
All authors Yusu Qian, Eli Bocek-Rivele, Liangchen Song, Jialing Tong, Yinfei Yang, Jiasen Lu, Wenze Hu, Zhe Gan
Computation and Language
Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
Xichen Zhang, Sitong Wu, Yinghao Zhu et al. (+4 more)
Reinforcement learning from verifiable rewards has emerged as a powerful technique for enhancing the complex
reasoning abilities of Large
Language Models (LLMs). However, these methods are fundamentally constrained by the ”learning cliff” phenomenon: when faced with problems far beyond their current capabilities, models consistently fail, yielding a persistent zero-reward signal. In policy optimization
algorithms like GRPO, this collapses the advantage calculation to zero, rendering these difficult problems invisible to the learning gradient and stalling progress. To overcome this, we introduce Scaf-GRPO (Scaffolded Group Relative Policy Optimization), a progressive training framework that strategically provides minimal guidance only when a model's independent learning has plateaued. The framework first diagnoses learning stagnation and then intervenes by injecting tiered in-
prompt hints, ranging from abstract concepts to concrete steps, enabling the model to construct a valid solution by itself. Extensive experiments on challenging mathematics
benchmarks demonstrate Scaf-GRPO's effectiveness, boosting the pass@1 score of the Qwen2.5-Math-7B model on the AIME24
benchmark by a relative 44.3% over a vanilla GRPO baseline. This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach, a critical step towards extending the frontier of autonomous
reasoning in LLM.
Read the full abstract ▾
All authors Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, Jiaya Jia
Computation and Language
The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
David Mora, Viraat Aryabumi, Wei-Yin Ko et al. (+3 more)
Synthetic data has become a cornerstone for scaling LLMs, yet its benefits for
multilingual modeling remain bottlenecked by translation. Translation for data augmentation inherits English-centric framing and style and neglects cultural dimensions, ultimately constraining model generalization. We argue that the overlooked
prompt space offers a more powerful lever for improving
multilingual performance. We introduce a lightweight framework for
prompt-space optimization, where translated
prompts are systematically transformed for Naturalness, Cultural Adaptation, and Difficulty Enhancement. Using an off-the-shelf
multilingual LLM, we apply these transformations to
prompts from 12
languages spanning 7 families. Under identical data conditions, our approach achieves substantial and consistent downstream improvements over the translation-only baseline: +4.7% accuracy on Global-MMLU, +2.4% XCometXL on Flores and +35.3% wins in preferences on mArenaHard. These gains are further corroborated by human preferences on open-ended creative writing. We establish
prompt-space optimization as a simple yet powerful paradigm for building
multilingual LLMs that are more robust, culturally grounded, and globally capable.
Read the full abstract ▾
All authors David Mora, Viraat Aryabumi, Wei-Yin Ko, Sara Hooker, Julia Kreutzer, Marzieh Fadaee
Computation and Language · Cross-listed from cs.LG
Blackbox Model Provenance via Palimpsestic Membership Inference
Rohith Kuditipudi, Jing Huang, Sally Zhu et al. (+3 more)
Suppose Alice trains an open-weight
language model and Bob uses a blackbox derivative of Alice's model to produce text. Can Alice prove that Bob is using her model, either by querying Bob's derivative model (query setting) or from the text alone (observational setting)? We formulate this question as an independence testing problem–in which the null hypothesis is that Bob's model or text is independent of Alice's randomized training run–and investigate it through the lens of palimpsestic memorization in
language models: models are more likely to memorize data seen later in training, so we can test whether Bob is using Alice's model using test statistics that capture correlation between Bob's model or text and the ordering of training examples in Alice's training run. If Alice has randomly shuffled her training data, then any significant correlation amounts to exactly quantifiable statistical evidence against the null hypothesis, regardless of the composition of Alice's training data. In the query setting, we directly estimate (via prompting) the likelihood Bob's model gives to Alice's training examples and order; we correlate the likelihoods of over 40 fine-tunes of various Pythia and OLMo base models ranging from 1B to 12B parameters with the base model's training data order, achieving a p-value on the order of at most 1e-8 in all but six cases. In the observational setting, we try two approaches based on estimating 1) the likelihood of Bob's text overlapping with spans of Alice's training examples and 2) the likelihood of Bob's text with respect to different versions of Alice's model we obtain by repeating the last phase (e.g., 1%) of her training run on reshuffled data. The second approach can reliably distinguish Bob's text from as little as a few hundred
tokens; the first does not involve any retraining but requires many more
tokens (several hundred thousand) to achieve high power.
Read the full abstract ▾
All authors Rohith Kuditipudi, Jing Huang, Sally Zhu, Diyi Yang, Christopher Potts, Percy Liang
Computation and Language
ToolDreamer: Instilling LLM Reasoning Into Tool Retrievers
Saptarshi Sengupta, Zhengyu Zhou, Jun Araki et al. (+4 more)
Tool calling has become increasingly popular for Large
Language Models (LLMs). However, for large tool sets, the resulting
tokens would exceed the LLM's context window limit, making it impossible to include every tool. Hence, an external retriever is used to provide LLMs with the most relevant tools for a query. Existing
retrieval models rank tools based on the similarity between a user query and a tool description (TD). This leads to suboptimal
retrieval as user requests are often poorly aligned with the
language of TD. To remedy the issue, we propose ToolDreamer, a framework to condition retriever models to fetch tools based on hypothetical (synthetic) TD generated using an LLM, i.e., description of tools that the LLM feels will be potentially useful for the query. The framework enables a more natural
alignment between queries and tools within the
language space of TD's. We apply ToolDreamer on the ToolRet
dataset and show that our method improves the performance of sparse and dense retrievers with and without training, thus showcasing its flexibility. Through our proposed framework, our aim is to offload a portion of the
reasoning burden to the retriever so that the LLM may effectively handle a large collection of tools without inundating its context window.
Read the full abstract ▾
All authors Saptarshi Sengupta, Zhengyu Zhou, Jun Araki, Xingbo Wang, Bingqing Wang, Suhang Wang, Zhe Feng
Computation and Language
Adapting Multilingual Models to Code-Mixed Tasks via Model Merging
Prashant Kodali, Vaishnavi Shivkumar, Swarang Joshi et al. (+3 more)
We study model merging as a practical alternative to conventional adaptation strategies for code-mixed NLP. Starting from a
multilingual base model, we: (i) perform continued pre-training (CPT) on unlabeled code-mixed text to obtain an adapted checkpoint, (ii) merge checkpoint with the base model, and (iii) fine-tune (FT) on the downstream task data. We evaluate our approach for sentence classification (sentiment and hate
speech) task in English-Hindi (En-Hi) and English-Spanish (En-Es) using XLM-R and Llama-3.2-1B models. Our results show that merged models consistently outperform full fine-tuning and CPT->FT. We observe gains of 2–5 points in F1 over full fine-tuning and ~1-2 points over CPT->FT, indicating that unlabeled data is leveraged more effectively via merging than via CPT alone. Zero-/few-shot prompting with larger LLMs (e.g., Llama-3.3-70B) lags behind fine-tuned and merged checkpoints, underscoring limits of in-context learning for code-mixed inputs. We further test cross-pair transfer by training on En-Hi and evaluating on En-Ta and En-Ml: merged checkpoints transfer more strongly than monolingual-English baselines (e.g., TV/TIES variants reaching 0.65-0.68 F1 vs 0.61-0.63 for full fine-tuning), suggesting that code-mixed knowledge is a more reliable substrate for low-resource pairs. We conclude with adaptation recipes matched to common data regimes (labeled only; labeled+unlabeled; transfer-only) and discuss limitations and scaling considerations for broader tasks and larger models.
Read the full abstract ▾
All authors Prashant Kodali, Vaishnavi Shivkumar, Swarang Joshi, Monojit Choudhary, Ponnurangam Kumaraguru, Manish Shrivastava
Computation and Language
AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
Yuezhou Hu, Jiaxin Guo, Xinyu Feng et al. (+1 more)
Speculative Decoding (SD) accelerates large
language model inference by employing a small draft model to generate predictions, which are then verified by a larger target model. The effectiveness of SD hinges on the
alignment between these models, which is typically enhanced by Knowledge
Distillation (KD). However, conventional KD methods aim to minimize the KL divergence between the draft and target models across all
tokens, a goal that is misaligned with the true objective of SD, which is to maximize
token acceptance rate. Therefore, draft models often struggle to fully assimilate the target model's knowledge due to capacity constraints, leading to suboptimal performance. To address this challenge, we propose AdaSPEC, a novel method that incorporates selective
token filtering into the KD process. AdaSPEC utilizes a reference model to identify and filter out difficult-to-fit
tokens, enabling the
distillation of a draft model that better aligns with the target model on simpler
tokens. This approach improves the overall
token acceptance rate without compromising generation quality. We evaluate AdaSPEC across diverse tasks, including arithmetic
reasoning, instruction-following, coding, and summarization, using model configurations of 31M/1.4B and 350M/2.7B parameters. Our results demonstrate that AdaSPEC consistently outperforms the state-of-the-art DistillSpec method, achieving higher acceptance rates across all tasks (up to 15%). The code is publicly available at https://github.com/yuezhouhu/adaspec.
Read the full abstract ▾
All authors Yuezhou Hu, Jiaxin Guo, Xinyu Feng, Tuo Zhao
Computation and Language · Cross-listed from cs.LG
GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
Anand Choudhary, Yasser Sula{\i}man, Lukas Mauch et al. (+3 more)
Sparse fine-tuning techniques adapt LLMs to downstream tasks by only tuning a sparse subset of model parameters. However, the effectiveness of sparse adaptation depends on optimally selecting the model parameters to be fine-tuned. In this work, we introduce a novel sparse fine-tuning technique named GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters, which fine-tunes only those model parameters which have the largest gradient magnitudes on downstream tasks and the smallest pre-trained magnitudes, intuitively prioritizing parameters that are highly task-relevant, but minimally disruptive to pre-trained knowledge. Our experimentation with LLaMA3 8B and Gemma 2B as base models shows that GaLLoP consistently improves or matches the in-distribution as well as out-of-distribution performance obtained via the usage of other leading parameter-efficient fine-tuning techniques, including LoRA, DoRA, and SAFT. Our analysis demonstrates that GaLLoP mitigates catastrophic forgetting and memorization of task data, as important pre-trained parameters remain unchanged, and stabilizes performance relative to other fine-tuning techniques, robustly generalizing across most random seeds.
Read the full abstract ▾
All authors Anand Choudhary, Yasser Sula{\i}man, Lukas Mauch, Ghouthi Boukli Hacene, Fabien Cardinaux, Antoine Bosselut
Computation and Language
SmartSwitch: Advancing LLM Reasoning by Overcoming Underthinking via Promoting Deeper Thought Exploration
Xichen Zhang, Sitong Wu, Haoru Tan et al. (+4 more)
The long chain-of-thought (LongCoT) capability is central to the recent breakthroughs achieved by large
language models in complex
reasoning tasks. However, the accompanying issue of ”underthinking”, where models exhibit shallow
reasoning by frequently switching thoughts without sufficient exploration, limits both performance and
token efficiency. To address this problem, we propose a simple yet effective
reasoning strategy: the SmartSwitch inference framework. This framework can be easily integrated into any large
language model as a plug-and-play solution, continuously monitoring the model's
reasoning process to detect underthinking and guide it toward deeper exploration of promising but overlooked thoughts. Specifically, the perception module identifies points where thoughts switch and evaluates the potential of the preceding thought using an off-the-shelf process reward model (PRM). If a high-potential thought is found to be prematurely abandoned, the intervention module interrupts the ongoing inference, backtracks to the point before the switch, and inserts a "deepening
prompt" to encourage further exploration along that promising path. Extensive experiments on challenging mathematical
reasoning benchmarks demonstrate that our method significantly enhances the performance of various large
language models of different sizes.
Read the full abstract ▾
All authors Xichen Zhang, Sitong Wu, Haoru Tan, Shaozuo Yu, Yinghao Zhu, Ziyi He, Jiaya Jia
Computation and Language
Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning
M. H. I. Abdalla, Zhipin Wang, Christian Frey et al. (+2 more)
Large
Language Model (LLM) conditioning refers to instructing an LLM to generate content in accordance with the norms and values of a specific culture, beliefs of a particular political orientation, or any desired text-specified semantic conditioning. Unfortunately,
prompt engineering does not ensure that LLMs behave in accordance with a desired conditioning due to the inductive bias of the pre-training and
alignment datasets. Prior works have focused on fine-tuning LLMs by directly conditioning the LoRA weights; however, such methods introduce a large number of parameters. As a remedy, we propose Zhyper, a parameter-efficient factorized hypernetwork framework that generates context-aware LoRA adapters from textual descriptions. Experiments on multiple
benchmarks show that Zhyper achieves competitive performance with up to 26x fewer parameters than the state-of-the-art baselines. Furthermore, we extend Zhyper to cultural
alignment, demonstrating improved generalization to out-of-domain settings and a better capturing of fine-grained contextual values.
Read the full abstract ▾
All authors M. H. I. Abdalla, Zhipin Wang, Christian Frey, Steffen Eger, Josif Grabocka