Computation and Language
OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs
Jaeseong Lee, seung-won hwang, Aurick Qiao et al. (+3 more)
Speculative decoding promises faster inference for large
language models (LLMs), yet existing methods fail to generalize to real-world settings.
Benchmarks typically assume short contexts (e.g., 2K
tokens), whereas practical workloads involve long contexts. We find current approaches degrade severely with long contexts; for instance, EAGLE3 even slows down the generation speed by 0.81x. We address these limitations by releasing a new long-context
benchmark (LongSpecBench) and introducing a novel model (OWL). OWL achieves about 5x higher acceptance length than EAGLE3 on long-context inputs through three innovations: (1) an LSTM-based drafter conditioned only on the last-
token state, making it generalize to various lengths, (2) a special
token [SPEC] in the verifier that produces richer representation for drafter, and (3) a hybrid
algorithm combining both tree and non-tree decoding methods. We release all code and
datasets to advance future research.
Read the full abstract ▾
All authors Jaeseong Lee, seung-won hwang, Aurick Qiao, Gabriele Oliaro, Ye Wang, Samyam Rajbhandari
Computation and Language
ParsTranslit: Truly Versatile Tajik-Farsi Transliteration
Rayyan Merchant and Kevin Tang
As a digraphic
language, the Persian
language utilizes two written standards: Perso-Arabic in Afghanistan and Iran, and Tajik-Cyrillic in Tajikistan. Despite the significant similarity between the dialects of each country, script differences prevent simple one-to-one mapping, hindering written communication and interaction between Tajikistan and its Persian-speaking “siblings”. To overcome this, previously-published efforts have investigated machine transliteration models to convert between the two scripts. Unfortunately, most efforts did not use
datasets other than those they created, limiting these models to certain domains of text such as archaic poetry or word lists. A truly usable transliteration system must be capable of handling varied domains, meaning that suck models lack the versatility required for real-world usage. The contrast in domain between data also obscures the task's true difficulty. We present a new state-of-the-art sequence-to-sequence model for Tajik-Farsi transliteration trained across all available
datasets, and present two
datasets of our own. Our results across domains provide clearer understanding of the task, and set comprehensive comparable leading
benchmarks. Overall, our model achieves chrF++ and Normalized CER scores of 87.91 and 0.05 from Farsi to Tajik and 92.28 and 0.04 from Tajik to Farsi. Our model, data, and code are available at https://anonymous.4open.science/r/ParsTranslit-FB30/.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
CompassLLM: A Multi-Agent Approach toward Geo-Spatial Reasoning for Popular Path Query
Md. Nazmul Islam Ananto, Shamit Fatin, Mohammed Eunus Ali et al. (+1 more)
The popular path query - identifying the most frequented routes between locations from historical trajectory data - has important applications in urban planning,
navigation optimization, and travel recommendations. While traditional
algorithms and machine learning approaches have achieved success in this domain, they typically require model training, parameter tuning, and retraining when accommodating data updates. As Large
Language Models (LLMs) demonstrate increasing capabilities in spatial and graph-based
reasoning, there is growing interest in exploring how these models can be applied to geo-spatial problems. We introduce CompassLLM, a novel multi-
agent framework that intelligently leverages the
reasoning capabilities of LLMs into the geo-spatial domain to solve the popular path query. CompassLLM employs its
agents in a two-stage pipeline: the SEARCH stage that identifies popular paths, and a GENERATE stage that synthesizes novel paths in the absence of an existing one in the historical trajectory data. Experiments on real and synthetic
datasets show that CompassLLM demonstrates superior accuracy in SEARCH and competitive performance in GENERATE while being cost-effective.
Read the full abstract ▾
All authors Md. Nazmul Islam Ananto, Shamit Fatin, Mohammed Eunus Ali, Md Rizwan Parvez
Computation and Language
When Thoughts Meet Facts: Reusable Reasoning for Long-Context LMs
Soyeong Jeong, Taehee Jung, Sung Ju Hwang et al. (+2 more)
Recent Long-Context
Language Models (LCLMs) can process hundreds of thousands of
tokens in a single
prompt, enabling new opportunities for knowledge-intensive multi-hop
reasoning by integrating large sets of retrieved documents or, in some cases, directly all necessary information. However, simply feeding more documents into the context window fails to capture how evidence should be connected. We address this gap with thought templates, which recast
reasoning as reusable thought caches, derived from prior problem solving traces, structuring how evidence is combined and guiding multi-hop inference with factual documents. To keep these templates effective, we propose an update strategy that iteratively refines templates derived from training data through natural-
language feedback. Across diverse
benchmarks and LCLM families, our approach delivers consistent gains over strong baselines in both
retrieval-based and
retrieval-free settings. Furthermore, we show that optimized templates can be distilled into smaller open-source models, demonstrating its broad applicability and transparent
reasoning reuse. We refer to our framework as Thought Template Augmented LCLMs (ToTAL).
Read the full abstract ▾
All authors Soyeong Jeong, Taehee Jung, Sung Ju Hwang, Joo-Kyung Kim, Dongyeop Kang
Computation and Language
Can Speech LLMs Think while Listening?
Yi-Jen Shih, Desh Raj, Chunyang Wu et al. (+6 more)
Recent advances in
speech large
language models (
speech LLMs) have enabled seamless spoken interactions, but these systems still struggle with complex
reasoning tasks. Previously, chain-of-thought (CoT) prompting or fine-tuning has been to shown to significantly improve the
reasoning abilities of text-based LLMs. In this work, we investigate the effect of CoT fine-tuning for multi-stream
speech LLMs, demonstrating that
reasoning in text space improves the accuracy of
speech LLMs by 2.4x, on average, over a suite of spoken
reasoning tasks. Beyond accuracy, the latency of the spoken response is a crucial factor for interacting with voice-based
agents. Inspired by the human behavior of "thinking while listening," we propose methods to reduce the additional latency from
reasoning by allowing the model to start
reasoning before the user query has ended. To achieve this, we introduce an
entropy-based metric, "question completeness," which acts as an indicator to guide the model on the optimal time to start
reasoning. This method provides greater control over the accuracy-latency trade-off compared with heuristic-based approaches and, under equivalent latency conditions, yields a 4% accuracy gain on ARC-Easy. Finally, we use Direct Preference Optimization (DPO) on preference data created using rejection sampling to push the accuracy-latency pareto frontier further, resulting in a 70% reduction in latency without loss in accuracy.
Read the full abstract ▾
All authors Yi-Jen Shih, Desh Raj, Chunyang Wu, Wei Zhou, SK Bong, Yashesh Gaur, Jay Mahadeokar, Ozlem Kalinli, Mike Seltzer
Computation and Language · Cross-listed from cs.AI
Evaluation of LLMs for Process Model Analysis and Optimization
Akhil Kumar, Jianliang Leon Zhao and Om Dobariya
In this paper, we report our experience with several LLMs for their ability to understand a process model in an interactive, conversational style, find syntactical and logical errors in it, and reason with it in depth through a natural
language (NL) interface. Our findings show that a vanilla, untrained LLM like ChatGPT (model o3) in a zero-shot setting is effective in understanding BPMN process models from images and answering queries about them intelligently at syntactic, logic, and semantic levels of depth. Further, different LLMs vary in performance in terms of their accuracy and effectiveness. Nevertheless, our empirical analysis shows that LLMs can play a valuable role as assistants for business process designers and users. We also study the LLM's "thought process" and ability to perform deeper
reasoning in the context of process analysis and optimization. We find that the LLMs seem to exhibit anthropomorphic properties.
Read the full abstract ▾
Computation and Language
Can Lessons From Human Teams Be Applied to Multi-Agent Systems? The Role of Structure, Diversity, and Interaction Dynamics
Rasika Muralidharan, Haewoon Kwak and Jisun An
Multi-
Agent Systems (MAS) with Large
Language Model (LLM)-powered
agents are gaining
attention, yet fewer studies explore their team dynamics. Inspired by human team science, we propose a multi-
agent framework to examine core aspects of team science: structure, diversity, and interaction dynamics. We evaluate team performance across four tasks: CommonsenseQA, StrategyQA, Social IQa, and Latent Implicit Hate, spanning commonsense and social
reasoning. Our results show that flat teams tend to perform better than hierarchical ones, while diversity has a nuanced impact. Interviews suggest
agents are overconfident about their team performance, yet post-task reflections reveal both appreciation for collaboration and challenges in integration, including limited conversational coordination.
Read the full abstract ▾
Computation and Language
AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding
Shuqing Luo, Yilin Guan, Pingzhi Li et al. (+2 more)
Test-time scaling (TTS) boosts LLM
reasoning via long chain-of-thought (CoT), but the linear KV-cache growth amplifies the
memory-bound bottleneck of LLM decoding. Query-aware page-level sparse decoding can achieve state-of-the-art performance under constrained FLOPs budgets, but is limited by both sequential-dependent page filtering and coarse-grained
token selection, hampering serving efficiency and model performance on TTS tasks under high concurrency and long CoT scenarios (consuming even higher runtime than the forward pipeline itself). In this paper, we first find that the current-step query state can be accurately approximated in a unified manner from a short window of recent queries, enabling training-free query-aware sparsity without waiting in the decoding loop. We propose AsyncSpade, an asynchronous framework for efficient TTS built on two core components: (1) a novel light-weight temporal-regressive module that predicts the next-
token query state; (2) an asynchronous and disaggregated framework that decouples the KV cache filtering from the auto-regressive decoding loop, overlapping the
token-level KV selection with the forward inference computation through asynchronism. To our knowledge, AsyncSpade is the first to eliminate the sequential dependence without sacrificing model performance. We validate the effectiveness of AsyncSpade on common LLM serving setups with an A100 node, where AsyncSpade fully overlaps KV-cache operations with the inference pipeline, achieving theoretical optimal time-per-output-
token (TPOT). Specifically, AsyncSpade delivers over 20% reduction on TPOT compared to SoTA baseline (i.e. Quest) and at least 50% TPOT reduction compared to full
attention on Qwen3-8B and Qwen3-32B models, while matching or surpassing their accuracy on various TTS
benchmarks (AIME-24/25, GPQA-Diamond, MATH-500).
Read the full abstract ▾
All authors Shuqing Luo, Yilin Guan, Pingzhi Li, Hanrui Wang, Tianlong Chen
Computation and Language
MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inference
Zheyuan Zhang, Lin Ge, Hongjiang Li et al. (+3 more)
Large
language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, and LLM-based
agents further extend these abilities to various practical workflows. While recent progress shows that multi-
agent systems (MAS) can outperform single
agents by coordinating specialized roles, designing effective MAS remains difficult due to
prompt sensitivity and the compounded instability MAS creates. To cope with the challenge, recent efforts in automated
prompt design have reduced manual effort. However, multi-
agent prompt optimization remains largely unexplored. Challenges like exponentially expanding search space and ambiguous credit assignment together make systematic design intractable without principled methods. Therefore, we introduce M}ulti-
Agent PRompt Optimization (MAPRO), a four-stage framework that first formulates MAS
prompt optimization as a Maximum a Posteriori (MAP) inference problem and solves it using a
language-guided variant of max-product belief propagation
algorithm. To address credit assignment and updates the system iteratively, MAPRO employs a
topology-aware refinement mechanism that integrates execution feedback and downstream blames to selectively update
agent prompts. Through this process, MAPRO progressively converges to a coordinated set of
agent-specific
prompt policies. Across
benchmarks in various tasks, MAPRO achieves state-of-the-art performance, consistently surpassing manually engineered baselines and recent automated alternatives. Beyond performance, our MAP-based formulation also delivers general guidelines for building more reliable and principled multi-
agent systems in the future
Read the full abstract ▾
All authors Zheyuan Zhang, Lin Ge, Hongjiang Li, Weicheng Zhu, Chuxu Zhang, Yanfang Ye
Computation and Language
Populism Meets AI: Advancing Populism Research with LLMs
Yujin J. Jung, Eduardo Ryô Tamaki, Julia Chatterley et al. (+5 more)
Measuring the ideational content of populism remains a challenge. Traditional strategies based on textual analysis have been critical for building the field's foundations and providing a valid, objective indicator of populist framing. Yet these approaches are costly, time consuming, and difficult to scale across
languages, contexts, and large corpora. Here we present the results from a rubric and anchor guided chain of thought (CoT) prompting approach that mirrors human coder training. By leveraging the Global Populism Database (GPD), a comprehensive
dataset of global leaders'
speeches annotated for degrees of populism, we replicate the process used to train human coders by prompting the LLM with an adapted version of the same documentation to guide the model's
reasoning. We then test multiple proprietary and open weight models by replicating scores in the GPD. Our findings reveal that this domain specific prompting strategy enables the LLM to achieve classification accuracy on par with expert human coders, demonstrating its ability to navigate the nuanced, context sensitive aspects of populism.
Read the full abstract ▾
All authors Yujin J. Jung, Eduardo Ryô Tamaki, Julia Chatterley, Grant Mitchell, Semir Dzebo, Cristóbal Sandoval, Levente Littvay, Kirk A. Hawkins
Computation and Language
Meaningful Pose-Based Sign Language Evaluation
Zifan Jiang, Colin Leong, Amit Moryossef et al. (+8 more)
We present a comprehensive study on meaningfully evaluating sign
language utterances in the form of human skeletal poses. The study covers keypoint distance-based,
embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through automatic meta-evaluation of sign-level
retrieval and a human correlation study of text-to-pose translation across different sign
languages. Our findings and the open-source pose-evaluation toolkit provide a practical and reproducible way of developing and evaluating sign
language translation or generation systems.
Read the full abstract ▾
All authors Zifan Jiang, Colin Leong, Amit Moryossef, Anne Göhring, Annette Rios, Oliver Cory, Maksym Ivashechkin, Neha Tarigopula, Biao Zhang, Rico Sennrich, Sarah Ebling
Computation and Language · Cross-listed from cs.CR
PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHing
Anthony Hughes, Vasisht Duddu, N. Asokan et al. (+2 more)
Language models (LMs) may memorize personally identifiable information (PII) from training data, enabling adversaries to extract it during inference. Existing defense mechanisms such as differential
privacy (DP) reduce this leakage, but incur large drops in utility. Based on a comprehensive study using
circuit discovery to identify the computational
circuits responsible PII leakage in LMs, we hypothesize that specific PII leakage
circuits in LMs should be responsible for this behavior. Therefore, we propose PATCH (
Privacy-Aware Targeted
Circuit PatcHing), a novel approach that first identifies and subsequently directly edits PII
circuits to reduce leakage. PATCH achieves better
privacy-utility trade-off than existing defenses, e.g., reducing recall of PII leakage from LMs by up to 65%. Finally, PATCH can be combined with DP to reduce recall of residual leakage of an LM to as low as 0.01%. Our analysis shows that PII leakage
circuits persist even after the application of existing defense mechanisms. In contrast, PATCH can effectively mitigate their impact.
Read the full abstract ▾
All authors Anthony Hughes, Vasisht Duddu, N. Asokan, Nikolaos Aletras, Ning Ma
Computation and Language
LASER: An LLM-based ASR Scoring and Evaluation Rubric
Amruta Parulekar and Preethi Jyothi
Standard ASR evaluation metrics like Word Error Rate (WER) tend to unfairly penalize morphological and syntactic nuances that do not significantly alter sentence semantics. We introduce an LLM-based scoring rubric
LASER that leverages state-of-the-art LLMs' in-context learning abilities to learn from
prompts with detailed examples. Hindi
LASER scores using Gemini 2.5 Pro achieved a very high correlation score of 94% with human annotations. Hindi examples in the
prompt were also effective in analyzing errors in other Indian
languages such as Marathi, Kannada and Malayalam. We also demonstrate how a smaller LLM like Llama 3 can be finetuned on word-pair examples derived from reference and ASR predictions to predict what kind of penalty should be applied with close to 89% accuracy.
Read the full abstract ▾
Computation and Language
Lemma Dilemma: On Lemma Generation Without Domain- or Language-Specific Training Data
Olia Toporkov, Alan Akbik and Rodrigo Agerri
Lemmatization is the task of transforming all words in a given text to their dictionary forms. While large
language models (LLMs) have demonstrated their ability to achieve competitive results across a wide range of NLP tasks, there is no prior evidence of how effective they are in the contextual lemmatization task. In this paper, we empirically investigate the capacity of the latest generation of LLMs to perform in-context lemmatization, comparing it to the traditional fully supervised approach. In particular, we consider the setting in which supervised training data is not available for a target domain or
language, comparing (i) encoder-only supervised approaches, fine-tuned out-of-domain, and (ii) cross-lingual methods, against direct in-context lemma generation with LLMs. Our experimental investigation across 12
languages of different morphological complexity finds that, while encoders remain competitive in out-of-domain settings when fine-tuned on gold data, current LLMs reach state-of-the-art results for most
languages by directly generating lemmas in-context without prior fine-tuning, provided just with a few examples. Data and code available upon publication: https://github.com/oltoporkov/lemma-dilemma
Read the full abstract ▾
Computation and Language
Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation
Mufei Li, Dongqi Fu, Limei Wang et al. (+10 more)
Modern long-context large
language models (LLMs) perform well on synthetic "needle-in-a-haystack" (NIAH)
benchmarks, but such tests overlook how noisy contexts arise from biased
retrieval and agentic workflows. We argue that haystack engineering is necessary to construct noisy long contexts that faithfully capture key real-world factors – distraction from heterogeneous biased retrievers and cascading errors in agentic workflows – to test models' long-context robustness. We instantiate it through HaystackCraft, a new NIAH
benchmark built on the full English Wikipedia hyperlink network with multi-hop questions. HaystackCraft evaluates how heterogeneous
retrieval strategies (e.g., sparse, dense, hybrid, and graph-based) affect distractor composition, haystack ordering, and downstream LLM performance. HaystackCraft further extends NIAH to dynamic, LLM-dependent settings that simulate agentic operations, where models refine queries, reflect on their past reasonings, and decide when to stop. Experiments with 15 long-context models show that (1) while stronger dense retrievers can introduce more challenging distractors, graph-based reranking simultaneously improves
retrieval effectiveness and mitigates more harmful distractors; (2) in agentic tests, even advanced models like Gemini 2.5 Pro and GPT-5 suffer cascading failures from self-generated distractors or struggle to perform early stops. These results highlight persistent challenges in agentic long-context
reasoning and establish HaystackCraft as a valuable testbed for future progress.
Read the full abstract ▾
All authors Mufei Li, Dongqi Fu, Limei Wang, Si Zhang, Hanqing Zeng, Kaan Sancak, Ruizhong Qiu, Haoyu Wang, Xiaoxin He, Xavier Bresson, Yinglong Xia, Chonglin Sun, Pan Li
Computation and Language
Inconsistent Affective Reaction: Sentiment of Perception and Opinion in Urban Environments
Jingfei Huang and Han Tu
The ascension of social media platforms has transformed our understanding of urban environments, giving rise to nuanced variations in sentiment reaction embedded within human perception and opinion, and challenging existing multidimensional sentiment analysis approaches in urban studies. This study presents novel methodologies for identifying and elucidating sentiment inconsistency, constructing a
dataset encompassing 140,750 Baidu and Tencent Street view images to measure perceptions, and 984,024 Weibo social media text posts to measure opinions. A reaction index is developed, integrating object detection and natural
language processing techniques to classify sentiment in Beijing Second Ring for 2016 and 2022. Classified sentiment reaction is analysed and visualized using regression analysis, image
segmentation, and word frequency based on land-use distribution to discern underlying factors. The perception affective reaction trend map reveals a shift toward more evenly distributed positive sentiment, while the opinion affective reaction trend map shows more extreme changes. Our mismatch map indicates significant disparities between the sentiments of human perception and opinion of urban areas over the years. Changes in sentiment reactions have significant relationships with elements such as dense buildings and pedestrian presence. Our inconsistent maps present perception and opinion sentiments before and after the pandemic and offer potential explanations and directions for environmental management, in formulating strategies for urban renewal.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
Yeskendir Koishekenov, Aldo Lipani and Nicola Cancedda
Most efforts to improve the
reasoning capabilities of large
language models (LLMs) involve either scaling the number of parameters and the size of training data, or scaling inference computation by letting models generate complex chains of thought. Motivated by interpretability studies showing that the crucial computation required for
reasoning tasks is concentrated in a limited range of layers, we introduce Encode-Think-Decode (ETD), a method that enhances the
reasoning capabilities of a base model by training it to iterate over a small subset of
reasoning-relevant layers during the mid-training stage. ETD amplifies latent
reasoning while preserving the original architecture, parameter count, hyperparameters, and training data composition. When iterating on the selected layers at inference time, ETD models yield substantial gains on 17
reasoning benchmarks, including +28.4% relative accuracy improvement on GSM8K and +36% on MATH with the OLMo-2 1B Base model. We also explore an adaptive depth strategy that adjusts the computation per input
token. Our results show that recursive latent
reasoning offers a simple and effective path to stronger LLM
reasoning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
ConCuR: Conciseness Makes State-of-the-Art Kernel Generation
Lingcheng Kong, Jiateng Wei, Hanzhang Shen et al. (+1 more)
GPU kernel generation by LLMs has recently experienced rapid development, leveraging test-time scaling and reinforcement learning techniques. However, a key challenge for kernel generation is the scarcity of high-quality data, as most high-quality kernels are proprietary and not open-source. This challenge prevents us from leveraging supervised fine-tuning to align LLMs to the kernel generation task. To address this challenge, we develop a pipeline that generates and curates high-quality CUDA kernels with
reasoning traces, motivated by a critical observation that concise yet informative
reasoning traces result in robust generation of high-performance kernels. Using this pipeline, we construct our
dataset ConCuR and introduce our model KernelCoder, which is the first model trained on a curated
dataset consisting of PyTorch,
reasoning, and CUDA kernel pairs, to our knowledge. In the KernelBench setup, our model achieves significant improvements over the existing top-performing model, QwQ-32B, and outperforms all open-source models fine-tuned for kernel generation, as well as frontier models such as DeepSeek-V3.1-Think and Claude-4-sonnet. Finally, we show that the average
reasoning length can serve as a metric to assess the difficulty of kernel generation tasks. The observations, metrics, and our data collection and curation pipeline can help obtain better data in the kernel generation task in the future.
Read the full abstract ▾
All authors Lingcheng Kong, Jiateng Wei, Hanzhang Shen, Huan Wang
Computation and Language
Artificial Hippocampus Networks for Efficient Long-Context Modeling
Yunhao Fang, Weihao Yu, Shu Zhong et al. (+3 more)
Long-sequence modeling faces a fundamental trade-off between the efficiency of compressive fixed-size
memory in RNN-like models and the fidelity of lossless growing
memory in
attention-based
Transformers. Inspired by the Multi-Store Model in cognitive science, we introduce a
memory framework of artificial neural networks. Our method maintains a sliding window of the
Transformer's KV cache as lossless short-term
memory, while a learnable module termed Artificial Hippocampus Network (AHN) recurrently compresses out-of-window information into a fixed-size compact long-term
memory. To validate this framework, we instantiate AHNs using modern RNN-like architectures, including Mamba2, DeltaNet, and GatedDeltaNet to augment open-weight LLMs. We also propose an efficient self-
distillation training method where the base model's all parameters are frozen and only the parameters from AHNs are optimized. For inference, our method sets a default large sliding window size of 32k for
attention, and AHNs activate only when the sequence length exceeds the 32k window, addressing the quadratic-complexity issue of
attention that emerges at that scale. Extensive experiments on long-context
benchmarks LV-Eval and InfiniteBench demonstrate that AHN-augmented models consistently outperform sliding window baselines and achieve performance comparable or even superior to full-
attention models, while substantially reducing computational and
memory requirements. For instance, augmenting the Qwen2.5-3B-Instruct with AHNs reduces inference FLOPs by 40.5% and
memory cache by 74.0%, while improving its average score on LV-Eval (128k sequence length) from 4.41 to 5.88. Code is available at: https://github.com/ByteDance-Seed/AHN.
Read the full abstract ▾
All authors Yunhao Fang, Weihao Yu, Shu Zhong, Qinghao Ye, Xuehan Xiong, Lai Wei
Computation and Language
SWE-IF: Aligning Code Evaluation with Human Preference
Ming Zhong, Xiang Zhou, Ting-Yun Chang et al. (+9 more)
Large
Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural
language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models' code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.
Read the full abstract ▾
All authors Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, Jiao Sun
Computation and Language
Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domain
Yue Li, Ran Tao, Derek Hommel et al. (+4 more)
Text-to-SQL
benchmarks have traditionally only tested simple data access as a translation task of natural
language to SQL queries. But in reality, users tend to ask diverse questions that require more complex responses including data-driven predictions or recommendations. Using the business domain as a motivating example, we introduce CORGI, a new
benchmark that expands text-to-SQL to reflect practical database queries encountered by end users. CORGI is composed of synthetic databases inspired by enterprises such as DoorDash, Airbnb, and Lululemon. It provides questions across four increasingly complicated categories of business queries: descriptive, explanatory, predictive, and recommendational. This challenge calls for causal
reasoning, temporal
forecasting, and strategic recommendation, reflecting multi-level and multi-step agentic intelligence. We find that LLM performance degrades on higher-level questions as question complexity increases. CORGI also introduces and encourages the text-to-SQL community to consider new automatic methods for evaluating open-ended, qualitative responses in data access tasks. Our experiments show that LLMs exhibit an average 33.12% lower success execution rate (SER) on CORGI compared to existing
benchmarks such as BIRD, highlighting the substantially higher complexity of real-world business needs. We release the CORGI
dataset, an evaluation framework, and a submission website to support future research.
Read the full abstract ▾
All authors Yue Li, Ran Tao, Derek Hommel, Yusuf Denizay Dönder, Sungyong Chang, David Mimno, Unso Eun Seo Jo
Computation and Language
Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning
Xue Zhang, Yunlong Liang, Fandong Meng et al. (+5 more)
Large
Reasoning Models (LRMs) have achieved remarkable performance on complex
reasoning tasks by adopting the “think-then-answer” paradigm, which enhances both accuracy and interpretability. However, current LRMs exhibit two critical limitations when processing non-English
languages: (1) They often struggle to maintain input-output
language consistency; (2) They generally perform poorly with wrong
reasoning paths and lower answer accuracy compared to English. These limitations significantly compromise the interpretability of
reasoning processes and degrade the user experience for non-English speakers, hindering the global deployment of LRMs. To address these limitations, we propose M-Thinker, which is trained by the GRPO
algorithm that involves a
Language Consistency (LC) reward and a novel Cross-lingual Thinking
Alignment (CTA) reward. Specifically, the LC reward defines a strict constraint on the
language consistency between the input, thought, and answer. Besides, the CTA reward compares the model's non-English
reasoning paths with its English
reasoning path to transfer its own
reasoning capability from English to non-English
languages. Through an iterative RL procedure, our M-Thinker-1.5B/4B/7B models not only achieve nearly 100%
language consistency and superior performance on two
multilingual benchmarks (MMATH and PolyMath), but also exhibit excellent generalization on out-of-domain
languages.
Read the full abstract ▾
All authors Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Kaiyu Huang, Yufeng Chen, Jinan Xu, Jie Zhou
Computation and Language · Cross-listed from cs.SD
AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
Peize He, Zichen Wen, Yubo Wang et al. (+13 more)
Processing long-form audio is a major challenge for Large Audio
Language models (LALMs). These models struggle with the quadratic cost of
attention ($O(N^2)$) and with modeling long-range temporal dependencies. Existing audio
benchmarks are built mostly from short clips and do not evaluate models in realistic long context settings. To address this gap, we introduce AudioMarathon, a
benchmark designed to evaluate both understanding and inference efficiency on long-form audio. AudioMarathon provides a diverse set of tasks built upon three pillars: long-context audio inputs with durations ranging from 90.0 to 300.0 seconds, which correspond to encoded sequences of 2,250 to 7,500 audio
tokens, respectively, full domain coverage across
speech, sound, and music, and complex
reasoning that requires multi-hop inference. We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows. We also study acceleration techniques and analyze the trade-offs of
token pruning and KV cache eviction. The results show large gaps across current LALMs and highlight the need for better temporal
reasoning and
memory-efficient architectures. We believe AudioMarathon will drive the audio and
multimodal research community to develop more advanced audio understanding models capable of solving complex audio tasks.
Read the full abstract ▾
All authors Peize He, Zichen Wen, Yubo Wang, Yuxuan Wang, Xiaoqian Liu, Jiajie Huang, Zehui Lei, Zhuangcheng Gu, Xiangqi Jin, Jiabing Yang, Kai Li, Zhifei Liu, Weijia Li, Cunxiang Wang, Conghui He, Linfeng Zhang
Computation and Language
On the Convergence of Moral Self-Correction in Large Language Models
Guangliang Liu, Haitao Mao, Bochuan Cao et al. (+4 more)
Large
Language Models (LLMs) are able to improve their responses when instructed to do so, a capability known as self-correction. When instructions provide only a general and abstract goal without specific details about potential issues in the response, LLMs must rely on their internal knowledge to improve response quality, a process referred to as intrinsic self-correction. The empirical success of intrinsic self-correction is evident in various applications, but how and why it is effective remains unknown. Focusing on moral self-correction in LLMs, we reveal a key characteristic of intrinsic self-correction: performance convergence through multi-round interactions; and provide a mechanistic analysis of this convergence behavior. Based on our experimental results and analysis, we uncover the underlying mechanism of convergence: consistently injected self-correction instructions activate moral concepts that reduce model uncertainty, leading to converged performance as the activated moral concepts stabilize over successive rounds. This paper demonstrates the strong potential of moral self-correction by showing that it exhibits a desirable property of converged performance.
Read the full abstract ▾
All authors Guangliang Liu, Haitao Mao, Bochuan Cao, Zhiyu Xue, Xitong Zhang, Rongrong Wang, Kristen Marie Johnson
Computation and Language
Online Rubrics Elicitation from Pairwise Comparisons
MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang et al. (+4 more)
Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work shows that reinforcement learning with rubric-based rewards leads to consistent gains in LLM post-training. Most existing approaches rely on rubrics that remain static over the course of training. Such static rubrics, however, are vulnerable to reward-hacking type behaviors and fail to capture emergent desiderata that arise during training. We introduce Online Rubrics Elicitation (OnlineRubrics), a method that dynamically curates evaluation criteria in an online manner through pairwise comparisons of responses from current and reference policies. This online process enables continuous identification and mitigation of errors as training proceeds. Empirically, this approach yields consistent improvements of up to 8% over training exclusively with static rubrics across AlpacaEval, GPQA, ArenaHard as well as the validation sets of expert questions and rubrics. We qualitatively analyze the elicited criteria and identify prominent themes such as transparency, practicality, organization, and
reasoning.
Read the full abstract ▾
All authors MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, Afra Feyza Akyürek
Computation and Language
Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models
Jonggeun Lee, Woojung Song, Jongwook Han et al. (+2 more)
Small
language models (SLMs) enable scalable tool-augmented multi-
agent systems where multiple SLMs handle subtasks orchestrated by a powerful coordinator. However, they struggle with tool-use tasks, particularly in selecting appropriate tools and identifying correct parameters. A common failure mode is schema misalignment: models hallucinate plausible tool names that are absent from the provided tool schema, due to different naming conventions internalized during pretraining. Rather than training models to adapt to unfamiliar schemas, we propose adapting schemas to align with models' pretrained knowledge. We introduce PA-Tool (Pretraining-Aligned Tool Schema Generation), a training-free method that leverages peakedness, a signal used in contamination detection that indicates pretraining familiarity, to rename tool components. By generating multiple candidates and selecting the candidate with the highest peakedness, PA-Tool identifies pretraining-aligned naming patterns. Experiments on MetaTool and RoTBench show improvements of up to 17%, with schema misalignment errors reduced by 80%. PA-Tool enables small models to substantially improve tool-use accuracy without retraining, showing that schema-level interventions can unlock the tool-use potential of resource-efficient models. Our code is available at https://github.com/holi-lab/PA-Tool.
Read the full abstract ▾
All authors Jonggeun Lee, Woojung Song, Jongwook Han, Haesung Pyun, Yohan Jo
Computation and Language
LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation
Joseph Enguehard, Morgane Van Ermengem, Kate Atkinson et al. (+8 more)
Evaluating large
language model (LLM) outputs in the legal domain presents unique challenges due to the complex and nuanced nature of legal analysis. Current evaluation approaches either depend on reference data, which is costly to produce, or use standardized assessment methods, both of which have significant limitations for legal applications. Although LLM-as-a-Judge has emerged as a promising evaluation technique, its reliability and effectiveness in legal contexts depend heavily on evaluation processes unique to the legal industry and how trustworthy the evaluation appears to the human legal expert. This is where existing evaluation methods currently fail and exhibit considerable variability. This paper aims to close the gap: a) we break down lengthy responses into 'Legal Data Points' (LDPs), self-contained units of information, and introduce a novel, reference-free evaluation methodology that reflects how lawyers evaluate legal answers; b) we demonstrate that our method outperforms a variety of baselines on both our proprietary
dataset and an open-source
dataset (LegalBench); c) we show how our method correlates more closely with human expert evaluations and helps improve inter-annotator agreement; and finally d) we open source our Legal Data Points for a subset of LegalBench used in our experiments, allowing the research community to replicate our results and advance research in this vital area of LLM evaluation on legal question-answering.
Read the full abstract ▾
All authors Joseph Enguehard, Morgane Van Ermengem, Kate Atkinson, Sujeong Cha, Arijit Ghosh Chowdhury, Prashanth Kallur Ramaswamy, Jeremy Roghair, Hannah R Marlowe, Carina Suzana Negreanu, Kitty Boxall, Diana Mincu
Computation and Language
Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense
Leitian Tao, Ilia Kulikov, Swarnadeep Saha et al. (+5 more)
Post-training for
reasoning of large
language models (LLMs) increasingly relies on verifiable rewards: deterministic checkers that provide 0-1 correctness signals. While reliable, such binary feedback is brittle–many tasks admit partially correct or alternative answers that verifiers under-credit, and the resulting all-or-nothing supervision limits learning. Reward models offer richer, continuous feedback, which can serve as a complementary supervisory signal to verifiers. We introduce HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way. HERO employs stratified normalization to bound reward-model scores within verifier-defined groups, preserving correctness while refining quality distinctions, and variance-aware weighting to emphasize challenging
prompts where dense signals matter most. Across diverse mathematical
reasoning benchmarks, HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks. Our results show that hybrid reward design retains the stability of verifiers while leveraging the nuance of reward models to advance
reasoning.
Read the full abstract ▾
All authors Leitian Tao, Ilia Kulikov, Swarnadeep Saha, Tianlu Wang, Jing Xu, Sharon Li, Jason E Weston, Ping Yu
Computation and Language
Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Experts
Christos Ziakas, Nicholas Loo, Nishita Jain et al. (+1 more)
Automated red-teaming has emerged as a scalable approach for
auditing Large
Language Models (LLMs) prior to deployment, yet existing approaches lack mechanisms to efficiently adapt to model-specific vulnerabilities at inference. We introduce Red-Bandit, a red-teaming framework that adapts online to identify and exploit model failure modes under distinct attack styles (e.g., manipulation, slang). Red-Bandit post-trains a set of parameter-efficient LoRA experts, each specialized for a particular attack style, using reinforcement learning that rewards the generation of unsafe
prompts via a rule-based safety model. At inference, a multi-armed bandit policy dynamically selects among these attack-style experts based on the target model's response safety, balancing exploration and exploitation. Red-Bandit achieves state-of-the-art results on AdvBench under sufficient exploration (ASR@10), while producing more human-readable
prompts (lower perplexity). Moreover, Red-Bandit's bandit policy serves as a diagnostic tool for uncovering model-specific vulnerabilities by indicating which attack styles most effectively elicit unsafe behaviors.
Read the full abstract ▾
All authors Christos Ziakas, Nicholas Loo, Nishita Jain, Alessandra Russo
Computation and Language
When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation
Xunyi Jiang, Dingyi Chang, Julian McAuley et al. (+1 more)
The rapid
evolution of large
language models (LLMs) and the real world has outpaced the static nature of widely used evaluation
benchmarks, raising concerns about their reliability for evaluating LLM factuality. While substantial works continue to rely on the popular but old
benchmarks, their temporal misalignment with real-world facts and modern LLMs, and their effects on LLM factuality evaluation remain underexplored. Therefore, in this work, we present a systematic investigation of this issue by examining five popular factuality
benchmarks and eight LLMs released across different years. An up-to-date fact
retrieval pipeline and three metrics are tailored to quantify
benchmark aging and its impact on LLM factuality evaluation. Experimental results and analysis illustrate that a considerable portion of samples in the widely used factuality
benchmarks are outdated, leading to unreliable assessments of LLM factuality. We hope our work can provide a testbed to assess the reliability of a
benchmark for LLM factuality evaluation and inspire more research on the
benchmark aging issue. Codes are available in https://github.com/JiangXunyi/BenchAge.
Read the full abstract ▾
All authors Xunyi Jiang, Dingyi Chang, Julian McAuley, Xin Xu