Computation and Language
Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents
Reid T. Johnson, Michelle D. Pain and Jordan D. West
We present Natural
Language Tools (NLT), a framework that replaces programmatic JSON tool calling in large
language models (LLMs) with natural
language outputs. By decoupling tool selection from response generation, NLT eliminates task interference and format constraints that degrade tool call performance. When evaluated across 10 models and 6,400 trials spanning customer service and mental health domains, NLT improves tool calling accuracy by 18.4 percentage points while reducing output variance by 70%. Open-weight models see the largest gains, surpassing flagship closed-weight alternatives, with implications for model training in both reinforcement learning and supervised fine-tuning stages. These improvements persist under
prompt perturbations and extend tool-calling capabilities to models lacking native support.
Read the full abstract ▾
Computation and Language
WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Models
Rui Wang, Ce Zhang, Jun-Yu Ma et al. (+10 more)
The hallmark of Deep Research
agents lies in compositional
reasoning, the capacity to aggregate distributed, heterogeneous information into coherent logical insights. However, current agentic systems are often
retrieval-heavy but
reasoning-light, where success is predominantly determined by simple entity-seeking rather than the multi-step aggregation of scattered evidence. To address this, we propose a data synthesis pipeline WebAggregator, designed to shift the agentic paradigm from
retrieval-centric to compositional aggregation. Our approach first employs Proactive Explorer to collect interconnected knowledge, then Compositional Logic Proposer to weave knowledge into complex questions using over 12 composition guidelines derived from a rigorous deconstruction of the Deep Research problem setting. By leveraging 10K verifiable QA pairs grounded on 50K websites, we curate a high-quality SFT
dataset via rejection sampling. Fine-tuning on this corpus fundamentally transforms
agent behavior, fostering deliberate composition
reasoning and reduced tool redundancy. The resulting WebAggregator-32B surpasses GPT-4.1 and matches Claude-3.7-Sonnet on
GAIA, WebWalkerQA, and XBench. To address the lack of
benchmarks that emphasize both
reasoning and
retrieval, we introduce the WebAggregatorQA testbed, which reveals that even with perfect
retrieval, top-tier models still underperformed. These results demonstrate that compositional
reasoning, not
retrieval, is the true performance ceiling for next-generation research
agents.
Read the full abstract ▾
All authors Rui Wang, Ce Zhang, Jun-Yu Ma, Jianshu Zhang, Hongru Wang, Yi Chen, Boyang Xue, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu, Kam-Fai Wong
Computation and Language
Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following
Qingyu Ren, Qianyu He, Powei Chang et al. (+5 more)
Language models often struggle to follow multi-constraint instructions that are crucial for real-world applications. Existing reinforcement learning (RL) approaches suffer from dependency on external supervision and sparse reward signals from multi-constraint tasks. We propose a label-free self-supervised RL framework that eliminates dependency on external supervision by deriving reward signals directly from instructions and generating pseudo-labels for reward model training. Our approach introduces constraint decomposition strategies and efficient constraint-wise binary classification to address sparse reward challenges while maintaining computational efficiency. Experiments show that our approach generalizes well, achieving strong improvements across 3 in-domain and 5 out-of-domain
datasets, including challenging agentic and multi-turn instruction following. The data and code are publicly available at https://github.com/Rainier-rq/verl-if
Read the full abstract ▾
All authors Qingyu Ren, Qianyu He, Powei Chang, Jie Zeng, Zeye Sun, Fei Yu, Jiaqing Liang, Yanghua Xiao
Computation and Language · Cross-listed from cs.AI
IMAGINE: Integrating Multi-Agent System into One Model for Complex Reasoning and Planning
Xikai Zhang, Bo Wang, Likang Xiao et al. (+4 more)
Although large
language models (LLMs) have made significant strides across various tasks, they still face significant challenges in complex
reasoning and planning. For example, even with carefully designed
prompts and prior information explicitly provided, GPT-4o achieves only a 7% Final Pass Rate on the TravelPlanner
dataset in the sole-planning mode. Similarly, even in the thinking mode, Qwen3-8B-Instruct and DeepSeek-R1-671B, only achieve Final Pass Rates of 5.9% and 40%, respectively. Although well-organized Multi-
Agent Systems (MAS) can offer improved collective
reasoning, they often suffer from high
reasoning costs due to multi-round internal interactions, long per-response latency, and difficulties in end-to-end training. To address these challenges, we propose a general and scalable framework called IMAGINE, short for Integrating Multi-
Agent System into One Model. This framework not only integrates the
reasoning and planning capabilities of MAS into a single, compact model, but also significantly surpass the capabilities of the MAS through a simple end-to-end training. Through this pipeline, a single small-scale model is not only able to acquire the structured
reasoning and planning capabilities of a well-organized MAS but can also significantly outperform it. Experimental results demonstrate that, when using Qwen3-8B-Instruct as the base model and training it with our method, the model achieves an 82.7% Final Pass Rate on the TravelPlanner
benchmark, far exceeding the 40% of DeepSeek-R1-671B, while maintaining a much smaller model size.
Read the full abstract ▾
All authors Xikai Zhang, Bo Wang, Likang Xiao, Yongzhi Li, Quan Chen, Wenjun Wu, Liu Liu
Computation and Language
MedTrust-RAG: Evidence Verification and Trust Alignment for Biomedical Question Answering
Yingpeng Ning, Yuanyuan Sun, Ling Luo et al. (+3 more)
Biomedical question answering (QA) requires accurate interpretation of complex medical knowledge. Large
language models (LLMs) have shown promising capabilities in this domain, with
retrieval-augmented generation (RAG) systems enhancing performance by incorporating external medical literature. However, RAG-based approaches in biomedical QA suffer from
hallucinations due to post-
retrieval noise and insufficient verification of retrieved evidence, undermining response reliability. We propose MedTrust-Guided Iterative RAG, a framework designed to enhance factual consistency and mitigate
hallucinations in medical QA. Our method introduces three key innovations. First, it enforces citation-aware
reasoning by requiring all generated content to be explicitly grounded in retrieved medical documents, with structured Negative Knowledge Assertions used when evidence is insufficient. Second, it employs an iterative
retrieval-verification process, where a verification
agent assesses evidence adequacy and refines queries through Medical Gap Analysis until reliable information is obtained. Third, it integrates the MedTrust-Align Module (MTAM) that combines verified positive examples with
hallucination-aware negative samples, leveraging Direct Preference Optimization to reinforce citation-grounded
reasoning while penalizing
hallucination-prone response patterns.
Read the full abstract ▾
All authors Yingpeng Ning, Yuanyuan Sun, Ling Luo, Yanhua Wang, Yuchen Pan, Hongfei Lin
Computation and Language
YNTP-100: A Benchmark for Your Next Token Prediction with 100 People
Shiyao Ding and Takayuki Ito
Large
language models (LLMs) trained for general next-
token prediction often fail to generate responses that reflect how specific individuals communicate. Progress on personalized
alignment is further limited by the difficulty of collecting real-world personal communication data due to
privacy constraints. We propose Your Next
Token Prediction (YNTP), a task that formulates personalized response generation as
token-level prediction conditioned on user interaction history. We introduce YNTP-100, a
benchmark built from
multilingual multi-day human–
agent conversations with 100 people, enabling systematic evaluation of user-specific response behavior. We evaluate external (parameter-preserving) and internal (parameter-updating)
alignment methods using metrics of substance similarity and stylistic consistency. The
dataset and results are publicly available at: https://github.com/AnonymousHub4Submissions/YNTP100.
Read the full abstract ▾
Computation and Language
Suicidal Comment Tree Dataset: Enhancing Risk Assessment and Prediction Through Contextual Analysis
Jun Li and Qun Zhao
Suicide remains a critical global public health issue. While previous studies have provided valuable insights into detecting suicidal expressions in individual social media posts, limited
attention has been paid to the analysis of longitudinal, sequential comment trees for predicting a user's evolving suicidal risk. Users, however, often reveal their intentions through historical posts and interactive comments over time. This study addresses this gap by investigating how the information in comment trees affects both the discrimination and prediction of users' suicidal risk levels. We constructed a high-quality annotated
dataset, sourced from Reddit, which incorporates users' posting history and comments, using a refined four-label annotation framework based on the Columbia Suicide Severity Rating Scale (C-SSRS). Statistical analysis of the
dataset, along with experimental results from Large
Language Models (LLMs) experiments, demonstrates that incorporating comment trees data significantly enhances the discrimination and prediction of user suicidal risk levels. This research offers a novel insight to enhancing the detection accuracy of at-risk individuals, thereby providing a valuable foundation for early suicide intervention strategies.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
Andrew Zhao, Reshmi Ghosh, Vitor Carvalho et al. (+4 more)
Large
language model (LLM) systems increasingly power everyday AI applications such as chatbots, computer-use assistants, and autonomous
robots, where performance often depends on manually well-crafted
prompts. LLM-based
prompt optimizers reduce that effort by iteratively refining
prompts from scored feedback, yet the security of this optimization stage remains underexamined. We present the first systematic analysis of poisoning risks in LLM-based
prompt optimization. Using HarmBench, we find systems are substantially more vulnerable to manipulated feedback than to query poisoning alone: feedback-based attacks raise attack success rate (ASR) by up to {\Delta}ASR = 0.48. We introduce a simple fake reward attack that requires no access to the reward model and significantly increases vulnerability. We also propose a lightweight highlighting defense that reduces the fake reward {\Delta}ASR from 0.23 to 0.07 without degrading utility. These results establish
prompt optimization pipelines as a first-class attack surface and motivate stronger safeguards for feedback channels and optimization frameworks.
Read the full abstract ▾
All authors Andrew Zhao, Reshmi Ghosh, Vitor Carvalho, Emily Lawton, Keegan Hines, Gao Huang, Jack W. Stokes
Computation and Language
PluriHopRAG: Exhaustive, Recall-Sensitive QA Through Corpus-Specific Document Structure Learning
Mykolas Sveistrys and Richard Kunert
Retrieval-Augmented Generation (RAG) has been used in question answering (QA) systems to improve performance when relevant information is in one (single-hop) or multiple (multi-hop) passages. However, many real life scenarios (e.g. dealing with financial, legal, medical reports) require checking all documents for relevant information without a clear stopping condition. We term these pluri-hop questions, and formalize them by 3 conditions - recall sensitivity, exhaustiveness, and exactness. To study this setting, we introduce PluriHopWIND, a
multilingual diagnostic
benchmark of 48 pluri-hop questions over 191 real wind-industry reports, with high repetitiveness to reflect the challenge of distractors in real-world
datasets. Naive, graph-based, and
multimodal RAG methods only reach up to 40% statement-wise F1 on PluriHopWIND. Motivated by this, we propose PluriHopRAG, which learns from synthetic examples to decompose queries according to corpus-specific document structure, and employs a cross-encoder filter at the document level to minimize costly LLM
reasoning. We test PluriHopRAG on PluriHopWIND and the Loong
benchmark built on financial, legal and scientific reports. On PluriHopWIND, our method shows 18-52% F1 score improvement across base LLMs, while on Loong, we show 33% improvement over long-context
reasoning and 52% improvement over naive RAG.
Read the full abstract ▾
Computation and Language
From Binary to Bilingual: How the National Weather Service is Using Artificial Intelligence to Develop a Comprehensive Translation Program
Joseph E. Trujillo-Falcon, Monica L. Bozeman, Liam E. Llewellyn et al. (+8 more)
To advance a Weather-Ready Nation, the National Weather Service (NWS) is developing a systematic translation program to better serve the 68.8 million people in the U.S. who do not speak English at home. This article outlines the foundation of an automated translation tool for NWS products, powered by artificial intelligence. The NWS has partnered with LILT, whose patented training process enables large
language models (LLMs) to adapt neural machine translation (NMT) tools for weather terminology and messaging. Designed for scalability across Weather Forecast Offices (WFOs) and National Centers, the system is currently being developed in Spanish, Simplified Chinese, Vietnamese, and other widely spoken non-English
languages. Rooted in best practices for
multilingual risk communication, the system provides accurate, timely, and culturally relevant translations, significantly reducing manual translation time and easing operational workloads across the NWS. To guide the distribution of these products, GIS mapping was used to identify
language needs across different NWS regions, helping prioritize resources for the communities that need them most. We also integrated ethical AI practices throughout the program's design, ensuring that transparency,
fairness, and human oversight guide how automated translations are created, evaluated, and shared with the public. This work has culminated into a website featuring experimental
multilingual NWS products, including translated warnings, 7-day forecasts, and educational campaigns, bringing the country one step closer to a national warning system that reaches all Americans.
Read the full abstract ▾
All authors Joseph E. Trujillo-Falcon, Monica L. Bozeman, Liam E. Llewellyn, Samuel T. Halvorson, Meryl Mizell, Stuti Deshpande, Bob Manning, Chris Rohrbach, Ian Blaylock, Angel Montanez, Todd Fagin
Computation and Language
Understanding the Ability of LLMs to Handle Character-Level Perturbation
Anyuan Zhuo, Xuefei Ning, Ningyuan Li et al. (+3 more)
This work investigates the resilience of contemporary large
language models (LLMs) against frequent character-level perturbations. We examine three types of character-level perturbations including introducing numerous typos within words, shuffling the characters in each word, and inserting a large number of invisible characters into the text. Surprisingly, even under severe perturbation, such as shuffling nearly all words character-wise to produce text that is almost unreadable to humans, or inserting invisible characters which are several times more than the visible ones as noise, many LLMs still maintain notable performance. We explore the underlying causes of this robustness and find that LLMs exhibit remarkable resilience to chaotic
segmentation and fragmented tokenization. Furthermore, we examine the mechanisms by which LLMs remove perturbations to correctly comprehend text, including both implicit and explicit mechanisms for character-level perturbation. We hope that our findings on the low-level robustness of LLMs will unveil their inherent architectural strengths, reveal the potential risks of their misuse, and inform the reliable deployment of LLMs across diverse application scenarios.
Read the full abstract ▾
All authors Anyuan Zhuo, Xuefei Ning, Ningyuan Li, Jingyi Zhu, Yu Wang, Pinyan Lu
Computation and Language · Cross-listed from cs.AI
AI for Service: Proactive Assistance with AI Glasses
Zichen Wen, Yiyu Wang, Chenfei Liao et al. (+10 more)
In an era where AI is evolving from a passive tool into an active and adaptive companion, we introduce AI for Service (AI4Service), a new paradigm that enables proactive and real-time assistance in daily life. Existing AI services remain largely reactive, responding only to explicit user commands. We argue that a truly intelligent and helpful assistant should be capable of anticipating user needs and taking actions proactively when appropriate. To realize this
vision, we propose Alpha-Service, a unified framework that addresses two fundamental challenges: Know When to intervene by detecting service opportunities from
egocentric video streams, and Know How to provide both generalized and personalized services. Inspired by the von Neumann computer architecture and based on AI glasses, Alpha-Service consists of five key components: an Input Unit for perception, a Central Processing Unit for task scheduling, an Arithmetic Logic Unit for tool utilization, a
Memory Unit for long-term personalization, and an Output Unit for natural human interaction. As an initial exploration, we implement Alpha-Service through a multi-
agent system deployed on AI glasses. Case studies, including a real-time Blackjack advisor, a museum tour guide, and a shopping fit assistant, demonstrate its ability to seamlessly perceive the environment, infer user intent, and provide timely and useful assistance without explicit
prompts.
Read the full abstract ▾
All authors Zichen Wen, Yiyu Wang, Chenfei Liao, Boxue Yang, Junxian Li, Weifeng Liu, Haocong He, Bolong Feng, Xuyang Liu, Yuanhuiyi Lyu, Xu Zheng, Xuming Hu, Linfeng Zhang
Computation and Language
CURE: Confidence-driven Unified Reasoning Ensemble Framework for Medical Question Answering
Ziad Elshaer and Essam A. Rashed
High-performing medical Large
Language Models (LLMs) typically require extensive fine-tuning with substantial computational resources, limiting accessibility for resource-constrained healthcare institutions. This study introduces a confidence-driven multi-model framework that leverages model diversity to enhance medical question answering without fine-tuning. Our framework employs a two-stage architecture: a confidence detection module assesses the primary model's certainty, and an adaptive routing mechanism directs low-confidence queries to Helper models with complementary knowledge for collaborative
reasoning. We evaluate our approach using Qwen3-30B-A3B-Instruct, Phi-4 14B, and Gemma 2 12B across three medical
benchmarks; MedQA, MedMCQA, and PubMedQA. Result demonstrate that our framework achieves competitive performance, with particularly strong results in PubMedQA (95.0%) and MedMCQA (78.0%). Ablation studies confirm that confidence-aware routing combined with multi-model collaboration substantially outperforms single-model approaches and uniform
reasoning strategies. This work establishes that strategic model collaboration offers a practical, computationally efficient pathway to improve medical AI systems, with significant implications for democratizing access to advanced medical AI in resource-limited settings.
Read the full abstract ▾
Computation and Language
Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contexts
Perapard Ngokpol, Kun Kerdthaisong, Pasin Buakhaw et al. (+4 more)
Large
language models (LLMs) are increasingly used as role-playing
agents, yet their capacity to faithfully and consistently portray version-specific characters – for example, superheroes across comic and cinematic universes – remains underexplored. Superhero canons such as Marvel and DC provide a rich testbed: decades of storytelling yield multiple incarnations of the same character with distinct histories, values, and moral codes. To study this problem, we introduce Beyond One World, a
benchmark for character-grounded roleplay spanning 30 iconic heroes and 90 canon-specific versions. The
benchmark comprises two tasks: (i) Canon Events, which probes factual recall of pivotal life stages, and (ii) Moral Dilemmas, which confronts models with ethically charged scenarios. We score responses for canonical accuracy and
reasoning fidelity under a framework that separates internal deliberation ("thinking") from outward decisions ("acting"). We further propose Think-Act Matching, a metric that quantifies
alignment between reasons and actions and serves as a proxy for model trustworthiness. Experiments across
reasoning- and non-
reasoning-oriented models yield three findings: (1) chain-of-thought prompting improves narrative coherence in weaker models but can reduce canonical accuracy in stronger ones; (2) cross-version generalization within a character remains a major obstacle; and (3) models often excel at either thinking or acting, but rarely both. Beyond One World exposes critical gaps in multiversal consistency and
reasoning alignment, offering a challenging evaluation for role-playing LLMs.
Read the full abstract ▾
All authors Perapard Ngokpol, Kun Kerdthaisong, Pasin Buakhaw, Pitikorn Khlaisamniang, Supasate Vorathammathorn, Piyalitt Ittichaiwong, Nutchanon Yongsatianchot
Computation and Language
A Robust Classification Method using Hybrid Word Embedding for Early Diagnosis of Alzheimer's Disease
Yangyang Li
Early detection of Alzheimer's Disease (AD) is greatly beneficial to AD patients, leading to early treatments that lessen symptoms and alleviating financial burden of health care. As one of the leading signs of AD,
language capability changes can be used for early diagnosis of AD. In this paper, I develop a robust classification method using hybrid word
embedding and fine-tuned hyperparameters to achieve state-of-the-art accuracy in the early detection of AD. Specifically, we create a hybrid word
embedding based on word vectors from Doc2Vec and ELMo to obtain perplexity scores of the sentences. The scores identify whether a sentence is fluent or not and capture semantic context of the sentences. I enrich the word
embedding by adding linguistic features to analyze syntax and semantics. Further, we input an embedded feature vector into logistic regression and fine tune hyperparameters throughout the pipeline. By tuning hyperparameters of the machine learning pipeline (e.g., model regularization parameter, learning rate and vector size of Doc2Vec, and vector size of ELMo), I achieve 91% classification accuracy and an Area Under the Curve (AUC) of 97% in distinguishing early AD from healthy subjects. Based on my knowledge, my model with 91% accuracy and 97% AUC outperforms the best existing NLP model for AD diagnosis with an accuracy of 88% [32]. I study the model stability through repeated experiments and find that the model is stable even though the training data is split randomly (standard deviation of accuracy = 0.0403; standard deviation of AUC = 0.0174). This affirms our proposed method is accurate and stable. This model can be used as a large-scale screening method for AD, as well as a complementary examination for doctors to detect AD.
Read the full abstract ▾
Computation and Language
Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL
Marwa Abdulhai, Ryan Cheng, Aryansh Shrivastava et al. (+3 more)
Large
Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare. However, their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns. The unpredictable nature of LLM behavior, combined with insufficient safeguards against
hallucination, misinformation, and user manipulation, makes their misuse a serious, real-world risk. In this paper, we investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception. We evaluate deception across four distinct dialogue scenarios, using five established deception detection metrics and our proposed metric. Our findings reveal this novel deception measure correlates more closely with human judgments than any existing metrics we test. Additionally, our benchmarking of eight state-of-the-art models indicates that LLMs naturally exhibit deceptive behavior in approximately 26% of dialogue turns, even when prompted with seemingly benign objectives. When prompted to deceive, LLMs are capable of increasing deceptiveness by as much as 31% relative to baselines. Unexpectedly, models trained with RLHF, the predominant approach for ensuring the safety of widely-deployed LLMs, still exhibit deception at a rate of 43% on average. Given that deception in dialogue is a behavior that develops over an interaction history, its effective evaluation and mitigation necessitates moving beyond single-utterance analyses. We introduce a multi-turn reinforcement learning methodology to fine-tune LLMs to reduce deceptive behaviors, leading to a 77.6% reduction compared to other instruction-tuned models.
Read the full abstract ▾
All authors Marwa Abdulhai, Ryan Cheng, Aryansh Shrivastava, Natasha Jaques, Yarin Gal, Sergey Levine
Computation and Language · Cross-listed from cs.AI
Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studies
Mason Nakamura, Abhinav Kumar, Saaduddin Mahmud et al. (+3 more)
A multi-
agent system (MAS) powered by large
language models (LLMs) can automate tedious user tasks such as meeting scheduling that requires inter-
agent collaboration. LLMs enable nuanced protocols that account for unstructured private data, user constraints, and preferences. However, this design introduces new risks, including misalignment and attacks by malicious parties that compromise
agents or steal user data. In this paper, we propose the Terrarium framework for fine-grained study on safety,
privacy, and security in LLM-based MAS. We repurpose the blackboard design, an early approach in multi-
agent systems, to create a modular, configurable testbed for multi-
agent collaboration. We identify key attack vectors such as misalignment, malicious
agents, compromised communication, and data poisoning. We implement three collaborative MAS scenarios with four representative attacks to demonstrate the framework's flexibility. By providing tools to rapidly prototype, evaluate, and iterate on defenses and designs, Terrarium aims to accelerate progress toward trustworthy multi-
agent systems.
Read the full abstract ▾
All authors Mason Nakamura, Abhinav Kumar, Saaduddin Mahmud, Sahar Abdelnabi, Shlomo Zilberstein, Eugene Bagdasarian
Computation and Language
MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking
Sathyanarayanan Ramamoorthy, Vishwa Shah, Simran Khanuja et al. (+5 more)
This paper introduces MERLIN, a novel testbed system for the task of
Multilingual Multimodal Entity Linking. The created
dataset includes BBC news article titles, paired with corresponding images, in five
languages: Hindi, Japanese, Indonesian, Vietnamese, and Tamil, featuring over 7,000 named entity mentions linked to 2,500 unique Wikidata entities. We also include several
benchmarks using
multilingual and
multimodal entity linking methods exploring different
language models like LLaMa-2 and Aya-23. Our findings indicate that incorporating visual data improves the accuracy of entity linking, especially for entities where the textual context is ambiguous or insufficient, and particularly for models that do not have strong
multilingual abilities. For the work, the
dataset, methods are available here at https://github.com/rsathya4802/merlin
Read the full abstract ▾
All authors Sathyanarayanan Ramamoorthy, Vishwa Shah, Simran Khanuja, Zaid Sheikh, Shan Jie, Ann Chia, Shearman Chua, Graham Neubig
Computation and Language
Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin et al. (+2 more)
Mathematical
reasoning remains one of the most challenging domains for large
language models (LLMs), requiring not only linguistic understanding but also structured logical deduction and numerical precision. While recent LLMs demonstrate strong general-purpose
reasoning abilities, their mathematical competence across diverse
languages remains underexplored. Existing
benchmarks primarily focus on English or a narrow subset of high-resource
languages, leaving significant gaps in assessing
multilingual and cross-lingual mathematical
reasoning. To address this, we introduce MATHMIST, a parallel
multilingual benchmark for mathematical problem solving and
reasoning. MATHMIST encompasses 2,890 parallel Bangla-English gold standard artifacts, totaling approximately 30K aligned question–answer pairs across thirteen
languages, representing an extensive coverage of high-, medium-, and low-resource linguistic settings. The
dataset captures linguistic variety, multiple types of problem settings, and solution synthesizing capabilities. We systematically evaluate a diverse suite of models, including open-source small and medium LLMs, proprietary systems, and
multilingual-
reasoning-focused models under zero-shot, chain-of-thought (CoT), perturbated
reasoning, and code-switched
reasoning paradigms. Our results reveal persistent deficiencies in LLMs' ability to perform consistent and interpretable mathematical
reasoning across
languages, with pronounced degradation in low-resource settings. All the codes and data are available at GitHub: https://github.com/mahbubhimel/MathMist
Read the full abstract ▾
All authors Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
Computation and Language
Constraint-Driven Small Language Models Based on Agent and OpenAlex Knowledge Graph: Mining Conceptual Pathways and Discovering Innovation Points in Academic Papers
Ziye Xia and Sergei S. Ospichev
In recent years, the rapid increase in academic publications across various fields has posed severe challenges for academic paper analysis: scientists struggle to timely and comprehensively track the latest research findings and methodologies. Key concept extraction has proven to be an effective analytical paradigm, and its automation has been achieved with the widespread application of
language models in industrial and scientific domains. However, existing paper databases are mostly limited to similarity matching and basic classification of key concepts, failing to deeply explore the relational networks between concepts. This paper is based on the OpenAlex opensource knowledge graph. By analyzing nearly 8,000 open-source paper data from Novosibirsk State University, we discovered a strong correlation between the distribution patterns of paper key concept paths and both innovation points and rare paths. We propose a
prompt engineering-based key concept path analysis method. This method leverages small
language models to achieve precise key concept extraction and innovation point identification, and constructs an
agent based on a knowledge graph constraint mechanism to enhance analysis accuracy. Through fine-tuning of the Qwen and DeepSeek models, we achieved significant improvements in accuracy, with the models publicly available on the Hugging Face platform.
Read the full abstract ▾
Computation and Language
Rethinking Schema Linking: A Context-Aware Bidirectional Retrieval Approach for Text-to-SQL
Md Mahadi Hasan Nahid, Davood Rafiei, Weiwei Zhang et al. (+1 more)
Schema linking – the process of aligning natural
language questions with database schema elements – is a critical yet underexplored component of Text-to-SQL systems. While recent methods have focused primarily on improving SQL generation, they often neglect the
retrieval of relevant schema elements, which can lead to
hallucinations and execution failures. In this work, we propose a context-aware bidirectional schema
retrieval framework that treats schema linking as a standalone problem. Our approach combines two complementary strategies: table-first
retrieval followed by column selection, and column-first
retrieval followed by table selection. It is further augmented with techniques such as question decomposition, keyword extraction, and keyphrase extraction. Through comprehensive evaluations on challenging
benchmarks such as BIRD and Spider, we demonstrate that our method significantly improves schema recall while reducing false positives. Moreover, SQL generation using our retrieved schema consistently outperforms full-schema baselines and closely approaches oracle performance, all without requiring query refinement. Notably, our method narrows the performance gap between full and perfect schema settings by 50%. Our findings highlight schema linking as a powerful lever for enhancing Text-to-SQL accuracy and efficiency.
Read the full abstract ▾
All authors Md Mahadi Hasan Nahid, Davood Rafiei, Weiwei Zhang, Yong Zhang
Computation and Language
PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering
Md Mahadi Hasan Nahid and Davood Rafiei
Retrieval plays a central role in multi-hop question answering (QA), where answering complex questions requires gathering multiple pieces of evidence. We propose PRISM, an agentic
retrieval framework that leverages large
language models (LLMs) in a structured loop to retrieve relevant evidence with high precision and recall. PRISM decomposes
retrieval into three specialized
agents: a Question Analyzer that breaks complex queries into sub-questions, a Selector that identifies the most relevant context for each sub-question (focusing on precision), and an Adder that brings in any missing evidence (focusing on recall). The iterative interaction between the Selector and Adder produces a compact yet comprehensive evidence set, avoiding both brittle error propagation and noisy context accumulation. It achieves higher
retrieval accuracy while filtering out distracting content, enabling downstream QA models to surpass full-context answer accuracy while relying on significantly less irrelevant information. Experiments on four challenging multi-hop QA
benchmarks, including HotpotQA, 2WikiMultiHopQA, MuSiQue, and MultiHopRAG, demonstrate that our approach consistently outperforms strong baselines.
Read the full abstract ▾
Computation and Language
Qwen3Guard Technical Report
Haiquan Zhao, Chenhan Yuan, Fei Huang et al. (+40 more)
As large
language models (LLMs) become more capable and widely used, ensuring the safety of their outputs is increasingly critical. Existing guardrail models, though useful in static evaluation settings, face two major limitations in real-world applications: (1) they typically output only binary "safe/unsafe" labels, which can be interpreted inconsistently across diverse safety policies, rendering them incapable of accommodating varying safety tolerances across domains; and (2) they require complete model outputs before performing safety checks, making them fundamentally incompatible with streaming LLM inference, thereby preventing timely intervention during generation and increasing exposure to harmful partial outputs. To address these challenges, we present Qwen3Guard, a series of
multilingual safety guardrail models with two specialized variants: Generative Qwen3Guard, which casts safety classification as an instruction-following task to enable fine-grained tri-class judgments (safe, controversial, unsafe); and Stream Qwen3Guard, which introduces a
token-level classification head for real-time safety monitoring during incremental text generation. Both variants are available in three sizes (0.6B, 4B, and 8B parameters) and support up to 119
languages and dialects, providing comprehensive, scalable, and low-latency safety moderation for global LLM deployments. Evaluated across English, Chinese, and
multilingual benchmarks, Qwen3Guard achieves state-of-the-art performance in both
prompt and response safety classification. All models are released under the Apache 2.0 license for public use.
Read the full abstract ▾
All authors Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, Baosong Yang, Chen Cheng, Jialong Tang, Jiandong Jiang, Jianwei Zhang, Jijie Xu, Ming Yan, Minmin Sun, Pei Zhang, Pengjun Xie, Qiaoyu Tang, Qin Zhu, Rong Zhang, Shibin Wu, Shuo Zhang, Tao He, Tianyi Tang, Tingyu Xia, Wei Liao, Weizhou Shen, Wenbiao Yin, Wenmeng Zhou, Wenyuan Yu, Xiaobin Wang, Xiaodong Deng, Xiaodong Xu, Xinyu Zhang, Yang Liu, Yeqiu Li, Yi Zhang, Yong Jiang, Yu Wan, Yuxin Zhou
Computation and Language
Retrofitting Small Multilingual Models for Retrieval: Matching 7B Performance with 300M Parameters
Lifu Tu, Yingbo Zhou and Semih Yavuz
Training effective
multilingual embedding models presents unique challenges due to the diversity of
languages and task objectives. Although small
multilingual models (<1 B parameters) perform well on
multilingual tasks generally, they consistently lag behind larger models (>1 B) in the most prevalent use case:
retrieval. This raises a critical question: Can smaller models be retrofitted specifically for
retrieval tasks to enhance their performance? In this work, we investigate key factors that influence the effectiveness of
multilingual embeddings, focusing on training data scale, negative sampling strategies, and data diversity. We find that while increasing the scale of training data yields initial performance gains, these improvements quickly plateau - indicating diminishing returns. Incorporating hard negatives proves essential for consistently improving
retrieval accuracy. Furthermore, our analysis reveals that task diversity in the training data contributes more significantly to performance than
language diversity alone. As a result, we develop a compact (approximately 300M)
multilingual model that achieves
retrieval performance comparable to or even surpassing current strong 7B models.
Read the full abstract ▾
Computation and Language
Less is More: Denoising Knowledge Graphs For Retrieval Augmented Generation
Yilun Zheng, Dan Yang, Jie Li et al. (+4 more)
Retrieval-Augmented Generation (RAG) systems enable large
language models (LLMs) instant access to relevant information for the generative process, demonstrating their superior performance in addressing common LLM challenges such as
hallucination, factual inaccuracy, and the knowledge cutoff. Graph-based RAG further extends this paradigm by incorporating knowledge graphs (KGs) to leverage rich, structured connections for more precise and inferential responses. A critical challenge, however, is that most Graph-based RAG systems rely on LLMs for automated KG construction, often yielding noisy KGs with redundant entities and unreliable relationships. This noise degrades
retrieval and generation performance while also increasing computational cost. Crucially, current research does not comprehensively address the denoising problem for LLM-generated KGs. In this paper, we introduce DEnoised knowledge Graphs for
Retrieval Augmented Generation (DEG-RAG), a framework that addresses these challenges through: (1) entity resolution, which eliminates redundant entities, and (2) triple reflection, which removes erroneous relations. Together, these techniques yield more compact, higher-quality KGs that significantly outperform their unprocessed counterparts. Beyond the methods, we conduct a systematic evaluation of entity resolution for LLM-generated KGs, examining different blocking strategies,
embedding choices, similarity metrics, and entity merging techniques. To the best of our knowledge, this is the first comprehensive exploration of entity resolution in LLM-generated KGs. Our experiments demonstrate that this straightforward approach not only drastically reduces graph size but also consistently improves question answering performance across diverse popular Graph-based RAG variants.
Read the full abstract ▾
All authors Yilun Zheng, Dan Yang, Jie Li, Lin Shang, Lihui Chen, Jiahao Xu, Sitao Luan
Computation and Language · Cross-listed from cs.LG
CAST: Compositional Analysis via Spectral Tracking for Understanding Transformer Layer Functions
Zihao Fu, Ming Liao, Chris Russell et al. (+1 more)
Large
language models have achieved remarkable success but remain largely black boxes with poorly understood internal mechanisms. To address this limitation, many researchers have proposed various interpretability methods including mechanistic analysis, probing classifiers, and activation visualization, each providing valuable insights from different perspectives. Building upon this rich landscape of complementary approaches, we introduce CAST (Compositional Analysis via Spectral Tracking), a probe-free framework that contributes a novel perspective by analyzing
transformer layer functions through direct transformation
matrix estimation and comprehensive spectral analysis. CAST offers complementary insights to existing methods by estimating the realized transformation
matrices for each layer using Moore-Penrose pseudoinverse and applying spectral analysis with six interpretable metrics characterizing layer behavior. Our analysis reveals distinct behaviors between encoder-only and decoder-only models, with decoder models exhibiting compression-expansion cycles while encoder models maintain consistent high-rank processing. Kernel analysis further demonstrates functional relationship patterns between layers, with CKA similarity
matrices clearly partitioning layers into three phases: feature extraction, compression, and specialization.
Read the full abstract ▾
All authors Zihao Fu, Ming Liao, Chris Russell, Zhenguang G. Cai
Computation and Language
Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behavior
Rahul Nadkarni, Yanai Elazar, Hila Gonen et al. (+1 more)
We present an experimental recipe for studying the relationship between training data and
language model (LM) behavior. We outline steps for intervening on data batches – i.e., “rewriting history” – and then retraining model checkpoints over that data to test hypotheses relating data to behavior. Our recipe breaks down such an intervention into stages that include selecting evaluation items from a
benchmark that measures model behavior, matching relevant documents to those items, and modifying those documents before retraining and measuring the effects. We demonstrate the utility of our recipe through case studies on factual knowledge acquisition in LMs, using both cooccurrence statistics and information
retrieval methods to identify documents that might contribute to knowledge learning. Our results supplement past observational analyses that link cooccurrence to model behavior, while demonstrating that extant methods for identifying relevant training documents do not fully explain an LM's ability to correctly answer knowledge questions. Overall, we outline a recipe that researchers can follow to test further hypotheses about how training data affects model behavior. Our code is made publicly available to promote future work.
Read the full abstract ▾
All authors Rahul Nadkarni, Yanai Elazar, Hila Gonen, Noah A. Smith
Computation and Language
MoM: Mixtures of Scenario-Aware Document Memories for Retrieval-Augmented Generation Systems
Jihao Zhao, Zhiyuan Ji, Simin Niu et al. (+3 more)
The traditional RAG paradigm, which typically engages in the comprehension of relevant text chunks in response to received queries, inherently restricts both the depth of knowledge internalization and
reasoning capabilities. To address this limitation, our research transforms the text processing in RAG from passive chunking to proactive understanding, defining this process as document
memory extraction with the objective of simulating human cognitive processes during reading. Building upon this, we propose the Mixtures of scenario-aware document
Memories (MoM) framework, engineered to efficiently handle documents from multiple domains and train small
language models (SLMs) to acquire the ability to proactively explore and construct document
memories. The MoM initially instructs large
language models (LLMs) to simulate domain experts in generating document logical outlines, thereby directing structured chunking and core content extraction. It employs a multi-path sampling and multi-perspective evaluation mechanism, specifically designing comprehensive metrics that represent chunk clarity and extraction completeness to select the optimal document
memories. Additionally, to infuse deeper human-like reading abilities during the training of SLMs, we incorporate a reverse
reasoning strategy, which deduces refined expert thinking paths from high-quality outcomes. Finally, leveraging diverse forms of content generated by MoM, we develop a three-layer document
memory retrieval mechanism, which is grounded in our theoretical proof from the perspective of probabilistic modeling. Extensive experimental results across three distinct domains demonstrate that the MoM framework not only resolves text chunking challenges in existing RAG systems, providing LLMs with semantically complete document
memories, but also paves the way for SLMs to achieve human-centric intelligent text processing.
Read the full abstract ▾
All authors Jihao Zhao, Zhiyuan Ji, Simin Niu, Hanyu Wang, Feiyu Xiong, Zhiyu Li
Computation and Language
Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs
Parsa Hejabi, Elnaz Rahmati, Alireza S. Ziabari et al. (+1 more)
Large
Language Models (LLMs) often produce inconsistent answers when faced with different phrasings of the same
prompt. In this paper, we propose Flip-Flop Consistency ($F^2C$), an unsupervised training method that improves robustness to such perturbations. $F^2C$ is composed of two key components. The first, Consensus Cross-
Entropy (CCE), uses a majority vote across
prompt variations to create a hard pseudo-label. The second is a representation
alignment loss that pulls lower-confidence and non-majority predictors toward the consensus established by high-confidence, majority-voting variations. We evaluate our method on 11
datasets spanning four NLP tasks, with 4-15
prompt variations per
dataset. On average, $F^2C$ raises observed agreement by 11.62%, improves mean $F_1$ by 8.94%, and reduces performance variance across formats by 3.29%. In out-of-domain evaluations, $F^2C$ generalizes effectively, increasing $\overline{F_1}$ and agreement while decreasing variance across most source-target pairs. Finally, when trained on only a subset of
prompt perturbations and evaluated on held-out formats, $F^2C$ consistently improves both performance and agreement while reducing variance. These findings highlight $F^2C$ as an effective unsupervised method for enhancing LLM consistency, performance, and generalization under
prompt perturbations. Code is available at https://github.com/ParsaHejabi/Flip-Flop-Consistency-Unsupervised-Training-for-Robustness-to-
Prompt-Perturbations-in-LLMs.
Read the full abstract ▾
All authors Parsa Hejabi, Elnaz Rahmati, Alireza S. Ziabari, Morteza Dehghani
Computation and Language · Cross-listed from cs.LG
Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models
Mehrzad Samadi, Aleksander Ficek, Sean Narenthiran et al. (+5 more)
Competitive programming has become a rigorous
benchmark for evaluating the
reasoning and problem-solving capabilities of large
language models (LLMs). The International Olympiad in Informatics (IOI) stands out as one of the most prestigious annual competitions in competitive programming and has become a key
benchmark for comparing human and AI-level programming ability. While several proprietary models have been claimed to achieve gold medal-level performance at the IOI, often with undisclosed methods, achieving comparable results with open-weight models remains a significant challenge. In this paper, we present GenCluster, a scalable and reproducible test-time compute framework that attains IOI gold-level performance using open-weight models. It combines large-scale generation, behavioral clustering, ranking, and a round-robin submission strategy to efficiently explore diverse solution spaces under limited validation budgets. Our experiments show that the performance of our proposed approach scales consistently with available compute, narrowing the gap between open and closed systems. Notably, we will show that GenCluster can achieve a gold medal at IOI 2025 for the first time with an open-weight model gpt-oss-120b, setting a new
benchmark for transparent and reproducible evaluation of
reasoning in LLMs
Read the full abstract ▾
All authors Mehrzad Samadi, Aleksander Ficek, Sean Narenthiran, Siddhartha Jain, Wasi Uddin Ahmad, Somshubra Majumdar, Vahid Noroozi, Boris Ginsburg