Computation and Language
MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs
Xinfeng Xia, Xiaofeng Hou, Jiacheng Liu et al. (+5 more)
Mixture-of-Experts (MoE) scales model capacity through sparse activation, and is becoming an important architecture for large
language models (LLMs). However, existing MoE serving systems typically execute all requests under a fixed routing configuration, limiting their ability to exploit heterogeneous computation requirements across requests. Routing top-$k$, which determines the number of routed experts activated per
token, directly controls routed-expert computation and provides a natural mechanism for request-level compute elasticity. Realizing this capability, however, requires finer-grained routing units and efficient runtime execution for heterogeneous routing budgets. We present MoE-Prism, a model and system support framework for request-level compute elasticity in MoE serving. MoE-Prismdecomposes monolithic experts into fine-grained sub-experts to expose denser routing operating points and provides a $k$-aware serving runtime that effectively serves heterogeneous routing budgets under both throughput-oriented and latency-sensitive workloads. We implement MoE-Prism on top of vLLM and evaluate it on three representative MoE models. MoE-Prism expands the number of available routing operating points by $4\times$, improves offline inference throughput by up to 33.9%, and reduces online serving TTFT under heterogeneous workloads. These results demonstrate practical elastic MoE serving with request-level routing targets.
Read the full abstract ▾
All authors Xinfeng Xia, Xiaofeng Hou, Jiacheng Liu, Wenfeng Wang, Mingxuan Zhang, Peng Tang, Chao Li, Minyi Guo
Computation and Language
Umar Butler, Abdur-Rahman Butler and Adrian Lucas Malec
We present the Massive Legal
Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source
benchmark for legal information
retrieval to date. MLEB consists of ten expert-annotated
datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the
datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information
retrieval landscape. We document our methodology in building MLEB and creating the new constituent
datasets, and release our code, results, and data openly to assist with reproducible evaluations.
Read the full abstract ▾
Computation and Language
LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts
Siyuan Wang, Gaokai Zhang, Li Lyna Zhang et al. (+4 more)
Reasoning over long contexts is essential for large
language models. While reinforcement learning (RL) enhances short-context
reasoning by inducing "Aha" moments in chain-of-thought, the advanced thinking patterns required for long-context
reasoning remain largely unexplored, and high-difficulty RL data are scarce. In this paper, we introduce LoongRL, a data-driven RL method for advanced long-context
reasoning. Central to LoongRL is KeyChain, a synthesis approach that transforms short multi-hop QA into high-difficulty long-context tasks by inserting UUID chains that hide the true question among large collections of distracting documents. Solving these tasks requires the model to trace the correct chain step-by-step, identify the true question, retrieve relevant facts and reason over them to answer correctly. RL training on KeyChain data induces an emergent plan-retrieve-reason-recheck
reasoning pattern that generalizes far beyond training length. Models trained at 16K effectively solve 128K tasks without prohibitive full-length RL rollout costs. On Qwen2.5-7B and 14B, LoongRL substantially improves long-context multi-hop QA accuracy by +23.5% and +21.1% absolute gains. The resulting LoongRL-14B reaches a score of 74.2, rivaling much larger frontier models such as o3-mini (74.5) and DeepSeek-R1 (74.9). It also improves long-context
retrieval, passes all 128K needle-in-a-haystack stress tests, and preserves short-context
reasoning capabilities.
Read the full abstract ▾
All authors Siyuan Wang, Gaokai Zhang, Li Lyna Zhang, Ning Shang, Fan Yang, Dongyao Chen, Mao Yang
Computation and Language
AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generation
Xianyang Liu, Yilin Liu, Shuai Wang et al. (+5 more)
The creation of high-quality
datasets to improve Large
Language Model (LLM)
reasoning remains a significant challenge, as current methods often suffer from generating low-quality/incorrect answers and limited information richness from available data sources. To address this, we propose AgenticMath, a novel agentic method for generating high-quality mathematical question-answer pairs to enhance the supervised fine-tuning of LLMs. Our method operates through four stages: (1) Seed Question Filter that selects questions with high information richness, complexity, and clarity; (2) an Agentic Question Rephrase step that employs a multi-
agent system to generate diverse, logically consistent paraphrases; (3) an Answer Augment step where rewrite answers using chain-of-thought
reasoning to enhance numerical and logical correctness, without reliance on human-provided labels; and (4) a final Question and Answer Evaluation that retains only the most superior pairs. Extensive experiments demonstrate that, fine-tuning 3B-8B parameter LLMs on AgenticMath generated
datasets (comprising only 30-60K math samples) achieves competitive or superior performance on diverse in domain and out-of-domain mathematical
reasoning benchmarks compared to baselines trained on much more data (e.g., 400K or 2.3M samples). Our work demonstrates that targeted, high-quality data generation is a more efficient path to improving mathematical
reasoning in LLMs than large-scale, low-quality alternatives.
Read the full abstract ▾
All authors Xianyang Liu, Yilin Liu, Shuai Wang, Hao Cheng, Andrew Estornell, Yuzhi Zhao, Jun Shu, Jiaheng Wei
Computation and Language
M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models
Yejin Kwon, Taewoo Kang, Hyunsoo Yoon et al. (+1 more)
We present M3-SLU, a new
multimodal large
language model (MLLM)
benchmark for evaluating multi-speaker, multi-turn spoken
language understanding. While recent models show strong performance in
speech and text comprehension, they still struggle with speaker-attributed
reasoning, the ability to understand who said what and when in natural conversations. M3-SLU is built from four open corpora (CHiME-6, MELD, MultiDialog, and AMI) and comprises over 12,000 validated instances with paired audio, transcripts, and metadata. It includes two tasks: (1) Speaker-Attributed Question Answering and (2) Speaker Attribution via Utterance Matching. We provide baseline results for both cascaded pipelines and end-to-end MLLMs, evaluated using an LLM-as-Judge and accuracy metrics. Results show that while models can capture what was said, they often fail to identify who said it, revealing a key gap in speaker-aware dialogue understanding. M3-SLU offers as a challenging
benchmark to advance research in speaker-aware
multimodal understanding.
Read the full abstract ▾
All authors Yejin Kwon, Taewoo Kang, Hyunsoo Yoon, Changouk Kim
Computation and Language
Modeling Turn-Taking with Semantically Informed Gestures
Varsha Suresh, M. Hamza Mughal, Christian Theobalt et al. (+1 more)
In conversation, humans use
multimodal cues, such as
speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this
dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in
multimodal turn-taking.
Read the full abstract ▾
All authors Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg
Computation and Language
Local Obfuscation by GLINER for Impartial Context Aware Lineage: Development and evaluation of PII Removal system
Prakrithi Shivaprakash, Lekhansh Shukla, Animesh Mukherjee et al. (+2 more)
Removing Personally Identifiable Information (PII) from clinical notes in Electronic Health Records (EHRs) is essential for research and AI development. While Large
Language Models (LLMs) are powerful, their high computational costs and the data
privacy risks of API-based services limit their use, especially in low-resource settings. To address this, we developed LOGICAL (Local Obfuscation by GLINER for Impartial Context-Aware Lineage), an efficient, locally deployable PII removal system built on a fine-tuned Generalist and Lightweight Named Entity Recognition (GLiNER) model. We used 1515 clinical documents from a psychiatric hospital's EHR system. We defined nine PII categories for removal. A modern-gliner-bi-large-v1.0 model was fine-tuned on 2849 text instances and evaluated on a test set of 376 instances using character-level precision, recall, and F1-score. We compared its performance against Microsoft Azure NER, Microsoft Presidio, and zero-shot prompting with Gemini-Pro-2.5 and Llama-3.3-70B-Instruct. The fine-tuned GLiNER model achieved superior performance, with an overall micro-average F1-score of 0.980, significantly outperforming Gemini-Pro-2.5 (F1-score: 0.845). LOGICAL correctly sanitised 95% of documents completely, compared to 64% for the next-best solution. The model operated efficiently on a standard laptop without a dedicated GPU. However, a 2% entity-level false negative rate underscores the need for human-in-the-loop validation across all tested systems. Fine-tuned, specialised
transformer models like GLiNER offer an accurate, computationally efficient, and secure solution for PII removal from clinical notes. This "sanitisation at the source" approach is a practical alternative to resource-intensive LLMs, enabling the creation of de-identified
datasets for research and AI development while preserving data
privacy, particularly in resource-constrained environments.
Read the full abstract ▾
All authors Prakrithi Shivaprakash, Lekhansh Shukla, Animesh Mukherjee, Prabhat Chand, Pratima Murthy
Computation and Language · Cross-listed from cs.LG
Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
Ling Team, Bin Han, Caizhi Tang et al. (+25 more)
In this technical report, we present the Ring-linear model series, specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0. Ring-mini-linear-2.0 comprises 16B parameters and 957M activations, while Ring-flash-linear-2.0 contains 104B parameters and 6.1B activations. Both models adopt a hybrid architecture that effectively integrates linear
attention and softmax
attention, significantly reducing I/O and computational overhead in long-context inference scenarios. Compared to a 32 billion parameter dense model, this series reduces inference cost to 1/10, and compared to the original Ring series, the cost is also reduced by over 50%. Furthermore, through systematic exploration of the ratio between different
attention mechanisms in the hybrid architecture, we have identified the currently optimal model structure. Additionally, by leveraging our self-developed high-performance FP8 operator library-linghe, overall training efficiency has been improved by 50%. Benefiting from the high
alignment between the training and inference engine operators, the models can undergo long-term, stable, and highly efficient optimization during the reinforcement learning phase, consistently maintaining SOTA performance across multiple challenging complex
reasoning benchmarks.
Read the full abstract ▾
All authors Ling Team, Bin Han, Caizhi Tang, Chen Liang, Donghao Zhang, Fan Yuan, Feng Zhu, Jie Gao, Jingyu Hu, Longfei Li, Meng Li, Mingyang Zhang, Peijie Jiang, Peng Jiao, Qian Zhao, Qingyuan Yang, Wenbo Shen, Xinxing Yang, Yalin Zhang, Yankun Ren, Yao Zhao, Yibo Cao, Yixuan Sun, Yue Zhang, Yuchen Fang, Zibin Lin, Zixuan Cheng, Jun Zhou
Computation and Language
Algorithmic Fairness in NLP: Persona-Infused LLMs for Human-Centric Hate Speech Detection
Ewelina Gajewska, Arda Derbent, Jaroslaw A Chudziak et al. (+1 more)
In this paper, we investigate how personalising Large
Language Models (Persona-LLMs) with annotator personas affects their sensitivity to hate
speech, particularly regarding biases linked to shared or differing identities between annotators and targets. To this end, we employ Google's Gemini and OpenAI's GPT-4.1-mini models and two persona-prompting methods: shallow persona prompting and a deeply contextualised persona development based on
Retrieval-Augmented Generation (RAG) to incorporate richer persona profiles. We analyse the impact of using in-group and out-group annotator personas on the models' detection performance and
fairness across diverse social groups. This work bridges psychological insights on group identity with advanced NLP techniques, demonstrating that incorporating socio-demographic attributes into LLMs can address bias in automated hate
speech detection. Our results highlight both the potential and limitations of persona-based approaches in reducing bias, offering valuable insights for developing more equitable hate
speech detection systems.
Read the full abstract ▾
All authors Ewelina Gajewska, Arda Derbent, Jaroslaw A Chudziak, Katarzyna Budzynska
Computation and Language
Slot Filling as a Reasoning Task for SpeechLLMs
Kadri Hacioglu, Manjunath K E and Andreas Stolcke
We propose integration of
reasoning into
speech large
language models (speechLLMs) for the end-to-end slot-filling task. Inspired by the recent development of
reasoning LLMs, we use a chain-of-thought framework to decompose the slot-filling task into multiple
reasoning steps, create a
reasoning dataset and apply the supervised fine-tuning strategy to a speechLLM. We distinguish between regular and
reasoning speechLLMs and experiment with different types and sizes of LLMs as their text foundation models. We demonstrate performance improvements by introducing
reasoning (intermediate) steps. However, we show that a
reasoning textual LLM developed mainly for math, logic and coding domains might be inferior as a foundation model for a
reasoning speechLLM. We further show that hybrid speechLLMs, built on a hybrid text foundation LLM and fine-tuned to preserve both direct and
reasoning modes of operation, have better performance than those fine-tuned employing only one mode of operation.
Read the full abstract ▾
Computation and Language
Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization
Junjie Song, Yiwen Liu, Dapeng Li et al. (+4 more)
Text summarization is a crucial task that requires the simultaneous optimization of multiple objectives, including consistency, coherence, relevance, and fluency, which presents considerable challenges. Although large
language models (LLMs) have demonstrated remarkable performance, enhanced by reinforcement learning (RL), few studies have focused on optimizing the multi-objective problem of summarization through RL based on LLMs. In this paper, we introduce hypervolume optimization (HVO), a novel optimization strategy that dynamically adjusts the scores between groups during the reward process in RL by using the hypervolume method. This method guides the model's optimization to progressively approximate the pareto front, thereby generating balanced summaries across multiple objectives. Experimental results on several representative summarization
datasets demonstrate that our method outperforms group relative policy optimization (GRPO) in overall scores and shows more balanced performance across different dimensions. Moreover, a 7B foundation model enhanced by HVO performs comparably to GPT-4 in the summarization task, while maintaining a shorter generation length. Our code is publicly available at https://github.com/ai4business-LiAuto/HVO.git
Read the full abstract ▾
All authors Junjie Song, Yiwen Liu, Dapeng Li, Yin Sun, Shukun Fu, Siqi Chen, Yuji Cao
Computation and Language
Fan Xu, Xinyu Hu, Zhenghan Yu et al. (+6 more)
The increasing reliance on natural
language generation (NLG) models, particularly large
language models, has raised concerns about the reliability and accuracy of their outputs. A key challenge is
hallucination, where models produce plausible but incorrect information. As a result,
hallucination detection has become a critical task. In this work, we introduce a comprehensive
hallucination taxonomy with 11 categories across various NLG tasks and propose the
HAllucination Detection (HAD) models https://github.com/pku0xff/HAD, which integrate
hallucination detection, span-level identification, and correction into a single inference process. Trained on an elaborate synthetic
dataset of about 90K samples, our HAD models are versatile and can be applied to various NLG tasks. We also carefully annotate a test set for
hallucination detection, called HADTest, which contains 2,248 samples. Evaluations on in-domain and out-of-domain test sets show that our HAD models generally outperform the existing baselines, achieving state-of-the-art results on HaluEval, FactCHD, and FaithBench, confirming their robustness and versatility.
Read the full abstract ▾
All authors Fan Xu, Xinyu Hu, Zhenghan Yu, Li Lin, Xu Zhang, Yang Zhang, Wei Zhou, Jinjie Gu, Xiaojun Wan
Computation and Language
KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls
Kailin Jiang, Hongbo Jiang, Ning Jiang et al. (+7 more)
Large
Multimodal Models encode extensive factual knowledge in their pre-trained weights. However, its knowledge remains static and limited, unable to keep pace with real-world developments, which hinders continuous knowledge acquisition. Effective knowledge injection thus becomes critical, involving two goals: knowledge adaptation (injecting new knowledge) and knowledge retention (preserving old knowledge). Existing methods often struggle to learn new knowledge and suffer from catastrophic forgetting. To address this, we propose KORE, a synergistic method of KnOwledge-oRientEd augmentations and constraints for injecting new knowledge into large
multimodal models while preserving old knowledge. Unlike general text or image data augmentation, KORE automatically converts individual knowledge items into structured and comprehensive knowledge to ensure that the model accurately learns new knowledge, enabling accurate adaptation. Meanwhile, KORE stores previous knowledge in the covariance
matrix of LMM's linear layer activations and initializes the adapter by projecting the original weights into the
matrix's null space, defining a fine-tuning direction that minimizes interference with previous knowledge, enabling powerful retention. Extensive experiments on various LMMs, including LLaVA-v1.5-7B, LLaVA-v1.5-13B, and Qwen2.5-VL-7B, show that KORE achieves superior new knowledge injection performance and effectively mitigates catastrophic forgetting.
Read the full abstract ▾
All authors Kailin Jiang, Hongbo Jiang, Ning Jiang, Zhi Gao, Jinhe Bi, Yuchen Ren, Bin Li, Yuntao Du, Lei Liu, Qing Li
Computation and Language
JointCQ: Improving Factual Hallucination Detection with Joint Claim and Query Generation
Fan Xu, Huixuan Zhang, Zhenliang Zhang et al. (+2 more)
Current large
language models (LLMs) often suffer from
hallucination issues, i,e, generating content that appears factual but is actually unreliable. A typical
hallucination detection pipeline involves response decomposition (i.e., claim extraction), query generation, evidence collection (i.e., search or
retrieval), and claim verification. However, existing methods exhibit limitations in the first two stages, such as context loss during claim extraction and low specificity in query generation, resulting in degraded performance across the
hallucination detection pipeline. In this work, we introduce JointCQ https://github.com/pku0xff/JointCQ, a joint claim-and-query generation framework designed to construct an effective and efficient claim-query generator. Our framework leverages elaborately designed evaluation criteria to filter synthesized training data, and finetunes a
language model for joint claim extraction and query generation, providing reliable and informative inputs for downstream search and verification. Experimental results demonstrate that our method outperforms previous methods on multiple open-domain QA
hallucination detection
benchmarks, advancing the goal of more trustworthy and transparent
language model systems.
Read the full abstract ▾
All authors Fan Xu, Huixuan Zhang, Zhenliang Zhang, Jiahao Wang, Xiaojun Wan
Computation and Language
TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
Reza Esfandiarpoor, Vishwas Suryanarayanan, Stephen H. Bach et al. (+2 more)
Since the introduction of the Model Context Protocol (MCP), the number of available tools for Large
Language Models (LLMs) has increased significantly. These task-specific tool sets offer an alternative to general-purpose tools such as web browsers, while being easier to develop and maintain than GUIs. However, current general-purpose
agents predominantly rely on web browsers for interacting with the environment. Here, we introduce TheMCPCompany, a
benchmark for evaluating tool-calling
agents on tasks that involve interacting with various real-world services. We use the REST APIs of these services to create MCP servers, which include over 18,000 tools. We also provide manually annotated ground-truth tools for each task. In our experiments, we use the ground truth tools to show the potential of tool-calling
agents for both improving performance and reducing costs assuming perfect tool
retrieval. Next, we explore
agent performance using tool
retrieval to study the real-world practicality of tool-based
agents. While all models with tool
retrieval perform similarly or better than browser-based
agents, smaller models cannot take full advantage of the available tools through
retrieval. On the other hand, GPT-5's performance with tool
retrieval is very close to its performance with ground-truth tools. Overall, our work shows that the most advanced
reasoning models are effective at discovering tools in simpler environments, but seriously struggle with navigating complex enterprise environments. TheMCPCompany reveals that navigating tens of thousands of tools and combining them in non-trivial ways to solve complex problems is still a challenging task for current models and requires both better
reasoning and better
retrieval models.
Read the full abstract ▾
All authors Reza Esfandiarpoor, Vishwas Suryanarayanan, Stephen H. Bach, Vishal Chowdhary, Anthony Aue
Computation and Language
Difficulty-Controllable Multiple-Choice Question Generation Using Large Language Models and Direct Preference Optimization
Yuto Tomikawa and Masaki Uto
Difficulty-controllable question generation for reading comprehension has gained significant
attention in the field of education as a fundamental tool for adaptive learning support. Although several neural question generation methods have recently succeeded in controlling difficulty, conventional approaches still face two major limitations. First, they cannot directly generate multiple-choice questions, which are the most widely used question type in educational contexts. Second, they are not explicitly trained to optimize the accuracy of difficulty control, leaving room for further improvement in difficulty controllability. To address these limitations, this study proposes a novel difficulty-controllable multiple-choice question generation method for reading comprehension which leverages a large
language model trained using a direct preference optimization technique to improve the accuracy of difficulty control.
Read the full abstract ▾
Computation and Language
SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheets
Ziwei Wang, Jiayuan Su, Mengyu Zhou et al. (+7 more)
Understanding and
reasoning over complex spreadsheets remain fundamental challenges for large
language models (LLMs), which often struggle with accurately capturing the complex structure of tables and ensuring
reasoning correctness. In this work, we propose SheetBrain, a neuro-symbolic dual workflow
agent framework designed for accurate
reasoning over tabular data, supporting both spreadsheet question answering and manipulation tasks. SheetBrain comprises three core modules: an understanding module, which produces a comprehensive overview of the spreadsheet - including sheet summary and query-based problem insight to guide
reasoning; an execution module, which integrates a Python sandbox with preloaded table-processing libraries and an Excel helper toolkit for effective multi-turn
reasoning; and a validation module, which verifies the correctness of
reasoning and answers, triggering re-execution when necessary. We evaluate SheetBrain on multiple public tabular QA and manipulation
benchmarks, and introduce SheetBench, a new
benchmark targeting large, multi-table, and structurally complex spreadsheets. Experimental results show that SheetBrain significantly improves accuracy on both existing
benchmarks and the more challenging scenarios presented in SheetBench. Our code is publicly available at https://github.com/microsoft/SheetBrain.
Read the full abstract ▾
All authors Ziwei Wang, Jiayuan Su, Mengyu Zhou, Huaxing Zeng, Mengni Jia, Xiao Lv, Haoyu Dong, Xiaojun Ma, Shi Han, Dongmei Zhang
Computation and Language
Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+
York Hay Ng, Aditya Khan, Xiang Lu et al. (+5 more)
Existing linguistic knowledge bases such as URIEL+ provide valuable geographic, genetic and typological distances for cross-lingual transfer but suffer from two key limitations. First, their one-size-fits-all vector representations are ill-suited to the diverse structures of linguistic data. Second, they lack a principled method for aggregating these signals into a single, comprehensive score. In this paper, we address these gaps by introducing a framework for type-matched
language distances. We propose novel, structure-aware representations for each distance type: speaker-weighted distributions for geography,
hyperbolic embeddings for genealogy, and a latent variables model for typology. We unify these signals into a robust, task-agnostic composite distance. Across multiple zero-shot transfer
benchmarks, we demonstrate that our representations significantly improve transfer performance when the distance type is relevant to the task, while our composite distance yields gains in most tasks.
Read the full abstract ▾
All authors York Hay Ng, Aditya Khan, Xiang Lu, Matteo Salloum, Michael Zhou, Phuong H. Hoang, A. Seza Doğruöz, En-Shiun Annie Lee
Computation and Language
DiSRouter: Distributed Self-Routing for LLM Selections
Hang Zheng, Hongshen Xu, Yongkai Lin et al. (+3 more)
The proliferation of Large
Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM
agents, each independently deciding whether to answer or route to other
agents based on its own self-awareness, its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-
agent systems.
Read the full abstract ▾
All authors Hang Zheng, Hongshen Xu, Yongkai Lin, Shuai Fan, Lu Chen, Kai Yu
Computation and Language · Cross-listed from q-fin.CP
Aligning Multilingual News for Stock Return Prediction
Yuntao Wu, Lynn Tao, Ing-Haw Cheng et al. (+4 more)
News spreads rapidly across
languages and regions, but translations may lose subtle nuances. We propose a method to align sentences in
multilingual news articles using optimal transport, identifying semantically similar content across
languages. We apply this method to align more than 140,000 pairs of Bloomberg English and Japanese news articles covering around 3500 stocks in Tokyo exchange over 2012-2024. Aligned sentences are sparser, more interpretable, and exhibit higher semantic similarity. Return scores constructed from aligned sentences show stronger correlations with realized stock returns, and long-short trading strategies based on these
alignments achieve 10% higher Sharpe ratios than analyzing the full text sample.
Read the full abstract ▾
All authors Yuntao Wu, Lynn Tao, Ing-Haw Cheng, Charles Martineau, Yoshio Nozawa, John Hull, Andreas Veneris
Computation and Language
When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue
Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou et al. (+3 more)
Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user,
agent, and tools. While existing evaluation methods assess either user satisfaction or
agents' tool-calling capabilities, they fail to capture critical errors in multi-turn tool-augmented dialogues-such as when
agents misinterpret tool results yet appear satisfactory to users. We introduce TRACE, a
benchmark of systematically synthesized tool-augmented conversations covering diverse error cases. Evaluation with state-of-the-art conversation evaluation frameworks reveals that all approaches remain far from ideal performance, demonstrating the fundamental difficulty of this
benchmark.
Read the full abstract ▾
All authors Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou, Shamik Roy, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah
Computation and Language
Interpretable Question Answering with Knowledge Graphs
Kartikeya Aneja, Manasvi Srivastava, Subhayan Das et al. (+1 more)
This paper presents a question answering system that operates exclusively on a knowledge graph
retrieval without relying on
retrieval augmented generation (RAG) with large
language models (LLMs). Instead, a small paraphraser model is used to paraphrase the entity relationship edges retrieved from querying the knowledge graph. The proposed pipeline is divided into two main stages. The first stage involves pre-processing a document to generate sets of question-answer (QA) pairs. The second stage converts these QAs into a knowledge graph from which graph-based
retrieval is performed using
embeddings and fuzzy techniques. The graph is queried, re-ranked, and paraphrased to generate a final answer. This work includes an evaluation using LLM-as-a-judge on the CRAG
benchmark, which resulted in accuracies of 71.9% and 54.4% using LLAMA-3.2 and GPT-3.5-Turbo, respectively.
Read the full abstract ▾
All authors Kartikeya Aneja, Manasvi Srivastava, Subhayan Das, Nagender Aneja
Computation and Language · Cross-listed from cs.AI
The Zero-Step Thinking: An Empirical Study of Mode Selection as Harder Early Exit in Reasoning Models
Yuqiao Tan, Shizhu He, Kang Liu et al. (+1 more)
Reasoning models have demonstrated exceptional performance in tasks such as mathematics and logical
reasoning, primarily due to their ability to engage in step-by-step thinking during the
reasoning process. However, this often leads to overthinking, resulting in unnecessary computational overhead. To address this issue, Mode Selection aims to automatically decide between Long-CoT (Chain-of-Thought) or Short-CoT by utilizing either a Thinking or NoThinking mode. Simultaneously, Early Exit determines the optimal stopping point during the iterative
reasoning process. Both methods seek to reduce the computational burden. In this paper, we first identify Mode Selection as a more challenging variant of the Early Exit problem, as they share similar objectives but differ in decision timing. While Early Exit focuses on determining the best stopping point for concise
reasoning at inference time, Mode Selection must make this decision at the beginning of the
reasoning process, relying on pre-defined fake thoughts without engaging in an explicit
reasoning process, referred to as zero-step thinking. Through empirical studies on nine baselines, we observe that
prompt-based approaches often fail due to their limited classification capabilities when provided with minimal hand-crafted information. In contrast, approaches that leverage internal information generally perform better across most scenarios but still exhibit issues with stability. Our findings indicate that existing methods relying solely on the information provided by models are insufficient for effectively addressing Mode Selection in scenarios with limited information, highlighting the ongoing challenges of this task. Our code is available at https://github.com/Trae1ounG/Zero_Step_Thinking.
Read the full abstract ▾
All authors Yuqiao Tan, Shizhu He, Kang Liu, Jun Zhao
Computation and Language
When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA
Nishanth Sridhar Nakshatri, Shamik Roy, Manoj Ghuhan Arivazhagan et al. (+3 more)
LLMs often fail to handle temporal knowledge conflicts–contradictions arising when facts evolve over time within their training data. Existing studies evaluate this phenomenon through
benchmarks built on structured knowledge bases like Wikidata, but they focus on widely-covered, easily-memorized popular entities and lack the dynamic structure needed to fairly evaluate LLMs with different knowledge cut-off dates. We introduce evolveQA, a
benchmark specifically designed to evaluate LLMs on temporally evolving knowledge, constructed from 3 real-world, time-stamped corpora: AWS updates, Azure changes, and WHO disease outbreak reports. Our framework identifies naturally occurring knowledge
evolution and generates questions with gold answers tailored to different LLM knowledge cut-off dates. Through extensive evaluation of 12 open and closed-source LLMs across 3 knowledge probing formats, we demonstrate significant performance drops of up to 31% on evolveQA compared to static knowledge questions.
Read the full abstract ▾
All authors Nishanth Sridhar Nakshatri, Shamik Roy, Manoj Ghuhan Arivazhagan, Hanhan Zhou, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah
Computation and Language
Think Straight, Stop Smart: Structured Reasoning for Efficient Multi-Hop RAG
Jihwan Bang, Juntae Lee, Seunghan Yang et al. (+1 more)
Multi-hop
retrieval-augmented generation (RAG) is a promising strategy for complex
reasoning, yet existing iterative prompting approaches remain inefficient. They often regenerate predictable
token sequences at every step and rely on
stochastic stopping, leading to excessive
token usage and unstable termination. We propose TSSS (Think Straight, Stop Smart), a structured multi-hop RAG framework designed for efficiency. TSSS introduces (i) a template-based
reasoning that caches recurring prefixes and anchors sub-queries to the main question, reducing
token generation cost while promoting stable
reasoning, and (ii) a retriever-based terminator, which deterministically halts
reasoning once additional sub-queries collapse into repetition. This separation of structured
reasoning and termination control enables both faster inference and more reliable answers. On HotpotQA, 2WikiMultiHop, and MuSiQue, TSSS achieves state-of-the-art accuracy and competitive efficiency among RAG-CoT approaches, highlighting its effectiveness in efficiency-constrained scenarios such as on-device inference.
Read the full abstract ▾
All authors Jihwan Bang, Juntae Lee, Seunghan Yang, Sungha Choi
Computation and Language · Cross-listed from cs.CR
OpenGuardrails: A Configurable, Unified, and Scalable Guardrails Platform for Large Language Models
Thomas Wang and Haowen Li
As large
language models (LLMs) are increasingly integrated into real-world applications, ensuring their safety, robustness, and
privacy compliance has become critical. We present OpenGuardrails, the first fully open-source platform that unifies large-model-based safety detection, manipulation defense, and deployable guardrail infrastructure. OpenGuardrails protects against three major classes of risks: (1) content-safety violations such as harmful or explicit text generation, (2) model-manipulation attacks including
prompt injection, jailbreaks, and code-interpreter abuse, and (3) data leakage involving sensitive or private information. Unlike prior modular or rule-based frameworks, OpenGuardrails introduces three core innovations: (1) a Configurable Policy Adaptation mechanism that allows per-request customization of unsafe categories and sensitivity thresholds; (2) a Unified LLM-based Guard Architecture that performs both content-safety and manipulation detection within a single model; and (3) a Quantized, Scalable Model Design that compresses a 14B dense base model to 3.3B via GPTQ while preserving over 98 of
benchmark accuracy. The system supports 119
languages, achieves state-of-the-art performance across
multilingual safety
benchmarks, and can be deployed as a secure gateway or API-based service for enterprise use. All models,
datasets, and deployment scripts are released under the Apache 2.0 license.
Read the full abstract ▾
Computation and Language
"You Are Rejected!": An Empirical Study of Large Language Models Taking Hiring Evaluations
Dingjie Fu and Dianxing Shi
With the proliferation of the internet and the rapid advancement of Artificial Intelligence, leading technology companies face an urgent annual demand for a considerable number of software and
algorithm engineers. To efficiently and effectively identify high-potential candidates from thousands of applicants, these firms have established a multi-stage selection process, which crucially includes a standardized hiring evaluation designed to assess job-specific competencies. Motivated by the demonstrated prowess of Large
Language Models (LLMs) in coding and
reasoning tasks, this paper investigates a critical question: Can LLMs successfully pass these hiring evaluations? To this end, we conduct a comprehensive examination of a widely used professional assessment questionnaire. We employ state-of-the-art LLMs to generate responses and subsequently evaluate their performance. Contrary to any prior expectation of LLMs being ideal engineers, our analysis reveals a significant inconsistency between the model-generated answers and the company-referenced solutions. Our empirical findings lead to a striking conclusion: All evaluated LLMs fails to pass the hiring evaluation.
Read the full abstract ▾
Computation and Language
Tibetan Language and AI: A Comprehensive Survey of Resources, Methods and Challenges
Cheng Huang, Nyima Tashi, Fan Gao et al. (+19 more)
Tibetan, one of the major low-resource
languages in Asia, presents unique linguistic and sociocultural characteristics that pose both challenges and opportunities for AI research. Despite increasing interest in developing AI systems for underrepresented
languages, Tibetan has received limited
attention due to a lack of accessible data resources, standardized
benchmarks, and dedicated tools. This paper provides a comprehensive survey of the current state of Tibetan AI in the AI domain, covering textual and
speech data resources, NLP tasks, machine translation,
speech recognition, and recent developments in LLMs. We systematically categorize existing
datasets and tools, evaluate methods used across different tasks, and compare performance where possible. We also identify persistent bottlenecks such as data sparsity, orthographic variation, and the lack of unified evaluation metrics. Additionally, we discuss the potential of cross-lingual transfer, multi-modal learning, and community-driven resource creation. This survey aims to serve as a foundational reference for future work on Tibetan AI research and encourages collaborative efforts to build an inclusive and sustainable AI ecosystem for low-resource
languages.
Read the full abstract ▾
All authors Cheng Huang, Nyima Tashi, Fan Gao, Yutong Liu, Jiahao Li, Hao Tian, Siyang Jiang, Thupten Tsering, Ban Ma-bao, Renzeg Duojie, Gadeng Luosang, Rinchen Dongrub, Dorje Tashi, Jin Zhang, Xiao Feng, Hao Wang, Jie Tang, Guojie Tang, Xiangxiang Wang, Jia Zhang, Tsengdar Lee, Yongbin Yu
Computation and Language · Cross-listed from cs.AI
A Multi-faceted Analysis of Cognitive Abilities: Evaluating Prompt Methods with Large Language Models on the CONSORT Checklist
Sohyeon Jeon and Hyung-Chul Lee
Despite the rapid expansion of Large
Language Models (LLMs) in healthcare, robust and explainable evaluation of their ability to assess clinical trial reporting according to CONSORT standards remains an open challenge. In particular, uncertainty calibration and metacognitive reliability of LLM
reasoning are poorly understood and underexplored in medical automation. This study applies a behavioral and metacognitive analytic approach using an expert-validated
dataset, systematically comparing two representative LLMs - one general and one domain-specialized - across three
prompt strategies. We analyze both cognitive adaptation and calibration error using metrics: Expected Calibration Error (ECE) and a baseline-normalized Relative Calibration Error (RCE) that enables reliable cross-model comparison. Our results reveal pronounced miscalibration and overconfidence in both models, especially under clinical role-playing conditions, with calibration error persisting above clinically relevant thresholds. These findings underscore the need for improved calibration, transparent code, and strategic
prompt engineering to develop reliable and explainable medical AI.
Read the full abstract ▾