Computation and Language
Typologically Informed Parameter Aggregation
Stef Accou and Wessel Poelman
Massively
multilingual language models enable cross-lingual generalization but underperform on low-resource and unseen
languages. While adapter-based fine-tuning offers a parameter-efficient solution, training
language-specific adapters at scale remains costly. We introduce Typologically Informed Parameter Aggregation (TIPA), a training-free method that constructs proxy
language adapters by aggregating existing ones, weighted by typological similarity. Integrated into the MAD-X framework, these proxies enable zero-shot cross-lingual transfer without additional training. We evaluate TIPA on five NLP tasks and over 230
languages. TIPA consistently outperforms or matches baselines such as English-only fine-tuning or selecting the typologically closest
language adapter. We see the largest gains for
languages lacking dedicated adapters. Our results demonstrate that typologically informed aggregation provides a viable alternative to
language-specific modules without any training needed.
Read the full abstract ▾
Computation and Language
MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages
Weerayut Buaphet, Thanh-Nhi Nguyen, Risa Kondo et al. (+8 more)
Social media data has been of interest to Natural
Language Processing (NLP) practitioners for over a decade, because of its richness in information, but also challenges for automatic processing. Since
language use is more informal, spontaneous, and adheres to many different sociolects, the performance of NLP models often deteriorates. One solution to this problem is to transform data to a standard variant before processing it, which is also called lexical normalization. There has been a wide variety of
benchmarks and models proposed for this task. The MultiLexNorm
benchmark proposed to unify these efforts, but it consists almost solely of
languages from the Indo-European
language family in the Latin script. Hence, we propose an extension to MultiLexNorm, which covers 5 Asian
languages from different
language families in 4 different scripts. We show that the previous state-of-the-art model performs worse on the new
languages and propose a new architecture based on Large
Language Models (LLMs), which shows more robust performance. Finally, we analyze remaining errors, revealing future directions for this task.
Read the full abstract ▾
All authors Weerayut Buaphet, Thanh-Nhi Nguyen, Risa Kondo, Tomoyuki Kajiwara, Yumin Kim, Jimin Lee, Hwanhee Lee, Holy Lovenia, Peerat Limkonchotiwat, Sarana Nutanong, Rob Van der Goot
Computation and Language
How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistants
Xueyang Feng, Weinan Gan, Xu Chen et al. (+2 more)
Large
language model (LLM)-powered assistants have recently integrated
memory mechanisms that record user preferences, leading to more personalized and user-aligned responses. However, irrelevant personalized
memories are often introduced into the context, interfering with the LLM's intent understanding. To comprehensively investigate the dual effects of personalization, we develop RPEval, a
benchmark comprising a personalized intent
reasoning dataset and a multi-granularity evaluation protocol. RPEval reveals the widespread phenomenon of irrational personalization in existing LLMs and, through error pattern analysis, illustrates its negative impact on user experience. Finally, we introduce RP-Reasoner, which treats
memory utilization as a pragmatic
reasoning process, enabling the selective integration of personalized information. Experimental results demonstrate that our method significantly outperforms carefully designed baselines on RPEval, and resolves 80% of the bad cases observed in a large-scale commercial personalized assistant, highlighting the potential of pragmatic
reasoning to mitigate irrational personalization. Our
benchmark is publicly available at https://github.com/XueyangFeng/RPEval.
Read the full abstract ▾
All authors Xueyang Feng, Weinan Gan, Xu Chen, Quanyu Dai, Yong Liu
Computation and Language
PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs
Jing Xu, Jiaqi Wang, Daxin Tan et al. (+1 more)
Although Large
Language Models (LLMs) excel in many tasks, their application to
Speech-to-
Speech Translation (S2ST) is underexplored and hindered by data scarcity. To bridge this gap, we propose PROST-LLM (PROgressive
Speech-to-
speech Translation) to enhance the S2ST capabilities in LLMs progressively. First, we fine-tune the LLMs with the CVSS corpus, employing designed tri-task learning and chain of modality methods to boost the initial performance. Then, leveraging the fine-tuned model, we generate preference pairs through self-sampling and back-translation without human evaluation. Finally, these preference pairs are used for preference optimization to enhance the model's S2ST capability further. Extensive experiments confirm the effectiveness of our proposed PROST-LLM in improving the S2ST capability of LLMs.
Read the full abstract ▾
All authors Jing Xu, Jiaqi Wang, Daxin Tan, Xiao Chen
Computation and Language
AuroraEdge-V-2B: A Faster And Stronger Edge Visual Large Language Model
Xiang Chen
Recently, due to the advancement of
multimodal technology, people are attempting to use visual large
language models (VLLMs) in industrial production. Many deep learning models (DLMs) deployed in the production environment are gradually being replaced by VLLMs. Compared with DLMs, VLLMs have some advantages in industrial applications: (1) Their strong generalization ability enables them to perform well across a wide range of tasks. (2) They are flexible and can deal with unfamiliar samples through context learning quickly. However, VLLMs also have obvious drawbacks: (1) VLLMs do not perform as well as custom-developed DLMs in specific domains. (2) The number of parameters in VLLMs is generally quite large, and their deployment requires substantial computational resources. (3) VLLMs generally operate much slower than DLMs, making real-time response challenging to achieve. To better utilize VLLMs in industrial applications, we introduce AuroraEdge-V-2B in this work, a compact, robust, and high-speed VLLM designed for edge deployment. To make the model run faster, we also propose a compression-fusion method to improve inference efficiency. AuroraEdge-V-2B has the following notable features: (1) Easy deployment and faster: It has only 2B parameters and is highly suitable for edge deployment, offering better real-time performance. (2) Fewer visual
tokens and cheaper: It significantly reduces the number of visual
tokens in the decoding process, thereby reducing the floating-point operations by half during inference and making it cheaper to use. (3) Strong performance: It gets a higher score on 9
benchmarks than models with the same number of parameter (e.g., Qwen2-VL-2B, Qwen2.5-VL-3B, InternVL-2.5-2B).
Read the full abstract ▾
Computation and Language
Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis
Jianyu Wen, Yang Wei, Xiongxi Yu et al. (+2 more)
As the development of Large
Language Models (LLMs) shifts from parameter scaling to inference-time collaboration, the Mixture-of-
Agents (MoA) framework has emerged as a general paradigm to harness collective intelligence by layering diverse models. While recent MoA variants have introduced dynamic routing and residual connections to improve efficiency, these methods often fail to facilitate deep semantic interaction between
agents, limiting the system's ability to actively correct
hallucinations and refine logic. In this paper, we introduce
Attention-MoA, a novel MoA-based framework that redefines collaboration through Inter-
agent Semantic
Attention. Complemented by an Inter-layer Residual Module with Adaptive Early Stopping Mechanism, our architecture mitigates information degradation in deep layers while improving computational efficiency. Extensive evaluations across AlpacaEval 2.0, MT-Bench, and FLASK demonstrate that
Attention-MoA significantly outperforms state-of-the-art baselines, achieving a 91.15% Length-Controlled Win Rate on AlpacaEval 2.0 and dominating in 10 out of 12 capabilities on FLASK. Notably,
Attention-MoA enables an ensemble of small open-source models to outperform massive proprietary models like Claude-4.5-Sonnet and GPT-4.1, achieving an MT-Bench score of 8.83 and an AlpacaEval 2.0 LC Win Rate of 77.36%.
Read the full abstract ▾
All authors Jianyu Wen, Yang Wei, Xiongxi Yu, Changxuan Xiao, Ke Zeng
Computation and Language
Retrieve-Refine-Calibrate: A Framework for Complex Claim Fact-Checking
Mingwei Sun, Qianlong Wang and Ruifeng Xu
Fact-checking aims to verify the truthfulness of a claim based on the retrieved evidence. Existing methods typically follow a decomposition paradigm, in which a claim is broken down into sub-claims that are individually verified. However, the decomposition paradigm may introduce noise to the verification process due to irrelevant entities or evidence, ultimately degrading verification accuracy. To address this problem, we propose a Retrieve-Refine-Calibrate (RRC) framework based on large
language models (LLMs). Specifically, the framework first identifies the entities mentioned in the claim and retrieves evidence relevant to them. Then, it refines the retrieved evidence based on the claim to reduce irrelevant information. Finally, it calibrates the verification process by re-evaluating low-confidence predictions. Experiments on two popular fact-checking
datasets (HOVER and FEVEROUS-S) demonstrate that our framework achieves superior performance compared with competitive baselines.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
A Collision-Free Hot-Tier Extension for Engram-Style Conditional Memory: A Controlled Study of Training Dynamics
Tao Lin
We investigate whether high-frequency key collisions are a primary bottleneck in Engram-style conditional
memory. To isolate the effect of collisions, we introduce Engram-Nine, a collision-free hot-tier extension that maps the most frequent n-grams through a Minimal Perfect Hash Function (MPHF) while retaining the original multi-head hashed lookup as a cold tier. Under a strictly iso-parameter setup, the collision-free design does not consistently improve validation loss. Through route-stratified evaluation (decomposing per-
token loss into hot/cold contributions), we uncover a consistent "hot-to-cold advantage flip" during training: hot (high-frequency) positions initially have lower loss, but cold positions eventually surpass them. Crucially, collision-free configurations flip earlier than collision-prone baselines, suggesting that collisions act as implicit regularization. We also identify a gating mismatch: the gate learns to favor hot positions early in training, but this preference persists even after the flip, assigning higher weights to positions with higher loss. Our findings suggest that improving lookup precision alone does not guarantee better training outcomes. The dominant limitation may lie in gating credit assignment rather than index accuracy, and collision-induced noise may provide beneficial regularization that should not be naively eliminated.
Read the full abstract ▾
Computation and Language
Curate-Train-Refine: A Closed-Loop Agentic Framework for Zero Shot Classification
Gaurav Maheshwari and Kevin El Haddad
Large
language models (LLMs) and high-capacity encoders have advanced zero and few-shot classification, but their inference cost and latency limit practical deployment. We propose training lightweight text classifiers using dynamically generated supervision from an LLM. Our method employs an iterative, agentic loop in which the LLM curates training data, analyzes model successes and failures, and synthesizes targeted examples to address observed errors. This closed-loop generation and evaluation process progressively improves data quality and adapts it to the downstream classifier and task. Across four widely used
benchmarks, our approach consistently outperforms standard zero and few-shot baselines. These results indicate that LLMs can serve effectively as data curators, enabling accurate and efficient classification without the operational cost of large-model deployment.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMs
Xianya Fang, Feiyang Ren, Xiang Chen et al. (+4 more)
Multimodal LLMs are powerful but prone to object
hallucinations, which describe non-existent entities and harm reliability. While recent unlearning methods attempt to mitigate this, we identify a critical flaw: structural fragility. We empirically demonstrate that standard erasure achieves only superficial suppression, trapping the model in sharp minima where
hallucinations catastrophically resurge after lightweight relearning. To ensure
geometric stability, we propose SARE, which casts unlearning as a targeted min-max optimization problem and uses a Targeted-SAM mechanism to explicitly flatten the loss landscape around hallucinated concepts. By suppressing
hallucinations under simulated worst-case parameter perturbations, our framework ensures robust removal stable against weight shifts. Extensive experiments demonstrate that SARE significantly outperforms baselines in erasure efficacy while preserving general generation quality. Crucially, it maintains persistent
hallucination suppression against relearning and parameter updates, validating the effectiveness of
geometric stabilization.
Read the full abstract ▾
All authors Xianya Fang, Feiyang Ren, Xiang Chen, Yu Tian, Zhen Bi, Haiyang Yu, Sheng-Jun Huang
Computation and Language · Cross-listed from cs.CV
TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning
Daixian Liu, Jiayi Kuang, Yinghui Li et al. (+8 more)
Multimodal Large
Language Models (MLLMs) have achieved remarkable progress in visual recognition and semantic understanding, yet precise compositional spatial
reasoning under
geometric constraints remains underexplored. Existing
benchmarks mainly assess coarse spatial relations and rarely support rigorous
geometric verification or multiple valid solutions in constructive tasks. To address these limitations, we introduce TangramPuzzle, a
benchmark comprising 668 validated configurations and 1,336 instances for evaluating compositional spatial
reasoning under
geometric constraints. We formulate the Tangram Construction Expression (TCE) to encode tangram configurations with machine-verifiable
geometric specifications, together with a multi-solution-aware verifier that separately evaluates
geometric validity and silhouette fidelity. TangramPuzzle covers two complementary evaluation tasks: part-to-whole Outline Prediction and whole-to-part End-to-End Assembly Generation. We conduct extensive evaluation experiments on advanced open-source and proprietary models, revealing an interesting insight: MLLMs tend to prioritize matching the target silhouette while neglecting
geometric constraints, leading to distortions or deformations of the pieces.
Read the full abstract ▾
All authors Daixian Liu, Jiayi Kuang, Yinghui Li, Yangning Li, Di Yin, Haoyu Cao, Xing Sun, Ying Shen, Hai-Tao Zheng, Liang Lin, Philip S. Yu
Computation and Language
SearchLLM: Detecting LLM Paraphrased Text by Measuring the Similarity with Regeneration of the Candidate Source via Search Engine
Hoang-Quoc Nguyen-Son, Minh-Son Dao and Koji Zettsu
With the advent of large
language models (LLMs), it has become common practice for users to draft text and utilize LLMs to enhance its quality through paraphrasing. However, this process can sometimes result in the loss or distortion of the original intended meaning. Due to the human-like quality of LLM-generated text, traditional detection methods often fail, particularly when text is paraphrased to closely mimic original content. In response to these challenges, we propose a novel approach named SearchLLM, designed to identify LLM-paraphrased text by leveraging search engine capabilities to locate potential original text sources. By analyzing similarities between the input and regenerated versions of candidate sources, SearchLLM effectively distinguishes LLM-paraphrased content. SearchLLM is designed as a proxy layer, allowing seamless integration with existing detectors to enhance their performance. Experimental results across various LLMs demonstrate that SearchLLM consistently enhances the accuracy of recent detectors in detecting LLM-paraphrased text that closely mimics original content. Furthermore, SearchLLM also helps the detectors prevent paraphrasing attacks.
Read the full abstract ▾
Computation and Language
Is Length Really A Liability? An Evaluation of Multi-turn LLM Conversations using BoolQ
Karl Neergaard, Le Qiu and Emmanuele Chersoni
Single-
prompt evaluations dominate current LLM benchmarking, yet they fail to capture the conversational dynamics where real-world harm occurs. In this study, we examined whether conversation length affects response veracity by evaluating LLM performance on the BoolQ
dataset under varying length and scaffolding conditions. Our results across three distinct LLMs revealed model-specific vulnerabilities that are invisible under single-turn testing. The length-dependent and scaffold-specific effects we observed demonstrate a fundamental limitation of static evaluations, as deployment-relevant vulnerabilities could only be spotted in a multi-turn conversational setting.
Read the full abstract ▾
Computation and Language
LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning
Obed Junias and Maria Leonor Pacheco
Commonsense
reasoning often involves evaluating multiple plausible interpretations rather than selecting a single atomic answer, yet most
benchmarks rely on single-label evaluation, obscuring whether statements are jointly plausible, mutually exclusive, or jointly implausible. We introduce LOGICAL-COMMONSENSEQA, a
benchmark that reframes commonsense
reasoning as logical composition over pairs of atomic statements using plausibility-level operators (AND, OR and NEITHER/NOR). Evaluating instruction-tuned,
reasoning-specialized, and fine-tuned models under zero-shot, few-shot, and chain-of-thought prompting, we find that while models perform reasonably on conjunctive and moderately on disjunctive
reasoning, performance degrades sharply on negation-based questions. LOGICAL-COMMONSENSEQA exposes fundamental
reasoning limitations and provides a controlled framework for advancing compositional commonsense
reasoning.
Read the full abstract ▾
Computation and Language
MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine
Liz Li and Wei Zhu
While
Retrieval-Augmented Generation (RAG) has been swiftly adopted in scientific and clinical QA systems, a comprehensive evaluation
benchmark in the medical domain is lacking. To address this gap, we introduce the Medical
Retrieval-Augmented Generation (MRAG)
benchmark, covering various tasks in English and Chinese
languages, and building a corpus with Wikipedia and Pubmed. Additionally, we develop the MRAG-Toolkit, facilitating systematic exploration of different RAG components. Our experiments reveal that: (a) RAG enhances LLM reliability across MRAG tasks. (b) the performance of RAG systems is influenced by
retrieval approaches, model sizes, and prompting strategies. (c) While RAG improves usefulness and
reasoning quality, LLM responses may become slightly less readable for long-form questions. We will release the MRAG-Bench's
dataset and toolkit with CCBY-4.0 license upon acceptance, to facilitate applications from both academia and industry.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configuration
Xinshuai Guo, Jiayi Kuang, Linyue Pan et al. (+6 more)
A reliable executable environment is the foundation for ensuring that large
language models solve software engineering tasks. Due to the complex and tedious construction process, large-scale configuration is relatively inefficient. However, most methods always overlook fine-grained analysis of the actions performed by the
agent, making it difficult to handle complex errors and resulting in configuration failures. To address this bottleneck, we propose EvoConfig, an efficient environment configuration framework that optimizes multi-
agent collaboration to build correct runtime environments. EvoConfig features an expert diagnosis module for fine-grained post-execution analysis, and a self-evolving mechanism that lets expert
agents self-feedback and dynamically adjust error-fixing priorities in real time. Empirically, EvoConfig matches the previous state-of-the-art Repo2Run on Repo2Run's 420 repositories, while delivering clear gains on harder cases: on the more challenging Envbench, EvoConfig achieves a 78.1% success rate, outperforming Repo2Run by 7.1%. Beyond end-to-end success, EvoConfig also demonstrates stronger debugging competence, achieving higher accuracy in error identification and producing more effective repair recommendations than existing methods.
Read the full abstract ▾
All authors Xinshuai Guo, Jiayi Kuang, Linyue Pan, Yinghui Li, Yangning Li, Hai-Tao Zheng, Ying Shen, Di Yin, Xing Sun
Computation and Language
Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic
Yichuan Ma, Linyang Li, Yongkang chen et al. (+5 more)
As large
language models (LLMs) increasingly tackle complex
reasoning tasks, test-time scaling has become critical for enhancing capabilities. However, in agentic scenarios with frequent tool calls, the traditional generation-length-based definition breaks down: tool latency decouples inference time from generation length. We propose Timely Machine, redefining test-time as wall-clock time, where models dynamically adjust strategies based on time budgets. We introduce Timely-Eval, a
benchmark spanning high-frequency tool calls, low-frequency tool calls, and time-constrained
reasoning. By varying tool latency, we find smaller models excel with fast feedback through more interactions, while larger models dominate high-latency settings via superior interaction quality. Moreover, existing models fail to adapt
reasoning to time budgets. We propose Timely-RL to address this gap. After cold-start supervised fine-tuning, we use reinforcement learning to enhance temporal planning. Timely-RL improves time budget awareness and consistently boosts performance across Timely-Eval. We hope our work offers a new perspective on test-time scaling for the agentic era.
Read the full abstract ▾
All authors Yichuan Ma, Linyang Li, Yongkang chen, Peiji Li, Xiaozhe Li, Qipeng Guo, Dahua Lin, Kai Chen
Computation and Language
TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization
Peiji Li, Linyang Li, Handa Sun et al. (+15 more)
Large
language models have demonstrated strong
reasoning capabilities in complex tasks through tool integration, which is typically framed as a Markov Decision Process and optimized with trajectory-level RL
algorithms such as GRPO. However, a common class of
reasoning tasks, iterative optimization, presents distinct challenges: the
agent interacts with the same underlying environment state across turns, and the value of a trajectory is determined by the best turn-level reward rather than cumulative returns. Existing GRPO-based methods cannot perform fine-grained, turn-level optimization in such settings, while black-box optimization methods discard prior knowledge and
reasoning capabilities. To address this gap, we propose Turn-Level GRPO (TL-GRPO), a lightweight RL
algorithm that performs turn-level group sampling for fine-grained optimization. We evaluate TL-GRPO on analog
circuit sizing (ACS), a challenging scientific optimization task requiring multiple simulations and domain expertise. Results show that TL-GRPO outperforms standard GRPO and
Bayesian optimization methods across various specifications. Furthermore, our 30B model trained with TL-GRPO achieves state-of-the-art performance on ACS tasks under same simulation budget, demonstrating both strong generalization and practical utility.
Read the full abstract ▾
All authors Peiji Li, Linyang Li, Handa Sun, Wenjin Mai, Yongkang Chen, Xiaozhe Li, Yue Shen, Yichuan Ma, Yiliu Sun, Jiaxi Cao, Zhishu He, Bo Wang, Xiaoqing Zheng, Zhaori Bi, Xipeng Qiu, Qipeng Guo, Kai Chen, Dahua Lin
Computation and Language
DeepEra: A Deep Evidence Reranking Agent for Scientific Retrieval-Augmented Generated Question Answering
Haotian Chen, Qingqing Long, Siyu Pu et al. (+6 more)
With the rapid growth of scientific literature, scientific question answering (SciQA) has become increasingly critical for exploring and utilizing scientific knowledge.
Retrieval-Augmented Generation (RAG) enhances LLMs by incorporating knowledge from external sources, thereby providing credible evidence for scientific question answering. But existing
retrieval and reranking methods remain vulnerable to passages that are semantically similar but logically irrelevant, often reducing factual reliability and amplifying
hallucinations.To address this challenge, we propose a Deep Evidence Reranking
Agent (DeepEra) that integrates step-by-step
reasoning, enabling more precise evaluation of candidate passages beyond surface-level semantics. To support systematic evaluation, we construct SciRAG-SSLI (Scientific RAG - Semantically Similar but Logically Irrelevant), a large-scale
dataset comprising about 300K SciQA instances across 10 subjects, constructed from 10M scientific corpus. The
dataset combines naturally retrieved contexts with systematically generated distractors to test logical robustness and factual grounding. Comprehensive evaluations confirm that our approach achieves superior
retrieval performance compared to leading rerankers. To our knowledge, this work is the first to comprehensively study and empirically validate innegligible SSLI issues in two-stage RAG frameworks.
Read the full abstract ▾
All authors Haotian Chen, Qingqing Long, Siyu Pu, Xiao Luo, Wei Ju, Meng Xiao, Yuanchun Zhou, Jianghua Zhao, Xuezhi Wang
Computation and Language
Persona Jailbreaking in Large Language Models
Jivnesh Sandhan, Fei Cheng, Tushar Sandhan et al. (+1 more)
Large
Language Models (LLMs) are increasingly deployed in domains such as education, mental health and customer support, where stable and consistent personas are critical for reliability. Yet, existing studies focus on narrative or role-playing tasks and overlook how
adversarial conversational history alone can reshape induced personas. Black-box persona manipulation remains unexplored, raising concerns for robustness in realistic interactions. In response, we introduce the task of persona editing, which adversarially steers LLM traits through user-side inputs under a black-box, inference-only setting. To this end, we propose PHISH (Persona Hijacking via Implicit Steering in History), the first framework to expose a new vulnerability in LLM safety that embeds semantically loaded cues into user queries to gradually induce reverse personas. We also define a metric to quantify attack success. Across 3
benchmarks and 8 LLMs, PHISH predictably shifts personas, triggers collateral changes in correlated traits, and exhibits stronger effects in multi-turn settings. In high-risk domains mental health, tutoring, and customer support, PHISH reliably manipulates personas, validated by both human and LLM-as-Judge evaluations. Importantly, PHISH causes only a small reduction in
reasoning benchmark performance, leaving overall utility largely intact while still enabling significant persona manipulation. While current guardrails offer partial protection, they remain brittle under sustained attack. Our findings expose new vulnerabilities in personas and highlight the need for context-resilient persona in LLMs. Our codebase and
dataset is available at: https://github.com/Jivnesh/PHISH
Read the full abstract ▾
All authors Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, Yugo Murawaki
Computation and Language
Finding What Matters: Anchoring Context Knowledge with Evolving Indices for Iterative Retrieval
Mingyan Wu, Zhenghao Liu, Xinze Li et al. (+7 more)
Retrieval-Augmented Generation (RAG) has become a dominant paradigm for mitigating
hallucinations in Large
Language Models (LLMs) by incorporating external knowledge. However, existing RAG systems often struggle to effectively integrate and reason over key evidence scattered across noisy retrieved documents, particularly in multi-hop scenarios. In this paper, we propose KAIR, a Knowledge Anchoring framework for Iterative
Retrieval that anchors knowledge within retrieved knowledge to guide LLMs to locate the key information. During iterative
retrieval, KAIR progressively updates the knowledge index to anchor salient evidence from retrieved documents. The evolving index serves as a navigational anchoring index that enables the LLM to assess knowledge sufficiency and formulate subsequent
retrieval queries. Finally, KAIR generates answers by jointly leveraging the retrieved documents and the finalized anchoring index. Experiments on four multi-hop question answering
benchmarks demonstrate that KAIR consistently outperforms strong RAG baselines. Further analysis shows that KAIR effectively anchors key knowledge and alleviates the context noise during iterative
retrieval, improving the LLM's ability to associate and reason over dispersed evidence across retrieved documents. All code and data are available at https://github.com/NEUIR/KAIR.
Read the full abstract ▾
All authors Mingyan Wu, Zhenghao Liu, Xinze Li, Yuqing Lan, Yukun Yan, Shuo Wang, Cheng Yang, Minghe Yu, Zheni Zeng, Maosong Sun
Computation and Language
Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of Go
Yichuan Ma, Linyang Li, Yongkang Chen et al. (+5 more)
Large
language models (LLMs) have demonstrated exceptional performance in
reasoning tasks such as mathematics and coding, matching or surpassing human capabilities. However, these impressive
reasoning abilities face significant challenges in specialized domains. Taking Go as an example, although AlphaGo has established the high performance ceiling of AI systems in Go, mainstream LLMs still struggle to reach even beginner-level proficiency, let alone perform natural
language reasoning. This performance gap between general-purpose LLMs and domain experts is significantly limiting the application of LLMs on a wider range of domain-specific tasks. In this work, we aim to bridge the divide between LLMs' general
reasoning capabilities and expert knowledge in domain-specific tasks. We perform mixed fine-tuning with structured Go expertise and general long Chain-of-Thought (CoT)
reasoning data as a cold start, followed by reinforcement learning to integrate expert knowledge in Go with general
reasoning capabilities. Through this methodology, we present LoGos, a powerful LLM that not only maintains outstanding general
reasoning abilities, but also conducts Go gameplay in natural
language, demonstrating effective strategic
reasoning and accurate next-move prediction. LoGos achieves performance comparable to human professional players, substantially surpassing all existing LLMs. Through this work, we aim to contribute insights on applying general LLM
reasoning capabilities to specialized domains. We will release the first large-scale Go
dataset for LLM training, the first LLM Go evaluation
benchmark, and the first general LLM that reaches human professional-level performance in Go at: https://github.com/Entarochuan/LoGos.
Read the full abstract ▾
All authors Yichuan Ma, Linyang Li, Yongkang Chen, Peiji Li, Jiasheng Ye, Qipeng Guo, Dahua Lin, Kai Chen
Computation and Language
Exploring the Effects of Alignment on Numerical Bias in Large Language Models
Ayako Sato, Hwichan Kim, Zhousi Chen et al. (+2 more)
"LLM-as-a-judge," which utilizes large
language models (LLMs) as evaluators, has proven effective in many evaluation tasks. However, evaluator LLMs exhibit numerical bias, a phenomenon where certain evaluation scores are generated disproportionately often, leading reduced evaluation performance. This study investigates the cause of this bias. Given that most evaluator LLMs are aligned through instruction tuning and preference tuning, and that prior research suggests
alignment reduces output diversity, we hypothesize that numerical bias arises from
alignment. To test this, we compare outputs from pre- and post-
alignment LLMs, and observe that
alignment indeed increases numerical bias. We also explore mitigation strategies for post-
alignment LLMs, including temperature scaling, distribution calibration, and score range adjustment. Among these, score range adjustment is most effective in reducing bias and improving performance, though still heuristic. Our findings highlight the need for further work on optimal score range selection and more robust mitigation strategies.
Read the full abstract ▾
All authors Ayako Sato, Hwichan Kim, Zhousi Chen, Masato Mita, Mamoru Komachi
Computation and Language · Cross-listed from cs.LG
Endless Terminals: Scaling RL Environments for Terminal Agents
Kanishk Gandhi, Shivam Garg, Noah D. Goodman et al. (+1 more)
Environments are the bottleneck for self-improving
agents. Current terminal
benchmarks were built for evaluation, not training; reinforcement learning requires a scalable pipeline, not just a
dataset. We introduce Endless Terminals, a fully autonomous pipeline that procedurally generates terminal-use tasks without human annotation. The pipeline has four stages: generating diverse task descriptions, building and validating containerized environments, producing completion tests, and filtering for solvability. From this pipeline we obtain 3255 tasks spanning file operations, log management, data processing, scripting, and database operations. We train
agents using vanilla PPO with binary episode level rewards and a minimal interaction loop: no
retrieval, multi-
agent coordination, or specialized tools. Despite this simplicity, models trained on Endless Terminals show substantial gains: on our held-out dev set, Llama-3.2-3B improves from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0%. These improvements transfer to human-curated
benchmarks: models trained on Endless Terminals show substantial gains on held out human curated
benchmarks: on TerminalBench 2.0, Llama-3.2-3B improves from 0.0% to 2.2%, Qwen2.5-7B from 2.2% to 3.4%, and Qwen3-8B-openthinker-sft from 1.1% to 6.7%, in each case outperforming alternative approaches including models with more complex agentic scaffolds. These results demonstrate that simple RL succeeds when environments scale.
Read the full abstract ▾
All authors Kanishk Gandhi, Shivam Garg, Noah D. Goodman, Dimitris Papailiopoulos
Computation and Language
Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuning
Qinglong Cao, Yuntian Chen, Chao Ma et al. (+1 more)
Multimodal large
language models (MLLMs) have shown remarkable capabilities in
multimodal perception and understanding tasks. However, their effectiveness in specialized domains, such as remote sensing and medical imaging, remains limited. A natural approach to domain adaptation is to inject domain knowledge through textual instructions,
prompts, or auxiliary captions. Surprisingly, we find that such input-level domain knowledge injection yields little to no improvement on scientific
multimodal tasks, even when the domain knowledge is explicitly provided. This observation suggests that current MLLMs fail to internalize domain-specific priors through
language alone, and that domain knowledge must be integrated at the optimization level. Motivated by this insight, we propose a reinforcement fine-tuning framework that incorporates domain knowledge directly into the learning objective. Instead of treating domain knowledge as descriptive information, we encode it as domain-informed constraints and reward signals, shaping the model's behavior in the output space. Extensive experiments across multiple
datasets in remote sensing and medical domains consistently demonstrate good performance gains, achieving state-of-the-art results on
multimodal domain tasks. Our results highlight the necessity of optimization-level domain knowledge integration and reveal a fundamental limitation of textual domain conditioning in current MLLMs.
Read the full abstract ▾
All authors Qinglong Cao, Yuntian Chen, Chao Ma, Xiaokang Yang
Computation and Language
Jacobian Scopes: A Unified Geometric Framework for Token-Level LLM Attribution
Toni J.B. Liu, Baran Zadeoğlu, Gurbir Arora et al. (+3 more)
Large
language models (LLMs) make next-
token predictions based on clues present in their context, such as semantic descriptions and in-context examples. Yet, elucidating which prior
tokens most strongly influence a given prediction remains challenging due to the proliferation of layers and
attention heads in modern architectures. We propose Jacobian Scopes, a suite of gradient-based,
token level causal attribution methods for interpreting LLM predictions. Grounded in perturbation theory and information geometry, Jacobian Scopes quantify how input
tokens influence various aspects of a model's prediction, such as specific logits, the full predictive distribution, and model uncertainty (effective temperature). Through case studies spanning instruction understanding, translation, and in-context learning (ICL), we demonstrate how Jacobian Scopes reveal implicit political biases, uncover word- and phrase-level translation strategies, and shed light on recently debated mechanisms underlying in-context time-series
forecasting. On
token-ablation faithfulness
benchmarks (LAMBADA and IWSLT2017) across six LLMs, the distribution-level Fisher and Temperature Scopes significantly outperform established gradient-based attribution methods in 10 of 12 model-
dataset settings. To facilitate exploration of Jacobian Scopes on custom text, we open-source our implementations and provide a cloud-hosted interactive demo at https://huggingface.co/spaces/Typony/JacobianScopes.
Read the full abstract ▾
All authors Toni J.B. Liu, Baran Zadeoğlu, Gurbir Arora, Raphaël Sarfati, Nicolas Boullé, Christopher J. Earls
Computation and Language
Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification
Zongwan Cao, Bingbing Wen and Lucy Lu Wang
Real-world visual question answering (VQA) may be context-dependent: an image-question pair could be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose Clarify-or-Answer (CoA), an ask-or-answer
agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it generates one or more clarification questions to iteratively enrich the context until ambiguity is resolved. To validate this framework, we introduce CONTEXTCLARIFY, a
dataset of ambiguous VQA questions along with a contrast set of non-ambiguous questions. We further introduce GRPO-CR(Clarification
Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLMs and three
datasets, CoA achieves consistent improvements at both the module and system levels, with a single clarification turn providing substantial gains and recursive multi-turn execution offering modest additional improvements in select settings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
White-Box Sensitivity Auditing with Steering Vectors
Hannah Cyberey, Yangfeng Ji and David Evans
Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators. Current audits of large
language models (LLMs) primarily rely on black-box evaluations that assess model behavior only through input-output testing. These methods are limited to tests constructed in the input space, often generated by heuristics. In addition, many socially relevant model properties (e.g., gender bias) are abstract and difficult to measure through text-based inputs alone. To address these limitations, we propose a white-box sensitivity
auditing framework for LLMs that leverages activation steering to conduct more rigorous assessments through model internals. Our
auditing method conducts internal sensitivity tests by manipulating key concepts relevant to the model's intended function for the task. We demonstrate its application to bias audits in four simulated high-stakes LLM decision tasks. Our method consistently indicates substantial dependence on protected attributes in model predictions, even in settings where standard black-box evaluations suggest little or no bias. Our code is openly available at https://github.com/hannahxchen/llm-steering-audit
Read the full abstract ▾
Computation and Language
From Attribution to Abstention: Training-Free Attention-Based Auditing for Clinical Summarization
Qianqi Yan, Huy Nguyen, Sumana Srivatsa et al. (+3 more)
Deploying
multimodal large
language models (MLLMs) for clinical summarization demands not only fluent generation but also transparency about where each statement originates-and a mechanism to flag when statements lack evidential support. We present ClinTrace, a training-free framework that extracts two clinically useful signals from the decoder
attention weights that every
transformer-based MLLM already produces during generation: (i) fine-grained source attributions linking each output sentence to supporting text spans or images, and (ii) per-sentence groundedness scores that identify poorly supported claims as candidate
hallucinations. Both signals are derived from the same
attention tensors in a single pass, requiring no retraining, no auxiliary models, and no additional inference cost. We evaluate on two clinical summarization tasks: doctor-patient dialogue summarization (CliConSummation) and radiology report summarization (MIMIC-CXR) using a general-purpose MLLM (Qwen3-8B) and a medical-finetuned model (HuatuoGPT-
Vision-7B). For source attribution, ClinTrace achieves over 92% text F1 on radiology and 88% on dialogue summarization, substantially outperforming
embedding-based and self-attribution baselines. For
hallucination detection, groundedness scores achieve 0.77 AUROC with the medical-finetuned model: competitive with
embedding-based confidence at zero additional cost, and enable an abstention mechanism that improves faithfulness from 61.7% to 72.6% by withholding the least: grounded 20% of output for clinician review. Notably, medical finetuning substantially improves the reliability of
attention-based
hallucination detection, suggesting that domain adaptation produces more semantically structured
attention patterns amenable to self-
auditing.
Read the full abstract ▾
All authors Qianqi Yan, Huy Nguyen, Sumana Srivatsa, Hari Bandi, Xin Eric Wang, Krishnaram Kenthapadi
Computation and Language
Cross-Lingual Activation Steering for Multilingual Language Models
Rhitabrat Pokharel, Ameeta Agrawal and Tanay Nagar
Large
language models exhibit strong
multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant
languages. Prior work attributes this gap to imbalances between shared and
language-specific
neurons in
multilingual representations. We propose Cross-Lingual Activation Steering (CLAS), a training-free inference-time intervention that selectively modulates
neuron activations. We evaluate CLAS on classification and generation
benchmarks, achieving average improvements of 2.3% (Acc.) and 3.4% (F1) respectively, while maintaining high-resource
language performance. We discover that effective transfer operates through functional divergence rather than strict
alignment; performance gains correlate with increased
language cluster separation. Our results demonstrate that targeted activation steering can unlock latent
multilingual capacity in existing models without modification to model weights.
Read the full abstract ▾