Computation and Language
LANGSAE EDITING: Improving Multilingual Information Retrieval via Post-hoc Language Identity Removal
Dongjun Kim, Jeongho Yoon, Chanjun Park et al. (+1 more)
Dense
retrieval in
multilingual settings often searches over mixed-
language collections, yet
multilingual embeddings encode
language identity alongside semantics. This
language signal can inflate similarity for same-
language pairs and crowd out relevant evidence written in other
languages. We propose LANGSAE EDITING, a post-hoc sparse autoencoder trained on pooled
embeddings that enables controllable removal of
language-identity signal directly in vector space. The method identifies
language-associated latent units using cross-
language activation statistics, suppresses these units at inference time, and reconstructs
embeddings in the original dimensionality, making it compatible with existing vector databases without retraining the base encoder or re-encoding raw text. Experiments across multiple
languages show consistent improvements in ranking quality and cross-
language coverage, with especially strong gains for script-distinct
languages.
Read the full abstract ▾
All authors Dongjun Kim, Jeongho Yoon, Chanjun Park, Heuiseok Lim
Computation and Language · Cross-listed from cs.AI
AT$^2$PO: Agentic Turn-based Policy Optimization via Tree Search
Zefang Zong, Dingwei Chen, Yang Li et al. (+6 more)
LLM
agents have emerged as powerful systems for tackling multi-turn tasks by interleaving internal
reasoning and external tool interactions. Agentic Reinforcement Learning has recently drawn significant research
attention as a critical post-training paradigm to further refine these capabilities. In this paper, we present AT$^2$PO (Agentic Turn-based Policy Optimization via Tree Search), a unified framework for multi-turn agentic RL that addresses three core challenges: limited exploration diversity, sparse credit assignment, and misaligned policy optimization. AT$^2$PO introduces a turn-level tree structure that jointly enables
Entropy-Guided Tree Expansion for strategic exploration and Turn-wise Credit Assignment for fine-grained reward propagation from sparse outcomes. Complementing this, we propose Agentic Turn-based Policy Optimization, a turn-level learning objective that aligns policy updates with the natural decision granularity of agentic interactions. ATPO is orthogonal to tree search and can be readily integrated into any multi-turn RL pipeline. Experiments across seven
benchmarks demonstrate consistent improvements over the state-of-the-art baseline by up to 1.84 percentage points in average, with ablation studies validating the effectiveness of each component. Our code is available at https://github.com/zzfoutofspace/ATPO.
Read the full abstract ▾
All authors Zefang Zong, Dingwei Chen, Yang Li, Qi Yi, Bo Zhou, Chengming Li, Bo Qian, Peng Chen, Jie Jiang
Computation and Language
JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergence
Shengyin Sun, Yiming Li, Renxi Liu et al. (+5 more)
Judge Decoding accelerates LLM inference by relaxing the strict verification of Speculative Decoding, yet it typically relies on expensive and noisy supervision. In this work, we revisit this paradigm from first principles, revealing that the “criticality” scores learned via costly supervision are intrinsically encoded in the draft-target distributional divergence. We theoretically prove a structural correspondence between learned linear judges and Kullback-Leibler (KL) divergence, demonstrating they rely on the same underlying logit primitives. Guided by this, we propose a simple, training-free verification mechanism based on KL divergence. Extensive experiments across
reasoning and coding
benchmarks show that our method matches or outperforms complex trained judges (e.g., AutoJudge), offering superior robustness to domain shifts and eliminating the supervision bottleneck entirely. Code is available at https://github.com/sunshy-1/JuDi
Read the full abstract ▾
All authors Shengyin Sun, Yiming Li, Renxi Liu, Weizhe Lin, Hui-Ling Zhen, Xianzhi Yu, Mingxuan Yuan, Chen Ma
Computation and Language
Differential syntactic and semantic encoding in LLMs
Santiago Acevedo, Alessandro Laio and Marco Baroni
We study how syntactic and semantic information is encoded in inner layer representations of Large
Language Models (LLMs), focusing on the very large DeepSeek-V3. We find that, by averaging hidden-representation vectors of sentences sharing syntactic structure or meaning, we obtain vectors that capture a significant proportion of the syntactic and semantic information contained in the representations. In particular, subtracting these syntactic and semantic “centroids” from sentence vectors strongly affects their similarity with syntactically and semantically matched sentences, respectively, suggesting that syntax and semantics are, at least partially, linearly encoded. We also find that the cross-layer encoding profiles of syntax and semantics are different, and that the two signals can to some extent be decoupled, suggesting differential encoding of these two types of linguistic information in LLM representations.
Read the full abstract ▾
Computation and Language
PILOT-Bench: A Benchmark for Legal Reasoning in the Patent Domain with IRAC-Aligned Classification Tasks
Yehoon Jang, Chaewon Lee, Hyun-seok Min et al. (+1 more)
The Patent Trial and Appeal Board (PTAB) of the USPTO adjudicates thousands of ex parte appeals each year, requiring the integration of technical understanding and legal
reasoning. While large
language models (LLMs) are increasingly applied in patent and legal practice, their use has remained limited to lightweight tasks, with no established means of systematically evaluating their capacity for structured legal
reasoning in the patent domain. In this work, we introduce PILOT-Bench, the first PTAB-centric
benchmark that aligns PTAB decisions with USPTO patent data at the case-level and formalizes three IRAC-aligned classification tasks: Issue Type, Board Authorities, and Subdecision. We evaluate a diverse set of closed-source (commercial) and open-source LLMs and conduct analyses across multiple perspectives, including input-variation settings, model families, and error tendencies. Notably, on the Issue Type task, closed-source models consistently exceed 0.75 in Micro-F1 score, whereas the strongest open-source model (Qwen-8B) achieves performance around 0.56, highlighting a substantial gap in
reasoning capabilities. PILOT-Bench establishes a foundation for the systematic evaluation of patent-domain legal
reasoning and points toward future directions for improving LLMs through
dataset design and model
alignment. All data, code, and
benchmark resources are available at https://github.com/TeamLab/pilot-bench.
Read the full abstract ▾
All authors Yehoon Jang, Chaewon Lee, Hyun-seok Min, Sungchul Choi
Computation and Language
Tool-MAD: A Multi-Agent Debate Framework for Fact Verification with Diverse Tool Augmentation and Adaptive Retrieval
Seyeon Jeong, Yeonjun Choi, JongWook Kim et al. (+1 more)
Large
Language Models (LLMs) suffer from
hallucinations and factual inaccuracies, especially in complex
reasoning and fact verification tasks. Multi-
Agent Debate (MAD) systems aim to improve answer accuracy by enabling multiple LLM
agents to engage in dialogue, promoting diverse
reasoning and mutual verification. However, existing MAD frameworks primarily rely on internal knowledge or static documents, making them vulnerable to
hallucinations. While MADKE introduces external evidence to mitigate this, its one-time
retrieval mechanism limits adaptability to new arguments or emerging information during the debate. To address these limitations, We propose Tool-MAD, a multi-
agent debate framework that enhances factual verification by assigning each
agent a distinct external tool, such as a search API or RAG module. Tool-MAD introduces three key innovations: (1) a multi-
agent debate framework where
agents leverage heterogeneous external tools, encouraging diverse perspectives, (2) an adaptive query formulation mechanism that iteratively refines evidence
retrieval based on the flow of the debate, and (3) the integration of Faithfulness and Answer Relevance scores into the final decision process, allowing the Judge
agent to quantitatively assess the coherence and question
alignment of each response and effectively detect
hallucinations. Experimental results on four fact verification
benchmarks demonstrate that Tool-MAD consistently outperforms state-of-the-art MAD frameworks, achieving up to 5.5% accuracy improvement. Furthermore, in medically specialized domains, Tool-MAD exhibits strong robustness and adaptability across various tool configurations and domain conditions, confirming its potential for broader real-world fact-checking applications.
Read the full abstract ▾
All authors Seyeon Jeong, Yeonjun Choi, JongWook Kim, Beakcheol Jang
Computation and Language
StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation
Huawei Zheng, Xinqi Jiang, Sen Yang et al. (+3 more)
Large
language models (LLMs) are increasingly applied in specialized domains such as finance and healthcare, where they introduce unique safety risks. Domain-specific
datasets of harmful
prompts remain scarce and still largely rely on manual construction; public
datasets mainly focus on explicit harmful
prompts, which modern LLM defenses can often detect and refuse. In contrast, implicit harmful
prompts-expressed through indirect domain knowledge-are harder to detect and better reflect real-world threats. We identify two challenges: transforming domain knowledge into actionable constraints and increasing the implicitness of generated harmful
prompts. To address them, we propose an end-to-end framework that first performs knowledge-graph-guided harmful
prompt generation to systematically produce domain-relevant
prompts, and then applies two-strategy obfuscation rewriting to convert explicit harmful
prompts into implicit variants via direct and context-enhanced rewriting. This framework yields high-quality
datasets combining strong domain relevance with implicitness, enabling more realistic red-teaming and advancing LLM safety research. We release our code and
datasets on GitHub.
Read the full abstract ▾
All authors Huawei Zheng, Xinqi Jiang, Sen Yang, Shouling Ji, Yingcai Wu, Dazhen Deng
Computation and Language
Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignment
Han Zhu, Jiale Chen, Chengkun Cai et al. (+8 more)
Despite remarkable capability in multi-modal understanding, deploying Multi-modal Large
Language Models (MLLMs) in open-ended conversational scenarios introduces safety risks that remain poorly addressed by existing
alignment methods. Unlike simple malicious visual question and answer (VQA) pairs , multi-turn interactions enable adversaries to incrementally reconstruct harmful intent across dialogues, progressively bypassing safety constraints in ways that are difficult to detect at any individual turn. Meanwhile, conventional reinforcement learning from human feedback (RLHF) approaches are unsuitable for this situation: designed primarily for VQA tasks, they neither capture cross-turn risk dynamics nor scale efficiently without costly manual preference annotation. To close this gap, we introduce MINT-Safe, an open-source visual multi-turn training
dataset comprising 11,270 multi-image dialogues and 500 refusal VQA pairs, constructed via multi-
agent interaction with text-to-image (T2I) tool-call augmentation. Building on MINT-Safe, we propose TAD-Align, a dialogue safety
alignment framework centered on a turn-aware dual-objective reward function. Rather than treating all dialogue turns uniformly, TAD-Align leverages rollout-based safety score variance to dynamically identify turns where the model exhibits inconsistent safety behavior, and adaptively up-weights these turns during optimization. Experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-7B demonstrate reductions of over 10% in Attack Success Rate (ASR), alongside improvements of at least 8% in harmlessness and 13% in helpfulness on multi-modal multi-turn safety
benchmarks, while preserving general model capabilities.
Read the full abstract ▾
All authors Han Zhu, Jiale Chen, Chengkun Cai, Shengjie Sun, Haoran Li, Yujin Zhou, Chi-Min Chan, Pengcheng Wen, Lei Li, Yike Guo, Sirui Han
Computation and Language · Cross-listed from cs.AI
Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Models
Shuyang Jiang, Yuhao Wang, Ya Zhang et al. (+2 more)
Current critic-free RL methods for large
reasoning models suffer from severe inefficiency when training on positive homogeneous
prompts (where all rollouts are correct), resulting in waste of rollouts due to zero advantage estimates. We introduce a radically simple yet powerful solution to \uline{M}ine \uline{in}trinsic mast\uline{er}y (Miner), that repurposes the policy's intrinsic uncertainty as a self-supervised reward signal, with no external supervision, auxiliary models, or additional inference cost. Our method pioneers two key innovations: (1) a
token-level focal credit assignment mechanism that dynamically amplifies gradients on critical uncertain
tokens while suppressing overconfident ones, and (2) adaptive advantage calibration to seamlessly integrate intrinsic and verifiable rewards. Evaluated across six
reasoning benchmarks on Qwen3-4B and Qwen3-8B base models, Miner achieves state-of-the-art performance among the other four
algorithms, yielding up to 4.58 absolute gains in Pass@1 and 6.66 gains in Pass@K compared to GRPO. Comparison with other methods targeted at exploration enhancement further discloses the superiority of the two newly proposed innovations. This demonstrates that latent uncertainty exploitation is both necessary and sufficient for efficient and scalable RL training of
reasoning models. Code is available at https://github.com/pixas/Miner.
Read the full abstract ▾
All authors Shuyang Jiang, Yuhao Wang, Ya Zhang, Yanfeng Wang, Yu Wang
Computation and Language
Automatic Classifiers Underdetect Emotions Expressed by Men
Ivan Smirnov, Segun T. Aroyehun, Paul Plener et al. (+1 more)
The widespread adoption of automatic sentiment and
emotion classifiers makes it important to ensure that these tools perform reliably across different populations. Yet their reliability is typically assessed using
benchmarks that rely on third-party annotators rather than the individuals experiencing the
emotions themselves, potentially concealing systematic biases. In this paper, we use a unique, large-scale
dataset of more than one million self-annotated posts and a pre-registered research design to investigate gender biases in
emotion detection across 414 combinations of models and
emotion-related classes. We find that across different types of automatic classifiers and various underlying
emotions, error rates are consistently higher for texts authored by men compared to those authored by women. We quantify how this bias could affect results in downstream applications and show that current machine learning tools, including large
language models, should be applied with caution when the gender composition of a sample is not known or variable. Our findings demonstrate that sentiment analysis is not yet a solved problem, especially in ensuring equitable model behaviour across demographic groups.
Read the full abstract ▾
All authors Ivan Smirnov, Segun T. Aroyehun, Paul Plener, David Garcia
Computation and Language · Cross-listed from cs.AI
Memory Matters More: Event-Centric Memory as a Logic Map for Agent Searching and Reasoning
Yuyang Hu, Jiongnan Liu, Jiejun Tan et al. (+2 more)
Large
language models (LLMs) are increasingly deployed as intelligent
agents that reason, plan, and interact with their environments. To effectively scale to long-horizon scenarios, a key capability for such
agents is a
memory mechanism that can retain, organize, and retrieve past experiences to support downstream decision-making. However, most existing approaches organize and store
memories in a flat manner and rely on simple similarity-based
retrieval techniques. Even when structured
memory is introduced, existing methods often struggle to explicitly capture the logical relationships among experiences or
memory units. Moreover,
memory access is largely detached from the constructed structure and still depends on shallow semantic
retrieval, preventing
agents from
reasoning logically over long-horizon dependencies. In this work, we propose CompassMem, an event-centric
memory framework inspired by Event
Segmentation Theory. CompassMem organizes
memory as an Event Graph by incrementally segmenting experiences into events and linking them through explicit logical relations. This graph serves as a logic map, enabling
agents to perform structured and goal-directed
navigation over
memory beyond superficial
retrieval, progressively gathering valuable
memories to support long-horizon
reasoning. Experiments on LoCoMo and NarrativeQA demonstrate that CompassMem consistently improves both
retrieval and
reasoning performance across multiple backbone models.
Read the full abstract ▾
All authors Yuyang Hu, Jiongnan Liu, Jiejun Tan, Yutao Zhu, Zhicheng Dou
Computation and Language
Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
Mingxin Li, Yanzhao Zhang, Dingkun Long et al. (+9 more)
In this report, we introduce the Qwen3-VL-
Embedding and Qwen3-VL-Reranker model series, the latest extensions of the Qwen family built on the Qwen3-VL foundation model. Together, they provide an end-to-end pipeline for high-precision
multimodal search by mapping diverse modalities, including text, images, document images, and
video, into a unified representation space. The Qwen3-VL-
Embedding model employs a multi-stage training paradigm, progressing from large-scale contrastive pre-training to reranking model
distillation, to generate semantically rich high-dimensional vectors. It supports Matryoshka Representation Learning, enabling flexible
embedding dimensions, and handles inputs up to 32k
tokens. Complementing this, Qwen3-VL-Reranker performs fine-grained relevance estimation for query-document pairs using a cross-encoder architecture with cross-
attention mechanisms. Both model series inherit the
multilingual capabilities of Qwen3-VL, supporting more than 30
languages, and are released in $\textbf{2B}$ and $\textbf{8B}$ parameter sizes to accommodate diverse deployment requirements. Empirical evaluations demonstrate that the Qwen3-VL-
Embedding series achieves state-of-the-art results across diverse
multimodal embedding evaluation
benchmarks. Specifically, Qwen3-VL-
Embedding-8B attains an overall score of $\textbf{77.8}$ on MMEB-V2, ranking first among all models (as of January 8, 2025). This report presents the architecture, training methodology, and practical capabilities of the series, demonstrating their effectiveness on various
multimodal retrieval tasks, including image-text
retrieval, visual question answering, and
video-text matching.
Read the full abstract ▾
All authors Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, Junyang Lin
Computation and Language
Identifying and Mitigating Bottlenecks in Role-Playing Agents: A Systematic Study of Disentangling Character Profile Axes
Yonghyun Jun, Junhyuk Choi, Jeonghyun Park et al. (+3 more)
While Large
Language Model (LLM) role-playing
agents have advanced rapidly, it remains unclear which profile elements genuinely drive role-playing quality. To bridge this gap, we introduce a systematic diagnostic framework that disentangles the impact of character profiles along three axes: Familiarity (Known vs. Unknown), Structure (Structured vs. Unstructured), and Disposition (Moral vs. Immoral). Utilizing a unified hierarchical schema (5 dimensions, 28 fields), we construct a controlled
dataset of 211 personas and evaluate five LLMs on both single- and multi-turn interactions. Our results reveal a striking asymmetry: Familiarity and Structure show negligible impact, while Disposition produces large, consistent performance degradation for immoral characters across all conditions. Further analyses suggest that the Moral–Immoral gap is amplified by post-SFT
alignment, and that this degradation varies substantially across profile attributes. To mitigate this bottleneck, we propose Field-Aware Contrastive Decoding (FACD), a training-free strategy that amplifies suppressed disposition-sensitive signals, significantly closing the performance gap without sacrificing moral-character performance.
Read the full abstract ▾
All authors Yonghyun Jun, Junhyuk Choi, Jeonghyun Park, Jihyeong Park, Liu Nicole Geumheon, Hwanhee Lee
Computation and Language
DSC2025 – ViHallu Challenge: Detecting Hallucination in Vietnamese LLMs
Anh Thi-Hoang Nguyen, Khanh Quoc Tran, Tin Van Huynh et al. (+3 more)
The reliability of large
language models (LLMs) in production environments remains significantly constrained by their propensity to generate
hallucinations – fluent, plausible-sounding outputs that contradict or fabricate information. While
hallucination detection has recently emerged as a priority in English-centric
benchmarks, low-to-medium resource
languages such as Vietnamese remain inadequately covered by standardized evaluation frameworks. This paper introduces the DSC2025 ViHallu Challenge, the first large-scale shared task for detecting
hallucinations in Vietnamese LLMs. We present the ViHallu
dataset, comprising 10,000 annotated triplets of (context,
prompt, response) samples systematically partitioned into three
hallucination categories: no
hallucination, intrinsic, and extrinsic
hallucinations. The
dataset incorporates three
prompt types – factual, noisy, and
adversarial – to stress-test model robustness. A total of 111 teams participated, with the best-performing system achieving a macro-F1 score of 84.80%, compared to a baseline encoder-only score of 32.83%, demonstrating that instruction-tuned LLMs with structured prompting and ensemble strategies substantially outperform generic architectures. However, the gap to perfect performance indicates that
hallucination detection remains a challenging problem, particularly for intrinsic (contradiction-based)
hallucinations. This work establishes a rigorous
benchmark and explores a diverse range of detection methodologies, providing a foundation for future research into the trustworthiness and reliability of Vietnamese
language AI systems.
Read the full abstract ▾
All authors Anh Thi-Hoang Nguyen, Khanh Quoc Tran, Tin Van Huynh, Phuoc Tan-Hoang Nguyen, Cam Tan Nguyen, Kiet Van Nguyen
Computation and Language
Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning
Feihu Jin, Shipeng Cen and Ying Tan
Fine-tuning large
language models (LLMs) achieves strong performance but is often limited by the
memory overhead of backpropagation. Zeroth-order (ZO) optimization avoids this overhead by estimating gradients through forward passes alone, yet it typically converges slowly because random Gaussian perturbations yield high-variance gradient estimates in high-dimensional parameter spaces. In this paper, we propose a plug-and-play framework that turns random perturbations into more effective descent directions. The key idea is to draw a small pool of candidate perturbations, evaluate their loss values, and then select or combine those that are best aligned with the optimization objective. We develop two instantiations of this idea: MeZO-GV, which forms a guiding vector from the contrast between low-loss and high-loss perturbation groups, and MeZO-Greedy, which keeps the single best perturbation within a fixed evaluation budget. We theoretically show that both strategies yield a larger per-step reduction in the objective than standard ZO estimation, leading to improved convergence rates. Experiments on LLMs of different scales and architectures confirm that the proposed methods integrate naturally with existing ZO optimizers and consistently improve convergence speed and task accuracy. On OPT-13B, our approach outperforms all ZO baselines across 11
benchmarks and exceeds gradient-based methods on 9 of them, while retaining the
memory efficiency of forward-only optimization.
Read the full abstract ▾
Computation and Language
PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards
Mukesh Ghimire, Aosong Feng, Liwen You et al. (+3 more)
Current techniques for post-training Large
Language Models (LLMs) rely either on costly human supervision or on external verifiers to boost performance on tasks such as mathematical
reasoning and code generation. However, as LLMs improve their problem-solving, any further improvement will potentially require high-quality solutions to difficult problems that are not available to humans. As a result, learning from unlabeled data is becoming increasingly attractive in the research community. Existing methods extract learning signal from a model's consistency, either by majority voting or by converting the model's internal confidence into reward. Although internal consistency metric such as
entropy or self-certainty require no human intervention, as we show in this work, these are unreliable signals for large-scale and long-term training. To address the unreliability, we propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence in the absence of ground-truth labels. We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance, and also keep the model's internal confidence in check. Code available at https://github.com/ghimiremukesh/PRISM.
Read the full abstract ▾
All authors Mukesh Ghimire, Aosong Feng, Liwen You, Youzhi Luo, Fang Liu, Xuan Zhu
Computation and Language · Cross-listed from cs.AI
TourPlanner: A Competitive Consensus Framework with Constraint-Gated Reinforcement Learning for Travel Planning
Yinuo Wang, Mining Tan, Wenxiang Jiao et al. (+5 more)
Travel planning is a sophisticated decision-making process that requires synthesizing multifaceted information to construct itineraries. However, existing travel planning approaches face several challenges: (1)
Pruning candidate points of interest (POIs) while maintaining a high recall rate; (2) A single
reasoning path restricts the exploration capability within the feasible solution space for travel planning; (3) Simultaneously optimizing hard constraints and soft constraints remains a significant difficulty. To address these challenges, we propose TourPlanner, a comprehensive framework featuring multi-path
reasoning and constraint-gated reinforcement learning. Specifically, we first introduce a Personalized Recall and Spatial Optimization (PReSO) workflow to construct spatially-aware candidate POIs' set. Subsequently, we propose Competitive consensus Chain-of-Thought (CCoT), a multi-path
reasoning paradigm that improves the ability of exploring the feasible solution space. To further refine the plan, we integrate a sigmoid-based gating mechanism into the reinforcement learning stage, which dynamically prioritizes soft-constraint satisfaction only after hard constraints are met. Experimental results on travel planning
benchmarks demonstrate that TourPlanner achieves state-of-the-art performance, significantly surpassing existing methods in both feasibility and user-preference
alignment.
Read the full abstract ▾
All authors Yinuo Wang, Mining Tan, Wenxiang Jiao, Xiaoxi Li, Hao Wang, Xuanyu Zhang, Yuan Lu, Weiming Dong
Computation and Language · Cross-listed from cs.AI
A Method for Constructing a Digital Transformation Driving Mechanism Based on Semantic Understanding of Large Models
Huayi Liu
In the process of digital transformation, enterprises are faced with problems such as insufficient semantic understanding of unstructured data and lack of intelligent decision-making basis in driving mechanisms. This study proposes a method that combines a large
language model (LLM) and a knowledge graph. First, a fine-tuned BERT (Bidirectional Encoder Representations from
Transformers) model is used to perform entity recognition and relationship extraction on multi-source heterogeneous texts, and GPT-4 is used to generate semantically enhanced vector representations; secondly, a two-layer graph neural network (GNN) architecture is designed to fuse the semantic vectors output by LLM with business metadata to construct a dynamic and scalable enterprise knowledge graph; then reinforcement learning is introduced to optimize decision path generation, and the reward function is used to drive the mechanism iteration. In the case of the manufacturing industry, this mechanism reduced the response time for equipment failure scenarios from 7.8 hours to 3.7 hours, the F1 value reached 94.3%, and the compensation for decision errors in the annual digital transformation cost decreased by 45.3%. This method significantly enhances the intelligence level and execution efficiency of the digital transformation driving mechanism by integrating large model semantic understanding with structured knowledge.
Read the full abstract ▾
Computation and Language
Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding
Sungmok Jung, Yeonkyoung So, Joonhak Lee et al. (+3 more)
Although negation is known to challenge large
language models (LLMs),
benchmarks for evaluating negation understanding-especially in Korean-are scarce. We conduct a corpus-based analysis of Korean negation and show that LLM performance degrades under negation. We then introduce Thunder-KoNUBench, a sentence-level negation understanding
benchmark that reflects the empirical distribution of Korean negation phenomena. Evaluating 47 LLMs on Thunder-KoNUBench, we analyze the effects of model size and instruction tuning, and perform error analysis to better understand model behavior. We further show that fine-tuning on Thunder-KoNUBench improves negation understanding and broader contextual comprehension in Korean.
Read the full abstract ▾
All authors Sungmok Jung, Yeonkyoung So, Joonhak Lee, Sangho Kim, Yelim Ahn, Jaejin Lee
Computation and Language
PEST: Parameter Efficient Steering of Blackbox VLMs via Agentic Few-shot Alignment for Hateful Meme Moderation
Naquee Rizwan, Subhankar Swain, Paramananda Bhaskar et al. (+3 more)
In this work, we examine hateful memes from three complementary angles - how to detect them, how to explain their content and how to intervene them before being posted - by applying a range of strategies built on top of generative AI models. To the best of our knowledge, explanation and intervention have typically been studied separately from detection, which does not reflect real-world conditions. Further, since curating large annotated
datasets for meme moderation is prohibitively expensive, we propose a novel framework - PEST - that leverages task-specific generative VLMs and the few-shot adaptability of large VLMs to cater to different types of memes. We believe this is the first work focused on generalizable hateful meme moderation under limited data conditions, and has strong potential for deployment in real-world production scenarios. Warning: Contains potentially toxic contents.
Read the full abstract ▾
All authors Naquee Rizwan, Subhankar Swain, Paramananda Bhaskar, Shehryaar Shah Khan, Gagan Aryan, Animesh Mukherjee
Computation and Language
ToolGate: Contract-Grounded and Verified Tool Execution for LLMs
Yanming Liu, Xinyue Peng, Jiannan Cao et al. (+5 more)
Large
Language Models (LLMs) augmented with external tools have demonstrated remarkable capabilities in complex
reasoning tasks. However, existing frameworks rely heavily on natural
language reasoning to determine when tools can be invoked and whether their results should be committed, lacking formal guarantees for logical safety and verifiability. We present ToolGate, a forward execution framework that provides logical safety guarantees and verifiable state
evolution for LLM tool calling. ToolGate maintains an explicit symbolic state space as a typed key-value mapping representing trusted world information throughout the
reasoning process. Each tool is formalized as a Hoare-style contract consisting of a precondition and a postcondition, where the precondition gates tool invocation by checking whether the current state satisfies the required conditions, and the postcondition determines whether the tool's result can be committed to update the state through runtime verification. Our approach guarantees that the symbolic state evolves only through verified tool executions, preventing invalid or hallucinated results from corrupting the world representation. Experimental validation demonstrates that ToolGate significantly improves the reliability and verifiability of tool-augmented LLM systems while maintaining competitive performance on complex multi-step
reasoning tasks. This work establishes a foundation for building more trustworthy and debuggable AI systems that integrate
language models with external tools.
Read the full abstract ▾
All authors Yanming Liu, Xinyue Peng, Jiannan Cao, Xinyi Wang, Songhang Deng, Jintao Chen, Jianwei Yin, Xuhong Zhang
Computation and Language · Cross-listed from cs.CV
Agri-R1: Agricultural Reasoning for Disease Diagnosis via Automated-Synthesis and Reinforcement Learning
Wentao Zhang, Mingkun Xu, Qi Zhang et al. (+6 more)
Agricultural disease diagnosis challenges VLMs, as conventional fine-tuning requires extensive labels, lacks interpretability, and generalizes poorly. While
reasoning improves model robustness, existing methods rely on costly expert annotations and rarely address the open-ended, diverse nature of agricultural queries. To address these limitations, we propose Agri-R1, a
reasoning-enhanced large model for agriculture. Our framework automates high-quality
reasoning data generation via
vision-
language synthesis and LLM-based filtering, using only 19% of available samples. Training employs Group Relative Policy Optimization (GRPO) with a novel reward function that integrates domain-specific lexicons and fuzzy matching to assess both correctness and linguistic flexibility in open-ended responses. Evaluated on CDDMBench, our resulting 3B-parameter model achieves performance competitive with 7B- to 13B-parameter baselines, showing a +27.9% relative gain in disease recognition accuracy, +33.3% in agricultural knowledge QA, and a +26.10-point improvement in cross-domain generalization over standard fine-tuning. These results suggest that automated
reasoning synthesis paired with domain-aware reward design may provide a broadly applicable paradigm for RL-based VLM adaptation in data-scarce specialized domains. Our code and data are publicly available at: https://github.com/CPJ-Agricultural/Agri-R1.
Read the full abstract ▾
All authors Wentao Zhang, Mingkun Xu, Qi Zhang, Shangyang Li, Derek F. Wong, Lifei Wang, Yanchao Yang, Lina Lu, Tao Fang
Computation and Language
Yifan Le and Yunliang Li
Multilingual large
language models (LLMs) achieve strong performance across
languages, yet how
language capabilities are organized at the
neuron level remains poorly understood. Prior work has identified
language-related
neurons mainly through activation-based heuristics, which conflate
language preference with functional importance. We propose CRANE, a relevance-based analysis framework that redefines
language specificity in terms of functional necessity, identifying
language-specific
neurons through targeted
neuron-level interventions. CRANE characterizes
neuron specialization by their contribution to
language-conditioned predictions rather than activation magnitude. Our implementation will be made publicly available.
Neuron-level interventions reveal a consistent asymmetric pattern: masking
neurons relevant to a target
language selectively degrades performance on that
language while preserving performance on other
languages to a substantial extent, indicating
language-selective but non-exclusive
neuron specializations. Experiments on English, Chinese, and Vietnamese across multiple
benchmarks, together with a dedicated relevance-based metric and base-to-chat model transfer analysis, show that CRANE isolates
language-specific components more precisely than activation-based methods.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Succeeding at Scale: Enterprise Retrieval Benchmark Construction and Index-Preserving Query Adaptation for Multi-Tenant Search
Prateek Jain, Shabari S Nair, Ritesh Goru et al. (+4 more)
Large-scale multi-tenant
retrieval systems generate extensive query logs but lack curated relevance labels for effective domain adaptation, resulting in substantial underutilized "dark data." This challenge is compounded by the high cost of model updates, as jointly fine-tuning query and document encoders requires full corpus re-indexing, which is impractical in multi-tenant settings with thousands of isolated indices. We introduce DevRev-Search, a passage
retrieval benchmark for technical customer support built via a fully automated pipeline. Candidate generation uses fusion across diverse sparse and dense retrievers, followed by an LLM-as-a-Judge for consistency filtering and relevance labeling. We further study and systematically evaluate index-preserving query-only adaptation strategies that fine-tune only the query-encoder while keeping the document indices fixed. Experiments on DevRev-Search, SciFact, and FiQA-2018 show that parameter-efficient fine-tuning of the query encoder delivers a remarkable quality-efficiency trade-off, enabling scalable and practical enterprise multi-tenant
retrieval.
Read the full abstract ▾
All authors Prateek Jain, Shabari S Nair, Ritesh Goru, Prakhar Agarwal, Ajay Yadav, Yoga Sri Varshan Varadharajan, Constantine Caramanis
Computation and Language · Cross-listed from cs.CR
DP-MGTD: Privacy-Preserving Machine-Generated Text Detection via Adaptive Differentially Private Entity Sanitization
Lionel Z. Wang, Yusheng Zhao, Jiabin Luo et al. (+6 more)
The deployment of Machine-Generated Text (MGT) detection systems necessitates processing sensitive user data, creating a fundamental conflict between authorship verification and
privacy preservation. Standard anonymization techniques often disrupt linguistic fluency, while rigorous Differential
Privacy (DP) mechanisms typically degrade the statistical signals required for accurate detection. To resolve this dilemma, we propose DP-MGTD, a framework incorporating an Adaptive Differentially Private Entity Sanitization
algorithm. Our approach utilizes a two-stage mechanism that performs noisy frequency estimation and dynamically calibrates
privacy budgets, applying Laplace and Exponential mechanisms to numerical and textual entities respectively. Crucially, we identify a counter-intuitive phenomenon where the application of DP noise amplifies the distinguishability between human and machine text by exposing distinct sensitivity patterns to perturbation. Extensive experiments on the MGTBench-2.0
dataset show that our method achieves near-perfect detection accuracy, significantly outperforming non-private baselines while satisfying strict
privacy guarantees.
Read the full abstract ▾
All authors Lionel Z. Wang, Yusheng Zhao, Jiabin Luo, Xinfeng Li, Lixu Wang, Yinan Peng, Haoyang Li, XiaoFeng Wang, Wei Dong
Computation and Language
SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation
Sirry Chen, Jieyi Wang, Wei Chen et al. (+1 more)
Medical consultations are intrinsically
speech-centric. However, most prior works focus on long-text-based interactions, which are cumbersome and patient-unfriendly. Recent advances in
speech language models (SpeechLMs) have enabled more natural
speech-based interaction, yet the scarcity of medical
speech data and the inefficiency of directly fine-tuning on
speech data jointly hinder the adoption of SpeechLMs in medical consultation. In this paper, we propose SpeechMedAssist, a SpeechLM natively capable of conducting
speech-based multi-turn interactions with patients. By exploiting the architectural properties of SpeechLMs, we decouple the conventional one-stage training into a two-stage paradigm consisting of (1) Knowledge & Capability Injection via Text and (2) Modality Re-
alignment with Limited
Speech Data, thereby reducing the requirement for medical
speech data to only 10k synthesized samples. To evaluate SpeechLMs for medical consultation scenarios, we design a
benchmark comprising both single-turn question answering and multi-turn simulated interactions. Experimental results show that our model outperforms all baselines in both effectiveness and robustness in most evaluation settings.
Read the full abstract ▾
All authors Sirry Chen, Jieyi Wang, Wei Chen, Zhongyu Wei
Computation and Language
MAGA-Bench: Machine-Augment-Generated Text via Alignment Detection Benchmark
Anyang Song, Ying Cheng, Yiqian Xu et al. (+1 more)
Machine-Generated Text (MGT) is becoming increasingly difficult to distinguish from Human-Written Text (HWT). This trend has exacerbated malicious activities such as fake news and online fraud. The generalization ability of fine-tuned detectors relies heavily on
dataset quality, and simply expanding the sources of MGT may become increasingly insufficient. Further augmentation of the generation process is required. Based on HC-Var's theory, enhancing the human-like
alignment of MGT not only facilitates robustness testing of existing detectors but also boosts the generalization ability of detectors fine-tuned on such aligned MGT
datasets. Therefore, we propose the Machine-Augment-Generated Text via
Alignment (MAGA) Detection
Benchmark. MAGA integrates several
alignment methods, ranging from
prompt construction to Generator-Detector Adversarial Reinforcement Learning (GDARL) and the
reasoning process. In our experiments, the RoBERTa detector fine-tuned on MAGA achieves an average improvement of 4.60% in generalization AUC. Conversely, the aligned MGTs in MAGA also lead to an average decrease of 8.13% in the AUC of selected detectors. We hope the MAGA
Benchmark will provide valuable insights for future research on the generalization ability of MGT detectors.
Read the full abstract ▾
All authors Anyang Song, Ying Cheng, Yiqian Xu, Rui Feng
Computation and Language
From National Curricula to Cultural Awareness: Constructing Open-Ended Culture-Specific Question Answering Dataset
Haneul Yoo, Won Ik Cho, Geunhye Kim et al. (+1 more)
Large
language models (LLMs) achieve strong performance on many tasks, but their progress remains uneven across
languages and cultures, often reflecting values latent in English-centric training data. To enable practical cultural
alignment, we propose a scalable approach that leverages national social studies curricula as a foundation for culture-aware supervision. We introduce CuCu, an automated multi-
agent LLM framework that transforms national textbook curricula into open-ended, culture-specific question-answer pairs for supervised fine-tuning (SFT). Applying CuCu to the Korean national social studies curriculum, we construct KCaQA, comprising 34.1k open-ended QA pairs. Our analyses and training experiments suggest that KCaQA covers culture-specific topics and produces responses grounded in local sociocultural contexts.
Read the full abstract ▾
All authors Haneul Yoo, Won Ik Cho, Geunhye Kim, Jiyoon Han
Computation and Language
Character-R1: Enhancing Role-Aware Reasoning in Role-Playing Agents via RLVR
Yihong Tang, Kehai Chen, Xuefeng Bai et al. (+4 more)
Current role-playing
agents (RPAs) are typically constructed by imitating surface-level behaviors, but this approach lacks internal cognitive consistency, often causing out-of-character errors in complex situations. To address this, we propose Character-R1, a framework designed to provide comprehensive verifiable reward signals for effective role-aware
reasoning, which are missing in recent studies. Specifically, our framework comprises three core designs: (1) Cognitive Focus Reward, which enforces explicit label-based analysis of 10 character elements (e.g., worldview) to structure internal cognition; (2) Reference-Guided Reward, which utilizes overlap-based metrics with reference responses as optimization anchors to enhance exploration and performance; and (3) Character-Conditioned Reward Normalization, which adjusts reward distributions based on character categories to ensure robust optimization across heterogeneous roles. Extensive experiments demonstrate that Character-R1 significantly outperforms existing methods in knowledge,
memory and others.
Read the full abstract ▾
All authors Yihong Tang, Kehai Chen, Xuefeng Bai, Benyou Wang, Zeming Liu, Haifeng Wang, Min Zhang
Computation and Language
When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation
Rhea Kapur, Robert Hawkins and Elisa Kreiss
Vision-
language models (VLMs) are increasingly used to make visual content accessible via text-based descriptions. In current systems, however, description specificity is often conflated with their length. We argue that these two concepts must be disentangled: descriptions can be concise yet dense with information, or lengthy yet vacuous. We define specificity relative to a contrast set, where a description is more specific to the extent that it picks out the target image better than other possible images. We construct a
dataset that controls for length while varying information content, and validate that people reliably prefer more specific descriptions regardless of length. We find that controlling for length alone cannot account for differences in specificity: how the length budget is allocated makes a difference. These results support evaluation approaches that directly prioritize specificity over verbosity.
Read the full abstract ▾