Computation and Language
From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text
Van-Truong Le
The complexity of Vietnam's legal texts presents a significant barrier to public access to justice. While Large
Language Models offer a promising solution for legal text simplification, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. This paper introduces a comprehensive dual-aspect evaluation framework to address this need. First, we establish a performance
benchmark for four state-of-the-art large
language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three key dimensions: Accuracy, Readability, and Consistency. Second, to understand the "why" behind these performance scores, we conduct a large-scale error analysis on a curated
dataset of 60 complex Vietnamese legal articles, using a novel, expert-validated error typology. Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy, while models like Claude 3 Opus achieve high Accuracy scores that mask a significant number of subtle but critical
reasoning errors. The error analysis pinpoints Incorrect Example and Misinterpretation as the most prevalent failures, confirming that the primary challenge for current LLMs is not summarization but controlled, accurate legal
reasoning. By integrating a quantitative
benchmark with a qualitative deep dive, our work provides a holistic and actionable assessment of LLMs for legal applications.
Read the full abstract ▾
Computation and Language
SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation
Deshan Sumanathilaka, Nicholas Micallef, Julian Hough et al. (+1 more)
Recent advances in
language models have substantially improved Natural
Language Understanding (NLU). Although widely used
benchmarks suggest that Large
Language Models (LLMs) can effectively disambiguate, their practical applicability in real-world narrative contexts remains underexplored. SemEval-2026 Task 5 addresses this gap by introducing a task that predicts the human-perceived plausibility of a word sense within a short story. In this work, we propose an LLM-based framework for plausibility scoring of homonymous word senses in narrative texts using a structured
reasoning mechanism. We examine the impact of fine-tuning low-parameter LLMs with diverse
reasoning strategies, alongside dynamic few-shot prompting for large-parameter models, on accurate sense identification and plausibility estimation. Our results show that commercial large-parameter LLMs with dynamic few-shot prompting closely replicate human-like plausibility judgments. Furthermore, model ensembling slightly improves performance, better simulating the agreement patterns of five human annotators compared to single-model predictions
Read the full abstract ▾
All authors Deshan Sumanathilaka, Nicholas Micallef, Julian Hough, Saman Jayasinghe
Computation and Language · Cross-listed from cs.CV
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
Yige Xu, Yongjie Wang, Zizhuo Wu et al. (+3 more)
Reasoning in
vision-
language models (VLMs) has recently attracted significant
attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine
vision-grounded
reasoning or relies predominantly on the
reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel
multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous
alignment effectively isolates modality-specific
reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual
reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct
reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts
reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual
reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.
Read the full abstract ▾
All authors Yige Xu, Yongjie Wang, Zizhuo Wu, Kaisong Song, Jun Lin, Zhiqi Shen
Computation and Language · Cross-listed from cs.LG
Detecting and Suppressing Reward Hacking with Gradient Fingerprints
Songtao Wang, Quang Hieu Pham, Fangcong Yin et al. (+4 more)
Reinforcement learning with verifiable rewards (RLVR) typically optimizes for outcome rewards without imposing constraints on intermediate
reasoning. This leaves training susceptible to reward hacking, where models exploit loopholes (e.g., spurious patterns in training data) in the reward function to achieve high scores without solving the intended task. These reward-hacking behaviors are often implicit, as the intermediate chain-of-thought (CoT) may appear plausible on the surface, limiting the effectiveness of purely text-based monitoring. We propose Gradient Fingerprint (GRIFT), a method for detecting reward hacking using models' internal computations. Given a
prompt and a model-generated CoT, GRIFT computes gradients of the CoT conditioned on the
prompt and compresses them into a compact representation, which is then used to assess whether the CoT reflects reward hacking behavior. Across verifiable
reasoning benchmarks spanning math, code, and logical
reasoning, GRIFT substantially outperforms strong baselines, including CoT Monitor and TRACE, achieving over 25% relative improvement in detecting reward hacking behavior. Moreover, integrating GRIFT into the rejection fine-tuning pipeline for
reasoning tasks reduces reward hacking and improves performance on the true task objective. Our results highlight a promising direction of leveraging gradient level representations for assessing the quality of CoT
reasoning traces. Our code is available at: https://github.com/songtao-x/reward_hack.
Read the full abstract ▾
All authors Songtao Wang, Quang Hieu Pham, Fangcong Yin, Xinpeng Wang, Jocelyn Qiaochu Chen, Greg Durrett, Xi Ye
Computation and Language
BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
Jiacheng Shen, Masato Hagiwara, Milad Alizadeh et al. (+9 more)
Large
language models have shown strong performance on broad-domain knowledge and
reasoning benchmarks, but it remains unclear how well
language models handle specialized animal-related knowledge under a unified closed-book evaluation protocol. We introduce BAGEL, a
benchmark for evaluating animal knowledge expertise in
language models. BAGEL is constructed from diverse scientific and reference sources, including bioRxiv, Global Biotic Interactions, Xeno-canto, and Wikipedia, using a combination of curated examples and automatically generated closed-book question-answer pairs. The
benchmark covers multiple aspects of animal knowledge, including taxonomy, morphology, habitat, behavior, vocalization, geographic distribution, and species interactions. By focusing on closed-book evaluation, BAGEL measures animal-related knowledge of models without external
retrieval at inference time. BAGEL further supports fine-grained analysis across source domains, taxonomic groups, and knowledge categories, enabling a more precise characterization of model strengths and systematic failure modes. Our
benchmark provides a new testbed for studying domain-specific knowledge generalization in
language models and for improving their reliability in biodiversity-related applications.
Read the full abstract ▾
All authors Jiacheng Shen, Masato Hagiwara, Milad Alizadeh, Ellen Gilsenan-McMahon, Marius Miron, David Robinson, Emmanuel Chemla, Sara Keen, Gagan Narula, Mathieu Laurière, Matthieu Geist, Olivier Pietquin
Computation and Language
Optimizing Korean-Centric LLMs via Token Pruning
Hoyeol Kim and Hyeonwoo Kim
This paper presents a systematic
benchmark of state-of-the-art
multilingual large
language models (LLMs) adapted via
token pruning - a compression technique that eliminates
tokens and
embedding parameters corresponding to
languages irrelevant to the target application. Focusing on Korean-centric natural
language processing (NLP) tasks, we evaluate architectures including Qwen3, Gemma-3, Llama-3, and Aya across three vocabulary configurations: Original, English-Korean (EnKo), and English-Korean-Chinese (EnKoZh). Performance is assessed using established
benchmarks for general aptitude, cultural literacy, instruction following, and machine translation. Our findings indicate that
token pruning significantly improves generation stability by eliminating
language confusion, and in the case of machine translation, frequently enhances performance on Korean-specific tasks. While instruction-following capabilities display architecture-dependent variance linked to latent cross-lingual representations, the significant reduction in vocabulary size validates
token pruning as a highly effective optimization strategy for
memory-constrained, domain-specific deployments, despite modest gains in inference latency.
Read the full abstract ▾
Computation and Language
Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations
Yanli Wang, Peng Kuang, Xiaoyu Han et al. (+2 more)
Large
language models are increasingly deployed in settings where reliability matters, yet output-level uncertainty signals such as
token probabilities,
entropy, and self-consistency can become brittle under calibration–deployment mismatch. Conformal prediction provides finite-sample validity under exchangeability, but its practical usefulness depends on the quality of the nonconformity score. We propose a conformal framework for LLM question answering that uses internal representations rather than output-facing statistics: specifically, we introduce Layer-Wise Information (LI) scores, which measure how conditioning on the input reshapes predictive
entropy across model depth, and use them as nonconformity scores within a standard split conformal pipeline. Across closed-ended and open-domain QA
benchmarks, with the clearest gains under cross-domain shift, our method achieves a better validity–efficiency trade-off than strong text-level baselines while maintaining competitive in-domain reliability at the same nominal risk level. These results suggest that internal representations can provide more informative conformal scores when surface-level uncertainty is unstable under distribution shift.
Read the full abstract ▾
All authors Yanli Wang, Peng Kuang, Xiaoyu Han, Kaidi Xu, Haohan Wang
Computation and Language · Cross-listed from cs.LG
JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
Alexandra Dragomir, Ioana Pintilie, Antonio Barbalau et al. (+6 more)
Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large
Language Models (LLMs), by sequentially learning a low-rank update
matrix for each task. To mitigate catastrophic forgetting, state-of-the-art approaches impose constraints on new adapters with respect to the previous ones, by targeting either subspace or coordinate-wise interference. In this paper, we propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating. The method achieves dynamic parameter isolation, which helps prevent task interference. We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
Read the full abstract ▾
All authors Alexandra Dragomir, Ioana Pintilie, Antonio Barbalau, Marius Dragoi, Florin Brad, Cristian Daniel Paduraru, Alexandru Tifrea, Elena Burceanu, Radu Tudor Ionescu
Computation and Language
AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliency
Max Henning Höth, Kristian Kersting, Björn Deiseroth et al. (+1 more)
Large
language models (LLMs) increasingly rely on chain-of-thought (CoT)
reasoning to solve complex tasks, yet a
reasoning trace may merely accompany the final answer rather than causally contribute to it or faithfully reflect how it was reached. We introduce differentiable
attention manipulation to train salient
reasoning with reinforcement learning – method which we call AtManRL. By optimizing an additive
attention mask that identifies the CoT
tokens needed to recover the model's answer, we derive a saliency reward and combine it with an outcome reward in GRPO with decoupled advantages. Because both saliency and faithfulness depend strongly on trace length, we evaluate against a sweep of length-penalty baselines that traces the trade-off a pure length penalty can reach, rather than against a single baseline. Across three models (Qwen2.5-3B, Granite-3.3-2B, and Phi-4-mini) and four
benchmarks (GSM8K, MMLU, ProofWriter, and QuestBench), AtManRL sits above this trade-off curve by 3.9-4.0pp in saliency, measured as answer changes under mistake injection, and by 5.7-7.7pp in faithfulness, measured as hint verbalization under hint injection, while matching the accuracy of baselines of similar length. These results suggest that training explicitly for saliency is a promising route towards chain-of-thought
reasoning that is more faithful to the model's actual decision process.
Read the full abstract ▾
All authors Max Henning Höth, Kristian Kersting, Björn Deiseroth, Letitia Parcalabescu
Computation and Language
On the Rejection Criterion for Proxy-based Test-time Alignment
Ayoub Hammal, Pierre Zweigenbaum and Caio Corro
Recent works proposed test-time
alignment methods that rely on a small aligned model as a proxy that guides the generation of a larger base (unaligned) model. The implicit reward approach skews the large model distribution, whereas the nudging approach defers the generation of the next
token to the small aligned model when the large base one is unconfident about its outcome. In this work, we first show that both approaches can be reduced to sampling from similar graphical models, where they differ only in the definition of a rejection criterion (or distribution). Moreover, we argue that the confidence criterion is ill-motivated due to linguistic phenomena like ambiguous phrasing. We propose a novel rejection criterion based on a conservative confidence bet. Experimentally, our novel approach outperforms previous work on several
datasets.
Read the full abstract ▾
Computation and Language
Sentiment Analysis of German Sign Language Fairy Tales
Fabrizio Nunnari, Siddhant Jain and Patrick Gebhard
We present a
dataset and a model for sentiment analysis of German sign
language (DGS) fairy tales. First, we perform sentiment analysis for three levels of valence (negative, neutral, positive) on German fairy tales text segments using four large
language models (LLMs) and majority voting, reaching an inter-annotator agreement of 0.781 Krippendorff's alpha. Second, we extract face and body motion features from each corresponding DGS
video segment using MediaPipe. Finally, we train an explainable model (based on XGBoost) to predict negative, neutral or positive sentiment from
video features. Results show an average balanced accuracy of 0.631. A thorough analysis of the most important features reveal that, in addition to eyebrows and mouth motion on the face, also the motion of hips, elbows, and shoulders considerably contribute in the discrimination of the conveyed sentiment, indicating an equal importance of face and body for sentiment communication in sign
language.
Read the full abstract ▾
Computation and Language
Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors
Jessica H. Zhu, Shayla Stringfield, Vahe Zaprosyan et al. (+3 more)
Firearm violence is a pressing public health issue, yet research into survivors' lived experiences remains underfunded and difficult to scale. Qualitative research, including in-depth interviews, is a valuable tool for understanding the personal and societal consequences of community firearm violence and designing effective interventions. However, manually analyzing these narratives through thematic analysis and inductive coding is time-consuming and labor-intensive. Recent advancements in large
language models (LLMs) have opened the door to automating this process, though concerns remain about whether these models can accurately and ethically capture the experiences of vulnerable populations. In this study, we assess the use of open-source LLMs to inductively code interviews with 21 Black men who have survived community firearm violence. Our results demonstrate that while some configurations of LLMs can identify important codes, overall relevance remains low and is highly sensitive to data processing. Furthermore, LLM guardrails lead to substantial narrative erasure. These findings highlight both the potential and limitations of LLM-assisted qualitative coding and underscore the ethical challenges of applying AI in research involving marginalized communities.
Read the full abstract ▾
All authors Jessica H. Zhu, Shayla Stringfield, Vahe Zaprosyan, Michael Wagner, Michel Cukier, Joseph B. Richardson Jr
Computation and Language · Cross-listed from cs.SE
LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning
Mahir Labib Dihan and Abir Muhtasim
The rapid proliferation of Large
Language Models (LLMs) in software development has made distinguishing AI-generated code from human-written code a critical challenge with implications for academic integrity, code quality assurance, and software security. We present LLMSniffer, a detection framework that fine-tunes GraphCodeBERT using a two-stage supervised contrastive learning pipeline augmented with comment removal preprocessing and an MLP classifier. Evaluated on two
benchmark datasets - GPTSniffer and Whodunit - LLMSniffer achieves substantial improvements over prior baselines: accuracy increases from 70% to 78% on GPTSniffer (F1: 68% to 78%) and from 91% to 94.65% on Whodunit (F1: 91% to 94.64%). t-SNE visualizations confirm that contrastive fine-tuning yields well-separated, compact
embeddings. We release our model checkpoints,
datasets, codes and a live interactive demo to facilitate further research.
Read the full abstract ▾
Computation and Language
Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures
Yutong Gao, Qinglin Meng, Yuan Zhou et al. (+1 more)
While Large
Language Models (LLMs) have achieved strong performance across many NLP tasks, their opaque internal mechanisms hinder trustworthiness and safe deployment. Existing surveys in explainable AI largely focus on post-hoc explanation methods that interpret trained models through external approximations. In contrast, intrinsic interpretability, which builds transparency directly into model architectures and computations, has recently emerged as a promising alternative. This paper presents a systematic review of the recent advances in intrinsic interpretability for LLMs, categorizing existing approaches into five design paradigms: functional transparency, concept
alignment, representational decomposability, explicit modularization, and latent sparsity induction. We further discuss open challenges and outline future research directions in this emerging field. The paper list is available at: https://github.com/PKU-PILLAR-Group/Survey-Intrinsic-Interpretability-of-LLMs.
Read the full abstract ▾
All authors Yutong Gao, Qinglin Meng, Yuan Zhou, Liangming Pan
Computation and Language
Stochasticity in Tokenisation Improves Robustness
Sophie Steger, Rui Li, Sofiane Ennadir et al. (+4 more)
The widespread adoption of large
language models (LLMs) has increased concerns about their robustness. Vulnerabilities in perturbations of tokenisation of the input indicate that models trained with a deterministic canonical tokenisation can be brittle to
adversarial attacks. Recent studies suggest that
stochastic tokenisation can deliver internal representations that are less sensitive to perturbations. In this paper, we analyse how
stochastic tokenisations affect robustness to
adversarial attacks and random perturbations. We systematically study this over a range of learning regimes (pre-training, supervised fine-tuning, and in-context learning), data sets, and model architectures. We show that pre-training and fine-tuning with uniformly sampled
stochastic tokenisations improve robustness to random and
adversarial perturbations. Evaluating on uniformly sampled non-canonical tokenisations reduces the accuracy of a canonically trained Llama-1b model by 29.8%. We find that training with
stochastic tokenisation preserves accuracy without increasing inference cost.
Read the full abstract ▾
All authors Sophie Steger, Rui Li, Sofiane Ennadir, Anya Sims, Arno Solin, Franz Pernkopf, Martin Trapp
Computation and Language
Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning
Jiaxi Bi, Tongxu Luo, Wenyu Du et al. (+2 more)
Parallel
reasoning enhances Large
Reasoning Models (LRMs) but incurs prohibitive costs due to futile paths caused by early errors. To mitigate this, path
pruning at the prefix level is essential, yet existing research remains fragmented without a standardized framework. In this work, we propose the first systematic taxonomy of path
pruning, categorizing methods by their signal source (internal vs. external) and learnability (learnable vs. non-learnable). This classification reveals the unexplored potential of learnable internal methods, motivating our proposal of STOP (Super
TOken for
Pruning). Extensive evaluations across LRMs ranging from 1.5B to 20B parameters demonstrate that STOP achieves superior effectiveness and efficiency compared to existing baselines. Furthermore, we rigorously validate the scalability of STOP under varying compute budgets - for instance, boosting GPT-OSS-20B accuracy on AIME25 from 84% to nearly 90% under fixed compute budgets. Finally, we distill our findings into formalized empirical guidelines to facilitate optimal real-world deployment. Code, data and models are available at https://bijiaxihh.github.io/STOP
Read the full abstract ▾
All authors Jiaxi Bi, Tongxu Luo, Wenyu Du, Zhengyang Tang, Benyou Wang
Computation and Language
Where does output diversity collapse in post-training?
Constantinos Karouzos, Xingwei Tan and Nikolaos Aletras
Post-trained
language models produce less varied outputs than their base counterparts. This output diversity collapse undermines inference-time scaling methods that rely on varied samples, and risks homogenizing model outputs on creative and value-laden tasks. Prior work attributes collapse to specific post-training methods, without separating the role of training data composition from the method, or the generation format from the model weights. We trace output diversity through three parallel post-training lineages of Olmo 3, Think (chain-of-thought
distillation), Instruct (broad multi-source data), and RL-Zero, across 15 tasks and four text diversity metrics. We find that the location of collapse co-varies with data composition: the Think lineage loses most semantic diversity at supervised fine-tuning, and the effect of DPO is larger in Instruct than in Think. Suppressing chain-of-thought
reasoning at inference in Think models drops accuracy on hard tasks, yet leaves answer-level diversity unchanged, showing that the collapse is embedded in the model weights by training data, not imposed by the generation format. Decomposing diversity loss on six verifiable tasks into a quality-control component (removal of incorrect outputs) and a residual component (genuine narrowing among correct outputs) reveals that the split is task-dependent, and Think models retain more correct-answer diversity than Instruct despite collapsing more in aggregate. Our results indicate that diversity collapse is determined during training by data composition and cannot be addressed at inference time alone.
Read the full abstract ▾
Computation and Language
AgentV-RL: Scaling Reward Modeling with Agentic Verifier
Jiazheng Zhang, Ziche Fu, Zhiheng Xi et al. (+13 more)
Verifiers have been demonstrated to enhance LLM
reasoning via test-time scaling (TTS). Yet, they face significant challenges in complex domains. Error propagation from incorrect intermediate
reasoning can lead to false positives for seemingly plausible solutions, while lacking external grounding makes verifiers unreliable on computation or knowledge-intensive tasks. To address these challenges, we propose Agentic Verifier, a framework that transforms reward modeling into a multi-turn, tool-augmented deliberative process. We introduce complementary forward and backward
agents: one traces solutions from premises to conclusions, while the other re-checks conclusions against their underlying premises. This bidirectional process enables a comprehensive, reliable, and interpretable assessment of solutions. To facilitate practical deployment, we propose AgentV-RL. Through proactive exploration and reinforcement learning, the verifier autonomously interleaves tool-use with internal
reasoning. Extensive experiments show that Agentic Verifier yields consistent performance gains under both parallel and sequential TTS. Notably, our 4B variant surpasses state-of-the-art ORMs by 25.2%, positioning it as a promising paradigm for agentic reward modeling.
Read the full abstract ▾
All authors Jiazheng Zhang, Ziche Fu, Zhiheng Xi, Wenqing Jing, Mingxu Chai, Wei He, Guoqiang Zhang, Chenghao Fan, Chenxin An, Wenxiang Chen, Zhicheng Liu, Haojie Pan, Dingwei Zhu, Tao Gui, Qi Zhang, Xuanjing Huang
Computation and Language
SCHK-HTC: Sibling Contrastive Learning with Hierarchical Knowledge-Aware Prompt Tuning for Hierarchical Text Classification
Ke Xiong, Qian Wu, Wangjie Gan et al. (+2 more)
Few-shot Hierarchical Text Classification (few-shot HTC) is a challenging task that involves mapping texts to a predefined tree-structured label hierarchy under data-scarce conditions. While current approaches utilize structural constraints from the label hierarchy to maintain parent-child prediction consistency, they face a critical bottleneck, the difficulty in distinguishing semantically similar sibling classes due to insufficient domain knowledge. We introduce an innovative method named Sibling Contrastive Learning with Hierarchical Knowledge-aware
Prompt Tuning for few-shot HTC tasks (SCHK-HTC). Our work enhances the model's perception of subtle differences between sibling classes at deeper levels, rather than just enforcing hierarchical rules. Specifically, we propose a novel framework featuring two core components: a hierarchical knowledge extraction module and a sibling contrastive learning mechanism. This design guides model to encode discriminative features at each hierarchy level, thus improving the separability of confusable classes. Our approach achieves superior performance across three
benchmark datasets, surpassing existing state-of-the-art methods in most cases. Our code is available at https://github.com/happywinder/SCHK-HTC.
Read the full abstract ▾
All authors Ke Xiong, Qian Wu, Wangjie Gan, Yuke Li, Xuhong Zhang
Computation and Language · Cross-listed from cs.AI
Weak-Link Optimization for Multi-Agent Reasoning and Collaboration
Haoyu Bian, Chaoning Zhang, Jiaquan Zhang et al. (+4 more)
LLM-driven multi-
agent frameworks address complex
reasoning tasks through multi-role collaboration. However, existing approaches often suffer from
reasoning instability, where individual
agent errors are amplified through collaboration, undermining overall performance. Current research mainly focuses on enhancing high-capability
agents or suppressing unreliable outputs to improve framework effectiveness, while systematic identification and reinforcement of performance-limiting
agents receive less
attention. To address this gap, we propose WORC, a \underline{w}eak-link \underline{o}ptimization framework for multi-
agent \underline{r}easoning and \underline{c}ollaboration, grounded in the weak-link principle. WORC follows a two-stage workflow. In the weak
agent localization stage, task features are constructed, and a meta-learning-based weight predictor trained on optimal configurations identified by swarm intelligence
algorithms (SIAs) enables zero-shot mapping from these features to
agent performance weights, where the
agent with the lowest predicted weight is identified as the weak
agent. In the weak-link optimization stage, an uncertainty-driven allocation strategy assigns additional
reasoning budgets to weak
agents, with lower predicted weights leading to larger repeated-sampling quotas to compensate for reliability deficiencies. Experimental results show that WORC achieves an average accuracy of 82.2% on
reasoning benchmarks while improving framework stability and cross-architecture generalization, suggesting that compensating for weak links, rather than reinforcing strengths alone, enhances the robustness of multi-
agent systems.
Read the full abstract ▾
All authors Haoyu Bian, Chaoning Zhang, Jiaquan Zhang, Xingyao Li, Yuanfang Guo, Wei Dong, Yang Yang
Computation and Language · Cross-listed from cs.CR
A Case Study on the Impact of Anonymization Along the RAG Pipeline
Andreea-Elena Bodea, Stephen Meisenbacher and Florian Matthes
Despite the considerable promise of
Retrieval-Augmented Generation (RAG), many real-world use cases may create
privacy concerns, where the purported utility of RAG-enabled insights comes at the risk of exposing private information to either the LLM or the end user requesting the response. As a potential mitigation, using anonymization techniques to remove personally identifiable information (PII) and other sensitive markers in the underlying data represents a practical and sensible course of action for RAG administrators. Despite a wealth of literature on the topic, no works consider the placement of anonymization along the RAG pipeline, i.e., asking the question, where should anonymization happen? In this case study, we systematically and empirically measure the impact of anonymization at two important points along the RAG pipeline: the
dataset and generated answer. We show that differences in
privacy-utility trade-offs can be observed depending on where anonymization took place, demonstrating the significance of
privacy risk mitigation placement in RAG.
Read the full abstract ▾
Computation and Language
RAGognizer: Hallucination-Aware Fine-Tuning via Detection Head Integration
Fabian Ridder, Laurin Lessel and Malte Schilling
Retrieval-Augmented Generation (RAG) is widely used to augment the input to Large
Language Models (LLMs) with external information, such as recent or domain-specific knowledge. Nonetheless, current models still produce closed-domain
hallucinations and generate content that is unsupported by the retrieved context. Current detection approaches typically treat
hallucination as a post-hoc problem, relying on black-box consistency checks or probes over frozen internal representations. In this work, we demonstrate that
hallucination detection based on internal state representation can also serve as a direct training signal. We introduce RAGognize, a
dataset of naturally occurring closed-domain
hallucinations with
token-level annotations, and RAGognizer, a
hallucination-aware fine-tuning approach that integrates a lightweight detection head into an LLM, allowing for the joint optimization of
language modeling and
hallucination detection. This joint objective forces the model to improve the separability of its internal states regarding
hallucinations while simultaneously learning to generate well-formed and meaningful responses. Across multiple
benchmarks, RAGognizer achieves state-of-the-art
token-level
hallucination detection while substantially reducing
hallucination rates during generation, without degrading
language quality or relevance.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SI
Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation
Nicolò Pagan, Christopher Barrie, Chris Andrew Bail et al. (+1 more)
Large
Language Models (LLMs) are increasingly deployed to curate and rank human-created content, yet the nature and structure of their biases in these tasks remains poorly understood: which biases are robust across providers and platforms, and which can be mitigated through
prompt design. We present a controlled simulation study mapping content selection biases across three major LLM providers (OpenAI, Anthropic, Google) on real social media
datasets from Twitter/X, Bluesky, and Reddit, using six prompting strategies (general, popular, engaging, informative, controversial, neutral). Through 540,000 simulated top-10 selections from pools of 100 posts across 54 experimental conditions, we find that biases differ substantially in how structural and how
prompt-sensitive they are. Polarization is amplified across all configurations, toxicity handling shows a strong inversion between engagement- and information-focused
prompts, and sentiment biases are predominantly negative. Provider comparisons reveal distinct trade-offs: GPT-4o Mini shows the most consistent behavior across
prompts; Claude and Gemini exhibit high adaptivity in toxicity handling; Gemini shows the strongest negative sentiment preference. On Twitter/X, where author demographics can be inferred from profile bios, political leaning bias is the clearest demographic signal: left-leaning authors are systematically over-represented despite right-leaning authors forming the pool plurality in the
dataset, and this pattern largely persists across
prompts.
Read the full abstract ▾
All authors Nicolò Pagan, Christopher Barrie, Chris Andrew Bail, Petter Törnberg
Computation and Language
Supriti Sinhamahapatra, Thai-Binh Nguyen, Yiğit Oğuz et al. (+3 more)
The goal of
multilingual speech technology is to facilitate seamless communication between individuals speaking different
languages, creating the experience as though everyone were a
multilingual speaker. To create this experience,
speech technology needs to address several challenges: Handling mixed
multilingual input, specific vocabulary, and code-switching. However, there is currently no
dataset benchmarking this situation. We propose a new
benchmark to evaluate current Automatic
Speech Recognition (ASR) systems, whether they are able to handle these challenges. The
benchmark consists of bilingual discussions on scientific papers between multiple speakers, each conversing in a different
language. We provide a standard evaluation framework, beyond Word Error Rate (WER) enabling consistent comparison of ASR performance across
languages. Experimental results demonstrate that the proposed
dataset is still an open challenge for state-of-the-art ASR systems. The
dataset is available in https://huggingface.co/
datasets/goodpiku/muscat-eval. Keywords:
multilingual,
speech recognition, audio
segmentation, speaker diarization
Read the full abstract ▾
All authors Supriti Sinhamahapatra, Thai-Binh Nguyen, Yiğit Oğuz, Enes Ugan, Jan Niehues, Alexander Waibel
Computation and Language · Cross-listed from cs.IR
Masahiro Suzuki and Hiroki Sakaji
We introduce JFinTEB, the first comprehensive
benchmark specifically designed for evaluating Japanese financial text
embeddings. Existing
embedding benchmarks provide limited coverage of
language-specific and domain-specific aspects found in Japanese financial texts. Our
benchmark encompasses diverse task categories including
retrieval and classification tasks that reflect realistic and well-defined financial text processing scenarios. The
retrieval tasks leverage instruction-following
datasets and financial text generation queries, while classification tasks cover sentiment analysis, document categorization, and domain-specific classification challenges derived from economic survey data. We conduct extensive evaluations across a wide range of
embedding models, including Japanese-specific models of various sizes,
multilingual models, and commercial
embedding services. We publicly release JFinTEB
datasets and evaluation framework at https://github.com/retarfi/JFinTEB to facilitate future research and provide a standardized evaluation protocol for the Japanese financial text mining community. This work addresses a critical gap in Japanese financial text processing resources and establishes a foundation for advancing domain-specific
embedding research.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents
Xing Zhang, Guanghui Wang, Yanwei Cui et al. (+4 more)
As LLM
agents scale to long-horizon, multi-session deployments, efficiently managing accumulated experience becomes a critical bottleneck.
Agent memory systems and
agent skill discovery both address this challenge, extracting reusable knowledge from interaction traces, yet a citation analysis of 1{,}136 references across 22 primary papers reveals a cross-community citation rate below 1%. We propose the Experience Compression Spectrum, a unifying framework that positions
memory, skills, and rules as points along a single axis of increasing compression (5–20$\times$ for episodic
memory, 50–500$\times$ for procedural skills, 1{,}000$\times$+ for declarative rules), directly reducing context consumption,
retrieval latency, and compute overhead. Mapping 20+ systems onto this spectrum reveals that every system operates at a fixed, predetermined compression level: none supports adaptive cross-level compression, a gap we term the missing diagonal. We further show that specialization alone is insufficient (both communities independently solve shared sub-problems without exchanging solutions), that evaluation methods are tightly coupled to compression levels, that transferability increases with compression at the cost of specificity, and that knowledge lifecycle management remains largely neglected. We articulate open problems and design principles for scalable, full-spectrum
agent learning systems.
Read the full abstract ▾
All authors Xing Zhang, Guanghui Wang, Yanwei Cui, Wei Qiu, Ziyuan Li, Bing Zhu, Peiyang He
Computation and Language
How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models
Judith Sieker and Sina Zarrieß
Large
language models (LLMs) are increasingly studied as repositories of linguistic knowledge. In this line of work, models are commonly evaluated both as generators of
language and as judges of linguistic output, yet these two roles are rarely examined in direct relation to one another. As a result, it remains unclear whether success in one role aligns with success in the other. In this paper, we address this question for pragmatic competence by comparing LLMs' performance as pragmatic listeners, judging the appropriateness of linguistic outputs, and as pragmatic speakers, generating pragmatically appropriate
language. We evaluate multiple open-weight and proprietary LLMs across three pragmatic settings. We find a robust asymmetry between pragmatic evaluation and pragmatic generation: many models perform substantially better as listeners than as speakers. Our results suggest that pragmatic judging and pragmatic generation are only weakly aligned in current LLMs, calling for more integrated evaluation practices.
Read the full abstract ▾
Computation and Language
DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition
Siun Kim and Hyung-Jin Yoon
Large
language models (LLMs) have advanced information extraction (IE) by enabling zero-shot and few-shot named entity recognition (NER), yet their generative outputs still show persistent and systematic errors. Despite progress through instruction fine-tuning, zero-shot NER still lags far behind supervised systems. These recurring errors mirror inconsistencies observed in early-stage human annotation processes that resolve disagreements through pilot annotation. Motivated by this analogy, we introduce DiZiNER (Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition), a framework that simulates the pilot annotation process, employing LLMs to act as both annotators and supervisors. Multiple heterogeneous LLMs annotate shared texts, and a supervisor model analyzes inter-model disagreements to refine task instructions. Across 18
benchmarks, DiZiNER achieves zero-shot SOTA results on 14
datasets, improving prior bests by +8.0 F1 and reducing the zero-shot to supervised gap by over +11 points. It also consistently outperforms its supervisor, GPT-5 mini, indicating that improvements stem from disagreement-guided instruction refinement rather than model capacity. Pairwise agreement between models shows a strong correlation with NER performance, further supporting this finding.
Read the full abstract ▾
Computation and Language
CiPO: Counterfactual Unlearning for Large Reasoning Models through Iterative Preference Optimization
Junyi Li, Yongqiang Chen and Ningning Ding
Machine unlearning has gained increasing
attention in recent years, as a promising technique to selectively remove unwanted
privacy or copyrighted information from Large
Language Models that are trained on a massive scale of human data. However, the emergence of Large
Reasoning Models (LRMs), which emphasize long chain-of-thought (CoT)
reasoning to address complex questions, presents a dilemma to unlearning: existing methods either struggle to completely eliminate undesired knowledge from the CoT traces or degrade the
reasoning performances due to the interference with the
reasoning process. To this end, we introduce Counterfactual Unlearning through iterative Preference Optimization (CiPO), a novel framework that redefines unlearning as the targeted intervention of the CoT
reasoning in LRMs. More specifically, given a desired unlearning target answer, CiPO instructs LRMs to generate a logically valid counterfactual
reasoning trace for preference tuning. As the LRM adjusts to the counterfactual trace, CiPO iteratively updates the preference learning data to increase the discrepancy from the original model. This iterative loop ensures both desirable unlearning and smooth optimization, effectively mitigating the dilemma. Experiments on challenging
benchmarks demonstrate that CiPO excels at unlearning, completely removing knowledge from both the intermediate CoT steps and the final answer, while preserving the
reasoning abilities of LRMs.
Read the full abstract ▾
Computation and Language
Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms
Tanja Baeumel, Josef van Genabith and Simon Ostermann
Large
language models (LLMs) have demonstrated impressive capabilities, yet their internal mechanisms for handling
reasoning-intensive tasks remain underexplored. To advance the understanding of model-internal processing mechanisms, we present an investigation of how LLMs perform arithmetic operations by examining internal mechanisms during task execution. Using early decoding, we trace how next-
token predictions are constructed across layers. Our experiments reveal that while the models recognize arithmetic tasks early, correct result generation occurs only in the final layers. Notably, models proficient in arithmetic exhibit a clear division of labor between
attention and MLP modules, where
attention propagates input information and MLP modules aggregate it. This division is absent in less proficient models. Furthermore, successful models appear to process more challenging arithmetic tasks functionally, suggesting
reasoning capabilities beyond factual recall.
Read the full abstract ▾