Computation and Language
Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation
Xiangxu Zhang, Lei Li, Yanyun Zhou et al. (+3 more)
Medical diagnostics is a high-stakes and complex domain that is critical to patient care. However, current evaluations of large
language models (LLMs) remain limited in capturing key challenges of clinical diagnostic scenarios. Most rely on
benchmarks derived from public exams, raising contamination bias that can inflate performance, and they overlook the confounded nature of real consultations beyond textbook cases. Recent dynamic evaluations offer a promising alternative, but often remain insufficient for diagnosis-oriented benchmarking, with limited coverage of clinically grounded confounders and trustworthiness beyond accuracy. To address these gaps, we propose DyReMe, a dynamic
benchmark for medical diagnostics that provides a controlled and scalable stress test of diagnostic robustness. Unlike static exam-style questions, DyReMe generates fresh, consultation-style cases that incorporate clinically grounded confounders, such as differential diagnoses and common misdiagnosis factors. It also varies expression styles to capture heterogeneous patient-style descriptions. Beyond accuracy, DyReMe evaluates LLMs on three additional clinically relevant dimensions: veracity, helpfulness, and consistency. Our experiments show that this dynamic approach yields more challenging assessments and exposes substantial weaknesses of stateof-the-art LLMs under clinically confounded diagnostic settings. These findings highlight the urgent need for evaluation frameworks that better assess trustworthy medical diagnostics 1 under clinically grounded confounders.
Read the full abstract ▾
All authors Xiangxu Zhang, Lei Li, Yanyun Zhou, Xiao Zhou, Yingying Zhang, Xian Wu
Computation and Language
CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation
Kaiwen Wei, Xiao Liu, Jie Zhang et al. (+11 more)
Multimodal Retrieval-Augmented Generation (MRAG) enables
Multimodal Large
Language Models (MLLMs) to generate responses with external
multimodal evidence, and numerous
video-based MRAG
benchmarks have been proposed to evaluate model capabilities across
retrieval and generation stages. However, existing
benchmarks remain limited in modality coverage and format diversity, often focusing on single- or limited-modality tasks, or coarse-grained scene understanding. To address these gaps, we introduce CFVBench, a large-scale, manually verified
benchmark constructed from 599 publicly available
videos, yielding 5,360 open-ended QA pairs. CFVBench spans high-density formats and domains such as chart-heavy reports, news broadcasts, and software tutorials, requiring models to retrieve and reason over long temporal
video spans while maintaining fine-grained
multimodal information. Using CFVBench, we systematically evaluate 7
retrieval methods and 14 widely-used MLLMs, revealing a critical bottleneck: current models (even GPT5 or Gemini) struggle to capture transient yet essential fine-grained
multimodal details. To mitigate this, we propose Adaptive Visual Refinement (AVR), a simple yet effective framework that adaptively increases frame sampling density and selectively invokes external tools when necessary. Experiments show that AVR consistently enhances fine-grained
multimodal comprehension and improves performance across all evaluated MLLMs
Read the full abstract ▾
All authors Kaiwen Wei, Xiao Liu, Jie Zhang, Zijian Wang, Ruida Liu, Yuming Yang, Xin Xiao, Xiao Sun, Haoyang Zeng, Changzai Pan, Yidan Zhang, Jiang Zhong, Peijin Wang, Yingchao Feng
Computation and Language
Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
Yongding Tao, Tian Wang, Yihong Dong et al. (+4 more)
Data contamination poses a significant threat to the reliable evaluation of Large
Language Models (LLMs). This issue arises when
benchmark samples may inadvertently appear in training sets, compromising the validity of reported performance. While detection methods have been developed for the pre-training and Supervised Fine-Tuning stages, a critical research gap exists for the increasingly significant phase of Reinforcement Learning (RL) post-training. As RL post-training becomes pivotal for advancing LLM
reasoning, the absence of specialized contamination detection methods in this paradigm presents a critical vulnerability. To address this, we conduct the first systematic study of data detection within RL post-training scenario and propose Self-Critique. Our method is motivated by a key observation: after RL phase, the output
entropy distribution of LLMs tends to collapse into highly specific and sparse modes. Self-Critique probes for the underlying policy collapse, i.e., the model's convergence to a narrow
reasoning path, which causes this
entropy reduction. To facilitate this research, we also introduce RL-MIA, a
benchmark constructed to simulate this specific contamination scenario. Extensive experiments show that Self-Critique significantly outperforms baseline methods across multiple models and contamination tasks, achieving an AUC improvement of up to 30%. Whereas existing methods are close to a random guess for RL-phase contamination, our method makes detection possible.
Read the full abstract ▾
All authors Yongding Tao, Tian Wang, Yihong Dong, Huanyu Liu, Kechi Zhang, Xiaolong Hu, Ge Li
Computation and Language
DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
Chenyang Gu, Yewen Pu, Bruce Yang et al. (+2 more)
Enhancing LLMs with the ability to actively search external knowledge is crucial for complex and real-world tasks. Current approaches either rely on prompting to elicit the model's innate
agent capabilities, or suffer from performance ceilings and collapse when applying RL to complex interactive tasks, leaving their true agentic potential untapped. To address this, we introduce Dynamic-filter Sequence-level Policy Optimization (DSPO), an improved RL
algorithm designed for robust
agent training through sequence-level optimization and dynamic sample filtering. We train our model purely through RL to interleave multi-turn search and
reasoning, obviating the need for supervised demonstration data. Across multiple QA
benchmarks, our 7B model improves over a comparable previous work by 34.1%, and even outperforms the 14B model from previous work in complex multihop QA such as HotpotQA by nearly 9% relative, maintaining exceptional training stability.
Read the full abstract ▾
All authors Chenyang Gu, Yewen Pu, Bruce Yang, Xiaofan Li, Huan Gao
Computation and Language
CrisiText: A dataset of warning messages for LLM training in emergency communication
Giacomo Gonella, Gian Maria Campedelli, Stefano Menini et al. (+1 more)
Effectively identifying threats and mitigating their potential damage during crisis situations, such as natural disasters or violent attacks, is paramount for safeguarding endangered individuals. To tackle these challenges, AI has been used in assisting humans in emergency situations. Still, the use of NLP techniques remains limited and mostly focuses on classification tasks. The significant potential of timely warning message generation using NLG architectures, however, has been largely overlooked. In this paper we present CrisiText, the first large-scale
dataset for the generation of warning messages across 13 different types of crisis scenarios. The
dataset contains more than 400,000 warning messages (spanning almost 18,000 crisis situations) aimed at assisting civilians during and after such events. To generate the
dataset, we started from existing crisis descriptions and created chains of events related to the scenarios. Each event was then paired with a warning message. The generations follow experts' written guidelines to ensure correct terminology and factuality of their suggestions. Additionally, each message is accompanied by three suboptimal warning types to allow for the study of different NLG approaches. To this end, we conducted a series of experiments comparing supervised fine-tuning setups with preference
alignment, zero-shot, and few-shot approaches. We further assessed model performance in out-of-distribution scenarios and evaluated the effectiveness of an automatic post-editor.
Read the full abstract ▾
All authors Giacomo Gonella, Gian Maria Campedelli, Stefano Menini, Marco Guerini
Computation and Language · Cross-listed from cs.CV
Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras
Jindong Hong, Wencheng Zhang, Shiqin Qiao et al. (+15 more)
Shoulder disorders, such as frozen shoulder (a.k.a., adhesive capsulitis), are common conditions affecting the health of people worldwide, and have a high incidence rate among the elderly and workers engaged in repetitive shoulder tasks. In regions with scarce medical resources, achieving early and accurate diagnosis poses significant challenges, and there is an urgent need for low-cost and easily scalable auxiliary diagnostic solutions. This research introduces
videos captured by consumer-grade devices as the basis for diagnosis, reducing the cost for users. We focus on the innovative application of
Multimodal Large
Language Models (MLLMs) in the preliminary diagnosis of shoulder disorders and propose a Hybrid Motion
Video Diagnosis framework (HMVDx). This framework divides the two tasks of action understanding and disease diagnosis, which are respectively completed by two MLLMs. In addition to traditional evaluation indicators, this work proposes a novel metric called Usability Index by the logical process of medical decision-making (action recognition, movement diagnosis, and final diagnosis). This index evaluates the effectiveness of MLLMs in the medical field from the perspective of the entire medical diagnostic pathway, revealing the potential value of low-cost MLLMs in medical applications for medical practitioners. In experimental comparisons, the accuracy of HMVDx in diagnosing shoulder joint injuries has increased by 79.6% compared with direct
video diagnosis, a significant technical contribution to future research on the application of MLLMs for
video understanding in the medical field.
Read the full abstract ▾
All authors Jindong Hong, Wencheng Zhang, Shiqin Qiao, Jianhai Chen, Jianing Qiu, Chuanyang Zheng, Qian Xu, Yun Ji, Qianyue Wen, Weiwei Sun, Hao Li, Huizhen Li, Huichao Wang, Kai Wu, Meng Li, Yijun He, Lingjie Luo, Jiankai Sun
Computation and Language · Cross-listed from eess.AS
Unsupervised lexicon learning from speech is limited by representations rather than clustering
Danel Slabbert, Simon Malan and Herman Kamper
Zero-resource word
segmentation and clustering systems aim to tokenise
speech into word-like units without access to text labels. Despite progress, the induced lexicons are still far from perfect. In an idealised setting with gold word boundaries, we ask whether performance is limited by the representation of word segments, or by the clustering methods that group them into word-like types. We combine a range of self-supervised
speech features (continuous/discrete, frame/word-level) with different clustering methods (K-means, hierarchical, graph-based) on English and Mandarin data. The best system uses graph clustering with dynamic time warping on continuous features. Faster alternatives use graph clustering with cosine distance on averaged continuous features or edit distance on discrete unit sequences. Through controlled experiments that isolate either the representations or the clustering method, we demonstrate that representation variability across segments of the same word type – rather than clustering – is the primary factor limiting performance.
Read the full abstract ▾
Computation and Language
IRIS: An Iterative and Integrated Framework for Verifiable Causal Discovery in the Absence of Tabular Data
Tao Feng, Lizhen Qu, Niket Tandon et al. (+1 more)
Causal discovery is fundamental to scientific research, yet traditional statistical
algorithms face significant challenges, including expensive data collection, redundant computation for known relations, and unrealistic assumptions. While recent LLM-based methods excel at identifying commonly known causal relations, they fail to uncover novel relations. We introduce IRIS (Iterative
Retrieval and Integrated System for Real-Time Causal Discovery), a novel framework that addresses these limitations. Starting with a set of initial variables, IRIS automatically collects relevant documents, extracts variables, and uncovers causal relations. Our hybrid causal discovery method combines statistical
algorithms and LLM-based methods to discover known and novel causal relations. In addition to causal discovery on initial variables, the missing variable proposal component of IRIS identifies and incorporates missing variables to expand the causal graphs. Our approach enables real-time causal discovery from only a set of initial variables without requiring pre-existing
datasets.
Read the full abstract ▾
All authors Tao Feng, Lizhen Qu, Niket Tandon, Gholamreza Haffari
Computation and Language
DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correction
Yiqi Li, Yusheng Liao, Zhe Chen et al. (+2 more)
When performing
reasoning tasks with user-specific requirements, such as strict output formats, large
language models (LLMs) often prioritize
reasoning over adherence to detailed instructions. Fine-tuning LLMs on supervised
datasets to address this is impractical due to high computational costs and limited parameter access. To tackle this, we propose DICE, a lightweight framework that guides small
language models (SLMs) to refine LLMs' outputs through chain-of-thought (CoT) correction. DICE decouples the process by first prompting LLMs to generate natural
language responses, then using trained SLMs to analyze and refine these outputs to meet structured output specifications. This framework preserves LLMs' broad knowledge and
reasoning capabilities while ensuring the outputs conform to user demands. Specifically, DICE first constructs structured CoT adaptation
datasets via a two-stage method and subsequently applies a dual-tuning strategy to fine-tune SLMs for generating structured outputs in an analyze-then-answer pattern. Experiments demonstrate that DICE improves the average format accuracy and content correctness of LLM outputs by 35.4% and 29.4%, respectively, achieving state-of-the-art (SOTA) performance over other competitive baselines.
Read the full abstract ▾
All authors Yiqi Li, Yusheng Liao, Zhe Chen, Yanfeng Wang, Yu Wang
Computation and Language · Cross-listed from cs.LG
Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs
Yumin Choi, Dongki Kim, Jinheon Baek et al. (+1 more)
Large
Language Models (LLMs) have shown remarkable success, and their
multimodal expansions (MLLMs) further unlock capabilities spanning images,
videos, and other modalities beyond text. However, despite this shift,
prompt optimization approaches, designed to reduce the burden of manual
prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of
multimodal prompt optimization, which expands the prior definition of
prompt optimization to the
multimodal space defined by the pairs of textual and non-textual
prompts. To tackle this problem, we then propose the
Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of
multimodal prompts through
alignment-preserving updates but also guides the selection process of candidate
prompts by leveraging earlier evaluations as priors in a
Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images,
videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing
multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.
Read the full abstract ▾
All authors Yumin Choi, Dongki Kim, Jinheon Baek, Sung Ju Hwang
Computation and Language
LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoning
Changjiang Gao, Zixian Huang, Jingyang Gong et al. (+3 more)
General Large
Language Models (LLMs) excel in
reasoning, but those enhanced for translation struggle with
reasoning tasks. To address this, we propose a novel translationenhanced recipe that begins with instruct models and applies layer-selective tuning only on parallel data. Following this pipeline, we introduce the Qwen3-XPlus models, which demonstrate significant improvements in translation performance across both high- and lowresource
languages, achieving 15+ spBLEU and 40+ xComet in low-resource
languages, like Swahili. Interestingly, training only with small parallel
datasets, Qwen3-XPlus achieves an average improvement of 1+ points on 7
multilingual tasks while maintaining proficiency comparable to the Qwen3 instruct model in 15 popular
reasoning datasets. This work offers a promising approach to
multilingual enhancement, significantly reducing complexity and enhancing accessibility for a wider range of
languages. The code and model are publicly available.
Read the full abstract ▾
All authors Changjiang Gao, Zixian Huang, Jingyang Gong, Shujian Huang, Lei Li, Fei Yuan
Computation and Language
Stronger Re-identification Attacks through Reasoning and Aggregation
Lucas Georges Gabriel Charpentier and Pierre Lison
Text de-identification techniques are often used to mask personally identifiable information (PII) from documents. Their ability to conceal the identity of the individuals mentioned in a text is, however, hard to measure. Recent work has shown how the robustness of de-identification methods could be assessed by attempting the reverse process of _re-identification_, based on an automated adversary using its background knowledge to uncover the PIIs that have been masked. This paper presents two complementary strategies to build stronger re-identification attacks. We first show that (1) the _order_ in which the PII spans are re-identified matters, and that aggregating predictions across multiple orderings leads to improved results. We also find that (2)
reasoning models can boost the re-identification performance, especially when the adversary is assumed to have access to extensive background knowledge.
Read the full abstract ▾
Computation and Language
Augmenting Dialog with Think-Aloud Utterances for Modeling Individual Personality Traits by LLM
Seiya Ishikura, Hiroaki Yamada, Tatsuya Hiraoka et al. (+2 more)
This study proposes augmenting dialog data with think-aloud utterances (TAUs) for modeling individual personalities in text chat by LLM. TAU is a verbalization of a speaker's thought before articulating the utterance. We expect "persona LLMs" trained with TAU-augmented data can mimic the speaker's personality trait better. We tested whether the trained persona LLMs obtain the human personality with respect to Big Five, a framework characterizing human personality traits from five aspects. The results showed that LLMs trained with TAU-augmented data more closely align to the speakers' Agreeableness and Neuroticism of Big Five than those trained with original dialog data. We also found that the quality of TAU-augmentation impacts persona LLM's performance.
Read the full abstract ▾
All authors Seiya Ishikura, Hiroaki Yamada, Tatsuya Hiraoka, Hiroaki Yamada, Takenobu Tokunaga
Computation and Language
DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation
Enze Zhang, Jiaying Wang, Mengxi Xiao et al. (+7 more)
Large
language models (LLMs) have substantially advanced machine translation (MT), yet their effectiveness in translating web novels remains unclear. Existing
benchmarks rely on surface-level metrics that fail to capture the distinctive traits of this genre. To address these gaps, we introduce DITING, the first comprehensive evaluation framework for web novel translation, assessing narrative and cultural fidelity across six dimensions: idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, and cultural safety, supported by over 18K expert-annotated Chinese-English sentence pairs. We further propose AgentEval, a
reasoning-driven multi-
agent evaluation framework that simulates expert deliberation to assess translation quality beyond lexical overlap, achieving the highest correlation with human judgments among seven tested automatic metrics. To enable metric comparison, we develop MetricAlign, a meta-evaluation
dataset of 300 sentence pairs annotated with error labels and scalar quality scores. Comprehensive evaluation of fourteen open, closed, and commercial models reveals that Chinese-trained LLMs surpass larger foreign counterparts, and that DeepSeek-V3 delivers the most faithful and stylistically coherent translations. Our work establishes a new paradigm for exploring LLM-based web novel translation and provides public resources to advance future research.
Read the full abstract ▾
All authors Enze Zhang, Jiaying Wang, Mengxi Xiao, Jifei Liu, Ziyan Kuang, Rui Dong, Eric Dong, Sophia Ananiadou, Min Peng, Qianqian Xie
Computation and Language
When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
Yongjie Wang, Yue Yu, Kaisong Song et al. (+2 more)
Large
Language Models (LLMs) have enabled a wide range of applications through their powerful capabilities in
language understanding and generation. However, as LLMs are trained on static corpora, they face difficulties in addressing rapidly evolving information or domain-specific queries.
Retrieval-Augmented Generation (RAG) was developed to overcome this limitation by integrating LLMs with external
retrieval mechanisms, allowing them to access up-to-date and contextually relevant knowledge. However, as LLMs themselves continue to advance in scale and capability, the relative advantages of traditional RAG frameworks have become less pronounced and necessary. Here, we present a comprehensive review of RAG, beginning with its overarching objectives and core components. We then analyze the key challenges within RAG, highlighting critical weakness that may limit its effectiveness. Finally, we showcase applications where LLMs alone perform inadequately, but where RAG, when combined with LLMs, can substantially enhance their effectiveness. We hope this work will encourage researchers to reconsider the role of RAG and inspire the development of next-generation RAG systems.
Read the full abstract ▾
All authors Yongjie Wang, Yue Yu, Kaisong Song, Jun Lin, Zhiqi Shen
Computation and Language
FrameEOL: Semantic Frame Induction using Causal Language Models
Chihiro Yano, Kosuke Yamada, Hayato Tsukagoshi et al. (+2 more)
Semantic frame induction is the task of clustering frame-evoking words according to the semantic frames they evoke. In recent years, leveraging
embeddings of frame-evoking words that are obtained using masked
language models (MLMs) such as BERT has led to high-performance semantic frame induction. Although causal
language models (CLMs) such as the GPT and Llama series succeed in a wide range of
language comprehension tasks and can engage in dialogue as if they understood frames, they have not yet been applied to semantic frame induction. We propose a new method for semantic frame induction based on CLMs. Specifically, we introduce FrameEOL, a
prompt-based method for obtaining Frame
Embeddings that outputs One frame-name as a Label representing the given situation. To obtain
embeddings more suitable for frame induction, we leverage in-context learning (ICL) and deep metric learning (DML). Frame induction is then performed by clustering the resulting
embeddings. Experimental results on the English and Japanese FrameNet
datasets demonstrate that the proposed methods outperform existing frame induction methods. In particular, for Japanese, which lacks extensive frame resources, the CLM-based method using only 5 ICL examples achieved comparable performance to the MLM-based method fine-tuned with DML.
Read the full abstract ▾
All authors Chihiro Yano, Kosuke Yamada, Hayato Tsukagoshi, Ryohei Sasano, Koichi Takeda
Computation and Language · Cross-listed from cs.CR
Exploiting Web Search Tools of AI Agents for Data Exfiltration
Dennis Rall, Bernhard Bauer, Mohit Mittal et al. (+1 more)
Large
language models (LLMs) are now routinely used to autonomously execute complex tasks, from natural
language processing to dynamic workflows like web searches. The usage of tool-calling and
Retrieval Augmented Generation (RAG) allows LLMs to process and retrieve sensitive corporate data, amplifying both their functionality and vulnerability to abuse. As LLMs increasingly interact with external data sources, indirect
prompt injection emerges as a critical and evolving attack vector, enabling adversaries to exploit models through manipulated inputs. Through a systematic evaluation of indirect
prompt injection attacks across diverse models, we analyze how susceptible current LLMs are to such attacks, which parameters, including model size and manufacturer, specific implementations, shape their vulnerability, and which attack methods remain most effective. Our results reveal that even well-known attack patterns continue to succeed, exposing persistent weaknesses in model defenses. To address these vulnerabilities, we emphasize the need for strengthened training procedures to enhance inherent resilience, a centralized database of known attack vectors to enable proactive defense, and a unified testing framework to ensure continuous security validation. These steps are essential to push developers toward integrating security into the core design of LLMs, as our findings show that current models still fail to mitigate long-standing threats.
Read the full abstract ▾
All authors Dennis Rall, Bernhard Bauer, Mohit Mittal, Thomas Fraunholz
Computation and Language
ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability
Chung-En Sun, Ge Yan, Akshay Kulkarni et al. (+1 more)
Recent advances in long chain-of-thought (CoT)
reasoning have largely prioritized answer accuracy and
token efficiency, while overlooking aspects critical to trustworthiness. We argue that usable
reasoning systems must be trustworthy, characterized by three properties: interpretability, faithfulness, and reliability. To this end, we propose ReFIne, a new training framework that integrates supervised fine-tuning with GRPO to encourage models to: (i) improve interpretability by producing structured, tag-based traces with high-level planning that are easier for humans to follow; (ii) enhance faithfulness by explicitly disclosing the decisive information guiding each solution, with consistent cross-section references; and (iii) promote reliability by providing self-assessments of both the derivation's soundness and the confidence of the final answer. We apply ReFIne to the Qwen3 models at multiple scales (1.7B/4B/8B) and evaluate across mathematical
benchmarks of varying difficulty. Our experimental results show that ReFIne models generate clearer and better-structured
reasoning traces (interpretability +44.0%), more faithfully expose their underlying decision process (faithfulness +18.8%), and offer informative confidence estimates (reliability +42.4%). These findings highlight an overlooked but important direction:
reasoning models should be optimized not only for accuracy, but also for broader dimensions of trustworthiness. Our code is available at: https://github.com/Trustworthy-ML-Lab/Training_Trustworthy_LRM_with_Refine
Read the full abstract ▾
All authors Chung-En Sun, Ge Yan, Akshay Kulkarni, Tsui-Wei Weng
Computation and Language
Muhammad Ali Shafique, Kanwal Mehreen, Muhammad Arham et al. (+3 more)
Developing a high-performing large
language models (LLMs) for low-resource
languages such as Urdu, present several challenges. These challenges include the scarcity of high-quality
datasets,
multilingual inconsistencies, and safety concerns. Existing
multilingual LLMs often address these issues by translating large volumes of available data. However, such translations often lack quality and cultural nuance while also incurring significant costs for data curation and training. To address these issues, we propose Alif-1.0-8B-Instruct, a
multilingual Urdu-English model, that tackles these challenges with a unique approach. We train the model on a high-quality,
multilingual synthetic
dataset (Urdu-Instruct), developed using a modified self-instruct technique. By using unique
prompts and seed values for each task along with a global task pool, this
dataset incorporates Urdu-native chain-of-thought based
reasoning, bilingual translation, cultural relevance, and ethical safety
alignments. This technique significantly enhances the comprehension of Alif-1.0-8B-Instruct model for Urdu-specific tasks. As a result, Alif-1.0-8B-Instruct, built upon the pretrained Llama-3.1-8B, demonstrates superior performance compared to Llama-3.1-8B-Instruct for Urdu specific-tasks. It also outperformed leading
multilingual LLMs, including Mistral-7B-Instruct-v0.3, Qwen-2.5-7B-Instruct, and Cohere-Aya-Expanse-8B, all within a training budget of under $100. Our results demonstrate that high-performance and low-resource
language LLMs can be developed efficiently and culturally aligned using our modified self-instruct approach. All
datasets, models, and code are publicly available at: https://github.com/traversaal-ai/alif-urdu-llm.
Read the full abstract ▾
All authors Muhammad Ali Shafique, Kanwal Mehreen, Muhammad Arham, Maaz Amjad, Sabur Butt, Hamza Farooq
Computation and Language · Cross-listed from cs.AI
Auto-scaling Continuous Memory for GUI Agent
Wenyi Wu, Kun Zhou, Ruoxin Yuan et al. (+4 more)
We study how to endow GUI
agents with scalable
memory that help generalize across unfamiliar interfaces and long-horizon tasks. Prior GUI
agents compress past trajectories into text
tokens, which balloons context length and misses decisive visual cues (e.g., exact widget size and position). We propose a continuous
memory that encodes each GUI trajectory into a fixed-length sequence of continuous
embeddings using the VLM itself as an encoder; these
embeddings are plugged directly into the backbone's input layer, sharply reducing context cost while preserving fine-grained visual information. As
memory size and
retrieval depth increase, performance improves monotonically, unlike text
memories that degrade with long
prompts. To grow
memory at low cost, we introduce an auto-scaling data flywheel that (i) discovers new environments via search, (ii) synthesizes tasks with an open-source VLM, (iii) rolls out trajectories with the
agent, and (iv) verifies success with the same VLM. Using this pipeline, we collect 100k+ trajectories for about \$4000 and fine-tune only the
memory encoder (LoRA on a Q-Former, 1.2% parameters) with 1,500 samples. On real-world GUI
benchmarks, our
memory-augmented
agent consistently improves success rates under long horizons and distribution shifts. Notably, Qwen-2.5-VL-7B + continuous
memory achieves performance comparable to state-of-the-art closed-source models (e.g., GPT-4o, Claude-4).
Read the full abstract ▾
All authors Wenyi Wu, Kun Zhou, Ruoxin Yuan, Vivian Yu, Stephen Wang, Zhiting Hu, Biwei Huang
Computation and Language
Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness
Chi Seng Cheang, Hou Pong Chan, Wenxuan Zhang et al. (+1 more)
Recent work suggests that LLMs "know what they don't know", positing that hallucinated and factually correct outputs arise from distinct internal processes and can therefore be distinguished using internal signals. However,
hallucinations have multifaceted causes: beyond simple knowledge gaps, they can emerge from training incentives that encourage models to exploit statistical shortcuts or spurious associations learned during pretraining. In this paper, we argue that when LLMs rely on such learned associations to produce
hallucinations, their internal processes are mechanistically similar to those of factual recall, as both stem from strong statistical correlations encoded in the model's parameters. To verify this, we propose a novel taxonomy categorizing
hallucinations into Unassociated
Hallucinations (UHs), where outputs lack parametric grounding, and Associated
Hallucinations (AHs), which are driven by spurious associations. Through mechanistic analysis, we compare their computational processes and hidden-state geometries with factually correct outputs. Our results show that hidden states primarily reflect whether the model is recalling parametric knowledge rather than the truthfulness of the output itself. Consequently, AHs exhibit hidden-state geometries that largely overlap with factual outputs, rendering standard detection methods ineffective. In contrast, UHs exhibit distinctive, clustered representations that facilitate reliable detection.
Read the full abstract ▾
All authors Chi Seng Cheang, Hou Pong Chan, Wenxuan Zhang, Yang Deng
Computation and Language
Exploring Cross-Lingual Knowledge Transfer via Transliteration-Based MLM Fine-Tuning for Critically Low-resource Chakma Language
Adity Khisa, Nusrat Jahan Lia, Tasnim Mahfuz Nafis et al. (+4 more)
As an Indo-Aryan
language with limited available data, Chakma remains largely underrepresented in
language models. In this work, we introduce a novel corpus of contextually coherent Bangla-transliterated Chakma, curated from Chakma literature, and validated by native speakers. Using this
dataset, we fine-tune six encoder-based
transformer models, including
multilingual (mBERT, XLM-RoBERTa, DistilBERT), regional (BanglaBERT, IndicBERT), and monolingual English (DeBERTaV3) variants on masked
language modeling (MLM) tasks. Our experiments show that fine-tuned
multilingual models outperform their pre-trained counterparts when adapted to Bangla-transliterated Chakma, achieving up to 73.54%
token accuracy and a perplexity as low as 2.90. Our analysis further highlights the impact of data quality on model performance and shows the limitations of OCR pipelines for morphologically rich Indic scripts. Our research demonstrates that Bangla-transliterated Chakma can be very effective for transfer learning for Chakma
language, and we release our
dataset to encourage further research on
multilingual language modeling for low-resource
languages.
Read the full abstract ▾
All authors Adity Khisa, Nusrat Jahan Lia, Tasnim Mahfuz Nafis, Zarif Masud, Tanzir Pial, Shebuti Rayana, Ahmedul Kabir
Computation and Language
Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise
Keno Harada, Lui Yoshida, Takeshi Kojima et al. (+2 more)
The performance of Large
Language Models (LLMs) is highly sensitive to the
prompts they are given. Drawing inspiration from the field of
prompt optimization, this study investigates the potential for enhancing Automated Essay Scoring (AES) by refining the scoring rubrics used by LLMs. Specifically, our approach
prompts models to iteratively refine rubrics by reflecting on models' own scoring rationales and observed discrepancies with human scores on sample essays. Experiments on the TOEFL11 and ASAP
datasets using GPT-4.1, Gemini-2.5-Pro, and Qwen-3-Next-80B-A3B-Instruct show Quadratic Weighted Kappa (QWK) improvements of up to 0.19 and 0.47, respectively. Notably, even with a simple initial rubric, our approach achieves comparable or better QWK than using detailed human-authored rubrics. Our findings highlight the importance of iterative rubric refinement in LLM-based AES to enhance
alignment with human evaluations.
Read the full abstract ▾
All authors Keno Harada, Lui Yoshida, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
Computation and Language
LitE-SQL: A Lightweight and Efficient Text-to-SQL Framework with Vector-based Schema Linking and Execution-Guided Self-Correction
Shengmin Piao, Jieun Lee and Sanghyun Park
The Text-to-SQL task translates natural
language questions into SQL queries, enabling intuitive database interaction for non-experts. While recent methods leveraging Large
Language Models (LLMs) achieve strong performance, their reliance on proprietary models raise concerns about deployment feasibility and data
privacy. In this work, we introduce LitE-SQL, a Lightweight and Efficient framework with two components: (i) a Schema Retriever that performs efficient schema linking using a vector database of pre-computed schema
embeddings, optimized with a hard-negative supervised contrastive objective to distinguish semantically similar but functionally irrelevant columns, and (ii) a SQL Generator fine-tuned in two stages-supervised fine-tuning followed by execution-guided reinforcement-enabling execution-guided self-correction without multi-candidate sampling, which is commonly required by prior LLM-based approaches. On BIRD, LitE-SQL achieves 72.10% execution accuracy, and on Spider 1.0 it reaches 88.45%, demonstrating comparable or superior performance to LLM-based methods despite using 2x to 30x fewer parameters. Our findings demonstrate that high-quality Text-to-SQL generation is feasible with lightweight models, offering a practical solution for
privacy-sensitive and resource-constrained settings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward
Yincen Qu, Huan Xiao, Feng Li et al. (+5 more)
In our deployed travel-planning service, most users give minimal inputs or free-form requests rather than the structured constraint checklists assumed by existing
benchmarks. We therefore present TripScore, a behavior-grounded
benchmark and evaluation framework built from real user logs and calibrated against 1,468 pairwise judgments by 203 travel experts. TripScore couples a hierarchical feasibility gate (format and commonsense) with a unified, point-wise reward that aggregates soft quality and preference fulfillment. Using TripScore as both evaluator and reward signal, we
benchmark direct prompting, test-time compute, neuro-symbolic solvers, code
agents, and fine-tuning. We find that reinforcement learning fine-tuning (e.g., GRPO) provides consistent gains over other approaches under the same base model and practical latency.
Read the full abstract ▾
All authors Yincen Qu, Huan Xiao, Feng Li, Gregory Li, Hui Zhou, Xiangying Dai, Xiaoru Dai, Xuan Huang
Computation and Language · Cross-listed from cs.CV
Hoigi Seo, Dong Un Kang, Hyunjin Cho et al. (+2 more)
Large
vision-
language models (LVLMs), which integrate a
vision encoder (VE) with a large
language model, have achieved remarkable success across various tasks. However, there are still crucial challenges in LVLMs such as object
hallucination, generating descriptions of objects that are not in the input image. Here, we argue that uncertain visual
tokens within the VE is a key factor that contributes to object
hallucination. Our statistical analysis found that there are positive correlations between visual
tokens with high epistemic uncertainty and the occurrence of
hallucinations. Furthermore, we show theoretically and empirically that visual
tokens in early VE layers that exhibit large representation deviations under small
adversarial perturbations indicate high epistemic uncertainty. Based on these findings, we propose a simple yet effective strategy to mitigate object
hallucination by modifying the VE only. Our method comprises a proxy method with
adversarial perturbations for identifying uncertain visual
tokens efficiently and a method to mask these uncertain visual
tokens during the self-
attention process in the middle layers of the VE, suppressing their influence on visual encoding and thus alleviating
hallucinations. Extensive experiments show that our method significantly reduces object
hallucinations in LVLMs and can synergistically work with other prior arts.
Read the full abstract ▾
All authors Hoigi Seo, Dong Un Kang, Hyunjin Cho, Joohoon Lee, Se Young Chun
Computation and Language
Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Models
Yutao Mou, Xiaoling Zhou, Yuxiao Luo et al. (+2 more)
Safety
alignment is essential for building trustworthy artificial intelligence, yet it remains challenging to enhance model safety without degrading general performance. Current approaches require computationally expensive searches for the optimal proportion of safety-critical and general-purpose data to balance safety and general performance, incurring high costs with limited gains. In this work, we show that LoRA-based Refusal-training enables performance-preserving safety
alignment even when trained solely on safety data, demonstrating that LoRA serves as cost-efficient, performance-preserving, and plug-and-play safety patches. Beyond empirical findings, we provide both theoretical and experimental evidence that LoRA effectively decouples safety into a low-rank subspace largely orthogonal to the model's intrinsic transformation space, ensuring that safety enhancements do not interfere with inherent capabilities.
Read the full abstract ▾
All authors Yutao Mou, Xiaoling Zhou, Yuxiao Luo, Shikun Zhang, Wei Ye
Computation and Language
DARO: Difficulty-Aware Reweighting Policy Optimization
Jingyu Zhou, Lu Ma, Hao Liang et al. (+3 more)
Recent advances in large
language models (LLMs) have shown that
reasoning ability can be significantly enhanced through Reinforcement Learning with Verifiable Rewards (RLVR). Group Relative Policy Optimization (GRPO) has emerged as the de facto approach for RLVR, inspiring numerous variants. However, our mathematical analysis reveals that these methods are fundamentally weighted variations of GRPO. We provide a unified view, demonstrating that their reliance on static or overly simplistic weighting schemes tied to sample difficulty prevents adaptation to a model's evolving capabilities. This creates a significant loss scale issue, where training disproportionately focuses on certain difficulty levels at the expense of others, hindering overall performance. To address these limitations, we introduce Difficulty-Aware Reweighting Policy Optimization (DARO), a method that dynamically adjusts the loss contribution of each difficulty group based on the model's learning state. Extensive experiments on Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Llama3.1-8B show that DARO outperforms four leading baselines across six math
benchmarks, achieving significantly faster convergence and superior final performance.
Read the full abstract ▾
All authors Jingyu Zhou, Lu Ma, Hao Liang, Chengyu Shen, Bin Cui, Wentao Zhang
Computation and Language
MASA: LLM-Driven Multi-Agent Systems for Autoformalization
Lan Zhang, Marco Valentino and André Freitas
Autoformalization serves a crucial role in connecting natural
language and formal
reasoning. This paper presents MASA, a novel framework for building multi-
agent systems for autoformalization driven by Large
Language Models (LLMs). MASA leverages collaborative
agents to convert natural
language statements into their formal representations. The architecture of MASA is designed with a strong emphasis on modularity, flexibility, and extensibility, allowing seamless integration of new
agents and tools to adapt to a fast-evolving field. We showcase the effectiveness of MASA through use cases on real-world mathematical definitions and experiments on formal mathematics
datasets. This work highlights the potential of multi-
agent systems powered by the interaction of LLMs and
theorem provers in enhancing the efficiency and reliability of autoformalization, providing valuable insights and support for researchers and practitioners in the field.
Read the full abstract ▾
Computation and Language
CAPC-CG: A Large-Scale, Expert-Directed LLM-Annotated Corpus of Adaptive Policy Communication in China
Bolun Sun, Charles Chang, Yuen Yuen Ang et al. (+4 more)
We introduce CAPC-CG, the Chinese Adaptive Policy Communication (Central Government) Corpus, the first open
dataset of Chinese policy directives annotated with a five-color taxonomy of clear and ambiguous
language categories, building on Ang's theory of adaptive policy communication. Spanning 1949-2023, this corpus includes national laws, administrative regulations, and ministerial rules issued by China's top authorities. Each document is segmented into paragraphs, producing a total of 3.3 million units. Alongside the corpus, we release comprehensive metadata, a two-round labeling framework, and a gold-standard annotation set developed by expert and trained coders. Inter-annotator agreement achieves a Fleiss's kappa of K = 0.86 on directive labels, indicating high reliability for supervised modeling. We provide baseline classification results with several large
language models (LLMs), together with our annotation codebook, and describe patterns from the
dataset. This release aims to support downstream tasks and
multilingual NLP research in policy communication.
Read the full abstract ▾
All authors Bolun Sun, Charles Chang, Yuen Yuen Ang, Ruotong Mu, Yuchen Xu, Zhengxin Zhang, Pingxu Hao