Computation and Language
Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning
Aozhe Wang, Yuchen Yan, Nan Zhou et al. (+5 more)
Reinforcement learning for code generation relies on verifiable rewards from unit test pass rates. Yet high-quality test suites are scarce, existing
datasets offer limited coverage, and static rewards fail to adapt as models improve. Recent self-play methods unify code and test generation in a single model, but face a inherent dilemma: white-box access leads to self-collusion where the model produces trivial tests for easy rewards, yet black-box restriction yields generic tests that miss implementation-specific bugs. We introduce Code-A1, an
adversarial co-
evolution framework that jointly optimizes a Code LLM and a Test LLM with opposing objectives. The Code LLM is rewarded for passing more tests, while the Test LLM is rewarded for exposing more defects. This architectural separation eliminates self-collusion risks and safely enables white-box test generation, where the Test LLM can inspect candidate code to craft targeted
adversarial tests. We further introduce a Mistake Book mechanism for experience replay and a composite reward balancing test validity with
adversarial difficulty. Experiments on Qwen2.5-Coder models demonstrate that Code-A1 achieves code generation performance matching or exceeding models trained on human-annotated tests, while significantly improving test generation capability.
Read the full abstract ▾
All authors Aozhe Wang, Yuchen Yan, Nan Zhou, Zhengxi Lu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
Computation and Language · Cross-listed from cs.RO
From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation
Yibin Liu, Yaxing Lyu, Daqi Gao et al. (+5 more)
Accurate process supervision remains a critical challenge for long-horizon robotic manipulation. A primary bottleneck is that current
video MLLMs, trained primarily under a Supervised Fine-Tuning (SFT) paradigm, function as passive "Observers" that recognize ongoing events rather than evaluating the current state relative to the final task goal. In this paper, we introduce PRIMO R1 (Process
Reasoning Induced Monitoring), a 7B framework that transforms
video MLLMs into active "Critics". We leverage outcome-based Reinforcement Learning to incentivize explicit Chain-of-Thought generation for progress estimation. Furthermore, our architecture constructs a structured temporal input by explicitly anchoring the
video sequence between initial and current state images. Supported by the proposed PRIMO
Dataset and
Benchmark, extensive experiments across diverse in-domain environments and out-of-domain real-world
humanoid scenarios demonstrate that PRIMO R1 achieves state-of-the-art performance. Quantitatively, our 7B model achieves a 50% reduction in the mean absolute error of specialized
reasoning baselines, demonstrating significant relative accuracy improvements over 72B-scale general MLLMs. Furthermore, PRIMO R1 exhibits strong zero-shot generalization on difficult failure detection tasks. We establish state-of-the-art performance on RoboFail
benchmark with 67.0% accuracy, surpassing closed-source models like OpenAI o1 by 6.0%.
Read the full abstract ▾
All authors Yibin Liu, Yaxing Lyu, Daqi Gao, Zhixuan Liang, Weiliang Tang, Shilong Mu, Xiaokang Yang, Yao Mu
Computation and Language · Cross-listed from cs.AI
OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data
Yuwen Du, Rui Ye, Shuo Tang et al. (+4 more)
Deep search capabilities have become an indispensable competency for frontier Large
Language Model (LLM)
agents, yet the development of high-performance search
agents remains dominated by industrial
giants due to a lack of transparent, high-quality training data. This persistent data scarcity has fundamentally hindered the progress of the broader research community in developing and innovating within this domain. To bridge this gap, we introduce OpenSeeker, the first fully open-source search
agent (i.e., model and data) that achieves frontier-level performance through two core technical innovations: (1) Fact-grounded scalable controllable QA synthesis, which reverse-engineers the web graph via topological expansion and entity obfuscation to generate complex, multi-hop
reasoning tasks with controllable coverage and complexity. (2) Denoised trajectory synthesis, which employs a retrospective summarization mechanism to denoise the trajectory, therefore promoting the teacher LLMs to generate high-quality actions. Experimental results demonstrate that OpenSeeker, trained (a single training run) on only 11.7k synthesized samples, achieves state-of-the-art performance across multiple
benchmarks including BrowseComp, BrowseComp-ZH, xbench-DeepSearch, and WideSearch. Notably, trained with simple SFT, OpenSeeker significantly outperforms the second-best fully open-source
agent DeepDive (e.g., 29.5% v.s. 15.3% on BrowseComp), and even surpasses industrial competitors such as Tongyi DeepResearch (trained via extensive continual pre-training, SFT, and RL) on BrowseComp-ZH (48.4% v.s. 46.7%). We fully open-source the complete training
dataset and the model weights to democratize frontier search
agent research and foster a more transparent, collaborative ecosystem.
Read the full abstract ▾
All authors Yuwen Du, Rui Ye, Shuo Tang, Xinyu Zhu, Yijun Lu, Yuzhu Cai, Siheng Chen
Computation and Language
Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation
Yanick Zengaffinen, Andreas Opedal, Donya Rooein et al. (+3 more)
Modeling student misconceptions in a realistic manner is critical for AI in education. In this work, we examine how large
language models (LLMs) reason about misconceptions when generating distractor answers for multiple-choice questions (MCQs), a task that requires producing answers that are incorrect, yet plausible. We introduce a taxonomy over
reasoning strategies for distractor generation that is grounded in learning-science literature and empirical observation, which we apply to LLM-generated
reasoning traces across math and science MCQs. On the math
dataset, we find that models follow a misconception-based process with potentially high diagnostic value: they recover the correct solution, articulate student errors, simulate them, and select plausible candidates. On the science
dataset, on the other hand, they tend to follow a less robust approach based on semantic similarity to the correct answer. We find the most frequent failure modes to be that the model is unable to generate a correct solution or that it discards plausible distractor candidates when performing selection. Providing the correct solution in the
prompt yields a relative improvement of 6.4% in
alignment with human-authored distractors, highlighting the critical role of anchoring distractor generation to the correct solution. Together, our findings offer an interpretable view of how LLMs model incorrect student
reasoning.
Read the full abstract ▾
All authors Yanick Zengaffinen, Andreas Opedal, Donya Rooein, Kv Aditya Srivatsa, Shashank Sonkar, Mrinmaya Sachan
Computation and Language
SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction
David Števaňák and Marek Šuppa
Keyphrase extraction for morphologically rich, low-resource
languages remains understudied, largely due to the scarcity of suitable evaluation
datasets. We address this gap for Slovak by constructing a
dataset of 227,432 scientific abstracts with author-assigned keyphrases – scraped and systematically cleaned from the Slovak Central Register of Theses – representing a 25-fold increase over the largest prior Slovak resource and approaching the scale of established English
benchmarks such as KP20K. Using this
dataset, we
benchmark three unsupervised baselines (YAKE, TextRank, KeyBERT with SlovakBERT
embeddings) and evaluate KeyLLM, an LLM-based extraction method using GPT-3.5-turbo. Unsupervised baselines achieve at most 11.6% exact-match $F1@6$, with a large gap to partial matching (up to 51.5%), reflecting the difficulty of matching inflected surface forms to author-assigned keyphrases. KeyLLM narrows this exact–partial gap, producing keyphrases closer to the canonical forms assigned by authors, while manual evaluation on 100 documents ($\kappa = 0.61$) confirms that KeyLLM captures relevant concepts that automated exact matching underestimates. Our analysis identifies morphological mismatch as the dominant failure mode for statistical methods – a finding relevant to other inflected
languages. The
dataset (https://huggingface.co/
datasets/NaiveNeuron/SlovKE) and evaluation code (https://github.com/NaiveNeuron/SlovKE) are publicly available.
Read the full abstract ▾
Computation and Language
Beyond the Covariance Trap: Unlocking Generalization in Same-Subject Knowledge Editing for Large Language Models
Xiyu Liu, Qingyi Si, Zhengxiao Liu et al. (+3 more)
While locate-then-edit knowledge editing efficiently updates knowledge encoded within Large
Language Models (LLMs), a critical generalization failure mode emerges in the practical same-subject knowledge editing scenario: models fail to recall the updated knowledge when following user instructions, despite successfully recalling it in the original edited form. This paper identifies the
geometric root of this generalization collapse as a fundamental conflict where the inner activation drifts induced by
prompt variations exceed the model's
geometric tolerance for generalization after editing. We attribute this instability to a dual pathology: (1) The joint optimization with orthogonal gradients collapses solutions into sharp minima with narrow stability, and (2) the standard covariance constraint paradoxically acts as a Covariance Trap that amplifies input perturbations. To resolve this, we introduce RoSE (Robust Same-subject Editing), which employs Isotropic
Geometric Alignment to minimize representational deviation and Hierarchical Knowledge Integration to smooth the optimization landscape. Extensive experiments demonstrate that RoSE significantly improves instruction-following capabilities, laying the foundation for robust interactive parametric
memory of LLM
agents.
Read the full abstract ▾
All authors Xiyu Liu, Qingyi Si, Zhengxiao Liu, Chenxu Yang, Naibin Gu, Zheng Lin
Computation and Language
ViX-Ray: A Vietnamese Chest X-Ray Dataset for Vision-Language Models
Duy Vu Minh Nguyen, Chinh Thanh Truong, Phuc Hoang Tran et al. (+4 more)
Vietnamese medical research has become an increasingly vital domain, particularly with the rise of intelligent technologies aimed at reducing time and resource burdens in clinical diagnosis. Recent advances in
vision-
language models (VLMs), such as Gemini and GPT-4V, have sparked a growing interest in applying AI to healthcare. However, most existing VLMs lack exposure to Vietnamese medical data, limiting their ability to generate accurate and contextually appropriate diagnostic outputs for Vietnamese patients. To address this challenge, we introduce ViX-Ray, a novel
dataset comprising 5,400 Vietnamese chest X-ray images annotated with expert-written findings and impressions from physicians at a major Vietnamese hospital. We analyze linguistic patterns within the
dataset, including the frequency of mentioned body parts and diagnoses, to identify domain-specific linguistic characteristics of Vietnamese radiology reports. Furthermore, we fine-tune five state-of-the-art open-source VLMs on ViX-Ray and compare their performance to leading proprietary models, GPT-4V and Gemini. Our results show that while several models generate outputs partially aligned with clinical ground truths, they often suffer from low precision and excessive
hallucination, especially in impression generation. These findings not only demonstrate the complexity and challenge of our
dataset but also establish ViX-Ray as a valuable
benchmark for evaluating and advancing
vision-
language models in the Vietnamese clinical domain.
Read the full abstract ▾
All authors Duy Vu Minh Nguyen, Chinh Thanh Truong, Phuc Hoang Tran, Hung Tuan Le, Nguyen Van-Thanh Dat, Trung Hieu Pham, Kiet Van Nguyen
Computation and Language
Invisible failures in human-AI interactions
Christopher Potts and Moritz Sudhof
AI systems fail silently far more often than they fail visibly. In an analysis of 100K human-AI interactions from the WildChat
dataset, we find that 79% of AI failures are invisible: something went wrong but the user gave no overt indication that there was a problem. These invisible failures cluster into eight archetypes that help us characterize where and how AI systems are failing to meet users' needs. In addition, the archetypes show systematic co-occurrence patterns indicating higher-level failure types. To address the question of whether these archetypes will remain relevant as AI systems become more capable, we also created and annotated a counterfactual
dataset in which WildChat's 2024-era responses are replaced by those from three present-day frontier LMs. This analysis indicates that failure rates have dropped substantially, but that the vast majority of failures remain invisible in our sense, and the distribution of failure archetypes seems stable. Finally, we illustrate how the archetypes help us to identify systematic and variable AI limitations across different usage domains. Overall, we argue that our invisible failure taxonomy can be a key component in reliable failure monitoring for product developers, scientists, and policy makers. Our code and data are available at https://github.com/bigspinai/bigspin-invisible-failure-archetypes
Read the full abstract ▾
Computation and Language
CLAG: Adaptive Memory Organization via Agent-Driven Clustering for Small Language Model Agents
Taeyun Roh, Wonjune Jang, Junha Jung et al. (+1 more)
Large
language model
agents heavily rely on external
memory to support knowledge reuse and complex
reasoning tasks. Yet most
memory systems store experiences in a single global
retrieval pool which can gradually dilute or corrupt stored knowledge. This problem is especially pronounced for small
language models (SLMs), which are highly vulnerable to irrelevant context. We introduce CLAG, a CLustering-based AGentic
memory framework where an SLM
agent actively organizes
memory by clustering. CLAG employs an SLM-driven router to assign incoming
memories to semantically coherent clusters and autonomously generates cluster-specific profiles, including topic summaries and descriptive tags, to establish each cluster as a self-contained functional unit. By performing localized
evolution within these structured neighborhoods, CLAG effectively reduces cross-topic interference and enhances internal
memory density. During
retrieval, the framework utilizes a two-stage process that first filters relevant clusters via their profiles, thereby excluding distractors and reducing the search space. Experiments on multiple QA
datasets with three SLM backbones show that CLAG consistently improves answer quality and robustness over prior
memory systems for
agents, remaining lightweight and efficient.
Read the full abstract ▾
All authors Taeyun Roh, Wonjune Jang, Junha Jung, Jaewoo Kang
Computation and Language · Cross-listed from cs.LG
Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities
Vanshaj Khattar, Md Rafi ur Rashid, Moumita Choudhury et al. (+4 more)
Test-time training (TTT) has recently emerged as a promising method to improve the
reasoning abilities of large
language models (LLMs), in which the model directly learns from test data without access to labels. However, this reliance on test data also makes TTT methods vulnerable to harmful
prompt injections. In this paper, we investigate safety vulnerabilities of TTT methods, where we study a representative self-consistency-based test-time learning method: test-time reinforcement learning (TTRL), a recent TTT method that improves LLM
reasoning by rewarding self-consistency using majority vote as a reward signal. We show that harmful
prompt injection during TTRL amplifies the model's existing behaviors, i.e., safety amplification when the base model is relatively safe, and harmfulness amplification when it is vulnerable to the injected data. In both cases, there is a decline in
reasoning ability, which we refer to as the
reasoning tax. We also show that TTT methods such as TTRL can be exploited adversarially using specially designed "HarmInject"
prompts to force the model to answer jailbreak and
reasoning queries together, resulting in stronger harmfulness amplification. Overall, our results highlight that TTT methods that enhance LLM
reasoning by promoting self-consistency can lead to amplification behaviors and
reasoning degradation, highlighting the need for safer TTT methods.
Read the full abstract ▾
All authors Vanshaj Khattar, Md Rafi ur Rashid, Moumita Choudhury, Jing Liu, Toshiaki Koike-Akino, Ming Jin, Ye Wang
Computation and Language
SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asia
Pengfei Yue, Xingran Zhao, Juntao Chen et al. (+6 more)
Multilingual document and scene text understanding plays an important role in applications such as search, finance, and public services. However, most existing
benchmarks focus on high-resource
languages and fail to evaluate models in realistic
multilingual environments. In Southeast Asia, the diversity of
languages, complex writing systems, and highly varied document types make this challenge even greater. We introduce SEA-
Vision, a
benchmark that jointly evaluates Document Parsing and Text-Centric Visual Question Answering (TEC-VQA) across 11 Southeast Asian
languages. SEA-
Vision contains 15,234 document parsing pages from nine representative document types, annotated with hierarchical page-, block-, and line-level labels. It also provides 7,496 TEC-VQA question-answer pairs that probe text recognition, numerical calculation, comparative analysis, logical
reasoning, and spatial understanding. To make such
multilingual, multi-task annotation feasible, we design a hybrid pipeline for Document Parsing and TEC-VQA. It combines automated filtering and scoring with MLLM-assisted labeling and lightweight native-speaker verification, greatly reducing manual labeling while maintaining high quality. We evaluate several leading
multimodal models and observe pronounced performance degradation on low-resource Southeast Asian
languages, highlighting substantial remaining gaps in
multilingual document and scene text understanding. We believe SEA-
Vision will help drive global progress in document and scene text understanding.
Read the full abstract ▾
All authors Pengfei Yue, Xingran Zhao, Juntao Chen, Peng Hou, Wang Longchao, Jianghang Lin, Shengchuan Zhang, Anxiang Zeng, Liujuan Cao
Computation and Language · Cross-listed from cs.CR
TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems
Kai Wang, Biaojie Zeng, Zeming Wei et al. (+7 more)
With the rapid development of LLM-based multi-
agent systems (MAS), their significant safety and security concerns have emerged, which introduce novel risks going beyond single
agents or LLMs. Despite attempts to address these issues, the existing literature lacks a cohesive safeguarding system specialized for MAS risks. In this work, we introduce TrinityGuard, a comprehensive safety evaluation and monitoring framework for LLM-based MAS, grounded in the OWASP standards. Specifically, TrinityGuard encompasses a three-tier fine-grained risk taxonomy that identifies 20 risk types, covering single-
agent vulnerabilities, inter-
agent communication threats, and system-level emergent hazards. Designed for scalability across various MAS structures and platforms, TrinityGuard is organized in a trinity manner, involving an MAS abstraction layer that can be adapted to any MAS structures, an evaluation layer containing risk-specific test modules, alongside runtime monitor
agents coordinated by a unified LLM Judge Factory. During Evaluation, TrinityGuard executes curated attack probes to generate detailed vulnerability reports for each risk type, where monitor
agents analyze structured execution traces and issue real-time alerts, enabling both pre-development evaluation and runtime monitoring. We further formalize these safety metrics and present detailed case studies across various representative MAS examples, showcasing the versatility and reliability of TrinityGuard. Overall, TrinityGuard acts as a comprehensive framework for evaluating and monitoring various risks in MAS, paving the way for further research into their safety and security.
Read the full abstract ▾
All authors Kai Wang, Biaojie Zeng, Zeming Wei, Chang Jin, Hefeng Zhou, Xiangtian Li, Chao Yang, Jingjing Qu, Xingcheng Xu, Xia Hu
Computation and Language
Fusian: Multi-LoRA Fusion for Fine-Grained Continuous MBTI Personality Control in Large Language Models
Zehao Chen and Rong Pan
Large
Language Models (LLMs) have demonstrated impressive capabilities in simulating diverse human behaviors and personalities. However, existing methods for personality control, which include
prompt engineering and standard Supervised Fine-Tuning (SFT), typically treat personality traits as discrete categories (e.g., "Extroverted" vs. "Introverted"), lacking the ability to precisely control the intensity of a trait on a continuous spectrum. In this paper, we introduce Fusian, a novel framework for fine-grained, continuous personality control in LLMs. Fusian operates in two stages: (1) Trajectory Collection, where we capture the dynamic
evolution of personality adoption during SFT by saving a sequence of LoRA adapters, effectively mapping the continuous
manifold of a trait; and (2) RL-based Dynamic Fusion, where we train a policy network using Reinforcement Learning to dynamically compute mixing weights for these frozen adapters. By sampling from a Dirichlet distribution parameterized by the policy network, Fusian fuses multiple adapters to align the model's output with a specific numerical target intensity. Experiments on the Qwen3-14B model demonstrate that Fusian achieves high precision in personality control, significantly outperforming baseline methods in aligning with user-specified trait intensities.
Read the full abstract ▾
Computation and Language
A Closer Look into LLMs for Table Understanding
Jia Wang, Chuanyu Qin, Mingyu Zheng et al. (+3 more)
Despite the success of Large
Language Models (LLMs) in table understanding, their internal mechanisms remain unclear. In this paper, we conduct an empirical study on 16 LLMs, covering general LLMs, specialist tabular LLMs, and Mixture-of-Experts (MoE) models, to explore how LLMs understand tabular data and perform downstream tasks. Our analysis focus on 4 dimensions including the
attention dynamics, the effective layer depth, the expert activation, and the impacts of input designs. Key findings include: (1) LLMs follow a three-phase
attention pattern – early layers scan the table broadly, middle layers localize relevant
cells, and late layers amplify their contributions; (2) tabular tasks require deeper layers than math
reasoning to reach stable predictions; (3) MoE models activate table-specific experts in middle layers, with early and late layers sharing general-purpose experts; (4) Chain-of-Thought prompting increases table
attention, further enhanced by table-tuning. We hope these findings and insights can facilitate interpretability and future research on table-related tasks.
Read the full abstract ▾
All authors Jia Wang, Chuanyu Qin, Mingyu Zheng, Qingyi Si, Peize Li, Zheng Lin
Computation and Language
When Does Sparsity Mitigate the Curse of Depth in LLMs
Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta et al. (+4 more)
Recent work has demonstrated the curse of depth in large
language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we provide evidence that sparsity-like mechanisms can dampen variance propagation and are associated with improved depth utilization Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and
attention sparsity induced by long-context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing in Grouped-Query
Attention and expert-activation sparsity in Mixtureof-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: mechanisms with reduced effective interaction density tend to exhibit lower output variance and better layer differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training depth-effective LLMs, yielding a notable 4.6 accuracy improvement on downstream tasks. Our results suggest that sparsity-like design choices are an important and previously underemphasized factor in effective depth scaling for LLMs. Code is available at https://github. com/pUmpKin-Co/SparsityAndCoD.
Read the full abstract ▾
All authors Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta, Max Zimmer, Nico Pelleriti, Thomas Hofmann, Shiwei Liu
Computation and Language · Cross-listed from cs.AI
CRASH: Cognitive Reasoning Agent for Safety Hazards in Autonomous Driving
Erick Silva, Rehana Yasmin and Ali Shoker
As AVs grow in complexity and diversity, identifying the root causes of operational failures has become increasingly complex. The heterogeneity of system architectures across manufacturers, ranging from end-to-end to modular designs, together with variations in
algorithms and integration strategies, limits the standardization of incident investigations and hinders systematic safety analysis. This work examines real-world AV incidents reported in the NHTSA database. We curate a
dataset of 2,168 cases reported between 2021 and 2025, representing more than 80 million miles driven. To process this data, we introduce CRASH, Cognitive
Reasoning Agent for Safety Hazards, an LLM-based
agent that automates
reasoning over crash reports by leveraging both standardized fields and unstructured narrative descriptions. CRASH operates on a unified representation of each incident to generate concise summaries, attribute a primary cause, and assess whether the AV materially contributed to the event. Our findings show that (1) CRASH attributes 64% of incidents to perception or planning failures, underscoring the importance of
reasoning-based analysis for accurate fault attribution; and (2) approximately 50% of reported incidents involve rear-end collisions, highlighting a persistent and unresolved challenge in autonomous driving deployment. We further validate CRASH with five domain experts, achieving 86% accuracy in attributing AV system failures. Overall, CRASH demonstrates strong potential as a scalable and interpretable tool for automated crash analysis, providing actionable insights to support safety research and the continued development of autonomous driving systems.
Read the full abstract ▾
Computation and Language
DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models
Xueyu Zhou, Yangrong Hu and Jian Huang
Masked
diffusion language models (MDLMs) have recently emerged as a new paradigm in
language modeling, offering flexible generation dynamics and enabling efficient parallel decoding. However, existing decoding strategies for pre-trained MDLMs predominantly rely on
token-level uncertainty criteria, while largely overlooking sequence-level information and inter-
token dependencies. To address this limitation, we propose Dependency-Oriented Sampler (DOS), a training-free decoding strategy that leverages inter-
token dependencies to inform
token updates during generation. Specifically, DOS exploits
attention matrices from
transformer blocks to approximate inter-
token dependencies, emphasizing information from unmasked
tokens when updating masked positions. Empirical results demonstrate that DOS consistently achieves superior performance on both code generation and mathematical
reasoning tasks. Moreover, DOS can be seamlessly integrated with existing parallel sampling methods, leading to improved generation efficiency without sacrificing generation quality.
Read the full abstract ▾
Computation and Language
Tagarela - A Portuguese speech dataset from podcasts
Frederico Santos de Oliveira, Lucas Rafael Stefanel Gris, Alef Iury Siqueira Ferreira et al. (+8 more)
Despite significant advances in
speech processing, Portuguese remains under-resourced due to the scarcity of public, large-scale, and high-quality
datasets. To address this gap, we present a new
dataset, named TAGARELA, composed of over 8,972 hours of podcast audio, specifically curated for training automatic
speech recognition (ASR) and text-to-
speech (TTS) models. Notably, its scale rivals English's GigaSpeech (10kh), enabling state-of-the-art Portuguese models. To ensure data quality, the corpus was subjected to an audio pre-processing pipeline and subsequently transcribed using a mixed strategy: we applied ASR models that were previously trained on high-fidelity transcriptions generated by proprietary APIs, ensuring a high level of initial accuracy. Finally, to validate the effectiveness of this new resource, we present ASR and TTS models trained exclusively on our
dataset and evaluate their performance, demonstrating its potential to drive the development of more robust and natural
speech technologies for Portuguese. The
dataset is released publicly, available at https://freds0.github.io/TAGARELA/, to foster the development of robust
speech technologies.
Read the full abstract ▾
All authors Frederico Santos de Oliveira, Lucas Rafael Stefanel Gris, Alef Iury Siqueira Ferreira, Augusto Seben da Rosa, Alexandre Costa Ferro Filho, Edresson Casanova, Christopher Dane Shulby, Rafael Teixeira Sousa, Diogo Fernandes Costa Silva, Anderson da Silva Soares, Arlindo Rodrigues Galvão Filho
Computation and Language
PYTHEN: A Flexible Framework for Legal Reasoning in Python
Ha-Thanh Nguyen and Ken Satoh
This paper introduces PYTHEN, a novel Python-based framework for defeasible legal
reasoning. PYTHEN is designed to model the inherently defeasible nature of legal argumentation, providing a flexible and intuitive syntax for representing legal rules, conditions, and exceptions. Inspired by PROLEG (PROlog-based LEGal
reasoning support system) and guided by the philosophy of The Zen of Python, PYTHEN leverages Python's built-in any() and all() functions to offer enhanced flexibility by natively supporting both conjunctive (ALL) and disjunctive (ANY) conditions within a single rule, as well as a more expressive exception-handling mechanism. This paper details the architecture of PYTHEN, provides a comparative analysis with PROLEG, and discusses its potential applications in autoformalization and the development of next-generation legal AI systems. By bridging the gap between symbolic
reasoning and the accessibility of Python, PYTHEN aims to democratize formal legal
reasoning for young researchers, legal tech developers, and professionals without extensive logic programming expertise. We position PYTHEN as a practical bridge between the powerful symbolic
reasoning capabilities of logic programming and the rich, ubiquitous ecosystem of Python, making formal legal
reasoning accessible to a broader range of developers and legal professionals.
Read the full abstract ▾
Computation and Language
CCTU: A Benchmark for Tool Use under Complex Constraints
Junjie Ye, Guoqiang Zhang, Wenjie Fu et al. (+4 more)
Solving problems through tool use under explicit constraints constitutes a highly challenging yet unavoidable scenario for large
language models (LLMs), requiring capabilities such as function calling, instruction following, and self-refinement. However, progress has been hindered by the absence of dedicated evaluations. To address this, we introduce CCTU, a
benchmark for evaluating LLM tool use under complex constraints. CCTU is grounded in a taxonomy of 12 constraint categories spanning four dimensions (i.e., resource, behavior, toolset, and response). The
benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios, each involving an average of seven constraint types and an average
prompt length exceeding 4,700
tokens. To enable reliable evaluation, we develop an executable constraint validation module that performs step-level validation and enforces compliance during multi-turn interactions between models and their environments. We evaluate nine state-of-the-art LLMs in both thinking and non-thinking modes. Results indicate that when strict adherence to all constraints is required, no model achieves a task completion rate above 20%. Further analysis reveals that models violate constraints in over 50% of cases, particularly in the resource and response dimensions. Moreover, LLMs demonstrate limited capacity for self-refinement even after receiving detailed feedback on constraint violations, highlighting a critical bottleneck in the development of robust tool-use
agents. To facilitate future research, we release the data and code.
Read the full abstract ▾
All authors Junjie Ye, Guoqiang Zhang, Wenjie Fu, Zelin Li, Tao Gui, Qi Zhang, Xuanjing Huang
Computation and Language
Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies
Giuseppe Samo and Paola Merlo
Large
language models (LLMs) have shown remarkable performance across various sentence-based linguistic phenomena, yet their ability to capture cross-sentence paradigmatic patterns, such as verb alternations, remains underexplored. In this work, we present curated paradigm-based
datasets for four
languages, designed to probe systematic cross-sentence knowledge of verb alternations (change-of-state and object-drop constructions in English, German and Italian, and Hebrew binyanim). The
datasets comprise thousands of the Blackbird
Language Matrices (BLMs) problems. The BLM task – an RPM/ARC-like task devised specifically for
language – is a controlled linguistic puzzle where models must select the sentence that completes a pattern according to syntactic and semantic rules. We introduce three types of templates varying in complexity and apply linguistically-informed data augmentation strategies across synthetic and natural data. We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the
datasets.
Read the full abstract ▾
Computation and Language
From Documents to Spans: Span-Centric Learning for LLM-based Explainable Medical Coding
Xu Zhang, Wenxin Ma, Chenxu Wu et al. (+5 more)
Medical coding, which assigns standardized ICD codes to clinical documents, is a fundamental yet challenging task in modern healthcare. It requires extensive knowledge of a large code space and the ability to identify and interpret sparse supporting evidence in long, noisy clinical documents. Most existing methods rely on document-level supervision, which provides limited guidance on supporting evidence and insufficient training signals for rare codes. To address these limitations, we propose Span-Centric Learning (SCL), which separates coding knowledge acquisition from its application in full documents. SCL first learns coding knowledge from short spans, then learns to use this knowledge in full documents from a small set of annotated data. Experiments show that, our approach enables an 8B LLM to surpass state-of-the-art methods using only 248 complete documents for training. It also provides explicit supporting evidence for each predicted code, with strong
alignment to human annotations. Further analysis shows that the model can learn knowledge of codes absent from documents through spans and apply that knowledge to full documents.
Read the full abstract ▾
All authors Xu Zhang, Wenxin Ma, Chenxu Wu, Rongsheng Wang, Zhiyang He, Xiaodong Tao, Kun Zhang, S. Kevin Zhou
Computation and Language · Cross-listed from cs.LG
Ye Wang, Jing Liu and Toshiaki Koike-Akino
The safety and reliability of
vision-
language models (VLMs) are a crucial part of deploying trustworthy agentic AI systems. However, VLMs remain vulnerable to jailbreaking attacks that undermine their safety
alignment to yield harmful outputs. In this work, we extend the Randomized
Embedding Smoothing and
Token Aggregation (RESTA) defense to VLMs and evaluate its performance against the JailBreakV-28K
benchmark of multi-modal jailbreaking attacks. We find that RESTA is effective in reducing attack success rate over this diverse corpus of attacks, in particular, when employing directional
embedding noise, where the injected noise is aligned with the original
token embedding vectors. Our results demonstrate that RESTA can contribute to securing VLMs within agentic systems, as a lightweight, inference-time defense layer of an overall security framework.
Read the full abstract ▾
Computation and Language
Practicing with Language Models Cultivates Human Empathic Communication
Aakriti Kumar, Nalin Poungpeth, Diyi Yang et al. (+2 more)
Empathy is central to human connection, yet people often struggle to express it effectively. In blinded evaluations, large
language models (LLMs) generate responses that are often judged more empathic than human-written ones. Yet when a response is attributed to AI, recipients feel less heard than when comparable responses are attributed to a human. We built a conversation platform in which participants are asked to offer empathic support to an LLM expressing realistic troubles and conducted a randomized experiment collecting 33,938 messages spanning 2,904 text-based conversations between 968 participants and their LLM conversational partners. We find participants report feeling empathy but systematically fail to express it, but an LLM coaching intervention offering personalized feedback on effective empathic communication significantly boosts it without homogenizing participants' responses. Moreover, we derive a data-driven taxonomy of idiomatic empathic expressions in naturalistic dialogues across personal and workplace trouble scenarios. These results advance the scientific understanding of how empathy is expressed and demonstrate a scalable, AI-based intervention for scaffolding and cultivating it.
Read the full abstract ▾
All authors Aakriti Kumar, Nalin Poungpeth, Diyi Yang, Bruce Lambert, Matthew Groh
Computation and Language
Bidirectional Chinese and English Passive Sentences Dataset for Machine Translation
Xinyue Ma, Pol Pastells, Mireia Farrús et al. (+1 more)
Machine Translation (MT) evaluation has gone beyond metrics, towards more specific linguistic phenomena. Regarding English-Chinese
language pairs, passive sentences are constructed and distributed differently due to
language variation, thus need special
attention in MT. This paper proposes a bidirectional multi-domain
dataset of passive sentences, extracted from five Chinese-English parallel corpora and annotated automatically with structure labels according to human translation, and a test set with manually verified annotation. The
dataset consists of 73,965 parallel sentence pairs (2,358,731 English words, 3,498,229 Chinese characters). We evaluate two state-of-the-art open-source MT systems with our
dataset, and four commercial models with the test set. The results show that, unlike humans, models are more influenced by the voice of the source text rather than the general voice usage of the source
language, and therefore tend to maintain the passive voice when translating a passive in either direction. However, models demonstrate some knowledge of the low frequency and predominantly negative context of Chinese passives, leading to higher voice consistency with human translators in English-to-Chinese translation than in Chinese-to-English translation. Commercial NMT models scored higher in metric evaluations, but LLMs showed a better ability to use diverse alternative translations.
Datasets and annotation script will be shared upon request.
Read the full abstract ▾
All authors Xinyue Ma, Pol Pastells, Mireia Farrús, Mariona Taulé
Computation and Language
Efficient Document Parsing via Parallel Token Prediction
Lei Li, Ze Zhao, Meng Li et al. (+6 more)
Document parsing, as a fundamental yet crucial
vision task, is being revolutionized by
vision-
language models (VLMs). However, the autoregressive (AR) decoding inherent to VLMs creates a significant bottleneck, severely limiting parsing speed. In this paper, we propose Parallel-
Token Prediction (PTP), a plugable, model-agnostic and simple-yet-effective method that enables VLMs to generate multiple future
tokens in parallel with improved sample efficiency. Specifically, we insert some learnable
tokens into the input sequence and design corresponding training objectives to equip the model with parallel decoding capabilities for document parsing. Furthermore, to support effective training, we develop a comprehensive data generation pipeline that efficiently produces large-scale, high-quality document parsing training data for VLMs. Extensive experiments on OmniDocBench and olmOCR-bench demonstrate that our method not only significantly improves decoding speed (1.6x-2.2x) but also reduces model
hallucinations and exhibits strong generalization abilities.
Read the full abstract ▾
All authors Lei Li, Ze Zhao, Meng Li, Zhongwang Lun, Yi Yuan, Xingjing Lu, Zheng Wei, Jiang Bian, Zang Li
Computation and Language
The Hrunting of AI: Where and How to Improve English Dialectal Fairness
Wei Li and Adrian de Wynter
It is known that large
language models (LLMs) underperform in English dialects, and that improving them is difficult due to data scarcity. In this work we investigate how quality and availability impact the feasibility of improving LLMs in this context. For this, we evaluate three rarely-studied English dialects (Yorkshire, Geordie, and Cornish), plus African-American Vernacular English, and West Frisian as control. We find that human-human agreement when determining LLM generation quality directly impacts LLM-as-a-judge performance. That is, LLM-human agreement mimics the human-human agreement pattern, and so do metrics such as accuracy. It is an issue because LLM-human agreement measures an LLM's
alignment with the human consensus; and hence raises questions about the feasibility of improving LLM performance in locales where low populations induce low agreement. We also note that fine-tuning does not eradicate, and might amplify, this pattern in English dialects. But also find encouraging signals, such as some LLMs' ability to generate high-quality data, thus enabling scalability. We argue that data must be carefully evaluated to ensure fair and inclusive LLM improvement; and, in the presence of scarcity, new tools are needed to handle the pattern found.
Read the full abstract ▾
Computation and Language
HindSight: Evaluating LLM-Generated Research Ideas via Future Impact
Bo Jiang
Evaluating AI-generated research ideas typically relies on LLM judges or human panels – both subjective and disconnected from actual research impact. We introduce HindSight, a time-split evaluation framework that measures idea quality by matching generated ideas against real future publications and scoring them by citation impact and venue acceptance. Using a temporal cutoff~$T$, we restrict an idea generation system to pre-$T$ literature, then evaluate its outputs against papers published in the subsequent 30 months. Experiments across 10 AI/ML research topics reveal a striking disconnect: LLM-as-Judge finds no significant difference between
retrieval-augmented and vanilla idea generation ($p{=}0.584$), while HindSight shows the
retrieval-augmented system produces 2.5$\times$ higher-scoring ideas ($p{<}0.001$). Moreover, HindSight scores are negatively correlated with LLM-judged novelty ($\rho{=}{-}0.29$, $p{<}0.01$), suggesting that LLMs systematically overvalue novel-sounding ideas that never materialize in real research.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation
Yitong Zhang, Chengze Li, Ruize Chen et al. (+4 more)
Large
Language Models (LLMs) have shown strong potential for code generation, yet they remain limited in private-library-oriented code generation, where the goal is to generate code using APIs from private libraries. Existing approaches mainly rely on retrieving private-library API documentation and injecting relevant knowledge into the context at inference time. However, our study shows that this is insufficient: even given accurate required knowledge, LLMs still struggle to invoke private-library APIs effectively. To address this limitation, we propose PriCoder, an approach that teaches LLMs to invoke private-library APIs through automatically synthesized data. Specifically, PriCoder models private-library data synthesis as the construction of a graph, and alternates between two graph operators: (1) Progressive Graph
Evolution, which improves data diversity by progressively synthesizing more diverse training samples from basic ones, and (2) Multidimensional Graph
Pruning, which improves data quality through a rigorous filtering pipeline. To support rigorous evaluation, we construct two new
benchmarks based on recently released libraries that are unfamiliar to the tested models. Experiments on three mainstream LLMs show that PriCoder substantially improves private-library-oriented code generation, yielding gains of over 20% in pass@1 in many settings, while causing negligible impact on general code generation capability.
Read the full abstract ▾
All authors Yitong Zhang, Chengze Li, Ruize Chen, Guowei Yang, Xiaoran Jia, Yijie Ren, Jia Li
Computation and Language
Indirect Question Answering in English, German and Bavarian: A Challenging Task for High- and Low-Resource Languages Alike
Miriam Winkler, Verena Blaschke and Barbara Plank
Indirectness is a common feature of daily communication, yet is underexplored in NLP research for both low-resource as well as high-resource
languages. Indirect Question Answering (IQA) aims at classifying the polarity of indirect answers. In this paper, we present two
multilingual corpora for IQA of varying quality that both cover English, Standard German and Bavarian, a German dialect without standard orthography: InQA+, a small high-quality evaluation
dataset with hand-annotated labels, and GenIQA, a larger training
dataset, that contains artificial data generated by GPT-4o-mini. We find that IQA is a pragmatically hard task that comes with various challenges, based on several experiment variations with
multilingual transformer models (mBERT, XLM-R and mDeBERTa). We suggest and employ recommendations to tackle these challenges. Our results reveal low performance, even for English, and severe overfitting. We analyse various factors that influence these results, including label ambiguity, label set and
dataset size. We find that the IQA performance is poor in high- (English, German) and low-resource
languages (Bavarian) and that it is beneficial to have a large amount of training data. Further, GPT-4o-mini does not possess enough pragmatic understanding to generate high-quality IQA data in any of our tested
languages.
Read the full abstract ▾