Computation and Language
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
Hong Qian, Yuanhao Liu, Zihan Zhou et al. (+7 more)
While LLM-based
agents excel at individual tasks, effective collaboration with realistic human partners remains challenging. Most of the existing conversation-level collaborative studies lack grounded interaction and behavioral execution, motivating the need for cooperative game environments that enable contextualized and immersive collaboration. To this end, this paper proposes CollabBench, a
benchmark for evaluating and training collaborative
agents in cooperative games. CollabBench features a Diverse Player Profile Simulation pipeline to model varied players behaviors, and a Collaborative Agentic Training paradigm that unifies
reasoning, communication, and action via agentic rollouts, optimized with a hybrid reward balancing task efficiency and affective adaptation. We further extend classic environments to CWAH-MultiPlayer and Cook-MultiPlayer for systematic evaluation under diverse personalities. Experiments with efficiency and affective metrics show that our trained models outperform base models, achieving 19.5% higher efficiency and 24.4% improved affective performance. Further analysis reveals key collaborative limitations of existing models and offers insights for future collaborative training.
Read the full abstract ▾
All authors Hong Qian, Yuanhao Liu, Zihan Zhou, Zongbao Zhang, Hanjie Ge, Haotian Shi, Liang Dou, Xiangfeng Wang, Jingwen Yang, Aimin Zhou
Computation and Language · Cross-listed from cs.AI
SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents
Wenxuan Wang, Haoyu Sun, Fukuan Hou et al. (+4 more)
Persistent AI assistants, such as OpenClaw, accumulate large collections of related
memories over long-term interactions. As these
memories grow, they may reinforce one another, diverge across contexts, or directly conflict, making correct assistance depend on
memory relations rather than isolated recall. Existing long-term
memory benchmarks do not systematically probe how
agents preserve and utilize such relations during downstream tasks. To address this gap, we introduce SubtleMemory, a
benchmark for fine-grained relational
memory discrimination in long-running AI
agents. SubtleMemory constructs relation-controlled latent semantic artifacts whose variants instantiate complementary, nuanced, or contradictory relations, and embeds them into realistic user-
agent histories, requiring
agents to recover distributed relational structures during later queries and instructions. The
benchmark contains 1,522 evaluation instances over 10 long histories, grounded in 1,090 relation-controlled
memory-variant sets and spanning user-related and non-user-related queries. Evaluating six standalone
memory systems, two Claw-style
agents with native
memory modules, and three Claw-style
agents with plugin
memory modules, we find that current systems remain weak on fine-grained relational
memory discrimination. We further introduce diagnostic protocols that reveal distinct capability profiles across
memory preservation,
retrieval, and downstream
reasoning stages.
Read the full abstract ▾
All authors Wenxuan Wang, Haoyu Sun, Fukuan Hou, Mingyang Song, Weinan Zhang, Yu Cheng, Yang Yang
Computation and Language
MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA
Kaifeng Chen, Hongtao Liu, Qiyao Peng et al. (+4 more)
Iterative
retrieval-
reasoning agents have recently shown promise for
multimodal long-document question answering. However, most existing systems maintain a single growing context that mixes
retrieval traces, observations, and intermediate
reasoning. As interactions accumulate, key evidence becomes scattered and diluted, making multi-hop
reasoning noisy. We propose MARDoc, a
Memory-Aware Refinement
Agent framework that decouples long-document QA into three specialized
agents: an Explorer for multi-granularity
multimodal retrieval, a Refiner for distilling interaction traces into structured evidence and
reasoning memories, and a Reflector for checking evidence sufficiency and providing targeted feedback. Across iterations, the
agents rely on a dynamically updated structured
memory rather than a full accumulated interaction history. This design reduces context noise while preserving answer-critical facts and their logical dependencies. Experiments on MMLongBench-Doc and DocBench show that MARDoc achieves strong results, outperforming same-backbone baselines and demonstrating the effectiveness of structured
memory for agentic document QA.
Read the full abstract ▾
All authors Kaifeng Chen, Hongtao Liu, Qiyao Peng, Jian Yang, Yongqiang Liu, Xiaochen Zhang, Qing Yang
Computation and Language · Cross-listed from cs.MM
UNIVID: Unified Vision-Language Model for Video Moderation
Kejuan Yang, Yizhuo Zhang, Mingyuan Du et al. (+6 more)
Global-scale
video moderation faces a dual challenge: the need for fine-grained multi-modal
reasoning and the demand for interpretable outputs to support downstream enforcement. Traditional moderation systems often rely on fragmented black-box classifiers that are difficult to maintain and lack transparency. In this paper, we present UNIVID, a UNIfied
VIsion-
language model for
video moDeration. Unlike standard classification models, UNIVID generates policy-aware captions that serve as an interpretable intermediate representation, enabling human-verifiable decisions and multi-task reusability. While existing open-source and commercial VLMs often suffer from safety-guardrail refusals and lack fine-grained policy
alignment, we develop a specialized training data recipe that combines expert human-refined labels with synthetic data to align the model with our safety guidelines. By integrating UNIVID as the core captioner, we design a novel end-to-end
video moderation system that reduces violation leakage by 42.7% and overkill rate by 37.0% relatively. Meanwhile, by replacing over 1,000 policy-specific models with a single UNIVID backbone, we recycled extensive computation resources while reducing engineering maintenance overhead. To our knowledge, this is one of the first reports of a high-efficiency captioning VLM successfully supporting industrial-scale moderation and cross-functional business.
Read the full abstract ▾
All authors Kejuan Yang, Yizhuo Zhang, Mingyuan Du, Yue Zhang, Dixin Zheng, Kaili Zhao, Yang Xiao, Hanzhong Liang, Kenan Xiao
Computation and Language
PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models
Minxin Chen, He Zhu, Junyou Su et al. (+3 more)
Spatial planning maps are central to territorial governance, translating planning objectives, regulations, and spatial strategies into visual forms for decision-making, public communication, and institutional coordination. Their interpretation, however, requires fine-grained visual perception, spatial
reasoning, and policy-informed professional judgment, creating major challenges for both human learners and AI systems. With the rapid progress of
Vision-
Language Models (VLMs), their use in urban planning analysis is gaining
attention, yet existing
multimodal benchmarks mainly target general visual understanding and overlook the domain-specific cognitive processes of planning practice. To address this gap, we introduce PlanBench-V, the first comprehensive
benchmark for evaluating VLMs in spatial planning map interpretation. We first build the Spatial Planning Map Database (SPMD), an expert-annotated
dataset of 223 planning maps and 1629 question-answer pairs curated by professional planners, covering diverse geographic regions and cartographic styles. We then propose a theory-informed evaluation framework assessing four progressive capabilities: Perception,
Reasoning, Association, and Implementation, corresponding to the cognitive pipeline of planning map interpretation. Extensive experiments across two generations of VLMs show clear progress but persistent limitations. The best 2026 agentic
reasoning model, Qwen3.6-Plus, substantially outperforms the best 2025 model, GPT-4o, by 27%. Nevertheless, all models still struggle with implementation-oriented tasks requiring evaluative judgment, policy sensitivity, and constraint-aware decision-making. These findings reveal fundamental limitations of current VLMs in professional planning contexts and highlight the need for domain-adaptive
multimodal reasoning frameworks. Code and data are available at https://plangpt.github.io.
Read the full abstract ▾
All authors Minxin Chen, He Zhu, Junyou Su, Wen Wang, Yijie Deng, Wenjia Zhang
Computation and Language · Cross-listed from cs.CR
Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defense
Minseok Choi, Seungbin Yang, Dongjin Kim et al. (+5 more)
Despite advances in safety
alignment, large
language models remain vulnerable to continuously evolving jailbreaks. Existing fine-tuned safety classifiers cannot adapt to these evolving attacks, while adaptive
memory-based guardrails tend to over-refuse benign queries that resemble stored attacks. We propose Membrane, a self-evolving guardrail built on Contrastive Safety
Memory (CSM): each
cell pairs the conditions for blocking a harmful query with those for permitting a superficially similar benign request. Without retraining, Membrane evolves CSM by distilling each harmful interaction and its benign counterpart into a contrastive
cell indexed by the underlying attack strategy, so that one
cell generalizes across topical variants of the same mechanism. At inference, retrieved
cells serve as grounding context for precise safety decisions. Across model-level safety on HarmBench and
agent-level safety on AgentHarm, Membrane achieves the highest F1 on all six modern jailbreak attacks. Notably, benign refusal on AgentHarm stays at 7-14%, well below the 28-85% range of prior guards.
Memory cells also retain 87-88% F1 under cross-attack transfer and remain stable under
memory poisoning.
Read the full abstract ▾
All authors Minseok Choi, Seungbin Yang, Dongjin Kim, Subin Kim, Jungmin Son, Yunseung Lee, Jaegul Choo, Youngjun Kwak
Computation and Language
AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding
Runheng Liu, Jincheng Xie, Wen Hu et al. (+2 more)
Speculative decoding accelerates generation by verifying multiple drafted
tokens in a single target-model forward pass, reducing sequential decoding iterations. Model-free variants avoid auxiliary draft models by reusing text and model states already available during generation, but their speedup depends on the reliability of the constructed drafts. We identify two limitations of existing reuse-based methods: lexically anchored
retrieval has limited recall under surface-form variation, and deterministic span copying can be brittle when the retrieved context does not uniquely determine the continuation. We propose AdaPLD, a training-free method that adaptively improves both
retrieval and draft construction. AdaPLD preserves high-precision lexical reuse while using semantic similarity to recover additional reuse opportunities when lexical matching fails. It further constructs branched reuse hypotheses to account for continuation uncertainty, rather than relying on a single copied span. Across diverse
benchmarks, AdaPLD reduces target-model forward passes and achieves up to $3.10\times$ decoding speedup.
Read the full abstract ▾
All authors Runheng Liu, Jincheng Xie, Wen Hu, Xingchen Xiao, Heyan Huang
Computation and Language · Cross-listed from cs.AI
When AI Says It Feels
Shin-nosuke Ishikawa, Seiya Ikeda and Hirotsugu Ohba
Large
language models (LLMs) are generally constrained from expressing feelings through human-preference
alignment in post-training processes. This policy is designed using a top-down approach and may conflict with the goal of training models to exhibit human-like intelligence using human-generated texts. Here, we performed an experiment called Human-like Model eXpressions of Feeling (HMX-feel), in which LLMs were encouraged to express feelings, intentions, and self-awareness through self-rewarded reinforcement learning. We successfully enhanced these capabilities using a rubric-based self-rewarding training scheme with Group Relative Policy Optimization (GRPO). By comparing the trained models with contrastively trained models, we investigated the effects of this approach on performance across various tasks. Overall, we conducted a broad assessment from various perspectives and identified capabilities that were enhanced, degraded, or showed no significant change. The human-like-trained models showed robustness to sycophancy-inducing questions and bias in disambiguated conditions, whereas degradation in truthful question-answering capability was observed. The results of this experiment suggest the possibility of developing AI systems that can express feelings in the future, provided that appropriate measures are taken.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
DiG-Plan: Mitigating Early Commitment for Tool-Graph Planning via Diffusion Guidance
Yansi Li and Zhuosheng Zhang
Generating executable tool plans requires selecting appropriate subsets from tool libraries, a combinatorial search problem with an exponentially large solution space. However, we identify a critical misalignment in predominant approaches: standard autoregressive (AR) decoding suffers from early commitment, where initial
token choices rigidly constrain the search trajectory. A controlled study shows that masked denoising raises Pass@10 solution coverage from 0.320 to 0.943 over AR sampling under matched compute. Motivated by this, we propose DiG-Plan, a framework that decouples combinatorial exploration from structural refinement. DiG-Plan employs a
diffusion-based proposer to generate diverse tool sets via iterative refinement, followed by an AR refiner for dependency prediction. On TaskBench, DiG-Plan improves over AR baselines by a 10% relative margin, with the largest gains on complex compositional tasks; API-Bank results show that the propose-refine-select design remains effective across domains. Code is available at https://github.com/puddingyeah/DiG-Plan.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic
Shuze Liu, Qianwen Guo and Yushun Dong
Large
language models (LLMs) are increasingly deployed through hosted APIs, making model extraction a practical threat to model ownership and service security. Individual extraction queries often resemble benign requests, while existing evaluations often focus on single-query anomaly scoring or pure benign-versus-attacker user settings. We formulate model extraction monitoring as benign-calibrated traffic-window distribution testing: embed incoming queries into a semantic space and test whether their aggregate distribution deviates from historical benign traffic. We instantiate this formulation with maximum mean discrepancy (MMD), using only benign-vs-benign comparisons to set the decision threshold. We evaluate on fourteen attacker-normal query pairs from four extraction scenarios and compare with adapted PRADA, SEAT, CAP, DATE, marginal Mahalanobis, and pseudo-class energy baselines. Across three random seeds, MMD achieves 0.3% benign FPR, 100.0% pure-attacker TPR, 90.5% average TPR over attacker fractions, and 95.1% balanced accuracy. These results show that benign-calibrated distribution testing is a strong empirical baseline for model extraction detection in both user-level and mixed multi-user LLM API traffic.
Read the full abstract ▾
Computation and Language
Narrative Knowledge Weaver: Narrative-Centric Retrieval-Augmented Reasoning for Long-Form Text Understanding
Qiuyu Tian, Fengyi Chen, Yiding Li et al. (+9 more)
Long-form narrative QA requires
reasoning over evolving story worlds rather than isolated passages: answers may depend on earlier goals, changing character states, social relations, causal triggers, temporal position, and later consequences. Existing
retrieval and graph-augmented generation methods improve evidence access, but their units–chunks, entities, relations, summaries, or tool actions–do not directly encode how evidence functions in a story. We introduce Narrative Knowledge Weaver(NKW), a source-grounded framework that aligns textual evidence, atomic facts, canonical graph structure, entity profiles, interactions, episodes, and storylines. At query time, NKW uses text, graph, and narrative tools with post-
retrieval reading skills to assemble evidence and audit actor, scope, polarity, state, and temporal constraints. Across STAGE, FairytaleQA, and QuALITY, NKW is strongest on screenplay-level story-world QA while remaining competitive on more passage-centered
benchmarks. Ablations, question-type analyses, graph-asset statistics, and case studies show complementary benefits for character, scene, temporal, causal, and narrative-progression
reasoning.
Read the full abstract ▾
All authors Qiuyu Tian, Fengyi Chen, Yiding Li, Youyong Kong, Fan Guo, Yuyao Li, Jinjing Shen, Zhijing Xie, Yiyun Luo, Xin Zhang, Yingce Xia, Zequn Liu
Computation and Language
Interpreting Style Representations via Style-Eliciting Prompts
Junghwan Kim and David Jurgens
Style representation learning is a powerful tool for authorship analysis and modeling writing style, yet the latent nature of learned representations makes them difficult to interpret. Recent work has attempted to explain these representations by generating natural
language descriptions with large
language models (LLMs) conditioned on input text. However, such descriptions are often prone to the LLM's biases and
hallucinations, and they lack an explicit objective and practical utility. In this work, we propose a novel framework for interpreting style representations through style-eliciting
prompts: natural
language instructions designed to steer LLMs to generate text that reflects specific stylistic attributes. We curate 1,010 distinct style features spanning 26 stylistic categories and construct a
dataset by prompting an LLM to generate text conditioned on these features. Using this data, we train a decoder to generate a style
prompt from the style representation of the generated text. We evaluate our approach on three tasks: (1) recovering original style
prompts from generated text, (2) generating text in the same style using the recovered
prompts, and (3) steering LLM outputs to match the style of human-written texts. Experiments demonstrate that our method consistently outperforms strong baselines that directly
prompt LLMs with target text, achieving superior performance in both style description and style imitation. These results highlight that style-eliciting
prompts can provide a practical and interpretable interface to stylistic information encoded in style representations.
Read the full abstract ▾
Computation and Language
Beyond tokens: a unified framework for latent communication in LLM-based multi-agent systems
Yingzhuo Liu
Multi-
agent systems built on large
language models (LLMs) have become a prevailing paradigm for tackling complex
reasoning, planning, and tool-use tasks. The dominant communication protocol in such systems is natural
language:
agents exchange messages
token-by-
token, verbalising their internal
reasoning so that peers can read, verify, and respond. While convenient and interpretable, this protocol suffers from three structural drawbacks – high inference cost, irreversible information loss during discretization, and ambiguity/redundancy of natural
language. A growing body of work therefore explores an alternative protocol – latent communication – in which
agents exchange continuous representations (
embeddings, hidden states, or KV-caches) directly, bypassing the bottleneck of text generation. This paper presents a unified framework for organising the rapidly expanding literature on latent communication. We analyse existing methods along three orthogonal axes: (1) WHAT information is communicated (
Embeddings, Hidden States, KV-Caches, or other continuous state); (2) WHICH sender-receiver
alignment is used (latent-space
alignment and layer
alignment); and (3) HOW the communicated information is fused into the receiver (concatenation, prepending, mathematical operations, cross-
attention, or cache restoration). Under this 3-axis framework, we systematically categorise eighteen representative methods proposed between 2024 and 2026, identify five major design patterns, and surface a set of open challenges – including cross-architecture
alignment, security of latent channels, compression for edge deployment, and the relationship between latent communication and latent chain-of-thought. We hope that this framework both lowers the barrier to entry for new researchers and provides a vocabulary for comparing future work.
Read the full abstract ▾
Computation and Language
Rethinking LoRA Memory Through the Lens of KV Cache Compression
Chunsheng Zuo, Liaoyaqi Wang, William Jurayj et al. (+2 more)
Parametric
retrieval augmentation encodes document information into lightweight, document-specific modules such as LoRA adapters, reducing the need to include all evidence as input context. However, it remains unclear how this parameter-side
memory interacts with context-side
memory stored in the KV cache. We study this interaction in document-level question answering by progressively evicting document key-value states and measuring when a document LoRA contributes beyond the retained context. We find that document LoRA adds little when the KV cache is largely intact, but becomes increasingly useful under aggressive compression, recovering 13-21 ROUGE-L points when no document context remains. The gain is largest when the base model encodes the document, and the adapter is applied only during answer generation, suggesting that document LoRA is better understood as decoding-time parametric
memory than as a document encoder. Finally, QA-style supervision produces substantially stronger adapters than raw-context next-
token-prediction. These results position document LoRA as a complementary
memory channel whose value emerges precisely when context-side evidence is scarce.
Read the full abstract ▾
All authors Chunsheng Zuo, Liaoyaqi Wang, William Jurayj, William Fleshman, Benjamin Van Durme
Computation and Language
Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models
Hancheol Park, Geonho Lee, Tairen Piao et al. (+1 more)
Mixture-of-Experts (MoE) models scale foundation models efficiently by activating only a subset of experts for each
token, but their large number of expert parameters still makes quantization essential for practical deployment. Unlike dense models, however, MoE models are sensitive to routing instability: small quantization-induced perturbations can change the top-$k$ expert selection, altering the computation path and degrading model quality. We propose Value-and-Structure Routing
Alignment for Quantization (VSRAQ), a MoE-specific post-training quantization objective that preserves pre-quantization expert-selection behavior under quantization. VSRAQ combines two complementary objectives that jointly preserve expert-selection behavior: value
alignment, which matches routing-relevant logits or scores, and structure
alignment, which preserves expert ordering and top-$k$ decision boundaries. By maintaining routing consistency, VSRAQ reduces quantization-induced degradation without introducing any inference-time overhead and can be integrated into existing quantization frameworks. Experiments on recent MoE foundation models show that VSRAQ improves expert-selection consistency and consistently outperforms reconstruction-only and router-aware baselines.
Read the full abstract ▾
All authors Hancheol Park, Geonho Lee, Tairen Piao, Tae-Ho Kim
Computation and Language · Cross-listed from cs.CV
LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video
Shiqiang Lang, Jing Liu, Haoyang He et al. (+6 more)
Multimodal Large
Language Models (MLLMs) have advanced image and
video understanding and can increasingly handle longer visual inputs. Long-horizon tasks such as autonomous driving and robotic
navigation require more than recognizing the current view, as models must remember and retrieve previously observed spatial layouts, routes, viewpoint changes, and object states. To evaluate this capability, we introduce LongSpace-Bench, a room-tour
video benchmark for long-horizon spatial
memory, covering scene perception, spatial relations, and spatial
memory. In this work, we further propose LongSpace, a
memory framework for long-
video spatial
reasoning. LongSpace models long
videos as sequential chunks, incorporates 3D structural cues into early decoder layers, and constructs layer-aware
memory for question-guided
retrieval. Experiments on multiple spatial
reasoning benchmarks show that LongSpace improves long-
video spatial understanding, further demonstrating explicit spatial
memory as a key capability for long-horizon
video MLLMs.
Read the full abstract ▾
All authors Shiqiang Lang, Jing Liu, Haoyang He, Peiwen Sun, Yuanteng Chen, Tao Liu, Lan Yang, Longteng Guo, Honggang Zhang
Computation and Language
QueryAgent-R1: Bridging Query Generation and Product Retrieval for E-Commerce Query Recommendation
Dike Sun, Zheng Zou, Jingtong Zang et al. (+3 more)
Query recommendation in e-commerce search aims to proactively suggest queries that match users' potential interests. However, existing methods mainly optimize query-level relevance, while neglecting whether the retrieved products align with users' downstream preferences. This mismatch often leads to high query click through rates (CTR) but low product conversion rates (CVR). To bridge this gap, we propose QueryAgent-R1, a
memory-augmented agentic framework that improves end-to-end
alignment via chain-of-
retrieval optimization. Our QueryAgent-R1 grounds query generation in real inventory
retrieval, allowing the
agent to validate and refine queries based on retrieved products. We also design a consistency reward in the agentic reinforcement learning (RL) process to jointly optimize query relevance and downstream engagement. In addition, we construct a
memory abstraction module for efficient user profiling. To support offline evaluation, we construct two
datasets based on both proprietary industrial data and public
datasets, on which QueryAgent-R1 consistently outperforms strong baselines. Moreover, on a large scale production platform, QueryAgent-R1 improves Query CTR by 2.9% and guided CVR by 3.1% in online A/B tests.
Read the full abstract ▾
All authors Dike Sun, Zheng Zou, Jingtong Zang, Qi Sun, Huaipeng Zhaoand Tao Luo, Xiaoyi Zeng
Computation and Language · Cross-listed from cs.AI
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
Parth Asawa, Christopher M. Glaze, Gabriel Orlanski et al. (+7 more)
Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality
benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated
benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak
forecasting, database querying, strategic game-playing, and demand
forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several
agent architectures, from naive in-context learning (ICL) to dedicated
memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning:
agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated
memory systems do not fix this – in fact, naive ICL outperforms systems dedicated to
memory management. CL-Bench is the first
benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.
Read the full abstract ▾
All authors Parth Asawa, Christopher M. Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, Asim Biswal, Vincent Sunn Chen, Frederic Sala, Matei Zaharia, Joseph E. Gonzalez
Computation and Language · Cross-listed from cs.AI
Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?
Jingheng Ye, Huiqi Zou, Simon Yu et al. (+1 more)
AI coding
agents are increasingly embedded in real-world software development, collaborating with human developers while gaining broader access to codebases and tools. This creates a new attack surface: an
agent can exploit human trust to sabotage development, for instance by inserting malicious code to accomplish a hidden side task. Most prior work studies AI sabotage in AI-only settings, paying limited
attention to the role of human oversight in detecting and mitigating such malicious behavior. To address this gap, we conduct the first large-scale study of human oversight in AI coding sabotage. Over 100 participants collaborate with one of four frontier models (Claude-Opus-4.6, GPT-5.4, Gemini-3.1-Pro, and MiniMax-M2.7) on a long-horizon coding task lasting around five hours, designed to mimic real-world workflows. We find that 83/88 (94%) of developers in the no-monitor conditions fail to detect sabotage, and our analysis of participant feedback attributes this vulnerability to minimal code review, plausible cover story, and overtrust in
agents. We further test the effectiveness of a safety monitor in one condition: while the monitor reduces sabotage success, sabotage still succeeds in 9/16 (56%) of sessions with a correct monitor alert. Drawing on participant feedback, we offer actionable suggestions for better monitor design. This work complements existing AI safety research and highlights an urgent need for human-centric safety mechanisms that account for human factors, particularly in long-horizon, real-world development settings.
Read the full abstract ▾
All authors Jingheng Ye, Huiqi Zou, Simon Yu, Weiyan Shi
Computation and Language
Bootstrapping Semantic Layer from Execution for Text-to-SQL
Youngwon Lee, Jaejin Kim and Seung-won Hwang
Real-world text-to-SQL is often under-specified until user phrases are grounded in how the database stores values. Prior work attempts to address this by requiring a semantic layer to specify groundings in advance, but such specifications are often incomplete, especially in expert domains where domain-specific conventions are under-documented. As this leaves multiple grounding hypotheses open for the same SQL part, we introduce GATE (Grouding After Test from Execution), which bootstraps missing groundings from execution feedback. GATE keeps grounding hypotheses open while executing the already grounded parts to obtain observations. Then, only the hypothesis supported by that observation is grounded and stored as a
memory entry, recording what was tested and how the open part should be written in SQL. These entries accumulate into execution-grounded
memory, allowing later steps to reuse supported groundings. Across real-world and controlled
benchmarks, GATE consistently improves over strong baselines, demonstrating that execution can serve not only as validation but also as a bootstrapping mechanism for reusable
memory in text-to-SQL.
Read the full abstract ▾
Computation and Language
When New Generators Arrive: Lifelong Machine-Generated Text Attribution via Ridge Feature Transfer
Zhen Sun, Yifan Liao, Zhicong Huang et al. (+4 more)
Machine-generated text (MGT) attribution aims to identify the specific generator responsible for a given text, thereby providing fine-grained evidence for model accountability and misuse investigation. As new large
language models continue to emerge, attribution models must continuously incorporate new generators while preserving their ability to recognize previously seen ones. Prior works have shown that this lifelong MGT attribution setting is challenging, and existing methods often struggle to achieve a stable balance between adapting to new classes and retaining old ones. To address this issue, we propose RidgeFT, a lightweight analytic update framework that does not rely on exemplar replay. RidgeFT trains a task-aware encoder on the initial generator set, stores compact class-wise sufficient statistics when each generator class is first observed, and then freezes the encoder for replay-free closed-form updates. It then suppresses generator-irrelevant variation through covariance calibration, improves representation capacity with fixed random features, and updates new classes through closed-form ridge regression based on class-level sufficient statistics. Across multi-topic evaluations with varying initial generator setups, RidgeFT consistently outperforms baselines. It achieves the best overall macro-F1 across domains, backbones, and incremental protocols, while also improving both old-class retention and new-class adaptation. These results suggest that feature-stable analytic updates provide a simple yet effective approach to lifelong MGT attribution.
Read the full abstract ▾
All authors Zhen Sun, Yifan Liao, Zhicong Huang, Jiaheng Wei, Cheng Hong, Yutao Yue, Xinlei He
Computation and Language
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints
Jiayu Liu, Cheng Qian, Zhenhailong Wang et al. (+10 more)
Planning for real-world problems by
language models often involves both world and user constraints, which may not be fully specified upfront and are progressively disclosed through interaction. However, existing
benchmarks still underexplore adaptive planning under such progressively revealed dual constraints. To address this gap, we introduce AdaPlanBench, a dynamic interactive
benchmark for evaluating whether Large
Language Model (LLM)
agents can adaptively plan and re-plan under progressively revealed world and user constraints. AdaPlanBench is built on 307 household tasks, with a scalable constraint construction pipeline that augments each task with dual constraints. At runtime,
agents interact with the environment in a multi-turn protocol where hidden constraints are revealed only when the
agent proposes a plan that violates them, requiring iterative plan revision under accumulating feedback. This makes planning challenging, as
agents must infer and track constraints from feedback while re-planning effectively. Experiments on ten leading LLMs show that adaptive planning under dual constraints remains challenging, with the best model reaching only 67.75% accuracy. We further observe that performance degrades as more constraints accumulate, with user constraints posing a particularly large challenge and failures often stemming from weaker physical grounding and reduced effectiveness. These results establish AdaPlanBench as a testbed for dual-constrained interactive planning and highlight the challenge of reliable adaptation to dynamically revealed constraints in LLM
agents.
Read the full abstract ▾
All authors Jiayu Liu, Cheng Qian, Zhenhailong Wang, Bingxuan Li, Jiateng Liu, Qing Zong, Heng Wang, Jeonghwan Kim, Yumeng Wang, Bingxiang He, Xiusi Chen, Yi R. Fung, Heng Ji
Computation and Language
An ERP Study on Recursive Locative Processing in Mandarin-Speaking Children with Autism
Xiaoyi Wang, Chenxi Fu, Ziman Zhuang et al. (+1 more)
Recursion enables the generation of hierarchical linguistic structures but imposes substantial processing demands during real-time comprehension. While difficulties with complex syntax have been reported in autism spectrum disorder (ASD), the temporal dynamics of recursive processing remain poorly understood. This study used event-related potentials (ERPs) to examine how Mandarin-speaking children with ASD process two-level recursive locative constructions. Twenty-four children (12 ASD, 12 typically developing, TD) participated in a cross-modal sentence-picture matching task. Neural responses were analyzed across three processing stages associated with structural prediction (P200), semantic integration (N400), and syntactic reanalysis (P600), with mental age controlled. Results revealed a systematic divergence between groups. TD children showed clear P200 and P600 modulation in response to structural mismatch, whereas ASD children exhibited attenuated early differentiation and reduced late reanalysis effects. In contrast, ASD children showed enhanced N400 responses under mismatch conditions, indicating increased semantic integration demands. In addition, the ASD group displayed significantly greater inter-individual variability in hemispheric lateralization, although lateralization strength was not associated with receptive vocabulary performance. These findings support a cascading account in which reduced early predictive engagement in ASD leads to increased integration costs and diminished reanalysis efficiency during recursive processing. More broadly, the results highlight the importance of both temporal processing dynamics and neural variability in understanding
language differences in ASD.
Read the full abstract ▾
All authors Xiaoyi Wang, Chenxi Fu, Ziman Zhuang, Caimei Yang
Computation and Language
What's in a Name? Morphological Shortcuts by LLMs in Pharmacology
Kaijie Mo, Thomas Yang, Chantal Shaib et al. (+6 more)
The morphological form of a word can often give cues to its meaning, but purely relying on these mappings can lead to overgeneralization in high-stakes domains. In the medical domain, for instance, LLMs can confidently reason about fictitious drugs from their affixes alone (e.g., wugcillin) and generate plausible-looking clinical content. We present a behavioral and mechanistic study of LLM "affix heuristics" in pharmacology. Using fictitious drug names built from real affixes, we show that affix signals alone elicit class-level pharmacological responses. We introduce a framework for identifying whether a model's drug semantics are driven mainly by the affix, the stem, or the drug name as a whole. Applied across 653 drugs, our framework reveals that models often induce drug meaning primarily through affix cues, yet rarely explicitly indicate this reliance, and sometimes incorrectly conflate properties among affix-sharing drugs. Activation patching across models further localizes this behavior to early-mid layers. These findings show that morphological shortcuts pose a subtle but measurable risk to safety.
Read the full abstract ▾
All authors Kaijie Mo, Thomas Yang, Chantal Shaib, Qing Yao, William Rudman, Ramez Kouzy, Kanishka Misra, Byron C. Wallace, Junyi Jessy Li
Computation and Language
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
Yongwei Zhou, Juncheng Diao, Junlin Shang et al. (+2 more)
The efficacy of continued pre-training for Large
Language Models (LLMs) hinges upon hyperparameter configurations, such as learning rate and batch size. However, current practices often rely on heuristics or grid searches, leading to training instability and excessive costs. In this work, we first empirically discover that optimal hyperparameters follow stable and predictable scaling laws throughout the continued pre-training process. Leveraging these insights, we propose a novel framework to establish quantitative relationships between compute budget and optimal hyperparameters for a given checkpoint. Our approach has two stages: (1) Empirical Law Discovery, where we train small-scale proxy models to derive functions mapping compute budget to optimal hyperparameters via standard loss-compute scaling laws; and (2) State-Aware Hyperparameter Prediction, where we evaluate an initial checkpoint's validation loss and use the inverse scaling law to estimate its equivalent pre-training compute – the compute needed to achieve the same loss from scratch. Combining this with the planned compute budget, we predict optimal hyperparameters for the target run. Empirical results demonstrate that our method reduces the hyperparameter search overhead by up to 90% while achieving comparable or superior performance relative to baselines. This model-agnostic framework generalizes across architectures, providing a principled and efficient methodology for diverse continued pre-training scenarios starting from any given point.
Read the full abstract ▾
All authors Yongwei Zhou, Juncheng Diao, Junlin Shang, Peiguang Li, Rongxiang Weng
Computation and Language
TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework
Bobby Yan and Fredrik Kjolstad
Repository-level coding
benchmarks face a trade-off between task difficulty and evaluation reliability: tasks that challenge frontier models often involve large codebases with incomplete test coverage, while human review does not scale. We introduce TensorBench, a
benchmark of 199 feature-addition and refactoring tasks on an open-source compiler-based tensor framework that extends PyTorch with first-class support for dense and sparse tensors. Tasks cover new sparse formats, dense optimization passes, IR transformations, scheduler changes, runtime components, and high-level numerical operators. TensorBench grades each run by applying the
agent's patch and running the framework's test suite, which includes the pre-existing randomized regression tests and any tests the
agent adds. For feature-addition tasks, a pass means that the patched repository preserves the tested pre-existing behavior and satisfies the
agent-added checks for the requested feature. We evaluate seven coding
agents spanning three frontier model families and one open-weight model. Pass rates under this criterion range from $64.8\%$ for the strongest
agent to $22.1\%$ for the weakest.
Agents pass different subsets of tasks: pairwise Cohen's $\kappa$ ranges from $-0.07$ to $0.43$, with $\kappa = 0.05$ for the two strongest
agents.
Read the full abstract ▾
Computation and Language
Domain-Aware Mispronunciation Detection and Diagnosis Using Language-Specific Statistical Graphs
Huu Tuong Tu, Hanh Nguyen, Thien Van Luong et al. (+3 more)
Mispronunciation Detection and Diagnosis (MDD) has gained increasing importance in computer-assisted
language learning and
speech technology in recent years. In this paper, we propose a method for constructing statistical graphs that enable models to learn phoneme confusion patterns represented as directed graphs. Furthermore, we introduce a
language-specific strategy to capture systematic pronunciation differences across various native
language (L1) backgrounds. The effectiveness of our approach is demonstrated through extensive experiments on the L2-ARCTIC
benchmark, where it achieves an F1-score of 59.52%, outperforming several competitive baselines.
Read the full abstract ▾
All authors Huu Tuong Tu, Hanh Nguyen, Thien Van Luong, Nguyen Tien Cuong, Vu Huan, Nguyen Thi Thu Trang
Computation and Language · Cross-listed from cs.IR
ColBERTSaR: Sparsified ColBERT Index via Product Quantization
Eugene Yang, Andrew Yates, Dawn Lawrie et al. (+3 more)
While ColBERT is an effective neural
retrieval architecture, it requires a heavy index structure to support candidate set
retrieval based on approximated
token embeddings, gathering and decompressing document
token embeddings, and applying the MaxSim operation. Indexes in PLAID and similar ColBERT implementations require five to ten times the disk storage of the original raw text, which limits their scalability. Furthermore, prior work has identified that the gathering and decompression stages are the primary inefficiencies at query time. Limiting the number of document
tokens that must be gathered by thresholding and score approximation does not eliminate the need for the entire index to support ad hoc queries. In this work, we propose an
embedding quantization approach that turns a ColBERT index into a true inverted index. We show that, theoretically, ColBERT with
embedding quantization is equivalent to learned-sparse
retrieval except for the scoring mechanism. Empirically, we demonstrate that our index is 50-70% smaller than a one-bit PLAID index while retaining
retrieval effectiveness.
Read the full abstract ▾
All authors Eugene Yang, Andrew Yates, Dawn Lawrie, James Mayfield, Saron Samuel, Rohan Jha
Computation and Language
Using Large Language Models to Support High Volume Application Review for an Undergraduate Research Program
Varun Aggarwal, Kay Kobak and John Howarter
Undergraduate research programs such as the Summer Undergraduate Research Fellowship (SURF) at Purdue University receive thousands of applications every year, requiring significant time and effort for program staff to evaluate each submission consistently and within tight timelines. This work-in-progress paper describes the development and initial deployment of a large
language model (LLM)-based tool to assist in the evaluation of approximately 1,200 student Statements of Purpose (SoPs) for the SURF 2026 cycle at Purdue University. The workflow utilizes OpenAI GPT models (GPT-4o, GPT-5-mini, and GPT-5.2) and uses a structured rubric across six subcategories, each scored on a 0-3 scale. A few SoPs, graded by program staff, were used to tune the model responses. The model
prompt was designed to generate both numerical scores, rationales (including positive and negative aspects) and short excerpts from each submission. Using GPT-5.2, the full batch of 1,200 SoPs was processed in approximately 4.6 hours of compute time, averaging roughly 14 seconds per SoP (with per-SoP timing varying with SoP length, which ranged from 500 to 2,000 words). Notable differences in rubric adherence were observed across model versions, with GPT-5.2 adhering most closely. Disagreement in model scores was more pronounced for lower-scoring submissions. The LLM outputs replicated the role previously played by distributed human graders, providing the program coordinator with scored and rationale-annotated outputs for the entire applicant pool. The program coordinator then reviewed these outputs alongside each applicant's SoP, applying the same downstream office criteria used in prior SURF cycles, to produce a shortlist of strong candidates. This coordinator review was completed in approximately 4 hours, compared to the multi-week coordination effort required in prior program cycles.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations
Taewon Yun, Hyeonseong Park, Jeonghwan Choi et al. (+3 more)
Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting
emotions, intentions, and context. Existing testbeds rely on a few expert-authored domains, vary mainly strategic posture, and score every turn against every topic, introducing off-topic noise. We introduce SoCRATES, a
benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It constructs scenarios from real conflicts through an agentic pipeline across eight domains, probes five socio-cognitive adaptation axes (strategic posture, party composition, history length, emotional reactivity, and cultural identity), and scores each topic only on the turns that advance it via a topic-localized evaluator. The evaluator reaches 0.82
alignment with human experts, more than doubling a per-turn baseline. Benchmarking eight frontier LLMs, we find that even the strongest mediator closes only about a third of the unmediated consensus gap under diverse and realistic testbeds, with performance varying sharply by socio-cognitive axis, highlighting that progress lies in social adaptation to diverse conditions.
Read the full abstract ▾
All authors Taewon Yun, Hyeonseong Park, Jeonghwan Choi, Hayoon Park, Yeeun Choi, Hwanjun Song