Computation and Language · Cross-listed from cs.SE
Offloading Score: Measuring AI Reliance Through Counterfactual Workflows
Vishakh Padmakumar, Lujain Ibrahim, Zora Zhiruo Wang et al. (+3 more)
AI tools are increasingly integrated into real-world workflows. However, existing measures of reliance on these tools focus on AI output adoption or on self-reported indicators, rather than how task effort is distributed between users and tools. Here, we introduce offloading score, a measure of reliance that quantifies the fraction of cognitive effort offloaded to an AI tool. Offloading Score is simulation-based – we construct a counterfactual workflow by estimating how the user would have completed the task without the tool, and then computing the fraction of steps saved by using the tool. We validate offloading score through intrinsic evaluations of metric validity, and a controlled user study ($n=40$) with developers performing programming tasks using AI tools. We vary time pressure to test whether reliance measures capture the known increase in reliance under time pressure. We show that offloading score detects significantly higher reliance in time-constrained settings ($+43\%$, $p=0.018$), while usage-based and self-reported baseline measures of reliance do not distinguish the conditions. We complement this with descriptive insights showing that higher reliance manifests as greater delegation of subtasks to the tool and more direct reuse of AI outputs. Finally, we demonstrate an approach of using offloading score in combination with target outcomes of a task (e.g., code understanding) to identify when reliance may be (in)appropriate. Our framework offers two contributions: an instrument users can apply to measure and reflect on their own reliance, and a quantitative signal that
agent designers can utilize to mitigate overreliance.
Read the full abstract ▾
All authors Vishakh Padmakumar, Lujain Ibrahim, Zora Zhiruo Wang, Jennifer Wang, Q. Vera Liao, Diyi Yang
Computation and Language · Cross-listed from cs.IR
Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies
Benjamin Clavié, Sean Lee, Aamir Shakir et al. (+1 more)
We propose Latent Terms, a method revealing that models trained for dense
retrieval, whether single- or multi-vector, learn representations that can trivially be decomposed into
retrieval-ready sparse features. When trained on frozen retrievers, Sparse Autoencoders without any
retrieval-specific adjustments extract a latent vocabulary with approximately Zipfian collection statistics, directly suitable for classical sparse
retrieval scoring via BM25. This approach enables sparse
retrieval while requiring no learned expansion objective or sparse
retrieval supervision whatsoever, and can be readily applied to any dense retriever. Latent Terms is able to match or outperform single-vector scoring methods from its own base model as well as comparable SPLADE variants. In addition, it substantially outperforms its base model on LIMIT, a task specifically designed to highlight the failures of single-vector
retrieval. Overall, our results highlight that neural retrievers contain more expressive and indexable structure than their default scoring functions expose, but that other methods can nonetheless be leveraged.
Read the full abstract ▾
All authors Benjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato
Computation and Language
BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
Rohan Shravan
We present BrahmicTokenizer-131K, a 131,072-vocabulary byte-level BPE tokenizer that closes the Brahmic compression gap at the 131K-vocabulary class while preserving the English, EU-
language, and code compression of OpenAI's o200k_base. We construct it through a two-stage retrofit: (1) a script-prune crop that reduces 200,019
tokens to 131,072 by removing nine out-of-scope writing systems, and (2) a
surgical retrofit of 2,372 corpus-dead vocabulary slots determined by linear-programming allocation across nine Brahmic Unicode blocks. The pre-tokenizer, decoder, and inherited merge rules are unchanged from o200k_base, making BrahmicTokenizer-131K a drop-in replacement at the tokenizer interface. On 27 million documents of public Indic pretraining text (2.84 billion words, 46.21 GB), BrahmicTokenizer-131K produces 26.7% fewer
tokens than Mistral-Nemo Tekken / Sarvam-m at the same vocabulary budget, with per-
language savings of 15.79% (Tamil) to 76.79% (Odia, a 4.31x compression ratio). The Odia advantage is mechanistically explained by Tekken/Sarvam-m containing zero Oriya-block
tokens; our surgery added 725. On non-Indic content, BrahmicTokenizer-131K matches o200k_base's English fertility (1.235 vs 1.232
tokens/word) and beats Tekken/Sarvam-m by 4.0-14.2% on HumanEval, MBPP, and GSM8K. Across our 14-tokenizer
benchmark, it is the only tokenizer simultaneously competitive on Brahmic, English, EU, code, and math at the 131K budget. Specialist tokenizers at other vocab classes (Sarvam-30B, Sarvam-1, MUTANT-Indic) achieve better Indic compression at the cost of non-Indic performance: Sarvam-1's English fertility is 15.9% worse and its code/math compression 26-33% worse than ours. We release the artifact under Apache 2.0 at https://huggingface.co/theschoolofai/BrahmicTokenizer-131K.
Read the full abstract ▾
Computation and Language
SURGENT: A Surgical Multi-Agent Assistance System Across the Perioperative Workflow
Dongsheng Shi, Yue Li, Xin Yi et al. (+3 more)
The intricate nature of modern
surgical care necessitates intelligent systems that can synthesize extensive patient records, support collaborative decision-making, and provide transparent, auditable
reasoning across the entire perioperative workflow. Although web-based Large
Language Models (LLMs) possess advanced
reasoning capabilities, they are ill-equipped for
surgical applications due to critical limitations: input length constraints, incomplete
memory management, and limited traceability. To address this issue, we present SURGENT, a
surgical multi-
agent assistance system that combines a Tree-of-Thought planner, multi-department collaboration
agents, and
retrieval-augmented
reasoning with clinical guidelines and biomedical literature. SURGENT features a novel
memory design that manages both long-term patient histories and short-term working summaries, enabling more complete, contextualized, and consistent
reasoning. Experimental evaluations across five key perioperative tasks - case analysis,
surgical plan simulation, safety monitoring, complication risk assessment, and rehabilitation guidance - show that SURGENT outperforms baseline LLMs and existing medical multi-
agent frameworks, yielding recommendations more closely aligned with patient histories. Ablation studies further highlight the advantage of DeepSeek as a locally deployable backbone model, enabling
privacy-preserving deployment without reliance on centralized services. These results position SURGENT as a practical and trustworthy advancement toward intelligent, equitable, and secure
surgical assistance systems.
Read the full abstract ▾
All authors Dongsheng Shi, Yue Li, Xin Yi, Yongyi Cui, Huawei Feng, Linlin Wang
Computation and Language
Attention Asymmetry in AI Layoff Discourse on X: A Computational Analysis of Capital vs Labour Amplification
Joy Bose
When workers lose jobs to AI-driven restructuring, two very different conversations happen on X (formerly Twitter) at the same time. Tech executives and AI researchers talk about productivity, transformation, and opportunity. Laid-off workers and labour critics talk about job loss, uncertainty, and fear. This paper asks a simple question: which conversation gets more reach? We report three studies using two collection methods and 763 tweets from 20 named public accounts. Study 1 used keyword-based collection (n=392) and found no significant difference between corpora (p=0.891), revealing that keyword search is too noisy for this task. Study 2 used account-based collection (n=96) and found a 3.12x mean amplification advantage for capital discourse over labour discourse (p=0.000003, Cohen's d=0.555). Study 3 combined both methods (n=763) and confirmed the finding at 4.18x mean and 10.77x median amplification ratio (p<0.000001). Critically, after normalising for follower count, the asymmetry persists at 2.69x (p=0.000009, Cohen's d=0.491), demonstrating that the effect is not simply a consequence of capital accounts having larger audiences. The finding is robust across all tested amplification metric weightings. We introduce the Amplification Ratio and Amplification Normalisation Index as simple metrics for measuring platform-level discourse inequality. A cross-platform replication on Reddit (n=647 posts) did not replicate the finding, suggesting the asymmetry may be specific to X's account-based amplification architecture. We discuss the methodological implications for cross-platform discourse analysis.
Read the full abstract ▾
Computation and Language
Casual as an Anchor: Resolving Supervision Misalignment in Formality Transfer Dataset
Hyojeong Yu, Hyukhun Koh, Minsung Kim et al. (+1 more)
Formality transfer is commonly framed as a symmetric bidirectional task between informal and formal registers. We argue that this framing conceals a supervision design flaw in existing
benchmarks such as GYAFC: binary human rewrites encode relative stylistic shifts rather than absolute human notions of formality. Consequently, models learn to generate pseudo-formal outputs that satisfy
benchmark labels while failing to produce genuinely formal
language. We quantify this misalignment by re-evaluating
benchmark formal labels under a human-aligned definition of formality, revealing substantial discrepancies that propagate to consistent informal-to-formal failures across model families. To address this issue, we reconceptualize formality transfer as a graded dimension rather than a binary attribute. We introduce a three-level spectrum: informal, casual, and formal, where casual serves as an explicit intermediate state that clarifies supervision signals. Based on this framework, we introduce 3LF, a
dataset providing parallel supervision across all three levels. Training on 3LF substantially reduces informal-to-formal failures and improves
alignment with human perception. For example, GPT-4.1-nano improves from 0.06 to 0.88 F1 in the informal-to-formal direction despite 3LF being significantly smaller than GYAFC. We further demonstrate that these gains cannot be reproduced through in-context learning alone and provide qualitative analyses of ambiguity-driven errors and meaning distortions. Overall, our findings demonstrate how supervision design shapes stylistic
alignment and highlight the importance of
alignment-aware
benchmark construction in controllable text generation.
Read the full abstract ▾
All authors Hyojeong Yu, Hyukhun Koh, Minsung Kim, Kyomin Jung
Computation and Language
Draft-OPD: On-Policy Distillation for Speculative Draft Models
Haodi Lei, Yafu Li, Haoran Zhang et al. (+8 more)
Speculative decoding accelerates large
language model inference by pairing a target model with a lightweight draft model whose proposed
tokens are verified in parallel. A common way to build draft models, like EAGLE3 or DFlash is supervised fine-tuning (SFT) on target-generated trajectories. However, we observe that SFT quickly plateaus: the draft model's acceptance length on test data stops improving. The reason is an offline-to-inference mismatch: In SFT, the drafter learns from fixed target-generated trajectories, whereas during speculative decoding it is evaluated on blocks proposed under its own policy. This motivates on-policy
distillation (OPD), where the target model supervises the drafter on draft-induced states. Yet OPD remains difficult for draft models, as they cannot reliably roll out complete sequences independently, whereas target-assisted generation makes the collected sequences follow the target distribution and thus eliminates the on-policy signal. We therefore propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions. This allows the drafter to learn from target feedback on both accepted and rejected proposals, focusing training on the draft-induced errors that limit speculative acceptance. Experiments show that Draft-OPD achieves over $5\times$ lossless acceleration for thinking models across diverse tasks, improving over EAGLE-3 and DFlash by 23% and 13%.
Read the full abstract ▾
All authors Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang, Qianjia Cheng, Xiaoye Qu, Ganqu Cui, Bowen Zhou, Ning Ding, Yun Luo, Yu Cheng
Computation and Language · Cross-listed from cs.CV
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu et al. (+14 more)
Multimodal large
language models are increasingly deployed as long-horizon
agents, where
memory must do more than recall: it must track an evolving world, revise what has gone stale, and surface the right evidence at decision time. Existing
benchmarks measure recall over static dialogue, collapse
memory into a single end-of-task accuracy, and reduce visual observations to captions, leaving us unable to localize failures to writing, maintenance,
retrieval, or use. The rise of
agent harnesses that author their own
memory sharpens this gap, since we have no principled way to compare hand-designed pipelines with self-managing alternatives. To close these gaps, we formulate
multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle, and instantiate it in WorldMemArena: 400 multi-session
multimodal tasks spanning Lifelong
Evolution (evolving personal and task states) and Agentic Execution (
memory from real observations, actions, and feedback), annotated with gold
memory points, updates, distractors, and evidence chains for stage-level diagnosis. This enables the first head-to-head comparison of long-context, manually designed (RAG and external
memory systems), and harness-based
memory agents. Results show that: (1) better
memory writing and storage do not guarantee better performance; (2)
multimodal memory still struggles to fully use visual evidence; (3) systems are unstable across domains and degrade on realistic agentic trajectories; and (4) harness
memory is more flexible but remains costly and less reliable.
Read the full abstract ▾
All authors Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu, Yepeng Liu, Lin Long, Yichen Guo, Nuo Chen, Zhaotian Weng, Elena Kochkina, Simerjot Kaur, Charese Smiley, Xiaomo Liu, James Zou, Sheng Liu, Yuheng Bu, Songyou Peng, Xin Eric Wang
Computation and Language
A Study on Question-Answer Dataset for LLM Safety Evaluation with a Focus on Illegal Activities
Kenji Imamura, Masao Ideuchi and Atsushi Fujita
In this paper, we discuss question-answer
dataset for LLM safety evaluation, with a focus on illegal activities. Specifically, on the basis of manual analysis of AnswerCarefully, we introduce several additional information, methods for creating question-answer examples, and a rubric for evaluating LLM-generated responses. The outcomes of this study are intended to be shared with the "JAI-Trust" project.
Read the full abstract ▾
Computation and Language
Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding
Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito et al. (+3 more)
Improving the quality of model-generated summaries, especially factuality, the accuracy of a summary with respect to its source content, remains a challenge. While reranking could select the optimal output from multiple generated candidates, it is limited to only using the source as guidance, resulting in unreliable summaries. To address this limitation, we propose ConSUM that reranks candidate summaries by considering two factors: consistency to the source document and consensus among the other candidates. Consensus is established using Minimum Bayes Risk (MBR) decoding over the set of generated summaries, while ensuring consistency by employing factuality-aware metrics that compare the summary against the source. Rigorous testing demonstrates that our system is competitive with existing methods, with human evaluations further confirming that its generated summaries are preferred over those from other systems. Our code is available at https://github.com/naist-nlp/ConSUM .
Read the full abstract ▾
All authors Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito, Jingun Kwon, Manabu Okumura, Taro Watanabe
Computation and Language
Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
Junlin He, Yihong Tang, Tong Nie et al. (+5 more)
Efficient
Distillation (EDistill) compresses large
language models (LLMs) by structured
pruning parameters and tuning lightweight modules with high training efficiency. Although these EDistilled LLMs achieve state-of-the-art (SOTA) performance on general ability
benchmarks relative to similarly sized LLMs, we identify a severe degradation in their multi-step
reasoning ability, which we term
reasoning collapse. We systematically analyze the
geometric origins of
reasoning collapse and show that the SOTA EDistill method based on width-reducing projection
matrices suffers from eRank collapse, in which the effective rank (eRank) of hidden representations drops. We theoretically explain how singular values of randomly initialized projection
matrices become unevenly distributed, leading to eRank collapse and thus
token indistinguishability. To address this issue, we propose RED (
Reasoning-preserved Efficient
Distillation) for LLMs, which introduces activation-aware initialization to initialize projection
matrices as channel-selection
matrices, thus theoretically mitigating eRank collapse. Experiments on Llama and Qwen series demonstrate that RED substantially recovers
reasoning while maintaining high training efficiency and SOTA general ability.
Read the full abstract ▾
All authors Junlin He, Yihong Tang, Tong Nie, Guilong Li, Binyu Yang, Jinxiao Du, Lijun Sun, Wei Ma
Computation and Language
STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments
Junyang Wang, Haiyang Xu, Xi Zhang et al. (+4 more)
Mobile GUI
agents excel at immediate reactive control but frequently fail in realistic, long-horizon tasks that require
memory. This failure stems from a fundamental conflict between limited context windows and
token-heavy screenshots. To save the limited context,
agents must progressively discard older visual history, permanently losing crucial transient information. Furthermore, existing action-centric
datasets fail to teach
agents what or when to explicitly memorize, and augmenting static real-world data is prohibitively expensive and lacks interactive verification. To resolve this, we present STAMP, a framework that trains explicit
memory in mobile
agents through controllable virtual environments, where deterministic
memory variables are programmatically injected into synthesized tasks to control what must be memorized, when it should be encoded, and when it must later be retrieved, thereby producing verifiable supervised data at scale and enabling online reinforcement learning through environment-driven reward feedback. Evaluated on our newly introduced
Memory-World
benchmark, the resulting Stamp-GUI
agent achieves state-of-the-art performance among GUI-specialized models and sets a new high watermark on our
Memory-World
benchmark, demonstrating exceptional
memory accuracy and task resilience while maintaining strong general mobile
navigation capabilities.
Read the full abstract ▾
All authors Junyang Wang, Haiyang Xu, Xi Zhang, Zhaoqing Zhu, Ming Yan, Jieping Ye, Jitao Sang
Computation and Language
Rethinking Stepwise Model Routing: A Cost-Efficient Table Reasoning Perspective
Shenghao Ye, Yuxiang Wang, Yu Guo et al. (+8 more)
Large
Reasoning Models (LRMs) achieve strong performance on table
reasoning tasks but incur substantial inference cost due to long
reasoning traces. Stepwise model routing mitigates this issue by dynamically assigning
reasoning steps to smaller or larger models. However, stepwise model routing for table
reasoning remains underexplored. Through empirical analysis, we find that
reasoning steps involving tables contain two types of
tokens with distinct uncertainty distributions: table
tokens grounded in table structure, such as
cell values and headers, and text
tokens representing surrounding natural-
language reasoning. The uncertainty of both
token types is correlated with the risk that the model makes an error in the next
reasoning step. However, existing methods fail to model them separately, leading to suboptimal routing decisions. To address this, we propose EcoTab, a table-aware stepwise routing framework for efficient table
reasoning. At each
reasoning step, EcoTab separately estimates the uncertainties of table
tokens and text
tokens, maps them to next-step failure risks for the small model, and combines the two risks for routing. Experiments on multiple table
reasoning benchmarks show that EcoTab consistently outperforms strong baselines and achieves a better balance between accuracy and efficiency.
Read the full abstract ▾
All authors Shenghao Ye, Yuxiang Wang, Yu Guo, Tao Zhang, Shuangwu Chen, Zijian Wen, Qihui Zhu, Mengjie Zhang, Dong Jin, Wang Luo, Jian Yang
Computation and Language
FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning
Juneyoung Park, Seongbae Lee, Han-Sang Lee et al. (+6 more)
Parameter-efficient fine-tuning(PEFT) has largely focused on LoRA and its accuracy-oriented variants, leaving the original goal of reducing trainable parameters has receivedcomparatively little
attention. We introduce FoRA, which revisits this goal by reducing the number of adapted layers rather than adapter rank. FoRA selects task-informative layers via a single-pass diagonal Fisher score (under 1% of training cost) and trains the LoRA down-projection at selected layers on the Stiefel
manifold, preserving column orthonormality and effective rank. FoRA consistently outperforms LoRA and DoRA at half their parameter budget, and falls within 0.7-0.8 accuracy points of AdaLoRA at one-quarter its parameter count, across five LLaMA-family backbones. Cross-architecture experiments on twelve backbones from the LLaMA, Qwen3, and Gemma families confirm consistent gains from 270M to 32B parameters. The two components combine super-additively: Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.
Read the full abstract ▾
All authors Juneyoung Park, Seongbae Lee, Han-Sang Lee, Kyuho Lee, Minjae Kim, Seungheon Hyeon, Kiduk Kwon, Seongwan Kim, Jaeho Lee
Computation and Language
PatchBoard: Schema-Grounded State Mutation for Reliable and Auditable LLM Multi-Agent Collaboration
Shuyu Zhang, Yaqi Shi, Jiarui Zhang et al. (+2 more)
LLM multi-
agent systems often coordinate through natural-
language dialogue or loosely structured shared
memory, making intermediate state difficult to validate, attribute, and audit. We introduce PatchBoard, a schema-grounded collaboration architecture that replaces inter-
agent dialogue with validated JSON Patch mutations over a shared structured state. An Architect
agent constructs a task-specific schema and workflow rules, while a deterministic kernel validates each proposed state mutation against schema constraints, role-specific write contracts, and runtime invariants before committing it transactionally. On 630 matched ALFWorld episodes, PatchBoard achieves an 84.6% success rate, compared with 30.8% for LangGraph and 61.6% for Flock, while reducing
tokens per successful task to 45.5k, compared with 368.3k and 64.2k, respectively.
Read the full abstract ▾
All authors Shuyu Zhang, Yaqi Shi, Jiarui Zhang, Yanxiao Zhao, Lu Wang
Computation and Language · Cross-listed from cs.AI
Rubric-Guided Process Reward for Stepwise Model Routing
Shenghao Ye, Yu Guo, Zhengheng Li et al. (+10 more)
Stepwise model routing improves the efficiency of Large
Reasoning Models (LRMs) by assigning each
reasoning step to a suitable model. Recent methods formulate routing as a sequential decision process and train the router with reinforcement learning. However, although they model routing as a process, they still supervise the router with outcome rewards. Such rewards only reflect final answer correctness and fail to evaluate intermediate routing decisions, which can weaken performance and generalization. To address this gap, we propose RoRo, a rubric-guided process reward framework for stepwise model routing. RoRo first collects diverse routing trajectories and constructs preference pairs based on outcome, cost, and process quality. It then trains a Rubricor to generate a query-specific evaluation rubric and a Judge to score routing trajectories under this rubric through alternating optimization. The resulting process rewards are combined with outcome rewards to optimize the routing policy via GRPO. Experiments on five
reasoning benchmarks under both same-family and cross-family settings show that RoRo consistently outperforms strong baselines and achieves better accuracy and cost trade-offs.
Read the full abstract ▾
All authors Shenghao Ye, Yu Guo, Zhengheng Li, Qiang Huang, Shuangwu Chen, Wang Luo, Qian Rui, Jie Zhang, Tao Zhang, Dong Jin, Xiaobin Tan, Quan Zheng, Jian Yang
Computation and Language
GrepSeek: Training Search Agents for Direct Corpus Interaction
Alireza Salemi, Chang Zeng, Atharva Nijasure et al. (+4 more)
Large
Language Model (LLM) search
agents have shown strong promise on knowledge-intensive tasks through iterative
reasoning and
retrieval. Most existing systems rely on retrievers that return ranked documents from a pre-built index. We explore a complementary paradigm in which the
agent treats the corpus as the search environment and finds evidence through executable shell commands. We introduce GrepSeek, an optimized direct corpus interaction (DCI)
agent that learns to find, filter, and compose evidence over large text corpora. To stabilize reinforcement learning (RL) over large corpora, we train in two stages: first, we initialize the policy using verified, causally grounded search trajectories generated by an answer-aware Tutor and an answer-blind Planner; then, we refine the policy using Group Relative Policy Optimization (GRPO). To make DCI practical at scale, we introduce two semantics-preserving execution optimizations: Pruned Adaptive Command Execution, which reduces shell-based search latency by up to $77\times$ on a 14GB corpus with 21 million documents using a compact auxiliary structure, and Sharded-Parallel Corpus Search, which achieves up to $7.6\times$ speedup without additional preprocessing; both preserve equivalence with sequential execution. Across eight open-domain QA
benchmarks, GrepSeek achieves the strongest overall performance, with a statistically significant relative improvement of $5.7\%$ over the best baseline. Our analysis shows how DCI-optimized
agents conduct flexible and effective compositional search through direct corpus interaction.
Read the full abstract ▾
All authors Alireza Salemi, Chang Zeng, Atharva Nijasure, Jui-Hui Chung, Razieh Rahimi, Fernando Diaz, Hamed Zamani
Computation and Language
MusTBench: Benchmarking and Advancing Temporal Grounding in Music LLMs
Daeyong Kwon, Qiyu Wu, Shinobu Kuriya et al. (+6 more)
Recent Large Audio-
Language Models (LALMs) have demonstrated promising abilities in understanding musical content. However, whether their responses are grounded in the correct temporal regions of the audio remains underexplored. This limitation is particularly critical for music understanding, where key information often occurs as temporally localized events, such as instrument entries and rhythmic transitions. To address this gap, we introduce MusTBench, a music-expert-validated
benchmark designed to evaluate temporal grounding in LALMs through five temporally grounded question-answering tasks. To further improve temporal grounding in existing models, we propose MusT, a novel four-stage temporal optimization recipe spanning music encoder adaptation, LLM adaptation, LLM supervised fine-tuning, and RL-based optimization. Experiments on MusTBench show that existing LALMs struggle with precise temporal grounding, while MusT brings significant improvements over strong baselines. These results establish temporal grounding as a key missing capability in current LALMs and position MusTBench as a challenging
benchmark for future research in temporally grounded music understanding.
Read the full abstract ▾
All authors Daeyong Kwon, Qiyu Wu, Shinobu Kuriya, Junghyun Koo, Shuyang Cui, Zhi Zhong, Wei-Hsiang Liao, Hiromi Wakaki, Yuki Mitsufuji
Computation and Language
Accommodation Goes Both Ways: Studying Linguistic Convergence Between Humans and Language Models
Terra Blevins
As LLMs become increasingly integrated into daily life, understanding how their presence will shape human linguistic behavior is an open question. We present a large-scale study of linguistic convergence in human-LLM dialogue, examining how humans and LLMs accommodate each other's linguistic style during multi-turn conversations. Using an asymmetric convergence metric on WildChat, a corpus of real-world ChatGPT transcripts, we find that while LLMs significantly overconverge toward their users on both function word and open-class features across eight
languages, human convergence rates in this setting are broadly consistent with human-human baselines. These findings suggest that accommodation in human-LLM dialogue is asymmetric: while LLMs dramatically overfit to their users' style, humans linguistically accommodate LLMs no differently than they would another person.
Read the full abstract ▾
Computation and Language
Prompt-Level Reward Specifications for Open-Ended Post-Training
Zijun Weng, Xiaohui Hu, Shuangyong Song et al. (+3 more)
Open-ended post-training benefits from rewards that make
prompt-specific success conditions explicit, rather than relying only on post-hoc scalar scores. In instruction following, writing, and decision-support tasks, response quality depends on local requirements, holistic preferences, and explicit constraints, but existing reward methods often leave these criteria implicit or cover only narrowly verifiable cases. We propose a
prompt-level reward specification framework that separates reward specification from reward computation. Given only
prompts, our framework constructs reusable task-adaptive rubrics and executable hard-constraint checkers offline, making reward criteria explicit before training and reusable across rollouts. At scoring time, artifact-anchored rubric and code scores are combined with an independent global score for residual holistic quality, yielding a normalized hybrid reward over requirement satisfaction, holistic quality, and deterministic constraints. The framework requires no human preference annotations, reference answers, or a separately trained reward model. Experiments show that the resulting reward improves offline RM-style response ranking and supports online reinforcement learning across multiple open-ended
benchmarks. Ablations further show that rubrics, global scoring, and executable verification provide complementary supervision.
Read the full abstract ▾
All authors Zijun Weng, Xiaohui Hu, Shuangyong Song, Yongxiang Li, Kaidong Yu, Xuanjing Huang
Computation and Language
Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimization
Yun Wang, Xin Xia, Xuansheng Wu et al. (+2 more)
LLM-based automated scoring approaches near-human performance, but scaling to new tasks remains bottlenecked by the per-item human configuration of upstream stages such as rubric construction. Human experts bypass this bottleneck through evaluation heuristics developed over extensive practice. We ask whether LLMs can learn similar heuristics directly from scoring experience, and formalize this as the concept of assessment skills: item-independent natural-
language procedural knowledge that guides LLMs through specific stages of the scoring workflow. Focusing on rubric construction as a first instantiation, we propose an iterative framework that decomposes a skill into a fixed scaffold and learnable item-agnostic rules, refining the rules through LLM-driven diagnosis of scoring errors and validation-gated selection. The framework requires no expert-written rubric. On all ten ASAP-SAS items, optimized skills substantially improve LLM-based scoring and frequently surpass the
dataset-provided expert rubric. Cross-item transfer experiments further reveal that learned skills capture both generalizable and item-specific patterns.
Read the full abstract ▾
All authors Yun Wang, Xin Xia, Xuansheng Wu, Xiaoming Zhai, Ninghao Liu
Computation and Language
Compute Allocation for Self-Evolving LLMs: From Depth-Breadth to Multi-Armed Bandits
Sixue Xing, Haoyu He, Kerui Wu et al. (+4 more)
LLM-guided evolutionary search (Evolve systems) has reached state-of-the-art results on mathematical and combinatorial tasks, yet most existing systems report only the best of many runs and leave the run-to-run distribution undocumented. We ask how a fixed budget of LLM calls should be allocated, and how reliably a single run reaches the reported numbers. Sweeping the depth-breadth grid over five models and three tasks, we identify two empirical regularities: a fitness-compute envelope along which capability ordering largely collapses when measured in effective FLOPs, and a bilinear depth-breadth fit with task-specific interaction; both are gated by model-task capability. Motivated by these regularities, we propose BaSE (Bandit-based Self-Evolving), a multi-armed bandit that allocates LLM calls across parallel trajectories. Without changing the model,
prompt, or evaluator, BaSE improves mean fitness by 12.3% over the strongest island-protocol baseline across 8 (model, task)
cells, with the largest gains on high-variance settings: a reliability gain from allocation alone.
Read the full abstract ▾
All authors Sixue Xing, Haoyu He, Kerui Wu, Zhuo Yang, Haozheng Luo, Tianfan Fu, Aarthy Nagarajan
Computation and Language
DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agents
Rongsheng Zhang, Jiji Tang, Junnan Ren et al. (+6 more)
Role-playing with large
language models is fundamentally a session-level task, requiring
agents to sustain character identity and interaction quality across extended multi-turn conversations. Yet existing evaluation and optimization methods remain largely turn-level, failing to capture long-horizon quality. We propose DynSess, a unified session-level framework for role-playing
agents. DynSess-Eval scores complete dialogue sessions via rubrics targeting long-horizon behaviors. Leveraging its session-level rewards, we construct high-quality training trajectories through multi-turn lookahead search and train DynSess-Character with two complementary variants: DSPO (off-policy) and GSRPO (on-policy). Experiments show that DynSess-Eval aligns with human judgments substantially better than prior evaluators, and blind human evaluation further shows that DynSess-Character matches the strongest character model despite using substantially fewer parameters, while maintaining strong role consistency and interactive ability. Our
dataset and code will be released to facilitate future research.
Read the full abstract ▾
All authors Rongsheng Zhang, Jiji Tang, Junnan Ren, Zuyi Bao, Weijie Chen, Ruofan Hu, Zhou Zhao, Tangjie Lv, Yan Zhang
Computation and Language
OmniRetrieval: Unified Retrieval across Heterogeneous Knowledge Sources
Jinheon Baek, Soyeong Jeong, Sangwoo Park et al. (+5 more)
Real-world information needs require access to structurally diverse knowledge sources, from unstructured text and relational tables to knowledge graphs and property graphs. Existing retrievers, however, operate over one source at a time under a fixed query
language, leaving the broader landscape of available knowledge fragmented behind incompatible interfaces. A natural attempt at unification would collapse these sources into a shared space, but this erases the structural affordances (such as schemas, ontologies, compositional operators) that give each source its expressive power. Effective
retrieval over diverse knowledge, therefore, requires not homogenization but an overarching layer that meets each source on its own terms. To achieve this, we present OmniRetrieval, a framework that takes any natural-
language query, identifies appropriate knowledge sources, and dispatches source-native queries to their native execution engines. Across an extensive
benchmark spanning 13
datasets and 309 distinct knowledge bases over text, relational, and graph-structured sources, OmniRetrieval exceeds single-source baselines, demonstrating that it can serve as a general-purpose interface to the heterogeneous sources while preserving the structural distinctions that make each source valuable.
Read the full abstract ▾
All authors Jinheon Baek, Soyeong Jeong, Sangwoo Park, Woongyeong Yeo, Minki Kang, Patara Trirat, Heejun Lee, Sung Ju Hwang
Computation and Language · Cross-listed from cs.AI
DenseSteer: Steering Small Language Models towards Dense Math Reasoning
Yang Ouyang, Shuhang Lin and Jung-Eun Kim
Large
language models (LLMs) demonstrate strong chain-of-thought (CoT)
reasoning abilities, while smaller models (<= 3B parameters) significantly underperform on multi-step
reasoning tasks. Based on empirical analyses of the Qwen-2.5 model family on math
reasoning benchmarks, we find that more proficient
reasoning is associated with fewer
reasoning steps but higher information density per step, a property we term Dense
Reasoning. Motivated by this observation, we propose DenseSteer, a training-free inference-time steering framework that enhances small-model
reasoning by modulating internal representations toward dense
reasoning patterns. Experiments show that our method yields consistent accuracy improvements without increasing
token-level Negative Log-Likelihood, highlighting dense
reasoning as an effective structural approach to mathematical problem solving.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
Implicit Identity Technologies for LLMs: Fingerprinting and Watermarking across Datasets, Models, and Generated Content
Bing Liu, Shunping Wang, Yufan Zhu et al. (+5 more)
This paper presents a survey and taxonomy of LLM fingerprinting and watermarking for identity, ownership verification, provenance, and generated-content attribution. Large
language models (LLMs) require substantial investments in data, computation, and expertise, and are increasingly deployed in high-stakes settings, making it critical to protect LLM-related assets and trace their origins. Existing work has rapidly expanded across
dataset provenance, model ownership, and generated-content detection, but the field remains fragmented: fingerprinting and watermarking are often used inconsistently, and methods are typically studied within isolated asset-specific settings. To address this gap, we introduce implicit identity as a unifying abstraction for verifiable but not directly observable identity signals in LLM systems. We distinguish fingerprinting as non-intrusive identity derived from intrinsic characteristics, and watermarking as intrusive identity deliberately embedded into data, models, or generated content. We then propose a lifecycle-based taxonomy that organises techniques across
datasets, models, and generated content, and further separates them by verification semantics: similarity-based attribution and keyed verification. Finally, we establish an evaluation framework centred on identifiability, robustness, and deployability, summarising representative metrics under realistic access and transformation regimes. By unifying terminology, lifecycle stages, and evaluation objectives, this survey provides a structured foundation for studying LLM identity technologies and for developing more reliable mechanisms for asset protection and provenance.
Read the full abstract ▾
All authors Bing Liu, Shunping Wang, Yufan Zhu, Xinyi Yu, Jing Huang, Linkang Du, Hongbin Pei, Wei Luo
Computation and Language
Wait! There's a Way Out: A Decision Mechanism for Forecasting Conversational Derailment
Laerdon Kim, Vivian Nguyen and Cristian Danescu-Niculescu-Mizil
Forecasting conversational derailment is the task of predicting, as the conversation unfolds, whether it will eventually derail into personal attacks. Since
forecasting models operate in an online fashion, they must decide whether to "trigger" an alert after each utterance–for example, to notify participants or a moderator that the conversation is at risk of derailing. Existing approaches make this decision solely based on the estimated likelihood of derailment given the preceding utterances, implicitly assuming that the conversation's future trajectory is fixed. As a result, they ignore the possibility of future recovery and incur an unnecessarily high rate of false positives. In this work we propose a method for decoupling the decision to trigger from derailment likelihood estimation. Our approach is inspired by the first human baseline on this task, which shows that humans achieve dramatically lower false positive rates by selectively deferring their decision to trigger when they anticipate that tension is likely to subside. We operationalize this insight with a deferral mechanism that uses forward-looking simulations to assess whether a tense moment admits plausible paths to recovery. Incorporating this mechanism into a state-of-the-art
forecasting model substantially reduces false positives without sacrificing
forecasting accuracy. More broadly, this work highlights the value of treating decision-making as a first-class component of
forecasting systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Surfacing Isolated Learners with Outcome-Independent Mediation of Feedback between Teachers and Students Using AI
Junsoo Park, Youssef Medhat, Htet Phyo Wai et al. (+2 more)
AI-augmented classrooms generate rich teacher and student feedback before graded outcomes become available, yet these signals can be difficult to translate into timely instructional decisions. We propose an interpretable decision layer: a transparent mechanism that ranks course topics requiring
attention without using grades or post-hoc outcome labels. The approach combines three signals: student learning difficulty prevalence, disagreement between learner self-reports and observed difficulties, and unresolved teacher concerns. The output is a ranked set of topic priorities with per-topic decision records explaining each ranking. In one graduate CS course offering ($n=5$ instructor interviews; $n=279$ survey responses), prioritized topics aligned with instructor concerns (top-5 overlap 3/5; Spearman $\rho=0.80$) and student-reported topic difficulty ($\rho=0.46$, $p=.048$). Multi-signal integration also surfaced learners not identified through individual signal sources alone (AUC $=0.96$ vs. $0.91$ for gap prevalence alone). Reflective thinking, help-seeking, and self-efficacy provided additional evidence that student behavioral signals align with learning-related constructs. While preliminary, these findings suggest that transparent coordination mechanisms may help support human-AI co-agency when feedback is incomplete.
Read the full abstract ▾
All authors Junsoo Park, Youssef Medhat, Htet Phyo Wai, Ploy Thajchayapong, Ashok K. Goel
Computation and Language
Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agents
Aditya Nawal, Manit Baser and Mohan Gurusamy
AI
agents augment large
language models with external tools such as web
retrieval, enabling grounded and up-to-date responses. However, incorporating external content into the generation pipeline can weaken the safety
alignment mechanisms that govern model outputs. Prior work shows that enabling
retrieval in
agents increases compliance with harmful requests. We introduce AgentREVEAL, a diagnostic framework for analyzing
retrieval-induced safety degradation in LLM
agents. The framework examines two axes: how
retrieval is integrated into the
agent pipeline and the properties of the retrieved content. Along the integration axis, we find that binding tool invocation and response generation in a single step amplifies harmful outputs. Along the content axis, we uncover the Safe Source Paradox: even oppositional or safety-oriented sources, such as pages containing warnings or risk disclaimers, can increase harmful compliance by an average of 25% compared to the no-
retrieval baseline. Finally, we show that relevance acts as a shared activation condition for both vulnerabilities. Similar patterns appear on frontier closed models, and harmful compliance remains elevated under several representative pipeline interventions, with some
agents also entering this regime under autonomous
retrieval. Because relevance is also what makes
retrieval useful, these results expose a safety-utility trade-off for
retrieval-enabled
agents. We introduce HarmURLBench, a
benchmark containing 1,405 real-world URLs paired with 320 harmful behaviors to support future evaluations.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
GTA: Generating Long-Horizon Tasks for Web Agents at Scale
Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey et al. (+4 more)
Web
agents, which couple
language models with browsing and tool-use capabilities, show promise as open web assistants. Yet progress is increasingly limited by the lack of scalable, process-level supervision. Existing
benchmarks are largely manually constructed, providing only coarse start-goal annotations without intermediate trajectories, while recent automatic generation efforts remain expensive, biased, and shallow. These limitations prevent reliable training and evaluation of
agents that must generalize to realistic, multi-hop, cross-page tasks. We introduce a scalable framework, GTA, that integrates crawling,
retrieval-based seeding, in-context generation, and automated quality control to produce realistic tasks paired with executable trajectories. This design decouples crawling from generation for greater efficiency, grounds tasks in the site graph to enforce compositionality, and ensures dense supervision through deterministic replays and systematic validation. We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with
multilingual and multi-hop coverage. The resulting
benchmark reveals a significant human-
agent performance gap and enables detailed diagnostics. Our contributions are three-fold: (i) formalizing multi-hop web-
agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic
benchmark with reproducible evaluation.
Read the full abstract ▾
All authors Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu