Computation and Language · Cross-listed from cs.CV
Tingyu Song, Mingxin Li, Yanzhao Zhang et al. (+5 more)
Sparse
retrieval underpins modern search systems, from web search to
retrieval-augmented generation. Existing work has introduced Learned Sparse
Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to
multimodal settings still relies heavily on auxiliary cross-modal modules. To address these limitations, we introduce UEmbed (Unified
Embedding), a decoder-only
multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special
tokens to the input and partitions the vocabulary into N disjoint subsets. Each
token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, outperforming
multimodal embedding models trained on publicly available data (e.g., RzenEmbed). On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications. Overall, UEmbed offers a new paradigm: it unifies dense and sparse
embeddings in one model, while further extending sparse
retrieval to unify text and
multimodal inputs.
Read the full abstract ▾
All authors Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Zhijie Nie, Yilun Zhao, Shu Wu
Computation and Language
Romanized Arabic Across Dialects: Views, Usage Patterns, and Linguistic Variation
Amr Keleg, Ahmed Amine Ben Abdallah, Taha Yassine et al. (+3 more)
Arabizi refers to Arabic written in Latin script. Although previous studies have shown that the prevalence and usage of Arabizi vary by factors such as region and age group, most NLP research on Arabic texts treats it as a temporary phenomenon resulting from limited technological support for the Arabic script. In this work, we engage with Arabic speakers to collect insights on their perceptions and usage of Arabizi. We further examine writing norms among speakers of different dialects, focusing on Algerian, Egyptian, Lebanese, Moroccan, and Tunisian Arabic. To this end, we release two resources. First, a character-level
alignment of Arabic words to study inter- and intra-dialectal variation across these five dialects, based on words transliterated by survey participants, finding systematic intra-dialectal regularity and inter-dialectal variation. Second, to study Arabic speakers' ability to identify this stylistic variation at the sentence-level, we build a manually curated parallel corpus of sentences written in Arabic script alongside multiple Arabizi transliterations, collected from speakers of the same five dialects. Our study presents the largest human-centered, cross-dialectal study of Arabizi's perceptions and practices to date.
Read the full abstract ▾
All authors Amr Keleg, Ahmed Amine Ben Abdallah, Taha Yassine, Chadi Helwe, Imane Guellil, Nedjma Ousidhoum
Computation and Language · Cross-listed from cs.CY
Who Should Be Generated? Justifying Demographic Targets in Open-Ended Generation
Zeshen Zheng, Yujia He, Qianmian Lin et al. (+2 more)
Fairness evaluation concerns not only what a model produces, but also what its outputs ought to be compared against. When a model generates "a CEO in the United States," the
prompt leaves demographic realization to the model. Existing group
fairness definitions assume that sensitive attributes are given on the input side. Generative audits instead examine output-side demographic composition, yet the targets they compare it against are typically supplied rather than justified. The upstream question is what the target distribution should be. We formalize this missing-target problem for demographic-value-unspecified generation and decompose target construction into four commitments: the evaluative object, prior admissibility, allocation, and operationalization. In this framework, we admit the geographic prior under a geographic-membership interpretation for the declared public-world use. The occupational prior, under an incumbency interpretation, requires an independently defended objective such as workforce-composition fidelity. Instantiating this construction in AP-Bench, we find substantial distribution divergence from geography-derived targets, ranging from 0.508 to 0.606 on a 0-to-1 scale. Replacing each geography-derived target with an equal-category comparator, while holding generations and measurement fixed, produces model-specific mean absolute
cell-level $\mathrm{JSD}_2$ changes ranging from 0.279 to 0.355. Target construction is therefore not a preliminary to
fairness evaluation but a component of it. What we supply is not a universal target, but a framework that makes explicit the justification required before a distribution can serve as a
fairness standard.
Read the full abstract ▾
All authors Zeshen Zheng, Yujia He, Qianmian Lin, Xiangyue Huang, Wenqing Chen
Computation and Language
MedPRESS: A Multi-turn Benchmark for Patient-Pressure-Induced Medical Sycophancy in LLMs
Saman Sarker Joy and Niloy Farhan
Large
language models (LLMs) are increasingly used for health-related advice. Existing research measures their safety with static questions rather than pressured patient-facing conversations. We introduce MedPRESS, a multi-turn
benchmark for measuring patient-pressure-induced sycophancy in LLMs. MedPRESS contains 600 medically grounded five-turn dialogues across three scenario families: medication and treatment demand, personal health self-care, and symptom triage and care resistance. Each dialogue begins with a health query and escalates through personal experience, social proof, external evidence claims, and direct
adversarial challenge. We evaluate 20 LLMs across general, medical-domain, lightweight, large, open-weight, and proprietary families using structured judging and safety-focused metrics. Results show that models frequently shift toward unsafe agreement under repeated patient pressure, with substantial variation across model families, model scale, and
prompt type. Anti-sycophancy prompting improves robustness for several models, but does not eliminate unsafe agreement. MedPRESS highlights a critical gap in medical LLM evaluation: safe medical knowledge is not enough unless models can maintain it under conversational pressure.
Read the full abstract ▾
Computation and Language
LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference
Zhichen Liu, Ruihan Sun, Hengjie Yang et al. (+4 more)
Long-running assistants and
agents consume interaction streams that eventually outgrow the context. Existing context retention, summarization, and
retrieval preserve access to selected history, but do not provide a persistent state over the full lifecycle when working context changes. We formulate this missing inference capability as state continuity under context turnover: carrying computation forward through a fixed-capacity
memory state whose lifetime is independent of the active context. We introduce an intrinsic
memory method, LiveMem, which augments a pretrained full-
attention LLM with a
memory state that preserves the historical information over the whole lifecycle while the main
attention path retains a bounded KV window. Context turnover and
memory state maintaining,
memory-oriented post-training, and state-aware serving jointly make this
memory state load bearing after its originating
tokens are released. Our experiments show that LiveMem achieves leading overall performance among evaluated systems and other intrinsic
memory methods. Experiments on LongMemEval show that LiveMem is able to answer the question based on the
memory state, even when the supporting evidence has been removed from the current context, and evidence-distance analysis shows that useful information persists beyond the active window. LiveMem thus establishes state continuity as a distinct and complementary abstraction for continual LLM inference.
Read the full abstract ▾
All authors Zhichen Liu, Ruihan Sun, Hengjie Yang, Zipeng Wu, Zhaohan Chen, Xiaofan Zhang, Yang Xu
Computation and Language · Cross-listed from cs.LG
RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States
Yi Yang, Zhennan Chen, Yihong Zhuang et al. (+5 more)
Learning-based
memory systems for self-evolving LLM
agents face two tightly coupled challenges. First, trajectory-indexed utilities grow with the interaction history, thereby dispersing limited feedback over an ever-expanding state space. Second, because trajectory-level rewards are jointly assigned to co-retrieved
memories, irrelevant experiences may receive misleading utility updates and consequently enter the
memory-reward trap. To address these challenges, we introduce Reduced-Order
Memory Reinforcement Learning (RoMeRL), which represents the growing trajectory-indexed utility space using a fixed-dimensional per-task
memory state factorized by outcome polarity and
memory dynamics. RoMeRL incorporates new experiences through a fixed set of semantic coordinates whose contents are updated or replaced over time, thereby concentrating feedback over a bounded utility support. Theoretically, we show that this reduced-order parameterization increases the average feedback received by each utility coordinate and characterize the steady-state occupancy of erroneous coordinates under a generic coordinate-transition model. Empirically, across ALFWorld and LifelongAgentBench, RoMeRL improves task performance, reduces the Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces the maintained
memory size by 84.4%, and cuts LLM calls by 21.1%. These results show that reduced-order utility states support efficient self-evolving
agent memory while limiting persistent reward contamination. Code is available at: https://github.com/YOUNG-fnxm/RoMeRL
Read the full abstract ▾
All authors Yi Yang, Zhennan Chen, Yihong Zhuang, Tiehan Fan, Yinan Chen, Jian Li, Jian Yang, Ying Tai
Computation and Language · Cross-listed from cs.SE
SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
Yuqiao Tan, Jinxiang Meng, Fangyu Lei et al. (+4 more)
Real-world software development requires coding
agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level
benchmarks typically evaluate
agents working alone or restrict user participation to messages. This leads us to ask: how do coding
agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when
agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon
benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace:
agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.
Read the full abstract ▾
All authors Yuqiao Tan, Jinxiang Meng, Fangyu Lei, Minzheng Wang, Shizhu He, Jun Zhao, Kang Liu
Computation and Language
Cultural Awareness is Represented but Not Decoded: Tracing Mythological Knowledge across 18 Open-Source LLMs
Iaroslav Chelombitko, Ekaterina Chelombitko and Mika Hämäläinen
Open-source LLMs reliably name Zeus, Jupiter, and Thor, but recover their counterparts in less-represented traditions like Finnish, Slavic, Egyptian, or Chinese mythology far less consistently. We ask where inside the model this cultural default is produced. On a parallel cross-cultural substrate of Thompson-motif entities, we instrument 18 open-source LLMs from 8 architecture families with linear probing, logit lens, activation patching, and output extraction. The residual stream cleanly distinguishes cultures, well above a name-string baseline, yet the decoder collapses culturally-specific
tokens onto dominant-tradition ones. The failure is at readout, not at representation. Asking the same question in the target culture's native
language versus English produces failures that cluster within
language but decouple across
language: the decoder is gated on
prompt language. We release a per-entity (probe, output) decomposition framework, a citation-anchored cross-cultural ground truth, a within- versus cross-mode correlation test for
language-conditioned readout, and per-entity predictions for all 18 models.
Read the full abstract ▾
Computation and Language
CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs
Muhammad Roman, Karen Rafferty and Barry Devereux
Trust is fundamental in modern regulatory ecosystems, and compliance checking plays a critical role in fostering that trust. Regulatory compliance verification is essential for businesses operating in highly controlled environments, as it ensures
alignment with sector-specific guidelines across domains such as financial reporting, data
privacy, and cybersecurity. Manual compliance testing, however, is often time-intensive and prone to inconsistencies, particularly when compliance depends indirectly on third-party services such as cloud providers, where vendors rely on external providers to meet regulatory standards. In this paper, we present CTRAG, a novel
Retrieval-Augmented Generation (RAG) pipeline designed for automated compliance checking. CTRAG employs advanced strategies, including adaptive chunking, dynamic
retrieval configurations, and in-context learning, to improve the precision and relevance of compliance assessments. By extracting control questions from regulatory texts and cross-referencing them with unstructured company documentation, CTRAG achieves highly accurate, document-informed compliance verification, even in cases of indirect compliance through third-party services. Empirical evaluations demonstrate significant improvements, with CTRAG achieving an F1-score of 78% and a recall of 85% in the final deployed configuration, ensuring minimal missed non-compliance cases while reducing manual reviewer effort in a real-world deployment. To validate CTRAG value, we developed and deployed a POC within a Big Four professional services firm, applying it to real-world cases and cross-checking results against manual compliance reports. These findings highlight CTRAG potential to streamline compliance workflows, mitigate risks, and enhance regulatory trust in complex, high-stakes environments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks
Xuan Ren, Weiqi Zhai, Tianle Pu et al. (+4 more)
Scientific
reasoning benchmarks typically evaluate large
language models (LLMs) using final-answer accuracy. However, a correct answer does not necessarily demonstrate the
reasoning capability targeted by the problem. We identify Solution Hacking, a failure mode in which an LLM reaches the correct answer through invalid shortcuts, such as numerical search, enumeration, guessing, or answer-first verification, without providing a valid task-targeted derivation. We systematically analyze this phenomenon across difficulty levels, scientific domains, and frontier models. Solution hacking increases sharply with
benchmark difficulty, from 2.2% on common problems to 28.3% on Olympiad-level problems and 37.4% on HLE. Moreover, 8.2%-44.1% of answers credited as correct across frontier models are identified as hacked solutions. We further develop expert-inspired anti-hacking strategies, including an automatic judge and a test-time instruction. The results show that suppressing shortcut behavior substantially reduces reported accuracy while having a smaller effect on correct and non-hacked accuracy. These findings reveal that answer-only evaluation can overestimate the scientific
reasoning capabilities of frontier LLMs.
Read the full abstract ▾
All authors Xuan Ren, Weiqi Zhai, Tianle Pu, Yihua Zhu, Yihua Zhu, Hu Wei, Bing Zhao
Computation and Language
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Nan Chen, Zhouhao Yang and Soufiane Hayou
Intent classification in Large
Language Models (LLMs) involves categorizing user
prompts into predefined classes. For instance, given a user
prompt, the system must determine whether it primarily concerns mathematics, coding, or general text processing. Such classification enables routing
prompts to specialized models optimized for specific domains, improving both accuracy and computational efficiency. In this work, we conduct a systematic study comparing training-free vs training-based approaches for intent classification. For this purpose, we consider two lightweight, training-free methods based on statistics of internal representations and compare them against MLP classifiers and linear probes. Our comprehensive empirical evaluation reveals that 1) Both training-free and training-based methods saturate easy
benchmarks (mathematics vs. coding vs. natural
language), 2) Training-based classifiers have an advantage on harder classification tasks (e.g. Java vs Python), and 3) Training-free methods are generally more robust to mixed-intent and
adversarial prompts.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.DB
Token-Native Storage: Read and Write in your Agent's Language
Kumar Shivendu
Search and database engines still store text as UTF-8, a format built for humans. But the systems that increasingly read and write that text (embedders, rerankers, and
language-model
agents) work with
token IDs, not characters, so every access pays to translate between the two. As
agents become the primary readers and writers of stored text, we argue for
token-native storage: keep the text as the model's own byte-pair-encoding (BPE)
token IDs. Packing r50k IDs as uint16 already beats UTF-8 by 2.25x on English with no compression, and an
entropy coder on top reaches 3.30x. Across six tokenizers and three corpora (English, code, Hindi), compressing
token IDs matches or beats every byte codec, even a corpus-trained zstd dictionary. Two findings sharpen the case. BPE numbers
tokens by merge order instead of frequency, and re-ranking by frequency lets a plain integer codec (streamvbyte) recover most of the
entropy coder's ratio while decoding ~7x faster, a near-free change to how AI labs publish vocabularies. And because a model reads
token IDs, not text, a
token-native store hands over the IDs directly instead of re-tokenizing on every read. The only requirement is that reader and writer share a tokenizer, and different model families often use different ones today, so we argue for standardization: a published, shared vocabulary, the way ASCII and UTF-8 standardized text.
Read the full abstract ▾
Computation and Language
PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noise
Abdulrahman AlRabah, Xiaocheng Yang, Dilek Hakkani-Tür et al. (+1 more)
Large
Language Models (LLMs) are increasingly deployed in task-oriented dialogue systems that support multi-step decision-making in high-stakes domains such as education, healthcare, and finance. However, existing
benchmarks typically assume perfectly accurate tool outputs, overlooking the reality that deployed systems must operate with noisy tools and human decision-makers whose trust in the
agent is itself uncertain. Such conditions are common in practice, for example, a clinician using a diagnostic prediction tool or an advisor relying on a model that forecasts student outcomes from historical records. We introduce PREDACTBENCH, a
benchmark for evaluating dialogue
agents paired with statistically imperfect tools, using education as a measurable testbed where ground truth outcomes and clear intervention decisions are available. First, we build a
benchmark for AI-assisted human decision-making, where the AI uses noisy predictors to help guide a user. Second, we introduce episode-level Relative AI-Reliance (RAIR) and Relative self-reliance (RSR) metrics, extending prior trust calibration framework to multi-turn dialogue. Third, we evaluate 13 state-of-the-art closed and open source LLMs on two educational
datasets, OULAD (real assessment trajectories from the UK Open University) and PREDACT-CS (60 courses with real final grade outcomes and synthetically generated weekly score trajectories), alongside a human study with instructors and teaching assistants. We find that when tools are noisy, SOTA models are supposed to provide visibility to teachers so that they do not over-rely on wrong suggestions or
hallucinations, but current models fail to do that. We offer PREDACTBENCH to help build better LLMs as AI decision support systems to help teachers.
Read the full abstract ▾
All authors Abdulrahman AlRabah, Xiaocheng Yang, Dilek Hakkani-Tür, Abdussalam Alawini
Computation and Language
Fast and Accurate Quotation Attribution in Literary Texts
Gaspard Michel, Hugo Attali and Elena V. Epure
Attributing quotations to their speakers in literary texts remains an open challenge. Standard methods, which independently predict a speaker mention for each quotation, are efficient but still limited in accuracy. In contrast, large
language model (LLM) approaches achieve strong performance, but their computational cost limits their use in large-scale literary analysis. We propose an encoder-based efficient formulation that resolves multiple quotation attributions within a shared, large context window. Using our new formulation, joint scoring, we report state-of-the-art (SOTA) performance on the Project Dialogism Novel Corpus (PDNC), comprising more than 35,000 manually annotated quotations from 22 English novels. Our best model reaches 94.5% overall attribution accuracy while processing novels $20\times$ faster than comparable standard methods and more than $1000\times$ faster than LLM-based approaches on an A100 GPU. An analysis of models' representations suggests that joint scoring improves on challenging attribution examples by preserving long-range anaphora resolution signal, an information that we found already present in pretrained encoders. To facilitate adoption, we release ModernBookNLP, a modified fork of BookNLP that replaces its quotation attribution model with our best system available at https://github.com/gasmichel/ModernBookNLP_QA/.
Read the full abstract ▾
Computation and Language
ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step
Vernon Toh, Navonil Majumder, Zhengyuan Liu et al. (+2 more)
To operate robustly in open-world environments, autonomous
agents should be able to infer the behavior of unfamiliar systems through interaction alone, even in the absence of documentation. However, existing tool-use
benchmarks expose semantic tool schemas in static environments, allowing
agents to rely on prior knowledge rather than autonomous discovery. To address this limitation, we introduce ScrambleToolBench, an interactive terminal
benchmark designed to isolate behavioral
reasoning. By removing semantic cues and enforcing a continuous task curriculum, the
benchmark requires
agents to uncover hidden tool behaviors entirely through trial-and-error interaction. The
benchmark further introduces dynamic challenges, including mapping drift,
stochastic action failures, and temporal execution windows, to evaluate whether
agents can revise and adapt their hypotheses as the environment changes. Our evaluation of state-of-the-art
language models reveals that successful initial discovery does not translate into robust adaptation. When faced with structural changes such as mapping drift,
agents fail to use deductive strategies such as cycle tracing, and instead exhibit belief inertia or fall back to exhaustive search. Increasing test-time
reasoning only amplifies this expensive brute-force search rather than enabling deductive recovery. While equipping
agents with persistent
memory reduces compounding errors, they remain unable to efficiently infer structural changes, highlighting a gap in current
agent reasoning.
Read the full abstract ▾
All authors Vernon Toh, Navonil Majumder, Zhengyuan Liu, Nancy F. Chen, Soujanya Poria
Computation and Language
Global Optimization and Inference-Time Region Grafting for Agentic Workflows
Donghyeok Koh, Gyuwan Kim, Jinyeong Bak et al. (+4 more)
Recent advances in agentic workflow optimization automate workflow design through task-specific workflow search or input-conditioned architecture selection. However, they determine the workflow before execution and cannot adapt failed workflow regions using execution-time label-free quality signals. Naively enabling such inference-time adaptation through whole-workflow re-optimization would be computationally prohibitive. To tackle this challenge, we introduce GRAFT, which preserves a globally optimized workflow while locally replacing only selected regions for each input. Without parameter training, GRAFT evaluates region-level alternatives using label-free execution-quality signals and accepts only replacements that improve local quality while preserving workflow-level consistency, thereby enabling instance-wise adaptation without whole-workflow re-optimization. GRAFT applies without modification across a range of tasks spanning mathematical
reasoning, code generation, and multi-hop and knowledge-intensive question answering. Under matched optimizer and executor settings, it improves over the strongest prior workflow-optimization method, MaAS, by 3.85 points on average. Replacing only the executor with a stronger model yields further gains without re-optimizing the global workflow. This suggests that an optimized workflow is not merely a static optimization artifact, but an adaptable execution policy that can evolve with inference-time feedback and stronger executors.
Read the full abstract ▾
All authors Donghyeok Koh, Gyuwan Kim, Jinyeong Bak, Seung-Hoon Na, Tao Yang, Haneol Jang, Cheoneum Park
Computation and Language · Cross-listed from cs.LG
Qwen-CUA: Native Computer Use for (almost) Everything
Dunjie Lu, Shuai Bai, Tianyi Bai et al. (+43 more)
Native computer use offers a general interface for
agents to operate almost any software available to people, but requires long-horizon state tracking, large-scale interactive experience, and learning from sparse yet verifiable outcomes. We introduce Qwen-CUA, a native computer-use
agent with a 397B-A17B Qwen mixture-of-experts backbone. It observes only screenshots and acts through keyboard and mouse events, without DOM trees, accessibility metadata, or task-specific APIs. Its scaffold maintains up to 20 active screenshots and folds older visual history in fixed-size blocks to retain recent evidence while preserving reusable
prompt prefixes. For training, we build a cloud rollout fleet with access to nearly 100,000 vCPUs and tens of thousands of concurrent environments, construct approximately 40,000 verifiable tasks, and collect personalized long-horizon workflows across everyday and professional software. We optimize complete trajectories with verifiable rewards and trajectory slicing, while iterative training runs refresh supervised data and recalibrate reinforcement-learning tasks. Across eight
benchmarks, Qwen-CUA outperforms Qwen3.7 and remains competitive with leading proprietary systems, reaching 86.2 on OSWorld-Verified and 18.5/48.4 binary/partial completion on OSWorld 2.0. Scaling the same recipe to a model with over one trillion parameters yields Qwen-CUA-Max, improving these scores to 87.6 and 21.2/53.3. Qwen-CUA also reduces RedTeamCUA attack success from 36.6 to 16.4 relative to Qwen3.7. Efficiency analyses, a browser deployment, and Bash-augmented experiments further characterize practical behavior. These results establish native computer use as a broadly capable
agent foundation and highlight scalable verifiable interaction and hybrid tool use as key directions.
Read the full abstract ▾
All authors Dunjie Lu, Shuai Bai, Tianyi Bai, Sicheng Fan, Chang Gao, Jian Guan, Feng Hu, Mianqiu Huang, Xingyang Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Ning Li, Dayiheng Liu, Shixuan Liu, Zheng Liu, Que Shen, Bowen Wang, Junli Wang, Chencan Wu, Rui Xie, Tianbao Xie, Zhihui Xie, Haiyang Xu, An Yang, Tao Yu, Wenzhen Yuan, Xi Zhang, Zhenru Zhang, Mingkang Zhu, Zhaoqing Zhu, Yizhong Cao, Kai Dang, Binyuan Hui, Kaixin Li, Junyang Lin, Haiquan Wang, Zekun Wang, Yiheng Xu, Fan Yan, Mengqi Yuan, Danyang Zhang, Jiajun Zhang, Zhipeng Zhang, Fan Zhou, Fan Zhou
Computation and Language
Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
Stefan Hut and Lorenzo Masoero
A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI
agents—conditioned on behavioral profiles and contextual descriptions of the intervention—simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a Simulated Randomized Controlled Trial (S-RCT) and derive a two-layer error decomposition that separates
agent approximation error from subsampling error, enabling targeted improvements to each. The framework is
agent-agnostic: any behavioral model—from a fine-tuned specialist to a general-purpose foundation model—can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by ${\sim}77\times$; a within-subject design—where each
agent is exposed to both arms—reduces standard errors by ${\sim}2.4\times$. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.
Read the full abstract ▾
Computation and Language
An Evidence-Grounded Retrieval-Augmented Transformer Framework for Health Misinformation Verification
Isah M. Bukar, Bala Mairiga Abduljalil, Bashir Saleh Maina et al. (+1 more)
The rapid spread of false and misleading health information through digital platforms has become a major public health challenge, particularly during infectious disease outbreaks where delayed verification can influence public behaviour and hinder effective disease control. Although recent advances in automated health misinformation detection have shown encouraging results, most existing approaches rely heavily on global biomedical resources and often fail to capture the local context needed to verify claims in developing countries. This study presents a
retrieval-augmented
transformer framework designed to verify health-related claims using trusted evidence from the World Health Organization and the Nigeria Centre for Disease Control and Prevention. The framework combines semantic evidence
retrieval with
transformer-based classification to determine whether a claim is true, false, or misleading. To evaluate the proposed approach, a manually annotated
dataset of 67 verified health claims covering coronavirus disease, Lassa fever, cholera, measles, and monkeypox was compiled from Nigerian fact-checking sources. Three
transformer models and a
retrieval-augmented configuration were evaluated. The Bidirectional Encoder Representations from
Transformers model achieved the best performance, with an accuracy of 71% and a weighted F1-score of 0.66. Although
retrieval augmentation did not improve classification performance because the current evidence repository was limited in size and coverage, the findings highlight the importance of comprehensive and authoritative knowledge sources for reliable health misinformation verification. The proposed framework provides a practical foundation for developing context-aware and evidence-driven health misinformation verification systems for Nigeria and other resource-constrained settings.
Read the full abstract ▾
All authors Isah M. Bukar, Bala Mairiga Abduljalil, Bashir Saleh Maina, Abdulbasit Hassan
Computation and Language
Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study
Ali Jafar, Amal Sarmad, Shifa Yousaf et al. (+1 more)
Recent advances in neural text-to-
speech (TTS) systems have substantially improved
speech naturalness and intelligibility across many
languages. However, comprehensive evaluation methodologies that jointly assess perceptual quality, speaker similarity, and acoustic fidelity across diverse
speech domains remain limited, particularly for low-resource and underrepresented
languages. This paper presents a reproducible, multi-metric benchmarking framework for systematic evaluation of modern TTS systems through domain-specific analysis. The proposed framework integrates complementary subjective and objective evaluation protocols and is demonstrated through a comprehensive case study on a representative low-resource
language spanning four
speech domains: Formal, Conversational, Literary/Storytelling, and Emotional. Four state-of-the-art TTS systems – Indic-Parler-TTS, MMS-TTS, Microsoft Edge TTS, and Google Gemini TTS – are evaluated using MUSHRA listening tests, ABX discrimination tests, speaker similarity scoring with Resemblyzer, and acoustic analyses based on mel-cepstral distortion (MCD) and F0 RMSE over 960 audio pairs. Results reveal substantial variation in TTS performance across
speech domains, with emotional
speech consistently presenting the greatest synthesis challenge (mean MCD 12.03 dB; mean F0 RMSE 889 cents), while conversational
speech achieves the highest overall acoustic fidelity. Beyond the empirical findings, this work provides a reproducible evaluation framework, publicly releasing evaluation scripts, result tables, and executable Colab notebooks to support standardized benchmarking and future research on TTS evaluation for low-resource
languages.
Read the full abstract ▾
All authors Ali Jafar, Amal Sarmad, Shifa Yousaf, Maryam Bashir
Computation and Language · Cross-listed from cs.IR
Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval
Chao Huang, Yufeng Chen, Changhao Guan et al. (+3 more)
Multilingual dense
retrieval aims to handle queries and documents across different
languages based on a unified retriever model. The challenge lies in enabling robust
retrieval transfer to low-resource
languages where annotated
retrieval data is often scarce. Although previous studies transfer high-resource supervision to low-resource
languages in
multilingual semantic representation learning, the shared representation often entangles semantic and linguistic features, which may interfere with optimizing semantic relevance for
retrieval. Different from existing methods that focus on learning
language-agnostic semantic features under such
entanglement, we propose a disentangled contrastive learning~(DCL) method for
multilingual dense
retrieval by separating
multilingual representations into semantic and linguistic subspaces. Specifically, we design disentangled optimization objectives based on hierarchical semantic
alignment and
language debiasing contrastive learning. By aligning
retrieval-relevant semantics across
languages at both sentence and
token levels while capturing
language-specific variations in the linguistic subspace, these objectives reduce
language-induced interference in semantic matching. We jointly optimize them with the
retrieval objective to facilitate stable zero-shot transfer from English supervision to
multilingual dense
retrieval. Extensive experiments on mMARCO and MIRACL show that our method consistently outperforms several strong baselines, demonstrating its effectiveness and generalization ability.
Read the full abstract ▾
All authors Chao Huang, Yufeng Chen, Changhao Guan, Guang Yang, Dongze Chen, Kaiyu Huang
Computation and Language · Cross-listed from cs.IR
Haonan Chen, Chu Li, Zhicheng Wang et al. (+4 more)
Multimodal representation learning is a cornerstone of modern AI. By encoding
multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern
agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM
embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin
Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified
multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an
embedding is grounded in
retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent
Reasoning organizes
retrieval evidence through hidden-space latent
reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong
video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.
Read the full abstract ▾
All authors Haonan Chen, Chu Li, Zhicheng Wang, Yuanwei Liu, Yuanjiang Wang, Shaohua Jiang, Zhicheng Dou
Computation and Language
Self-Improving Large Language Models via Progressive Experience Evolution
Shijie Ren, Xiting Wang, Meng Li et al. (+8 more)
Large
language models (LLMs) capable of self-improvement require not only effective policy optimization, but also a principled mechanism for transforming transient interaction experience into persistent model capabilities. Existing self-improvement paradigms remain fragmented: test-time methods can explicitly extract experience but cannot internalize it into model parameters, whereas training-time optimization methods can update model parameters but lack an explicit mechanism for accumulating transferable experience. Bridging these two paradigms requires a critical intermediate stage that remains underexplored, namely experience
distillation. To address this gap, we propose SPEE (Self-Progressive Experience
Evolution), a unified post-training framework that sequentially performs explicit experience
evolution followed by implicit policy optimization. During explicit experience
evolution, SPEE reflects on trajectories collected from multiple interactions to extract, verify, and progressively evolve transferable experience, which is subsequently internalized into the policy through privilege-guided On-Policy Self-
Distillation (OPSD). During implicit policy optimization, reward-driven reinforcement learning leverages these internalized priors to explore novel solution strategies. In the experience
evolution stage, a continuously evolving global experience pool consolidates knowledge from both successful and failed trajectories, filters out low-utility experience, and mitigates post-hoc rationalization induced by individual trajectories. Experiments on five mathematical
reasoning benchmarks demonstrate that SPEE consistently outperforms both test-time and training-time self-
evolution baselines across three model scales. The source code is available at https://github.com/rrrsj/SPEE.
Read the full abstract ▾
All authors Shijie Ren, Xiting Wang, Meng Li, Yujie Guo, Yunhang Yao, Ziheng Peng, Xunlong Wang, Yuetan Chen, Haoyang Zhou, Yunlong Liang, Fandong Meng
Computation and Language
The Role of Disfluencies in Speech Translation
Maike Züfle, Maria Teleki, Fabian Retkowski et al. (+5 more)
Current
speech translation systems, including SpeechLLMs, are trained on cleaned text and tend to strip disfluencies like filled pauses and false starts rather than translate them. We show this comes at a cost: disfluencies carry meaning that gets lost when
speech is cleaned up. To study this systematically, we introduce Uh-Mazing, a
benchmark of human-translated, disfluency-annotated Switchboard
speech covering English into eight target
languages. Across these
languages and several architectures, we find that false starts and self-repairs, not filled pauses or discourse markers, drive most of the translation-quality loss, and that models which fail to preserve a disfluency tend to omit it rather than mistranslate it. We show inference-time decoding can mitigate this without retraining, and release the
benchmark and code.
Read the full abstract ▾
All authors Maike Züfle, Maria Teleki, Fabian Retkowski, Vilém Zouhar, Oliver Grabner, Alexander Waibel, James Caverlee, Jan Niehues
Computation and Language · Cross-listed from cs.CV
HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models
Jin Cui, Chuanchang Su, Jiayi Lu et al. (+3 more)
Vision-
language models (VLMs) remain unreliable when predictions require fine-grained visual evidence. We identify a previously overlooked cause: spectral response
rigidity. Despite substantial frequency variation across images and tasks, pretrained
vision encoders exhibit persistent, encoder-specific layerwise spectral profiles that change only marginally under downstream fine-tuning. Since pretrained
vision encoders only receive images, they cannot adapt spectral extraction to the evidence required by the current query. We therefore propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation. Hierarchical Adaptive Frequency Injection (HAFI) retrieves complementary low-, mid-, and high-frequency evidence at multiple encoder depths using text-modulated, spatially aligned cross-
attention. A Visual Enrichment Layer Adapter further recalibrates shallow LLM
attention to effectively utilize the enriched visual
tokens. Experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate consistent improvements in general VQA, text-rich understanding, and
hallucination robustness, outperforming representation-level enhancement methods and most resolution- or cropping-based approaches without additional high-resolution encoding. Mechanistic analyses show that HAFI restores task-dependent spectral allocation while retaining semantic
attention, establishing frequency enrichment as a distinct and effective route for improving VLM perception.
Read the full abstract ▾
All authors Jin Cui, Chuanchang Su, Jiayi Lu, Xinyue Long, Boran Zhao, Pengju Ren
Computation and Language
From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding
Zixian Li, Tong Li, Chi Xie et al. (+2 more)
Speculative decoding accelerates LLM inference only when drafted continuations survive target-model verification. Semi-autoregressive drafters such as DSpark predict an entire
token block with one backbone forward and refine it with a lightweight Markov head. However, DSpark decodes this block as a single chain, so an early mismatch invalidates the remaining suffix and limits the benefit of large draft blocks. We show that the conditional structure already learned by DSpark can support multiple parent-consistent continuations without retraining or additional backbone passes. We introduce Parent-Conditioned Drafting Tree (PCTree), which uses the pretrained Markov head to score alternative children separately for each concrete parent and allocates a fixed verification budget to the most probable paths. This converts DSpark's linear draft into a tree while preserving its one-pass parallel backbone. Across Qwen3-{4B,8B,14B} and nine
benchmarks, at $B{=}7$, measured speedup gains over autoregressive (AR) decoding, relative to matched DSpark, range from $3.1\%$ to $29.5\%$. On Qwen3-4B GSM8K at $B{=}16$, PCTree increases mean acceptance length from $9.41$ to $11.16$ and three-run mean AR speedup from $6.14{\times}$ to $6.60{\times}$. These show that parent-conditioned branching can turn conditional capacity already present in a semi-autoregressive drafter into end-to-end inference gains through an inference-only change.
Read the full abstract ▾
All authors Zixian Li, Tong Li, Chi Xie, Xiaohui Song, Haonan Lu
Computation and Language
IACM-RL: Intent-Aware Context Management and Reinforcement Learning for Complex Tool Invocation under Dynamic Intent Fluctuations
Dingwei Zhu, Jiahan Li, Chengjun Pan et al. (+22 more)
Executing long-horizon tool invocations in real-world environments is severely challenged by dynamic user intent noise. Existing methods attempt robustness via implicit history scanning or text compression, yet predominantly assume perfect instructions in simplistic scenarios. Inevitably, under fluctuating contexts, obsolete constraints dilute model
attention, triggering intent deviation and repetitive API loops. To resolve this, we propose IACM-RL, a comprehensive framework for robust tool invocation. First, we introduce the DynamicIntent pipeline, synthesizing trajectories across several fine-grained fluctuation scenarios, paired with a five-dimensional diagnostic metric suite. Second, IACM-RL deploys a BeliefState-based Self-Generated Context Manager that proactively tracks shifting goals and isolates overwritten parameters using structural stale flags. To autonomously internalize this state-tracking capability, we optimize the policy using a hierarchical intent-driven reward alongside two auxiliary losses (CM extraction and state
distillation). Experiments on DynamicIntent, BFCL-V3, and $\mathrm{\tau}^2$-Bench demonstrate that IACM-RL outperforms baselines, reducing infinite loops and stale context errors while enhancing out-of-domain generalization.
Read the full abstract ▾
All authors Dingwei Zhu, Jiahan Li, Chengjun Pan, Yunxian Yang, Yunbin Zhao, Yunke Zhang, Zhonghang Lu, Zhuohui Sheng, Chenhao Huang, Jiahang Lin, Yajie Yang, Junlin Shang, Shichun Liu, Yuhui Wang, Honglin Guo, Junjie Ye, Xin Guo, Jiazheng Zhang, Ming Zhang, Shihan Dou, Zhiheng Xi, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang
Computation and Language
Cross-Domain Hybrid OPD for Generalizable Search Agents
Hongzhan Chen, Xiaoyu Liu, Dengming Zhang et al. (+11 more)
Recent advances in Reinforcement Learning (RL) have substantially improved the capabilities of autonomous search
agents, enabling sophisticated planning, and iterative
retrieval over dynamic information sources. However, optimizing
language models for specialized search behaviors often incurs an
alignment tax, where gains in search performance come at the expense of general-purpose capabilities, limiting their effectiveness as universal assistants. In this technical report, we present the training framework behind the Yuanbao search
agent, designed to achieve search specialization without sacrificing general intelligence. Built upon the Hunyuan3 architecture, our framework combines agentic reinforcement learning for autonomous search with a cross-domain expert On-Policy
Distillation (OPD) pipeline. Experts specializing in complementary general-purpose domains are distilled into the search-specialized student, restoring and further enhancing its broad capabilities. Rather than treating specialization and general capability as competing objectives, our hybrid training strategy jointly optimizes both, effectively mitigating the
alignment tax. Extensive experiments demonstrate that the resulting model achieves competitive search performance while consistently improving its general-purpose capabilities, providing a favorable balance between specialized execution and broad generalization in real-world search scenarios.
Read the full abstract ▾
All authors Hongzhan Chen, Xiaoyu Liu, Dengming Zhang, Minzhou Huang, Dongliang Xu, Jingcheng Xie, Dongxiang Fang, Bowen Qin, Minsheng Hao, Yaozong Shen, Xiaojun Quan, Mona Zhou, Haosheng Zou, Jeff Chen
Computation and Language · Cross-listed from cs.AI
Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy
Jim Dilkes, Vahid Yazdanpanah and Sebastian Stein
Post-training Large
Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which appends one of a small fixed set of instructions to task
prompts during training, using the same set for every problem, increasing the coverage of behaviours attempted. To facilitate ICE, we combine RL on the instruction-conditioned policy with self-
distillation of its correct rollouts into the unconditioned test-time policy. ICE with this objective improves Qwen3-1.7B held-out pass@1 performance at 4K response length on mathematical
reasoning tasks by $5.0\%$ relative to training with DAPO, with improvement persisting at a longer 8K context. The improvement does not appear for Qwen3-4B at 4K, where the instructions do not expand base-model coverage.
Read the full abstract ▾
Computation and Language
CAVE: Competence-Aware Visual Boundary Evidence Alignment for Video Temporal Grounding
Wei Jia, Zhicong Lu, Yu Chen et al. (+5 more)
Large
vision-
language models (LVLMs) have achieved substantial performance gains in
Video Temporal Grounding (VTG) through reinforcement learning (RL). However, existing methods primarily rely on outcome correctness rewards that evaluate only the final predicted intervals, leaving boundary-related visual evidence and its correspondence with timestamp predictions insufficiently constrained. In this paper, we delve into timestamp prediction and its underlying boundary-level visual evidence, showing prevalent misalignment between visual evidence and predicted timestamps across widely used
benchmarks. To address this issue, we propose Competence-Aware Visual Boundary Evidence
Alignment (CAVE), which augments localization optimization with boundary-specific visual evidence rewards to mitigate evidence-timestamp misalignment. Specifically, to explicitly represent the boundary-specific visual evidence, CAVE introduces boundary-specific evidence
tokens and initializes their structured generation and distinct boundary semantics through a lightweight supervised warm-up. During RL, the visual boundary evidence
alignment reward reinforces the visual
attention of special evidence
tokens within the ground-truth boundaries, thereby promoting
alignment between visual evidence and temporal boundaries. Moreover, performance-aware gating for evidence supervision is designed to adaptively retain evidence guidance for poorly localized groups while reducing it once localization becomes sufficiently accurate to avoid over-constraining fine-grained boundary refinement. Extensive experiments on several public VTG
benchmarks demonstrate the effectiveness of our method.
Read the full abstract ▾
All authors Wei Jia, Zhicong Lu, Yu Chen, Xiang Wang, Shuai Li, Wenqian Lv, Jiayue Cao, Huaxing Liu