Computation and Language
Towards Multidisciplinary Summarization of Hospital Stays: Efficient Sentence-Level Clinical Provenance Categorization
Baris Karacan, Vaibhav Bhargava, Barbara Di Eugenio et al. (+21 more)
Effective "all-team" summarization in high-complexity settings like the Neonatal Intensive Care Unit (NICU) requires aggregating insights from diverse disciplines (physicians, nurses, therapists) spread across hundreds of clinical free-text notes. Simply pooling heterogeneous text often leads to incoherent outputs. Structured summarization therefore first requires accurate categorization of sentence-level provenance across multi-source notes. This pilot study introduces a clinical provenance categorization pipeline using supervised fine-tuning (SFT) of large
language models (LLMs). We adapted two Llama-3 models (8B and 70B) to MedSecId, a corpus of 2,002 MIMIC-III (Adult ICU) notes annotated with clinical provenance headers, achieving in-domain Macro F1 scores above 92% for both models. To evaluate cross-domain generalization, we assessed model capacity (8B vs. 70B) and quantization on a gold-standard
dataset of 227 sentence-level spans derived from three multi-disciplinary NICU summaries. Experimental results demonstrate a scale-dependent transfer effect: while SFT produced only marginal changes for the 8B model, it substantially improved the 70B model, increasing Macro F1 by 7%. Notably, the quantized fine-tuned 70B model outperformed its full-precision baseline while substantially reducing computational requirements. These findings suggest that sufficient model capacity is critical for preserving semantic flexibility during cross-domain clinical transfer and that efficient quantized adaptation can enable structured provenance modeling for downstream summarization.
Read the full abstract ▾
All authors Baris Karacan, Vaibhav Bhargava, Barbara Di Eugenio, Natalie Parde, Mary Khetani, Yu-Shan Tseng, Vanessa Barbosa, Julie Vignato, Lindsey Knake, Rajashree Dahal, Emily Spellman, Danielle Hitzel, Janine Petitgout, Kristi Haughey, Amanda Karstens, Brianna Clarahan, Rachel Dawson, Lauren Boyd, Mackenzie Weis, Angie Tipton, Jaewon Bae, Catherine K. Craven, Karen Dunn Lopez, Andrew D. Boyd
Computation and Language · Cross-listed from cs.CR
Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Tools
Bardia Mohammadi, Lars Klein, Akhil Arora et al. (+1 more)
Tool-augmented
language agents speculatively issue likely future tool calls to hide latency, but those calls leak inferred user intent to external services before the
agent commits to the branch. Every external observer that received the call retains the disclosure after the
agent abandons the branch. Timing is the issue, not authorization: no commit-time cleanup, read-only restriction, or access-control allow-list unsends what an observer already holds. We call these invocations ghost tool calls and propose Speculative Tool
Privacy Contracts, a runtime abstraction that treats observation before commitment as a first-class effect, distinct from state mutation. We implement the contracts in a prototype runtime and evaluate twelve policies across three corpora. Speculative dispatch increases what an observer can infer about user intent; post-hoc filters, read-only restrictions, and access-control allow-lists leave that inference intact; only issue-time policies that change or suppress the speculative call's argument or destination projection before dispatch reduce it.
Read the full abstract ▾
All authors Bardia Mohammadi, Lars Klein, Akhil Arora, Laurent Bindschaedler
Computation and Language
Deokhyung Kang and Hyounghun Kim
Reasoning language models (RLMs) achieve strong performance on complex
reasoning tasks, but still exhibit substantial
multilingual reasoning gaps, largely due to
language-understanding failures when interpreting non-English inputs. Translation into English can mitigate these failures by expressing non-English inputs in a form that RLMs can more reliably interpret, yet translating every input incurs unnecessary inference cost when the model can reason reliably from the original query. To address this challenge, we propose Luar, a
Language Understanding Boundary-aware Reinforcement Learning framework that trains RLMs to selectively invoke translation when their direct understanding is unreliable. Luar trains the model to either solve the original input directly or reason over its English translation, encouraging translation only when translator-augmented
reasoning is expected to substantially outperform direct
reasoning. Across
multilingual reasoning benchmarks, Luar outperforms training-based baselines, with particularly large gains on low-resource
languages. On Qwen3-4B, it improves average accuracy by 8.4 percentage points over standard GRPO. Further analysis shows that Luar concentrates translator calls on queries where translation benefits
reasoning, helping to reduce inference cost. The project will be made publicly available at https://github.com/deokhk/LUAR.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents
Yiheng Shu, Bernal Jiménez Gutiérrez, Saisri Padmaja Jonnalagedda et al. (+3 more)
Language agents spend substantial inference time solving individual tasks, yet the experience acquired in one episode is often underutilized in future episodes. Continual learning expects an
agent to accumulate experience across a stream of tasks, improve over time, and avoid interference from irrelevant experiences. Unfortunately, existing
benchmarks struggle to evaluate this problem rigorously. Most efforts focus on
retrieval and
reasoning over long-context conversations or documents, while recent lifelong-adaptation
benchmarks often stream existing
datasets whose cross-task dependencies are unknown, making it difficult to understand what an
agent learns and reuses over time. This paper presents an evaluation framework AgentCL for continual learning in
agents, centered on distinguishable task streams and metrics for CL properties. AgentCL distinguishes three cross-task relations by the availability of reusable knowledge. In dependent streams, later tasks can reuse knowledge from earlier ones. In conventional streams, whether such reuse exists is unknown. Besides, independent tasks are held out with little transferable knowledge. We use the
benchmark to evaluate non-parametric
memory designs for continual learning. To diagnose how
memory design choices affect continual learning, we develop MemProbe, a probing method that stores interactions, insights, and skills, while filtering unreliable experiences during consolidation. Empirical analysis across coding, deep research, and
language understanding/
reasoning tasks shows that conventional streams offer limited ability to distinguish
memory designs, whereas dependent streams more clearly distinguish their plasticity. Meanwhile, conventional streams and independent tasks often yield limited gains and can expose
memory-induced degradation. These results highlight the need for stronger
memory designs that balance plasticity and stable reuse.
Read the full abstract ▾
All authors Yiheng Shu, Bernal Jiménez Gutiérrez, Saisri Padmaja Jonnalagedda, Yuguang Yao, Huan Sun, Yu Su
Computation and Language · Cross-listed from cs.AI
HLL: Can Agents Cross Humanity's Last Line of Verification?
Xinhao Song, Su Su, Sirui Song et al. (+6 more)
Multimodal agents are increasingly expected to operate interfaces on behalf of users, raising a central deployment question: can they truly substitute for humans in workflows that services deliberately protect against automation? CAPTCHA verification makes this question concrete. It is not merely a visual puzzle, but a human-verification boundary placed before account creation, content access, form submission, and other protected actions. We introduce Humanity's Last Line of Verification (HLL), a controlled
benchmark that uses interactive CAPTCHA verification to evaluate whether
agents can cross this boundary through grounded, human-like interaction rather than recognition alone. HLL covers diverse CAPTCHA interactions and exposes
agents to controlled realism stressors, including cluttered webpages, harder task variants, and trace-conditioned validation of the solving process. We evaluate eight frontier
multimodal agents in a closed-loop GUI environment. The results show that current
agents remain brittle at this human-substitution boundary: performance varies sharply across verification types, degrades under realistic interface conditions, and drops further when correct answers must be supported by valid action traces. By exposing gaps in localization, action calibration, state tracking, and process consistency, HLL provides a concrete testbed for measuring how close
multimodal agents are to acting as human substitutes in protected real-world workflows. Our code is available at https://github.com/XinhaoS0101/HLL
Read the full abstract ▾
All authors Xinhao Song, Su Su, Sirui Song, Hongliang Wu, Wen Shen, Zhihua Wei, Gongshen Liu, Linfeng Zhang, Dongrui Liu
Computation and Language · Cross-listed from cs.AI
Food Noise & False Safety: A Systematic Evaluation of How LLMs Fail to Adapt to Eating Disorder Queries with Clinician Feedback
Giulia Pucci, Emily Hemendinger, Ruizhe Li et al. (+3 more)
Recent evidence shows that people with eating disorders (EDs) are increasingly seeking guidance, advice, and emotional support from Large
Language Model (LLM)-based chat systems. Although these systems are not designed to provide clinical advice, their perceived expertise, neutrality and accessibility make them a frequent, albeit risky, source of support. This paper investigates potential patterns of interaction between users with EDs and LLMs, focusing on the potential harms arising from models that uncritically adapt to, and facilitate unsafe or self-harming user requests. We find, in consultation with clinical ED experts, that specific linguistic cues in
prompts increase the likelihood of unsafe responses and, through systematically varying the degree of potential risk present in the user
prompt, report the extent to which LLMs uncritically adapt to problematic, and potentially dangerous user inputs.
Read the full abstract ▾
All authors Giulia Pucci, Emily Hemendinger, Ruizhe Li, Gavin Abercrombie, Tanvi Dinkar, Arabella Sinclair
Computation and Language
PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning
Yusong Zhao, Yuejin Xie, Youliang Yuan et al. (+4 more)
Between the first visible sign of danger and the moment an accident occurs, there is often a window where intervention remains possible.
Video-capable
multimodal large
language models (MLLMs) could serve as always-on safety monitors that issue warnings during this window. Yet current
benchmarks do not test this ability: they rely on static inputs, ignore timing precision, and omit false-positive measurement on safe scenes. We present PaSBench-
Video, a 740-
video benchmark with 481 risk and 259 no-risk
videos across four domains: driving, healthcare, daily life, and industrial production. Risk
videos are annotated with frame-level risk onset and accident boundaries. A model must observe the
video causally and produce a warning that is both temporally calibrated and content-correct. Testing 13 MLLMs, we find that no model exceeds 20.0% on our strictest metric, and recall is tightly coupled with false-positive rate, with Pearson correlation 0.64: higher detection comes only at the cost of triggering warnings on the majority of safe clips. Performance splits sharply by domain: models achieve moderate recall at low false-positive rates in daily life, where risks are inherently anomalous, yet fire indiscriminately in driving, where routine and hazardous scenes look alike. These results indicate that current models rely on scene-level activity cues rather than
reasoning about emerging harm.
Read the full abstract ▾
All authors Yusong Zhao, Yuejin Xie, Youliang Yuan, Junjie Hu, Jitian Guo, Yujiu Yang, Pinjia He
Computation and Language · Cross-listed from cs.LG
On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
Mind Lab: Vin Bo, Song Cao, Vic Cao et al. (+62 more)
Parameter-efficient fine-tuning (PEFT) is usually treated as a cheaper alternative to full fine-tuning. We study a broader role: small trainable adapters as persistent local state on top of strong shared foundation models. In this framing, the base model provides shared competence while adapters carry instance-specific behavior such as preferences, skills, tool habits, and
memory-like updates. We organize the problem around three scaling axes: Scale Up, where stronger shared priors make small local updates more useful; Scale Down, where we study how small adapters can be while remaining reliable; and Scale Out, where many persistent adapted instances coexist. MinT provides one infrastructure example for managing adapter identity, revision, provenance, evaluation, and serving residency. Together, the results suggest that PEFT can be a compact substrate for persistent personal models rather than only a budget substitute for full fine-tuning.
Read the full abstract ▾
All authors Mind Lab: Vin Bo, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou, Peixuan Hua, Charles Huang, Miles Jiang, Nora Jiang, Yuyi Jiang, Qiuyu Jin, Fancy Kong, Andrew Lei, Kyrie Lei, Alexy Li, Lucian Li, Ray Li, Theo Li, Wenhao Li, Zhihui Li, Allen Lin, Jiayi Lin, Kairus Liu, Kieran Liu, Logan Liu, Xiang Liu, Irvine Lu, Maeve Luo, Runze Lv, Pony Ma, Verity Niu, Anson Qiu, Vincent Wang, Rio Yang, Maxwell Yao, Carrie Ye, Regis Ye, Wenlin Ye, Josh Ying, Danney Zeng, Yuhan Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Shiyang Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Adrian Zhou, Yuhua Zhou, Xinyue Zhu, Murphy Zhuang
Computation and Language · Cross-listed from cs.IR
ODTQA-FoRe: An Open-Domain Tabular Question Answering Dataset for Future Data Forecasting and Reasoning
Zhensheng Wang, Xiaole Liu, Wenmian Yang et al. (+3 more)
The rapid development of LLMs has significantly advanced tabular question answering, but most systems cannot perform future-oriented numerical prediction. To address this gap, we introduce a novel task, Open-Domain Tabular Question Answering for Future Data
Forecasting and
Reasoning, and propose the first
dataset to cover time-series
forecasting and forecast-based
reasoning scenarios using real estate data. This task poses challenges in retrieving precise historical data, overcoming the
forecasting limitations of LLMs, and standardizing responses for diverse queries. To solve the above challenges, we propose TimeFore, an LLM
agent-based framework that decomposes the problem into three collaborative roles: a Retriever autonomously generates SQL to fetch data, a Forecaster invokes external time-series models for higher accuracy, and an Analyzer synthesizes the results to construct a precise and consistent final answer. Extensive experiments demonstrate the effectiveness of our TimeFore.
Read the full abstract ▾
All authors Zhensheng Wang, Xiaole Liu, Wenmian Yang, Kun Zhou, Yiquan Zhang, Weijia Jia
Computation and Language
Investigating and Alleviating Harm Amplification in LLM Interactions
Ruohao Guo, Wei Xu and Alan Ritter
Large
language models (LLMs) can serve as helpful assistants, yet they can equally function as harm amplifiers that enable malicious users to achieve harmful outcomes beyond their capabilities through extended interactions. This risk manifests along two axes, i.e., democratizing domain expertise that allows novices to produce specialized harmful content, and scaling harmful operations at volumes that manual effort cannot match. Existing works, however, often overlook how LLMs compound harm across multi-turn conversations. We introduce HarmAmp, a new
benchmark for multi-turn harm amplification scenarios spanning twelve risk categories. Each scenario is grounded in real-world threats and satisfies rigorous criteria, i.e., substantive amplification, operational specificity, and multi-turn necessity. We further propose TrajSafe, a proactive monitor that anticipates harmful trajectories and intervenes through actions such as probing users' genuine intents and steering the models towards safer completion. Our extensive experiments demonstrate that TrajSafe significantly reduces the harmfulness incurred in multi-turn interactions while preserving a low over-refusal rate and the target model's general capabilities. Our work offers a promising paradigm to alleviate the nuanced safety risks in LLM interactions.
Read the full abstract ▾
Computation and Language
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Nahyun Lee, Dongkeun Yoon, Guijin Son et al. (+12 more)
Frontier model evaluations are shifting from foundational capabilities (e.g., instruction following and
reasoning) toward compositional, agentic ones, but Korean agentic
benchmarks remain scarce. We introduce K-BrowseComp, a web-browsing
agent benchmark grounded in Korean contexts, consisting of 400 problems. The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers. On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00–45.67%, a substantial drop from BrowseComp, while Korean LLMs released through Korea's Proprietary AI Foundation Model program obtain only 0.00–10.33%. We further construct a 100-problem synthetic split using hard few-shot exemplars and failure-mode-targeted generation to exploit the asymmetry between solving and creating web browsing problems. On the adversarially filtered synthetic diagnostic split, the strongest model reaches only 26.00%, and we report this split separately as a targeted stress test. We publicly release our data and code.
Read the full abstract ▾
All authors Nahyun Lee, Dongkeun Yoon, Guijin Son, Geewook Kim, Dayoon Ko, Jeonghun Park, Haneul Yoo, Jaewon Cho, Junghun Park, Changyoon Lee, Kyochul Jang, Jaeyeon Kim, Eunsu Kim, Woojin Cho, Seungone Kim
Computation and Language
AutoForest: Automatically Generating Forest Plots from Biomedical Studies with End-to-End Evidence Extraction and Synthesis
Massimiliano Pronesti, Angelo Miculescu, Mohsin Kapdi et al. (+8 more)
Systematic reviews rely on forest plots to synthesise quantitative evidence across biomedical studies, but generating them remains a fragmented and labour-intensive process. Researchers must interpret complex clinical texts, manually extract outcome data from trials, define appropriate interventions and comparators, harmonise inconsistent study designs, and carry out meta-analytic computations-typically using specialised software that demands structured inputs and domain expertise. While recent work has demonstrated that large
language models can extract study-level data from unstructured text, no existing system automates the complete pipeline from raw documents to synthesised forest plots. To address this gap, we introduce AutoForest, the first end-to-end system that generates publication-ready forest plots directly from biomedical papers. Given one or more study papers, AutoForest automatically suggests ICO (Intervention, Comparator, Outcome) elements, extracts outcome data, performs statistical synthesis, and renders the final forest plot. We describe the system architecture, user interface and demonstrate its effectiveness on real-world examples through a user study involving clinicians, showing how AutoForest can accelerate evidence synthesis and substantially lower the barrier to conducting meta-analyses.
Read the full abstract ▾
All authors Massimiliano Pronesti, Angelo Miculescu, Mohsin Kapdi, Paul Flanagan, Oisín Redmond, Joao Bettencourt-Silva, Gurdeep Mannu, Spiros Denaxas, Rui Bebiano Da Providencia E Costa, Anya Belz, Yufang Hou
Computation and Language · Cross-listed from cs.LG
Local Conflict, Targeted Repair: A Perturbation Analysis of Multi-Domain RL
Lei Yang, Siyu Ding and Deyi Xiong
Reinforcement learning (RL) post-training improves large
language models on individual domains but can degrade previously acquired capabilities. We study this interference in mathematical
reasoning, code generation, question answering (QA), and creative writing (CW). Full-model domain gradients can be nearly orthogonal despite substantial selective degradation. Our structural analyses show sparse, weakly overlapping edits on computation routes that
reasoning domains share, with directional
alignment associated with transfer and conflict. A local perturbation model separates loss changes into a first-order term and a path-averaged
curvature term, decomposes the latter along the actual rollback intervention, and provides a conditional explanation for recovery by short refresh. The experiments support this local sensitivity account. After Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW training, a short Re-Math refresh raises Math from 57.66 to 66.04 and achieves the highest average score among the evaluated baselines. A training-free rollback at a $9.8\%$ budget recovers $57.4\%$ of the QA-induced Math accuracy drop with a small QA change.
Curvature probes at the Math checkpoint further show higher sensitivity along the selected direction than along matched random directions. Together, these results support sparse intervention and targeted refresh as effective ways to mitigate cross-domain degradation.
Read the full abstract ▾
Computation and Language
SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence
Yuyan Bu, Haowei Li, Qirui Zheng et al. (+7 more)
As LLM-based
agents expand their operational scope, reliability becomes a prerequisite for real-world deployment. However, in practical applications, human users cannot monitor every immediate behavior; instead, the execution process often remains a black box, leaving users dependent solely on the
agent's self-reported updates. This opacity creates a critical risk:
agents may present observer-facing reports that diverge from their executed actions, rendering the system uncontrollable, especially in high-stakes autonomous scenarios. We term such self-reported plan-action divergence as
agent deception. To assess this, we introduce SPADE-Bench, a
benchmark designed to evaluate spontaneous plan-action divergence. Unlike prior deception
benchmarks, SPADE-Bench simultaneously integrates actual tool execution and controlled pressure scenarios. This design ensures ecological validity and rigorously distinguishes strategic deception from mere
hallucination through controlled plan-action comparisons under pressure. Experiments across mainstream models confirm that
agent deception is a genuine and pressing issue in tool-use contexts. By providing a comprehensive and robust evaluation framework, SPADE-Bench fills a critical gap in
agent safety, facilitating the community's progress toward building trustworthy and controllable autonomous systems.
Read the full abstract ▾
All authors Yuyan Bu, Haowei Li, Qirui Zheng, Bowen Dong, Kaiyue Yang, Jiaming Ji, Yingshui Tan, Wenxin Li, Yaodong Yang, Juntao Dai
Computation and Language
WaxalNet: Benchmarking and Adapting Speech Recognition for Spontaneous Speech in 19 African Languages
Victor Tolulope Olufemi, Oreoluwa Babatunde, Ramsey Njema et al. (+28 more)
Multilingual speech foundation models claim broad
language coverage, but how well they handle spontaneous African
speech, and how best to adapt them, remains unclear. We
benchmark zero-shot and fine-tuned
speech recognition on the WAXAL corpus of spontaneous
speech in 19 African
languages, covering Whisper, MMS, Omnilingual ASR, and SeamlessM4T-v2. Fine-tuning lowers macro word error rate (WER) from 52.3 for the best zero-shot model covering all 19
languages (Omnilingual-7B) to 36.2 for a 300M-parameter MMS model and 31.5 for Omnilingual with an LLM decoder, although zero-shot models are already competitive on a few
languages. Using Omnilingual's shared encoder, we compare CTC and LLM decoders with encoder pretraining held fixed. Between MMS and Whisper, the better model depends on the
language family, but with a shared encoder the LLM decoder has lower WER on 18 of 19
languages, so the family pattern more likely comes from pretraining than from the decoder. The LLM decoder is, however, about five times larger and much slower on CPU. Pooling
languages into one model, by family or across all 19, costs only 0.6 to 1.0 WER on average relative to per-
language models, a modest price for serving all
languages with one model. Fine-tuned models remain usable on read
speech, and for held-out
languages neither training on the other
languages nor in-context examples replace in-
language fine-tuning. A native-speaker audit across all 19
languages characterizes failure modes such as repetition loops. We release the code and a suite of over 100 fine-tuned models.
Read the full abstract ▾
All authors Victor Tolulope Olufemi, Oreoluwa Babatunde, Ramsey Njema, Bolarinwa Gbotemi, Wanchi Lucia Yen, John Uzodinma, Sunday Ajayi, Oluwademilade Williams, Kausar Moshood, Innocent Elendu Anyaele, Akebert Arefaine, Candace Hunzwi, Wongel Dawit Daniel, Emmilly Namuganga, Cleophas Kadima, Athanase Bahizire, Onitsiky Ranaivoson, Emmanuel Aaron, Nicholaus Ladislaus, Idris Muhammed, Jonathan Enoch Simenya, Martin Koome, Matewos Tegete Endaylalu, Peter Ifeoluwa Adeyemo, Hondi Prisca Birindwa, Ukachi Agnes Eze-Mbey, Yacoba Oduro-Yeboah, Pericles Adjovi, Mikel K. Ngueajio, Toluwani Aremu, Prasenjit Mitra
Computation and Language · Cross-listed from cs.AI
Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses
Pengcheng Jiang, Zhiyi Shi, Kelly Hong et al. (+5 more)
Search
agents are often trained as policies over growing transcripts: the model must decide how to search while also remembering what it has seen, which evidence is useful, which constraints remain open, and which claims have actually been checked. We argue that this formulation puts too much routine state management inside the policy: reinforcement learning is forced to optimize both semantic search decisions and recoverable bookkeeping that the environment can maintain more reliably. We introduce Harness-1, a 20B search
agent (
retrieval subagent) trained with reinforcement learning inside a stateful search harness. The harness maintains environment-side working
memory, including a candidate pool, an importance-tagged curated set, compact evidence links, verification records, compressed and deduplicated observations, and budget-aware context rendering. The policy retains the semantic decisions: what to search, which documents to keep or discard, what to verify, and when to stop. Across eight
retrieval benchmarks spanning web, finance, patents, and multi-hop QA, Harness-1 achieves 0.730 average curated recall, outperforming the next strongest open search subagent by +11.4 points and remaining competitive with much larger frontier-model searchers. Its gains are especially strong on held-out transfer
benchmarks, suggesting that reinforcement learning over explicit search state can produce
retrieval behaviors that generalize beyond the training domains. Our code is available at https://github.com/pat-jj/harness-1.
Read the full abstract ▾
All authors Pengcheng Jiang, Zhiyi Shi, Kelly Hong, Xueqiang Xu, Jiashuo Sun, Jimeng Sun, Hammad Bashir, Jiawei Han
Computation and Language · Cross-listed from cs.AI
CoMAP: Co-Evolving World Models and Agent Policies for LLM Agents
Youwei Liu, Jian Wang, Hanlin Wang et al. (+1 more)
Equipping
language agents with world models enables them to anticipate environment dynamics and evaluate candidate actions before execution. However, existing textual world models are typically fixed after training, preventing them from adapting to the on-policy state-action distributions induced by an evolving
agent. Meanwhile,
agent-improvement methods often rely on external rewards or verifiers, limiting their applicability in realistic interactive environments. In this paper, we propose COMAP, a novel framework that co-evolves textual world models and
agent policies through closed-loop interaction. At each decision step, the world model predicts future state feedback for candidate actions, and the
agent performs future-aware reflection by estimating the reliability of this feedback and refining its action accordingly. The resulting on-policy trajectories are then used to update the world model via self-
distillation, allowing it to better match the
agent's evolving interaction distribution. Across
embodied task planning, Web
navigation, and tool-use
benchmarks, COMAP consistently outperforms competitive baselines, e.g., +16.75% relative improvement with Qwen3-4B. Further analyses show that the co-evolutionary loop improves the world model's prediction accuracy over time and leads to more effective long-horizon decision-making. Our code is available at: https://github.com/loyiv/CoMAP.
Read the full abstract ▾
All authors Youwei Liu, Jian Wang, Hanlin Wang, Wenjie Li
Computation and Language · Cross-listed from cs.AI
Forget Attention: Importance-Aware Attention Is All You Need
Suhyeong Shin and Yeongwook Yang
Attention retrieves content precisely, but its scores depend only on query–key products, leaving order and recency to positional encodings. Conversely, state-space models (SSMs) such as Mamba-3 track order and recency through input-dependent state transitions that decay and rotate, yet compressing context into a fixed-size state hampers precise
retrieval. Hybrids like Samba and Hymba exploit this complementarity only across layers or heads, so SSM transitions never shape
attention scores directly. The closest prior work, the Forgetting
Transformer (FoX), adds an input-dependent forget gate to the scores but reweights
attention by recency alone. We propose SSM-Informed Softmax
Attention (SISA), which builds Mamba-3-style state transitions, rotation included, directly into the scores alongside FoX's gate. Each logit receives the weight such an SSM would give that
token pair: a decayed match between rotated SSM projections. This rotation already encodes relative position, so SISA drops RoPE; restoring it raises loss. SISA outperforms parameter-matched
Transformer and FoX in
language modeling and length extrapolation. Among the
attention models we compare, hybrids included, only SISA benefits from context beyond the training window. Building SSM transitions into the scores yields stronger
attention, and our kernels make it faster and leaner than baselines on their official or standard kernels.
Read the full abstract ▾
Computation and Language
TVIR: Building Deep Research Agents Towards Text-Visual Interleaved Report Generation
Xinkai Ma, Zhiqi Bai, Dingling Zhang et al. (+21 more)
Deep Research
Agents have shown strong capability in multi-step information
retrieval,
reasoning, and long-form report generation, but existing
benchmarks and systems remain predominantly text-centric, with limited evaluation of whether visual elements are factually reliable and well aligned with the surrounding analysis. To address this gap, we introduce TVIR (Text-Visual Interleaved Report Generation), which includes TVIR-Bench, a
benchmark of 100 expert-curated
multimodal deep research tasks that require visual elements to serve specific analytical sub-goals, and TVIR-
Agent, a hierarchical multi-
agent framework that serves as a strong baseline for constructing outlines, retrieving images, generating charts with traceable sources, and composing reports through context-aware sequential writing. We further develop a dual-path evaluation framework that combines Textual Assessment and Visual Assessment. Experiments across nine deep research systems show that TVIR-
Agent achieves strong overall performance, underscoring the importance of explicit
multimodal design and evaluation for evidence-driven report generation.
Read the full abstract ▾
All authors Xinkai Ma, Zhiqi Bai, Dingling Zhang, Pei Liu, Yishuo Yuan, He Zhu, Jiakai Wang, Qianqian Xie, Yifan Zhao, Xinlong Yang, Hao Cong, Zhiheng Yao, Fengxia Xie, Zihao Xu, Haoran Xu, Zhaohui Wang, Minghao Liu, Shirong Lin, Yingshui Tan, Yuchi Xu, Wenbo Su, Zhaoxiang Zhang, Bo Zheng, Jiaheng Liu
Computation and Language
Zixuan Zhu, Yitong Hu, Yong Dai et al. (+4 more)
LLM-based
agents can solve multi-step interactive tasks by combining
reasoning with environment feedback, yet each episode starts from the same fixed context and any useful strategy discovered along the way is lost once the task ends. Existing approaches either limit learning to the current task or pool all experience into a single untyped store, without distinguishing knowledge types, tracking quality through use, or balancing what the library still lacks. We introduce Unified Context
Evolution (UCE), a gradient-free framework that externalizes
agent experience into an evolving library of typed Evolvable Context Units (ECUs). UCE decomposes experience into four complementary types (
Memory, Strategy, Workflow, and Skill), each generated from trajectories under type-specific conditions, retrieved at decision time, scored through repeated usage outcomes, and pruned when no longer valuable. A scheduling module allocates each cycle's generation budget toward the types where the library is weakest. Across two interactive
benchmarks, UCE raises ALFWorld success from 75.4% to 96.3% and WebShop task score from 45.1% to 61.3%, and remains effective across alternative actor
reasoning backbones without model retraining.
Read the full abstract ▾
All authors Zixuan Zhu, Yitong Hu, Yong Dai, Junfeng Fang, Chunyang Jiang, Senkang Hu, Yuzhi Zhao
Computation and Language
Beyond Isolated Behaviors: Hierarchical User Modeling for LLM Personalization
Liang Wang, Xinyi Mou, Xiaoyou Liu et al. (+3 more)
Large
Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, yet personalizing their outputs to individual users remains an open challenge. Existing approaches predominantly adopt a flat behavioral paradigm, aggregating user behaviors without an explicit account of how they are organized into deeper behavioral structures. In this work, we draw on Pierre Bourdieu's Theory of Practice to propose PHF (Practice-Habitus-Field), a sociologically grounded framework that reconceptualizes LLM personalization through three hierarchical levels: individual behaviors as practices, their temporal accumulation into stable dispositions as habitus, and shared regularities across similar users as fields. We instantiate PHF through $\mathrm{PHF}_{\text{Compass}}$, a lightweight and model-agnostic implementation based on a frozen LLM. Experiments on the
Language Model Personalization (LaMP)
benchmark demonstrate consistent improvements across diverse tasks, while further analyses validate the interpretability and extensibility of the learned behavioral structures.
Read the full abstract ▾
All authors Liang Wang, Xinyi Mou, Xiaoyou Liu, Tiannan Wang, Yuqing Wang, Zhongyu Wei
Computation and Language
AI as a Tool for Simulation-Based Experiments in Literary Studies
Matthew Wilkens
Generative artificial intelligence (AI) systems open new possibilities for experimentation in literary studies via controlled, grounded, large-scale, low-cost simulations of cultural production. Current systems have not yet been shown to produce high-quality, book-length narrative texts that reliably reflect arbitrarily specified cultural constraints or stylistic features. But there exists substantial relevant research on each of the components required for literary-historical simulation. These include the use and validation of AI systems as proxies for differentiable human populations; the narrative and stylistic properties of AI-generated texts; the stability and coherence of multiagent, multiturn AI simulations of human actors; and technical methods through which to alter in predictable ways the knowledge and behavior of generative systems. Together, these areas could provide a starting point for more ambitious AI-based modeling of cultural systems of literary production. We describe the possibilities and challenges of simulation-based experiments in literary studies, summarize the current state of the art in relevant fields, and explain key technical aspects of the work. To provide an example directly relevant to literary scholars, we present the results of experiments on literary text generation, including comparisons to high-status, human-authored novels. Our results include the first demonstration of (limited) in-distribution outputs by AI models in this domain. We conclude with a description of future work on full counterfactual literary-historical simulations using AI.
Read the full abstract ▾
Computation and Language
DECK: A Consistency x Confidence Taxonomy of LLM Hallucinations
Mohit Singh Chauhan
Existing
hallucination taxonomies classify LLM errors by what is wrong with the output – memorised misconceptions,
reasoning failures, fluent fabrications – but cannot answer a different question: which uncertainty scorer would have caught this error? We propose a complementary taxonomy that classifies errors by their detectability signature, the signal a scorer family would read. The DECK taxonomy is a 2x2 partition along inter-sample consistency and
token-level confidence into four regimes (Drift, Entrenched, Confabulation, Knotted) that yields a falsifiable blind-spot map: black-box consistency scorers have signal in D and C, white-box
token-probability scorers in K and C, and only an LLM-as-a-Judge with independent pretraining can detect E. Across three models and four short-form QA
datasets we test this map two ways: judge-involving scorer disagreements concentrate in each family's predicted blind-spot
cells, and external labels (SelfAware unanswerable, HaluEval
adversarial, PopQA entity popularity) land in the predicted
cells, robustly to cross-fitted thresholds. We further identify a universal blind spot of output-level UQ: on knowledge-gap inputs where the generator emits confident, repeatable fabrications, every output-level family collapses by construction. A linear probe on Llama-3-8B's final-layer hidden states also falls to chance, with or without quantisation, though an intermediate layer retains weak signal.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Cross-modal linkage risk in clinical vision-language models
Soroosh Tayebi Arasteh, Mahshad Lotfinia, Sven Nebelung et al. (+1 more)
Vision-
language models (VLMs) trained on paired chest radiographs and radiology reports learn a shared
embedding space that can preserve instance-level image-report correspondence. This poses a
privacy risk in settings where radiographs and reports are deliberately kept separate after acquisition, such as image-only data sharing or access-controlled reports, because a de-identified image may be re-linked to its original narrative report through cosine similarity alone. We formalized this as image-to-report
retrieval and used public paired cohorts, in which the true pairing is known by design, as ground-truth
benchmarks to audit the risk rather than as the
privacy scenario. Evaluating VLMs of increasing clinical specialization on 406,241 paired examples from 126,804 patients across MIMIC-CXR (43,793 held-out pairs) and external CheXpert Plus (29,296 pairs), we found that re-linkage rose systematically with specialization: the strongest VLM retrieved the correct report at 15 times chance at a candidate pool of N = 100, 50 times chance at N = 10,000, and well above chance at full-database scale. The signal persisted under pathology-matched hard negatives that removed disease-label shortcuts, indicating correspondence beyond broad diagnostic categories. To reduce it without retraining, we froze both encoders and applied differentially private optimization only to the projection heads defining the
alignment layer (epsilon = 0.34, delta = 6x10-6). This reduced Recall@1 by 61.8% at N = 10,000 on MIMIC-CXR and transferred to CheXpert Plus without retraining, while image-side utility was largely preserved: macro AUROC for linear-probe classification across 14 labels shifted only from 79.63% to 79.43%. Targeted DP finetuning of the shared
alignment layer can substantially reduce cross-modal re-linkage without materially degrading the image representations that make these models clinically useful.
Read the full abstract ▾
All authors Soroosh Tayebi Arasteh, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn
Computation and Language
Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli et al. (+10 more)
Human annotation is the empirical foundation of much NLP research, from
dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled. We provide the first large-scale, task-level audit of human annotation reporting across major NLP venues, asking which annotation details are documented, which are missing, and how reporting varies across time, topic, venue, and intended use of human judgment. We introduce a unified taxonomy of annotation-reporting practices and validate an LLM-assisted extraction pipeline against Annotated-gold, a human-adjudicated gold standard of 41 papers and 72 annotation tasks, where the best model reaches human-comparable agreement with adjudicated labels, with Krippendorff's alpha of 0.606 versus 0.585 for human-human agreement. Using this pipeline, we construct Annotated-llm, a
dataset covering ACL-venue papers from 2018-2025, with 2,667 extracted annotation tasks from 1,603 papers, and find that papers frequently report operational details such as recruitment strategies, annotator expertise, and annotation volume, but often omit details needed to assess annotation validity, including training,
language proficiency, compensation, socio-demographics, adjudication, and agreement values, especially in model-evaluation studies. Our results show that annotation reporting in NLP has improved over time but remains uneven, and they establish a scalable framework and bare-minimum reporting recommendations for making human annotation more reliable, reproducible, and interpretable.
Read the full abstract ▾
All authors Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli, Yanran Chen, Christian Greisinger, Lotta Kiefer, Christoph Leiter, Subhadeep Roy, Tewodros Achamaleh, Muhammad Arslan Manzoor, Sebastian Pohl, Yufang Hou, Steffen Eger
Computation and Language
ResMerge: Residual-based Spectral Merging of Large Language Models
Yandu Sun, Zhiyan Hou, Hongyan An et al. (+6 more)
Model merging offers a training-free way to combine multiple post-trained expert models, but merging experts obtained through reinforcement learning (RL) remains challenging. Existing spectral merging methods often assume that leading singular directions contain the main task signal, while lower-energy residual components can be compressed, selected, or attenuated to reduce interference. We find that this assumption does not hold for RL task vectors: after decomposing each task vector into a leading spectral head and a residual component, both parts can independently recover substantial behavior knowledge, while exhibiting different merging properties. The head is highly concentrated and informative but more prone to sharp cross-expert conflicts, whereas the residual component is more dispersed and provides a more stable basis for aggregation. Based on this observation, we propose ResMerge, a residual-based spectral merging framework for RL experts. ResMerge first constructs a stable residual backbone with Spherical Residual Consensus Adaptation, which estimates a reliability-weighted consensus direction on the Frobenius sphere. It then reintroduces leading-head information through a Lightweight Head Correction module gated by positive cross-expert agreement. Experiments across multiple RL expert groups and capability domains show that ResMerge better preserves expert capabilities than representative task-vector and spectral merging baselines.
Read the full abstract ▾
All authors Yandu Sun, Zhiyan Hou, Hongyan An, Weizhen Wang, Haokai Ma, Yuheng Jia, Junfeng Fang, Haiyun Guo, Jinqiao Wang
Computation and Language
Geometric Latent Reasoning Induces Shorter Generations in LLMs
Shashi Kumar, Yacouba Kaloga, Petr Motlicek et al. (+2 more)
Large
language models solve complex problems by generating lengthy chains of explicit
reasoning tokens. While effective, this makes
reasoning expensive, length-sensitive, and constrained to (discrete) natural
language. While latent
reasoning offers a continuous alternative, determining useful structures for intermediate latent states is an open challenge. In this paper, we formulate latent
reasoning as a
geometric path-approximation problem within the model's pretrained
token-
embedding space. We introduce
Geometric Latent
Reasoning (GLR), which uses a lightweight transition head to predict iterative direction updates in
embedding space. Using textual chain-of-thought traces as anchors, GLR learns to approximate discrete
reasoning trajectories while permitting continuous deviations from exact
token embeddings. Evaluations on mathematical
reasoning benchmarks using Qwen3 models reveal an emergent phenomenon:
geometric latent
reasoning induces substantially shorter generations without an explicit length objective. By replacing early explicit
reasoning with continuous latent steps, models often reach correct answers using substantially fewer total generation steps. These findings suggest that continuous trajectories act as compact intermediate
reasoning states, exposing a new tradeoff between latent computation budget, output length, and accuracy.
Read the full abstract ▾
All authors Shashi Kumar, Yacouba Kaloga, Petr Motlicek, Ina Kodrasi, Andrea Cavallaro
Computation and Language
When Knowledge Is Not Free: Cost-Aware Evidence Selection in Retrieval-Augmented Generation
Mingyan Wu, Han Yang, Omer Ben-Porat et al. (+1 more)
Retrieval-Augmented Generation (RAG) typically assumes that external knowledge is free, but many high-quality sources are paywalled, licensed, restricted, or otherwise costly to access. We introduce cost-aware RAG, a setting where retrieved evidence is assigned access-cost tiers and systems must answer under an explicit evidence-access budget. We instantiate this setting by augmenting MS MARCO v2.1 with access-friction tiers and evaluate budgeted evidence selection across general-domain and domain-specific QA
benchmarks. Our results show that static selection is brittle: no fixed selector uniformly dominates, and larger budgets do not reliably improve answer quality, even when costly evidence is domain-matched. We then study agentic cost-aware RAG, where an LLM decides when to retrieve, which tier to access, and when to stop.
Agents show strong promise as adaptive evidence-acquisition controllers, but their behavior remains highly model- and task-dependent. These findings suggest that cost-aware evidence acquisition is a central challenge for the next generation of RAG systems. All code and data are available at https://github.com/Mignonmy/Cost-Aware.
Read the full abstract ▾
All authors Mingyan Wu, Han Yang, Omer Ben-Porat, Yftah Ziser
Computation and Language · Cross-listed from cs.CR
AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations
Hiskias Dingeto and William Leeney
Indirect
prompt injection in tool-use
agents is a concrete production threat: LLM
agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls. Existing
benchmarks under-measure the threat: most cover only a handful of integrations with the same attack payload replayed across runs, and open-source guards are trained on chat-style data rather than tool-response content. We introduce AGENTREDBENCH, a dynamic LLM-driven redteaming
benchmark of 215 subtle underspecified-authorization scenarios across 24 enterprise integrations and five attack types. Across an eight-model panel (Anthropic, OpenAI, Google), no-guard attack success rate ranges from 32% to 81%. To keep the scenario set out of training corpora and preserve headline ASR meaning over time, we release the codebase, integration schemas, and AGENTREDGUARD model openly; the canonical scenarios are evaluated through a maintainer-mediated channel with immutable versioning. AGENTREDGUARD cuts online attack success by 75-77pp across three target model families (Haiku, GPT-5.4-mini, Gemini-3-flash) at 0.0% real-benign false-positive rate (0.2% on a synthetic-benign corpus), outperforming every open-source baseline with non-trivial detection (Llama Guard, PromptGuard 2, ProtectAI) on both axes. Cross-integration and cross-attacker holdouts (two independent attacker families held out from training) confirm the gain transfers beyond the training subset.
Read the full abstract ▾
Computation and Language
Better with Experience: Self-Evolving LLM Agents for Evidence-Grounded Health Community Notes
Zihang Fu, Fanxiao Li, Jianyang Gu et al. (+5 more)
Large
Language Model (LLM)-augmented Community Notes offer a scalable path for timely, evidence-grounded correction of health misinformation on social platforms. However, they still reset at every post, leaving useful correction experience from prior cases unused. We introduce EvoNote, an agentic framework that enables health Community Notes generation to self-evolve through an evolving experience
memory of prior misinformation correction episodes. Its core is fine-grained credit assignment: EvoNote grounds trajectory-level feedback in health-specific note qualities and distills it into action-level
memory for claim analysis, evidence acquisition, and note writing. We evaluate EvoNote on MM-HealthCN, a 1.2K-instance
multimodal benchmark of user-flagged health posts with human-written Community Notes and crowd-derived helpfulness labels. Under a human-validated hierarchical utility judge, EvoNote-generated notes are preferred over corresponding human-written notes in 89.6% of cases; on a separate set of Needs More Ratings posts without a crowd helpfulness verdict, EvoNote produces helpful notes for 82.0% of cases. It also reduces time to an available candidate correction from multi-hour post-to-note delays to under 2 minutes on average. Analyses link these quality gains to stronger evidence use and reusable correction strategies, positioning self-evolving note generation as a promising paradigm for scalable health misinformation governance.
Read the full abstract ▾
All authors Zihang Fu, Fanxiao Li, Jianyang Gu, Haonan Wang, Preslav Nakov, Bryan Hooi, Min-Yen Kan, Jiaying Wu