Computation and Language · Cross-listed from cs.CV
Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations
Ali Khoramfar, Mohammad Javad Dousti, Alireza Mohamadian et al. (+1 more)
Standard accuracy metrics for VLMs often mask significant reliability failures in sensitive domains. In this work, we utilize a histopathology-validated
brain MRI
dataset to systematically assess the diagnostic robustness of four VLM families under evidence-preserving perturbations. By reordering anatomical slices and swapping target label positions, we evaluate whether models maintain consistent predictions when clinical evidence remains invariant. Our results reveal significant vulnerabilities in presentation-order stability, with models exhibiting prediction flips in up to 48.9% of cases under simple sequence reversals. We further identify a textual selection bias, where label reordering triggers inconsistent diagnoses in up to 67.8% of cases despite identical visual inputs. Negative-control tests further reveal diagnostic overcommitment: models generate categorical diagnoses in up to 76.1% of cases after expert-annotated lesion slices are removed. These results demonstrate that high accuracy can overestimate clinical reliability, masking sensitivity to sequential presentation and textual framing that is not captured by aggregate accuracy. Our findings highlight the necessity of stability-based metrics for the deployment of VLMs in safety-critical clinical applications. Our evaluation data and code will be made public upon acceptance.
Read the full abstract ▾
All authors Ali Khoramfar, Mohammad Javad Dousti, Alireza Mohamadian, Heshaam Faili
Computation and Language
A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination
Wenxiao Zhao, Dong Liu, Kaiyi Xu et al. (+9 more)
Symbolic regression aims to discover closed-form equations from data, but existing LLM-guided methods often rely on a unified proposal loop that compresses heterogeneous search failures into a scalar score and a single
prompt. We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views. A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process
memory. During search, evaluator feedback characterizes reliability and productivity, updates role-level utilities, and routes elite motifs, failure traces, and validity diagnostics to different
agents. The framework self-evolves at two timescales: within a run, it adapts the search process without updating LLM parameters; across runs, recorded trajectories can be distilled into open-source LLMs as role-conditioned proposal priors. Averaged over the four LSR-Synth scientific domains in LLM-SRBench, A-SR improves Acc@0.01 over baselines from 25.79% to 48.30% with Llama3.1-8B, while A-SR-LoRA improves the corresponding Qwen3-4B result from 24.58% to 38.29%. On four real-world scientific discovery tasks, A-SR obtains the best in-distribution or out-of-distribution normalized mean squared error on 7 of 8 reported metrics.
Read the full abstract ▾
All authors Wenxiao Zhao, Dong Liu, Kaiyi Xu, Feng Liu, Zhen Zhao, Fei Ben, Shu Wang, Wenhao Li, Ying Nian Wu, Fenghua Ling, Haobo Li, Lei Bai
Computation and Language
Preverbal Uninflected and Underived Roots in Mapudungun. Wuno and Its Implications
Andres Chandia
This study examines the grammatical status of preverbal uninflected and underived roots in Mapudungun, with particular focus on wuno 'return/re-'. Through a critical review of scholarly classifications–auxiliaries (Smeets, 2008), modal prefixes (Longkon, 2011), and preverbal particles/complex verb stems (Zúñiga, 2006)–we demonstrate the limitations of existing frameworks. A diachronic corpus analysis spanning four centuries (1606-present) reveals that these elements exhibit three distinct profiles: stable V1 compounds (kim, shinge), volatile V1 rates reflecting orthographic shift (pepi, wuno), and a true particle (kalli). The discovery of V2 attestations for kim and kupa confirms their status as full lexical verbs. We propose a prosodic-orthographic hypothesis: apparent "variable binding" results from the fossilization of prosodic pauses transcribed by early missionaries as spaces, a convention later reanalysed by speakers as syntactic boundaries. The evidence supports Zúñiga's radical concatenation as the correct grammatical model, with implications for the study of
languages with no pre-contact written tradition.
Read the full abstract ▾
Computation and Language
Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content
Arianna Denitto and Beatrice Savoldi
Despite its cultural relevance and
diffusion, queer slang remains underrepresented in Natural
Language Processing research. Towards addressing this gap, we introduce Slang-Q, a manually curated
dataset of naturally user-generated English sentences paired with queer slang terms and reference definitions, built upon a newly constructed taxonomy of 118 queer terms. We use this resource to conduct a first exploratory evaluation of
language models on their ability to understand and define queer slang under varying prompting conditions. Slang-Q is intended as a basis for studying how current models handle sensitive, community-specific
language and whether they can provide accurate and reliable information about such forms of identity and linguistic expression.
Read the full abstract ▾
Computation and Language
Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?
Jinyi Han, Yuanjian Xu, Ying Liao et al. (+6 more)
Large
language model (LLM)
agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. Existing evaluations mostly judge the quality of a skill or its contribution to task success, leaving unexamined whether an
agent can recognize a relevant skill and apply it on its own. We introduce Skill-Use, a
benchmark that evaluates skill use under progressive disclosure, where an
agent sees only a skill's name and short description and must retrieve the full procedure before following it. Skill-Use separates three facets of skill use. Trigger measures whether the
agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations. A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered. Skill-Use pairs 79 real skills with 177 executable tasks across nine domains, each grounded in real files, run in an isolated Docker sandbox, and scored by a trajectory-based rubric. Evaluating eight LLMs under two
agent harnesses, we find that reliable skill use remains out of reach, as the strongest configuration reaches an SU of only 0.613. Triggering and procedural compliance fail as independent bottlenecks, and both scores and model rankings shift with the harness, so skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.
Read the full abstract ▾
All authors Jinyi Han, Yuanjian Xu, Ying Liao, Xinyi Wang, Zishang Jiang, Zixiang Di, Fanyang Lu, Zhichao Hu, Yanghua Xiao
Computation and Language
A Modular Part-of-Speech Tagger for Scottish Gaelic using spaCy
Peter Stefan, Peter J Barclay and Alistair Lawson
Part-of-
speech tagging for low-resource
languages remains challenging due to limited annotated data, especially for linguistically complex
languages. Gaidhlig (Scottish Gaelic) is a morphologically rich and endangered
language with limited digital resources, making it suitable for examining a lightweight
language processing approach. This paper describes using the modular spaCy Natural
Language Processing framework to build part-of-
speech taggers for Gaidhlig using the Annotated Reference Corpus of Scottish Gaelic. We train two models with minimal pre-processing and configuration: one using a fine-grained tagset and another using a reduced coarse-grained tagset. Both models are trained without external
embeddings or pre-trained
language models, using only supervised learning from the available corpus. The fine-grained model achieves 88.6% tagging accuracy, while the coarse-grained model achieves 93.7%. The results are comparable to those of the two previously published Gaidhlig taggers, indicating that simple, off-the-shelf
language processing pipelines can demonstrate good performance in low-resource and morphologically complex linguistic settings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation
Yi Yang, Cong Qin, Xiaodan Liu et al. (+8 more)
Large
language model
agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual
tokens should be updated. On-Policy Self-
Distillation (OPSD) addresses this by re-scoring generated
tokens under a privileged replay view to obtain dense,
token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs
token scores. To resolve this confounding, we propose Observation-Calibrated Self-
Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate
token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at https://github.com/yiy1x/OCSD.
Read the full abstract ▾
All authors Yi Yang, Cong Qin, Xiaodan Liu, Chishui Chen, Qing Dong, Yan Zhang, Cao Liu, Zhao Yang, Lu Pan, Jiaye Lin, Yi Feng
Computation and Language
Reachability in 3-VAS
Łukasz Kamiński and Sławomir Lasota
We settle the exact complexity of the reachability problem in (stateless) vector addition systems (VAS) in fixed low dimension. In dimensions 2-4 it has only been known to be sandwiched between NP and PSPACE. We prove PSPACE-hardness of the reachability problem for symmetric vector addition systems in dimension 3 (3-VAS), a restricted fragment of general 3-VAS. Combined with previously established PSPACE upper bounds, our result settles the complexity of the problem to be PSPACE-complete in 3-VAS and 4-VAS, as well as in their symmetric fragments.
Read the full abstract ▾
Computation and Language
Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent
Chenyu Wang, Yi Liu, Baoqing Li et al. (+2 more)
Scaling supervision for multi-turn medical
agents is difficult because expert dialogue annotation is costly and clinical conversations are
privacy-restricted. We introduce Guideline-as-Oracle (GAO), which compiles American Academy of Ophthalmology guidance into a 70-row operational rule table and uses it as the sole source of instance-level supervision for 3,000 training dialogues, reserving human labeling for evaluation. Because converting rules into dialogues is itself a design problem, we catalog eight construction strategies, including cited-row tier assignment, one-fact boundary pairs, metadata-only repair, and label repair, and characterize the evidential status of each: labeling mechanism, null, confounded, or evaluated only as a package. Fine-tuning a 9B backbone on this corpus yields GAO-Triage, improving agreement with a 201-case operational reference from 61.7% to 74.1% (exact McNemar p=0.0046) and emergent-case recall from 9.5% to 69.0%; the gains persist across a second seed and patient simulator. None of the seven general-purpose systems we test dominates GAO-Triage on both metrics, and GAO-Triage requires no frontier model at inference time. Permuting label-dialogue assignments collapses the model to a constant-routine predictor, indicating that the signal lies in guideline-derived assignment rather than dialogue surface form. Label repair coincides with the disappearance of a late-training safety degradation.
Read the full abstract ▾
All authors Chenyu Wang, Yi Liu, Baoqing Li, Min Tu, Diping Song
Computation and Language
InsightEmb: Learning Action-Intent Embeddings for Agentic Insight Retrieval
Tsz Ting Chung, Jiangnan Li, Jie Zhou et al. (+1 more)
Self-improving
agents accumulate reusable insights from prior trajectories, making
retrieval increasingly important for turning accumulated experience into actionable guidance. At each decision step, retrieving the right insight can help the
agent progress toward its goal, a setting we refer to as agentic insight
retrieval. However, existing
retrieval methods primarily model semantic similarity, while overlooking whether a retrieved insight resolves the
agent's current decision bottleneck. We propose InsightEmb, a contrastive
embedding framework that learns transferable progress-oriented
retrieval geometry using only mathematical
reasoning data. InsightEmb jointly learns to align concrete situations with abstract heuristic rules and to cluster
reasoning trajectories with similar progress structures. We evaluate InsightEmb on dynamic
agent tasks and a static skill-
retrieval benchmark. Without any environment-specific training, InsightEmb improves over all these evaluations, surpassing the performance of existing
reasoning embedding models. These results suggest that the geometry of state-insight matching can transfer across domains, enabling effective training from publicly available
reasoning data without expensive environment-specific supervision.
Read the full abstract ▾
All authors Tsz Ting Chung, Jiangnan Li, Jie Zhou, Mo Yu
Computation and Language · Cross-listed from cs.CV
Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs
Yang Yang, Jiawei Chen, Tairan Chen et al. (+1 more)
Although
Multimodal Large
Language Models (MLLMs) have made substantial progress, their spatial
reasoning may still produce intermediate judgments inconsistent with the input image, allowing errors to propagate through the
reasoning chain and affect the final answer. Existing methods mainly improve spatial
reasoning through training or additional spatial information, without considering whether the
reasoning process itself is faithful to the model input. Our study shows that unfaithful
reasoning chains significantly reduce final-answer accuracy. To address this issue, we propose a modular and training-free framework for spatial
reasoning verification and correction. The framework constructs a Spatial Evidence Graph (SEG), which associates atomic spatial evidence extracted from Chain-of-Thought
reasoning with visual entities, spatial relations, source steps, and visual evidence. Spatial Evidence Reliability Assessment (SERA) evaluates the reliability of visual evidence based on object existence, localization, and
geometric measurements. The framework then identifies the earliest spatial evidence unit contradicted by reliable visual evidence and guides the original MLLM to revise the subsequent
reasoning and final answer. Across 15 model-
dataset settings, our method achieves an average accuracy of 68.94%, outperforming the compared baselines by 8.55 percentage points on average. Our code will be open-sourced.
Read the full abstract ▾
All authors Yang Yang, Jiawei Chen, Tairan Chen, Zhaoxia Yin
Computation and Language · Cross-listed from cs.CV
Simile Understanding in Text-to-Image Models: An Evaluation Framework
Luecheng Wang, Shintaro Ozaki, Hidetaka Kamigaito et al. (+4 more)
Similes provide a compact and expressive way to describe visual characteristics in text
prompts. Recent text-to-image models (t2i models) can produce visually compelling outputs from simile
prompts, yet even frontier models frequently misinterpret the metaphorical vehicle and confuse it with the object. These systematic failures reveal a gap between figurative
language and object-level visual grounding in t2i models. To investigate this issue, we propose a scalable evaluation framework for simile understanding. Our framework includes (1) a controlled simile
dataset in which metaphorical vehicles are drawn from a predefined set of object-detectable categories and combined with diverse templates, (2) automatic grounding metrics based on YOLO (You Only Look Once) detection, and (3) text encoder layer analysis using
Diffusion Lens to track how metaphorical vehicles emerge during generation. Experiments across architecturally diverse t2i models reveal consistent literalization failure patterns. We further discuss potential mitigation strategies for improving simile grounding in t2i models.
Read the full abstract ▾
All authors Luecheng Wang, Shintaro Ozaki, Hidetaka Kamigaito, Katsuhiko Hayashi, Jingun Kwon, Manabu Okumura, Taro Watanabe
Computation and Language
Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems
Kartikey Singh Bhandari, Aarya Wadhwani, Dhruv Kumar et al. (+1 more)
LLM
agents that persist across sessions accumulate stored
memories whose validity varies enormously by content type, yet existing
memory architectures treat all
memories as equally persistent and systematically contaminate retrieved context with outdated facts. We show that per-
memory, type-conditioned temporal decay, a property of western scrub jay episodic
memory, can be operationalized as an auto-classified coefficient $\pi_i$ in an external LLM-
agent memory store, yielding ScrubJay-MEM: each
memory is encoded as a jointly-bound What–Where–When tuple with an estimated perishability $\pi_i$ and utility horizon $\tau_i$, retrieved by query-adaptive scoring, and revised retroactively at $O(1)$ LLM calls per update. We introduce the Temporal Generalization Test (TGT), a
benchmark with held-out retention intervals and a Generalization Gap (GenGap) metric. On TGT, ScrubJay-MEM is the only
retrieval-based system with substantially positive GenGap ($+0.108$); on MemoryAgentBench EventQA-64k it improves F1 by $+2.66$ over Mem0 and $+3.09$ over Qwen3-
Embedding-4B under a llm backbone. A decay ablation collapses GenGap by $5.7\times$, establishing type-conditioned decay as necessary for the result. Gains narrow under stronger backbones and reverse on fact-consolidation tasks, scoping the contribution to temporal
reasoning over perishable facts.
Read the full abstract ▾
All authors Kartikey Singh Bhandari, Aarya Wadhwani, Dhruv Kumar, Pratik Narang
Computation and Language
EmpaAva: An Open-source Agentic 3D-Avatar Empathetic Live Chatbot
Jie Yang, Wenhao Xu, Shuhui Lin et al. (+1 more)
This paper presents EmpaAva, to our knowledge the first open-source, agentic 3D-avatar empathetic chatbot, which carries empathetic response generation (ERG) from text-only exchanges into live, face-to-face interaction. Through a
video-call-like interface, a user speaks to a 3D digital human that reads their affect from
speech and optional
vision, and replies with emotional
speech, lip-synced facial motion, and photorealistic 3D Gaussian rendering. At its core, an LLM coordinates a Tri-
Agent Architecture, in which perception, empathetic response planning, and
embodied rendering form a closed loop, paired with a Response Planning layer that compiles each reply into an executable
multimodal plan, keeping voice, expression, and rendering on one empathetic intent. Building on strong open-source modules, EmpaAva supplies the intelligence that binds them into one controllable, inspectable experience. In automatic and human evaluations, EmpaAva surpasses text-only, 2D talking-face, and
multimodal avatar baselines in
emotion understanding, response quality, and audio-visual consistency. We open-source EmpaAva with an online live demo.
Read the full abstract ▾
All authors Jie Yang, Wenhao Xu, Shuhui Lin, Hao Fei
Computation and Language
IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)
Shahd Gaben, Heba Sbahi, Samer Rashwani et al. (+5 more)
Large
language models (LLMs) are increasingly used for question answering, education, and research, including in religious and cultural domains where answers depend on specialised source traditions. Yet in Islamic Studies, key concepts, methods, and debates preserved in the authoritative scholarly tradition, known as turath, lack high-quality annotated resources. We introduce IslamicTurathBench (ISTB), a multi-task, multi-discipline
dataset for evaluating LLMs on classical Islamic scholarship. Developed and reviewed by domain experts, ISTB contains 3,465 question-answer items drawn from 35 recognised source works spanning more than 12 centuries of scholarship across seven key fields of Islamic Studies. To enable comprehensive profiling of model capabilities, ISTB is structured along two axes: scholarly demand (Beginner, Intermediate, and Advanced) and task format (multiple-choice questions, passage-based comprehension, and open-ended knowledge questions). ISTB includes aggregated scores from a scholarly human reference panel and zero-shot baselines from ten systems. The
dataset supports reproducible evaluation of
language-model behaviour across source works, disciplines, scholarly-demand levels, and question formats in a historically layered scholarly domain.
Read the full abstract ▾
All authors Shahd Gaben, Heba Sbahi, Samer Rashwani, Abdessalam Bouchekif, Mutaz Al-Khatib, Emad Mohamed, Somaya Eltanbouly, Mohammed Ghaly
Computation and Language
Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention
George Fountzoulas
Papers 1-2 of the Kathleen series showed that a byte-level,
attention-free architecture built from a wavetable encoder and multi-scale reverberant state can match strong baselines on classification at ~450-700K parameters, without pretraining. We ask whether the same ingredients can generate. (1) Scaling: on byte-level
language modeling (WikiText-103, raw UTF-8, no tokenizer), the reverberant model beats a parameter-matched
transformer at every
dataset scale measured (2-512 MB), e.g. 1.84 vs 2.04 bits/byte at 512 MB with ~0.5M parameters; the
transformer needs more than 512 MB to match what the
attention-free model learns from 32 MB. (2) Measurement: we introduce FORM DISTANCE, a non-parametric, gaming-resistant instrument for "reads like text": nine statistical axes of human text define a reference cloud, and five constructed fakes are all rejected. (3) Generation: decoding policy dominates architecture – widening the sampler halves the same model's distance (3.17 to 1.52), and a
retrieval-augmented decoding scheme takes the frozen model further (1.52 to 1.14) with no training step involved; the ablation attributes the gain to the sparse phrase dose itself, not the selection gate. The gain has a sharp boundary condition: the phrases must come from the model's own training corpus – a 40x larger foreign library helps not at all, an effect the
attention twin shares, consistent with in-context integration being a capability of scale. We also report four architectural additions that did not help, and a computed lexicon reaching 94% of a learned table's top-1 accuracy at one fifth of the parameters. Everything runs offline; all experiments are reproducible on a free Kaggle T4.
Read the full abstract ▾
Computation and Language
Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark
Enrico Mensa, Lorenzo Zane, Calogero Jerik Scozzaro et al. (+3 more)
Large
Language Models (LLMs) have transformed computational linguistics and achieved remarkable performance across numerous natural
language processing tasks, yet significant gaps persist in understanding how these systems process culturally embedded linguistic expressions. This paper introduces ProverbIT, a novel Italian
benchmark comprising 100 multiple-choice questions designed to evaluate LLMs' ability to complete Italian proverbs. We assess 13 frontier models, including Large
Reasoning Models (LRMs) and traditional LLMs, across three tasks: proverb completion, multiple-choice selection with correct answers, and multiple-choice selection without correct answers. Our evaluation reveals surprising results: while nearly all models demonstrate knowledge of the proverbs through successful completion tasks, performance drops dramatically when transitioning to multiple-choice formats without correct answers, with even state-of-the-art
reasoning models showing substantial degradation. Through detailed Chain-of-Thought analysis of two LRMs, we uncover that models exhibit a strong bias toward selecting literal synonyms and frequently mention correct proverb endings during
reasoning without successfully identifying their absence from the given options. These findings suggest that current LLMs rely heavily on memorized patterns rather than deeper semantic understanding of culturally grounded expressions, highlighting important limitations in their
reasoning capabilities for figurative
language comprehension.
Read the full abstract ▾
All authors Enrico Mensa, Lorenzo Zane, Calogero Jerik Scozzaro, Matteo Delsanto, Tommaso Milani, Daniele Paolo Radicioni
Computation and Language
Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
Ian B. de Haan, Peter van der Putten and Max van Duijn
Large
language models (LLMs) have recently shown strong performance on Theory of Mind (ToM) tests, prompting debate about the nature and validity of the underlying capabilities. At the same time,
reasoning-oriented LLMs trained via reinforcement learning with verifiable rewards have demonstrated notable improvements across a range of
benchmarks. In this work, we examine the behavior of such
reasoning models in ToM tasks using novel adaptations of machine psychological experiments together with results from established
benchmarks. We observe that
reasoning models consistently exhibit increased robustness to
prompt variations and task perturbations. Our analysis suggests these gains come at least partly from models being more robust at reaching the correct answer under
prompt and task variation. We read this as evidence for a robustness-based account rather than for a new ToM-specific ability.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
AI Literacy for Legal Translation: Developing Digital Resilience
Łucja Biel
Generative AI is transforming legal translation by introducing opportunities alongside linguistic, technical, legal, ethical and cognitive risks. This chapter examines the implications of AI for professional legal translation and proposes an AI literacy framework tailored to the profession. It argues that AI does not change the fundamental objectives of legal translation but requires an extension of professional competence through AI literacy. The proposed framework comprises four mutually reinforcing dimensions, foundational, procedural, critical and strategic, and conceptualises AI literacy as a transversal component of legal translation competence that fosters digital resilience. It further discusses the pedagogical implications of this framework by proposing classroom activities designed to develop AI literacy in legal translator education, enabling future translators to integrate AI critically, responsibly and in accordance with professional standards.
Read the full abstract ▾
Computation and Language
When Absence Is Evidence: Evaluating Completeness-Sensitive Negative Reasoning in Large Language Models
Byoungjae Min, Kennedy Edemacu, Sae-Hong Cho et al. (+3 more)
Large
language models (LLMs) are often asked whether something is absent from a record, list, or retrieved context. Yet non-observation licenses a negative answer only when evidence completely covers the query scope; otherwise, the answer should remain unknown. We call this completeness-sensitive negative
reasoning. We introduce CROWN-QA, comprising CROWN-Synth, a controlled paired core that fixes the question and observed facts while varying only query-relative coverage, and CROWN-Real, a real-document contrast-set evaluation with controlled coverage variants. Across three LLM families, models show unstable closure judgments and substantial over-closure, failing to reliably distinguish a justified negative answer (Certified-Negative) from insufficient evidence (Unknown). The dominant CROWN-Synth failure is asymmetric: models often recognize implicitly complete evidence yet treat implicitly partial evidence as query-covering. Prompting redistributes errors between over- and under-closure rather than consistently resolving them. Structured certificate elicitation traces many errors to evidence-coverage mischaracterization. CROWN-Real shows that the core partial-coverage asymmetry persists on real-document content, while its strength and the balance between over- and under-closure vary by model,
prompt, and source.
Read the full abstract ▾
All authors Byoungjae Min, Kennedy Edemacu, Sae-Hong Cho, Yoonhyuk Choi, Beakcheol Jang, Jong Wook Kim
Computation and Language
E$^3$-Orch: Towards Effective, Efficient, and Extensible Agentic Orchestration with Reinforcement Learning
Junnan Liu, Linhao Luo, Thuy-Trang Vu et al. (+1 more)
Agentic orchestration enables multiple autonomous
agents to solve complex tasks through adaptive decomposition, delegation, and execution. However, existing orchestrators often rely on hand-crafted logic and prompting strategies, limiting adaptation and generalization across tasks and executor configurations. We propose E$^3$-Orch, a reinforcement learning framework for effective, efficient, and extensible agentic orchestration based on a milestone-plan-act workflow. Instead of planning all subtasks upfront, E$^3$-Orch organizes execution around milestones, a mid-level abstraction that scopes planning around meaningful intermediate objectives and allows orchestration decisions to adapt as execution progresses. For each milestone, the orchestrator builds a dependency-aware plan, assigns subtasks to suitable executors, and executes independent subtasks in parallel. We train the orchestration policy from execution feedback, using milestone and plan decisions as units for fine-grained credit assignment. Tree-structured rollouts compare alternative decisions under shared execution histories, while complementary rewards optimize task performance, execution cost, and planning completeness, including an uncertainty-aware performance reward for
stochastic downstream outcomes. Across seven
benchmarks, E$^3$-Orch achieves the best task performance under multiple executor configurations, improving over the strongest baselines by $0.7$–$3.8$ points and delivering $1.16$–$1.59\times$ higher intelligence efficiency. The learned policy also transfers to unseen executor configurations introduced only at evaluation time, supporting extensible agentic orchestration.
Read the full abstract ▾
All authors Junnan Liu, Linhao Luo, Thuy-Trang Vu, Gholamreza Haffari
Computation and Language
Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
Yexing Du, Kaiyuan Liu, Youcheng Pan et al. (+4 more)
Multimodal large
language models (MLLMs) have achieved significant success in
speech-to-text translation (S2TT). However, when processing
multilingual speech inputs, a single
speech encoder shared across all
languages suffers from the curse of multilinguality:
languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource
speech. To address this problem and improve
multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of
Speech Encoders (MoSE). MoSE uses an explicit
language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource
language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource
languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per
language for effective
alignment. We conduct extensive experiments on 45
languages, systematically evaluating all $45 \times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource
languages simultaneously, with the largest gains on low-resource
speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future
multilingual S2TT research, we release our code and models.
Read the full abstract ▾
All authors Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Chengpeng Fu, Yu Wang, Ming Liu
Computation and Language
Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study
Yuanjun Zhang and Mourad Oussalah
Humanitarian reports are long, noisy, and multi-topic, making it difficult to consolidate decision-relevant causal evidence. We present a ReliefWeb study (2000-2024) and a two-stage Large
Language Model (LLM) pipeline that extracts structured intervention-outcome records with direction and strength attributes. Query-conditioned extraction restricts output to a specified intervention class, reducing
retrieval-induced over-extraction, while snippet grounding links each relation to supporting text for auditability and classification. In an expert-annotated
dataset of 100 reports, the best closed-source LLM achieved a weighted F1 score of 90.73% with strong cost-efficiency, while Llama-3.1-8B with supervised fine-tuning reached 94.15% weighted F1 score. We further propose context-preserving triangulation that aggregates strength-weighted evidence within disaster$\times$source
cells, applies Laplace smoothing and equally weights
cells to quantify cross-context convergence via a Level-of-Evidence score. Applied to cash assistance, food-related outcomes show strong positive convergence (LoE=0.865) and stable long-horizon trajectories.
Read the full abstract ▾
Computation and Language
When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents
Yushi Sun and Yanjie Zhang
Memory-augmented VLM
agents act on persistent spatial knowledge, yet that knowledge silently goes stale as the environment changes. We ask what happens when an
agent must reconcile a confident
memory claim with a contradicting observation, and whether current models can catch the conflict before it becomes a safety-relevant mistake. Using a dynamic FrozenLake testbed, we pair a staleness-detection task with a downstream
navigation task across three closed-source models and three open-weight VLMs under both text and image inputs (1,800 detection runs, and 12,000 text-mode
navigation episodes over four LLM navigators at a shared 50-seed scale). Three findings emerge. First, text solvability does not imply visual grounding: models that flag stale entries reliably from text nonetheless span
vision F1 from 0.887 down to 0.067 on the identical grids, and the weakest keeps making fluent, confident decisions that ignore the image. Second, consuming stale
memory without an audit is a safety liability: in our primary GPT-4o setting, an
agent that trusts raw
memory dies more than twice as often as the same
agent given no
memory at all. Third,
auditing helps but does not close the gap: a transparent read-time filter removes much of the safety cost in text mode, yet even oracle stale labels bring no further significant gain on the current grid size, and when visual
auditing is unreliable, filtering yields no consistent benefit. Together these results frame spatial-
memory staleness as a safety failure mode and isolate reliable visual grounding and action selection under
memory–observation conflict as the central open challenges for
memory-augmented
agents.
Read the full abstract ▾
Computation and Language
The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
Yushi Sun, Yanjie Zhang and Rui Sheng
Personalized LLMs with persistent
memory are increasingly deployed, yet the faithfulness of their user models remains unexamined. We study over-inference (OI): the phenomenon where LLMs fabricate user attributes beyond what evidence supports. We introduce MirageBench, comprising 150 personas balanced across stereotypical, counter-stereotypical, and neutral profiles, 6 personalization tasks spanning an “imagination gradient”, a four-way faithfulness taxonomy operationalized by an independent judge (validated against a blind human annotator on 400 claims: Cohen's kappa = 0.863 four-class, kappa = 0.900 binary), and a leaderboard of 12 models across 7 families on 143616 judged claims. We find that over-inference is pervasive: every one of the 12 models over-infers 35%–49% of its claims (cross-model mean 41.6%; claim-weighted 41.8%), with no model in this evaluation escaping it. Most strikingly, we surface a Self-Monitoring Inversion: at the model-selection level, models' self-assessed OI is negatively rank-correlated with their judge-measured OI (rho = -0.60, p = 0.044; exploratory, wide bootstrap CI [-0.90, +0.06], n = 12). The models that report the least over-inference tend to be flagged as fabricating the most, so self-reported confidence is a misleading signal for comparing models, even though within a single model self-audit still ranks that model's own claims moderately well (AUROC 0.58–0.83). We further show that OI is task-dependent (27%–59%) and that, in a multi-turn pilot, inferred attributes accumulate approximately linearly with little revision. MirageBench positions external verification, rather than model self-report, as a more reliable foundation for trustworthy personalization.
Read the full abstract ▾
Computation and Language
Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression
Zhengpei Hu, Kai Li, Dapeng Fu et al. (+5 more)
Hard
prompt compression reduces long-context inference cost by independently scoring
tokens, sentences, or chunks and retaining the highest-scoring units under a budget. We identify a structural failure in this procedure: independent selection can split dependent evidence pairs, retaining one member while deleting the other. When retained text contains an answer but deleted text defines the entity needed to interpret it, we call the result referential dangling. At a compression ratio of 0.30, Beaver, which ranks coherent chunks using Qwen3-0.6B
embeddings, leaves the answer path incomplete in 34-54% of bridge examples across three multi-hop question answering
datasets. On a shared HotpotQA bridge set, all six hard compressors we test exhibit dangling at rates up to 60%, and every document in LongBench-v2 Single-Document QA contains at least one dangling reference. On dangling examples evaluated with Qwen3-8B, reinserting the missing supporting paragraph while removing nonsupporting paragraphs to maintain the
token budget improves accuracy by 29-34 percentage points (p < 0.0001), recovering at least 88% of the gap to contexts retaining both supporting paragraphs. Stronger answer models do not absorb the loss: on MuSiQue, GPT-5.5 is 8.8 points less accurate on compressed contexts than on contexts retaining both supporting paragraphs. Finally, we train a compact classifier to rank omitted sentences by whether they are needed to interpret retained text and reinsert the top-ranked candidates without support annotations at inference. On HotpotQA with Qwen3-8B, this automatic restoration improves accuracy by 4.7 points while changing the compression ratio only from 0.30 to 0.31. Hard compressors should optimize both relevance and referential completeness.
Read the full abstract ▾
All authors Zhengpei Hu, Kai Li, Dapeng Fu, Xuechao Zou, Yuanhao Tang, Yue Li, Tengfei Cao, Jianqiang Huang
Computation and Language
STRIVE: Probing Reasoning Limits in Graded Plausibility Generation and Evaluation
Bhiman Kumar Baghel, Anna Chrabaszcz, Tessa Warren et al. (+3 more)
Event knowledge concerns who does what to whom. Psycholinguists use event-plausibility judgments to examine how this knowledge supports human
language processing. To isolate plausibility effects, these studies require controlled event sets in which one event slot varies across plausibility levels while all other event features remain fixed. Constructing such sets manually is labor-intensive. We therefore introduce STRIVE, an LLM-based framework for jointly generating and evaluating controlled event sets crossing plausibility class (plausible vs. implausible) with intended classification difficulty (easy vs. hard). Given a verb, STRIVE constructs a shared event frame, then produces one event per condition by varying one slot while holding all others fixed. In experiments with six models across 60 verbs, GPT-5.1 produced high-quality sets only 16.7% of the time using the baseline generation
prompt. Adding a global
reasoning scratchpad and evaluator-guided refinement raised this rate to 75.0%. Greater
reasoning effort also improved evaluator–human agreement. Nevertheless, events near the plausibility boundary remain most difficult. They elicit the greatest human disagreement, and the best evaluator reaches only 57% accuracy on the implausible-hard condition, indicating a need for human input. Overall, STRIVE offers a scalable approach to reducing manual effort by automating initial event-set generation and evaluation for psycholinguistic studies.
Read the full abstract ▾
All authors Bhiman Kumar Baghel, Anna Chrabaszcz, Tessa Warren, Michael Walsh Dickey, Haley C. Dresang, Xiang Lorraine Li
Computation and Language · Cross-listed from cs.CR
Breadcrumbing Search Agents
Xuebin Li, Hanqing Zhao, Siyuan Liang et al. (+4 more)
LLM-based search
agents are widely used for information-seeking tasks, but their reliance on external tool returns introduces a critical security risk: web content retrieved during execution is untrusted, exposing
agents to
prompt injection and goal hijacking. Prior work on search-
agent safety primarily focuses on static web-content injection, but modern
agents issue follow-up queries and cross-check competing sources, so a single injected page is often diluted or rejected. We show that the channel delivering search and page observations is a fragile security boundary: beyond exposing the
agent to a single poisoned page, a mediated search interface can repeatedly steer how the
agent gathers evidence and forms its final answer. Under a constrained tool-intermediary threat model, appending only one controlled result per query can substantially increase attack success when the evidence is coordinated across the
agent's trajectory. We study this setting with a strategy-driven long-horizon attack system and introduce Authority-Chain Hijack (ACH), an expert-refined strategy that turns isolated search-result and page-content manipulations into a coherent evidence chain across seemingly corroborating sources. ACH achieves the highest Overall ASR among all baselines, reaching 55.9% / 83.3% ASR / MaxN ASR on the full SafeSearch test split. We further introduce Trace-Guided Strategy
Evolution (TGSE), which automatically improves attacker strategies from execution traces, replacing manual redesign with trace-driven refinement; its strongest single setting reaches 71.4% / 95.0% in held-out evaluation.
Read the full abstract ▾
All authors Xuebin Li, Hanqing Zhao, Siyuan Liang, Kejiang Chen, Weiming Zhang, Dacheng Tao, Nenghai Yu
Computation and Language
Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling
Han Chen, Ming Li, Hong Jiao et al. (+1 more)
Predicting item difficulty from content can provide an initial estimate for newly developed questions before sufficient student responses are available. Existing approaches typically represent the question stem and answer choices as text. When mathematics items contain visual components, a common pipeline first textualizes that evidence and then applies a text predictor. We ask: how should visual evidence be represented for item difficulty prediction? We compare question text alone, visual textualization, which expresses visual evidence in
language, and image-native modeling, which retains the original image. Using Eedi items with difficulty calibrated from student responses, we train large
language models (LLMs) and
vision-
language models (VLMs) directly for difficulty regression. Both visual interfaces achieve the lowest point estimates, although the leading systems cannot be reliably ordered. Open-VLM textualization yields lower RMSE point estimates for all evaluated LLMs, while broader adaptation does so for all image-native VLMs. Test-time interventions show dependence on the paired full-item image, but do not isolate the additional visual component. The two visual interfaces also make partially complementary item-level errors and differ substantially in computational workflow. Thus, textualization should not be treated as the only practical interface: image-native modeling is a competitive alternative whose effectiveness depends on how the VLM is adapted.
Read the full abstract ▾
All authors Han Chen, Ming Li, Hong Jiao, Tianyi Zhou
Computation and Language
Relational Response Fields: A General Theory of Black-Box LLM Response Consistency and Recovery
Song Zichen
Black-box
language-model reliability is commonly pursued by sampling, prompting, voting, verifying, or iteratively revising individual answers. We ask a prior question: what determines whether a collection of black-box responses is recoverable at all? We represent responses to typed transformations of a query as a relational response field (RRF). Edge transports encode how valid responses must change under paraphrase, scaling, decomposition, refactoring, or other task
symmetries; anchors encode independently trusted evidence such as execution or a verifier. For relation operator $D$, anchor operator $A$, and at most $k$ corrupted response nodes, we identify $\gamma_k(D,A)$ as the intrinsic difficulty of black-box response recovery. It is positive exactly when every $k$-node corruption is identifiable; it gives a deterministic stability bound proportional to $1/\gamma_k$; and a matching two-point minimax lower bound shows that no
estimator can improve this dependence. Thus consistency is not truth: relation-only methods are blind to null directions, including shared
hallucinations. We derive sparse field-repair
algorithms while separating information-theoretic identifiability from the stronger null-space conditions required by convex optimization. Controlled
theorem tests and black-box mathematics/code experiments evaluate four theory-fixed consequences: consistency–truth separation, anchor phase transitions, redundancy saturation, and cross-model, cross-task prediction of repair difficulty. The results support $\gamma_k(D,A)$ as a measurable property of a response-recovery instance, rather than a score attached to one repair heuristic.
Read the full abstract ▾