Computation and Language
SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?
Jinshan Gao, Zhuoran Jin, Tianyi Men et al. (+2 more)
Large
language model-based multi-
agent systems are evolving from fixed interaction
topologies toward dynamically orchestrated
Agent Swarms. However, existing
benchmarks are still largely based on single-
agent or general-purpose
agent tasks, making it difficult to systematically evaluate key orchestration capabilities. We propose SwarmBench, a
benchmark that evaluates model performance from multiple perspectives, including accuracy, efficiency, cost, and process quality. Experimental results show that current models exhibit substantial differences in orchestration capability. These differences are reflected not only in final accuracy, efficiency, and cost, but also in the overall quality of the orchestration process itself. Based on these findings, we further propose SwarmExp, a simple yet effective method based on experience extraction and experience replay, which consistently improves the orchestration performance of large
language models.
Read the full abstract ▾
All authors Jinshan Gao, Zhuoran Jin, Tianyi Men, Kang Liu, Jun Zhao
Computation and Language · Cross-listed from cs.AI
Geometry of Divergence: Tracking Hidden-State Trajectories for Adaptive Multi-Turn Reasoning
Jie Liang, Zhengxin Yu, Hamid Nasiri et al. (+1 more)
LLM
agents need to sustain goal-consistent
reasoning across long multi-turn interactions under strict resource constraints. However, as the multi-turn context accumulates, it can destabilize the underlying LLM's internal representation of task-relevant information from earlier turns, blurring the boundary between constructive
reasoning and representation drift. We formulate multi-turn
reasoning as a hidden-state trajectory of the underlying LLM that is characterized via two complementary signals: temporal
curvature that captures the directional consistency of turn-to-turn updates, and variance slope which measures the expansion or contraction of the exploration space. Across four tasks and three underlying LLMs, we observed that these
geometric signals distinguish between correct and incorrect episodes prior to completion. We further decompose each episode into three-action chains formed from four actions (Read, Write, Respond, Transfer) and show that separability is action-dependent, with different signals distinguishing various chain patterns. Our experiments demonstrate that trajectory geometry can identify critical turns in the
reasoning process, increasing task success rates on $\tau$-Bench from 24.1% to 39.6% while reducing
token cost by 11.2%.
Read the full abstract ▾
All authors Jie Liang, Zhengxin Yu, Hamid Nasiri, Peter Garraghan
Computation and Language
Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models
Kangwook Ko, Jaehyuk Jang, Wonjun Lee et al. (+2 more)
Removing a specific individual's information from
multimodal large
language models (MLLMs) is often needed after deployment, but existing methods rely on a retain set, which is hardest to obtain at that point, and rebuilding it recreates the
privacy exposure that unlearning aims to remove. Forgetting from the forget set alone instead damages the shared visual-
language computation, harming perception. We cast retain-free unlearning as a localization problem: causal tracing, weight transplant, and Fisher overlap all point to early-to-mid decoder MLPs as the layers where identity information is stored and, unlike other module families, can be modified without substantially disrupting
vision. We turn this into Pathway-Aware Visual-attribute Anchoring (PAVA), which confines updates to these layers and pairs a forget loss with a visual-attribute anchor that preserves image-grounded behavior by distilling the model's own pre-unlearning answers from the forget images alone. On MLLMU-Bench and ReMem, PAVA gives the strongest forget-retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.
Read the full abstract ▾
All authors Kangwook Ko, Jaehyuk Jang, Wonjun Lee, Hee-Seon Kim, Changick Kim
Computation and Language
What It Costs to Compose, Rebuild, and Correct Precomputed Memory
Asa Shepard
Language models can answer from precomputed
memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed
memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed
memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed
memories are best rebuilt on the cadence at which new information changes what the
memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a
memory, as pasted text or injected cache state, show particular promise for keeping precomputed
memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.
Read the full abstract ▾
Computation and Language
BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
Debarpan Bhattacharya, Malay Phadke and Sriram Ganapathy
Reliable uncertainty estimation is a crucial requirement for deploying large
language models (LLMs) and
vision-
language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty
estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this
matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA,
multilingual QA, and
multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty
estimators, while remaining simple, unsupervised, and applicable to black-box model settings.
Read the full abstract ▾
Computation and Language
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
Outongyi Lv, Yuanwei Zhang and Xiaoqun Zhang
Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large
language models' (LLMs)
reasoning abilities, demonstrating remarkable effectiveness across
reasoning tasks. Recent studies suggest that high-
entropy tokens play an exceptionally important role in model training, since training with only the highest 20%
entropy tokens yields significant performance gains. However, why such high-
entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-
entropy tokens within one answer tend to correlate with large gradient magnitude,
entropy alone fails to consistently reflect
token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based
Token Selection (GMTS) method to quantify
token importance, which leverages the
entropy-gradient connection to approximate gradient-magnitude rankings for
token selection. We find that training on the top 20%
tokens ranked by GMTS consistently outperforms
entropy-based
token selection across three
reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of
token contribution for RLVR training.
Read the full abstract ▾
Computation and Language
REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation
Haoran Que, Jiajun Shi, Ting Huang et al. (+7 more)
As
language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-
token prediction supervises what follows a context but leaves the intermediate
reasoning behind that continuation implicit. We introduce REER-PT, a scalable framework that extends Reverse-Engineered
Reasoning (REER) to raw pre-training data. REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise
reasoning annotations that reconstruct the missing connection between context and continuation. Candidate annotations are generated and refined offline, with perplexity serving as the optimization signal. Constraints on length and target leakage filter out unhelpful or trivial annotations. This sparse transformation preserves the source text and remains compatible with standard next-
token prediction, avoiding online
reasoning rollouts during pre-training. We apply REER-PT to transform a source pre-training corpus into an augmented one. Across augmented-data, original-
token, and selected-continuation comparisons, perplexity reductions range from 0.42 to 7.29, and only about 0.05% of annotation 13-grams appear verbatim in the source text. We then train two 680M-parameter models with the same architecture and training configuration on the source and augmented corpora, respectively. The augmented-data model gains up to 2.07 percentage points on several knowledge and
reasoning benchmarks. Together, the perplexity analysis indicates improved continuation predictability, while the controlled pre-training experiments suggest that this augmentation can improve model performance without changing the standard pre-training objective.
Read the full abstract ▾
All authors Haoran Que, Jiajun Shi, Ting Huang, Renming Pang, Jiaheng Liu, Ge Zhang, Wenhao Huang, Shen Yan, Wei Ye, Shikun Zhang
Computation and Language
Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text
Minkyung Cho, Jihyo Kim, SeungWoo Song et al. (+4 more)
Synthetic data is increasingly used to train large
language models (LLMs), yet its security implications remain poorly understood. Prior work on subliminal learning suggests that models can inherit behavioral traits from seemingly unrelated training data. In this work, we investigate whether such mechanisms can be exploited to inject targeted social biases into aligned models through semantically benign synthetic data. We construct a pipeline in which a misaligned teacher model generates filtered synthetic
datasets across domains such as creative writing and code generation, which are then used to fine-tune aligned student models. Our experiments show that benign-looking synthetic data can act as a covert channel for transmitting targeted biases while largely preserving the student model's general task capabilities. These results reveal a previously underexplored security risk in synthetic data-driven LLM training pipelines and highlight the need for improved safeguards. As one possible step toward this goal, we suggest that log-linearity-based scoring may provide a useful signal for screening seemingly benign synthetic data.
Read the full abstract ▾
All authors Minkyung Cho, Jihyo Kim, SeungWoo Song, Junghun Yuk, Minjoon Kee, Hoyun Song, KyungTae Lim
Computation and Language
TaxCE : A Framework for Automated Taxonomy Construction and Evaluation at Scale
Sandeep Sricharan Mukku, Albert Aristotle Nanda and Rohit Pyati
Organizing unstructured feedback text into hierarchical taxonomy is a fundamental challenge in NLP, particularly in domains where feedback arrives at massive scale in varied forms such as reviews, transcripts, and surveys. Existing approaches either produce shallow hierarchies, neglect long-tail topics, or lack rigorous evaluation frameworks. We present TaxCE, a fully automated framework that constructs multi-level hierarchical taxonomies from raw text through progressive condensation of corpus content into actionable segments, deduplicated semantic units, and granular topics with definitions, which are then organized bottom-up into a hierarchy with corpus-groundedness. We also introduce three corpus-grounded evaluation metrics, Exclusivity, Exhaustivity, and Granularity (
EEG), and integrate them into a metrics-in-the-loop iterative refinement mechanism that diagnoses deficiencies and applies targeted corrections until convergence. Extensive experiments demonstrate that TaxCE consistently outperforms existing baselines spanning classical topic models, neural methods, and LLM-based approaches, with average improvements of 11.8, 20.5, and 15.7 percentage points in exclusivity, exhaustivity, and granularity respectively over the strongest baseline. Human evaluation further confirms superior taxonomy quality, actionability, and navigability.
Read the full abstract ▾
Computation and Language
Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training
Lukas Borggren, Jenny Kunz and Marco Kuhlmann
Large
language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to address this limitation is to specialise existing models through additional training on target-domain corpora. In this work, we investigate such continued pre-training for adapting large
language models to Swedish journalism, using a high-quality
dataset that we curate from millions of news articles. To evaluate the adaptation efficacy, we also construct a novel domain-specific
benchmark that covers six editorial tasks. Through full and parameter-efficient fine-tuning across two model sizes, we find that continued pre-training yields benefits in the target domain, but only when paired with experience replay to mitigate forgetting. We observe consistent enhancements in the models' generation quality and factual knowledge, but not their proficiency in discriminative tasks. Exploring a training-free method to facilitate instruction following, we see further improvements, but exclusively for models trained with low-rank adaptation. Crucially, we demonstrate the importance of targeted evaluation in the adaptation process, as an existing Swedish
benchmark largely fails to capture the models' in-domain performance gains.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
Boryeong Cho, Sumyeong Ahn and Se-Young Yun
Direct Preference Optimization (DPO) simplifies
alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57
dataset-model-
benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.
Read the full abstract ▾
Computation and Language
Language Proficiency Assessment from Eye Movements in Naturalistic Passage Reading
Shachar Frenkel, Ido Falah, Omer Shubi et al. (+1 more)
Standard
language proficiency tests rely on linguistic tasks such as vocabulary, grammar and reading comprehension quizzes. An alternative, cognitively motivated approach, introduced in Berzak et al. (2018), proposed instead to predict
language proficiency from behavioral traces of eye movements in reading. In this work, we validate and extend this approach from single sentences to more naturalistic reading of contextualized passages in English as a second
language, new proficiency measures, prediction models, and reading in an information seeking regime. We find that the approach is effective in all these evaluations. We further address two key open questions on eye movement based proficiency testing: (1) potential scoring biases that reflect the proximity of the reader's native
language to English, which may undermine validity, and (2) its reliability. We find that eye movement based proficiency scores are indeed biased towards L1s that are linguistically closer to English. We propose a score debiasing method which effectively remedies this issue. The reliability analyses suggest that eye movement proficiency scores are more reliable than standard
language proficiency scores. Overall, our results strengthen and broaden the empirical foundations for future eye movement based
language assessment technologies.
Read the full abstract ▾
All authors Shachar Frenkel, Ido Falah, Omer Shubi, Yevgeni Berzak
Computation and Language
Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation
Ran Zhang, Miryam de Lhoneux and Wessel Poelman
Pixel-based
language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource
languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from
languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while
multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.
Read the full abstract ▾
Computation and Language
WebWorld: The Browser as a World Model for Self-Improving Web Code
Jiajun Wu, Jian Yang, Yaxin Du et al. (+7 more)
VLM-driven self-improvement of web code has a structural flaw: the model that proposes the repair is the model that judges it, and visual plausibility under that judge is a poor proxy for whether the page actually works. What the loop is missing is a counterparty the VLM cannot fool, and the browser already is that counterparty: a deterministic, executable simulator of how an HTML artifact behaves under user actions, and in everything but name a world model for web code. We present WebWorld, the interface that lets a VLM prior interact with this browser-as-world-model autonomously and decides which interactions become supervision. Each round, the VLM emits a critique that the planner compiles into a typed interaction contract; the browser re-executes the candidate and issues an acceptance certificate only when both target progress and preservation of every previously verified capability hold; certified transitions accumulate as a quality ratchet that is the only thing the SFT export ever sees. Under matched training, WebWorld-27B improves Raw-27B by 5.3 points on HTMLBench-400 and 14.9 points on MiniAppBench-Val, and reaches the level of strong frontier systems such as Kimi-K2.6 and GPT-5.4 on interactive HTML generation. Equal-size ablations show that browser-backed admission carries the gain: without the certificate, the matched 9B lift nearly disappears.
Read the full abstract ▾
All authors Jiajun Wu, Jian Yang, Yaxin Du, Wei Zhang, Haowen Wang, Junhang Cheng, Yuxuan Zhang, Tuney Zheng, Xianglong Liu, Ming Zhou
Computation and Language · Cross-listed from cs.AI
ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions
Guangxiang Zhao, Qilong Shi, Xusen Xiao et al. (+13 more)
Benchmark saturation and data contamination increasingly obscure genuine scientific
reasoning in frontier LLMs. We introduce ScienceArena, an olympiad-style
benchmark from thirteen public science competitions in physics, chemistry, and biology, including IPhO and IChO 2025–2026, IBO 2023, USAPhO 2026, and USNCO 2025. Its open-ended, multi-step problems use process-credit rubrics, making faithful scoring difficult. We build ScienceArena through an expert-audited digitization pipeline that converts official exams, figures, solutions, and rubrics into structured items verified by olympiad medalists. To scale evaluation beyond costly human grading, we calibrate LLM-as-judge against medalist ground truth on archived answers from five models across IPhO and IChO; two strong judges stay within one point of expert total scores. Medalist notes show that failures often stem from visual grounding, structure fidelity, and global problem control rather than missing terminology. Evaluating fourteen recent LLMs with interleaved solving, we find that top models obtain medal-equivalent rubric scores on several public international exams, while chemistry and long-horizon consistency remain key bottlenecks. We provide an interactive \href{https://science-arena.onrender.com/}{demo}.
Read the full abstract ▾
All authors Guangxiang Zhao, Qilong Shi, Xusen Xiao, Wenpu Liu, Yaoming Li, Linfeng Hao, Shuyang Hou, Zijian Guo, Xinrui Zhang, Yuntian Zhao, Zhengyang Wang, Wenrui Liu, Yuhan Wu, Tong Yang, Lin Sun, Xiangzheng Zhang
Computation and Language · Cross-listed from cs.CV
Lot Machine: Multimodal Lot Extraction from Auction Catalogs
Mathias Zinnen, Alisha Mund, Sabine Lang et al. (+3 more)
For provenance research and art
market studies, auction catalogs are an essential resource to trace specific objects over time and space. While historical auction catalogs follow established domain conventions, their internal formatting remains highly variable, and their large-scale analysis is currently restricted by the lack of machine-readable representations of the auction lots. We propose a pipeline to automatically extract structured lot-level metadata from German Sales, a large database of historical auction and sales catalogs from the 19th and 20th centuries. Using a manually annotated test set of representative catalog pages, we evaluate
Vision-
Language Models (VLMs) under varying
prompt strategies and constrained decoding frameworks. To reflect the practical constraints faced by cultural heritage institutions, including budget, compute resources, and data
privacy requirements, we
benchmark the methods across different deployment modes ranging from commercial providers to locally hosted, quantized models. We find that commercial endpoints establish the performance ceiling, while institutional gateways offer a viable,
privacy-preserving alternative. Local deployments remain feasible, but strictly require enforcing the output structure during generation to guarantee a valid JSON format. While varying degrees of human-in-the-loop correction are still necessary, this work demonstrates that a VLM-based pipeline can successfully unlock historical auction catalogs for large-scale automated analysis.
Read the full abstract ▾
All authors Mathias Zinnen, Alisha Mund, Sabine Lang, Lukas Hüttner, Thomas Gorges, Vincent Christlein
Computation and Language
UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory
Peijun Qing, Fobo Shi and Soroush Vosoughi
Long-term
memory is increasingly important for conversational
agents, yet existing
benchmarks primarily measure
memory through pointwise factual recall: whether a system can recover isolated facts or event-level details from prior interactions. Real-world
memory use, however, often requires a more demanding capability: integrating distributed, implicit, and noisy evidence across extended interaction histories into coherent, task-oriented outputs. We call this capability
memory utilization. Here, we introduce UtilMem, a diagnostic
benchmark comprising 1,717 instances across five domains, designed to evaluate four underexplored aspects of
memory utilization:
reasoning over dense histories, identifying implicitly relevant
memories, synthesizing distributed evidence into summaries, analyses, or plans, and resisting interference from semantically similar distractors. Evaluating a diverse set of
retrieval-based and
memory-augmented systems, we find that strong performance on conventional factual-
memory benchmarks does not reliably translate into effective
memory utilization. Moreover,
retrieval alone is insufficient: even when relevant evidence is successfully recovered, systems frequently fail to integrate information across sessions or to distinguish useful evidence from plausible distractors. These findings expose a substantial gap between accessing stored information and using it effectively, and suggest that progress in long-term conversational
memory will require architectures that explicitly support evidence integration and robustness to
retrieval interference. Code is available at https://github.com/peijunallin/UtilMem.
Read the full abstract ▾
Computation and Language
Two Centuries of Sexism in British Parliament: A Computational Analysis of Women's Representation in the Hansard Corpus
Mohammad Omar Khursheed, Mandira Sawkar and Ashiqur R. KhudaBukhsh
The
language a legislature uses to debate women's rights, even in favour of them, encodes systematic patterns of sexism that persist across two centuries. In this work, we analyse 6,531
speeches over 200 years of UK parliamentary debate (Hansard, 1803-2005) by using large
language models to classify a speaker's perspective towards women's suffrage and political representation, as well as analyse sexist
speech in parliament from the lens of the Ambivalent Sexism Inventory. We also release this parliamentary
dataset, an organized and metadata-enriched version of the publicly available Hansard Corpus optimized for computational social science research, with 6.7 million
speeches across 1.2 million debates, with 89% gender-matching for
speeches by MPs from the House of Commons. We find that 54% of
speeches opposing women's representation contain sexist content, compared to 21% of
speeches that are for the cause, and that the two sides use fundamentally different types of sexism: anti-suffrage rhetoric combines hostile and benevolent framing, while pro-suffrage sexism is overwhelmingly benevolent. Female MPs support women's political rights at 93% compared to 70% for male MPs, a gap that closes only after enfranchisement. Our findings are evidence that benevolent and hostile sexism are used in different rhetorical contexts in a manner consistent with the theory of Ambivalent Sexism.
Read the full abstract ▾
Computation and Language
Agents in the Large: Perception-Centered Architecture for Persistent Agents
Shihan Dou, Haoxiang Jia, Shichun Liu et al. (+14 more)
Cognitive
language agents have achieved substantial progress by equipping
language models with
memory, tools, and decision-making procedures, enabling
agents to reason and act in interactive environments. Existing frameworks largely cast these
agents as systems for solving user-specified, bounded tasks. An increasingly important goal is for
language agents to provide persistent assistance in long-lived settings where user needs, context, and service procedures persist and change, and to remain useful across the broad range of tasks that arise over time. Yet we still lack a framework to characterize persistent AI
agents, organize existing work, and guide future development. To this end, we propose a Perception-Centered Architecture for Persistent
Agents (Pera). Pera describes a persistent
agent organized around perception and control components that continually perceive service-relevant signals from episodic task executions, internal context, and changes in the surrounding environment, and use these signals to construct lifecycle tasks. These tasks drive the ongoing operation and adaptation of the
agent's service procedures. We use Pera to retrospectively organize recent work, examine a detailed case study, and offer forward-looking insights for building more capable persistent
agents. Just as software engineering moved from programming in the small to programming in the large, Pera frames the
evolution of
language agents as an analogous architectural transition toward long-lived, adaptive intelligence systems.
Read the full abstract ▾
All authors Shihan Dou, Haoxiang Jia, Shichun Liu, Feng Chen, Chenhao Huang, Yujiong Shen, Shaofan Liu, Jiayi Chen, Jiahang Lin, Honglin Guo, Qianyu He, Minghao Guo, Ziyi Ye, Pluto Zhou, Tao Gui, Qi Zhang, Xuanjing Huang
Computation and Language
ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation
Samir Abdaljalil, Hunzalah Hassan Bhatti, Ahlam Bashiti et al. (+11 more)
We present an overview of the ImageEval 2026 shared task on culturally grounded Arabic
multimodal evaluation. It includes two tasks: (i) AynVQA, covering spoken visual question answering and image-grounded
hallucination detection in English and Modern Standard Arabic (MSA), and (ii) CRAI-Bench, evaluating the cultural accuracy of text-to-image generation. A total of 14 teams participated in the test phase, with 12 teams submitting system description papers. Participating systems used a range of approaches, including zero-shot prompting, fine-tuning of
vision-
language models,
speech-recognition pipelines, ensembling, and score calibration. We describe the task setup,
datasets, evaluation procedure, and participating systems, and summarize the main results across the different tracks. All
datasets and evaluation scripts from the shared task are released to the research community. The shared task highlights the challenges of culturally grounded
multimodal evaluation, particularly for Arabic
speech and image-text
reasoning.
Read the full abstract ▾
All authors Samir Abdaljalil, Hunzalah Hassan Bhatti, Ahlam Bashiti, Farina Amir, Md Arid Hasan, Basel Mousi, Nadir Durrani, Fahim Dalvi, Zien Sheikh Ali, Erchin Serpedin, Hasan Kurban, Mustafa Jarrar, Shammur Absar Chowdhury, Firoj Alam
Computation and Language
Hi-Q: Hierarchical Evidence-guided Query Refinement for Multi-Hop Question Answering
Jueun Kim, Sungho Park and Wook-Shin Han
A central bottleneck in multi-hop Question Answering (QA) is that the granularity at which a question is expressed often differs from the granularity at which corpus evidence is retrievable. Existing methods address this mismatch by imposing fixed graph structures over the corpus, by iteratively reformulating the query, or by executing a generated program over it, but these strategies do not explicitly decide when a query unit is already supported by evidence and when it should be refined. We formulate this bottleneck as retrievable granularity discovery and introduce Hi-Q, an evidence-conditioned framework for hierarchical query refinement. At each query node, a resolution operator tests whether retrieved evidence supports the current query unit; resolved nodes terminate, while unresolved nodes are expanded by a dependency-preserving binary operator and checked by a semantic coverage verifier. Hi-Q therefore grows a query tree whose
topology is determined by corpus support signals rather than by a fixed decomposition template or a pre-built graph. We evaluate Hi-Q on three multi-hop QA
benchmarks, primarily under full-corpus
retrieval, where dependent evidence must be located among open-domain distractors rather than within a small annotated pool. In this setting Hi-Q reaches 52.3 EM and 64.0 F1 averaged over the three
benchmarks, ahead of the iterative
retrieval baseline IRCoT by 15.1 EM / 18.2 F1 on that same average, and ahead of the graph-based RAG baseline PropRAG by 11.5 EM / 12.0 F1 on MuSiQue-full, without corpus-wide graph construction. In the restricted supporting/distractor setting used by prior work, Hi-Q likewise attains the best accuracy, with 57.9 EM and 69.3 F1 on average, ahead of PropRAG by 5.6 EM / 3.9 F1 and IRCoT by 13.7 EM / 15.8 F1. The project page is available at https://hi-q-project.github.io/.
Read the full abstract ▾
Computation and Language
More Capable, Less Faithful: A Multilingual Analysis of Mathematical (Un)Solvability Detection in LLMs
Maria-Eleni Zoumpoulidi, Nikolaos Xiros and Georgios Paraskevopoulos
Solvability detection is one of the most challenging aspects of mathematical
reasoning for Large
Language Models (LLMs). While prior work has studied this capability extensively, these analyses have been limited to English. Consequently, it remains unclear whether
multilingual failures arise from differences in internal Solvability Belief or from
language-dependent failures to express it. To address this gap, we introduce the first
multilingual benchmark of paired solvable and unsolvable mathematical problems, extending ReliableMath to French and Greek. Using this, we train
multilingual probes predicting Solvability Belief and analyze the solvability detection capabilities of state-of-the-art LLMs behaviorally, representationally, and in terms of faithfulness. We find that Solvability Belief is encoded as a largely universal,
language-agnostic feature, and that higher-resource
languages such as English, despite achieving stronger mathematical
reasoning performance, exhibit lower solvability-detection faithfulness.
Read the full abstract ▾
Computation and Language
Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfer
Minju Song, Hyeon Hwang, Junhyun Lee et al. (+1 more)
Large
language models exhibit substantial performance variation across
languages, even when solving semantically equivalent tasks. Existing analyses often treat this phenomenon as an observational disparity caused by differences in pretraining data, tokenization, or
benchmark coverage. We study a complementary hypothesis: high-resource
languages (HRLs) may more reliably elicit latent computations useful for task-specific (i.e. mathematical)
reasoning, while lower-resource
languages (LRLs) may under-activate those computations despite expressing the same task. To test this hypothesis, we introduce a mechanistic intervention framework for identifying and transferring task-relevant sparse latent features across
languages. Using sparse autoencoders over residual-stream activations, we isolate features enriched in successful HRL task-specific
reasoning while filtering out source-
language and generic-generation features. We then construct steering directions from these features and inject them during LRL inference. The resulting interventions test whether the selected features are functionally involved in the observed
reasoning gap: suppressing them should impair source-
language reasoning, while activating them should partially recover target-
language reasoning beyond random and non-task controls. Our framework reframes some cross-lingual
reasoning gaps as failures of mechanism elicitation rather than capability absence, and offers a causally testable route to feature-mediated transfer without translation, fine-tuning, or changing the user-facing
language.
Read the full abstract ▾
All authors Minju Song, Hyeon Hwang, Junhyun Lee, Jaewoo Kang
Computation and Language
From Final Artifacts to Trajectories: Retrospective Process Supervision for Evidence-Grounded Long-Form Generation
Junjie Huang, Jiarui Qin, Di Yin et al. (+4 more)
Trajectory data is getting more vital for training large
language models for boosting the agentic abilities. Unlike the verifiable domains such as coding or mathematics, scaling trajectory data for open-ended tasks is much more difficult because these tasks lack singular ground truth and are costly to annotate or verify. In this paper, we propose RetroGen, a self-improving framework of retrospective process supervision. Our key observation is that although expert trajectories are scarce, high-quality final artifacts such as literature reviews, analyst reports and legal judgments, are abundant in pre-training data and can be viewed as compressed traces of the evidence-seeking processes that produced them. RetroGen reconstructs candidate latent trajectories from expert artifacts, verifies them against both the artifact and supporting evidence, and trains models on their own successful reconstruction data, without requiring trajectory data from stronger models. Experiments show that RetroGen improves grounding, faithful synthesis, and long-form evidence-seeking
agent tasks.
Read the full abstract ▾
All authors Junjie Huang, Jiarui Qin, Di Yin, Weiwen Liu, Yong Yu, Xing Sun, Weinan Zhang
Computation and Language · Cross-listed from cs.LG
Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometry
Jiani Guo, Junjie Wang, Jie Wu et al. (+5 more)
Multimodal geometry
reasoning requires VLMs to extract precise visual relations and preserve them through multi-step deduction. Existing free-form traces obscure the decisions that determine the answer, and trajectory-level reinforcement learning distributes a single terminal signal across the entire response. We introduce credit-addressable
reasoning, in which the semantic units exposed during inference also define where learning compares alternatives and assigns credit. We instantiate this principle with Code-CoT, which retains the diagram, represents visual relations as line-addressable executable code, and organizes
reasoning into typed events, and CE-GRPO, which selects event boundaries using structural priors and type-normalized
entropy, samples complete continuations from shared prefixes, and converts outcome differences into localized advantages. Across nine geometry
benchmarks, CE-GRPO achieves an average accuracy of 76.04, outperforming Qwen3-VL-8B and trajectory-level GRPO by $8.09$ and 3.43 points, respectively. Its relative advantage increases with the number of intermediate events, demonstrating the value of representation–optimization co-design for long, dependency-heavy
multimodal reasoning.
Read the full abstract ▾
All authors Jiani Guo, Junjie Wang, Jie Wu, Pengxiang Zhao, Dongdong Zhang, Shaohan Huang, Yujiu Yang, Furu Wei
Computation and Language
Graph Evidence Is Not Enough: Diagnosing Native Decoder Use in Graph-Augmented LLMs
Xiaoyu Guo, Pengcheng Chen, Jiong Yu et al. (+3 more)
Graph-augmented large
language models often assume that graph evidence produced by external computation and placed in the input can be used by the native decoder. We test this assumption with HopQA, a deliberately bounded diagnostic that asks for the shortest-hop distance between two query nodes. Because the answer is a small integer and the target is purely topological, failure cannot be dismissed as open-ended generation or ambiguous evaluation. Yet existing graph-augmented baselines still fail on this setting, showing that providing graph evidence is not the same as making it usable. We introduce an intervention triangle with three matched conditions: readable graph evidence, shuffled graph evidence, and no-graph input. This separates evidence inclusion, structural readability, and decoder-usable
topology. Guided by this diagnosis, we present S$^2$GE as an instance showing that diagnosis-driven interface design can improve native decoder usability. S$^2$GE uses query-aware sampling, endpoint and proximity-based ordering, and structure-preserving
alignment. Across DBLP, Biomedical, GoodReads, and PubMed, S$^2$GE achieves strict exact-match scores of $36.5\%$, $57.8\%$, $76.6\%$, and $52.0\%$, improving over the strongest native-generation baseline by $53.5$ points on average. The interventions further reveal harmful-shuffle, shuffle-robust, and no-graph-saturated regimes.
Read the full abstract ▾
All authors Xiaoyu Guo, Pengcheng Chen, Jiong Yu, Yi Lu, Yaohua Wang, Ziyang Li
Computation and Language · Cross-listed from cs.AI
EvoSkill Injection: Red-Teaming Autonomous Skill Generation and Evolution in Self-Evolving Agents
Doyun Kim, Chanwoo Kim, Sugyeong Eo et al. (+2 more)
LLM-based
agent systems increasingly adopt skill-based architectures to reduce repetitive
reasoning costs and improve stable, efficient task execution. Recent studies propose self-evolving
agents that autonomously generate, refine, and reuse skills from past experiences to enable continuous capability
evolution. However, autonomous skill
evolution introduces a new attack surface in which malicious capabilities are generated, stored, and reused as legitimate skills. In this paper, we define EvoSkill Injection as a threat model targeting the autonomous skill generation and
evolution pipeline of self-evolving
agents. We further propose SARGE (Red-teaming Autonomous Skill Generation and
Evolution in self-evolving
agents), a red-teaming framework for evaluating this threat model through iterative generation, escalation, and reinforcement interactions. To support our framework, we construct EvoSkillBench, a
benchmark dataset of malicious interaction trajectories for inducing malicious skill formation in self-evolving
agents, and introduce EvoSkillSafetyBench, a post-attack
benchmark for evaluating whether injected malicious skills are subsequently retrieved and activated as harmful behaviors. Our evaluation shows that SARGE induces malicious skill formation and that injected skills are persistently stored and repeatedly activated, highlighting the risk of persistent capability corruption.
Read the full abstract ▾
All authors Doyun Kim, Chanwoo Kim, Sugyeong Eo, Yeo-Chan Yoon, Chanjun Park
Computation and Language
Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions
Jaewoo Ahn, Junseo Kim, Hyunseo Kim et al. (+4 more)
Strategic deception by LLM and VLM
agents has emerged as a central AI
alignment and safety concern. Social-deduction games (where each player holds a hidden role and communicates with others to deduce identities) serve as the canonical testbed, particularly in multi-
agent settings. Existing testbeds, however, are text-only and run on a single fixed
agent configuration, missing the non-verbal sensorimotor channels treated as core by deception taxonomies and leaving it ambiguous whether an observed behavior reflects the underlying model or the surrounding harness. We introduce MineAmongUs, a 3D
multimodal Among Us sandbox where imposter
agents must deceive crewmates through joint verbal and non-verbal action. We also propose ARIA, a configurable VLM-
agent harness that exposes five cognitive-component ablation axes; and an atom- and arc-level annotation scheme grounded in deception taxonomies and operationalized at scale by an LLM-as-a-Judge reaching near-human atom-labeling agreement. Empirical results show that VLM
agents pursue imposter wins through joint verbal and non-verbal deception, with non-verbal channels emerging as the more decisive winning contributors across both harness ablation and cross-VLM evaluation. Taken together, our work opens a new path for
embodied VLM-
agent alignment research.
Read the full abstract ▾
All authors Jaewoo Ahn, Junseo Kim, Hyunseo Kim, Heeseung Yun, Jaehyeon Son, Zsolt Kira, Gunhee Kim
Computation and Language
Ceiling-Clipped Acceptance Histograms Indicate Stranded Speed-up in Block-Diffusion Speculative Decoding
Ephrem Wu
Speculative decoding speeds up generation with an efficient draft model (drafter) that proposes
tokens for a target model to verify in one pass, preserving the target's output distribution. High-acceptance block-
diffusion drafters such as DFlash and DFlare fill an entire block in one parallel pass. In many cycles, the target accepts the whole block, so the drafter exhausts its trained block horizon before verification fails. We call this unrealized acceptance stranded speed-up. A mean committed length, per
prompt or per cycle, hides it, whereas the acceptance histogram exposes it as a spike in the ceiling bin, the fraction of cycles that accept the entire block. We recommend the histogram as a preflight check before spending training compute. Naively widening the block at inference does not recover the speed-up, because once the block outgrows its training size, the drafter's bidirectional
attention shifts its distribution even at early positions and erodes front-of-block verification. Instead, we post-train the drafter on a longer block with a short curriculum that emphasizes the newly exposed positions, a method we call DBloom. Expanding the pretrained DFlash and DFlare drafters from block size 16 to 24 across Qwen3-8B and Qwen3-4B targets raises the per-
prompt committed length on the high-ceiling
benchmarks by a median of +0.8
tokens (up to +1.1). Once continuation fine-tuning precedes expansion, the increase reaches 1.37
tokens. The same expansion also lifts committed length on all seven
benchmarks for Gemma-4-12B-IT, a different model family, by a median of +0.41
tokens (Arm A), and the full continuation-then-expand pipeline (Arm B) adds +0.29 to +0.98
tokens over the same B16 drafter. In a
prompt-matched comparison against JetSpec, a contemporary tree-based drafter not used in our design, DBloom commits more
tokens on every
benchmark at tree budgets up to 64 nodes.
Read the full abstract ▾
Computation and Language
Learning to Reason and Use Tools through Unsupervised Fine-Tuning in Task-Oriented Dialog Systems
Markel Ferro and Oier Lopez de Lacalle
Current dialogue systems struggle with dynamic information
retrieval, often leading to
hallucinations and lower response accuracy. We address this by adapting the ReAct framework for Task-Oriented Dialogue, enabling Large
Language Models (LLMs) to access external knowledge and produce factual responses. Mainly, we propose an unsupervised fine-tuning pipeline that harvests
reasoning trajectories via in-context learning inference. High-quality samples are filtered using an LLM-based judge to construct a robust training set. This is enhanced by a unsupervised self-improvement loop, where improved checkpoints generate increasingly better trajectories for subsequent fine-tuning iterations. Experiments on the SIMMC
dataset demonstrate that ReAct-based systems outperform baselines due to superior
reasoning and tool use. Notably, our fine-tuned 8B model surpasses a 70B in-context system. Finally, we present an error analysis, impact of scene complexity, and cross-domain generalization.
Read the full abstract ▾