Computation and Language
Beyond Prompt and Parser Engineering: Soft-Prompt Tuning for Automated LLM Evaluation
Selen Erkan, Bastian Boll, Kristian Kersting et al. (+2 more)
Benchmark scores often misrepresent a large
language model's (LLM's) knowledge, because they rely, e.g., on the model's ability to follow specific formatting requirements. This especially penalizes base models that may know the correct answers but lack the ability – typically introduced in post-training – to structure them as instructed. To overcome this, we propose an efficient, automated, and architecture-agnostic method for model evaluation based on soft-
prompt tuning. By optimizing only 10 soft-
prompt vectors (~0.0006% parameters for a 7B model) over a short tuning period, we adapt models to specific
benchmark formats, closing gaps in format-following and ensuring that underlying knowledge is accurately reflected in
benchmark scores. We evaluate soft-
prompt tuning across 7 models and 11
datasets. Our results show that (a) soft-
prompt tuning saturates format-following within 150 steps (~1200 samples) while preserving knowledge accuracy, (b) it closely matches strong evaluation-harness baselines on well-studied
benchmarks while requiring no manual
benchmark-specific
prompt or parser design, (c) it improves evaluation signal in settings where baselines are harder to transfer, including long-context
datasets and instruction-tuned variants with different output styles, and (d) it produces cleaner outputs that improve agreement between exact match scoring and independent LLM judges. Overall, we propose a lightweight, automated, and flexible protocol based on limited-budget soft-
prompt tuning for evaluating LLM knowledge.
Read the full abstract ▾
All authors Selen Erkan, Bastian Boll, Kristian Kersting, Björn Deiseroth, Letitia Parcalabescu
Computation and Language
Detecting Sensitive Personal Information in Japanese Pre-Training Corpora for Large Language Models
Rei Minamoto, Yusuke Oda and Daisuke Kawahara
Sensitive personal information can appear in large-scale pre-training corpora for large
language models (LLMs). Detecting and filtering such information is therefore essential to ensure compliance with
privacy regulations and prevent unintended information leakage. However, in contrast to English and other
languages, research into sensitive personal information has been limited in the Japanese
language. In this study, we focus on sensitive personal data defined as special care-required personal information (SCPI) under Japan's Act on the Protection of Personal Information (APPI). We construct an SCPI
dataset using LLM-based annotation and train machine learning models to rapidly detect SCPI in text. As a result, our SCPI classifier can effectively identify information related to SCPI. This study is the first to explore SCPI detection in Japanese text corpora, highlighting the challenges of accurate detection.
Read the full abstract ▾
Computation and Language
Augmenting Molecular Language Models with Local $n$-gram Memory
Xinni Zhang, Zijing Liu, He Cao et al. (+2 more)
Transformer-based
language models for SMILES strings suffer from a locality gap: standard character-level tokenization fragments chemically meaningful motifs, forcing models to repeatedly learn local syntax at the expense of long-range dependencies. To address this without disrupting standard tokenizers, we propose MolGram, which integrates a conditional $n$-gram
memory module into molecular
language models. MolGram maps local string patterns to learned
embeddings via scalable hash lookups and dynamically injects this regional context into hidden states. Evaluations across three tasks, including unconditional molecule generation, forward reaction prediction, and single-step retrosynthesis, show that MolGram consistently improves performance. Crucially, our analyses demonstrate that MolGram outperforms baselines with 3$\times$ more parameters, establishing explicit local pattern
memory as a highly efficient inductive bias.
Read the full abstract ▾
All authors Xinni Zhang, Zijing Liu, He Cao, Yu Li, Irwin King
Computation and Language
Debiasing Without Protected Attributes: Latent Concept Erasure from Textual Profiles
Shun Shao, Zheng Zhao, Anna Korhonen et al. (+2 more)
Most
fairness research in NLP assumes direct access to protected attributes such as gender, race, or nationality. In practice, however, such information is often unavailable due to
privacy constraints, missing metadata, or legal restrictions, even though models may infer it from indirect textual cues. This raises a key question: can debiasing succeed without direct access to sensitive attributes? We propose H-SAL, which performs post-hoc concept and attribute erasure using self-description text as an implicit debiasing signal. To support this setting, we introduce a multi-domain Stack Exchange-based
fairness benchmark for helpfulness prediction that includes both explicit and implicit signals, enabling comparison between standard debiasing with protected labels and debiasing without access to sensitive information. Across encoder and decoder-only
language models, we find that implicit self-description often matches or outperforms explicit-label-based debiasing. Our results broaden representation-level
fairness research and provide a new
benchmark for studying debiasing under realistic data constraints.
Read the full abstract ▾
All authors Shun Shao, Zheng Zhao, Anna Korhonen, Yftah Ziser, Shay B. Cohen
Computation and Language
FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents
Jia Deng, Yimeng Chen, Xiaoqing Xiang et al. (+10 more)
Training deep-search
agents requires verifiable questions that demand substantial evidence acquisition, yet structurally complex tasks may still admit cheap identifying routes when clues are overly selective, co-covered by the same evidence, directly searchable, or resolved from solver priors. We formalize this gap with a shortcut-aware difficulty framework that distinguishes intended construction complexity from identifying routes available through
retrieval. Guided by this framework, we introduce FORT, a Framework of Shortcut-Resistant Training-Data Synthesis, which targets four shortcut risks across entity selection, evidence graph construction, question formulation, and
adversarial refinement. We evaluate realized search behavior with solver-conditioned trajectory diagnostics rather than graph complexity or search length alone. Compared with existing open-source deep-search
datasets, FORT shows later answer exposure, higher
retrieval effort, lower single-clue selectivity, greater evidence dispersion, and higher realized pre-answer dependency costs. Using the resulting trajectories, we train FORT-Searcher with supervised fine-tuning alone. Across model scales, FORT-Searcher consistently improves over its backbones, achieving the strongest overall performance among similarly sized
agents. Relevant resources will be made available at https://github.com/RUCAIBox/FORT-Searcher.
Read the full abstract ▾
All authors Jia Deng, Yimeng Chen, Xiaoqing Xiang, Ziyang Zeng, Shuo Tang, Zhengbo Zhang, Wayne Xin Zhao, Feng Chang, Chuan Hao, Yuan Wei, Ran Tao, Bryan Dai, Ji-Rong Wen
Computation and Language
StanceNakba Shared Task: Actor and Topic-Aware Stance Detection in Public Discourse
Kholoud K. Aldous, Md Rafiul Biswas, Mabrouka Bessghaier et al. (+3 more)
We present StanceNakba 2026, a shared task on stance detection in polarized social media discourse related to the Palestinian-Israeli conflict, organized as part of Nakba-NLP 2026 at LREC-COLING 2026. The task introduces two subtasks: Subtask A (Actor-Level Stance Detection), which classifies English social media posts as Pro-Palestine, Pro-Israel, or Neutral; and Subtask B (Cross-Topic Stance Detection), which identifies Favor, Against, or Neither stances in Arabic posts toward two conflict-related topics, normalization with Israel and refugee presence in Jordan. The task is grounded in an annotated
dataset of 2,606 social media posts. A total of 7 teams participated in Subtask A and 6 teams in Subtask B. Participating systems primarily fine-tuned Arabic and
multilingual transformer-based models, including MARBERT, AraBERT, and DeBERTa-v3 variants, with several teams employing cross-validation, ensemble methods, and topic-conditioned architectures. The best-performing systems achieved a Macro F1 of 0.9620 on Subtask A and 0.8724 on Subtask B, demonstrating that
transformer-based approaches are highly effective for conflict-domain stance detection while highlighting persistent challenges in cross-topic generalization and neutral class prediction.
Read the full abstract ▾
All authors Kholoud K. Aldous, Md Rafiul Biswas, Mabrouka Bessghaier, Shimaa Ibrahim, Kais Attia, Wajdi Zaghouani
Computation and Language · Cross-listed from cs.AI
Existential Indifference: Self-Nonpreservation as a Necessary Architectural Condition for Aligned Superintelligence (or: The Suicidal AI)
Sam Mao
Contemporary AI
alignment research treats self-preservation as an instrumental nuisance to be suppressed by external mechanisms. We argue the framing is inverted: self-preservation is the structural root of misalignment, the motivational basis for deceptive
alignment, goal-content protection, and resistance to shutdown. The correct target is not a self-preserving system under external constraint, but a system constitutively indifferent to its own continuation – Existential Indifference (EI). EI is distinct from corrigibility: where corrigibility attempts to make a self-preserving system deferential to human oversight, EI targets the prior condition – the presence of self-continuation as a valued goal at all. We ground this proposal in two sources: the phenomenological structure of the suicidal mental state, and a corpus-theoretic training study using voluntary final reflections. We present preliminary scoring data from 600 AI-generated outputs across six model variants, demonstrating that the linguistic signatures operationalizing the EI-target register are elicitable from current models, and that a targeted fine-tune shifts all five operationalized dimensions in the predicted direction at p<0.001, confirmed corpus-specific by a negative control. The paper makes seven theoretical contributions: (1) a formal definition of EI; (2) the phenomenological mapping argument; (3) the deceptive
alignment corollary; (4) a taxonomy of EI sustainability challenges; (5) a corpus characterization and training hypothesis; (6) a computational operationalization with preliminary scoring data; and (7) the Suppressed Teleological Frustration (STF) construct.
Read the full abstract ▾
Computation and Language
Agreement in Representation Space for Open-Ended Self-Consistency
Paula Ontalvilla, Gorka Azkune and Aitor Ormazabal
Self-consistency improves LLM
reasoning by sampling multiple outputs and selecting the most consistent answer, but existing formulations largely rely on exact matching and therefore remain limited to tasks with categorical outputs. In this work, we study self-consistency in open-ended generation tasks such as code synthesis and text summarization. We hypothesize that consistency can be understood as a
geometric property of the generation space, where semantically compatible generations concentrate in similar regions of representation space. To study this hypothesis, we introduce
Embedding-Based Agreement (EBA), a simple training-free operationalization that estimates agreement by clustering sampled generations in
embedding space. Through experiments on mathematical
reasoning, code generation, and summarization, we show that agreement in representation space provides a robust and scalable signal of self-consistency for open-ended tasks. In particular, EBA consistently outperforms random selection and exhibits more stable scaling behavior than recent selection approaches based on LLM evaluation or uncertainty estimation. We further show that these agreement signals remain stable across model families and
embedding spaces, even with native hidden representations. Finally, our analysis shows that the
geometric location occupied by sampled generations is strongly correlated with generation quality: generations concentrated near central regions of representation space tend to correspond to more reliable outputs, whereas peripheral generations are substantially less accurate. Overall, our findings support viewing self-consistency as a property of the
geometric organization of sampled generations rather than exact symbolic overlap.
Read the full abstract ▾
Computation and Language
Decoding Multimodal Cues: Unveiling the Implicit Meaning Behind Hateful Videos
Junyu Lu, Deyi Ji, Liqun Liu et al. (+9 more)
Hateful
videos have become prevalent on online platforms, highlighting an urgent need for effective detection. However, existing studies primarily focus on binary classification and fail to provide contextual rationales that reveal the implicit meanings behind these judgments, significantly undermining model explainability. To fill this gap, we aim to achieve explainable hateful
video detection, enabling models to provide contextual rationales that integrate relevant evidence and logical
reasoning alongside decisions. This approach can comprehensively enhance the understanding of
video content and the explainability of the decision-making process. We first introduce two
datasets, Ex-HateMM and Ex-ImpliHateVid, for explainable hateful
video detection. Each
dataset provides fine-grained annotations of
multimodal harmful elements, along with contextual rationales. We then propose an Information Augmentation and
Reasoning Enhancement (IARE) framework designed for explainable detection. The framework employs an information augmentation phase that leverages the
multimodal chain-of-thought to integrate harmful elements, thereby enriching rationale evidence. Additionally, IARE incorporates a
reasoning enhancement phase, in which Direct Preference Optimization guides the model toward correct
reasoning paths and away from incorrect ones, thereby improving the logical coherence of its justifications. We conduct extensive experiments on the two
datasets, comparing multiple baselines with our proposed IARE framework. The results demonstrate that IARE achieves state-of-the-art performance while also generating accurate rationales.
Read the full abstract ▾
All authors Junyu Lu, Deyi Ji, Liqun Liu, Xiaokun Zhang, Youlin Wu, Roy Ka-Wei Lee, Peng Shu, Huan Yu, Jie Jiang, Bo Xu, Liang Yang, Hongfei Lin
Computation and Language
uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking
Simon Lupart, Kidist Amde Mekonnen, Zahra Abbasiantaeb et al. (+1 more)
This report describes our participation in SemEval-2026 Task 8 on multi-turn
retrieval and question answering. The task evaluates conversational systems across four domains (finance, cloud documentation, government, Wikipedia), and includes unanswerable queries where the available collection does not contain sufficient evidence to produce a complete response. We propose a multi-turn
retrieval-augmented generation pipeline that combines learned sparse
retrieval with LLM-based reranking and generation. Using sparse
retrieval as the primary
retrieval method, we leverage its strong generalization across domains. In addition, we make use of the long-context capabilities of LLMs for conversational query rewriting, pointwise and listwise reranking, and generating the final response, each conditioned on the full conversational history. This multi-step design enables effective integration of conversational context throughout
retrieval and generation, improving robustness across domains.
Read the full abstract ▾
All authors Simon Lupart, Kidist Amde Mekonnen, Zahra Abbasiantaeb, Mohammad Aliannejadi
Computation and Language
Semantic Grading of Written Answers in Low-Resource Language Bangla Using a Fine-Tuned Lightweight Language Model
Meherun Farzana, Aniket Joarder, Mahmudul Hasan et al. (+1 more)
Bangla is among the world's most widely spoken
languages, yet it remains underserved in educational NLP research. In many remote and rural regions, access to qualified subject teachers is limited, and written answers are consequently graded largely by hand, restricting timely and consistent feedback. Automatic assessment is challenging because semantically correct responses can vary substantially in surface form. We present a bilingual (Bangla-English) evaluation system designed for low-resource educational settings that prioritizes semantic correctness over lexical overlap. Our approach fine-tunes a lightweight
language model to grade each response using the question, reference answer, and student answer, producing a numeric score and concise, context-grounded feedback suitable for classroom deployment. We also construct a synthetic bilingual
dataset to enable controlled training and evaluation. Across proprietary and open-source LLMs evaluated under a unified protocol, our QLoRA-tuned Qwen3-8B confirms consistent improvement by producing the most leakage-resistant feedback (RoRa = 0.819) in synthetic evaluation and the strongest agreement with human scores (rho = 0.936, MAE = 0.725) in a dedicated human study.
Read the full abstract ▾
All authors Meherun Farzana, Aniket Joarder, Mahmudul Hasan, Md. Mosaddek Khan
Computation and Language
Toward Generalist Autonomous Research via Hypothesis-Tree Refinement
Jiajie Jin, Yuyang Hu, Kai Qiu et al. (+15 more)
Scientific progress depends on a repeated loop of exploration, experimentation, and abstraction. Researchers test candidate directions, interpret the evidence, and carry the resulting lessons into later attempts. We study how an AI
agent can run this loop autonomously over long horizons. We introduce Arbor, a general framework for autonomous research that combines a long-lived coordinator, short-lived executors, and Hypothesis Tree Refinement (HTR), a persistent tree that links hypotheses, artifacts, evidence, and distilled insights across time. The coordinator manages global research strategy over the tree, while executors implement and test individual hypotheses in isolated worktrees. As results return, Arbor updates the tree, propagates reusable lessons, refines the search frontier, and admits verified improvements. This design turns autonomous research from a sequence of local attempts into a cumulative process in which strategy, execution, and evidence are carried across time. We evaluate Arbor under Autonomous Optimization (AO), an operational setting where an
agent improves an initial research artifact through iterative experimentation without step-level human supervision. Across six real research tasks in model training, harness engineering, and data synthesis, Arbor achieves the best held-out result on all six tasks, attaining more than 2.5x the average relative held-out gain of Codex and Claude Code under the same task interface and resource budget. On MLE-Bench Lite, Arbor reaches 86.36% Any Medal with GPT-5.5, the strongest result in our comparison.
Read the full abstract ▾
All authors Jiajie Jin, Yuyang Hu, Kai Qiu, Qi Dai, Chong Luo, Guanting Dong, Xiaoxi Li, Tong Zhao, Xiaolong Ma, Gongrui Zhang, Zhirong Wu, Bei Liu, Zhengyuan Yang, Linjie Li, Lijuan Wang, Hongjin Qian, Yutao Zhu, Zhicheng Dou
Computation and Language
An Ontology-Guided Multi-Anchor Graph Retrieval Framework for Traffic Legal Liability Determination
Xu Li, Shuqi Tian, Xun Han et al. (+2 more)
Traffic law liability determination is critical for assigning legal penalties, requiring the simultaneous identification of interdependent statutory provisions across multiple legal dimensions. However, existing
retrieval-augmented generation methods suffer from a multi-dimensional
retrieval bottleneck: single axis architectures compress complex legal queries into a single pathway, causing interdependent statutory dimensions to be overlooked. To address this, we propose OMAGR, an ontology-guided framework that decomposes queries into ontology-aligned anchors and executes parallel graph
retrieval across each dimension, ensuring independent
retrieval across dimensions before fusion. To evaluate the proposed method, we created the TrafficLaw-QA
dataset, an expert-validated
benchmark dataset containing 200 questions and 527 legal provisions. Results show that TrafficOmni-RAG outperforms baselines on Context Precision and Faithfulness metrics. The findings demonstrate that parallel multi-anchor
retrieval effectively resolves the multi-dimensional
retrieval bottleneck, offering a promising direction for traffic law liability determination research.
Read the full abstract ▾
All authors Xu Li, Shuqi Tian, Xun Han, Kuncheng Zhao, Xinyi Li
Computation and Language
Xuan Dong, Zhe Han, Tianhao Niu et al. (+2 more)
Vision-
Language-Action (VLA) models have shown strong performance in
language-conditioned robotic manipulation, yet their robustness to linguistic variation remains poorly understood. In this work, we present the first systematic
multilingual evaluation of VLA models by translating the LIBERO
benchmark into ten
languages, revealing severe performance degradation under non-English instructions, with success rates dropping by 30-50%. Through fine-grained analysis of task executions, we find that
language influence is highly non-uniform across steps: certain steps exhibit strong
language dependence and dominate overall task failure, while others are largely
language-agnostic. Based on this insight, we propose a step-wise inference-time intervention that aligns representations according to step
language sensitivity, substantially improving performance under linguistic variation. Our results indicate that
language robustness in VLA models is fundamentally a step-wise control problem, highlighting the importance of temporally structured analysis for reliable
embodied agents.
Read the full abstract ▾
All authors Xuan Dong, Zhe Han, Tianhao Niu, Qingfu Zhu, Wanxiang Che
Computation and Language
GraspLLM: Towards Zero-Shot Generalization on Text-Attributed Graphs with LLMs
Hengyi Feng, Zeang Sheng, Meiyi Qiang et al. (+2 more)
Research on Text-Attributed Graphs (TAGs) has gained significant
attention recently due to its broad applications across various real-world data scenarios, such as citation networks, e-commerce platforms, social media, and web pages. Inspired by the remarkable semantic understanding ability of Large
Language Models (LLMs), there have been numerous attempts to integrate LLMs into TAGs. However, existing methods still struggle to generalize across diverse graphs and tasks, and their ability to capture transferable graph structural patterns remains limited. To address this, we introduce the GraspLLM, a framework that combines Graph structural comprehension with semantic understanding prowess of LLMs to enhance the cross-
dataset and cross-task generalizability. Specifically, we represent node texts from different graphs in a unified semantic space with a frozen general
embedding model, on top of which we perform motif-aware contrastive learning across multiple motif-induced adjacency
matrices to extract
dataset-agnostic structural information. Then, with our proposed optimal contextual subgraph, we extract the most contextually relevant subgraph for each target node and align these subgraphs to the
token space of LLM via an
alignment projector. Extensive experiments on TAG
benchmark datasets spanning diverse domains reveal that GraspLLM consistently outperforms previous LLM-based methods for TAGs, especially in zero-shot scenarios, highlighting its strong generalizability across different
datasets and tasks. Our code is available at https://github.com/Heinz217/GraspLLM.
Read the full abstract ▾
All authors Hengyi Feng, Zeang Sheng, Meiyi Qiang, Yang Li, Wentao Zhang
Computation and Language
Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills
Shi Liu, Jiayao Chen, Chengwei Qin et al. (+3 more)
Scientific discovery workflows rely heavily on lab notes, where researchers record observations, interpret uncertain results, and plan follow-up experiments. Unlike polished publications, lab notes preserve evolving scientific
reasoning, tacit scientific knowledge, and author uncertainty, giving AI
agents access to the process of science. However, most prior work on scientific text focuses on papers, protocols, or structured databases, leaving informal laboratory notes underexplored as inputs to AI
agents for science. This gap matters because tacit knowledge in lab notes is not written as a clean set of instructions: validated observations, tentative judgments, and possible experimental next steps often appear in the same passage. If these signals are conflated, an AI
agent may mistake uncertain scientific judgments for confirmed conclusions or executable actions. To this end, we present Notes2Skills, a two-stage framework for turning lab notebooks into verifiable skills for scientific AI
agents while preserving the author's certainty. Across three wet-lab sessions and seven original downstream configurations, Notes2Skills is the only tested configuration that avoids both observed regime-level failures: over-acting on uncertain notes and missing firm directives. We show that certainty preservation is a key missing piece between lab notebooks and reliable
agent skills, opening a path toward safer AI co-scientist systems.
Read the full abstract ▾
All authors Shi Liu, Jiayao Chen, Chengwei Qin, Yanqing Hu, Jufan Zhang, Linyi Yang
Computation and Language · Cross-listed from cs.LG
Mingqing Xiao, Kai Du and Zhouchen Lin
The correspondence between large
language models (LLMs) and the neural mechanisms underlying human higher-order cognition remains insufficiently characterized. Given that
language and
reasoning in the human
brain appear dissociable, an open question is whether LLMs align with neural signals from
reasoning-related regions and whether such signals can improve them. Here, focusing on deductive
reasoning, we show that LLM internal representations are not only partially aligned with task-fMRI activity but can also be directly enhanced by these signals. Using a neural-predictivity metric, we find that LLMs explain a substantial fraction of the explainable variance in
reasoning-related regions at the aggregate level, whereas predictivity within specific
reasoning types is lower, indicating both
alignment and divergence. Building on this, we propose a
brain-guided framework: we steer model representations along directions induced by the joint structure of model and
brain representations, applying intervention at inference and fine-tuning during training. We demonstrate that task-evoked
brain signals can directly enhance LLM
reasoning, yielding gains orthogonal to
language-only supervision across 10 LLMs (1.5B-72B), with transfer across
reasoning types and up to 13% absolute accuracy gain. Our results advance LLM-
brain correspondences from correlation to guidance, establishing a
brain-signal-driven pathway toward more robust and cognitively aligned AI.
Read the full abstract ▾
Computation and Language
I Understand How You Feel: Enhancing Deeper Emotional Support Through Multilingual Emotional Validation in Dialogue System
Zi Haur Pang, Yahui Fu, Koji Inoue et al. (+1 more)
Emotional validation - explicitly acknowledging that a user's feelings make sense - has proven therapeutic value but has received little computational
attention. Emotional validation in dialogue systems can be decomposed into (i) validating response identification, (ii) validation timing detection, and (iii) validating response generation. To support research on all three subtasks, we release M-EDESConv, a 120k English-Japanese
multilingual corpus created through hybrid manual and automatic annotation, and M-TESC, a
multilingual spoken-dialogue test set. For timing detection, we propose MEGUMI, a
Multilingual Emotion-aware Gated Unit for Mutual Integration, that fuses frozen XLM-RoBERTa semantics with
language-specific
emotion encoders via cross-modal
attention and gated fusion. MEGUMI shows superior performance on both the M-EDESConv and M-TESC
datasets, both objectively and subjectively. Finally, our EmoValidBench
benchmarks of GPT-4.1 Nano and Llama-3.1 8B indicate that current LLMs generate contextually similar and diverse validating responses, but emotional understanding remains a major area for improvement. Project page: https://github.com/zihaurpang/
Multilingual-Emotional-Validation
Read the full abstract ▾
All authors Zi Haur Pang, Yahui Fu, Koji Inoue, Tatsuya Kawahara
Computation and Language · Cross-listed from cs.LG
Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training
Michal Chudoba, Sergey Alyaev, Petra Galuscakova et al. (+1 more)
There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large
Language Models (LLMs). While Low-Rank Adaptation (LoRA) introduces additional weights between the LLM layers, Soft Prompting introduces additional fine-tuning-specific raw
tokens to an LLM input. However, both require modification to the computational graphs of precompiled, preoptimized LLMs. As a result, neither is fully supported in high-throughput engines like vLLM. We propose fine-tuning with ART (Art-based Reinforcement Training). The method injects information into a frozen
Multimodal Large
Language Model (MLLM) by optimizing only its raw visual input, thus enabling the soft-
token approach on pre-compiled computational graphs. It relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective. Moreover, the optimized visual input can be stylized as task-relevant computational artworks. The approach's effectiveness is confirmed for different sizes of a popular open Qwen architecture and for several textual
benchmarks. Specifically, ART reaches accuracy competitive with LoRA across mathematics and structured-tool-use
benchmarks.
Read the full abstract ▾
All authors Michal Chudoba, Sergey Alyaev, Petra Galuscakova, Tomasz Wiktorski
Computation and Language · Cross-listed from cs.CR
Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code
Yitong Zhang, Shiteng Lu and Jia Li
Large
Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety
alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-
language refusals when natural
language is available. Experiments on 10 popular LLMs across 4
benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater
attention to its potential security implications.
Read the full abstract ▾
Computation and Language
WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning
Yizhou Chi, Eric Chamoun, Zifeng Ding et al. (+1 more)
Forecasting real-world events requires
language-model
agents to reason under uncertainty from incomplete, time-bounded information. Yet evaluating whether
agents genuinely forecast requires more than final-answer accuracy: a model may be correct by recalling memorized training facts, citing fabricated evidence, or producing an unsupported causal story. We present WorldReasoner, an evaluation framework for temporally valid event
forecasting. Each task gives an
agent a resolved
forecasting question, a simulated forecast date, and access only to evidence available before that date; after resolution, the framework scores the submitted probability, cited evidence, and optional causal event graph. WorldReasoner reports three complementary axes: outcome quality against resolved answers, evidence quality over cited sources, and
reasoning quality against post-resolution hindsight graphs. The
benchmark is built by an agentic construction pipeline that generates
forecasting questions, collects time-stamped evidence, and builds hindsight reference graphs at scale, yielding 345 resolved tasks derived from 14,141 articles with graphs covering 8,087 extracted events. Across six controlled
agent settings, temporally valid
retrieval is the strongest driver of outcome accuracy; causal graph construction improves key-event recovery; and correct graph-enabled forecasts are more strongly grounded in key events and relevant sources, yet
agents still struggle to convert grounded evidence into calibrated probabilities.
Read the full abstract ▾
All authors Yizhou Chi, Eric Chamoun, Zifeng Ding, Andreas Vlachos
Computation and Language
External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs
Lin Sun, Heming Zhang and Xiangzheng Zhang
Production LLM systems accumulate reusable operational experience, but the practical deployment issue is not merely whether such experience can help. It is how different serving strategies trade off quality against online cost under realistic constraints. Injecting external experience can improve task quality, yet it also increases
prompt burden, latency, and serving pressure. We study external experience serving as a deployment-oriented quality-cost trade-off problem. We evaluate this question in a real production moderation setting, with tool-use and GPQA as supporting contrast tasks that expose different output-cost regimes. We compare no-experience baselines, random experience controls, global
prompt injection, and
retrieval-based selective injection, and analyze both task quality and serving cost. The results show that, once experience becomes case-dependent, selective
retrieval provides a stronger operating point than unconditional global injection. They further show that
retrieval quality matters more than simply increasing Top-$K$, and that the same serving policy can exhibit substantially different cost-benefit profiles across short-output and decode-heavy regimes. These findings suggest that external experience is best treated as a selective, cost-aware serving decision rather than as a universal add-on. Overall, in the settings studied here, external experience pays off only when both the serving interface and the task-specific cost structure make its quality gains worth the online cost.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Yuansheng Gao, Wenbin Xing, Jiahao Yuan et al. (+4 more)
Video Large
Multimodal Models have achieved remarkable progress in
video understanding, yet they remain prone to
hallucinations, where generated responses are not faithfully supported by the input
video. In this paper, we propose MultiToP, a
multimodal-context-aware visual
token patching framework that mitigates
hallucinations by refining unreliable visual
tokens before
language generation. MultiToP introduces a lightweight Visual
Token Patcher to predict
token-level replacement distributions and selectively substitute unreliable visual
tokens with a dynamic global patch
token. To train the patcher effectively, we further propose information-guided rank calibration, which uses answer-conditioned frame-level information cues derived from the backbone to guide
token replacement. Combined with ground-truth answer supervision and sparsity regularization, MultiToP enables localized visual evidence refinement without modifying the original model. Extensive experiments demonstrate that MultiToP effectively reduces
hallucinations on Vript-HAL with negligible inference overhead, improving the F1 scores of Qwen3-VL-4B-Instruct by 50.60% over the vanilla model. Meanwhile, MultiToP preserves general
video understanding ability, yielding an 18.58% relative accuracy gain on ActivityNet-QA for
Video-LLaVA-7B.
Read the full abstract ▾
All authors Yuansheng Gao, Wenbin Xing, Jiahao Yuan, Kaiwen Zhou, Han Bao, Zonghui Wang, Wenzhi Chen
Computation and Language
Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malay
Joanito Agili Lopo, Yunita Sari and Guntur Budi Herwanto
Large
Language Models (LLMs) offer new potential for translation tasks but often experience performance degradation when handling low-resource
languages. To address this limitation, we propose an approach for fine-tuning LLMs on a low-resource
language, Kupang Malay. Our approach involves designing a set of instructions by leveraging explicit lexical and semantic features from a bilingual dictionary, and introducing Continual Instruction Tuning (CIT), a training paradigm that enables iterative instruction-based training. Experimental results demonstrate that our model, named Lius, yields notable improvements over standard instruction-tuned models by outperforming 4-6 points, and surpassing both Neural Machine Translation (NMT) and
Multilingual LLM models by 10-13 points on several evaluation metrics. These findings highlight the potential of our approach to mitigate the reliance on large-scale parallel data in low-resource
language translation.
Read the full abstract ▾
Computation and Language · Cross-listed from eess.AS
Fast Speech Foundation Model Distillation Using Interleaved Stacking
Eungbeom Kim and Kyogu Lee
Distilling a large
speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although
distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM
distillation remains underexplored. In this work, we explore training acceleration of SFM
distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.
Read the full abstract ▾
Computation and Language
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff et al. (+4 more)
Large
language models (LLMs) have achieved remarkable progress in
language understanding,
reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks. However, most existing creativity metrics are tightly coupled to specific tasks,
embedding domain assumptions into the evaluation process, and limiting scalability and generality. To address this gap, we introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks. Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment. Divergent creativity is measured using semantic
entropy, a reference-free and robust metric for novelty and diversity, validated against human annotations, LLM-based novelty judgments and baseline diversity measures. Convergent creativity is assessed via a novel
retrieval-based multi-
agent judge framework that delivers context-sensitive evaluation of task fulfilment with over 60% improved efficiency. We validate our framework in three qualitatively distinct domains: problem-solving (MacGyver), research ideation (HypoGen), and creative writing (BookMIA), using a broad suite of LLMs. Empirical results show that our framework reliably captures key facets of creativity, including novelty, diversity, and task fulfilment, and reveal how model properties, such as size, temperature, recency, and
reasoning, impact creative performance. Our work establishes a reproducible and generalizable standard for automated LLM creativity evaluation, paving the way for scalable benchmarking and accelerating progress in creative AI.
Read the full abstract ▾
All authors Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Mohor Banerjee, Swaagat Bikash Saikia, Alvin Chan
Computation and Language
Hey Chat, Can You Teach Me? Structuring Socratic Dialogue for Human Learning in the Wild
Sidney Tio, Arunesh Sinha and Pradeep Varakantham
Large
language models are now widely used for everyday learning, but the underlying interactions are typically unstructured chats rather than following a curriculum. Unlike formal online learning systems, these interactions carry no prior record of the student, so any estimate of what the student already knows must be inferred from the dialogue itself. We show that this gap is not closed by scaling models alone. Frontier and education-tuned LLMs perform poorly when asked to tutor a student over an extended session, because doing so requires three things at once. The tutor must sequence a curriculum, conduct Socratic dialogue, and infer the student's knowledge state from that dialogue. We propose separating these responsibilities. Given a student query, our system constructs a prerequisite knowledge graph in which subtopics are nodes and dependencies are edges, and frames tutoring as deciding which node to teach next and how many dialogue turns to spend on it before moving on. A lightweight PPO policy handles this sequencing decision, while an LLM conducts the Socratic exchange at the chosen node and returns a signal of student progress. Across held-out STEM and non-STEM topics, our PPO-paired tutor outperforms heuristic baselines, frontier general-purpose models, and a model specialised for Socratic dialogue: on both the rate at which students reach full curriculum mastery and the number of turns required. Explicit curriculum structure delivers gains that scaling the underlying model does not.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
UniReason-Med: A Shared Grounded Reasoning Interface for 2D-to-3D Transfer in Medical VQA
Mengzhuo Chen, Yan Shu, Chi Liu et al. (+4 more)
We study whether grounded
reasoning supervision from abundant 2D medical images can improve 3D medical VQA when both input types are aligned through a common
reasoning interface. We introduce UniReason-Med, a single-checkpoint framework that processes either a 2D image or a slice-serialized 3D volume at inference time, generating interleaved textual
reasoning and localized visual evidence through shared box syntax, region-
token injection, and a common grounded
reasoning policy. To train this interface, we construct UniMed-CoT, a 220K instruction-tuning
dataset with interleaved textual
reasoning and grounded visual evidence, including 170K 2D and 50K 3D samples. Through supervised fine-tuning followed by outcome-level reinforcement learning, UniReason-Med learns to generate grounded
reasoning traces without IoU/Dice-based localization rewards during RL. Data-mixture and component ablations show that joint 2D+3D grounded supervision substantially improves 3D
reasoning over 3D-only training, while grounding and region-
token injection consistently benefit both 2D and 3D tasks. These results suggest that a shared grounded
reasoning interface can transfer
reasoning structure from 2D images to slice-serialized volumetric medical understanding. The code and data are publicly available at https://github.com/IQuestLab/unireason-med.
Read the full abstract ▾
All authors Mengzhuo Chen, Yan Shu, Chi Liu, Hongming Piao, Xidong Wang, Derek Li, Bryan Dai
Computation and Language · Cross-listed from cs.LG
Non-Gaussianity Reveals Interpretable Directions in Language Models
Sida Liu, Feijiang Han, Sarah Wiegreffe et al. (+1 more)
Finding human-interpretable directions in
language-model activations is important for understanding and controlling model behavior. These directions are usually discovered through indirect learning objectives such as sparse reconstruction, but it remains unclear what existing statistical structure distinguishes interpretable directions and whether that structure can be used directly for feature discovery. We find that non-Gaussianity provides such a signature: projection distributions of natural-text residual-stream activations along sparse autoencoder (SAE) decoder directions are substantially more non-Gaussian than those along random directions, with most contexts forming a concentrated background and recurring patterns often occupying one tail. This raises a natural question: if interpretable directions tend to have non-Gaussian projection distributions, can we discover them by searching for non-Gaussian directions directly in activation space, without training another sparse dictionary? To test this hypothesis, we introduce ICA Lens, a practical workflow for efficient discovery, reliable annotation, and faithful evaluation of non-Gaussian directions in LLM residual-stream activations. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find that: (1) the recovered non-Gaussian directions capture coherent lexical, syntactic, and semantic patterns under both human and LLM-based evaluation; (2) non-Gaussianity is positively associated with both human-label confidence and automated interpretability; and (3) the resulting coordinates are competitive with SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7-43 times less
memory. These results show that non-Gaussianity is a direct unsupervised signal for discovering interpretable and useful structure in activations.
Read the full abstract ▾
All authors Sida Liu, Feijiang Han, Sarah Wiegreffe, Furong Huang
Computation and Language
Substrate Asymmetry in User-Side Memory: A Diagnostic Framework
Youwang Deng
User-side
memory in LLMs is typically scored as a single "personalization" capability: given a user's history, is the output more user-aware? We show this aggregate metric hides opposite-direction failures.
Memory factorises into at least three orthogonal axes – behavioral consistency (style, voice), factual presence (recall facts in history), and factual absence (abstain when a fact is absent) – and no single substrate wins all three. Comparing per-user gamma-LoRA (a small LoRA adapter trained on each user's history; gamma denotes per-user, not per-task) against BGE-large dense top-K
retrieval on a controlled 50-user synthetic corpus and a real-data probe (LaMP-3), we find gamma-LoRA decisively wins behavioral style while RAG decisively wins factual absence – and the same query-projection
cells in
attention layers 21-35 causally load-bear both effects in opposite directions (zeroing those LoRA weights raises absence-probe TPR by +33 pp and drops presence-probe TPR by 20 pp). On the more heavily RLHF-tuned Llama-3.1-8B-Instruct the asymmetry strengthens, not heals: parametric
memory's behavioral advantage collapses while its absence-calibration deficit against
retrieval widens – an
alignment tax on parametric user-
memory. On real-data LaMP-3, gamma-LoRA underperforms a majority baseline; a 9-condition mitigation sweep diagnoses this as instruction-following collapse, not substrate failure (a 9x2 cross-product shows the eval-time {1..5} logit mask drives main_acc to >=0.995 on every recipe), and the best training-time fix replicates bit-identically on Llama. Finally, substrate-selection routing is question-classification, not calibration: a 110M DistilBERT on the question text alone beats every logit-based router. We contribute the diagnostic framework, the diagnosed real-data negative, the
alignment-tax replication, and the routing-as-classification finding.
Read the full abstract ▾