Computation and Language
Beyond Tokens in Language Models: Interpreting Activations through Text Genre Chunks
Éloïse Benito-Rodriguez, Einar Urdshals, Jasmina Nasufi et al. (+1 more)
Understanding Large
Language Models (LLMs) is key to ensure their safe and beneficial deployment. This task is complicated by the difficulty of interpretability of LLM structures, and the inability to have all their outputs human-evaluated. In this paper, we present the first step towards a predictive framework, where the genre of a text used to
prompt an LLM, is predicted based on its activations. Using Mistral-7B and two
datasets, we show that genre can be extracted with F1-scores of up to 98% and 71% using scikit-learn classifiers. Across both
datasets, results consistently outperform the control task, providing a proof of concept that text genres can be inferred from LLMs with shallow learning models.
Read the full abstract ▾
All authors Éloïse Benito-Rodriguez, Einar Urdshals, Jasmina Nasufi, Nicky Pochinkov
Computation and Language
TurkColBERT: A Benchmark of Dense and Late-Interaction Models for Turkish Information Retrieval
Özay Ezerceli, Mahmoud El Hussieni, Selva Taş et al. (+4 more)
Neural information
retrieval systems excel in high-resource
languages but remain underexplored for morphologically rich, lower-resource
languages such as Turkish. Dense bi-encoders currently dominate Turkish IR, yet late-interaction models – which retain
token-level representations for fine-grained matching – have not been systematically evaluated. We introduce TurkColBERT, the first comprehensive
benchmark comparing dense encoders and late-interaction models for Turkish
retrieval. Our two-stage adaptation pipeline fine-tunes English and
multilingual encoders on Turkish NLI/STS tasks, then converts them into ColBERT-style retrievers using PyLate trained on MS MARCO-TR. We evaluate 10 models across five Turkish BEIR
datasets covering scientific, financial, and argumentative domains. Results show strong parameter efficiency: the 1.0M-parameter colbert-hash-nano-tr is 600$\times$ smaller than the 600M turkish-e5-large dense encoder while preserving over 71% of its average mAP. Late-interaction models that are 3–5$\times$ smaller than dense encoders significantly outperform them; ColmmBERT-base-TR yields up to +13.8% mAP on domain-specific tasks. For production-readiness, we compare indexing
algorithms: MUVERA+Rerank is 3.33$\times$ faster than PLAID and offers +1.7% relative mAP gain. This enables low-latency
retrieval, with ColmmBERT-base-TR achieving 0.54 ms query times under MUVERA. We release all checkpoints, configs, and evaluation scripts. Limitations include reliance on moderately sized
datasets ($\leq$50K documents) and translated
benchmarks, which may not fully reflect real-world Turkish
retrieval conditions; larger-scale MUVERA evaluations remain necessary.
Read the full abstract ▾
All authors Özay Ezerceli, Mahmoud El Hussieni, Selva Taş, Reyhan Bayraktar, Fatma Betül Terzioğlu, Yusuf Çelebi, Yağ{\i}z Asker
Computation and Language · Cross-listed from cs.RO
Xiaoshuai Hao, Lei Zhou, Zhijian Huang et al. (+41 more)
We open-source
MiMo-
Embodied, the first cross-
embodied foundation model to successfully integrate and achieve state-of-the-art performance in both Autonomous Driving and
Embodied AI.
MiMo-
Embodied sets new records across 17
embodied AI
benchmarks in Task Planning, Affordance Prediction and Spatial Understanding, while also excelling in 12 autonomous driving
benchmarks across Environmental Perception, Status Prediction, and Driving Planning. Across these tasks,
MiMo-
Embodied significantly outperforms existing open-source, closed-source, and specialized baselines. Our results indicate that through multi-stage learning, curated data construction, and CoT/RL fine-tuning, these two domains exhibit strong positive transfer and mutually reinforce one another. We provide a detailed analysis of our model design and training methodologies to facilitate further research. Code and models are available at https://github.com/XiaomiMiMo/
MiMo-
Embodied.
Read the full abstract ▾
All authors Xiaoshuai Hao, Lei Zhou, Zhijian Huang, Zhiwen Hou, Yingbo Tang, Lingfeng Zhang, Guang Li, Zheng Lu, Shuhuai Ren, Xianhui Meng, Yuchen Zhang, Jing Wu, Jinghui Lu, Chenxu Dang, Jiayi Guan, Jianhua Wu, Zhiyi Hou, Hanbing Li, Shumeng Xia, Mingliang Zhou, Yinan Zheng, Zihao Yue, Shuhao Gu, Hao Tian, Yuannan Shen, Jianwei Cui, Wen Zhang, Shaoqing Xu, Bing Wang, Haiyang Sun, Zeyu Zhu, Yuncheng Jiang, Zibin Guo, Chuhong Gong, Chaofan Zhang, Wenbo Ding, Kun Ma, Guang Chen, Rui Cai, Diyun Xiang, Heng Qu, Fuli Luo, Hangjun Ye, Long Chen
Computation and Language · Cross-listed from cs.IR
Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluation
Elena V. Epure, Yashar Deldjoo, Bruno Sguerra et al. (+2 more)
Music Recommender Systems (MRSs) have long relied on an information
retrieval framing, where progress is measured mainly through accuracy on
retrieval-oriented subtasks. While effective, this reductionist paradigm struggles to address the deeper question of what makes a good recommendation. Attempts to broaden evaluation, through user studies or
fairness analyses, have had limited impact. The emergence of Large
Language Models (LLMs) disrupts this framework: LLMs are generative rather than ranking-based, making standard accuracy metrics questionable. They also introduce challenges such as
hallucinations, knowledge cutoffs, non-determinism, and opaque training data, rendering traditional train/test protocols difficult to interpret. At the same time, LLMs create new opportunities, enabling natural
language (NL) interaction and even allowing models to act as evaluators. This work argues that the shift toward LLM-driven MRSs requires rethinking evaluation. We first review how LLMs have impacted user modeling, item modeling, and NL-based recommendation in music. We then analyze evaluation practices from NLP, highlighting methodologies and open challenges relevant to MRSs. Finally, we synthesize insights, focusing on how LLM prompting applies to MRSs, to outline a structured set of success and risk dimensions. Our goal is to provide the MRSs community with an updated, pedagogical, and cross-disciplinary perspective on evaluation.
Read the full abstract ▾
All authors Elena V. Epure, Yashar Deldjoo, Bruno Sguerra, Markus Schedl, Manuel Moussallam
Computation and Language
Arctic-Extract Technical Report
Mateusz Chiliński, Julita Ołtusek and Wojciech Jaśkowski
Arctic-Extract is a state-of-the-art model designed for extracting structural data (question answering, entities and tables) from scanned or digital-born business documents. Despite its SoTA capabilities, the model is deployable on resource-constrained hardware, weighting only 6.6 GiB, making it suitable for deployment on devices with limited resources, such as A10 GPUs with 24 GB of
memory. Arctic-Extract can process up to 125 A4 pages on those GPUs, making suitable for long document processing. This paper highlights Arctic-Extract's training protocols and evaluation results, demonstrating its strong performance in document understanding.
Read the full abstract ▾
Computation and Language
Anatomy of an Idiom: Tracing Non-Compositionality in Language Models
Andrew Gomes
We investigate the processing of idiomatic expressions in
transformer-based
language models using a novel set of techniques for
circuit discovery and analysis. First discovering
circuits via a modified path patching
algorithm, we find that idiom processing exhibits distinct computational patterns. We identify and investigate “Idiom Heads,”
attention heads that frequently activate across different idioms, as well as enhanced
attention between idiom
tokens due to earlier processing, which we term “augmented reception.” We analyze these phenomena and the general features of the discovered
circuits as mechanisms by which
transformers balance computational efficiency and robustness. Finally, these findings provide insights into how
transformers handle non-compositional
language and suggest pathways for understanding the processing of more complex grammatical constructions.
Read the full abstract ▾
Computation and Language
ESGBench: A Benchmark for Explainable ESG Question Answering in Corporate Sustainability Reports
Sherine George and Nithish Saji
We present ESGBench, a
benchmark dataset and evaluation framework designed to assess explainable ESG question answering systems using corporate sustainability reports. The
benchmark consists of domain-grounded questions across multiple ESG themes, paired with human-curated answers and supporting evidence to enable fine-grained evaluation of model
reasoning. We analyze the performance of state-of-the-art LLMs on ESGBench, highlighting key challenges in factual consistency, traceability, and domain
alignment. ESGBench aims to accelerate research in transparent and accountable ESG-focused AI systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Models
Li Zhang, Zhongxuan Han, XiaoHua Feng et al. (+5 more)
Efficient and lightweight adaptation of pre-trained
Vision-
Language Models (VLMs) to downstream tasks through collaborative interactions between local clients and a central server is a rapidly emerging research topic in
federated learning. Existing adaptation
algorithms are typically trained iteratively, which incur significant communication costs and increase the susceptibility to potential attacks. Motivated by the one-shot
federated training techniques that reduce client-server exchanges to a single round, developing a lightweight one-shot
federated VLM adaptation method to alleviate these issues is particularly attractive. However, current one-shot approaches face certain challenges in adapting VLMs within
federated settings: (1) insufficient exploitation of the rich
multimodal information inherent in VLMs; (2) lack of specialized adaptation strategies to systematically handle the severe data heterogeneity; and (3) requiring additional training resource of clients or server. To bridge these gaps, we propose a novel Training-free One-shot
Federated Adaptation framework for VLMs, named TOFA. To fully leverage the generalizable
multimodal features in pre-trained VLMs, TOFA employs both visual and textual pipelines to extract task-relevant representations. In the visual pipeline, a hierarchical
Bayesian model learns personalized, class-specific prototype distributions. For the textual pipeline, TOFA evaluates and globally aligns the generated local text
prompts for robustness. An adaptive weight calibration mechanism is also introduced to combine predictions from both modalities, balancing personalization and robustness to handle data heterogeneity. Our method is training-free, not relying on additional training resources on either the client or server side. Extensive experiments across 9
datasets in various
federated settings demonstrate the effectiveness of the proposed TOFA method.
Read the full abstract ▾
All authors Li Zhang, Zhongxuan Han, XiaoHua Feng, Jiaming Zhang, Yuyuan Li, Linbo Jiang, Jianan Lin, Chaochao Chen
Computation and Language
Classification of worldwide news articles by perceived quality, 2018-2024
Connor McElroy, Thiago E. A. de Oliveira and Chris Brogly
This study explored whether supervised machine learning and deep learning models can effectively distinguish perceived lower-quality news articles from perceived higher-quality news articles. 3 machine learning classifiers and 3 deep learning models were assessed using a newly created
dataset of 1,412,272 English news articles from the Common Crawl over 2018-2024. Expert consensus ratings on 579 source websites were split at the median, creating perceived low and high-quality classes of about 706,000 articles each, with 194 linguistic features per website-level labelled article. Traditional machine learning classifiers such as the Random Forest demonstrated capable performance (0.7355 accuracy, 0.8131 ROC AUC). For deep learning, ModernBERT-large (256 context length) achieved the best performance (0.8744 accuracy; 0.9593 ROC-AUC; 0.8739 F1), followed by DistilBERT-base (512 context length) at 0.8685 accuracy and 0.9554 ROC-AUC. DistilBERT-base (256 context length) reached 0.8478 accuracy and 0.9407 ROC-AUC, while ModernBERT-base (256 context length) attained 0.8569 accuracy and 0.9470 ROC-AUC. These results suggest that the perceived quality of worldwide news articles can be effectively differentiated by traditional CPU-based machine learning classifiers and deep learning classifiers.
Read the full abstract ▾
Computation and Language
AICC: Parse HTML Finer, Make Models Better – A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser
Ren Ma, Jiantao Qiu, Chao Xu et al. (+26 more)
While web data quality is crucial for large
language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter
language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our
benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8% ROUGE-N F1 compared to Trafilatura's 63.6%, with exceptional structured element preservation (90.9% for code blocks, 94.0% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion
token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B
tokens) achieve 50.8% average accuracy across 13
benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key
benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction.
Read the full abstract ▾
All authors Ren Ma, Jiantao Qiu, Chao Xu, Pei Chu, Kaiwen Liu, Pengli Ren, Yuan Qu, Jiahui Peng, Linfeng Hou, Mengjie Liu, Lindong Lu, Wenchang Ning, Jia Yu, Rui Min, Jin Shi, Haojiong Chen, Peng Zhang, Wenjian Zhang, Qian Jiang, Zengjie Hu, Guoqiang Yang, Zhenxiang Li, Fukai Shang, Runyuan Ma, Chenlin Su, Zhongying Tu, Wentao Zhang, Dahua Lin, Conghui He
Computation and Language
Learning from Sufficient Rationales: Analysing the Relationship Between Explanation Faithfulness and Token-level Regularisation Strategies
Jonathan Kamp, Lisa Beinborn and Antske Fokkens
Human explanations of natural
language, rationales, form a tool to assess whether models learn a label for the right reasons or rely on
dataset-specific shortcuts. Sufficiency is a common metric for estimating the informativeness of rationales, but it provides limited insight into the effects of rationale information on model performance. We address this limitation by relating sufficiency to two modelling paradigms: the ability of models to identify which
tokens are part of the rationale (through
token classification) and the ability of improving model performance by incorporating rationales in the input (through
attention regularisation). We find that highly informative rationales are not likely to help classify the instance correctly. Sufficiency conversely captures the classification impact of the non-rationalised context, which interferes with rationale information in the same input. We also find that incorporating rationale information in model inputs can boost cross-domain classification, but results are inconsistent per task and model type. Finally, sufficiency and
token classification appear to be unrelated. These results exemplify the complexity of rationales, showing that metrics capable of systematically capturing this type of information merit further investigation.
Read the full abstract ▾
Computation and Language
NLP Datasets for Idiom and Figurative Language Tasks
Blake Matheny, Phuong Minh Nguyen, Minh Le Nguyen et al. (+1 more)
Idiomatic and figurative
language form a large portion of colloquial
speech and writing. With social media, this informal
language has become more easily observable to people and trainers of large
language models (LLMs) alike. While the advantage of large corpora seems like the solution to all machine learning and Natural
Language Processing (NLP) problems, idioms and figurative
language continue to elude LLMs. Finetuning approaches are proving to be optimal, but better and larger
datasets can help narrow this gap even further. The
datasets presented in this paper provide one answer, while offering a diverse set of categories on which to build new models and develop new approaches. A selection of recent idiom and figurative
language datasets were used to acquire a combined idiom list, which was used to retrieve context sequences from a large corpus. One large-scale
dataset of potential idiomatic and figurative
language expressions and two additional human-annotated
datasets of definite idiomatic and figurative
language expressions were created to evaluate the baseline ability of pre-trained
language models in handling figurative meaning through idiom recognition (detection) tasks. The resulting
datasets were post-processed for model agnostic training compatibility, utilized in training, and evaluated on slot labeling and sequence tagging.
Read the full abstract ▾
All authors Blake Matheny, Phuong Minh Nguyen, Minh Le Nguyen, Stephanie Reynolds
Computation and Language · Cross-listed from cs.AI
OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
Kaichen Zhang, Keming Wu, Zuhao Yang et al. (+6 more)
Recent advancements in large
reasoning models have fueled growing interest in extending such capabilities to
multimodal domains. However, despite notable progress in visual
reasoning, the lack of transparent and reproducible data curation and training strategies remains a major barrier to scalable research. In this work, we introduce OpenMMReasoner, a fully transparent two-stage recipe for
multimodal reasoning spanning supervised fine-tuning (SFT) and reinforcement learning (RL). In the SFT stage, we construct an 874K-sample cold-start
dataset with rigorous step-by-step validation, providing a strong foundation for
reasoning capabilities. The subsequent RL stage leverages a 74K-sample
dataset across diverse domains to further sharpen and stabilize these abilities, resulting in a more robust and efficient learning process. Extensive evaluations demonstrate that our training recipe not only surpasses strong baselines but also highlights the critical role of data quality and training design in shaping
multimodal reasoning performance. Notably, our method achieves a 11.6% improvement over the Qwen2.5-VL-7B-Instruct baseline across nine
multimodal reasoning benchmarks, establishing a solid empirical foundation for future large-scale
multimodal reasoning research. We open-sourced all our codes, pipeline, and data at https://github.com/EvolvingLMMs-Lab/OpenMMReasoner.
Read the full abstract ▾
All authors Kaichen Zhang, Keming Wu, Zuhao Yang, Bo Li, Kairui Hu, Bin Wang, Ziwei Liu, Xingxuan Li, Lidong Bing
Computation and Language
Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
Jiashu Yao, Heyan Huang, Shuang Zeng et al. (+6 more)
Through reinforcement learning (RL) with outcome correctness rewards, large
reasoning models (LRMs) with scaled inference computation have demonstrated substantial success on complex
reasoning tasks. However, the one-sided reward, focused solely on final correctness, limits its ability to provide detailed supervision over internal
reasoning process. This deficiency leads to suboptimal internal
reasoning quality, manifesting as issues like over-thinking, under-thinking, redundant-thinking, and disordered-thinking. Inspired by the recent progress in LRM self-rewarding, we introduce self-rewriting framework, where a model rewrites its own
reasoning texts, and subsequently learns from the rewritten
reasoning to improve the internal thought process quality. For
algorithm design, we propose a selective rewriting approach wherein only "simple" samples, defined by the model's consistent correctness, are rewritten, thereby preserving all original reward signals of GRPO. For practical implementation, we compile rewriting and vanilla generation within one single batch, maintaining the scalability of the RL
algorithm and introducing only ~10% overhead. Extensive experiments on diverse tasks with different model sizes validate the effectiveness of self-rewriting. In terms of the accuracy-length tradeoff, the self-rewriting approach achieves improved accuracy (+0.6) with substantially shorter
reasoning (-46%) even without explicit instructions in rewriting
prompts to reduce
reasoning length, outperforming existing strong baselines. In terms of internal
reasoning quality, self-rewriting achieves significantly higher scores (+7.2) under the LLM-as-a-judge metric, successfully mitigating internal
reasoning flaws.
Read the full abstract ▾
All authors Jiashu Yao, Heyan Huang, Shuang Zeng, Chuwei Luo, WangJie You, Jie Tang, Qingsong Liu, Yuhang Guo, Yangyang Kang
Computation and Language
SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
Wei Xia and Zhi-Hong Deng
With the rapid advancement of large
language models (LLMs), their deployment in real-world applications has become increasingly widespread. LLMs are expected to deliver robust performance across diverse tasks, user preferences, and practical scenarios. However, as demands grow, ensuring that LLMs produce responses aligned with human intent remains a foundational challenge. In particular, aligning model behavior effectively and efficiently during inference, without costly retraining or extensive supervision, is both a critical requirement and a non-trivial technical endeavor. To address the challenge, we propose SDA (Steering-Driven Distribution
Alignment), a training-free and model-agnostic
alignment framework designed for open-source LLMs. SDA dynamically redistributes model output probabilities based on user-defined
alignment instructions, enhancing
alignment between model behavior and human intents without fine-tuning. The method is lightweight, resource-efficient, and compatible with a wide range of open-source LLMs. It can function independently during inference or be integrated with training-based
alignment strategies. Moreover, SDA supports personalized preference
alignment, enabling flexible control over the model response behavior. Empirical results demonstrate that SDA consistently improves
alignment performance across 8 open-source LLMs with varying scales and diverse origins, evaluated on three key
alignment dimensions, helpfulness, harmlessness, and honesty (3H). Specifically, SDA achieves average gains of 64.4% in helpfulness, 30% in honesty and 11.5% in harmlessness across the tested models, indicating its effectiveness and generalization across diverse models and application scenarios.
Read the full abstract ▾
Computation and Language
SeSE: Black-Box Uncertainty Quantification for Large Language Models Based on Structural Information Theory
Xingtao Zhao, Hao Peng, Dingli Su et al. (+4 more)
Reliable uncertainty quantification (UQ) is essential for deploying large
language models (LLMs) in safety-critical scenarios, as it enables them to abstain from responding when uncertain, thereby avoiding
hallucinations, i.e., plausible yet factually incorrect responses. However, while semantic UQ methods have achieved advanced performance, they overlook latent semantic structural information that could enable more precise uncertainty estimates. In this paper, we propose \underline{Se}mantic \underline{S}tructural \underline{E}ntropy ({SeSE}), a principled black-box UQ framework applicable to both open- and closed-source LLMs. To reveal the intrinsic structure of the semantic space, SeSE constructs its optimal hierarchical abstraction through an encoding tree with minimal structural
entropy. The structural
entropy of this encoding tree thus quantifies the inherent uncertainty within LLM semantic space after optimal compression. Additionally, unlike existing methods that primarily focus on simple short-form generation, we extent SeSE to provide interpretable, granular uncertainty estimation for long-form outputs. We theoretically prove that SeSE generalizes semantic
entropy, the gold standard for UQ in LLMs, and empirically demonstrate its superior performance over strong baselines across 24 model-
dataset combinations.
Read the full abstract ▾
All authors Xingtao Zhao, Hao Peng, Dingli Su, Xianghua Zeng, Chunyang Liu, Jinzhi Liao, Philip S. Yu
Computation and Language · Cross-listed from cs.CV
Can MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactions
Caixin Kang, Yifei Huang, Liangyang Ouyang et al. (+3 more)
Despite their advanced
reasoning capabilities, state-of-the-art
Multimodal Large
Language Models (MLLMs) still lack a core component of human intelligence: the ability to "read the room" and assess deception in complex social interactions. To rigorously evaluate this limitation, we introduce
Multimodal Interactive Deception Assessment (MIDA), a new task and
multimodal dataset comprising synchronized
video and text with verifiable ground-truth labels for every statement. We
benchmark 12 state-of-the-art open- and closed-source MLLMs and identify a substantial performance gap: even powerful models such as GPT-4o struggle to reliably distinguish truthful statements from deceptive ones. Our analysis shows that current models have difficulty grounding
language in
multimodal social cues and modeling what others know, believe, or intend. To address these limitations, we propose a Social Chain-of-Thought (SoCoT)
reasoning pipeline and a Dynamic Social Epistemic
Memory (DSEM) module. Experimental results show that structured epistemic
memory consistently improves deception assessment, while
multimodal social-cue
reasoning remains a challenging and important direction for future research.
Read the full abstract ▾
All authors Caixin Kang, Yifei Huang, Liangyang Ouyang, Mingfang Zhang, Ruicong Liu, Yoichi Sato
Computation and Language · Cross-listed from cs.CR
PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization
Huseein Jawad and Nicolas Brunel
System
prompts are critical for guiding the behavior of Large
Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a
prime target for extraction attacks.
Adversarial queries can successfully elicit these hidden instructions, posing significant security and
privacy risks. Existing defense mechanisms frequently rely on heuristics, incur substantial computational overhead, or are inapplicable to models accessed via black-box APIs. This paper introduces a novel framework for hardening system
prompts through shield appending, a lightweight approach that adds a protective textual layer to the original
prompt. Our core contribution is the formalization of
prompt hardening as a utility-constrained optimization problem. We leverage an LLM-as-optimizer to search the space of possible SHIELDs, seeking to minimize a leakage metric derived from a suite of
adversarial attacks, while simultaneously preserving task utility above a specified threshold, measured by semantic fidelity to baseline outputs. This black-box, optimization-driven methodology is lightweight and practical, requiring only API access to the target and optimizer LLMs. We demonstrate empirically that our optimized SHIELDs significantly reduce
prompt leakage against a comprehensive set of extraction attacks, outperforming established baseline defenses without compromising the model's intended functionality. Our work presents a paradigm for developing robust, utility-aware defenses in the escalating landscape of LLM security. The code is made public on the following link: https://github.com/psm-defense/psm
Read the full abstract ▾
Computation and Language
SemanticCite: Citation Verification with AI-Powered Full-Text Analysis and Evidence-Based Reasoning
Sebastian Haan
Effective scientific communication depends on accurate citations that validate sources and guide readers to supporting evidence. Yet academic literature faces mounting challenges: semantic citation errors that misrepresent sources, AI-generated hallucinated references, and traditional citation formats that point to entire papers without indicating which sections substantiate specific claims. We introduce SemanticCite, an AI-powered system that verifies citation accuracy through full-text source analysis while providing rich contextual information via detailed
reasoning and relevant text snippets. Our approach combines multiple
retrieval methods with a four-class classification system (Supported, Partially Supported, Unsupported, Uncertain) that captures nuanced claim-source relationships and enables appropriate remedial actions for different error types. Our experiments show that fine-tuned lightweight
language models achieve performance comparable to large commercial systems with significantly lower computational requirements, making large-scale citation verification practically feasible. The system provides transparent, evidence-based explanations that support user understanding and trust. We contribute a comprehensive
dataset of over 1,000 citations with detailed
alignments, functional classifications, semantic annotations, and bibliometric metadata across eight disciplines, alongside fine-tuned models and the complete verification framework as open-source software. SemanticCite addresses critical challenges in research integrity through scalable citation verification, streamlined peer review, and quality control for AI-generated content, providing an open-source foundation for maintaining citation accuracy at scale.
Read the full abstract ▾
Computation and Language
TS-PEFT: Unveiling Token-Level Redundancy in Parameter-Efficient Fine-Tuning
Dabiao Ma, Ziming Dai, Zhimin Xin et al. (+3 more)
Current Parameter-Efficient Fine-Tuning (PEFT) methods typically operate under an implicit assumption: Once a target module is selected, every
token passing through it contributes equally to the downstream task and requires a parameter update. In this paper, we challenge this convention by revealing a pervasive
token-level redundancy in the fine-tuning of large models (LMs). We propose TS-PEFT, a theoretical framework utilizing proximal optimization that acts as a dynamic probe to identify
token-level redundancy during the fine-tuning process. Extensive experiments demonstrate that indiscriminately updating all
tokens is not only computationally superfluous but often introduces optimization noise. Surprisingly, by discarding 30%-70% of
token updates, TS-PEFT consistently matches or exceeds the performance of dense baselines such as LoRA, DoRA. Our in-depth analysis shows that the learned
token-level sparsity is a superior indicator of module importance compared to traditional weight criteria, providing a novel data-driven perspective on the intrinsic adaptation mechanism of LMs.
Read the full abstract ▾
All authors Dabiao Ma, Ziming Dai, Zhimin Xin, Shu Wang, Jian Yang, Haojun Fei
Computation and Language
ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
Qing Zhang, Bing Xu, Xudong Zhang et al. (+9 more)
The remarkable performance of Large
Language Models (LLMs) highly relies on crafted
prompts. However, manual
prompt engineering is a laborious process, creating a core bottleneck for practical application of LLMs. This phenomenon has led to the emergence of a new research area known as Automatic
Prompt Optimization (APO), which develops rapidly in recent years. Existing APO methods such as those based on evolutionary
algorithms or trial-and-error approaches realize an efficient and accurate
prompt optimization to some extent. However, those researches focus on a single model or
algorithm for the generation strategy and optimization process, which limits their performance when handling complex tasks. To address this, we propose a novel framework called Ensemble Learning based
Prompt Optimization (ELPO) to achieve more accurate and robust results. Motivated by the idea of ensemble learning, ELPO conducts voting mechanism and introduces shared generation strategies along with different search methods for searching superior
prompts. Moreover, ELPO creatively presents more efficient
algorithms for the
prompt generation and search process. Experimental results demonstrate that ELPO outperforms state-of-the-art
prompt optimization methods across different tasks, e.g., improving F1 score by 7.6 on ArSarcasm
dataset.
Read the full abstract ▾
All authors Qing Zhang, Bing Xu, Xudong Zhang, Yifan Shi, Yang Li, Chen Zhang, Yik Chung Wu, Ngai Wong, Yijie Chen, Hong Dai, Xiansen Chen, Mian Zhang
Computation and Language
Early science acceleration experiments with GPT-5
Sébastien Bubeck, Christian Coester, Ronen Eldan et al. (+11 more)
AI models like GPT-5 are an increasingly valuable tool for scientists, but many remain unaware of the capabilities of frontier AI. We present a collection of short case studies in which GPT-5 produced new, concrete steps in ongoing research across mathematics, physics, astronomy, computer science, biology, and materials science. In these examples, the authors highlight how AI accelerated their work, and where it fell short; where expert time was saved, and where human input was still key. We document the interactions of the human authors with GPT-5, as guiding examples of fruitful collaboration with AI. Of note, this paper includes four new results in mathematics (carefully verified by the human authors), underscoring how GPT-5 can help human mathematicians settle previously unsolved problems. These contributions are modest in scope but profound in implication, given the rate at which frontier AI is progressing.
Read the full abstract ▾
All authors Sébastien Bubeck, Christian Coester, Ronen Eldan, Timothy Gowers, Yin Tat Lee, Alexandru Lupsasca, Mehtaab Sawhney, Robert Scherrer, Mark Sellke, Brian K. Spears, Derya Unutmaz, Kevin Weil, Steven Yin, Nikita Zhivotovskiy
Computation and Language
Learning Tractable Distributions Of Language Model Continuations
Gwen Yidou-Weng, Ian Li, Anji Liu et al. (+4 more)
Controlled generation imposes sequence-level constraints (syntax, style, safety) that depend on future
tokens, making exact conditioning of an autoregressive LM intractable. Tractable surrogates such as HMMs can approximate continuation distributions and steer decoding, but standard surrogates are often weakly context-aware. We propose Learning to Look Ahead (LTLA), a hybrid method that uses base-LM
embeddings to condition a globally learned tractable surrogate: a neural head predicts only a prefix-dependent latent prior, while a shared HMM answers continuation queries exactly. LTLA is designed to avoid two common efficiency traps when adding neural context. First, it avoids vocabulary-sized prefix rescoring (V extra LM evaluations) by scoring all next-
token candidates via a single batched HMM forward update. Second, it avoids predicting a new HMM per prefix by learning one shared HMM and conditioning only the latent prior, which enables reuse of cached future-likelihood (backward) messages across decoding steps. Empirically, LTLA improves continuation likelihood over standard HMM surrogates, enables lookahead control for
vision–
language models by incorporating continuous context, achieves 100% syntactic constraint satisfaction, and improves detoxification while adding only a 14% decoding-time overhead.
Read the full abstract ▾
All authors Gwen Yidou-Weng, Ian Li, Anji Liu, Oliver Broadrick, Yuchen Cui, Guy Van den Broeck, Benjie Wang
Computation and Language
Liars' Bench: Evaluating Lie Detectors for Language Models
Kieron Kretschmar, Walter Laurito, Sharan Maiya et al. (+1 more)
Prior work has introduced techniques for detecting when large
language models (LLMs) lie, that is, generate statements they believe are false. However, these techniques are typically validated in narrow settings that do not capture the diverse lies LLMs can generate. We introduce LIARS' BENCH, a testbed consisting of 72,863 examples of lies and honest responses generated by four open-weight models across seven
datasets. Our settings capture qualitatively different types of lies and vary along two dimensions: the model's reason for lying and the object of belief targeted by the lie. Evaluating three black- and white-box lie detection techniques on LIARS' BENCH, we find that existing techniques systematically fail to identify certain types of lies, especially in settings where it's not possible to determine whether the model lied from the transcript alone. Overall, LIARS' BENCH reveals limitations in prior techniques and provides a practical testbed for guiding progress in lie detection.
Read the full abstract ▾
All authors Kieron Kretschmar, Walter Laurito, Sharan Maiya, Samuel Marks
Computation and Language · Cross-listed from cs.AI
SpellForger: Prompting Custom Spell Properties In-Game using BERT supervised-trained model
Emanuel C. Silva, Emily S. M. Salum, Gabriel M. Arantes et al. (+3 more)
Introduction: The application of Artificial Intelligence in games has evolved significantly, allowing for dynamic content generation. However, its use as a core gameplay co-creation tool remains underexplored. Objective: This paper proposes SpellForger, a game where players create custom spells by writing natural
language prompts, aiming to provide a unique experience of personalization and creativity. Methodology: The system uses a supervisedtrained BERT model to interpret player
prompts. This model maps textual descriptions to one of many spell prefabs and balances their parameters (damage, cost, effects) to ensure competitive integrity. The game is developed in the Unity Game Engine, and the AI backend is in Python. Expected Results: We expect to deliver a functional prototype that demonstrates the generation of spells in real time, applied to an engaging gameplay loop, where player creativity is central to the experience, validating the use of AI as a direct gameplay mechanic.
Read the full abstract ▾
All authors Emanuel C. Silva, Emily S. M. Salum, Gabriel M. Arantes, Matheus P. Pereira, Vinicius F. Oliveira, Alessandro L. Bicho
Computation and Language · Cross-listed from cs.IR
QueryGym: A Toolkit for Reproducible LLM-Based Query Reformulation
Amin Bigdeli, Radin Hamidi Rad, Mert Incesu et al. (+3 more)
We present QueryGym, a lightweight, extensible Python toolkit that supports large
language model (LLM)-based query reformulation. This is an important tool development since recent work on llm-based query reformulation has shown notable increase in
retrieval effectiveness. However, while different authors have sporadically shared the implementation of their methods, there is no unified toolkit that provides a consistent implementation of such methods, which hinders fair comparison, rapid experimentation, consistent benchmarking and reliable deployment. QueryGym addresses this gap by providing a unified framework for implementing, executing, and comparing llm-based reformulation methods. The toolkit offers: (1) a Python API for applying diverse LLM-based methods, (2) a
retrieval-agnostic interface supporting integration with backends such as Pyserini and PyTerrier, (3) a centralized
prompt management system with versioning and metadata tracking, (4) built-in support for
benchmarks like BEIR and MS MARCO, and (5) a completely open-source extensible implementation available to all researchers. QueryGym is publicly available at https://github.com/radinhamidi/QueryGym.
Read the full abstract ▾
All authors Amin Bigdeli, Radin Hamidi Rad, Mert Incesu, Negar Arabzadeh, Charles L. A. Clarke, Ebrahim Bagheri
Computation and Language · Cross-listed from cs.AI
CARE-RAG - Clinical Assessment and Reasoning in RAG
Deepthi Potluri, Aby Mammen Mathew, Jeffrey B DeWitt et al. (+4 more)
Access to the right evidence does not guarantee that large
language models (LLMs) will reason with it correctly. This gap between
retrieval and
reasoning is especially concerning in clinical settings, where outputs must align with structured protocols. We study this gap using Written Exposure Therapy (WET) guidelines as a testbed. In evaluating model responses to curated clinician-vetted questions, we find that errors persist even when authoritative passages are provided. To address this, we propose an evaluation framework that measures accuracy, consistency, and fidelity of
reasoning. Our results highlight both the potential and the risks:
retrieval-augmented generation (RAG) can constrain outputs, but safe deployment requires assessing
reasoning as rigorously as
retrieval.
Read the full abstract ▾
All authors Deepthi Potluri, Aby Mammen Mathew, Jeffrey B DeWitt, Alexander L. Rasgon, Yide Hao, Junyuan Hong, Ying Ding
Computation and Language
TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialogues
Sarik Ghazarian, Abhinav Gullapalli, Swair Shah et al. (+4 more)
In real-world task-oriented dialogue (TOD) settings,
agents are required to strictly adhere to complex instructions while conducting multi-turn conversations with customers. These instructions are typically presented in natural
language format and include general guidelines and step-by-step procedures with complex constraints. Existing TOD
benchmarks often oversimplify the complex nature of these instructions by reducing them to simple schemas composed of intents, slots, and API call configurations. To address this gap and systematically
benchmark LLMs' instruction-following capabilities, we propose TOD-ProcBench, a challenging
benchmark featuring complex process instructions with intricate, fine-grained constraints that evaluates various LLMs' abilities to understand and follow instructions in multi-turn TODs. Our
benchmark dataset comprises instruction documents derived from the high-quality ABCD
dataset with corresponding conversations under human quality control. We formulate fine-grained constraints and action procedures as multi-level condition-action instruction statements. We design three tasks to comprehensively
benchmark LLMs' complex instruction-following capabilities in multi-turn TODs. Task 1 evaluates how LLMs retrieve the most relevant statement from a complex instruction and predict the corresponding next action. In Task 2, we synthesize instruction-violating responses by injecting inconsistencies and manipulating the original instructions, and then we analyze how effectively LLMs can identify instruction-violating responses. Task 3 investigates LLMs' abilities in conditional generation of instruction-following responses based on the original complex instructions. Additionally, we conduct studies on the impact of
multilingual settings and different instruction text formats on compliance performance. We release our
benchmark under the Llama 3.3 Community License Agreement.
Read the full abstract ▾
All authors Sarik Ghazarian, Abhinav Gullapalli, Swair Shah, Anurag Beniwal, Nanyun Peng, Narayanan Sadagopan, Zhou Yu
Computation and Language · Cross-listed from cs.AI
JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation
Zhenyu Bi, Gaurav Srivastava, Yang Li et al. (+4 more)
While small
language models (SLMs) have shown promise on various
reasoning tasks, their ability to judge the correctness of answers remains unclear compared to large
language models (LLMs). Prior work on LLM-as-a-judge frameworks typically relies on comparing candidate answers against ground-truth labels or other candidate answers using predefined metrics like entailment. However, this approach is inherently indirect and difficult to fully automate, offering limited support for fine-grained and scalable evaluation of
reasoning outputs. In this work, we propose JudgeBoard, a novel evaluation pipeline that directly queries models to assess the correctness of candidate answers without requiring extra answer comparisons. We focus on two core
reasoning domains: mathematical
reasoning and science/commonsense
reasoning, and construct task-specific evaluation leaderboards using both accuracy-based ranking and an Elo-based rating system across five
benchmark datasets, enabling consistent model comparison as judges rather than comparators. To improve judgment performance in lightweight models, we propose MAJ (Multi-
Agent Judging), a novel multi-
agent evaluation framework that leverages multiple interacting SLMs with distinct
reasoning profiles to approximate LLM-level judgment accuracy through collaborative deliberation. Experimental results reveal a significant performance gap between SLMs and LLMs in isolated judging tasks. However, our MAJ framework substantially improves the reliability and consistency of SLMs. On the MATH
dataset, MAJ using smaller-sized models as backbones performs comparatively well or even better than their larger-sized counterparts. Our findings highlight that multi-
agent SLM systems can potentially match or exceed LLM performance in judgment tasks, with implications for scalable and efficient assessment.
Read the full abstract ▾
All authors Zhenyu Bi, Gaurav Srivastava, Yang Li, Meng Lu, Swastik Roy, Morteza Ziyadi, Xuan Wang