Computation and Language
Model-Aware Tokenizer Transfer
Mykola Haltiuk and Aleksander Smywinski-Pohl
Large
Language Models (LLMs) are trained to support an increasing number of
languages, yet their predefined tokenizers remain a bottleneck for adapting models to lower-resource or distinct-script
languages. Existing tokenizer transfer methods typically rely on semantic heuristics to initialize new
embeddings, ignoring higher-layer model dynamics and limiting transfer quality. We propose Model-Aware Tokenizer Transfer (MATT), a method that incorporates model internals into the tokenizer transfer process. MATT introduces an
Attention Influence Modeling (AIM) objective that distills inter-
token communication patterns from a source model into a target model with a new tokenizer, providing an efficient warm-up before standard
language modeling. Unlike approaches that focus solely on
embedding similarity, MATT leverages
attention behavior to guide
embedding initialization and adaptation. Experiments across diverse linguistic settings show that MATT recovers a large fraction of the original model's performance within a few GPU hours, outperforming heuristic baselines. These results demonstrate that incorporating model-level signals offers a practical and effective path toward robust tokenizer transfer in
multilingual LLMs.
Read the full abstract ▾
Computation and Language
Explaining and Mitigating Crosslingual Tokenizer Inequities
Catherine Arnett, Tyler A. Chang, Stella Biderman et al. (+1 more)
The number of
tokens it takes to encode parallel text in different
languages is known to vary. These disparities are called
token premiums. Having high
token premiums leads to less throughput during training and increases costs at inference. In this paper, we show that even after controlling for
dataset size, vocabulary size, and data content, monolingual tokenizers exhibit a wide range of
token premiums across
languages. To understand the cross-linguistic differences that cause these
token premiums, we train a suite of approximately 7,000 comparable monolingual tokenizers for 97
languages, manipulating tokenization
algorithm, vocabulary size, and
dataset size. We measure
token premiums and test for a relationship between factors such as data similarity (between tokenizer training and evaluation), vocabulary size, and pre-tokenization. We also investigate the role of
language-specific features such as writing system and word length. We find that similarity between training and test data does not impact
token premiums, but vocabulary size and pre-tokenization do. While simply increasing vocabulary size does not lead to reduced
token premium effects, we can determine an “optimal” vocabulary size for each
language to achieve significantly reduced
token premium effects. We also train superword tokenizers which allow merges over whitespaces, and we find that they both reduce
token premium effects and improve compression overall. Thus, intervening on the vocabulary size or the pre-tokenizer significantly reduces crosslingual
token premium effects.
Read the full abstract ▾
All authors Catherine Arnett, Tyler A. Chang, Stella Biderman, Benjamin K. Bergen
Computation and Language
Deep Literature Survey Automation with an Iterative Workflow
Hongbo Zhang, Han Cui, Yidong Wang et al. (+6 more)
Automatic literature survey generation has attracted increasing
attention, yet most existing systems follow a one-shot paradigm, where a large set of papers is retrieved at once and a static outline is generated before drafting. This design often leads to noisy
retrieval, fragmented structures, and context overload, ultimately limiting survey quality. Inspired by the iterative reading process of human researchers, we propose \ours, a framework based on recurrent outline generation, in which a planning
agent incrementally retrieves, reads, and updates the outline to ensure both exploration and coherence. To provide faithful paper-level grounding, we design paper cards that distill each paper into its contributions, methods, and findings, and introduce a review-and-refine loop with visualization enhancement to improve textual flow and integrate
multimodal elements such as figures and tables. Experiments on both established and emerging topics show that \ours\ substantially outperforms state-of-the-art baselines in content coverage, structural coherence, and citation quality, while producing more accessible and better-organized surveys. To provide a more reliable assessment of such improvements, we further introduce Survey-Arena, a pairwise
benchmark that complements absolute scoring and more clearly positions machine-generated surveys relative to human-written ones. The code is available at https://github.com/HancCui/IterSurvey\_Autosurveyv2.
Read the full abstract ▾
All authors Hongbo Zhang, Han Cui, Yidong Wang, Yijian Tian, Qi Guo, Cunxiang Wang, Jian Wu, Chiyu Song, Yue Zhang
Computation and Language
Understanding Network Behaviors through Natural Language Question-Answering
Mingzhe Xing, Chang Tian, Jianan Zhang et al. (+4 more)
Modern large-scale networks introduce significant complexity in understanding network behaviors, increasing the risk of misconfiguration. Prior work proposed to understand network behaviors by mining network configurations, typically relying on domain-specific
languages interfaced with formal models. While effective, they suffer from a steep learning curve and limited flexibility. In contrast, natural
language (NL) offers a more accessible and interpretable interface, motivating recent research on NL-guided network behavior understanding. Recent advances in large
language models (LLMs) further enhance this direction, leveraging their extensive prior knowledge of network concepts and strong
reasoning capabilities. However, three key challenges remain: 1) numerous router devices with lengthy configuration files challenge LLM's long-context understanding ability; 2) heterogeneity across devices and protocols impedes scalability; and 3) complex network
topologies and protocols demand advanced
reasoning abilities beyond the current capabilities of LLMs. To tackle the above challenges, we propose NetMind, a novel framework for querying networks using NL. Our approach introduces a tree-based configuration chunking strategy to preserve semantic coherence while enabling efficient partitioning. We then construct a unified fact graph as an intermediate representation to normalize vendor-specific configurations. Finally, we design a hybrid imperative-declarative
language to reduce the
reasoning burden on LLMs and enhance precision. We contribute a
benchmark consisting of NL question-answer pairs paired with network configurations. Experiments demonstrate that NetMind achieves accurate and scalable network behavior understanding, outperforming existing baselines.
Read the full abstract ▾
All authors Mingzhe Xing, Chang Tian, Jianan Zhang, Lichen Pan, Peipei Liu, Zhaoteng Yan, Yinliang Yue
Computation and Language
Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation
Dhrupad Bhardwaj, Julia Kempe and Tim G. J. Rudner
To deploy large
language models (LLMs) in high-stakes application domains that require substantively accurate responses to open-ended
prompts, we need reliable, computationally inexpensive methods that assess the trustworthiness of long-form responses generated by LLMs. However, existing approaches often rely on claim-by-claim fact-checking, which is computationally expensive and brittle in long-form responses to open-ended
prompts. In this work, we introduce semantic isotropy – the degree of uniformity across normalized text
embeddings on the unit sphere – and use it to assess the trustworthiness of long-form responses generated by LLMs. To do so, we generate several long-form responses, embed them, and estimate the level of semantic isotropy of these responses as the angular dispersion of the
embeddings on the unit sphere. We find that higher semantic isotropy – that is, greater
embedding dispersion – reliably signals lower factual consistency across samples. Our approach requires no labeled data, no fine-tuning, and no hyperparameter selection, and can be used with open- or closed-weight
embedding models. Across multiple domains, our method consistently outperforms existing aggregate trust signals in predicting nonfactuality using only a handful of samples, offering a practical, low-cost first-pass signal that complements claim-level verification in real-world LLM workflows.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
Jonathan Bragg, Mike D'Arcy, Nishant Balepur et al. (+36 more)
AI
agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such
agents, ranging from general-purpose "deep research" systems to specialized science-specific
agents, such as AI Scientist and AIGS. Rigorous evaluation of these
agents is critical for progress. Yet existing
benchmarks fall short on several fronts: they often (1) lack reproducible
agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick
agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline
agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking
agents. Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta
agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta
agents and numerous baselines. Our extensive evaluation of 57
agents across 22
agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.
Read the full abstract ▾
All authors Jonathan Bragg, Mike D'Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi, Sergey Feldman, Dany Haddad, Jena D. Hwang, Peter Jansen, Varsha Kishore, Bodhisattwa Prasad Majumder, Aakanksha Naik, Sigal Rahamimov, Kyle Richardson, Amanpreet Singh, Harshit Surana, Aryeh Tiktinsky, Rosni Vasu, Guy Wiener, Chloe Anastasiades, Stefan Candra, Jason Dunkelberger, Dan Emery, Rob Evans, Malachi Hamada, Regan Huff, Rodney Kinney, Matt Latzke, Jaron Lochner, Ruben Lozano-Aguilera, Cecile Nguyen, Smita Rao, Amber Tanaka, Brooke Vlahos, Peter Clark, Doug Downey, Yoav Goldberg, Ashish Sabharwal, Daniel S. Weld
Computation and Language · Cross-listed from cs.LG
Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
Faisal Hamman, Pasan Dissanayake, Yanjun Fu et al. (+1 more)
Knowledge
distillation is a promising approach to transfer capabilities from complex teacher models to smaller, resource-efficient student models that can be deployed easily, particularly in task-aware scenarios. However, existing methods of task-aware
distillation typically require substantial quantities of data which may be unavailable or expensive to obtain in many practical scenarios. In this paper, we address this challenge by introducing a novel strategy called Counterfactual-explanation-infused
Distillation CoD for few-shot task-aware knowledge
distillation by systematically infusing counterfactual explanations. Counterfactual explanations (CFEs) refer to inputs that can flip the output prediction of the teacher model with minimum perturbation. Our strategy CoD leverages these CFEs to precisely map the teacher's decision boundary with significantly fewer samples. We provide theoretical guarantees for motivating the role of CFEs in
distillation, from both statistical and
geometric perspectives. We mathematically show that CFEs can improve parameter estimation by providing more informative examples near the teacher's decision boundary. We also derive
geometric insights on how CFEs effectively act as knowledge probes, helping the students mimic the teacher's decision boundaries more effectively than standard data. We perform experiments across various
datasets and LLMs to show that CoD outperforms standard
distillation approaches in few-shot regimes (as low as 8-512 samples). Notably, CoD only uses half of the original samples used by the baselines, paired with their corresponding CFEs and still improves performance.
Read the full abstract ▾
All authors Faisal Hamman, Pasan Dissanayake, Yanjun Fu, Sanghamitra Dutta
Computation and Language
The Universal Landscape of Human Reasoning
Qiguang Chen, Jinhao Liu, Libo Qin et al. (+14 more)
Understanding how information is dynamically accumulated and transformed in human
reasoning has long challenged cognitive psychology, philosophy, and artificial intelligence. Existing accounts, from classical logic to probabilistic models, illuminate aspects of output or individual modelling, but do not offer a unified, quantitative description of general human
reasoning dynamics. To solve this, we introduce Information Flow Tracking (IF-Track), that uses large
language models (LLMs) as probabilistic encoder to quantify information
entropy and gain at each
reasoning step. Through fine-grained analyses across diverse tasks, our method is the first successfully models the universal landscape of human
reasoning behaviors within a single metric space. We show that IF-Track captures essential
reasoning features, identifies systematic error patterns, and characterizes individual differences. Applied to discussion of advanced psychological theory, we first reconcile single- versus dual-process theories in IF-Track and discover the
alignment of artificial and human cognition and how LLMs reshaping human
reasoning process. This approach establishes a quantitative bridge between theory and measurement, offering mechanistic insights into the architecture of
reasoning.
Read the full abstract ▾
All authors Qiguang Chen, Jinhao Liu, Libo Qin, Yimeng Zhang, Yihao Liang, Shangxu Ren, Chengyu Luan, Dengyun Peng, Hanjing Li, Jiannan Guan, Zheng Yan, Jiaqi Wang, Mengkang Hu, Yantao Du, Zhi Chen, Xie Chen, Wanxiang Che
Computation and Language · Cross-listed from cs.AI
DeepAgent: A General Reasoning Agent with Scalable Toolsets
Xiaoxi Li, Wenxiang Jiao, Jiarui Jin et al. (+8 more)
Large
reasoning models have demonstrated strong problem-solving abilities, yet real-world tasks often require external tools and long-horizon interactions. Existing
agent frameworks typically follow predefined workflows, which limit autonomous and global task completion. In this paper, we introduce DeepAgent, an end-to-end deep
reasoning agent that performs autonomous thinking, tool discovery, and action execution within a single, coherent
reasoning process. To manage long-horizon interactions, we introduce an autonomous
memory folding mechanism that compresses past interactions into structured episodic, working, and tool
memories, reducing error accumulation while preserving critical information. To teach general-purpose tool use efficiently and stably, we develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation
tokens. Extensive experiments on eight
benchmarks, including general tool-use tasks (ToolBench, API-Bank, TMDB, Spotify, ToolHop) and downstream applications (ALFWorld, WebShop,
GAIA, HLE), demonstrate that DeepAgent consistently outperforms baselines across both labeled-tool and open-set tool
retrieval scenarios. The code and demo are available at https://github.com/RUC-NLPIR/DeepAgent.
Read the full abstract ▾
All authors Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Guanting Dong, Jiajie Jin, Yinuo Wang, Hao Wang, Yutao Zhu, Ji-Rong Wen, Yuan Lu, Zhicheng Dou
Computation and Language
RETuning: Upgrading Inference-Time Scaling for Stock Movement Prediction with Large Language Models
Xueyuan Lin, Cehao Yang, Ye Ma et al. (+7 more)
Recently, large
language models (LLMs) have demonstrated outstanding
reasoning capabilities on mathematical and coding tasks. However, their application to financial tasks-especially the most fundamental task of stock movement prediction-remains underexplored. We study a three-class classification problem (up, hold, down) and, by analyzing existing
reasoning responses, observe that: (1) LLMs follow analysts' opinions rather than exhibit a systematic, independent analytical logic (CoTs). (2) LLMs list summaries from different sources without weighing
adversarial evidence, yet such counterevidence is crucial for reliable prediction. It shows that the model does not make good use of its
reasoning ability to complete the task. To address this, we propose Reflective Evidence Tuning (RETuning), a cold-start method prior to reinforcement learning, to enhance prediction ability. While generating CoT, RETuning encourages dynamically constructing an analytical framework from diverse information sources, organizing and scoring evidence for price up or down based on that framework-rather than on contextual viewpoints-and finally reflecting to derive the prediction. This approach maximally aligns the model with its learned analytical framework, ensuring independent logical
reasoning and reducing undue influence from context. We also build a large-scale
dataset spanning all of 2024 for 5,123 A-share stocks, with long contexts (32K
tokens) and over 200K samples. In addition to price and news, it incorporates analysts' opinions, quantitative reports, fundamental data, macroeconomic indicators, and similar stocks. Experiments show that RETuning successfully unlocks the model's
reasoning ability in the financial domain. Inference-time scaling still works even after 6 months or on out-of-distribution stocks, since the models gain valuable insights about stock movement prediction.
Read the full abstract ▾
All authors Xueyuan Lin, Cehao Yang, Ye Ma, Ming Li, Rongjunchen Zhang, Yang Ni, Xiaojun Wu, Chengjin Xu, Jian Guo, Hui Xiong
Computation and Language · Cross-listed from cs.IR
Doc-Researcher: A Unified System for Multimodal Document Parsing and Deep Research
Kuicai Dong, Shurui Huang, Fangda Ye et al. (+9 more)
Deep Research systems have revolutionized how LLMs solve complex questions through iterative
reasoning and evidence gathering. However, current systems remain fundamentally constrained to textual web data, overlooking the vast knowledge embedded in
multimodal documents Processing such documents demands sophisticated parsing to preserve visual semantics (figures, tables, charts, and equations), intelligent chunking to maintain structural coherence, and adaptive
retrieval across modalities, which are capabilities absent in existing systems. In response, we present Doc-Researcher, a unified system that bridges this gap through three integrated components: (i) deep
multimodal parsing that preserves layout structure and visual semantics while creating multi-granular representations from chunk to document level, (ii) systematic
retrieval architecture supporting text-only,
vision-only, and hybrid paradigms with dynamic granularity selection, and (iii) iterative multi-
agent workflows that decompose complex queries, progressively accumulate evidence, and synthesize comprehensive answers across documents and modalities. To enable rigorous evaluation, we introduce M4DocBench, the first
benchmark for Multi-modal, Multi-hop, Multi-document, and Multi-turn deep research. Featuring 158 expert-annotated questions with complete evidence chains across 304 documents, M4DocBench tests capabilities that existing
benchmarks cannot assess. Experiments demonstrate that Doc-Researcher achieves 50.6% accuracy, 3.4xbetter than state-of-the-art baselines, validating that effective document research requires not just better
retrieval, but fundamentally deep parsing that preserve
multimodal integrity and support iterative research. Our work establishes a new paradigm for conducting deep research on
multimodal document collections.
Read the full abstract ▾
All authors Kuicai Dong, Shurui Huang, Fangda Ye, Wei Han, Zhi Zhang, Dexun Li, Wenjun Li, Qu Yang, Gang Wang, Yichao Wang, Chen Zhang, Yong Liu
Computation and Language
Automated Quality Control for Language Documentation: Detecting Phonotactic Inconsistencies in a Kokborok Wordlist
Kellen Parker van Dam and Abishek Stephen
Lexical data collection in
language documentation often contains transcription errors and undocumented borrowings that can mislead linguistic analysis. We present unsupervised anomaly detection methods to identify phonotactic inconsistencies in wordlists, applying them to a
multilingual dataset of Kokborok varieties with Bangla. Using character-level and syllable-level phonotactic features, our
algorithms identify potential transcription errors and borrowings. While precision and recall remain modest due to the subtle nature of these anomalies, syllable-aware features significantly outperform character-level baselines. The high-recall approach provides fieldworkers with a systematic method to flag entries requiring verification, supporting data quality improvement in low-resourced
language documentation.
Read the full abstract ▾
Computation and Language
From Polyester Girlfriends to Blind Mice: Creating the First Pragmatics Understanding Benchmarks for Slovene
Mojca Brglez and Špela Vintar
Large
language models are demonstrating increasing capabilities, excelling at
benchmarks once considered very difficult. As their capabilities grow, there is a need for more challenging evaluations that go beyond surface-level linguistic competence. Namely,
language competence involves not only syntax and semantics but also pragmatics, i.e., understanding situational meaning as shaped by context as well as linguistic and cultural norms. To contribute to this line of research, we introduce SloPragEval and SloPragMega, the first pragmatics understanding
benchmarks for Slovene that contain altogether 405 multiple-choice questions. We discuss the difficulties of translation, describe the campaign to establish a human baseline, and report pilot evaluations with LLMs. Our results indicate that current models have greatly improved in understanding nuanced
language but may still fail to infer implied speaker meaning in non-literal utterances, especially those that are culture-specific. We also observe a significant gap between proprietary and open-source models. Finally, we argue that
benchmarks targeting nuanced
language understanding and knowledge of the target culture must be designed with care, preferably constructed from native data, and validated with human responses.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.MA
ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystem
Fangwen Wu, Zheng Wu, Jihong Wang et al. (+11 more)
With the rapid development of (
multimodal) large
language model-based
agents, the landscape of agentic service management has evolved from single-
agent systems to multi-
agent systems, and now to massive-
agent ecosystems. Current massive-
agent ecosystems face growing challenges, including impersonal service experiences, a lack of standardization, and untrustworthy behavior. To address these issues, we propose ColorEcosystem, a novel blueprint designed to enable personalized, standardized, and trustworthy agentic service at scale. Concretely, ColorEcosystem consists of three key components:
agent carrier,
agent store, and
agent audit. The
agent carrier provides personalized service experiences by utilizing user-specific data and creating a digital twin, while the
agent store serves as a centralized, standardized platform for managing diverse agentic services. The
agent audit, based on the supervision of developer and user activities, ensures the integrity and credibility of both service providers and users. Through the analysis of challenges, transitional forms, and practical considerations, the ColorEcosystem is poised to power personalized, standardized, and trustworthy agentic service across massive-
agent ecosystems. Meanwhile, we have also implemented part of ColorEcosystem's functionality, and the relevant code is open-sourced at https://github.com/opas-lab/color-ecosystem.
Read the full abstract ▾
All authors Fangwen Wu, Zheng Wu, Jihong Wang, Yunku Chen, Ruiguang Pei, Heyuan Huang, Xin Liao, Xingyu Lou, Huarong Deng, Zhihui Fu, Weiwen Liu, Zhuosheng Zhang, Weinan Zhang, Jun Wang
Computation and Language
Are the LLMs Capable of Maintaining at Least the Language Genus?
Sandra Mitrović, David Kletz, Ljiljana Dolamic et al. (+1 more)
Large
Language Models (LLMs) display notable variation in
multilingual behavior, yet the role of genealogical
language structure in shaping this variation remains underexplored. In this paper, we investigate whether LLMs exhibit sensitivity to linguistic genera by extending prior analyses on the MultiQ
dataset. We first check if models prefer to switch to genealogically related
languages when
prompt language fidelity is not maintained. Next, we investigate whether knowledge consistency is better preserved within than across genera. We show that genus-level effects are present but strongly conditioned by training resource availability. We further observe distinct
multilingual strategies across LLMs families. Our findings suggest that LLMs encode aspects of genus-level structure, but training data imbalances remain the primary factor shaping their
multilingual performance.
Read the full abstract ▾
All authors Sandra Mitrović, David Kletz, Ljiljana Dolamic, Fabio Rinaldi
Computation and Language
Document Understanding, Measurement, and Manipulation Using Category Theory
Jared Claypoole, Yunye Gong, Noson S. Yanofsky et al. (+1 more)
We apply category theory to extract
multimodal document structure which leads us to develop information theoretic measures, content summarization and extension, and self-supervised improvement of large pretrained models. We first develop a mathematical representation of a document as a category of question-answer pairs. Second, we develop an orthogonalization procedure to divide the information contained in one or more documents into non-overlapping pieces. The structures extracted in the first and second steps lead us to develop methods to measure and enumerate the information contained in a document. We also build on those steps to develop new summarization techniques, as well as to develop a solution to a new problem viz. exegesis resulting in an extension of the original document. Our question-answer pair methodology enables a novel rate distortion analysis of summarization techniques. We implement our techniques using large pretrained models, and we propose a
multimodal extension of our overall mathematical framework. Finally, we develop a novel self-supervised method using RLVR to improve large pretrained models using consistency constraints such as composability and closure under certain operations that stem naturally from our category theoretic framework.
Read the full abstract ▾
All authors Jared Claypoole, Yunye Gong, Noson S. Yanofsky, Ajay Divakaran
Computation and Language
InterpDetect: Interpretable Signals for Detecting Hallucinations in Retrieval-Augmented Generation
Likun Tan, Kuan-Wei Huang, Joy Shi et al. (+1 more)
Retrieval-Augmented Generation (RAG) integrates external knowledge to mitigate
hallucinations, yet models often generate outputs inconsistent with retrieved content. Accurate
hallucination detection requires disentangling the contributions of external context and parametric knowledge, which prior methods typically conflate. We investigate the mechanisms underlying RAG
hallucinations and find they arise when later-layer FFN modules disproportionately inject parametric knowledge into the residual stream. To address this, we explore a mechanistic detection approach based on external context scores and parametric knowledge scores. Using Qwen3-0.6b, we compute these scores across layers and
attention heads and train regression-based classifiers to predict
hallucinations. Our method is evaluated against state-of-the-art LLMs (GPT-5, GPT-4.1) and detection baselines (RAGAS, TruLens, RefChecker). Furthermore, classifiers trained on Qwen3-0.6b signals generalize to GPT-4.1-mini responses, demonstrating the potential of proxy-model evaluation. Our results highlight mechanistic signals as efficient, generalizable predictors for
hallucination detection in RAG systems.
Read the full abstract ▾
All authors Likun Tan, Kuan-Wei Huang, Joy Shi, Kevin Wu
Computation and Language
Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models
Omer Moussa and Mariya Toneva
Pretrained
language models are remarkably effective in aligning with human
brain responses elicited by natural
language stimuli, positioning them as promising model organisms for studying
language processing in the
brain. However, existing approaches for both estimating and improving this
brain alignment are participant-dependent and highly affected by the amount of data available per participant, hindering both generalization to new participants and population-level analyses. In this work, we address these limitations by introducing a scalable, generalizable
brain-tuning method, in which we fine-tune pretrained
speech language models to jointly predict fMRI responses from multiple participants. We demonstrate that the resulting
brain-tuned models exhibit strong individual
brain alignment while generalizing across participants. Specifically, our method leads to 1) a 5-fold decrease in the amount of fMRI data needed to predict
brain data from new participants, 2) up to a 50% increase in the overall
brain alignment, and 3) strong generalization to new unseen
datasets. Furthermore, this multi-participant
brain-tuning additionally improves downstream performance on semantic tasks, suggesting that training using
brain data from multiple participants leads to more generalizable semantic representations. Taken together, these findings demonstrate a bidirectional benefit between neuroscience and AI, helping bridge the gap between the two fields. We make our code and models publicly available at https://github.com/bridge-ai-neuro/multi-
brain-tuning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Lorenzo Basile, Valentino Maiorca, Diego Doimo et al. (+2 more)
Language and
vision-
language models have shown impressive performance across a wide range of tasks, but their internal mechanisms remain only partly understood. In this work, we study how individual
attention heads in text-generative models specialize in specific semantic or visual attributes. Building on an established interpretability method, we reinterpret the practice of probing intermediate activations with the final decoding layer through the lens of signal processing. This lets us analyze multiple samples in a principled way and rank
attention heads based on their relevance to target concepts. Our results show consistent patterns of specialization at the head level across both unimodal and
multimodal transformers. Remarkably, we find that editing as few as 1% of the heads, selected using our method, can reliably suppress or enhance targeted concepts in the model output. We validate our approach on
language tasks such as question answering and toxicity mitigation, as well as
vision-
language tasks including image classification and captioning. Our findings highlight an interpretable and controllable structure within
attention layers, offering simple tools for understanding and editing large-scale generative models.
Read the full abstract ▾
All authors Lorenzo Basile, Valentino Maiorca, Diego Doimo, Francesco Locatello, Alberto Cazzaniga
Computation and Language · Cross-listed from cs.SE
Wisdom and Delusion of LLM Ensembles for Code Generation and Repair
Fernando Vallecillos-Ruiz, Max Hort and Leon Moonen
Today's pursuit of a single Large
Language Model (LMM) for all software engineering tasks is resource-intensive and overlooks the potential benefits of complementarity, where different models contribute unique strengths. However, the degree to which coding LLMs complement each other and the best strategy for maximizing an ensemble's potential are unclear, leaving practitioners without a clear path to move beyond single-model systems. To address this gap, we empirically compare ten individual LLMs from five families, and three ensembles of these LLMs across three software engineering
benchmarks covering code generation and program repair. We assess the complementarity between models and the performance gap between the best individual model and the ensembles. Next, we evaluate various selection heuristics to identify correct solutions from an ensemble's candidate pool. We find that the theoretical upperbound for an ensemble's performance can be 83% above the best single model. Our results show that consensus-based strategies for selecting solutions fall into a "popularity trap," amplifying common but incorrect outputs. In contrast, a diversity-based strategy realizes up to 95% of this theoretical potential, and proves effective even in small two-model ensembles, enabling a cost-efficient way to enhance performance by leveraging multiple LLMs.
Read the full abstract ▾
Computation and Language
MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization
Chenglong Wang, Yang Gan, Hang Zhou et al. (+10 more)
Recent advances in
diffusion language models (DLMs) have presented a promising alternative to traditional autoregressive large
language models (LLMs). However, DLMs still lag behind LLMs in
reasoning performance, especially as the number of denoising steps decreases. Our analysis reveals that this shortcoming arises primarily from the independent generation of masked
tokens across denoising steps, which fails to capture the
token correlation. In this paper, we define two types of
token correlation: intra-sequence correlation and inter-sequence correlation, and demonstrate that enhancing these correlations improves
reasoning performance. To this end, we propose a Multi-Reward Optimization (MRO) approach, which encourages DLMs to consider the
token correlation during the denoising process. More specifically, our MRO approach leverages test-time scaling, reject sampling, and reinforcement learning to directly optimize the
token correlation with multiple elaborate rewards. Additionally, we introduce group step and importance sampling strategies to mitigate reward variance and enhance sampling efficiency. Through extensive experiments, we demonstrate that MRO not only improves
reasoning performance but also achieves significant sampling speedups while maintaining high performance on
reasoning benchmarks.
Read the full abstract ▾
All authors Chenglong Wang, Yang Gan, Hang Zhou, Chi Hu, Yongyu Mu, Kai Song, Murun Yang, Bei Li, Chunliang Zhang, Tongran Liu, Jingbo Zhu, Zhengtao Yu, Tong Xiao
Computation and Language · Cross-listed from cs.CR
SBASH: a Framework for Designing and Evaluating RAG vs. Prompt-Tuned LLM Honeypots
Adetayo Adebimpe, Helmut Neukirchen and Thomas Welsh
Honeypots are decoy systems used for gathering valuable threat intelligence or diverting attackers away from production systems. Maximising attacker engagement is essential to their utility. However research has highlighted that context-awareness, such as the ability to respond to new attack types, systems and attacker
agents, is necessary to increase engagement. Large
Language Models (LLMs) have been shown as one approach to increase context awareness but suffer from several challenges including accuracy and timeliness of response time, high operational costs and data-protection issues due to cloud deployment. We propose the System-Based
Attention Shell Honeypot (SBASH) framework which manages data-protection issues through the use of lightweight local LLMs. We investigate the use of
Retrieval Augmented Generation (RAG) supported LLMs and non-RAG LLMs for Linux shell commands and evaluate them using several different metrics such as response time differences, realism from human testers, and similarity to a real system calculated with Levenshtein distance, SBert, and BertScore. We show that RAG improves accuracy for untuned models while models that have been tuned via a system
prompt that tells the LLM to respond like a Linux system achieve without RAG a similar accuracy as untuned with RAG, while having a slightly lower latency.
Read the full abstract ▾
Computation and Language
REMONI: An Autonomous System Integrating Wearables and Multimodal Large Language Models for Enhanced Remote Health Monitoring
Thanh Cong Ho, Farah Kharrat, Abderrazek Abid et al. (+1 more)
With the widespread adoption of wearable devices in our daily lives, the demand and appeal for remote patient monitoring have significantly increased. Most research in this field has concentrated on collecting sensor data, visualizing it, and analyzing it to detect anomalies in specific diseases such as diabetes, heart disease and depression. However, this domain has a notable gap in the aspect of human-machine interaction. This paper proposes REMONI, an autonomous REmote health MONItoring system that integrates
multimodal large
language models (MLLMs), the Internet of Things (IoT), and wearable devices. The system automatically and continuously collects vital signs, accelerometer data from a special wearable (such as a smartwatch), and visual data in patient
video clips collected from cameras. This data is processed by an anomaly detection module, which includes a fall detection model and
algorithms to identify and alert caregivers of the patient's emergency conditions. A distinctive feature of our proposed system is the natural
language processing component, developed with MLLMs capable of detecting and recognizing a patient's activity and
emotion while responding to healthcare worker's inquiries. Additionally,
prompt engineering is employed to integrate all patient information seamlessly. As a result, doctors and nurses can access real-time vital signs and the patient's current state and mood by interacting with an intelligent
agent through a user-friendly web application. Our experiments demonstrate that our system is implementable and scalable for real-life scenarios, potentially reducing the workload of medical professionals and healthcare costs. A full-fledged prototype illustrating the functionalities of the system has been developed and being tested to demonstrate the robustness of its various capabilities.
Read the full abstract ▾
All authors Thanh Cong Ho, Farah Kharrat, Abderrazek Abid, Fakhri Karray
Computation and Language · Cross-listed from cs.SE
Does Model Size Matter? A Comparison of Small and Large Language Models for Requirements Classification
Mohammad Amin Zadenoori, Vincenzo De Martino, Jacek Dabrowski et al. (+2 more)
[Context and motivation] Large
language models (LLMs) show notable results in natural
language processing (NLP) tasks for requirements engineering (RE). However, their use is compromised by high computational cost, data sharing risks, and dependence on external services. In contrast, small
language models (SLMs) offer a lightweight, locally deployable alternative. [Question/problem] It remains unclear how well SLMs perform compared to LLMs in RE tasks in terms of accuracy. [Results] Our preliminary study compares eight models, including three LLMs and five SLMs, on requirements classification tasks using the PROMISE, PROMISE Reclass, and SecReq
datasets. Our results show that although LLMs achieve an average F1 score of 2% higher than SLMs, this difference is not statistically significant. SLMs almost reach LLMs performance across all
datasets and even outperform them in recall on the PROMISE Reclass
dataset, despite being up to 300 times smaller. We also found that
dataset characteristics play a more significant role in performance than model size. [Contribution] Our study contributes with evidence that SLMs are a valid alternative to LLMs for requirements classification, offering advantages in
privacy, cost, and local deployability.
Read the full abstract ▾
All authors Mohammad Amin Zadenoori, Vincenzo De Martino, Jacek Dabrowski, Xavier Franch, Alessio Ferrari
Computation and Language
Redefining Retrieval Evaluation in the Era of LLMs
Giovanni Trappolini, Florin Cuconasu, Simone Filice et al. (+2 more)
Traditional Information
Retrieval (IR) metrics, such as nDCG, MAP, and MRR, assume that human users sequentially examine documents with diminishing
attention to lower ranks. This assumption breaks down in
Retrieval Augmented Generation (RAG) systems, where search results are consumed by Large
Language Models (LLMs), which, unlike humans, process all retrieved documents as a whole rather than sequentially. Additionally, traditional IR metrics do not account for related but irrelevant documents that actively degrade generation quality, rather than merely being ignored. Due to these two major misalignments, namely human vs. machine position discount and human relevance vs. machine utility, classical IR metrics do not accurately predict RAG performance. We introduce a utility-based annotation schema that quantifies both the positive contribution of relevant passages and the negative impact of distracting ones. Building on this foundation, we propose UDCG (Utility and Distraction-aware Cumulative Gain), a metric using an LLM-oriented positional discount to directly optimize the correlation with the end-to-end answer accuracy. Experiments on five
datasets and six LLMs demonstrate that UDCG improves correlation by up to 36% compared to traditional metrics. Our work provides a critical step toward aligning IR evaluation with LLM consumers and enables more reliable assessment of RAG components
Read the full abstract ▾
All authors Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, Fabrizio Silvestri
Computation and Language
Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
Abderrazek Abid, Thanh-Cong Ho and Fakhri Karray
As generative AI continues to evolve,
Vision Language Models (VLMs) have emerged as promising tools in various healthcare applications. One area that remains relatively underexplored is their use in human activity recognition (HAR) for remote health monitoring. VLMs offer notable strengths, including greater flexibility and the ability to overcome some of the constraints of traditional deep learning models. However, a key challenge in applying VLMs to HAR lies in the difficulty of evaluating their dynamic and often non-deterministic outputs. To address this gap, we introduce a descriptive caption data set and propose comprehensive evaluation methods to evaluate VLMs in HAR. Through comparative experiments with state-of-the-art deep learning models, our findings demonstrate that VLMs achieve comparable performance and, in some cases, even surpass conventional approaches in terms of accuracy. This work contributes a strong
benchmark and opens new possibilities for the integration of VLMs into intelligent healthcare systems.
Read the full abstract ▾
Computation and Language
HalleluBERT: Let Every Token That Has Meaning Bear Its Weight
Raphael Schmitt
Transformer-based models have advanced NLP, yet Hebrew still lacks a RoBERTa encoder that is trained at scale and released in both base and large variants. We present HalleluBERT, a RoBERTa-based encoder family trained from scratch on 49.1~GB of deduplicated Hebrew web text and Wikipedia using a Hebrew-specific byte-level BPE vocabulary. On native Hebrew
benchmarks for named entity recognition (BMC, NEMO) and sentiment classification (SMCD), HalleluBERT outperforms monolingual and
multilingual baselines, and yields the highest unweighted mean score across the three
benchmarks. We release model weights and tokenizer under the MIT license to support reproducible Hebrew NLP research.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.MA
HIKMA: Human-Inspired Knowledge by Machine Agents through a Multi-Agent Framework for Semi-Autonomous Scientific Conferences
Zain Ul Abideen Tariq, Mahmood Al-Zubaidi, Uzair Shah et al. (+2 more)
HIKMA Semi-Autonomous Conference is the first experiment in reimagining scholarly communication through an end-to-end integration of artificial intelligence into the academic publishing and presentation pipeline. This paper presents the design, implementation, and evaluation of the HIKMA framework, which includes AI
dataset curation, AI-based manuscript generation, AI-assisted peer review, AI-driven revision, AI conference presentation, and AI archival dissemination. By combining
language models, structured research workflows, and domain safeguards, HIKMA shows how AI can support - not replace traditional scholarly practices while maintaining intellectual property protection, transparency, and integrity. The conference functions as a testbed and proof of concept, providing insights into the opportunities and challenges of AI-enabled scholarship. It also examines questions about AI authorship, accountability, and the role of human-AI collaboration in research.
Read the full abstract ▾
All authors Zain Ul Abideen Tariq, Mahmood Al-Zubaidi, Uzair Shah, Marco Agus, Mowafa Househ
Computation and Language
SindBERT, the Sailor: Charting the Seas of Turkish NLP
Raphael Schmitt and Stefan Schweter
Transformer models have revolutionized NLP, yet many morphologically rich
languages remain underrepresented in large-scale pre-training efforts. With SindBERT, we set out to chart the seas of Turkish NLP, providing the first large-scale RoBERTa-based encoder for Turkish. Trained from scratch on 312~GB of Turkish text (mC4, OSCAR23, Wikipedia), SindBERT is released in both base and large configurations, representing the first large-scale encoder-only
language model available for Turkish. We evaluate SindBERT on part-of-
speech tagging, named entity recognition, offensive
language detection, and the TurBLiMP linguistic acceptability
benchmark. Our results show that SindBERT performs competitively with existing Turkish and
multilingual models, with the large variant achieving the best scores in two of four tasks but showing no consistent scaling advantage overall. This flat scaling trend, also observed for XLM-R and EuroBERT, suggests that current Turkish
benchmarks may already be saturated. At the same time, comparisons with smaller but more curated models such as BERTurk highlight that corpus quality and diversity can outweigh sheer data volume. Taken together, SindBERT contributes both as an openly released resource for Turkish NLP and as an empirical case study on the limits of scaling and the central role of corpus composition in morphologically rich
languages. The SindBERT models are released under the MIT license and made available in both fairseq and Huggingface formats.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models
Zihao Fu, Ryan Brown, Shun Shao et al. (+3 more)
Text-to-image
diffusion models, such as Stable
Diffusion, have demonstrated remarkable capabilities in generating high-quality and diverse images from natural
language prompts. However, recent studies reveal that these models often replicate and amplify societal biases, particularly along demographic attributes like gender and race. In this paper, we introduce FairImagen (https://github.com/fuzihaofzh/FairImagen), a post-hoc debiasing framework that operates on
prompt embeddings to mitigate such biases without retraining or modifying the underlying
diffusion model. Our method integrates Fair Principal Component Analysis to project CLIP-based input
embeddings into a subspace that minimizes group-specific information while preserving semantic content. We further enhance debiasing effectiveness through empirical noise injection and propose a unified cross-demographic projection method that enables simultaneous debiasing across multiple demographic attributes. Extensive experiments across gender, race, and intersectional settings demonstrate that FairImagen significantly improves
fairness with a moderate trade-off in image quality and
prompt fidelity. Our framework outperforms existing post-hoc methods and offers a simple, scalable, and model-agnostic solution for equitable text-to-image generation.
Read the full abstract ▾
All authors Zihao Fu, Ryan Brown, Shun Shao, Kai Rawal, Eoin Delaney, Chris Russell